Deep Generative Model for Periodic Graphs
Authors: Shiyu Wang
Xiaojie Guo
Liang Zhao
Venue: NeurIPS 2022 Accept (2022)
Keywords: periodic graph
deep generative model
disentangled learning
variational autoencoder

Abstract: Periodic graphs are graphs consisting of repetitive local structures, such as crystal nets and polygon mesh. Their generative modeling has great potential in real-world applications such as material design and graphics synthesis. Classical models either rely on domain-specific predefined generation principles (e.g., in crystal net design), or follow geometry-based prescribed rules. Recently, deep generative models have shown great promise in automatically generating general graphs. However, their advancement into periodic graphs has not been well explored due to several key challenges in 1) maintaining graph periodicity; 2) disentangling local and global patterns; and 3) efficiency in learning repetitive patterns. To address them, this paper proposes Periodical-Graph Disentangled Variational Auto-encoder (PGD-VAE), a new deep generative model for periodic graphs that can automatically learn, disentangle, and generate local and global graph patterns. Specifically, we develop a new periodic graph encoder consisting of global-pattern encoder and local-pattern encoder that ensures to disentangle the representation into global and local semantics. We then propose a new periodic graph decoder consisting of local structure decoder, neighborhood decoder, and global structure decoder, as well as the assembler of their outputs that guarantees periodicity. Moreover, we design a new model learning objective that helps ensure the invariance of local-semantic representations for the graphs with the same local structure. Comprehensive experimental evaluations have been conducted to demonstrate the effectiveness of the proposed method.

Deep Generative Model for Periodic Graphs
Shiyu Wang Emory University shiyu. wang@emory.en
Xiaojie Guo
IBM Thomas Watson Research Center
edu edu
xguo7@gmu. edu
Liang Zhao Emory University
1iang. www.lemory.cceoommoooooo edu
Abstract
Periodic graphs are graphs consisting of repetitive local structures, such as crystal nets and polygon mesh. Their generative modeling has great potential in worldworld applications such as material design and graphics synthesis. Classical models either rely on domain-specdt or follow geometry-basee prescribed rules. Recently, deep generative models have shown great promise in automatically generating general graphs. However, their advancement into periodic graphs has not been well explored due to several key challenges in 1) maintaining graph periodicity; 2) disentangling local and global patterns; and 3) efficiency in learning repetitive patterns. To address them, this paper proposes Periodical-Graas Graph Disentangled Variational Auto-encoder (PGD- VAE), new deep generative model for periodic graphs that automatically learn, disentangle, and generate local and global graph patterns. Specifically, we develop new periodic graph encoder consisting of global-pattern encoder and local-pattern encoder that ensures to disentangle the representation into global and local seman- ties. We then propose new periodic graph decoder consisting of local structure decoder, neighborhood decoder, and global structure decoder, as well as the assem- bler of their outputs that guarantees periodicity. Moreover, we design learning objective that helps ensure the invariance of local-semantic representations for the graphs with the same local structure. Comprehensive experimental evalua- tions have been conducted to demonstrate the effectiveness of the proposed method. The code and appendix is available in the Supplemental Material. The code of PGD-VAL VAE is available at wnttttt/////
predefined generation principles (e.g.. crystal net design),
model
com/ shi suri-yyure wang/ Baonngg/PDDD.... PGD-VAA VAE
1 Introduction
Graphs is ubiquitous data structure that describes objects and their relations. As a special type of graphs, periodic graphs graphs that consist of repetitive local structures. They naturally character- ize many real-world applications ranging from crys- tal nets containing repetitive unit cells to polygon mesh data containing repetitive grids (Figure TT (5T). Hence, understanding, modeling, and generating peri- odic graphs have great potential in real-world applica- tions such as material design and graphics synthesis. Different from general graphs, modeling generative process of periodic graphs requires ensure the peri-
Figure I: Examples of periodic graphs where
basic units highlighed: (a) structure of tridymite; (b) structure of graphene and (c)
triangular grids (3
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

odicity. Therefore, conventional graph models such as random graphs, swall-world models cannot be used effectively. Classical studies on periodic graph generative models have a long history. where conventional methods can be categorized into two types: domain peeifficbasedd and ceometry-hased Specifically, domain specific- specific-based models generate periodic graphs relying on properties handcrafted based on domain knowledge [63] 5 [58. In addition, ccometry-basel methods generate periodic graphs either according to predefined geometric rules to craft the targeted periodic graphs [22 56) or
simply gluing the initial fragment into periodic graphs [41].
As mentioned above, both domain specific- baseed and geometry graphs require predefined properties that are either domain-speciin or cemeericc-aseede enabling them to fit well towards the properties that the predefined principles are tailored for. but usually they cannot do well for the others. For example, a geometry geometry-bused generative model can generate graphs within the predefined geometric rules but cannot generalize to other geometric properties. However, in many domains, network properties and generation principles are largely unknown, such as those explaining the thermal probability [53] and solubilities [19] of materials. The lack of predefined principles leads to the incapability of existing methods to model periodic graphs. Hence, it is imperative to fill the gap between the lack of powerful models and the complex generative process.
soometry-based -based generative models for periodic
Expressive generative models that can directly learn the underlying data generative process auto- matically have been an extremely challenging task to accomplish, until very recent years when the advancement of deep generative models have exhibited great success in data such as images and graphs 49
18 17. Under this area. deep graph generative models aim at learning the distribution of general graph data representation extracted by graph neural networks [70]. However, despite the progress in graph neural networks for general graphs, they cannot be directly applied to handle periodic graphs due to the additional requirements of the generated graphs such as periodicity. requires significant efforts to advance the current deep graph generative models in order to handle periodic graphs due to the following unique challenges: (1) Difficulty in ensuring the periodicity of the generated graphs. Existing works for generic graph generative models typically are not able to identify or generate the "basic unit", namely the repetitive local subgraphs. The conventional learning objective such as reconstruction errors or adversarial errors can only measure the generated graph quality globally but cannot ensure the identity of local structures as well as how different local structures are connected. (2) How to jointly learn local and global topological patterns. In periodic graph, there two key patterns, namely the local structure which specifies the basic unit and the global topology which characterizes how local structures are connected to form the whole graph topology. Therefore, it is imperative for the generative models to be able to factorize these two different patterns for respective characterization and control. However, existing works for general graphs only learn representation for the graph instead of disentangling them into different representations for local and global patterns; and (3) Efficiency in learning large periodic graphs. Due to the ubiquitousness of repetitive structures, periodic graphs are full of redundant information which makes existing works for generic graphs highly inefficient to model them. It is imperative to establish models that can automatically deduplicate the modeled generative process of periodic graphs.
To address these challenges, we developed Feriodic-Graph. Disentangled Variational Auto-encoder (PGD-VAE), a deep generative model for periodic graphs that can automatically learn, disentangle and generate local and global graph patterns. Specially, we proposed novel encoder that contains a local-pattern encoder and a global-pattern encoder to disentangle the local and global semantics from the periodic graph representation. Besides, we designed periodic graph decoder consisting of a local structure decoder, a global structure decoder, neighborhood decoder and an assembler to assemble local and global patterns while preserving periodicity. To achieve the disentanglement of local and global semantics, we proposed new learning objective that can simultancously ensure the invariance of the local-semantic representations of graphs that have the same basic unit. Our contributions can be summarized as follows:
We propose a new problem of end-to-end periodic graph generation by deep generative models.
We develop PGD- VAF, a novel variational autoencoder that can decompose and assemble pe- riodic graphs with the learned patterns of basic units, their pairwise relations, and their local neighborhoods.
We propose a new learning objective to achieve the disentanglement of the local and global semantics by ensuring the invariance of the local-semantic representations of graphs that contain the same local structure.
2

---

We conducted extensive experiments to demonstrate the effectiveness and efficiency of the proposed methods with multiple real-world and synthetic datasets.
2 Related works
2.1 Graph Representation Learning and Graph Generation
Surging development of deep-learning techniques, recent work on graph representation learning + 9,34], especially graph neural networks (GNNs) [60 74 45 B1] is attracting considerable attention.
Significant success has been achieved on developing various GNNs models to handle domain- -ppcciff
graphs, such as traffic network [67 69 8), social network 772 20 [5], knowledge graph [ [2] [4]
Graph generation is one of the most important topics among the domain of graph representation learning. Due to the development of deep generative models, most
and materials structure
GNN- graph generation models borrow the framework VAE 37 38 35 59 or generative
adversarial nets (GANs) [25 68 73.. For example, Graph VAE learns the distribution of given graphs
and generates small graphs under the framework of VAE [52]. By contrast, MolGAN was proposed
to generate small molecular graphs by means of GANs
13
2.2 Disentangled Representation Learning
Disentangled representation learning has been attracting considerable attentions of the community, especially in the domain of computer vision 7 55 71 32]. The goal of disentangled representation learning is to separate underlying semantic factors accounting for the variation of the data in the learned representation. This disentangled representation has been shown to be resilient to the complex factors involved 16] and can enhance the generalizability and improve robustness against adversarial attack [27] 1 Intuitively, disentangled representation learning can achieve superior interpretability over regular graph representation learning tasks to better understand the graphs in various domains (28). This motivates the surge of a few VAE-based approaches that modify the VAE objective by adding, removing or adjusting the weight of individual terms for deep graph generation tasks [27 10 36). Moreover, Martinkus et al. [46] proposed GAN-based model global and local graph structure via graph spectrum. Disentangled representation learning is important in modeling periodical graphs where the process requires to distinguish the repeatable patterns from the others.
2.3 Periodical Graph Learning
Periodic graph is the graph that consists repetitive basic units in its structure, and can into the data structure in number of domains, such as crystal structures in materials science 4 14] 62 24
polygon mesh in computer graphics [1 and motion planning in robotics [23]. Several graph representation learning tasks have been conducted on this type of graph. For example, periodic graph-structured information in the airport network can be formatted into a periodic graph and a graph convolutional cewwork-based (GCN-based) was developed for flight delay prediction [6]. The multiple periodic spatial-temporal graphs of human mobility was employed to learn the community structure based collective embedding model [57]. In addition, graph generation model in periodic graphs can be categorized in to two scenarios: (1) domain specific- based models and (2) geometry- based models. For example, CDVAE borrows specific bonding preferences between different atoms and generates the periodic structure of materials in a diffusion process [63]. Moreover, [48) achieved geometric construction of periodic graphs by projecting two-dimensional hyperbolic tilings onto triply periodic minimal surfaces. They both need predefined rules to generate valid periodic graphs. Besides, the periodic graphs are usually large, the large time or space complexity of existing graph generative models limits the advance of periodic graph generation, motivating us to propose an end-to-end periodic graph generative model, PGD-VAE, that bears superior scalability on large periodic graphs.
3 Methodology
In this section, the problem formulation is first provided before moving on to derive the overall objective, following which the novel encoder of PGD-VAE that can respectively extract local and global information of the graph structure introduced. Afterwards, the decoder that generates local and global structure, and the algorithm of the assembler are described.
3

---

GNN
Node embedding clustering
400
H
Local
local-pattern encoder
Neighborhood
Ma
Global-pattern encoder
6
Global
decoder
4
Figure 2: Overview of PGD-VAE. The graph representation 27 and 2g that respectively capture local and global semantics are learned by local-patted and global-pattern encoders, respectively. Then A A()) and A(a) are generated by local decoder, neighborhood decoder and global decoder, respectively. Finally, the generated graph is obtained by assembling A A(n) and A Note that A sticks to the intrinsic node ordering and permutation naturally followed by the graphs in the training set.
3.1 Notations and Problem Definition
3.1.1 Periodic graphs
Define a periodic graph as = &V,E, E,A), A), where V is the set of nodes and & S V V is the set of edges of the graph G. The repeated local structures number of nodes is n). Suppose the periodic graph G consists of m basic units as well as their connections, this means the node
be defined basic units of size n (i.e.,
U u==1Vi, Vi,u is the union of nodes in all basic units, and = -UU==110j. U U (ij.e..ee. is the union of all edges within and across basic units of the graph. A E NxN corresponds to the adjacency matrix of the graph, in which Ai = 1 if Cij E and Aij otherwise. In addition,
define three matrices: (I) A Rexx is the adjacency matrix of the basic unit of the graph; (2) A(s) m the adjacency matrix denoting the pairwise relations among basic units; and (3) (mm) Ruxn is the incidence matrix regarding how the nodes in adjacent basic units are connected. Without loss of generality, all the adjacency matrices stick to specific and intrinsic node ordering (i.c., permutation) where A occupy the diagonal blocks A. The reason that say it is "intrinsic" is because such node ordering is usually naturally provided by the data (see the dataset description in Section 4.1 for details).
3.1.2 Represent A by A A(9), and (m)
As the adjacency matrix A of the periodic graph contains repeated local structures, we want to use an alternative "deduplicated" way to represent A efficiently. This will be achieved by concisely representing A with a function of ( A and (r); in the following and illustrated in the right part of Figure 2 First, define binary block matrix
((A()) ((s)) A A())), which is detailed
P E {0, 1 s.t. P- pT = J, where is the block diagonal matrix whose diagonal blocks are
all I's. Then, A the augmentation of (9) can be achieved as:
A PA PT.
Additionally, we define the matrix Q E Ramxn the concatenation of m identity matrices of size n. Then we can obtain A and (m)), the repetition of 4 and ((n), respectively, along both dimensions by m
A QF A Q".
(2)

---

Consequently, we can deterministically obtain A from AO,, A(a) and A via the function f:
A = I(A , ) (M @ QA (M @ Q4 Q²)")) @ PA p" + M @ QA Q" (M A()) + (M @ y A + M @ A
(3)
where M() and M(R) are two mask matrices. Specifically, the off-diagonal blocks of size n in M are O's while the diagonal blocks in M are I's. We also specify that the upper-triangule blocks of
size n in Min) are I's and other blocks in M(") are O's. In the following, we may exchangeably use G and to denote the whole graph since it is determined by its topology.
3.1.3 Deep generative models of periodic graphs
Learning generative models for a periodic graph aims to learn the conditional distribution p( Z)) Define (2): 2g) as the learned graph representation based on the structure of graphs. 2 is designed to capture the local semantic factors that characterize the topological structure of basic units, while 2g is designed to represent the global semantics on how basic units are topologically connected.
3.2 Learning Deep Periodic Graph Generative models
We first introduce the variational inference of the generative model and learning objective. Then we introduce our local and global encoders, as well as the new periodic graph decoder.
3.2.1 Periodic graph generative model inference
As discussed in the problem definition, we aim to learn the conditional distribution of G given Z = (2), 2g). So we aim maximize the marginal likelihood of the observed periodic graph G' in expectation over the distribution of the latent variables (z1. xy)= )- For a given periodic graph G, we define the prior distribution of the latent representation as p(21, zg) which intractable to infer. As a result, we leverage variational inference, in which the posterior distribution is approximated by (2(2), zy|G), Hence, another goal is to minimize the Kullback-Leible (KL) divergence between the true prior and the approximated posterior distributions. To encourage the disentanglement of 2 and Zg. we introduce constraint by matching the inferred posterior configurations of the latent factors (2(21, =g|G) to the prior distribution po =g)):
max EG-D [ 1 log Po poppo (G)a,, #g]]
0.6
s.t.Eo-D |K L(g 2g G) (po 2n]]] < I,,
(4)
where D refers to the observed graphs and I is a constant. The detailed derivation of the VAE objective can be seen at Appendix ?? Then, we can decompose the constraint term in Eq. given the assumption that 2 and 2g are independent, leading to:
(o(2), =n) = (=:) 4o(i) Cg = % (=(2)())o((g)G))
(5)
Based on the problem formulation, G can be represented as (A A(t), A()). Since 2 should capture and only capture the local patterns, thus A()) and (n) are dependent on 21 and conditionally
independent from each other given 21, namely A 1 A IzI. And 2g should capture and only capture the global patterns, we have
pe =G) zg) = (A (1) = (A (2)) (2)p0
(6)
Then, the objective function be written as:
max Ec-n (B Pd (z))o) (AA (2))o (A (1)))))))
s.L.Eg-D (.EE--D[LL(gd (x where 1, and Ty are the information bottleneck to control the information captured by the latent vector ž and zg. Based on the objective we derived above, we propose a novel objective for PGD-VAF by extending Fq. in order to not only fulfill the optimization objective of the general VAE framework (Eq, 4h. but also secure the disentanglemennt of local and global semantics of the graph. Specifically, we force the 2 equal for graphs with the same basic unit (Eq, 8) At the meanwhile, we distinguish 21 's of graphs with different basic units (Eq. g)
I; EG (icc.D[[K[(e [K[(go(e//(]llpo(a))) < Ig,
max Bo-D (Egg log po (A (2) (A (1) 1p (A (6) |x))]
P.0
S.t.Eo-D K KL Iloo < T3 Ba- [LL (K(oo///(((aaass)CCC)oo((oaa))) < Ig
(8)
2,,
(1)
= J J = 1, Nr 1, N; # 21,, I, N, I,.., N, #
5

---

where .(0 and _G] are respectively the local and global representations of the j-th graph for the FI, i-th basic unit. The derived objective function for PGI--VA (Eq. () can meet the objective in our problem formulation, as stated in Theorem 3.1] The proof of Theorem 3.J provided in Appendix ??.
Theorem 3.1. The proposed objective function in Eq. guarantees that and 2g satisfy:
1. 24 capture and only capture the local semantic of the graph;
2. bg capture and only capture the global semantic of the graph.
3.2.2 Learning objectives
To optimize the overall objective in Eq. 8 while satisfying the constraint, we transform the inequality constraint into tractable form. Given that I and ly are constants, we rewrite the first two constraints of Eq. g using Lagrangian algorithm under KKT condition
LK -3 K ( (=)po/oo)))))) - 82 ())l
(9)
where G represents a set of observed graphs, the Lagrangian multipliers B1 and 32 are regularization coefficients that constraint the capacity of latent information channels and Zg. respectively, and put implicit pressure of independence on the learned posterior.
Additionally, the goal of the last two constraints of Eq. 8 is to enforce the graphs that contain the same repeat patterns (i.e., local information) sharing the same and the graphs that contain different repeat patterns having different 2t. Thus, to both minimize the distances of among the graphs with the same unit cell and maximize the distances of 21 among the graphs with different unit cells, we propose to minimize a contrastive loss Leontra as:
exp(sim(s ( ((1) PLi
Leontra -3 ΣN Σ/e EN log N.N Dezt. D exp(sim(s (///)
where is a predefined temperature parameter, and B3 is the weights added the contrastive loss to adjust the convergence of 2 in graphs with the same basic unit and the divergence in graphs with different basic units. sim() is the cosine similarity borrowed in computing the contrastive loss (66].
In summary, the final implemented loss function minimized L includes three parts: (1) Tree to minimize the reconstruction loss of the input and generated graphs; (2) LKL to minimize the KL divergence of the posterior and its approximation: and (3) Leontra. the contrastive loss to force 21 equal for graphs with the same basic unit and different for those with different basic units:
EG--D(Busstn (G) log po (A |20)po (2))p)))) (o) [z4]] LKL Leontra Lree + LKL Leontra
3.3 Architecture of PGD-VAE
Based on the above periodic graph generative model and its learning objective, we are proposing our new Periodic Graph Disentangled VAE model (PGD-VAE). In general, PGD-VAN VAE takes the whole adjacency matrix A the input and can automatically identify A A and (()) by model architecture and learning objective. Specifically, PGD-VAE contains a local-pattern encoder and global-pattern encoder to disentangle the graph representation into local and global semantics. respectively. It also contains a local structure decoder, a global structure decoder, neighborhood decoder together with an assembler to formalize the graph while ensuring the periodicity.
An overview of PGD-VAE is shown in Figure 2 The proposed framework has two encoders, each of which models one of the distributions go g0(zg|G) and (o(2)[G); and three decoders to model
2x). and po (A(0)))=2)) respectively. Each type of representations is sampled using its own inferred mean and standard derivation. For example. the representation vectors are sampled as 2 = fn + oi 77, where 7) follows standard normal distribution.
p=(A())=)) po (A(())=))
3.3.1 Encoder
The encoder of PGD-VAE is designed to learn and 2g such that 2 captures only the local semantics of the graph and 2g captures only the global semantics of the graph. Thus, we propose local-pattern encoder to encode the local information and a global-pattern encoder to encode the global information,
as illustrated in Figure 2
Local-pattern encoder We designed the encoder to learn an assignment matrix Arep E RCxN N to identify representative nodes of the graph (Eq. 10) where C' is predefined as the number of

---

representative nodes to be assigned. This node embedding clustering module can enforce to only capture the local information, namely excluding the global information (e.g,, size of the graph) from 21, because: 1) Each node embedding preserves local pattern since GIN embeds local information of nodes; 2) Since node embedding of periodic graph is periodic, embedding of a node in a basic unit is the same as the corresponding node in another basic unit. By clustering all node embedding, each cluster preserves repetitive node embedding patterns across different local regions. In our setting, these representative nodes capture different embedding with each other. The column of Arep corresponding to the node y can be calculated:
Avep, = SOFTMAX
V))
(10)
where ()) is the embedding of node v at the last layer of Graph Isomorphism Network (GIN) 64] GIN is provably the most expressive among the class of GNNs and is as powerful as the
Weisfeilerl.ehmal graph isomorphism test [64]. TTo preserve all information contained by the
representative nodes, We concatenate all representative node embedding obtain μ and dy:
μ = MLP(CONCAVEELL. -Are 166 E V))
MLP(CONCATEEDI. -Arep 1000 V)),
(11)
where Darep has the degree of each cluster in its diagonal. can then be sampled according the VAE framework: 2 ~ (p),
Global-pattery encoder To enforce that 2g captures all the global information, GIN is leveraged to generate node-level embedding of the graph. Additionally, PGD-VAE leverages sum aggregator as the graph-level readout (Eq. 12) obtain the graph-level embedding 2g while achieving maximal
discriminative power [64]
= SUM(MLP( V)), SUM(MLPP(M JeE V))
(12)
The (K) represents the embedding of node v obtained at the last layer of GIN. 2g can then be sample according to the VAE framework: Fg ~ (N og).
3.3.2 Decoder
As shown in Figure 2 the decoder contains four parts: (1) the local-structure decoder to generate
AO; (2) the global-structure decoder to generate A (3) the neighborhood decoder to generate
A (m); and (4) an assembler f {A A(s), A - A which generates A according to Eq. (3). The local structure decoder takes the local-semantic representation as the input to generate the structure of the basic unit. The global structure decoder takes the global-semantic representation 2g as the input generate the global organization of basic units. Since how basic units connected is regarded as the local structure of the graph, the neighborhood decoder takes the local-semantic representation 2 as the input to generate the connection between adjacent basic units. The assembler f is closed-form function defined in Section 3.1 to assemble A Als) and A(r) into A:
A= = ((A)) A (())).
(13)
Here fi. fg. fn are deterministic functions serving as three decoders to generate A and A(), respectively. They are approximated by three multilayer perceptrons (MIPss) in implementation. Edges are sampled from the Bernoulli distribution given the probability output of MLP.
3.4 Complexity Analysis
We theoretically evaluate the scalability of our model and comparison models in the number of generation steps, similar to the analysis in [43). Since the decoder of PGD-VAE follows an one-shot strategy using MLP, it requires (()) generation steps. Additionally, the space complexity of PGD- VAE is (max((n (n². m²)), }), since the generated graph is determined by basic units, the connection of adjacent basic units and the global pattern. The graph size can be calculated as mn. The number of generation steps of PGD- PGD-VAE is aligned with that of VGAE, but is lower than that of Graph and GRAN. Nevertheless, PGD-VAL has the smallest space complexity. For time complexity, PGD-VAE can achieve O(m² + + mn)) by using matrix replication and concatenation tricks in implementation, better than that of other comparison models. The number of generation steps, time complexity and the space complexity of TGD-VAE and comparison models are summarized in Table] I
7

---

Table 1: Complexity of PGD-VAE and comparison models (n is the size of the basic unit, m is the size of the global pattern)
Model
Number of generation steps
Time complexity
Space complexiry
VGAE
((1)
([
O[(rmm)
CiapERNN
([(m)))
O[
GRAN
O[um)
PGD-VAB
O(I)
O' + mm)]
Vimar ,m
Table 2: PGD-VAE compared to state-of-the-art deep generative models QMOF, MeshSeg and Synthetic
datasets using KL.D (KI.D_clustee cluuster: KI.D clustering coefficient, _eD__ess:: KLD of graph density, PGD-
VAF-1: PGD-VAE without regularization, PGD-VAR-2: PGD-VAE replacing GIN with GCN in encoder,
PGD-VAB-3: RGE with single MLP as decoder)
Method
QMOF
MeshScg
Synthetic
KLD_cluster cluster
KLD deuse
KLD cluster
KLD densc
Klu_cluster
KLD dense
VGAE[8
1.7267
1.5688
0,9723
0,9642
4.6196
1.4222
Grap
2.2093
28441
L.365
1,1236
3,5928
03901
GraphRNNN 65
3.1055
5.0048
1.4316
1.2711
6.4719
3.6064
GRAN
3.3107
1,2466
0.8879
5.7501
2,3020
PGD VAE.
1,1933
0,4921
3,2672
0.8399
PGD- VAE-2
0,8508
0,9332
0.4539
0.4862
4.1596
2,7684
PGD VAE
0.9597
L0224
0.3496
0,5099
3,3850
0.9366
VAE
1.164
LO512
0.3977
0,4538
3.1863
0.6824
Table 3: PGD- VAE compared to ofte-of-the-ar deep generative models on QMOF, MeshSeg and Synthetic datasets according to uniqueness and novelty IPGD-VAE-1: PGD-VAL VAE without regularization, PGD-VAB-2:
RGD-VAE replacing GIN with GCN encoder, PGD- VAR-3: RGD-VAF with single MI.P as decoder)
QMOI
MeshSey
Synthenic
Method
Uniqueness
Novelty
Uniqueness
Novelly
Uniqueness
Novelyy
VGAE[38
100%
100%
100%
100%
100%
100%
GmphVAF
100%
100%
100%
100%
100%
100%
GraphRNN
23,34%
100%
71.39%
100%
26.76%
100%
GRAN[41
8.00%
100%
15.33%
100%
18.7%
100%
PGD-VAE-1
100%
100%
100%
100%
100%
100%
PGD-VAE-2
100%
100%
100 %
100%
100%
100%
RGD-VAE-
100%
100%
100%
100%
PGD-VAE
100%
100%
100 %
100%
100%
100%
4 Experiments
This section reports the results of both quantitative and qualitative experiments that were performed to evaluate PGD-VAF with other comparison models. Two real-world datasets and one synthetic datasets of periodic graphs were employed in the experiments. All experiments were conducted on the 64-bit machine with an NVIDIA GPU, NVIDIA GeForce RTX 3090.
4.1 Data
Two real-world datasets and one synthetic dataset were employed to evaluate the performance of PGD- VAE and other comparison models. All the comparison and our proposed methods only take the whole adjacency matrix A. The information of (()), A(9), and (a) are hidden from all the methods. Also, in all the datasets, the adjacency matrix of each graph followed its intrinsic permutation when they were generated, namely all the diagonal blocks are occupied by the basic unit A(0, as shown in Figure So all the graphs generated by all the models trained by these datasets also follow such fixed permutation which naturally saves them from considering node permutation. 1) QMOF is a publicaly available database of computed quantum-chemical properties and molecular structures (50) We selected metal-organic frameworks (MOFs) from QMOF consisting of unit cells as the basic unit (i.c., A A(s) and (T) can be automatically determined by Atomic Simulation Environment (ASE) [40]. Finally, the whole adjacency matrix A was the assemble A A and (()) via Eq. 3 2) MeshSeg contains 380 meshes for quantitative analysis of how people decompose objects into parts and for comparison of mesh segmentation algorithms [Л] We extracted meshes from MeshSeq and made 10 replicates for each mesh. The whole adjacency matrix A was directly obtained from the graphs in this dataset. 3) Synthetic dataset. Three synthetic datasets were generated based on three different basic units (i.e., A()), namely triangle, grid and hexagon. For each of the above basic unit types, different numbers of basic units were used to compose the whole graph (i.c., represented by
8

---

Figure 3: Visualizing variations of generated periodical graphs regarding local and global semantics
the whole adjacency matrix A) via Equation (4), with self-designed A and A All adjacency matrices were padded into fixed size with zeroes in implementation. The statistics of the three datasets have been summarized in Appendix, Table ?? and more details can be found in Appendix ??.
4.2 Comparison Models
Four state-of-tt sftt-off-the-art deep generative models were employed comparison models to compare with PGD- PGD-VAE VAE
(1) VGAE, a VAE-based graph generative model [38] by learning the graph-level representation and using ML.P to generate the adjacent matrix; (2) Graph VAB-based graph generative model [52) by modeling the existence of nodes and edges, as well as their attributes independent random variables in a probabilistic graph:(3) GraphRNN, deep autoregressive model that accounts for non-local dependencies of edges by decomposing generation process into sequence of node and edge conformations [65]; and (4) GRAN, a deep autoregressive model (43 which generates a block of nodes and associated edges at a time, and captures the auto-regressive conditioning between the generratd and to-be-generated pieces of the graph using GNNs with attention. PGD- -AEE and all comparison methods only take the whole adjacency matrix A as input, without extra information on A A ((s) or A Noticeably, we used fixed permutation that restricts the adjacency matrix A fixed node ordering as illustrated in Figure 2 Graphs with predefined permutation can easily be provided in periodic graph datasets as aforementioned.
In addition, three ablation study are conducted by evaluating four variants of PGD-VAE: (1) PGD-VAE without regularization term; (2) PGD-VAE replacing GIN with GCN in the encoder; (3) PGD- VAE with single MLP decoder. The comparison models are implemented using their default settings. The implementation details of PGD- DAE are presented in Appendix ??
4.3 Evaluating the Generated Graphs
Quantitative evaluation. To quantitatively evaluate generation performance of PGD- VAD-VAE and comparison models, we compute the KI. Divergence (KLD) between generated and ground-truth graphs to measure the similarity of their distributions regarding: (1) average clustering coefficient and (2) density of graphs (29). Moreover, we adopt uniqueness and novelty to evaluate generated graphs The uniqueness measures the diversity of generated graphs. The novelty reflects whether the model has generated unseen graphs by measuring the proportion of generated graphs that are not in the set of training graphs. The KLD-based results
calculated and shown in Table 2 The results show that, PGD- VAE achieves KLD smaller than comparison models by 0.8191 for clustering coefficient and 0.5062 for graph density average MeshSeg dataset, and 2.4275 for clustering coefficient as well as 1.7611 for graph density on average on Synthetic dataset. Interestingly, PGD-VAL without disentanglement (PGD-VAF-1) achieves 3.2672 and 0.8399 of KL.D clustering coefficient and density for Synthetic dataset, which worse than the full model by 0.0809 and 0.1575 as well. In addition, QMOF dataset, PGD-VAE that replaces GIN with GCN (AF-2) achieves KLD of 0,8508 regarding clustering coefficient and 0.9332 regarding graph density, outperforming
'Some graph generative models, such as Graph VAE that employs O(n*) on its max max- pooling matching algorithm, are not included in the comparison due to its poor scalability to large graphs, which is common in QMOF and MeshSeg datasets.

---

other models, which is likely due to the fact that the basic unit of MOF is usually more complex and needs more powerful GNN capture the local information of the graph. Also, the PGD- -AE (PGI-VAF-33) with a single MLP as the decoder hurts the performance of the model (Table 2 indicating the efficiency of the proposed hierarchical decoder. The measuring of uniqueness and novelty of generated graphs by different models has been summarized in Table PGD-VAE and all its variants achieve better performance than GraphRNN and GRAN, as they tend to generate unique graphs, which is desired for tasks such materials designing and object synthesis.
Qualitative evaluation for disentanglement To validate the disentanglement of local and global semantics in the graph representation, we qualitatively explore whether PGD-VAE can uncover and disentangle these two latent factors. To this end, we continuously changing the value of 2 or 2g while keeping another latent vector fixed and simultaneously visualizing the change of generated graphs. Figure shows the variation of generated graphs when traversing two latent factors 2 and Eg-
lal
Figure 4: Periodic graphs of MOFs generated by PGD-VAE. Nodes of the graph are colored by their degrees representing atoms of the MOF. Edges represent chemical bonds of the MOF: (a) structure of C16H8Na8Cu4NN 16516 composed of C, H, Na, Cu, N and S atoms; (b) structure of C24H32CI8CuAN8 which has a different basic unit consisting of C, H, CI, Cu and N atoms; (c) structure of C16H116M88020020 composed of relatively large basic unit with C, H, Mg, g, O and S atoms.
case study of generating MOFs We showcase several representative periodic graph structures gen- erated by the proposed VAE for MOF design in this section. As shown in Figure there three diverse and novel generated MOFs. The first one can be interpreted as C16H88 N aaC o44N166611 for have generated the basic unit (i.e., each other by incident chemical bonds. The second one can be interpreted as C24H33CI8CC4N3. which is highly different from the first in terms of atom diversity. The third generated MOF is C16H1 17/16.2 6.M 98020522, which has a larger basic unit, namely, C4H4M g20555. In addition, the basic units of these generated structures diversity, which demonstrates the effectiveness of the proposed PGD-VAE. More details of the MOF decoration and visualization are provided in Appendix ??.
which PGD-VA
that are connected with
all constructed in a periodic way with high validity and
Evaluating scalability We evaluated the time complexity of RGD-VAE and comparison models by running 10 epochs graphs in stratified synthesis dataset. Expectedly, all comparison models consume more time than PGD-VAE. The details are presented Appendix ??
Conclusion
We propose PGD-VAE, novel deep generative model for periodic graphs that can automatically learn, disentangle, and generate local and global patterns. Our method can guarantee the periodicity of generated graphs and outperforms existing graph generative models in periodic graph generation, both quantitatively and qualitatively. In the future, we attempt to introduce the node and edge features into the model to advance domains such as materials design and object synthesis.
Acknowledgments and Disclosure of Funding
This work was supported by the NSF Grant No. 2007716, No. 2007976, No. 1942594, No. 1907805, No. 1841520, No. 1755850, Meta Research Award, NEC Lab, Amazon Research Award, NVIDIA GPU Grant, and Design Knowledge Company (subcontract number: 10827.002.
References
[1] A. Alemi, Fischer, Dillon, and K. Murphy. Deep variational information bottleneck. In ICLR, 2017. htthttp:: /arxiv.ivv. org/ www.aa//e...o.oo 00410
10

---

[2] S. Arora. A survey on graph neural networks for knowledge graph completion. arXiv preprint N.Xiv:2077.1237/ 2020.
[3] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-188 1828, 2013.
[4] V. A. Blatov and D. M. Proserpio. Periodic-graph approaches in crystal structure prediction.
2010.
[5] P. V. Bushlanov, V. A. Blatov, and A. R. Oganov. Topology-basee crystal structure generator. Computer Physics Communications, 236:1-7, 2019.
[6] K. Cai, Y. Li, Y.-P. Fang, and Y. Zhu. A deep learning approach for flight delay prediction through time-evolving graphs. IEEE Transactions Intelligent Transportation Systems, 2021.
[7] A. Chartsias, T. Joyce. G. Papanastasiou, S. Semple, M. Williams, D. E. Newby, R. Dhar- makumar, and S. A. Tsaftaris. Disentangled representation learning in cardiac image analysis. Medical image analysis, 58:101535, 2019.
[8] C. Chen, K. Li, S. G. Teo, X. Zou, K. Wang, Wang, and Zeng. Gated residual recurrent graph neural networks for traffic prediction. In Proceedings of the AAAI conference artificial intelligence, volume 33, pages 485-499, 2019.
[9] F. Chen, Y.-.. Wang, B. Wang, and C.-C. J. Kuo. Graph representation learning: A survey. APSIPA Transactions on Signal and Information Processing, 9, 2020.
[10] R. Chen, X. Li, R. Grosse, and D. K. Duvenaud. Isolating sources of disentanglement in variational autoencoders, Advances in neural information processing systems, 31, 2018
[11] X. Chen, A. Golovinskiy, and T. Funkhouser. A benchmark for 3D mesh segmentation. ACM Transactions on Graphics (Proc. SIGGRAPH), 28(3), Aug. 2009.
[12] Y. Chen, L. Wu, and M. J. Zaki. Toward subgraph guided knowledge graph question generation with graph neural networks. arXiv preprint A.KV::2004.000115 2020.
[13] N. De Cao and T. Kipf. MolGAN: An implicit generative model for small molecular graphs. ICML 2018 workshop Theoretical Foundations and Applications of Deep Generative Models, 2018.
[14] O. Delgado-Friedriite S. T. Hyde, M. O' O'eeffe, and O. M. Yaghi. Crystal structures as periodic graphs: the topological genome and graph databases. Structural Chemistry, 28(1):39-44) 2017.
[15] Y. Du, Wang, X. Guo, H. Cao, S. Hu, J. Jiang, Varala, A. Angirekula, and L. Zhao. Graphgt: Machine learning datasets for graph generation and transformation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[16] Y. Du, X. Guo, H. Cao, Y. Ye, and L. Zhao. Disentangled spatiotemporal graph generative models. arXiv preprint htt::22033004111 2022.
[17] Y. Du, X. Guo, A. Shehu, and L. Zhao. Interpretable molecular graph generation via monotonic constraints. In Proceedings of the 2022 SIAM International Conference on Data Mining (SDM), pages 73-81. SIAM, 2022.
[18] Y. Du, X. Guo, Y. Wang, A. Shehu, and L. Zhao. Small molecule generation via disentangled representation learning. Bioinformatics (Oxford, England), page btac296, 2022.
[19] F. Fan, H. Yuan, Y. Feng, F. Liu, I.. Zhang, X. Liu, X. Zhu, W. An, S. Rohani, and J. Lu. Molecular simulation approaches for the prediction of unknown crystal structures and solubilities of (r)-and (r, s)-crizotinib in organic solvents. Crystal Growth and Design, 19810):5882-5855 2019.
[20] W. Fan, Y. Ma, Li, Li, Y. He, E. Zhao, Tang, and D. Yin. Graph neural networks for social recommendation. In The World Wide Web Conference, pages 417-426, 2019.

---

[21] D. Foley, A. Van Dam, S. K. Feiner, J. F. Hughes, and R. L. Phillips. Introduction to computer graphics, volume 55. Addison- Weeleey Reading, 1994.
[22] O. D. Friedrichs, A. W. Dress, D. H. Huson, J. Klinowski, and A. L. Mackay, Systematic enumeration of erystalline networks. Nature, 40066745):614-447, 1999.
[23] N. Fu, A. Hashikura, and H. Imai. Geometrical treatment of periodic graphs with coordinate system using axis-fiber and application to motion planning. In 20/2 Ninth International Symposium on Voronoi Diagrams in Science and Engineering, pages 115-121. IEEE, 2012.
[24] H. Gaur, B. Khidhir, and R. K. Manchiryal. Solution of structural mechanic's problems by machine learning. International Journal of Hydromechatronics, 5(1):22 43, 2022.
[25] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks, 2014.
[26] X. Guo and L. Zhao. A systematic survey on deep generative models for graph generation. arXiv preprint arXiv:2007.060: 2020.
[27] X. Guo, L. Zhao, Z. Qin, L. Wu, A. Shehu, and Y. Ye. Interpretable deep graph generation with node-edge co-disentanglement. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1697-1707, 2020.
[28] X. Guo, L. Zhao, Z. Qin, L. Wu, A. Shehu, and Y. Ye. Node-edge co-disentangled representation learning for attributed graph generation. In International Conference on Knowledge Discovery and Data Mining (SIGKDD), 2020.
[29] X. Guo, Y. Du, and L. Zhao. Deep generative models for spatial networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 505-515, 2021.
[30] X. Guo, S. Wang, and L. Zhao. Graph neural networks: Graph transformation. In Graph Neural Networks: Foundations, Frontiers, and Applications, pages 251-275. Springer, 2022.
[31] X. Guo, L. Wu, and L. Zhao. Deep graph translation. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[32] T. V. Hahn and C. K. Mechefske. Self-supervised learning for tool wear monitoring with a International Journal of Hydrromechatromies, 4(1):69-98,
arnpraaggdd-varrrattoaaaaaaaeeee 2021.
[33] W. L.. Hamilton. Graph representation learning. Synthesis Lectures on Artifical Intelligence and Machine Learning, 14(3):1-159 2020.
[34] W. L. Hamilton, R. Ying, and J. Leskovec. Representation learning graphs: Methods and applications. IEEE Data Eng. Bull., 40(3):52-74, 2017. URL http::/sittss. computer
org//deall//ATaeptppp53 pdf
[35] Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.
[36] H. Kim and A. Mnih. Disentangling by factorising, In International Conference on Machine Learning, pages 2649-2658. PMLR, 2018
[37] D. P. Kingma and M. Welling, Auto-encoding variational bayes. In ICLR, 2014. URL https://arxiv: http:/aaxvvvorrar///e/// abs//312..1111
[38] T. N. Kipf and M. Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07300 2016.
[39] Korn and I.. Linardakis. conservative discretization of the shallow-water equations on triangular grids. Journal of Computational Physics, 375:871-900, 2018.
12

---

[40] A. H. Larsen, J. J. Mortensen, J. Blomqvist, I. E. Castelli, R. Christensen, M. Dulak, J. Friis,
M. N. Groves. Hammer, Hargus, et al. The atomic simulation e python library for working with atoms. Journal of Physics: Condensed Matter, 29(27):273002, 2017.
[41] A. Le Bail. Inorganic structure prediction with grinsp. Journal of applied crystallography, 38 (2):389-395, 2005.
[42] J. Lee and R. Asahi. Transfer learning for materials informatics using crystal graph convolutional neural network. Computational Materials Science, 190: 10314, 2021.
[43] R. Liao, Y. Li, Y. Song, S. Wang, W. Hamilton, D. K. Duvenaud, R. Urtasun, and R. 7emel. Effi cient graph generation with graph recurrent attention networks. Advances in Neural Information Processing Systems, 32, 2019.
[44] C. Ling, C. Yang, and L.. Zhao. Deep generation of heterogeneous networks. In 2021 IEEE International Conference on Data Mining (ICDM), pages 379-388, 2021.
[45] C. LLing, H. Cao, and L. Zhao. Stgen: Deep continuous-timg time spatiotemporal graph generation. In 2022 European Conference on Machine Learning and Principles of Knowledge Discovery in Databases, 2022.
[46] K. Martinkus, A. Loukas, N. Perraudin, and R. Wattenhofer. Spectre: Spectral condition- ing helps to overcome the expressivity limits of one-shot graph generators. arXiv preprint arXiv:2204.01613, 2022.
[47] C. W. Park and C. Wolverton. Developing an improved crystal graph convolutional neural network framework for accelerated materials discovery. Physical Review Materials, 4(6): 063801, 2020.
[48] S. Ramsden, V. Robins, and S. Hyde. Three-dimensional euclidean nets from two-dimensional hyperbolic tilings: kaleidoscopic examples. Acta Crystallographica Section A: Foundations of Crystallograhhy 65(2):81-108, 2009.
[49] M. Ranzato, J. Susskind, V. Mnih, and G. Hinton. On deep generative models with applications to recognition. In CVPR 2011, pages 2857-2864. IEEE, 2011.
[50] A. S. Rosen, S. M. Iyer, D. Ray, Z. Yao, A. Aspuru- Guzik, L. Gagliardi. J. M. Notestein, and R. Q. Snurr. Machine learning the quantum-chemiett properties of metal-organic frameworks for accelerated materials discovery. Matter, 4(5):1578-159) 2021.
[51] M. Safa, M. Ahmadi, J. Mehrmashadi, D. Petkovic, M. Mohammadhassani, Y. Zandi, and Sedghi. Selection of the most influential parameters on vectorial crystal growth of highly oriented vertically aligned carbon nanotubes by adaptive neuro-fuzzy technique. International Journal of Hydrromechatronics, 3(3):238-22 251,, 2020.
[52] M. Simonovsky and N. Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In International conference artificial neural networks, pages 412 422. Springer, 2018
[53] P. Tabero, A. Frackowiak, E. Filipek, G. Dbrowska, Z. Homonnay, and P. Szilagyi. Synthesis, thermal stability and unknown properties of fel -xlxx04 solid solution. Ceramics International, 44(15):17559-1776 2018.
[54] A. Tian, C. Zhang, M. Rang, X. Yang, and Z. Zhan. Ra-gen: Relational aggregation graph convolutional network for knowledge graph completion. In Proceedings of the 2020 12th International Conference on Machine Learning and Computing, pages 580-586, 2020.
[55] L. Tran, X. Yin, and X. Liu. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 1415-1424, 2017.
[56] M. Treacy, I. Rivin, E. Balkovsky, K. Randall, and M. Foster. Enumeration of periodic tetrahedral frameworks. polynodal graphs. Microporous and Mesoporous Materials, 74(1-3):121-132, 2004.
13

---

[57] P. Wang, Y. Fu, J. Zhang, X. Li, and D. Lin. Learning urban community structures: A collective embedding perspective with periodic -eatialltemporal mobility graphs. ACM Transactions on Intelligent Systems and Technology (TIST), 96)):128, 2018.
[58] S. Wang, Y. Du, X. Guo, B. Pan, and L. Zhao. Controllable data generation by deep learning: A review. arXiv preprint arXiv:2207 009000..999..... 2022.
[59] S. Wang. X. Guo, X. Lin, B. Pan, Y. Du, Y. Wang, Y. Ye, A. Petersen, A. Leitgeb, S. AlKhalifa, K. Minbiole, B. Wuest, A. Shehu, and L. Zhao. Multi-objectivve deep data generation with correlated property control, 2022.
[60] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1):4-24, 2020.
[61] T. Xie and J. C. Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters, 120(14):14530,, 2018.
[62] T. Xie, X. Fu, O.-E. Ganea, R. Barzilay, and T. Jaakkola. Crystal diffusion variational au- toencoder for periodic material generation. Computer Physics Communications, 236:1-7, 2019.
[63] T. Xie, X. Fu, O.-E Ganea, R. Barzilay, and T. Jaakkola. Crystal diffusion variational autoen- coder for periodic material generation, 2021.
[64] K. Xu, W. Hu, Leskovec, and Jegelka. How powerful are graph neural networks? In ICLR, 2019. httptttts////iiii org/ atspa/a/.////.../11/////
[65] J. You, R. Ying, X. Ren, W. Hamilton, and J. Leskovec. Graphrnn: Generating realistic graphs with deep auto-regressive models. In International conference on machine learning, pages 5708-5717. PMLR, 2018
[66] Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33:5812-5822, 2020.
[67] J. Q. Yu and J. Gu. Real-time traffic speed estimation with graph convolutional generative autoencoder. IEEE Transactions Intelligent Transportation Systems, (0(10):34403351) 2019.
[68] L. Zhang, L. Zhao, S. Qin, D. Pfoser, and C. Ling. Tg-gan: Continuous -iiereoouu---eeeeee temporal graph deep generative models with time-validity constraints. In Proceedings of the Web Conference 2021, pages 2104-2116 16, 2021.
[69] Y. Zhang, S. Wang, B. Chen, and J. Cao. Gegan: Generative adversarial nets with graph cnn for setwork-scale traffic prediction. In 2019 International Joint Conference on Neural Networks (IJCNN, pages 1-8. JEEE, 2019.
[70] Z. Zhang and L. Zhao. Representation learning spatial networks. Advances in Neural Information Processing Systems, 34:2303-2318, 2021.
[71] J. Zhao, Y. Cheng, Y. Cheng, Y. Yang. F. Zhao, J. Li, H. Liu, S. Yan, and J. Feng. Look across clapse: Disentangled representation learning and photorealistic cross-age face synthesis for age-invariant face recognition. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 9251-9258. 2019.
[72] T. Zhong, T. Wang, Wang. J. Wu, and Zhou. Multiple-aspect attentional graph neural networks for online social network user localization. IEEE Access, 8:95223-95234, 2020.
[73] D. Zhou, L. Zheng, J. Xu, and J. He. Misc-gan: A multi-scale generative model for graphs. Frontiers in big Data, 2:3, 2019.
[74] J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun. Graph neural networks: A review of methods and applications. Al Open, 1:57-81, 2020.
14


Reviews:
[{"review_id": "tSRw8S7UH8b", "reviewer": "Reviewer_bnmq", "summary": "This work proposed a novel VAE-based method to generate periodic graphs with applications to MOF, mesh and synthetic graphs generation. The proposed VAE-based method (PGD-VAE) disentangles the graph representation into local and global semantics by generating $A^{(l)}, A^{(g)}, A^{(n)} $ and then assembles them into a complete periodic graph. Overall, I enjoyed reading the paper. But there are some concerns to be addressed.", "questions": "Please refer to the *weaknesses* for my questions.", "limitations": "The limitation of this work is that the generated periodic graphs may not be applicable to real-world applications.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well-written and easy to follow\n* To the best of my knowledge, the proposed PGD-VAE is novel in generating periodic graphs, and the proposed disentangled periodic graph representation is interesting.\n* The results are impressive on the evaluation metrics.\n\nWeaknesses:\n* Why not use the disentangled periodic graph representation as input to the PGD-VAE as well?\n* More Ablation studies on the necessity of disentanglement should be added to validate the claim of the benefits of using disentangled generation.\n* The generation results look good on the evaluation metrics. However, it is unknown how the generated graphs can be applicable to real-world applications. For instance, are the generated MOF structures stable and synthesizable? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "q5Fs1EMrkF", "reviewer": "Reviewer_GS3z", "summary": "The paper proposes to tackle periodic graph generation, by learning to generate basic structural units and to compose them into a larger graph. This is achieved by using a hierarchical VAE approach with a loss term which encourages disentanglement.", "questions": "How exactly is m an n selected for each dataset? As I understood it, you use analytic methods (domain knowledge) to determine the optimal substructure sizes. This already seems to give a lot of ground-truth information for the model.\n\nWhat ordering do the GraphRNN and GRAN use? The default BFS/DFS orderings that were originally used for these models or do you also provide them with the true graph ordering you use for your model?\n\nIn Figure 3, are the trees in the lower left corner part of the dataset? It is unclear which dataset this figure comes from. I presume it to be the synthetic one, but it does not have trees. If it is the synthetic dataset, it seems strange trees are generated even though all base units in the dataset where polygons.", "limitations": "As I mentioned in the weaknesses, the assumption that the adjacency matrix has a true canonical ordering is quite problematic. Even though for regular graphs in some cases we have something close to a canonical ordering. It essentially goes against all of the graph machine learning wisdom that is based on equivariance to the point that it is unclear we even need graph machine learning to solve this problem when this assumption trully holds.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Mostly, current deep graph generation approaches aim to generate the graph by focusing on local structure so any attempt to introduce more focus on the global structure or decouple global and local structure generation is very welcome. Recently, a more general approach to decouple local and global structure generation, not focused on periodic graphs  has been proposed [1]. I think this should be mentioned in the related work. I haven't seen other work that focuses in particular on periodic graph generation using deep learning. Which makes this particular problem quite novel and potentially useful in problems such as crystal structure design.\n\nWhat I find problematic, is the assumption that the node order in the graph is known and can be consistently determined.\nFirst, it limits us to a small set of problems where we can even hope to determine such an ordering (e.g. atom lattices). Further, even for the problems considered here it's not entirely clear to me that we can determine such an ordering. For example lets consider the MeshSeq dataset. In this dataset we have a sequence of cuts people used to partition a given mesh. Even though the cuts are sequential, they gives us two equivalent parts (we build a binary tree, not a sequence) and depending on the particular mesh of the same type of object or its orientation which part is the \"left\" of the split and which is \"right\" could differ. Also, In general for mesh processing various equivariant architectures are normally used, due to the problem that in general we don't have a way to find a canonical orientation (ordering) of a mesh. So I don't see how we can get a canonical representation of a mesh here to use a loss that is not invariant to permutations.\nIn context of this an important reference [2] discusses, that for graph generation, you need a permutation invariant loss function. Which we do not have in the approach proposed here. I also think this reference should be included in the paper.\nThe second problem I see with assuming that we can get this canonical graph structure (ordering) is that in this case we are essentially just dealing with a set of matrices. So we can just use a traditional and very powerful matrix processing models such as convolutional neural networks. In fact the proposed architecture is a bit like a CNN already as it considers a bunch of patches and how they combine/overlap. So under this assumption I'm not even sure we need a GNN-based architecture.\n\nA small note on the Global pattern encoder - it's true that using GIN and global pooling gives you 1-WL representation power, but 1-WL is inherently a local procedure, so I'm not sure that's the best architecture to capture the global graph structure. Models that usually perform better on graph-level tasks could be considered, such as the more expressive GNN architectures or graph transformers, which do not suffer from having only local interactions.\n\nWhy only two graph similarity measures (density and clustering coefficient) are used? Even the baselines that were used such as GRAN used more advanced measures, such as orbital counts or spectral features. Using only density and clustering coefficient are determined by local features, so there are no measures that show how well the global graph structure is reproduced. Which is one of the main goals of the work.\n\nI'm also confused why the GraphVAE baseline was not used. It certainly has the closest overall architecture (GNN encoder + MLP decoder). Authors argue that it is too expensive, but the expense comes from the graph matching used to compute a permutation invariant loss. But authors claim, that for the graphs they consider we can assume the node order to fixed as otherwise their proposed method does not work. Also, GRAN used GraphVAE as a baseline on datasets with graphs as large as the ones considered in this paper (Protein graphs used in GRAN paper had up to 500 nodes). In GRAN paper they also just discarded GraphVAE matching loss and used the node ordering as given to GRAN model itself, which worked fine.\n\nWriting quality could certainly be improved. The text is wordy, there's a lot of typos, sometimes words are missing in the sentences, even multiple words per sentence (e.g. line 125). I would strongly suggest the authors to at least use some sort of a spelling and grammar correction software (e.g. Grammarly).\n\n[1] Martinkus et al. \"SPECTRE: Spectral Conditioning Helps to Overcome the Expressivity Limits of One-shot Graph Generators.\" ICML 2022\n[2] Vignac and Frossard \"Top-N: Equivariant Set and Graph Generation without Exchangeability\" ICLR 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SnyJmnu75DC", "reviewer": "Reviewer_9Uud", "summary": "This paper focuses on the problem of periodic graph generation, and proposes PGD-VAE, a Variational AutoEncoder based generative model for periodic graphs that can automatically learn, disentangle, and generate local and global patterns. The authors formulate the generation of periodic graphs as the generation of a finite number of basic units.", "questions": "* If the goal is to make the generation of local structures and global repeating patterns independent, why do you  introduce a contrastive loss to make the generation of local structures conditioned on the global repeating patterns?", "limitations": "Yes, they addressed the limitations of the proposed method.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n* originality: This paper formulates a new problem setting for periodic graph generation. Instead of generating basic units and repeating patterns, they generate the periodic graph consist of a fixed number of basic units. \n\n## Weaknesses\n* According to the authors, this paper's main idea of the generation process is to make the generation of local structures and global repeating patterns independent. But they introduce a contrastive loss to make the generation of local structures actually **conditioned** on the global repeating patterns.\n* In this paper, the generation of node type such as atom type in QMOF is ignored. Only the structure is generated.\n* Permutation invariance is ignored. The proposed method can not satisfy permutation invariance when the node type needs to be considered. The application of the proposed method is limited because of breaking permutation invariance. \n* Because of above considerations, the quality and significance of the proposed method is limited.\n* Clarity: Hard to read. There are variables coming from nowhere and not shown in the equation, such as Equation (8). Many sentences hard to understand.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KRQFV49EHnO", "reviewer": "Reviewer_WzHK", "summary": "Generating novel periodic graphs is an interesting and important task as the application examples given in the paper. Three challenges towards this task are provided by the authors, which seem reasonable to me. To address these challenges, the authors propose a novel model to generate periodic graphs by automatically learning, disentangling, generating and assembling local and global patterns of periodic graphs. This idea of “hierarchically”generating periodic graphs is novel and it is suitable and efficient to handle this type of graphs as suggested by the complexity analysis part. In addition, the experimental results show the effectiveness of this model compared with other comparison models.\n", "questions": "1. The author can add and discuss the missing references, such as [1] in the Related Works session.\n2. Please check and fix the typos, e.g., that in Figure 2.\n", "limitations": "The limitations of the proposed work are discussed in the Conclusion session.\nThis work does not have potential negative societal impacts.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1 The framework proposed by the paper solves an important task to generate periodic graphs and is novel to me as it generates periodic graphs in a hierarchical fashion. This strategy looks natural since the periodic graph (e.g., graph structure of materials) can be very huge and is intuitively more efficient since it only needs to learn and generate local and global patterns. Also, to my understanding, the node embedding clustering module embedded in the local-pattern encoder can learn the local structure of the graph and perform at least as well as the sum pooling on the whole graph. \n2 The analysis on the model is comprehensive both theoretically and experimentally. The learning objective has been derived properly. The complexity of the model is better than the comparison models as shown in Table 1. The proposed model also has better performance than the comparison models.\n3 The presentation of the paper is good to me. Figure 2 is organized very well and is intuitive to show how the framework functions. The ability to disentangle local and global patterns has been demonstrated by Figure 3.\n4 The authors published the code.\n\n\nWeakness:\n1 Some typos: for example, in Figure 2, it should be A^{(l)}, A^{(g)} and A^{(n)} rather than A_l, A_g and A_n.\n2 Missing references: for example, [1] below aims to generate periodic structures, although seems using different strategies and different type of input data.\n \n[1] Xie, Tian, et al. \"Crystal diffusion variational autoencoder for periodic material generation.\" arXiv preprint arXiv:2110.06197 (2021).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept