Towards Symmetry-Aware Generation of Periodic Materials
Authors: Youzhi Luo
Chengkai Liu
Shuiwang Ji
Venue: NeurIPS 2023 spotlight (2023)
Keywords: material generation
symmetries
variational auto-encoder
score-based diffusion model
TL;DR: We propose SyMat, a novel deep generative model for periodic material generation.

Abstract: We consider the problem of generating periodic materials with deep models. While symmetry-aware molecule generation has been studied extensively, periodic materials possess different symmetries, which have not been completely captured by existing methods.
In this work, we propose SyMat, a novel material generation approach that can capture physical symmetries of periodic material structures. SyMat generates atom types and lattices of materials through generating atom type sets, lattice lengths and lattice angles with a variational auto-encoder model. In addition, SyMat employs a score-based diffusion model to generate atom coordinates of materials, in which a novel symmetry-aware probabilistic model is used in the coordinate diffusion process. We show that SyMat is theoretically invariant to all symmetry transformations on materials and demonstrate that SyMat achieves promising performance on random generation and property optimization tasks. Our code is publicly available as part of the AIRS library (https://github.com/divelab/AIRS).

Towards Symmetry-Aware Generation of Periodic Materials
Youzhi Luo, Chengkai Liu, Shuiwang
Department of Computer Science & Engineering Texas A&M University
College Station, TX 77843 {yzluo, liuchengaa, , sj @t@aamueedu
Abstract
We consider the problem of generating periodic materials with deep models. While symmetry-aware molecule generation has been studied extensively, periodic mate- rials possess different symmetries, which have not been completely captured by existing methods. In this work, we propose SyMat, novel material generation ap- proach that can capture physical symmetries of periodic material structures. SyMat generates atom types and lattices of materials through generating atom type sets, lattice lengths and lattice angles with variational auto-encoder model. In addition, SyMat employs a score-based diffusion model to generate atom coordinates of mate- rials, in which novel symmetry-awann probabilistic model is used in the coordinate diffusion process. We show that SyMat is theoretically invariant to all symme- try transformations materials and demonstrate that SyMat achieves promising performance on random generation and property optimization tasks. Our code is publicly available as part of the AIRS library fnttss:/grhubb.codddilbb///////
1 Introduction
Designing synthesizing novel periodic materials with target properties is fundamental problem in many real-world applications, such as designing new periodic materials for solar cells and batteries [ For long time, this challenging task heavily relies either manually designing material structures based on the experience of chemical experts, or running very expensive and time-consuming density functional theory (DFT) based simulation. Recently, thanks to the progress of deep learning tech- niques, many studies have applied advanced deep generative models [22
13 to generate or discover novel chemical compounds, such molecules [10] 39 50) and proteins 48 [6] However, while
deep learning has been widely used in periodic material representation learning S
40
4 [7] 25
generating novel periodic materials with deep generative models remains largely under-explored.
The major challenge of employing deep generative models to the generation of periodic materials capturing physical symmetries of periodic material structures. Ideally, deep generative models for periodic materials should maintain invariance to symmetry transformations of periodic materials, including permutation, rotation, translation, and periodic transformations (59 9 660. To achieve this target, we propose SyMat, a novel symmetry-aware periodic material generation method. SyMat transforms the atom types and lattices of materials to symmetry-awan generation targets. which generated by a variational auto-encoder model [22]. Besides, SyMat adopts novel symmetry-awee aware probabilistic diffusion process to generate atom coordinates with score-based diffusion model [4]. Experiments show that SyMat can achieve promising performance on random generation and property optimization tasks.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Relations with Prior Methods. We note that several existing studies are related to our proposed SyMat, such as CDVAE [52]. Generally, the major difference of SyMat between them lies in the capture of symmetries. We will discuss the detailed differences in Section 3.5
Background and Related Work
2.1 Molecule Generation
Designing or discovering novel molecules with target properties has long been a challenging problem in many chemical applications. Re-
cently, motivated from the significant progress
in deep generative learning [22 [2 44
[3..
lot of studies have applied advanced deep gener- ative models to this problem. Some early stud- 27 consider molecules as 2D topology graphs and design graph genera- tion models to discover novel molecules. More recently, considering the importance of 3D struc- tures in many molecular properties, many stud- ies have also proposed advanced generative mod-
Molecule
Material
ies 19 58
32
Figure 1: An illustration of molecule (left) and periodic material (right) structures. Different from molecules, atoms in materials periodically repeat themselves infinitely in 3D space (we 21) visualization here for simplicity).
els that can generate 3D molecular conformations either from scratch 110 39
31
15 50 or from
input conditional information [33 56 [43 (33 [54, 42 55 20 28). However, all these existing studies designed for non-periodic molecules while materials have periodic structures, in which atoms periodically repeat themselves in 3D space (see Figure T for an illustration of differences between
molecules and periodic materials). Hence, they cannot be directly applied to the periodic material generation problem that we study in this work.
2.2 Periodic Material Generation
While molecule generation has been extensively studied, currently the periodic material design or generation problem remains largely under-explored. Early periodic material design methods 34 7 only generate compositions of chemical elements in periodic materials but do not generate 3D structures. More recent studies have proposed use variational auto-encoders (VAEs) 22 or generative adversarial networks (GANs) [2] to generate 3D structures of periodic materials. However, these methods use coordinate matrices 36
61 or voxel images [1 of materials as the direct generation targets, which results in the violation of physical symmetries of periodic material structures their captured probabilistic distributions. Also, the VAE and GAN models used by them are not powerful enough to capture the distribution of complicated 3D periodic material structures. Different from them, our method employs a more powerful score-based diffusion model [44) to generate 3D periodic material structures, and the model is designed to capture physical symmetries in materials.
21
2.3 Score-Based Diffusion Models for 3D Structure Generation
Diffusion models a family of deep generative models that have achieved outstanding generation performance in a variety of data modalities, such as images [38] 37). audios (24). and molecular conformations (55] 15 20 Currently, score-based diffusion models 44 and denoising diffusion probabilistic models 13 are two most commonly used diffusion models, and our method employs score-based diffusion models for 3D periodic material generation. For given data distribution p(æ) in the high-dimensiooal data space, the score-based diffusion model trains a parameterized score model so with parameters
accurately approximate the score function V_ log p(æ) from To improve the score approximation accuracy in the regions where the training data is sparse, denoising score matching [46] [4 is proposed to more effectively train s-(-). Specifically, let {o:) be set of positive scalars satisfying
> 02 > > dr, the denoising score matching method first perturbs the data point æ sampled from p(æ) by sequence of Gaussian noises with noise magnitude levels (ox ) iee. sampling き from Po. (m|=) = N ( of I) for every noise level o, The score model so(-) is optimized to predict the denoising score function V log Pc, (Ø a:) from क at each noise level, where the ground truth of Va log Po. (æ x)) can be easily computed from the mathematical formula

---

of (á of Formally, s-(-) is optimized to minimize the following training objective L:
T
C 2T
Σ
88 2 2
at
As shown in Vincent [66] when de is small enough, after so(-) is well trained with the objective in Equation the output (a) can accurately approximate the exact score function for any imput æ. New data samples can be generated by running multiple steps of the Langevin dynamics sampling algorithm 49 with the score functions approximated by the score model.
Several existing studies have applied score-based diffusion models to 3D molecular and material structure generation. ConfGF [42] is the first score- diffusion model for 3D molecular confor- mation generation. It proposes novel theoretic framework achieve -oootrranllaiinal equivariance in score functions. Based on the theoretic framework of ConfGF, DGSM (30) further improves the performance of ConIGF by dynamic graph score matching. Though they have achieved good performance in 3D molecules, they do not consider periodic invariance so they cannot be applied to 3D periodic materials. Recently, a novel approach CDVAE [52] proposed for periodic material generation, and score-basee based diffusion models are used in CDVAE to generate atom coordinates in materials. However, as we discussed in Section [.3] CDVAE fails to achieve translational invariance in calculating the denoising score function, so it does not capture all physical symmetries.
3 SyMat: Symmetry-Aware Generation of Periodic Materials
While a variety of deep generative models have been proposed to capture all physical symmetries in 3D molecular conformations, it remains challenging to do it for periodic materials they have more complicated 3D periodic structures. In this section, we present SyMat, a novel periodic material generation method. In the following subsections, we will first introduce the problem of symmetry- aware material generation, then elaborate how our proposed SyMat approach achieves invariance to all symmetry transformations of periodic materials.
3.1 Symmetry-Aware Generation
For any periodic material structure, we can consider as periodic repetitions of one unit cell in 3D space, where any unit cell is the smallest repeatable structure of the material. Hence, for any periodic material M, we can describe its complete structure information with one its unit cells and the three lattice vectors describing its periodic repeating directions. Specifically, assuming there n atoms in any unit cell of M, we represent M as M = (A, P, L), where A E Z" P E R3xn, and L 33x the atom type vector, coordinate matrix, and lattice matrix, respectively. Here, the i-th element a; of = [a1...., a, and the i-th column vector p; of P = [Pi,.. Pal denote the atom type, i.e., atomic number and the 3D Cartesian coordinate of the i-th atom in the unit cell, respectively. The lattice matrix = [e,, l2, La] is formed by three lattice vectors L,, l2 &3 indicating how the atoms in a unit cell periodically repeat themselves in 3D space.
In this work, we consider the problem of generating periodic material structures with generative Im
models. Formally, we given dataset M = {M,} where each periodic material data M, is assumed to be sampled from the distribution p(-). Because of the physical symmetry properties in periodic materials [59 9. p(-) is restricted to be invariant to the following symmetry transformations.
Permutation. For any M = (A, L), we permute A, ie., exchange elements in A to obtain A', and use the same permutation order to permute the column vectors of P to obtain P', we actually change nothing but atom orders in unit cell representations. Hence, let M' (A', P L), the probability densities of M and M' are actually the same, i.e., p(M) p(M').
Rotation and translation. For any M (A.P, D). we can rotate and translate its atom positions in 3D space to obtain a new material M' (A, P', L'), where P' p'= QP + b17 and L' = QL. Here Q E 3x3 is the rotation matrix satisfying Q" Q = I, E R the translation vector, and 1 is an nt-dimensional vector whose elements are all 1. It can be intuitively understood that rotation and translation do not change any internal structures of periodic materials, so M and M' are actually different representations of the same material, pMM) = p(M') should hold.
3

---

KL-distance
Reconstruction loss
l2
4122
VAE encoder
VAE decoder
2
Atom
Lattice lengths
Lattice angles
C
+
z zL
type set
Score model
So 5M(M)
Score matching loss
M (A.
(A.P.L)
(a) The overall process of training the SyMat model on a data sample M (A, L) from the dataset.
WAE decoder
x2
4 6
P.2
Intiaian Pu 708 7
75.7 *,(M,
sy(M
70.7
/60
10 9/-
z1- =! NV(0.)) .r(o.)
P)
(AP
(b) The overall process of generating a new material with the SyMat model.
Figure 2: Illustrations of training and generation process in our proposed SyMat method.
Periodic transformation. Because every atom from a unit cell periodically repeats itself along lattice vectors in 3D space, any coordinate matrix P 3 has infinite number of periodically equivalent coordinate matrices. Formally, a periodically equivalent coordinate matrix pK P can be obtained by periodic transformation pk P LK, where K Z3x" is an arbitrary integer matrix. Since pK and P are equivalent and the periodic transformations between them are invertible, consider M (A, L) and M (A, PK L) the same material so p(M) p( KM) should hold.
In the periodic material generation problem, target to learn generative model po with parameters 0 from the given dataset M so that the model can capture the real data distribution p(-), and the learned generation model generate valid material structure M with high probability Pe (M). In addition, we require the distribution Pe(-) captured by the generation model to have the same symmetry-related restrictions the real data distribution p(-). The main motivation modeling symmetries comes from property optimization. A significant target of developing generative models for periodic materials is to generate novel periodic materials with desirable chemical properties (e. g., low energies). Notably, chemical properties are invariant to physical symmetry transformations. For instance, rotating a material structure in 3D space does not change its energy. In other words, chemical properties only depend on the 3D geometric information that is invariant to symmetry transformations (e.g., interatomic distances). Using symmetry-aware modeling forces generative models only capture the distribution of these 3D geometric information, which facilitates searching materials with desirable properties in the latent space.
3.2 An Overview of SyMat
Generally, to generate a new material M = (A,P, our proposed MyMat method first generates the atom type vector A and lattice matrix L, then generates the coordinate matrix P conditioned on A and L. The probability of generating M = (A, P, L) can then be described as
po [M) po A A) (P)((P)
Since the structures of atom types and lattices are relatively simple, we find that VAE 22 model already has adequate capability to capture their distributions. Hence, we use VAE model to generate A and L. However, the structures of coordinates are much more complicated, so we use powerful score-based diffusion model 44 for their generation. Next, we will elaborate the details of generating atom types and lattices Section 3.3 and the details of generating coordinates Section 3.4 overview of the proposed SyMat method is illustrated in Figure]

---

3.3 Atom Type and Lattice Generation
For the generation of atom types and lattices, a natural idea is using the VAE model to directly generate and capture the distribution of A and L. However, this fails to capture the invariance to symmetry transformations described in Section [.T] including the permutation invariance of A and rotation invariance of L. To tackle this issue, we transform A and L to items that are invariant to symmetry transformations, and make these items be the direct generation targets of the VAE model. Specifically, for A = [a1,-.., .... an.] we can count the number of atoms for every atom type existed in A, and represent A with an unordered k-element atom type set с = {[e1, m1) (ex, 7. which indicates that for i 1, ...., k. there exist n, atoms with the atom type e, in A. We can easily find that e is invariant to any permutations on A so we use C as the direct generation targets. Besides, for the lattice matrix L [e1, Lz. Lz). we do not generate the three lattice vectors l, l2, Eg directly. Instead, we use six rotation-invann
items as the generation targets, including three lattice lengths l [6,, l2, E2). where &, = ||e/122. and three lattice angles ø [012. 013, Φ23), where Øiy is the angle between L; and lz. Overall, we transform A and L to generation targets c, l, Φ so that the generation process and captured distributions po ()). po LL) are ensured to be symmetry-ave
The VAE model used for generating c, L, and Φ consists of an encoder model and a decoder model. The encoder model is based on a 3D graph neural network (GNN) that takes a material M = (A, P. L) as inputs and outputs a d-dimensional latent variable = E IRd and an f-dimensional latent variable z E R/. This 3D GNN employs the symmetry-aware SphereNet [29] architecture so 2 x will not change if applying any transformations described in Section 3.1 to the input material. We will introduce more details of this 3D GNN model in Appendix B The decoder model is composed of four multi-layer perceptrons (MLPs) including MLP", MLP*. .LP",, MLP", which are used to predict c = {(e1. 71), (Ek, n2)) (ek,7 ). l = [C.. Eg fa), and Φ [612, $13 02sl from the latent variables A and x _ Specifically, let E be the number of all considered atom types and N be the largest possible number of atoms that can exist in material, MLP predicts vector P. E [0. 1] which contains the existing probability for each of the E atom type from the input = and MLP* predicts the size of the set e from the input A The & atom types 61, Gk with top-k probabilities in Pe chosen as the atom types in c, and MLP" predicts the number of atoms R1,.... n2 for them, where each n, is predicted from e; and A. Here, the prediction of Pe is considered as E binary classification tasks for each of E atom types, and the prediction of and 21,... nl are both considered N N-category classification tasks. Besides, MLPé is used to predict lattice items [e, +) = [61, l2, fa. 012, P13, 223] from z², which is considered as a regression task.
We now describe the training and generation process of the VAE model. When training the VAE model on material dataset M, for any material M (A, P. L) in M, the encoder model first maps it to latent variable 2 and zL. Afterwards, the decoder model uses A and z to reconstruct the exact c, l, and & obtained from A and L. The VAE model is optimized to minimize the reconstruction error of c, l, & together with the KL-distance between 2A, =² and the standard Gaussian distribution (0, 1). The reconstruction error losses are defined as cross entropy loss and minimum squared error loss for classification and regression tasks, respectively. After the model is trained, we can generate A and L by first sampling latent variables x x² from (O I), then using the decoder to map V^, xx to C. L, Φ. Finally, we can produce A from {(e1, n21 ) .... (ck, n4)] } as
A le ...y CI, C2 C2 Cbyy Ck
and produce L from l, Φb by constructing three lattice vectors with lengths in l and pairwise angles in Φb The detailed procedure of producing three lattice vectors is described in Appendix
3.4 Coordinate Generation
After the atom type vector A and lattice matrix L are generated, our method will generate the coordinate matrix P conditioned on A and L. We use a score-based diffusion model for coordinate generation. However, because the original score-based diffusion model proposed by Song and Ermon [4 does not consider any invariance to symmetry transformations in data distributions, we cannot directly apply its theoretic framework to coordinate generation. To overcome this limitation, we propose a novel probabilistie modeling process to calculate the score function of P, i.e., the score matrix Vp log P A, L), and design the architecture and training framework of the score model to
5

---

achieve invariance to all symmetry transformations of crystal materials. We will elaborate the details of our proposed coordinate generation framework in the next paragraphs.
We first elaborate the restrictions of the score matrix Vp log p( PIA, L) that is supposed to be captured by the score model. By the following Proposition 3.T (see its proof in Appendix [A.]]. we can show that if the material distribution p(M is invariant to all symmetry transformations described in Section 3.1 the score matrix V p log p( LlA, L) will be invariant translation and periodic transformations, and equivariant to permutation and rotation transformations.
Proposition 3.1. If the material distribution p(M is invariant to permutation, rotation, transla- tion, and periodic transformations described in Section 3.1 V log p( PIA, L) will satisfy the following properties: (1) Vp log pl PlA. L) is equivariant to permulations on A and P; (2)
V VoP+bir and b E R²; (3) LK log p LK A. L) = Vp log (PP)A, holds for any K E Z
+ QL) QV log API L) holds for any Q E Rax3 Q²
To satisfy all properties in Proposition 3.1 we formulate Vp log PIA, L) as function of score functions of edge distances in a graph created by multi-graph method (5T]. For a periodic material M with coordinate matrix P = [Pi.... ....... ......]] and lattice matrix L, multi-graph method produces an n-node undirected graph on M. In this graph, the node corresponds to the i-th atom in the unit cell, and an edge is added between any two nodes i, j if one of the interatomic distances between them is smaller than a pre-defined cutoff r. Here, the interatomic distances between i, j are the distances between their corresponding atoms in the complete infinite structure of M, including both the distance within the unit cell and the distances crossing the unit cell boundary. Formally, the set of {i. k) ||p, + Lk p,ll2 I k E Z 1 S j S n). Note that there may exist more than one edges connecting the nodes i, if more than one interatomic distances between them is smaller than r. Let di, = I|p; + LA - -Pyll be the distance of the edge ji,j, k). we consider log p( P A, L) as function of the distances of all edges in E(M). Denoting 8, as the score function of pa. then Vp log ((P|A, = [$1, s,.].
all edges in the graph constructed on M can be written as E ((M)
From the chain rule of derivatives, we can calculate 8;
Pj + Lk Pj 8i.j.h-
8; = Σ
Velgi log (esP(PL) P)A, V dij.k =
Σ (A))EN())
di.j.
(3)
(L)))N)())
where the scalar 81.j. = Vdl.A log p( Pl PlA,L) is the score function of the distance di and ((1) = (G.k) (i.j.k) E E(M)}. With this probabilistic modeling process, Vp p(g L) can be approximated by first approximating $iJ.A for every edge in the multi graph representation of
M, then calculating each column vector of VP log p( [|| A, L) by Equation (3). By the following Proposition 3.2 (see its proof in Appendix A.2). can show that if Si.j.k is invariant to all symmetry transformations described in Section 3.1 for every edge (i.j, k), the score matrix Vp log (P|A, P|A,L) L) will satisfy all properties in Proposition
3.1
[s1..... ..., sn) where each s; is calculated by Equation (3) on a multi- graph representation of M = (A, (A L). if si.jk k is invariant to permutation, rotation, translation, and periodic transformations described in Section for any edge (i. k), then the calculated Vp log pl Pl PIA, L) will always satisfy the properties in Proposition 3.1
Proposition 3.2. For V log (Ppg,((PAAAA)
In our method, we use a 3D GNN model the score model s+(+) to approximate V Togppp(P ((P|A, L). so(-) takes the multi-graph representation of M as input and outputs oj.j.A as the approximated $i.j every edge (i, j. k) in the input graph. We use SphereNet [29] as the backbone architecture of s-(.). which ensures that the output Oz.j. is always invariant to any symmetry transformation. More details about SphereNet will be introduced in Appendix B To train the score model, we follow the most common operations in denoising score matching framework [44). Specifically, for material M- (A, P, L) in the dataset, we will perturb it to multiple noisy materials by a sequence of or. Let the noisy material at the t-th noise level be M- = (A, the noisy coordinate matrix P [p ... P.] is obtained by adding Gaussian noise o(0.of1) to P. The score model sp iPP,A,,
Gaussian noises with different noise magnitude levels (a:) in which > 02 >
is trained to predict the denoising score matrix V log Po, Here, each 8; is calculated in a similar way as in Equation
[s1,-., ..., 8,) from M. 3 with the approximated distance
score $. = Vā logPo. (PIP,, A, L)). where de = p. + Lk p2112- Motivated from
6

---

Algorithm 1: Langevin Dynamics Sampling Algorithm for Coordinate Generation
Input: A, L, atom number n. noise magnitudes {ot)t= step size c, step number score
model (.(.)
Sample Fn E [R from uniform distribution (O, 1) and set Po LFo
for 1 to 7 do
P. Pt-1
a; c
for j 10 do
Mt (A.
Sample Z RRxn from fromN((0. (0,
Obtain the approximated edge distance score from so(-) which takes M,-1 as inputs
Obtain so (M(M-1 = [ 8n) where each is computed by Equation (3 with edge
distance scores. P: P. (M.80(M)) + Z2,Z
end for end for
Output Pr as the finally generated coordinate matrix
we model the distribution of di.5,k Gaussian distribution N (d. of where the computation process of the mean dijj. and the variance of is described Appendix We can calculate & directly by taking the derivative of the mathematical formula of this Gaussian distribution. The score model s-(-) is trained to minimize the difference
the strategies of existing methods 30 42
55
between the output Ozjik from so(-) and $i,,k k for every edge (i.j, in M with the loss
T
Oi,j.k ō, + dr di.j. o7
Σ
C = 2T
o M.M 1=1
(A..A)E((M)
After the score model is trained, we can use it to generate coordinate matrix P from given A and with annealed Langevin dynamies sampling algorithm [49]. Specifically, we first randomly initializes a coordinate matrix Po. then starting from M, = (A, Po. L), we iteratively update
M = (A, P:-1, L) to M, = (A, L) (t = I.., 2) with the approximated score matrix se sa(M (M, by running multiple Langevin dynamics sampling steps. See Algorithm | for the detailed coordinate generation algorithm pseudocodes.
3.5 Discussions and Relations with Prior Methods
Advantages and limitations. In our method, the atom type vector and lattice matrix are first transformed items that are invariant to symmetry transformations, then VAE model is used to capture the distributions of these items. In addition, our method adopts powerful score-based diffusion models generate atom coordinates, and novel probabilistic modeling framework for score approximation is employed to satisfy all symmetry relattd restrictions. With all these strategies, our method successfully incorporates physical symmetries of periodic materials into generative models, thereby capturing the underlying distributions of material structures more effectively, We will show these advantages through experiments in Section 4 The major limitations of our method lie in that the speed of generating atom coordinates with score-based diffusion models is slow because running thousands of Langevin dynamics sampling steps is needed, and our method cannot be applied to non-periodic materials. We leave more discussions about limitations and broader impacts in Appendix C
Motivations in Model Design. The model design of our SyMat method has the following three moti- vations. (1) First, the structures of atom types and lattices are relatively simple and their symmetry- invariant representations (i.e., atom type sets, lattice lengths and angles) are now-dimessional vectors. Their distributions are simple enough to be captured by VAE models. Also, VAE models are faster than diffusion models in generation speed. Hence, we use VAE models for atom type and lattice generation. (2) Second, the structure of atom coordinates much more complicated than atom types or lattices, use more powerful score-based diffusion models to generate them. More
7

---

importantly, using score- pore-baseddne diffusion models for atom coordinates enables us to trickily convert the prediction of coordinate scores to that of interatomic distance scores by the chain rule of derivatives. This strategy maintains invariance to rotation, translation and periodic transformations. We think it is hard to develop such a symmetry-awar probabilistic modeling for other generative models. Hence, we use bosee diffusion models for coordinate generation. (3) Third, SphereNet is chosen as the backbone network of our model. SphereNet is a 3D graph neural network for 3D graph representation learning. It has powerful capacities in 3D structure feature extraction because it considers complete 3D geometric information contained in pairwise distances, line angles and plane angles (torsion angles). Also, the features extracted by SphereNet are invariant to rotation, translation and periodic transformations when input 3D graphs are created by multi multi-graph method. Because of its powerful capacity and symmetry-awend feature extraction, we use SphereNet the backbone network.
Relations with prior methods. Here we will discuss several studies that are related to our work. Some studies 42 [3)] have also applied score-based diffusion models to generate atom coordinates from molecular graphs, but there are not periodic structures in 3D molecules. Differently, keeping invariant to periodic transformations is significant in periodic material generation, and our score-based diffusion model uses periodic invariant multi-graph representations of materials as inputs to satisfy this property. Besides, compared with recently proposed periodic material generation method CDVAE [52], our SyMat has two major differences. (1) First, to generate the atom type vector, CDVAE generates total atom numbers and compositions (the proportion of atom numbers for every atom type to the total atom number) by a VAE model, in which compositions are vectors composed of continuous numbers. While compositions have complicated structures and are hard to predict, SyMat uses relatively simpler atom type sets as generation targets and the prediction of them can be easily converted to integer prediction classification problems (see Section 3.3 (2) Second, in coordinate generation, CDVAE also uses the denoising score matching framework, but the denoising score matrix V s log Po. (PIP. A, L) is calculated as the direct difference between P and a coordinate matrix obtained by aligning P, which makes Vp log (PIP, A, L) not invariant to translations on P. However, SyMat calculates it from the edge distance score functions ensure invariance to all symmetry transformations (see Section 3.4p. In addition, another two studies, PGD- VAE [47] and DiffCSP [8], are also related to our SyMat approach. PGD-VAE proposes VAE model for periodic graph generation. However, PGD-VAE VAE is fundamentally different from SyMat in that only generates 2D periodic topology graphs without 3D structures, while SyMat is developed for 3D periodic material structures. PGD- VAE cannot be applied to 3D periodic materials so we do not compare SyMat with PGD-VA VAE in our experiments. DifICSP proposes novel method for generating 3D periodic material structures from their atom types. Different from SyMat, DifICSP uses denoising diffusion models to jointly generate lattices and atom coordinates. In addition, DiffCSP does not generate atom types of periodic materials, but lattices and atom coordinates from the input atom types, so DiffCSP cannot be applied design novel periodic materials from scratch. Hence, do not compare SyMat with DifiCSP in our experiments as all our experiments evaluate the performance of different methods in generating novel periodic materials.
4 Experiments
In this section, we evaluate our proposed SyMat method in two periodic material generation tasks, including random generation and property optimization. We show that our proposed SyMat can achieve promising performance in both tasks.
4.1 Experimental Setup
Data. We evaluate SyMat on three benchmark datasets curated by Xie et al. [52] including Perov- 5 2 Carbon-24 [35] and MP-20 [17. Perov-5 is a dataset with collection of 18,928 perovskite materials. The chemical element compositions of all materials in Perov-5 can be denoted by general chemical formula ABX;, which means that there are 3 different atom types and 5 atoms in each unit cell. The Carbon-24 dataset we used has 10,153 materials. All these material structures consist of only carbon elements and have 6 24 atoms in each unit cell, and the 3D structures of them are obtained by DFT simulation. MP-20 is a dataset curated from Materials Project [17 library. includes 45,231 materials with various structures and compositions, and in all materials, there exist at
8

---

Table 1: Random generation performance Perov-5, Carbon-24, and MP-20 datasets. Here means higher metric values lead to better performance, while & means the opposite. Because all materials in the Carbon-24 dataset are composed of only carbon atoms, all methods easily achieve 100% in composition validity and 0.0 in # element EMD so we do not use these two metrics. Also, Cond-DFC-VAL can only be used to Perov-5 dataset in which materials have cubic structures. Bold and underline numbers highlight the best and second best performance, respectively.
Dutaset
Method
Composition validity
Structure validity"
Element EMD+
Density EMD
Energy EMD+
COV-R
COV-pt
FTCP
54.24%
0.24%
0.6297
10.27
156.0
0.00%
0.00%
Cond-DFC-VAS VAE
82.95%
73.60%
0,8373
2.268
4.111
77.80%
12.38%
Perov-5
G SchNet
98.79%
99.92%
0.0368
1.625
4.746
0.25%
0.37%
P-G-SchNet
99.13%
79.63%
0,4552
0.2755
1.388
0.56%
0.41%
CDVAE
98.59%
100.00%
0.0628
0.1258
0.0264
99.50%
98.93%
SyMat (ours)
97.40 %
100.00% %
0.0177
0.1893
0.2364
99.68%
98. 64%
FTCP
0.08%
5,206
19.05
0.00%
0.00%
G-SchNet
99
0.9427
1.320
0.00%
0.00%
Carbon- 24
P-G SchNet
48.39%
1.533
134.7
0.00%
0.00%
CDVAF
100.00 %
0.1407
0.2850
100.00 %
99.98%
SyMat (ours)
100.00%
0.1195
3.9576
100.00% %
59.E
FTCP
48.37%
1.55%
0.7363
23,71
160.9
5.26%
0.23%
G-SchNet
75.96%
99.65%
0.6411
3.034
42.09
41 68%
99. 65%
MP-20
P-G-SchNet
76.40%
77.51%
0.6234
4.04
2.448
44.89%
99. 79.76%
CDVAE
86.70%
100.00 %
1.432
0.6875
0.2778
99.17%
99 644%
SyMat (ours) 88.26%
100.00 %
0.5067
0.3805
0.3506
98.97%
99.97%
most 20 atoms in each unit cell. For all three datasets, we split them with ratio of 3:1:1 as training, validation, and test sets in our experiments.
Tasks, We evaluate the performance of SyMat in random generation and property optimization tasks. In the random generation task, SyMat generates novel periodic materials from randomly sampled latent variables and we employ variety of validity and statistic metrics to evaluate the quality of the generated periodic materials. In the property optimization task. SyMat is evaluated by the ability of discovering novel periodic materials with good properties. Besides, to show that latent representations are informative enough to reconstruct periodic materials, we also evaluate the performance by the material reconstruction task. See Appendix D.2 for experiment results of this task.
Baselines. We compare SyMat with two early periodic material generation methods FTCP 36) and Cond-DFC-VAE (6). which generate material structures through generating Fourier-transfommed crystal property matrices and 3D voxel images with VAE models, respectively. Note that Cond- DAC-VAE can only be used to generate cubic systems, so it is only used for Perov-5 dataset in which materials have cubic structures. In addition, we compare with an autoregressive 3D molecule generation method G-SchNet [10]. and its variant P-G-SchNet that incorporates periodicity in G- SchNet. The latest periodic material generation method CDVAE (52) is also compared with our method. Note that all these baseline methods are compared with SyMat in the random generation task, but G-SchNet and P-G-SchNet are not compared in the property optimization task because they cannot produce latent representations with fixed dimensions for different periodic materials.
4.2 Random Generation
Metrics. In the random generation task, we adopt seven metries to evaluate physical and statistic properties of materials generated by our method and baseline methods. We evaluate the composition validity and structure validity, which are the percentages of generated materials with valid atom type vectors and 3D structures, respectively. Here, an atom type vector is considered as valid if its overall charge computed by SMACT package X is neutral, and following Court et [ a 3D structure is considered valid if the distances between any pairwise atoms larger than 0.5A. In addition, we evaluate the similarity between the generated materials and the materials in the test set by five statistic metries, including the earth mover's 's distance (EMD) between the distributions in chemical element number, density (g/cm²) 2), and energy predicted by a GNN model. Also, we evaluate the percentage of the test set materials that cover at least one of the generated materials (COV-R), and the percentage of generated materials that cover at least one of the test set materials (COV-P). All seven metrics are evaluated on 10,000 generated periodic materials. See Appendix [D.T for more information about evaluation metries and experimental settings in the random generation task.

---

Table 2: Property optimization performance on Perov-5, Carbon- 24, and MP-20 datasets. We report the success rate (SR), i.e., the rate of periodic materials that can achieve top 5%, 10%, and 15% of the property distribution in the dataset after being optimized. Cond-DFC- VAE can only be used to Perov-5 dataset in which materials have cubic structures. Bold and underline numbers highlight the best and second best performance, respectively.
Perov-5
Carbon
24
MP-20
Method
SR5
SR10
SR15
SR5
SR10
SR15
SR5
SRI0
SR15
FTCP
0.06
0.11
0.16
0.00
0.00
0.00
0.02
0.04
0.05
Cond-DFC-VAD VAE
0.55
0.64
0.69
CDVAE
0.52
0.65
0.79
0.00
0.06
0.06
0.78
0.86
0.90
SyMat (ours)
0.73
0.80
0.87
0.06
0.13
0.13
0.92
0.97
0.97
The random generation performance results of all methods on three datasets are presented in Table Among the total 19 metrics over three datasets in the table, our SyMat method achieves top- 1 performance in 11 metrics and top-2 performance in 17 metrics. Particularly, SyMat can achieve excellent performance in composition and structure validity, which demonstrates that Sy Mat can accurately learn the rules of forming chemically valid and stable periodic materials so it generates chemically valid periodic materials with high chances. Overall, SyMat achieves promising perfor- mance in the random generation task, showing that SyMat is an effective approach to capture the distribution of complicated periodic material structures.
Results.
We visualize some generated periodic materials in Figure of Appendix and use additional metrics to evaluate the diversity of the generated periodic materials in Appendix D.3 To demonstrate the effectiveness of coordinate generation module in SyMat, we conduct an ablation study using several evaluation metries in the random generation task and present this part in Appendix
D.4 D.4
4.3 Property Optimization
the property optimization task, we evaluate the ability of SyMat in discovering novel periodic materials with low energies. We jointly train the SyMat model with an energy prediction model that predicts the energies from the latent representations. Afterwards, we follow the evaluation procedure in [52) to optimize the latent representations of testing materials with 5000 gradient descent steps, decode 100 periodic materials from the optimized latent representations to periodic materials, and report the success rates (SR), i.e., the rate of decoded materials whose properties in the top 5%, 10%, and 15% of the energy distribution in the dataset. We use the same evaluation procedure to baseline methods and compare them with SyMat in optimization success rates.
Metrics.
Results. The property optimization performance results of all methods on three datasets are summa- rized in Table 2 On three datasets, SyMat outperforms all baseline methods in optimization success rates, showing that SyMat has the highest chances to generate periodic materials with low energies. This demonstrates that SyMat models have strong capacities in discovering novel periodic materials with good properties. The good performance of SyMat in the property optimization method shows that it can be effective tool for designing novel periodic materials with desired properties.
We visualize the changes of material structures and energies of some periodic materials after property optimization in Figure of Appendix B We also visualize the material energy distribution before and after property optimization by SyMat models in Figure of Appendix
5 Conclusion
We propose SyMat, a novel deep generative model for periodic material generation. SyMat achieves invariance to physical symmetry transformations of periodic materials, including permutation, ro- tation, translation, and periodic transformations. In SyMat, atom types and lattices of materials are generated in the form of atom type sets, lattice lengths and lattice angles with a VAE model. Additionally, SyMat uses a score-based diffusion model based on a symmetry-awar coordinate diffusion process to generate atom coordinates. Experiments on random generation and property optimization tasks show that SyMat a promising approach for discovering novel materials. In the future, we will study the problem of accelerating atom coordinate generation in SyMat.
10

---

Acknowledgments and Disclosure of Funding
We thank Xiang Fu for his help on providing technical details about running baseline methods. This work was supported in part by National Science Foundation under grants IIS-1908220 and IIS-2006861.
References
[1] Keith T Butler, Daniel W Davies, Hugh Cartwright, Olexandr Isayev, and Aron Walsh. Machine learning for molecular and materials science. Nature, 559(77155:547-555, 2018.
[2] Ivano Castelli, David Landis, Kristian Thygesen, Søren Dahl, Ib Chorkendorff, Thomas Jaramillo, and Karsten W Jacobsen. New cubic perovskites for one-and two-photon water splitting using the computational materials repository. Energy & Environmental Science, 5(10); 0034-9043, 2012.
[3] Ivano E Castelli, Thomas Olsen, Soumendu Datta, David D Landis, Søren Dahl, Kristian S Thygesen, and Karsten W Jacobsen, Computational screening of perovskite metal oxides for optimal solar light capture. Energy & Environmental Science, 5(2):
2012.
[4] Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. Graph networks universal machine learning framework for molecules and crystals. Chemistry of Materials, 31 (9):3564-3572,) 2019.
[5] Kamal Choudhary and Brian DeCost. Atomistic line graph neural network for improved materials property predictions. npj Computational Materials, 7(1):1-8, 2021.
[6] Callum Court, Batuhan Yildirim, Apoorv Jain, and Jacqueline M Cole. 3-D inorganic crystal structure generation and property prediction via representation learning. Journal of chemical information and modeling, 60(10):4518 4535, 2020.
[7] Yabo Dan, Yong Zhao, Xiang Li, Shaobo Li, Ming Hu, and Jianjun Hu. Generative adversarial networks (GAN) based efficient sampling of chemical composition space for inverse design of inorganic materials. upj Computational Materials, 6(1):1-7, 2020.
[8] Daniel W Davies, Keith Butler, Adam Jackson, Jonathan M Skelton, Kazuki Morita, and Aron Walsh. SMACT: Semiconducting materials by analogy and chemical theory. Journal of Open Source Software, 4(38):3311, 2019.
[9] Mildred Dresselhaus, Gene Dresselhaus, and Ado Jorio. Group theory: application to the physies of condensed matter. Springer Science & Business Media, 2007.
[10] Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmetry-adaptedd generation of 3d point sets for the targeted discovery of molecules. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[11] Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Doina Precup and Yee whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Pro- ceedings of Machine Learning Research, pages 1263-1272, International Convention Centre, Sydney, Australia, 2017.
[12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27, pages 2672-2688 Curran Associates, Inc., 2014.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic mod- els. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, edi- tors, Advances in Neural Information Processing Systems, volume 33, pages 6840- 6851. Curran Associates, Inc., 2020. URL. httpp://wooeeednnsseeeeeeiccc/pp Wc5befece55Aatttoo77Tlbbbllll 79c444 4

---

[14] Jordan Hoffmann, Louis Maestrati, Yoshihide Sawada, Jian Tang, Jean Michel Sellier, and Yoshua Bengio. Data-driven approach to encoding and decoding 3-D crystal structures. arXiv preprini arXiv: 0094,, 2019.
[15] Emiel Hoogeboom, Víctor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant dif- fusion for molecule generation in 3D. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Confer- ence on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 8867-8887. PMLR, 17-23 Jul 2022. URL http:://poeecddnnsssmippss/////s wttps//ewww///////ooo/ooooooo html
[16] John Ingraham, Max Baramov, Zak Costello, Vincent Frappier, Ahmed Ismail, Shan Tie, Wujie Wang, Vincent Xue, Fritz Obermeyer, Andrew Beam, and Gevorg Grigoryan. Illuminating protein space with a programmable generative model. bioRxiv, 2022. doi: 10.1101/2022.12. 12.00. 518682.
URL http:///www.bbbaaoooo////tttlltt 22001111882
[17] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. Commentary: The materials project: A materials genome approach accelerating materials innovation. APL materials, 1(1):01 1002, 2013.
[18] Rui Jiao, Wenbing Huang, Peijia Lin, Jiaqi Han, Pin Chen, Yutong Lu, and Yang Liu. Crystal structure prediction by joint equivariant diffusion lattices and fractional coordinates. In Workshop on "Machine Learning for Materials" ICLR 2023, 2023. URL https://openteeiiww Ci
[19] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junetion tree variational autoencoder for molecular graph generation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2323-2332. 2018.
[20] Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi S. Jaakkola. Torsional diffusion for molecular conformer generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL
[21] Sungwon Kim, Juhwan Noh, Geun Ho Gu, Alan Aspuru-Guzik, and Yousung Jung. Generative adversarial networks for crystal structure prediction. ACS central science, 688):1412-1420, 2020.
[22] Diederik Kingma and Max Welling. Auto-Encoding variational bayes. In 2nd International Conference on Learning Representations, 2014.
[23] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. On density estimation with diffusion models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL htt.h//s//oppeeeviee.ete/otooooe
id=21 dB
[24] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. URL http::///ppaaevieeee///////////a
xFK5
[25] Yuchao Lin, Keqiang Yan, Youzhi Luo, Yi Liu, Xiaoning Qian, and Shuiwang Ji. Efficient approximations of complete interatomic potentials for crystal property prediction. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors. Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pages 21260-21287. PMIR, 23-29 Jul 2023. URL
[26) Meng Liu, Youzhi Luo, Limei Wang, Yaochen Xie, Hao Yuan, Shurui Gui, Haiyang Yu, Zhao Xu, Jingtun Zhang, Yi Liu, Keqiang Yan, Haoran Liu, Cong Fu, Bora M Oztekin, Xuan Zhang, and Shuiwang Ji. DIG: A turnkey library for diving into graph deep learning research. Journal of Machine Learning Research, 22(240):1-9, 2021. URL http://mmmlarggaaaeew22222-1-110
12

---

[27] Meng Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. GraphEBM: Molecular graph generation with energy-based models. In Energy Based Models Workshop-ICLR 2021, 2021.
[28] Meng Liu, Youzhi Luo, Kanji Uchino, Koji Maruhashi, and Shuiwang Ji. Generating 3D molecules for target protein binding. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari. Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 13912-13924, PMLR, 17-23 Jul 2022. URL https://proceedings wwt.press/vvee 162/hu22m html
[29] Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3D molecular graphs In International Conference on Learning Representations, 2022. URL https///oeeneeeeeeeeee////onnndd
[30] Shitong Luo, Chence Shi, Minkai Xu, and Jian Tang. Predicting molecular conformation via dynamic graph score matching. In M. Ranzato, Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neurul Information Processing Systems, volume 34, pages 19784-19795. Curran Associates, Inc., 2021. URL /ttt///ss///ooeedigseerrrppppppp 202 wwweeeeeff11222aaalo da 81889999999.9ppp...
[31] Youzhi Luo and Shuiwang Ji. An autoregressive flow model for 3D molecular geometry generation from scratch. In International Conference on Learning Representations, 2022. URL
https:/openevvie.neefffnmm http://wwwe.ooommmoooo//////A// Cid=Cosjj--NSS
[32] Youzhi Luo, Keqiang Yan, and Shuiwang Ji. GraphDF: A discrete flow model for molecular graph generation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th Inter- national Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7192-7203. PMLR, 18-24 Jul 2021.
[33] Elman Mansimov, Omar Mahmood, Seokho Kang, and Kyunghyun Cho. Molecular geometry prediction using a deep generative graph neural network. Scientific reports, 9(1):1-13, 2019.
[34] Yashaswi Pathak, Karandeep Singh Juneja, Girish Varma, Masahiro Ehara, and U Deva Priyaku- mar. Deep learning enabled inorganic material generator. Physical Chemistry Chemical Physics, (((46):2693526944 2020.
[35] Chris J. Pickard. Airss data for carbon at 10gpa and the c+n+h+o system at Igpa, 2020. URL
[36] Zekun Ren, Siyu Isaac Parker Tian, Juhwan Noh. Felipe Oviedo, Guangzong Xing, Jiali Li, Qiaohao Liang, Ruiming Zhu, Armin G. Aberle, Shijing Sun, Xiaonan Wang, Yi Liu, Qianxiao Li. Senthilnath Jayavelu, Kedar Hippalgaonkar, Yousung Jung, and Tonio Buonassisi, An invert- ible crystallographic representation for general inverse design of inorganic crystals with targeted properties. Matter, 5(1):314-335, 2022. ISSN 2590- 2385. doi: http:://doi.on org/ 101 616.66/.....
2021.11.032.
[37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[38] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://ppenceiie..eeffommm -id- n5-
[39] Victor Garcia Satorras, Emiel Hoogeboom, Fabian Bernd Fuchs, Ingmar Posner, and Max Welling. E(n) equivariant normalizing flows. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL
13

---

[40] Jonathan Schmidt, Love Pettersson, Claudio Verdozzi, Silvana Botti, and Miguel A. L. Marques. Crystal graph attention networks for the prediction stable materials. Science Advances, 7(99): eabi7948, 2021. doi: 10.1 126/sciadvabbi7958
[41] Chence Shi*, Minkai Xu Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. GraphAF: a flow-based autoregressive model for molecular graph generation. In 8th Interna- tional Conference on Learning Representations. 2020.
[42] Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation generation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9558-9568. PMLR, 18-24 Jul 2021.
[43] Gregor Simm and Jose Miguel Hernandez-Loobtto. A generative model for molecular distance geometry. In Hal Daumé III and Aarti Singh, editors, Proceedings the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1949-8955 PMLR, 13-18 Jul 2020.
[44] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL http:://wpocccci.s....uuscccaaap 3001ef257400d5537 hatp://5337770aa0000447700000000
[45] Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021. URL https://openneviiew.nee/
forum''d==XTIGI2RRRSS
[46] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-1674, 2011.
[47] Shiyu Wang, Xiaojie Guo, and Liang Zhao. Deep generative model for periodic graphs. In Alice H. Oh. Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https///ppeneeiiieweettfffrmmm lgNGDjWRTo-
[48] Joseph L. Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas Milles, al. De novo design of protein structure and function with rfdiffusion. Nature, 62897976):008-11100 2023.
[49] Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient Langevin dy- namics. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11, page 681-688, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306195.
[50] Lemeng Wu, Chengyue Gong, Xingchao Liu, Mao Ye, and qiang liu. Diffusion-based molecule generation with informative prior bridges. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information P Processing Systems, 2022. URL https:/ppparrevie..eeffffumm Pidffforuunnd-TTTUNNNIIII
[51] Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for accurate and interpretable prediction of material properties. Physical review letters, 120( (20(14):14530) 2018.
[52] Tian Xie, Xiang Fu, Octavian -ugann Ganea, Regina Barzilay, and Tommi S. Jaakkola. Crystal diffusion variational autoencoder for periodic material generation. In International Conference on Learning Representations, 2022. URL
[53] Minkai Xu, Shitong I Luo, Yoshua Bengio, Jian Peng. and Jian Tang. Learning neural generative dynamies for molecular conformation generation. In International Conference on Learning Representations, 1021. http:://wwwaaawwwwwwooo//o////ea
14

---

[54] Minkai Xu, Wujie Wang. Shitong Luo, Chence Shi. Yoshua Bengio, Rafael Gomez-Bombardee Bombarelli, and Jian Tang. An end-to-end framework for molecular conformation generation via bilevel programming. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11537-11547, PMLR, 18-24 Jul 2021.
[55] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. GeoDiff: A geometric diffusion model for molecular conformation generation. In International Conference on Learning Representations, 2022. URL
[56] Zhao Xu, Youzhi Luo, Xuan Zhang, Xinyi Xu, Yaochen Xie, Meng Liu, Kaleb Dickerson, Cheng Deng, Maho Nakata, and Shuiwang Ji. Molecule3D: A benchmark for predicting 3D geometries from molecular graphs. arXiv preprint arXiv:21 2021.
[57] Keqiang Yan, Yi Liu, Yuchao Lin, and Shuiwang Ji. Periodic graph transformers for crys- tal material property prediction. In Alice H. Oh, Alekh Agarwal. Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022. URL
[58] Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, pages 6410-6421. Curran Associates, Inc., 2018.
[59] Anthony Zee. Group theory in a nutshell for physicists, volume 17, Princeton University Press, 2016.
[60] Xuan Zhang, Limei Wang, Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, Meng Liu, Yuchao Lin, Zhao Xu, Keqiang Yan, Keir Adams, Maurice Weiler, Xiner Li, Tianfan Fu, Yucheng Wang, Haiyang Yu, YuQing Xie, Xiang Fu, Alex Strasser, Shenglong Xu, Yi Liu, Yuanqi Du, Alexandra Saxton, Hongyì Ling, Hannah Lawrence, Hannes Stärk, Shurui Gui, Carl Edwards, Nicholas Gao, Adriana Ladera, Tailin Wu, Elyssa F. Hofgard, Aria Mansouri Tehrani, Rui Wang, Ameya Daigavane, Montgomery Bohde, Jerry Kurtin, Qian Huang, Tuong Phung, Minkai Xu, Chaitanya K. Joshi, Simon Mathis, Kamyar Arizzadenesheli, Ada Fang, Alán Aspuru-Guzik, Erik Bekkers, Michael Bronstein, Marinka Zitnik, Anima Anandkumar, Stefano Ermon, Pietro Liò, Rose Yu, Stephan Günnemann. Jure Leskovee, Heng Ji. Jimeng Sun, Regina Barzilay, Tommi Jaakkola, Connor W. Colcy, Xiaoning Qian, Xiaofeng Qian, Tess Smidt, and Shuiwang Artificial intelligence for science quantum, atomistic, and continuum systems. arXiv preprint w:Xiv:2307.008233 2023.
[61] Yong Zhao, Mohammed Al-Fahdi, Ming Hu,, Edirisuriya MD Siriwardane, Yuqi Song, Alireza Nasiri, and Jianjun Hu. High-throughput discovery of novel cubic crystal materials using deep generative neural networks. Advanced Science, 8(20):2100566 2021.
15

---

A Proof of Propositions
A.1 Proof of Proposition 3.1
We first prove Proposition 3.1 i.e., if the material distribution p(M) is invariant to permuta- tion, rotation, translation, and periodic transformations described in Section 3.1 the score matrix Vp log p P/ L) will be invariant to translation and periodic transformations, and equivariant to permutation and rotation transformations.
(1) For M' = (AA', P', L) where A' and P' are obtained by permuting the elements of A and column vectors of P with the same order, from p(M) p (M"), we know that p( A'' L) = ppPP A, A L) holds, so P log p( P' L) actually equals to the matrix obtained by permuting the column vectors of Vp p log p( P| P|A, L) with the same permutation order. In other words, Vp log (PlA, L)
is permutation-equuiarrinn
(2) For M' = (A, PP, Lt) obtained by rotating M (A. with the rotation matrix Q and translating it with the translation vector b, because p(M') = p(M and P' = QP + b1 b1" we have p( P' |A, L') p(l|A, L) and V log (PP A, L') = QVp log (PlA, L). In other words,
Vp log p A, L) is cranslation-ivvarii and rotation-equivariant.
(3) For M
(A,PA L) where pk P+ LK is obtained by periodic transfor- mation, since p(M) = -pp((MM p(MK), we can easily find that p pP A, L) = p( p(P|A, L) and V px log (PppkK |A, L) = Vp log L) hold. In other words, Vp logo is invari-
ant to periodic transformations.
A.2 Proof of Proposition 3.2
We next prove Proposition 3.2 i.e., if St.j.k is always invariant to permutation, rotation, translation, and periodic transformations described Section 3.1 the score matrix Vp log ((lug(((AAA)) P|A, L) computed by Equation (3) will always be invariant to translation and periodic transformations, and equivariant to permutation and rotation transformations.
(L) The score matrix Vp lo P(IA, L)= [si..... s..] is formed by stacking coordinate score vector 8, according to the atom order, and this stacking operation is clearly equivariant to permutations. Also, from Equation (3), we can easily find that if stjjk is always invariant to permutation, the computed coordinate score vector s; for each coordinate p, will also be invariant to permutations. Hence, Vp log (PllA. L) from $1...., Sn is equivariant to permutations.
R 3x3 Q Q I and b R3. let P' [p's ..., P'J p'- Qp: + b ,.... 1, n,
(2) For any Q
QP b1 where L' = QL. and M' (A, L'). We can calculate the score vector sf
for pl in the way similar to Equation
s pp (ogg((P(A,,)))
Σ
V&, S.
log P L') V..
+ L'k -
Σ
1p + L'k 10--pp. -D'Il2'
(1))
QP, - b 11p, bb+ QLK QP, b112'
Qp: + b QLk
Σ
(1))
Q(p: Lk P.)) |pp + Lk -1
Σ
Si.j.
(LA))N())
Qsi,
where (' (G,k) : (i,j,k) E(M')) (i) k(,k) (L,j,)) E E(M)). Note that the
second to the last equation holds because ||@0| 2 = ||ズ||z holds for any E R if Q I, and the edge set of multi-graph, representation and Si.j. is invariant to any rotation and transla- tion transformations, hence E(M') = E(M and $.j,k Sij.A. So far, we have shown that V AA L) is invariant to translations and equivariant to rotations.
16

---

+ (3) For any K E Z let p [ - LK and M = (A, L). Because the multi- graph method is strictly invariant periodic transformations, there must exist bijective mapping f: ((M) + E (MK) such that for any (i, k) E E(M), (i, k') = [ J. k) satisfies that
P. Lk - p LK' - P)j Also, because st & is invariant to periodic transformations, we
Lk can easily find that calculating the score vectors for ps and p by Equation 33) are actually summing up the same of vectors, these score vectors equal and then Vp log A, L) is invariant to periodic transformations.
17

---

B Additional Implementation Details
SphereNet Model. In SyMat, we use SphereNet model 2 as the backbone GNN architecture for both the encoder model in the VAE for atom type and lattice generation and the score model in the score-based diffusion model for atom coordinate generation. SphereNet is a powerful 3D GNN model proposed for 3D molecular property prediction, but in our method, we will use the multi-graph representations materials as inputs to SphereNet models. Generally, SphereNet model first initializes the embedding vectors of nodes and edges with look-up atom type embedding vectors and spherical basis functions, then employ multiple message passing [1T] layers to iteratively update the node and edge embeddings. Note that this overall process of initializing and updating embeddings is always invariant to all symmetry transformations described in Section 3.1 The encoder model of VAE model will use sum pooling over the final node embeddings to obtain global representation vector, which is passed into an MLP to obtain V^ and z²_ The score model will use another MLP to predict the edge distance score scalar from the final edge embeddings. Note that independent SphereNet models whose parameters are not shared for the VAE encoder model and score model. We use the implementations of SphereNet in DIG package [26].
two
Obtaining lattice matrix from lattice lengths and angles. From the lattice lengths = [61, E2, Eg] and lattice angles & [012, 223, @1a]. let 7 = arccos cos dbga cus d13 101--ome -cos d21,, L = 1e,, L2, Es] can be computed as
one lattice matrix
sim (22 sin 918
f [e1 sin Ø13, 0, l cos 01s)";
L Eg sin d22 cos 7/ lg sin 022 sin y, l2 cos d2s)",
l3 = [0, 0, 6s)"
Calculating the denoising score matrix. In the training of score-based diffusion model in SyMat, add Gaussian noise with noise magnitude og to the coordinate matrix of a material M (A, P. L)
in the dataset to obtain a noisy material M = (A. L. and the score model se(-) is optimized to predict the denoising score matrix V logPo. (PIP, A.P. Then the score model s.(:) tries (PIP,A,L) for every edge (i, k) in in molecule generation, we can assume that the distribution edge distance d'.j.k in M is a Gaussian distribution find that using coordinate matrix P f Pn] obtained by aligning P [p..... Pn. w.r.t. P (p..... .... Pal to calculate the corresponding edge distance in M can achieve the best performance. Specifically, we follow alignment procedure as in CDVAE [52] and calculate each p; as p; = Pt + Lu, where = arg min
to predict the edge distance score &ij.k Vā.,, logPo. Po.
the multi-graph representation of M. Motivated from existing methods 30 42 55
centered around the edge distance of the same edge in M. Practically,
|lp: + L p:l12. The mean d the Gaussian distribution of the edge distance ds.j. is calculated as di.j, μp; + Lk 1,112. Besides, we assume the Gaussian distributions of di.jA for every edge have the same standard deviations ot, which is pre-computed by manually perturbing all materials in the dataset to noisy materials, collecting the edge distances in all noisy materials, and calculating the empirical standard deviation dz.
18

---

C More Discussions
Limitations. Generally, there are three major limitations in our proposed SyMat approach. (1) First, SyMat may take as long as an hour to generate atom coordinates of periodic materials with score-based diffusion models, since this process requires run thousands of Langevin dynamics sampling steps. Actually, this is a common limitation for many other molecule or material generation using score-based diffusion models. Nonetheless, we argue that compared with the huge time cost of laborious lab experiments or DFT based calculation, the time cost of SyMat relatively lower and acceptable. We will explore accelerating SyMat with SDE based diffusion models [45] (3) in the future. (2) Second, SyMat is designed to maintain invariance to periodic transformations, so it cannot be applied to non non-periodic materials as this will lead to incorrect probabilistic modeling. Though some materials are non non-periodic, such as amorphous solids and polymers, we focus on periodic materials because they occupy a large part of materials used in real-world applications. We believe that developing non-periodic material generation method need to rely on effective periodic material generation methods, but the problem of generating relatively simpler periodic materials still remains challenging and largely unsolved. Hence, will focus generating periodic materials in this work and leaves the problem of generating non-periodic materials the future. (3) Third, our work follows previous material generation studies [52] evaluate the quality of the generated materials by some basic metrics, such as composition and structure validity. However, they do not use metrics evaluate the synthesizability of the generated materials, though these metries are useful for practical applications. In the future, we will collaborate with experts in material science to come up with metrics for evaluating synthesizability of the materials generated by our method.
methods [42 30 52
Broader impacts. In this work, we propose novel method for periodic material generation. Our work can be used to design novel materials with desired properties for many real-world applications, such batteries or solar cells. Nonetheless, our method may unexpectedly generate materials that produce negative impacts to human life. For example, manufacturing some materials generated by model may cause environmental pollution. Hence, believe strict evaluation or validation process need to be done to estimate the related environmental or other social impacts before using the materials generated by our model real-world scenarios.
19

---

D Additional Experiment Details and Results
D.1 Additional Details in Experimental Settings
In our SyMat model, the SphereNet model is composed of 4 message passing layers and the hidden size is set to 128. Also, all MLP models in the VAE decoder is composed of linear layers with ReLU function between them, and the hidden size is set to 256. During training, we set the learning rate to 0,001, the batch size to 128, and the epoch number to 1,000. The total training needs around 5, and 10 hours on GTX 2080 GPU for Perov-5, Carbon-24, and MP-20 datasets, separately. We use different weights for different loss terms. Specifically, we set the weights of the reconstruction losses for atom type set size, atom types, atom numbers of each atom type, lattice items 1.0, 30.0, 1.0, 10.0, respectively, and use a weight 0.01 for KL-distance loss and weight of 10.0 for denoising
matching loss. In the generation of atom coordinates with Algorithm T we step size e as 0.0001 and step number q as 100. In addition, we use a geometrically decreasing series between 10 and 0.01 {or) and set T 50. In the random generation task, for COV-R and COV-P metrics, we consider one material cover another material if the distances of their chemical element composition fingerprints and 3D structure fingerprints
smaller than the thresholds å, and 6.. respectively. For Perov-5 dataset, we use 6, 6, 6. = 0.8; for Carbon-24 dataset, we use 6, = 4,
5. 1.0; for MP-24 dataset, we use 6 12, 8g 0.6.
D.2 Material Reconstruction
In the material reconstruction task, we reconstruct materials in the test sets of Perov-5, Carbon- 24 and MP-20 datasets from the encoded latent representations A and =². We evaluate the reconstruction performance by several metrics. First, evaluate the percentage of materials whose atom types can be exactly reconstructed (Atom Type Match Rate), and the root mean square error in lattice lengths and angles between the target and reconstructed materials (Lattice RMSE). Second, to show the score-based diffusion model has the capacity to approximately reconstruct 3D structures, we evaluate the root mean square error in interatomic distances between the target and reconstructed materials (Distance RMSE). We present the experimental results of SyMat together with baseline methods in Table 3 Results show that method performs well in atom types, lattices and interatomic distances reconstruction.
Table 3: Reconstruction performance Perov-5, Carbon-24, and MP-20 datasets. Here t means higher metric values lead to better performance, while + means the opposite. Note that Cond-DFC. VAE can only be used to Perov-5 dataset in which materials have cubic structures. Bold and underline numbers highlight the best and second best performance, respectively.
Atom type match rate
Dataset
Method
Lattice RMSE!
Distance RMSE
FTCP
99.67 %
0.0786
0.2953
Cond-DFC-VAE
58.92%
0.0765
0.2281
Perov-5
CDVAE
98.16%
0.0231
0.1072
SyMat (ours)
98.30%
0.0224
0.0723
FTCP
100.00 %
0.1349
0.3759
Carbon-24
CDVAR
100.00 %
0.0624
0.1745
SyMat (ours)
100.00%
0.0632
0.1186
FTCP
71.46%
1057
0.2062
MP-20
CDVAE
68.32%
0.0532
0.0975
SyMat (ours)
72.10%
0.0510
0.0876
D.3
Diversity Evaluation
We evaluate the diversity of the randomly generated atom types by uniqueness percentage, i.e., percentage of unique atom types among all the generated atom types. We present the uniqueness percentages of SyMat and baseline methods on Perov-5 and MP-20 datasets Table Note that Cond-DFC-Van VAB can only be used to Perov-5 dataset in which materials have cubic structures, and
20

---

Carbon-24 is not used here because all materials in Carbon-24 are composed of only carbon atoms, so the uniqueness is expected be low. Results show that our method performs well generating materials with diverse atom types.
Table 4: Uniqueness percentages on Perov-5 and MP-20 datasets.
Dataset
FTCP
Cond-DFC-VAE G-SchNet
P-G-SchNet
CDVAR
SyMat
Perov-5
72.33% 79.04%
80.36%
98.74% 99.23%
98.57% 99.03%
98.61% %
99.43 %
MP-20
99.84%
99.98 %
D.4
Ablation Study
We conduct ablation study experiment show that the of distance score matching in coordinate generation module significant. We implement a variant of SyMat which the score model in coordinate generation module is trained by applying the denoising score matching loss directly to coordinates, instead of using our proposed distance score matching loss. We evaluate its performance the random generation task by the structural validity, COV-R, and COV-P metrics as these metrics evaluate the quality of the generated atom coordinates. We summarize the results in Table From the results, we can clearly find that using coordinate score matching always achieves lower COV-R R and COV-P than using distance score matching. This demonstrates that distance score matching can more accurately capture the distribution of material structures, so the generated periodic materials are more similar to periodic materials in datasets, thereby achieving higher COV-R and COV-P.
Table 5: Comparison of SyMat with coordinate score matching and distance score matching. Here means higher metric values lead better performance. Bold numbers highlight the best best performance.
Structure validity
Dataset
Method
COV-R1
COV-Pt
SyMat with coordinate score matching
100.00 % 100.00
97.46%
95.32%
Perov-5
SyMat with distance score matching
99.68
98.64%
SyMat with coordinate score matching
100.00 %
96.72%
93.44
Carbon-24
SyMat with distance score matching
100.00
100.00 %
97.59 %
SyMat with coordinate score matching
100.00%
94. 94.25%
95.61%
MP-20
SyMat with distance score matching
100.00 %
98.97 %
99.97%
21

---

E
Visualization Results
Perov-5
Carbon-24
MP-20
Figure 3: Illustrations some periodic materials generated by SyMat models trained on Perov-5, Carbon-24, and MP-20 datasets.
-1.871
-0.237
-0.355
-0.521
Figure 4: Illustrations of some periodic materials before and after being optimized by SyMat models in the property optimization task. Numbers show how energies (c (/atom)) changes optimization.
0.23
6.28
Ale Op
beedy
0.19
0.3
0.6
-044
-76
-1NE4 2M4 2045
(a) Perov-5 Datasct.
(b) Carbon-24 24 Datasct.
(c) MP-20 Dataset.
Figure 5: Illustrations material energy (e (/ttm))) distribution before and after being optimized by Mat models in the property optimization task.
22


Reviews:
[{"review_id": "sueo1Z0EGS", "reviewer": "Reviewer_oTTM", "summary": "This paper introduces a deep learning based method for the automatic generation of periodic molecules. At the core of their method lies a variational autoencoder and a score-based diffusion models, which generate lattice lengths and angles, and atom coordinates, respectively. Theoretical derivations show that their method is invariant to symmetric transformations and quantitative validation show that the proposed method can be used for optimization problems. \n\nI have read the rebuttal. I am more positive about this submission after the rebuttal and reading the other reviews. The authors have addressed my concerns regarding empirical analysis of limitations, and the quality of writing and figures.", "questions": "- Will code be provided? \n- Why is denoising score matching \"more effective to train\" the generative model?", "limitations": "In my opinion, the authors could provide more discussion of the limitations of their work.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is overall well written and easy to follow. \n- Extensive validation is provided, with comparisons with previous work across a variety of metrics and with ablation studies of the components of their own method. \n- Invariance to symmetric transformations is achieved through theoretical derivations, which makes the model more robust to these transformations than data augmentation-based methods. \n- The provided supplementary material is quite solid and shows very interesting results and ablations. \n- The paper and supplementary material contain enough detail to help reproducibility of the results.", "weaknesses": "- I believe the paper could do a better job at motivating the need for modelling symmetry in molecule generation. I am no expert in this topic and I lacked some motivation onto why the problem is salient. \n- The figures in the paper could be improved, particularly Figure 2 could benefit from better illustrations. \n- Some design choices could be motivated better, particularly related to the model design. \n- The paper needs, in my opinion, more qualitative results like those provided in Figure 3 in the supplementary material. \n- The potential limitations of SyMat are not thorougly analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and insightful comments! We hope your concerns and questions can be addressed by the following responses.\n\n> **Q1: The paper could do a better job at motivating the need for modeling symmetry.**\n\nThanks for your suggestion. The main motivation of modeling symmetry comes from property optimization. A significant target of developing generative models for periodic materials is to generate novel periodic materials with desirable chemical properties (e.g., low energies). Notably, chemical properties are invariant to physical symmetry transformations. For instance, rotating a material structure in 3D space does not change its energy. In other words, chemical properties only depend on the 3D geometric information that is invariant to symmetry transformations (e.g., interatomic distances). Using symmetry-aware modeling forces generative models to only capture the distribution of these 3D geometric information, which facilitates searching materials with desirable properties in the latent space. We will add the above discussions to the revision of our paper.\n\n> **Q2: The figures in the paper could be improved, particularly Figure 2 could benefit from better illustrations.**\n\nThanks for your suggestion. We have improved Figure 2 by adding intuitive graphics instead of only using mathematical symbols to represent the generation targets (see Figure 1 in the rebuttal PDF attached in author rebuttal to all reviewers). We will update Figure 2 in the revision of our paper.\n\n> **Q3: Some design choices could be motivated better, particularly related to the model design.**\n\nThanks for your suggestion. Due to the space limitation in author rebuttal to each reviewer, we post the response to this question in author rebuttal to all reviewers.\n\n> **Q4: The paper needs more qualitative results like those provided in Figure 3.**\n\nThanks for your suggestion. We have included more qualitative visualization results of several optimized periodic materials obtained in the property optimization task in Figure 2 of the rebuttal PDF attached in author rebuttal to all reviewers. We will add them to the revision of our paper.\n\n> **Q5: The potential limitations of SyMat are not thoroughly analyzed.**\n\nThanks for your suggestion. We thoroughly analyze the potential limitations of SyMat below.\n- First, SyMat takes as long as an hour to generate periodic materials in one time, and most of the time is spent on atom coordinate generation. This is because generating atom coordinates requires to run thousands of Langevin dynamics sampling steps. To tackle this limitation, we will explore using stochastic differential equation (SDE) based diffusion models [1,2] for atom coordinate generation in the future as they need much less sampling steps.\n- Second, SyMat is designed to maintain invariance to periodic transformations, so it cannot be applied to non-periodic materials as this will lead to incorrect probabilistic modeling. Though periodic materials occupy a large part of materials used in real-world applications, this limitation hampers the application to some useful non-periodic materials, such as amorphous solids and polymers. In the future, we will explore developping generative models that can capture symmetry properties in non-periodic materials.\n- Third, our work follows previous material generation studies [3] to evaluate the quality of the generated materials by some basic metrics, such as composition and structure validity. However, they do not use metrics to evaluate the synthesizability of the generated materials, though these metrics are useful for practical applications. In the future, we will collaborate with experts in material science to come up with metrics for evaluating synthesizability of the materials generated by our method.\n\nWe will add the above discussions to the revision of our paper.\n\n[1] Song, Yang, et al. \"Score-based generative modeling through stochastic differential equations.\" ICLR 2021.  \n[2] Kingma, Diederik, et al. \"Variational diffusion models.\" NeurIPS 2021.  \n[3] Xie, Tian, et al. \"Crystal diffusion variational autoencoder for periodic material generation.\" ICLR 2022.\n\n> **Q6: Will code be provided?**\n\nYes, we will release our code after the paper decision is out.\n\n> **Q7: Why is denoising score matching \"more effective to train\" the generative model?**\n\nDenoising score matching can effectively train the model because it significantly improves the prediction accuracy of scores in low data density regions. In most practical application scenarios, the data whose distribution will be captured by generative models is not universally scattered in the whole data space, but concentrated in one or several small regions. Hence, there exist large amounts of low data density regions where the probabilistic density of data distribution is low. If the model is only trained to accurately predict scores of data samples in the given dataset, it will not perform well in predicting scores in low data density regions. However, denoising score matching first moves dataset samples out from high data density regions by adding Gaussian noise to them, then the model is trained to predict the score of the noised data sample. This mechanism forces the model to achieve high score prediction accuracy in low data density regions. Since the data generation process of models (Langevin dynamics sampling) needs accurate scores in the whole data space, the model trained by denoising score matching will generate high-quality data in more chances. We recommend Section 3 of [1] as an excellent introduction about this topic."}, {"review_id": "OZqt9gs2ls", "reviewer": "Reviewer_QiTe", "summary": "This paper proposes SyMat, a new approach to generate materials that capture physical properties of perodic structures. This method generates atom types, lattice matrix of the unit cell, and then atom positions, all while maintaining invariance to permutations, rotation, translation, and periodic transformation. They use a VAE (with SphereNet as the encoder and MLPs as decoder) to generate the elements and lattice vectors, and score-based diffusion model to generate the atom positions. They evaluate SyMat in random generation and property optimization tasks.", "questions": "- For success rate of property optimization, \"the rate of decoded materials whose properties are in the top 5%, 10%, and 15% of the energy distribution in the dataset\", what is \"the dataset\" referring to - is it just some energy threshold determined by the Nth percentile of the test set? Would it be helpful to report the distribution of energies genereated in addition to success rate at thresholds?\n- How is the energy of the generated materials obtained? Is it verified by something like DFT? i.e. to make sure that the model isn't cheating by reporting artificially low energies", "limitations": "- The authors acknowledge limitations such as model inference speed and that the work is limited to task of periodic structures", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper tackles an important problem of generating periodic materials, useful for the field of materials discovery given that datasets of known materials for inference are limited\n- Good techniques of how to frame the prediction task in a symmetry-aware way\n- Good comparison to related work and including several of them as baselines\n- Generally strong results, with SyMat performing best or second best in nearly all metrics", "weaknesses": "- I don't fully agree with not needing to evaluate the ability of the model to reconstruct materials from their latent representations. 1: even if the goal is to generate novel materials, we still want to ensure that the latent representation is valid and useful, and reconstruction of known materials seems to be a good way of validating the latent representations. 2: Agree that you can't use absolute coordinates for evaluation, but you should be able to evaluate it based on interatomic distances (or perhaps at least something like graph connectivity).\n- Is there some way to evaluate diversity in the random samples generated? Also, ideally it would be great to see if the generated materials are stable (e.g. able to converge from some relaxation), but perhaps that's out of scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback and helpful suggestions! We believe your concerns and questions can be addressed by the following responses.\n\n> **Q1: I don't fully agree with not needing to evaluate the ability of the model to reconstruct materials from their latent representations. 1: even if the goal is to generate novel materials, we still want to ensure that the latent representation is valid and useful, and reconstruction of known materials seems to be a good way of validating the latent representations. 2: Agree that you can't use absolute coordinates for evaluation, but you should be able to evaluate it based on interatomic distances (or perhaps at least something like graph connectivity).**\n\nThanks for your insightful comments and suggestions! We agree that material reconstruction experiments can help ensure that the latent representation is valid and useful. We have done the reconstruction experiment and evaluated the performance by the following metrics on the test sets of Perov-5, Carbon-24 and MP-20 datasets .\n- First, to show the validity and usefulness of latent representations $z^A$ and $z^L$, we evaluate the percentage of materials whose atom types can be exactly reconstructed (Atom Type Match Rate), and the root mean square error in lattice lengths and angles between the target and reconstructed materials (Lattice RMSE).\n- Second, to show the score-based diffusion model has the capacity to approximately reconstruct 3D structures, we evaluate the root mean square error in interatomic distances between the target and reconstructed materials (Distance RMSE).\n\nWe present the experimental results of SyMat together with baseline methods on Perov-5, Carbon-24 and MP-20 datasets in the following three tables.\n\n|Method|Atom Type Match Rate|Lattice RMSE|Distance RMSE|\n|----|----|----|----|\n|FTCP|**99.67%**|0.0786|0.2953|\n|Cond-DFC-VAE|58.92%|0.0765|0.2281|\n|CDVAE|98.16%|0.0231|0.1072|\n|SyMat|98.3%|**0.0224**|**0.0723**|\n\n|Method|Atom Type Match Rate|Lattice RMSE|Distance RMSE|\n|----|----|----|----|\n|FTCP|**100%**|0.1349|0.3759|\n|CDVAE|**100%**|**0.0624**|0.1745|\n|SyMat|**100%**|0.0632|**0.1186**|\n\n|Method|Atom Type Match Rate|Lattice RMSE|Distance RMSE|\n|----|----|----|----|\n|FTCP|71.46%|0.1057|0.2062|\n|CDVAE|68,32%|0.0532|0.0975|\n|SyMat|**72.1%**|**0.0510**|**0.0876**|\n\nNote that Cond-DFC-VAE can only be used to Perov-5 dataset in which materials have cubic structures, and G-SchNet and P-SchNet do not produce latent representations so they cannot do reconstruction. Results show that our method performs well in atom types, lattices and interatomic distances reconstruction.\n\n> **Q2: Is there some way to evaluate diversity in the random samples generated? Also, ideally it would be great to see if the generated materials are stable (e.g. able to converge from some relaxation), but perhaps that's out of scope.**\n\n- In our opinion, we can evaluate diversity of the randomly generated atom types by uniqueness percentage, which is the percentage of unique atom types among all the generated atom types. However, we think there is no way to evaluate the diversity in 3D structures. We present the uniqueness percentages of SyMat and baseline methods on Perov-5 and MP-20 datasets (Carbon-24 is not used because all materials in Carbon-24 are composed of only carbon atoms, so the uniqueness is expected to be low) in the following table.\n\n|Method|Perov-5|MP-20|\n|----|----|----|\n|FTCP|72.33%|79.04%|\n|Cond-DFC-VAE|80.36%|---|\n|G-SchNet|98.74%|99.23%|\n|P-G-SchNet|98.57%|99.03%|\n|CDVAE|98.61%|99.84%|\n|SyMat|**99.43%**|**99.98%**|\n\nNote that Cond-DFC-VAE can only be used to Perov-5 dataset in which materials have cubic structures. Results show that our method performs well in generating materials with diverse atom types.\n\n- We agree that it would be practically meaningful to evaluate if the generated materials are stable. However, it needs very expensive computational resources to do the structure relaxation, so we and existing studies [1,2] in material generation evaluate the stability of generated materials by structure validity, i.e., the percentage of generated materials whose interatomic distances are all larger than 0.5Å. We think the evaluation by structure relaxation is out of scope and leave it to the future work.\n\n[1] Court, Callum J., et al. \"3-D inorganic crystal structure generation and property prediction via representation learning.\" Journal of Chemical Information and Modeling 60.10 (2020): 4518-4535.  \n[2] Xie, Tian, et al. \"Crystal diffusion variational autoencoder for periodic material generation.\" ICLR 2022.\n\n> **Q3: For success rate of property optimization, \"the rate of decoded materials whose properties are in the top 5%, 10%, and 15% of the energy distribution in the dataset\", what is \"the dataset\" referring to - is it just some energy threshold determined by the Nth percentile of the test set? Would it be helpful to report the distribution of energies generated in addition to success rate at thresholds?**\n\nYes, success rates are computed based on the energy threshold determined by the 5, 10, and 15 percentile of the test set. We agree that it is helpful to report the distribution of energies in addition to success rate. We have visualized the energy distribution before and after property optimization in Figure 3 of the rebuttal PDF attached in author rebuttal to all reviewers.\n\n> **Q4: How is the energy of the generated materials obtained? Is it verified by something like DFT? i.e. to make sure that the model isn't cheating by reporting artificially low energies.**\n\nTo ensure fair comparison, we follow the setting in CDVAE [1] to obtain energies by predicting them from a GNN model provided in CDVAE official codes. From our observation, this GNN model does not report unusually low energies.\n\n[1] Xie, Tian, et al. \"Crystal diffusion variational autoencoder for periodic material generation.\" ICLR 2022."}, {"review_id": "K4qTYqSiKN", "reviewer": "Reviewer_sa6H", "summary": "this paper proposes a method to enforce required symmetric properties in output generated by diffusion models. this method is applied to the problem of generation of new materials with desired symmetric properties. the method comprise of VAE for the generation of atom numbers, A, and lattice structures, L, while diffusion is used for the generation of coordinates, P. The main assumption in this work is that the probability of a molecule does not change under permutation/rotation+translation. ", "questions": "could you please explain why P(M) = P(M') in Section 3.1? There are inter-atom, A, interactions, which may affect the properties of a molecule, including its energy etc. The probability to find M' in nature can be significantly different from that of M.\n\nSo, I am not sure that P(M)=P(M') under 'Permutation' and/or 'Rotation and translation' of A and P is a reasonable assumption in general. If I am wrong, please explain. Otherwise, please explain the limitations of this assumption.  \n\nCould you please explain \"Since the structures of atom types and lattices are relatively simple..\"?\nIs not it a combinatoric problem? What does it mean simple in this context? How is this simplicity related to VAE? VAE has quite a lot of drawbacks. \n\nWhy do you use different approaches for A+L and for P, VAE, and Diffusion, respectively? You mention that Diffusion is more powerful than VAE, so it can not 'harm' the quality of A+L in comparison to VAE?\nWhy to complicate? \n\nline 187: \"Afterwards, the decoder model uses zA and zL to reconstruct the exact $c$, $\\ell$...\". VAE usually has a reconstruction error. How do you guarantee \"exact reconstruction\"? \n\nCan you explain why \"our method cannot be applied to non-periodic materials\"?\n\nWhat are the possible directions for the acceleration of \"atom coordinate generation\"? \n\n\n", "limitations": "the assumption about the equiprobable molecule generation P(M) = P(M'), is limiting in my option. \n\nthe authors adequately address the limitations in Section 3.5.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "an insightful approach for incorporating required symmetric properties.\nFigure 2 helps to understand the entire method.  ", "weaknesses": "the assumption on equiprobable molecules, P(M)=P(M'), is not true in general. Though it can work in some cases.\n\nthe assumption that \"type vector A\" is independent on \"lattice matrix L\" in the generation of a new material sounds heuristic. could you please elaborate of this assumption and its limitations? \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and comments! We make detailed responses below to address your concerns and questions. We hope you will raise your score if we address your concerns.\n\n> **Q1: Could you please explain why $P(M) = P(M')$ in Section 3.1? There are inter-atom, $A$, interactions, which may affect the properties of a molecule, including its energy etc. The probability to find $M'$ in nature can be significantly different from that of $M$. So, I am not sure that $P(M)=P(M')$ under 'Permutation' and/or 'Rotation and translation' of $A$ and $P$ is a reasonable assumption in general.**\n\nWe think there is some misunderstanding. $M$ and $M'$ in Section 3.1 are not different periodic materials in nature, but actually different representations of the same periodic material in nature. Intuitively, for a certain periodic material in nature, we can write its atoms in any order, and think the 3D material structure as a rigid body and rotate or translate this rigid body arbitrarily in 3D space. These atom order permutation, rotation or translation do not change any internal structures or chemical properties of this material, but simply produce different equivalent representations of this material. In Section 3.1, $M$ and $M'$ are actually the equivalent representations of the same material produced in this way, so $P(M)=P(M')$.\n\n> **Q2: The assumption that \"type vector $A$\" is independent on \"lattice matrix $L$\" in the generation of a new material sounds heuristic. Could you please elaborate of this assumption and its limitations?**\n\nThis assumption comes from the the observation that the distribution of lattices is approximately independent of atom types in real-world periodic materials. Practically, the synthesis of different periodic materials can have very different chemical reaction environments, e.g., different temperatures or pressures. Even if the atom type composition in reactants is fixed, under very different chemical reaction environments, the synthesized periodic materials will have very different 3D structures and a wide range of different lattice lengths and angles. Hence, lattices are often assumed to be independent of atom types in periodic materials and our method (and also previous periodic material generation method [1,2]) takes this assumption in probabilistic modeling. Actually, this assumption is reflected in our used datasets --- materials in Perov-5 dataset have very similar lattices but extremely different atom types, while materials in Carbon-24 dataset have very similar atom types but extremely different lattices. The potential limitations of this assumption are that in theory, generative models may accidently generate periodic materials whose lattice lengths are so small that the unit cell volume is not enough to cover all atoms in a unit cell (though we have not observed such phenomena in our experiments).\n\n[1] Ren, Zekun, et al. \"An invertible crystallographic representation for general inverse design of inorganic crystals with targeted properties.\" Matter 5.1 (2022): 314-335.  \n[2] Xie, Tian, et al. \"Crystal diffusion variational autoencoder for periodic material generation.\" ICLR 2022.\n\n> **Q3: Could you please explain \"Since the structures of atom types and lattices are relatively simple..\"? Is not it a combinatorial problem? What does it mean simple in this context? How is this simplicity related to VAE? VAE has quite a lot of drawbacks.**\n\nThe structures of atom types and lattices are simple because their symmetry-invariant representations (i.e., atom type sets, lattice lengths and angles) are low-dimensional vectors whose distributions are easy to be captured by VAE models. Generating atom types is not a combinatorial problem because we do not brute-forcely traverse all combinations, but sample from the distribution learned on datasets, whose atom types only occupy a small part of all combinatiions. We agree that VAE models have drawbacks and are not as powerful as diffusion models, but they are powerful enough for atom types and lattices and are fast in generation, so we use VAE models.\n\n> **Q4: Why do you use different approaches for $A$+$L$ and for $P$, VAE, and Diffusion, respectively? You mention that Diffusion is more powerful than VAE, so it can not 'harm' the quality of A+L in comparison to VAE? Why to complicate?**\n\nDiffusion models are more powerful than VAE models, but are slower than VAE models in generation. For the atom type $A$ and lattice $L$, their distributions are simple enough (see the response to Q3) to be captured by both VAE models and diffusion models. To achieve fast generation, we use VAE models. Such designs are also used by previous material generation methods [1,2].\n\n[1] Ren, Zekun, et al. \"An invertible crystallographic representation for general inverse design of inorganic crystals with targeted properties.\" Matter 5.1 (2022): 314-335.  \n[2] Xie, Tian, et al. \"Crystal diffusion variational autoencoder for periodic material generation.\" ICLR 2022.\n\n> **Q5: Line 187: \"Afterwards, the decoder model uses $z^A$ and $z^L$ to reconstruct the exact $c$, $\\ell$\". VAE usually has a reconstruction error. How do you guarantee \"exact reconstruction\"?**\n\nWe agree that there is no guarantee for \"exact reconstruction\". What we mean here is that in the training of VAE models, we use the exact $c$, $\\ell$ as the reconstruction target to calculate the reconstruction loss function. We will clarify it in the revision of our paper.\n\n> **Q6: Can you explain why \"our method cannot be applied to non-periodic materials\"?**\n\nDue to the space limitation in author rebuttal to each reviewer, we post the response to this question in author rebuttal to all reviewers.\n\n> **Q7: What are the possible directions for the acceleration of \"atom coordinate generation\"?**\n\nDue to the space limitation in author rebuttal to each reviewer, we post the response to this question in author rebuttal to all reviewers."}, {"review_id": "U6iOiLovMO", "reviewer": "Reviewer_ukbg", "summary": "The problem of generating periodic materials with deep models is considered, focusing on capturing their unique symmetries. SyMat, a novel material generation approach, is proposed, using a variational auto-encoder to generate atom types, lattice lengths, and angles. A score-based diffusion model with a symmetry-aware probabilistic model is employed for generating atom coordinates, demonstrating promising performance in random generation and property optimization tasks.", "questions": "- Eq. 2: Why did you decide to factorize $p(A, L)$ into $p(A) * p(L)$. I would have guessed that coupling $A$ and $L$ is important.\n- Did you consider tackling the permutation symmetry in $[\\ell, \\phi]$, where permuting $\\ell_1$ with $\\ell_2$ leaves $\\phi_{1,2}$ unaffected (or does it?).\n", "limitations": "As the authors comment, diffusion models are computationally intensive. I don't consider that a direct problem of the paper, though.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear presentation\n- Good structure\n- Fair comparison to prior work (CDVAE)\n- Important application (materials discovery)\n- Convincing empirical results\n- Clear limitations section", "weaknesses": "- Contributions to ML method development minor", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and comments! We hope your concerns and questions can be addressed by the following responses.\n\n> **Q1: Contributions to ML method development are minor.**\n\nWe agree that our contributions to general generative models or other machine learning methods are not significant, as we focus on developing novel periodic material generation models in this work. Nonetheless, we make significant technical contributions in developing novel generative learning methods to achieve symmetry-aware generation of 3D periodic material structures. Specifically, for the purpose of achieving invariance to permutation, rotation, translation and periodic transformations, we propose a series of novel techniques in the probabilistic modeling of periodic material structures.\nThey include transforming atom types and lattices to symmetry-invariant generation targets, and calculating coordinate score functions by the score functions of edge distances in multi-edge graphs. We believe these are highly non-trivial technical contributions, and our proposed novel model or techniques will potentially be very useful to developing automatic design systems for 3D periodic material structures or other 3D periodic point cloud structures.\n\n> **Q2: Eq. 2: Why did you decide to factorize $p(A,L)$ into $p(A) * p(L)$. I would have guessed that coupling $A$ and $L$ is important.**\n\nThe factorization of $p(A,L)$ into $p(A) * p(L)$ comes from the observation that the distribution of lattices is approximately independent of atom types in real-world periodic materials. Practically, the synthesis of different periodic materials can have very different chemical reaction environments, e.g., different temperatures or pressures. Even if the atom type composition in reactants is fixed, under very different chemical reaction environments, the synthesized periodic materials will have very different 3D structures and a wide range of different lattice lengths and angles. Hence, lattices are often assumed to be independent of atom types in periodic materials and our method (and also previous periodic material generation method [1,2]) takes this assumption in probabilistic modeling. Actually, this assumption is reflected in our used datasets --- materials in Perov-5 dataset have very similar lattices but extremely different atom types, while materials in Carbon-24 dataset have very similar atom types but extremely different lattices.\n\n[1] Ren, Zekun, et al. \"An invertible crystallographic representation for general inverse design of inorganic crystals with targeted properties.\" Matter 5.1 (2022): 314-335.  \n[2] Xie, Tian, et al. \"Crystal diffusion variational autoencoder for periodic material generation.\" ICLR 2022.\n\n> **Q3: Did you consider tackling the permutation symmetry in $[\\ell, \\phi]$, where permuting $\\ell_1$ with $\\ell_2$ leaves $\\phi_{1,2}$ unaffected (or does it?)?**\n\nWe did not consider the permutation symmetry in $[\\ell, \\phi]$. In data preprocessing, we always produce the three lattice lengths $\\ell=[\\ell_1,\\ell_2,\\ell_3]$ in an order satisfying $\\ell_1\\le\\ell_2\\le\\ell_3$. We will clarify it in the revision of our paper.\n"}]

Author rebuttal:
We sincerely appreciate all reviewers' hard work in reviewing our paper and writing comments. We are glad that all reviewers give positive feedbacks to our work and make many insightful suggestions to help improve our paper.

In the rebuttal PDF file, we include an updated version of the original Figure 2 in our paper following **reviewer oTTM**'s suggestion, visualize several optimized periodic materials obtained in the property optimization task following **reviewer oTTM**'s suggestion and visualize the energy distribution before and after property optimization following **reviewer QiTe**'s suggestion.

We provide detailed responses to resolve each reviewer's questions or concerns in the author rebuttal to each reviewer. As the space of author rebuttal to each reviewer is limited, we have included some additional responses here.

Additional responses to **reviewer oTTM**:

> **Q3: Some design choices could be motivated better, particularly related to the model design.**

Thanks for your suggestion. We elaborate the motivation of our model design choices below.
- First, the structures of atom types and lattices are relatively simple and their symmetry-invariant representations (i.e., atom type sets, lattice lengths and angles) are low-dimensional vectors. Their distributions are simple enough to be captured by VAE models [1]. Also, VAE models are faster than diffusion models in generation speed. Hence, we use VAE models for atom type and lattice generation.
- Second, the structure of atom coordinates is much more complicated than atom types or lattices, so we use more powerful score-based diffusion models [2] to generate them. More importantly, using score-based diffusion models for atom coordinates enables us to trickly convert the prediction of coordinate scores to that of interatomic distance scores by the chain rule of derivatives. This strategy maintains invariance to rotation, translation and periodic transformations. We think it is hard to develop such a symmetry-aware probabilistic modeling for other generative models. Hence, we use score-based diffusion models for coordinate generation.
- Third, SphereNet [3] is chosen as the backbone network of our model. SphereNet is a 3D graph neural network for 3D graph representation learning. It has powerful capacities in 3D structure feature extraction because it considers complete 3D geometric information contained in interatomic distances, line angles and plane angles (torsion angles). Also, the features extracted by SphereNet are invariant to rotation, translation and periodic transformations when input 3D graphs are created by multi-graph method [4]. Because of its powerful capacity and symmetry-aware feature extraction, we use SphereNet as the backbone network.

We will add the above discussions to the revision of our paper.

[1] Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." ICLR 2014.  
[2] Song, Yang, and Stefano Ermon. "Generative modeling by estimating gradients of the data distribution." NeurIPS 2019.  
[3] Liu, Yi, et al. "Spherical message passing for 3D molecular graphs." ICLR 2022.  
[4] Xie, Tian, and Jeffrey C. Grossman. "Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties." Physical review letters 120.14 (2018): 145301.

Additional responses to **reviewer sa6H**:

> **Q6: Can you explain why "our method cannot be applied to non-periodic materials"?**

Our method is designed to maintain invariance to periodic transformations, but non-periodic materials are not invariant to periodic transformations, so applying our method to non-periodic materials leads to incorrect probabilistic modeling.

> **Q7: What are the possible directions for the acceleration of "atom coordinate generation"?**

We think possible and promising directions for acceleration are using stochastic differential equation (SDE) based diffusion models [1,2] because they need significantly less sampling steps in generation. We will explore these directions in the future.

[1] Song, Yang, et al. "Score-based generative modeling through stochastic differential equations." ICLR 2021.  
[2] Kingma, Diederik, et al. "Variational diffusion models." NeurIPS 2021.

Meta-review: The paper received positive ratings. The reviewers in general appreciate the paper’s effort to address an important problem of generating periodic materials, it's clear presentation, and the verification of the model through convincing and extensive results. The ACs agree with the reviewers and recommend the acceptance. The final version should, where possible, include a more detailed description of the model design as suggested by the reviewers.
Decision: Accept (spotlight)