Crystal Diffusion Variational Autoencoder for Periodic Material Generation
Authors: Tian Xie
Xiang Fu
Octavian-Eugen Ganea
Regina Barzilay
Tommi S. Jaakkola
Venue: ICLR 2022 Poster (2022)
Keywords: materials
graph neural networks
periodic
diffusion models
score matching
molecule
3D
generative

Abstract: Generating the periodic structure of stable materials is a long-standing challenge for the material design community. This task is difficult because stable materials only exist in a low-dimensional subspace of all possible periodic arrangements of atoms: 1) the coordinates must lie in the local energy minimum defined by quantum mechanics, and 2) global stability also requires the structure to follow the complex, yet specific bonding preferences between different atom types. Existing methods fail to incorporate these factors and often lack proper invariances. We propose a Crystal Diffusion Variational Autoencoder (CDVAE) that captures the physical inductive bias of material stability. By learning from the data distribution of stable materials, the decoder generates materials in a diffusion process that moves atomic coordinates towards a lower energy state and updates atom types to satisfy bonding preferences between neighbors. Our model also explicitly encodes interactions across periodic boundaries and respects permutation, translation, rotation, and periodic invariances. We significantly outperform past methods in three tasks: 1) reconstructing the input structure, 2) generating valid, diverse, and realistic materials, and 3) generating materials that optimize a specific property. We also provide several standard datasets and evaluation metrics for the broader machine learning community.

Published as a conference paper at ICLR 2022
CRYSTAL DIFFUSION VARIATIONAL AUTOENCODER FOR PERIODIC MATERIAL GENERATION
Tian Xie, Xiang Fu, Octavian-Eugen Ganea; Regina Barzilay, Tommi Jaakkola Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology Cambridge, MA 02139, USA (txie, xi xiangfu, oct, regina, tormi Jécai 1 l.mit.ee edu
ABSTRACT
Generating the periodic structure of stable materials a long-standing challenge for the material design community. This task is difficult because stable materials only exist
oow-dimensional subspace of all possible periodic arrangements of atoms: 1) the coordinates must lie in the local energy minimum defined by quan- mechanics, and 2) global stability also requires the structure to follow the complex, yet specific bonding preferences between different atom types. Existing methods fail to incorporate these factors and often lack proper invariances. We propose a Crystal Diffusion Variational Autoencoder (CDVAE) that captures the physical inductive bias of material stability. By learning from the data distribu- tion of stable materials, the decoder generates materials in a diffusion process that moves atomic coordinates towards a lower energy state and updates atom types to satisfy bonding preferences between neighbors. Our model also explicitly encodes interactions across periodic boundaries and respects permutation, translation, rota- tion, and periodic invariances. We significantly outperform past methods in three tasks: 1) reconstructing the input structure, 2) generating valid, diverse, and real- istic materials, and 3) generating materials that optimize specific property. We also provide several standard datasets and evaluation metries for the broader ma- chine learning community.
INTRODUCTION
Solid state materials, represented by the periodic arrangement of atoms in the 3D space, are the foundation of many key technologies including solar cells, batteries, and catalysis (Butler et al., 2018). Despite the rapid progress of molecular generative models and their significant impact on drug discovery, the problem of material generation has many unique challenges. Compared with small molecules, materials have more complex periodic 3D structures and cannot be adequately represented by a simple graph like molecular graphs (Figure 1). In addition, materials can be made up of more than 100 elements in the periodic table, while molecules are generally only made up of a small subset of atoms such as carbon, oxygen, and hydrogen. Finally, the data for training ML models for material design is limited. There are only ~200 experimentally known inorganic materials, collected by the ICSD (Belsky al., 2002), in contrast close a billion molecules in ZINC (Irwin & Shoichet, 2005).
The key challenge of this task is in generating stable materials. Such materials only exist in low- dimensional subspace of all possi- ble periodic arrangements of atoms: 1) the atom coordinates must lie in the local energy minimum defined by quantum mechanics (QM); 2) global stability also requires the structure to follow the complex, yet specific
(a),
(ag. sog)
C
C
10.1
Figure 1: The periodic structure of diamond. The left shows the infinite periodic structure, the middle shows a unit cell representing the periodic structure, and the right shows multi-graph (Xie & Grossman, 2018) representation.
"Equal contribution. Correspondence to: Tian Xic at txie@csail.mit.cdh
Code and data available ht pttt://///e thub. www.tt////////o..cco 93/dvaa

---

Published as a conference paper at ICLR 2022
bonding preferences between different atom types (section 3.2). The issue of stability is unique to material generation because valency checkers assessing molecular stability are not applicable to materials. Moreover, we also have to encode the interactions crossing periodic boundaries (Figure 1, middle). and satisfy permutation, translation, rotation, and periodic invariances (section 3.1). Our goal is to learn representations that can learn features of stable materials from data, while adhering to the above invariance properties.
We address these challenges by learning a variational autoencoder (VAE) (Kingma & Welling, 2014) to generate stable 3D materials directly from latent representation without intermediates like graphs. The key insight is to exploit the fact that all materials in the data distribution are stable, therefore if noise is added to the ground truth structure, denoising it back to its original structure will likely increase stability. We capture this insight by designing a noise conditional score network (NCSN) (Song & Ermon, 2019) as our decoder: 1) the decoder outputs gradients that drive the atom coordinates to the energy local minimum; 2) it also updates atom types based on the neigh- bors to capture the specific local bonding preferences (e.g., Si-O is preferred over Si-Si and O-O in SiO2). During generation, materials generated using Langevin dynamies that gradually deforms an initial random structure to a stable structure. To capture the necessary invariances and encode the interactions crossing periodic boundaries, we use SE(3) equívariant graph neural networks adapted with periodicity (PGNNs) for both the encoder and decoder of our VAE.
Our theoretical analysis further reveals an intriguing connection between the gradient field learned by our decoder and an harmonic force field. De facto, the decoder utilizes the latter to estimate the forces on atoms when their coordinates deviate from the equilibrium positions. Consequently, this formulation provides an important physical inductive bias for generating stable materials.
In this work, we propose Crystal Diffusion Variational AutoEncoder (CDVAE) to generate stable materials by learning from the data distribution of known materials. Our main contributions include:
We curate 3 standard datasets from QM simulations and create a set of physically mean- ingful tasks and metrics for the problem of material generation.
We incorporate stability as an inductive bias by designing noise conditional score net- work as the decoder of our VAF, which allows us to generate significantly more realistic materials.
We encode permutation, translation, rotation, and periodic invariances, as well as interac- tions crossing periodic boundaries with SE(3) equivariant GNNs adapted with periodicity.
Empirically, our model significantly outperforms past methods in tasks including recon- structing an input structure, generating valid, diverse, and realistic materials, and generat- ing materials that optimize specific properties.
2 RELATED WORK
Material graph representation learning. Graph neural networks have made major impacts in ma- terial property prediction. They were first applied to the representation learning of periodic materials by Xie & Grossman (2018) and later enhanced by many studies including Schütt al. (2018): Chen et al. (2019). The Open Catalyst Project (OCP) provides platform for comparing different architec- tures by predicting energies and forces from the periodic structure of catalytic surfaces (Chanussot et al., 2021). Our encoder and decoder PGNNS directly use GNN architectures developed for the OCP (Klicpera al., 2020b; 2021; Shuaibi et al., 2021; Godwin et al., 2021), which are also closely related to SE(3) equivariant networks (Thomas et al., 2018; Fuchs et al., 2020).
Quantum mechanical search of stable materials. Predicting the structure of unknown materials requires very expensive random search and QM simulations, and is considered a grand challenge in materials discovery (Oganov et al., 2019). State-of-the-art methods include random sampling (Pickard & Needs, 2011), evolutionary algorithms (Wang et al., 2012: Glass et al., 2006), substi- tuting elements in known materials (Hautier et al., 2011), 1), etc., but they generally have low success rates and require extensive computation even on relatively small problems.
Material generative models. Past material generative models mainly focus on two different ap- proaches, and neither incorporate stability as an inductive bias. The first approach treats materials as 3D voxel images, but the process of decoding images back to atom types and coordinates often results in low validity, and the models are not rotationally invariant (Hoffmann et al., 2019; Noh et al., 2019; Court al., 2020; Long al., 2021). The second directly encodes atom coordinates,
2

---

Published as a conference paper at ICLR 2022
types, and lattices as vectors (Ren al., 2020; Kim al., 2020; Zhao et al., 2021), but the models are generally not invariant to any Euclidean transformations. Another related method to train a force field from QM forces and then apply the learned force field to generate stable materials by minimizing energy (Deringer et al., 2018: Chen & Ong. 2022). This method is conceptually similar to our decoder, but it requires additional force data which is expensive to obtain. Remotely related works include generating contact maps from chemical compositions (Hu et al., 2021; Yang al., 2021) and building generative models only for chemical compositions (Sawada et al., 2019; Pathak et al., 2020; Dan et al., 2020).
Molecular conformer generation and protein folding. Our decoder that generates the 3D atomic structures via diffusion process is closely related to the diffusion models used for molecular con- former generation (Shi et al., 2021; Xu al., 2021b). The key difference is that our model does not rely on intermediate representations like molecular graphs. G-SchNet (Gebauer et al., 2019) is more closely related our method because it directly generates 3D molecules atom-by-atom without re- lying on graph. Another closely related work is E-NFs (Satorras et al., 2021) that use a flow model to generate 3D molecules. In addition, for molecular graph generation (Liu et al., 2021) and protein folding (Wu et al., 2021), Flow models have also been used for molecular graph generation (Shi et al., 2020; Luo et al., 2021). However, these generative models do not incorporate periodicity. which makes them unsuitable for materials.
nooeebbaed and energy- based models have also been used
3 PRELIMINARIES
3.1 PERIODIC STRUCTURE MATERIALS
Any material structure can be represented as the periodic arrangement of atoms in the 3D space. As illustrated in Figure 1, we can always find repeating unit, i.c. a unit cell, to describe the infinite periodic structure of material. unit cell that includes N atoms can be fully described by lists: 1) atom types A = (ao..... aN) E A where Å denotes the set of all chemical elements; 2) atom coordinates X = (Fg EN) E RNx3, and 3) periodic lattice L = (22,42,4 (g) E R3x3 The periodic lattice defines the periodic translation symmetry of the material. Given M (A, the infinite periodic structure can be represented as,
((d,, 2b))a, = 0s; ar x; + kil1 h2ll2 + kglg, k1, 12 hig Z),
where A2, ky are any integers that translate the unit cell using L to tile the entire 3D space.
The chemical composition of material denotes the ratio of different elements that the material is composed of. Given the atom types of a material with N atoms A E A the composition can be represented as c E RIAI, where e; > 0 denotes the percentage of atom type and Σ, C = 1. For example, the composition of diamond in Figure/ has C and c; 0 for # 6 because 6 is the atomic number of carbon.
Invariances for materials. The structure of a material does not change under several invariances. 1) Permutation invariance. Exchanging the indices of any pair of atoms will not change the material. 2) Translation invariance. Translating the atom coordinates X by an arbitrary vector will not change the material. 3) Rotation invariance. Rotating X and L together by an arbitrary rotation matrix will not change the material. 4) Periodic invariance. There are infinite different ways of choosing unit cells with different shapes and sizes, e.g., obtaining bigger unit cell of a smaller unit cell using integer translations. The material will again not change given different choices of unit cells.
integer multiplier
Multi-graph representation for materials. Materials can be represented directed multi-graph g = [V.8) E to encode the periodic structures following (Wells et al., 1977: O' O'eefffe & Hyde, 1980; Xie & Grossman, 2018), where V {U1,... ..., UN) } is the set of nodes representing atoms and & (eij,(k) j. E {1,..., N}. k1, k2, ky E Z} is the set of edges representing bonds. Cjj.(k, denotes directed edge from node i at the original unit cell to node j at the cell translated by k + k2ll2 + ka4y (in Figure right, k ka) are labeled on top of edges). For materials, there is no unique way to define edges (bonds) and the edges are often computed using k-nearest neighbor (KNN) approaches under periodicity or more advanced methods such as CrystalNN (Pan et al., 2021). Given this directed multi-graph, message-passing neural networks and -quiaaaqu networks can be used for the representation learning of materials.
3

---

Published as a conference paper at ICLR 2022
Add noises A.
A.X
Denoise A.
M (1,
Luwa
Conditional
PGGNNDe ( (M(2)
Encode
Predict
0.5
PGNN
MLF (xa
Compasition
Lattica
(A.
M (A,
Rand. Inf.
Training
Langevin dynamica
Conditional
PGNN Dec (M
=)
Figure 2: Overview of the proposed CDVAE approach.
3.2 PROBLEM DEFINITION AND ITS PHYSICAL ORIGIN
Our goal to generate novel, stable materials M X, x RNX3, x R3x3, The space of stable materials is a subspace in AN Rxxx x R333 that satisfies the following constraints. 1) The materials lie in the local minimum of the energy landscape defined by quantum mechanics, with 0 and 0F 0. 2) The material is
respect to the atom coordinates and lattice, i.e. 0F/0X globally stable and thus cannot decompose into nearby phases. Global stability is strongly related to bonding preferences between neighboring atoms. For example, in SiO,, each Si is surrounded by and each O is surrounded by 2 Si. This configuration is caused by the stronger bonding preferences between Si-O than Si-Si and O-O.
Generally, finding novel, stable materials requires very expensive random search and quantum me- chanical simulations. To bypass this challenge, we aim to learn a generative model p( ((M) from the empirical distribution of experimentally observed stable materials. A successful generative model will be able to generate novel materials that satisfy the above constraints, which can then be verified using quantum mechanical simulations.
3.3 DIFFUSION MODELS
Diffusion models are a new class of generative models that have recently shown great success in generating high-quality images (Dhariwal & Nichol, 2021), point clouds (Cai et al., 2020; Luo & Hu, 2021), and molecular conformations (Shi al., 2021). There are several different types of diffusion models including diffusion probabilistic models Sohl-DDiksteein et al., 2015), noise- conditioned score networks (NCSN) (Song & Ermon, 2019), and denoising diffusion probabilistic models (DDPM) (Ho et al., 2020). We follow ideas from the NCSN (Song & Ermon, 2019) and learn score network sp(x) to approximate the gradient of probability density V V=P(2)) at different noise levels. Let {a:) be sequence of positive scalars that satisfies 01 01/02 =
11.-1/01, 1. We define the data distribution perturbed by Gaussian noise as go (x) = f (t) (t)W o²1) dt. network to jointly estimate the scores of all perturbed data
The goal of NCSN is to learn distributions, i.e. Yo E {at) (0 a) ss Valo (x:) During generation, NCSN uses an annealed Langevin dynamics algorithm to produce samples following the gradient estimated by the score network with gradually reduced noise level.
PROPOSED METHOD
Our approach generates new materials via a two-step process: 1) We sample a from the latent space and use it to predict aggregated properties of a material: composition (c), lattice (L), and number of atoms (N), which are then used to randomly initialize a material structure M (A, X, L). 2) We perform Langevin dynamics to simultancously denoise X and Á conditioned on to improve both the local and global stability of M and generate the final structure of the new material.
To train our model, we optimize networks concurrently using stable materials M (A, L) sampled from the data distribution. 1) A periodic GNN encoder PGNN (M (M) that encodes M into a latent representation 2. 2) A property predictor MLP (2a that prediets the c, L, and N of M from 3) periodic GNN decoder PGNNDEC M 2) that denoises both X and Á conditioned
4

---

Published as a conference paper at ICLR 2022
on 2. For 3), the noisy structure M = (A. L) is obtained by adding different levels of noise to X and A. The noise schedules are defined by the predicted aggregated properties, with the motivation of simplifying the task for our decoder from denoising an arbitrary random structure from over ~100 elements to a constrained random structure from predicted properties. We train all three networks together by minimizing a combined loss including the aggregated property loss CAsa, decoder denoising loss LDEC, and KL. divergence loss Lxt. for the VAE.
To capture the interactions across periodic boundaries, we employ a multi-graph representation (sec- tion 3.1) for both M and M. We also use SE(3) equivariant GNNs adapted with periodicity as both the encoder and the decoder to ensure the permutation, translation, rotation, and periodic invariances of our model. The CDVAE is summarized in Figure 2 and we explain the individual components of our method below. The implementation details can be found in Appendix B.
Periodic material encoder. PGNN ENc (M) encodes material M as latent representation z RD following the reparameterization trick in VAE (Kingma & Welling, 2014). We use the multi- graph representation (refer to section 3.1) to encode M, and PGNNENC can be parameterized with an SE(3) invariant graph neural network.
Prediction of aggregated properties. MLP (z) predicts aggregated properties of the encoded material from its latent representation 2. It is parameterized by 3 separate multilayer perceptrons (MLPs). 1) Composition с E || is predicted by minimizing the cross entropy between the ground truth composition and predicted composition, ie. - -E,pi log c,. 2) Lattice L E R3x3 is reduced to unique, rotation invariant parameters with the Niggli algorithm (Grosse- Kunstleve al., 2004), i.e. the lengths of the 3 lattice vectors, the angles between them, and the values are predicted with an MLP after being normalized to the same scale (Appendix B.1) with an L2 loss. 3) Number of atoms N' E {1, } is predicted with softmax classification loss from the set of possible number of atoms. LAGG is a weighted sum of the above losses.
Conditional score matching decoder. PGNN (MNN))((()) (M]=) is a PGNN that inputs noisy mate- rial M with type noises AA, coordinate noises ox, as well as a latent z, and outputs 1) a score x (M [2; a A. ox) E Nx3 to denoise the coordinate for each atom towards its ground truth value, and 2) a probability distribution of the true atom types PA (M A ox) N AA, We use a SE(3) graph network to ensure the equivariance of 8x with respect to the rotation of M. To obtain the noisy structures M, we sample a A and ox from two geometric sequences of the same length: {o 1==11 {ox,) 1/=11 and add the noises with the following methods. For type noises, we use the type distribution defined by the predicted composition e to linearly perturb true type distribution aa Pc), where PA.ij = 1 if atom has the true atom type j and PA,ij = 0 for all other js. and Pe is the predicted composition. For coordinate noises, we add Gaussian noises to the true coordinates X NNX.o>x))
A ~ 110 PA + 1+0
PGNNDEC is parameterized by SE(3) equivariant PGNN that inputs multi-graph representation (section 3.1) of the noisy material structure and the latent representation. The node embedding for node i is obtained by the concatenation of the element embedding of ā, and the latent representation z, followed by a MLP, h MLP(e, and e, is a learned embedding for elements. After K message-passing layers, PGNNDEC outputs a vector per node that is equivariant to the rotation of M. These vectors are used to predict the scores, and we follow Song& & Ermon (2019); Shi et al. (2021) to parameterize the score network with noise scaling: sx [M] z; A, xx) sx (M 2)) / The node representations h!" are used to predict the distribution of true atom types, and the type predictor is the same at all noise levels: PA( [M x; A. xx = PA (M [=), PA (M Ix), = softmax(MLP
2), where || denotes concatenation of two vectors
Periodicity influences denoising target. Due to periodicity, specific atom may move out of the unit cell defined by L when the noise is sufficiently large. This leads two different ways define the scores for node i. 1) Ignore periodicity and define the target score as x; T.; or 2) Define the target score as the shortest possible displacement between æ, and क; considering periodicity, i.e. (x;, );) = ming, kg (z; x; + k14 + k2l2 + kylz). We choose 2) because the scores are the same given two different X that are periodically equivalent, which is mathematically grounded for periodic structures, and empirically results in much more stable training.
5

---

Published as a conference paper at ICLR 2022
The training loss for the decoder Liec can be written as,
2L Erasuttrr T x (MMM)) (11x((1112) dwin (X, x)
da La (. dAj
xx,
(2)
where Aa denotes coefficient for balancing the coordinate and type losses, La denotes the cross entropy loss over atom types, PA denotes the true atom type distribution. Note that to simplify the equation, we follow the loss coefficients in Song & Ermon (2019) for different ox and A, and factor them into Equation 2.
Material generation with Langevin dy- namics. After training the model, we can generate the periodic structure of mate- rial given latent representation z. First,
Algorithm Langevin Dynamies
Material Generation via Annealed
latent representation z, type and coordinate noise levels {a {oxx. step size e, number of sampling steps 7
Input:
we use z to predict the aggregated prop-
erties: 1) composition c, 2) lattice L. and 3) the number of atoms N. Then, we randomly initialize an initial peri- odic structure (Ag, Xu, L) with the ag- gregated properties and perform an an- nealed Langevin dynamics (Song & Er- mon, 2019) using the decoder, simulta- neously updating the atom types and co- ordinates. During the coordinate update. 11:
2: Predict aggregated properties c, L, N from 2.
3: Uniformly initialize Xo within the unit cell by L.
4: Randomly initialize Ag with c.
5: for j + 1 to do
6:
aj f- e. c. //ox////x /
for t + to do
8x,: < 8x (A-11 XI-1, L
9:
PA.: f PA (A, x,-1; -
10:
Draw X; ~ ((0,1)
X; + XI-1 + a,ax,1 v
we map the coordinates back to the unit cell at each step atoms move out of the cell. The algorithm is summarized in Al- 14:
12:
X, & back_to, celkstto_ccllaa2 X
13:
A. argmax PA.
Xo XT, Ao AT
gorithm 1.
Connection between the gradient field and a harmonic force field. The gradient field 8 (M [z) is used to update atom coordinates in Langevin dynamics via the force term, a In Appendix A,
we show that Sy5x,t is mathematically equivalent to harmonic force field F(X) = -x(X X) when the noises are small, where X is the equilibrium position of the atoms and k: is a force constant. Harmonic force field, i.e. spring-like force field, is a simple yet general physical model that approximates the forces on atoms when they close to their equilibrium locations. This indicates that our learned gradient field utilizes the harmonic approximation to approximate QM forces without any explicit force data and generates stable materials with this physically motivated inductive bias.
EXPERIMENTS
We evaluate multiple aspects of material generation that are related to real-world material discovery process. Past studies in this field used very different tasks and metries, making it difficult to compare different methods. Building upon past studies (Court et al., 2020; Ren et al., 2020), we create a set of standard tasks, datasets, and metrics to evaluate and compare models for material generation. Experiment details can be found in Appendix D.
Tasks. We focus on tasks for material generation. 1) Reconstruction evaluates the ability of the model to reconstruct the original material from its latent representation z. 2) Generation evaluates the validity, property statistics, and diversity of material structures generated by the model. 3) Property optimization evaluates the model's ability to generate materials that are optimized for specific property.
Datasets. We curated 3 datasets representing different types of material distributions. 1) Perov- 5 (Castelli et al., 201 2a;b) includes 18928 perovskite materials that share the same structure but differ in composition. There are 56 elements and all materials have 5 atoms in the unit cell. 2) Carbon-24 (Pickard, 2020) includes 10153 materials that are all made up of carbon atoms but differ in structures. There is element and the materials have 6 24 atoms in the unit cells. 3) MP-20 (Jain et al., 2013) includes 45231 materials that differ in both structure and composition. There are
In fact, this also true for the original formulation of NCSN (Song & Ermon, 2019)

---

Published as a conference paper at ICLR 2022
Perow-5
Carbon-24
MP-20
Ground
FTGP
Cons-DFC-VAE
COVAE
Figure 3: Reconstructed structures of randomly selected materials in the test set. Note our model reconstructs rotated (translated) version of the original material due to the SE(3) invariance.
Table 1: Reconstruction performance.
Match rate (%)
Method
Perov-5
Carhon-24 24
MP-20
Perov-5
Carbon-24
MP-20
FTCP
99.34
62.28
69.89
0.0259
0,2563
0.1393
Cond-DFC-VAL
51.65
0.0217
CDVAE
97.52
55.22
45.43
0.0156
0.1251
0.0356
89 elements and the materials have 1 - 20 atoms in the unit cells. We use 60-20-20 random split for our experiments. Details regarding dataset curation can be found at Appendix C.
Stability of materials in datasets. Structures in all 3 datasets are obtained from QM simulations and all structures are local energy minima. Most materials in Perov-5 and Carbon-24 are hypothetical, ie. they may not have global stability (section 3.2) and likely cannot he synthesized. MP-20 is a realistic dataset that includes most experimentally known inorganic materials with most 20 atoms in the unit cell, most of which are globally stable. A model achieving good performance in MP-20 has the potential to generate novel materials that can be experimentally synthesized.
Baselines. We compare CDVAE with the following baselines, which include the latest coordinate- based, voxel- voxel-based, and 3D molecule generation methods. FTCP (Ren et al., 2020) is a crystal representation that concatenates real-space properties (atom positions, atom types, etc.) and Fourier- transformed momentum-spann representation for crystal generation. Cond-DFC-Vas VAE (Court et al., 2020) encodes and generates crystals with 3D density maps, while employing several modifications over the previous Voxel- VAE (Hoffmann et al., 2019) method. However, the effectiveness is only demonstrated for cubic systems, limiting its usage to the Perov-5 dataset. G-SchNet (Gebauer et al., 2019) an auto-regressive model that generates 3D molecules by performing atom-by-aton completion using SchNet (Schütt al., 2018). Since G-SchNet is unaware of periodicity and cannot generate the lattice L. We adapt G-SchNet to our material generation tasks by constructing the smallest oriented bounding box with PCA such that the introduced periodicity does cause structural invalidity. P-G-SchNet is our modified G-SchNet that incorporates periodicity. During training, the SchNet encoder inputs the partial periodic structure to predict next atoms. During generation, we first randomly sample lattice from training data and autoregressively generate the periodic structure.
properties (diffraction pattern). A ID CNN-VAE is trained over this
5.1 MATERIAL RECONSTRUCTION
Setup. The first task is to reconstruct the material from its latent representation. We evaluate recon- struction performance by matching the generated structure and the input structure for all materials in the test set. We use St ructureMat cher from pyma gen (Ong et al., 2013), which finds the best match between percentage of materials satisfying the criteria stol=0.5 5, angle tol-10, 1tol=0.3, 3. The RMSE is averaged over all matched materials. Because the inter-atomic distances vary signifi- cantly for different materials, the RMSE is normalized by / \/V/N, roughly the average atom radius per material. Note G-SchNet not a VAE so we do not evaluate its reconstruction performance.
structures considering all invariances of materials. The match rate is the
The reconstructed structures are shown in Figure 3 and the metrics are in Table 1. Since our model SE(3) invariant, the generated structures may be translated (or rotated) version of the ground truth structure. Our model has a lower RMSE than all other models, indicating its stronger capability to reconstruct the original stable structures. FTCP has higher match rate than model.
Results.

---

Published as a conference paper at ICLR 2022
FTGP
Cond-OFC-VAE
Schhit
G-SchNet (perodic
COVAE
Figure 4: Structures sampled from No,, 1) and filtered by the validity test.
Table 2: Generation performance
Validity (%)
COV (%)T
Property Statistics 1
Method
Data
Siruc.
Comp.
R.
FTCP
Ferw-5
0.24
54.24
0.000
0.00
10:27
156.0
0.6297
Carbon-24
0.00
5,206
19.05
MP-20
1.55
48.37
4.72
0.09
23.71
160.9
0.7363
Cond-DFC-VAE
Perov-5
73.60
82,95
73.92
10.13
2.268
4.111
0.8373
G-SchNet
Pemv-5
99.92
98.79
0.18
0.23
1.625
4.746
0.03684
Carbon-24
99.94
0.00
0,9427
1.320
MP-20
99.65
75.96
38.33
99.57
3.034
42.09
0.641
P-G-SehNet
Perov-5
79.63
99.13
0.37
0.25
0.2755
1,388
0,4552
Carbon-24
39
0.00
0,00
1,533
134.7
MP-20
77.51
41.93
99.74
4.04
2.448
0.6234
CDVAE
Pemw-5
100.0
98.59
99.45
98.46
0.1258
0.0264
0.0628
Carbon-24
100.0
99.80
83.08
0.1407
0.2850
MP-20
100.0
86.70
99.15
99.49
0.6875
0.2778
1.432
This can be explained by the fact that the same set of local structures can be assembled into different stable materials globally (e.g., two different crystal forms of ZnS). Our model is SE(3) invariant and only encodes local structures, while FTCP directly encodes the absolute coordinates and types of each atom. In Figure 5, we show that CDVAE can generate different plausible arrangements of atoms by sampling 3 Langevin dynamics with different random seeds from the same z. We note that this capability could be an advantage since it generates more diverse structures than simply reconstructing the original ones.
5.2 MATERIAL GENERATION
Setup. The second task is generate novel, stable materials that are distributionally similar to the test materials. The only high-fidelity evaluation of stability of generated materials is to perform QM calculations, but it is computationally prohibitive use QM for computing evaluation metrics. We developed several physically meaningful metrics to evaluate the validity, property statistics, and diversity of generated materials. 1) Validity. Following Court et al. (2020), structure is valid as long the shortest distance between any pair of atoms is larger than 0.5 A, which is relative weak criterion. The composition is valid if the overall charge is neutral as computed by SMACT (Davies et al., 2019). 2) Coverage (COV). Inspired by Xu et al. (202 Ganea et al. (2021), we define two coverage metrics, COV-R (Recall) and COV-P (Precision), to measure the similarity between ensem- bles of generated materials and ground truth materials in test set. Intuitively, COV-R measures the percentage of ground truth materials being correctly predicted, and COV-P measures the percentage of predicted materials having high quality (details in Appendix G). 3) Property statistics. We com- pute the earth mover's distance (EMD) between the property distribution of generated materials and test materials. We use density (p. unit g/cm²), energy predicted by an independent GNN (E, unit eV //atoom), and number of unique elements (# elem.) as properties. Validity and coverage computed over 10,000 materials randomly sampled from (0,0 1). Property statisties computed over 1,000 valid materials randomly sampled from those that pass the validity test.
Some metrics unsuitable for specific datasets have "_" values in the table (explained in Appendix D.1). 4 Forr comparison, the ground truth structure validity is 100.0 % for all datasets, and ground truth composition validity is 98.6 60 %. 100.0 %. 91.13 % for Perov-5, Carbon-24, and MP-20. DDue the low validity of FTCP, instead randomly generate 100,000 materials from A/ 1) and 1,000 materials from those valid ones compute diversity and property statistics metrics.
8

---

Published as a conference paper at ICLR 2022
Table 3: Property optimization performance.
Perov-5
Carbon-24
MPP20
Method
SR5
SR10
SR15
SR5
SRI0
SRIS
SR5
SR1D
SRI5
FTCP
0.06
0.11
0.16
0.0
0.0
0.0
0.02
0.04
0.05
CORd-DFC-VAE
0.55
0.64
0.69
CDVAE
0.52
0.65
0.79
0.0
0.06
0.06
0.78
0.86
0.90
Results. The generated structures are shown in Figure 4 and the metrics are in Table 2. Our model achieves a higher validity than FTCP, Cond-DFC-VAE, and P-G-SchNet, while G-SchNet achieves a similar validity as ours. The lower structural validity in P-G-SchNet than G-SchNet is likely due to the difficulty of avoiding atom collisions during the autoregressive generation inside a finite periodic box. On the contrary, our G-SchNet baseline constructs the lattice box after the 3D positions of all atoms are generated, and the construction explicitly avoids introducing invalidity. Furthermore, our model also achieves higher COV-R and COV-P than all other models, except in MP-20 our COV-P is similar to G-SchNet and P-G-SchNet. These results indicate that our model generates both diverse (COV-R) and high quality (COV-P) materials. More detailed results on the choice of thresholds for COV-R and COV-P, as well as additional metrics can be found in Appendix G. Finally, our model also significantly outperforms all other models in the property statisties of density and energy, further confirming the high quality of generated materials. We observe that our method tends to generate more elements in material than ground truth, which explains the lower performance in the statistics of # of elems. than G-SchNet. We hypothesize this due to the non-Gaussian statistical structure of ground truth materials (details in Appendix D.3), and using more complex prior, e.g., a flow- mooee-trannsormeed Gaussian (Yang et al., 2019), might resolve this issue.
5.3 PROPERTY OPTIMIZATION
Setup. The third task is to generate materials that optimize specific property. Following Jin et al. (2018), we jointly train a property predictor F parameterized by an MLP predict properties of training materials from latent z. To optimize properties, we start with the latent representations of testing materials and apply gradient ascent in the latent space to improve the predicted property F(.). After applying 5000 gradient steps with step sizes of x 10-3, 10 materials are decoded from the latent trajectories every 500 steps. We use independently trained property predictor to select the best one from the 10 decoded materials. Cond-DFC-VAL is a conditional VAE so we directly condition on the target property, sample 10 materials, and select the best one using the property predictor. For all methods, we generate 100 materials following the protocol above. We use the independent property predictor to predict the properties for evaluation. We report the success rate (SR) as the percentage of materials achieving 5, 10, and 15 percentiles of the target property distribution. Our task is to minimize formation energy per atom for all 3 datasets.
Results. The performance is shown in Table 3. We significantly outperform FTCP, while having a similar performance as Cond-DFC- VAE in Perov-5 (Cond-DFC-VAr) VeAee-VAAeeeaaae cannot work for Carbon-24 and MP-20). Both G-SchNet and P-G-SchNet are incapable of property optimization 6, We note that all models perform poorly on the Carbon-24 -24 dataset, which might be explained by the complex and diverse 3D structures of carbon.
6 CONCLUSIONS AND OUTLOOK
We have introduced Crystal Diffusion Variational Autoencoder (CDVAE) to generate the periodic structure of stable materials and demonstrated that significantly outperforms past methods on the tasks of reconstruction, generation, and property optimization. We note that the last two tasks are far more important for material design than reconstruction because they can be directly used to generate new materials whose properties can then be verified by QM simulations and experiments. We believe CDVAE opens up exciting opportunities for the inverse design of materials for various important applications. Meanwhile, our model just a first step towards the grand challenge of material design. We provide our datasets and evaluation metrics to the broader machine learning community to collectively develop better methods for the task of material generation.
6 'Very recently the authors published an improved version for conditional generation (Gebauer al., 2021) but the code has not been released yet.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We have made the following efforts ensure reproducibility: 1) We provide code ht // gi titbbb ccm/txxl txie- e- 93/ cdvae; 2eWe provide our data and corresponding vaain/vaidationnfen splits at https: /github......... com/ txie--ie 93/ ww////vveeeeeeeeeee /ta///ainiii/////aaa//iiii daaaa; 3) We provide de- tails on experimental configurations in Appendix D.
ACKNOWLEDGMENTS
We thank Peter Mikhael, Jason Yim, Rachel Wu, Bracha Laufer, Gabriele Corso, Felix Faltings, Bowen Jing, and the rest of the RB and TJ group members for their helpful comments and sugges- tions. The authors gratefully thank DARPA (HR00111920023 the consortium Machine Learning for Pharmaceutical Discovery and Synthesis (mlpds.mit.edu), and MIT-GIST collaboration for sup- port.
REFERENCES
Alec Belsky, Mariette Hellenbrandt, Vicky Lynn Karen, and Peter Luksch. New developments in the inorganic crystal structure database (iesd): accessibility in support of materials research and design. Acta Crystallographica Section B: Structural Science, 58(3):364-369, 2002. 1, 16
Keith T Butler, Daniel W Davies, Hugh Cartwright, Olexandr Isayev, and Aron Walsh. Machine 2018.
learning for molecular and materials science. Nature, 559(7715):547
Ruojin Cai, Guandao Yang. Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 364-381. Springer, 2020. 4
Ivano E Castelli, David Landis, Kristian S Thygesen, Søren Dahl, Chorkendorff, Thomas Jaramillo, and Karsten W Jacobsen. New cubic perovskites for one-and two-photon water splitting using the computational materials repository. Energy & Environmental Science, 5(10);0034- 9043, 2012a. 6, 16
Ivano E Castelli, Thomas Olsen, Soumendu Datta, David D Landis, Søren Dahl, Kristian $ Thyge- sen, and Karsten W Jacobsen. Computational screening of perovskite metal oxides for optimal solar light capture. Energy & Environmental Science, 5(2):58 14-5819, 2012b. 6, 16
Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Heras- -ominno Caleb Ho, Weihua Hu, al. Open catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 11100)6059-6072 2021. 2, 15
Chi Chen and Shyue Ping Ong. A universal graph deep learning interatomic potential for the periodic table. arXiv preprint 2202::2202. 02440,, 2022.
Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. Graph networks universal machine learning framework for molecules and crystals. Chemistry of Materials, 3 31(9))3564- 3572, 2019. 2
Callum J Court, Batuhan Yildirim, Apoorv Jain, and Jacqueline M Cole. 3-d inorganic crystal structure generation and property prediction via representation learning. Journal of chemical information and modeling, 60(10):4518 45355 2020. 2, 6, 8
Yabo Dan, Yong Zhao, Xiang Li, Shaobo Li, Ming Hu, and Jianjun Hu. Generative adversarial networks (gan) based efficient sampling of chemical composition space for inverse design of inorganic materials. npj Computational Materials, 6(1):1-7, 2020. 3
Daniel W Davies, Keith T Butler, Adam J Jackson, Jonathan M Skelton, Kazuki Morita, and Aron Walsh. Smact: Semiconducting materials by analogy and chemical theory. Journal of Open Source Software, 4(38):1361, 2019. 8, 16
Volker I. Deringer, Chris Pickard, and Gábor Csányi. Data-driven learning of total and local energies in elemental boron. Physical review letters, 120(15):15601,, 2018.
10

---

Published as a conference paper at ICLR 2022
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2105.05233, 2021. 4
Fabian Fuchs, Daniel E. Worrall, Volker Fischer, and Max Welling. tran)trranforrm::::: 3d roto- translation equivariant attention networks. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Process- ing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: https::/proceeeiinngggg////////t paper/2020 hash hwshhhhshssae
newrips ca/ 9911 3b 3ww.2cccccccc33332 wtw.:ecccce55e-4--baarreee html. 2
-ugeenn--Eugen Ganea, Lagnajit Pattanaik, Connor W Coley, Regina Barzilay, Klavs F Jensen, William H Green, and Tommi S Jaakkola. Geomol: Torsional geometric generation of molecular 3d conformer ensembles. arXiv preprint
2021. 8, 18
Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. In H. Wallach, H. Larochelle, Beygelzimer, F. d'Alché-Buc E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. 7.7
Niklas WA Gebauer, Michael Gastegger, Stefaan SP Hessmann, Klaus-Robert Müller, and Kristof T Schütt. Inverse design of 3d molecular structures with conditional generative neural networks. arXiv preprint arXiv:2109.04822 2021. 9
Colin W Glass, Artem R Oganov, and Nikolaus Hansen. Uspex-evolutionary crystal structure prediction. Computer physics communcatiions, 175(11-12)713--20, 2006. 2
Jonathan Godwin, Michael Schaarschmidt, Alexander Gaunt, Alvaro Sanchez-Gonzalez, Yulia Rubanova, Petar Veličković, James Kirkpatrick, and Peter Battaglia. Very deep graph neural networks via noise regularisation. arXiv preprint arXiv:2106.0797 2021.
Ralf W for the computation of reduced unit cells. Acta Crystallographica Section A: Foundations of Crystallography, 60(1):1-6, 2004. 5, 15
Nicholas K Sauter, and Paul D Adams. Numerically stable algorithms
Geoffroy Hautier, Chris Fischer, Virginie Ehrlacher, Anubhav Jain, and Gerbrand Ceder. Data mined ionic substitutions for the discovery of new compounds. Inorganic chemistry, 50(2):656-66) 2011. 2
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httpps:/ppoooeeeei/itee wwwrroceedingg..eeee. neuri innurips... cc/ http:/pper/2222//// 4c5bc Frcbeeec8588 f0d967 a 111100199c4440 Abst.act........................ Aba html.
Jordan Hoffmann, Louis Maestrati. Yoshihide Sawada, Jian Tang, Jean Michel Sellier, and Yoshua Bengio. Data-driven approach to encoding and decoding 3-d crystal structures. arXiv preprint arXiv: arXiv:1909.000 2019. 2,
Jianjun Hu, Wenhui Yang, Rongzhi Dong, Yuxin Li, Xiang Li, Shaobo Li, and Edirisuriya MD Siriwardane. Contact map based crystal structure prediction using global optimization. Crys- 2021.
1EngComm, 23(8):1765-11)
John J Irwin and Brian K Shoichet. Zinc- free database of commercially available compounds for virtual screening. Journal of chemical information and modeling, 4571):177-182, 2005. 1
Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, al. Commentary: The ma- terials project: A materials genome approach to accelerating materials innovation. APL materials, 1(1):01 1002, 2013. 6, 16
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning, pp. 2323-2332. PMLR, 2018. 9
11

---

Published as a conference paper at ICLR 2022
Sungwon Kim, Juhwan Noh, Geun Ho Gu, Alan Aspuru-Guzis and Yousung Jung. Generative adversarial networks for crystal structure prediction. ACS central science, 6(8):1412-1420, 2020. 3
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL abs/1312. 6114. 2,5
Johannes Klicpera, Shankari Giri, Johannes T Margraf, and Stephan Günnemann. Fast and uncertainty-awart directional message passing for non-equilibriun molecules. arXiv preprint 1rXi::2011.14115. 2020a. 15
Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molec- ular graphs. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020b. URL https://///eoc pttetpss///peerev/// /tpp///veeeeee....eeeeee.. net/ hert/form?? dd=31 eWbxStPH. 2, 15
Johannes Klicpera, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. arXiv preprint vrXiv:2106.08003 2021. 2. 15
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint arXiv:2106.00133) 2021. 17
Meng Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. Graphebm: Molecular graph generation with energy-based models. arXiv preprint arXiv:2102.00044 2021. 3
Teng Long, Nuno M Fortunato, Ingo Opahle, Yixuan Zhang, Ilias Samathrakis, Chen Shen, Oliver Gutfleisch, and Hongbin Zhang. Constrained crystals deep convolutional generative adversarial network for the inverse design of crystal structures. npj Computational Materials, 7(1):1-7, 2021. 2
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2837-2845, 2021. 4
Youzhi Luo, Keqiang Yan, and Shuiwang Ji. Graphdf: A discrete flow model for molecular graph generation. arXiv preprint arXiv:2102.01 189, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-81181 8171. PMLR, 2021. 17
Juhwan Noh, Jaehoon Kim, Helge S Stein, Benjamin Sanchez-Lengeling, John M Gregoire, Alan Aspuru-Guzze Guzik, and Yousung Jung. Inverse design of solid-state materials via a continuous repre- sentation. Matter, 16)):1370-1884, 2019. 2
Artem R Oganov, Chris Pickard, Qiang Zhu, and Richard J Needs. Structure prediction drives materials discovery. Nature Reviews Materials, 4(5):331-343 2019.
M. Oeeefffe and B. G. Hyde. Plane in crystal chemistry. Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences, 295(1417):533-618 1980. ISSN 00804614. URL http: /wwww jstor org/ sttp//taalleeeee /36188.3366.. 3
Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L Chevrier, Kristin A Persson, and Gerbrand Ceder. Python materials genomics (pymatgen): A robust, open-source python library for materials analysis. Computational Materials Science, 68:314 2013. 7, 15, 18
Hillary Pan, Alex M Ganose, Matthew Horton, Muratahan Aykol, Kristin A Persson, Nils ER Zim- mermann, and Anubhav Jain. Benchmarking coordination number prediction algorithms on inor- ganic crystal structures. Inorganic chemistry, 6003):1500-1603 2021. 3, 15
Yashaswi Pathak, Karandeep Singh Juneja, Girish Varma. Masahiro Ehara, and Deva Priyakumar. Deep learning enabled inorganic material generator. Physical Chemistry Chemical Physics, 22 (46):2633522693,, 2020.
12

---

Published as a conference paper at ICLR 2022
Chris J. Pickard. Airss data for carbon 10gpa and the c+n+h+o system at Igpa, 2020. URL attps:://arcn www.mmaaaaaaloo.oo.. org/ record/2020. 2ttp:///22200002/vv1.. 2tt.:///...o..oeeooo//////000000
Chris Pickard and RJ Needs. High-pressun phases of silane. Physical review letters, 97(4)004550 2006. 16
Chris J Pickard and RJ Needs. Ab initio random structure searching. Journal of Physics: Condensed Matter, 23(5):053201, 2011. 2, 16
Zekun Ren, Juhwan Noh, Siyu Tian, Felipe Oviedo, Guangzong Xing. Qiaohao Liang, Armin Aberle, Yi Liu, Qianxiao Li, Senthilnath Jayavelu, al. Inverse design of crystals using gen- eralized invertible crystallographic representation. arXiv preprint arXiv:2005.07609, 2020. 3, 6, 7, 16
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint (xXiv:2222.005122 2022. 17
Victor Garcia Satorras, Emiel Hoogeboom, Fabian B Fuchs, Ingmar Posner, and Max Welling. E (n) equivariant normalizing flows for molecule generation 3d. arXiv preprint :XXv:22105000166 2021. 3
Yoshihide Sawada. Koji Morikawa, and Mikiya Fujüi. Study of deep generative models for inorganic chemical compositions. arXiv preprint arXiv: 1499, 2019.
Kristof T' Schütt, Huziel Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24):241222,2 2018. 2.7
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: flow-based autoregressive model for molecular graph generation. arXiv preprini arXiv:2001.0338 2020. 3
Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular con- formation generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th Inter- national Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, vol- ume 139 of Proceedings of Machine Learning Research, pp. 9558 9568. PMLR, 2021. URL http:/ // http://poceeennngeeee/ee mlr. www.ppressuveee hates//vll/sssse shillb htt.2llb........mmmmmm..mmm..... 3,4, 5,5,
Muhammed Shuaibi, Adeesh Kolluru, Abhishek Das, Aditya Grover, Anuroop Sriram, Zachary Ulissi, and C Lawrence Zitnick. Rotation invariant graph neural networks using spin convolutions. arXiv preprint arXiv:2106.09577 2021. 2
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015. 4
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Informa- tion Processing Systems 32: Annual Conference Neural Information Processing Sys- tems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11895- 11907, 2019, URL httts://procceeiing neurips.. Abstrratt. html. 2,
wttprrisssccccppeeee wttpcc//ppppaeeee htt/ppppr/2219/haa/e
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivarianl neural networks for 3d point clouds. arXiv preprint arXiv:1802.082/11 2018.
Yanchao Wang, Jian Lv, Li Zhu, and Yanming Ma. Calypso: A method for crystal structure predic- tion. Computer Physics Communications, 183(10):2063-207) 2012. 2
Logan Ward, Ankit Agrawal, Alok Choudhary, and Christopher Wolverton. A general-purpose machine learning framework for predicting properties of inorganic materials. upj Computational Materials, 2(1):1-7, 2016. 18
13

---

Published as a conference paper at ICLR 2022
Alexander Frank Wells et al. Three dimensional nets and polyhedra. Wiley, 1977. 3
Jiaxiang Wu, Tao Shen, Haidong Lan, Yatao Bian, and Junzhou Huang. Se (3)-qquivariant energy- based models for end-to-end protein folding. bioRxiv, 2021. 3
Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters, 125314))44330,,, 2018. 1, 2.3
Minkai Xu, Shitong Luo, Yoshua Bengio, Jian Peng, and Jian Tang. Learning neural generative dynamics for molecular conformation generation. In International Conference on Learning Rep- resentations, 2021a. URL https::/openreviiieee/ wwt.eriwwwweeeeeeeeeeee.eeeteeee forum?dd--mmmmmmm gfw.k 8,
18
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: geo- metric diffusion model for molecular conformation generation. In International Conference Learning Representations, 2021b.
Guandao Yang, Xun Huang. Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1541-45500 2019.
Wenhui Yang, Edirisuriya Dilanga Siriwardane, Rongzhi Dong, Yuxin Li, and Jianjun Hu. Crystal structure prediction of materials with high symmetry using differential evolution. arXiv preprint arXiv:2104.09764 2021. 3
Yong Zhao, Mohammed Al-Fahdi, Ming Hu, Edirisuriya Siriwardane, Yuqi Song, Alireza Nasiri, and Jianjun Hu. High-throughput discovery of novel cubic crystal materials using deep generative neural networks. arXiv preprint arXiv:2102.01880, 2021.
Nils ER Zimmermann and Anubhav Jain. Local structure order parameters and site fingerprints for quantification of coordination environment and crystal structure similarity. RSC Advances, 10 (10):6063-6081,2)) 2020. 18
14

---

Published as a conference paper at ICLR 2022
A PROOF FOR THE CONNECTION TO A HARMONIC FORCE FIELD
We assume the loss in Equation can be minimized to when the noises small, meaning that
(A, X, L|2 dwin (X,
J,
(3)
xx
where xx E { and any noise smaller than ox,J is considered as small.
The force term the Langevin dynamics 8,88,t can then be written as
138x(A, Z Z. xx. e xx,5/0x,1 8x (x X, L L
(4)
Xxj,
dmmi (X,X)
Vj >
(5)
X,L
X
dmin (X, X)) Vj
(6)
X1L
we write 3/ hi then,
Lzz = kdmin [X
Xj.
(7)
If the noises are small enough that atoms do not cross the periodic boundaries, then we have dminn (X,)) = X. Therefore,
0,8x (Á, X, L|2; - x).
B IMPLEMENTATION DETAILS
B. PREDICTION OF LATTICE PARAMETERS
There are infinitely many different ways of choosing the lattice for the same material. We compute the Niggli reduced lattice (Grosse-Kunstlews et al., 2004) with pymatgen (Ong et al., 2013), which is instead
unique lattice for any given material. Since the lattice matrix L is not rotation invariant, predict the 6 lattice parameters, i.e. the lengths of the 3 lattice vectors and the angles between them.
We normalize the lengths of lattice vectors with VN, where N the number of atoms, to ensure that the lengths for materials of different sizes the same scale.
B.2 MULTI-GRAPH CONSTRUCTION
For the encoder, we use CrystalNN (Pan et al., 2021) to determine edges between atoms and build multi-graph representation. For the decoder, since it inputs noisy structure generated on the fly, the multi-graph must also be built on the fly for both training and generation, and CrystalNN is too slow for that purpose. We use KNN algorithm that considers periodicity to build the decoder graph where K 20 all of our experiments.
B.3 GNN ARCHITECTURE
We use DimeNet++ adapted for periodicity (Klicpera et al., 2020a;b) as the encoder, which is SE(3) invariant the input structure. The decoder needs output an vector per node that is SE(3) equiv- ariant to the input structure. We use GemNet-dQ (Klicpera et al., 2021) as the decoder. We used implementations from the Open Catalysis Project (OCP) (Chanussot et al., 2021), but reduced the size of hidden dimensions to 128 for faster training. The encoder has 2.2 million parameters and the decoder has 2.3 million parameters.
C DATASET CURATION
C.I PEROV-5
Perovskite is a class of materials that share a similar structure and have the general chemical formula ABX;, The ideal perovskites have cubic structure, where the site A atom sits at corner position,
15

---

Published as a conference paper at ICLR 2022
the site B atom sits at body centered position and site X atoms sit at face centered positions. Perovskite materials are known for their wide applications. We curate the Perov-5 dataset from an open database that was originally developed for water splitting (Castelli et al., 2012a;b).
All 18928 materials in the original database are included. In the database, A, B can be any non- radioactive metal and X can be one or several elements from O, N, S, and F. Note that there can be multiple different X atoms in the same material. All materials in Perov-5 density functional theory (DFT), and their relaxed structure can deviate significantly from the ideal structures. A significant portion of the materials are not thermodynamically stable, i..., they will decompose to nearby phases and cannot be synthesized.
relaxed using
C.2 CARBON-24
Carbon-24 includes various carbon structures obtained via ab initio random structure searching (AIRSS) (Pickard & Needs, 2006; 201 1) performed at 10 GPa.
The original dataset includes 101529 carbon structures, and selected the 10% of the carbon structure with the lowest energy per atom to create Carbon-24. All 10153 structures in Carbon-24 relaxed using DFT. The most stable structure is diamond at 10 GPa. All remaining structures are thermodynamically unstable but may be kinetically stable. Most of the structures cannot be synthesized.
C.3 MP-20
MP-20 includes almost all experimentally stable materials from the Materials Project (Jain et al., 2013) with unit cells including at most 20 atoms. We only include materials that are originally from ICSD (Belsky et al., 2002) to ensure the experimental stability, and these materials represent the majority of experimentally known materials with most 20 atoms in unit cells.
To ensure stability, we only select materials with energy above the hull smaller than 0.08 eV/atom and formation energy smaller than 2 eV/atom, following Ren et al. (2020). Differing from Ren et al. (2020), do not constrain the number of unique elements per material. All materials MP-20 are relaxed using DFT. Most materials are thermodynamcially stable and have been synthesized.
D EXPERIMENT DETAILS
D.1 REASONS FOR THE UNSUITABILITY OF SOME METRICS FOR SPECIFIC DATASETS
In Table 2, property statistics are computed by comparing the earth mover's distance between the property distribution of generated materials and ground truth materials. So, they are not meaningful for ground truth data.
Materials in Perov-5 have the same structure, so it is not meaningful to require higher structure diversity.
Materials in Carbon-22 24 have the same composition (carbon), so it is not meaningful to require higher composition diversity. In addition, all models have 100% composition validity, so it is not com- pared in the table.
D.2 COMPOSITION VALIDITY CHECKER
We modified the charge neutrality checker from SMACT (Davies et al., 2019) because the original checker is not suitable for alloys. The checker is based on list of possible charges for each element and it checks if the material can be charge neutral by enumerating all possible charge combinations. However, it does consider that metal alloys can be mixed with almost any combination. As result, for materials composed of all metal elements, we always assume the composition is valid in validity checker.
For the ground truth materials MP-20, the original checker gives a composition validity of 550% which significantly underestimates the validity of MP-20 materials (because most of them are exper- imentally synthesizable and thus valid). Our checker gives a composition validity of 990, which is far more reasonable. We note again that these checkers are all empirical and the only high-fidelity evaluation of material stability requires QM simulations.
16

---

Published as a conference paper at ICLR 2022
D.3 NON-GAUSSIAN STATISTICAL STRUCTURE OF MATERIALS
The material datasets are usually biased towards certain material groups. For example, there are lots of lithium-containing materials in MP-20 because started with battery research. We also find that our decoder tends to underfit the data distribution with larger 33 in Equation 9. We believe these observations indicate that the statistical structure of the ground truth materials are far from Gaussian. As a result, sampling from (0, 1) may lead to out-of-distee distirrbibution materials, which explains why our method tends to generate more elements per material than the ground truth.
D.4 HYPERPARAMETERS AND TRAINING DETAILS
The total loss can be written as,
L= LAGG
LDEC + LXL dcfe ALLL + ANLN + +xxx + AACA + 3LKL. (9)
We aim to keep each loss term at a similar scale. For all three datasets, we use Ae 1, = 10, Av Ax 10, A = 1.
We tune between 0.01, 0.03, 0.1 for all three datasets and select the model with best validation loss. For Perov-5, MP-20, use 0.01, and for Carbon-24, we use / = 0.03.
For the noise levels in {oA. fax. f we follow Shi et al. (2021) and set 50. For all three datasets,
0 A,max 5, 0 A,min 0.01, ox max 10, ox ,ii 0.01.
During the training. we use an initial learning rate 0.001 and reduce the learning rate by a factor of 0.6 if the validation loss does not improve after 30 epochs. The minimum learning rate is 0.0001.
During the generation, we use 0.0001 and run Langevin dynamics for 100 steps each noise level.
E VISUALIZATION MULTIPLE RECONSTRUCTED STRUCTURES
Perov-5
Carbon-24
MP-20
Ground Truth
Sample
Sample 2
Sample
Figure 5: Different reconstructed structures from CDVAE from the same z, following Langevin
dynamics sampling with different random seeds.
F SAMPLING SPEED FOR MATERIAL GENERATION
We summarize the speed for generating 10,000 materials for all models in Table 4. FTCP is sig- nificantly faster, but the quality of generated materials is very poor as shown in Table 2. Cond- DFC-VAE is faster than our method in Perov-5, but has a lower quality than our method and only works for cubic systems. It is also unclear how it will perform on larger materials in Carbon-24 and MP-20, because the compute increases cubicily with the increased size of the density map. G- SchNer/P-G-Schin
have a comparable sampling time as our method, but have a lower quality. We also note that we did not optimize sampling speed in current work. It is possible to reduce sampling time by using fewer sampling steps without significantly influencing generation quality. There also many recent works that aim to speed up the sampling process for diffusion models (Nichol & Dhariwal, 2021: Kong & Ping, 2021; Salimans & Ho, 2022).
17

---

Published as a conference paper at ICLR 2022
Table 4: Time used for generating 10,000 materials single RTX 2080 Ti GPU.
FTCP
Band-DFC-VAE
G-SchNet
P-G-SchNet
CDVAE
Perov-5
< 1 min
0.5 h
2.0
2.0
3.1
Carbon-24
< 1 min
6.2
6.3 h
5.3 h
MP-20
< 1 min
6.3 h
6.3 h
5.8 h
G COVERAGE METRICS FOR MATERIAL GENERATION
Inspired by Xu et al. (2021a); Ganea et al. (2021), we define six metrics to compare two ensembles of materials: materials generated by a method ]Ms ke/L..K. and ground truth materials in test data
We use the Euclidean distance of the CrystalNN fingerprint (Zimmermann & Jain, 2020) and normalized Magpie fingerprint (Ward et al., 2016) to define the structure distance and composi- tion distance between generated and ground truth materials, respectively. They can be written as Datrue. (M,, M2) and Doomp (M₃, M2)) We further define the thresholds for the structure and composition distance as Ōstrue. and dcomp. respectively.
Following the established classification metrics of Precision and Recall, we define the coverage metrics as:
1 111 E [1..1] :3k [1..K], Datrue. (M&. Mf) < Ostrue.>
COV-R (Recall)
Doomp. (M₃, M;) < 6coup. H
(10)
AMSD-R (Recall) =
Σ
min Detruc. (Mr, Mi) kE[l..K]
(11)
L
LE(1..]]
AMCD-R (Recall)
Σ
min Deomp. MM), kE[L..K]
(12)
T
[E[1...]
where COV is "Coverage", AMSD "Average Minimum Structure Distance", AMCD is "Average Minimum Composition Distance", and COV-P (precision), AMSD-P (precision), AMCD-P (pre- cision) are defined as above equations, but with the generated and ground truth material sets swapped. The recall metrics measure how many ground truth materials are correctly predicted, while the precision metries measure how many generated materials are of high quality (more dis- cussions can be found in Ganea et al. (2021)).
We several points why define the metries in their current forms. COV requires both structure and composition distances to be within the thresholds, because generating materials that structurally close to ground truth material and compositionally close to another is not mean- ingful. As a result, AMSD and AMCD are less useful than COV. 2) We use fingerprint distance, rather than RMSE from StructurMa t cher (Ong et al., 2013), because the material space is too large for the models to generate enough materials to exactly match the ground truth materials. St struut eraMat cher first requires the compositions of two materials to exactly match, which will cause all models to have close-to-zero coverage.
For Perov-5 and Carbon-24, we choose 8strue. = 0.2, &comp. = 4. For MP-20, we choose Satrue. 0.4, 6comp. 10. In Figure 6, Figure 7, Figure 8, show how both COV-R and COV-P change by varying Ostrue. and dcomp. in all three datasets.
18

---

Published as a conference paper at ICLR 2022
Table 5: Full coverage metrics for the generation task.
Method
Data
COV-R
AMSD-R
AMCD-R
COV-P 1
AMSD-P
AMCD-P
FICP
Perov-5
0.00
0.7447
7.212
0,00
0.3582
3.390
Carbon-24
0.00
1.181
0.00
0.00
0.8822
24.16
MP-20
4.72
0.6542
9.271
0.09
0.1954
4.378
CORd-DFC-VAE
Perov-5
73.92
0.1508
2.773
10.13
0.3162
4.257
G-SchNet
Perov-5
0.18
0.5962
0,23
0,4259
1.3163
Carbon-24
0.00
0.5887
0.00
0,00
0.5970
0.00
MP-20
38.33
0.5365
3,233
P-G-SchNNet
Perov-5
0.37
0,5510
1.0264
0.25
0.3967
1.316
Carbon-24
0.00
0.6308
0.8166
0.00
MP-20
41.93
0.5327
3.274
99.74
0.1985
3.567
CDVAE
Perov-5
99,45
0,0482
0.6969
0,0593
Carbon-24
99.80
0.0489
0.00
83.08
0.1343
0.00
MP-20
99.15
0.1549
3.621
0.1883
1.0
0.0
0.2
0.6
1.0
15
Struc. recall Bhreancles
Comp. recal Strasho da
0.2 0.4
0.
1.0
25
Figure 6: Change of COV-R and COV-P by varying Satrae. and deamp. for Perov-5. Dashed line denotes the current chosen thresholds.
19

---

Published as a conference paper at ICLR 2022
0.4
0.2
04
0.6
1.0
20
25
0.0
Struc.
stresholta
Camp. thrnshalds
FTCP
S 5-Schher
G-Schhet (peracici
COURE
Figure 7: Change of COV-R and COV-P by varying Istrue. and δoomp. for Carbon-24, Dashed line denotes the current chosen thresholds.
0.2
0.4
0.6
1.0
15
0.0
0.0
Strc.
Threhhlls
Comp. peocis thrishildd
FTCP
GoShhict (perloski
COVRE
Figure 8: Change of COV-R and COV-P by varying S and 8comp. for MP-20. Dashed line denotes the current chosen thresholds.
20


Reviews:
[{"review_id": "iIu8DP67iay", "reviewer": "Reviewer_5P5h", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The approach is well motivated and novel. Being able to train a generative model without requiring large datasets of non-equilirbium configurations is very relevant considering the computational cost of electronic structure calculations. \n\nThe proposed CDVAE incorporates the necessary invariances and a connection to the harmonic approximation is shown. CDVAE shows state-of-the-art results in validity and diversity of the generated materials. The comparison G-SchNet is not ideal, as a proper adaption to materials would have included periodic boundary conditions. These should be relatively straightforward to implement since they are already available for SchNet. The evaluation could be improved by performing at least a limited number of QM calculations on a small random subset of the generated structures.\n\nThe optimization of the formation energy is another strong point of the paper.", "summary_of_the_paper": "The paper proposes an approach to generate stable crystal structures using variational autoencoders without having to train on out-of-equilibrium data, which is scarce for materials. Beyond that, benchmark datasets and tasks for materials generation are proposed.", "summary_of_the_review": "The proposed approach is a novel and relevant contribution to the domain of materials generation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Sx_jTdgDa3", "reviewer": "Reviewer_vK4U", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**\n- This paper proposes an entire workflow to tackle crystal materials generation. It is an important step towards the grand challenge of materials design. \n- The authors demonstrate the proposed method by using several datasets and baselines. Although the performance can be improved and some metrics are not very reasonable, these efforts can be identified as benchmarks for further exploration in the AI/ML community.  \n\n**Weakness**\n\nGeneral comments: \n- The technical contribution on AI/ML side is probably marginal. Multiple recent models, like SE(3) with PGNNs, diffusion models (NCSN), CrystalNN, DimeNet++, GemNet-dQ, are simply used and combined without improvements. Although building the workflow is not trivial, I did not get an overall insight, e.g., why do you choose such models?  \n\n- Even though this paper is probably classified as an application paper, the contribution to the domain area may not be very substantial. First, the problem of crystal generation has been posted before and several studies already exist. Second, It is difficult to evaluate the success of the paper in crystal generation as it lacks good absolute metrics or a clear performance goal for a successful model. This approach does outperform existing methods in certain areas, but these existing methods are either too limited or too underperforming themselves to serve as a sufficiently challenging benchmark.\n\nDetailed comments:\n\n- The authors claim to generate stable structures due to some physical or chemical motivations. However, I did not get the insight that why stable data distribution will guarantee a stable structure generation via noising and denoising. Does it depend on how do you define the stable structure, it is statistical stable or physical stable?  I suppose that is the reason why you use the NCSN model for generation tasks but it might be easy to follow the intuition.  Why not using energy-based models [1] or flow-based models [2, 3], which are widely used for molecular generation? \n\n[1] Liu, Meng, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. \"GraphEBM: Molecular graph generation with energy-based models.\" arXiv preprint arXiv:2102.00546 (2021).\n\n[2] Shi, C., Xu, M., Zhu, Z., Zhang, W., Zhang, M. and Tang, J., 2020. Graphaf: a flow-based autoregressive model for molecular graph generation. arXiv preprint arXiv:2001.09382.\n\n[3] Luo, Youzhi, Keqiang Yan, and Shuiwang Ji. \"GraphDF: A discrete flow model for molecular graph generation.\" arXiv preprint arXiv:2102.01189 (2021).\n\n- If NCSN works better than the other generative models, how do you define the sensitive noise parameters? The suggested noise levels are specifically designed for image generation, but do they work well for materials generation as well? If not, do you provide an ablation study to show the difference? Since now I do not see a code submission, I am curious about the reproducibility because the workflow looks pretty complex consisting of multiple-step training, rather than a straightforward end-to-end generation. \n\n- Another weakness is the reconstruction loss from the VAE architecture. Visually, it is difficult to tell in Figure 3 if the reconstruction structures have shifted significantly or not. How big an impact does this reconstruction loss have on properties such as formation energy, for example? Is it also possible for elements to change upon reconstruction? In Figure 3 on the MP-20 dataset, it looks like the element has shifted for the second structure; is this really the case or a plotting error?  \n\n- A characteristic of this method is that the crystal parameters (composition and # of atoms, crystal lattice) are decoupled and predicted separately from the atomic structure. In addition, the generation is from random initiation, how do you define the upper and lower bound of your initialization, for different parameters? We know composition, # of atoms, lattice parameter share different scales and they are a mixture of continuous and discrete, do you normalize them? Have you considered dequantization?  It would be helpful to understand how to set up the dataset if these details can be provided. \n\n- Validity criterion is likely too weak and likely not useful in evaluating generation performance, as most structures should not have interatomic distances anywhere near 0.5 Angstroms. Charge neutrality may also not be a good criterion due to the existence of defect structures or materials such as electrides. \n\n- Authors use three existing datasets from the literature, containing (1) perovskites, (2) carbon, (3) and inorganic crystals. From what I can tell, the main contribution to these datasets by the authors is truncating the inorganic crystal dataset to a maximum of 20 atoms. Is 45000 for the inorganic crystal dataset a sufficiently large number of data for this type of problem, and to serve as a benchmark dataset? For the perovskite dataset, is it really necessary to use generative models here since they have a fixed ABO3 composition and do not have structural diversity, which defeats the purpose of a 3-D crystal generation dataset.\n\n- Property optimization is not a fair comparison since the author’s method chooses the best out of ten, whereas other methods like Cond-DFC-VAE only use a single sample from conditioned generation. More importantly, property optimization based on optimized latent space should be able to generate realistic materials, but it might not be guaranteed due to several optimization issues. For example, gradient-based optimization in latent space (similar to JTVAE 2018 paper) may be trapped into local minima, do you consider a global optimizer, e.g., Bayesian optimization or CMA-ES?  I understand it also has scalability issues if the dimension of latent space is relatively high. In addition, some papers [GraphAF and conformation generation] use RL methods to generate a goal-oriented design. \n\n- For property metrics, there is a high possibility of generating very out-of-distribution crystal samples from the training data; since the property is predicted by an independent GNN trained on the same data, there is a strong concern about the property predictions will be inaccurate and meaningless.\n\n- It looks like G-SchNet is a strong contender in this paper, however the authors implement it without considering periodicity. This is not necessarily a failure of the G-SchNet method, but rather it is used improperly and cannot perform well, and allows the work in this paper to perform favorably compared to it.\n\n- Scalability issue. The dataset used here is still a small-scale graph, does it work well for a large-scale crystal generation, such as more than 20 atoms in Materials Project?  Do you anticipate any limitations of the current workflow? If so, could you add more discussion about the existing limitations of CDVAE? \n\n", "summary_of_the_paper": "This paper aims to address challenging stable crystal materials generation problems via diffusion variational autoencoder with graph representation learning. Several recent advances in generative models and GNN are combined together to develop the entire workflow from data distribution learning, prediction to sample generation and property optimization. The authors demonstrate their method by using three datasets and compare with three baseline methods. ", "summary_of_the_review": "The paper proposes to tackle challenging problems and provide a reasonably workable framework with several benchmarks. However, the core contributions from either the technical side or domain science side are not substantial and significant. Multiple pieces of work need more clear justification and explanation, and the metrics need to be improved with a more physically reasonable definition. I also expect to see a better comparison with the current baselines. Probably, the work is still ongoing so I recommend to reject it in the current form but expect to see more improvements in the near future. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "94DfE_45_af", "reviewer": "Reviewer_Fg5a", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The tackled problem is specific but new and interesting for the machine learning research community, and the paper nicely provided 3 benchmarks to standardize the material generation problem.\n- The paper is well-written and easy to follow. Both technical and empirical parts are clearly illustrated.\n- The experiments are extensive to demonstrate the effectiveness across different datasets and metrics.\n\nWeaknesses:\n- Some detail of the \"conditional score matching decoder\" is confusing for me: how you sample the noisy atom type $\\tilde{A} ~ (p_A +\\sigma_A p_c)$. In my understanding, $p_A +\\sigma_A p_c$ is not a valid categorical distribution, so I'm concerned with the sampling. Is there any softmax function imposed on it?\n- Another concern is about the experiments. I appreciate the authors' efforts to provide several meaningful benchmarks, but the metrics in Sec 5.2 seem still very weak by just comparing validity, property, and fingerprints. There are some recent papers working on molecular conformation generation (samples are listed below), where the two conformation sets are compared by **coverage** and **matching** score. The main differences are the periodic and unconditional nature without graph input, but I think these two evaluations can still be easily adopted since you have also reported similar metrics in Sec 5.1. Could you provide more empirical justifications on the coverage and matching score?\n\n References for weakness 2:\n- Xu, Minkai, et al. \"Learning neural generative dynamics for molecular conformation generation.\" arXiv preprint arXiv:2102.10240 (2021).\n- Ganea, Octavian-Eugen, et al. \"GeoMol: Torsional Geometric Generation of Molecular 3D Conformer Ensembles.\" arXiv preprint arXiv:2106.07802 (2021).\n", "summary_of_the_paper": "The paper proposed a new generative model for 3D periodic material molecular structure. They designed a special variational autoencoder, where the decoder is parameterized by the denoising score-matching framework. Experiments demonstrate the model can successfully generate valid and realistic material, and optimize desired properties.", "summary_of_the_review": "The problem is specific but new to the general machine learning audience, and the proposed VAE model with equivariant score matching decoder is interesting for me. However, I still hold some technical confusion (weakness 1), and I think some important metrics from the highly related topic are missed in the material generation task, which is the most important empirical part of this paper. So currently I vote for a weak rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0jEoPBJWcY", "reviewer": "Reviewer_oZsg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**strengths**\n* The proposed score-based generative model for crystal structure generation, i.e., CDVAE, is novel. Although similar techniques have already been explored in other structure generation problems [1,2], I still believe the proposed model is quite novel as it addresses several important challenges in material structure generation such as **periodic boundary condition and composition prediction.**\n* The proposed model is SE(3) equivariant and periodic boundary condition-sensitive, based on the recent advance on equivariant graph neural networks. It also considers the Periodicity influences on denoising target.\n* The experimental design seems comprehensive, which may serve as standard benchmarks for inorganic crystal structure generation. The experimental results seems quite promising.\n* The authors promise to provide the source code of the project after the review process.\n\n**weaknesses**\n* All the datasets included are inorganic. It would be very nice to see some initial results on organic crystal structure prediction.\n* The lattice constant **L** is predicted from the latent vector and seems it is fixed during the decoding process.  You say that the structure of materials shoudl satisfy $\\partial E / \\partial L = 0$. How is this condition modeled in CDVAE? Can we update the lattice constant during decoding?\n* The crystal structures often exhibit some symmetrcity, e.g. space group. I do not see any discussion related to this concept. Can CDVAE tackle the space group, or is this concept not applicable to the current datasets?\n\n**Reference**\n1. Learning gradient fields for molecular conformation generation\n2. SE(3)-Equivariant Energy-based Models for End-to-End Protein Folding\n", "summary_of_the_paper": "The paper proposes a score-based Diffusion Variational Autoencoder to generate the periodic structure of stable materials (crystal structure generation). \n\n**Main contributions include**\n* Creating a set of standard tasks, datasets and metrics to unify the evaluation process of inorganic crystal structure generation\n* Proposing a **SE(3) equivariant, periodic boundary condition-sensitive** decoder to generate crystal structure structures based on a latent vector.\n\n", "summary_of_the_review": "Overall, I think the submission is of high quality and reaches the bar of ICLR.\nIt would be nice if the authors could address the concerns listed above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a model, named Crystal Diffusion Variational Autoencoder (CDVAE), that can learn to sample valid material structures. It accounts for known symmetries (SE(3), permutation) of the structure via SE(3)-equivariant GNNs.

The proposed model is a complicated combination of many existing models / modeling techniques (VAEs, NCSNs, diffusion models) but it is not entirely ad hoc; the revised paper does a reasonable job in justifying the many different modeling choices made. 
Existing model components, often designed in the context of molecule generation, do not account for the periodicity of the crystal's lattice structure; so this paper introduces modifications to account for this periodicity.

The paper evaluates the model on several datasets and also introduces new benchmarks that can be used for further research. The experimental results look promising but there are a few remaining clarity issues with the metrics used (cf. reviews).
Decision: Accept (Poster)