SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To satisfy all properties in Proposition 3.1, we formulate  $\nabla_{P}\log p(P|A,L)$  as a function of score functions of edge distances in a graph created by multi-graph method [51]. For a periodic material  $M$  with coordinate matrix  $P = [p_1,\dots,p_n]$  and lattice matrix  $L$ , multi-graph method produces an  $n$ -node undirected graph on  $M$ . In this graph, the node  $i$  corresponds to the  $i$ -th atom in the unit cell, and an edge is added between any two nodes  $i,j$  if one of the interatomic distances between them is smaller than a pre-defined cutoff  $r$ . Here, the interatomic distances between  $i,j$  are the distances between their corresponding atoms in the complete infinite structure of  $M$ , including both the distance within the unit cell and the distances crossing the unit cell boundary. Formally, the set of all edges in the graph constructed on  $M$  can be written as  $\mathbf{E}(M) = \{(i,j,k):||\pmb{p}_i + \pmb{L}\pmb{k} - \pmb{p}_j||_2\leq r,\pmb {k}\in \mathbb{Z}^3,1\leq i,j\leq n\}$ . Note that there may exist more than one edges connecting the nodes  $i,j$  if more than one interatomic distances between them is smaller than  $r$ . Let  $d_{i,j,k} = ||\pmb {p}_i + \pmb {L}\pmb {k} - \pmb {p}_j||_2$  be the distance of the edge  $(i,j,k)$ , we consider  $\log p(P|A,L)$  as a function of the distances of all edges in  $E(M)$ . Denoting  $\boldsymbol{s}_i$  as the score function of  $\boldsymbol{p}_i$ , then  $\nabla_P\log p(P|A,L) = [s_1,\dots,s_n]$ . From the chain rule of derivatives, we can calculate  $\boldsymbol{s}_i$  as
$$
\boldsymbol {s} _ {i} = \sum_ {(j, \boldsymbol {k}) \in \mathcal {N} (i)} \nabla_ {d _ {i, j, \boldsymbol {k}}} \log p (\boldsymbol {P} | \boldsymbol {A}, \boldsymbol {L}) \cdot \nabla_ {\boldsymbol {p} _ {i}} d _ {i, j, \boldsymbol {k}} = \sum_ {(j, \boldsymbol {k}) \in \mathcal {N} (i)} s _ {i, j, \boldsymbol {k}} \cdot \frac {\boldsymbol {p} _ {i} + \boldsymbol {L} \boldsymbol {k} - \boldsymbol {p} _ {j}}{d _ {i , j , \boldsymbol {k}}}, \tag {3}
$$
where the scalar  $s_{i,j,\pmb{k}} = \nabla_{d_{i,j,\pmb{k}}} \log p(\pmb{P}|\pmb{A},\pmb{L})$  is the score function of the distance  $d_{i,j,\pmb{k}}$  and  $\mathcal{N}(i) = \{(j,\pmb{k}):(i,j,\pmb{k})\in E(\pmb{M})\}$ . With this probabilistic modeling process,  $\nabla_P \log p(\pmb{P}|\pmb{A},\pmb{L})$  can be approximated by first approximating  $s_{i,j,\pmb{k}}$  for every edge in the multi-graph representation of  $M$ , then calculating each column vector of  $\nabla_P \log p(\pmb{P}|\pmb{A},\pmb{L})$  by Equation (3). By the following Proposition 3.2 (see its proof in Appendix A.2), we can show that if  $s_{i,j,\pmb{k}}$  is invariant to all symmetry transformations described in Section 3.1 for every edge  $(i,j,\pmb{k})$ , the score matrix  $\nabla_P \log p(\pmb{P}|\pmb{A},\pmb{L})$  will satisfy all properties in Proposition 3.1.
Proposition 3.2. For  $\nabla_{P}\log p(P|A,L) = [s_{1},\dots,s_{n}]$  where each  $s_i$  is calculated by Equation (3) on a multi-graph representation of  $M = (\mathbf{A},\mathbf{P},\mathbf{L})$ , if  $s_{i,j,\mathbf{k}}$  is invariant to permutation, rotation, translation, and periodic transformations described in Section 3.1 for any edge  $(i,j,k)$ , then the calculated  $\nabla_{P}\log p(P|A,L)$  will always satisfy the properties in Proposition 3.1.
In our method, we use a 3D GNN model as the score model  $s_{\theta}(\cdot)$  to approximate  $\nabla_P \log p(P|A,L)$ .  $s_{\theta}(\cdot)$  takes the multi-graph representation of  $M$  as input and outputs  $o_{i,j,k}$  as the approximated  $s_{i,j,k}$  at every edge  $(i,j,k)$  in the input graph. We use SphereNet [29] as the backbone architecture of  $s_{\theta}(\cdot)$ , which ensures that the output  $o_{i,j,k}$  is always invariant to any symmetry transformation. More details about SphereNet will be introduced in Appendix B. To train the score model, we follow the most common operations in denoising score matching framework [44]. Specifically, for a material  $M = (A,P,L)$  in the dataset, we will perturb it to multiple noisy materials by a sequence of Gaussian noises with different noise magnitude levels  $\{\sigma_t\}_{t=1}^T$ , in which  $\sigma_1 > \sigma_2 > \ldots > \sigma_T$ . Let the noisy material at the  $t$ -th noise level be  $\widetilde{M} = (A,\widetilde{P},L)$ , the noisy coordinate matrix  $\widetilde{P} = [\tilde{p}_1,\dots,\tilde{p}_n]$  is obtained by adding Gaussian noise  $\mathcal{N}(0,\sigma_t^2 I)$  to  $P$ . The score model  $s_{\theta}(\cdot)$  is trained to predict the denoising score matrix  $\nabla_{\widetilde{P}} \log p_{\sigma_t}(\widetilde{P}|P,A,L) = [\tilde{s}_1,\dots,\tilde{s}_n]$  from  $\widetilde{M}$ . Here, each  $\tilde{s}_i$  is calculated in a similar way as in Equation (3) with the approximated distance score  $\tilde{s}_{i,j,k} = \nabla_{\tilde{d}_{i,j,k}} \log p_{\sigma_t}(\widetilde{P}|P,A,L)$ , where  $\tilde{d}_{i,j,k} = ||\tilde{p}_i + Lk - \tilde{p}_j||_2$ . Motivated from
Algorithm 1: Langevin Dynamics Sampling Algorithm for Coordinate Generation
Input:  $A, L$ , atom number  $n$ , noise magnitudes  $\{\sigma_t\}_{t=1}^T$ , step size  $\epsilon$ , step number  $q$ , score model  $s_\theta(\cdot)$   
Sample  $F_0 \in \mathbb{R}^{3 \times n}$  from uniform distribution  $U(0,1)$  and set  $P_0 = LF_0$   
for  $t = 1$  to  $T$  do  
 $P_t = P_{t-1}$ $\alpha_t = \epsilon \cdot \sigma_t^2 / \sigma_T^2$   
for  $j = 1$  to  $q$  do  
 $M_t = (A, P_t, L)$   
Sample  $Z \in \mathbb{R}^{3 \times n}$  from  $\mathcal{N}(0, I)$   
Obtain the approximated edge distance score from  $s_\theta(\cdot)$  which takes  $M_{t-1}$  as inputs  
Obtain  $s_\theta(M_{t-1}) = [s_1, \dots, s_n]$  where each  $s_i$  is computed by Equation (3) with edge distance scores.  
 $P_t = P_t + \alpha_t s_\theta(M_t) + \sqrt{2\alpha_t} Z$   
end for  
end for  
Output  $P_T$  as the finally generated coordinate matrix
the strategies of existing methods [30, 42, 55], we model the distribution of  $\tilde{d}_{i,j,\pmb{k}}$  as a Gaussian distribution  $\mathcal{N}\left(\hat{d}_{i,j,\pmb{k}},\hat{\sigma}_t^2\right)$ , where the computation process of the mean  $\hat{d}_{i,j,\pmb{k}}$  and the variance  $\hat{\sigma}_t^2$  is described in Appendix B. We can calculate  $\tilde{s}_{i,j,\pmb{k}}$  directly by taking the derivative of the mathematical formula of this Gaussian distribution. The score model  $s_{\theta}(\cdot)$  is trained to minimize the difference between the output  $o_{i,j,\pmb{k}}$  from  $s_{\theta}(\cdot)$  and  $\tilde{s}_{i,j,\pmb{k}}$  for every edge  $(i,j,\pmb{k})$  in  $\widetilde{M}$  with the loss  $\mathcal{L}$ :
$$
\mathcal {L} = \frac {1}{2 T} \sum_ {t = 1} ^ {T} \hat {\sigma} _ {t} ^ {2} \mathbb {E} _ {\widetilde {M}, M} \left[ \sum_ {(i, j, \boldsymbol {k}) \in \boldsymbol {E} (\widetilde {M})} \left| \left| \frac {o _ {i , j , \boldsymbol {k}}}{\hat {\sigma} _ {t}} + \frac {\tilde {d} _ {i , j , \boldsymbol {k}} - \hat {d} _ {i , j , \boldsymbol {k}}}{\hat {\sigma} _ {t} ^ {2}} \right| \right| _ {2} ^ {2} \right].
$$
After the score model is trained, we can use it to generate coordinate matrix  $\pmb{P}$  from given  $\pmb{A}$  and  $\pmb{L}$  with annealed Langevin dynamics sampling algorithm [49]. Specifically, we first randomly initializes a coordinate matrix  $\pmb{P}_0$ , then starting from  $M_0 = (A, P_0, L)$ , we iteratively update  $M_{t-1} = (A, P_{t-1}, L)$  to  $M_t = (A, P_t, L)$  ( $t = 1, \dots, T$ ) with the approximated score matrix  $s_\theta(M_{t-1})$  by running multiple Langevin dynamics sampling steps. See Algorithm 1 for the detailed coordinate generation algorithm pseudocodes.
3.5 Discussions and Relations with Prior Methods
Advantages and limitations. In our method, the atom type vector and lattice matrix are first transformed to items that are invariant to symmetry transformations, then a VAE model is used to capture the distributions of these items. In addition, our method adopts powerful score-based diffusion models to generate atom coordinates, and a novel probabilistic modeling framework for score approximation is employed to satisfy all symmetry-related restrictions. With all these strategies, our method successfully incorporates physical symmetries of periodic materials into generative models, thereby capturing the underlying distributions of material structures more effectively. We will show these advantages through experiments in Section 4. The major limitations of our method lie in that the speed of generating atom coordinates with score-based diffusion models is slow because running thousands of Langevin dynamics sampling steps is needed, and our method cannot be applied to non-periodic materials. We leave more discussions about limitations and broader impacts in Appendix C.
Motivations in Model Design. The model design of our SyMat method has the following three motivations. (1) First, the structures of atom types and lattices are relatively simple and their symmetry-invariant representations (i.e., atom type sets, lattice lengths and angles) are low-dimensional vectors. Their distributions are simple enough to be captured by VAE models. Also, VAE models are faster than diffusion models in generation speed. Hence, we use VAE models for atom type and lattice generation. (2) Second, the structure of atom coordinates is much more complicated than atom types or lattices, so we use more powerful score-based diffusion models to generate them. More
importantly, using score-based diffusion models for atom coordinates enables us to trickily convert the prediction of coordinate scores to that of interatomic distance scores by the chain rule of derivatives. This strategy maintains invariance to rotation, translation and periodic transformations. We think it is hard to develop such a symmetry-aware probabilistic modeling for other generative models. Hence, we use score-based diffusion models for coordinate generation. (3) Third, SphereNet is chosen as the backbone network of our model. SphereNet is a 3D graph neural network for 3D graph representation learning. It has powerful capacities in 3D structure feature extraction because it considers complete 3D geometric information contained in pairwise distances, line angles and plane angles (torsion angles). Also, the features extracted by SphereNet are invariant to rotation, translation and periodic transformations when input 3D graphs are created by multi-graph method. Because of its powerful capacity and symmetry-aware feature extraction, we use SphereNet as the backbone network.
Relations with prior methods. Here we will discuss several studies that are related to our work. Some studies [42, 30] have also applied score-based diffusion models to generate atom coordinates from molecular graphs, but there are not periodic structures in 3D molecules. Differently, keeping invariant to periodic transformations is significant in periodic material generation, and our score-based diffusion model uses periodic invariant multi-graph representations of materials as inputs to satisfy this property. Besides, compared with a recently proposed periodic material generation method CDVAE [52], our SyMat has two major differences. (1) First, to generate the atom type vector, CDVAE generates total atom numbers and compositions (the proportion of atom numbers for every atom type to the total atom number) by a VAE model, in which compositions are vectors composed of continuous numbers. While compositions have complicated structures and are hard to predict, SyMat uses relatively simpler atom type sets as generation targets and the prediction of them can be easily converted to integer prediction or classification problems (see Section 3.3). (2) Second, in coordinate generation, CDVAE also uses the denoising score matching framework, but the denoising score matrix  $\nabla_{\widetilde{P}}\log p_{\sigma_t}(\widetilde{P}|P,A,L)$  is calculated as the direct difference between  $\widetilde{P}$  and a coordinate matrix obtained by aligning  $P$ , which makes  $\nabla_{\widetilde{P}}\log p_{\sigma_t}(\widetilde{P}|P,A,L)$  not invariant to translations on  $\widetilde{P}$ . However, SyMat calculates it from the edge distance score functions to ensure invariance to all symmetry transformations (see Section 3.4). In addition, another two studies, PGD-VAE [47] and DiffCSP [18], are also related to our SyMat approach. PGD-VAE proposes a VAE model for periodic graph generation. However, PGD-VAE is fundamentally different from SyMat in that it only generates 2D periodic topology graphs without 3D structures, while SyMat is developed for 3D periodic material structures. PGD-VAE cannot be applied to 3D periodic materials so we do not compare SyMat with PGD-VAE in our experiments. DiffCSP proposes a novel method for generating 3D periodic material structures from their atom types. Different from SyMat, DiffCSP uses denoising diffusion models to jointly generate lattices and atom coordinates. In addition, DiffCSP does not generate atom types of periodic materials, but lattices and atom coordinates from the input atom types, so DiffCSP cannot be applied to design novel periodic materials from scratch. Hence, we do not compare SyMat with DiffCSP in our experiments as all our experiments evaluate the performance of different methods in generating novel periodic materials.
In this section, we evaluate our proposed SyMat method in two periodic material generation tasks, including random generation and property optimization. We show that our proposed SyMat can achieve promising performance in both tasks.
Data. We evaluate SyMat on three benchmark datasets curated by Xie et al. [52], including Perov-5 [3, 2], Carbon-24 [35], and MP-20 [17]. Perov-5 is a dataset with a collection of 18,928 perovskite materials. The chemical element compositions of all materials in Perov-5 can be denoted by a general chemical formula  $\mathrm{ABX}_3$ , which means that there are 3 different atom types and 5 atoms in each unit cell. The Carbon-24 dataset we used has 10,153 materials. All these material structures consist of only carbon elements and have 6 - 24 atoms in each unit cell, and the 3D structures of them are obtained by DFT simulation. MP-20 is a dataset curated from Materials Project [17] library. It includes 45,231 materials with various structures and compositions, and in all materials, there exist at
Table 1: Random generation performance on Perov-5, Carbon-24, and MP-20 datasets. Here  $\uparrow$  means higher metric values lead to better performance, while  $\downarrow$  means the opposite. Because all materials in the Carbon-24 dataset are composed of only carbon atoms, all methods easily achieve  $100\%$  in composition validity and 0.0 in # element EMD so we do not use these two metrics. Also, Cond-DFC-VAE can only be used to Perov-5 dataset in which materials have cubic structures. Bold and underline numbers highlight the best and second best performance, respectively.
Dataset	Method	Composition validity†	Structure validity†	# Element EMD↓	Density EMD↓	Energy EMD↓	COV-R†	COV-P†
Perov-5	FTCP	54.24%	0.24%	0.6297	10.27	156.0	0.00%	0.00%
Cond-DFC-VAE	82.95%	73.60%	0.8373	2.268	4.111	77.80%	12.38%
G-SchNet	98.79%	99.92%	0.0368	1.625	4.746	0.25%	0.37%
P-G-SchNet	99.13%	79.63%	0.4552	0.2755	1.388	0.56%	0.41%
CDVAE	98.59%	100.00%	0.0628	0.1258	0.0264	99.50%	98.93%
SyMat (ours)	97.40%	100.00%	0.0177	0.1893	0.2364	99.68%	98.64%
Carbon-24	FTCP	-	0.08%	-	5.206	19.05	0.00%	0.00%
G-SchNet	-	99.94%	-	0.9427	1.320	0.00%	0.00%
P-G-SchNet	-	48.39%	-	1.533	134.7	0.00%	0.00%
CDVAE	-	100.00%	-	0.1407	0.2850	100.00%	99.98%
SyMat (ours)	-	100.00%	-	0.1195	3.9576	100.00%	97.59%
MP-20	FTCP	48.37%	1.55%	0.7363	23.71	160.9	5.26%	0.23%
G-SchNet	75.96%	99.65%	0.6411	3.034	42.09	41.68%	99.65%
P-G-SchNet	76.40%	77.51%	0.6234	4.04	2.448	44.89%	99.76%
CDVAE	86.70%	100.00%	1.432	0.6875	0.2778	99.17%	99.64%
SyMat (ours)	88.26%	100.00%	0.5067	0.3805	0.3506	98.97%	99.97%
most 20 atoms in each unit cell. For all three datasets, we split them with a ratio of 3:1:1 as training, validation, and test sets in our experiments.
Tasks. We evaluate the performance of SyMat in random generation and property optimization tasks. In the random generation task, SyMat generates novel periodic materials from randomly sampled latent variables and we employ a variety of validity and statistic metrics to evaluate the quality of the generated periodic materials. In the property optimization task, SyMat is evaluated by the ability of discovering novel periodic materials with good properties. Besides, to show that latent representations are informative enough to reconstruct periodic materials, we also evaluate the performance by the material reconstruction task. See Appendix D.2 for experiment results of this task.
Baselines. We compare SyMat with two early periodic material generation methods FTCP [36] and Cond-DFC-VAE [6], which generate material structures through generating Fourier-transformed crystal property matrices and 3D voxel images with VAE models, respectively. Note that Cond-DFC-VAE can only be used to generate cubic systems, so it is only used for Perov-5 dataset in which materials have cubic structures. In addition, we compare with an autoregressive 3D molecule generation method G-SchNet [10], and its variant P-G-SchNet that incorporates periodicity in G-SchNet. The latest periodic material generation method CDVAE [52] is also compared with our method. Note that all these baseline methods are compared with SyMat in the random generation task, but G-SchNet and P-G-SchNet are not compared in the property optimization task because they cannot produce latent representations with fixed dimensions for different periodic materials.
4.2 Random Generation
Metrics. In the random generation task, we adopt seven metrics to evaluate physical and statistic properties of materials generated by our method and baseline methods. We evaluate the composition validity and structure validity, which are the percentages of generated materials with valid atom type vectors and 3D structures, respectively. Here, an atom type vector is considered as valid if its overall charge computed by SMACT package [8] is neutral, and following Court et al. [6], a 3D structure is considered as valid if the distances between any pairwise atoms is larger than  $0.5\mathring{\mathrm{A}}$ . In addition, we evaluate the similarity between the generated materials and the materials in the test set by five statistic metrics, including the earth mover's distance (EMD) between the distributions in chemical element number, density  $(\mathrm{g} / \mathrm{cm}^3)$ , and energy predicted by a GNN model. Also, we evaluate the percentage of the test set materials that cover at least one of the generated materials (COV-R), and the percentage of generated materials that cover at least one of the test set materials (COV-P). All seven metrics are evaluated on 10,000 generated periodic materials. See Appendix D.1 for more information about evaluation metrics and experimental settings in the random generation task.
Table 2: Property optimization performance on Perov-5, Carbon-24, and MP-20 datasets. We report the success rate (SR), i.e., the rate of periodic materials that can achieve top  $5\%$ ,  $10\%$ , and  $15\%$  of the property distribution in the dataset after being optimized. Cond-DFC-VAE can only be used to Perov-5 dataset in which materials have cubic structures. Bold and underline numbers highlight the best and second best performance, respectively.
Method	Perov-5	Carbon-24	MP-20
SR5	SR10	SR15	SR5	SR10	SR15	SR5	SR10	SR15
FTCP	0.06	0.11	0.16	0.00	0.00	0.00	0.02	0.04	0.05
Cond-DFC-VAE	0.55	0.64	0.69	-	-	-	-	-	-
CDVAE	0.52	0.65	0.79	0.00	0.06	0.06	0.78	0.86	0.90
SyMat (ours)	0.73	0.80	0.87	0.06	0.13	0.13	0.92	0.97	0.97
Results. The random generation performance results of all methods on three datasets are presented in Table 1. Among the total 19 metrics over three datasets in the table, our SyMat method achieves top-1 performance in 11 metrics and top-2 performance in 17 metrics. Particularly, SyMat can achieve excellent performance in composition and structure validity, which demonstrates that SyMat can accurately learn the rules of forming chemically valid and stable periodic materials so it generates chemically valid periodic materials with high chances. Overall, SyMat achieves promising performance in the random generation task, showing that SyMat is an effective approach to capture the distribution of complicated periodic material structures.
We visualize some generated periodic materials in Figure 3 of Appendix E, and use additional metrics to evaluate the diversity of the generated periodic materials in Appendix D.3. To demonstrate the effectiveness of coordinate generation module in SyMat, we conduct an ablation study using several evaluation metrics in the random generation task and present this part in Appendix D.4.
4.3 Property Optimization
Metrics. In the property optimization task, we evaluate the ability of SyMat in discovering novel periodic materials with low energies. We jointly train the SyMat model with an energy prediction model that predicts the energies from the latent representations. Afterwards, we follow the evaluation procedure in [52] to optimize the latent representations of testing materials with 5000 gradient descent steps, decode 100 periodic materials from the optimized latent representations to periodic materials, and report the success rates (SR), i.e., the rate of decoded materials whose properties are in the top  $5\%$ ,  $10\%$ , and  $15\%$  of the energy distribution in the dataset. We use the same evaluation procedure to baseline methods and compare them with SyMat in optimization success rates.
Results. The property optimization performance results of all methods on three datasets are summarized in Table 2. On three datasets, SyMat outperforms all baseline methods in optimization success rates, showing that SyMat has the highest chances to generate periodic materials with low energies. This demonstrates that SyMat models have strong capacities in discovering novel periodic materials with good properties. The good performance of SyMat in the property optimization method shows that it can be an effective tool for designing novel periodic materials with desired properties.
We visualize the changes of material structures and energies of some periodic materials after property optimization in Figure 4 of Appendix E. We also visualize the material energy distribution before and after property optimization by SyMat models in Figure 5 of Appendix E.
We propose SyMat, a novel deep generative model for periodic material generation. SyMat achieves invariance to physical symmetry transformations of periodic materials, including permutation, rotation, translation, and periodic transformations. In SyMat, atom types and lattices of materials are generated in the form of atom type sets, lattice lengths and lattice angles with a VAE model. Additionally, SyMat uses a score-based diffusion model based on a symmetry-aware coordinate diffusion process to generate atom coordinates. Experiments on random generation and property optimization tasks show that SyMat is a promising approach for discovering novel materials. In the future, we will study the problem of accelerating atom coordinate generation in SyMat.
We thank Xiang Fu for his help on providing technical details about running baseline methods. This work was supported in part by National Science Foundation under grants IIS-1908220 and IIS-2006861.