SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[54] Minkai Xu, Wujie Wang, Shitong Luo, Chence Shi, Yoshua Bengio, Rafael Gomez-Bombarelli, and Jian Tang. An end-to-end framework for molecular conformation generation via bilevel programming. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11537-11547. PMLR, 18-24 Jul 2021.
[55] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. GeoDiff: A geometric diffusion model for molecular conformation generation. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=PzcvxEMzvQC.
[56] Zhao Xu, Youzhi Luo, Xuan Zhang, Xinyi Xu, Yaochen Xie, Meng Liu, Kaleb Dickerson, Cheng Deng, Maho Nakata, and Shuiwang Ji. Molecule3D: A benchmark for predicting 3D geometries from molecular graphs. arXiv preprint arXiv:2110.01717, 2021.
[57] Keqiang Yan, Yi Liu, Yuchao Lin, and Shuiwang Ji. Periodic graph transformers for crystal material property prediction. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=pqCT3L-BU9T.
[58] Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, pages 6410-6421. Curran Associates, Inc., 2018.
[59] Anthony Zee. Group theory in a nutshell for physicists, volume 17. Princeton University Press, 2016.
[60] Xuan Zhang, Limei Wang, Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, Meng Liu, Yuchao Lin, Zhao Xu, Keqiang Yan, Keir Adams, Maurice Weiler, Xiner Li, Tianfan Fu, Yucheng Wang, Haiyang Yu, YuQing Xie, Xiang Fu, Alex Strasser, Shenglong Xu, Yi Liu, Yuanqi Du, Alexandra Saxton, Hongyi Ling, Hannah Lawrence, Hannes Stärk, Shurui Gui, Carl Edwards, Nicholas Gao, Adriana Ladera, Tailin Wu, Elyssa F. Hofgard, Aria Mansouri Tehrani, Rui Wang, Ameya Daigavane, Montgomery Bohde, Jerry Kurtin, Qian Huang, Tuong Phung, Minkai Xu, Chaitanya K. Joshi, Simon V. Mathis, Kamyar Azizzadenesheli, Ada Fang, Alán Aspuru-Guzik, Erik Bekkers, Michael Bronstein, Marinka Zitnik, Anima Anandkumar, Stefano Ermon, Pietro Liò, Rose Yu, Stephan Gunnemann, Jure Leskovec, Heng Ji, Jimeng Sun, Regina Barzilay, Tommi Jaakkola, Connor W. Coley, Xiaoning Qian, Xiaofeng Qian, Tess Smidt, and Shuiwang Ji. Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv preprint arXiv:2307.08423, 2023.
[61] Yong Zhao, Mohammed Al-Fahdi, Ming Hu, Edirisuriya MD Siriwardane, Yuqi Song, Alireza Nasiri, and Jianjun Hu. High-throughput discovery of novel cubic crystal materials using deep generative neural networks. Advanced Science, 8(20):2100566, 2021.
A Proof of Propositions
We first prove Proposition 3.1, i.e., if the material distribution  $p(M)$  is invariant to permutation, rotation, translation, and periodic transformations described in Section 3.1, the score matrix  $\nabla_{P} \log p(P|A, L)$  will be invariant to translation and periodic transformations, and equivariant to permutation and rotation transformations.
(1) For  $M' = (A', P', L)$  where  $A'$  and  $P'$  are obtained by permuting the elements of  $A$  and column vectors of  $P$  with the same order, from  $p(M) = p(M')$ , we know that  $p(P'|A', L) = p(P|A, L)$  holds, so  $\nabla_{P'} \log p(P'|A', L)$  actually equals to the matrix obtained by permuting the column vectors of  $\nabla_P \log p(P|A, L)$  with the same permutation order. In other words,  $\nabla_P \log p(P|A, L)$  is permutation-equivariant.
(2) For  $M' = (A, P', L')$  obtained by rotating  $M = (A, P, L)$  with the rotation matrix  $Q$  and translating it with the translation vector  $b$ , because  $p(M') = p(M)$  and  $P' = QP + b1^T$ , we have  $p(P'|A, L') = p(P|A, L)$  and  $\nabla_{P'} \log p(P'|A, L') = Q \nabla_P \log p(P|A, L)$ . In other words,  $\nabla_P \log p(P|A, L)$  is translation-invariant and rotation-equivariant.
(3) For  $M^K = (A, P^K, L)$  where  $P^K = P + LK$  is obtained by periodic transformation, since  $p(M) = p(M^K)$ , we can easily find that  $p(P^K | A, L) = p(P | A, L)$  and  $\nabla_{P^K} \log p(P^K | A, L) = \nabla_P \log p(P | A, L)$  hold. In other words,  $\nabla_P \log p(P | A, L)$  is invariant to periodic transformations.
We next prove Proposition 3.2, i.e., if  $s_{i,j,k}$  is always invariant to permutation, rotation, translation, and periodic transformations described in Section 3.1, the score matrix  $\nabla_P \log p(P|A,L)$  computed by Equation (3) will always be invariant to translation and periodic transformations, and equivariant to permutation and rotation transformations.
(1) The score matrix  $\nabla_{\pmb{P}}\log p(\pmb {P}|\pmb {A},\pmb {L}) = [s_1,\dots,s_n]$  is formed by stacking coordinate score vector  $s_i$  according to the atom order, and this stacking operation is clearly equivariant to permutations. Also, from Equation (3), we can easily find that if  $s_{i,j,k}$  is always invariant to permutation, the computed coordinate score vector  $s_i$  for each coordinate  $\pmb{p}_i$  will also be invariant to permutations. Hence,  $\nabla_P\log p(P|A,L)$  from  $s_1,\ldots ,s_n$  is equivariant to permutations.
(2) For any  $Q \in \mathbb{R}^{3 \times 3}$ ,  $Q^T Q = I$  and  $b \in \mathbb{R}^3$ , let  $P' = [p_1', \dots, p_n'] = QP + b\mathbf{1}^T$  where  $p_i' = Qp_i + b$ ,  $i = 1, \dots, n$ ,  $L' = QL$ , and  $M' = (A, P', L')$ . We can calculate the score vector  $s_i'$  for  $p_i'$  in the way similar to Equation (3) as
$$
\begin{array}{l} \boldsymbol {s} _ {i} ^ {\prime} = \nabla_ {\boldsymbol {p} _ {i} ^ {\prime}} \log p (\boldsymbol {P} ^ {\prime} | \boldsymbol {A}, \boldsymbol {L} ^ {\prime}) \\ = \sum_ {(j, \mathbf {k}) \in \mathcal {N} ^ {\prime} (i)} \nabla_ {d _ {i, j, \mathbf {k}} ^ {\prime}} \log p (\boldsymbol {P} ^ {\prime} | \boldsymbol {A}, \boldsymbol {L} ^ {\prime}) \cdot \nabla_ {\boldsymbol {p} _ {i} ^ {\prime}} d _ {i, j, \mathbf {k}} ^ {\prime} \\ = \sum_ {(j, \boldsymbol {k}) \in \mathcal {N} ^ {\prime} (i)} s _ {i, j, \boldsymbol {k}} ^ {\prime} \cdot \frac {\boldsymbol {p} _ {i} ^ {\prime} + \boldsymbol {L} ^ {\prime} \boldsymbol {k} - \boldsymbol {p} _ {j} ^ {\prime}}{| | \boldsymbol {p} _ {i} ^ {\prime} + \boldsymbol {L} ^ {\prime} \boldsymbol {k} - \boldsymbol {p} _ {j} ^ {\prime} | | _ {2}}, \\ = \sum_ {(j, \mathbf {k}) \in \mathcal {N} ^ {\prime} (i)} s _ {i, j, \mathbf {k}} ^ {\prime} \cdot \frac {Q p _ {i} + \mathbf {b} + Q L \mathbf {k} - Q p _ {j} - \mathbf {b}}{\| Q p _ {i} + \mathbf {b} + Q L \mathbf {k} - Q p _ {j} - \mathbf {b} \| _ {2}}, \\ = \sum_ {(j, \boldsymbol {k}) \in \mathcal {N} (i)} s _ {i, j, \boldsymbol {k}} \cdot \frac {\boldsymbol {Q} (\boldsymbol {p} _ {i} + \boldsymbol {L} \boldsymbol {k} - \boldsymbol {p} _ {j})}{\| \boldsymbol {p} _ {i} + \boldsymbol {L} \boldsymbol {k} - \boldsymbol {p} _ {j} \| _ {2}}, \\ = Q s _ {i}, \\ \end{array}
$$
where  $\mathcal{N}'(i) = \{(j,k):(i,j,k)\in E(M')\}$ ,  $\mathcal{N}(i) = \{(j,k):(i,j,k)\in E(M)\}$ . Note that the second to the last equation holds because  $||Qx||_2 = ||x||_2$  holds for any  $\pmb{x}\in \mathbb{R}^3$  if  $Q^T Q = I$ , and the edge set of multi-graph representation and  $s_{i,j,k}$  is invariant to any rotation and translation transformations, hence  $E(M') = E(M)$  and  $s_{i,j,k}' = s_{i,j,k}$ . So far, we have shown that  $\nabla_P\log p(P|A,L)$  is invariant to translations and equivariant to rotations.
(3) For any  $\pmb{K} \in \mathbb{Z}^{3 \times 3}$ , let  $P^{\pmb{K}} = [p_1', \dots, p_n'] P + L\pmb{K}$  and  $M^{\pmb{K}} = (A, P^{\pmb{K}}, L)$ . Because the multi-graph method is strictly invariant to periodic transformations, there must exist a bijective mapping  $f: E(M) \to E(M^{\pmb{K}})$  such that for any  $(i,j,k) \in E(M)$ ,  $(i,j,k') = f(i,j,k)$  satisfies that  $p_i + Lk - p_j = p_i' + Lk' - p_j'$ . Also, because  $s_{i,j,k}$  is invariant to periodic transformations, we can easily find that calculating the score vectors for  $p_i$  and  $p_i'$  by Equation (3) are actually summing up the same set of vectors, so these two score vectors equal and then  $\nabla_P \log p(P|A,L)$  is invariant to periodic transformations.
SphereNet Model. In SyMat, we use SphereNet model [29] as the backbone GNN architecture for both the encoder model in the VAE for atom type and lattice generation and the score model in the score-based diffusion model for atom coordinate generation. SphereNet is a powerful 3D GNN model proposed for 3D molecular property prediction, but in our method, we will use the multi-graph representations of materials as inputs to SphereNet models. Generally, SphereNet model first initializes the embedding vectors of nodes and edges with a look-up atom type embedding vectors and spherical basis functions, then employ multiple message passing [11] layers to iteratively update the node and edge embeddings. Note that this overall process of initializing and updating embeddings is always invariant to all symmetry transformations described in Section 3.1. The encoder model of VAE model will use a sum pooling over the final node embeddings to obtain a global representation vector, which is passed into an MLP to obtain  $z^{A}$  and  $z^{L}$ . The score model will use another MLP to predict the edge distance score scalar from the final edge embeddings. Note that we use two independent SphereNet models whose parameters are not shared for the VAE encoder model and score model. We use the implementations of SphereNet in DIG package [26].
Obtaining lattice matrix from lattice lengths and angles. From the lattice lengths  $\ell = [\ell_1, \ell_2, \ell_3]$  and lattice angles  $\phi = [\phi_{12}, \phi_{23}, \phi_{13}]$ , let  $\gamma = \arccos \frac{\cos\phi_{23}\cos\phi_{13} - \cos\phi_{12}}{\sin\phi_{23}\sin\phi_{13}}$ , one lattice matrix  $L = [\ell_1, \ell_2, \ell_3]$  can be computed as
$$
\begin{array}{l} \boldsymbol {\ell} _ {1} = \left[ \ell_ {1} \sin \phi_ {1 3}, 0, \ell_ {1} \cos \phi_ {1 3} \right] ^ {T}, \\ \boldsymbol {\ell} _ {2} = \left[ - \ell_ {2} \sin \phi_ {2 3} \cos \gamma , \ell_ {2} \sin \phi_ {2 3} \sin \gamma , \ell_ {2} \cos \phi_ {2 3} \right] ^ {T}, \\ \boldsymbol {\ell} _ {3} = [ 0, 0, \ell_ {3} ] ^ {T}. \\ \end{array}
$$
Calculating the denoising score matrix. In the training of score-based diffusion model in SyMat, we add Gaussian noise with noise magnitude  $\sigma_{t}$  to the coordinate matrix of a material  $M = (A,P,L)$  in the dataset to obtain a noisy material  $\widetilde{M} = \left(A,\widetilde{P},L\right)$ , and the score model  $s_{\theta}(\cdot)$  is optimized to predict the denoising score matrix  $\nabla_{\widetilde{P}}\log p_{\sigma_t}\left(\widetilde{P} |P,A,L\right)$ . Then the score model  $s_{\theta}(\cdot)$  tries to predict the edge distance score  $\tilde{s}_{i,j,k} = \nabla_{\tilde{d}_{i,j,k}}\log p_{\sigma_t}\left(\widetilde{P} |P,A,L\right)$  for every edge  $(i,j,k)$  in the multi-graph representation of  $\widetilde{M}$ . Motivated from existing methods [30, 42, 55] in molecule generation, we can assume that the distribution of edge distance  $\tilde{d}_{i,j,k}$  in  $\widetilde{M}$  is a Gaussian distribution centered around the edge distance of the same edge in  $M$ . Practically, we find that using a new coordinate matrix  $\widehat{P} = [\hat{p}_1,\dots,\hat{p}_n]$  obtained by aligning  $P = [p_1,\dots,p_n]$  w.r.t.  $\widetilde{P} = [\tilde{p}_1,\dots,\tilde{p}_n]$  to calculate the corresponding edge distance in  $M$  can achieve the best performance. Specifically, we follow alignment procedure as in CDVAE [52] and calculate each  $\hat{p}_i$  as  $\hat{p}_i = p_i + L\boldsymbol{u}$ , where  $\boldsymbol{u} = \arg \min_{\boldsymbol{v}\in \mathbb{Z}^3}||\boldsymbol{p}_i + \boldsymbol{L}\boldsymbol{v} - \tilde{\boldsymbol{p}}_i||_2$ . The mean  $\hat{d}_{i,j,k}$  of the Gaussian distribution of the edge distance  $\tilde{d}_{i,j,k}$  is calculated as  $\hat{d}_{i,j,k} = ||\hat{p}_i + Lk - \hat{p}_j||_2$ . Besides, we assume the Gaussian distributions of  $\hat{d}_{i,j,k}$  for every edge have the same standard deviations  $\sigma_t$ , which is pre-computed by manually perturbing all materials in the dataset to noisy materials, collecting the edge distances in all noisy materials, and calculating the empirical standard deviation as  $\sigma_t$ .
C More Discussions
Limitations. Generally, there are three major limitations in our proposed SyMat approach. (1) First, SyMat may take as long as an hour to generate atom coordinates of periodic materials with score-based diffusion models, since this process requires to run thousands of Langevin dynamics sampling steps. Actually, this is a common limitation for many other molecules or material generation methods [42, 30, 52] using score-based diffusion models. Nonetheless, we argue that compared with the huge time cost of laborious lab experiments or DFT based calculation, the time cost of SyMat is relatively lower and acceptable. We will explore accelerating SyMat with SDE based diffusion models [45, 23] in the future. (2) Second, SyMat is designed to maintain invariance to periodic transformations, so it cannot be applied to non-periodic materials as this will lead to incorrect probabilistic modeling. Though some materials are non-periodic, such as amorphous solids and polymers, we focus on periodic materials because they occupy a large part of materials used in real-world applications. We believe that developing non-periodic material generation method need to rely on effective periodic material generation methods, but the problem of generating relatively simpler periodic materials still remains challenging and largely unsolved. Hence, we will focus on generating periodic materials in this work and leaves the problem of generating non-periodic materials to the future. (3) Third, our work follows previous material generation studies [52] to evaluate the quality of the generated materials by some basic metrics, such as composition and structure validity. However, they do not use metrics to evaluate the synthesizeability of the generated materials, though these metrics are useful for practical applications. In the future, we will collaborate with experts in material science to come up with metrics for evaluating synthesizeability of the materials generated by our method.
Broader impacts. In this work, we propose a novel method for periodic material generation. Our work can be used to design novel materials with desired properties for many real-world applications, such as batteries or solar cells. Nonetheless, our method may unexpectedly generate materials that produce negative impacts to human life. For example, manufacturing some materials generated by our model may cause environmental pollution. Hence, we believe strict evaluation or validation process need to be done to estimate the related environmental or other social impacts before using the materials generated by our model to real-world scenarios.
D.1 Additional Details in Experimental Settings
In our SyMat model, the SphereNet model is composed of 4 message passing layers and the hidden size is set to 128. Also, all MLP models in the VAE decoder is composed of 2 linear layers with a ReLU function between them, and the hidden size is set to 256. During training, we set the learning rate to 0.001, the batch size to 128, and the epoch number to 1,000. The total training needs around 3, 5, and 10 hours on a GTX 2080 GPU for Perov-5, Carbon-24, and MP-20 datasets, separately. We use different weights for different loss terms. Specifically, we set the weights of the reconstruction losses for atom type set size, atom types, atom numbers of each atom type, lattice items as 1.0, 30.0, 1.0, 10.0, respectively, and use a weight of 0.01 for KL-distance loss and a weight of 10.0 for denoising score matching loss. In the generation of atom coordinates with Algorithm 1, we set step size  $\epsilon$  as 0.0001 and step number  $q$  as 100. In addition, we use a geometrically decreasing series between 10 and 0.01 as  $\{\sigma_t\}_{t=1}^T$  and set  $T = 50$ . In the random generation task, for COV-R and COV-P metrics, we consider one material cover another material if the distances of their chemical element composition fingerprints and 3D structure fingerprints are smaller than the thresholds  $\delta_c$  and  $\delta_s$ , respectively. For Perov-5 dataset, we use  $\delta_c = 6$ ,  $\delta_s = 0.8$ ; for Carbon-24 dataset, we use  $\delta_c = 4$ ,  $\delta_s = 1.0$ ; for MP-24 dataset, we use  $\delta_c = 12$ ,  $\delta_s = 0.6$ .
D.2 Material Reconstruction
In the material reconstruction task, we reconstruct materials in the test sets of Perov-5, Carbon-24 and MP-20 datasets from the encoded latent representations  $z^A$  and  $z^L$ . We evaluate the reconstruction performance by several metrics. First, we evaluate the percentage of materials whose atom types can be exactly reconstructed (Atom Type Match Rate), and the root mean square error in lattice lengths and angles between the target and reconstructed materials (Lattice RMSE). Second, to show the score-based diffusion model has the capacity to approximately reconstruct 3D structures, we evaluate the root mean square error in interatomic distances between the target and reconstructed materials (Distance RMSE). We present the experimental results of SyMat together with baseline methods in Table 3. Results show that our method performs well in atom types, lattices and interatomic distances reconstruction.
Table 3: Reconstruction performance on Perov-5, Carbon-24, and MP-20 datasets. Here  $\uparrow$  means higher metric values lead to better performance, while  $\downarrow$  means the opposite. Note that Cond-DFCAVE can only be used to Perov-5 dataset in which materials have cubic structures. Bold and underline numbers highlight the best and second best performance, respectively.
Dataset	Method	Atom type match rate↑	Lattice RMSE↓	Distance RMSE↓
Perov-5	FTCP	99.67%	0.0786	0.2953
Cond-DFC-VAE	58.92%	0.0765	0.2281
CDVAE	98.16%	0.0231	0.1072
SyMat (ours)	98.30%	0.0224	0.0723
Carbon-24	FTCP	100.00%	0.1349	0.3759
CDVAE	100.00%	0.0624	0.1745
SyMat (ours)	100.00%	0.0632	0.1186
MP-20	FTCP	71.46%	0.1057	0.2062
CDVAE	68.32%	0.0532	0.0975
SyMat (ours)	72.10%	0.0510	0.0876
D.3 Diversity Evaluation
We evaluate the diversity of the randomly generated atom types by uniqueness percentage, i.e., the percentage of unique atom types among all the generated atom types. We present the uniqueness percentages of SyMat and baseline methods on Perov-5 and MP-20 datasets in Table 4. Note that Cond-DFC-VAE can only be used to Perov-5 dataset in which materials have cubic structures, and
Carbon-24 is not used here because all materials in Carbon-24 are composed of only carbon atoms, so the uniqueness is expected to be low. Results show that our method performs well in generating materials with diverse atom types.
Table 4: Uniqueness percentages on Perov-5 and MP-20 datasets.
Dataset	FTCP	Cond-DFC-VAE	G-SchNet	P-G-SchNet	CDVAE	SyMat
Perov-5	72.33%	80.36%	98.74%	98.57%	98.61%	99.43%
MP-20	79.04%	-	99.23%	99.03%	99.84%	99.98%
We conduct an ablation study experiment to show that the use of distance score matching in our coordinate generation module is significant. We implement a variant of SyMat in which the score model in coordinate generation module is trained by applying the denoising score matching loss directly to coordinates, instead of using our proposed distance score matching loss. We evaluate its performance in the random generation task by the structural validity, COV-R, and COV-P metrics as these metrics evaluate the quality of the generated atom coordinates. We summarize the results in Table 5. From the results, we can clearly find that using coordinate score matching always achieves lower COV-R and COV-P than using distance score matching. This demonstrates that distance score matching can more accurately capture the distribution of material structures, so the generated periodic materials are more similar to periodic materials in datasets, thereby achieving higher COV-R and COV-P.
Table 5: Comparison of SyMat with coordinate score matching and distance score matching. Here  $\uparrow$  means higher metric values lead to better performance. Bold numbers highlight the best best performance.
Dataset	Method	Structure validity↑	COV-R↑	COV-P↑
Perov-5	SyMat with coordinate score matching	100.00%	97.46%	95.32%
SyMat with distance score matching	100.00%	99.68%	98.64%
Carbon-24	SyMat with coordinate score matching	100.00%	96.72%	93.44%
SyMat with distance score matching	100.00%	100.00%	97.59%
MP-20	SyMat with coordinate score matching	100.00%	94.25%	95.61%
SyMat with distance score matching	100.00%	98.97%	99.97%
E Visualization Results
Perov-5
Carbon-24
MP-20
Figure 3: Illustrations of some periodic materials generated by SyMat models trained on Perov-5, Carbon-24, and MP-20 datasets.
-0.355
Figure 4: Illustrations of some periodic materials before and after being optimized by SyMat models in the property optimization task. Numbers show how energies (eV/atom) changes in optimization.
-0.521
(a) Perov-5 Dataset.
(b) Carbon-24 Dataset.
(c) MP-20 Dataset.
Figure 5: Illustrations of material energy (eV/atom) distribution before and after being optimized by SyMat models in the property optimization task.