SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here  $\varphi_{m}$  and  $\varphi_{h}$  are MLPs. The function  $\psi_{\mathrm{FT}}:(-1,1)^3\to [-1,1]^{3\times K}$  is Fourier Transformation of the relative fractional coordinate  $f_{j} - f_{i}$ . Specifically, suppose the input to be  $\pmb {f} = [f_1,f_2,f_3]^\top$  then the  $c$ -th row and  $k$ -th column of the output is calculated by
$$
\boldsymbol {m} _ {i j} ^ {(s)} = \varphi_ {m} \left(\boldsymbol {h} _ {i} ^ {(s - 1)}, \boldsymbol {h} _ {j} ^ {(s - 1)}, \boldsymbol {L} ^ {\top} \boldsymbol {L}, \psi_ {\mathrm {F T}} \left(\boldsymbol {f} _ {j} - \boldsymbol {f} _ {i}\right)\right), \tag {6}
$$
$$
\boldsymbol {m} _ {i} ^ {(s)} = \sum_ {j = 1} ^ {N} \boldsymbol {m} _ {i j} ^ {(s)}, \tag {7}
$$
$$
\boldsymbol {h} _ {\dot {i}} ^ {(s)} = \boldsymbol {h} _ {\dot {i}} ^ {(s - 1)} + \varphi_ {h} \left(\boldsymbol {h} _ {\dot {i}} ^ {(s - 1)}, \boldsymbol {m} _ {\dot {i}} ^ {(s)}\right). \tag {8}
$$
$\psi_{\mathrm{FT}}(\pmb {f})[c,k] = \sin (2\pi mf_c)$ , if  $k = 2m$  (even); and  $\psi_{\mathrm{FT}}(\pmb {f})[c,k] = \cos (2\pi mf_c)$ , if  $k = 2m + 1$  (odd).  $\psi_{\mathrm{FT}}$  extracts various frequencies of all relative fractional distances that are helpful for crystal structure modeling, and more importantly,  $\psi_{\mathrm{FT}}$  is periodic translation invariant, namely,  $\psi_{\mathrm{FT}}(w(\pmb {f}_j + \pmb {t}) - w(\pmb {f}_i + \pmb {t})) = \psi_{\mathrm{FT}}(\pmb {f}_j - \pmb {f}_i)$  for any translation  $\pmb{t}$ , which is proved in Appendix A.3.
After  $S$  layers of message passing conducted on the fully connected graph, the lattice noise  $\hat{\epsilon}_L$  is acquired by a linear combination of  $L$ , with the weights given by the final layer:
$$
\hat {\boldsymbol {\epsilon}} _ {L} = \boldsymbol {L} \varphi_ {L} \left(\frac {1}{N} \sum_ {i = 1} ^ {N} \boldsymbol {h} _ {i} ^ {(S)}\right), \tag {9}
$$
where  $\varphi_L$  is an MLP with output shape as  $3\times 3$ . The fractional coordinate score  $\hat{\epsilon}_{F}$  is output by:
$$
\hat {\boldsymbol {\epsilon}} _ {F} [:, i ] = \varphi_ {F} \left(\boldsymbol {h} _ {i} ^ {(S)}\right), \tag {10}
$$
where  $\hat{\epsilon}_{F}(:,i]$  defines the  $i$ -th column of  $\hat{\epsilon}_F$ , and  $\varphi_F$  is an MLP on the final representation.
We apply the inner product term  $L^{\top}L$  in Eq. (6) to achieve O(3)-invariance, as  $(QL)^{\top}(QL) = L^{\top}L$  for any orthogonal matrix  $Q\in \mathbb{R}^{3\times 3}$ . This leads to the O(3)-invariance of  $\varphi_L$  in Eq. (10), and we further left-multiply  $L$  with  $\varphi_L$  to ensure the O(3)-equivariance of  $\hat{\epsilon}_L$ . Therefore, the above formulation of the denoising model  $\phi (L,F,A,t)$  ensures the following property.
Proposition 3. The score  $\hat{\epsilon}_L$  by Eq. (9) is  $O(3)$ -equivariant, and the score  $\hat{\epsilon}_F$  from Eq. (10) is periodic translation invariant. Hence, the generated distribution by DiffCSP is periodic  $E(3)$  invariant.
Comparison with multi-graph representation Previous methods [9, 15, 29, 52] utilize Cartesian coordinates, and usually describe crystals with multi-graph representation to encode the periodic structures. They create multiple edges to connect each pair of nodes where different edges refer to different integral cell translations. Here, we no longer require multi-graph representation, since we employ fractional coordinates that naturally encode periodicity and the Fourier transform  $\psi_{\mathrm{FT}}$  in our message passing is already periodic translation invariant. We will ablate the benefit in Table 3.
Table 1: Results on stable structure prediction task.
# of samples	Perov-5	Carbon-24	MP-20	MPTS-52
Match rate↑	RMSE↓	Match rate↑	RMSE↓	Match rate↑	RMSE↓	Match rate↑	RMSE↓
RS [21]	20	29.22	0.2924	14.63	0.4041	8.73	0.2501	2.05	0.3329
5,000	36.56	0.0886	14.63	0.4041	11.49	0.2822	2.68	0.3444
BO [21]	20	21.03	0.2830	0.44	0.3653	8.11	0.2402	2.05	0.3024
5,000	55.09	0.2037	12.17	0.4089	12.68	0.2816	6.69	0.3444
PSO [21]	20	20.90	0.0836	6.40	0.4204	4.05	0.1567	1.06	0.2339
5,000	21.88	0.0844	6.50	0.4211	4.35	0.1670	1.09	0.2390
P-cG-SchNet [53]	1	48.22	0.4179	17.29	0.3846	15.39	0.3762	3.67	0.4115
20	97.94	0.3463	55.91	0.3551	32.64	0.3018	12.96	0.3942
CDVAE [9]	1	45.31	0.1138	17.09	0.2969	33.90	0.1045	5.34	0.2106
20	88.51	0.0464	88.37	0.2286	66.95	0.1026	20.79	0.2085
DiffCSP	1	52.02	0.0760	17.54	0.2759	51.49	0.0631	12.19	0.1786
20	98.60	0.0128	88.47	0.2192	77.93	0.0492	34.02	0.1749
Perov-5	Carbon-24	MP-20	MPTS-52
Ground Truth
DiffCSP
CDVAE
P-cG-SchNet
Figure 3: Visualization of the predicted structures from different methods. We select the structure of the lowest RMSE over 20 candidates. We translate the same predicted atom by all methods to the origin for better comparison. Our DiffCSP accurately delivers high quality structure predictions.
In this section, we evaluate the efficacy of DiffCSP on a diverse range of tasks, by showing that it can generate high-quality structures of different crystals in § 5.1, with lower time cost comparing with DFT-based optimization method in § 5.2. Ablations in § 5.3 exhibit the necessity of each designed component. We further showcase the capability of DiffCSP in the ab initio generation task in § 5.4.
5.1 Stable Structure Prediction
Dataset We conduct experiments on four datasets with distinct levels of difficulty. Perov-5 [54, 55] contains 18,928 perovskite materials with similar structures. Each structure has 5 atoms in a unit cell. Carbon-24 [56] includes 10,153 carbon materials with  $6\sim 24$  atoms in a cell. MP-20 [57] selects 45,231 stable inorganic materials from Material Projects [57], which includes the majority of experimentally-generated materials with at most 20 atoms in a unit cell. MPTS-52 is a more challenging extension of MP-20, consisting of 40,476 structures up to 52 atoms per cell, sorted according to the earliest published year in literature. For Perov-5, Carbon-24 and MP-20, we apply the 60-20-20 split in line with Xie et al. [9]. For MPTS-52, we split 27,380/5,000/8,096 for training/validation/testing in chronological order.
Baselines We contrast two types of previous works. The first type follows the predict-optimize paradigm, which first trains a predictor of the target property and then utilizes certain optimization algorithms to search for optimal structures. Following Cheng et al. [21], we apply MEGNet [52] as the predictor of the formation energy. For the optimization algorithms, we choose Random Search (RS), Bayesian Optimization (BO), and Particle Swarm Optimization (PSO), all iterated over 5,000 steps. The second type is based on deep generative models. We follow the modification in Xie et al. [9], and leverage cG-SchNet [53] that utilizes SchNet [29] as the backbone and additionally
considers the ground-truth lattice initialization for encoding periodicity, yielding a final model named P-cG-SchNet. Another baseline CDVAE [9] is a VAE-based framework for pure crystal generation, by first predicting the lattice and the initial composition and then optimizing the atom types and coordinates via annealed Langevin dynamics [49]. To adapt CDVAE into the CSP task, we replace the original normal prior for generation with a parametric prior conditional on the encoding of the given composition. More details are provided in Appendix B.2.
Evaluation metrics Following the common practice [9], we evaluate by matching the predicted candidates with the ground-truth structure. Specifically, for each structure in the test set, we first generate  $k$  samples of the same composition and then identify the matching if at least one of the samples matches the ground truth structure, under the metric by the StructureMatcher class in pymatgen [58] with thresholds  $\mathrm{stol} = 0.5$ , angle_tol=10, ltol=0.3. The Match rate is the proportion of the matched structures over the test set. RMSE is calculated between the ground truth and the best matching candidate, normalized by  $\sqrt[3]{V / N}$  where  $V$  is the volume of the lattice, and averaged over the matched structures. For optimization methods, we select 20 structures of the lowest energy of all 5,000 structures from all iterations during testing as candidates. For generative baselines and our DiffCSP, we let  $k = 1$  and  $k = 20$  for evaluation. We provide more details in Appendix B, C.1 and I.
Results Table 1 conveys the following observations. 1. The optimization methods encounter low Match rates, signifying the difficulty of locating the optimal structures within the vast search space. 2. In comparison to other generative methods that construct structures atom by atom or predict the lattice and atom coordinates in two stages, our method demonstrates superior performance, highlighting the effectiveness of jointly refining the lattice and coordinates during generation. 3. All methods struggle with performance degradation as the number of atoms per cell increases, on the datasets from Perov-5 to MPTS-52. For example, the Match rates of the optimization methods are less than  $10\%$  in MPTS-52. Even so, our method consistently outperforms all other methods.
Visualization Figure 3 provides qualitative comparisons.DiffCSP clearly makes the best predictions.
5.2 Comparison with DFT-based Methods
We further select 10 binary and 5 ternary compounds in MP-20 testing set and compare our model with USPEX [59], a DFT-based software equipped with the evolutionary algorithm to search for stable structures.
Table 2: Overall results over the 15 selected compounds.
Match rate (%)↑	Avg. RMSD↓	Avg. Time↓
USPEX [59]	53.33	0.0159	12.5h
DiffCSP	73.33	0.0172	10s
For our method, we sample 20 candidates for each compound following the setting in Table 1. We select the model trained on MP-20 for inference, with a training duration of 5.2 hours. For USPEX, we apply 20 generations, 20 populations for each compound, and select the best sample in each generation, leading to 20 candidates as well. We summarize the Match rate over the 15 compounds, the Averaged RMSD over the matched structures, and the Averaged Inference Time to generate 20 candidates for each compound in Table 10. The detailed results for each compound are listed in Appendix F. DiffCSP correctly predicts more structures with higher match rate, and more importantly, its time cost is much less than USPEX, allowing more potential for real applications.
We ablate each component of DiffCSP in Table 3, and probe the following aspects. 1. To verify the necessity of jointly updating the lattice  $\pmb{L}$  and fractional coordinates  $\pmb{F}$ , we construct two variants that separate the joint optimization into two stages, denoted as  $\pmb{L} \rightarrow \pmb{F}$  and  $\pmb{F} \rightarrow \pmb{L}$ . Particularly,  $\pmb{L} \rightarrow \pmb{F}$  applies two networks to learn the reverse processes  $p_{\theta_1}(L_{0:T-1}|A, F_T, L_T)$  and  $p_{\theta_2}(F_{0:T-1}|A, F_T, L_0)$ . During inference, we first sample  $L_T, F_T$  from their prior distributions, acquiring  $L_0$  via  $p_{\theta_1}$ , and then  $F_0$  by  $p_{\theta_2}$  based on  $L_0$ .  $F \rightarrow L$  is similarly executed but with the generation order of  $L_0$  and  $F_0$  exchanged. Results indicate that  $L \rightarrow F$  performs better than the  $F \rightarrow L$ , but both are inferior to the joint update in DiffCSP, which endorses our design. We conjecture that the joint diffusion fashion enables  $L$  and  $F$  to update synergistically, which makes the generation process more tractable to learn and thus leads to better performance. 2. We explore the necessity of preserving the O(3) invariance when generating  $L$ , which is ensured by the inner product  $L^\top L$  in Eq. (6).
When we replace it with  $\pmb{L}$  and change the final output as  $\hat{\epsilon}_{\pmb{L}} = \varphi_{L}\left(\frac{1}{N}\sum_{i=1}^{N}h_{i}^{(S)}\right)$  in Eq. (9) to break the equivariance, the model suffers from extreme performance detriment. Only 1.66% structures are successfully matched, which obviously implies the importance of incorporating O(3) equivariance. Furthermore, we introduce the chirality into the denoising model by adding sign  $(|\pmb{L}|)$ , the sign of the determinant of the lattice matrix, as an additional input in Eq. 6. The adapted model is SO(3)-invariant, but breaks the reflection symmetry and hence is NOT O(3)-invariant. There is no essential performance change, indicating the chirality is not quite crucial in distinguishing different crystal structures for the datasets used in this paper. 3. We further assess the importance of periodic translation invariance from two perspectives. For the generation process, we generate  $\pmb{F}$  via the score-based
model with the Wrapped Normal (WN) distribution. We replace this module with DDPM under standard Gaussian as  $q(\pmb{F}_t|\mathcal{M}_0) = \mathcal{N}\big(\pmb{F}_t|\sqrt{\bar{\alpha}_t}\pmb{F}_0, (1 - \bar{\alpha}_t)\pmb{I}\big)$  similarly defined as Eq. (3). A lower match rate and higher RMSE are observed for this variant. For the model architecture, we adopt Fourier Transformation(FT) in Eq. (6) to capture periodicity. To investigate its effect, we replace  $\psi_{\mathrm{FT}}(\pmb{f}_j - \pmb{f}_i)$  with  $\pmb{f}_j - \pmb{f}_i$ , and the match rate drops from 51.49% to 29.15%. Both of the two observations verify the importance of retaining the periodic translation invariance. 4. We further change the fully-connected graph into the multi-graph approach adopted in Xie and Grossman [15]. The multi-graph approach decreases the match rate, since the multi-graphs constructed under different intermediate structures may differ vibrationally during generation, leading to substantially higher training difficulty and lower sampling stability. We will provide more discussions in Appendix E.
Table 3: Ablation studies on MP-20. MG: Multi-Graph edge construction [15], FT: Fourier-Transformation.
Match rate (%) ↑	RMSE ↓
DiffCSP	51.49	0.0631
w/o Joint Diffusion
L → F	50.03	0.0921
F → L	36.73	0.0838
w/o O(3) Equivariance
w/o inner product	1.66	0.4002
w/ chirality	49.68	0.0637
w/o Periodic Translation Invariance
w/o WN	34.09	0.2350
w/o FT	29.15	0.0926
MG Edge Construction
MG w/ FT	25.85	0.1079
MG w/o FT	28.05	0.1314
5.4 Ab Initio Crystal Generation
DiffCSP is extendable to ab initio crystal generation by further conducting discrete diffusion on atom types  $A$ . We contrast DiffCSP against five generative methods following [9]: FTCP [28], Cond-DFC-VAE [7], G-SchNet [60] and its periodic variant P-G-SchNet, and the original version of CDVAE [9]. Specifically for our DiffCSP, we gather the statistics of the atom numbers from the training set, then sample the number based on the pre-computed distribution similar to Hoogeboom et al. [41], which allows DiffCSP to generate structures of variable size. Following [9], we evaluate the generation performance in terms of there metrics: Validity, Coverage, and Property statistics, which respectively return the validity of the predicted crystals, the similarity between the test set and the generated samples, and the property calculation regarding density, formation energy, and the number of elements. The detailed definitions of the above metrics are provided in Appendix G.
Results Table 4 show that our method achieves comparable validity and coverage rate with previous methods, and significantly outperforms the baselines on the similarity of property statistics, which indicates the high reliability of the generated samples.
**Limitation 1. Composition generation.** Our model yields slightly lower compositional validity in Table 4. We provide more discussion in Appendix G, and it is promising to propose more powerful generation methods on atom types. **2. Experimental evaluation.** Further wet-lab experiments can better verify the effectiveness of the model in real applications.
Conclusion In this work, we present DiffCSP, a diffusion-based learning framework for crystal structure prediction, which is particularly curated with the vital symmetries existing in crystals. The diffusion is highly flexible by jointly optimizing the lattice and fractional coordinates, where
<sup>3</sup>Composition-based metrics are not meaningful for Carbon-24, as all structures are composed of carbon.
Table 4: Results on ab initio generation task. The results of baseline methods are from Xie et al. [9].
Data	Method	Validity (%) ↑	Coverage (%) ↑	Property ↓
Struc.	Comp.	COV-R	COV-P	dρ	dE	delem
Perov-5	FTCP [28]	0.24	54.24	0.00	0.00	10.27	156.0	0.6297
Cond-DFC-VAE [7]	73.60	82.95	73.92	10.13	2.268	4.111	0.8373
G-SchNet [60]	99.92	98.79	0.18	0.23	1.625	4.746	0.0368
P-G-SchNet [60]	79.63	99.13	0.37	0.25	0.2755	1.388	0.4552
CDVAE [9]	100.0	98.59	99.45	98.46	0.1258	0.0264	0.0628
DiffCSP	100.0	98.85	99.74	98.27	0.1110	0.0263	0.0128
Carbon-24³	FTCP [28]	0.08	-	0.00	0.00	5.206	19.05	-
G-SchNet [60]	99.94	-	0.00	0.00	0.9427	1.320	-
P-G-SchNet [60]	48.39	-	0.00	0.00	1.533	134.7	-
CDVAE [9]	100.0	-	99.80	83.08	0.1407	0.2850	-
DiffCSP	100.0	-	99.90	97.27	0.0805	0.0820	-
MP-20	FTCP [28]	1.55	48.37	4.72	0.09	23.71	160.9	0.7363
G-SchNet [60]	99.65	75.96	38.33	99.57	3.034	42.09	0.6411
P-G-SchNet [60]	77.51	76.40	41.93	99.74	4.04	2.448	0.6234
CDVAE [9]	100.0	86.70	99.15	99.49	0.6875	0.2778	1.432
DiffCSP	100.0	83.25	99.71	99.76	0.3502	0.1247	0.3398
the intermediate distributions are guaranteed to be invariant under necessary transformations. We demonstrate the efficacy of our approach on a wide range of crystal datasets, verifying the strong applicability of DiffCSP towards predicting high-quality crystal structures.
This work was jointly supported by the following projects: the National Natural Science Foundation of China (61925601 & 62006137); Beijing Nova Program (20230484278); the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (23XNKJ19); Alibaba Demo Research Fund; CCF-Ant Research Fund (CCF-AFSG RF20220204); National Key R&D Program of China (2022ZD0117805).
[1] Gautam R Desiraju. Cryptic crystallography. Nature materials, 1(2):77-79, 2002.
[2] Keith T Butler, Daniel W Davies, Hugh Cartwright, Alexandr Isayev, and Aron Walsh. Machine learning for molecular and materials science. Nature, 559(7715):547-555, 2018.
[3] Walter Kohn and Lu Jeu Sham. Self-consistent equations including exchange and correlation effects. Physical review, 140(4A):A1133, 1965.
[4] Chris J Pickard and RJ Needs. Ab initio random structure searching. Journal of Physics: Condensed Matter, 23(5):053201, 2011.
[5] Tomoki Yamashita, Nobuya Sato, Hiori Kino, Takashi Miyake, Koji Tsuda, and Tamio Oguchi. Crystal structure prediction accelerated by bayesian optimization. Physical Review Materials, 2 (1):013803, 2018.
[6] Artem R Oganov, Chris J Pickard, Qiang Zhu, and Richard J Needs. Structure prediction drives materials discovery. Nature Reviews Materials, 4(5):331-348, 2019.
[7] Callum J Court, Batuhan Yildirim, Apoorv Jain, and Jacqueline M Cole. 3-d inorganic crystal structure generation and property prediction via representation learning. Journal of chemical information and modeling, 60(10):4518-4535, 2020.
[8] Wenhui Yang, Edirisuriya M Dilanga Siriwardane, Rongzhi Dong, Yuxin Li, and Jianjun Hu. Crystal structure prediction of materials with high symmetry using differential evolution. Journal of Physics: Condensed Matter, 33(45):455902, 2021.
[9] Tian Xie, Xiang Fu, Octavian-Eugen Ganea, Regina Barzilay, and Tommi S Jaakkola. Crystal diffusion variational autoencoder for periodic material generation. In International Conference on Learning Representations, 2021.
[74] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. EGSDE: Unpaired image-to-image translation via energy-guided stochastic differential equations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[75] Fan Bao, Min Zhao, Zhongkai Hao, Peiyao Li, Chongxuan Li, and Jun Zhu. Equivariant energy-guided SDE for inverse molecular design. In The Eleventh International Conference on Learning Representations, 2023.
1 Introduction 1
2 Related Works 2
3 Preliminaries 3
4 The Proposed Method: DiffCSP 3
4.1 Symmetries of Crystal Structure Distribution 3
4.2 Joint Equivariant Diffusion 4
4.3 The Architecture of the Denoising Model 6
5 Experiments 7
5.1 Stable Structure Prediction 7
5.2 Comparison with DFT-based Methods 8
5.3 Ablation Studies 8
5.4 Ab Initio Crystal Generation 9
6 Discussions 9
7 Acknowledgement 10
A Theoretical Analysis 18
A.1 Proof of Proposition 1 18
A.2 Proof of Proposition 2 19
A.3 Proof of Proposition 3 20
A.4 Discussion on Periodic Translation Invariance 21
B Implementation Details 22
B.1 Approximation of the Wrapped Normal Distribution 22
B.2 Adaptation of CDVAE 22
B.3 Algorithms for Training and Sampling 23
B.4 Hyper-parameters and Training Details 24
C Exploring the Effects of Sampling and Candidate Ranking 25
C.1 Impact of Sampling Numbers 25
C.2 Diversity 25
C.3Ranking among Multiple Candidates 25
D Impact of Noise Schedulers 26
E Learning Curves of Different Variants 26
F Comparison with DFT-based Methods 26
F.1 Implementation Details 26
F.2 Results 28
G Extension to More General Tasks 28
G.1 Overview 28
G.2 Ab Initio Generation 28
G.3 Property Optimization 30
H Computational Cost for Inference 32
I Error Bars 32
J More Visualizations 32
A Theoretical Analysis
We first introduce the following definition to describe the equivariance and invariance from the perspective of distributions.
Definition 4. We call a distribution  $p(x)$  is  $G$ -invariant if for any transformation  $g$  in the group  $G$ ,  $p(g \cdot x) = p(x)$ , and a conditional distribution  $p(x|c)$  is  $G$ -equivariant if  $p(g \cdot x|g \cdot c) = p(x|c), \forall g \in G$ .
We then provide and prove the following lemma to capture the symmetry of the generation process.
Lemma 1 (Xu et al. [10]). Consider the generation Markov process  $p(x_0) = p(x_T) \int p(x_{0:T-1}|x_t) dx_{1:T}$ . If the prior distribution  $p(x_T)$  is  $G$ -invariant and the Markov transitions  $p(x_{t-1}|x_t), 0 < t \leq T$  are  $G$ -equivariant, the marginal distribution  $p(x_0)$  is also  $G$ -invariant.
Proof. For any  $g \in G$ , we have
$$
\begin{array}{l} p (g \cdot x _ {0}) = p (g \cdot x _ {T}) \int p (g \cdot x _ {0: T - 1} | g \cdot x _ {t}) d x _ {1: T} \\ = p (g \cdot x _ {T}) \int \prod_ {t = 1} ^ {T} p (g \cdot x _ {t - 1} | g \cdot x _ {t}) d x _ {1: T} \\ = p \left(x _ {T}\right) \int \prod_ {t = 1} ^ {T} p \left(g \cdot x _ {t - 1} \mid g \cdot x _ {t}\right) d x _ {1: T} \\ = p (x _ {T}) \int \prod_ {t = 1} ^ {T} p (x _ {t - 1} | x _ {t}) d x _ {1: T} \\ = p \left(x _ {T}\right) \int p \left(x _ {0: T - 1} \mid x _ {t}\right) d x _ {1: T} \\ = p (x _ {0}). \\ \end{array}
$$
Hence, the marginal distribution  $p(x_0)$  is G-invariant.
The proposition Proposition 1 is rewritten and proved as follows.
Proposition 1. The marginal distribution  $p(L_0)$  by Eq. (3) is  $O(3)$ -invariant if  $\hat{\epsilon}_{\pmb{L}}(\mathcal{M}_t,t)$  is  $O(3)$ -equivariant, namely  $\hat{\epsilon}_{\pmb{L}}(\pmb{Q}\pmb{L}_{t},\pmb{F}_{t},\pmb{A},t) = \pmb{Q}\hat{\epsilon}_{\pmb{L}}(\pmb{L}_{t},\pmb{F}_{t},\pmb{A},t), \forall \pmb{Q}^{\top}\pmb{Q} = \pmb{I}$ .
Proof. Consider the transition probability in Eq. (3), we have
$$
p \left(\boldsymbol {L} _ {t - 1} \mid \boldsymbol {L} _ {t}, \boldsymbol {F} _ {t}, \boldsymbol {A}\right) = \mathcal {N} \left(\boldsymbol {L} _ {t - 1} \mid a _ {t} \left(\boldsymbol {L} _ {t} - b _ {t} \hat {\epsilon} _ {\boldsymbol {L}} \left(\boldsymbol {L} _ {t}, \boldsymbol {F} _ {t}, \boldsymbol {A}, t\right)\right), \sigma_ {t} ^ {2} \boldsymbol {I}\right),
$$
where  $a_{t} = \frac{1}{\sqrt{\alpha_{t}}}, b_{t} = \frac{\beta_{t}}{\sqrt{1 - \bar{\alpha}_{t}}}, \sigma_{t}^{2} = \beta_{t} \cdot \frac{1 - \bar{\alpha}_{t - 1}}{1 - \bar{\alpha}_{t}}$  for simplicity, and  $\hat{\epsilon}_{L}(\mathcal{M}_{t}, t)$  is completed as  $\hat{\epsilon}_{L}(L_{t}, F_{t}, A, t)$ .
As the denoising term  $\hat{\epsilon}_{L}(L_{t},F_{t},A,t)$  is O(3)-equivariant, we have  $\hat{\epsilon}_{L}(QL_{t},F_{t},A,t) = Q\hat{\epsilon}_{L}(L_{t},F_{t},A,t)$  for any orthogonal transformation  $Q\in \mathbb{R}^{3\times 3}$ ,  $Q^{\top}Q = I$ .