SimXRD-4M: Big Simulated X-ray Diffraction Data and Crystal Symmetry Classification Benchmark
Authors: Bin CAO
Yang Liu
Zinan Zheng
Ruifeng Tan
Jia Li
Tong-yi Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Symmetry Classification
benchmark and dataset
Primary area: datasets and benchmarks

Abstract: Powder X-ray diffraction (XRD) patterns are highly effective for crystal identification and play a pivotal role in materials discovery. While machine learning (ML) has advanced the analysis of powder XRD patterns, progress has been constrained by the limited availability of training data and established benchmarks. To address this, we introduce SimXRD, the largest open-source simulated XRD pattern dataset to date, aimed at accelerating the development of crystallographic informatics. We developed a novel XRD simulation method that incorporates comprehensive physical interactions, resulting in a high-fidelity database. SimXRD comprises 4,065,346 simulated powder XRD patterns, representing 119,569 unique crystal structures under 33 simulated conditions that reflect real-world variations. We benchmark 21 sequence models in both in-library and out-of-library scenarios and analyze the impact of class imbalance in long-tailed crystal label distributions. Remarkably, we find that: (1) current neural networks struggle with classifying low-frequency crystals, particularly in out-of-library situations; (2) models trained on SimXRD can generalize to real experimental data.

Published as a conference paper at ICLR 2025
SIMXRD-4M: BIG SIMULATED X-RAY DIFFRACTION DATA AND CRYSTAL SYMMETRY CLASSIFICATION BENCHMARK
Bin Cao'; Yang Zinan Zheng"; Ruifeng Tan 1.3, bant.,,jin Jia Li Tong-yi Zhang
1 Guangghouu Municipal Key Laboratory of Materials Informatics, Advanced Materials Thrust,
The Hong Kong University of Science and Technology (Guangzhou)
Data Science and Analytics Thrust, The Hong Kong University of Science and Technology (Guangzhou)
Sustainable Energy and Environment Thrust, The Hong Kong University of Science and Technology (Guangzhou)
"The Hong Kong University of Science and Technology
ABSTRACT
Powder X-ray diffraction (XRD) patterns are highly effective for crystal identi- fication and play pivotal role in materials discovery. Although machine learn- ing (ML) has advanced the analysis of powder XRD patterns, progress has been constrained by the limited availability of training data and established bench- marks. To address this, we introduce SimXRD-4M, the largest open-source simulated XRD pattern dataset date, aimed at accelerating the development of crystallographic informatics. We developed a novel XRD simulation method that incorporates comprehensive physical interactions, resulting high-fidelity database. SimXRD comprises 4,065,346 simulated powder XRD patterns, repre- senting 119,569 unique crystal structures under 33 simulated conditions that re- flect real-world variations. We benchmark 21 sequence models in both in-library and out-of-library scenarios and analyze the impact of class imbalance long- tailed crystal label distributions. Remarkably, we find that: (1) current neural networks struggle with classifying low-frequency crystals, particularly out-of- library situations; (2) models trained on SimXRD can generalize to real experi- mental data.
INTRODUCTION
Symmetry identification is fundamental and crucial step for materials characterization and de- sign. Specifically, Powder X-ray diffraction (XRD) analysis exceptionally potent in probing mi- crostructure, due to its sensitivity to atomic arrangement and the element specificity of atom scat- tering power. The diffraction pattern reflects the atomic arrangement of the diffracting material, as depicted in Figure serving fingerprint for the crystals.
Traditional methods involve a search-match pro- cess AAloomree et al. (008) among numerous known powder difffaction patterns. Given a target XRD pattern, iterates through candidates in databases until satisfactory alignment is achieved, known as the search-match method. Although this approach is prevalent, it is time-consuming (Altomare et al. 2008 Lv al.) 2024] International Centre for Diffraction Data) due to two issues: (1) high depen- dency on human intervention. Matching processes rely domain-specific programs that require fre- quent human-assisted tuning during program execu-
200
PSS0L
1500
Figure 1: The X-ray diffraction patterns of crystal TbyBaCoOs and PbSO
Equual contribution. Corresponding authors (jialee@uss ust.. mezhangt@hhuustgg..ddt

---

Published as a conference paper at ICLR 2025
tion; (2) diffraction is complex, mulli-physiccll coupling process, making structure analysis chal- lenging even for experienced domain experts.
Inspired by the remarkable progress of machine learning models, recent studies (Lee et al. 2020
Szymanski et al. 202T] Maffettone et al. 202T; Wang et al. 2020 Salgado et al., (023) have at- tempted to train neural networks simulate XRD datasets increase the efficiency of symmetry identification. They treat XRD patterns as sequences and aim to classify them into specific crystal
systems or space groups. More recent studies (Guo et al. 2024a b Li al., 2024] Cao al. [025] Riesel et al. 2024 Lai et al. 2024) on powder XRD for crystal prediction and generation have employed advanced ML architectures, achieving significant progress and noteworthy results.
Despite significant progress, several limitations cxist in current model development and evaluation:
Lack of large-scale and high-quality dataset: Previous research has largely been confined to specific materials (Fatimah et al. [2022] Lee et al.] 20200 Wang et al.] (020)) For example, Lee al. (Lee et al. 2020) focus on mixtures of 38 distinct binary and ternary crystals in the Sr-Li- AI-O inorganic compounds. Additionally, the geometric similarity of crystals and experimental physical factors can result in XRD patterns with similar peak distributions. The introduction of large-scale databases presents new challenges. Therefore, sufficient coverage of structures and close aligmment with experimental patterns under different physical conditions is necessary for developing models with high generalization ability.
Insufficient evaluation on real XRD patterns: symmetry from experimental measurements, most studies train and evaluate model performance on simulated data. Considering the potential bias between simulated and real XRD patterns, mod- els may overfit simulated data and fail to generalize to real data.
While symmetry classification aims to recognize
Limited exploration on out-library identification: Most studies focus on in-library identifica- tion. This involves training neural network to generalize across different experimental settings for the same set of crystals. In contrast, out-library identification aims to generalize to entirely new crystals. This approach is crucial for discovering novel materials but receives insufficient attention.
The limited scale of experimental data presents two significant challenges in this field: (1) insuf- ficient data to adequately test generative capabilities, and (2) difficulty in effectively utilizing such small-scale experimental datasets for model tuming, alone for training a model solely based on experimental data, To address this dilemma, we devel- oped high-fidelity simulation data solution. Our results demonstrate no observable differences in model performance between simulated and experimental test datasets, providing evidence of the high quality of the simulations. Crafting high-fidelity simulated XRD pattern dataset is challeng- ing due to the complexity of domain-specific processes, such as determining crystal stability, con- figuring instruments, and accounting for physical environmental factors. In this work, we present a novel XRD simulation method (Appendix (.44 that incorporates comprehensive physical interac- tions. Using this method, we developed SimXRD, the largest open- source and physically detailed dataset aimed at advancing this interdisciplinary field. We source crystals from the Materials Project (MP) (Jain et al., 2013) database and apply rigorous filtering to remove crystals with broken sym- metry, duplicates, or discrepancies in space group classification. Consequently, SimXRD comprises 4,065,346 X-ray powder diffraction patterns covering I 119,569 distinct crystal structures simulated under various conditions, including grain size, orientation, internal stress, inelastic scattering, ther- mal vibration, instrumental zero shift, noise and others.
within transfer learning framework, let
We benchmark both the in librirrrry and out-library symmetry identification performance on SimXRD of 21 models, which can be divided into three types of sequence models: convolution neural net- works (i.e., the backbone of existing symmetry classification models), recurrent models, and trans- formers. Through extensive experiments, we discover that:
Long-tail distribution: The class labels follow a heavy long-tailed distribution. Most models are biased toward the high-frequency classes and fail to predict the fre.-uueuuccy classes, especially in space group classification. To provide insights into model designs, evaluate the model performance under different objective functions and find that label smoothing and focal loss yield better results.
2

---

Published as a conference paper at ICLR 2025
Table I: Summaries of existing powder XRD datasets. ICSD refers the commercial Inorganic
Crystal Structure Database. MP denotes the open-sourced Material Project.
Dataset
Open Simulated Crystal Source
RRUFF LLffuente al 2015
3,002
3,002 169,530 536
2015
XRDSP (Sazuli PC
2012
169.536
ICSD
2020
CNN Lzee et
1,785.405
170
ICSD
2020
PONet [Tong
250.000
ICSD
2021
XRDAnto mallyne nalyzer Styminisk Pr mi
38,250
150
ICSD
2021
IlYDaAllYoNNeee Lee al. 2022
328.503
189,476.
139.021
ICSD&MM
2022
AivancdXRDAmmah
29.569.633
197.
ICSD
2023
CrySTINel (Chen 2012
100
100
ICSD
2024
CPICANINYCC
20124
692. 190
23.073
COD
2024
SimXRD
4,065,346
119,569
MP
2024
Out-library generalization: Out-library classification is challenging, as it requires models to learn from long-tailed data and generalize to XRD patterns of unobserved crystals. Thus, the model performance of out-library classification is generally reduced compared to that of in-library classification.
Experimental data generalization: The ability to generalize to real experimental data is a gold standard for symmetry identification models. Remarkably, we found that models trained on the proposed SimXRD are able to achieve comparable or even better results on the real dataset (RRUFF), validating the contribution of our dataset construction.
We highlight that our findings provide the research community a clearer insight into the cur- rent progress in XRD analysis. Additionally, we have made the SimXRD database, simulation code, benchmark models, evaluation process and tutorial notebooks into a repository: https: 7/github. com/ Bin-Cao: Caain--CooSiimRR SimXRD
RELATED WORK
2.1 EXISTING DATASETS
Several datasets have been developed to train neural models for crystallography. Table provides a summary of these datasets, compared across several key dimensions:
Dataset Size: Existing open-source databases either contain relatively small structures tailored to specific purposes (Chen al.) 2024 Dong et al. 2021; Szymanski et al. 2021 Lee et al. 2020 or
consider limited environmental settings (Chen et al. (2024) Suzuki et al. 2020). Crystal structure
databases form the backbone of XRD pattern databases. Although the database developed by
Lee et al. (Lee et al. 2022 2023) encompasses nearly all the crystals in the Inorganic Crystal Structure Database (ICSD) Zagorac et al., 2019) and the Materials Project (MP) (Jain et al., 2013), it contains a significant number of repeated crystals with broken symmetries. Additionally, it does sufficiently account for various physical conditions, resulting in a relatively small XRD pattern database. SimXRD, by contrast, employs the entire MP database of January 2024, totaling 154,718 crystal structures. During the generation of the SimXRD database, we applied screening methods to improve crystal quality. Additionally, we simulate XRD patterns under various environments to meet the academic-industry requirements.
Most pattern datasets are openly available, as indicated in Table though some datasets can only be retrieved upon reasonable request (Lee et al. 2020 2022 2023 Additionally, published diffraction pattern datasets are saved in various formats, requiring specific data loaders provided by the authors. In contrast, the SimXRD-4M dataset developed in our work is fully accessible and features an easier workflow for machine learning training, easily integrable with TensorFlow PyTorch frameworks.
Availability:
Among the existing datasets, only the RRUFF Project (La fuente al. 2015) aims to create comprehensive of high-quality spectral data from well- characterized minerals. Since its inception in 2005 (RRUFF 2005), RRUFF has compiled a col- lection of 3,002 high-quality experimental powder XRD patterns of March 2024. Conducting
Simulation vs. Experiments:
Union of ICSD and MP datahase, existing the structures.

---

Published as a conference paper at ICLR 2025
the experiments is time-consuming. Thus, recent studies have employed domain-speeiff soft- ware to generate simulated data. Earlier studies primarily relied on widely-used software tools, such Pymatgen (Ong al. 2013), FullProff (Rodriguez-Carvaji
2001), and GSAS-II (Toby & Von Dreele 2013), to simulate patterns for training. However, these approaches often showed varying degrees of performance degradation when applied to experimental test sets. SimXRD addresses these limitations by employing the newly developed PysimXRD, which accounts for a wider range of real-world conditions, including grain size, internal stress, external temperature variations, grain orientation, instrument drift, instrument noise, detector geometry, and scattering induced background. This comprehensive approach ensures higher fidelity in data generation, improving the robustness of the resulting models.
2.2 SYMMETRY IDENTIFICATION
Most existing methods for symmetry identification rely on one-dimensional convolutional neural networks for building classification models. The models developed in recent years have primarily focused on CNN architectures (Dong et al.] 2021 Wang et al.] 2020) Park et al. 2017). In partic- ular, they focus designing CNNs with deep layers Lee et al. 2022) ) and large kernel sizes (Le) iet al. (002)), ensemble CNNs (Maffettone et al. 2021 Szymanski et al. 2021). and CNNs with- dropout (Lee al., 2020) and pooling layers (Salgado et al., (023). However, whether CNNs can address long-tailed distributions, as well as the performance of other sequence models, remain underexplored.
3
SIMXRD-4M DATASET
3.1 PRELIMINARIES
Crystal symmetry Symmetry in crystallography fundamental property of the orderly arrange- ments of atoms in crystals (Su et al. 2024). Each atomic arrangement possesses certain symmetry elements, which are transformations that leave the arrangement unchanged. These elements include rotation, translation, reflection, and inversion. The specific symmetry elements present in a crys- talline solid determine its shape and influence physical properties. Crystals are classified based on their geometry and symmetry elements, falling into one of 7 crystal systems, and could further divide into 230 space groups (Clegg 2023). Further details are provided in AppendixB.1
XRD pattern Powder XRD patterns offer one-dimensional representation of the three- dimensional diffraction pattern and are the most common experimental measurement. Specifically, each one mie-dmmeniionl powder X-ray diffraction pattern in SimXRD is presented in the format of d-I (lattice plane distance-intensityy, where the x-axis represents lattice plane distances and the y-axis represents corresponding intensities, as shown in Figure] 2 The diffraction pattern we provide is ID tensor with 3501 elements. These points represent lattice plane distances, equally spaced from 1.199 Á to 8.853 A, with step size of 0.0176 A. This corresponds to typical experimental setup for a two-theta diffractometer (copper target), where the diffraction angle scanned from 10 to 80 with step size of 0.02%. The d-I pattern is independent of the wavelength of the incident X-rays, making it universally applicable across all experimental X-ray sources. Lattice plane distances de- note the detected atomic directions. Some distances lack peaks due to geometric constraints from the lattice cell and extinction effects (Lund et al. 2010)) under diffraction. The relative intensity is normalized, with the maximum intensity set 100, making the pattern independent of the incident X-ray intensity.
3.2 DATA GENERATION AND PROCESSING
Crystal data The crystal structures used this study were sourced from Material Project (Jain) et al. 2013), comprehensive, searchable database containing information on solid-state materials and molecules. It provides detailed data on the physical properties of these materials, such as elas- tic tensors, band structures, and formation energies, derived from electronic structure calculations, providing a more comprehensive reference for crystal studies. We utilize the latest dataset, denoted as MP-2024.1, 1, which includes total of 154,718 erystallographic structures as of January 2024.
4

---

Published as a conference paper at ICLR 2025
A
Crystal Database
Models
Classification
CNNs
Recurrent
Space groups SG,, SG2, SG.
XR Simulation
Ni ooientaiin
Transformers
MatMul
MatMul
Crystal systems CS,, CS2, ..., C,,CC,
tQ
fk V
Diffraction Rings
B
Physical
100
Experiment Simulation Residuals
Structures
setting
80
grain size
jm
(arb.u..)
60
stress
temperature
40
orientation
hackground drift noise
20
3
7
d (A))
Figure 2: (A) The workflow of the large simulated XRD database and and symmetry classification bench- mark consists of four main components: (1) Retrieval of crystal data from the Materials Project database. (2) Multiphysical coupling simulations to generate high-fidelity powder XRD patterns. (3) Application of three machine learning model types, CNNs, recurrent models, and transform- ers, for benchmarking the database.(4) Symmetry classification, which maps powder XRD patterns to specific space groups and crystal systems. (B) Experimental XRD pattern of a Li-rich layered oxide cathode (LizMnn,; was compared with simulated pattern generated using PysimXRD. The simulation incorporates multiphysical coupling, producing patterns that closely match experimental measurement with minimal residual errors.
Crystal Filtering To ensure consistency between recorded space group numbers (crystal systems) and atomic arrangements, and to enhance the quality of crystal structures, we performed thorough examination of each structure in the MP database. We used Spglib (Togo al. 22024). a library designed for identifying and managing crystal symmetries, to analyze all structures. Structures exhibiting broken symmetry, duplication, or discrepancies in space groups tionally, only structures containing up to 500 atoms per lattice cell (conventional unit cell) were retained, effectively covering nearly all inorganic materials in the MP dataset. A total of 119,569 crystal structures were screened. Further details are given Appendix A.
excluded. Addi-
Simulation in SimXRD Figure illustrates the overall simulation process. The experimental XRD patterns depend on both the crystal's intricate structure and practical factors, including the sample's state and instrumental parameters. Our goal is to simulate this process using domain- specific custom software Pysimxrd (Appendix [..4). First, we specify the environmental settings and then compute the XRD patterns for the given crystals. By varying the environmental param-
5

---

Published as a conference paper at ICLR 2025
eters, generate all samples in the SimXRD dataset. We systematically analyze the physical conditions affecting the XRD profiles, categorizing them into factors that influence the sample's internal structure and those affect instrumental measurements. These parameters, along with their respective value ranges, are outlined in the Appendix [8.4, For each input crystal, we randomly combine these physical conditions within reasonable ranges, repeating the process 33 times sim- ulate XRD patterns across diverse environments. Each XRD pattern is standardized, with the x-axis values (lattice plane distance) uniformly set based on the experimental settings, resulting in a vector of 3501 dimensions. The y-axis values (diffraction intensity) are simulated according to the crystal structure and diffraction conditions.
3.3 DATA ANALYSIS
Long-tailed distribution
The distribution of space groups logarithmic scale is shown in Fig- The space groups in the MP dataset are ranked in descending order, and their frequencies plotted on a logarithmic scale. This reveals differences in frequency spanning several orders of magnitude. The data clearly exhibit a highly imbalanced, long-tailed distribution. Since MP-2024.1 serves as comprehensive database of commonly occurring material systems, this distribution im- balance is inherent to the physical population of crystals in nature. This imbalance can lead to biased model performance, as the model may be more inclined to predict the majority class, leading to lower accuracy for the minority classes. Since minority crystals are also important for material discovery, addressing long-tailed sequence learning is crucial for symmetry identification. However, existing models do not consider this challenge, making it necessary to evaluate their performance across different classes. The detailed space group distribution and the energy distribution crystals presented in Appendix(A.]
ure 3
Case study A case study is conducted to investi- gate the difference between experimental and simu- lated XRD patterns of LizMnO3. Initially, the phys- ical states of the tested sample and the experimen- tal settings were obtained using a refinement method (see Appendix B.7). Subsequently, the XRD pattern was generated inversely using the Pysimxrd. Fig- ure (B) shows the results where we can observe that the generated profile exhibits extreme similarity to the experimental measurements(Lei et al.] 22024). Therefore, by altering the physical states within reasonable range, other patterns corresponding to different experimental settings can be generated. Such simulation allows us to obtain high-quality data relatively efficient manner, compared to experimental patterns.
10-
Ingt Frequeenee)))))
Space Groups
Figure 3: The distribution of space groups logarithmic scale
BENCHMARK
4.1 PROBLEM DEFINITION
Symmetry classification Formally, SimXRD treats the problem as multi-class sequence classi- fication task. Given the XRD pattern X = [21, x2, ... xr,] E I." where Σ; is the i-th intensity value and X is arranged by lattice plane distance, the goal is predict its symmetry Y R Here n is the feature dimension which is 3501 in our dataset. For crystal system classification, k = 7, and for space group classification, k 230. We did not consider practical factors such as average grain size and stress as input variables, since these are not directly measurable in real-world diffraction experiments.
In/Out-library crystallographic tasks In-library classification aims to generalize across different experimental environments for the same set of structures. Consequently, the training and testing datasets consist the same structures but with varying simulation parameters. In contrast, out-library classification seeks to generalize to different crystals. Therefore, the training and testing datasets consist of different types of crystals. More details are provided in Appendix B.2

---

Published as a conference paper at ICLR 2025
Table 2: Results of in library crystal system classification and space group classification. Inference time is measured for batch size of 100 samples and experiments are run on GeForce RTX 3090 GPU.
Model # Conv. Deopont Pooling Ensemble
Accuracy
Crystal System
Precision
(ms). Accuracy
Precision Recall
Group
(ms)
CNNI
JagPool
Park
0.427
0.241
CNN2
LAFer
0.456
0.222
0.341
CNN3
MaxPool
X
40.0
0.551
0.328
0.315
0.369
3.5
0.241
0.022
0.001
0.05
3.6
CNN4
V
MaxPool
1070
0.316
0.069
0.045
0.143
144
0.241
0.002
0.001
0.005
14
CNN5
AngRool
Maneutooe 21
2021
0.517
0.394
0.439
0.378
0.5
0.295
0.022
0022
0.026
0.6
CNN6
MaxPool
EX15
0.316
0.099
0.045
0.143
65.0
0.241
0.022
0.001
0,005
65.0
CNN7
MaxPool
0.588
0.147
CNN8
MaxPool
CNN9
0.526
0.599
CNNIO
MaxPoel
0.870
0.892
0.705
CNNII
None
Sagian 26
0.902
0.922
0.532
0.914
4.8
0.758
0.750
0.735
0.828
MLP
0.316
0.069
0.045
0.143
1.6
0.241
0.022
0.001
0.005
1.6
RNN
0.361
0. 183
0.200
0.202
8.0
0.245
0.003
0/002
0.007
8.1
LSTM
0.728
0.741
0.762
0.728
14.6
0.515
0.156
0.224
0.151
14.6
GRU
0.765
0.788
0.902
0.777
0.575
0.273
0:400
0.251
15.1
Bidirectionl--NN.
0.365
0.197
0.245
0,002
Bidirectional-L
0.791
0.559
Ridircctional-CRI
0.800
0.826
0.540
0.816
30.3
0.627
0/60)
0.408
Transformer
0.338
0.172
0.241
0.001
Hransformer
0.627
0.611
0,652
0.399
1,9
0.388
0.135
0.320
0.118
1.9
PatchTST
0.720
0.752
0.766
0.740
3,3
0.631
0.811
0,850
0.781
3,3
4.2 EXPERIMENTAL SETTING
Dataset split For in-library classification, fundamental task in crystallography, the dataset is randomly split according to the types of simulated environments, resulting in 119,569 x 30 training instances, 119,569 x validation instances, and 119,569 x testing instances. The data recorded for training, validation, and testing include crystals from different simulated environments (e.g., grain size, stress, temperature). Under out-of-library settings, the training and testing XRD patterns are generated from non-overlapping crystals. This setup yields 83,698 x 33 training instances, 11,957 33 validation instances, and 23,914 33 testing instances.
Baselines
We consider the following three types of sequence classification models:
We evaluate all existing CNN architectures proposed for symmetry identi- fication. Since most models do not have specific names, we classify them based on their number of convolution layers, pooling layers, and whether they use ensemble learning or dropout layers. Table summarizee the existing 11 CNN models.
CNN-based Models:
We select three basic recurrent neural networks: RNN (Medsker & Jain, 1999), LSTM (Hochreiter & Schmidhuber) 1997). and GRU (Chung et al., 2014). Additionally, since XRD patterns can be viewed from both forward and backward, we also evaluate the perfor- mance of bidirectional RNN, LSTM, and GRU (Schuster Paliwal 1997).
Recurrent Models:
Transformers: Transformers have proven effective in variety of sequence modeling tasks. We evaluate the performance of the raw transformer (Vaswani et al.] (017)) and two advance trans- former models iTransformer (Liu et al. 2023) and PatchTST (Nie et al. 2023).
Besides these models, we also include MLP as a straightforward baseline. For recurrent models and transformers, we first use them to learn sequence representations, and then employ an MLP to predict the symmetry.
Implementation details We use the following size of 128 and learning rate of 2.5 x 10-4 All models are trained for 50 epochs with an early stopping patience 3. We use the Cross-Entropy function to measure the difference between pre- dictions and the ground truth. The performance of the models is evaluated using accuracy, macro Fl-score, macro precision, and macro recall as metrics. All models are implemented using the PyTorch (Pasıke et al. 2019) library and trained on GeForce RTX 3090 GPU.
across all experiments: Batch
7

---

Published as a conference paper at ICLR 2025
Table 3: Results of weighted classification, label smoothing, and Focal loss crystal system and
space group classification.
Crystal System Classification
Training F1
Weighted classification
Label smoothing
Focal loss
Metric
Precision
Recall
F1
Precision
Recall
Fl
Precision
Recall
CNNT
0.565
0.538
0.642
0.578
0.587
0.575
0.607
0.608
0.610
CNN2
0.690
0.659
0.734
0.497
0.493
0.506
0.511
0.506
0.518
CNN3
0.699
0.665
0.765
0,723
0.765
0.698
0.536
0.529
0.546
0.191
0.390
0.043
CNNS
0.397
0,379
0.468
0.382
0.431
0.375
0.313
0.339
0.321
CNN6
0,069
0.046
0.142
0.069
0.046
0.142
0.069
0.046
0.142
CNN7
0.740
0.710
0.793
0.863
0.887
0.844
0.851
0.871
0.836
CNN8
0.546
0.533
0.625
0.566
0.590
0.559
0.554
0.687
0.547
CNN9
0.780
0,748
0.824
0.802
0.820
0.788
0.814
0.803
0.807
CNNI0
0.800
0.767
0.848
0.877
0.898
0,860
0.844
0.857
0.834
0.811
0.797
0.845
MLP
0.069
0,046
0,142
0.069
0.046
0.142
0.069
0.046
0.142
RNN
0.154
0.184
0.236
0.180
0.200
0.196
0.197
0.187
LSTM
0.086
0.153
0.171
0.706
0.721
0.697
0.701
0.712
0.701
GRU
0.769
0.737
0.821
0,783
0.799
0.771
0.727
0.742
0.717
Bidirectional-RNN
0.169
0.184
0.238
0.144
0.192
0.176
0.159
0.150
0.179
Addinoctinaal-STTe
0.717
0.684
0,789
0.795
0,819
0.775
0.622
0.653
0.602
Bidirectional--Ge
0,863
Transformer
0.134
0.161
0.178
0.130
0.169
0.166
0.
0.171
0.156
Transformer
0.545
0.516
0.643
0.590
0,580
0.604
0.630
PatchTST
0.672
0.643
0.745
0,710
0,735
0.691
0.714
0.722
0.709
Space Group Classilication
CNNI
0.000
0,000
0.005
0,002
0.001
0,005
0.002
0.001
0.005
CNN2
0.000
0.000
0.005
0.002
0.001
0.005
0.002
0.001
0.005
CNN3
0.000
0.000
0.005
0.002
0.001
0.005
0.002
0.001
0.005
CNN4
0.001
0,000
0.005
0.002
0.001
0.005
0.002
0.001
0.005
CNNS
0.033
0.028
0.056
0.005
0.001
0.005
CNN6
0,000
0
0.005
0.002
0.001
0.005
0.002
0.001
0.005
CNN8
0.000
0.000
0.005
0.002
0.001
0.005
0.002
0.001
0.005
CNN9
0.391
0.325
0.739
0.370
0.648
0.303
0.582
0.674
0.535
CNN10
0.200
0.223
0.649
0.542
0,806
0.452
0.774
0,851
0.732
CNNTI
0,000
0.000
0,005
0,615
0,836
0.525
0.751
0.741
0.815
0.000
0.000
0.005
0.002
0.001
0.005
0.002
0.001
0.005
RNN
0.000
0.002
0.006
0.003
0.001
0.006
0.003
0.001
0.005
GRU
0.215
0,180
0.610
0.185
0.261
0.176
0.227
0.192
0.653
Bidirectional-RNN
0.004
0.0016
0.010
0.006
0.004
0.005
0.009
Bidirectional- -SSM
0.148
0.135
0.518
0.177
0.278
0.166
0.177
0.275
0.159
Bidirectional-GRU
0.088
0.099
0,276
0,297
0.475
0.246
0.472
0.638
0.416
Transformer
0.001
0.001
0.006
0.002
0.002
0.005
0.002
0.002
0.005
Transformer
0.083
0.096
0.387
0.092
0,172
0.090
0.173
0.365
0.149
PatchTST
0.121
0.123
0.386
0.609
0.808
4.3 RESULTS
In-library classification Table 2 displays the baseline performance and inference time for the classification of crystal systems and space groups. Based on the results, we make the following observations:
Most existing CNN models are unsuitable for symmetry identification within the large scale of the SimXRD database. is clear that the crystal system classification accuracies of several CNNs. e.g., CNN 4 and 6 are on par with that of MI.P. As shown in the additional results in Appendix[AA. the reason for this is that their predictions are heavily biased by high frequency classes, making them unable to predict low-frequency classes effectively. This limitation is even more apparent in space group classification, where the accuracy of most CNNs (i.e., CNN 1-4, 6, and 8) is 0.241, indicating that their outputs almost fixed to the most frequent space group.
Convolutional neural networks without pooling have achieved the best performance in most tasks, consistent with the findings of previous work (Salgado et al. 2023). Since the peak maximum and the relative value among peaks in XRD patterns closely related to the atom arrangement of crystals, employing pooling layers may lead to information loss, which can affect peak identifica- tion and result in lower accuracy.
8

---

Published as a conference paper at ICLR 2025
Table 4:
Results of out-library symmetry identification.
Task
Crystal System Classification
Space Group Classfiiction
Model
Accuracy
F1
Precision
Recall
Accuracy
F1
Precision
Recall
CNNI
0.627
0.437
0.565
0.472
0,285
0.003
0.002
0.006
CNN2
0.606
0.419
0.428
0.425
0.285
0.003
0.002
0.006
CNN3
0.659
0.501
0.496
0.507
0,285
0.003
0.002
0.006
CNN4
0.378
0.078
0.054
0.142
0.285
0.003
0.002
0.006
CNN5
0.495
0.278
0.285
0.28.3
0,285
0.003
0.002
0.006
0,285
CNN7
0.673
0.607
0.633
0.608
0,429
0.053
0.050
0.072
CNN8
0.612
0.452
0.448
0.497
0,286
0.003
0.001
0.006
CNN9
0.675
0.632
0.629
0.644
0.439
0.099
0.137
0.113
CNN10
0.692
0.659
0.650
0.674
0,430
0.107
0.168
0.121
CNN11
0.702
0.672
0.659
0.690
0.481
0.136
0.167
0.150
MLP
0.378
0.078
0.054
0.142
0,285
0.003
0.002
0.006
0.409
0.149
0,274
0.009
LSTM
0.657
0.575
0.583
0.589
0,431
0.070
0.092
0,085
0.707
0,480
0.143
0.125
Bidirectionl-RNN
0.403
0.157
0.146
0.174
0.295
0.004
0.003
0.008
NadirectionllLLSTR
0.704
0.663
0.654
0.678
0,349
0.035
0.044
0.051
Bidirectional GRU
0.722
0.699
0.697
0.705
0.498
0.138
0.192
0.149
furmer
0.376
0.138
0.231
0.158
0,285
0.003
0.005
0,006
PatchTST
0.656
0.616
0.612
0.627
0.375
0.067
0.093
0.082
Bidirectional recurrent models consistently outperform their unidirectional counterparts. These results highlight an important physical attribute of XRD patterns: ensuring the consistent retrieval crystal information through bidirectional pattern reading. This property distinct from previ- ous sequence tasks such text and time-series classification, necessitates further model design considerations.
The raw transformer encounters difficulties when dealing with the long-tailed sequence prob- lem. However, compared to the raw transformer, we observe that PatchTSTT achieves significant performance improvement, demonstrating that subsequence-level patches are beneficial for peak identification and further enhancing classification performance.
Long-tailed objective functions The long-tailed distribution significantly impedes the symmetry identification of XRD patterns. To address this issue, we further train all baseline models using the following techniques: (1)Weighted classification that reweights training instances based on the inverse of class frequency; (2) Label smoothing that applies cross-entropy with targets smoothed towards a uniform distribution; (3) Focal loss that focuses on hard-to-classify examples by down- weighting the loss for easy examples. The results are presented in Table 3 These results show that weighted classification does not improve model performance. This lack of improvement is likely because increasing the importance of minority classes leads to a decrease accuracy for the majority classes. In contrast, label smoothing and focal loss generally generally yield better results for both crystal system and space group classification, suggesting promising research directions for model designs.
Out-library classification Different splitting mechanisms are essential for preliminary experi- ments. Consequently, conduct experiments using out-of-library settings, where the training and testing XRD patterns come from non-overlapping structures. To ensure that the training, validation, and test datasets consist of distinct crystals, we randomly partition the dataset by crystal type. The results are detailed in Table Based on the findings, we have the following observations:
Performance decrease compared to in-library classification. This decline anticipated, as out-of- library classification requires models generalize across different crystal types, which presents greater challenge.
The relative model performances align with the results from in-library identification, with Bidirectional-GRU and CNN11 achieving higher scores. However, there is noticeable decline in out-of-library cases, as the models need to account for varying extinction effects across different space groups, which introduces significant challenges.
Models such as CNN1-4, CNN6, CNN8, MLP, and Transformer exhibit poor performance in out-of-library identification, with accuracy scores around 24%, close random prediction. This

---

Published as a conference paper at ICLR 2025
Table 5: Generalization experiments RRUFF dataset.
Task
Crystal System Classification
Space Group Classification
Model
Accuracy
F1
Precision
Recall
Accuracy
F1
Precision
Recall
CNNI
0.623
0.542
0.563
0.539
0.241
0.004
0.002
0.010
CNN2
0.425
0.210
0.178
0.143
0.241
0.004
0.002
0.010
CNN3
0.506
0.313
0.397
0.332
0.241
0.004
0.002
0.010
CNN4
0.297
0.065
0.042
0.143
0.241
0.004
0.002
0.010
CNNS
0,330
0.402
0.511
0.384
0,302
0.043
0.045
0.051
CNN6
0.297
0.065
0.042
0.143
0.241
0.004
0.002
0.010
CNN7
0.868
0.877
0.899
0.860
0.577
0.202
0.205
0.222
CNN8
0.297
0.065
0.042
0.143
0.241
0.004
0.002
0.010
CNN9
0.792
0.819
0.832
0.809
0.615
0.500
0.555
0.486
CNN10
0.669
CNNTI
0.893
0.920
0.930
0.914
0.766
0,700
0.754
0.687
MLP
0.325
0.070
0.046
0.143
0.241
0.004
0.002
0.010
RNN
0.377
0.165
0.203
0.185
0.245
0.006
0.004
0.013
LSTM
0.723
0.735
0.747
0.731
0.522
0.230
0.275
0.230
GRU
0.774
0.421
0.357
Bidirectional-NNS
0.365
0.152
0.190
0.184
0.245
0,005
0.003
0.013
Bidirectional- LSTM
0.788
0.820
0.354
Ridirectional GRU
0.804
0.845
0.864
0.834
0.636
0,448
0.512
0,429
Transformer
0.354
0.138
0.219
0.165
0.241
0.004
0.002
0.010
iTransfommrr
0.629
0.612
0.658
0.607
0.392
0,143
0.156
0.151
PatchTST 0.723
0.764
0.785
0.750
0.635
0.696
0.748
0.687
underscores the need for further development of -ppaiin-ppecifc models to effectively address the challenges of oft-or-library identification.
Experimental data generalization A generalization experiment is conducted using the experi- mental RRUFF dataset. We train and validate all baseline models on SimXRD and then test their performance on RRUFF. The results are presented in Table 5 The findings indicate that the models generally achieve consistent performance on both SimXRD and RRUFF. Notably, CNN 11 consis- tently delivers the best accuracy, surpassing its performance SimXRD (e.g.. CNN 11 in space group classification). Models that exhibited lower accuracy on SimXRD (c.g.. CNN 1-6) also perform poorly RRUFF.
CONCLUSION
In this paper, we introduce SimXRD, the largest open-source XRD pattern dataset for symmetry identification. Data analysis reveals that the symmetry labels follow long-tailed distribution. We evaluate models on two different splitting patterns (in-library and out-of-library) and find that most existing models struggle to accurately predict the symmetry of low-frequency classes, even when addressing for class imbalance. This limitation hinders their real-world applicability. Our results emphasize the importance of modeling long-tailes sequence classification and conducting comprehensive comparison to accurately assess the capabilities of various models. To promote further research and advancements in symmetry identification algorithms, make both SimXRD and the simulation code available as open-source.
Limitations While SimXRD improves upon previous datasets terms of sample size, crystal quality, and accessibility, the long-tailed and imbalanced label distribution remains inherent limi- tation. The distribution of crystallographic symmetry arises through natural processes and is invari- ant to human intervention, making such biases difficult to address any dataset. Potential solutions include developing insalaebbd algorithms or incorporating additional information about the target crystals.
Future works We view SimXRD as evolving project and are committed to its ongoing devel- opment. Future plans include: (1) Developing long-tailed sequence classification models through techniques such data augmentation (Chu et al. 2020: Zheng et al.) 2024) or enhancing features via graph inputs] Liu et al. (024a)): (2) Exploring training approaches like multi-stage learning to address the imbalanced nature of XRD patterns; (3) Designing data augmentation strategies Liu iet al. (2024b) for XRD patterns, while ensuring that methods like random swapping do not generate instances that violate underlying physical principles.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by the Guangzzhou-HKKSS((Z)) Joint Funding Program (Nos. 2023A03J0003 and 2023A03J0103) and is funded by National Natural Science Foundation of China Grant No. 72371217. the Guangzhou Industrial Informatic and Intelligence Key Laboratory No. 2024A03J0628, the Nansha Key Area Science and Technology Project No. 20232D003, and Project No. 2021JC02X191.
REFERENCES
Angela Altomare, Corrado Cuocci, Carmelo Giacovazzo, Anna Moliterni, and Rosanna Rizzi. Qualx: a computer program for qualitative analysis using powder diffraction data. Journal of Applied Crystallography, 41(4))815-817, 2008.
BH Armstrong. Spectrum line profiles: the voigt function. Journal of Quantitative Spectroscopy and Radiative Transfer, 7(1):61-88, 1967.
G Caglioti, AT Paoletti, and FP Ricci. Choice of collimators for crystal spectrometer for neutron diffraction. Nuclear Instruments, 3(4):223-228, 1958.
Barnie Cao, Daniel Anderson, and Luke Davis. Asugnn: asymmetric-unit-haset graph neural network for crystal property prediction. Applied Crystallographyy 2025.
Bin Cao. http:////w...aaoa..../////////aa 2024. Accessed: 2024-05-21.
ht
Litao Chen, Bingxu Wang, Wentao Zhang, Shisheng Zheng, Zhefeng Chen, Mingzheng Zhang, Cheng Dong. Feng Pan, and Shunning Li. Crystal structure assignment for unknown compounds from x-ray diffraction patterns with deep learning. Journal of the American Chemical Society, 146(12) (206112))898811111 8109, 2024.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long-tailed data. In ECCV, volume 12374, pp. 694-710, 2020.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of 2014.
gated recurrent neural networks sequence modeling. arXiv preprint
William Clegg. Space groups- the final frontier: a tutorial guided tour some entries in international tables for crystallography volume Crystallography Reviews, 29(4):228-24 2023.
Hongyang Dong, Keith T Butler, Dorota Matras, Stephen WT Price, Yaroslav Odarchenko, Rahul Khatry, Andrew Thompson, Vesna Middelkoop, Simon DM Jacques, Andrew M Beale, et al. A deep convolutional neural network for real-time full profile analysis of big powder diffraction data. NPJ Computational Materials, 7(1)74,, 2021.
Siti Fatimah, Risti Ragadhita, Dwi Fitria Al Husaeni, and Asep Bayu Dani Nandiyanto. How to calculate crystallite size from x-ray diffraction (xrd) using scherrer method. ASEAN Journal of Science and Engineering, 2(1)x65-76, 2022.
Gabe Guo, Judah Goldfeder, Ling Lan, Aniv Ray, Albert Hanming Yang, Boyuan Chen, Simon JL Billinge, and Hod Lipson. Towards end-to-end structure determination from x-ray diffraction data using deep learning. npj Computational Materials, 10(1):209, 2024a.
Gabe Guo. Tristan Saidi, Maxwell Terban, Simon JL Billinge, and Hod Lipson. Diffusion models are promising for ab imitio structure solutions from nanocrystalline powder diffraction data, arXiv preprint arXiv:2406. 10796, 2024b.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
JH Hubbell, WH McMaster, Kerr Del Grande, and JH Mallett. International tables for x-ray crystallography, Vol. IV, pp. 45-70, 1974.

---

Published as a conference paper at ICLR 2025
International Centre for Diffraction Data. Mdi jade software. indi-jade/ Accessed: 2024-05-18 18.
https: www. icdd. com/
Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, al. Commentary: The ma- terials project: A materials genome approach to accelerating materials innovation. materials 1(1), 2013.
Barbara Lafuente, Robert T Downs, Hexiong Yang, Nate Stone, Thomas Armbruster, Rosa Micaela Danisi, et al. The power of databases: the rruff project. Highlights in mineralogical crystallogra- phy, 1:25, 2015.
Qingsi Lai, Lin Yao. Zhifeng Gao, Siyuan Liu, Hongshuai Wang, Shuqi Lu, Di He, Liwei Wang, Cheng Wang, and Guolin Ke. End-to-end crystal structure prediction from powder x-ray diffrac- tion. arXiv preprint arXiv:2401.03862. 2024.
Nam Q Le, Michael Pekala, Alexander New, Edwin B Gienger, Christine Chung, Timothy J Mon- talbano, Elizabeth A Pogue, Janna Domenico, and Christopher D Stiles. Deep learning models to identify common phases across material systems from x-ray diffraction. The Journal of Physical 2023.
Chemistry C, 127(
Armel Le Bail. Whole powder pattern decomposition methods and applications: A retrospection. Powder diffraction, 2064):316-326, 2005.
Armel Le Bail, Huguette Duroy, and Jean Louis Fourquet. Ab-initio structure determination of lisbwo6 by x-ray powder diffraction. Materials Research Bulletin, 23(3):447-452 1988.
Byung Do Lee, Jin- Woong Lee, Woon Bac Park, Joonsco Park, Min- Young Cho. Satendra Pal Singh, Min-Young Myoungho Pyo, and Kee-Sun Sohn. Powder x-ray diffraction pattern is all you need for machine- learning-based symmetry identification and property prediction. Advanced Intelligent Systems, (7):2200042, 2022.
Byung Do Lee, Jin- Woong Lee, Junuk Ahn, Seonghwan Kim, Woon Bae Park, and Kee-Sun Sohn. A deep learning approach to powder x-ray diffraction pattern analysis: Addressing generalizability and perturbation issues simultaneously. Advanced Intelligent Systems, (0))2230, 2023.
Jin-Woon Woong Lee, Woon Bae Park, Jin Hee Lee, Satendra Pal Singh, and Kee-Sun Sohn. A deep- learning technique for phase identification in multiphase inorganic compounds using synthetic xrd powder patterns. Nature communications, 11(1):86, 2020.
Tongxing Lei, Bin Cao, Wenbo Fu, Xiuling Shi, Zhiyu Ding, Qi Zhang, Junwei Wu, Kaikai Li, and Tong- Yi Zhang. A li-rich layered oxide cathode with remarkable capacity and prolonged cycle life. Chemical Engineering Journal, pp. 151522, 2024.
Qi Li, Rui Jiao, Liming Wu, Tiannian Zhu, Wenbing Huang, Shifeng Jin, Yang Liu, Hongming Weng, and Xiaolong Chen. Powder diffraction crystal structure determination using generative models. arXiv preprint arXiv:2409.04722, 2024.
Yang Liu, Jiashun Cheng, Haihong Zhao, Tingyang Xu, Peilin Zhao, Fugee Tsung, Jia Li, and Yu Rong. SEGNO: Generalizing equivariant graph neural networks with physical inductive bi- In ICLR, 2024a.
Yang Liu, Zinan Zheng, Yu Rong, and Jia Li. Equivariant graph learning for high-density crowd trajectories modeling. TMLR, 2024b. ISSN 2835-8856.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang. Lintao Ma, and Mingsheng Long. itransformer: Inverted transfonmers are effective for time series forecasting. CoRR, abs/2310.06625, 2023.
Kristina Lund, Norihiro Muroyama, and Osamu Terasaki. Accidental extinction in powder xrd intensity of porous crystals: Mesoporous carbon crystal cmk-5 and layered zeolite-nant nanosheets. Microporous and mesoporous materials, 128(1-3):11-77, 2010.
moniit-nanosheets..
12

---

Published as a conference paper at ICLR 2025
Luca Lutterotti, Henry Pilliere, Christophe Fontugne, Philippe Boullay, and Daniel Chateigner. Full- profile search-mater by the rietveld method. Journal of applied crystallography, 52(3):587-598, 2019.
Binfeng Lv, Zhenjie Feng, Xiaowei Sun, Jian Cao. Yu Gao, Shihui Chang, Cheng Dong, and Jincang Zhang. iqual: new computer program for qualitative phase analysis with efficient search-match capability. Journal of Applied Crystallography, 57(2), 2024.
Phillip M Maffettone, Lars Banko, Peng Cui, Yury Lysogorskiy, Mare A Little, Daniel Olds, Alfred Ludwig, and Andrew Cooper. Crystallography companion agent for high high-throughput materials discovery. Nature Computational Science, (4)):990-29) 2021.
K Maniammal, G Madhu, and V Biju. X-ray diffraction line profile analysis of nanostructured nickel oxide: Shape factor and convolution of crystallite size and microstrain contributions. Physica E: Low-dimensional Systems and Nanostruclures, 85:214-222, 2017.
Larry Medsker and Lakhmi Jain. Recurrent neural networks: design and applications. 1999.
RP Millane. R factors in x-ray fiber diffraction. ii. largest likely factors. Acta Crystallographica Section A: Foundations of Crystallography, 45(8):573-576, 1989.
MAR Miranda and JM Sasaki. The limit of application of the scherrer equation. Acta Crystallo- graphica Section A: Foundations and Advances, 74(1):54-65, 2018.
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In ICLR, 2023.
Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent I. Chevrier, Kristin A Persson, and Gerbrand Ceder. Python materials genomics (pymatgen): robust, open-source python library for materials analysis. Computational Materials Science, 68:3 (4-314-319, 2013.
Woon Bac Park, Jiyong Chung, Jacyoung Jung, Keemin Sohn, Satendra Pal Singh, Myoungho Pyo, Namsoo Shin, and K-S Sohn. Classification of crystal structure using a convolutional neural network. IUCrl, 4(4):486 4944 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
GS Pawley. Unit-cell refinement from powder diffraction scans. Journal of Applied Crystallography, 14(6))357-361, 1981.
Eric A Riesel, Tsach Mackey, Hamed Nilforoshan, Minkai Xu, Catherine K Badding, Alison B Altman, Jure Leskovee, and Danna E Freedman. Crystal structure determination from powder diffraction patterns with generative machine learning. Journal of the American Chemical Society, 146(44):33340-33348 2024.
HM Rietveld. Line profiles of neutron powder-diffraction peaks for structure refinement. Acta Crystallographica, 22(1):151-152, 1967.
Hugo M Rictveld. A profile refinement method for nuclear and magnetic structures. Journal of applied Crystallography, 2(2):65-71, 1969.
Juan Coariguez-Carvvaaal
introduction to the program fullprof 2000. Version July, 54, 2001.
Mario Enrique Rodriguez-Garcia, Martin Adelaido Mermandez-LLnddavedd Jose Miguel Delgado, Cristian Felipe Ramirez-Gutierrez, Marius Ramirez-Cardona, Beatriz M Millan-Malo, and San- dra Milena Londoño-Resteo Restrepo. Crystalline structures of the main components of starch, Current Opinion in Food Science, 37:107-111, 2021.
RRUFF. Rruff, 2005.
URL ht https: http::///saa.cooff..... confex. com httpf/eww..oo//aa///aeaa http://gg/////////////////// wwwpoggraa.......mmmmmmmmm. Accessed: 2024-05-21.
Paper92745. html
13

---

Published as a conference paper at ICLR 2025
Jerardo E Salgado, Samuel Lerman, Zhaotong Du, Chenliang Xu, and Niaz Abdolrahim. Auto- mated classification of big x-ray diffraction data using deep learning models. upj Computational Materials, 91):2114, 2023.
Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(1 1):2633-2681, 1997.
Tianhao Su, Bin Cao, Shunbo Hu, Musen Li, and Tong- Yi Zhang. Cgwgan: crystal generative framework based on wyckoff generative adversarial network. Journal of Materials Informatics, (4):N-A, 2024.
Yuta Suzuki, Hideitsu Hino, Takafumi Hawai, Kotaro Saito, Masato Kotsugi, and Kanta Ono. Sym- metry prediction and knowledge discovery from x-ray diffraction patterns using an interpretable machine learning approach. Scientific reports, 10(1):21790, 2020.
EC Svensson, VF Sears, ADB Woods, and Martel. Neutron-diffraction study of the static structure factor and pair correlations in liquid he 4. Physical Review B, 21(8):3638, 1980.
Nathan J Szymanski, Christopher J Bartel, Yan Zeng. Qingsong Tu, and Gerbrand Ceder. Proba- bilistic deep learning approach to automate the interpretation of multi-phase diffraction spectra. Chemistry of Materials, 33(1 (((1))4204-2255 2021.
Brian H Toby. R factors in rietveld analysis: How good is good enough? Powder diffraction, 21(1): 67-70, 2006.
Brian H Toby and Robert Von Dreele. Gsas-ii: the genesis of a modern open-source all purpose crystallography software package. Journal of Applied Crystallography. 46(2):544 549, 2013.
Atsushi Togo, Kohei Shinohara, and Isao Tanaka. Spglib: a software library for crystal symmetry search, 2024.
David Tumbull.. Phase changes. In Solid state physics, volume 3, pp. 225-306. 1956.
University College London. X-ray powder diffraction quality assurance tables, date accessed. URL http://pd. chem. ucl ac uk pxrd/ qa/ smtable. htm.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9111), 2008
Van Laar and WB Yelon. The peak in neutron powder diffraction. Journal of Applied crystallog raphy, 17(2):47-54) 1984.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. In Neurips, pp. 5998-6000 2017.
Hong Wang, Yunchao Xie, Dawei Li, Heng Deng, Yunxin Zhao, Ming Xin, and Jian Lin. Rapid identification of x-ray diffraction patterns based on very limited data by interpretable convolu- tional neural networks. Journal of chemical information and modeling, 60(4):2004-20) 2011,, 2020.
Dejan Zagorac, H Müller, S Ruehl, Zagorac, and Silke Rehme. Recent developments in the inor- ganic crystal structure database: thcoretical crystal structure data and related features. Journal of applied crystallography, 52(5):918
2019.
Zinan Zheng, Yang Liu, Jia Li, Jianhua Yao, and Yu Rong. Relaxing continuous constraints of equivariant graph neural networks for broad physical dynamics learning. In KDD, pp. 4548- 4558, 2024.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL RESULTS
A.1 LONG-TAILED DISTRIBUTION AND FORMATION ENERGY
The distributions of space groups and crystal systems are shown in Figure 4 and B. The forma- tion energy (Ey) of configuration represents the energy required or released during its formation. Thus, the formation process can be either endothermic or exothermic. A more negative formation energy indicates greater structural and thermal stability [Trnbbll] (1956). Figures C and illus- trate the formation energy per atom across 154,718 crystal structures in MP-2024.1 and 119.569 crystal structures in SimXRD. Figure E highlights the high-quality crystal data MP-2024.1, with 90.33% of structures exhibiting negative formation energy. In comparison, the crystal data in SimXRD shows negative formation energy rate of 92. 18%, indicating that the excluded crystals are primarily unstable. This high-quality structural data forms the foundation of XRD pattern databases.
30.0
Hosagg
4:0 OOmmmmmnnii
14
2m
1000
340
397
CA09
230
302
eryats spsterss
27
30.33%
1750
1259
1142
130 pas
240 Bano
509
1477
63 Desr
277
337
11AC
12.18%
10
Tumuldie Predaiig
Preeesecy
1120
314
Fraquency
Fraquency
Fraquency
Frequency
Fraquancy
enargy (allaton))
Figure 4: The statics of dataset. (A) The distribution of structures contained in the SimXRD database across various space groups (uniformly divided into 5 categories according the frequency) and (B) crystal systems (C) The distribution of -formation energy/atom of crystals contained in MP-2024.1 and (D) SimXRD. Space groups 89, 103, and 104 have been excluded from Figure A enhance clarity, each appearing only once.
A.2 PERFORMANCE W.R.T. LONG-TAILED DISTRIBUTION
To further investigate the model's performance, we display the accuracy of each crystal system and space group at different frequencies in Figure Junder visualize the model performance space groups, we divided the results of 280 space groups into categories according to their frequency (the same as Figure From the results, we observe the following:
under in-library symmetry identification.
better
In crystal system classification, the performance of most CNN models, the raw RNN, and trans- formers is affected by the frequency of crystal systems. They cannot predict the XRD patterns of Cubic and Hexagonal symmetry (their accuracies almost zero). In contrast, multiple domain- specific CNN (i.c., CNN 7, 9, 10. 11), LSTM, GRU, and advanced Transformers (i.c., iTrans- former and PatchTST) can mitigate long-tail distributions.
15

---

Published as a conference paper at ICLR 2025
Temragonnal
Hesageooal
0.8
0.5
04
DWN
CNN
ONNY
CleNs
OWNT
OHNTI
LOTM
Biaard
Tranffomrr
Transform
Palanaar
1-575
647-122
(04084)))
[N-10945]
(
(1-260)
(1
pert
2015
oec
ANS
cen
Figure 5: Classification accuracy of each model w.r.t. crystal systems and space groups. The same as Figure AA), Space groups are classified into categories based on their frequency. N denotes the number of crystals.
Space group classification more challenging than crystal system classification. Most models can only predict space groups with high frequency and struggle predict most low-frequency space groups. Surprisingly, PatchTST achieves comparable performance with the best models in all low-frequency space groups Given that PatchTST is model specifically tailored for time se- ries forecasting, its performance on XRD pattern classification still has considerable room for improvement.
A.3 PARAMETERS INTERVALS STUDY
To study how simulation parameters influence model performance on SimXRD, we perform an additional comparison by dividing the testing instances into three distinct intervals based on their key simulation parameters:
Category 1: Grain size (nm) E [2, 10]. atomic thermal offset (A) E [0.4, 0.5), orientation random- ness (%) E [30, 40), zero shifting (?) 1.2].
Category 2: Grain size (nm) [40, 50], atomic thermal offset (A) [0, 0.1], orientation random- ness (%) E [0, 10), zero shifting (*) E [0, 0.2].
Category 3: The parameter domain excludes Category and Category 2.
The three parameter categories represent three simulated experimental scenarios: Category reflects high degree of peak broadening and external influence, while Category 2 is aligned with relative ideal conditions, exhibiting lower broadening and noise. Category represents an intermediate state between the two. The accuracy of space group classification is displayed in Table 6 The results indicate that the baselines generally perform better in Category 2 and exhibit the lowest accuracy in Category 1, underscoring the challenges of modeling experimental XRD patterns in more complex physical situations. These insights highlight the need for developing a high-fidelity database that encompasses diverse crystals and covers sufficient range of practical scenarios.
16

---

Published as a conference paper at ICLR 2025
Table 6: Parameter Categories comparison of crystal systems & space group
Task
Crystal System Classification
Space Group Classification
Metrie
Categoryl
Category2
Category3
Categoryl
Category2
Category3
CNNI
0.523
0.553
0.554
0.250
0.233
0.241
CNN2
0.452
0.467
0.464
0.250
0.233
0.241
0,318
0.384
0.307
0.250
0.233
0.241
CNN4
0.375
0,305
0.302
0,250
0.233
0.241
CNN5
0.500
0.545
0.526
0.312
0.286
0.295
CNN6
0.375
0.305
0.302
0.250
0.233
0.241
CNN7
0.781
0.894
0,862
0.437
0.566
0,575
0.375
0.305
0.250
0,233
CNN9
0.781
0.823
0.785
0,343
0.646
0.609
CNNI0
0.812
0.908
0.868
0.406
0.727
0.704
CNN11
0.656
0.908
0.895
0.562
0.823
0.775
0.312
0.327
0.250
0.233
0.241
RNN
0.468
0.383
0.367
0.218
0.236
0.245
LSTM
0.593
0.761
0.727
0.281
0.577
0.515
GRU
0.531
0.793
0.765
0.437
0.596
0.575
Bidirectional-NN RNN
0.343
0.362
0.365
0.250
0.236
0.245
Bidirectional-L... LSTM
0.781
0.837
0.790
0.437
0.600
0.559
Bidirectional GRU
0.656
0,818
0.800
0.437
0.596
0.575
Transformer
0.375
0.364
0,250
0.233
0.240
iTransformer
0.500
0.662
0.627
0.312
0.378
0.388
PatchTST
0.562
0.745
0.720
0.437
0.672
0.631
A.4 FEATURE ANALYSIS
We also analyze the relative feature importance map to provide model interpretability, shown in Figure 6 By masking features in powder XRD patterns, we observe the impact of the masked features on the model's inference by calculating the relative accuracy drop. This approach reflects the relative importance the masked features for model classification. As expected, masking critical features results in significant drop in accuracy.
To further investigate, we categorize peak features into five groups based on their intensity: (1) Peaks with intensities within the 0- 0-20% maximum intensity range, (2) Peaks within the 20 intensity range, (3) Peaks within the 40-60% intensity range, (4) Peaks within the 60-80% intensity range, and (5) Peaks within the 80-100% intensity range. We evaluate the model's inference performance with these masked features, as shown in Figure The results demonstrate a clear trend: when high- intensity peaks, particularly those category are masked, significant accuracy drop is observed across all baselines. This indicates that the MIL model heavily relies on high-intensity peaks, similar the "characteristic peaks" identified in search- match approaches.
Interestingly, for models that perform well on relative tasks, masking peaks in categories 3 or 4 also noticeably affects performance. This suggests that these models capture more comprehensive pattern characteristics by giving attention to relatively weak intensity peaks. Such behavior is reasonable for deep neural networks, as they extract more detailed information rather than relying solely on the 'three strongest diffraction peaks'. This leads to better performance.
Comparing the results of CNNI1, BiGRU, and PatchTST, three models that performed relatively well in these symmetry identification tasks, observe distinct strategies for decision-making. CNN11 places greater emphasis on characteristic peaks in category 5, indicating that its inference approach aligns closely with the traditional search-mater match method. PatchTST follows similar trend, but the relative accuracy drop is more pronounced, suggesting that PatchTST captures a broader range of peak characteristics compared to CNN11. In contrast, BiGRU exhibits gradient in the accuracy drop, indicating that it selectively focuses on relatively strong peaks while progressively reducing emphasis on weaker peaks based on their intensity. This strategy is intuitively reasonable, as weaker peaks are more susceptible to noise. These observations suggest that bidirectional ar- chitectures, such as BiGRU, may provide promising solution for studying powder XRD data, as concluded in the main text.
17

---

Published as a conference paper at ICLR 2025
CNNI
0.000
0.000
0.000
0.000
Relative
Accuracy
-0,001
-0.121
-0.079
-0.143
3
5
1
2
3
5
2
5
CNNS
CNN6
CNN7
CNNN
0.000
0.000
0,000
0.000
Relative
Accuracy
-0,001
-0.124
-0.43
-0,036
3
5
2
2
5
CNN9
CNN10
CNNII
MLP
0.000
0.000
0.000
0.000
Relative
Accuracy
-0.001
-0.331
-0.398
-0.414
3
RNN
LSTM
GRU
BIRNN
0.000
0.000
0.000
0.000
Relative
Accuracy
-0,048
-0.318
-0.385
-0.044
BILSTM
BIGRU
Transformer
iTransformer
Relative
Accuracy
-0.421
-0.430
0.002
PatchTST
Relative
Accuracy
-0.245
Figure 6: The relative classification accuracy drops when peak features are masked across all base- lines. These features are categorized into five groups based on their intensity. Peaks with inten- sities within the 0-20% maximum intensity range, 2: Peaks within the 20-40% intensity range, 3: Peaks within the 40-60% intensity range, 4: Peaks within the 60-80% intensity range, and 5: Peaks within the 80-100% intensity range.
18

---

Published as a conference paper at ICLR 2025
B ADDITIONAL BACKGROUND
B.1 MORE DETAILS ABOUT CRYSTAL SYMMETRY
The seven crystal systems are cubic(#1), hexagonal(#2), tetragonal(#3), orthorhombic(#4), trigo- na5,, m06), and triclinic(#7). Crystals can be diagrammatically represented by an or- derly stacking of lattice cells, whose shape determines the crystal system which they belong. Unit cells of identical shape can have lattice points, representing centers or faces, in addition to the corners. These additional lattice points further divide the seven crystal systems into 14 Bravais lattices, which are then subdivided into 32 crystal classes, point groups. Each point group corresponds to a possible combination of rotations, reflections, inversions, and improper rotations. When translational elements are included, these point groups yield the 230 space groups (Clegg 2023).
atom or group of atoms, at their
B.2 IN-LIBRARY AND OUT-OF-LIBRARY TASKS
In-library identification is fundamental task in crystallography that aims to accurately identify crys- tal types based on XRD patterns measured in various environments. Since 1938, crystallographers have been documenting all discovered structures and archiving them as Powder Diffraction Files determine the structures of stud-
(PDFs). By comparing and retrieving these PDFs, researchers ied materials by matching them with historical data. With advancements in computing, numerous software programs have been developed to assist in the search-mater process.
A related concept is out-of-library identification, which involves discovering previously unknown structures that lack recorded data. This process depends on symmetry identification to determine the basic space group information, followed by subsequent refinement. Traditionally, the geometric characteristics of crystal system are determined based on physical properties, such as electrical conductivity, optical behavior, and thermal properties. The space group is then identified by ex- amining extinction effects. Next, the ideal chemical formula is determined based on the estimated number of atoms and Wyckoff positions. Validation is performed using techniques such Rietveld refinement and site optimization.
Therefore, provided two types of splits:
In-library Classification: In our in-library benchmark, the dataset is divided based on different simulation environments. Both the training and testing datasets contain the same structures. but under varying simulation conditions. This setup corresponds to in-library identification in XRD phase analysis.
Out-of-library Classification: In out-of-library benchmark, the dataset is split according to crystal types. The training and testing datasets consist different structures, mirroring the out- of-library identification process in XRD phase analysis.
B.3 SEARCH-MATCH APPROACH
Conventionally, the primary method for in-library phase identification in X-ray powder diffraction patterns is the search-match approach, which consists of three main steps. Initially, (d, I) values, rep- resenting interplanar spacing (d) and intensity (I) each peak, extracted from the pattern. Subse- quently, potential phases are sought from diffraction databases based on characteristic values, such as Hanawalt indexes (University College London date accessed). Following this, candidate phases are compared to the (d, 1) values of the pattern using scoring system to assess alignment, aiding in selecting the most suitable candidate phase. This process iterates until satisfactory alignment achieved for most (d, I) values. A recent study (Lutterotti et al., 2019) employs an approach utilizing the Rietveld method for conducting a Full Profile Search-Mateh (FPSM). Crystal structures selected from databases are automatically aligned with raw data via the Rietveld method, culminating in score determined by Rwp (Millane 1989). The phase with the highest score is acknowledged, and this iterative procedure persists until specified criteria met. Subsequently. Rietveld quantifica- tion executed to ascertain the weight fraction of each phase. Despite advancements in computer technology that have enabled new qualitative methods, the core workflow of the search-match ap-
19

---

Published as a conference paper at ICLR 2025
proach remains unchanged: matching experimental data with database entries, which contain partial information abstracted from raw diffraction patterns, and calculating a corresponding score.
B.4 XRD PATTERN SIMULATION
Powder XRD provides a one-dimensional representation of three-dimensional diffraction patterns and is the most common experimental measurement, shown in Figure g These patterns reflect the relative arrangement of atoms in paree-dimensionll space. The interaction of X-rays with condensed materials during diffraction involves elastic collisions between photons and electrons. Consequently, factors that affect atomic arrangement and electron behavior influence the shape of the diffraction pattern, emphasizing the need for high-fieelity simulation technologies. To address this gap, we developed Pysimxrd.
The diffraction vector G defined as
2sin
G
À
where À is the wavelength of the scattered particle and 0 28/2, with 20 being the angle between the incident and diffracted beams. In three dimensions, scattering occurs only at a discrete set of reciprocal vectors, K, forming the reciprocal lattice,
K = hat kbb +lc",
where a", b", and c* the reciprocal lattice vectors, and h, k, and constants. The diffraction condition is defined as
2x G K.
(3)
The x-axis of simulation patterns d is the reciprocal of |G'), i.e., d = 1/|G"|
The diffraction intensity each diffraction vector G* for single phase determined by
I(G = SF FøLPOD + BBC
(4)
where S denotes the scale factor, F' is the structure factor, F* is the complex conjugate of F, ø the profile function, L is the Lorentz-polarization factor, P is the multiplicity, O is the preferred orientation factor, D is the Debye- Waller factor, and fBG is the background intensity. The structure factor is computed as:
N F
= Σ fee
R
(5)
j=1
where fj is the form factor Hubbell et al. (1774) in XRD, R. is the lattice coordinate of atom j. and N is the total number of atoms in a lattice cell.
The Lorentz-pollarization calculated by,
L + cos 20
(6)
sin" O cos
where 0 = aresin
AA'' 2
The multiplicity P is determined by counting the number of diffraction
vectors present within an Ewald diffraction sphere. The Debye-Waller factor D calculated by
D
2M
6 mkga (o ( + ( (sin" 8) />, Planck's constant, m is atom mass, is Boltz- mann constant, θ is the average characteristic temperature, T is absolute temperature, and d(0/T) is the Debye function.
where M
The profile function (Ø) modeled by convolving various factors, including diffraction, detector geometry, and noise factors etc. The simulated peaks are calculated by:
v()) = W G
(8)
20

---

Published as a conference paper at ICLR 2025
Here, denotes the convolution process, and W, G, and S represent the contributions to the observed XRD pattern from diffraction emission, instrumental factors, and the noise mixture, respectively. S modeled Gaussian peak.
The W a Voigt function Armstrong 1967..
(20 t)² 22 exp (_(20
(20 - 1)2
W
dt.
(9)
2m
2x²
Peak broadening is correlated with the Full Width Half Maximum (FWHM. D) where 2 2/ 2o I Caglioti et al. (1958). I' is calculated by Scherrer's equation Miranda Sasaki (2018) related to finite grain size.
The geometrical factor Van Laar & Yelon (9844) G, accounts for the actual dimensions of the de- tectors and the powder sample. It is defined as follows:
81(20) 8(2a)
D(2a, 20)
d(2a).
(10)
L
G(2o,; 20) 4H cos(2))
dz,
(11)
where 2a represents the Bragg angle. The detector is slit-shaped, with a height of 2H, and the sample has a height of 2S. L denotes the distance between the sample and the detector.
The simulation environments in our study encompass three key aspects. The first aspect pertains to the sample, where we simulate the average grain size, the ideal orientation effect of the powder, and the lattice cell's s extinction and torsional deformation under internal stress. The second aspect involves the testing conditions, accounting for factors such as atomic thermal vibrations at room temperature, air ceattering-inducee signal noise, and instrumental -ndduee noise. The third aspect relates to the diffractometer, including the zero shift of the angular position and detector's geometry.
The adjustable parameters of the simulation environments include:
Finite grain size (for Voigt peaks): the average grain size is set as a random number in the range of 2 nm to 50 nm replicate the degree of experimental broadening (Maniammal et al. 2017 7). Instrumental broadening becomes apparent the experimental patterns as convolution with the crystal peak occurs. Typically, in the refinement process, it is common separate observed peaks into Voigt and instrumental components. However, quantifying the instrumental compo- nent proves challenging
varies with the diffraction angle. Therefore, in our simulations, we opt for smaller grain size than realistic to accurately replicate the degree of experimental broad- ening. Pysimxrd can also model the asymmetric peak by convolving the instrumental geometry factors in equation [1] If a convolution peak is applied, the grain size range must be adjusted accordingly.
Orientation: the ideal powder sample is assumed to have no orientation, however, this impossi- ble. The orientation effect arises from the uneven distribution of small grains in the incident beam, leading to an uneven distribution of reciprocal sites across the reciprocal spheres the powder sample. To simulate the impact of orientation, set the intensity by perturbation within 40% range based on the ideal simulation to mimic the orientation randomness.
Thermal vibration: the temperature is converted to the kinetic energy of atoms, which is simulated by allowing atoms to shift from their average positions within a range of 0.01-0.5 angstrom.
Internal stress: internal stress is simulated by applying elastic deformation of the recorded lattice constant by up to 10%.
Instrument zero shifting: the zero shift is simulated by randomly translating 20 within -1.2 to 1.2 degrees.
Instrument noise: 2% Gaussian white noise is added to the entire pattern to reflect instrument noise.
21

---

Published as a conference paper at ICLR 2025
Inelastic Scattering: A sixth-order polynomial function is employed to simulate the background distribution, encompassing phenomena such as Compton scattering, fluorescence, and multiple scattering. This simulated background is then integrated into the XRD pattern with 2% ratio.
100
Parameter set
Porameter set
A
80
80
Intensity (a.u.
Intensity (a.u.)
40
20
(A)
(A)
100
Parameter
Experiment
C
D
80
Intensity (a u.u.)
Intensity (a.u
60
40
20
d (A)
(A)
Figure 7: Comparison of simulation pattern with experimental pattern of PbSO, (A) The generated PbSO4 powder XRD without considering the simulation environment. (B) The generated PbSO, powder XRD based on set of parameters that account for finite grain size, orientation, thermal vibrations, and internal stress. (C) The generated PbSO, powder XRD with an extended set of pa- rameters that further include zero-shift correction, noise, peak convolution, and background effects. (D) The experimentally observed data. Note: The PbSO, structure corresponds entry 1010950.cin from the Crystallography Open Database. The parameter sets used in the simulations were opti- mized by minimizing the differences between the simulated and experimental patterns, rather than being randomly assigned. All patterns were converted to Q-space using wavelength of 1.54 Á.
As illustrated in Figure 7 the multiphysical coupling observed in experimental XRD can be incor- porated into the simulation parameter space by introducing more realistic conditions. Specifically, considering factors such orientation, thermal vibrations, zero-shift corrections, noise, and other sources of randomness significantly improves the realism the simulations. These parameters col- lectively influence the arrangement of peaks, their intensities, and overall shapes in the diffraction pattern.
B.5 CRYSTAL AND DIFFRACTION
Among various investigative methods, diffraction analysis stands out exceptionally potent for probing microstructure, primarily due to its sensitivity to atomic arrangement and the element speci- ficity of atom scattering power. Each intensity maximum, termed line profile peak, within diffraction pattern reflects the atomic arrangement along certain direction of the diffracting material.
Powder X-ray diffraction,, in particular, furnishes remarkably diverse array of structural details, encoded in the material- and instrument-specine distribution of coherently scattered monochromatic wave intensity, with wavelengths corresponding to lattice spacing. The X-rays generated by cathode ray tube, filtered to produce monochromatic radiation, collimated for concentration, and directed towards the sample. When conditions satisfy Bragg's Law, which relates the wavelength
22

---

Published as a conference paper at ICLR 2025
of electromagnetic radiation to the diffraction angle and lattice spacing in crystalline sample, the incident rays interact with the sample to produce constructive interference and a diffracted ray, as shown in Figure g
The diffracted X-rays are subsequently detected, processed, and quantified. By scanning the sample across range of diffraction angles, all possible diffraction directions of the lattice accessible due to the random orientation of powder sample.
B.6 SYMMETRY RECOGNITION IN STRUCTURE DETERMINATION
Pattern refinement (Rietveld] 1967 1969) is essential for accurately determining the crystal structure in diffraction studies. In XRD refinement, the initial step involves identifying the crystal phase of the given XRD pattern. This is crucial because it facilitates the calculation of the static structure factor (Svensson et 1980), which aids determining atomic sites during refinement.
Using existing crystal databases, the commonly employed search- match approach retrieves potential structures that match the ob- served pattern, effectively performing in-library identification. The matched structure, along with basic crystallographic information such crystal symmetry, serves foundational element for fur- ther analysis. In out-of-library situations, the geometry the crys- tal lattice must be established from scratch, there is no recorded structure history. By solving the diffraction indices corresponding set of lattice planes for each diffraction peak, a process known as indexing calculation (Rodriguez-Garrin et al. 2021), the crystal symmetry be determined.
cathode
filter collimtor
Somple
fiiter
detector
Whether the observed patterns are from in-library or out-of-library crystals, symmetry identification combined with indexing is a uni- versal method for determining lattice geometry to initiate the re- finement process. This is why many researchers seek general method for deriving the symmetry of structures based on diffrac- tion patterns. Once the fundamental crystallographic information is obtained, techniques such as pattern decomposition and Rietveld refinement (Rietveld] [967; 1969. Le Bail, 2005) are employed. These methodologies help determine atomic sites, temperature factors, grain sizes, residual stress, and other intricate structural details from the diffraction patterns, thereby contributing to the com- prehensive understanding of crystal structure.
experiment
crystal
Figure 8: Experimental pro- cedure for recording XRD patterns.
B.7 WHOLE POWDER PATTERN FITTING AND REFINEMENT
Structure refinement is effective method for determining crystal structures. Traditionally, refine- ment begins with a set of reasonably estimated parameters with physical significance. Accurately identifying the structure's space group aids in establishing rational parameters. However, without prior structural knowledge, this process becomne trial and eиror. Although identifying the space group initiates the process, its validation relies on refinement outcomes.
A significant advancement in powder diffraction fitting for structure refinement emerged with the Rietveld method 1967 1969), which pioneered whole pattern fitting over the analysis of individual, non-overlapping Bragg diffraction peaks. The Whole Powder Pattern Fitting (WPPF) approach within the Rietveld method uses profile intensity calculations and squares algorithm for structure refinement. By minimizing the disparities between observed experimental profiles and theoretical profiles, extracts structural information. Once fitting reaches its limit, the parameters in the thcoretical model represent the only determinable physical information. SimXRD's 's pattern simulation also relies on the structural information contained within the theoretical model. The Pawley(Pawle) 1981) and Le Bail Bail et al. 1988) methods, two widely used WPPF techniques, are developed after the Rietveld method.
These parameters with physical significance diverse, and important type is peak shape pa- rameters. The parameters of each diffraction peak (positions, intensities, and shapes) within a whole XRD pattern offer detailed structural insights from different aspects, making their quantitative and
23

---

Published as a conference paper at ICLR 2025
accurate extraction from the overall XRD profile highly significant. This quantification of peak information and the subsequent derivation of physical parameters constitute the essence the re- finement endeavor.
B.8 SIM2REAL GAP
To illustrate the advantages of Pysimxrd in generating high-fidelity powder XRD patterns, we com- pared its performance with a widely recognized software, GSAS-II (Toby & Von Dreele 2013). The simulation process involves the use of refinement software, starting with the determination of suitable parameter sets through the refinement procedure provided by each software. The lattice constants derived from these refinements are summarized in Table Once the parameters are con- firmed, the corresponding simulation patterns are generated.
Table 7: Derived crystal structures in refinement.
Lattice Constants (Á)
Models
Crystals
a
Pysimxrd
Mn,O;
9.4082(2)
9,4082(2)
9,4082(2)
90
90
90
Pysimxrd
RuO,
475
4.5381(7)
3.1248(4)
90
90
90
GSAS-II
Mn,O;
9.4197(1)
9.4197(1)
9.4197(1)
90
90
90
GSAS-II
RuOz
4.5351(8)
4.5351(8)
3.1226(0)
90
90
90
Two experimental powder XRD patterns obtained using X'Pert Pro MPD (Panalytical) in
geometry with copper X-ray source (Ax-a = 1.5406 Å). The samples used were Mn₂O₃ and RuO, crystals. As shown in Figure Pysimxrd and GSAS-II demonstrated comparable accuracy in reconstructing the MnzO; pattern. However, Pysimxrd showed significant advantages in reconstructing the RuO₃ pattern. This superiority stems from Pysimxrd's ability account for a broader range of simulation environments and its greater flexibility in tuning parameters, which resulted more accurate reconstruction with lower R-factors.
G245-11
60300
60000
A, 2.40%
40320
Ruy 3.15%
42000
3.3
Interdityyy
20320
20003
11111
20
50
60
10
10
30
60
10
50
(degrees)
(degrees)
Fycmant Expar mant
cesaniment
M0300
Recidval
E0005
Residua!
Rg 020
40320
40003
Rog 7054
Inteosityy
-20020
40 50
50
69
70
50
28 (diggrass)
29 (degreos)
Comparison of simulated and experimental patterns: (A) The MnşO, pattern simulated using Pysimxrd. (B) The Mn;O, pattern simulated using GSAS-II (C) The RuO, pattern simulated using Pysimxrd. (D) The RuO, pattern simulated using GSAS-II. The R-factors, calculated as (Toby) 2006), are used to evaluate the quality of the fitting.
Figure 9:
24

---

Published as a conference paper at ICLR 2025
B.9 THE T-SNE PLOTS OF POWDER XRD PATTERNS
We also computed t-SNE plots (Van der Maaten & Hinton 2008 based on XRD patterns, along with their corresponding crystal systems and space groups, shown in Figure TO The results indi- cate that the projection plots for both SimXRD and RRUFF exhibit bilateral symmetry and distinct manifolds. However, due the limited data in RRUFF, the two datasets show some discrepancies, reflected in the differences in crystal system and space group distributions across the XRD patterns.
A
across Crystal Systems
across Space Groups
1-NEE
1-SNE
5-SNE
B
XRD across Crystal Systems
XRD across Space Groups
1-ONE
SSNE
Figure 10: (A)The t-SNE plot for XRD patterns in the SimXRD validation dataset. (B) The t-SNE plot for XRD patterns in the RRUFF dataset. Numbers 1-7 represent the seven crystal systems (Appendix B.I. while 1-230 correspond to the space groups
25


Reviews:
[{"review_id": "2KbQORKvcZ", "reviewer": "Reviewer_Z7kM", "summary": "The paper introduces a benchmark dataset \"SimXRD\" which comprises 4,065,346 simulated powder XRD patterns that represent 119,569 unique crystal structures. This dataset aims to advance crystallographic informatics by providing high-quality training data for machine learning models. The authors evaluated 21 sequence models on their XRD dataset, testing both in-library and out-of-library classification scenarios. Their analysis revealed that current models struggle with rare crystal types and showed how different physical conditions affect model performance. The results demonstrate that models trained on simulated data can work with real XRD patterns.", "questions": "1. Model Interpretability: Could you provide analysis of what patterns your best performing models learn from XRD data, and what features differentiate successful classifications from failures, especially for low-frequency crystals?\n\n2. While not very important, it would be helpful to understand systematic differences between simulated and real XRD patterns beyond the Li2MnO3 case study, and identify which simulation parameters most affect model performance on real data.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is essentially a benchmark paper that introduces a dataset that can be a useful resource for the community.\n\n1. Creates the largest open-source XRD dataset (4M+ patterns) with validation against experimental data, enabling advancement in crystallographic machine learning research\n2. Provides benchmarking across 21 models (CNNs, RNNs, Transformers) with detailed performance metrics \n3. Tests both in-library and out-of-library scenarios and also evaluates against real experimental data (RRUFF dataset)\n4. Identifies key challenges like long-tail distribution in crystal structures, providing directions for future machine learning model development", "weaknesses": "1. The paper benchmarks existing approaches without introducing new architectures to address the identified challenges, particularly for low-frequency crystal classification. This limits its technical novelty from an architectural perspective.\n\n2. While generating experimental data is challenging, the validation against real-world data is limited. The RRUFF dataset (~3K patterns) represents only 0.07% of the simulated dataset size (4M+ patterns), and the comparison between simulated and real-world patterns is demonstrated only for Li2MnO3. A more comprehensive analysis of sim and real world gaps would be helpful.\n\n3. The paper lacks investigation of model failure modes and interpretability analysis. For example, analyzing what patterns CNNs learn and how they make decisions could reveal important insights and potentially guide future architectural improvements. Such analysis would provide valuable direction for developing more effective models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gfbBmfZ97u", "reviewer": "Reviewer_Qfc3", "summary": "Introduces a new dataset of simulated X-ray diffraction dataset. It is generated with a novel technique based on known physical interactions. This dataset is the largest symmetric dataset to date. They train existing model architectures on their dataset, and show that some models have superior performance using their dataset. However, CNNs (one of the most commonly used architectures for this task) do not perform well. They also struggle to handle low-frequency class labels, even with re-weighting of sample importance.", "questions": "1 - In line 36: Transitional methods for what? I assume symmetry identification, though extra clarity would be appreciated.\n\n2 - Line 156 claims that existing symmetry methods rely on 1D CNNs? As it is currently written, it is saying that every technique that has been introduced uses 1D CNNs, which contradicts lines 296-308. \n\n3 - Lines 296-308: is this supposed to be a comprehensive list of techniques? As it is written, that is what it seems to be saying.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The novel technique for generating samples of data is quite interesting; gathering experimental x-ray diffraction data on scale for large machine learning projects is difficult. This gives a strong motivation and need for this paper in the research community. The explanation for the techniques (in appendix B4) is very strong and extremely easy to read. Overall, the work and explanation provided in the appendix is one of the strongest parts of the paper, and is excellent in both content and writing quality. \n\nThe testing and the results appear to have had a significant amount of effort put into them and cover a variety of models. This provides strong support for the claims made and the generalizability of the results.", "weaknesses": "Overall: I think that this submission is on a very good track. As someone who has worked with x-ray diffraction data before, I understand the need for this work. However, the limitations of the dataset are too severe for practical use, and so I do not think that it is ready for use. \n\nThe two main weaknesses are: \n\nA - There is no validation of SimXRD to show that the patterns it creates match reality. The results from the machine learning models trained on the data show that to ML algorithms it seems close enough, but analytical results would be nice. The only justification provided is the technique used to create the data and the results from training with SimXRD then testing on other datasets.\n\nB - The poor performance of the models on the long-tailed distribution. These results undercut the quality of the distribution; they show that the dataset does not allow for training of models that perform accurately. \n\nC - I do not understand Figure 2. There are just lots of images, with very little explanation. Since it is referenced as a primary part of the paper and does not have extensive description in the text, I do not understand what it is describing.\n\nD - Overall writing clarity (outside of the appendix) could use some improvement. Explanation seems like it would be sufficient for someone already familiar with the work, but not somebody learning about this for the first time. Additionally, there are some grammatical mistakes which cause confusion (missing three commas in the sentence on line 69-70 for example).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gBW50bEIb5", "reviewer": "Reviewer_xPRn", "summary": "This paper presents SimXRD, a simulated X-ray diffraction (XRD) pattern dataset for benchmarking crystalline symmetry classification tasks. The patterns are simulated using PySimXRD, a custom software based on diffraction physics, for (filtered) crystal structures in the Materials Project database. Using the dataset, analyses of crystal symmetry data and comparisons of ML models are performed.", "questions": "- The simulated dataset uses lattice plane distance as the x-axis. How is it related to 2$\\theta$ which is more commonly used in XRD data, and what’s the reason for this choice?\n- The generalizability of models to experimental data is assessed by training on SimXRD and testing on RRUFF. Is RRUFF representative of experimental data influenced by various factors (materials type, physical condition, equipment, etc.)? How do the authors ensure this is a rigorous assessment?\n- Line 157, “almost all related research…due to the lack of easily accessible datasets”, what does this mean?\n- Line 345, is “being biased by classes of high frequency” a problem of CNN? It should be more related to the objective/loss function.\n- Regarding presentation clarity:\n  - Line 48, structure analysis being challenging for domain experts is irrelevant to the search-match method.\n  - Line 59, what challenges do introducing large-scale databases present?\n  - Line 79, repeated words.\n  - Line 216, the details on the Materials Project are irrelevant to this work.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The presented large-scale simulated XRD dataset is open-sourced, which could benefit the community.", "weaknesses": "- The main contribution of this work is unclear. The analyses and benchmarks do not provide much insight into ML research. If the PySimXRD software contains new/better functions than previous XRD simulators, they should be specified; even so, the contributions are in physics or materials science, not ML. Rather than ICLR, venues like *Scientific Data* seem more suitable for this paper.\n- The presentation logic and clarity of this paper are poor. Examples are listed in Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Imr30ESMxS", "reviewer": "Reviewer_UoDH", "summary": "Overall, the work presents a valuable dataset and simulation environment for PXRD patterns, which is a great topic of interest in the materials science community.", "questions": "- Are you able to generate PXRD patterns of varying Q-ranges and/or x-ray wavelengths?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Accurate simulation of PXRD patterns is a timely topic of great recent research interest, and actually something I have grappled with in my own work, so I was pleased to read of your work that systematically does this in an easy software package.\n- Allows users to simulate PXRD patterns with a wide variety of realistic settings not offered in other packages: grain size, stress, temperature, orientation, background, drift, noise. This is very important for addressing the sim2real gap in computational materials science.\n- Some really cool figures (Figure 1, Figure 2) that show that we can simulate PXRD patterns that look like real experimental data. Other leading packages like Pymatgen are known to be unable to do this.\n- Validates that models trained on these simulated PXRD patterns can generalize to experiments on real data.\n- Well-presented and easily usable open-source code, especially this early in the review process. It gives me faith that this is a reproducible, adaptable contribution to the community.\n- Brings to light the weaknesses of previous works (CNN-based) in dealing with the unbalanced symmetry classification task. This is especially important, as it would presumably be more important to analyze the less common symmetry groups.", "weaknesses": "I have a number of important suggestions for improvement, but most of these are easily addressable, rather than fundamental limitations of your work:\n- Baseline for Dataset: This work is under the datasets and benchmarks task. You certainly addressed the benchmark part of this, by evaluating multiple space group classification models on your work. However, it is necessary to have baselines for the dataset. In particular, if you simulated PXRD patterns with Pymatgen and GSAS-II [6] and trained models on these, how would it perform on experimental data, as compared to training on SimXRD? The value of the dataset presumably comes from the fact that training on it would reflect real-world conditions with better fidelity, and thus, lead to boosted performance in testing, as compared to training on less realistic patterns. Thus, you need to show that other available datasets/simulators (Pymatgen, GSAS-II) cannot lead to the same sim2real transfer.\n- Presentation of Figures: It's really cool that you can simulate PXRD patterns that look like actual experimental data. I'd like to see some more examples of these in a larger diagram. I only saw some small pictures in Figure 1 and Figure 2. I think displaying 4-6 examples of these in a large figure (preferably in the main text) would really drive home the point that your work provides the most realistic simulations of PXRD patterns available. In line with the previous comment, you should also compare to what other simulators (Pymatgen, GSAS-II) can give.\n- Simulation Fidelity Evaluation: Can you take maybe 10 experimental patterns with known ground truth structures, and simulate patterns with SimXRD, Pymatgen, and GSAS-II? Then see the R_w goodness of fit value as compared to the experimentally observed? Of course, in this experiment, you'd need to play with the parameters (if available) to get the best fit possible from each software (although I think only GSAS-II, SimXRD, and possibly TOPAS, if you have that; make this possible). I see that you already have the embryo of this in Figure 1, so hopefully it should not be that hard to do on a few patterns. Again, this would emphasize the point that you have the most realistic simulator available.\n- Address Structure Solution: Although symmetry classification is important, it's just one step towards the grand challenge of end-to-end structure determination from PXRD. This end-to-end solution paradigm has gained a lot of interest, as of 2024. You seem not to cite the relevant works in the field: please look at them and cite them [1, 2, 3, 4, 5]. It's probably not feasible to train or evaluate such generative structure prediction models in such a short period of time, especially since most of them are very new and not-yet open source. But at least mention them, because I think in the future, this dataset will be very valuable for training the PXRD structure solvers, and you should let the relevant authors see that your work is relevant to them, so that the impact of your work can be maximized!\n- Writing: It seems that your intro is fixated on the benchmark part of your contribution, but I think you can emphasize more how this dataset itself is valuable. Again, in my own experience, I have actually been looking for a dataset like this, to address the sim2real gap common in computational crystallography. For instance, out of the bullets on the bold bulleted list at the end of page 2, I actually find the third (\"Experimental Data Generalization\") more interesting, and I think you could emphasize that direction more in your presentation. \n\n[1] Guo, G., Goldfeder, J., Lan, L., Ray, A., Yang, A. H., Chen, B., ... & Lipson, H. (2024). Towards end-to-end structure determination from x-ray diffraction data using deep learning. npj Computational Materials, 10(1), 209.\n\n[2] Lai, Q., Yao, L., Gao, Z., Liu, S., Wang, H., Lu, S., ... & Ke, G. (2024). End-to-End Crystal Structure Prediction from Powder X-Ray Diffraction. arXiv preprint arXiv:2401.03862.\n\n[3] Guo, G., Saidi, T., Terban, M., Billinge, S. J., & Lipson, H. (2024). Diffusion Models Are Promising for Ab Initio Structure Solutions from Nanocrystalline Powder Diffraction Data. arXiv preprint arXiv:2406.10796.\n\n[4] Li, Q., Jiao, R., Wu, L., Zhu, T., Huang, W., Jin, S., ... & Chen, X. (2024). Powder Diffraction Crystal Structure Determination Using Generative Models. arXiv preprint arXiv:2409.04727.\n\n[5] Riesel, E. A., Mackey, T., Nilforoshan, H., Xu, M., Badding, C. K., Altman, A. B., ... & Freedman, D. E. (2024). Crystal structure determination from powder diffraction patterns with generative machine learning. Journal of the American Chemical Society.\n\n[6] Toby, B. H., & Von Dreele, R. B. (2013). “GSAS-II: the genesis of a modern open-source all purpose crystallography software package”. Journal of Applied Crystallography, 46(2), 544-549. doi:10.1107/S0021889813003531", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)