Wyckoff Transformer: Generation of Symmetric Crystals
Authors: Nikita Kazeev
Wei Nong
Ignat Romanov
Ruiming Zhu
Andrey E Ustyuzhanin
Shuya Yamazaki
Kedar Hippalgaonkar
Venue: ICML 2025 poster (2025)
Keywords: material design
machine learning
crystal generation
space group symmetry
Transformer
Wyckoff position
generative model
autoregressive model
permutation invariance
Primary area: applications->chemistry_physics_and_earth_sciences
TL;DR: A fast generative model for crystals conditioned on space group symmetry; based on Wyckoff positions.

Abstract: Crystal symmetry plays a fundamental role in determining its physical, chemical, and electronic properties such as electrical and thermal conductivity, optical and polarization behavior, and mechanical strength. Almost all known crystalline materials have internal symmetry. However, this is often inadequately addressed by existing generative models, making the consistent generation of stable and symmetrically valid crystal structures a significant challenge. We introduce WyFormer, a generative model that directly tackles this by formally conditioning on space group symmetry. It achieves this by using Wyckoff positions as the basis for an elegant, compressed, and discrete structure representation. To model the distribution, we develop a permutation-invariant autoregressive model based on the Transformer encoder and an absence of positional encoding. Extensive experimentation demonstrates WyFormer's compelling combination of attributes: it achieves best-in-class symmetry-conditioned generation, incorporates a physics-motivated inductive bias, produces structures with competitive stability, predicts material properties with competitive accuracy even without atomic coordinates, and exhibits unparalleled inference speed.

Wyckoff Transformer:
Generation of Symmetric Crystals
Nikita Kazeev
Wei Nong? ?
Ignat Romanov Ruiming Zhu? Andrey Ustyuzhanin 451
Shuya Yamazaki
Kedar Hippalgaonkar 126
Abstract
1. Introduction
Crystal symmetry plays a fundamental role in determining its physical, chemical, and elec- tronic properties such as electrical and thermal conductivity, optical and polarization behavior, and mechanical strength. Almost all known crystalline materials have internal symmetry. However, this is often inadequately addressed by existing generative models, making the consistent generation of stable and symmetrically valid crystal structures a significant challenge. We introduce WyFormer, a generative model that directly tackles this by formally conditioning on space group symmetry. It achieves this by using Wyckoff positions elegant, compressed, and discrete structure representation. To model the distribution, we develop a permutation-invariann autoregressive model based on the Transformer encoder and an absence of positional encoding. Extensive experimentation demonstrates WyFormer's compelling combination of attributes: it achieves best-in-class connuettyy-cnditionne generation, incorporates physics-motivated inductive bias, produces structures with competitive stability, predicts material properties with competitive accuracy even without atomic coordinates, and exhibits unparalleled inference speed. https: //github. com/ Symmet ryAdvantage/ MyckoffTransforne
Discovery of materials with desirable properties is the cor- nerstone of civilization from the stone age the bronze age and now in the silicon age, the ability to wield materi- als with different properties and function has transformed society Pyzer-Knapp et al., 2022). However, for the most part, the search of new materials as well as new function- alities, has proceeded through a traditional route of trial- and-error, also called the Edisonian approach (Wang et al., 2024). The space of all possible combinations of atoms forming periodic structures is intractably large, Cao et al. (2024) gauge it at 10160, It is not possible to fully screen this space or even to enumerate it. Materials that exist under realistic conditions, however, occupy a small part of this set of possibilities (Curtarolo et al., 2013). It consists of the meergticalll--avoree combinations of atoms that are held together through covalent, ionic, metallic and other chemical bonding. generative model that outputs novel priori stable materials will speed up automated material design by orders of magnitude.
the basis for an
1.1. Space groups and Wyckoff' positions
A crystal structure can be represented by lattice vectors and atomic basis. The lattice provides a periodic geomet- ric framework in three-dimensional space, defined by the lattice matrix = [1, 12, 1g] E p3x3 with a basis of an atom (or group of atoms) that occupy any lattice point. The atomic positions in real space are hence given by X [2, XN] R3xN where N is the number of atoms in the unit cell. These positions fractional coordinates F [f fz. ........ fw] [0, 1)3xN related to real-space coordinates by F L X, ensuring atomic positions remain consistent within the periodic lat- tice. The periodic arrangement can be further constrained by the space group G, finite set of symmetry operations g G defined as g X RX + where R O(3) is 3 x 3 orthogonal transformation matrix representing rotations, reflections, combinations thereof, and t E R is 3 x 1 translation vector. These symmetry operations collectively form the 230 distinct space groups, which com- prehensively classify all possible crystal symmetries in three dimensions (Fedorow, 1892; Hahn al., 1983). Each space group defines the allowable positions for atoms within the
also be expressed in
'Institute for Functional Intelligent Materials University of Sin- gapore, Block $9, Level 9, Science Drive 2, Singapore 117544 School of Materials Science and Engineering, Nanyang Techno- logical University, Singapore 639798 HSS University, Myasnit- skaya Ulitsa, 20, Moscow, Russia, 101000 "Constructor University, Bremen, Campus Ring 1, 28759, Germany 'Constructor Knowl- edge Labs, Bremen, Campus Ring 1, 28759, Germany "Institute of Materials Research and Engineering, Agency for Science Technol- ogy and Research, Fusionopolis Way. Singapore, 138634. Cor- respondence Nikita Kazeev <kazeevn @ gmail.com>. Kedar Hippalgaonkar <kedar edu....
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Wyckoff Transformer: Generation of Symmetrie Crystals
(Wyckoff, 1922). Mathematically, WP encompasses all points whose site symmetry groups are conjugate subgroups of the full space group (Kantorovich, 2004). An illustra- tion of WPs is present in Figure 1. Two different WPs in the same space group can share the same site symmetry. This is called symmetry equivalence and occurs when the Wyckoff positions can be mapped onto each other using higher-order symmetry operations. Continuing with the Fm-3m space group example, Wyckoff positions 4 (0,0,0) and 4b (12,12 appear distinct under conventional symme- try, but lattice center translation reveals their higher-order symmetry equivalence (see Figure 3), Such a transforma- tion is a coset representative of the affine normalizer, which introduces symmetry operations beyond the space group's symmetry operations, G. The Euclidean normalizer is de- fined as the largest symmetry group preserving G, but al- lowing additional transformations like centering translations or scaling, mapping Wyckoff positions onto each other in a higher-symmetry framework, forming the basis for enu- meration and augmentation in the next sections. We further explore this idea in (Yamazaki et al., 2025).
Figure 1. toy 2D crystal (Goodall al., 2020). It contains 4 mirror lines, and one rotation center. There are four Wyckoff positions, illustrated by shading. Magenta the Wyckoff position that is invariant under all the transformations, it only contains single point; red and yellow lie on the mirror lines, and teal only invariant under the identity transformation and occupies the rest of the space. Markers of the corresponding colors show one of the possible locations of an atom belonging to the corresponding Wyckoff position.
WPs for given space group are enumerated by Latin letters. typically in order of decreasing site symmetry. Each WP has defined multiplicity, which represents the number of equivalent atomic positions in the unit cell related by the symmetry operations of that space group. For example WP 2a has the highest site symmetry and multiplicity 2. The number of distinct WPs in a space group is finite, ranging from single WP in the simplest symmetry group P1 to as many 27 in the most complex space groups. Wyckoff positions can represent ID lines, 2D planes, open 3D regions within the unit cell. These fundamental concepts lattice, atomic basis, space groups, site symmetry, and Wyckoff positions - define a framework to unequivocally describe crystal structures, which is the foundation to our representation. See also Appendix A for an illustration.
unit cell. Every periodic crystal possesses at least the sim- plest level of symmetry, P1, which consists only of trans- lational symmetry. Most known crystals have additional internal symmetry, see Figure 2. This is not merely mathe- matical observation; optical, electrical, magnetic, structural and other properties are determined by symmetry, as shown by Malgrange et al. (2014); Yang et al. (2005), as well as our results in Section 3.2.
Within a given space group G, a subgroup forms the site symmetry, referring to the set of symmetry operations G, = [g G g f. f.} C G that leave specific point in the crystal invariant. These operations describe the local symmetrical environment, such as mirrors, screw axes, or inversions centered on a given region. Atoms located at the representative fractional coordinates f, generate equiva- lent positions RR,f: + 1 where n. the multiplicity of the -ymmetry--equivalent position. Higher site symmetry is in regions where multiple symmetry elements intersect, while those with lower site symmetry include only one sym- metry operation. Taking space group 225 Fm- 3m as an example, F' represents a face-centered lattice, site symmetry subgroup m-3m represents a highly symmetric environment at the center of cubie unit cell, where multiple symme- try elements intersect, including mirror planes and 3-fold rotoinversion axis (Hahn et al., 1983). In contrast, another lower site symmetry subgroup 3m corresponds to less symmetric environment with only a 3-fold rotation axis and a mirror plane.
1.2. Our contribution
Representing a crystal as an unordered set of tokens fused from the chemical elements and Wyckoff posi- tions; Section 2.1.
2. Encoding Wyckoff positions using their universally de- fined symmetry point groups and symmetry operations descriptors based on spherical harmonics; Section 1.
3. Wyckoff Transformer architecture and training proto- col that combine autoregressive probability factoriza- tion with pernutation invariance; Section 2.3.
4. Model invariance with respect to the arbitrary choice of the coset representative of the space group Euclidean normalizer, Sections 2.1, 2.3.
These site symmetry points, classified by their symme- try properties, are grouped into Wyckoff positions (WPs)
2

---

Wyckoff Transformer: Generation of Symmetric Crystals
MP-20
WyFormer
WyForDiffCSP++
CrystalFormer
+
WyCryst
DifcCSP
FlowMM
P-1
C2/m
Pnma Cmcm
p4/mmm
14/mmm
R-3m
P63/mmc
Pm-3m
Fm-3m
Figure 2. Distribution of space groups in MP-20 dataset (Xie et al., 2021) and generated samples. 10 space groups most frequent in MP-20 are labeled, 98% MP-20 structures belong to symmetry groups other than P1, Plot design by Levy (2024). The comparison of the distribution of generated samples' space groups the ground truth distribution is presented in Table 1, column Space Group x".
(3d=(4/mm.mm, 1) (1/2, 0, 0)
suitable partner, DiffCSP++, evaluate thoroughly.
(1b=fm-3mm, 1)) (1/2, 1/2, 1/2)
Wyckoff positions and machine learning. The concept of Wyckoff positions was originally published more than a 100 years ago (Wyckoff, 1922). Given the elegance of the representation, naturally, in modern times WPs have found their way into machine learning. The main limiting factor in their adoption was the ability of machine learn- ing algorithms to handle discrete structured data which is formed by WPs. WP-based representation was used for property prediction (Goodall et al., 2020; Jain & Bligaard, 2018; Möller et al., 2018; Goodall et al., 2022), and re- cently in generative models. Our work is inspired by Zhu et al. (2024), the first such model. uses a VAE over one-hot-encoul information about WPs, as opposed to our Transformer encoder, which is generally superior archi- tecture for categorical data. A14Science et al. (2023) use GFlowNet (Bengio et al., 2023) to sample space group and chemical composition, but not the full Wyckoff' representa- tion. A concurrent preprint (Cao et al., 2024) independently explores a Transformer-basee approach similar to ours; an- other concurrent work (Levy et al., 2024) uses diffusion Wyckoff position site symmetry, fractional coordinates, and lattice parameters.
Ti (1mmm, 0,
(1a=[m-3m, 0}) (0, 0,
(1b=(mm3m, 1)) (1/2, 1/2, 1/2)
(3c=[4mmmmm, 0)) (0, 1/2, 1/2)
Figure 3. Two equivalent Wyckoff representations of SrTiO3 mp-4651, depending the lattice center choice:
(Ti, (m-3m, 0) J. [Sr, (m-3mm) 1) ]. [O, (4/mm. m, 1)
(Ti, (m- 3m, }. J. [Sr, (m-3m, 0) [O, (4/mm. m, 0)]
5. Empirically, our model outperforms baseline methods in generating novel, symmetric, diverse materials con- ditioned on space group symmetry; Section 3.2.
6. Despite not using the information about atom coordi- nates, our model achieves property prediction perfor- mance competitive with the machine learning models that use the full structure; Section 3.2.
The main difference between our and most other approaches, that are based on Wyckoff positions is that they use Wyck- off letters as the representation. Wyckoff letter definitions depend on the space group, unlike site symmetry, leading to data fragmentation. Levy al. (2024) also use WP site symmetry, with mne-hot-encodig of symmetry operations per axis to represent it; Goodall et al. (2022) use the sum of mne-hot-encoding of sites to represent a WP; we treat site symmetry as a categorical variable and use learnable embed- dings. Zhu et al. (2024); Cao et al. (2024) don't take into account dependency of the Wyckoff letters on the arbitrary choice of the coset representative of the space group Eu- clidean normalizer. Finally, Cao et al. (2024) use positional encoding to establish the relationship between the chemi- cal elements and Wyckoff positions they occupy, while we combine them in one token.
1.3. Related work
Crystal generation is a burgeoning field, with most state- of-the-art models using a differentiable non- oon-inerrtible SO(3) invariant representation constructed from atom coordinates, such as a graph neural network. Then they use diffusion or flow matching to solve the generation problem (Jiao al., 2024a;b; Cao et al., 2024; Yang et al., 2023; Zeni et al., 2025; Xie et al., 2021; Klipfel al., 2023; Luo et al., 2024; Sinha et al., 2024). Our approach uses discrete Wyckoff space and fast autoregressive sampling, compared to gradual refine- ment in the aforementioned works. WyFormer complements them naturally by providing symmetry constraints and/or initial structure approximation the synergy with the most
3

---

Wyekoff Transformer: Generation of Symmetric Crystals
Spherical harmonics are widely used build fixed-length descriptor for spatial relationships (Bartók et al., 2013).
where 8; is the site symmetry, and n, is the enumeration, and F (e1y-..,,mi Cm]] are the chemical elements occupy- ing them. Spglib (Atsushi Togo & Tanaka, 2024) provides us mapping from crystal C' = (L, F. F)) to R, which is used to preprocess the training dataset. The problem solved by Wyckoff Transformer is sampling the distribution P (R) 3C : p(C') is stable), which is enabled by learning the following probabilities: p(e: E,-1, St-1, NN-1), and p(n, E1, SI, N₂-1), where E [el,..., , ei-1], S , -:-1) N [ ......,---1)
2. Wyckoff Transformer (WyFormer)
2.1. Tokenization
Our work is based on the inductive bias that for stable ma- terials space group symmetry and Wyckoff sites almost completely define the structure more than 98% of the ma- terials in MP-20 (Xie et al., 2021) and MPTS-52 (Baird et al., 2024) datasets, which together contain almost all experimen- tally stable structures from the Materials Project (Jain et al., 2013), have unique Wyckoff representations. Therefore, it is safe to assume that for almost any Wyckoff representation there is either none, or just one stable material conforming to it. Symmetry captured by this discrete part is sufficient to determine properties of material, such as piezoelec- tricity via non-centrosymmmetry; direct/indirect band gap via positions of the valenceecoonduction bands in the Bril- louin Zone, while the fractional coordinates can be linked to the magnitude of that property. We additionally prove this assumption by predicting various material properties; see Section 3.2. Given a Wyckoff representation, coordinates can be determined as discussed in Section 2.4.
p(s E,,, 5,,1,
2.1.1. SPHERICAL HARMONICS
Enumerations are defined by an arbitrary convention, in this respect they are no better than Wyckoff letters. We ad- dress this with a representation that is defined consistently across space groups. Consider a Wyckoff position consist- ing of set of k symmetry operations {R, + tt, i L...k.. We apply these operations to points I1 = [0. 0, 0j and
D2 = [1, 1, obtaining two matrices (1) and W w = R.æ, + L.a.. Finally, we convolve the transformed coordinates with spherical harmonics:
Φ arctan (Wr6) wW))); arccos((W(11)
We represent each structure set of tokens, as shown in Figure4. The first token contains the space group; the rest are divided into groups of three tokens, each representing a specific WP. The first token in each group is responsible for the type of atom that occupies the position following the site symmetry, while the last token is for the so-called enumeration. Several WPs can have the same site symmetry. To differentiate those WPs we enumerate them separately within each space group and site symmetry according to the conventional WP order (Aroyo et al., 2006). For example, in space group 225 present in Figure4 WP 4 is encoded as (m-3m, 0), 4b as (m-3m, 1), and 8 as (-43m, 0). A more comprehensive example can be found in Ap- pendix S. The purpose of this encoding is to take advantage of the fact that, unlike Wyckoff letters, site symmetry defini- tion is universal across different space groups. An ablation study comparing our representation with Wyckoff letters is in Appendix N.
& 1) Σ (w (0 (10) (. ,
1/kk,
where is the degree of spherical harmonics, parameter, and the resulting complex vectors h2) and (2) each have n + 1 dimensions. = 2 is enough to disambiguate all Wyckoff positions with the same site symmetry belonging to the same space groups; n = 1 is not. Finally, we obtain the final 2n + 2 dimensional descriptor 8 by concatenation: s = R(h ()) + h (2) @ ((h (1) @ (2)). The harmonic repre- sentation is not directly invertible; in the main section of the paper, only use it for property prediction, which results in a slight performance increase, as shown in Appendix O. A way to adapt the harmonics-batt ture generation is discussed in Appendix P.
representation for struc-
2.2. Model architecture
Elements, site symmetries, and emmmraations are each em- bedded with a simple lookup table with trainable weights, the embeddings are concatenated. Then we apply a linear layer to provide each head of the multihead attention with information from all three parts of token.
Such an encoding has an additional advantage. For a given crystal, the conventional unit cell can sometimes be chosen in several equivalent ways, which changes the Wyckoff positions (see Figure 3) corresponding to each atom, but not their site symmetries. We collect all the arbitrariness in one variable, which leaves the rest of the representation strictly invariant to that choice.
Since our model is conditioned space group, preventing data fragmentation is of utmost importance. To this end, the space group is not encoded just as a categorical variable. Building upon (A14Science et al., 2023) and similarly to Levy et al. (2024) we use Xtal to get one-hot-encoded 15 x 10 matrix that represents symmetry elements on each
Formally, we define Wyckoff representation of a structure as R = (G, E. W), where G is the space group, W = [w],.... wm.] are the Wyckoff positions with w; = (st, ni),
4

---

Wyckoff Transformer: Generation of Symmetric Crystals
Element
Mg
Tm
Hg
Space
Site Symmetry
225
m-3m
m-3m
-43m
Group
Enumeration
0
0
Figure 4. An example of structure tokenization, TmMgHga mp-865981
axis for each space group, flatten discard the positions that do not vary across the dataset and use the resulting vector as the space group embedding. Then we apply linear layer, the representation becomes learnable- but still transferable between space groups.
De novo generation The training pipeline and architecture are shown in Figure 5. We train the model to predict next part of token in cascade fashion: first the chemical ele- ment conditioned on the previous tokens, then site symmetry conditioned on the previous tokens and the element and, fi- nally, enumeration conditioned on the previous tokens, the element and the site symmetry. On each training iteration we randomly sample known sequence length and the part of the cascade predict; place MASK tokens as necessary, input the known parts of the sequences into the model, com- pute cross-entropy loss between the predicted scores and the target.
Token sequences are used as input for a Transformer en- coder (Vaswani, 2017: Devlin, 2018). Wyckoff represen- tation is germutation-invariin so Transformer; we do use positional encoding, making the model formally permutation-invariann with respect to the input.
De novo generation We use enumerations representation. We additionally add a STOP token to each structure. To represent states where some parts of token are known and others are not, we replace those values with MASK. We also add a fully-connected neural network for each part of the token that want to predict, three in total. To get the prediction, take the output of Transformer encoder the token containing MASK value(s), concatenate it with one-hot vector encoding presence in the input sequence of each possible value for this token part, and use it as the input for the corresponding fully-comnected network.
Unlike Transformer itself, auto-regressive generation not invariant. The number WPs is small, the average in MP-20 just 3.0; this again allows us to train the model be invariant with augmentation by shuffling the order of every Wyckoff representation every training epoch. Moreover, we use multi-class loss when training to predict the first cascade part, chemical element, further reducing learning complexity.
On MP-20 the model is trained for 9 x 10a epochs using SGD optimizer without batching; due to the efficiency of the representation gradient backpropagation for the entire dataset fits into GPU memory. We the loss the vali- dation dataset for early stopping, learning rate scheduling, and manual hyperparameter tuning.
Property prediction We take the Transformer encoder out- puts tokens, excluding the token corresponding the space group, compute weighted average with weights being equal to the multiplicities of WPs, and use the result as input for fully-connected neural network that outputs scalar predicted value.
Property prediction The model is trained using MSE loss with batch size 500, and Adam optimizer. For both MP-20 and AFLOW training takes around 5k epochs.
2.3. Training
Following the approach of Wang et al. (2023); Abramson et al. (2024), we use simple architecture and do not strictly enforce invariance with respect to the choice of the equiva- lent Wyckoff representations, but rather leave it as a training goal by picking a randomly selected equivalent representa- tion every training epoch. It is especially viable because of the low number of variants; in MP-20 dataset for 96% structures there are less than 10.
Hyperparameters are available in Appendix L.
2.4. Structure generation
We generate crystals conditioned on space group number which is sampled from the combination of training and validation datasets, illustrated in Figure 7. Wyckoff repre- sentation is then autoregressively sampled using WyFormer. We use two ways to generate the final crystal structure con- ditioned on the representation, the details are described in Appendix C. They both start with randomly sampling a structure conditioned on the Wyckoff representation with Xtal (Fredericks al., 2021). Then it's relaxed with CrySPR (Nong et al., 2024) and CHGNet (Deng et al., 2023)
The experimental results we present were obtained by train- ing separate models for property prediction and de novo generation. A single model do both is possible, we leave for the future work.
5

---

Wyckoff Transformer: Generation of Symmetric Crystals
WyFormer
One-hot encoded symmetry per axis
Eu(Cu Eg/mmm)) Ezn(0)
mmm
Layar for Black Mixing
Prediction
Cross Entrr-Entrppy
Encoder
Eu[C] Egg(m2m) E..
m22
m22
Eg(Ai) MASK
m2m
MASK
Linear
MASK
MASK
Figure 5. Model training pipeline. (I) The crystal is converted into token sequence where the first token is the space group number and then token triplets in the order atom, site, symmetry and enumeration. Then the triplets are randomly shuffled.. (2) Randomly sample the number of fully known Wyckoff positions and the part of the next triplet to be predicted; mask unknown tokens, remove unknown Wyckoff positions. (3) Embed the tokens using simple lookup tables; for cach Wyckoff positions concatenate tokens corresponding to in the emhedding dimension. (4) A linear layer mixes the features to provide homogencous input to multiple attention heads. (5) The sequence passed through the Transformer Encoder. (6) An MLP is applied the token the output sequence. (7) The loss is cross entropy of the prediction and the true value of the token being predicted.
DiffCSP+ (Jiao al., 2024b).
Xie al. (2021) and then followed by most of the subse- quent work.
3. Experimental Evaluation
Validity Xie et al. (2021) proposed verifying crystal feasi- bility according to two criteria:
3.1. De novo generation
Structural validity means that no two atoms are closer than 0.5Á. All structures in MP-20 and almost all structures produced by state-of-the-art models fulfill it.
3. DATASETS
We use MP-20 (Xie et al., 2021), which contains almost all experimentally stable materials in Materials Project (Jain et al., 2013) with maximum within 0.08 V/atoo of the convex hull, and formation en- ergy smaller than 2 e//ttom, 45 229 structures in total, split 60/20/20 into train, validation and test parts. Additionally, we train and evaluate WyFormer MPTS-52 (Baird et al., 2024), a more challenging subset of Materials Projects con- taining materials with up to 52 atoms per unit cell.
Compositional validity means having neutral charge (Davies et al., 2019). Only 90% of MP-20 structures pass this test meaning that nonconforming structures are physically pos- sible if somewhat rare.
atoms per unit cell,
Novelty and uniqueness The purpose of de novo generation is to obtain materials. Generated materials that already exist in the training dataset increase the model performance according to structure stability and similarity metrics, but such structures are useless for material design and just in- crease the gap between the proxy metrics and the model fitness for its purpose. Therefore we exclude generated materials that are not novel and unique from metric compu- tation. On a deeper level, generative models for materials
3.1.2. METRICS
Structure property similarity metrics Coverage and Prop- erty EMD (Wasserstein) distance, have been proposed low-cost proxy metric for de novo structure generation by
6

---

Wyckoff Transformer:
Generation of Symmetric Crystals
Table Evaluation. Symmetry metrics are computed only using novel structurally valid examples. Note that the 1000 and 105-example metrics are computed using MP-20 train and validation as reference datasets for novelty. while the 10 10000 100-example S.U.N. only uses MP-20 train remain compatible with the reported values. Bold indicates the values within p 0.1 statistical significance threshold from the best. Values marked by were computed by Miller et al. (2024), the rest by us; see note H.1 for important caveat; in short, the values (brackets) are less accurate, but are compatible with each other.
Method/Metric
Novel Unique
P1 (%)
Space Group S.U.N. % t S.S.U.N. %
S.U.N. % f
Templates (#)
ref 1.7
x²
Emall < 80 meV
Emall < 0 meV
Sample size Relaxation
1000
1000
1000
105
105
10 000
CHGNet
CHGNet
CHGNet
DFT
DFT
DFT
WyForeerCHGNet
180
3.24
0.223
23.1
22.3
NyFormerDiffCCS++
186
1.46
0.212
22.2
21.1
3.83 (4.14)
DiffCSP++
10
2.57
0,255
14.4
14.4
CrystalFormer
74
0.91
0.276
20.1
20.1
SymmCD
2.35
0.24
20.7
20.7
WyCryst
165
4.79
0.710
5.5
5.5
DiffCSP
76
36.57
7.989
22.2
20.6
(3.34*)
FlowMM
44.27
12.423
17.8
16.9
- (2.34
WyFormer MPTS-52
386
0
0.225
are subject to exploration/explootttinn trade-off: the more physically similar are the sampled materials to the train- ing dataset, the more likely they are stable and distributed similar the data, but the less useful they are for the pur- pose of material design. From purely machine learning point of view, novelty percentage serves proxy metric for overfitting.
We argue that presence of symmetry is good proxy value for structure feasibility that difficult to capture in standard DFT computations, and would require finite-temperature calculations and/or improved methodologies.
Novel Unique Templates is the number of the novel unique element-agnostic Wyckoff representations (Section 2.1) in the generated sample. Element-agnostic means that we remove the chemical element, while retaining the sym- metry information. For example, for the TmMgHga in Figure 4, it will be (X, (m-3m, 0)), (X, (m-3m, 1)), (x, (-43m, 0) and its equivalent. An impor- tant difference between our work and (Levy et al., 2024) is that we take into account equivalence of Wyckoff represen- tations. The metric provides a lower limit on overfitting and physically meaningful sample novelty: if two materials have different symmetry templates, their physical properties will be different, while the inverse is not always true. serves as addition to the strict structure novelty, which provides the upper bound. Finally, the ability of model to generate new templates allows it generate more structures before starting to repeat itself, as we demonstrate in Appendix J.
Stability determines whether the material, fact, exists under normal conditions. is estimated by computing en- threshold.
ergy above convex hull, and comparing Materials Project is the source of the reference structures for the hull. The details are in Appendix G.
S.U.N. (Zeni et al., 2025) combines the above into the frac- tion of stable unique novel structures.
Symmetry of the structures has paramount physical im- portance. Controlling symmetries also leads to control physical, electronic, and mechanical behavior, which is desirable in property-directed inverse design of materi- als. For example, in electronic materials, higher symmetry can improve carrier mobility and uniformity in electronic band structure, enhancing performance in applications such as semiconductors or optoelectronics. Furthermore, high- symmetry structures often exhibit isotropic properties, mean- ing their behaviors are the same in all directions, making them more versatile for industrial use. We use four metrics for evaluating the ability of the generative models to repro- duce the symmetry present in the data and, ultimately, in nature:
Space Group x² is the x² statistic of difference of the frequencies of space groups between the generated and test datasets.
S.S.U.N. is the percentage of the structures that sym- metric (space group not P1), stable, unique and novel.
3.1.3. METHODOLOGY
P1 is the percentage of the structures that have symmetry group P1. In MP-20 the corresponding number is just 1.7%.
WyFormer was trained using MP-20 dataset follow- ing the original train/test/validtiion split. We sampled
7

---

Wyckoff Transformer: Generation of Symmetric Crystals
10 Wyckoff representations, then obtained structures using CrySPR+CHGNer (NyFormerCHGNet) and Dif- ICSP++ (WyFomeeDDifiCSPt)) approaches described in Section 3.1.3.
lar to the training dataset. It also produces sizable fraction of a priori structurally invalid crystals.
WyCryst suffers from even lower novelty, stability and distribution similarity metrics.
WyCryst (Zhu et al., 2024) only supports a limited number of unique elements per structure, therefore we trained it on subset of MP-20 containing only binary and temaary compounds, 35 575 in total. An evaluation of WyFormer trained on the same dataset is present in Appendix K. As WyCryst also produces Wyckoff representations, and not structures, the same CrySPR+CHGNet procedure was used to obtain them.
DMfCCSP and FlowMM can not be conditioned on the sym- metry group, and produce a large fraction of unrealistic asymmetric structures.
SymmCD is concurrent work based on similar princi- ples, and achieves similar performance, except for lesser number of Novel Unique Templates.
On MPTS-52, as expected, WyFormer shows higher nov- elty as well as template novelty. In terms of distribution similarity metries WyFormer performs largely similarly on MP-20 and MPTS-52. We used CHGNet predict for- mation energies estimate S.S.U.N.: 24.4% MPTS-52, compared to 35.2% on MP-20. This reflects the increased difficulty, and shows that WyFormer is still very much capa- ble of generating stable structures in this setting.
CrystalFormer (Cao et al., 2024) code and weights pub- lished by the authors were used by us to produce the sample, conditioned on the space groups sampled from MP-20.
DifCCSP (Jiao et al., 2024a), DiffCSP++ (Jiao et al., 2024b), and Symm CD (Levy et al., 2024) samples were provided by the authors. The DiffCSP++ sampling process is con- ditioned on Wyckoff templates from the training dataset, which includes the space group.
3.2. Material property prediction
For each model a data sample containing 1000 structures was relaxed using CHGNet. The generated samples were filtered for uniqueness, more than 99.5% of structures for every method passed the filtering. We computed for DFT for 105 novel structures for each method; detailed description of the settings is available in Appendix H.
MP-20 dataset contains two properties: formation energy and band gap. which we predict using WyFormer. The re- sults are shown Table WyFormer achieves competitive results with the models that use full structures.
We also utilize the AFLOW database (Curtarolo et 2012), which contains 4905 compounds spanning diverse range of chemistries and crystal structures. We predict four prop- erties: thermal conductivity, Debye temperature, bulk mod- ulus, and shear modulus. The data are divided into training, validation, and test sets using a 60/20/20 split. The results are presented in Table 4; WyFormer demonstrated supe- rior performance in predicting thermal conductivity. For the remaining three properties, the model's performance is comparable to the baselines.
Additionally, we computed DFT for 10 000 structures from WyFormer, and compared S.U.N. values to the values re- ported by Miller al. (2024).
3.1.4. DE NOVO STRUCTURE GENERATION RESULTS
Evaluation results are present in Tables and 2; a sample of generated structures is illustrated in Figure 9.
WyFormer achieves 24% higher S.U.N. on the 10000- structure sample compared the best available baseline; best template novelty, fraction of asymmetric structures and space group distribution reproduction. On the 105-structure sample, the difference WyFormer, CrystalFormer, DiffCSP, FlowMM, and SymmCD the difference between S.U.N. and S.S.U.N. values is not statistically significant.
From this we argue that the symmetries and composition of a crystal alone already carry a considerable amount of information about its properties. This is especially true for band gap, where Brillouin zones are defined by symme- try, and thermal conductivity, which is non-equilibrium phonon transport property conditioned on underlying sym- metry of the structure; according to the first order approx- imation kinetic theory, higher symmetry crystals typically have higher thermal conductivity due to (1) higher group velocities and (2) longer scattering times due to lower an- harmonicity (Newnham, 2004; Yang et al., 2021).
DiffCSP++ has lower stability, despite using a priori valid structure templates from the data. As we show in Ap- pendix J, the lack of template novelty limits the diversity, and the model starts to repeat itself. DiffCSP++ oversam- ples the structures with the large number of unique elements, WyFormer matches the distribution most closely, as depicted in Figure 10.
4. Conclusions and Limitations
CrystalFormer has lower novelty, which means that the model has been overfitted, and the structures are more simi-
Emal determined from formation energy as a proxy for sta- bility is commonly used, but is imperfect, as it doesn't 'ttt take
8

---

Wyckoff Transformer: Generation of Symmetrie Crystals
Table Evaluation of the methods according to validity and property distribution metrics. Structures were relaxed with CHGNet. Following the reasoning in Section 3.1.2, we apply filtering by novelty and structural validity, and do not discard structures based on compositional validity. evaluation following protocol proposed by Xie et (2021) is available in Appendix
Method
Novelty
Validity (%) f
Coverage (%) t
Property EMD
(%) f
Struct.
Comp.
COV-R
COV-P
p
E
Nelem
WyFot
90.00
99.56
80.44
98.67
96.72
0.74
0.053
0.097
89.50
99.66
80.34
99.22
96.79
0.67
0.050
0.098
DiffCSP++
89.69
100.00
85.04
99.33
95.80
0.15
0.036
0.504
CrystalFormer
76.92
86.84
82.37
99.87
95.13
0.52
0.100
0.163
SymmCD
88.77
95.82
84.88
99.55
94.66
0.62
0.102
0.525
WyCryst
52.62
99.81
75.53
98.85
87.10
0.96
0,113
0.286
DiffCSP
90.06
100.00
80.94
99.55
96,21
0.82
0.052
0.294
FlowMM
89.44
100.00
81.93
99.67
99.64
0.49
0.036
0.131
WyFormer MPTS-52
98.7%
99.3%
76.7%
0.698
0.108
0.228
Table One-shot energy and band gap prediction. We computed CHGNet energy predictions on the MP-20 dataset, the rest of the baseline values are from (Lin al., 2023); The MP-20 test set is part of CHGNet training set. Xie & Grossman (201 8); Jha et al. (2019) report the error between DFT-computed and experimental results x 0.08 eV for energy, and zz 0.6 eV for band gap.
Table 4. MAE values for AFLOW dataset; baseline values are by Wang et al. (2021).
Method
Thermal conductivity
Debye temperature
Bulk modulus
Shear modulus
Roost
2.70
37.17
8.82
9.98
CrabNet
2.32
33,46
8.69
9.08
Method
Energy meV
Band gap meV
Train
Test
HotCrab
2,25
35.76
9.10
9.43
ElemNet
3.32
45.72
12.12
13.32
RF
2.66
36.48
11.91
10.09
CGCNN
31
292
WyFormer
2.20
36.36
9.63
10.14
SchNet
33
345
Materials Project
MEGNet
30
307
GATGNN
33
280
2018.6.1
ALIGNN
22
218
An important part of the future work is Crystal Structure Prediction (CSP). Unlike the models that work with atoms and coordinates, it is hard ensure that WyFormer output strictly conforms to given stoichiometry. But we can add the stoichiometry as generation condition, like space group. Then, as as we show in Appendix 6, WyFormer is four orders of magnitude faster than other CSP solutions, which allows to simply use rejection sampling.
Matformer
21
211
PotNet
19
204
CHGNet
34
MPTrj
MP-20
WyFormer
25
234
MP-20
into account configurational and vibrational entropic con- tributions, and hull determination relies on already known structures. Moreover, our results, along with Miller al. (2024) show that generated structures with space symmetry group P1 are consistently found stable at much higher rate than they occur in nature. There are two logical ex- planations: either DifICSP and FlowMM have, in passing, discovered a new class of asymmetric materials - or our stability estimation methodology systematically flawed. In our biased opinion the latter is much more likely.
In conclusion, we demonstrate that WyFormer represents novel advancement in generation of realistic symmetric crystals by leveraging Wyckoff positions encode material symmetries. WyFormer achieves higher degree of struc- ture diversity compared to baselines by encoding the discrete symmetries of space groups without relying on atomic co- ordinates. This unique tokenization of symmetry elements enables the model to explore a reduced, yet highly repre- sentative space of possible configurations. resulting in more stable and purportedly synthesizable crystals. The model respects the inherent symmetry of crystalline materials, out- performs existing models in generating both novel and phys- ically meaningful structures. These innovations underscore the method's 's potential in accelerating material discovery while maintaining accuracy in predicting key properties like formation energy and band gap.
Novelty and diversity evaluation is crucial and open ques- tion. A model can generate structures that are similar to the ones in the training dataset, and are valid, but not very useful for new material design. Counting complete duplicates is a step in the right direction, but doesn't 't measure substantial sample diversity (Hicks et al., 2021).

---

Wyckoff Transformer: Generation of Symmetrie Crystals
Acknowledgements
Mater, Meth., 4():2384622-2384886 2024. doi: 10. URL http:::/dd/oo /doi. 0080277660400.22222222228 20222.22222222 2384822.
We thank Lei Wang for insights on symmetry-conditioned generation; Andrey Okhotin for insights on permutation invariance and the 10k CHGNet computation; Benjamin Miller for discussion of the evaluation metrics; Rui Jiao and Daniel Levy for providing data samples.
Baird, S. G., Sayeed. H. M., Montoya, J., and Sparks, A Python library for
T. D. matbench-genmetrie benchmarking crystal structure generative models using time- splits of Materials Project structures. Jour- nal of Open Source Software, 9(97);5618, 2024. doi: 10.21105//oss.056188 URL petpss://oli htt.ddii.oogg/ooo// 21105/ joss. 05618.
This research/project is supported by the Ministry of Edu- cation, Singapore, under its Research Centre of Excellence award to the Institute for Functional Intelligent Materials (I-FIM, project No. EDUNC-33- 1----18-27--- 27)-) This re- search/project is supported by the National Research Foun- dation, Singapore under its AI Singapore Programme (AISG Award No: AISG3-RP-2022-00
Bartók, A. P., Kondor, R., and Csányi, G. On representing chemical environments. Physical Review B-Condenseed Matter and Materials Physics, 87(18): (0(18))184114) 184115, 2013.
and from the MAT-GITT Program at S via the AME Programmatic Fund by the Agency for Science Technology and Research under Grant No. M24N4b0034. The computational work for this article was performed on resources at the National Super- computing Centre of Singapore (NSCC). Computational work involved in this research work is partially supported by NUS IT's Research Computing group. The research used computational resources provided by Constructor Tech. This research was supported in part through computational resources of HPC facilities at HSE University.
Bengio, Y., Lahlou, S., Deleu, T., Hu, E. J., Tiwari, M., and Bengio, E. GFlowNet foundations. The Journal of Machine Learning Research,
2023.
Cao, Z., Luo, X., Jv, and Wang, L. Space group informed transformer for crystalline materials generation. arXiv preprint arXiv:2403. 15734, 2024.
Curtarolo, S., Setyawan, W., Hart, G. I.., Jahnatek, M., Chepulskii, R. V., Taylor, R. H., Wang, S., Xue, J., Yang, K., Levy, O., Mehl, M. J., Stokes, H. T., Dem- chenko, D. O., and Morgan, D. AFLOW: An auto- matic framework for high-throughput materials discov- ery. Computational Materials Science, 58:218-226. 2012. ISSN 09270256. doi: 10.1016/j.commm 11016ff.commattcc22011 02.005. URL http://dx doi. org/10. commatsci. 2012. 02.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Curtarolo, S., Hart, G. L.., Nardelli, M. B., Mingo, N., San- vito, S., and Levy, O. The high-throughput highway to computational materials design. Nature materials, 12(3): 191 2013.
References
Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T.,
Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature, pp. 1-3, 2024.
Davies, D. W., Butler, K. T., Jackson, A. J., Skelton, J. M., Morita, K., and Walsh, A. Smact: Semiconducting ma- terials by analogy and chemical theory. Journal of Open Source Software, 4(38):1361 2019.
A14Science, M., Hernandez-Gaarci, A., Duval, A., Volokhova, A., Bengio, Y., Sharma, D., Carrier, P. L., Benabed, Y., Koziarski, M., and Schmidt, V. Crystal- GFN: sampling crystals with desirable properties and constraints. arXiv preprint arXiv:2310.04925 2023.
Deng, B., Zhong, P., Jun, K., Riebesell, J., Han, K., Bartel, C. J., and Ceder, G. Chgnet as pretrained universal neural network potential for charge-informed atomistic modelling, Nature Machine Intelligence, 103::1331114,,,, 2023.
Aroyo, M. I., Perez-Mato. J. M., Capillas, C., Kroumova, E., Ivantchev, S., Madariaga, G., Kirov, A., and Won- dratschek, H. Bilbao crystallographic server: I. databases and crystallographic computing programs. Zeitschrift für Saapallogrrhh---rystllliie Materials, 221(1)115-27, 2006.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint 1eXiv:811004880 2018
Fedorow, E. v. II. Zusammenstellung der krystallographis- chen Resultate des Herrn Schoenflies und der meinigen. Zeitschrift für riistalloraphie-CCysttlllll Materials, 20 (1-6):25-75, 1892.
Atsushi Togo, K. S. and Tanaka, I. Spglib: a software library for crystal symmetry search. Sei. Technol. Adv.
10

---

Wyckoff Transformer: Generation of Symmetrie Crystals
Fredericks, S., Parrish, K., Sayre, D., and Zhu, Q. PyXtal: A Python library for crystal structure generation and sym- metry analysis. Computer Physics Communications, 261: 107810, 2021.
Jha, D., Choudhary, K., and Tavazza, F. Enhancing ma- terials property prediction by leveraging computational and experimental data using deep transfer learning. Nat Commun 10, 2019.
Ganose, A. M., Sahasrabuddhe, H., Asta, M., Beck, K., Biswas, T., Bonkowski, A., Bustamante, J., Chen, X., Chiang, Y., Chrzan, D., Clary, J., Cohen, O., Ertural, C., Gallant, M., George, J., Gerits, S., Goodall, R., Guha, R., Hautier, G., Horton, M., Kaplan, A., Kingsbury, R., Kuner, M., Li, B., Linn, X., McDer- mott, M., Mohanakrishnan, R. S., Naik, A., Neaton, J., Persson, K., Petretto, G., Purcell, T., Ricci, F., Rich, B., Riebesell, J., Rignanese, G.-M., Rosen, A., Scheffler, M., Schmidt, J., Shen, J.-X., Sobolev, A., Sundararaman, R., Tezak, C., Trinquet, V., Varley, J. Vigil-Fowler, D., Wang, D., Waroquiers, D., Wen, M.. Yang, H., Zheng, H., Zheng, J., Zhu, Z., and Jain, A. Atomate2: Modular Workflows for Materials Science. ChemRxiv, 2025. https: /cemrxivv http://ngageeehherxr/ree/ee http:e////aaattcleeeeeaalaleaa 678e7 765l (edeeaddd333999557563333
Jiao, R., Huang, W., Lin, P., Han, J., Chen, P., Lu, Y., and Liu, Y. Crystal structure prediction by joint equivariant diffusion. Advances in Neural Information Processing Systems, 36, 2024a.
Jiao, R., Huang, W., Liu, Y., Zhao, D., and Liu, Y. Space group constrained crystal generation. arXiv preprint arXi::2402.03992 2024b.
Kantorovich, L. Quantum theory of the solid state: an introduction, volume 136. Springer Science & Business Media, 2004.
Klipfel, A., Frégier, Y., Sayede, A., and Bouraoui. Z. Uni- fied model for crystalline material generation. arXiv preprint mrXiv:2306.045112 2023.
Kresse, G. and Furthmüller, J. Efficient iterative schemes for ab initio total-energy calculations using a plave-wave basis set. Phys. Rev. B, 54:11169 11116691111111 11186, Oct 1996. doi: 10. 10.1105/PhysRev..4..1111 aps ort./oi/oool...ooo 1 1103/P 1w1.0/0y//e/........ 54 11169.
Goodall, R. E., Parackal, A. S., Faber, F. A., and Armiento, R. Wyckoff set regression for materials discovery. In Third Workshop on Machine Learning and the Physical Sciences (NeurIPS 2020), Vancouver, Canada., 2020.
URL https://ling
Kresse, G. and Joubert, D. From ultrasoft pseudopotentials to the projector augmented-wave method. Phys. Rev. B. 59:1758-1772 Jan 1999. doi: 10.1103/Physeev.. opw.or///////10..
Goodall, R. E.,, Parackal, A. S., Faber, F. A., Armiento, R.,
and Lee, A. A. Rapid discovery of stable materials by
coordinate- coarse graining. Science advances, 8(30): cabn4117 2022.
59.1758. URL https:::: 1103 1103/Phystie Pwwp//yyssseeeeeeeeeeeeeaaeereee 1758..
Levy, D., Panigrahi, S. S., Kaba, S.-O., Zhu, Q., Galkin, M., Miret, S., and Ravanbakhsh, S. SymmCD: Symmetry- Preserving Crystal Generation with Diffusion Models. In Al for Accelerated Materials Design-NeurIPS 2024, 2024.
Gruver, N., Sriram, A., Madotto, A., Wilson, A. G., Zitnick, C. L., and Ulissi, Z. Fine-tuned language models gen- erate stable inorganic materials text. arXiv preprint arXiv:2402.04377 043479 2024.
Hahn, T., Shmueli, U., and Arthur, J. W. International tables for erystallography, volume Reidel Dordrecht, 1983.
Lin, Y., Yan, K., Luo, Y., Liu, Y., Qian, X., and Ji, Effi- cient approximations of complete interatomic potentials for crystal property prediction. Proceedings of the 40-th International Conference on Machine Learning, 2023.
Hicks, D., Toher, C., Ford, D. C., Rose, F., Santo, C. D., Levy, O., Mehl, M. J., and Curtarolo, S. AFLOW- XtalFinder: a reliable choice to identify crystalline proto- types. npj Computational Materials, 7(1):00, 2021.
Luo, X., Wang, Z., Gao, P., Lv, J., Wang, Y., Chen, C., and Ma, Y. Deep learning generative model for crystal structure prediction. npj Computational Materials, 10(I): 254, 2024.
Jain, A. and Bligaard, T. Atomic-positionn independent descriptor for machine learning of material properties. Physical Review B, 98(21):214112, 2018
Malgrange, C., Ricolleau, C., and Schlenker, M. Symmetry and physical properties of crystals. Springer, 2014.
Jain, A., Ong. S. P., Hautier, G., Chen, W., Richards, W. D., Dacck, S., Cholia, S., Gunter, D., Skinner, D., Ceder, G., et al. Commentary: The Materials Project: A materials genome approach to accelerating materials innovation. APL materials, 1(1), 2013.
Miller, B. K., Chen, R. T., Sriram, A., and Wood, B. M. FlowMM: Generating Materials with Riemannian Flow Matching. ICML 2024; arXiv preprint 04..4.2406.04713, 2024.

---

Wyckoff Transformer: Generation of Symmetrie Crystals
Möller, J. J., Körner, W., Krugel, G., Urban, D. F., and Elsässer, C. Compositional optimization of hard- magnetic phases with machine-learning models. Acta Materialia, 153:53-61 61, 2018.
Wang, Y., Elhag, A. A., Jaitly, N., Susskind, J. M., and Bautista, M. Á. Swallowing the bitter pill: Simplified scalable conformer generation. In Forty-first Interna- tional Conference on Machine Learning, 2023.
Newnham, R. E. Thermal conductivity. In Prop erties of Materials: Anisotropy, Symmetry, Struc- ture. Oxford University Press, 11 2004. ISBN
Wang, Z., Chen, A., Tao, K., Han, Y., and Li, J. Matgpt: A vane of materials informatics from past, present, to future. Advanced Materials, 36(6))2306733, 2024.
9780198520757. doi: 10.1003//00///33115577777774
Wyckoff, R. W. G. The Analytical Expression of the Results of the Theory of Space- Space-groups, volume 318. Carnegie institution of Washington, 1922.
0020. URL hwwpttttt//ctc http://doooooo..o///// 10t////0..99////////// 9780198520757.003.
Nong, W., Zhu, R., and Hippalgaonkar, K. CrySPR: A Python interface for implementation of crys- tal structure pre-relaxation and prediction us- ing machine-learning interatomic potentials. ChemRxiv, 2024. doi: hutps://doi.or//1.226443 bhemrxiv-2024-r4wwa URL https://chemzxiv. wtt.eegggee/cemmmm///aaiiiioiiee 56b308550103479655..o
Xie, T. and Grossman, J. C. Crystal graph convolutional neural networks for accurate and interpretable predic- tion of material properties. Physical review letters, 120 (14):143301, 2018.
Xie, T., Fu, X., Ganea, O.-F., Barzilay, R., and Jaakkola, T. Crystal diffusion variational autoencoder for peri- odic material generation. ICLR 2022, arXiv preprint arXhv:2110061999 2021.
Ong. S. P., Richards, W. D., Jain, A., Hautier, G., Kocher, M., Cholia, S., Gunter, D., Chevrier, V. L., Persson, K. A., and Ceder, G. Python materials genomics (pymatgen): A robust, open-source python library for materials analysis. Computational Materials Science, 68:3 64-314- 2013.
Yamazaki, S., Nong, W., Zhu, R., Novoselov, K. S., Ustyuzhanin, A., and Hippalga K. Multi-property directed generative design of inorganic materials through Wyckoff-augmented transfer learning. arXiv preprint arXiv:2503. 16784, 2025.
Perdew, J. P., Burke, K., and Ernzerhof, M. Generalized gradient approximation made simple. Phys. Rev. Lett., 77:3865- 3368,, Oct 1996. doi: 10.1 Ro.e. 77.3865. URL https:/ //iikk//nnn//..//// aps. otgg/oo///..oooo 1103 1103/PhysRevIett.om 77. 3865.
Yang, J. et al. An introduction to the theory of piezoelectric- ity, volume 9. Springer, 2005.
Yang, M., Cho. K., Merchant, A., Abbeel, P., Schuurmans, D., Mordatch, I., and Cubuk, E. D. Scalable diffusion for materials generation, 2023. URL org/ abs / 22112 09235.
Knapp- E. O., Pitera, J. W. Staar, P. W., Takeda, S., Laino, T., Sanders, D. P., Sexton, J., Smith, J. R., and Curioni, A. Accelerating materials discovery using artificial intelligence, high performance computing and robotics. npj Computational Materials, 8(1):84, 2022.
Yang, R., Yue, S., Quan, Y., and Liao, B. Crystal symme- try based selection rules for anharmonic phonon-phonon scattering from a group theory formalism. Phys. Rev. B, 103:184302, May 2021. doi: 10.1103/PhyseevB.ll 103. 184302. URL htwps: http://linnkkppp/e www.pp.......oo..ee...o.oo /ow//oo//////oooooooo/////oo/// /10.
Riebesell, J., Goodall, R. E., Jain, A., Benner, P., Persson, K. A., and Lee, A. A. Matbench discovery-an evalu- ation framework for machine learning crystal stability prediction. arXiv preprint arXiv:2308. 14920, 2023.
103. 184302.
Sinha, A., Jia, S., and Fung, V. Representation-spaae dif- fusion models for generating periodic materials. arXiv preprint arXiv:2408.0722
Zeni, C., Pinsler, R., Zügner, D., Fowler, A., Horton, M., Fu, X., Wang, Z., Shysheya, A., Crabbé, J., Ueda, S., Sordillo, R., Sun, L., Smith, J., Nguyen, B., Schulz, H., Lewis, S., Huang. C.-W., Lu, Z., Zhou, Y., Yang, H., Hao, H. Li, J., Yang. Li, W., Tomioka, R., and Xie, T. A generative model for inorganic materials design. Nature, 639(8055):624-633, Mar 2025. ISSN 1476-4687. doi: 10.108884413866-055-088888 URL https: / doi doi.orrg10.1038//ttt
2024.
Sommer, T., Willa, R., Schmalian, J. and Friederich, P. 3DSC-a dataset of superconductors including crystal structures. Scientific Data, 10(1):816, 2023.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
025 08628-88888-888 5.
Wang, A. Y.TT. Kauwe, S. K., Murdock. R. J., and Sparks, T. D. Compositionally restricted attention- based network for materials property predictions. Npj Computational Materials, 711):77, 2021.
Zhu, R., Nong, W., Yamazaki, S., and Hippalgaonkar, K. WyCryst: Wyckoff inorganic crystal gener- ator framework. Matter, 2024. ISSN 2590- 2385. doi: https://doi.or//10. 1016/ 2024.05.
12

---

Wyckoff Transformer: Generation of Symmetric Crystals
042. URL htt.:/ttpps//ww..eeeeeeeeeeree.t com/
13

---

Wyckoff Transformer: Generation of Symmetric Crystals
A. Wyckoff representation with fractional coordinates
crystal can be represented as space group, a set of WPs and chemical elements occupying them, the fractional coordinates of the WP degrees of freedom, and free lattice parameters. Such representation reduces the number of parameters by an order magnitude without information loss. For example, see Figure 6.
Group: I4/mmm (139)
Lattice: a a= 8.9013,
c= 5.1991,
90.0, B = 90.0, 7 90.0
Wyckoff sites:
Nd C 000000
0.0000
0.00001, WP [2a] Site [4/m2 /22 / mm
A1 ( 0.2788
0.5000
0.00001, WP [8j] Site [mm2
A1 e 0. 6511
0.0000
0.0000], WP [8i] Site [mm2
Cu
@ 0.2500
0.2500
0.2500], WP
[8f] Site
[.2
Figure 6. Wyckoff representation of Nd(Al;Cu)4 (mp-974729), variable parameters in bold. If represented point cloud, the structure has 13[atoms] x 3[coordinates] + 6[lattice] = 42 parameters; if represented using WPs, it has just 4 continuous parameters (WPs 8 and j each have free parameter, and the tetragonal lattice has two), and discrete parameters (space group number, and WPs for each atom).
B. WyFormer Description
Structure generation is shown in Figure and described in Algorithm 1; training in Algorithm 2; model itself in Algorithm 3.
C. Structure generation details
The process of obtaining crystal structures from Wyckoff representations using PyXtal (Fredericks et al., 2021) begins by specifying a space group and defining WPs. PyXtal allows users to input atomic species, stoichiometry, and symmetry preferences. Based on these parameters, Xtal generates a random crystal structure that respects the symmetry requirements of the space group. Once the initial structure is generated, we then perform energy relaxation using CHGNet. CHGNet neural network- based model designed to predict atomic forces and energies, significantly speeding up calculations that would traditionally require density functional theory (DFT). We repeat the process for six random initializations and pick the structure with the lowest energy. Energy distribution among the initializations is presented in Figure 8. Energy relaxation involves optimizing the atomic positions reach minimum energy configuration, which represents the most stable form of the material. CHGNet, trained on vast DFT datasets, can efficiently relax crystal structures by adjusting atomic positions to reduce the total energy. This approach ensures that the final structure is not only symmetrical but also physically realistic in terms of energy stability.
For the 2nd structure generation method, D is diffusion-basse crystal structure prediction model that focuses on generating purportedly stable crystal structures by sampling from an energy landscape in a physically consistent manner. DifiCSP++ generation also starts with PtXtal sampling.
D. Training computational requirements
Our tests were done on a single NVIDIA RTX 6000 Ada, 24 CPU cores and MP-20 dataset. The results are present in Figure5. We have also tried batched training for next token prediction training, but it just becomes slower without improved
Table 5. WyFormer training resources requirements on the MP-20 dataset.
Prediction target
Time
Batch size
Number of epochs
GPU memory, MiB
GPU load
Next token
11h
27136
900k
2000
50%
Formation energy
26m
1000
5.2k
1700
45%
Band gap
10m
1000
3k
1700
25%
quality. The reason might be that we choose a random known sequence length and part of the token, token permutation, and
14

---

Wyckoff Transformer: Generation of Symmetric Crystals
STEP 1
STEP 2
STEP 3
STEP 4
62
Br
-1
0
Model for structure relaxation
62
WyFormer
Se
-1
1
Dy
.m.
0
Figure 7. High-level flowchart of structure generation with WyFormer. In step I space group which is sampled from the training data distribution and as the initial token for WyFormer, in step WyFormer autoregressively generates tokens; step the Wyckoff representation is converted JSON and stored. Finally, in step 4, the Wyckoff representation is passed to DifCSSP+/CrySPR for structure generation as described Section 2.4.
15

---

end end if end if Wyckoff Transformer: Generation of Symmetric Crystals
Algorithm Generation of Crystal Structure using Wyckoff Transformer Model
1: Load Trained Wyckoff Transformer Model
2: Select or sample spuce-group
3: Initialize sequence [rnuc)
4: loopShouldEnd + false
5: current.iokenccom + length(sequence)
6: repeat
7: if' current_toker www.eooooeottttttteeeteeoooooote count 2 max max then
8: loopShouldEnd true
9: end if
10: if not loopShouldEnd then
11:
Predict element using Model(sequence)
12:
element= == STOP then
13:
loopShouldEnd t- true
else
15:
append element sequence current token count + current token count if current wwwttttooeeeeoooeee 2 max_length then loopShouldEnd t- true end end if end if
16:
17:
18:
19:
20:
if not loopShouldEnd then
23:
Predict site -ymmemyyyyr using Model(sequence)
24:
if site site.symmerry STOP [Less likely, but possible] then
25:
loopShouldEnd + true
26:
else
27:
append site symmmetry to sequence current token count +- current token count +
if current token_ttn 2 max_length then
30:
loopShouldEnd t- true
31:
end if end if
32:
33:
end if
34: if not loopShouldEnd then
Predict enumeration using Model(sequence)
if enumeration STOP [Less likely, but possible] then
37:
loopShouldEnd true else
38:
39:
append enumeration to sequence current_toker lowee__ouennnnni +- current waattttoooeeooooeeeeeee +
40:
41:
current token count 2 max length then loopShouldEnd +- true
42:
44:
45:
46:
until loopShouldEnd = true
47:
Convert generated sequence of (element, site_symmetry, enmmeration) tokens into list of {element, Wyckoff position letter) pairs for the chosen space group.
48: Use Xtal library with the Wyckoff representation to create an initial 3D crystal structure. Relax the structure MLIP (CHGNet, etc.), DiffCSP++, or DFT return the crystal structure.
16

---

Wyckoff Transformer: Generation of Symmetric Crystals
Algorithm 2 Wyckoff Transformer Training Algorithm
Require:
Training dataset Drain (crystal structures represented as sequences of tokens: space-group + list of [element,
site_symmetry. enumeration])
Initialize Wyckoff Transformer Model M with random weights
2: Initialize Optimizer O SGD)
3: for epoch to MaxEpochs do
4: for each crystal structure sequence S in Duain do
5: Sang S
6: Randomly shuffle the order of [element, site.symmetry, enumeration] tokens within Smgg {AJugmentation for
permutation invariance
7:
Randomly choose one of the equivalent Wyckoff representations for Sang
8:
poSeetet Randomly pick position in Sang to predict
9:
parget Randomly pick which part of the token (element, site_symmetry, enumeration) to predict at pos
10:
Replace pagett pogeet in Sang with a MASK token
11:
Also mask any subsequent parts of the token at urger remove the tokens after it
12:
Pped + (S {Forward Pass: Model predicts the masked part]
13:
Vacuual < the true value of pagtt at poStarget in the original Sang
14:
L CrossEntropyLoss Lossspyosssppsssss (pred, Victma) {Use multi-class variant if part element)
Calculate gradients V L based on the loss w.r.t M's parameters
16:
Update M's weights using (VV.)
17:
end for
18:
Optional: Validate model M's performance on separate validation dataset Dval periodically.
19:
Optional: Adjust learning rate or implement early stopping based on validation performance.
20:
end for
21:
return Wyckoff Transformer Model M.
500
400
Counts
300
mean 0.14 eV/atom
200
100
2
3
Standard deviation of energy (eV/atom)
Figure 8. Distribution of CHGNet-predicted energy standard deviation across six random ptXt initializations for 1000 Wyckoff representations.
17

---

Wyckoff Transformer: Generation of Symmetric Crystals
Algorithm 3 Model Forward Pass
1: Define element embedding layer (lookup table)
2: Define site symmeetry.mbeeddinggg layee (lookup table)
3: Define enumeration embedding layer (lookup table)
4: Define space-group-ae group_embedding Jayer (special encoding linear layer)
5: Define embedding mixer (linear layer) {Mixes concatenated [element, site aate_symmetr. enumeration] embeddings)
6: Define
cettrrmerr_encder_bbie (standard Transformer Encoder layers, NO positional encoding)
7: Define element prediction head (Fully-connecter Neural Network)
8: Define site citt__symmety pwwddctioonneeeeeeeee (Fully-connecte) Neural Network)
9: Define
head (Fully- ceuneccc Neural Network)
10:
11: Function Model_Forward
space group embedding + space group embedding layer(space group token)
13: Sycckof_eemedddinn Jlist []
14: for each token in sequence .ff.ffff./o.... wyckoff tokens do
15:
element_embl
deeement_embeddeee hawpeeeddddinnllyeteeeeeeeeeeeee syver(token.elemenn
16:
site_sym.emb + site_symmetry http:ettrreeeeeddiingleallleeeee law.aaaaecctnntttteeee..e.. wttp://enssstssssmmmti.ie. ssymmetry)
17:
enum_emb + enumeration enuueeratonneembeddigg... eew.eeeeiingaaaarrroeeeeee..
18:
concatenated.co .en + concaternateellemet.. emb, site_sym sym_em,, enum_emb)
mixed wyckoff emb <- embedding mixer(concattenated emb)
append mixed_ wyckodfwwcooofemmmt to sreckoff.embeeddnngs list
21: end for
22:
aaapattntttelppcegggooeeeeebeeee group.embedding. wyckoff.embeddingg eebeediiggs.lstt) 23: transformer output sequence t- transformer encoder block(full sequence embeddings)
concatenate(space
24: targetsscores + htwp://wwrrouuuutteuuueeeeeeeeee {the masked token is the last one}
25: Optional: target.embedding + concaaeenateettgeeeemmllldle
presence_vector)
if predicting.elemen then
predicted.probabilitee + element elementt.pedictton
28: else if predicting site. symmetry then
29:
t- site_symmetry www.ssmmmm--yddiiiieeee prw.iiiiiiooeeeeeeetttett.eooee.
30: else if predicting ennumeration then
31:
probabiiiir +- citmmeraaioonpredictii preelcction.eelllaggetemmmmaoaai
32: end if
33: return
18

---

Wyckoff Transformer: Generation of Symmetric Crystals
*enumerations* variant every batch; this can help avoid sharp minimum even when gradients are computed over the whole dataset. For comparison, training DiffCSP++ took 19.5 hours and 32000 MiB of GPU memory.
E. Inference speed
We conducted experiments on a machine with NVIDIA RTX 6000 Ada and 24 physical CPU cores. For baselines, we used source code, model hyperparameters and weights published by the authors. Assuming that the downstream costs of structure relaxation by DFT machine-leaming interaction potential are fixed, the inference cost per S.U.N. structure is present in the Figure6.
Table Inference time per S.U.N. structure. When GPU is running, also occupies a CPU core, which is taken into account. S.U.N. rates mcasured according to DFT stability estimation. CHGNet not used anywhere, for WyFormerRaw we sample structure with py Xtal and use it directly as an input for DFT.
Method
S.U.N.
GPU ms per
CPU per
(%)
structure
S.U.N.
structure
S.U.N.
WyFormerRaw
4.8
0.05
1.0
0.105
2.2
WyForDiffCSP++
12.8
840
5957
0.940
6.7
DiffCSP DiffCSP++
19.7
360
1731
0.360
1.73
7.6
1250
14705
1.35
15.9
Generating batch of Wyckoff representations takes 25 seconds, of which 5 seconds spent generating Py PyTorch tensors, and 20 seconds on decoding them into Python dictionaries containing Wyckoff representations. The latter part has not been optimized. In total, generation takes 0.05 GPU ms and 4.8 CPU ms per structure.
Obtaining unrelaxed structures using XtXtal takes 100 CPU ms structure.
Relaxing the structure the most expensive step. DiffCSP++ takes 14 minutes to produce 1000 structures 840 GPU ms structure. Note that modified the code to remove the inference of atom types, it runs faster compared the original version. CHGNet: 112 GPU s structure for MP-20 NVIDIA A40
Baselines
the authors don't report speed. On our machine, generating 10000 structures on GPU took I hour, 360 GPU ms per structure.
DiffCSP:
DiffCSP++: the authors don't report speed. On machine, generating 27135 structures took 6 hours, 1.25 CPU+GPU seconds per structure
CrystalFormer; Cao et al. (2024): "It takes 520 seconds to generate a batch size 13,000 crystal samples on a single A100 GPU", which translates to generation speed of 40 ms per sample.
FlowMM: The authors also do not publish inference time or model weights. They claim to be 3x faster than DiffCSP in terms of integration steps.
WyCryst; Zhu et al. (2024): "Lateet space sampling CPU second/2000 structures; Xtal generation 2 CPU core seconds/structure"
F. Plots
Figure 10 contains the number of unique elements per structure for MP-20 and novel generated structures.
G. Energy above hull calculations
For CHGNet. to obtain the Enuli- we firstly constructed the reference convex hull data by querying all 153235 structures from the Materials Project (MP); then, using the pymatggnn mallyss. phase diagram sub-module the Fron for
19

---

Wyckoff Transformer: Generation of Symmetric Crystals
(a) stable
(b) unstable
NS
Nd
KyAsAuSg, C2 (5)
CegBig, P6y/mom (193)
NdSbS, P4/nmm (129)
NdBig, Pm-3m (221)
LigMn,Og, P1 (1)
Ag
Nd
EuCoSig, 14mm (107) Ce R-3m (166)
DySeBr, Pnma (62) KyNaAgF Fm-3m (225)
Nd R-3c (167)
Figure 10 structures generated from WyFormerDifCSSP+ and presented without additional relaxation. labels contain the chemical formula, followed by the space group symbol in the short Mermann-Mauguin notation, and space group number: To the left 8 structures were randomly chosen from 15 stable structures as validated by DFT calculations, the right from unstable structures. The solid lines represent the primitive cell.
WyFormer
0.7
CrystalFormer
Normalized frequency
0.6
DiffCSP++
DiffCSP
0.5
FlowMM
WyCryst
0.4
MP-20
0.3
0.2
0.1
0.0
2
3
4
5
6+
Unique elements per stucture
Figure 10. Distribution of the number of unique elements per structure for MP-20 and novel generated structures.
20

---

(Ganose Wyckoff Transformer: Generation of Symmetric Crystals
1.00
Empirical cumulative density function
0.75
0.50
D
DiffCSP
WyFormer/CHGNet
0.25
WyFormerRaw
WyForDiffCSP++ WyCryst/CHGNet
0.00
0.0
0.1
0.2
0.3
Root mean squared deviation (A)
Figure The empirical cumulative density function (ECDF) root mean squared deviation (RMSD) DFT-unrelaxed structures from DFT-relaxed counterparts, RMSD is calculated using pymat gen. analysis. StructureMaa cher. r. in which only the RMSD of matched structure pairs reported. WyFormerCCHGN and WyCryst/CHGNet refer to models that use CHGNet-relaxed structures as inputs for DFT relaxations, while WyFormerRaw refers to WyFormer directly using pyxtal-generated unrelaxed structures (Section C).
each entry of generated structure was computed by referencing to the MP convex hull, Emall max (E, where AE; is the decomposition energy of any possible path for a structure decomposing into the reference convex hull. For the DFT data we used the MP convex hull -2--07-pp--m pkk.. gz distributed by matbench-di srabeenchddicooeryy (Riebesell al., 2023) was used the reference hull.
H. DFT details
We use DFT settings from Materials Project
bothoodology/materrtt meetthoodol hoyoollgy//aooo hww.ehhhoologg///lllllllaaaa.tt ctt.c/alcllcaicooccccccccooccall ansiioonddalllsee / oon-eetallssggaauu-alll culatitiiooo ons/
parameters and convergnce for structure relaxation and energy computation. In particular, we do GGA and GGA+U calculations with atomate2. vasp. flows mm. MRGGADoulLeRRllaasttttt....a al., 2025), which in pymatggn. wymatgen.io.vaspe.. wannp....v.sp..mm sets Wwets..PStat.ii.. MPStaticSet (Ong et al., 2013). Computations themselves VASP (Kresse & Furthmülller, 1996) version 5.4.4. with the plane-wave basis set (Kresse & Furthmüller, 1996). The electron-ion interaction is described by the projector augmented wave (PAW) pseudo-potentials (Kresse & Joubert, 1999). The exchange-correlation of valence electrons is treated with the generalized gradient approximation (GGA) (Perdew et al., 1996).
relies
pymatgen. www.. MMMRelaXeee and done with
(PBE) functional within the
For small fraction (1 15%) of the generated structures, the DFT failed to converge. We consider such structures be unstable for the purposes S.U.N. computation. The effect especially strong for CrystalFormer, as 13% of the structures it generates are structurally invalid, that is have atoms closer than 0.5 Á.
The 105-sample relaxations used structures as produced by the MI. models. For the 10 000-sample run we used CHGNet pre-relaxation to speed up the computations.
The raw total energies computed by DFT were corrected with wwt.keetaaaeeeeeeeeeoooaoaabbbba before putting into the PhaseDi agram to obtain the DFT Fhuil-
21

---

Wyckoff Transformer: Generation of Symmetrie Crystals
H.1. DFT setting difference between Materials Project and (Miller et al., 2024)
The settings used by Miller et al. (2024) are entirely the same as the ones used by the Materials Project, us, and Zeni et al. (2025). Both in the paper (section A.7.) and in the code /lowmm/blob// 6a 6a9ceec 6eba89f666aeeeeeee
inputs.py$Ise Miller et al. (2024) refer to using pymatgen. ic. vasp .MPRelaxSeee and doing so only once. The Materials Project procedure consists of relaxing all cell and atomic positions two times in consecutive runs, followed by a more precise static calculation https:/ //ccss materialsprojjct wwtpmmehtooooggg//////ooooooo/// materials how.olsosstthhoooogg...ooo /aalyy//aaaa cttcc/alccaaattaaaoeaaaaaaaaaaaa details.
To estimate the effect and make direct comparison, in Figurel report the S.U.N. obtained from structures relaxed with single Me axSet. in (brackets) and more accurate MPGGADoubl eRe axStat aessstaiiMaa cMaker result without.
I. Legacy metrics
For completeness sake, in Figure7 we present the metrics computed following the protocol set up by Xie et al. (2021). We would like to again reiterate issues with it. Firstly, the metrics are negatively correlated with structure novelty, the raison d' d'être for material generative models. Secondly, filtering by charge neutrality aka compositional validity means discarding viable structures.
Table 7. Method comparison according the protocol set up by Xic al. (2021). COV-P depends on the generated sample size, so compute it we uniformly subsample structures.
(a) Directly using structures produced by the methods, without additional relaxation. Note that CHGNet is an integral part of generating structures with Wyckoff Transformer and WyCryst, so it's used.
Method
Validity (%)+ f
Coverage (%)
Property EMD +
Struct.
Comp.
COV-R
COV-P
E
Nelem
WyckoffTransformer
99.60
81.40
98.77
95.94
0.39
0.078
0.081
DifCSst
99.80
81.40
99.51
95.81
0.36
0.083
0.079
DifICSP++
99.94
85.13
99.67
95.71
0.31
0.069
0.399
CrystalFormer
93.39
84.98
99.62
94.56
0.19
0.208
0.128
SymmCD
100.00
86.27
99.50
94.82
0.06
0.160
0.402
WyCryst
99.90
82.09
99.63
96.16
0.44
0.330
0.322
DiffCSP
100.00
83.20
99.82
96.84
0.35
0.095
0.347
FlowMM
96.87
83.11
99.73
95.59
0.12
0.073
0.094
(b) All structures have been relaxed with CHGNet. Note that for some models we didn't compute CHNet relaxation for all the structures, so the sample size is smaller.
Method
Validity (%) f
Coverage (%) 1
Property EMD E Netem
Struct.
Comp.
COV-R
COV-P
TyckoffTranssfrmer
99.60
81.40
98.77
95.94
0.39
0.078
0.081
99.70
81.40
99.26
95.85
0.33
0.070
0.078
DiffCSP++
100.00
85.80
99.42
95.48
0.13
0.036
0.453
CrystalFormer
89.92
84.88
99.87
95.45
0.19
0.139
0.119
SymmCD
95.49
85.86
99.19
96.05
0.32
0.095
0.392
WyCryst
99.90
82.09
99.63
96.16
0.44
0.330
0.322
DiffCSP
100.00
82.50
99.64
95.18
0.46
0.075
0.321
FlowMM
100.00
82.83
99.71
95.83
0.17
0.046
0.093
J. Template Novelty and Diversity
To asses the impact of template novelty on the diversity of the generated data can be assessed by evaluating the number of unique structures as the function of the total dataset size. We sampled 1 18k examples from the model with the lowest template novelty, DiffCSP++. and the highest, WyFormer. We present the number of unique samples as function of the
22

---

Wyckoff Transformer: Generation of Symmetric Crystals
generated sample size in Figure 12. Di uniqueness is clearly lower; due its high inference costs (see Appendix 6), were unable prepare larger sample.
le5
1.00
D
Total Number of Unique Structures
Wyckoffrransformer
1.0
Fraction Unique Structures
0.95
0.90
0.8
0.85
0.6
0.80
0.4
0.75
0.2
0.70
+
WyFormer
0.0
102
10a
104
10
0.0
0.2
0.4
0.6
0.8
1.0
Sample Size
Sample Size
le5
Figure 12. Fraction of unique structures and total number of unique structures function of sample size. For Wyckoff' Transformer used only the Wyckoff representations for uniqueness assessment, meaning that the uniqueness is likely to be slightly underestimated.
Evaluation on MP-20 binary & ternary
Comparison of WyFormer to WyCryst is presented in tables 8 and 9, Both models were trained on a subset of MP-20 training data containing only binary and ternary structures, and similarly selected subset of MP-20 testing dataset is used as the reference for property distributions. All generated structures were relaxed with CHGNet.
WyFormer outperforms WyCryst across the board. S.U.N. values are close, but this achieved by WyCryst sacrificing sample diversity and property similarity metrics, with about half of the generated structures already existing the training dataset.
Table Evaluation methods according to symmetry metrics. Aside from Template Novelty, metrics are computed only using novel structurally valid structures. Stability estimated with CHGNet.
Method
Template Novelty
P1 (%)
S.S.U.N. Space Group
(%)
ref 1.7
x²
(%)
WyFormer 25.63
1.43
0.224
37.9
WyCryst
18.51
4.79
0.815
35.2
Table 9. Evaluation of the methods according to validity and property distribution metrics. Following the reasoning in Section 3.1.2, we apply filtering by novelty and structural validity, and do not discard structures based on compositional validity. Validity is also computed only for novel structures. Stability estimated with CHGNet.
Method
Novelty
Validity (%) f
Coverage (%) 1
Property EMD
S.U.N.
(%) f
Struct.
Comp.
COV-R
COV-P
E
Netem
(%)
WyFormer
91.19
99.89
77.28
98.90
96.75
0.83
0.064
0.084
38,4
WyCryst
52.62
99.81
75.53
98.85
89.27
1.35
0.128
0.003
36.6
23

---

Wyckoff Transformer: Generation of Symmetric Crystals
L. Hyperparaaeeeess
L.1. Next token prediction MP-20
Model:
WP representation: Site symmetry + enumeration
Element embedding size: 16
Site symmetry embedding size: 16
Site enumerations embedding size: 8
Number of fully-connected layers:
Number of attention heads:
Dimension of feed-forward layers inside Encoder: 128
Dropout inside Encoder: 0.2
Number of Encoder layers: 3
Optimizer:
Loss function: Cross Entropy, multi-class for element, single-class for other token parts, no averaging
Batch size: 27136 (full MP-20 train)
Optimizer: SGD
Initial learning rate: 0.2
Scheduler: ReduceLROnPlateee attau
Scheduler patience: 2 104t epochs
Early stopping patience: 10 epochs of no improvement in validation loss
clip grad norm: max norm=2
L.2. Energy prediction MP-20
Model:
WP representation: Site symmetry + harmonics
Element embedding size: 32
Site symmetry embedding size: 64
Harmonics vector size: 12
Embedding dropout: 0.03
Number of fully-connected layers:
Fully-connected dropout:
Number of attention heads:
Dimension of feed-forward layers inside Encoder: 128
24

---

Wyckoff Transformer: Generation of Symmetric Crystals
Dropout inside Encoder: 0.1
Number of Encoder layers: 4
Optimizer:
Loss function: Mean squared error (MSE), averaged over batch
Batch size: 1000
Optimizer: Adam
Initial learning rate: 0.001
Scheduler: POntacel
Scheduler patience: 200 epochs
Scheduler factor: 0.5
Early stopping patience: 10 epochs of no improvement in validation loss
clip grad norm: max norm=2
L.3. Band gap prediction MP-20
Model:
WP representation: Site symmetry + harmonics
Element embedding size: 32
Site symmetry embedding size: 64
Harmonics vector size: 12
Embedding dropout: 0.05
Number of fully-connected layers:
Fully-connected dropout: 0.03
Number of attention heads:
Dimension of feed-forward layers inside Encoder: 128
Dropout inside Encoder: 0.2
Number Encoder layers:
Optimizer:
Loss function: Mean squared error (MSE), averaged over batch
Batch size: 1000
Optimizer: Adam
Initial learning rate: 0.001
Scheduler: RodlaeaL
Scheduler patience: 200 epochs
Scheduler factor: 0.5
Early stopping patience: 102 epochs of no improvement in validation loss
clip grad norm: max norm=2
25

---

Wyckoff Transformer: Generation of Symmetric Crystals
M. Fine-tuning LLM with Wyckoff representation
To challenge Wyckoff Transformer's architecture, we compared with pre-trained language models that were used in vanilla mode as well as after fine-tuning, essentially combining approach by Gruver et al. (2024) with Wyckoff representation. We explored two different textual representations of crystals corresponding a given space group:
Naive, which contains the specifications of atoms at particular symmetry groups encoded by Wyckoff' symmetry la- bels: Na at a, Na at a, Na at Mn at a, Co at a, Ni at o at a, o at 0 at a, O at a, O at a, at a
Augmented, which contains the specifications of atom types with its" symmetries and site enumerations: Na O, Na 8 e 0, Na
8
O, Mn m o, m O, where the set of valid
C m o m e C, Ni
0, o
m D, oeeme D, O
0, O
O, o
symmetries is: (22.22',
' 44/mmm'
'1' 1-3.. 6mm "m-3m' '2' 33m" 'm',
-6mm2m', 4mm', '3'2" 3222'
.2mm 1-1 .mm".. .m.m', 'm.2m' 3.3m".
3m', 'm2m.', 2mm' 3-32/m.²
22
.2.',
....' 22/m', 4-3m', 4/mm.m'.
'...', 12/m2/m.', '23.' 222'
m.
mm.
r-3.', 'm-3.', 13.', 4/m.."
, 1.-3m', 2m. 3-32/m/ mm 42m
m. mm
14.." 'm.m2', 442'', 132.' '2..
16222m2', 3m. -3.3 mmm.
222.
mm2 .', 4m2
1 2 22mm mm2'
-3m2/m² 1-4m 2', 2mm 3
- 42 ml
...2//m 4m. m²
1 1-4 1-4.. 6/ m/mm2 m
'22m' 'm..', mm mmm.
mmm 32",
'm'. '6..''
We fine-tuned the OpenAl chattoP--40-mnn-22222----------- with the vanilla OpenAl
model using different representations and compared 2024-08-2---- model. For each of the cases prompt looked like: Provide example a material for spacegroup number The table below contains details of the model training:
Model
Base Model
Representation
Hyperparameters
Training
Inference Time
Number of Parameters
Time
WyLLM- vanilla
gpt-4o-
Naive
74m
2 200B
2024-08-06
WyLLM-
gpt-4o-
Naive
epochs: 1, batch: 24, learn- 51m ing rate multiplier: 1.8
51m
8B
naive
mini-2024-
07-18
WyLLM-
gpt-4o-
Site Symmetry
epochs: 1, batch: 24, learn- 95m ing rate multiplier: 1.8
37m
~ 8B
site-
mini-2024-
symmetry
07-18
Table 10. Comparison of different models and their characteristics. Number of parameters is known exactly and is taken from public sources as an approximate estimation. For reference, WyFormer has 150k parameters.
Both training and inference times were measured using batch job execution on OpenAl's cloud. The fine-tuned model returned JSON string that was easy to parse, while the vanilla model required additional parsing of its output.
Comparison the WyFormer to WyLLM is present in Figurel1. 1. When fine-tuned, an LLM using Wyckoff representations shows similar performance to WyFormer at a much greater computational cost. Using site symmetries instead of Wyckoff letters doesn't 't unequivocally increase the LLM performance, possible explanation that since this representation our original proposition, the LLM is less able to take advantage of pre-training that contained letter-based Wyckoff representations. Without fine-tuning, the majority of LLM outputs are formally invalid, and the distribution of the valid ones doesn't match MP-20.
N. Ablation study: letters vs site symmetries
To evaluate the effect of using a representation based on site symmetry, as opposed on Wyckoff letters, we trained a WyFormer model with the same hyperparameters, but using a Wyckoff letters, and not site symmetry + enumeration representation. The baseer variant underperforms, as show Figure12.
26

---

Wyckoff Transformer: Generation of Symmetric Crystals
Table 1.. Comparison for WyFormer to different variant of WyLLM. All structures have been relaxed with DiffCSP+++ Sample size is 1000 structures per model. The metrics deseribed in Scction 3.1.2. nan is placed where the generated structures contained a rare element that crashed the property computation code. Wyckoff Validity refers the percentage of the generated outputs that are valid Wyckoff representations. Aside from LLM-specific problems, such as non-existent elements, a Wyckoff representation can be invalid if it places several atoms Wyckoff position without degrees of freedom, refers Wyckoff positions that do not exist in the space group. Stability computed with DFT.
Method
Novelty (%)
Validity (%) f
Coverage (%) f
Property EMD +
Struct.
Comp.
COV-R
COV-P
P
E
Nelem
WyFormer
89.50
99.66
80.34
99.22
96.79
0.67
0.050
0.098
WyLLM-naive
94.67
99.79
82.89
98.72
94.97
0.39
0.067
0.015
WyLLM-vanilla
95.59
99.82
88.75
94.46
59.67
2.23
0.234
0.253
WyLLMM-ite-
89.58
99.89
83.89
99.44
96.32
0.29
nan
0.039
Method
Wyckoff Validity
Novel Unique Templates (ff)
P1 (%)
Space Group x²
S.U.N.
S.S.U.N.
(%)
ref 1.7
%
t
WyFormer
97.8
186
1.46
0.212
22.2
21.3
M-L.ie
94.9
237
1.38
0.167
11.7
11.7
WyLLM-vvnnlla
28.7
87
2.03
0.621
WyLLMesiit-yymmetry
89.6
191
2.24
0.158
Method/Metric
Novel Unique Templates (#)
P1 (%)
Space Group x²
S.U.N.
S.S.U.N.
ref = 1.7
% 1
% f
WyFormerDiffCSPPt Wyformer--eterr--fffffSS-+
186
1.46
0.21
22.2
21.1
250
1.16
0.21
16.0
16.0
Table 12. WyFormer using Wyckoff letters (WyFommerr-eteess-iifCCSS--- vs WyFormer using site symmetry--euumeration (WyFormerDif-
ICSP+))
27

---

Wyckoff Transformer: Generation of Symmetric Crystals
O. Performance analysis of encoding WPs with spherical harmonics
To assess the impact of spherical harmonics we compare the performance of models with the same set of hyperparameters for the property prediction task on MP-20, leaving generative performance comparison for the future work. The results are presented in Figure13, hyperparameters in Figure14.
Table 13. Performance of WyFormer with different representation. The values are slightly different from Figure3, as there we have tuned hyperparameters.
Representation
Energy MAE, meV
Band Gap MAE, meV
Site symmetry only
31.7
247.8
Wyckoff letter
30.5
234.0
Site symmetry Enumeration
30.7
244.1
Site symmetry & Harmonics
29.7
238.7
Parameter
Value
Element embedding size Wyckoff letter embedding size Site symmetry embedding size
16
27
16
Site enumerations embedding size
7
Harmonic vector length
12
Batch size
500
Number of fully-connected layers
3
Number of attention heads
Dimension of ferd-forward layers inside Encoder
128
Dropout inside Encoder
0.2
Number of Encoder layers
Table 14. Hyperparameters used in the ablation study.
P. Sampling harmonic-encoded WPs
WP harmonic representation a real-valued vector. But for each space group it can only take up to possible values, so learning the full distribution of such vectors is not necessary. We tried the following procedure:
1. Take the harmonic representations of all the WPs in all space group
2. Use K-means clustering to find cluster centers.
3. Separately for each space group, assign harmonic labels to each emumeration:
(a) Compute the Euclidean distances between all cluster centers and all WPs in the SG
(b) Choose the smallest distance. Assign the WP to the corresponding cluster, remove WP and the cluster center from consideration.
(c) Repeat until all WPs are assigned
This way all we obtain discrete prediction target with one-to-one mapping with enumerations, but where physically-simibe values are grouped together. Experimentally, however, this modification reduces performance. When predicting spherical harmonics clusters, S.U.N. based on 1k CHGNet-relaxed structures was 34.0% as compared to 36.6% for enumerations-bant model; S.U. U.N. based on 105 DFT structures S.U.N. was 19.1 1% vs 22.2%.
28

---

Wyckoff Transformer: Generation of Symmetric Crystals
Q. Superconductor critical temperature prediction
used WyFormer to predict the critical temperature superconductors on 3DSC dataset (Sommer et al., 2023); obtained MLSE 0.81
R. Token analysis
R.1. WyFomer tokens
Tokens are formed from three parts: (element, site symnetry, cnumeration) for example: (O, .m., 0). Considering all choices of space group Euclidean normalizer, there are 10904 unique tokens in MP-20. The distribution for MP-20 is present Figure 13; for MPTS-52 in Figure 14.
0, 0
Li 1,
S, 3m. 1
3m.,
Te, 3m., 1
Ga, m...
Fe, 0,
O m. 0
o,
O, mm2,
Se, 3m.,
Se, 3m., 0
Te, 3m.,
H, 1,
0, 2mm.
So, m,
Mn.
To, 3m.,
O, m.
3m.,
Se.
0,000
0.025
0,050
0.075
0.100
0.125
0.150
0.175
Token frequency MP-20 for top tokens
Token frequency distribution in MP-20 for 10901 tokens with frequency < 0.(
104
103
Number of tokens
102
10
100
0.000
0.001
0.003 Token frequency in MP-20
0.002
0.004
0.005
Figure 13. Distribution of tokens in MP-20
29

---

Wyckoff Transformer: Generation of Symmetric Crystals
m...
0
C.1.0
O,
m.
Na, 0,
Cd, 3m,
m, 0
1, 0
P,
3m3,, 0
1,10
K.1.0
As, 1.
1,
1,1,0
1.3m, 2
1,
1,
Cd, 3m.
Cu. 1,
Token freq uequy MPTS-52 for tokens
Token frequency distribution in MPTS-52 for 11066 tokens with frequency
104
103
Number of tokens
102
10
100
0.000
0.002
0.004
0.006
0.008
0.010
Token frequency in MPTS-52
Figure 14. Distribution of tokens in MPTS-52
30

---

Wyckoff Transformer: Generation of Symmetric Crystals
R.2. Template tokens
In this section, we consider different token structure (space group number, site symmetry. enumeration] which will call template token. It does not correspond to token structure inside WyFormer, but the analysis of such tokens is interesting from the template novelty point of view. Considering all choices of space group Euclidean normalizer, there are 1047 unique template tokens in MP-20. The distribution is plotted in Figure 15. Wyckoff Transformer generates templates tokens not present in the training and validation datasets. For sample size of 9046 produced 20 new template tokens; for comparison, the similarly-sized test dataset contains 21 new template tokens.
1,
1, 0
62, .m.,
8. m, 0
216, -43m, 1
5, 0
47, mm2, 3
m, m.
14, 1,
71, mm2,
71, m...
216, 43m,
47, mm2,
225, 4mn.,
225, m-3m,
156, 3m..
166,
m2m,
47, mm2,
12, 1, 4mm,
225, m-3m.
156, 3m.,
mm2.
216, -43m,
225, -43m,
156, 3m,, 2
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Template token frequency in MP-20 for top 30 tokens
Template token frequency distribution in MP-20 for 1033 tokens with frequency 0.01
10?
Number of tokens
10
100
0.000
0.002
0.004
0.006
0.008
0.010
Token frequency MP-20
Figure 15. Distribution of template tokens in MP-20
31

---

Wyckoff Transformer: Generation of Symmetric Crystals
S. Comparison of enumerations for full Space groups
Figure 16.
Wyckoff Positions: Groups 3 & 84 (P4./m) with Enumeration
Wyckoff Letter
Group 3 (P2)
Group 84 (P4./m)
Mult.
Site Sym.
Enum.
Coordinates
Mult.
Site Sym.
Enum.
Coordinates
0
()))))
2
2/m..
0
(90,A)
2
2/m..
(1/2,112,01 (12
(122,y00)
2
2/m..
(9.12,0) (12,0012)
(1/2gx122)
2/m..
(2.1)2.120.11200)
(6222) (+4g-4)
4...
(0.0,149) (0.0,34)
2
4.
(1/2.112 (M), (122.12234)
4
2..
(0.0.),, (0.0.rr13)) (0.0.00 (00,2210))
4
2..
(122,122)) (12,1222+112), (122,1/2-)) 12..122)11
4
2.
(0.1/2,5), (1/20,2+1z)) (0,122
4
m..
(xyxlt, (+tyy R3 (epoctr2yy (y-e,122)
(uazz (x XAt (50224122) b. x2+122) (x.y 033, (eotue1102 (198,48122)
Figure 16. Different WPs can have common site symmetry. In this case, they differ in coordinates. The corresponding column indicates the triples of coordinates of all the included atoms, where x, and z are the unfixed parameters that change from to 1. Such collisions connected to symmetries and differ significantly between
could be resolved using letters. However, as in the table, letters space groups Therefore, we use an approach that numbers positions within a group of WPs with the same site symmetry. The ordering performed accordance with (Aroyo al., 2006).
32


Reviews:
[{"review_id": "ArLFVZiZYz", "reviewer": "Reviewer_YPYZ", "summary": "This paper introduces WyFormer, a Transformer-based architecture to generate Wyckoff sites of crystal structures. The key idea is the tokenization approach to convert the structure into a sequence of the space group and Wyckoff sites, and a permutation-invariant auto-regressive transformer for sequence generation. The proposed model can further generate the entire structures with PyXtal-based initializations and refinement via CHGNet or DiffCSP++. Results on MP-20 showcases the effectiveness of this method.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and for recognizing the strengths of our paper, particularly the **importance of generating high–symmetry crystal structures,** the **well–designed experiments, and the clarity of evidence** they produce!\n\nWe appreciate your detailed comments aimed at bettering the clarity and presentation of our work, and the willingness to improve the score upon enchantment the paper readability. As per the ICML guidelines, we are unable to update the submitted PDF at this stage. However, we want to assure you that we have implemented the proposed changes to enhance the readability and clarity of our paper. **The following represents the text of the key changes that will be incorporated into the camera-ready version:**\n\n1. **Enumerations**. Different WPs can share the site symmetry. *Enumerations* allow us to disambiguate them by enumerating such WPs from 0 up to 6 in the conventional order from [ITA](https://www.iucr.org/news/newsletter/volume-25/number-1/it-vol-a-6th-ed) (one that’s used for assigning letters). Enumeration is separate for each site symmetry. **[Here](https://www.notion.so/Enumerations-1c775a35da3680efb760f6dcb7c03ab1?pvs=21) we have prepared a figure illustrating the concept.** The key advantage of using site symmetry plus *enumeration* over Wyckoff letters is that they allow us to compartmentalise the part of the token whose definition depends on the space group into the *enumeration,* leaving the universally-defined site symmetries to be learned by the model.\n2. **Content allocation.** We will restructure the introduction to be more concise and move some of the theoretical background on space groups and Wyckoff positions, as well as the detailed data representation, into a dedicated Appendix as you suggested. The content currently in Appendix C on spherical harmonics will be moved to this revised Section 2 to provide a more detailed explanation of the method in the main body of the paper. We have also prepared a [flowchart](https://www.notion.so/WyFormer-Flowcharts-1c775a35da368002a4caf977dd980296?pvs=21) and [pseudocode](https://www.notion.so/Pseudocode-1c875a35da36808289a8fcd3e43b56dc?pvs=21) to further explain WyFormer.\n3. **Figure 4** is corrected [here](https://www.notion.so/Figure-4-Tokens-1c775a35da368051b878c36c46c58d56?pvs=21).\n4. **Typos** fixed\n5. **Multihead attention.** The sentence \"The reason is that in multihead attention, different heads look at continuous blocks of the input vector\" refers to a potential limitation when applying multihead attention to the concatenated embeddings of different features (element, site symmetry, and enumeration).\n    1. Concatenating embeddings creates distinct, contiguous blocks of information in the input vector.\n    2. Standard multihead attention processes the input by splitting it into several independent heads. If these heads operate on continuous blocks corresponding to single feature types (e.g., only element embeddings), they might fail to learn relationships between different types of features (e.g., the relationship between an element and its site symmetry).\n    3. To address this, we apply a linear layer after concatenating the embeddings. This step helps to mix the features, ensuring that each attention head has access to information from all embedding types and can therefore learn cross-feature relationships.\n\n> Could the authors give more experimental evidence of the advantages of using the site symmetries and enumerations instead of Wyckoff letters and multiplicities?\n\nWe have conducted an experiment by training a variant of WyFormer that uses Wyckoff letters instead of site symmetry + enumeration, with the same hyperparameters, and generating 1k examples. We then used DiffCSP++ to obtain the structures, and computed DFT for 105 structures. While using letters results in a higher number of novel unique templates, crucially, **site symmetry achieves 2x S.U.N. and S.S.U.N. compared to letters**. We’ll add these results to the camera-ready version.\n\n| Method/Metric | Novel Uniques Templates (#) ↑ | P1 (%) ref = 1.7 | Space Group χ2 ↓ | DFT S.(S).U.N. (%) ↑ |\n| --- | --- | --- | --- | --- |\n| WyFormer-letters-DiffCSP++ | **250** | 1.16 | **0.21** | 6.7 (6.7) |\n| WyFormer-DiffCSP++ | 186 | **1.45** | **0.21** | **12.8 (12.8)** |\n\nA small remark: multiplicity mentioned by the reviewer can be used with both letters and site symmetries. We have tried including it in preliminary experiments, which didn’t lead to improvement, so we don’t use it.", "claims_and_evidence": "The claims in this paper are supported by clear evidence.", "methods_and_evaluation_criteria": "The main part of the proposed method is described vaguely. The reviewer finds that the concept of \"enumerations\" plays a crucial role in the tokenization process. Unlike the traditional encoding of Wyckoff positions using letters and multiplicities, the proposed method employs \"enumerations.\" The authors are suggested to provide a clearer explanation of the differences between these representations and the advantages of the enumeration-based approach.", "theoretical_claims": "The theoretical definitions and claims in this paper are correct.", "experimental_designs_or_analyses": "The experiments in this work are well-designed. Notably, Table 1 introduces symmetry-based metrics to evaluate the model’s symmetry-aware generation performance.", "relation_to_broader_scientific_literature": "Generating high-symmetry crystal structures is an important topic. Recently, DiffCSP++ [A] proposed a generation framework that conditions on space groups and Wyckoff sites. The proposed framework extends this approach by generating these conditions directly and introducing a two-stage generation process to sample high-symmetry crystal structures from scratch.\n\n[A] Jiao, Rui, et al. \"Space group constrained crystal generation.\" ICLR 2024.", "essential_references_not_discussed": "Most of the essential references in this field are already be discussed.", "other_strengths_and_weaknesses": "1. The allocation of content in this paper is a key issue. The introduction spends a significant amount of space discussing space groups and Wyckoff positions, while the method section (Section 2) is very brief, which hinders the reader's understanding of the proposed method. The reviewer suggests that the authors consider restructuring the introduction by moving the theoretical background and data representation into a preliminary section, or relocating some of the redundant content to the appendix. This would free up more space to provide a more detailed explanation of the method (for instance, the content in Appendix C would be more appropriate for the main body of the paper). The reviewer would be willing to improve the score if the authors enhance the readability of the paper.\n\n2. Figure 4 is somewhat ambiguous and may lead readers to mistakenly believe that element, site symmetry, and enumeration are integrated as a single input-output token. However, as described in lines 229-240, these three aspects are auto-regressive outputs. The reviewer suggests that the authors revise Figure 4 to reduce this ambiguity.", "other_comments_or_suggestions": "Some typos or confusing sentences could be found:\n\nLine 179 (Left): is it -> it is\n\nLine 247 (Left): fist -> first\n\nLine 165-166 (Right): \"The reason is that in multihead attention, different heads look at continuous blocks of the input vector.\" The meaning of this sentence is unclear. Could the authors clarify what is meant here?", "questions_for_authors": "1. Could the authors give more experimental evidence of the advantages of using the site symmetries and enumerations instead of Wyckoff letters and multiplicities?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QWmXngv8K6", "reviewer": "Reviewer_PsvD", "summary": "The paper proposes a transformer-based model called WyFormer to learn the representation of crystal structures considering their symmetry information. The model represents crystals as discrete tokens encoding space group, chemical elements, and Wyckoff positions rather than using 3D coordinates. Through their experiments, the authors demonstrate that WyFormer can generate novel materials with proper crystal symmetry, outperforming baselines on symmetry metrics while maintaining competitive performance in terms of diversity, efficiency and other metrics. Additionally, the Wyckoff representation can complement other crystal generation methods.\n\n## Update after rebuttal\nI believe the authors have presented convincing evidence and analysis to support their experimental results. With the complementary results added to the draft, it should be a solid contribution to the field.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback and recognition of our main claim that **WyFormer addresses  the limitation in existing generative models and achieves better symmetry in generated crystals with competitive stability**, and its support by **reasonably designed experiments**. We are especially encouraged by the recognition of integration of Wyckoff representation with diffusion models as a promising research direction for crystal structure generation.\n\nWe acknowledge the simplicity of ML techniques. The core concept—combining autoregressive generation of a discrete high-level representation with diffusion-based refinement—may hold potential, if modest, interest for the wider ML community, possibly extending to areas like music/video generation and robotic motion planning. Simplicity also facilitates easier understanding and implementation, quoting an ICML–[endorsed](https://icml.cc/Conferences/2025/ReviewerInstructions) [meme](https://researchonresearch.blog/wp-content/uploads/2024/10/slide9.jpeg).\n\n> How does WyFormer's performance scale with larger sample size, as currently very limited structures are sampled and reported in Table 1 and even fewer structures are sent for DFT evaluation?\n\nEach sampled structure is sampled independently, so the performance per se doesn’t change with sample size – **except for uniqueness.** In Appendix K we present the number of unique structures as a function of the sample size, highlighting the ability of WyFormer to generate a diverse set of structures.\n\n> Have you analyzed the discrepancy between crystal structures before and after relaxation? How much does the performance rely on relaxation?\n\n- For all symmetry–based methods the space group is the same before and after DFT for > 90% examples; 44% for FlowMM and 55% for DiffCSP\n- Figure 9 presents root mean squared deviation (RMSD) of DFT-unrelaxed structures from DFT-relaxed; for > 90% of examples RMSD < 0.2 Å\n- Table 7 presents the very similar proxy metric values for generated structures with and without CHGNet relaxation\n\nIn conclusion, the materials before and after relaxation are similar, and relaxation meaningfully affects performance for about 10% of examples.\n\n> The S.U.N. metric reported across methods is relatively low (<20%), which may not fully capture the uniqueness and diversity of crystal structures. Have you explored alternative evaluation metrics?\n\nGood question! Firstly low S.U.N. value is actually good in the sense that this metric is not saturated, as opposed to the ones proposed by [Xie et. al in 2021](https://arxiv.org/abs/2110.06197). We additionally explore the uniqueness and diversity in two ways:\n\n1. By counting novel unique templates. This captures physically meaningful sample diversity: if two materials have different templates, their physical properties will definitely be different, while two structures which don’t match precisely can be similar.\n2. In Appendix K we present an evaluation on uniqueness of WyFormer-generated structures as a function of sample size.\n\n> You mentioned selecting one sample with the lowest energy out of 6 random initializations. What variance in energy did you observe across these initializations?\n\nThe mean std is 0.14 eV/atom, see the [figure](https://www.notion.so/CHGNet-energy-standard-deviation-1c775a35da3680fbac4de4cdaa1a3e40?pvs=21) for the full distribution. We’ll add it to the camera–ready version.\n\n> How are structures generated with WyLLM variations being relaxed? What considerations behind not adopting M3GNet used in the CrystalTextLLM paper?\n\nInitially we used M3GNet, but then switched to newer CHGNet. However, for WyLLM experiments we used DiffCSP++, as it showed better performance compared to CHGNet-based structure generation, see WyFormer vs WyForDiffCSP++ in tables 1 and 2.\n\n> have you observed differences in decomposition energy when using different relaxation methods\n\nWe used two relaxation and energy estimation methods: CHGNet and DFT. CHGNet inflates S.U.N., for example see the table below. Pearson correlation between structures’ stability determined by DFT and CHGNet was in range 0.3–0.4, so CHGNet is still useful.\n\n> what are the key tradeoffs between WyFormer's specialized architecture and fine-tuned LLMs based on your comparative experiments?\n\nThe main one is, of course, the computational cost: WyFormer has 150k parameters to gpt-4o-mini-2024-07-18 8B; WyFormer’s inference time is at least an order of magnitude faster. The comparison is also unfair as the LLM has seen much more data than WyFormer. *Despite all of these advantages, there is no clear gain from using an LLM.* In addition to the proxy metrics reported in the paper, we have computed DFT & CHGNet relaxation:\n\n|  | DFT S.(S).U.N. (%)↑ | CHGNet S.(S).U.N. (%) ↑ |\n| --- | --- | --- |\n| WyLLM-naive-DiffCSP++ | 9.0 (9.0) | 31.6 (30.9) |\n| WyForDiffCSP++ | **12.8 (12.8)** | **36.6 (35.9)** |\n\nWe’ll update the camera–ready version to include these results.", "claims_and_evidence": "The main claim is that WyFormer generates crystal structures with better symmetry properties than baseline methods while maintaining competitive stability. I believe this claim is supported by the experiments.", "methods_and_evaluation_criteria": "Yes. The authors evaluate the generated structures on metrics across elemental distribution, stability and space group distribution.", "theoretical_claims": "The claims are valid.", "experimental_designs_or_analyses": "Experiments are reasonably designed.", "relation_to_broader_scientific_literature": "The paper demonstrates the integration of Wyckoff representation with diffusion models as a future direction for crystal structure generation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The focus on symmetry addresses the limitation in existing generative models like CDVAE, DiffCSP and FlowMM.\n2. The authors provide supplementary experiments on fine-tuning an LLM like CrystalLLM, which offers some interesting insights.\n3. The experimental results with WyFormerDiffCSP++ suggest a future direction of combining discrete symmetry modeling with continuous coordinate refinement.\n\nWeakness:\n1. The formulation and machine learning techniques explored in this approach are relatively simple, with limited technical innovation from a machine learning perspective, offering modest insights for broader ML researchers.", "other_comments_or_suggestions": "1. I suggest clear definitions for some abbreviations before referencing, for example WyFormerDiffCSP++ and WyLLM.", "questions_for_authors": "1. How does WyFormer's performance scale with larger sample size, as currently very limited structures are sampled and reported in Table 1 and even fewer structures are sent for DFT evaluation?\n2. Have you analyzed the discrepancy between crystal structures before and after relaxation? How much does the performance rely on relaxation?\n3. The S.U.N. metric reported across methods is relatively low (<20%), which may not fully capture the uniqueness and diversity of crystal structures. Have you explored alternative evaluation metrics?\n4. You mentioned selecting one sample with the lowest energy out of 6 random initializations. What variance in energy did you observe across these initializations?\n5. How are structures generated with WyLLM variations being relaxed? What considerations behind not adopting M3GNet used in the CrystalTextLLM paper? Some additional thoughts, have you observed differences in decomposition energy when using different relaxation methods, and what are the key tradeoffs between WyFormer's specialized architecture and fine-tuned LLMs based on your comparative experiments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mKQwPqb6Sc", "reviewer": "Reviewer_CSRK", "summary": "The author uses Wyckoff positions as the basis for structure representation and develops a permutation-invariant autoregressive model based on the Transformer architecture, with the absence of positional encoding.\n\n## update after rebuttal\n\nMy concern has been addressed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and constructive review of our submission. We greatly appreciate the time and effort you dedicated to evaluating our work. We are particularly encouraged by your positive feedback on the key aspects of our paper: **reasonable experimental results leading to  clear and convincing evidence that WyFormer outperforms baseline methods in generating novel, symmetric, diverse materials conditioned on space group symmetry**.\n\nRegarding the weaknesses you pointed out, we sincerely regret that, according to the ICML guidelines, we are unable to update the submitted PDF at this stage. However, we want to assure you that we have taken your feedback to heart and will include the suggested improvements into the camera-ready version:\n\n1. We will **introduce mathematical formulas and [pseudocode](https://wyckoff.notion.site/Pseudocode-1c875a35da36808289a8fcd3e43b56dc) to describe the problem our method aims to solve and the method itself**\n2. [Updated flowcharts](https://www.notion.so/WyFormer-Flowcharts-1c775a35da368002a4caf977dd980296)\n\n>Does WyFormer only generate Wyckoff positions? …\n\nWyckoff positions and chemical elements\n>what advantages does WyFormer have compared to methods like DiffCSP++, which can generate the entire structure in one step while ensuring the accuracy of the Wyckoff positions? How does WyFormer compare to SymmCD, which also utilizes Wyckoff positions?\n\n**WyFormer just works better than 1–step methods & its inference speed is four orders of magnitude faster than that of diffusion models** (Appendix F). Similarly to the situation in LLMs, autoregressive generation provides a better inductive bias for discrete Wyckoff positions, while diffusion provides a better one for continuous coordinates\n1. DiffCSP++ *requires as input* a template with Wyckoff positions. WyFormer generates those templates. In the DifffCSP++ paper the authors use templates from the training dataset. As we show in Appendix K, the lack of template novelty limits the generated sample diversity. DiffCSP++ also has lower S.(S).U.N.: 7.6% vs 12.8%.\n2. CrystalFormer has low novelty, which means that the model has been overfitted, and the structures are similar to the training dataset. It also produces a sizable fraction of structurally invalid crystals.\n3. WyCryst suffers from abyssal novelty, stability and distribution similarity metrics.\n4. SymmCD is a [concurrent work](https://icml.cc/Conferences/2025/ReviewerInstructions), and was not included in the experiments in the original submission. We have done them and will include in the camera–ready version:\n\n|Method|Novel Uniques Templates (#) ↑|P1 (%) ref = 1.7|Space Group χ2 ↓|DFT S.(S).U.N. (%) ↑|CHGNet S.(S).U.N. (%) ↑|\n|-|-|-|-|-|-|\n|SymmCD |161|2.35|**0.24**|**12.1 (12.1)**|**34.1 (33.2)**|\n|WyForDiffCSP++ | **186**|**1.46**|**0.21**|**12.7 (12.7)**|**36.6 (35.9)**|\n\nWyFormer achieves higher number of novel uniques templates; higher S.(S).U.N., but this difference is not statistically significant.\n>Based on your description in the main text and your title, your focus seems to be on generation methods. Why, then, is your method also applied to prediction tasks? Moreover, your prediction method shows limited effectiveness, and the comparison methods in Table 3 are not the latest, as they do not include the baseline proposed in 2024.\n\nWe show that it is possible to do reasonable property prediction in the Wyckoff space, laying groundwork for property-conditioned generation.\n\nPredicting properties also serves to support our core assumption: crystal symmetries play a crucial role in the properties of matter, including the ones not included in MP–20, but crucial for material design. We don’t intend for a model using only symmetries to outperform models using whole structures (although we sometimes see this), just prove that symmetry information alone largely (but not completely) determines properties of the material.\n> In the experimental section, methods WyFormerRaw and WyForDiffCSP++ are mentioned. What is the difference between these two methods? …\n\nApologies for the confusion, we’ll add the definitions prominently to the camera-ready version. Different suffixes denote different ways to obtain the final structure from the Wyckoff representation:\n1. WyFormer uses CrySPR and CHGNet\n2. WyForDiffCSP++ uses DiffCSP++\n3. WyFormerRaw samples an unrelaxed structure with pyXtal\n\n> The performance of WyFormer–related methods in Table 2 does not seem to be particularly outstanding.\n\n*None of the methods are outstanding in Table 2*, because the **metrics in Table 2, except novelty, were proposed in [2021](https://arxiv.org/abs/2110.06197) and are mostly saturated,** the most important metrics are in Table 1.\n\n1. WyFormer is **within 1% of the best value in 4 out of 5** %-based metrics: Novelty, Structural Validity, COV-R and COV-P.\n2. In terms of EMD, out of 6, WyFormer ranks 3rd for $ρ$, tied 3rd–4th for $E$, and 1st for $N_\\text{elem}$", "claims_and_evidence": "Claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Proposed methods and evaluation criteria make sense for the problem or application at hand.", "theoretical_claims": "Proofs for theoretical claims are correct.", "experimental_designs_or_analyses": "I have reviewed the experimental designs, and they are reasonable.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\nEmpirically, the proposed model outperforms baseline methods in generating novel, symmetric, diverse materials conditioned on space group symmetry.\n\n\nWeaknesses:\n\nThe writing of the paper is quite poor. In section 2, the problem and method to be solved are not described using mathematical formulas but only explained with text. Additionally, section 2 lacks a flowchart of the proposed method. This makes it difficult for readers to follow and understand the content.", "other_comments_or_suggestions": "1. To facilitate reading, you could use mathematical formulas in section 2 to describe the problem your method aims to solve, as well as the method itself, rather than only describing it in text.\n\n2. You may consider adding a flowchart of the proposed method in section 2. If your Figure 6 is a flowchart, it would be better to move it to section 2.", "questions_for_authors": "1. In the experimental section, methods WyFormerRaw and WyForDiffCSP++ are mentioned. What is the difference between these two methods, and why are they not described in section 2? Are they related to DiffCSP++ mentioned in section 2.4?\n\n2. Does WyFormer only generate Wyckoff positions? How are the lattice matrix-related parameters generated? According to the description in section 2.4, \"Structure generation,\" is WyFormer used to generate the Wyckoff representation first, and then another method is used to generate the lattice matrix-related parameters? If so, what advantages does WyFormer have compared to methods like DiffCSP++, which can generate the entire structure in one step while ensuring the accuracy of the Wyckoff positions? How does WyFormer compare to SymmCD [1], which also utilizes Wyckoff positions?\n\n[1]Levy, Daniel, et al. \"SymmCD: Symmetry-Preserving Crystal Generation with Diffusion Models.\" AI for Accelerated Materials Design-NeurIPS 2024.\n\n3. Based on your description in the main text and your title, your focus seems to be on generation methods. Why, then, is your method also applied to prediction tasks? Moreover, your prediction method shows limited effectiveness, and the comparison methods in Table 3 are not the latest, as they do not include the baseline proposed in 2024.\n\n4. The performance of WyFormer-related methods in Table 2 does not seem to be particularly outstanding.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aLbB7d2A45", "reviewer": "Reviewer_QeGe", "summary": "This paper proposes WyFormer, a novel generative model for materials design that leverages Wyckoff positions to encode space group symmetry. The authors argue that symmetry rules are crucial for determining material properties, and that traditional material discovery approaches are limited by the vastness of the possible combinations of atoms. WyFormer aims to address this by generating stable materials with desired symmetries and properties using a permutation-invariant autoregressive model based on the Transformer architecture.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful feedback. We appreciate their recognition of the **novelty of WyFormer's symmetry–focused and efficient tokenization approach**, **comprehensive evaluation across various metrics and datasets**, and the **explainability** provided by the use of Wyckoff positions.\n\nWe will add the proposed references.\n\n**Larger structures**\n\nWe have trained WyFormer on MPTS-52, which contains up to 52 atoms per unit cell, without any issues. The values of the metrics are present in the last rows of Tables 1 and 2. As expected, on MPTS-52 WyFormer shows higher novelty and template novelty. We estimated S.S.U.N with CHGNet: 24.4% on MPTS-52, vs 35.2% on MP-20. This reflects the increased difficulty, and shows that WyFormer is still very much capable of generating stable structures in this setting.\n\n**We are glad you have asked about computational speed!**\n\nWyFormer **has the fastest inference** of all the models. Inference speed evaluation results are present in Appendix F Table 6; we did some more experiments is addition to it, will add them to the camera-ready version:\n\nGenerating a batch of $10^5$ Wyckoff representations takes 25 seconds, of which 5 seconds are spent generating pytorch tensors, and 20 seconds on decoding them into Python dictionaries containing Wyckoff representations. The latter part hasn’t been optimized. In total, generation takes **0.05 GPU ms and 4.8 CPU ms per structure**.\n\nObtaining unrelaxed structures using pyXtal takes 100 CPU ms / structure.\n\nRelaxing the structure is the most expensive step:\n\n1. DiffCSP++ takes 14 minutes to produce 1000 structures at 840 GPU ms / structure. Note that we modified the code to remove the inference of atom types, so it runs faster compared to the original version.\n2. CHGNet: 112 GPU s / structure for MP-20 on NVIDIA A40\n\n**Baselines**\n\nDiffCSP: the authors don’t report speed. On our machine, generating 10000 structures on GPU took 1 hour, at **360 GPU ms per structure**.\n\nDiffCSP++: the authors don’t report speed. On our machine, generating 27135 structures took 6 hours, at **1.25 CPU+GPU seconds per structure**\n\nCrystalFormer paper: “It takes 520 seconds to generate a batch size 13,000 crystal samples on a single A100 GPU”, which translates to a generation speed of **40 ms per sample**.\n\nFlowMM: The authors also do not publish inference time or model weights. They claim to be 3x faster than DiffCSP in terms of integration steps.\n\nWyCryst paper: “Latent space sampling 1 CPU second/2000 structures; PyXtal generation 2 CPU core seconds/structure”\n\n> Comparison with other Wyckoff-based models: While the paper compares WyFormer with a few existing models, a more comprehensive comparison with other Wyckoff-based models would be beneficial.\n\nWyCryst, CrystalFormer, and DiffCSP++ are Wyckoff-based models, and are included in our baselines. In response to CSRK, we have also made a comparison with a concurrent work, SymmCD:\n\n1. WyCryst underperforms on all the metrics\n2. DiffCSP++ has lower stability, and as we show in appendix K, the lack of template novelty limits the diversity\n3. CrystalFormer has low novelty – a sign of overfitting. It also produces a sizable fraction of a priori structurally invalid crystals.\n4. SymmCD has similar stability, but lower template novelty\n\n> How does WyFormer handle cases where multiple Wyckoff positions are possible for a given site symmetry and enumeration?\n\n1. In terms of group theory, the combination of space group, site symmetry and enumeration uniquely defines a WP. This is essentially the definition of *enumerations*, see the [figure](https://www.notion.so/Enumerations-1c775a35da3680efb760f6dcb7c03ab1?pvs=21) for more details.\n2. It is possible for different crystallographic orbits to have the same WP; if the WP contains a free parameter, the atoms can still occupy different locations in 3D space. WyFormer handles it naturally by repeating the site symmetry and enumeration.\n\n> What are the limitations of the spherical harmonics-based enumeration representation?\n\nThey are not invertible and can’t be directly used for structure generation. We have implemented the clustering algorithm from Appendix P, results pending.\n\n> How does WyFormer handle materials with mixed-valence properties?\n\nWyFormer handles materials with mixed-valence properties implicitly, rather than explicitly. Its tokenization scheme and symmetry-aware representation, which explicitly include space group symmetry and site symmetry, allow it to generate and learn from materials where atoms occupy non-equivalent crystallographic sites with differing local environments, often the cause of mixed valence, without directly specifying valence states \n\nAdding explicit oxidation states should be possible as an additional feature in WyFormer, a direction for future research. Thank you for the idea!", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "WyFormer builds upon the existing literature on crystal structure generation and material property prediction.", "essential_references_not_discussed": "some missing refs about the transformer framework for CSP task:\n1. Jin, Luozhijie, et al. \"Transformer-generated atomic embeddings to enhance prediction accuracy of crystal properties with machine learning.\" Nature Communications 16.1 (2025): 1210.\n2. Lin P, Chen P, Jiao R, et al. Equivariant Diffusion for Crystal Structure Prediction[C]//Forty-first International Conference on Machine Learning.\n3. Yan, Keqiang, et al. \"Periodic graph transformers for crystal material property prediction.\" Advances in Neural Information Processing Systems 35 (2022): 15066-15080.\n4. Yan, Keqiang, et al. \"Complete and efficient graph transformers for crystal material property prediction.\" arXiv preprint arXiv:2403.11857 (2024).", "other_strengths_and_weaknesses": "Strengths:\n\nNovelty of approach: WyFormer’s focus on symmetry and its efficient tokenization approach are unique and innovative.\nComprehensive evaluation: The authors provide a thorough evaluation of WyFormer’s performance using various metrics and datasets.\nExplainability: The use of Wyckoff positions and their encoding provides a transparent and interpretable representation of material structures.\n\nWeaknesses:\n\nLimited scalability: The current implementation of WyFormer is limited to structures with a maximum of 20 atoms per unit cell. Scalability to larger structures is a challenge that needs to be addressed in future work.\nComputational cost: The inference process for WyFormer can be computationally expensive, especially for larger structures. Optimizations and parallelization techniques are needed to improve efficiency.\nComparison with other Wyckoff-based models: While the paper compares WyFormer with a few existing models, a more comprehensive comparison with other Wyckoff-based models would be beneficial.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. How does WyFormer handle cases where multiple Wyckoff positions are possible for a given site symmetry and enumeration?\n2. What are the limitations of the spherical harmonics-based enumeration representation?\n3. How does WyFormer handle materials with mixed-valence properties?\n4. Are there any plans to extend WyFormer to handle larger structures?\n5. How can the computational cost of WyFormer be reduced?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No"}}]

Author rebuttal:

Meta-review: This paper presents a method named WyFormer, which is designed to generate crystal structures autoregressively, by generating the space group of the unit cell, the elements and the Wyckoff positions. There is no consensus among reviewers about the overall assessment and recommendation of the paper. On the one hand, the method seems solid and the results are convincing though not groundbreaking. On the other hand, the structure, clarity and writing of the paper could be largely improved. This has been noted by several reviewers and is also my own assessment. For these reasons, I am recommending a shy acceptance. I think the technical quality of the method deserves acceptance, but the presentation might not meet the minimum standards.
Decision: Accept (poster)