# LeMat-GenBench: A Unified Evaluation Framework for
Crystal Generative Models

Siddharth Betala, Samuel P Gleason, Ali Ramlaoui, Andy Xu, Georgia
Channing, Daniel Levy, Clémentine Fourrier, Nikita Kazeev, Chaitanya K
Joshi, Sékou-Oumar Kaba, et al.

► To cite this version:

Siddharth Betala, Samuel P Gleason, Ali Ramlaoui, Andy Xu, Georgia Channing, et al. LeMat-GenBench: A Unified Evaluation Framework for Crystal Generative Models. NeurIPS AI4Mat, Dec 2025, San Diego (CA), United States. 10.48550/arXiv.2512.04562. hal-05510249

HAL Id: hal-05510249

https://hal.science/hal-05510249v1

Submitted on 13 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# LEMAT-GENBENCH: A Unified Evaluation Framework for Crystal Generative Models

Siddharth Betala<fnref n="1" />,*, Samuel P. Gleason<fnref n="1" />, Ali Ramlaoui<fnref n="1" />, Andy Xu², Georgia Channing³, Daniel Levy⁴, Clementine Fourrier³, Nikita Kazeev⁵, Chaitanya K. Joshi⁶, Sekou-Oumar Kaba⁴, Felix Therrien⁴, Alex Hernandez-Garcia⁴, Rocío Mercado⁷, N. M. Anoop Krishnan⁸, Alexandre Duval<fnref n="1" />,∗

¹Entalpic, France, ²Harvey Mudd College, USA, ³Hugging Face, France, ⁴Mila, Canada, ⁵National University of Singapore, Singapore, ⁶University of Cambridge, UK, ⁷Chalmers University of Technology, Sweden, ⁸Indian Institute of Technology Delhi, India

*Corresponding authors

Generative machine learning (ML) models hold great promise for accelerating materials discovery through the inverse design of inorganic crystals, enabling an unprecedented exploration of chemical space. Yet, the lack of standardized evaluation frameworks makes it challenging to evaluate, compare, and further develop these ML models meaningfully. In this work, we introduce LEMAT-GENBENCH, a unified benchmark for generative models of crystalline materials, supported by a set of evaluation metrics designed to better inform model development and downstream applications. We release both an open-source evaluation suite and a public leaderboard on Hugging Face, and benchmark 12 recent generative models. Results reveal that an increase in stability leads to a decrease in novelty and diversity on average, with no model excelling across all dimensions. Altogether, LEMAT-GENBENCH establishes a reproducible and extensible foundation for fair model comparison and aims to guide the development of more reliable, discovery-oriented generative models for crystalline materials.

Code: https://github.com/LeMaterial/lemat-genbench

Leaderboard: https://huggingface.co/spaces/LeMaterial/LeMat-GenBench

Correspondence: siddharth.betala-ext@entalpic.ai, alexandre.duval@entalpic.ai

ENTALPIC

## 1 Introduction

Discovery of inorganic crystalline materials has traditionally relied on an Edisonian cycle of expert intuition and experimental validation [Schmidt et al., 2019], occasionally aided by quantum simulations such as density functional theory (DFT) [Kohn et al., 1996, Sholl and Steckel, 2009]. While such quantum simulations provide valuable insights into structure and stability, they remain computationally expensive and require a fully specified atomic configuration (crystal lattices†<fnref n="1" /> and atomic positions), information that is rarely available when proposing novel materials. Machine learning (ML) models [Deringer et al., 2019, Unke et al., 2021], particularly those based on geometric graph neural networks [Duval et al., 2023], offer fast proxies for DFT and enable scalable evaluation of candidate structures. Yet, similar to DFT, these models are limited to predicting the properties (such as energy and forces) for given atomic structure and are not designed to explore or propose new crystal geometries. This has motivated a growing wave of generative ML models for materials discovery [Mila AI4Science et al., 2023, Levy et al., 2025, Kazeev et al., 2025a, Zeni et al., 2025].

These models, ranging from variational autoencoders (VAEs) [Kingma and Welling, 2013] and diffusion models [Song et al., 2021] to GFlowNets [Bengio et al., 2023] and large language models (LLMs) [Brown et al., 2020], aim to learn the distribution of valid crystal structures and sample from it, guided by target properties when applicable. This inverse design paradigm promises to unlock previously inaccessible regions of chemical space and accelerate the discovery of practically relevant materials.

<footnote n="1">¹Terms marked with † are defined in Box 2 of the Supplementary Material.</footnote>
<figure><img src="image_2.png" /><figcaption>**Figure 1** LeMat-GenBench pipeline from raw model outputs to comprehensive evaluation. The framework begins by filtering generated crystals through rigorous validity checks. Valid structures are enriched with structural fingerprints, crystallographic descriptors, and MLIP-based energetic properties. Then, LeMat-GenBench computes a unified set of metrics capturing stability, novelty, uniqueness, diversity, distributional alignment, practical synthesizability considerations, and model efficiency. The resulting metric suite provides a standardized, leaderboard-ready assessment of generative models for inorganic crystal design.</figcaption></figure>

The rapid development of generative models for crystal structures has revealed a critical challenge: the absence of standardized evaluation protocols. Studies vary widely in how they assess stability, define novelty, or validate structures. For instance, researchers use different reference datasets, fingerprinting methods, energy estimators [Batatia et al., 2023, Unke et al., 2021], relaxation procedures, energy thresholds, and more, making the direct comparison of the performance of these models challenging. Without shared benchmarks designed for the fair comparison of generative models, it remains difficult to disentangle genuine model improvements from differences in evaluation design. While standardized benchmarks for predictive modeling of materials, such as Matbench [Dunn et al., 2020], have enabled progress in property prediction by providing systematic model comparison, no analogous established framework exists for the evaluation of generative models of crystal structures. This gap makes progress difficult to quantify, and leaves the community without shared reference for scientific advancement in data-driven materials design.

To this end, we introduce LEMAT-GENBENCH, a benchmarking framework aimed at standardizing the evaluation of generative models for inorganic crystal structures with the following key aspects:

* **Standardized evaluation protocol:** unified metrics suite centered on (Meta)Stable, Unique, Novel ((M.)S.U.N.) rate alongside validity, diversity, and efficiency;

* **Principled evaluation methodology:** carefully designed choices for stability estimation, validity filtering, and reference dataset construction that improve reliability and reduce common sources of error;

* **Open-source toolkit:** a public Python suite for reproducible metric computation and model evaluation;

* **Comprehensive benchmark analysis:** evaluation of 12 contemporary models, revealing clear trade-offs across metrics made available through a leaderboard to establish performances and provide a fillip to the development of better models.

By establishing shared protocols and rigorous evaluation standards, LEMAT-GENBENCH aims to enable more systematic, fair, and transparent model comparisons. The framework is not set in stone; it is designed to evolve with methodological advancements while supporting both research and practical applications in AI-driven materials.
# 2 Related Works

The field of generative modeling for inorganic crystals has rapidly grown, with an increasing diversity of proposed architectures, training strategies, representations, and target outputs. As the challenge of materials discovery is addressed by an expanding array of methodologies, this heterogeneity complicates the task of fair comparison. In what follows, we provide an overview of current modeling approaches (Section 2.1), followed by an analysis of existing evaluation practices and their limitations (Section 2.2). Together, these point to the need for standardized benchmarks and motivate the framework we introduce in this work.

## 2.1 Generative Models Overview

Generative modeling for inorganic crystals has emerged as a promising strategy for inverse design, enabling the proposal of candidate structures with desired stability, symmetry†, or functional properties. Most of these ML models are trained on large datasets of relaxed crystal structures to generate new candidates, either by sampling from a learned distribution of valid crystals, or conditioned on specific target properties. Over the past few years, an extensive range of architectural paradigms have been explored. We briefly discuss the prominent families of models below. A more comprehensive survey of the different approaches to generative modeling crystals can be found in Section A, with a taxonomy presented in Figure 6.

Among the earliest approaches were latent-variable models. **Variational AutoEncoders** (VAEs) [Noh et al., 2019, Hoffmann et al., 2019, Court et al., 2020] encode crystal representations into a continuous latent space to enable interpolation and sampling, but struggle with decoding to valid atomic configurations [Zhao et al., 2023]. **GAN-based** methods, on the other hand, attempt to generate crystal representations via adversarial training [Kim et al., 2020]. Despite being widely used years ago, their use has been drastically reduced due to training instabilities, limited diversity, and poor domain adaptation to 3D periodic systems [Zhao et al., 2021].

**Diffusion models** have become the dominant paradigm, learning to gradually transform Gaussian noise into 3D representations of periodic atomic structures [Xie et al., 2021, Jiao et al., 2023, Zeni et al., 2025]. The diffusion process is able to incorporate geometric and physical inductive biases, resulting in physically plausible crystals. However, they suffer from slow inference times due to the increased number of steps required during the denoising process. **Flow-based** models offer a related but computationally faster alternative than diffusion, and learn velocity fields that map between a base distribution and the data distribution [Miller et al., 2024]. Recent variants of these models have incorporated explicit symmetry conditioning [Jiao et al., 2024, Levy et al., 2025, Chang et al., 2025, Puny et al., 2025], textual conditioning from pretrained language models [Das et al., 2025, Park et al., 2025, Sriram et al., 2024], and joint training on non-crystal datasets [Joshi et al., 2025].

An alternative paradigm uses **sequential generation** strategies, where the task of crystal generation is decomposed into a series of discrete actions or tokens. Autoregressive transformers have been used to learn stepwise generation paths of crystals using tokenizations based on Wyckoff positions† [Cao et al., 2024, Kazeev et al., 2025a]. **Large language models** (LLMs) have recently been adapted to crystal generation by tokenizing CIF† formats. They can be fine-tuned for specific tasks, and due to their pretraining, can flexibly leverage textual prompts [Gruver et al., 2024, Xu et al., 2025a, Mishra et al., 2024] and incorporate multimodal conditioning [Johansen et al., 2025, Moro et al., 2025]. **Reinforcement learning** (RL) [Zamaraeva et al., 2023, Govindarajan et al., 2024] and **Generative Flow Networks** (GFlowNets) [Mila AI4Science et al., 2023, Cipcigan et al., 2024, Podina et al., 2025] decompose generation into stepwise actions guided by a reward. They differ from the previously described generative models: rather than training on a dataset of known crystals and sampling from this learned distribution, these methods are instead directly trained to generate samples that optimize a reward, such as crystal stability and validity, or some target property.

Taken together, these methods reflect an increasingly diverse and dynamic modeling landscape for crystal generation. In designing these methods, researchers have chosen different priorities and tradeoffs, balancing aspects such as physical plausibility, efficiency, diversity, and the ability to condition on properties. These inherent variability leaves the evaluation and comparison of crystal generative models as a major open challenge—due to the lack of standardized tasks, metrics, and data splits.
## 2.2 Evaluation Metrics and Benchmarking Efforts

Evaluating generative models for crystal structures is inherently more complex than supervised tasks such as property prediction, where objective metrics such as Root Mean Square Error (RMSE) or Mean Absolute Error (MAE) are well established. In generative settings, one must assess both structural plausibility (i.e., whether the generated atomic arrangement could exist as a real material), and functional relevance, often in the absence of direct ground-truth targets. As a result, current evaluation protocols vary considerably, hindering systematic comparisons across the literature.

Most studies focus on generating structures that are valid and stable. Validity is typically defined via heuristic filters such as charge neutrality (requiring that the sum of oxidation states across all atoms equals zero) and interatomic distances (being greater than a target physical value) [Xie et al., 2021]. Some approaches also measure validity by comparing the space group number distribution of generated structures against reference datasets through distributional distance metrics [Baird et al., 2024]. However, these checks vary across works, serve different purposes, either as hard pre-filters or as indicative sanity checks, and are not always consistent with real-world data. For example, a notable number of experimentally observed structures in Materials Project [Jain et al., 2013] entries fail charge neutrality tests due to phenomena such as partial oxidation, which can be challenging to quantify (see Section 5).

Thermodynamic stability is usually evaluated through formation energy† (*E*<sub>f</sub>), which measures the energy difference between the entire crystal and its constituent elements, or through energy above the convex hull† (*E*<sub>hull</sub>), which quantifies how far a structure lies above the lowest-energy combination of competing phases [Batatia et al., 2023]. Studies differ in how they define stability thresholds (some requiring structures on the hull, others allowing up to 0.1 eV/atom above it), in their relaxation strategies†, and in their choice of reference datasets. Besides, recent studies often favor Machine Learning Interatomic Potentials (MLIPs) to predict energies, since they scale better than DFT. However, MLIPs tend to incorporate biases related to the distribution of their training datasets [Fu et al., 2022]. Only a few works quantify this uncertainty or reconcile MLIP-based scores with DFT-based convex hulls.

Building upon stability, the S.U.N. framework (Stable, Unique, Novel) [Zeni et al., 2025] rewards models for generating thermodynamically viable structures (Stable) that are both distinct from one another (Unique) and absent from known databases (Novel). This metric has emerged as a composite diagnostic, but lacks a consensus implementation. Uniqueness and novelty rely on structure-based matching, using tools such as **StructureMatcher** [Ong et al., 2013] or BAWL [Siron et al., 2025] which compare atomic arrangements under symmetry operations to determine structure equivalence, whose results depend on the reference set, tolerance thresholds, and whether novelty is computed over all samples or only the stable ones. Most works rely on a single scheme without systematically comparing alternatives or handling disorder (e.g., partial occupancies or mixed species) [Siron et al., 2025] affecting the relevance of S.U.N scores.

Additional metrics like diversity and distribution similarity are frequently used. Distribution similarity, often measured by Maximum Mean Discrepancy (MMD)<fnref n="2" /> [Gretton et al., 2012] or Earth Mover's Distance (EMD)<fnref n="3" /> [Rubner et al., 2000], often rewards memorization rather than novelty, misaligned with discovery goals. On the other hand, diversity lacks a unified definition, implementation, and visualization across studies. Lastly, efficiency metrics like training time, memory usage, and inference cost are under-reported, though increasingly important as generation receives more attention and model sizes grow.

To address this fragmentation, we introduce LEMAT-GENBENCH: a unified, extensible benchmark that proposes a list of standardized metrics and tasks, enabling reproducible model comparison through public tools and a live leaderboard.

<footnote n="2">²MMD compares whether samples come from the same distribution by comparing their averaged representations</footnote>

<footnote n="3">³EMD computes the minimum displacement, under a chosen metric, needed to transform one distribution into another)</footnote>
# 3 Benchmark Methodology

## 3.1 Evaluation Tasks

Crystal generative modeling spans a range of practical use cases, from learning broad structural priors to targeting application-specific properties. To reflect these different levels of supervision, we conceptually distinguish three evaluation scenarios: (i) *unconditional generation*, which assesses a model’s ability to sample realistic crystalline materials without guidance; (ii) *conditional generation*, where models are steered toward property-aware design using inexpensive oracle^4^ evaluations; and (iii) *conditional generation under limited budget*, which reflects settings where each oracle call (e.g., DFT or lab experiment) is costly and sample efficiency becomes paramount. We consider that these three scenarios form the conceptual landscape in which generative models for materials discovery operate.

In this work, we focus exclusively on the unconditional generation setting. This choice reflects both the maturity of unconditional modeling techniques and the need for standardized metrics to evaluate general-purpose crystal generators before they are deployed or adapted for downstream, property-driven tasks. Establishing rigorous and comparable unconditional metrics is a prerequisite to meaningfully assessing conditional or constrained discovery workflows. We therefore introduce a unified evaluation suite centered on validity, stability, novelty, uniqueness, and diversity, along with standardized reference datasets and MLIP-based energy evaluation protocols. The two conditional settings remain important for real-world discovery, which will be pursued as future extensions of LEMAT-GENBENCH.

## 3.2 Unconditional Generation Metrics

To evaluate unconditional generation, a representative number of structures (e.g., 2,500) are sampled from the trained model and evaluated using the following metrics, whose formal definitions are provided in Section B.

**Validity.** We define a new validity metric, building on Xie et al. [2021]. It serves as a pre-filtering step to exclude malformed or nonphysical structures, catching common failure modes and reducing computational overhead. These checks include CIF† readability, minimum interatomic distances (> 0.7 Å), and density bounds (mass density of 0.01–25 g/cm³ and atomic density of 10⁻⁵–0.5 atoms/Å³). We also check for reasonable lattice parameters (*a*, *b*, *c* within 1–100 Å and 0 < *α*, *β*, *γ* < 180 degrees), a crystal symmetry adhering to a space group determinable by the SpacegroupAnalyzer module in *pymatgen* Jain et al. [2013], and charge neutrality (via oxidation state plausibility analysis, with a tolerance threshold). All results are aggregated into a single validity score: the percentage of structures passing the validity test. Failing any of the above metrics results in an invalid label, and these structures are discarded. While submitters may pre-filter their structures, the ensuing standardized criteria ensure consistency across evaluations. An analysis of our proposed validity metric can be found in Section 5.

**Stability.** Thermodynamic stability is a key proxy for real-world material feasibility. While formation energy has been widely used to assess stability, it introduces notable biases, such as favoring strongly bonded, electronegative compositions, and is therefore an unreliable proxy for stability. We thus utilize energy above the convex hull and adopt disjoint categories: structures with $E_{\text{hull}} \le 0$ are stable, those with $0 < E_{\text{hull}} \le 0.1 \text{ eV/atom}$ are metastable†. We use the term (*meta*)stable to refer to the union of both these sets, aligning with the inclusive threshold common in ML workflows.

A key methodological detail is how the convex hull is constructed. Mixing MLIP-predicted total energies with a DFT-based convex hull leads to systematic inconsistencies because MLIPs and DFT operate on different energy references depending on the MLIP’s training dataset. As demonstrated empirically (see Section C.3), such mixed-reference hulls can degrade stability prediction accuracy when de-referencing energies to make them comparable. For this reason, LEMAT-GENBENCH adopts a self-consistent MLIP-based convex hull, where each MLIP both evaluates candidate structures and defines the reference convex hull. This approach cancels potential-specific errors and yields more reliable $E_{\text{hull}}$ estimates, particularly under strict stability thresholds.

<footnote n="4">⁴An oracle is a function that returns a material property score. It can be an ML model, a DFT simulation, a lab experiment or any other evaluation method. Oracles vary in computational cost and accuracy.</footnote>
The convex hull is constructed from LeMat-Bulk [Siron et al., 2025], which contains approximately 5 million structures and provides the broadest coverage of competing phases relaxed under consistent DFT settings. We choose LeMat-Bulk as the reference because its breadth yields a tighter and a more realistic convex hull for stability assessment. This better reflects the needs of end-users seeking genuinely novel materials, rather than reproducing a narrow training distribution. The rationale for choosing LeMat-Bulk over datasets like MP-20 is expanded in Section 5. To enable scalable and open evaluation, we compute $E_{hull}$ using an ensemble of MLIPs: MACE-MP [Batatia et al., 2023], UMA [Wood et al., 2025], and Orb-v3 [Rhodes et al., 2025]. For each MLIP, we construct its own convex hull and report the mean and standard deviation of $E_{hull}$ across models, providing robustness and uncertainty estimates. To avoid expensive on-the-fly computation, we pre-compute MLIP energies for all LeMat-Bulk structures; at evaluation time, we simply filter to the relevant compositional subspace and construct the phase diagram for each generated candidate.

We additionally assess structural quality via a relaxation check: each structure is relaxed with every MLIP in the ensemble, and we compute the root mean square deviation (RMSD) between the initial and relaxed atomic positions (averaged over each MLIP). Low RMSD indicates that the submitted structure already lies near a local energy minimum. We report both *pre-relaxation* scores (computed on submitted structures) and *post-relaxation* scores (computed after MLIP relaxation); full results for both settings appear in Section C. For the public leaderboard, we favor pre-relaxation evaluation: users are encouraged to submit structures that have already been relaxed as part of their generation pipeline, ensuring that stability metrics reflect the model's actual outputs rather than post-hoc refinement, therefore improving comparability across submissions.

**Novelty** measures the fraction of generated crystals not found in a reference dataset, using structural fingerprints, to identify previously unseen materials. Again, we use LeMat-Bulk [Siron et al., 2025] as the reference database for novelty evaluation: the same 5 million structures that define competing phases for the convex hull also define what counts as “known” for novelty assessment. To assess if two 3D structures point to the same material, we apply the MatterGen-adapted *StructureMatcher* [Zeni et al., 2025], partly because it handles symmetry, disorder, and known edge cases reliably. To ensure scalability, we restrict comparisons to structures of matching composition. For workflows requiring comparison across large numbers of model outputs, we also implement the Short-BAWL fingerprint [Siron et al., 2025] as a computationally efficient alternative within the codebase. Note that novelty should be treated with caution. Structural difference does not always imply functional difference, and the boundary between truly novel and incrementally different materials is inherently fuzzy. Still, standardizing fingerprint methods, reference datasets, and thresholds is critical for enabling meaningful comparisons and tracking progress.

**Uniqueness** quantifies non-redundancy among generated structures using the same fingerprinting approach as novelty, yielding the percentage of unique structures. Importantly, we apply validity, uniqueness and novelty sequentially, i.e., we also compute novelty on generated structures that are valid and unique, which offers different information compared to doing it on the whole set of structures.

**(M.)S.U.N. rate.** We aggregate core evaluation criteria into a single standardized metric following the S.U.N. framework introduced by MatterGen [Zeni et al., 2025]: the S.U.N. rate measures the fraction of generated structures that are *stable*, *unique*, and *novel*, computed sequentially from the submitted set in a funnel-like manner. This serves as our primary benchmark for unconditional generation in LEMAT-GENBENCH, with each component precisely defined using fixed thresholds, reference datasets, and fingerprinting methods. To account for synthesizable but metastable materials, we also report the M.S.U.N. rate, relaxing stability to $0 < E_{hull} \le 0.1$ eV/atom. (M.)S.U.N. combines both and thus sets a practical upper bound on generative performance and provides a consistent, interpretable measure to compare models.

**Distribution metrics** capture the spread of generated samples across lattice parameters, space groups, elemental compositions, and structural descriptors. We compute the per-feature Shannon entropy and provide options for visualization tools in the LEMAT-GENBENCH codebase.

**Diversity metrics** compare the distribution of generated structures against reference datasets across key crystallographic and compositional attributes. We elaborate further on the diversity and distribution-based metrics offered as part of LEMAT-GENBENCH in Section C.

**Model efficiency.** Beyond output quality, we also report efficiency metrics related to training and inference. Specifically, authors must provide: (i) training compute in FLOPs and time (CPU/GPU days), (ii) inference
time to generate 2500 structures on a reference CPU/GPU (e.g., Nvidia A100)<fnref n="5" />, and (iii) peak memory usage during inference, together with number of model parameters. Reporting these values will inform model cost and scalability, and support analysis of tradeoffs between performance (e.g., (M.)S.U.N. rate) and deployment feasibility.

# 4 Towards an Open Benchmark Framework

## 4.1 Leaderboard Implementation

To converge towards a community-wide benchmarking framework, we adopt several concrete steps. Specifically, we provide a fully open-source implementation of the proposed evaluation metrics for unconditional evaluation, which can be used and updated by the community as the field evolves. Our contributions include: (i) LeMat-Bulk as the reference dataset for (M.)S.U.N, (ii) an ensemble of MLIPs for robust formation energy prediction, with uncertainty quantification, (iii) energy above hull calculations between an MLIP prediction and its corresponding convex hull, instead of a MLIP vs DFT-hull; (iv) enhanced validity criteria, (v) standardized diversity scores, (vi) resource efficiency reporting, (vii) pre-relaxation and post-relaxation metrics, (viii) RMSD check, (ix) a public leaderboard on Hugging Face to further facilitate fair model comparisons.

For the leaderboard, the submission process is as follows: (i) authors submit 2500 generated crystal structures; (ii) authors may also submit their packaged model, granting a compliance badge (optional); (iii) evaluation metrics are computed using our open reference implementation; (iv) results are displayed with multiple views to support comparison across tasks and generation scenarios. The code is available at https://github.com/LeMaterial/lemat-genbench.

This evaluation framework aims to provide clear, fair, and rigorous standards for evaluating generative models in crystal generation, while remaining flexible to evolving research needs and application contexts. Going further, we aim to extend this benchmark to the conditional generation tasks, to further bridge the gap between computational prediction and experimental realization. This infrastructure becomes particularly important as the field moves toward closed-loop discovery pipelines that integrate computational prediction, experimental validation, and synthesis planning.

## 4.2 Benchmarking Results and Discussion

We evaluate 12 generative models for crystalline materials on LEMAT-GENBENCH. The models considered are: ADiT [Joshi et al., 2025], Crystal-GFN [AI4Science et al., 2023], CrystalFormer [Cao et al., 2024], DiffCSP [Jiao et al., 2023], DiffCSP++ [Jiao et al., 2024], LLaMat2-CIF, LLaMat3-CIF [Mishra et al., 2024], MatterGen [Zeni et al., 2025], PLaID++ [Xu et al., 2025b], SymmCD [Levy et al., 2025], WyFormer [Kazeev et al., 2025a], and WyFormer-DFT [Kazeev et al., 2025a]. All models were trained on the MP-20 dataset. These span a broad range of architectures: diffusion models (DiffCSP, DiffCSP++, MatterGen, SymmCD), autoregressive transformers (CrystalFormer, WyFormer, WyFormer-DFT), LLM-based approaches (LLaMat2-CIF, LLaMat3-CIF), flow matching with VAE (ADiT), RL-guided models (PLaID++) and GFlowNets (Crystal-GFN). For each model, we evaluate 2500 generated structures obtained either directly from the authors or from public repositories [Kazeev et al., 2025a]. CrystalFormer is the only exception, for which only 1000 structures were available. A detailed listing of all data sources is provided in Table 3. The percentages of novel, unique, and stable structures are calculated over all submitted structures but only count valid structures, effectively setting an upper bound on achievable performance equal to the validity rate. This prevents inflated scores for models with low validity (e.g., novelty appearing high simply because the valid set is small) and ensures the reported values reflect the true frequency of valid-and-novel (or stable, unique, etc.) structures in the model's full output distribution.

To provide reference points for interpreting generative model performance, we establish baselines by directly sampling structures from existing materials databases. These baselines are not intended as fair comparisons, but rather to establish upper bounds on certain metrics (validity, stability) and lower bounds on others

<footnote n="5">⁵While inference cost is typically negligible compared to the cost of downstream validation (e.g., DFT or experiments), standardized reporting of these metrics provides important context for practical usability and future integration into closed-loop discovery pipelines.</footnote>
(novelty). They help contextualize what performance levels are achievable when simply reproducing known structures versus genuinely discovering new ones. We sample structures from MP-Exp (MP-20 entries marked as experimental by setting the 'Theoretical' indicator to false), AFLOW [Curtarolo et al., 2012], Alexandria [Schmidt et al., 2021, 2024], and OQMD [Kirklin et al., 2015, Saal et al., 2013]. For each database, we first randomly sample 10,000 structures without replacement, then draw 2,500 samples with replacement from this subset. This two-step procedure emulates the sampling behavior of generative models, which naturally produce duplicates when sampling from a learned distribution. Sampling with replacement ensures that uniqueness metrics remain meaningful—without it, uniqueness would trivially be 100%, unlike real generative models.

To ensure evaluations remain consistent, scalable, and physically meaningful, all metrics are computed through a simple sequential pipeline: we first apply a standardized validity check (CIF readability, density bounds, charge neutrality, minimum distances, symmetry consistency), and then compute stability, fingerprint-based, distributional, and diversity metrics on the valid subset of generated structures. This guarantees that all downstream quantities are computed on physically plausible samples and that heavy computations—such as MLIP relaxation or structural matching—remain tractable. Importantly, all evaluations are conducted on the submitted structures *without re-relaxation*. While this simplifies benchmarking and ensures reproducibility, it likely underestimates metrics such as S.U.N. and M.S.U.N. We encourage future submissions to include relaxed structures to better reflect thermodynamic viability. More detailed results can be found in Section 5 and Section C, where we report results of using MP-20 as reference set Section C.4 and of post-relaxation metrics (Table 9, Table 10, Table 15, Table 16).

<table><caption>**Table 1** Core model evaluation metrics calculated using MLIP ensemble with LeMat-Bulk as the reference set. Submissions are organized into three categories separated by horizontal lines: (1) models that incorporate relaxation as part of their generation pipeline (MatterGen, PLaid++, WyFormer, WyFormer-DFT), (2) other generative models, (3) samples from real-world datasets. Best values are shown in **bold**, second-best values are **underlined**. Arrows indicate whether higher (↑) or lower (↓) values are better. The energy-based metrics are computed over the full generated sets which pass the initial validity check, and the relatively high standard deviations reflect the stochasticity in the structural generation process rather than measurement error.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Valid% ↑</th><th rowspan="2">Unique% ↑</th><th rowspan="2">Novel% ↑</th><th colspan="3">Energy-based</th><th colspan="2">Stability</th><th colspan="2">Metastability</th></tr><tr><th>E<sub>f</sub> (Std) ↓</th><th>E<sub>hull</sub> (Std) ↓</th><th>RMSD (Std) ↓</th><th>Stable% ↑</th><th>SUN% ↑</th><th>Metastable% ↑</th><th>MSUN% ↑</th></tr></thead><tbody><tr><td>MatterGen</td><td>95.7</td><td><b>95.1</b></td><td><b>70.5</b></td><td>-0.70 ± 0.79</td><td>0.18 ± 0.18</td><td>0.39 ± 0.50</td><td>2.0</td><td>0.2 (6)</td><td>33.4</td><td><b>15.0</b></td></tr><tr><td>PLaid++</td><td><b>96.0</b></td><td>77.8</td><td>24.2</td><td>-0.50 ± 0.44</td><td><b>0.09 ± 0.16</b></td><td><b>0.13 ± 0.29</b></td><td><b>12.4</b></td><td><b>1.0 (26)</b></td><td><b>60.7</b></td><td>7.6</td></tr><tr><td>WyFormer</td><td>93.4</td><td>93.0</td><td><b>66.4</b></td><td>-0.43 ± 0.95</td><td>0.50 ± 0.51</td><td>0.81 ± 0.98</td><td>0.5</td><td>0.1 (3)</td><td>15.7</td><td>1.9</td></tr><tr><td>WyFormer-DFT</td><td>95.2</td><td><b>95.0</b></td><td><b>66.4</b></td><td>-0.67 ± 0.91</td><td>0.27 ± 0.36</td><td>0.42 ± 0.60</td><td><b>3.7</b></td><td><b>0.4 (9)</b></td><td>24.8</td><td>7.8</td></tr><tr><td>ADI</td><td>90.6</td><td>87.8</td><td>26.0</td><td>-0.73 ± 0.93</td><td>0.33 ± 0.45</td><td>0.38 ± 0.40</td><td>0.4</td><td>0.0 (0)</td><td><b>36.5</b></td><td>1.0</td></tr><tr><td>Crystal-GFN</td><td>51.7</td><td>51.7</td><td>51.7</td><td>-1.30 ± 2.63</td><td>2.09 ± 2.38</td><td>1.87 ± 0.97</td><td>0.0</td><td>0.0 (0)</td><td>0.0</td><td>0.0</td></tr><tr><td>Crystalformer</td><td>69.9</td><td>69.4</td><td>31.8</td><td>-0.17 ± 1.48</td><td>0.70 ± 1.33</td><td>0.66 ± 1.05</td><td>1.4</td><td>0.0 (0)</td><td>28.8</td><td>3.1</td></tr><tr><td>DiffCSP</td><td><b>95.7</b></td><td>94.8</td><td>66.2</td><td>-0.64 ± 0.96</td><td>0.28 ± 0.56</td><td>0.59 ± 0.63</td><td>2.3</td><td>0.1 (3)</td><td>29.8</td><td><b>8.5</b></td></tr><tr><td>DiffCSP++</td><td>95.3</td><td><b>95.1</b></td><td>62.0</td><td>-0.52 ± 0.87</td><td>0.41 ± 0.44</td><td>0.69 ± 0.82</td><td>1.0</td><td>0.2 (4)</td><td>26.4</td><td>5.0</td></tr><tr><td>LLaMat2-CIF</td><td>84.4</td><td>81.4</td><td>30.0</td><td>-0.47 ± 1.01</td><td>0.44 ± 0.71</td><td>0.54 ± 0.71</td><td>0.7</td><td>0.1 (3)</td><td>34.7</td><td>2.1</td></tr><tr><td>LLaMat3-CIF</td><td>15.4</td><td>15.2</td><td>10.5</td><td>0.74 ± 2.69</td><td>1.71 ± 2.48</td><td>1.01 ± 1.10</td><td>0.1</td><td>0.0 (0)</td><td>2.1</td><td>0.2</td></tr><tr><td>SymmCD</td><td>73.4</td><td>73.0</td><td>47.0</td><td>-0.02 ± 1.22</td><td>0.88 ± 1.07</td><td>0.87 ± 1.09</td><td>1.4</td><td>0.1 (2)</td><td>18.6</td><td>2.4</td></tr><tr><td>Alexandria</td><td>93.4</td><td>93.4</td><td>0.0</td><td>-0.18 ± 0.94</td><td>0.44 ± 0.61</td><td>0.14 ± 0.32</td><td>2.3</td><td>0.0 (0)</td><td>27.4</td><td>0.0</td></tr><tr><td>OQMD</td><td>96.8</td><td>96.4</td><td>0.0</td><td>-0.23 ± 0.88</td><td>0.39 ± 0.46</td><td>0.14 ± 0.28</td><td>5.3</td><td>0.0 (0)</td><td>29.1</td><td>0.0</td></tr><tr><td>AFLOW</td><td>91.4</td><td>91.4</td><td>21.5</td><td>-0.19 ± 0.86</td><td>0.35 ± 0.42</td><td>0.12 ± 0.44</td><td>9.3</td><td>0.0 (0)</td><td>30.4</td><td>0.7</td></tr><tr><td>MP-Exp</td><td>98.0</td><td>85.6</td><td>1.9</td><td>-1.02 ± 0.97</td><td>0.09 ± 0.45</td><td>0.20 ± 0.35</td><td>15.7</td><td>0.3 (8)</td><td>64.1</td><td>0.6</td></tr></tbody></table>

**Key takeaways.** Tables 1 and 2 show that no single model dominates across all evaluation dimensions. Instead, models occupy different regions of the trade-off space between thermodynamic stability, novelty, distribution fidelity and diversity, which reinforces the need for multi-faceted benchmarking in generative materials design. Validity is often pretty high across models (typically > 90%), indicating that generative approaches have mostly solved low-level structural correctness.

Among generative models, MatterGen attains the highest novelty fraction (70.5%) and the strongest M.S.U.N. rate (15.0%), indicating that it frequently discovers metastable structures that are also unique and novel relative to LeMat-Bulk. WyFormer-DFT offers a balanced trade-off profile, combining strong novelty and diversity with reasonable energy quality. Diffusion-based approaches demonstrate that continuous denoising processes can effectively explore regions beyond the training distribution. In contrast, Crystal-GFN and PLaid++ are clearly (over)optimised for formation energy and stability, respectively. The latter even shows the best stability metrics and S.U.N, but has substantially lower novelty (24.2%) and uniqueness (77.8%).
<figure><img src="image_3.png" /><figcaption>**Figure 2** Spider plots comparing generative models using LeMat-Bulk as reference. Left: (M)S.U.N. metrics measuring validity, uniqueness, novelty, (meta)stability, and the combined (M)S.U.N. score. Right: Quality metrics including energy above hull (E<sub>hull</sub>), structural relaxation (RMSD), diversity (average of elemental, space group, and size diversity normalized), JS divergence (Jensen-Shannon divergence measuring distributional similarity to reference). All metrics normalized to same scale where outer positions indicate better performance.</figcaption></figure>

than MatterGen and WyFormer-like models, suggesting a tighter focus on well-explored regions of the MP-20 distribution. Models which incorporate relaxation in their generation pipelines (MatterGen, PLaID++, WyFormer, WyFormer-DFT) achieve systematically lower RMSD values than purely unrelaxed approaches. We examine the interplay between relaxation strategy and benchmark performance in Section 5.3. Overall, these results show that generative models have substantial scope for improvement, as their S.U.N rate never exceeds 1%. More broadly, the benchmark reveals that no single model excels across all dimensions: strong stability often comes at the cost of novelty, and high exploration capacity does not guarantee thermodynamic feasibility. This underscores the value of multi-metric evaluation in capturing the diverse trade-offs inherent to materials discovery. A visual comparison of the different models can be found in Figure 2.

**Dataset baselines.** Because LeMat-Bulk unifies multiple major resources including Alexandria [Schmidt et al., 2021, 2024] and OQMD [Kirklin et al., 2015, Saal et al., 2013], baseline samples drawn from these databases provide interpretive anchors for generative model performance. As expected, dataset baselines deliver very relaxed structures with near-perfect validity and uniqueness, strong stability metrics but weak novelty, since these structures already appear in the LeMat-Bulk reference. Their distribution-matching scores (particularly for Alexandria, with the lowest JS and MMD) effectively define the reference behavior that generative models aim to approximate, as shown in Table 2. AFLOW [Curtarolo et al., 2012] provides a complementary baseline: because it does not contribute directly to LeMat-Bulk, it attains non-zero novelty (21.5%) alongside strong stability (9.3% stable, 30.4% metastable), illustrating the performance of a large, external real dataset under our benchmark. Its relatively high Herfindahl-Hirschman Index (HHI)<fnref n="6" /> scores reflect a relatively higher representation of lower abundance/higher cost elements, emphasizing that real-world materials collections can also be composed of elements which are not fully supply chain optimized. The set of real synthesized materials (MP-Exp) outperforms all generative models and computational databases on SGDiv and SiteDiv, suggesting that generative models still underexplore the diversity of crystallographic environments found in nature. Together, these baselines establish that strong performance approaching dataset levels on stability indicates realistic structure generation, while strong novelty performance signals genuine exploration beyond known materials.

<footnote n="6">⁶Quantifies supply risk concentration for materials by measuring the concentration of element production sources and reserves. More details on how it is computed in Section B</footnote>
**Diversity** metrics in Table 2 reveal additional aspects of model behavior. WyFormer achieves some of the highest elemental and site diversity (ElemDiv and SiteDiv), demonstrating that Wyckoff-based autoregressive architectures are particularly effective at exploring a wide range of compositions and symmetry environments. SymmCD and DiffCSP++ achieve strong space-group diversity (SGDiv), which is consistent with their explicit treatment of symmetry. Crystal-GFN reaches the highest diversity and the lowest HHI values among generative models, but at the cost of lower stability and validity metrics, highlighting that unconstrained chemical exploration can generate many structurally or thermodynamically implausible candidates. More broadly, we observe that models with very high diversity and novelty often incur larger JS, MMD, and FID values, confirming that aggressive exploration of chemical space naturally deviates from the reference training distribution. Additionally, higher novelty does not guarantee higher diversity. Mattergen illustrates this: novelty measures how combined structural and compositional attributes deviate from the reference dataset, not each attribute independently.

Taken together, these results show how **architectural choices** map to distinct performance profiles. LLM-based approaches with reinforcement learning (PLaID++) produce great stability and validity but tend to concentrate in a narrower region of chemical space. Diffusion-based models (DiffCSP, DiffCSP++, MatterGen) strike a balance between stability and novelty, with symmetry-aware variants improving physical validity and distributional alignment. Autoregressive Wyckoff-based transformers (WyFormer, WyFormer-DFT) and Crystal-GFN systematically enhance diversity and novelty by respectively encoding crystallographic constraints directly in the representation and leveraging the exploration property offered by GFlowNets [Bengio et al., 2023]. By exposing these trade-offs quantitatively, LEMAT-GENBENCH enables users to choose models based on their discovery priorities—whether exploration, thermodynamic plausibility, structural diversity, or supply-chain robustness.

<table><caption>**Table 2** Model evaluation metrics calculated using MLIP ensemble with LeMat-Bulk as the reference set: distribution, diversity, and HHI. Submissions are organized into three categories separated by horizontal lines: (1) models that incorporate relaxation as part of their generation pipeline (MatterGen, PLaID++, WyFormer, WyFormer-DFT), (2) other generative models, (3) samples from real-world datasets. Best values are shown in **bold**, second-best values are underlined. Arrows indicate whether higher (↑) or lower (↓) values are better. Lower values are better for distribution metrics: JS (Jensen-Shannon divergence), MMD (maximum mean discrepancy), and FID (Fréchet Inception distance) and for HHI (Herfindahl-Hirschman index) metrics. Higher values indicate greater diversity across ElemDiv (elemental diversity), SGDiv (space-group diversity), SizeDiv (crystal size diversity), and SiteDiv (atomic site diversity). “Prod” and “Res” denote production- and reserve-side HHI components, respectively. These metrics are described in detail in Section 3.2.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">Distribution</th><th colspan="4">Diversity</th><th colspan="2">HHI</th></tr><tr><th>JS ↓</th><th>MMD ↓</th><th>FID ↓</th><th>ElemDiv ↑</th><th>SGDiv ↑</th><th>SizeDiv ↑</th><th>SiteDiv ↑</th><th>Prod ↓</th><th>Res ↓</th></tr></thead><tbody><tr><td>MatterGen</td><td>0.44</td><td>0.006</td><td>2.13</td><td>0.64</td><td>0.14</td><td>0.24</td><td>12.78</td><td>3.52</td><td>2.72</td></tr><tr><td>PLaID++</td><td>0.43</td><td>0.029</td><td>2.69</td><td>0.68</td><td>0.26</td><td>0.21</td><td>6.01</td><td>5.23</td><td>3.36</td></tr><tr><td>WyFormer</td><td>0.22</td><td>0.006</td><td><u>1.55</u></td><td><u>0.74</u></td><td><u>0.50</u></td><td>0.27</td><td><u>23.84</u></td><td>3.55</td><td>2.73</td></tr><tr><td>WyFormer-DFT</td><td>0.25</td><td>0.010</td><td>1.72</td><td>0.73</td><td>0.49</td><td>0.26</td><td>23.56</td><td>3.49</td><td>2.74</td></tr><tr><td>ADiT</td><td>0.51</td><td><strong>0.002</strong></td><td>1.72</td><td>0.71</td><td>0.03</td><td>0.23</td><td>14.18</td><td>3.52</td><td>2.78</td></tr><tr><td>Crystal-GFN</td><td>0.55</td><td>0.121</td><td>43.17</td><td><strong>0.75</strong></td><td>0.29</td><td><strong>0.32</strong></td><td><strong>32.06</strong></td><td><u>3.48</u></td><td><strong>2.34</strong></td></tr><tr><td>Crystalformer</td><td>0.26</td><td><u>0.003</u></td><td>2.31</td><td>0.71</td><td>0.35</td><td><u>0.31</u></td><td>18.59</td><td>3.78</td><td>2.82</td></tr><tr><td>DiffCSP</td><td>0.45</td><td>0.008</td><td>2.25</td><td>0.73</td><td>0.14</td><td>0.24</td><td>14.42</td><td><strong>3.29</strong></td><td><strong>2.60</strong></td></tr><tr><td>DiffCSP++</td><td><strong>0.21</strong></td><td>0.003</td><td>1.28</td><td>0.72</td><td><strong>0.51</strong></td><td>0.27</td><td>20.84</td><td>3.56</td><td>2.77</td></tr><tr><td>LLaMat2-CIF</td><td>0.32</td><td><strong>0.002</strong></td><td><strong>1.23</strong></td><td>0.71</td><td>0.25</td><td>0.24</td><td>9.86</td><td>3.94</td><td>2.86</td></tr><tr><td>LLaMat3-CIF</td><td>0.34</td><td>0.005</td><td>9.48</td><td>0.70</td><td>0.10</td><td>0.29</td><td>13.15</td><td>3.83</td><td>2.81</td></tr><tr><td>SymmCD</td><td><u>0.22</u></td><td>0.006</td><td>1.65</td><td>0.73</td><td>0.49</td><td>0.27</td><td>19.45</td><td>3.54</td><td>2.74</td></tr><tr><td>Alexandria</td><td>0.09</td><td>0.000</td><td>1.31</td><td>0.71</td><td>0.50</td><td>0.25</td><td>13.50</td><td>4.02</td><td>3.33</td></tr><tr><td>OQMD</td><td>0.27</td><td>0.004</td><td>0.83</td><td>0.63</td><td>0.48</td><td>0.24</td><td>12.57</td><td>4.22</td><td>3.29</td></tr><tr><td>AFLOW</td><td>0.28</td><td>0.004</td><td>1.78</td><td>0.69</td><td>0.46</td><td>0.25</td><td>10.07</td><td>4.12</td><td>3.22</td></tr><tr><td>MP-Exp</td><td>0.33</td><td>0.021</td><td>2.90</td><td>0.72</td><td>0.71</td><td>0.28</td><td>54.79</td><td>2.78</td><td>2.27</td></tr></tbody></table>

# 5 Benchmark Design Rationale

The metrics introduced in Section 3.2 are not purely intrinsic properties of a model: they depend on concrete methodological choices in the evaluation pipeline. For example, stricter validity filters change how many structures proceed to stability checks, the way we construct the convex hull shifts stability thresholds, and
the choice of reference dataset directly affects both novelty and stability via the phase diagram. Therefore, in this section we motivate and empirically justify four key design decisions in LEMAT-GENBENCH: the validity criteria, the self-consistent MLIP-based hull construction, the pre- vs. post-relaxation protocol, and the choice of reference dataset.

## 5.1 Validity

<figure><img src="image_4.png" /><figcaption>**Figure 3** Validity comparison between SMACT and our proposed method. Five random samples of 1,000 structures from LeMat-Bulk were evaluated using both methods. **Left:** Overall validity rates with standard deviation across seeds. **Right:** Agreement breakdown. Our method recovers 12.6% of structures that SMACT rejects, primarily f-block and metalloid containing structures and elements with multiple oxidation states, while disagreeing on only 1.0% in the opposite direction. Seed-level results in Figure 8.</figcaption></figure>

Existing validity checks often reject a large fraction of experimentally confirmed structures, penalizing generative models for producing chemically plausible outputs and distorting benchmark comparisons. This is especially true for SMACT [Davies et al., 2019], which only marks ~83% of LeMat-Bulk structures [Siron et al., 2025] as valid, with most invalid labels arising from strict charge-neutrality assumptions, fixed oxidation state rules, and metallic structures improperly assessed using oxidation state assignments. Such rigidity causes systematic false negatives for particularly mixed-valence structures, alloys, and structures containing f-block elements.

Our hierarchical validity metric addresses these issues by incorporating empirical oxidation-state frequencies from LeMat-Bulk and ICSD and by introducing a flexible oxidation state assignment protocol for structures failing these strict checks. This enables more flexible treatment of metallic structures and structures containing ambiguous and uncommon oxidation states. As shown in Figure 3 and Figure 8, this approach increases validity from 82.7% (SMACT) to 94.3% while maintaining a low ~1% false-positive rate. It correctly recovers many structures, such as rare-earth oxides and metalloid containing structures, that SMACT incorrectly rejects.

Because this metric relies on empirical priors, it may slightly favor well-documented elements, but the low false-positive rate suggests this bias is limited in practice. Importantly, many valid structures may still contain scarce, toxic, or potentially radioactive elements; rather than marking them as invalid, we manage these undesirable compositions downstream through the HHI index. This separation ensures that validity reflects structural and chemical soundness, while compositional preferences remain application-specific.

## 5.2 Self-Consistent MLIP Convex Hull

Once structures pass validity screening, stability assessment relies on computing their energy above the convex hull ($E_{hull}$). Since DFT computations are expensive, a widely used shortcut is to predict candidate energies with an MLIP while comparing them against a DFT-derived hull from a reference dataset. This
<figure><img src="image_5.png" /><figcaption>**Figure 4** F1-score for stability prediction on LeMat-Bulk. Three MLIPs (orb-v3, uma-s1p1, mace-mp) are evaluated at different $E_{\text{hull}}$ thresholds. (a) MLIP energies compared against a DFT-constructed hull. (b) Self-consistent approach where each MLIP defines its own hull. The self-consistent method yields consistently higher F1-scores, particularly at strict thresholds.</figcaption></figure>

mixed-reference setup introduces systematic inconsistencies: MLIPs and DFT operate on different energy baselines determined by their respective training sets and reference choices. Small shifts, even a few tens of meV/atom, can flip stability labels. This is problematic because $E_{\text{hull}}$ is often treated as a hard decision boundary when evaluating the quality of structures coming out of generative models.

To eliminate these inconsistencies, LEMAT-GENBENCH adopts a self-consistent approach: each MLIP both predicts energies and defines its own convex hull. To compare this approach against the mixed-reference strategy, we treat DFT stability labels from LeMat-Bulk as ground truth. A structure is considered truly stable if its DFT energy lies within a given threshold of the DFT convex hull. We then evaluate how well each MLIP recovers these labels using either (i) the DFT hull or (ii) its self-consistent hull.

As shown in Figure 4, F1-scores<fnref n="7" /> improve markedly under the self-consistent setup e.g., Orb-v3: 0.66 → 0.81, UMA: 0.67 → 0.83 at $E_{\text{hull}}$ < 0.01 eV/atom. The gain likely stems from model-specific energy biases canceling out when the same potential defines both candidate energies and its reference hull. Additional analyses are provided in Section C.3. Overall, these results show that the self-consistent MLIP-based hull is systematically more reliable for stability assessment and should be preferred for benchmarking generative models.

## 5.3 Pre-Relaxation vs. Post-Relaxation

All results in Table 1 are computed on the structures exactly as generated, without applying any additional relaxation. This is crucial for fair comparison: it avoids introducing the bias of a single external relaxer and directly evaluates the geometric quality of the structures produced by each model. In practice, high-quality generative pipelines should already output relaxed or near-relaxed structures, so evaluating them in their raw form is both more meaningful and more desirable. Nevertheless, since some models explicitly integrate relaxation into their generation pipeline, while others do not, we also compute a post-relaxations metrics. In particular, we apply a uniform UMA relaxation to all generated structures (Table 9), to understand how relaxation affects evaluation.

Models that internally relax their outputs show minimal change. PLAiD++ and AFLOW display the lowest pre-relaxation RMSD values (0.10 and 0.11 Å), confirming that outputs already reside near MLIP energy minima.

Models without internal relaxation change substantially after refinement. ADiT shows the strongest shift: stability increases from 0.1% to 17.7% and metastability from 35.4% to 62.6%. DiffCSP improves similarly (stability: 2.0% to 5.9%, M.S.U.N.: 8.2% to 19.4%). These gains indicate that several models produce

<footnote n="7">⁷$F_1 = 2 \frac{\text{precision}-\text{recall}}{\text{precision}+\text{recall}}$, where precision is the fraction of predicted-stable structures that are stable according to DFT and recall is the fraction of truly stable structures correctly identified.</footnote>
geometrically plausible but not fully relaxed structures, and thus benefit greatly from an additional energy minimization step.

Given these large differences, we report in this paper both pre- and post-relaxation results (Table 7, 9), since the magnitude of the shift carries important diagnostic information about each model’s geometry quality and reliance on downstream refinement. This being said, the LEMAT-GENBENCH leaderboard adopts pre-relaxation metrics as the default, to reflect the true generative capability of each model and encourage pipelines that produce well-relaxed structures directly.

## 5.4 Reference Dataset: LeMat-Bulk vs MP-20

Choosing a reference dataset requires balancing two objectives. Model developers benefit from evaluations aligned with the training distribution, where differences isolate architectural and algorithmic contributions. End-users, however, require metrics that reflect real discovery potential, where stability, novelty, and phase competition are assessed against the broadest available set of known materials. To serve both needs, we report MP-20 results in Section C.4 but adopt LeMat-Bulk as the primary reference for the public leaderboard.

LeMat-Bulk (~5M structures) provides the comprehensive phase coverage needed for realistic evaluation. In contrast, MP-20’s limited compositional and structural variety leads to inflated novelty and stability estimates, since many “new” or “stable” structures simply fall outside its constrained reference set. This difference is substantial: novelty drops from 83.1% to 70.5% for MatterGen; stability rates are divided by 2-3× (e.g., PLaID++: 31.0% to 12.4%; DiffCSP: 7.2% to 2.3%); and S.U.N. rates collapse further (e.g., DiffCSP: 4.1% to 0.1%). These shifts illustrate that smaller references can substantially overestimate a model’s practical usefulness.

As generative models scale toward deployment in real discovery settings, evaluation against a comprehensive reference set becomes essential. MP-20 remains valuable for controlled methodological benchmarking, but LeMat-Bulk provides a more reliable basis for assessing discovery-oriented performance. We anticipate expanding reference sets as the field matures and more large-scale, consistently curated materials databases become available.

# 6 Conclusions and Outlook

Generative models for crystalline materials have advanced rapidly, yet the absence of shared evaluation standards has made progress difficult to measure. LEMAT-GENBENCH addresses this gap by introducing the first unified, open benchmarking framework for unconditional crystal generation. It builds on standardized validity checks, self-consistent MLIP-based stability assessments, and well-defined measures of uniqueness, novelty, and diversity. By grounding stability and novelty evaluations in the large-scale LeMat-Bulk dataset, the benchmark offers a stringent and realistic assessment of a model’s capacity to propose genuinely new materials.

Our evaluation of 12 state-of-the-art generative models highlights the diversity of modeling trade-offs: diffusion and autoregressive architectures tend to explore more compositionally and structurally diverse regions of chemical space, while RL- and LLM-guided approaches produce more stable but less novel candidates. No model dominates all metrics, underscoring the need for multi-dimensional evaluation and improved generative paradigms. The LEMAT-GENBENCH leaderboard and open-source implementation provide a common reference point for future model development, reproducibility, and community-driven extensions. We see this as a first step toward closing the loop between computational generation and experimental validation.

**Limitations and Future Directions.** This release focuses on unconditional generation evaluation, with conditional generation benchmarks and expanded model implementations planned for future updates. Key challenges remain. Data quality is a persistent bottleneck: widely used datasets often lack compositional diversity, structural metadata, or negative examples necessary for robust generative model training. Most generative models assume idealized, defect-free crystals, overlooking critical phenomena like disorder, doping, and non-stoichiometry that shape real-world functionality. Moreover, stability assessment relies on ensembles of MLIPs, which, despite averaging, can deviate systematically from DFT, especially near stability thresholds. Finally, although we assess thermodynamic plausibility, our framework does not yet capture kinetic barriers,
synthesis feasibility, or real-world constraints. Bridging these gaps, especially toward synthesis-aware and experimentally grounded pipelines, will require tighter integration between data, modeling, and validation across disciplines. Environmental and sustainability considerations are discussed in Section D.

# 7 Acknowledgments

We acknowledge the fruitful discussions and insights brought by Alfonso Amayuelas, Anuroop Sriram, Benjamin Kurt Miller, Edvin Fako, Hashim Piracha, Kishalay Das, Tristan Deleu, Victor Schmidt, Zekun Danny Ren, Aron Walsh and Tian Xie.

# References

Jonathan Schmidt, Mário RG Marques, Silvana Botti, and Miguel AL Marques. Recent advances and applications of machine learning in solid-state materials science. *npj Computational Materials*, 5(1):83, 2019.

Walter Kohn, Axel D Becke, and Robert G Parr. Density functional theory of electronic structure. *The Journal of Physical Chemistry*, 100(31):12974-12980, 1996.

David S. Sholl and Janice A. Steckel. *Density Functional Theory: A Practical Introduction*. Wiley, March 2009. ISBN 9780470447710. doi:10.1002/9780470447710. URL http://dx.doi.org/10.1002/9780470447710.

Volker L Deringer, Miguel A Caro, and Gábor Csányi. Machine learning interatomic potentials as emerging tools for materials science. *Advanced Materials*, 31(46):1902765, 2019.

Oliver T Unke, Stefan Chmiela, Huziel E Sauceda, Michael Gastegger, Igor Poltavsky, Kristof T Schutt, Alexandre Tkatchenko, and Klaus-Robert Muüller. Machine learning force fields. *Chemical Reviews*, 121 (16):10142-10186, 2021.

Alexandre Duval, Simon V Mathis, Chaitanya K Joshi, Victor Schmidt, Santiago Miret, Fragkiskos D Malliaros, Taco Cohen, Pietro Liò, Yoshua Bengio, and Michael Bronstein. A hitchhiker's guide to geometric GNNs for 3D atomic systems. *arXiv preprint arXiv:2312.07511*, 2023.

Mila AI4Science, Alex Hernandez-Garcia, Alexandre Duval, Alexandra Volokhova, Yoshua Bengio, Divya Sharma, Pierre Luc Carrier, Michał Koziarski, and Victor Schmidt. Crystal-GFN: sampling crystals with desirable properties and constraints. *AI for Accelerated Materials Design Workshop (NeurIPS)*, 2023.

Daniel Levy, Siba Smarak Panigrahi, Sékou-Oumar Kaba, Qiang Zhu, Kin Long Kelvin Lee, Mikhail Galkin, Santiago Miret, and Siamak Ravanbakhsh. SymmCD: Symmetry-preserving crystal generation with diffusion models. In *The Thirteenth International Conference on Learning Representations*, 2025.

Nikita Kazeev, Wei Nong, Ignat Romanov, Ruiming Zhu, Andrey E Ustyuzhanin, Shuya Yamazaki, and Kedar Hippalgaonkar. Wyckoff transformer: Generation of symmetric crystals. In *Forty-Second International Conference on Machine Learning*, 2025a. URL https://openreview.net/forum?id=eFHfRQRjJo.

Claudio Zeni, Robert Pinsler, Daniel Zügner, Andrew Fowler, Matthew Horton, Xiang Fu, Zilong Wang, Aliaksandra Shysheya, Jonathan Crabbé, Shoko Ueda, et al. A generative model for inorganic materials design. *Nature*, 639(8055):624-632, 2025.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2013. URL https://arxiv.org/abs/1312.6114.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In *International Conference on Learning Representations*, 2021. URL https://openreview.net/forum?id=PxTIG12RRHS.

Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J Hu, Mo Tiwari, and Emmanuel Bengio. GFlowNet foundations. *Journal of Machine Learning Research (JMLR)*, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. *Advances in Neural Information Processing Systems*, 33:1877–1901, 2020.

Ilyes Batatia, Philipp Benner, Yuan Chiang, Alin M. Elena, Dávid P. Kovács, Janosh Riebesell, Xavier R. Advincula, Mark Asta, Matthew Avaylon, William J. Baldwin, Fabian Berger, Noam Bernstein, Arghya Bhowmik, Samuel M. Blau, Vlad Cărare, James P. Darby, Sandip De, Flaviano Della Pia, Volker L. Deringer, Rokas Elijošius, Zakariya El-Machachi, Fabio Falcioni, Edvin Fako, Andrea C. Ferrari, Annalena Genreith-Schriever, Janine George, Rhys E. A. Goodall, Clare P. Grey, Petr Grigorev, Shuang Han, Will Handley, Hendrik H. Heenen, Kersti Hermansson, Christian Holm, Jad Jaafar, Stephan Hofmann, Konstantin S. Jakob, Hyunwook Jung, Venkat Kapil, Aaron D. Kaplan, Nima Karimitari, James R. Kermode, Namu Kroupa, Jolla Kullgren, Matthew C. Kuner, Domantas Kuryla, Guoda Liepuoniute, Johannes T. Margraf, Ioan-Bogdan Magdău, Angelos Michaelides, J. Harry Moore, Aakash A. Naik, Samuel P. Niblett, Sam Walton Norwood, Niamh O'Neill, Christoph Ortner, Kristin A. Persson, Karsten Reuter, Andrew S. Rosen, Lars L. Schaaf, Christoph Schran, Benjamin X. Shi, Eric Sivonxay, Tamás K. Stenczel, Viktor Svahn, Christopher Sutton, Thomas D. Swinburne, Jules Tilly, Cas van der Oord, Eszter Varga-Umbrich, Tejs Vegge, Martin Vondrák, Yangshuai Wang, William C. Witt, Fabian Zills, and Gábor Csányi. A foundation model for atomistic materials chemistry. *arXiv preprint arXiv: 2401.00096*, 2023.

Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain. Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm. *npj Computational Materials*, 6(1):138, 2020.

Juhwan Noh, Jaehoon Kim, Helge S. Stein, Benjamin Sanchez-Lengeling, John M. Gregoire, Alan Aspuru-Guzik, and Yousung Jung. Inverse design of solid-state materials via a continuous representation. *Matter*, 1 (5):1370–1384, 2019. ISSN 2590-2385. doi:10.1016/j.matt.2019.08.017.

Jordan Hoffmann, Louis Maestrati, Yoshihide Sawada, Jian Tang, Jean Michel Sellier, and Yoshua Bengio. Data-driven approach to encoding and decoding 3D crystal structures, 2019. URL http://arxiv.org/abs/1909.00949.

Callum J. Court, Batuhan Yildirim, Apoorv Jain, and Jacqueline M. Cole. 3D inorganic crystal structure generation and property prediction via representation learning. *Journal of Chemical Information and Modeling*, 60(10):4518–4535, 2020. ISSN 1549-9596, 1549-960X. doi:10.1021/acs.jcim.0c00464. URL https://pubs.acs.org/doi/10.1021/acs.jcim.0c00464.

Yong Zhao, Edirisuriya M. Dilanga Siriwardane, Zhenyao Wu, Nihang Fu, Mohammed Al-Fahdi, Ming Hu, and Jianjun Hu. Physics guided deep learning for generative design of crystal materials with symmetry constraints. *npj Computational Materials*, 9(1):38, 2023. ISSN 2057-3960. doi:10.1038/s41524-023-00987-9. URL https://doi.org/10.1038/s41524-023-00987-9.

Sungwon Kim, Juhwan Noh, Geun Ho Gu, Alan Aspuru-Guzik, and Yousung Jung. Generative adversarial networks for crystal structure prediction. *ACS Central Science*, 6(8):1412–1420, 2020. doi:10.1021/acscentsci.0c00426. URL https://doi.org/10.1021/acscentsci.0c00426. PMID: 32875082.

Yong Zhao, Mohammed Al-Fahdi, Ming Hu, Edirisuriya M. D. Siriwardane, Yuqi Song, Alireza Nasiri, and Jianjun Hu. High-throughput discovery of novel cubic crystal materials using deep generative neural networks. *Advanced Science*, 8(20):2100566, 2021. doi:https://doi.org/10.1002/advs.202100566. URL https://advanced.onlinelibrary.wiley.com/doi/abs/10.1002/advs.202100566.

Tian Xie, Xiang Fu, Octavian-Eugen Ganea, Regina Barzilay, and Tommi Jaakkola. Crystal diffusion variational autoencoder for periodic material generation. *arXiv preprint arXiv:2110.06197*, 2021.

Rui Jiao, Wenbing Huang, Peijia Lin, Jiaqi Han, Pin Chen, Yutong Lu, and Yang Liu. Crystal structure prediction by joint equivariant diffusion. *Advances in Neural Information Processing Systems*, 36:17464–17497, 2023.

Benjamin Miller, Ricky Chen, Anuroop Sriram, and Brandon Wood. FlowMM: Generating materials with Riemannian flow matching. In *International Conference on Machine Learning (ICML)*. ICML, jun 2024.
Rui Jiao, Wenbing Huang, Yu Liu, Deli Zhao, and Yang Liu. Space group constrained crystal generation. *arXiv preprint arXiv:2402.03992*, 2024.

Rees Chang, Angela Pak, Alex Guerra, Ni Zhan, Nick Richardson, Elif Ertekin, and Ryan P Adams. Space group equivariant crystal diffusion. *arXiv preprint arXiv:2505.10994*, 2025.

Omri Puny, Yaron Lipman, and Benjamin Kurt Miller. Space group conditional flow matching. *arXiv preprint arXiv:2509.23822*, 2025.

Kishalay Das, Subhojyoti Khastagir, Pawan Goyal, Seung-Cheol Lee, Satadeep Bhattacharjee, and Niloy Ganguly. Periodic materials generation using text-guided joint diffusion model. In *The Thirteenth International Conference on Learning Representations*, 2025.

Hyunsoo Park, Anthony Onwuli, and Aron Walsh. Exploration of crystal chemical space using text-guided generative artificial intelligence. *Nature Communications*, 16(1):4379, 2025.

Anuroop Sriram, Benjamin Miller, Ricky Chen, and Brandon Wood. FlowLLM: Flow matching for material generation with large language models as base distributions. In *Neural Information Processing Systems (NeurIPS)*. NeurIPS, jan 2024.

Chaitanya K Joshi, Xiang Fu, Yi-Lun Liao, Vahe Gharakhanyan, Benjamin Kurt Miller, Anuroop Sriram, and Zachary W Ulissi. All-atom diffusion transformers: Unified generative modelling of molecules and materials. In *International Conference on Learning Representations*, 2025.

Zhendong Cao, Xiaoshan Luo, Jian Lv, and Lei Wang. Space group informed transformer for crystalline materials generation. *arXiv preprint arXiv:2403.15734*, 2024.

Nate Gruver, Anuroop Sriram, Andrea Madotto, Andrew Gordon Wilson, C Lawrence Zitnick, and Zachary Ulissi. Fine-tuned language models generate stable inorganic materials as text. *arXiv preprint arXiv:2402.04379*, 2024.

Andy Xu, Rohan Desai, Larry Wang, Gabriel Hope, and Ethan T. Ritz. PLaID: Preference aligned language model for targeted inorganic materials design. In *AI for Accelerated Materials Design Workshop (ICLR)*, 2025a. URL https://openreview.net/forum?id=7aoP3ZeBfy.

Vaibhav Mishra, Somaditya Singh, Dhruv Ahlawat, Mohd Zaki, Vaibhav Bihani, Hargun Singh Grover, Biswajit Mishra, Santiago Miret, N M Anoop Krishnan, et al. Foundational large language models for materials research. *arXiv preprint arXiv:2412.09560*, 2024.

Frederik Lizak Johansen, Ulrik Friis-Jensen, Erik Bjørnager Dam, Kirsten Marie Ørnsbjerg Jensen, Rocío Mercado, and Raghavendra Selvan. deCIFer: Crystal structure prediction from powder diffraction data using autoregressive language models. *arXiv preprint arXiv:2502.02189*, 2025.

Viggo Moro, Charlotte Loh, Rumen Dangovski, Ali Ghorashi, Andrew Ma, Zhuo Chen, Samuel Kim, Peter Y Lu, Thomas Christensen, and Marin Soljačić. Multimodal foundation models for material property prediction and discovery. *Newton*, 2025.

Elena Zamaraeva, Christopher M Collins, Dmytro Antypov, Vladimir V Gusev, Rahul Savani, Matthew S Dyer, George R Darling, Igor Potapov, Matthew J Rosseinsky, and Paul G Spirakis. Reinforcement learning in crystal structure prediction. *Digital Discovery*, 2(6):1831–1840, 2023.

Prashant Govindarajan, Santiago Miret, Jarrid Rector-Brooks, Mariano Phielipp, Janarthanan Rajendran, and Sarath Chandar. Learning conditional policies for crystal design using offline reinforcement learning. *Digital Discovery*, 3(4):769–785, 2024.

Flaviu Cipcigan, Jonathan Booth, Rodrigo Neumann Barros Ferreira, Carine Ribeiro Dos Santos, and Mathias Steiner. Discovery of novel reticular materials for carbon dioxide capture using GFlowNets. *Digital Discovery*, 2024.

Lena Podina, Christina Humer, Alexandre Duval, Victor Schmidt, Ali Ramlaoui, Shahana Chatterjee, Yoshua Bengio, Alex Hernandez-Garcia, David Rolnick, and Félix Therrien. Catalyst GFlowNet for electrocatalyst design: A hydrogen evolution reaction case study. *arXiv preprint arXiv:2510.02142*, 2025.
Sterling G Baird, Hasan M Sayeed, Joseph Montoya, and Taylor D Sparks. matbench-genmetrics: A Python library for benchmarking crystal structure generative models using time-based splits of Materials Project structures. *Journal of Open Source Software*, 9(97):5618, 2024.

Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, and Kristin A. Persson. Commentary: The Materials Project: A materials genome approach to accelerating materials innovation. *APL Materials*, 1(1), 7 2013. doi:10.1063/1.4812323.

Xiang Fu, Zhenghao Wu, Wujie Wang, Tian Xie, Sinan Keten, Rafael Gomez-Bombarelli, and Tommi Jaakkola. Forces are not enough: Benchmark and critical evaluation for machine learning force fields with molecular simulations. arXiv preprint arXiv:2210.07237, 2022.

Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L Chevrier, Kristin A Persson, and Gerbrand Ceder. Python Materials Genomics (pymatgen): A robust, open-source python library for materials analysis. *Computational Materials Science*, 68:314–319, 2013.

Martin Siron, Inel Djafar, Ali Ramlaoui, Etienne du Fayette, Amandine Rossello, Edvin Fako, Matthew McDermott, Felix Therrien, Luis Barroso-Luque, Flaviu Cipcigan, et al. LeMat-Bulk: Aggregating, and de-duplicating quantum chemistry materials databases. *arXiv preprint arXiv:2511.05178*, 2025.

Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. *The journal of machine learning research*, 13(1):723–773, 2012.

Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance as a metric for image retrieval. *International journal of computer vision*, 40(2):99–121, 2000.

Brandon M Wood, Misko Dzamba, Xiang Fu, Meng Gao, Muhammed Shuaibi, Luis Barroso-Luque, Kareem Abdelmaqsoud, Vahe Gharakhanyan, John R Kitchin, Daniel S Levine, et al. UMA: A family of universal models for atoms. *arXiv preprint arXiv:2506.23971*, 2025.

Benjamin Rhodes, Sander Vandenhaute, Vaidotas Šimkus, James Gin, Jonathan Godwin, Tim Duignan, and Mark Neumann. Orb-v3: atomistic simulation at scale. *arXiv preprint arXiv:2504.06231*, 2025.

Mila AI4Science, Alex Hernandez-Garcia, Alexandre Duval, Alexandra Volokhova, Yoshua Bengio, Divya Sharma, Pierre Luc Carrier, Yasmine Benabed, Michał Koziarski, and Victor Schmidt. Crystal-gfn: sampling crystals with desirable properties and constraints. *arXiv preprint arXiv:2310.04925*, 2023.

Andy Xu, Rohan Desai, Larry Wang, Gabriel Hope, and Ethan Ritz. Plaid++: A preference aligned language model for targeted inorganic materials design. *arXiv preprint arXiv:2509.07150*, 2025b.

Stefano Curtarolo, Wahyu Setyawan, Gus LW Hart, Michal Jahnatek, Roman V Chepulskii, Richard H Taylor, Shidong Wang, Junkai Xue, Kesong Yang, Ohad Levy, et al. Aflow: An automatic framework for high-throughput materials discovery. *Computational Materials Science*, 58:218–226, 2012.

Jonathan Schmidt, Love Pettersson, Claudio Verdozzi, Silvana Botti, and Miguel AL Marques. Crystal graph attention networks for the prediction of stable materials. *Science Advances*, 7(49):eabi7948, 2021.

Jonathan Schmidt, Tiago FT Cerqueira, Aldo H Romero, Antoine Loew, Fabian Jäger, Hai-Chen Wang, Silvana Botti, and Miguel AL Marques. Improving machine-learning models in materials science through large datasets. *Materials Today Physics*, 48:101560, 2024.

Scott Kirklin, James E Saal, Bryce Meredig, Alex Thompson, Jeff W Doak, Muratahan Aykol, Stephan Rühl, and Chris Wolverton. The Open Quantum Materials Database (OQMD): Assessing the accuracy of DFT formation energies. *npj Computational Materials*, 1(1):1–15, 2015.

James E Saal, Scott Kirklin, Muratahan Aykol, Bryce Meredig, and Christopher Wolverton. Materials design and discovery with high-throughput density functional theory: The Open Quantum Materials Database (OQMD). *Jom*, 65(11):1501–1509, 2013.
Daniel W Davies, Keith T Butler, Adam J Jackson, Jonathan M Skelton, Kazuki Morita, and Aron Walsh. SMACT: Semiconducting materials by analogy and chemical theory. *Journal of Open Source Software*, 4 (38):1361, 2019.

Zekun Ren, Siyu Isaac Parker Tian, Juhwan Noh, Felipe Oviedo, Guangzong Xing, Jiali Li, Qiaohao Liang, Ruiming Zhu, Armin G. Aberle, Shijing Sun, Xiaonan Wang, Yi Liu, Qianxiao Li, Senthilnath Jayavelu, Kedar Hippalgaonkar, Yousung Jung, and Tonio Buonassisi. An invertible crystallographic representation for general inverse design of inorganic crystals with targeted properties. *Matter*, 5(1):314–335, 2022. ISSN 2590-2385. doi:10.1016/j.matt.2021.11.032. URL https://www.sciencedirect.com/science/article/pii/S2590238521006251.

Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. *Advances in Neural Information Processing Systems*, 27, 2014.

Teng Long, Nuno M Fortunato, Ingo Opahle, Yixuan Zhang, Ilias Samathrakis, Chen Shen, Oliver Gutfleisch, and Hongbin Zhang. Constrained crystals deep convolutional generative adversarial network for the inverse design of crystal structures. *npj Computational Materials*, 7(1):66, 2021.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, *Proceedings of the 32nd International Conference on Machine Learning*, volume 37 of *Proceedings of Machine Learning Research*, pages 2256–2265, Lille, France, 07–09 Jul 2015. PMLR.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, *Advances in Neural Information Processing Systems*, volume 33, pages 6840–6851. Curran Associates, Inc., 2020.

Youzhi Luo, Chengkai Liu, and Shuiwang Ji. Towards symmetry-aware generation of periodic materials. In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023. URL https://openreview.net/forum?id=Jkc74vn1aZ.

Sherry Yang, KwangHwan Cho, Amil Merchant, Pieter Abbeel, Dale Schuurmans, Igor Mordatch, and Ekin Dogus Cubuk. Scalable diffusion for materials generation. arXiv preprint arXiv:2311.09235, 2023.

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In *The Eleventh International Conference on Learning Representations*, 2023. URL https://openreview.net/forum?id=PqvMRDCJT9t.

Michael Samuel Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In *The Eleventh International Conference on Learning Representations*, 2023. URL https://openreview.net/forum?id=li7qeBbCR1t.

Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In *The Eleventh International Conference on Learning Representations*, 2023. URL https://openreview.net/forum?id=XVjTT1nw5z.

Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli, Kevin Patrick Murphy, and Tim Salimans. Diffusion models and gaussian flow matching: Two sides of the same coin. In *The Fourth Blogpost Track at ICLR 2025*, 2025. URL https://openreview.net/forum?id=C8Yyg9wy0s.

Richard S Sutton and Andrew G Barto. *Reinforcement learning: An introduction*. MIT Press Cambridge, 1998.

Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. *IEEE Signal Processing Magazine*, 34(6):26–38, 2017.

Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. *Advances in Neural Information Processing Systems (NeurIPS)*, 2021.
Daniil Tiapkin, Nikita Morozov, Alexey Naumov, and Dmitry Vetrov. Generative flow networks as entropy-regularized RL. *International Conference on Artificial Intelligence and Statistics (AISTATS)*, 2024.

Tristan Deleu, Padideh Nouri, Nikolay Malkin, Doina Precup, and Yoshua Bengio. Discrete probabilistic inference as control in multi-path environments. *Conference on Uncertainty in Artificial Intelligence (UAI)*, 2024.

Yu Zhang, Xiusi Chen, Bowen Jin, Sheng Wang, Shuiwang Ji, Wei Wang, and Jiawei Han. A comprehensive survey of scientific large language models and their applications in scientific discovery. *arXiv preprint arXiv:2406.10833*, 2024.

Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. *arXiv preprint arXiv:2309.10668*, 2023.

Daniel Flam-Shepherd and Alán Aspuru-Guzik. Language models can generate molecules, materials, and protein binding sites directly in three dimensions as xyz, cif, and pdb files. *arXiv preprint arXiv:2305.05708*, 2023.

Luis M Antunes, Keith T Butler, and Ricardo Grau-Crespo. Crystal structure generation with autoregressive large language modeling. *Nature Communications*, 15(1):1–16, 2024.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. *arXiv preprint arXiv:2307.09288*, 2023.

Pierre-Paul De Breuck, Hashim A Piracha, Gian-Marco Rignanese, and Miguel AL Marques. A generative material transformer using wyckoff representation. *arXiv preprint arXiv:2501.16051*, 2025.

Nikita Kazeev, Wei Nong, Ignat Romanov, Ruiming Zhu, Andrey E Ustyuzhanin, Shuya Yamazaki, and Kedar Hippalgaonkar. Wyckoff Transformer: Generation of Symmetric Crystals. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, *Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of *Proceedings of Machine Learning Research*, pages 29495–29526. PMLR, 13–19 Jul 2025b. URL https://proceedings.mlr.press/v267/kazeev25a.html.

Theo Hahn, Uri Shmueli, and JC Wilson Arthur. *International tables for crystallography*, volume 1. Reidel Dordrecht, 1983.

Ruiming Zhu, Wei Nong, Shuya Yamazaki, and Kedar Hippalgaonkar. WyCryst: Wyckoff inorganic crystal generator framework. *Matter*, 7(10):3469–3488, 2024.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. *Advances in Neural Information Processing Systems*, 36, 2023.

Botian Wang, Yawen Ouyang, Yaohui Li, Yiqun Wang, Haorui Cui, Jianbing Zhang, Xiaonan Wang, Wei-Ying Ma, and Hao Zhou. Moma: A modular deep learning framework for material property prediction. *arXiv preprint arXiv:2502.15483*, 2025.

Onur Boyar, Indra Priyadarsini, Seiji Takeda, and Lisa Hamada. LLM-Fusion: A novel multimodal fusion model for accelerated material discovery. *arXiv preprint arXiv:2503.01022*, 2025.

Nathan J Szymanski, Bernardus Rendy, Yuxing Fei, Rishi E Kumar, Tanjin He, David Milsted, Matthew J McDermott, Max Gallant, Ekin Dogus Cubuk, Amil Merchant, et al. An autonomous laboratory for the accelerated synthesis of novel materials. *Nature*, 624(7990):86–91, 2023.

Dejan Zagorac, H Müller, S Ruehl, J Zagorac, and Silke Rehme. Recent developments in the inorganic crystal structure database: theoretical crystal structure data and related features. *Applied Crystallography*, 52(5): 918–925, 2019.

Shyue P Ong, Lei Wang, Byoungwoo Kang, and Gerbrand Ceder. Li-Fe-P-O2 phase diagram from first principles calculations. *Chemistry of Materials*, 20(5):1798–1807, 2008.
Martin Siron, Inel Djafar, Lucile Ritchie, Etienne Du-Fayet, Amandine Rossello, Ali Ramlaoui, Leandro von Werra, Thomas Wolf, and Alexandre Duval. LeMat-Bulk Dataset, 2024. URL https://huggingface.co/datasets/LeMaterial/LeMat-Bulk.

Dan Friedman and Adji Bousso Dieng. The Vendi score: A diversity evaluation metric for machine learning. arXiv preprint arXiv:2210.02410, 2022.

Bent Fuglede and Flemming Topsoe. Jensen-Shannon divergence and Hilbert space embedding. In *International Symposium on Information Theory (ISIT)*, page 31. IEEE, 2004.

Ilya O Tolstikhin, Bharath K Sriperumbudur, and Bernhard Schölkopf. Minimax estimation of maximum mean discrepancy with radial kernels. *Advances in Neural Information Processing Systems*, 29, 2016.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. *Advances in Neural Information Processing Systems*, 30, 2017.

Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Gunter Klambauer. Fréchet ChemNet distance: a metric for generative models for molecules in drug discovery. *Journal of Chemical Information and Modeling*, 58(9):1736-1741, 2018.

Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor Coley. Sample efficiency matters: a benchmark for practical molecular optimization. *Advances in Neural Information Processing Systems*, 35:21342-21357, 2022.

Aria Mansouri Tehrani, Leila Ghadbeigi, Jakoah Brgoch, and Taylor D Sparks. Balancing mechanical properties and sustainability in the search for superhard materials. *Integrating Materials and Manufacturing Innovation*, 6(1):1-8, 2017.
# Supplementary Material

## Box 1: Key Terms in Generative Modeling

**Generative Model:** A machine learning model that learns a data distribution $p(\mathbf{x})$ (or a conditional distribution $p(\mathbf{x}|\mathbf{z})$ or $p(\mathbf{x}|\mathbf{c})$) and can generate new samples $\mathbf{x}' \sim p(\mathbf{x})$ that resemble the training data.

**Latent Space:** A lower-dimensional representation space $\mathbf{z} \in \mathbb{R}^d$ learned by models such as VAEs or GANs, where semantic attributes of the data are often encoded.

**Prior Distribution:** A predefined distribution (e.g., Gaussian) over the latent variables, typically denoted as $p(\mathbf{z})$, from which samples are drawn during generation.

**Decoder / Generator:** A neural network (often denoted $G(\mathbf{z})$) that maps latent codes $\mathbf{z}$ to data samples $\mathbf{x}$.

**Reconstruction Loss:** A metric used in training autoencoders and VAEs that measures how well the generated sample $\hat{\mathbf{x}}$ matches the original input $\mathbf{x}$:

$$ \mathcal{L}_{\text{recon}} = \|\hat{\mathbf{x}} - \mathbf{x}\|^2 \quad \text{or} \quad -\log p(\mathbf{x}|\mathbf{z}). $$

**KL Divergence:** A measure of how much one probability distribution differs from another. Commonly used in VAEs to regularize the encoder:

$$ \mathcal{L}_{\text{KL}} = D_{\text{KL}}(q(\mathbf{z}|\mathbf{x})\|p(\mathbf{z})). $$

**Mode Collapse:** A failure mode in GANs where the generator produces samples with limited diversity, collapsing to a few modes of the data distribution.

**Conditional Generation:** Generation of samples $\mathbf{x}$ based on specified properties or constraints $\mathbf{c}$, e.g., $p(\mathbf{x}|\mathbf{c})$, enabling property-guided design.

**Inverse Design:** The process of searching the input space (e.g., structure, composition) that maps to a desired target property, often using a generative model or an optimization loop in latent space.

**Diffusion Models:** A class of generative models that learn to reverse a stochastic diffusion process. Data $\mathbf{x}_0$ is gradually perturbed into noise via:

$$ q(\mathbf{x}_t|\mathbf{x}_0) = \mathcal{N}(\mathbf{x}_t; \sqrt{\alpha_t}\mathbf{x}_0, (1-\alpha_t)I). $$

and a neural network is trained to denoise $\mathbf{x}_t$ to recover $\mathbf{x}_0$ through a learned reverse process $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_t)$.

**Score-Based Models:** Closely related to diffusion models, they learn the score function $\nabla_{\mathbf{x}} \log p(\mathbf{x})$ and use Langevin dynamics or ODE solvers to sample from the data distribution.

$$ \log p(\mathbf{x}) = \log p(\mathbf{z}) + \sum_{k=1}^{K} \log \left| \det \frac{\partial f_k^{-1}}{\partial \mathbf{x}} \right|. $$

**Flow Matching:** A recent generative approach that avoids training score functions or simulating diffusion. It directly learns a vector field $\mathbf{v}_{\theta}(\mathbf{x}, t)$ that maps noise to data through an ODE:

$$ \frac{d\mathbf{x}}{dt} = \mathbf{v}_{\theta}(\mathbf{x}, t). $$

This method can be trained via supervised learning on synthetic trajectories or velocity fields between the base and target distributions.
Box 2: Key Terms in Crystallography & Materials Science

**Crystal Lattice:** A crystal structure is periodic in three dimensions. This periodicity is described by the lattice, which is defined as

$$L = \{ l_1 a_1 + l_2 a_2 + l_3 a_3 \mid l_1, l_2, l_3 \in \mathbb{Z} \},$$

where $a_1, a_2, a_3$ are basis vectors of $\mathbb{R}^3$.

**Unit Cell:** A unit cell is the smallest unit that can be translated to define the whole lattice. In three dimensions, it is always a parallelepiped.

**Lattice Parameters:** A lattice is typically defined in two ways: either as a set of three basis vectors, or as a set of lattice parameters $(a, b, c, \alpha, \beta, \gamma)$, where $a, b, c$ are the lengths of edges of the unit cell, and $\alpha, \beta, \gamma$ are the angles between them.

**Symmetry:** An object's symmetry is given by the set of geometric transformations that map the object onto itself, leaving it invariant.

**Space Group:** Crystals can be classified by their symmetries. They possess the translational symmetry of their crystal lattices, and they may also have the point group symmetries of rotations and reflections within a unit cell. The combination of translational and point group symmetries can yield more transformations that a crystal can be symmetric to, including screw and glide symmetries. The full set of symmetric transformations that leave a crystal invariant defines the space group of the crystal. In three dimensions, there are 230 distinct space groups.

**Wyckoff Position:** Applying symmetry operations to a crystal may leave some atoms unaffected; for example, a rotation about an axis leaves atoms on the axis in the same position. The set of symmetry operations that do not move a position defines that position's site symmetry. A Wyckoff position is a set of positions that all have the same site symmetries, or conjugate site symmetries. For example, all points along a mirror plane may belong to the same Wyckoff position, while a point at the origin of a unit cell may have its own. Every point in a crystal can be assigned a Wyckoff position.

**Formation Energy ($E_f$):** The formation energy of a crystal is the difference in energy between the crystal and its constituent elements. It can be calculated using: $E_f = E_{\text{total}} - \sum_i n_i \mu_i$

**Energy Above Convex Hull ($E_{\text{hull}}$):** The convex hull gives linear combinations of known phases that represent the lowest-energy mixtures of materials; if a material has an energy above the hull ($E_{\text{hull}} > 0$), it is energetically favorable for it to decompose into a combination of stable phases and is therefore thermodynamically unstable. Mathematically, this can be represented as $E_{\text{hull}} = E_{\text{total}} - E_{\text{hull}}^{\text{min}}$. For example, the convex hull of table salt, NaCl, also includes pure stable Na, pure Cl, as well as NaCl$_3$. However, Na$_2$Cl has a higher formation energy than the combination of NaCl and pure Na, so it is unstable.

**Metastable:** Even if a crystal is not in its lowest possible energy state, it may still be metastable, meaning that a potential energy barrier prevents it from easily transitioning to a lower-energy state. A crystal having a low energy above the convex hull while also being at an energy minimum may indicate that it is metastable. Metastable materials are still important: for example, diamond is metastable, but does not readily convert to a lower energy state under normal conditions.

**CIF:** A Crystallographic Information File is a string-based encoding of a crystal that includes information such as atom positions, unit cell parameters, and chemical elements.

**Relaxation Trajectory:** sequence of intermediate atomic configurations produced during an energy-minimization procedure (e.g., via DFT, MLIPs, or classical force fields). Starting from an initial crystal structure, the optimizer iteratively updates atomic positions and lattice parameters to reduce the system's total energy.
<figure><img src="image_6.png" /><figcaption>**Figure 5** An overview of the generative AI paradigm for candidate structure generation and optimization that underpins much of the work reviewed herein.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Figure 6** Overview of generative models for materials discovery discussed in this work. (a) Evolution of major model architectures over time, showing the early dominance of VAEs and the recent rise of LLMs. (b) Treemap of target properties optimized across models; box size reflects the proportion of papers mentioning each property. Space group, composition, lattice parameters, and formation energy are the most common. (c) Model distribution for unconditional (left) and conditional (right) generation tasks. Most conditional models can also perform unconditional generation, but not vice versa. Methods are grouped by their two dominant architecture families (1° and 2° model classes) for simplicity, with all colors matching panel (a). As many things have been simplified for this figure, please see Section A for details on the various methods presented. Each model is annotated with its primary input data type. **Abbreviations**: LLM = large language model; VAE = variational autoencoder; RL = reinforcement learning; NL prompt = natural language prompt; PXRD = powder X-ray diffraction. “CIF prefix” typically includes composition, space group, and lattice parameters; “Crystallographic file” refers to any file encoding structure data (e.g., XYZ, PDB, CIF).</figcaption></figure>
# A Survey of Generative Models for Crystalline Materials

## A.1 Early Deep Learning Approaches

Variational autoencoders (VAEs) [Kingma and Welling, 2013] were among the first deep generative models applied to crystal structure generation, offering a powerful framework for encoding materials in continuous latent spaces. Early efforts by iMatGen [Noh et al., 2019] employed 3D grid-based image representations of crystal structures to learn a latent space of inorganic materials, further refined by a binary classifier that distinguished stable from unstable structures based on formation energies. This approach successfully demonstrated the ability to generate novel vanadium oxide materials. Subsequent work by Hoffmann et al. [2019] advanced this methodology by combining U-Net segmentation with VAEs in a two-step approach: first generating 3D density maps via a VAE, then segmenting these maps into 3D voxel grids using a U-Net model. Building on this foundation, Court et al. [2020] introduced conditional deep feature consistent VAEs (Cond-DFC-VAE) to enable property-guided crystal generation. A limitation of these early approaches is the use of voxel grid representations that are difficult to decode to physically meaningful atom coordinates and types. A significant advancement came from Ren et al. [2022], who developed a generalized invertible representation encoding both structural information (atom coordinates and types) and lattice parameters through reciprocal space transformations in the FTPC framework. This representation facilitated inverse design of inorganic crystals with targeted properties such as formation energy and bandgap, representing an important step toward property-conditioned generation.

Generative adversarial networks (GANs) [Goodfellow et al., 2014] have also been used for crystal structure prediction and candidate generation. In this framework, a discriminator and a generator neural network are trained, with the first being trained to distinguish between real data and generated samples, and the second trained adversarially to produce samples that fool the discriminator. Kim et al. [2020] used GANs to predict crystal structures conditioned on composition. It was notably one of the first methods to introduce a coordinate-based representation for crystal structure generation. Long et al. [2021] introduced a method to constrain generation to desired properties by additionally optimizing with respect to a property prediction network. CubicGAN [Zhao et al., 2021] introduced a learnable space group aware component to the representation, which proved influential, although restricted to cubic systems.

## A.2 Diffusion and Flow-based Models

Diffusion models [Sohl-Dickstein et al., 2015, Ho et al., 2020, Song et al., 2021] have become a staple in image generation and have also been adapted successfully for crystal generation. They have shown promising capabilities in generating stable 3D periodic structures for novel crystalline materials. These approaches operate on the principle of gradually denoising randomly sampled configurations to produce physically plausible structures, incorporating both geometric and chemical constraints.

Early implementations such as CDVAE [Xie et al., 2021] and SyMat [Luo et al., 2023] combined VAEs with score-based denoising networks operating directly on atomic coordinates. These hybrid models first predict lattice parameters and atomic composition via the VAE component, and then predict atomic coordinates through score-based diffusion. Subsequent architectures like DiffCSP [Jiao et al., 2023] and MatterGen [Zeni et al., 2025] advanced this approach by adopting joint diffusion frameworks that simultaneously model atomic composition, fractional coordinates, and lattice parameters. These methods leverage SE(3)-equivariant neural networks adapted for periodicity [Duval et al., 2023] to predict the denoised composition, coordinates and lattice parameters, ensuring both Euclidean and periodic invariance in the learned distribution. An alternative representation strategy was proposed in UniMat [Yang et al., 2023], which introduced a unified crystal representation using a 4D tensor where atomic positions are stored within corresponding atom entries in the periodic table. This approach employs interleaved attention and convolution layers as denoising networks, effectively addressing the challenge of jointly modeling discrete atom types and continuous atomic coordinates.

Recent work has further extended diffusion models to incorporate domain knowledge and constraints. DiffCSP++ [Jiao et al., 2024] and SymmCD [Levy et al., 2025] enforce space group symmetry constraints during generation, significantly reducing the effective dimensionality of the generative task. By constraining atomic coordinates to follow specified Wyckoff positions and space group symmetries, these models not only
improve computational efficiency but also ensure that generated structures adhere to physically meaningful crystallographic rules. DiffCSP++ is based on using predefined structural templates for each space group, whereas in SymmCD the templates are also generated, resulting in increased diversity.

Text-guided diffusion models represent another frontier, with approaches like TGDMat [Das et al., 2025] and Chemeleon [Park et al., 2025] incorporating contextual representations from pretrained language models to enable generation of stable periodic materials that align with conditions specified in natural language descriptions.

Flow matching (FM) methods [Lipman et al., 2023; Albergo and Vanden-Eijnden, 2023; Liu et al., 2023] are closely related to diffusion models [Gao et al., 2025], and learn a time-dependent velocity field that smoothly transforms an arbitrary base distribution into the target distribution of stable materials. FlowMM [Miller et al., 2024] pioneered this approach for crystal generation, defining specialized representations that enforce global rotational and translational symmetries while respecting periodic boundary conditions. Flow matching notably offers faster generation than diffusion models, since it requires less evaluations of the neural network. Building on this work, FlowLLM [Sriram et al., 2024] introduced a hybrid approach that leverages language models to provide a chemically informed base distribution. Rather than starting from simple distributions, FlowLLM generates initial material representations using an LLM trained on known structures [Gruver et al., 2024], then refines these samples through flow matching that preserves composition while updating atomic positions and lattice parameters. Sampling from a base distribution defined by an LLM resulted in an increase in the rate of stable materials compared to sampling a generic base distribution.

Recent work demonstrates the potential of encoding both periodic (crystalline) and non-periodic (molecular) structures using a single unified flow matching framework. ADiT [Joshi et al., 2025] utilizes an autoencoder to learn a joint latent representation of both crystals and molecules, followed by flow matching in the unified latent space to generate new latent representations and decode them to valid crystals or molecules. This approach enables learning from and generating diverse materials, including metal-organic frameworks (MOFs), which exhibit both periodic and molecular characteristics. Unlike previous approaches that require specialized equivariant architectures, ADiT simplifies the generative process by using a standard Gaussian base distribution and Transformer architecture as the time-dependent velocity field in latent space. This simplification leads to faster generation times while still demonstrating strong performance on generation of stable structures.

## A.3 Reinforcement Learning and GFlowNets

By contrast with generative models that learn from datasets of existing structures, reinforcement learning (RL) [Sutton and Barto, 1998; Arulkumaran et al., 2017] methods instead train an agent (i.e., a neural network) to optimize a given quantity or reward, without data. The model thus learns a behaviour policy by taking actions that act on a defined environment, like moving pieces (actions) on a chessboard (environment) to play a game, exploring actions that are likely to lead to high rewards (e.g., winning the game). This method has been used by Zamaraeva et al. [2023] with actions such as swapping atom positions and changing unit cell parameters, and the reward defined as the a prediction of the negative change in energy from that action. Challenges however lie in access to accurate, but inexpensive energy calculation, as well as efficient exploration strategies. The reinforcement learning approach was extended to crystal design in work by Govindarajan et al. [2024]. In this work, actions consist of choosing new atoms to sequentially add to sites in a crystal. An offline RL framework is used, which allows to use a more expensive DFT-based method for computing rewards.

Generative Flow Networks (GFlowNets) [Bengio et al., 2021, 2023] offer a distinct set of methods, drawing inspiration from reinforcement learning [Tiapkin et al., 2024; Deleu et al., 2024]. GFlowNets formulate generation as a sequential decision process, constructing molecules or materials step-by-step using policy network(s) acting on pre-defined environments. In Crystal-GFN [Mila AI4Science et al., 2023], a space group is first sampled, then the atomic composition and finally the lattice parameters form a crystal. It also encodes rigorously physico-chemical and symmetry constraints by masking impossible actions along the generation process. For materials discovery, the reward function typically corresponds to properties of interest such as formation energy, bandgap, or stress. Crystal-GFN demonstrated this capability by generating structures with low formation energy, targeted bandgaps, and high cell density, while other implementations have extended the approach to generate reticular frameworks for CO₂ capture [Cipcigan et al., 2024].
## A.4 Large Language Models and Multimodal ML Systems

Large language models (LLMs) are increasingly being repurposed across the natural sciences as general-purpose priors for reasoning over sequences, graphs, and spatial data [Zhang et al., 2024]. This paradigm has recently been extended to materials generation. By representing crystal structures through textual descriptions of unit cells and atomic positions, language models trained to generate discrete tokens have shown competitive and sometimes superior performance compared to specialized geometric models [Gruver et al., 2024]. This unexpected effectiveness may stem from language models' efficiency as information compressors [Delétang et al., 2023], enabling sample-efficient learning in data-constrained materials domains.

Early explorations by Flam-Shepherd and Aspuru-Guzik [2023] demonstrated that decoder-based transformers trained on text sequences could generate molecules, crystals, and protein binding sites. Building on this foundation, CrystalLLM [Antunes et al., 2024] fine-tuned language models on Crystallographic Information Files (CIF) for crystal structure generation. The approach was further refined by deCIFer [Johansen et al., 2025] and Gruver et al. [2024]; whereas deCIFer conditioned a decoder-only transformer on powder X-ray diffraction (PXRD) signals to generate crystal structures that agree with observed diffraction patterns in CIF format, Gruver et al. [2024] fine-tuned LLaMA-2 [Touvron et al., 2023] on condensed crystal format descriptions that included unit cell parameters and fractional atomic coordinates. This achieved great generation performance while enabling flexible natural language conditioning, supporting unconditional generation, property-targeted generation, and structure completion.

Matra-Genoa [De Breuck et al., 2025] is another recent decoder-based autoregressive transformer model designed for inorganic crystal generation. By decomposing crystal structures into an interpretable token-based vocabulary, it efficiently generates structures conditioned on target properties such as composition, space group, and formation energies. Further, Mishra et al. [2024] recently demonstrated that foundational materials domain LLMs (LLaMat) with additional training on semantic and syntactic tasks related to CIF (LLaMat-CIF) enables superior crystal generation with high diversity.

Domain knowledge integration has been explored in models like WyFormer [Kazeev et al., 2025b] and CrystFormer [Cao et al., 2024], which incorporate crystallographic priors of space groups and Wyckoff positions [Hahn et al., 1983, Zhu et al., 2024]. Further improvements have come through reinforcement learning techniques such as direct preference optimization (DPO) [Rafailov et al., 2023], demonstrated in PLaID [Xu et al., 2025a,b], which significantly improved stability and S.U.N. rates by learning from comparative evaluations of generated structures by machine learning interatomic potentials. Their work showcases the potential of leveraging reinforcement learning as a framework for optimizing future long-tail conditional generation tasks.

Multimodal approaches represent another frontier, attempting to integrate structural, spectroscopic, and textual information for more comprehensive materials modeling. Systems like MultiMat [Moro et al., 2025], MoMa [Wang et al., 2025], and LLM-Fusion [Boyar et al., 2025] demonstrate early efforts to combine representations across modalities, while agentic frameworks like A-Lab [Szymanski et al., 2023] point toward autonomous laboratories that can propose, execute, and refine materials synthesis guided by computational predictions.

## B Evaluation Metrics for Materials Generation

Unconditional generation refers to the task of producing valid, stable crystal structures without targeting specific properties or constraints. The following metrics assess the fundamental quality of generated structures:

*Fundamental Validity Metrics.* These ensure the outputs are physically meaningful and chemically plausible. In different terms, they serve as a sanity check both for model development and inference time. Note that all metrics may not be relevant for every material system.

* **Charge Neutrality:** The total valence charge of all atoms must sum to zero:

$$ \sum_{i=1}^{N} q_{i} = 0 \qquad (1) $$
where $q_i$ is the nominal oxidation state of atom $i$ in the structure. For this to be calculated, the oxidation states of every atom in the structure must first be assigned. Here, we have developed a hierarchical structure for determining oxidation states and charge neutrality:

1. If all atoms are metals, each atom is assigned a nominal oxidation state of zero, and the structure is labeled as charge balanced. This metallic assignment is determined by using the Pymatgen "BVAnalyzer" class and the SMACT "metallicity-score" function with a cutoff of 0.7 [Davies et al., 2019]. If either of these are satisfied, the structure is labeled as a metal and therefore labeled charge-balanced.

2. If all atoms are not metals, the Pymatgen "get-oxi-state-decorated-structure" function [Ong et al., 2013] is used to assign oxidation states and determine charge balance.

3. However, the function used above can fail to find oxidation states for structures that are not well optimized. It is still necessary to determine whether these structures are charge-balanced, particularly in the case of generative model benchmarks, when many structures may be too far from typical structures for the Pymatgen functions to analyze them. Here, we determine charge neutrality using a data-driven approach from LeMat-Bulk [Siron et al., 2025]. First, this workflow determines all the possible charge-balanced compositions of oxidation states based on the observed oxidation states in LeMat-Bulk or the ICSD [Zagorac et al., 2019]. The most likely oxidation state assignments for this particular composition are determined by assigning each composition a score based on how probable that particular oxidation state configuration is, as determined by the distribution of oxidation states seen in LeMat-Bulk/ICSD. This score is determined by taking the geometric mean of all probabilities for each individual oxidation state. The geometric mean was chosen to normalize the score to the number of atoms in the structure, so scores between structures with two atoms and structures with ten atoms are comparable. Under the current implementation, if a composition exists from ICSD or LeMat-Bulk oxidation states that charge balances the structure, it passes the validity test. However, it is also possible to introduce a threshold here to filter out structures that require a combination of oxidation states which are highly uncommon, if a stricter validity metric is desired.

4. If no charge-balanced composition can be made using oxidation states from LeMat-Bulk/ICSD, the structure is passed to the next stage of the validity assessment. In this stage, all historically observed oxidation states for each element, including highly irregular ones, are used to see if the structure can be charge-balanced. This list of oxidation states is drawn from Pymatgen's "Element" class. Since this list includes some very irregular oxidation states (negative charges on metals, for example), an additional check is used to determine whether this charge-balanced structure is viable. Since we don't have access to probabilities for these very uncommon states, the oxidation state of each element in the system is correlated to the Pauli electronegativity of that element. A negative correlation is desired, as this would show that increasing electronegativity leads to a decreasing oxidation state. Any structures that show a positive correlation are labeled invalid, as this would reflect a very chemically implausible structure overall.

* **Minimum Interatomic Distance:** All interatomic distances $d_{ij}$ must exceed a cutoff value $d_{min}$ to prevent atomic overlap. We use 0.7 Å in this work.

$$d_{ij} > d_{min} \quad \forall i \neq j \qquad (2)$$

* **Mass density and atomic number density** should be within reasonable ranges. Mass density is given by $\rho = \frac{M_{total}}{V_{cell}}$, in $(g/cm^3)$. The latter is expressed in atoms/Å³. We take upper bounds of 25 g/cm³ and 0.5 atoms/Å³, respectively.

* **Valid crystallographic representation** is a good proxy to determine whether a structure is CIF-readable using *pymatgen*.

* **Lattice parameters** should be within reasonable ranges. We take upper bounds of 100 Å for a,b,c, and 180 degrees for $\alpha, \beta, \gamma$ respectively, and lower bounds of 1 Å and zero degrees for a,b,c and $\alpha, \beta, \gamma$, respectively.
<figure><img src="image_8.png" /><figcaption>**Figure 7** The elemental composition of the structures labeled invalid by SMACT yet labeled valid by our model across the 5 random samples of 1000 structures drawn from LeMat-Bulk. Of these 5000 structures, 631 are labeled valid by our model but invalid by SMACT. The counts indicate the number of times a structure containing that element is found in the overall list of structures. For example, 70 of the 631 structures contain boron.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>**Figure 8** **Seed variation in validity outcomes.** Five independent samples of 1,000 structures each from LeMat-Bulk. Categories match the agreement breakdown from Figure 3. Dashed lines indicate mean values. High consistency across seeds demonstrates robustness of both validation methods, with our approach consistently validating ~126 additional structures per seed while rejecting only ~10 SMACT-approved structures.</figcaption></figure>
Stability metrics. These assess the thermodynamic and energetic properties of generated structures:

* **Formation Energy ($E_f$):**

$$E_f = E_{\text{tot}}(\text{compound}) - \sum_i n_i \mu_i \quad (3)$$

where $E_{\text{tot}}$ is the total energy of the crystal, $n_i$ is the number of atoms of element $i$, and $\mu_i$ is the chemical potential of the pure element. The result is normalized per atom: $E_f^{\text{per atom}} = \frac{E_f}{\sum_i n_i}$. We want it to be as small (and negative) as possible.

The chemical potentials $\mu_i$ are derived from the LeMaterial-Bulk dataset using the same methodology as used in Ong et al. [2008].

**Multi-MLIP Ensemble Implementation:** The formation energy metric supports ensemble statistics across multiple MLIPs (ORB, MACE, UMA). For each structure, ensemble statistics are computed as:

$$\langle E_f \rangle = \frac{1}{N_{\text{MLIP}}} \sum_{k=1}^{N_{\text{MLIP}}} E_f^{(k)} \quad (4)$$

$$\sigma_{E_f} = \sqrt{\frac{1}{N_{\text{MLIP}} - 1} \sum_{k=1}^{N_{\text{MLIP}}} (E_f^{(k)} - \langle E_f \rangle)^2} \quad (5)$$

where $E_f^{(k)}$ is the formation energy predicted by the k-th MLIP. The implementation extracts pre-computed ensemble statistics from structure properties (formation\_energy\_mean, formation\_energy\_std) or calculates them from individual MLIP results (formation\_energy\_orb, formation\_energy\_mace, etc.). A minimum of 2 MLIPs is required for ensemble statistics.

* **Energy Above Convex Hull ($E_{\text{hull}}$):**

$$E_{\text{hull}} = E_{\text{tot}} - E_{\text{hull}}^{\min} \quad (6)$$

Structures with $E_{\text{hull}} \le 0$ are considered stable, while values below approximately 0.1 eV/atom are often deemed metastable. We take LeMat-Bulk [Siron et al., 2024] as reference point for calculating the convex hull.

The convex hull is constructed by filtering the LeMat-Bulk dataset to include only compounds containing elements present in the target composition, creating PDEntry objects, and using Pymatgen's PhaseDiagram.get_decomp_and_eAbove_hull() method. The implementation handles charged species by extracting neutral elements before phase diagram construction. Multi-MLIP ensemble statistics follow the same formulation as formation energy: $\langle E_{\text{hull}} \rangle = \frac{1}{N_{\text{MLIP}}} \sum_{k=1}^{N_{\text{MLIP}}} E_{\text{hull}}^{(k)}$ with corresponding standard deviation calculations.

* **Relaxation Stability:** We use an ensemble of Machine Learning Interatomic Potentials to relax the generated structures (each one is done independently). Then, compute the Root Mean Square Deviation (RMSD) between pre- and post-relaxation atomic positions:

$$\text{RMSD} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} \| \mathbf{r}_{i}^{\text{init}} - \mathbf{r}_{i}^{\text{relax}} \|^2} \quad (7)$$

Low RMSD indicates minimal distortion and structural robustness under optimization.

The implementation calculates individual RMSD values for each MLIP relaxation, then computes ensemble statistics: $\langle \text{RMSD} \rangle = \frac{1}{N_{\text{MLIP}}} \sum_{k=1}^{N_{\text{MLIP}}} \text{RMSD}^{(k)}$ where RMSD$^{(k)}$ is the relaxation RMSD from the k-th MLIP. The metric extracts pre-computed values from structure properties (relaxation\_rmsd\_mean, relaxation\_rmsd\_std) or calculates ensemble statistics from individual MLIP results (relaxation\_rmsd\_orb, relaxation\_rmsd\_mace, etc.). We use a force convergence threshold of 0.02 eV/Å and a maximum of 50 optimization steps for reporting RMSD in this work.

*Novelty, Uniqueness, and Diversity Metrics.* These evaluate how effectively a model explores the chemical space:
* **Novelty (N):** Evaluates the fraction of generated structures that are not present in a reference dataset of known materials. The novelty score is defined as:

$$N = \frac{|\{x \in G \mid x \notin T\}|}{|G|} \qquad (8)$$

where G is the set of generated structures and T is the reference dataset (LeMat-Bulk).

The implementation supports two comparison methods: **BAWL fingerprinting** using crystallographic hash strings with the Weisfeiler-Lehman algorithm, and **structure matching** using Pymatgen's symmetry-aware structural comparison algorithms. For BAWL, novelty is determined by checking if the generated structure's fingerprint exists in the pre-computed reference fingerprint set. For structure matching, each generated structure is compared against reference structures with overlapping elemental compositions using space group analysis and atomic position matching with configurable tolerances. In our paper, we report results using *pymatgen's StructureMatcher* approach for more robust structural comparison against the LeMat-Bulk reference dataset.

* **Uniqueness (U):** Measures the fraction of unique structures within the generated set to assess internal diversity. The uniqueness score is defined as:

$$U = \frac{|\text{unique}(G)|}{|G|} \qquad (9)$$

where unique(G) returns the set of unique structures based on their fingerprints.

The metric is implemented as a structure-level continuous scoring system rather than binary classification. For BAWL fingerprinting, individual uniqueness scores are assigned as $u_i = 1/c_i$, where $c_i$ is the count of structures sharing the same fingerprint within the generated set. This assigns a score of 1.0 to truly unique structures while proportionally penalizing duplicated structures. For structure matching, the implementation uses pairwise comparison with an ordered approach: structure i is considered unique if it is not equivalent to any structure j where $j < i$, ensuring deterministic selection of the first occurrence as the unique representative. The overall uniqueness metric is computed as $U = \frac{1}{|G|} \sum_{i=1}^{|G|} u_i$. Both BAWL fingerprinting and structure matching methods are supported, with structure matching used for reporting results.

* **S.U.N. and M.S.U.N. Rates:** Proportion of generated structures that are simultaneously Stable (or Metastable), Unique, and Novel:

$$\text{S.U.N. Rate} = \frac{|\text{novel}(\text{unique}(x \in G \mid E_{\text{hull}}(x) \le 0))|}{|G|} \qquad (10)$$

$$\text{M.S.U.N. Rate} = \frac{|\text{novel}(\text{unique}(x \in G \mid 0 < E_{\text{hull}}(x) \le \tau))|}{|G|} \qquad (11)$$

The implementation follows a hierarchical computation order: **Stability → Uniqueness → Novelty**. First, structures are classified as stable ($E_{\text{hull}} \le 0$) or metastable ($0 < E_{\text{hull}} \le \tau$) using energy above hull values computed by the Multi-MLIP stability preprocessor. Then, uniqueness is evaluated within each stability class separately using the chosen comparison method. Finally, novelty is assessed for unique structures from each stability class. This hierarchical approach provides detailed metrics at each evaluation stage: stability counts, unique-within-stable/metastable counts, and final SUN/MSUN counts. The Multi-MLIP preprocessor assigns ensemble stability properties (e.g., e_above_hull_mean) to structure objects, enabling robust stability classification across multiple MLIPs (ORB, MACE, UMA). We set $\tau$ to 0.1 eV/atom for assembling results.

* **Diversity:** We assess diversity by comparing the distribution of generated structures against reference datasets across key crystallographic and compositional attributes. This includes space groups, elemental compositions, lattice parameters, and atomic environments. In addition to distribution plots, we compute feature-level entropy metrics to quantify the spread of generated samples.

- **Composition, Space Group, Lattice, and Atomic Site Entropy:** Given a set of N generated structures, we first compute the frequency $f_i$ of each category (e.g., element i for composition,
space group index for symmetry, discretized lattice bins, or Wyckoff site labels). These frequencies are normalized into a probability distribution $p_i = \frac{f_i}{\sum_j f_j}$. We then compute the Shannon entropy

$$H = -\sum_i p_i \log p_i,$$

together with the Vendi score [Friedman and Dieng, 2022], defined as the exponential of the Shannon entropy. Although illustrated here for composition entropy, the same procedure is applied to the other diversity attributes in our benchmark.

*Distribution-Level Metrics.* To evaluate how well the distribution of generated structures matches that of real materials, we use the following metrics:

* **Jensen-Shannon Distance** [Fuglede and Topsoe, 2004]:

$$JSD(P, Q) = \sqrt{\frac{1}{2}D_{KL}(P \| M) + \frac{1}{2}D_{KL}(Q \| M)} \quad (12)$$

where $P$ and $Q$ are the distributions of generated and real samples, respectively, $M = \frac{1}{2}(P + Q)$ is their mixture distribution, and $D_{KL}$ denotes the Kullback-Leibler divergence.

* **Maximum Mean Discrepancy (MMD)** [Tolstikhin et al., 2016]:

$$MMD^2(P, Q) = \mathbb{E}_{x,x'}[k(x, x')] + \mathbb{E}_{y,y'}[k(y, y')] - 2\mathbb{E}_{x,y}[k(x, y)] \quad (13)$$

where $P$ and $Q$ are distributions of generated and real samples, and $k$ is a kernel function.

* **Fréchet Distance Metrics** [Heusel et al., 2017, Preuer et al., 2018]: Variants such as the Fréchet ChemNet Distance (FCD) compare the distributions of generated and reference structures:

$$FD(G, T) = \|\mu_G - \mu_T\|^2 + \mathrm{Tr}(\Sigma_G + \Sigma_T - 2(\Sigma_G \Sigma_T)^{1/2}) \quad (14)$$

where $\mu$ and $\Sigma$ denote the mean and covariance in embedding space.

*Model Efficiency.* This measures how effectively a model learns from limited training data [Gao et al., 2022]:

* **Generic Metrics**: training dataset size, number of model parameters, number of epochs required for training, training time and associated computational infrastructure, inference time on 10k structures.

* **Learning Curve Analysis**: Performance (e.g., S.U.N. rate, property prediction accuracy) as a function of the number of expensive function evaluations (e.g., DFT calculations) required for training, i.e., the number of labeled data points.

*Herfindahl-Hirschman Index (HHI) Metrics.* The Herfindahl-Hirschman index quantifies supply risk concentration for materials by measuring the concentration of element production sources and reserves. The elemental data for calculating HHI metrics are derived from the calculator<fnref n="8" /> made available by Mansouri Tehrani et al. [2017]. For a given crystal structure with composition, we compute:

* **Compound HHI Value**: For a compound with chemical formula represented by composition $C$:

$$HHI_{\text{compound}} = \sum_{i} x_{i} \cdot HHI_{i} \quad (15)$$

where $x_i$ is the fractional composition of element $i$ in the compound, and $HHI_i$ is the element-specific HHI value.

<footnote n="8">⁸https://sparks.mse.utah.edu/about.html</footnote>
* **Production HHI:** Measures supply risk based on the concentration of element production sources (market concentration):

$$ \text{HHI}_{\text{production}} = \sum_{j} s_{j}^{2} \times 10000 \qquad (16) $$

where $s_j$ is the market share of producer j for a given element.

* **Reserve HHI:** Measures long-term supply risk based on the concentration of element reserves (geographic distribution):

$$ \text{HHI}_{\text{reserve}} = \sum_{k} r_{k}^{2} \times 10000 \qquad (17) $$

where $r_k$ is the fraction of global reserves held by country/region k.

* **Scaling Convention:** HHI values are typically scaled from the classical range [0, 10000] to a convenience range [0, 10]:

$$ \text{HHI}_{\text{scaled}} = \frac{\text{HHI}_{\text{classical}}}{1000} \qquad (18) $$

* **Combined HHI Score:** The final benchmark score combines both production and reserve metrics using weighted averaging:

$$ \text{HHI}_{\text{combined}} = w_{\text{prod}} \cdot \text{HHI}_{\text{production}} + w_{\text{res}} \cdot \text{HHI}_{\text{reserve}} \qquad (19) $$

where $w_{\text{prod}} = 0.25$ and $w_{\text{res}} = 0.75$ by default, prioritizing long-term supply security over short-term market dynamics.

* **Missing Element Handling:** Elements not found in the HHI lookup tables are assigned the maximum risk value (10000 unscaled / 10 scaled) to represent maximum supply uncertainty for rare or untracked elements.

* **Risk Categories:** For the scaled [0, 10] range:

$$ \text{Low Risk : HHI}_{\text{scaled}} \leq 2.0 \qquad (20) $$

$$ \text{Moderate Risk : } 2.0 < \text{HHI}_{\text{scaled}} \leq 5.0 \qquad (21) $$

$$ \text{High Risk : HHI}_{\text{scaled}} > 5.0 \qquad (22) $$

# C Benchmark Results

<table><caption>**Table 3** Training datasets and data sources used for the reported generative crystal structure models.</caption><thead><tr><th>Model</th><th>Training Dataset</th><th>Source of Submitted Structures</th></tr></thead><tbody><tr><td>ADiT [Joshi et al., 2025]</td><td>MP-20</td><td>Authors of [Joshi et al., 2025]</td></tr><tr><td>Crystal GFN [AI4Science et al., 2023]</td><td>MP-20</td><td>Authors of [AI4Science et al., 2023]</td></tr><tr><td>Crystalformer[Cao et al., 2024]</td><td>MP-20</td><td>Figshare of [Kazeev et al., 2025a]<sup>9</sup></td></tr><tr><td>DiffCSP [Jiao et al., 2023]</td><td>MP-20</td><td>Figshare of [Kazeev et al., 2025a]<sup>9</sup></td></tr><tr><td>DiffCSP++ [Jiao et al., 2024]</td><td>MP-20</td><td>Figshare of [Kazeev et al., 2025a]<sup>9</sup></td></tr><tr><td>LLaMat2 [Mishra et al., 2024]</td><td>MP-20</td><td>Authors of [Mishra et al., 2024]</td></tr><tr><td>MatterGen [Zeni et al., 2025]</td><td>MP-20</td><td>Figshare of [Kazeev et al., 2025a]<sup>9</sup></td></tr><tr><td>PLaID++ [Xu et al., 2025a]</td><td>MP-20</td><td>Authors of [Xu et al., 2025a]</td></tr><tr><td>SymmCD [Levy et al., 2025]</td><td>MP-20</td><td>Figshare of [Kazeev et al., 2025a]<sup>9</sup></td></tr><tr><td>WyFormer-DiffCSP++ [Kazeev et al., 2025a]</td><td>MP-20</td><td>Authors of [Kazeev et al., 2025a]</td></tr><tr><td>WyFormer-DiffCSP++-DFT [Kazeev et al., 2025a]</td><td>MP-20</td><td>Authors of [Kazeev et al., 2025a]</td></tr><tr><td>Alexandria [Schmidt et al., 2021, 2024]</td><td>—</td><td>Sampled from source database</td></tr><tr><td>QOMD [Saal et al., 2013, Kirklin et al., 2015]</td><td>—</td><td>Sampled from source database</td></tr><tr><td>AFLOW [Curtarolo et al., 2012]</td><td>—</td><td>Sampled from source database</td></tr></tbody></table>

As summarized in Table 3, since most models included in this study were trained on MP-20, we report results using the multi-MLIP ensemble benchmark on both the LeMat-Bulk and MP-20 datasets as references. Evaluating on both reference sets ensures consistency between the training distribution and the evaluation

<footnote n="9">⁹https://figshare.com/articles/dataset/Generated_crystals_for_WyFormer_DiffCSP_DiffCSP_WyCryst_SymmCD_CrystalFormer_MiAD/29145101</footnote>
database, allowing authors to better interpret model behavior without being penalized for discrepancies arising solely from dataset choice.

However, LeMat-Bulk is currently the most comprehensive and diverse collection of inorganic crystal structures. Performance on this dataset, therefore, provides the most informative assessment of generative modeling capability and serves as the primary reference for the public leaderboard. Metrics that depend on a reference dataset are expected to vary between the two settings, including novelty scores, energy-based metrics, stability and metastability rates, and embedding-based distributional metrics such as the Fréchet distance.

Several generative models considered here perform structure relaxation as part of their generation pipeline (MatterGen, PLaID++, WyFormer, and WyFormer-DFT). To provide a fair and transparent comparison, we report two sets of results for all models and baselines: (i) metrics computed directly on the submitted structures, and (ii) metrics computed after relaxing the structures using the UMA s-1 potential. In this setting, only UMA s-1 is used for evaluating formation energies and constructing the convex hull. Using the full multi-MLIP ensemble would not be meaningful, since one of the potentials is already used for relaxation and the goal is to measure geometric and energetic quality in a self-consistent manner.

For all relaxation procedures, we use the default configuration of our relaxation pipeline: a force convergence threshold of 0.02 eV/Å and a maximum of 500 optimization steps. These settings correspond to the default parameters of the relaxation module:

```
--fmax 0.02
```

```
--steps 500
```

This ensures that all post-relaxation metrics are evaluated under a uniform and reproducible protocol across models and datasets.

## C.1 Model Training Details

<table><caption>**Table 4** Training and inference times for 2,500 samples for the reported generative models, with respective compute information.</caption><thead><tr><th>Model</th><th>Training Time (hrs)</th><th>Train GPU</th><th>Inference Time (s)</th><th>Inference GPU</th></tr></thead><tbody><tr><td>SymmCD</td><td>26</td><td>V100</td><td>1,400</td><td>RTX8000</td></tr><tr><td>Crystalformer</td><td>13</td><td>A100</td><td>100</td><td>V100</td></tr><tr><td>ADiT</td><td>576</td><td>V100</td><td>300</td><td>V100</td></tr><tr><td>PLaID++</td><td>10</td><td>H100</td><td>345</td><td>H100</td></tr><tr><td>MatterGen</td><td>80</td><td>A100</td><td>18,000</td><td>V100</td></tr><tr><td>DiffCSP</td><td>-</td><td>-</td><td>2,335</td><td>A40</td></tr><tr><td>DiffCSP++</td><td>19.5</td><td>RTX 6000</td><td>6,150</td><td>A40</td></tr><tr><td>WyFormer</td><td>11.5</td><td>RTX 6000</td><td>0.625</td><td>RTX6000</td></tr><tr><td>Crystal-GFN</td><td>12</td><td>CPU</td><td>3000</td><td>CPU</td></tr><tr><td>LLaMat2</td><td>3264</td><td>A100</td><td>3897</td><td>A100</td></tr><tr><td>LLaMat3</td><td>144</td><td>CS2 Cerebras</td><td>4235</td><td>A100</td></tr></tbody></table>

## C.2 Benchmark Results with LeMat-Bulk as the reference database

We visualize S.U.N. and M.S.U.N. structures with respect to LeMat-Bulk from various generative models using t-SNE, a dimensionality reduction method in Figure 9. The resulting plots show that most models produce samples that are broadly dispersed throughout crystallographic space, rather than collapsing into small or isolated clusters. This indicates that many contemporary crystal generators are capable of proposing chemically diverse structures. One standout model in these plots is MatterGen, whose samples appear to have the highest coverage of ORB's embedding space.

We report the model efficiency metrics in Figure 10. Our pareto frontier demonstrates that their does exist a correlation between model parameters and general M.S.U.N. performance, however, careful architectural design is still critical. On the inference-time axis, WyFormer occupies the extreme low-latency regime while MatterGen sits at the opposite end, delivering the highest M.S.U.N.% at the expense of the longest runtimes. While the reported runtimes are not perfectly comparable—since models were evaluated on heterogeneous hardware—they nonetheless highlight the importance of prioritizing architectures that deliver
strong performance under realistic computational budgets, which we view as critical for scalable downstream discovery.

A similar picture emerges when comparing M.S.U.N.% against the number of model parameters. WyFormer and Crystalformer achieve competitive M.S.U.N.% with comparatively small parameter counts, and DiffCSP and MatterGen extend this frontier to larger models. In contrast, SymmCD, ADiT, and especially LLaMat2/3 use one to two orders of magnitude more parameters than the frontier models yet deliver comparable or worse M.S.U.N.%. Together, these results suggest that architectures explicitly tailored to crystal generation (WyFormer, Crystalformer, DiffCSP, and MatterGen) provide better M.S.U.N.%-per-parameter and M.S.U.N.%-per-second than more generic, heavily overparameterized generators.

<figure><img src="image_10.png" /><figcaption>**Figure 9** Two-dimensional t-SNE visualizations (perplexity = 30) of ORB graph embeddings of generative model structures computed with LeMatBulk as the reference set. The visualizations demonstrate how MatterGen, DiffCSP, and PlaID++ generate a wide diversity of novel crystal structures which cover a breadth of chemical space.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>**Figure 10** Performance-efficiency trade-offs for crystal structure generation models. (a) Inference time (log scale) versus MSUN% shows the computational cost required to achieve MSUN structures. Models on the Pareto frontier (connected by blue line) represent optimal time-performance trade-offs. (b) Model parameter count (log scale) versus MSUN% illustrates the relationship between model size and generation quality. WyFormer, Crystalformer, DiffCSP, and MatterGen form the Pareto frontier, demonstrating that larger models can achieve better performance when properly designed.</figcaption></figure>
<table><caption>**Table 5** Model evaluation metrics calculated using MLIP ensemble with LeMat-Bulk as the reference set: Validity, Energy, Stability, and Metastability. All models provide 2500 structures except Crystalformer (1000 structures). Submissions are organized into four categories separated by horizontal lines: (1) models that incorporate relaxation as part of their generation pipeline (MatterGen, PLaID++, WyFormer, WyFormer-DFT), (2) other generative models, (3) samples from real-world datasets that contribute to LeMat-Bulk (Alexandria, OQMD) as baselines, and (4) samples from the AFLOW dataset and MP-Exp structures. Best values are shown in **bold**, second-best values are **underlined**. Arrows indicate whether higher (↑) or lower (↓) values are better.</caption><thead><tr><th rowspan="2">Model</th><th colspan="4">Validity</th><th colspan="2">Novel% ↑</th><th colspan="3">Energy-based</th><th colspan="3">Stability</th><th colspan="3">Metastability</th></tr><tr><th>Valid% ↑</th><th>CN% ↑</th><th>MinDist% ↑</th><th>PhysPlau% ↑</th><th>FormE (Std) ↓</th><th>E<sub>bull</sub> (Std) ↓</th><th>RMSD (Std) ↓</th><th>Stable% ↑</th><th>U-Stable% ↑</th><th>SUN% ↑</th><th>Metastable% ↑</th><th>U-Meta% ↑</th><th>MSUN% ↑</th></tr></thead><tbody><tr><td>MatterGen</td><td><strong>95.7</strong></td><td>95.9</td><td>99.8</td><td><strong>100.0</strong></td><td><strong>95.1</strong></td><td><strong>70.5</strong></td><td>-0.70 ± 0.79</td><td>0.18 ± 0.18</td><td>0.39 ± 0.50</td><td>2.0 (49)</td><td>2.0 (49)</td><td>0.2 (6)</td><td>33.4 (835)</td><td>32.9 (823)</td><td><strong>15.0</strong> (374)</td></tr><tr><td>PLaID++</td><td><strong>96.0</strong></td><td>96.0</td><td>96.2</td><td><strong>96.2</strong></td><td><strong>77.8</strong></td><td>24.2</td><td>-0.50 ± 0.44</td><td><strong>0.09 ± 0.16</strong></td><td><strong>0.13 ± 0.29</strong></td><td><strong>12.4</strong> (311)</td><td><strong>8.6</strong> (215)</td><td><strong>1.0</strong> (26)</td><td><strong>60.7</strong> (1517)</td><td><strong>47.2</strong> (1181)</td><td>7.6 (189)</td></tr><tr><td>WyFormer</td><td><strong>93.4</strong></td><td>95.2</td><td>98.2</td><td><strong>100.0</strong></td><td><strong>93.0</strong></td><td>66.4</td><td>-0.43 ± 0.95</td><td>0.50 ± 0.51</td><td>0.81 ± 0.58</td><td>0.5 (12)</td><td>0.5 (12)</td><td>0.1 (3)</td><td>15.7 (393)</td><td>15.5 (387)</td><td>1.9 (48)</td></tr><tr><td>WyFormer-DFT</td><td><strong>95.2</strong></td><td>95.2</td><td><strong>100.0</strong></td><td><strong>100.0</strong></td><td><strong>95.0</strong></td><td>66.4</td><td>-0.67 ± 0.91</td><td>0.27 ± 0.36</td><td>0.42 ± 0.60</td><td>3.7 (92)</td><td>3.5 (88)</td><td>0.4 (9)</td><td>24.8 (621)</td><td>24.8 (619)</td><td>7.8 (195)</td></tr><tr><td>ADiT</td><td><strong>90.6</strong></td><td><strong>99.0</strong></td><td>91.5</td><td><strong>100.0</strong></td><td><strong>87.8</strong></td><td>26.0</td><td>-0.73 ± 0.93</td><td>0.33 ± 0.45</td><td>0.38 ± 0.40</td><td>0.4 (10)</td><td>0.4 (10)</td><td>0.0 (0)</td><td>36.5 (913)</td><td>35.0 (874)</td><td>1.0 (25)</td></tr><tr><td>Crystal-GFN</td><td><strong>51.7</strong></td><td>58.3</td><td>91.1</td><td><strong>96.3</strong></td><td>51.7</td><td>51.7</td><td>-1.30 ± 2.63</td><td>2.09 ± 2.38</td><td>1.87 ± 0.97</td><td>0.0 (0)</td><td>0.0 (0)</td><td>0.0 (0)</td><td>0.0 (0)</td><td>0.0 (0)</td><td>0.0 (0)</td></tr><tr><td>Crystalformer</td><td><strong>69.9</strong></td><td>96.3</td><td>72.9</td><td><strong>90.9</strong></td><td>69.9</td><td>69.9</td><td>-0.17 ± 1.48</td><td>0.70 ± 1.33</td><td>0.66 ± 1.05</td><td>1.4 (14)</td><td>1.4 (14)</td><td>0.0 (0)</td><td>28.4 (288)</td><td>28.5 (288)</td><td>3.1 (31)</td></tr><tr><td>DiffCSP</td><td><strong>95.7</strong></td><td>96.2</td><td>99.5</td><td><strong>100.0</strong></td><td><strong>94.8</strong></td><td>66.2</td><td>-0.64 ± 1.96</td><td>0.28 ± 0.56</td><td>0.59 ± 0.63</td><td>2.3 (58)</td><td>2.2 (55)</td><td>0.1 (3)</td><td>29.8 (745)</td><td>29.0 (726)</td><td>8.5 (212)</td></tr><tr><td>DiffCSP++</td><td><strong>95.3</strong></td><td>95.5</td><td>99.7</td><td><strong>100.0</strong></td><td><strong>95.1</strong></td><td>62.0</td><td>-0.52 ± 1.87</td><td>0.41 ± 0.44</td><td>0.99 ± 0.82</td><td>1.0 (25)</td><td>1.0 (25)</td><td>0.2 (4)</td><td>26.4 (660)</td><td>26.2 (655)</td><td>5.0 (124)</td></tr><tr><td>LLaMat2</td><td><strong>84.4</strong></td><td>97.0</td><td>87.1</td><td><strong>90.6</strong></td><td><strong>81.4</strong></td><td>30.2</td><td>-0.16 ± 1.01</td><td>0.44 ± 0.72</td><td>0.54 ± 0.73</td><td>0.7 (18)</td><td>0.7 (18)</td><td>0.1 (3)</td><td>38.1 (667)</td><td>38.1 (811)</td><td>2.1 (52)</td></tr><tr><td>LLaMat3</td><td><strong>15.4</strong></td><td>82.5</td><td>16.8</td><td><strong>84.2</strong></td><td>15.2</td><td>10.5</td><td>0.74 ± 0.99</td><td>1.71 ± 2.48</td><td>1.01 ± 1.10</td><td>0.1 (2)</td><td>0.1 (2)</td><td>0.0 (0)</td><td>2.1 (53)</td><td>2.0 (51)</td><td>0.2 (4)</td></tr><tr><td>SymmCD</td><td><strong>73.4</strong></td><td>95.9</td><td>76.4</td><td><strong>100.0</strong></td><td><strong>73.0</strong></td><td>47.0</td><td>-0.02 ± 1.22</td><td>0.88 ± 1.07</td><td>0.87 ± 1.09</td><td>1.4 (34)</td><td>1.4 (34)</td><td>0.1 (2)</td><td>18.6 (464)</td><td>18.3 (457)</td><td>4.2 (59)</td></tr><tr><td>Alexandria</td><td><strong>93.4</strong></td><td>93.4</td><td>99.0</td><td><strong>99.0</strong></td><td><strong>93.4</strong></td><td>0.0</td><td>-0.18 ± 0.94</td><td>0.44 ± 0.61</td><td>0.14 ± 0.32</td><td>2.3 (57)</td><td>2.3 (57)</td><td>0.0 (0)</td><td>27.4 (684)</td><td>27.4 (684)</td><td>0.0 (0)</td></tr><tr><td>OQMD</td><td><strong>96.8</strong></td><td>96.8</td><td>99.4</td><td><strong>99.5</strong></td><td><strong>96.4</strong></td><td>0.0</td><td>-0.23 ± 0.88</td><td>0.39 ± 0.46</td><td>0.14 ± 0.28</td><td>5.3 (132)</td><td>5.3 (132)</td><td>0.0 (0)</td><td>29.1 (727)</td><td>29.0 (724)</td><td>0.0 (0)</td></tr><tr><td>AFLOW</td><td><strong>91.4</strong></td><td>91.4</td><td>100.0</td><td><strong>100.0</strong></td><td><strong>91.4</strong></td><td>21.5</td><td>-0.19 ± 0.86</td><td>0.35 ± 0.42</td><td>0.12 ± 0.44</td><td>9.3 (232)</td><td>8.6 (216)</td><td>0.0 (0)</td><td>30.4 (761)</td><td>27.6 (690)</td><td>0.7 (17)</td></tr><tr><td>MP-Exp</td><td><strong>98.0</strong></td><td>98.0</td><td>100.0</td><td><strong>100.0</strong></td><td><strong>85.6</strong></td><td>1.9</td><td>-1.02 ± 0.97</td><td>0.09 ± 0.45</td><td>0.20 ± 0.35</td><td>15.7 (395)</td><td>13.6 (339)</td><td>0.3 (8)</td><td>64.1 (1602)</td><td>56.5 (1412)</td><td>0.6 (14)</td></tr></tbody></table>

<table><caption>**Table 6** Model evaluation metrics calculated using MLIP ensemble with LeMat-Bulk as the reference set: Distribution, Diversity, and HHI. All models provide 2500 structures except Crystalformer (1000 structures). Submissions are organized into four categories separated by horizontal lines: (1) models that incorporate relaxation as part of their generation pipeline (MatterGen, PLaID++, WyFormer, WyFormer-DFT), (2) other generative models, (3) samples from real-world datasets that contribute to LeMat-Bulk (Alexandria, OQMD) as baselines, and (4) samples from the AFLOW dataset and MP-Exp structures. Best values are shown in **bold**, second-best values are **underlined**. Arrows indicate whether higher (↑) or lower (↓) values are better.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">Distribution</th><th colspan="4">Diversity</th><th colspan="2">HHI</th></tr><tr><th>JS ↓</th><th>MMD ↓</th><th>FID ↓</th><th>ElemDiv ↑</th><th>SGDiv ↑</th><th>SizeDiv ↑</th><th>SiteDiv ↑</th><th>Prod ↓</th><th>Res ↓</th></tr></thead><tbody><tr><td>MatterGen</td><td>0.44</td><td>0.006</td><td>2.13</td><td>0.64</td><td>0.14</td><td>0.24</td><td>12.78</td><td>3.52</td><td>2.72</td></tr><tr><td>PLaID++</td><td>0.43</td><td>0.029</td><td>2.69</td><td>0.68</td><td>0.26</td><td>0.21</td><td>6.01</td><td>5.23</td><td>3.36</td></tr><tr><td>WyFormer</td><td><strong>0.22</strong></td><td>0.006</td><td>1.55</td><td><strong>0.74</strong></td><td><strong>0.50</strong></td><td>0.27</td><td><strong>23.84</strong></td><td>3.55</td><td>2.73</td></tr><tr><td>WyFormer-DFT</td><td>0.25</td><td>0.010</td><td>1.72</td><td>0.73</td><td>0.49</td><td>0.26</td><td>23.56</td><td>3.49</td><td>2.74</td></tr><tr><td>ADiT</td><td>0.51</td><td><strong>0.002</strong></td><td>1.72</td><td>0.71</td><td>0.03</td><td>0.23</td><td>14.18</td><td>3.52</td><td>2.78</td></tr><tr><td>Crystal-GFN</td><td>0.55</td><td>0.121</td><td>43.17</td><td><strong>0.75</strong></td><td>0.29</td><td><strong>0.32</strong></td><td><strong>32.06</strong></td><td><strong>3.48</strong></td><td><strong>2.34</strong></td></tr><tr><td>Crystalformer</td><td>0.26</td><td>0.003</td><td>2.31</td><td>0.71</td><td>0.35</td><td>0.31</td><td>18.59</td><td>3.78</td><td>2.82</td></tr><tr><td>DiffCSP</td><td>0.45</td><td>0.008</td><td>2.25</td><td>0.73</td><td>0.14</td><td>0.24</td><td>14.42</td><td><strong>3.29</strong></td><td><strong>2.60</strong></td></tr><tr><td>DiffCSP++</td><td><strong>0.21</strong></td><td>0.003</td><td>1.28</td><td>0.72</td><td><strong>0.51</strong></td><td>0.27</td><td>20.84</td><td>3.56</td><td>2.77</td></tr><tr><td>LLaMat2</td><td>0.32</td><td><strong>0.002</strong></td><td><strong>1.23</strong></td><td>0.71</td><td>0.25</td><td>0.24</td><td>9.86</td><td>3.94</td><td>2.86</td></tr><tr><td>LLaMat3</td><td>0.34</td><td>0.005</td><td>9.48</td><td>0.70</td><td>0.10</td><td>0.29</td><td>13.15</td><td>3.83</td><td>2.81</td></tr><tr><td>SymmCD</td><td><strong>0.22</strong></td><td>0.006</td><td>1.65</td><td>0.73</td><td>0.49</td><td>0.27</td><td>19.45</td><td>3.54</td><td>2.74</td></tr><tr><td>Alexandria</td><td>0.09</td><td>0.000</td><td>1.31</td><td>0.71</td><td>0.50</td><td>0.25</td><td>13.50</td><td>4.02</td><td>3.33</td></tr><tr><td>OQMD</td><td>0.27</td><td>0.004</td><td>0.83</td><td>0.63</td><td>0.48</td><td>0.24</td><td>12.57</td><td>4.22</td><td>3.29</td></tr><tr><td>AFLOW</td><td>0.28</td><td>0.004</td><td>1.78</td><td>0.69</td><td>0.46</td><td>0.25</td><td>10.07</td><td>4.12</td><td>3.22</td></tr><tr><td>MP-Exp</td><td>0.33</td><td>0.021</td><td>2.90</td><td>0.72</td><td>0.71</td><td>0.28</td><td>54.79</td><td>2.78</td><td>2.27</td></tr></tbody></table>

<table><caption>**Table 7** Model evaluation metrics calculated using uma with LeMat-Bulk as the reference set: Validity, Energy, Stability, and Metastability. All models provide 2500 structures except Crystalformer (1000 structures). Submissions are organized into four categories separated by horizontal lines: (1) models that incorporate relaxation as part of their generation pipeline (MatterGen, PLaID++, WyFormer, WyFormer-DFT), (2) other generative models, (3) samples from real-world datasets that contribute to LeMat-Bulk (Alexandria, OQMD) as baselines, and (4) samples from the AFLOW dataset and MP-Exp structures. Best values are shown in **bold**, second-best values are **underlined**. Arrows indicate whether higher (↑) or lower (↓) values are better.</caption><thead><tr><th rowspan="2">Model</th><th colspan="4">Validity</th><th colspan="2">Novel% ↑</th><th colspan="3">Energy-based</th><th colspan="3">Stability</th><th colspan="3">Metastability</th></tr><tr><th>Valid% ↑</th><th>CN% ↑</th><th>MinDist% ↑</th><th>PhysPlau% ↑</th><th>FormE (Std) ↓</th><th>E<sub>bull</sub> (Std) ↓</th><th>RMSD (Std) ↓</th><th>Stable% ↑</th><th>U-Stable% ↑</th><th>SUN% ↑</th><th>Metastable% ↑</th><th>U-Meta% ↑</th><th>MSUN% ↑</th></tr></thead><tbody><tr><td>MatterGen</td><td><strong>95.7</strong></td><td>95.9</td><td>99.8</td><td><strong>100.0</strong></td><td><strong>95.1</strong></td><td><strong>70.5</strong></td><td>-0.69 ± 0.88</td><td>0.19 ± 0.19</td><td>0.26 ± 0.30</td><td>1.4 (32)</td><td>1.2 (31)</td><td>0.1 (3)</td><td>32.7 (818)</td><td>32.2 (806)</td><td><strong>14.2</strong> (354)</td></tr><tr><td>PLaID++</td><td><strong>96.0</strong></td><td>96.0</td><td>96.2</td><td><strong>100.0</strong></td><td><strong>77.8</strong></td><td>24.2</td><td>-0.43 ± 0.48</td><td>0.09 ± 0.17</td><td>0.10 ± 0.20</td><td><strong>10.0</strong> (250)</td><td><strong>6.9</strong> (173)</td><td><strong>1.0</strong> (24)</td><td><strong>62.1</strong> (1522)</td><td><strong>47.5</strong> (194)</td><td>7.4 (185)</td></tr><tr><td>WyFormer</td><td><strong>93.4</strong></td><td>95.2</td><td>98.2</td><td><strong>100.0</strong></td><td><strong>93.0</strong></td><td>66.4</td><td>-0.41 ± 0.97</td><td>0.51 ± 0.52</td><td>0.53 ± 0.54</td><td>0.5 (12)</td><td>0.5 (12)</td><td>0.0 (1)</td><td>15.2 (381)</td><td>15.0 (375)</td><td>2.1 (53)</td></tr><tr><td>WyFormer-DFT</td><td><strong>95.2</strong></td><td>95.2</td><td><strong>100.0</strong></td><td><strong>100.0</strong></td><td><strong>95.0</strong></td><td>66.4</td><td>-0.65 ± 0.97</td><td>0.28 ± 0.65</td><td>0.19 ± 0.28</td><td>2.2 (56)</td><td>2.2 (55)</td><td>0.2 (5)</td><td>25.9 (648)</td><td>25.7 (643)</td><td>7.8 (195)</td></tr><tr><td>ADiT</td><td><strong>90.6</strong></td><td><strong>99.0</strong></td><td>91.5</td><td><strong>100.0</strong></td><td><strong>87.8</strong></td><td>26.0</td><td>-0.72 ± 0.95</td><td>0.34 ± 0.45</td><td>0.31 ± 0.29</td><td>0.1 (3)</td><td>0.1 (3)</td><td>0.0 (0)</td><td>35.4 (886)</td><td>33.9 (848)</td><td>0.9 (23)</td></tr><tr><td>Crystal-GFN</td><td><strong>51.7</strong></td><td>58.3</td><td>96.3</td><td><strong>96.3</strong></td><td>51.7</td><td>51.7</td><td>-0.56 ± 1.46</td><td>2.83 ± 2.23</td><td>0.83 ± 0.34</td><td>0.0 (0)</td><td>0.0 (0)</td><td>0.0 (0)</td><td>0.0 (0)</td><td>0.0 (0)</td><td>0.0 (0)</td></tr><tr><td>Crystalformer</td><td><strong>69.9</strong></td><td>96.3</td><td>72.9</td><td><strong>90.9</strong></td><td>69.9</td><td>69.9</td><td>-0.17 ± 1.48</td><td>0.71 ± 1.33</td><td>0.46 ± 0.61</td><td>1.4 (14)</td><td>0.0 (0)</td><td>0.0 (0)</td><td>28.1 (284)</td><td>28.1 (284)</td><td>3.2 (32)</td></tr><tr><td>DiffCSP</td><td><strong>95.7</strong></td><td>96.2</td><td>99.5</td><td><strong>100.0</strong></td><td><strong>94.8</strong></td><td>66.2</td><td>-0.62 ± 0.98</td><td>0.28 ± 0.58</td><td>0.37 ± 0.38</td><td>2.0 (49)</td><td>1.8 (46)</td><td>0.2 (4)</td><td>29.7 (743)</td><td>29.1 (727)</td><td>8.2 (205)</td></tr><tr><td>DiffCSP++</td><td><strong>95.3</strong></td><td>95.5</td><td>99.7</td><td><strong>100.0</strong></td><td><strong>95.1</strong></td><td>62.0</td><td>-0.50 ± 0.88</td><td>0.42 ± 0.44</td><td>0.43 ± 0.49</td><td>1.0 (26)</td><td>1.2 (30)</td><td>0.2 (4)</td><td>25.4 (634)</td><td>25.2 (629)</td><td>4.8 (120)</td></tr><tr><td>LLaMat2</td><td><strong>84.4</strong></td><td>97.0</td><td>87.1</td><td><strong>90.6</strong></td><td><strong>81.4</strong></td><td>30.2</td><td>-0.45 ± 1.03</td><td>0.44 ± 0.72</td><td>0.58 ± 0.61</td><td>0.0 (0)</td><td>0.0 (0)</td><td>0.0 (0)</td><td>38.1 (667)</td><td>38.1 (811)</td><td>1.9 (48)</td></tr><tr><td>LLaMat3</td><td><strong>15.4</strong></td><td>82.5</td><td>16.8</td><td><strong>84.2</strong></td><td>15.2</td><td>10.5</td><td>0.79 ± 2.73</td><td>1.74 ± 2.52</td><td>0.75 ± 0.84</td><td>0.0 (0)</td><td>0.0 (0)</td><td>0.0 (0)</td><td>2.2 (54)</td><td>2.1 (52)</td><td>0.2 (5)</td></tr><tr><td>SymmCD</td><td><strong>73.4</strong></td><td>95.9</td><td>76.4</td><td><strong>100.0</strong></td><td><strong>73.0</strong></td><td>47.0</td><td>0.00 ± 1.23</td><td>0.89 ± 1.08</td><td>0.58 ± 0.61</td><td>1.2 (29)</td><td>1.2 (29)</td><td>0.0 (0)</td><td>18.6 (466)</td><td>18.4 (460)</td><td>2.4 (60)</td></tr><tr><td>Alexandria</td><td><strong>94.3</strong></td><td>94.3</td><td>100.0</td><td><strong>100.0</strong></td><td><strong>94.3</strong></td><td>0.0</td><td>-0.16 ± 0.97</td><td>0.45 ± 0.64</td><td>0.11 ± 0.39</td><td>1.2 (29)</td><td>1.2 (29)</td><td>0.0 (0)</td><td>28.1 (695)</td><td>28.1 (695)</td><td>0.0 (0)</td></tr><tr><td>OQMD</td><td><strong>97.3</strong></td><td>97.3</td><td>100.0</td><td><strong>100.0</strong></td><td><strong>96.9</strong></td><td>0.0</td><td>-0.21 ± 0.93</td><td>0.40 ± 0.66</td><td>0.13 ± 0.30</td><td>3.6 (90)</td><td>3.6 (90)</td><td>0.0 (0)</td><td>30.6 (762)</td><td>30.5 (759)</td><td>0.0 (0)</td></tr><tr><td>AFLOW</td><td><strong>91.4</strong></td><td>91.4</td><td>100.0</td><td><strong>100.0</strong></td><td><strong>87.1</strong></td><td>21.5</td><td>-0.18 ± 0.86</td><td>0.35 ± 0.42</td><td>0.11 ± 0.40</td><td>5.8 (146)</td><td>5.4 (136)</td><td>0.0 (0)</td><td>33.5 (838)</td><td>30.4 (760)</td><td>0.7 (17)</td></tr><tr><td>MP-Exp</td><td><strong>98.0</strong></td><td>98.0</td><td>100.0</td><td><strong>100.0</strong></td><td><strong>85.6</strong></td><td>1.9</td><td>-0.99 ± 1.36</td><td>0.11 ± 0.95</td><td>0.19 ± 0.39</td><td>8.8 (221)</td><td>7.4 (186)</td><td>0.4 (9)</td><td>70.1 (1753)</td><td>61.8 (1545)</td><td>0.5 (13)</td></tr></tbody></table>
<table><caption>**Table 8** Model evaluation metrics calculated using uma with LeMat-Bulk as the reference set: Distribution, Diversity, and HHI. All models provide 2500 structures except Crystalformer (1000 structures). Submissions are organized into four categories separated by horizontal lines: (1) models that incorporate relaxation as part of their generation pipeline (MatterGen, PLaID++, WyFormer, WyFormer-DFT), (2) other generative models, (3) samples from real-world datasets that contribute to LeMat-Bulk (Alexandria, OQMD) as baselines, and (4) samples from the AFLOW dataset and MP-Exp structures. Best values are shown in **bold**, second-best values are **underlined**. Arrows indicate whether higher (↑) or lower (↓) values are better.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">Distribution</th><th colspan="4">Diversity</th><th colspan="2">HHI</th></tr><tr><th>JS ↓</th><th>MMD ↓</th><th>FID ↓</th><th>ElemDiv ↑</th><th>SGDiv ↑</th><th>SizeDiv ↑</th><th>SiteDiv ↑</th><th>Prod ↓</th><th>Res ↓</th></tr></thead><tbody><tr><td>MatterGen</td><td>0.44</td><td>0.006</td><td>0.38</td><td>0.64</td><td>0.14</td><td>0.24</td><td>12.78</td><td>3.52</td><td>2.72</td></tr><tr><td>PLaID++</td><td>0.43</td><td>0.029</td><td>0.96</td><td>0.68</td><td>0.26</td><td>0.21</td><td>6.01</td><td>5.23</td><td>3.36</td></tr><tr><td>WyFormer</td><td><u>0.22</u></td><td>0.006</td><td><u>0.10</u></td><td><u>0.74</u></td><td><u>0.50</u></td><td><u>0.27</u></td><td><u>23.84</u></td><td>3.55</td><td>2.73</td></tr><tr><td>WyFormer-DFT</td><td>0.25</td><td>0.010</td><td>0.26</td><td>0.73</td><td>0.49</td><td>0.26</td><td>23.56</td><td>3.49</td><td>2.74</td></tr><tr><td>ADiT</td><td>0.51</td><td><u>0.002</u></td><td>0.27</td><td>0.71</td><td>0.03</td><td>0.24</td><td>14.18</td><td>3.52</td><td>2.78</td></tr><tr><td>Crystal-GFN</td><td>0.55</td><td>0.121</td><td>1.70</td><td><u>0.75</u></td><td>0.29</td><td><u>0.32</u></td><td>32.06</td><td>3.48</td><td><u>2.34</u></td></tr><tr><td>Crystalformer</td><td>0.26</td><td><u>0.003</u></td><td>0.74</td><td>0.71</td><td>0.35</td><td>0.31</td><td>18.59</td><td>3.78</td><td>2.82</td></tr><tr><td>DiffCSP</td><td>0.45</td><td>0.008</td><td>0.11</td><td>0.73</td><td>0.13</td><td>0.24</td><td>14.42</td><td><u>3.29</u></td><td><u>2.60</u></td></tr><tr><td>DiffCSP++</td><td><u>0.21</u></td><td><u>0.003</u></td><td><u>0.08</u></td><td>0.72</td><td><u>0.51</u></td><td><u>0.27</u></td><td>20.84</td><td>3.56</td><td>2.77</td></tr><tr><td>LLaMat2</td><td>0.32</td><td><u>0.002</u></td><td>0.16</td><td>0.71</td><td>0.25</td><td>0.24</td><td>9.86</td><td>3.94</td><td>2.86</td></tr><tr><td>LLaMat3</td><td>0.34</td><td>0.005</td><td>5.14</td><td>0.70</td><td>0.10</td><td>0.30</td><td>13.15</td><td>3.83</td><td>2.81</td></tr><tr><td>SymmCD</td><td><u>0.22</u></td><td>0.006</td><td>0.96</td><td>0.73</td><td>0.49</td><td>0.27</td><td>19.45</td><td>3.54</td><td>2.74</td></tr><tr><td>Alexandria</td><td>0.09</td><td>0.000</td><td>0.01</td><td>0.71</td><td>0.50</td><td>0.25</td><td>13.50</td><td>4.02</td><td>3.33</td></tr><tr><td>OQMD</td><td>0.27</td><td>0.004</td><td>0.14</td><td>0.63</td><td>0.48</td><td>0.24</td><td>12.57</td><td>4.22</td><td>3.29</td></tr><tr><td>AFLOW</td><td>0.28</td><td>0.004</td><td>0.12</td><td>0.69</td><td>0.46</td><td>0.25</td><td>10.07</td><td>4.12</td><td>3.22</td></tr><tr><td>MP-Exp</td><td>0.33</td><td>0.021</td><td>0.69</td><td>0.72</td><td>0.71</td><td>0.28</td><td>54.79</td><td>2.78</td><td>2.27</td></tr></tbody></table>

<table><caption>**Table 9** Model evaluation metrics calculated using uma with LeMat-Bulk as the reference set **after relaxation**: Validity, Energy, Stability, and Metastability. All models provide 2500 structures except Crystalformer (1000 structures). Submissions are organized into four categories separated by horizontal lines: (1) models that incorporate relaxation as part of their generation pipeline (MatterGen, PLaID++, WyFormer, WyFormer-DFT), (2) other generative models, (3) samples from real-world datasets that contribute to LeMat-Bulk (Alexandria, OQMD) as baselines, and (4) samples from the AFLOW dataset and MP-Exp structures. Best values are shown in **bold**, second-best values are **underlined**. Arrows indicate whether higher (↑) or lower (↓) values are better.</caption><thead><tr><th rowspan="2">Model</th><th colspan="4">Validity</th><th colspan="2">Energy-based</th><th colspan="3">Stability</th><th colspan="3">Metastability</th></tr><tr><th>Valid% ↑</th><th>CN% ↑</th><th>MinDist% ↑</th><th>PhysPlau% ↑</th><th>FormE (Std) ↓</th><th>E<sub>bull</sub> (Std) ↓</th><th>RMSD (Std) ↓</th><th>Stable% ↑</th><th>U-Stable% ↑</th><th>SUN% ↑</th><th>Metastable% ↑</th><th>U-Meta% ↑</th><th>MSUN% ↑</th></tr></thead><tbody><tr><td>MatterGen</td><td>95.9</td><td>95.9</td><td><b>100.0</b></td><td><b>100.0</b></td><td>95.3</td><td>70.4</td><td>0.14 ± 0.14</td><td>0.00 ± 0.02</td><td>4.5 (113)</td><td>4.4 (111)</td><td>0.8 (20)</td><td>39.5 (987)</td><td>39.0 (976)</td><td><b>21.7 (543)</b></td></tr><tr><td>PLaID++</td><td><b>99.7</b></td><td><b>99.8</b></td><td><b>100.0</b></td><td><b>100.0</b></td><td><b>100.0</b></td><td>80.8</td><td>-0.74 ± 0.80</td><td>0.08 ± 0.15</td><td>16.8 (404)</td><td>11.3 (272)</td><td><b>1.3 (32)</b></td><td>59.4 (1430)</td><td>46.7 (1123)</td><td>8.2 (198)</td></tr><tr><td>WyFormer</td><td>95.2</td><td>95.2</td><td><b>100.0</b></td><td><b>100.0</b></td><td><b>100.0</b></td><td>94.7</td><td>-0.49 ± 0.47</td><td><b>0.08 ± 0.15</b></td><td>0.00 ± 0.01</td><td>4.0 (101)</td><td>3.9 (97)</td><td>27.5 (687)</td><td>27.2 (679)</td><td>10.4 (260)</td></tr><tr><td>WyFormer-DFT</td><td>95.2</td><td>95.2</td><td><b>100.0</b></td><td><b>100.0</b></td><td><b>100.0</b></td><td>95.0</td><td>-0.68 ± 0.89</td><td>0.25 ± 0.28</td><td>0.01 ± 0.05</td><td>3.8 (94)</td><td>3.6 (90)</td><td>27.7 (691)</td><td>27.6 (689)</td><td>10.3 (257)</td></tr><tr><td>ADiT</td><td>99.0</td><td>99.0</td><td><b>100.0</b></td><td><b>100.0</b></td><td>95.2</td><td>24.3</td><td>-1.04 ± 0.90</td><td><b>0.06 ± 0.09</b></td><td>0.00 ± 0.01</td><td><b>17.7 (441)</b></td><td><b>16.5 (413)</b></td><td>0.6 (16)</td><td><b>62.6 (1562)</b></td><td><b>60.3 (1506)</b></td><td>10.0 (250)</td></tr><tr><td>Crystal-GFN</td><td>56.1</td><td>58.2</td><td>99.3</td><td>97.0</td><td>56.1</td><td>56.1</td><td>-2.54 ± 2.08</td><td>0.86 ± 1.74</td><td>0.12 ± 0.14</td><td>0.0 (0)</td><td>0.0 (0)</td><td>1.0 (24)</td><td>1.0 (24)</td><td>1.0 (24)</td></tr><tr><td>Crystalformer</td><td>94.7</td><td>96.3</td><td>98.4</td><td><b>100.0</b></td><td><b>100.0</b></td><td>94.2</td><td>-0.73 ± 0.86</td><td>0.17 ± 0.24</td><td>0.01 ± 0.08</td><td>10.0 (96)</td><td>9.9 (95)</td><td>41.2 (396)</td><td>40.9 (393)</td><td>10.2 (98)</td></tr><tr><td>DiffCSP</td><td>96.2</td><td>96.2</td><td><b>100.0</b></td><td><b>100.0</b></td><td><b>100.0</b></td><td>94.8</td><td>-0.76 ± 0.82</td><td>0.14 ± 0.15</td><td>0.01 ± 0.05</td><td>5.9 (148)</td><td>5.3 (133)</td><td>0.8 (20)</td><td>40.0 (1000)</td><td>39.5 (988)</td><td>19.4 (484)</td></tr><tr><td>DiffCSP++</td><td>95.5</td><td>95.5</td><td><b>100.0</b></td><td><b>100.0</b></td><td><b>100.0</b></td><td>95.2</td><td>-0.72 ± 0.86</td><td>0.20 ± 0.21</td><td>0.01 ± 0.04</td><td>5.3 (132)</td><td>5.2 (131)</td><td>0.5 (11)</td><td>31.6 (789)</td><td>31.4 (785)</td><td>10.1 (253)</td></tr><tr><td>LLaMat2</td><td>97.0</td><td>97.0</td><td>99.9</td><td><b>100.0</b></td><td><b>100.0</b></td><td>93.8</td><td>-0.77 ± 0.85</td><td>0.13 ± 0.19</td><td>0.00 ± 0.04</td><td>11.3 (282)</td><td>10.6 (266)</td><td>1.0 (24)</td><td>48.1 (1202)</td><td>45.8 (1145)</td><td>9.1 (227)</td></tr><tr><td>LLaMat3</td><td>83.7</td><td>84.6</td><td>98.6</td><td><b>99.9</b></td><td><b>83.4</b></td><td>77.4</td><td>-0.38 ± 0.71</td><td>0.27 ± 0.26</td><td>0.02 ± 0.06</td><td>0.8 (17)</td><td>0.8 (17)</td><td><b>0.2 (4)</b></td><td><b>17.6 (366)</b></td><td><b>17.5 (364)</b></td><td>14.5 (302)</td></tr><tr><td>SymmCD</td><td>95.6</td><td>95.9</td><td>99.7</td><td><b>100.0</b></td><td><b>95.2</b></td><td>60.7</td><td>-0.68 ± 0.84</td><td>0.23 ± 0.25</td><td>0.01 ± 0.05</td><td>5.6 (140)</td><td>5.5 (138)</td><td>0.6 (15)</td><td>30.6 (764)</td><td>30.3 (758)</td><td>9.4 (236)</td></tr><tr><td>Alexandria</td><td>94.3</td><td>94.3</td><td>100.0</td><td>100.0</td><td>94.3</td><td>4.6</td><td>-0.18 ± 0.95</td><td>0.43 ± 0.60</td><td>0.00 ± 0.02</td><td>1.7 (43)</td><td>1.7 (43)</td><td>0.0 (1)</td><td>28.4 (702)</td><td>28.4 (702)</td><td>0.8 (20)</td></tr><tr><td>OQMD</td><td>97.3</td><td>97.3</td><td>100.0</td><td>100.0</td><td>96.9</td><td>4.3</td><td>-0.22 ± 0.90</td><td>0.38 ± 0.45</td><td>0.00 ± 0.00</td><td>6.4 (158)</td><td>6.3 (156)</td><td>0.2 (4)</td><td>29.4 (730)</td><td>29.3 (729)</td><td>0.9 (22)</td></tr><tr><td>AFLOW</td><td>91.4</td><td>91.4</td><td>100.0</td><td>100.0</td><td>87.1</td><td>21.6</td><td>-0.18 ± 0.87</td><td>0.34 ± 0.42</td><td>0.00 ± 0.00</td><td>12.3 (307)</td><td>11.3 (282)</td><td>0.0 (0)</td><td>28.3 (708)</td><td>25.6 (639)</td><td>0.8 (21)</td></tr><tr><td>MP-Exp</td><td>98.0</td><td>98.0</td><td>100.0</td><td>100.0</td><td>85.9</td><td>8.0</td><td>-1.03 ± 0.89</td><td>0.07 ± 0.18</td><td>0.00 ± 0.02</td><td>25.3 (633)</td><td>22.4 (560)</td><td>1.0 (24)</td><td>55.4 (1386)</td><td>48.6 (1214)</td><td>2.0 (51)</td></tr></tbody></table>
<table><caption>**Table 10** Model evaluation metrics calculated using uma with LeMat-Bulk as the reference set **after relaxation**: Distribution, Diversity, and HHI. All models provide 2500 structures except Crystalformer (1000 structures). Submissions are organized into four categories separated by horizontal lines: (1) models that incorporate relaxation as part of their generation pipeline (MatterGen, PLaID++, WyFormer, WyFormer-DFT), (2) other generative models, (3) samples from real-world datasets that contribute to LeMat-Bulk (Alexandria, OQMD) as baselines, and (4) samples from the AFLOW dataset and MP-Exp structures. Best values are shown in **bold**, second-best values are **underlined**. Arrows indicate whether higher (↑) or lower (↓) values are better.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">Distribution</th><th colspan="4">Diversity</th><th colspan="2">HHI</th></tr><tr><th>JS ↓</th><th>MMD ↓</th><th>FID ↓</th><th>ElemDiv ↑</th><th>SGDiv ↑</th><th>SizeDiv ↑</th><th>SiteDiv ↑</th><th>Prod ↓</th><th>Res ↓</th></tr></thead><tbody><tr><td>MatterGen</td><td>0.35</td><td>0.006</td><td>0.51</td><td>0.64</td><td>0.32</td><td>0.24</td><td>12.81</td><td>3.51</td><td>2.72</td></tr><tr><td>PLaID++</td><td>0.43</td><td>0.029</td><td>1.02</td><td>0.68</td><td>0.30</td><td>0.21</td><td>6.01</td><td>5.23</td><td>3.36</td></tr><tr><td>WyFormer</td><td><u>0.28</u></td><td>0.008</td><td><u>0.22</u></td><td><u>0.74</u></td><td><u>0.51</u></td><td><u>0.27</u></td><td><u>24.27</u></td><td>3.52</td><td>2.71</td></tr><tr><td>WyFormer-DFT</td><td>0.29</td><td>0.010</td><td>0.26</td><td><u>0.73</u></td><td>0.49</td><td><u>0.27</u></td><td><u>23.57</u></td><td>3.49</td><td>2.74</td></tr><tr><td>ADiT</td><td>0.36</td><td>0.003</td><td>1.03</td><td>0.71</td><td>0.40</td><td>0.24</td><td>14.50</td><td><b>3.41</b></td><td>2.70</td></tr><tr><td>Crystal-GFN</td><td>0.64</td><td>0.092</td><td>2.94</td><td><b>0.74</b></td><td>0.12</td><td><b>0.32</b></td><td>31.50</td><td>3.45</td><td><b>2.33</b></td></tr><tr><td>Crystalformer</td><td>0.29</td><td>0.006</td><td>0.43</td><td>0.71</td><td>0.39</td><td>0.31</td><td>21.57</td><td>3.56</td><td><u>2.69</u></td></tr><tr><td>DiffCSP</td><td>0.35</td><td>0.009</td><td>0.30</td><td><u>0.73</u></td><td>0.34</td><td>0.25</td><td>14.42</td><td>3.27</td><td>2.59</td></tr><tr><td>DiffCSP++</td><td><b>0.27</b></td><td>0.004</td><td>0.43</td><td>0.72</td><td>0.50</td><td>0.27</td><td>20.90</td><td>3.55</td><td>2.77</td></tr><tr><td>LLaMat2</td><td><u>0.28</u></td><td><b>0.002</b></td><td>0.77</td><td>0.71</td><td>0.38</td><td>0.24</td><td>11.91</td><td>3.84</td><td>2.79</td></tr><tr><td>LLaMat3</td><td>0.43</td><td><u>0.003</u></td><td><b>0.15</b></td><td>0.72</td><td>0.17</td><td>0.27</td><td>28.26</td><td>3.48</td><td>2.93</td></tr><tr><td>SymmCD</td><td>0.28</td><td>0.008</td><td>0.28</td><td><u>0.73</u></td><td>0.50</td><td>0.27</td><td>20.87</td><td>3.51</td><td>2.68</td></tr><tr><td>Alexandria</td><td>0.14</td><td>0.000</td><td>0.01</td><td>0.71</td><td>0.50</td><td>0.26</td><td>13.50</td><td>4.02</td><td>3.33</td></tr><tr><td>OQMD</td><td>0.28</td><td>0.003</td><td>0.16</td><td>0.63</td><td>0.46</td><td>0.24</td><td>12.57</td><td>4.22</td><td>3.29</td></tr><tr><td>AFLOW</td><td>0.29</td><td>0.004</td><td>0.13</td><td>0.69</td><td>0.50</td><td>0.25</td><td>10.07</td><td>4.12</td><td>3.22</td></tr><tr><td>MP-Exp</td><td>0.35</td><td>0.022</td><td>0.74</td><td>0.72</td><td>0.68</td><td>0.28</td><td>54.79</td><td>2.78</td><td>2.27</td></tr></tbody></table>

### C.3 Using a Self-Consistent MLIP-Based Convex Hull

A key challenge in evaluating generated structures with Machine-Learned Interatomic Potentials (MLIPs) is the accurate calculation of the energy above the convex hull ($E_{hull}$). Mixing total energies from an MLIP with a convex hull built from reference DFT calculations introduces inconsistencies: the MLIP and DFT methods operate on different energy references and possess distinct systematic errors. Since stability thresholds are tight (often a few tens of meV/atom), even small systematic shifts can change a structure's classification from stable to unstable.

To address this, our benchmark constructs the convex hull using the same MLIP that evaluates the candidate structures. This self-consistent approach allows systematic errors to cancel, yielding more reliable $E_{hull}$ estimates.

*Empirical validation.* Figures 4 and 11 compare F1-scores for stability prediction under the two approaches. On both LeMat-Bulk and MP-20, the self-consistent hull (panel b) consistently outperforms the mixed-reference DFT hull (panel a), with the largest improvements at the strictest threshold (< 0.01 eV/atom). For example, on LeMat-Bulk the ORB F1-score improves from 0.66 to 0.81.

Figure 12 reveals why. The self-consistent approach (top row) yields systematically lower MAE and higher correlation with DFT ground truth than the mixed-reference approach (bottom row). For ORB, MAE decreases from 0.012 to 0.008 eV/atom; for UMA, from 0.012 to 0.007 eV/atom.

*Ensemble composition.* Although MACE-MP exhibits higher variance and lower correlation with DFT ($r \approx 0.7$) than ORB and UMA ($r > 0.9$), we retain it in the ensemble to ensure model diversity. ORB and UMA were both trained on OMat24 in addition to Materials Project, making their predictions partially correlated; relying on them alone risks collapsing the ensemble toward a single effective model. MACE-MP, trained exclusively on Materials Project, provides an independent perspective that guards against this collapse. On the self-consistent hull, systematic biases cancel within each model, and the remaining variance-type errors are effectively reduced by averaging across architectures with diverse training data. This design choice reflects a deliberate trade-off: accepting slightly higher individual-model error in exchange for more robust and diverse ensemble estimates.
<figure><img src="image_12.png" /><figcaption>**Figure 11** F1-score for stability prediction on MP-20. Format matches Figure 4.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>**Figure 12** Correlation between MLIP-predicted and DFT-calculated *E*<sub>hull</sub> on LeMat-Bulk. *Top row (Own Hull)*: Self-consistent approach yields high correlation (r > 0.94 for ORB and UMA), low error (MAE ≤ 0.008 eV/atom), and tight clustering around the diagonal. The ensemble (rightmost) achieves intermediate performance (MAE = 0.011 eV/atom). *Bottom row (DFT Hull)*: Mixed-reference approach shows degraded correlation and higher MAE. Notably, the ensemble (MAE = 0.035 eV/atom) performs worse than all individual models, illustrating how averaging heterogeneous biases can compound error.</figcaption></figure>
## C.4 Benchmark Results with MP-20 as the Reference Database

We visualize S.U.N. and M.S.U.N. structures from various generative models using t-SNE, a dimensionality reduction method in Figure 13. Across both plots, most models produce structures that are spread out broadly throughout the embedding space rather than collapsing into small, isolated regions. This suggests that current crystal generators are relatively capable of proposing chemically diverse candidates and are not restricted to narrow pockets of the MP-20 distribution. Although there is some overlap, different models still preferentially populate distinct regions of the embedding manifold. This complementary coverage supports the idea that combining multiple generative models within a discovery pipeline could increase the overall diversity of viable candidates.

One notable pattern appears in Figure 13b, where Crystal-GFN produces a comparatively tight, well-defined cluster. Despite having the second highest S.U.N. rate, this indicates the model may be biased toward a narrower slice of chemically stable structures relative to other methods. In contrast, models like PLaid++, WyFormer, and MatterGen reflect broader exploration of the structural landscape.

We report model efficiency metrics in Figure 14 and Figure 15. The Pareto frontier in inference-time space is again spanned by WyFormer, Crystalformer, and Plaid++, which interpolate between low-latency, moderate-quality generation and slower, higher S.U.N.% regimes. WyFormer remains at the extreme low-latency corner, whereas Plaid++ attains the highest S.U.N.% at substantially increased runtime, with MatterGen, DiffCSP, and other approaches lying close to but generally below this curve.

A similar picture appears when plotting S.U.N.% versus parameter count. WyFormer and Crystalformer achieve competitive S.U.N.% with relatively compact models, and DiffCSP and MatterGen extend this frontier to moderately larger parameter budgets. PLaid++ pushes to the highest S.U.N.% but does so with a substantially larger model, while SymmCD, ADiT, and the LLaMat variants sit well below the frontier—using one to two orders of magnitude more parameters than the most efficient models yet achieving comparable or worse S.U.N.%. Since PLaid++ directly optimizes for S.U.N. with respect to MP-20, it is unsurprising that it has the highest S.U.N. rate. At the same time, its performance highlights the promise of reinforcement learning-based optimization which could be applied to smaller models to improve material discovery rates across a range of model sizes.

<figure><img src="image_14.png" /><figcaption>**Figure 13** Two-dimensional t-SNE visualizations (perplexity = 30) of ORB graph embeddings of generative model structures computed with MP-20 as the reference step. The visualizations demonstrate how most models generate a wide diversity of novel crystal structures which cover a breadth of chemical space.</figcaption></figure>
<figure><img src="image_15.png" /><figcaption>(a) Inference time vs MSUN%</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>(b) Model parameters vs MSUN%</figcaption></figure>

**Figure 14** Performance-efficiency trade-offs for crystal structure generation models evaluated on MP-20 measured using MSUN. (a) Inference time (log scale) versus MSUN% shows the computational cost required to achieve MSUN structures. Models on the Pareto frontier (connected by blue line) represent optimal time-performance trade-offs. (b) Model parameter count (log scale) versus MSUN% illustrates the relationship between model size and generation quality. WyFormer, Crystalformer, DiffCSP, and MatterGen form the Pareto frontier.

<figure><img src="image_17.png" /><figcaption>(a) Inference time vs SUN%</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>(b) Model parameters vs SUN%</figcaption></figure>

**Figure 15** Performance-efficiency trade-offs for crystal structure generation models evaluated on MP-20 measured using SUN. (a) Inference time versus S.U.N.% highlights the computational budget required to produce stable and unique structures. The Pareto frontier shows that WyFormer, Crystalformer, and PLAIID++ jointly define the best speed-quality trade-offs, with PLAIID++ achieving the highest S.U.N.% at a moderate additional cost. (b) Model size versus S.U.N.% illustrates how parameter count correlates with generation quality. With WyFormer, CrystalFormer, DiffCSP and PLAIID++ on the pareto frontier, this demonstrates that larger models can achieve better performance when properly designed.
<table><caption>**Table 11** Model evaluation metrics calculated using MLIP ensemble with MP-20 as the reference set: Validity, Energy, Stability, and Metastability. All models provide 2500 structures except Crystalformer (1000 structures). Submissions are organized into four categories separated by horizontal lines: (1) models that incorporate relaxation as part of their generation pipeline (MatterGen, PLaID++, WyFormer, WyFormer-DFT), (2) other generative models, (3) samples from real-world datasets that contribute to LeMat-Bulk (Alexandria, OQMD) as baselines, and (4) samples from the AFLOW dataset and Materials Project Experimental structures. Best values are shown in **bold**, second-best values are **underlined**. Arrows indicate whether higher (↑) or lower (↓) values are better.</caption><thead><tr><th rowspan="2">Model</th><th colspan="4">Validity</th><th colspan="2">Unique% ↑</th><th colspan="2">Energy-based</th><th colspan="3">Stability</th><th colspan="2">Metastability</th></tr><tr><th>Valid% ↑</th><th>CN% ↑</th><th>MinDist% ↑</th><th>PhysPlau% ↑</th><th>FormE (Std) ↓</th><th>Novel% ↑</th><th>E<sub>best</sub> (Std) ↓</th><th>RMSD (Std) ↓</th><th>Stable% ↑</th><th>U-Stable% ↑</th><th>SUN% ↑</th><th>Metastable% ↑</th><th>U-Meta% ↑</th><th>MSUN% ↑</th></tr></thead><tbody><tr><td>MatterGen</td><td>95.7</td><td>95.9</td><td>99.8</td><td><b>100.0</b></td><td><b>95.1</b></td><td>83.1</td><td>-0.70 ± 0.79</td><td>0.16 ± 0.18</td><td>0.39 ± 0.50</td><td>6.0 (150)</td><td>3.6 (91)</td><td></td><td>34.6 (865)</td><td>34.1 (853)</td><td>24.7 (618)</td></tr><tr><td>PLaID++</td><td>96.0</td><td>96.0</td><td>96.2</td><td><b>96.2</b></td><td>77.8</td><td>63.9</td><td>-0.50 ± 0.44</td><td><b>0.06 ± 0.17</b></td><td>0.11 ± 0.29</td><td><b>31.0 (776)</b></td><td><b>21.1 (528)</b></td><td><b>11.6 (289)</b></td><td><b>45.1 (1128)</b></td><td><b>38.0 (933)</b></td><td>21.2 (605)</td></tr><tr><td>WyFormer</td><td>93.4</td><td>95.2</td><td>98.2</td><td><b>100.0</b></td><td>93.0</td><td>80.4</td><td>-0.43 ± 0.56</td><td>0.47 ± 0.52</td><td>0.81 ± 0.97</td><td>2.8 (71)</td><td>2.8 (70)</td><td>2.1 (52)</td><td>16.1 (403)</td><td>16.0 (400)</td><td>6.6 (166)</td></tr><tr><td>WyFormer-DFT</td><td>95.2</td><td>95.2</td><td><b>100.0</b></td><td><b>100.0</b></td><td>95.0</td><td>83.3</td><td>-0.67 ± 0.91</td><td>0.24 ± 0.38</td><td>0.42 ± 0.59</td><td>8.2 (205)</td><td>8.0 (201)</td><td>4.0 (100)</td><td>24.4 (610)</td><td>24.4 (609)</td><td>16.8 (419)</td></tr><tr><td>ADiT</td><td>90.6</td><td><b>99.0</b></td><td>91.5</td><td><b>100.0</b></td><td>87.8</td><td>31.3</td><td>-0.73 ± 0.93</td><td>0.32 ± 0.45</td><td>0.38 ± 0.40</td><td>1.0 (25)</td><td>1.0 (25)</td><td>0.6 (15)</td><td><b>38.0 (994)</b></td><td><b>38.0 (951)</b></td><td>3.1 (78)</td></tr><tr><td>Crystal-GFN</td><td>51.7</td><td>58.3</td><td>96.1</td><td><b>100.0</b></td><td>51.7</td><td>51.7</td><td>-1.29 ± 2.65</td><td>1.60 ± 2.51</td><td>1.86 ± 0.90</td><td>5.6 (141)</td><td>5.6 (141)</td><td>5.6 (141)</td><td>-0.9 (22)</td><td>-0.9 (22)</td><td>0.9 (22)</td></tr><tr><td>Crystalformer</td><td>69.9</td><td>72.9</td><td>99.9</td><td><b>100.0</b></td><td>69.4</td><td>44.2</td><td>-0.17 ± 1.14</td><td>0.68 ± 1.34</td><td>0.69 ± 1.03</td><td>5.2 (52)</td><td>5.2 (52)</td><td>3.3 (33)</td><td>27.9 (278)</td><td>27.5 (278)</td><td>9.4 (94)</td></tr><tr><td>DiffCSP</td><td>95.7</td><td>96.2</td><td>99.5</td><td><b>100.0</b></td><td>94.8</td><td>81.9</td><td>-0.64 ± 0.96</td><td>0.25 ± 0.56</td><td>0.59 ± 0.63</td><td>7.2 (179)</td><td>7.0 (176)</td><td>4.1 (102)</td><td>28.8 (720)</td><td>28.1 (702)</td><td>18.3 (457)</td></tr><tr><td>DiffCSP++</td><td>95.3</td><td>95.5</td><td>99.7</td><td><b>100.0</b></td><td>95.1</td><td>81.1</td><td>-0.52 ± 0.87</td><td>0.39 ± 0.44</td><td>0.70 ± 0.83</td><td>4.8 (121)</td><td>4.8 (121)</td><td>3.4 (85)</td><td>26.3 (657)</td><td>26.1 (652)</td><td>14.1 (353)</td></tr><tr><td>LLaMat2</td><td>84.4</td><td>87.1</td><td>99.6</td><td><b>100.0</b></td><td>81.4</td><td>53.0</td><td>-0.47 ± 1.03</td><td>0.42 ± 0.71</td><td>0.51 ± 0.71</td><td>3.8 (94)</td><td>3.8 (94)</td><td>2.9 (73)</td><td>35.8 (823)</td><td>37.8 (788)</td><td>10.6 (266)</td></tr><tr><td>LLaMat3</td><td>15.4</td><td>82.7</td><td>96.6</td><td><b>84.2</b></td><td>15.2</td><td>13.8</td><td>0.74 ± 2.06</td><td>1.68 ± 2.48</td><td>1.10 ± 1.10</td><td>0.5 (12)</td><td>0.5 (12)</td><td>0.4 (11)</td><td>2.1 (52)</td><td>2.0 (50)</td><td>0.8 (21)</td></tr><tr><td>SymmCD</td><td>73.4</td><td>95.9</td><td>16.8</td><td><b>100.0</b></td><td>73.0</td><td>61.3</td><td>-0.02 ± 1.22</td><td>0.85 ± 1.07</td><td>0.87 ± 1.10</td><td>4.3 (108)</td><td>4.3 (107)</td><td>2.2 (54)</td><td>17.8 (446)</td><td>17.6 (440)</td><td>9.0 (225)</td></tr><tr><td>Alexandria</td><td>93.4</td><td>93.4</td><td>99.0</td><td><b>93.4</b></td><td>93.4</td><td>92.4</td><td>-0.18 ± 0.94</td><td>0.41 ± 0.62</td><td>0.15 ± 0.36</td><td>10.7 (268)</td><td>10.7 (268)</td><td>10.5 (263)</td><td>24.6 (616)</td><td>24.6 (616)</td><td>24.0 (599)</td></tr><tr><td>OQMD</td><td><b>96.8</b></td><td>96.8</td><td>99.4</td><td><b>96.4</b></td><td>99.5</td><td><b>94.2</b></td><td>-0.10 ± 0.88</td><td>0.36 ± 0.47</td><td>0.14 ± 0.28</td><td>17.5 (438)</td><td>17.5 (437)</td><td>16.6 (415)</td><td>19.9 (495)</td><td>19.8 (495)</td><td>18.2 (456)</td></tr><tr><td>AFLOW</td><td>91.4</td><td>91.4</td><td><b>100.0</b></td><td><b>100.0</b></td><td>87.1</td><td>57.0</td><td>-0.19 ± 0.86</td><td>0.33 ± 0.42</td><td><b>0.12 ± 0.44</b></td><td>16.0 (401)</td><td>15.0 (376)</td><td>3.0 (74)</td><td>25.2 (631)</td><td>22.8 (571)</td><td>4.2 (105)</td></tr><tr><td>Experimental</td><td>98.0</td><td>98.0</td><td>100.0</td><td><b>100.0</b></td><td>85.6</td><td>58.6</td><td>-1.02 ± 0.97</td><td>0.06 ± 0.46</td><td>0.20 ± 0.36</td><td>40.5 (1013)</td><td>35.4 (885)</td><td>25.0 (624)</td><td>41.5 (1038)</td><td>36.6 (914)</td><td>13.0 (326)</td></tr></tbody></table>

<table><caption>**Table 12** Model evaluation metrics calculated using MLIP ensemble with MP-20 as the reference set: Distribution, Diversity, and HHI. All models provide 2500 structures except Crystalformer (1000 structures). Submissions are organized into four categories separated by horizontal lines: (1) models that incorporate relaxation as part of their generation pipeline (MatterGen, PLaID++, WyFormer, WyFormer-DFT), (2) other generative models, (3) samples from real-world datasets that contribute to LeMat-Bulk (Alexandria, OQMD) as baselines, and (4) samples from the AFLOW dataset and MP-Exp structures. Best values are shown in **bold**, second-best values are **underlined**. Arrows indicate whether higher (↑) or lower (↓) values are better.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">Distribution</th><th colspan="4">Diversity</th><th colspan="2">HHI</th></tr><tr><th>JS ↓</th><th>MMD ↓</th><th>FID ↓</th><th>ElemDiv ↑</th><th>SGDiv ↑</th><th>SizeDiv ↑</th><th>SiteDiv ↑</th><th>Prod ↓</th><th>Res ↓</th></tr></thead><tbody><tr><td>MatterGen</td><td>0.43</td><td><b>0.000</b></td><td><b>14.38</b></td><td>0.64</td><td>0.14</td><td>0.21</td><td>12.78</td><td>3.52</td><td>2.72</td></tr><tr><td>PLaID++</td><td>0.48</td><td>0.049</td><td>15.48</td><td>0.68</td><td>0.26</td><td>0.21</td><td>6.01</td><td>5.23</td><td>3.36</td></tr><tr><td>WyFormer</td><td>0.42</td><td><b>0.001</b></td><td>15.03</td><td><b>0.74</b></td><td><b>0.50</b></td><td>0.27</td><td><b>23.84</b></td><td>3.55</td><td>2.73</td></tr><tr><td>WyFormer-DFT</td><td>0.42</td><td>0.002</td><td>21.23</td><td>0.73</td><td>0.49</td><td>0.26</td><td>23.56</td><td>3.49</td><td>2.74</td></tr><tr><td>ADiT</td><td>0.43</td><td><b>0.001</b></td><td><b>14.41</b></td><td>0.71</td><td>0.03</td><td>0.24</td><td>14.18</td><td>3.52</td><td>2.78</td></tr><tr><td>Crystal-GFN</td><td>0.56</td><td>0.098</td><td>50.20</td><td><b>0.75</b></td><td>0.29</td><td><b>0.32</b></td><td><b>32.06</b></td><td><b>3.48</b></td><td><b>2.34</b></td></tr><tr><td>Crystalformer</td><td><b>0.41</b></td><td>0.002</td><td>16.60</td><td>0.71</td><td>0.35</td><td><b>0.31</b></td><td>18.59</td><td>3.78</td><td>2.82</td></tr><tr><td>DiffCSP</td><td>0.44</td><td>0.002</td><td>14.73</td><td>0.73</td><td>0.13</td><td>0.24</td><td>14.42</td><td><b>3.29</b></td><td><b>2.60</b></td></tr><tr><td>DiffCSP++</td><td>0.42</td><td>0.003</td><td>14.97</td><td>0.72</td><td><b>0.51</b></td><td>0.27</td><td>20.84</td><td>3.56</td><td>2.77</td></tr><tr><td>LLaMat2</td><td><b>0.41</b></td><td>0.003</td><td>14.91</td><td>0.71</td><td>0.25</td><td>0.24</td><td>9.86</td><td>3.94</td><td>2.86</td></tr><tr><td>LLaMat3</td><td>0.45</td><td>0.012</td><td>27.67</td><td>0.70</td><td>0.10</td><td>0.30</td><td>13.15</td><td>3.83</td><td>2.81</td></tr><tr><td>SymmCD</td><td><b>0.41</b></td><td><b>0.001</b></td><td>16.33</td><td>0.73</td><td>0.49</td><td>0.27</td><td>19.45</td><td>3.54</td><td>2.74</td></tr><tr><td>Alexandria</td><td><b>0.38</b></td><td>0.004</td><td>14.82</td><td>0.71</td><td><b>0.50</b></td><td>0.25</td><td>13.50</td><td>4.02</td><td>3.33</td></tr><tr><td>OQMD</td><td>0.43</td><td>0.014</td><td>15.32</td><td>0.63</td><td>0.48</td><td>0.24</td><td>12.57</td><td>4.22</td><td>3.29</td></tr><tr><td>AFLOW</td><td>0.44</td><td>0.003</td><td>15.34</td><td>0.69</td><td>0.46</td><td>0.25</td><td>10.07</td><td>4.12</td><td>3.22</td></tr><tr><td>MP-Exp</td><td>0.49</td><td>0.011</td><td>14.73</td><td>0.72</td><td>0.71</td><td>0.28</td><td>54.79</td><td>2.78</td><td>2.27</td></tr></tbody></table>

<table><caption>**Table 13** Model evaluation metrics calculated using uma with MP-20 as the reference set: Validity, Energy, Stability, and Metastability. All models provide 2500 structures except Crystalformer (1000 structures). Submissions are organized into four categories separated by horizontal lines: (1) models that incorporate relaxation as part of their generation pipeline (MatterGen, PLaID++, WyFormer, WyFormer-DFT), (2) other generative models, (3) samples from real-world datasets that contribute to LeMat-Bulk (Alexandria, OQMD) as baselines, and (4) samples from the AFLOW dataset and MP-Exp structures. Best values are shown in **bold**, second-best values are **underlined**. Arrows indicate whether higher (↑) or lower (↓) values are better.</caption><thead><tr><th rowspan="2">Model</th><th colspan="4">Validity</th><th colspan="2">Unique% ↑</th><th colspan="2">Energy-based</th><th colspan="3">Stability</th><th colspan="2">Metastability</th></tr><tr><th>Valid% ↑</th><th>CN% ↑</th><th>MinDist% ↑</th><th>PhysPlau% ↑</th><th>FormE (Std) ↓</th><th>Novel% ↑</th><th>E<sub>best</sub> (Std) ↓</th><th>RMSD (Std) ↓</th><th>Stable% ↑</th><th>U-Stable% ↑</th><th>SUN% ↑</th><th>Metastable% ↑</th><th>U-Meta% ↑</th><th>MSUN% ↑</th></tr></thead><tbody><tr><td>MatterGen</td><td>95.7</td><td>95.9</td><td>99.8</td><td><b>100.0</b></td><td><b>95.1</b></td><td>83.1</td><td>-0.69 ± 0.80</td><td>0.17 ± 0.13</td><td>0.40 ± 0.60</td><td>5.4 (136)</td><td>5.4 (135)</td><td>3.5 (88)</td><td>35.4 (835)</td><td>33.0 (824)</td><td>23.1 (578)</td></tr><tr><td>PLaID++</td><td>96.0</td><td>96.0</td><td>96.2</td><td><b>96.2</b></td><td>77.8</td><td>63.9</td><td>-0.48 ± 0.45</td><td><b>0.07 ± 0.17</b></td><td>0.12 ± 0.32</td><td><b>29.6 (741)</b></td><td><b>20.6 (514)</b></td><td>12.0 (299)</td><td><b>45.6 (1141)</b></td><td><b>37.4 (935)</b></td><td>23.1 (578)</td></tr><tr><td>WyFormer</td><td>93.4</td><td>95.2</td><td>98.2</td><td><b>100.0</b></td><td>93.0</td><td>80.4</td><td>-0.41 ± 0.97</td><td>0.49 ± 0.54</td><td>0.80 ± 1.05</td><td>2.4 (61)</td><td>2.4 (60)</td><td>1.9 (47)</td><td>16.2 (405)</td><td>16.0 (401)</td><td>6.7 (168)</td></tr><tr><td>WyFormer-DFT</td><td>95.2</td><td>95.2</td><td><b>100.0</b></td><td><b>100.0</b></td><td>95.0</td><td>83.3</td><td>-0.65 ± 1.07</td><td>0.26 ± 0.66</td><td>0.39 ± 0.60</td><td>7.5 (187)</td><td>7.4 (184)</td><td>3.8 (94)</td><td>24.4 (611)</td><td>24.4 (609)</td><td>16.4 (409)</td></tr><tr><td>ADiT</td><td>90.6</td><td><b>99.0</b></td><td>91.5</td><td>100.0</td><td>87.8</td><td>31.3</td><td>-0.72 ± 0.98</td><td>0.32 ± 0.45</td><td>0.38 ± 0.41</td><td>1.0 (26)</td><td>1.0 (26)</td><td>0.6 (16)</td><td>38.2 (954)</td><td>36.6 (915)</td><td>3.0 (76)</td></tr><tr><td>Crystal-GFN</td><td>51.7</td><td>58.3</td><td>96.1</td><td>99.3</td><td>51.7</td><td>51.7</td><td>-0.56 ± 1.46</td><td>2.38 ± 2.48</td><td>1.86 ± 1.10</td><td>5.4 (134)</td><td>5.4 (134)</td><td>5.4 (134)</td><td>-0.8 (19)</td><td>0.8 (19)</td><td>0.8 (19)</td></tr><tr><td>Crystalformer</td><td>69.9</td><td>96.8</td><td>72.9</td><td>99.9</td><td>69.4</td><td>44.2</td><td>-0.15 ± 1.48</td><td>0.69 ± 1.09</td><td>0.68 ± 1.24</td><td>5.1 (51)</td><td>5.1 (51)</td><td>3.2 (32)</td><td>27.9 (279)</td><td>27.5 (279)</td><td>9.5 (95)</td></tr><tr><td>DiffCSP</td><td>95.7</td><td>96.2</td><td>99.5</td><td>100.0</td><td>94.8</td><td>81.9</td><td>-0.62 ± 0.98</td><td>0.26 ± 0.58</td><td>0.59 ± 0.63</td><td>6.4 (160)</td><td>6.2 (155)</td><td>3.8 (95)</td><td>28.8 (719)</td><td>28.2 (704)</td><td>17.8 (445)</td></tr><tr><td>DiffCSP++</td><td>95.3</td><td>95.5</td><td>99.7</td><td>100.0</td><td>95.1</td><td>81.1</td><td>-0.50 ± 0.89</td><td>0.40 ± 0.45</td><td>0.69 ± 0.88</td><td>4.7 (118)</td><td>4.7 (118)</td><td>3.2 (79)</td><td>25.6 (640)</td><td>25.4 (635)</td><td>13.6 (311)</td></tr><tr><td>LLaMat2</td><td>84.4</td><td>97.0</td><td>97.1</td><td>99.6</td><td>81.4</td><td>53.0</td><td>-0.45 ± 1.03</td><td>0.43 ± 0.72</td><td>0.54 ± 0.71</td><td>4.5 (116)</td><td>4.5 (112)</td><td>3.2 (86)</td><td>33.3 (832)</td><td>33.7 (782)</td><td>9.7 (243)</td></tr><tr><td>LLaMat3</td><td>15.4</td><td>82.5</td><td>96.6</td><td>100.0</td><td>84.2</td><td>15.2</td><td>0.79 ± 2.73</td><td>1.72 ± 2.52</td><td>1.10 ± 1.26</td><td>0.4 (10)</td><td>0.4 (10)</td><td>0.4 (9)</td><td>2.2 (55)</td><td>2.2 (54)</td><td>1.0 (24)</td></tr><tr><td>SymmCD</td><td>73.4</td><td>95.9</td><td>76.4</td><td><b>100.0</b></td><td>73.0</td><td>61.3</td><td>-0.00 ± 1.23</td><td>0.87 ± 1.08</td><td>0.85 ± 1.13</td><td>3.7 (92)</td><td>3.6 (91)</td><td>0.4 (9)</td><td>18.2 (456)</td><td>18.0 (451)</td><td>8.8 (219)</td></tr><tr><td>Alexandria</td><td>93.4</td><td>93.4</td><td>99.0</td><td><b>93.4</b></td><td>92.4</td><td></td><td>-0.16 ± 0.97</td><td>0.42 ± 0.64</td><td><b>0.12 ± 0.40</b></td><td>9.9 (247)</td><td>9.9 (247)</td><td>9.8 (245)</td><td>24.5 (612)</td><td>24.5 (612)</td><td><b>23.7 (592)</b></td></tr><tr><td>OQMD</td><td><b>96.8</b></td><td>96.8</td><td>99.4</td><td><b>96.4</b></td><td><b>94.2</b></td><td></td><td>-0.21 ± 0.89</td><td>0.37 ± 0.47</td><td><b>0.13 ± 0.29</b></td><td>17.2 (429)</td><td>17.1 (428)</td><td><b>16.3 (407)</b></td><td>19.6 (491)</td><td>19.6 (489)</td><td>18.0 (449)</td></tr><tr><td>AFLOW</td><td>91.4</td><td>91.4</td><td><b>100.0</b></td><td><b>100.0</b></td><td>87.1</td><td>57.0</td><td>-0.18 ± 0.86</td><td>0.33 ± 0.42</td><td><b>0.11 ± 0.40</b></td><td>12.9 (322)</td><td>12.1 (302)</td><td>2.6 (66)</td><td>28.0 (699)</td><td>25.3 (632)</td><td>4.5 (112)</td></tr><tr><td>MP-Exp</td><td>98.0</td><td>98.0</td><td>100.0</td><td><b>100.0</b></td><td>85.6</td><td>58.6</td><td>-0.99 ± 1.36</td><td>0.08 ± 1.06</td><td>0.19 ± 0.39</td><td>35.0 (874)</td><td>30.5 (763)</td><td>23.8 (594)</td><td>46.7 (1168)</td><td>41.1 (1027)</td><td>14.0 (351)</td></tr></tbody></table>
<table><caption>**Table 14** Model evaluation metrics calculated using uma with MP-20 as the reference set: Distribution, Diversity, and HHI. All models provide 2500 structures except Crystalformer (1000 structures). Submissions are organized into four categories separated by horizontal lines: (1) models that incorporate relaxation as part of their generation pipeline (MatterGen, PLaID++, WyFormer, WyFormer-DFT), (2) other generative models, (3) samples from real-world datasets that contribute to LeMat-Bulk (Alexandria, OQMD) as baselines, and (4) samples from the AFLOW dataset and MP-Exp structures. Best values are shown in **bold**, second-best values are **underlined**. Arrows indicate whether higher (↑) or lower (↓) values are better.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">Distribution</th><th colspan="4">Diversity</th><th colspan="2">HHI</th></tr><tr><th>JS ↓</th><th>MMD ↓</th><th>FID ↓</th><th>ElemDiv ↑</th><th>SGDiv ↑</th><th>SizeDiv ↑</th><th>SiteDiv ↑</th><th>Prod ↓</th><th>Res ↓</th></tr></thead><tbody><tr><td>MatterGen</td><td>0.43</td><td><strong>0.000</strong></td><td>0.18</td><td>0.64</td><td>0.14</td><td>0.24</td><td>12.78</td><td>3.52</td><td>2.72</td></tr><tr><td>PLaID++</td><td>0.48</td><td>0.049</td><td>0.30</td><td>0.68</td><td>0.26</td><td>0.21</td><td>6.01</td><td>5.23</td><td>3.36</td></tr><tr><td>WyFormer</td><td>0.42</td><td>0.001</td><td>1.24</td><td><strong>0.74</strong></td><td><strong>0.50</strong></td><td>0.27</td><td><strong>23.84</strong></td><td>3.55</td><td>2.73</td></tr><tr><td>WyFormer-DFT</td><td>0.42</td><td>0.002</td><td><strong>0.18</strong></td><td>0.73</td><td>0.49</td><td>0.26</td><td>23.56</td><td>3.49</td><td>2.74</td></tr><tr><td>ADI</td><td>0.43</td><td>0.001</td><td>0.37</td><td>0.71</td><td>0.03</td><td>0.24</td><td>14.18</td><td>3.52</td><td>2.78</td></tr><tr><td>Crystal-GFN</td><td>0.56</td><td>0.098</td><td>1.70</td><td><strong>0.75</strong></td><td>0.29</td><td><strong>0.32</strong></td><td><strong>32.06</strong></td><td>3.48</td><td><strong>2.34</strong></td></tr><tr><td>Crystalformer</td><td><strong>0.41</strong></td><td>0.002</td><td>2.24</td><td>0.71</td><td>0.35</td><td><strong>0.31</strong></td><td>18.59</td><td>3.78</td><td>2.82</td></tr><tr><td>DiffCSP</td><td>0.44</td><td>0.002</td><td>0.60</td><td>0.73</td><td>0.13</td><td>0.24</td><td>14.42</td><td><strong>3.29</strong></td><td><strong>2.60</strong></td></tr><tr><td>DiffCSP++</td><td>0.42</td><td>0.003</td><td><strong>0.09</strong></td><td>0.72</td><td><strong>0.51</strong></td><td>0.27</td><td>20.84</td><td>3.56</td><td>2.77</td></tr><tr><td>LLaMat2</td><td>0.41</td><td>0.003</td><td>0.66</td><td>0.71</td><td>0.25</td><td>0.24</td><td>9.86</td><td>3.94</td><td>2.86</td></tr><tr><td>LLaMat3</td><td>0.45</td><td>0.012</td><td>9.31</td><td>0.70</td><td>0.10</td><td>0.30</td><td>13.15</td><td>3.83</td><td>2.81</td></tr><tr><td>SymmCD</td><td>0.41</td><td><strong>0.001</strong></td><td>3.34</td><td>0.73</td><td>0.49</td><td>0.27</td><td>19.45</td><td>3.54</td><td>2.74</td></tr><tr><td>Alexandria</td><td><strong>0.38</strong></td><td>0.004</td><td>0.97</td><td>0.71</td><td>0.50</td><td>0.25</td><td>13.50</td><td>4.02</td><td>3.33</td></tr><tr><td>OQMD</td><td>0.43</td><td>0.014</td><td>0.55</td><td>0.63</td><td>0.48</td><td>0.24</td><td>12.57</td><td>4.22</td><td>3.29</td></tr><tr><td>AFLOW</td><td>0.44</td><td>0.003</td><td>0.55</td><td>0.69</td><td>0.46</td><td>0.25</td><td>10.07</td><td>4.12</td><td>3.22</td></tr><tr><td>MP-Exp</td><td>0.49</td><td>0.011</td><td>0.16</td><td>0.72</td><td>0.71</td><td>0.28</td><td>54.79</td><td>2.78</td><td>2.27</td></tr></tbody></table>

<table><caption>**Table 15** Model evaluation metrics calculated using uma with MP-20 as the reference set **after relaxation**: Validity, Energy, Stability, and Metastability. All models provide 2500 structures except Crystalformer (1000 structures). Submissions are organized into four categories separated by horizontal lines: (1) models that incorporate relaxation as part of their generation pipeline (MatterGen, PLaID++, WyFormer, WyFormer-DFT), (2) other generative models, (3) samples from real-world datasets that contribute to LeMat-Bulk (Alexandria, OQMD) as baselines, and (4) samples from the AFLOW dataset and MP-Exp structures. Best values are shown in **bold**, second-best values are **underlined**. Arrows indicate whether higher (↑) or lower (↓) values are better.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">Validity</th><th colspan="2">Unique% ↑</th><th colspan="2">Energy-based</th><th colspan="3">Stability</th><th colspan="3">Metastability</th></tr><tr><th>Valid% ↑</th><th>CN% ↑</th><th>MinDist% ↑</th><th>PhysPlau% ↑</th><th>Novel% ↑</th><th>FormE (Std) ↓</th><th>E<sub>best</sub> (Std) ↓</th><th>RMSD (Std) ↓</th><th>Stable% ↑</th><th>U-Stable% ↑</th><th>SUN% ↑</th><th>Metastable% ↑</th><th>U-Meta% ↑</th><th>MSUN% ↑</th></tr></thead><tbody><tr><td>MatterGen</td><td>95.8</td><td>95.8</td><td>100.0</td><td>100.0</td><td>95.3</td><td>-0.74 ± 0.80</td><td>0.12 ± 0.14</td><td>0.00 ± 0.05</td><td>10.2 (255)</td><td>10.1 (252)</td><td>5.8 (145)</td><td>40.2 (1006)</td><td>39.8 (995)</td><td><strong>31.9 (797)</strong></td></tr><tr><td>PLaID++</td><td>96.0</td><td>96.0</td><td>96.2</td><td>96.2</td><td>77.7</td><td>-0.49 ± 0.47</td><td>0.06 ± 0.16</td><td>0.00 ± 0.01</td><td><strong>34.3 (857)</strong></td><td>23.4 (586)</td><td>12.8 (320)</td><td>42.3 (1057)</td><td>35.4 (886)</td><td>23.4 (585)</td></tr><tr><td>WyFormer</td><td>95.0</td><td>95.1</td><td>99.8</td><td>99.9</td><td>94.6</td><td>-0.67 ± 0.90</td><td>0.22 ± 0.32</td><td>0.00 ± 0.02</td><td>9.6 (239)</td><td>9.3 (233)</td><td>4.8 (120)</td><td>26.0 (664)</td><td>26.3 (658)</td><td>18.1 (452)</td></tr><tr><td>WyFormer-DFT</td><td>95.2</td><td>95.2</td><td>100.0</td><td>100.0</td><td>94.9</td><td>-0.71 ± 0.90</td><td>0.20 ± 0.28</td><td>0.04 ± 0.21</td><td>9.4 (236)</td><td>9.3 (232)</td><td>4.7 (118)</td><td>26.6 (666)</td><td>26.6 (664)</td><td>19.4 (486)</td></tr><tr><td>ADI</td><td><strong>98.9</strong></td><td><strong>99.0</strong></td><td>99.9</td><td>100.0</td><td>95.1</td><td>-1.04 ± 0.90</td><td><strong>0.04 ± 0.09</strong></td><td>0.00 ± 0.03</td><td>33.9 (848)</td><td><strong>32.2 (805)</strong></td><td>5.6 (140)</td><td><strong>50.0 (1251)</strong></td><td>3.4 (84)</td><td>13.4 (335)</td></tr><tr><td>Crystal-GFN</td><td>55.3</td><td>57.4</td><td>98.0</td><td>95.7</td><td>55.3</td><td>-2.54 ± 0.88</td><td>0.41 ± 0.90</td><td>0.72 ± 0.75</td><td>15.6 (389)</td><td>15.6 (389)</td><td>15.6 (389)</td><td>34.7 (347)</td><td>34.6 (346)</td><td>18.3 (183)</td></tr><tr><td>Crystalformer</td><td>92.0</td><td>93.8</td><td>95.6</td><td>97.2</td><td>91.5</td><td>-0.74 ± 0.85</td><td>0.15 ± 0.25</td><td>0.00 ± 0.02</td><td>17.9 (179)</td><td>17.6 (176)</td><td>6.9 (69)</td><td>34.7 (347)</td><td>38.0 (965)</td><td>30.4 (761)</td></tr><tr><td>DiffCSP</td><td>96.2</td><td>96.2</td><td><strong>100.0</strong></td><td><strong>100.0</strong></td><td>94.8</td><td>-0.76 ± 0.82</td><td>0.12 ± 0.18</td><td>0.07 ± 0.28</td><td>13.4 (336)</td><td>12.8 (321)</td><td>7.1 (178)</td><td>38.6 (965)</td><td>38.0 (951)</td><td>30.4 (761)</td></tr><tr><td>DiffCSP++</td><td>95.5</td><td>95.5</td><td>100.0</td><td>100.0</td><td>95.2</td><td>-0.72 ± 0.86</td><td>0.18 ± 0.23</td><td>0.04 ± 0.21</td><td>12.2 (304)</td><td>12.1 (302)</td><td>6.8 (171)</td><td>29.1 (728)</td><td>29.0 (725)</td><td>39.9 (498)</td></tr><tr><td>LLaMat2</td><td><strong>97.0</strong></td><td><strong>97.0</strong></td><td>99.9</td><td>100.0</td><td>93.8</td><td>-0.77 ± 0.85</td><td>0.11 ± 0.21</td><td>0.03 ± 0.20</td><td>23.9 (597)</td><td>22.6 (566)</td><td>8.5 (213)</td><td>39.2 (980)</td><td>37.6 (939)</td><td>20.4 (510)</td></tr><tr><td>LLaMat3</td><td>69.8</td><td>70.5</td><td>82.2</td><td>83.3</td><td>69.6</td><td>-0.38 ± 0.71</td><td>0.24 ± 0.27</td><td>0.17 ± 0.46</td><td>2.5 (63)</td><td>2.5 (62)</td><td>1.7 (43)</td><td>16.0 (399)</td><td>15.9 (398)</td><td>15.0 (374)</td></tr><tr><td>SymmCD</td><td>95.6</td><td>95.4</td><td>99.7</td><td>100.0</td><td>95.1</td><td>-0.67 ± 0.85</td><td>0.25 ± 0.27</td><td>0.00 ± 0.03</td><td>12.5 (313)</td><td>12.3 (308)</td><td>6.0 (151)</td><td>27.4 (686)</td><td>27.3 (683)</td><td>18.4 (461)</td></tr><tr><td>Alexandria</td><td>93.4</td><td>93.4</td><td>99.0</td><td>99.0</td><td>93.4</td><td>-0.18 ± 0.95</td><td>0.40 ± 0.61</td><td>0.00 ± 0.02</td><td>10.5 (263)</td><td>10.5 (263)</td><td>10.2 (258)</td><td>24.8 (621)</td><td>24.8 (621)</td><td>24.2 (604)</td></tr><tr><td>OQMD</td><td>96.8</td><td>96.8</td><td>99.5</td><td>99.5</td><td><strong>96.4</strong></td><td>-0.22 ± 0.90</td><td>0.35 ± 0.47</td><td>0.00 ± 0.02</td><td>15.6 (466)</td><td>18.5 (463)</td><td><strong>17.3 (432)</strong></td><td>19.3 (497)</td><td>19.9 (497)</td><td>18.6 (466)</td></tr><tr><td>AFLOW</td><td>91.4</td><td>91.4</td><td>100.0</td><td>100.0</td><td>87.1</td><td>-0.18 ± 0.87</td><td>0.32 ± 0.42</td><td><strong>0.00 ± 0.00</strong></td><td>20.0 (499)</td><td>18.5 (462)</td><td>3.2 (79)</td><td>22.0 (549)</td><td>19.7 (492)</td><td>4.3 (108)</td></tr><tr><td>MP-Exp</td><td>98.0</td><td>98.0</td><td>100.0</td><td>100.0</td><td>85.9</td><td>-1.03 ± 0.89</td><td>0.04 ± 0.19</td><td>0.00 ± 0.02</td><td>50.3 (1257)</td><td>43.8 (1096)</td><td>26.0 (651)</td><td>32.9 (822)</td><td>29.2 (730)</td><td>13.3 (332)</td></tr></tbody></table>
<figure><img src="image_19.png" /><figcaption>**Figure 16** Generative models evaluation across all metric categories. Top row: MP-20 reference. Middle row: LeMat-Bulk pre-relaxation. Bottom row: LeMat-Bulk post-relaxation. Columns from left to right: SUN Metrics, MSUN Metrics, Energy & Stability, Distribution & Diversity. All rows use consistent scaling for direct comparison. Models include relaxation-based generators (MatterGen, PLaID++, WyFormer, WyFormer-DFT) and diffusion/autoregressive approaches.</figcaption></figure>
<figure><img src="image_20.png" /><figcaption>**Figure 17** Dataset baselines evaluation across all metric categories. Top row: MP-20 reference. Middle row: LeMat-Bulk pre-relaxation. Bottom row: LeMat-Bulk post-relaxation. Columns from left to right: SUN Metrics, MSUN Metrics, Energy & Stability, Distribution & Diversity. All rows use consistent scaling for direct comparison. Baselines include Alexandria, OQMD, AFLOW, and MP-Exp.</figcaption></figure>