M$^2$Hub: Unlocking the Potential of Machine Learning for Materials Discovery
Authors: Yuanqi Du
Yingheng Wang
Yining Huang
Jianan Canal Li
Yanqiao Zhu
Tian Xie
Chenru Duan
John Gregoire
Carla P Gomes
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Materials Discovery

Abstract: We introduce M$^2$Hub, a toolkit for advancing machine learning in materials discovery. Machine learning has achieved remarkable progress in modeling molecular structures, especially biomolecules for drug discovery. However, the development of machine learning approaches for modeling materials structures lag behind, which is partly due to the lack of an integrated platform that enables access to diverse tasks for materials discovery. To bridge this gap, M$^2$Hub will enable easy access to materials discovery tasks, datasets, machine learning methods, evaluations, and benchmark results that cover the entire workflow. Specifically, the first release of M$^2$Hub focuses on three key stages in materials discovery: virtual screening, inverse design, and molecular simulation, including 9 datasets that covers 6 types of materials with 56 tasks across 8 types of material properties. We further provide 2 synthetic datasets for the purpose of generative tasks on materials. In addition to random data splits, we also provide 3 additional data partitions to reflect the real-world materials discovery scenarios. State-of-the-art machine learning methods (including those are suitable for materials structures but never compared in the literature) are benchmarked on representative tasks. Our codes and library are publicly available at \url{https://github.com/yuanqidu/M2Hub}.

MuHub: Unlocking the Potential of Machine Learning for Materials Discovery
Yuanqi Du
Yingheng Wang'
Yining Huang² Jianan Canal Li
Yanqiao Zhu
Tian Xie
Chenru Duan
John M. Gregoire" Carla P. Gomes'
Cornell Northwestern 3 UCB 4 *UCLA
MSR A14Science
Microsoft Quantum Caltech Eqqual Contribution
Abstract
We introduce Mubu,, toolkit for advancing machine learning in materials dis- covery. Machine learning has achieved remarkable progress modeling molecular structures, especially biomolecules for drug discovery. However, the development of machine learning approaches for modeling materials structures lag behind, which is partly due the lack of an integrated platform that enables access diverse tasks for materials discovery. To bridge this gap, HuHub will enable easy access to materials discovery tasks, datasets, machine learning methods, evaluations, and benchmark results that cover the entire workflow. Specifically, the first release of Hub focuses on three key stages materials discovery: virtual screening, inverse design, and molecular simulation, including 9 datasets that covers 6 types of materials with 56 tasks across types of material properties. We further provide 2 synthetic datasets for the purpose of generative tasks on materials. In addition to random data splits, we also provide additional data partitions to reflect the real-world materials discovery scenarios. State-of-the-art machine learning meth- ods (including those are suitable for materials structures but never compared the literature) benchmarked representative tasks. Our codes and library are publicly available at lttps: //tggguttt///t/// com com/yuanqidd//H2Hu
1 Introduction
With the methodological advancements in machine learning, an increasing number of machine learning models have been developed and applied solve scientific problems, from simulating molecular systems with millions of particles to predicting accurate protein structures Zhhang et al. (2018), Jumper al. [2021]. The primary focus of machine learning the chemical sciences has remained in the domain of molecular structures. molculess including small molecules, proteins, RNAs, etc. Atz al. 202T Rives et al.] [021]. Townshend al. (202T]. However, materials constitute large portion of the chemical space which have been significantly less studied, especially in the machine learning community. Among scientific problems, materials discovery plays a vital role in driving innovations and progress across various fields spanning energy, electronics, healthcare, and sustainability Sanchez-Lengelineg and Aspuru-Guzik [2018), Gomes et al.) [202]]. However, traditional trial-and-error approach to materials discovery is expensive and time-consuming. Over decades, classical machine learning methods have already been widely applied in assisting materials discovery, Schmmidt et al. [2019] yet the impact of machine learning for solid state materials lags behind efficacy in other areas of chemical science.
Witnessing the success of machine learning in solving grand challenges in science Wang et al. 2018) Jumper al. [2021), one of the key ingredients is the infrastructure that supports the machine learning community to build the machine learning workflow: data preparation/processing, model development, performance evaluation, and model improvement based on the evaluation feedback. While effort
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

2 Hub
WM Candidutes
t$
Simulation
DOS
Virtaa Screening
Generative ModeliiggCommbinttriiaal Optimization
Design
@ Machine Learning Forcefield Pottern Demixing
Symthesis
2
Active Reiiringgeeinoorrmmeentt Learning
Function
Database
Representation Learning on Materials
Experiment Vilidation
e Generative Sootllnng/Serrchh/aapinngg
Fendlass coop
Fendhool Loop
Figure 1: Hub: Materials discovery meets Machine learning. A-F on the left figure demonstrates machine learning approaches used in each stage of the materials discovery pipeline on the right figure (dashed lines denote currently unavailable experiment-relate tasks).
has been made to make materials datasets available to the machine learning community Blaiszik al. 2019]. Dunn al. (2020), Clement al. 2020), Qayyum al. [022 Durdy et al. [2023), existing work mainly focus on providing data servers that allow the users to query materials data and predefined benchmark sets. However, bridge the gap between the molecular and solid state materials, we identify the need for a unified platform to facilitate the development of machine learning approaches for materials discovery purpose, including (1) centralized data sources with diverse materials, property and task types, (2) clear problem formulations, (3) realistic problem settings (e.g. data split), and (4) appropriate benchmarks and transparent comparisons with prior methods.
In order to address the aforementioned challenges, establish Hub, which integrates and connects different machine learning building blocks the materials discovery (Fig. []. The cornerstone of H is benchmark that incorporates several key aspects: (i) it integrates three key tasks: virtual screening, molecular dynamics simulation, and inverse design, which are translated using machine learning constructs such materials representation learning, machine learning forcefield, and generative materials design; (ii) it is underpinned by curated set of 11 datasets that encompass types of materials with 56 tasks across 8 distinct material properties. In addition to the standard random split, we have included 3 realistic (out-of-distribution) data splits to enhance the robustness of model evaluation; (iii) a distinctive feature of our benchmark is the emphasis on the generative design of materials. For this, we provide machine learning formulations, evaluation metrics, and oracle functions to facilitate further research and development in this area; (iv) finally, our benchmarks evaluate not only the commonly used material representation learning methods, but also those designed for non-periodic molecular structures. These methods are applied to 13 representative tasks for material property prediction.
The flow of this paper is as follows: we introduce the background and related work developing machine learning methods for materials discovery in Sec. 2 we present the overview of Mub including problem formulation, dataset curation, data processing, evaluation and oracle function for inverse design in Sec. 3 in Sec. 4 we detail the implemented machine learning models, benchmarking results, observations, and insights emerging from the results.
2 Background
2.1 Materials Representation Learning
Material representation learning refers to representing materials structures in an expressive and machine-readable format for downstream studies, from property prediction to materials generation. Recent advances graph neural networks bring a wave of representing materials structures as graphs where nodes represent atoms and edges represent bonds or interactions. A line of works has been proposed to adapt this structured inductive bias into deep learning models Mondels.GCNNN\X and Grossman 2018) introduces multi-edge graph representation to capture periodicity, MEGNet Chen al.

---

2019 unifies molecules and crystal structures representations by graph neural networks representing each atom as node and interaction/bond between atoms edge. More recently, ALIGNN considers both atomistic and line graphs which externally capture angular information Choudhary and DeCost
[202T]. Equivariant graph neural networks (e.g., E3NN [eeiger and Smidtj [2022] have also been applied thanks to its roto-translational equivariance property. This task has also attracted increasing
interests through years Kaba and Ravanbakhsh [2022] Yan et al. [2022]. DAS et al. 20233) Lin et al.
2023
Das et al. 2023
2.2 Machine Learning Forcefields
Molecular dynamics simulation has become essential tool to understand the microscopic dynamical behaviors of molecular systems. It is worth noting there is a common trade-off between two popular diagrams, empirical forcefields and ab initio molecular dynamics. Empirical forcefields often rely on the hand-crafted parameters which are efficient yet inaccurate, while ab initio molecular dynamics rely on quantum-mechane mechanical calculations which are precise but inefficient. Inspired by recent advances of deep learning in automated parameters learning and transferability, large amount of works has been developed to learn machine learning forceficlds from between accuracy and efficiency. Specifically, it is expected to be more accurate than empirical forcefields and more efficient than seaatuu--mecanical calculations. Most representative work include DeepMD Zhang et al. 20188) ANI-1 [Smith et al. [2017), and NeugIP|Batzner Batzner et al. [2022].
mechhannca data to strike a balance
2.3 Materials Inverse Design
Designing new materials structures is long-standing challenge, often known as the inverse design problem, in materials science Du et al. [2022a], Manica et al. [023]] Before deep generative models have been applied to this problem, traditional computational methods often leverage quantum mechanical search over the possible stable materials including random search, evolutionary algorithm, element substitutions over known materials Glass et al. (206)] |Pickard and Needs [201T). HHutier eet al.] [011T. One line works leverages a learned force field to minimize the energy of the structure to reach stable material Deringer al. (2018). Later, deep generative models have been applied to this problem where the models aim to model the distribution of the known crystal structures and learn to sample new structures from it Early work leverage 3D voxel representation but it is nontrivial
to fit atom from the generated voxels Holfmann et al. [2019] Noh et al. [201]] Later work instead leverage atomic representation directly Zhao al. [202]]. G-SchNet Gebauer et al. 2019 instead proposes an auto-regressive model that generates each atom in a sequential way. Notably, it remains largely unexplored for efficient and controllable crystal structure generation with machine learning methods. A recent work Xie et al. (2022) builds upon the recent success of diffusion models on images and adapts them for crystal structure generation and optimization in an iterative refinement manner instead of one-shot or auto-regressive sequential generation.
Gebauer
Overview of MDHub
3.1 Problem Formulation
pobki
noviflor
Material representation. Material structures can be represented as a set of atoms M = (ma, m1, , mx) in 3D space with atom types H = (hg, h4...,, hy) E RRNKK x and atomic positions X (Io, .xN)
Materials
Machine Learning
Nxx where N denotes number of atoms and K de- notes number of atom types (C, O, Fe, AI, etc.). Most materials are crystal structures which pe- riodically repeat their unit cells in 3D space. In such cases, lattice vectors L = (4, 12, 1a) E R333 are utilized to describe the periodicity in 3D space. Note I, is not rotation invariant, 6 invariant lattice parameters (lengths of lattice parameters and angles between them) can also
Seech
Sctences
das
e00l
modta
Figure 2: Regular workflow for studying materials with machine learning approaches (green colored steps denote materials science expertise and blue colored steps denote machine learning expertise.
3

---

Table 1: Curated materials datasets in MDHub. Materials type and property type are detailed in Sec. 3.2. and Sec. 3.2.2 dim refers to data dimensionality; PBC refers to perodic boundary condition; method refers to how properties are obtained (sim short for simulation).
Duta MaiBench
Mascrials inorganic bulk
Dim
PBC
Prop.
Task type
tasks
A
Size
Method
scalar
312-152.752
28.00
Sim.
QMOF
metal-unganic framowurk
3D
(T.
eloc. energetic eloc.
scalar scalarrvocior
>20,000
113.67.:66
Sim
OC20
bulk adsochate interface
3D
(1: F)
660,081
73:26:300 96
Sim.
DMDB
organic crystal
3D
(E. T.
scalar
12,500
82.291226.5 55
Sim.
DFT3D
inorganic bulk
3D
(T. T)
elec htt.///hhhhtthemm......
sealar
20
55,722
9.59=8.04
Sim.
DFT2D
bulk
3D
stab. elec.
6:36
7.19:4.25
Sim.
EDOS-PDDSS
inorganic bulk
3D
dist.
9.50-8.45
Sim.
BIQQM
transition metal complex
3D
(E.FF F)
eloc.
86,665
65.99=27001
Sim
QM9
organic molecalls
3D
(F.E.
eloc.
scalar
12
~134,000
18.02=2944
Sim.
Carbon24
inorganic bulk
3D
(T. T:
NIA
N/A
10,153
9.21=3.58
NIA
Parov5
inorganic bulk
3D
T,T)
N/A
NIA
18,928
5,0000000
N/A
be used to describe the lattice (la, lc. le, a. B, r) Overall, material is denoted as M it is periodic and otherwise M = (H,
if
Material graph representation. Regardless of periodicity, materials structures can be naturally represented as graphs G = =VVE), where V is a set of vertices and m = (esy(k, k2 ka)li, E {1,2...., N). k k2, ky E Z} } is set of D edges (kg, kg. ks denotes the translation of the unit cell using lattice vector L, none if not periodic); H E R RNNR denotes the node features; XN x33 denotes the atomic positions; E E DDx denotes F' edge features (such bonds distances between each pair of nodes). The graph connections can be determined in multiple ways such as distance threshold, detailed in Sec. 3.3
Predictive tasks. Predictive tasks often have paired input material M and label Y, where given an input material M, we aim predict the expected label as p J.M. The label Y could be of various format such as binary, scalar, vector and distribution, detailed in Sec. (3.4] Both material representation learning and machine learning forcefield are considered as predictive tasks.
Generative tasks. Generative tasks could be divided into two parts: (1) distribution learning and (2) goal-oriented generation. Given set of J materials M {M.) distribution learning aims to learn the distribution of p(M) and sample new materials Mnew ~ p( (M). Goal-oriented generation aims to sample molecules fulfilling specific design targets (often defined by an oracle function f (M) such that M arg (P(M)) (M)
3.2 Data Description
We aim to curate set of datasets covering the diversity of material, property, and task types and data amounts which will enable a variety of perspectives for machine learning model developments.
3.2.1 Material types
Inorganic bulks refer to solid substances that lacks carbon -hydoonn bonds, that is, substances that are not organic compounds, such as metals, alloys, ceramics, and minerals. They are typically large-scale structures and are used in various applications, ranging from construction materials to electronics, due to their robustness and electrical/thermal conductivity.
Organic crystals are composed of carbon-based molecules arranged in a highly ordered pattern. They exhibit distinct molecular structures and are often used in the field of optoelectronics, such as organic light-emitting diodes (OLEDs), due to their unique optical properties. They can range in size from microscopic to macroscopic.
Organic molecules are individual carbon-based compounds that can be small in size, consisting of few atoms, or large, such as polymers. They have diverse chemical structures and are widely used in pharmaceuticals, plastics, and organic electronics due their flexibility in design and funetionality.
Bulk-adsorben interfaces refer to the boundary between a bulk material and an absorbed species, such as gases or liquids, on its surface. They are typically at the nanoscale and play a crucial role in various fields, including catalysis, gas sensing, transport, and energy storage, by influencing the interaction and reactivity of the absorbed species with the material.

---

Transition metal complexes are coordination compounds consisting of central transition metal atom(s) surrounded by ligands. They exhibit unique electronic and magnetic properties and are commonly used in catalysis, medicine, and material science due to their ability to undergo redox reactions and flexible and tunable coordination environment with organic ligands.
Metal-organd frameworks (MOFs) are crystalline materials composed of metal ions or clusters coordinated with organic linkers. MOF' poss high surface area and tunable porious structure, making them useful in applications such as gas storage, separations, and catalysis. They can range in size from microscopic crystals to bulk materials.
3.2.2 Property types
Electrical properties refer to the characteristics of material related to its ability to conduct or resist the flow of electric current. These properties include conductivity, resistivity, and dielectric constant, which determine how well a material can conduct or insulate against electrical charges.
Mechanical properties describe how a material behaves under applied forces loads. These properties include strength, stiffness, ductility, toughness, and elasticity. They determine how the material responds to stress, strain, and deformation, and are essential in understanding its structural integrity and performance.
Stability refers to a material's 's ability to maintain its properties and resist degradation over time. encompasses chemical stability (resistance to chemical reactions or corrosion), thermal stability (ability to withstand high temperatures), and mechanical stability (ability to resist physical changes or mechanical stress).
Optical properties pertain to material's interaction with light. These properties include absorption, reflection, transmission, and emission of light. Optical properties determine material's color, transparency, opacity, and light-emitting capabilities, and are crucial in fields such as optics, photonics, and display technologies.
Thermal properties describe how a material conducts, stores, and dissipates heat. These properties include thermal conductivity, specific heat capacity, thermal expansion coefficient, and thermal diffu- sivity. Thermal properties influence a material's 's ability to transfer heat, its response to temperature changes, and its behavior in thermal management applications.
Energetic properties are computational models that use machine learning algorithms to predict the behavior of materials at the atomic or molecular level. They employ large datasets to learn the relationships between atomic arrangements and energies, enabling the simulation and understanding of complex material systems.
Semiconductor properties refer to the electrical behavior of materials that exhibit an intermediate conductivity between conductors and insulators. These materials can be controlled to selectively allow or impede the flow of electrons, making them ideal for electronic devices. Semiconductor properties are characterized by parameters such as band gap, carrier mobility, and doping concentration, and are crucial in the design and functionality of transistors, diodes, and integrated circuits.
3.2.3 Datasets
Materials Project (MP) Jain et al. (0133 (license: CC-BY-4.0) is database that curates inorganic materials with computed properties including but not limited thermal, electrical, mechanical, etc.
MatBench Dunn et al. [2020) (MIT license) is a benchmark that provides a standardized framework for evaluating and comparing the performance of different machine learning models on various materials science tasks. It curates data from multiple sources with MP as a main source. However, they do not provide machine learning ready data preparation implemented machine learning models and workflow.
(license: CC-BY-4.0) is comprehensive database that focuses on metal-organic frameworks (MOFs) with quantum-chemical properties. The MOFs are optimized by DFT derived from both experimental and hypothetical MOF databases.
Quantum MOF Database (QMOF) Rosen et al. 2022
5

---

Organic Materials Database (OMDB) Borysov et al. [2017] (open access but no license specified)
a repository of organic materials. The properties are calculated using DFT for crystal structures contained in the COD database (in Appendix B additional data sources).
Joint Automated Repository for Various Integrated Simulations (JARVIS) Choudhary al. [2200] (license: GNU v3.0) is database that integrates materials data from various sources, including quantum mechanical caleulations, materials simulations, machine learning predictions and high- throughput databases. Our datasets DFT3D, DFT2D and EDOS-PDOS are all from JARVIS database.
Open Catalyst (OC) Chanussot al. 2021] (MIT license) is database focused catalytic materials. It includes three tasks: Structure to Energy and Forces (S2EF). Initial Structure to Relaxed Structure (IS2RS) and Relaxed Energy (IS2RE).
Transition Metal Quantum Materials Database (tmQM) Balcells and Skjelstad 2020 (license: CC BY-NC 4.0) is a comprehensive database focused on transition metal-based materials. It com- piles experimentally derived and computationally predicted data on the structure, composition, and electronic properties of transition metal compounds.
Quantum Machines 9 (QM9) Ramakrishnan et al. 20144 (open access but no license specified) comprises small organic molecules up to 9 heavy atoms with 12 quantum chemical properties.
Carbon24 Pickard 2020 (license: CC-BY-4.0) is synthetic dataset that includes materials made up by carbon atoms but with different structures obtained by ab initio random structure searching.
Perov5 Castelli et al. [2012a b)
(license: CC-BY-4.0) is a synthetic dataset that includes perovskite materials with the same structure but different compositions.
3.3 Machine Learning Ready Dataset Preparation
Raw data format The raw data format for both molecules and crystals is 3D structures and atomic types. Other features (such angular information) can be derived from them.
Machine learning ready data format As explained in Sec. 3.1 materials includes atomic types denote the atomic number of a given atom and are often converted to one-hot embeddings; atomic coordinates denote the positions of given atom and often need to be used careful if equivariance needs to be guaranteed; edge features denote information attached to each edge which often include bond types, interatomic distances, etc.
machine learning ready format for
Graph construction Three common graphs are constructed to represent the materials: mulli-graph construction is a common way represent materials as graphs which considers edges with repeated atoms (outside the unit cell) as multiple edges with the same atom; line graph construction for materials representation is first proposed in Choudhary and DeCost [202]] which a bond adjacency graph (i.c. line graph) is constructed to capture the bond and angular information.
Data split The test scenarios often out-of- -istrribbiittiionn of the training set. While previously common use data split random split, it is crucial to develop data splits that mimic the real scenarios: composition split (e.g. AxBy vs AzBy) refers to splitting the dataset with same materials compositions but varying ratios; system split (e.g. AB, AC vs ABC) refers to splitting the dataset with unseen materials systems; time split refers to splitting the dataset into training, validation and test set by the date when the materials are published. Note that time split is only available for MP dataset now as the publication information for each material structure is provided in MP.
3.4 Evaluations
3.4.1 Predictive Evaluations
The evaluation metrics for predictive tasks depend on the type of prediction label: (i) scalar value prediction:
common evaluation metrics include R², mean absolute error, and mean squared er- ror; (ii) classification: common evaluation metrics are accuracy and Area under the ROC Curve (AUC-ROC) score; (iii) vector/tensor value prediction: common evaluation metrics are similar to scalar value prediction, including R², mean squared error, and mean absolute error. However, the distance measurement between two vector or tensor values may need to take into account the symmetry, e.g., rotation invariance, such that the distance between the rotated crystal structure and

---

the original structure is zero: (iv) distribution prediction: common evaluation metrics include cosine similarity, KL divergence, Wasserstein distance, etc.
3.4.2 Generative Evaluations
Evaluating generative tasks has been a notoriously challenging problem in machine learning. The evaluation metrics can generally be divided into three categories: (i) reconstruction: This evaluates the performance of the generative methods in reconstructing the exact material in the training set. (ii) basic requirement: This assesses the minimum requirement for the generated materials, such as structure or composition validity. (iii) distribution: This measures whether the generative model is capable of learning the data distribution (in terms of structure, property, etc.) in the training set, and whether can interpolate or generalize to unseen materials.
We include all three types of evaluations metrics developed by Xie et al. (2022]: :: Materials match reconstruction metric check if the generated material reconstructs structure the test set. Following Xie et al. [2022]. this is done using StructureMatcher from pymatgen Ong et al. [2013] which considers the match of two materials considering invariances. Validity is basic metric to check if the generated materials are valid. Following Court et al. [2020), material structure is valid if the shortest distance between any pair of atoms is greater than 0.5A. Structure coverage is a distribution metric check if the generated material structures cover the training distribution. We follow Xie et al.) [2022] to utilize the CrystalNN fingerprint Zimmermann and Jain (2020) and normalized MagPic fingerprint Ward al. [0166] define the structure and composition distance, respectively. Property statistics is a distribution metric to check if the properties of generated materials are close to those in the training dataset.
3.5 Oracle Functions for Generative Materials Design
In efforts facilitate the generative design of materials, we have established two categories of oracle functions. Drawing inspiration from oracle functions designed for drug discovery via machine learning Huang al. (202I). we initially offer a fingerprint (FP)-based oracle function. This function utilizes conventional materials descriptors in tandem with classical machine learning algorithms predict properties of interest. More specifically, we have pre-trained random forest model for each material property prediction task across 13 different datasets as proposed by Dunn al. 2020]. Consequently, by harnessing the pre-trained models with extracted features based on the Sine Coulomb Matrix and MagPie fcaturization algorithms,, we can predict the properties of an input material. While the predictive accuracy of these classical, FP-based materials descriptors may not rival that of deep learning- based models, we underscore the importance of their inclusion. Their utilization enables generalization where rules apply and mitigates the risk of biasing the optimization process towards deep learning. Our second oracle function is structure-based oracle function which aids selecting an appropriate substrate for given material (film). By taking into account their respective structures, we have incorporated oracle function that matches film with list of substrates. Specifically, this method analyzes the compatibility between a thin film and various potential substrates, particularly in terms of crystallographic orientation, matching area, potential strain, and clastic energy. This is achieved by loading the structural information of the film and substrates from respective files, then calculating and grouping matches based on substrate Miller indices. Each match, characterized by a minimum match area, is recorded with relevant details such the substrate formula, orientations of the film and substrate, and, if available, elastic energy and strain. Then the most suitable matches -thse with the smallest matching area area-for for each substrate orientation identified. This method ultimately returns list of all matches, providing comprehensive overview of how well the film could potentially fit on each substrate. Details can be found in Appendix B
Benchmarking Machine Learning Models
4.1 Existing approach
A burgeoning amount of machine learning models have been developed for learning molecular representations suitable for variety of downstream tasks, especially machine learning potential and molecular property prediction Wu et al. 20018) Ramakrishhan et al. (0144) Chmiela et al. (017).
7

---

Table 2: Representative work in modeling molecular and crystal structures.
Method
Representation
Symmetry
Graph construction
Angular
CGCNN Xie and Grossman 2018
Graph
Perm. + E(3) Inv.
Multi-graph
None
ALIGNNCChoudhan Choudhary and DeCost
2021
Graph
Perm. + E(3) Inv.
Multigraph + Line graph
Explicit
SchiNetSchult
2018
Graph
Pern. E(3) Inv.
Multi-gruph
None
BGNN Satoras el al.
202T
Graph
Perm. E(3) Equiv.
Multi-graph graph
Implicit
DimeNet=+ Gastesper et 2020
Graph
Perm. + E(3) Inv.
Multi-graph
Explicit
GemNer Gistegger et 2021
Graph
Perm. SE(3) Inv.
Multi- -ruli-
Explicit
Equiformer Lio and Smint
Graph
Perm. E(3WSE((3) Equiv.
Multi-graph
Implicit
LEFTNNe ef al 2025
Graph
Pern. E(3)SB(3) Equiv.
Multi-graph
Implicit
However, most of existing work focus on molecules without periodicity. Around the same time, another branch of work motivated directly by modeling crystal structures have been developed. We implement and benchmark models from both branches to facilitate the development of new methods in realization of both directions. Specifically, we detail them below and summarize them in Table 2
Learning crystal structures. CGCNN Xie and Grossman 2018 is E(3) invariant graph neural network that leverages pairwise distances as edge features. ALIGNN Choudhary and DeCost (2021) is an E(3) invariant graph neural network that builds an extra line graph to explicilly encode the bond angle information in addition to the original atomistic graph similar to CGCNN.
Learning on molecular structures. SchNer Schütt et al. 2018 is an E(3) invariant graph neural network that leverages pairwise distances with continuous filter convolution to construct the message. EGNN (Satorras et al.] [021] is an E(3) equivariant graph neural network that leverages relative positions between each pair of nodes and pairwise distances as the message function to update both invariant and equivariant features. DimeNet+ + Gasteiger et al. (2020) is an E(3) invariant graph neural network that introduces bond angles to improve expressiveness. However, it requires triplet of atom representations model the bond angle. GemNer Gasteiger et al. [2021] is SE(3) invariant graph neural network that leverages dihedral angles for better expressiveness. However, requires learning on quadruplet representations of atoms. Equiformer Liao and Smidī SE(3)E((3)
equivariant graph transformer network. Equiformer equips previous transformers with equivariant operations such tensor product to learn equivariant features built from irreducible representations. LEFTNer Du et al. 2023) is an SE(3)E(3) equivariant graph neural network based on equivariant local frames. LEFTNNe first scalarizes vector and tensor features during message passing and convert them back by tensorizing the scalars through the equivariant frames proposed in ClofNet Du et al. 20226 LEFTNet introduces local structure encoding and frame transition encoding components to further improve the expressiveness.
4.2 Experiment Set-ups
We build on top of the Open Catalyst Project (OCP) which provides reproducible inplementations of commonly used 3D graph neural networks with benchmarks on OC datasets Chanussot et al. [ We further implement CGCNN, ALIGNN, EGNN, Equiformer and LEFTNet as they are not inc luded in OCP. We test all the methods a list of 13 representative tasks from our benchmarks with three data splits (random, composition and system). We mostly use the default hyperparameters provided in the open-source code of each method and reported them in Appendix F As OC20 and QM9 have been largely adopted in the community, we directly take the results and report in Appendix C Most of our experiments are conducted single 16GB V100s while some experiments with nemory-intensiie models on single 80GB A100s.
4.3 Results and Discussions
Several observations can be gleaned from our benchmark results shown in Table (i performance (observation /): despite the competitive performance of advanced equivariant graph neural networks, invariant models such as DimeNett+ and ALIGNN continue be among the state-of-the-ar methods; (ii) efficiency (observation 2): there is a significant variation in efficiency across the benchmarked models. ALIGNN, DimeNet++, GemNet, and Equiformer, as illustrated in Table have particularly slow runtimes. LEFTNet presents desirable balance of accuracy and efficiency; (iii) data split (observation 3): more realistic data splits indeed increase the challenge of the task, particularly the system split. However, this trend does not hold for all properties, with dielectric being an exception;
8

---

Table 3: Benchmark on materials property prediction tasks (different colors denote distinct property types: purple (electrical), yellow (stability), green (thermal), red (optical), blue (mechanical), denotes missing results due to extremely small test set after data split). The best numbers are darkly shaded, and the second-best numbers are lightly shaded.
Metlesh
qusifuadga
ny buakgg
p30n
.1.23
ponoskit
1rg (S_ge13
Tog. Kg.kkhh
evcrage racking
COCNN
0.3151
0,2966
0.2759
010105
0.0055
56.3534
0,2000
0,8426
02 0..5555
0:3123
0.0524
[0011]
Raddes
S:NNa
0.239
0.3166
0.329
$5.456
0.1.3
0.500
$3.5380
0,3713
0.8447
0.3295
0.30%
06775
BANN
033134
BL984
0.3381
18.15%
QUI13
16005
51.1220
IL2445
01827
65 03.3310
0.3994
00288
16113
6.2
Darzhetie
08:3500
112771
01821
39:36-3
002099
0.3000
03321
0.3090
$5.336
0.023
0.003
53.4700
0.0653
04556
60.7030
0.2443
0.2833
06671
62
LEPTNN
03143
03326
01139
50.966
00/10
0.00%%
49:3550
0.0396
0.055
38.0130
0.9033
0.770
00519
ALEONN
0.313
50:00
10000
0.001
00249
0.5533
00761
ScaNaa
0.6634
0 13530
0. 0.336
$7.79G
00131
0.00L
00113
0.791
0.300
6.9
51.31%
0H117
03825
GeaNe::
02353
02321
0.2213
15.22%
121119
03279
02801
N6411
Tepiieenor
03110
0 0.911
ST.66
0.014
0.005
03837
0.389
0,0818
TYRENN
0/351
310300
11115
619T
011124
105566
ALRGNN
01311
01097
K0075
00175
432280
012631
0U1T1
61:000
0.163
02890
13
Sc3Nea
0.693
01111
0.506:
1080
0.340
031133
166.05T
0.197
0.119
DuncSetis
D3419
0.5095
80087
0.1133
0.504
35,0240
0.0866
0A224
87 67.5M0
03123
02898
00566
GcaNN::
03412
0.5075
15.41%
011122
01000
34,4400
0.0651
113.3400
0.1383
0,651
N0712
0.2999
16.015
0/0/1
[10045
1400190
013190
010386
1200
001377
0,0815
BCTT
Table 4: Benchmark the efficiency of machine learning models with materials in different sizes (pdos~10, e form~30,, qmof- 100) a single V100 GPU (each row with same batch size except when exceeding the maximum memory, running time for 10 epochs).
CGCNN
ALIGNN
SchNet
EGNN
DimeNet++
GemNet
Equiformer
LEFTNet
pdos e_oor
68s
623s
77s
158s
203s
713s
117s
8572s
41343s
10589s
12591s
35622s
40801s
62344s
13797s
qmof bandgap
678s
2277s
512s
1336s
5240s
4572s
27884s
2405s
average ranking
1.33
1.67
5.67
4.33
(iv) material type (observation 4): the performance trends across various models remain consistent for a given material property. For instance, for the bandgap property, organic crystals (OMDB) demonstrate the smallest values, followed by metal-organic frameworks (QMOF), while inorganic bulk materials (MP) exhibit the largest values.
5 Conclusion, Limitation and Future Outlook
In this paper, we introduce HuH as a comprehensive platform for machine learning development in materials discovery. Hub is toolkit that consists of problem formulation, data downloading, data processing, machine learning methods implementations, machine learning training and evaluation procedures, and benchmark results. We cover not only the commonly considered predictive tasks on materials but also provides tools to enable the study of generative tasks on materials. Specifically, curate 9 datasets constructed by 6 types of materials with 65 tasks across 8 property types for the predictive task. We further provide 2 synthetic datasets for the purpose of generative tasks on materials. We design 3 extra challenging and realistic data split schemes in addition to previously used random split. We believe Hu will serve as an essential role in machine learning for materials discovery with datasets, infrastructures and benchmarks.
Despite we formulate the matcrials discovery pipeline in the machine learning language supported by datasets, infrastructures and benchmarks, most of the tasks do not involve experiments. However, in reality, experiment is the golden standard to test new materials. It remains a challenge to develop datasets and benchmarks for machine learning models to grow in assisting the experiment phase of materials discovery such as phase demixing and experiment planning (related work is summarized in Appendix(M (A).
Limitation:
There are multiple future directions to extend Mub is to improve the usability for materials science community (similar to previous work Ward et al. 2018 Jacobs et al. (2020). ), e.g. collect pre-trai models) Xia et al. Wang et al. [2021a]] benchmark machine learning models on
Future work:
specific tasks Kong et al. 2022
Bai et al. [2023]
etc.

---

Maintenance: We are committed to maintaining and extending the usability of this toolkit to wider machine learning and materials science community. We plan maintain this toolkit by adding new datasets, new tasks and evaluations, new oracle functions, and pre-trained models as this community continues to grow benefit more researchers.
Acknowledgement
This project is partially supported by the Eric and Wendy Schmidt AI in Science Postdoctoral Fellowship, a Schmidt Futures program; the National Science Foundation (NSF), the Air Force Office of Scientific Research (AFOSR); the Department of Energy; and the Toyota Research Institute (TRI).
References
Linfeng Zhang, Jiequn Han, Han Wang, Roberto Car, and EJPRL Weinan. Deep potential molecular dynamics: scalable model with the accuracy of quantum mechanics. Physical review letters, 120 114):143001, 2018.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool. Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583 -899, 2021.
Kenneth Atz, Francesca Grisoni, and Gisbert Schneider. Geometric deep learning on molecular representations. Nature Machine Intelligence, 3 3(12): 2021.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 1 118(15))2006623111 18, 2021. doi: 10. (0.1073/paas.2112239 18. URI.https. bioRxiv 10.1101/622803.
//www pnas wwtpoo/o////o////////o//oo 10/33///1...73/aa.... wtt.:////.....1211111111111 2016239118
wwws.pnas.oo
Raphael JL Townshend, Stephan Eismann, Andrew M Watkins, Ramya Rangan, Maria Karelina, Rhiju Das, and Ron O Dror. Geometric deep learning of ma structure. Science, 373(6558): 1047-1051, 2021. 2021.
Benjamin Sanchez-Lengelingg and Alán Aspuru-Guzik. Inverse molecular design using machine learning: Generative models for matter engineering. Science, 361(640)) 2018.
Carla P Gomes, Daniel Fink, R Bruce van Dover, and John M Gregoire. Computational sustainability meets materials science. Nature Reviews Materials, 6(8):455 647,,447755 2021.
Jonathan Schmidt, Mário RG Marques, Silvana Botti, and Miguel AL. Marques. Recent advances and applications of machine learning in solid-state materials science. npj Computational Materials, (1):83, 2019.
Han Wang, Linfeng Zhang. Jiequn Han, and E Weinan. Deepmd-kit: A deep learning package for many-body potential energy representation and molecular dynamics. Computer Physics Communications, 228: 178-184. 2018.
Ben Blaiszik, Logan Ward, Marcus Schwarting, Jonathon Gaff, Ryan Chard, Daniel Pike, Kyle Chard, and lan Foster. A data ecosystem to support machine learning in materials science. MRS Communications, (44):1225-11333 2019.
Alexander Dunn, Qi Wang. Alex Ganose, Daniel Dopp, and Anubhav Jain. Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm. npj Computational Materials, 6(1):138, 2020.
Conrad Clement, Steven Kauwe, and Taylor D Sparks. Benchmark aflow data sets for machine learning. Integrating Materials and Manufacturing Innovation, 9: 1:33-155, 2020.
Faiza Qayyum, Do-Hyeun Kim, Seon-Jong Bong, Su- Young Chi, and Yo-Han Choi. A survey of datasets, preprocessing, modeling mechanisms, and simulation tools based on for material analysis and discovery. Materials, 15(4):1428, 2022.
10

---

Samantha Durdy, Cameron J Hargreaves, Mark Dennison, Benjamin Wagg, Michael Moran, Jon A Newnham, Michael W Gaultois, Matthew Rosseinsky, and Matthew Dyer. The liverpool materials discovery server: A suite of computational tools for the collaborative discovery of materials. 2023.
Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review leiters, 120(14): 145301, 2018.
Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong- Graph networks as universal machine learning framework for molecules and crystals. Chemistry Materials, 31(9):3564-3572 2019.
Kamal Choudhary and Brian DeCost. Atomistic line graph neural network for improved materials property predictions. upj Computational Materials, 7(1):185, 2021.
Mario Geiger and Tess Smidt. e3nn: Euclidean neural networks. arXiv preprint arXiv:2207.09453, 2022.
Oumar Kaba and Siamak Ravanbakhsh. Equivariant networks for crystal structures. Advances in Neural Information Processing Systems, 35:4150-4166, 2022.
Keqiang Yan, Yi Liu, Yuchao Lin, and Shuiwang Ji. Periodic graph transformers for crystal material property prediction. Advances in Neural Information Processing Systems, 35:15666-15080, 2022.
KISHALAY DAS, Bidisha Samanta, Pawan Goyal. Scung-Cheol Lee, Satadeep Bhattacharjee, and Niloy Ganguly. Crysgnn: Distilling pre-trained knowledge to enhance property prediction for crystalline materials. In Workshop on" Machine Learning for Materials' MiterialssILLA. "ICLR 2023, 2023.
Yuchao Lin, Keqiang Yan, Youzhi Luo, Yi Liu, Xiaoning Qian, and Shuiwang Ji. Efficient approxi- mations of complete interatomic potentials for crystal property prediction. 2023.
Kishalay Das, Pawan Goyal, Seung-Cheol Cheol Lee, Satadeep Bhattacharjee, and Niloy Ganguly. Crys- mmnet: Multimodal representation for crystal property prediction. In Uncertainty Artificial Intelligence, pages 507-517. PMLR, 2023.
Justin S Smith, Olexandr Isayev, and Adrian E Roitberg. Ani-l: an extensible neural network potential with dft accuracy at force field computational cost. Chemical science, 1(4):31,,-3203, 2017.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Korbbluth, Nicola Molinari, Tess Smidt, and Boris Kozinsky. E (3)-equivrriant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):1-11, 2022.
Yuanqi Du, Tianfan Fu, Jimeng Sun, and Shengchao Liu. Molgensurvey: A systematic survey in machine learning models for molecule design. arXiv preprint arXiv:2203. 2022a.
Matteo Manica, Jannis Born, Joris Cadow, Dimitrios Christofidellis, Ashish Dave, Dean Clarke, Yves Gaetan Nana Teukam, Giorgio Giannone, Samuel C Hoffman, Matthew Buchan, al. Accelerating material design with the generative toolkit for scientific discovery. npj Computational Materials, (1):69, 2023.
Colin W Glass, Artem R Oganov, and Nikolaus Hansen. Uspex-evolutee evolutionoootyyyy crystal structure prediction. Computer physies communications, 175(11-12)7113-20 2006.
Chris J Pickard and RJ Needs. Ab initio random structure searching. Journal of Physics: Condensed Matter, 23(5):0 2011.
Geoffroy Hautier, Chris Fischer, Virginie Ehrlacher, Anubhav Jain, and Gerbrand Ceder. Data mined ionic substitutions for the discovery of new compounds. Inorganic chemistry, 50(2):656-66) 2011.
Volker Deringer, Chris J Pickard, and Gábor Csányi. Data-driven learning of total and local energies in elemental boron. Physical review letters, 120615):15600,, 2018.

---

Jordan Hoffmann, Louis Maestrati, Yoshihide Sawada, Jian Tang, Jean Michel Sellier, and Yoshua Bengio. Data-driven approach to encoding and decoding 3-d crystal structures. arXiv preprint wwXiv:/90900094 2019.
Juhwan Noh, Jaehoon Kim, Helge $ Stein, Benjamin Sanchez-Lengeling, John M Gregoire, Alan Aspuru-Guzik, and Yousung Jung. Inverse design of solid-state materials via continuous repre- sentation. Matter, 13):1370-1384, 2019.
Yong Zhao, Mohammed Al-Fahdi, Ming Hu, Edirisuriya MD Siriwardane, Yuqi Song, Alireza Nasiri, and Jianjun Hu. High-throughput discovery of novel cubic crystal materials using deep generative neural networks. Advanced Science, 8(20):2100566 2021.
Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. Advances in neural information processing systems, 32, 2019.
Tian Xie, Xiang Fu, Octavian- Ganea, Regina Barzilay, and Tommi S Jaakkola. Crystal diffusion variational autoencoder for periodic material generation. In International Conference on Learning Representations, 2022.
Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL materials, 1(1):01 1002, 2013.
Andrew S Rosen, Victor Fung, Patrick Huck, Cody T O' O'Donnell, Matthew K Horton, Donald G Truhlar, Kristin A Persson, Justin M Notestein, and Randall Q Snurr. High-throughput predictions of metal-organic framework electronic properties: theoretical challenges, graph neural networks. and data exploration. npj Computational Materials, 8(1):112, 2022.
Stanislav S Borysov, R Matthias Geilhufe, and Alexander Balatsky, Organic materials database: An open-access online database for data mining. PloS one, 011001 2017.
Kamal Choudhary, Kevin F Garrity, Andrew CE Reid, Brian DeCost, Adam Biacchi, Angela R Hight Walker, Zachary Trautt, Jason Hattrick-Simporr A Gilad Kusne, Andrea Centrone, et al. The joint automated repository for various integrated simulations (jarvis) for data-driven materials design. npj computational materials, 6(1):173, 2020.
Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Heras-Doming Domingo, Caleb Ho, Weihua Hu, et al. Open catalyst 2020 (oc20) dataset and community challenges. Acs Catalysis, 11(10):6059-607) 2021.
David Balcells and Bastian Bjerkem Skjelstad. tmqm 86k transition metal complexes. Journal of chemical information and modeling, 60(12):6135-6146, 2020.
geometries and properties of
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):177, 2014.
Chris J. Pickard. Airss data for carbon at 10gpa and the c+n+h+o system at 1gpa, 2020. URL rec////cordd22200 2tt.:o////020000/0//1111
httts://archive.
Ivano E Castelli, David D Landis, Kristian S Thygesen, Søren Dahl, Ib Chorkendorff, Thomas Jaramillo, and Karsten W Jacobsen. New cubic perovskites for one-and two-photon water splitting using the computational materials repository. Energy & Environmental Science, 5(10):9034 9043, 2012a.
Ivano E Castelli, Thomas Olsen, Soumendu Datta, David D Landis, Søren Dahl, Kristian S Thyygesen, and Karsten W Jacobsen. Computational screening of perovskite metal oxides for optimal solar light capture. Energy & Environmental Science, 5(2):5814-5819 2012b.
Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent I. Chevrier, Kristin A Persson, and Gerbrand Ceder. Python materials genomics (pymatgen): robust, open-source python library for materials analysis. Computational Materials Science, 68:314-319, 2013.
12

---

Callum J Court, Batuhan Yildirim, Apoorv Jain, and Jacqueline M Cole. 3-d inorganic crystal structure generation and property prediction via representation learning. Journal of Chemical Information and Modeling, 60(10):4518 (0(10))4518-4535, 2020.
Nils ER Zimmermann and Anubhav Jain. Local structure order parameters and site fingerprints for quantification of coordination environment and crystal structure similarity. RSC advances, 10(10); 1081, 2020.
Logan Ward, Ankit Agrawal, Alok Choudhary, and Christopher Wolverton. A general-purpose machine learning framework for predicting properties of inorganic materials. upj Computational Materials, 2(1):1-7, 2016.
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeuties data commons: Machine learning datasets and tasks for drug discovery and development. Advances in neural information processing systems, 2021.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 9(2):513-530, 2018.
Stefan Chmiela, Alexandre Tkatchenko, Huziel E Sauceda, Igor Poltavsky, Kristof T Schütt, and Klaus-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science advances, 365)ee600015, 2017.
Kristof T Schütt, Huziel E Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24):241722, 2018.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference machine learning, pages 9323-99222993 9332.. PMLR, 2021.
Johannes Gasteiger, Shankari Giri, Johannes T Margraf, and Stephan Günnemann. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. arXiv preprint 1:Xiv:2011.1411:: 2020.
Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34: 6790-6802, 2021.
Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs.
Weitao Du, Yuanqi Du, Limei Wang, Dieqiao Feng, Guifeng Wang, Shuiwang Ji, Carla Gomes, and Zhi-Ming Ma. Efficient and expressive equivariant graph neural networks. under review, 2023.
Weitao Du, He Zhang, Yuanqi Du, Qi Meng, Wei Chen, Nanning Zheng, Bin Shao, and Yin Liu. Se (3) equivariant graph neural networks with complete local frames. In International Conference on Machine Learning, pages 5583-5608. PMLR, 2022b.
Logan Ward, Alexander Dunn, Alireza Faghaninia, Nils ER Zimmermann, Saurabh Bujaj. Qi Wang. Joseph Montoya, Jiming Chen, Kyle Bystrom, Maxwell Dylla, et al. Matminer: An open source toolkit for materials data mining. Computational Materials Science, 152:60-69, 2018.
Ryan Jacobs, Tam Mayeshiba, Ben Afflerbach, Luke Miles, Max Williams, Matthew Tumnrr, Raphael Finkel, and Dane Morgan. The materials simulation toolkit for machine learning (mast-ml): An automated open source toolkit accelerate data-driven materials research. Computational Materials Science, 176:109544 2020.
Jun Xia, Yanqiao Zhu, Yuanqi Du, and Stan Z Li. systematic survey of chemical pre-trained models.
Yingheng Wang, Yaosen Min, Erzhuo Shao, and Ji Wu. Molecular graph contrastive learning with pa- rameterized explainable augmentations. In 202/ IEFE International Conference on Bioinformatics and Biomedicine (BIMM), pages 1558-1563, IEEE, 2021a.
13

---

Yingheng Wang, Xin Chen, Yaosen Min, and Ji Wu. Molcloze: a unified cloze-style self-supervised molecular structure learning model for chemical property prediction. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 2896-2903. IEEF, 2021b.
Shufeng Kong, Francesco Ricci, Dan Guevarra, Jeffrey Neaton, Carla P Gomes, and John M Gregoire. Density of states prediction for materials discovery via contrastive learning from probabilistic embeddings. Nature communications, 13(1):949, 2022.
Junwen Bai, Yuanqi Du, Yingheng Wang, Shufeng Kong, John Gregoire, and Carla Gomes, Xtal2dos: Attention-based crystal to sequence learning for density of states prediction. arXiv preprint arXiv:2302.014485 2023.
Di Chen, Yiwei Bai, Sebastian Ament, Wenting Zhao, Dan Guevarra, Lan Zhou, Bart Selman, R Bruce van Dover, John M Gregoire, and Carla P Gomes. Automating crystal-structure phase mapping by combining deep learning with constraint reasoning. Nature Machine Intelligence, (9):812-822, 2021.
Sebastian Ament, Maximilian Amsler, Duncan R Sutherland, Ming- ShingChiang Chang, Dan Guevarra, Aine Connolly, John M Gregoire, Michael O Thompson, Carla P Gomes, and R Bruce van Dover. Autonomous materials synthesis via hierarchical active learning of nonequilibrium phase diagrams. Science Advances, 7(51 ):eabg4930, 2021.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese. Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414 2022.
Evgeny Blokhin and P Villars. Materials platform for data science: from big data towards materials genome. 2019.
Saulius Gražulis, Adriana Daškevič, Andrius Merkys, Daniel Chateigner, Luca Lutterotti, Miguel Quiros, Nadezhda R Serebryanaya, Peter Moeck, Robert T Downs, and Armel Le Bail. Crystal- lography open database (cod): an open-access collection of crystal structures and platform for world-wide collaboration. Nucleic acids research, 40(D1):D400-447, 2012.
James E Saal, Scott Kirklin, Muratahan Aykol, Bryce Meredig, and Christopher Wolverton, Materials design and discovery with high-throughput density functional theory: the open quantum materials database (oqmd). Jom, 65:1501-1509, 2013.
Mariette Hellenbrandt. The inorganic crystal structure database (iesd) present and future. Crystal- lography Reviews, 10(1):17-22 2004.
Colin Groom, Ian J. Bruno, Matthew P. Lightfoot, and Suzanna C. Ward. The Cambridge Structural Database. Acta Crystallographica Section B. 72()):1111779, Apr 2016. doi: 10.1107/ $20525206160039944 URL Jtttsss::///do: http:////.ooooog/....o 1107 /111177777022222220666633355
Claudia Draxl and Matthias Scheffler. The nomad laboratory: from data sharing to artificial intelli- doi: (0.1088/2115-7639a1 13bb.
gence. Journal of Physics: Materials, 2(3):03600,, may 2019.
URL hattps://da doi
Leopold Talirz, Snehal Kumbhar, Elsa Passaro, Aliaksandr V. Yakutovich, Valeria Granata, Fernando Gargiulo, Marco Borelli, Martin Uhrin, Sebastiaan P. Huber, Spyros Zoupanos, Carl S. Adorf, Casper Welzel Andersen, Ole Schütt, Carlo A. Pignedoli, Daniele Passerone, Joost Vande Vondele, Thomas C. Schulthess, Berend Smit, Giovanni Pizzi, and Nicola Marzari. Materials cloud, platform for open computational science. Scientific Data, 7(1):999, Sep 2020. ISSN 2052-4463, doi: 10.103884441597-0000006666
URL https: www/oog///111 10. 1038/ 8 841557-0000-00677--
Matthias Scheffler, Martin Aeschlimann, Martin Albrecht, Tristan Bereau, Hans-Joachim Bungartz, Claudia Felser, Mark Greiner, Axel GroB, Christoph T. Koch, Kurt Kremer, Wolfgang E. Nagel, Markus Scheidgen, Christof Wöll, and Claudia Draxl. Fair data enabling new horizons for materials research. Nature, 604(7907))635 642, Apr 2022. ISSN 1476-4687. doi: 10. 10.1038/ s411566-02-04501-x URL hRtps://///do org/ 10. 10.100010000 1038/ /4158////486666 022 04501-x
14

---

Stefano Curtarolo, Wahyu Setyawan, Gus L.W. Hart, Michal Jahnatek, Roman V. Chepulskii, Richard H. Taylor, Shidong Wang. Junkai Xue. Kesong Yang. Ohad Levy, Michael J. Mehl,
Harold T. Stokes, Denis O. Demchenko, and Dane Morgan. Aflow: An automatic frame-
work for high-throughput materials discovery. Computational Materials Science, 58:218- 226, 2012. ISSN 0927-0256. doi: http://oooooglll..ooo.ccommmmmmp
URL
https://wwai http://wwww....ttttto...ooo.octa
15

---

Appendix for Hub
Contents
Additional Related Work
16
A.1 Materials Synthesis
16
A.2 Experiment Control with Machine Learning
16
Additional Data Sources
17
Additional Experimental Results
17
D Dataset Statistics
17
Oracle Function Details
18
Experimental Details
19
F1 Hyperparameters
19
A
Additional Related Work
As mentioned in Sec. 5 main focus in the current version is virtual screening, inverse design and molecular simulation while ignoring tasks related to experiments where problem formulation, dataset curation, evaluation and benehmarking are much more challenging. In this section, we introduce additional related work that leverage machine learning to assist in the experiment phase of materials discovery.
A.1 Materials Synthesis
In addition to designing materials computationally, new materials have to be synthesized with experiments. Even though computational methods have been deployed to simulate or predict material properties, it is much harder to predict how materials can be synthesized. Thus, it is a nontrivial problem to study material synthesis. Moreover, the synthesis process is challenging to predict as well. Material scientists have to post-process the synthesized material to identify the synthesized structures. Specifically, X-ray Diffraction (XRD) is a commonly used technology to detect crystal structures. Chen et al. [021 combines deep learning with reasoning module to incorporate physical constraints and identity crystal structure phase compositions from the experimental results with XRD patterns.
A.2 Experiment Control with Machine Learning
Experiment is an indispensable step to validate the properties of the materials. However, experiments are usually very expensive and even inaccessible in real scenarios. In addition, current experiment highly relies on human expert design which may be suboptimal. Therefore, automated experiment design becomes an urgent yet challenging problem. One way to improve the efficiency of experiment design leverage machine learning models for uncertainty estimation. Specifically, active learning can be utilized to construct a loop of decision and feedback. Machine learning models suggest the next experiment and experiment provides feedback to improve machine learning models Ament et al. [2021]. Another promising direction is to leverage reinforcement learning methods which have an agent attempting to achieve some goal by the feedback provided by the environment (Dgrave et al. 2022
16

---

B Additional Data Sources
In addition to the datasets included in our current version (Sec. 3). there are other available data sources which may be used in different purposes and we will consider add in future version.
Materials Platform for Data Science (MPDS) Blokhin and Villars 2019 (no license) is data repository that collects experimental and computational materials data through data mining from the scientific publications. There, around half million articles were manually processed and systematized, covering broad spectrum of physical sciences, such as physics, chemistry, materials science, environmental science, engineering, and geology.
Crystallography Open Database (COD) Gražulis et al. [2012] (license: CCO 1.0): COD is a open-access database containing crystallographic data on inorganic and organic compounds. It includes experimentally determined crystal structures along with associated metadata for organic, inorganic, metal-organic compounds and minerals.
Open Quantum Materials Database (OQMD) Saal et al. (013 (license: CC-BY 4.0):
OQMD is database that focuses on quantum- guantum-mechanica calculations of materials prop- ertics. It contains calculated data on crystal structures, electronic structures, formation energies, and other material properties for wide range of inorganic compounds.
Inorganic Crystal Structure Database (ICSD) Hellenbrandt [2004] (commercial): ICSD is a comprehensive database that compiles > 281,000 experimentally determined crystal structures of inorganic compounds. To ensure the high quality of structures in ICSD, structure has to be fully characterized and passed thorough quality checks by its expert editorial team before inclusion. The information ICSD updated biannually.
Cambridge Structural Database (CSD) Groom et al. [2016] (commercial): CSD contains over 1.1M accurate 3D experimentally crystalized structures with data from X-ray and neutron diffraction analyses. It contains diverse types of organic crystal structure (drug, pigment, etc.) and metal-organic crystals (transition metal complex, metal-organic framework, etc.).
Novel Materials Discovery (NOMAD) Draxi and Scheffler [0199] (license: CC-BY 4.0):
NOMAD is a data management platform for materials science data where users can share data freely. Here, NOMAD is a web-application and database that allows centrally publish data. But you can also use the its utilities to build your own local database.
2020) (license: CCO 1.0): MC is built enable the seamless sharing and dissemination of resources in computational materials science, offering educational, research, and archiving tools; simulation software and services; and curated and raw data. These underpin published results and empower data-based discovery, compliant with data management plans and the FAIR principles SShefflerr al. (2022). In addition to database, MC also provides lectures for computational materials science, various visualization and simulation tools.
Materials Cloud (MC) Talirz
AFLOW Curtarolo et al. 2012 (MIT license) Similar to MC, AFLOW is a composite platform includes materials database, search and visualization, simulation, and machine learning models.
C Additional Experimental Results
In addition to benchmark results, two popular benchmarking datasets, OC20 and QM9 have been extensively tested in previous work. We directly take the experimental results from Liao and Smidi for OC20 ( and Du et al. [2023) for QM9 (Table as reference performance of existing approaches. Note that there are two commonly used data splits for QM9 in previous literature and they both reported.
D Dataset Statistics
In Table 7 we report the statistics of each dataset with number of samples and number of atoms in each material.
17

---

Table 5: Benchmark on machine learning forcefield (OC20 IS2RE test set) (results taken from LIao
and Smidt). (The best results are bolded.)
Energy
Methods
ID
0OD Ads
OOD Cat
OOD Both
Average
1D
OOD Ads
OOD Cat
OOD Both
Average
CGCNN
0.6149
0.9155
0.6219
0.8511
0.7309
3.40
1.93
3.10
2,00
2.61
SchNet
0.7342
0.7037
0.6846
2.96
2.94
DimeNet++
0.5621
0.7252
0.5756
0.6613
0.6311
4.25
2.07
4.10
2.41
3.21
Equiformer
0.5037
0.6881
0.5213
0.6301
0.5858
5.14
2.41
4.67
2.69
3,73
Table Benchmark on molecular property prediction (QM9) (results taken from Du al. 2023]
(The best results are bolded.)
Task
As
EnGMо meV V
FLUMO meVV
C₁
C
HI
R
U
Un meV
ZPVE
Units
bohr
meV
cal/mol K
meV
meV
hohr
meV
meV
EGNN
.011
48
29
25
.029
.031
12
12
.106
12
11
1.55
Equiformer
.056
33
10
.227
11
10
1.32
LEFTNet
.048
40
.012
.023
.109
7
1.33
SchNet
.235
63
.033
.033
14
.073
19
14
1.70
DimeNet++
(444
33
25
20
.030
.033
8
.331
6
1.21
LEFTNet
.039
39
23
18
.011
.022
.994
5
5
1.19
Table 7: Dataset statistics (number of samples in each dataset and size of systems in each dataset.
3339
43E4
23787
D037
195.133
13325
1200
E
Oracle Function Details
FP-hased oracle function This method generates desired properties for any specific input materials. The core concept is SCM/MagPie featurization and machine learning prediction. The oracle function first reads CIF files to extract the structures or compositions of the materials. Then, the function preprocess and prepares the appropriate data format based the task at hand. Before training, the materials data are transformed by the Sine Coulomb Matrix and MagPie fcaturization algorithms convert the raw data into form that can be used by the machine learning model. After that, standard machine learning pipeline is built to predict the target property for these materials. The pipeline uses random forest the machine learning model. Finally, the method saves the predictions and returns them for user convenience.
Structure-based oracle function This method is used match given film with list of substrates. The function first reads the input film and substrates to get their structures. Then, the SubstrateAn= alyzer from pymatgen Ong al.) [013 called calculate possible matches between the film and each substrate. It finds the best matches (with the smallest matching area) for each orientation of the substrate. Finally, all the match information is stored and returned for users. The match information includes the substrate's 's formula, orientations of the substrate and film, the matching area, and optionally, the elastic energy and strain. This substrate matching process can be useful in thin film deposition processes, where you want to match the crystal structure of a thin film material to substrate material to ensure good adhesion and minimize defects.
Table 8: Hyparameters for benchmarked machine learning models.
CGCNN
ALIGNN
SchNet
EGNN
DimeNettt
GemNet
Equiformer
LEFTNet
cutoff
6.0
8.0
6.0
6.0
6.0
6.0
6.0
6.0
max, reighhees
N/A
N/A
N/A
N/A
50
500
N/A
layers
3
6
4
3
hidden. dimension
256
128
128
128
192
128
512
128
learning rate
le-4
optimizer
Adam/
AdamW
AdamW
Adam
AdamW
Adam
AdamW
Adam
scheduler
N/A
N/A
N/A
N/A
N/A
ReduceL ReduceLROaafatteu ROaPlateuu
Cosine
N/A
training epochs
500
500
500
500
500
500
500
500
18

---

Experimental Details
F.1 Hyperparameters
We report the general hyperparamters shared across models in Table 8 For model-specific parameters,
we report in haahaaggiidd///u//ooooontt
19

---

Table 9: Detailed system size statistics for each task in each dataset.
Data Name
Task
Mean
std
Carbon24
carbon24
9.21
3.58
DFT2D
jdft2d
7.19
4.35
DFT3D
avg_elec_mass
8.58
6.70
DFT3D
avg hole mass bulk modulus kv
8.58
6.70
DFT3D
6.71
5.11
DFT3D
dfft__pizzo max_ lil__ddlcctrrcioonnn ionic
9.83
6.41
DFT3D
dfpt_piezo saso__ma__d_____
7.99
4.96
DFT3D
dfpt_ppezo__axx max_ze_
9.83
6.41
DFT3D
ehull
10.34
8.84
DFT3D
encut
10.33
8.83
DFT3D
epsx
8.57
7.15
DFT3D
epsy
8.57
7.15
DFT3D
epsz
8.57
7.15
DFT3D
exfoliation_enaa _nerryy
9,42
6.79
DFT3D
formation formation_energy seergyyperrato
10.34
8.84
DFT3D
Leontt_engghhaoo unit
10.33
8.83
DFT3D
magmom_outer
10.21
8.72
DFT3D
max max_eeg
5.79
3.53
DFT3D
mbj_bandgap
8.01
6.37
DFT3D
mepsx
8.06
6.40
DFT3D
mepsy
8.06
6.40
DFT3D
mepsz
8.06
6.40
DFT3D
n-Seebeck
10.92
8.59
DFT3D
n-powerfact
10.92
8.59
DFT3D
optb88vdw __ndgapppappppppp
10.34
8.84
DFT3D
optb88vdw total_ Lotrgy
10.34
8.84
DFT3D
p-Seebeck
10.92
8.59
DFT3D
P-powerfact
10.92
8.59
DFT3D
shear bhear_modulus..
6.71
5.11
DFT3D
slme
10.98
7.28
DFT3D
spillage
7.58
5.74
EDOS-PDOS
edos_up
10.08
9.06
EDOS-PDOS
pdos_elast
7.23
5.46
MatBench
dielectric
16.89
14.67
MatBench
log_gvrh
8.63
8.66
MatBench
log_kvrh
8.63
8.66
MatBench
mp_e_form
29.15
30.10
MatBench
mp_gap
30.02
29.94
MatBench
mp mp_is istal
30.02
29.94
MatBench
perovskites
5.00
0.00
MatBench
phonons
7.53
3.74
Perov5
perov5
5.00
0.00
OC20
S2EF
73.25
30.96
OC20
IS2RE
77.75
31.40
OC20
IS2RS
77.75
31.40
OMDB
band_
82.29
26.55
QM9
all
18.02
2.94
QMOF
bandgap
113.67
68.86
tmQM
all
65.99
27.01
20


Reviews:
[{"review_id": "fu8JjMZSsY", "reviewer": "Reviewer_GSxn", "summary": "", "questions": "", "limitations": "The framework collects various information (datasets, baseline and metrics) in one place and perform evaluation on the proposed tasks. \nIn a sense this is a positive aspect (to merge different information and tools) on the other hand is limited in novelty.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. One place with various training and evaluation tasks for material design. \n2. dataset separated per task\n3. evaluation metrics for each of the task\n4. the authors performed evaluation of the proposed tasks against the baselines", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "2DwuHr2t9a", "reviewer": "Reviewer_ouYe", "summary": "", "questions": "", "limitations": "n/a", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "* The formulated tasks are diverse covering four different predictive tasks and three generative tasks. Data instances cover various material types and properties. This should provide a thorough and reliable evaluation to ML approaches and make their adaptation into material science production much easier.\n* The work should be of interest to both the boarder machine learning and material science communities. The task formulation and dataset construction\n* A concrete benchmarking based on the constructed tasks and datasets.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "wE8vxpKZz9", "reviewer": "Reviewer_PX85", "summary": "", "questions": "", "limitations": "The current limitation section is merged with future work, it may be better to clearly write the limitations.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "In recent times we have seen rapid interest among the ML and Materials science community to develop ML models for fast and accurate property prediction of materials and new materials generation.We have witnessed many DL methods being proposed by different researchers in Top venues in ML and Material Science Community. But evaluating all the models was getting difficult because everyone was coming up with new models, conducting experiments with different datasets with different train test splits, and using different evaluation criteria. Hence a predefined benchmark set consisting of all the diverse datasets, different predictive and generative tasks, train-test data split, evaluation metrics, etc is needed to perform comparative studies and evaluate the existing as well as upcoming works. This paper makes a significant contribution toward addressing this need by introducing a comprehensive benchmark. With the availability of this benchmark, researchers can now assess and compare the performance of different models without encountering the challenges posed by inconsistent methodologies in the previous works.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "taZLJgJDHX", "reviewer": "Reviewer_1xJA", "summary": "", "questions": "", "limitations": "Graph neural network-based architectures are very popular and this work showcases many of them; I was wondering if it would be easy to use the framework for non-graph-based architectures, such as Allegro? ", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "I like the variety of datasets exposed (spanning single-molecule systems, interfaces between metals and molecules, and bulk materials) and the implementation of multiple neural network architectures in a common framework is very useful for comparison when developing new methods or adapting architectures to new problem domains.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This submission presents a benchmark for material design, in particular considering 3D molecular information. All reviewers agree this is a good paper and should be accepted, and therefore I too recommend acceptance.
Decision: Accept (Poster)