SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\mathbf{M}^{2}\mathbf{H}\mathbf{ub}$ : Unlocking the Potential of Machine Learning for Materials Discovery
Yuanqi  $\mathbf{Du}^{1,*}$  Yingheng Wang $^{1,*}$  Yining Huang $^{2}$  Jianan Canal Li $^{3}$  Yanqiao Zhu $^{4}$
Tian Xie $^{5}$  Chenru Duan $^{6}$  John M. Gregoire $^{7}$  Carla P. Gomes $^{1}$
1 Cornell 2 Northwestern 3 UCB 4 UCLA
5 MSR AI4Science 6 Microsoft Quantum 7 Caltech * Equal Contribution
We introduce M²Hub, a toolkit for advancing machine learning in materials discovery. Machine learning has achieved remarkable progress in modeling molecular structures, especially biomolecules for drug discovery. However, the development of machine learning approaches for modeling materials structures lag behind, which is partly due to the lack of an integrated platform that enables access to diverse tasks for materials discovery. To bridge this gap, M²Hub will enable easy access to materials discovery tasks, datasets, machine learning methods, evaluations, and benchmark results that cover the entire workflow. Specifically, the first release of M²Hub focuses on three key stages in materials discovery: virtual screening, inverse design, and molecular simulation, including 9 datasets that covers 6 types of materials with 56 tasks across 8 types of material properties. We further provide 2 synthetic datasets for the purpose of generative tasks on materials. In addition to random data splits, we also provide 3 additional data partitions to reflect the real-world materials discovery scenarios. State-of-the-art machine learning methods (including those are suitable for materials structures but never compared in the literature) are benchmarked on representative tasks. Our codes and library are publicly available at https://github.com/yuanqidu/M2Hub.
With the methodological advancements in machine learning, an increasing number of machine learning models have been developed and applied to solve scientific problems, from simulating molecular systems with millions of particles to predicting accurate protein structures Zhang et al. [2018], Jumper et al. [2021]. The primary focus of machine learning in the chemical sciences has remained in the domain of molecular structures, (bio)molecules including small molecules, proteins, RNAs, etc. Atz et al. [2021], Rives et al. [2021], Townshend et al. [2021]. However, materials constitute a large portion of the chemical space which have been significantly less studied, especially in the machine learning community. Among scientific problems, materials discovery plays a vital role in driving innovations and progress across various fields spanning energy, electronics, healthcare, and sustainability Sanchez-Lengeling and Aspuru-Guzik [2018], Gomes et al. [2021]. However, the traditional trial-and-error approach to materials discovery is expensive and time-consuming. Over decades, classical machine learning methods have already been widely applied in assisting materials discovery, Schmidt et al. [2019] yet the impact of machine learning for solid state materials lags behind its efficacy in other areas of chemical science.
Witnessing the success of machine learning in solving grand challenges in science Wang et al. [2018], Jumper et al. [2021], one of the key ingredients is the infrastructure that supports the machine learning community to build the machine learning workflow: data preparation/processing, model development, performance evaluation, and model improvement based on the evaluation feedback. While effort
$\mathbf{M}^{2}$  Hub
A Generative Modeling/Combinatorial Optimization
B Machine Learning Forcefield C Pattern Demixing
$\langle \widehat{\mathbb{C}}\rangle$  Active Learning/Reinforcement Learning
$\mathbb{E}$  Representation Learning on Materials
F Generative Modeling/Search/Sampling
Figure 1:  $\mathbf{M}^2\mathbf{H}\mathbf{ub}$ : Materials discovery meets Machine learning. A-F on the left figure demonstrates machine learning approaches used in each stage of the materials discovery pipeline on the right figure (dashed lines denote currently unavailable experiment-related tasks).
has been made to make materials datasets available to the machine learning community Blaiszik et al. [2019], Dunn et al. [2020], Clement et al. [2020], Qayyum et al. [2022], Durdy et al. [2023], existing work mainly focus on providing data servers that allow the users to query materials data and predefined benchmark sets. However, to bridge the gap between the molecular and solid state materials, we identify the need for a unified platform to facilitate the development of machine learning approaches for materials discovery purpose, including (1) centralized data sources with diverse materials, property and task types, (2) clear problem formulations, (3) realistic problem settings (e.g. data split), and (4) appropriate benchmarks and transparent comparisons with prior methods.
In order to address the aforementioned challenges, we establish  $\mathbf{M}^2\mathbf{Hub}$ , which integrates and connects different machine learning building blocks in the materials discovery (Fig. 1). The cornerstone of  $\mathbf{M}^2\mathbf{Hub}$  is a benchmark that incorporates several key aspects: (i) it integrates three key tasks: virtual screening, molecular dynamics simulation, and inverse design, which are translated using machine learning constructs such as materials representation learning, machine learning forcefield, and generative materials design; (ii) it is underpinned by a curated set of 11 datasets that encompass 6 types of materials with 56 tasks across 8 distinct material properties. In addition to the standard random split, we have included 3 realistic (out-of-distribution) data splits to enhance the robustness of model evaluation; (iii) a distinctive feature of our benchmark is the emphasis on the generative design of materials. For this, we provide machine learning formulations, evaluation metrics, and oracle functions to facilitate further research and development in this area; (iv) finally, our benchmarks evaluate not only the commonly used material representation learning methods, but also those designed for non-periodic molecular structures. These methods are applied to 13 representative tasks for material property prediction.
The flow of this paper is as follows: we introduce the background and related work on developing machine learning methods for materials discovery in Sec. 2; we present the overview of  $\mathbf{M}^2$ Hub including problem formulation, dataset curation, data processing, evaluation and oracle function for inverse design in Sec. 3; in Sec. 4, we detail the implemented machine learning models, benchmarking results, observations, and insights emerging from the results.
2.1 Materials Representation Learning
Material representation learning refers to representing materials structures in an expressive and machine-readable format for downstream studies, from property prediction to materials generation. Recent advances in graph neural networks bring a wave of representing materials structures as graphs where nodes represent atoms and edges represent bonds or interactions. A line of works has been proposed to adapt this structured inductive bias into deep learning models.CGCNN Xie and Grossman [2018] introduces a multi-edge graph representation to capture periodicity. MEGNet Chen et al.
[2019] unifies molecules and crystal structures representations by graph neural networks representing each atom as node and interaction/bond between atoms as edge. More recently, ALIGNNN considers both atomistic and line graphs which externally capture angular information Choudhary and DeCost [2021]. Equivariant graph neural networks (e.g., E3NN Geiger and Smidt [2022]) have also been applied thanks to its roto-translational equivariance property. This task has also attracted increasing interests through years Kaba and Ravanbakhsh [2022], Yan et al. [2022], DAS et al. [2023], Lin et al. [2023], Das et al. [2023].
2.2 Machine Learning Forcefields
Molecular dynamics simulation has become an essential tool to understand the microscopic dynamical behaviors of molecular systems. It is worth noting there is a common trade-off between two popular diagrams, empirical forcefields and ab initio molecular dynamics. Empirical forcefields often rely on the hand-crafted parameters which are efficient yet inaccurate, while ab initio molecular dynamics rely on quantum-mechanical calculations which are precise but inefficient. Inspired by recent advances of deep learning in automated parameters learning and transferability, a large amount of works has been developed to learn machine learning forcefields from quantum-mechanical data to strike a balance between accuracy and efficiency. Specifically, it is expected to be more accurate than empirical forcefields and more efficient than quantum-mechanical calculations. Most representative work include DeepMD Zhang et al. [2018], ANI-1 Smith et al. [2017], and NeuqIP Batzner et al. [2022].
2.3 Materials Inverse Design
Designing new materials structures is a long-standing challenge, often known as the inverse design problem, in materials science Du et al. [2022a], Manica et al. [2023]. Before deep generative models have been applied to this problem, traditional computational methods often leverage quantum mechanical search over the possible stable materials including random search, evolutionary algorithm, element substitutions over known materials Glass et al. [2006], Pickard and Needs [2011], Hautier et al. [2011]. One line of works leverages a learned force field to minimize the energy of the structure to reach a stable material Deringer et al. [2018]. Later, deep generative models have been applied to this problem where the models aim to model the distribution of the known crystal structures and learn to sample new structures from it. Early work leverage 3D voxel representation but it is nontrivial to fit atom from the generated voxels Hoffmann et al. [2019], Noh et al. [2019]. Later work instead leverage atomic representation directly Zhao et al. [2021]. G-SchNet Gebauer et al. [2019] instead proposes an auto-regressive model that generates each atom in a sequential way. Notably, it remains largely unexplored for efficient and controllable crystal structure generation with machine learning methods. A recent work Xie et al. [2022] builds upon the recent success of diffusion models on images and adapts them for crystal structure generation and optimization in an iterative refinement manner instead of one-shot or auto-regressive sequential generation.
3 Overview of  $\mathbf{M}^2\mathbf{H}\mathbf{u}\mathbf{b}$
Material representation. Material structures can be represented as a set of atoms  $M = (m_0, m_1, \ldots, m_N)$  in 3D space with atom types  $H = (h_0, h_1, \ldots, h_N) \in \mathbb{R}^{N \times K}$  and atomic positions  $X = (x_0, x_1, \ldots, x_N) \in \mathbb{R}^{N \times 3}$  where  $N$  denotes number of atoms and  $K$  denotes number of atom types (C, O, Fe, Al, etc.). Most materials are crystal structures which periodically repeat their unit cells in 3D space. In such cases, lattice vectors  $L = (l_1, l_2, l_3) \in \mathbb{R}^{3 \times 3}$  are utilized to describe the periodicity in 3D space. Note  $L$  is not rotation invariant, 6 invariant lattice parameters (lengths of lattice parameters and angles between them) can also
Figure 2: Regular workflow for studying materials with machine learning approaches (green colored steps denote materials science expertise and blue colored steps denote machine learning expertise.
Table 1: Curated materials datasets in  $\mathbf{M}^2\mathbf{Hub}$ . Materials type and property type are detailed in Sec. 3.2.1 and Sec. 3.2.2; dim refers to data dimensionality; PBC refers to periodic boundary condition; method refers to how properties are obtained (sim short for simulation).
Data name	Materials type	Dim	PBC	Prop. type	Task type	# tasks	# data	Size	Method
MatBench	inorganic bulk	3D	(T, T, T)	elec./mech./stab./opt/ther.	scalar	8	312-132,752	27.08±28.27	Sim.
QMOF	metal-organic framework	3D	(T, T, T)	elec.	scalar	1	>20,000	113.67±68.86	Sim.
OC20	bulk-adsorbate interface	3D	(T, T, F)	energetic	scalar/vector	3	640,081	73.26±30.96	Sim.
OMDB	organic crystal	3D	(T, T, T)	elec.	scalar	1	12,500	82.29±26.55	Sim.
DFT3D	inorganic bulk	3D	(T, T, T)	elec./mech./stab./semi.	scalar	29	55,722	9.59±8.04	Sim.
DFT2D	inorganic bulk	3D	(T, T, F)	stab.	scalar	1	636	7.19±4.35	Sim.
EDOS-PDOS	inorganic bulk	3D	(T, T, T)	elec./ther.	1D dist.	2	48,469	9.50±8.45	Sim.
tmQM	transition metal complex	3D	(F, F, F)	elec.	scalar	8	86,665	65.99±27.01	Sim.
QM9	organic molecules	3D	(F, F, F)	elec.	scalar	12	~134,000	18.02±2.94	Sim.
Carbon24	inorganic bulk	3D	(T, T, T)	N/A	N/A	1	10,153	9.21±3.58	N/A
Perov5	inorganic bulk	3D	(T, T, T)	N/A	N/A	1	18,928	5.00±0.00	N/A
be used to describe the lattice  $(l_a, l_b, l_c, \alpha, \beta, \gamma)$ . Overall, a material is denoted as  $M = (H, X, L)$  if it is periodic and otherwise  $M = (H, X)$ .
Material graph representation. Regardless of periodicity, materials structures can be naturally represented as graphs  $G = (\mathcal{V}, \mathcal{E})$ , where  $\mathcal{V}$  is a set of vertices and  $\mathcal{E} = \{e_{ij}(k_1, k_2, k_3) | i, j \in \{1, 2, \dots, N\}, k_1, k_2, k_3 \in \mathbb{Z}\}$  is a set of  $D$  edges ( $k_1, k_2, k_3$  denotes the translation of the unit cell using lattice vector  $L$ , none if not periodic);  $H \in \mathbb{R}^{N \times K}$  denotes the node features;  $X^{N \times 3}$  denotes the atomic positions;  $E \in \mathbb{R}^{D \times F}$  denotes  $F$  edge features (such as bonds or distances between each pair of nodes). The graph connections can be determined in multiple ways such as distance threshold, detailed in Sec. 3.3.
Predictive tasks. Predictive tasks often have paired input material  $M$  and label  $Y$ , where given an input material  $M$ , we aim to predict the expected label as  $p(Y|M)$ . The label  $Y$  could be of various format such as binary, scalar, vector and distribution, detailed in Sec. 3.4. Both material representation learning and machine learning forcefield are considered as predictive tasks.
Generative tasks. Generative tasks could be divided into two parts: (1) distribution learning and (2) goal-oriented generation. Given a set of  $J$  materials  $\mathcal{M} = \{M_i\}_{i=1}^J$ , distribution learning aims to learn the distribution of  $p(M)$  and sample new materials  $M_{\mathrm{new}} \sim p(M)$ . Goal-oriented generation aims to sample molecules fulfilling specific design targets (often defined by an oracle function  $f(M)$  such that  $M^{\star} = \arg \max_{M \sim p(M)} f(M)$ ).
3.2 Data Description
We aim to curate a set of datasets covering the diversity of material, property, and task types and data amounts which will enable a variety of perspectives for machine learning model developments.
3.2.1 Material types
Inorganic bulks refer to solid substances that lacks carbon-hydrogen bonds, that is, substances that are not organic compounds, such as metals, alloys, ceramics, and minerals. They are typically large-scale structures and are used in various applications, ranging from construction materials to electronics, due to their robustness and electrical/thermal conductivity.
Organic crystals are composed of carbon-based molecules arranged in a highly ordered pattern. They exhibit distinct molecular structures and are often used in the field of optoelectronics, such as organic light-emitting diodes (OLEDs), due to their unique optical properties. They can range in size from microscopic to macroscopic.
Organic molecules are individual carbon-based compounds that can be small in size, consisting of a few atoms, or large, such as polymers. They have diverse chemical structures and are widely used in pharmaceuticals, plastics, and organic electronics due to their flexibility in design and functionality.
Bulk-adsorbate interfaces refer to the boundary between a bulk material and an absorbed species, such as gases or liquids, on its surface. They are typically at the nanoscale and play a crucial role in various fields, including catalysis, gas sensing, transport, and energy storage, by influencing the interaction and reactivity of the absorbed species with the material.
Transition metal complexes are coordination compounds consisting of central transition metal atom(s) surrounded by ligands. They exhibit unique electronic and magnetic properties and are commonly used in catalysis, medicine, and material science due to their ability to undergo redox reactions and flexible and tunable coordination environment with organic ligands.
Metal-organic frameworks (MOFs) are crystalline materials composed of metal ions or clusters coordinated with organic linkers. MOFs possess a high surface area and tunable porous structure, making them useful in applications such as gas storage, separations, and catalysis. They can range in size from microscopic crystals to bulk materials.
3.2.2 Property types
Electrical properties refer to the characteristics of a material related to its ability to conduct or resist the flow of electric current. These properties include conductivity, resistivity, and dielectric constant, which determine how well a material can conduct or insulate against electrical charges.
Mechanical properties describe how a material behaves under applied forces or loads. These properties include strength, stiffness, ductility, toughness, and elasticity. They determine how the material responds to stress, strain, and deformation, and are essential in understanding its structural integrity and performance.
Stability refers to a material's ability to maintain its properties and resist degradation over time. It encompasses chemical stability (resistance to chemical reactions or corrosion), thermal stability (ability to withstand high temperatures), and mechanical stability (ability to resist physical changes or mechanical stress).
Optical properties pertain to a material's interaction with light. These properties include absorption, reflection, transmission, and emission of light. Optical properties determine a material's color, transparency, opacity, and light-emitting capabilities, and are crucial in fields such as optics, photonics, and display technologies.
Thermal properties describe how a material conducts, stores, and dissipates heat. These properties include thermal conductivity, specific heat capacity, thermal expansion coefficient, and thermal diffusivity. Thermal properties influence a material's ability to transfer heat, its response to temperature changes, and its behavior in thermal management applications.
Energetic properties are computational models that use machine learning algorithms to predict the behavior of materials at the atomic or molecular level. They employ large datasets to learn the relationships between atomic arrangements and energies, enabling the simulation and understanding of complex material systems.
Semiconductor properties refer to the electrical behavior of materials that exhibit an intermediate conductivity between conductors and insulators. These materials can be controlled to selectively allow or impede the flow of electrons, making them ideal for electronic devices. Semiconductor properties are characterized by parameters such as band gap, carrier mobility, and doping concentration, and are crucial in the design and functionality of transistors, diodes, and integrated circuits.
3.2.3 Datasets
Materials Project (MP) Jain et al. [2013] (license: CC-BY-4.0) is a database that curates inorganic materials with computed properties including but not limited to thermal, electrical, mechanical, etc.
MatBench Dunn et al. [2020] (MIT license) is a benchmark that provides a standardized framework for evaluating and comparing the performance of different machine learning models on various materials science tasks. It curates data from multiple sources with MP as a main source. However, they do not provide machine learning ready data preparation nor implemented machine learning models and workflow.
Quantum MOF Database (QMOF) Rosen et al. [2022] (license: CC-BY-4.0) is a comprehensive database that focuses on metal-organic frameworks (MOFs) with quantum-chemical properties. The MOFs are optimized by DFT derived from both experimental and hypothetical MOF databases.
Organic Materials Database (OMDB) Borysov et al. [2017] (open access but no license specified) is a repository of organic materials. The properties are calculated using DFT for crystal structures contained in the COD database (in Appendix B additional data sources).
Joint Automated Repository for Various Integrated Simulations (JARVIS) Choudhary et al. [2020] (license: GNU v3.0) is a database that integrates materials data from various sources, including quantum mechanical calculations, materials simulations, machine learning predictions and high-throughput databases. Our datasets DFT3D, DFT2D and EDOS-PDOS are all from JARVIS database.
Open Catalyst (OC) Chanussot et al. [2021] (MIT license) is a database focused on catalytic materials. It includes three tasks: Structure to Energy and Forces (S2EF), Initial Structure to Relaxed Structure (IS2RS) and Relaxed Energy (IS2RE).
Transition Metal Quantum Materials Database (tmQM) Balcells and Skjelstad [2020] (license: CC BY-NC 4.0) is a comprehensive database focused on transition metal-based materials. It compiles experimentally derived and computationally predicted data on the structure, composition, and electronic properties of transition metal compounds.
Quantum Machines 9 (QM9) Ramakrishnan et al. [2014] (open access but no license specified) comprises small organic molecules up to 9 heavy atoms with 12 quantum chemical properties.
Carbon24 Pickard [2020] (license: CC-BY-4.0) is a synthetic dataset that includes materials made up by carbon atoms but with different structures obtained by ab initio random structure searching.
Perov5 Castelli et al. [2012a,b] (license: CC-BY-4.0) is a synthetic dataset that includes perovskite materials with the same structure but different compositions.
3.3 Machine Learning Ready Dataset Preparation
Raw data format The raw data format for both molecules and crystals is 3D structures and atomic types. Other features (such as angular information) can be derived from them.
Machine learning ready data format As explained in Sec. 3.1, a machine learning ready format for materials includes atomic types denote the atomic number of a given atom and are often converted to one-hot embeddings; atomic coordinates denote the positions of a given atom and often need to be used careful if equivariance needs to be guaranteed; edge features denote information attached to each edge which often include bond types, interatomic distances, etc.
Graph construction Three common graphs are constructed to represent the materials: multi-graph construction is a common way to represent materials as graphs which considers edges with repeated atoms (outside the unit cell) as multiple edges with the same atom; line graph construction for materials representation is first proposed in Choudhary and DeCost [2021] which a bond adjacency graph (i.e. line graph) is constructed to capture the bond and angular information.
Data split The test scenarios are often out-of-distribution of the training set. While previously common use data split is random split, it is crucial to develop data splits that mimic the real scenarios: composition split (e.g. AxBy vs AzBy) refers to splitting the dataset with same materials compositions but varying ratios; system split (e.g. AB, AC vs ABC) refers to splitting the dataset with unseen materials systems; time split refers to splitting the dataset into training, validation and test set by the date when the materials are published. Note that time split is only available for MP dataset now as the publication information for each material structure is provided in MP.
3.4.1 Predictive Evaluations