SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The evaluation metrics for predictive tasks depend on the type of prediction label: (i) scalar value prediction: common evaluation metrics include  $\mathbb{R}^2$ , mean absolute error, and mean squared error; (ii) classification: common evaluation metrics are accuracy and Area under the ROC Curve (AUC-ROC) score; (iii) vector/tensor value prediction: common evaluation metrics are similar to scalar value prediction, including  $\mathbb{R}^2$ , mean squared error, and mean absolute error. However, the distance measurement between two vector or tensor values may need to take into account the symmetry, e.g., rotation invariance, such that the distance between the rotated crystal structure and
the original structure is zero; (iv) distribution prediction: common evaluation metrics include cosine similarity, KL divergence, Wasserstein distance, etc.
3.4.2 Generative Evaluations
Evaluating generative tasks has been a notoriously challenging problem in machine learning. The evaluation metrics can generally be divided into three categories: (i) reconstruction: This evaluates the performance of the generative methods in reconstructing the exact material in the training set. (ii) basic requirement: This assesses the minimum requirement for the generated materials, such as structure or composition validity. (iii) distribution: This measures whether the generative model is capable of learning the data distribution (in terms of structure, property, etc.) in the training set, and whether it can interpolate or generalize to unseen materials.
We include all three types of evaluations metrics developed by Xie et al. [2022]: Materials match is a reconstruction metric to check if the generated material reconstructs structure in the test set. Following Xie et al. [2022], this is done using StructureMatcher from pymatgen Ong et al. [2013] which considers the match of two materials considering invariances. Validity is a basic metric to check if the generated materials are valid. Following Court et al. [2020], a material structure is valid if the shortest distance between any pair of atoms is greater than  $0.5\mathring{\mathrm{A}}$ . Structure coverage is a distribution metric to check if the generated material structures cover the training distribution. We follow Xie et al. [2022] to utilize the CrystalNN fingerprint Zimmermann and Jain [2020] and normalized MagPie fingerprint Ward et al. [2016] to define the structure and composition distance, respectively. Property statistics is a distribution metric to check if the properties of generated materials are close to those in the training dataset.
3.5 Oracle Functions for Generative Materials Design
In our efforts to facilitate the generative design of materials, we have established two categories of oracle functions. Drawing inspiration from oracle functions designed for drug discovery via machine learning Huang et al. [2021], we initially offer a fingerprint (FP)-based oracle function. This function utilizes conventional materials descriptors in tandem with classical machine learning algorithms to predict properties of interest. More specifically, we have pre-trained a random forest model for each material property prediction task across 13 different datasets as proposed by Dunn et al. [2020]. Consequently, by harnessing the pre-trained models with extracted features based on the Sine Coulomb Matrix and MagPie featurization algorithms, we can predict the properties of an input material. While the predictive accuracy of these classical, FP-based materials descriptors may not rival that of deep learning-based models, we underscore the importance of their inclusion. Their utilization enables generalization where rules apply and mitigates the risk of biasing the optimization process towards deep learning. Our second oracle function is structure-based oracle function which aids in selecting an appropriate substrate for a given material (film). By taking into account their respective structures, we have incorporated an oracle function that matches a film with a list of substrates. Specifically, this method analyzes the compatibility between a thin film and various potential substrates, particularly in terms of crystallographic orientation, matching area, potential strain, and elastic energy. This is achieved by loading the structural information of the film and substrates from respective files, then calculating and grouping matches based on substrate Miller indices. Each match, characterized by a minimum match area, is recorded with relevant details such as the substrate formula, orientations of the film and substrate, and, if available, elastic energy and strain. Then the most suitable matches—those with the smallest matching area—for each substrate orientation are identified. This method ultimately returns a list of all matches, providing a comprehensive overview of how well the film could potentially fit on each substrate. Details can be found in Appendix E.
4 Benchmarking Machine Learning Models
4.1 Existing approach
A burgeoning amount of machine learning models have been developed for learning molecular representations suitable for a variety of downstream tasks, especially machine learning potential and molecular property prediction Wu et al. [2018], Ramakrishnan et al. [2014], Chmiela et al. [2017].
Table 2: Representative work in modeling molecular and crystal structures.
Method	Representation	Symmetry	Graph construction	Angular
CGCNN Xie and Grossman [2018]	Graph	Perm. + E(3) Inv.	Multi-graph	None
ALIGNN Choudhary and DeCost [2021]	Graph	Perm. + E(3) Inv.	Multi-graph + Line graph	Explicit
SchNet Schütt et al. [2018]	Graph	Perm. + E(3) Inv.	Multi-graph	None
EGNN Satorras et al. [2021]	Graph	Perm. + E(3) Equiv.	Multi-graph	Implicit
DimeNet++ Gasteiger et al. [2020]	Graph	Perm. + E(3) Inv.	Multi-graph	Explicit
GemNet Gasteiger et al. [2021]	Graph	Perm. + SE(3) Inv.	Multi-graph	Explicit
Equiformer Liao and Smidt	Graph	Perm. + E(3)/SE(3) Equiv.	Multi-graph	Implicit
LEFTNet Du et al. [2023]	Graph	Perm. + E(3)/SE(3) Equiv.	Multi-graph	Implicit
However, most of existing work focuses on molecules without periodicity. Around the same time, another branch of work motivated directly by modeling crystal structures have been developed. We implement and benchmark models from both branches to facilitate the development of new methods in realization of both directions. Specifically, we detail them below and summarize them in Table 2.
Learning on crystal structures. CGCNN Xie and Grossman [2018] is a E(3) invariant graph neural network that leverages pairwise distances as edge features. ALIGNNN Choudhary and DeCost [2021] is an E(3) invariant graph neural network that builds an extra line graph to explicitly encode the bond angle information in addition to the original atomistic graph similar to CGCNN.
Learning on molecular structures. SchNet Schutt et al. [2018] is an E(3) invariant graph neural network that leverages pairwise distances with a continuous filter convolution to construct the message. EGNN Satorras et al. [2021] is an E(3) equivariant graph neural network that leverages relative positions between each pair of nodes and pairwise distances as the message function to update both invariant and equivariant features. DimeNet++ Gasteiger et al. [2020] is an E(3) invariant graph neural network that introduces bond angles to improve expressiveness. However, it requires triplet of atom representations to model the bond angle. GemNet Gasteiger et al. [2021] is an SE(3) invariant graph neural network that leverages dihedral angles for better expressiveness. However, it requires learning on quadruplet representations of atoms. Equformer Liao and Smidt is an SE(3)/E(3) equivariant graph transformer network. Equformer equips previous transformers with equivariant operations such as tensor product to learn equivariant features built from irreducible representations. LEFTNet Du et al. [2023] is an SE(3)/E(3) equivariant graph neural network based on equivariant local frames. LEFTNet first scalarizes vector and tensor features during message passing and convert them back by tensorizing the scalars through the equivariant frames proposed in ClofNet Du et al. [2022b]. LEFTNet introduces a local structure encoding and frame transition encoding components to further improve the expressiveness.
4.2 Experiment Set-ups
We build on top of the Open Catalyst Project (OCP) which provides reproducible implementations of commonly used 3D graph neural networks with benchmarks on OC datasets Chanussot et al. [2021]. We further implement CGCNN, ALIGNNN, EGNN, Equiformer and LEFTNet as they are not included in OCP. We test all the methods on a list of 13 representative tasks from our benchmarks with three data splits (random, composition and system). We mostly use the default hyperparameters provided in the open-source code of each method and reported them in Appendix F. As OC20 and QM9 have been largely adopted in the community, we directly take the results and report in Appendix C. Most of our experiments are conducted on single 16GB V100s while some experiments with memory-intensive models on single 80GB A100s.
4.3 Results and Discussions
Several observations can be gleaned from our benchmark results as shown in Table 3: (i) performance (observation 1): despite the competitive performance of advanced equivariant graph neural networks, invariant models such as DimeNet++ and ALIGNNN continue to be among the state-of-the-art methods; (ii) efficiency (observation 2): there is a significant variation in efficiency across the benchmarked models. ALIGNNN, DimeNet++, GemNet, and Equiformer, as illustrated in Table 4, have particularly slow runtimes. LEFTNet presents a desirable balance of accuracy and efficiency; (iii) data split (observation 3): more realistic data splits indeed increase the challenge of the task, particularly the system split. However, this trend does not hold for all properties, with dielectric being an exception;
Table 3: Benchmark on materials property prediction tasks (different colors denote distinct property types: purple (electrical), yellow (stability), green (thermal), red (optical), blue (mechanical), - denotes missing results due to extremely small test set after data split). The best numbers are darkly shaded, and the second-best numbers are lightly shaded.
Methods	omdb bandgap	qmof bandgap	mp bandgap	ismetal	edos	podos	df2d	perovskites	eform	phonons	dielectric	log_gvrh	log_kvrh	average ranking
Random	CGCNN	0.3351	0.2906	0.2759	89.16%	0.0108	0.0035	56.2634	0.0600	0.0426	62.5355	0.3122	0.0924	0.0731	5.1
ALIGNN	0.2499	0.2285	0.2010	90.26%	0.0104	0.0036	56.6110	0.0366	0.0246	29.4330	0.2827	0.0734	0.0557	2.5
SchNet	0.4396	0.3746	0.3298	88.45%	0.0120	0.0040	53.5080	0.0713	0.0447	97.6870	0.3295	0.1096	0.0775	7.3
EGNN	0.5134	0.3964	0.3263	88.15%	0.0113	0.0035	51.1220	0.0443	0.0627	65.3510	0.3004	0.0968	0.0735	6.2
DimeNet++	0.2764	0.2379	0.2062	90.69%	0.0111	0.0036	48.5600	0.0371	0.0219	39.3630	0.2658	0.0731	0.0522	2.4
GemNet	0.2516	0.2327	0.2014	89.75%	0.0104	0.0034	52.5870	0.0411	0.0236	48.1810	0.2985	0.0856	0.0555	2.9
Equiformer	0.3900	0.3221	0.3050	88.23%	0.0123	0.0039	53.4700	0.0653	0.0568	64.7630	0.2942	0.0983	0.0671	6.2
LEFTNet	0.3143	0.2328	0.1839	89.96%	0.0110	0.0038	49.3590	0.0398	0.0256	38.0120	0.3030	0.0792	0.0529	3.3
Composition	CGCNN	0.3516	0.2965	0.2840	88.20%	0.0111	0.0031	-	-	0.0411	-	0.3665	0.0946	0.0610	5
ALIGNN	0.2631	0.2266	0.2139	89.00%	0.0108	0.0031	-	-	0.0249	-	0.3533	0.0761	0.0488	2.3
SchNet	0.4624	0.3670	0.3236	87.75%	0.0121	0.0034	-	-	0.0432	-	0.3792	0.1029	0.0600	6.9
EGNN	0.5177	0.3831	0.3433	87.31%	0.0117	0.0030	-	-	0.0625	-	0.3622	0.1070	0.0685	7
DimeNet++	0.2793	0.2339	0.2234	89.37%	0.0116	0.0032	-	-	0.0224	-	0.3303	0.0762	0.0413	2.7
GemNet	0.2637	0.2321	0.2203	88.33%	0.0110	0.0029	-	-	0.0245	-	0.3279	0.0807	0.0417	2.4
Equiformer	0.4050	0.3350	0.3014	87.66%	0.0128	0.0032	-	-	0.0527	-	0.3495	0.0938	0.0552	5.8
LEFTNet	0.3822	0.2299	0.2062	88.79%	0.0116	0.0033	-	-	0.0276	-	0.3515	0.0797	0.0423	3.6
System	CGCNN	-	0.4224	0.6021	78.71%	0.0127	0.0040	53.0300	0.1132	0.0557	142.5260	0.1593	0.1124	0.0966	5.3
ALIGNN	-	0.3431	0.4997	80.07%	0.0125	0.0041	43.7480	0.0821	0.0331	63.4010	0.1636	0.0890	0.0766	3.3
SchNet	-	0.4863	0.6562	76.59%	0.0131	0.0044	41.1090	0.1405	0.0512	196.0570	0.1571	0.1194	0.0971	6.8
EGNN	-	0.4923	0.7350	75.04%	0.0130	0.0038	30.8940	0.0981	0.0826	151.5430	0.1495	0.1181	0.0974	5.8
DimeNet++	-	0.3419	0.5086	80.87%	0.0130	0.0041	35.6240	0.0806	0.0294	87.5960	0.1128	0.0893	0.0698	2.3
GemNet	-	0.3412	0.5676	78.47%	0.0122	0.0039	34.8430	0.0851	0.0422	113.2400	0.1383	0.0951	0.0712	3.2
Equiformer	-	0.4272	0.6381	75.02%	0.0136	0.0043	36.7680	0.0955	0.0603	168.2930	0.1375	0.1066	0.0841	5.9
LEFTNet	-	0.3468	0.4550	78.01%	0.0134	0.0043	34.0180	0.0790	0.0288	89.1200	0.1277	0.0915	0.0710	3.1
Table 4: Benchmark the efficiency of machine learning models with materials in different sizes (pdos~10, e_form~30, qmof~100) on a single V100 GPU (each row with same batch size except when exceeding the maximum memory, running time for 10 epochs).
CGCNN	ALIGNN	SchNet	EGNN	DimeNet++	GemNet	Equiformer	LEFTNet
pdos	68s	623s	77s	87s	158s	203s	713s	117s
e_form	8572s	41343s	10589s	12591s	35622s	40801s	62344s	13797s
qmof bandgap	678s	2277s	512s	1336s	5240s	4572s	27884s	2405s
average ranking	1.33	6	1.67	3	5.67	6	8	4.33
(iv) material type (observation 4): the performance trends across various models remain consistent for a given material property. For instance, for the bandgap property, organic crystals (OMDB) demonstrate the smallest values, followed by metal-organic frameworks (QMOF), while inorganic bulk materials (MP) exhibit the largest values.
5 Conclusion, Limitation and Future Outlook
In this paper, we introduce  $\mathbf{M}^2$ Hub as a comprehensive platform for machine learning development in materials discovery.  $\mathbf{M}^2$ Hub is a toolkit that consists of problem formulation, data downloading, data processing, machine learning methods implementations, machine learning training and evaluation procedures, and benchmark results. We cover not only the commonly considered predictive tasks on materials but also provides tools to enable the study of generative tasks on materials. Specifically, we curate 9 datasets constructed by 6 types of materials with 65 tasks across 8 property types for the predictive task. We further provide 2 synthetic datasets for the purpose of generative tasks on materials. We design 3 extra challenging and realistic data split schemes in addition to previously used random split. We believe  $\mathbf{M}^2$ Hub will serve as an essential role in machine learning for materials discovery with datasets, infrastructures and benchmarks.
**Limitation:** Despite we formulate the materials discovery pipeline in the machine learning language supported by datasets, infrastructures and benchmarks, most of the tasks do not involve experiments. However, in reality, experiment is the golden standard to test new materials. It remains a challenge to develop datasets and benchmarks for machine learning models to grow in assisting the experiment phase of materials discovery such as phase demixing and experiment planning (related work is summarized in Appendix A).
Future work: There are multiple future directions to extend  $\mathbf{M}^2$ Hub is to improve the usability for materials science community (similar to previous work Ward et al. [2018], Jacobs et al. [2020]), e.g. collect pre-trained models Xia et al., Wang et al. [2021a,b], benchmark machine learning models on specific tasks Kong et al. [2022], Bai et al. [2023], etc.
Maintenance: We are committed to maintaining and extending the usability of this toolkit to wider machine learning and materials science community. We plan to maintain this toolkit by adding new datasets, new tasks and evaluations, new oracle functions, and pre-trained models as this community continues to grow to benefit more researchers.
This project is partially supported by the Eric and Wendy Schmidt AI in Science Postdoctoral Fellowship, a Schmidt Futures program; the National Science Foundation (NSF), the Air Force Office of Scientific Research (AFOSR); the Department of Energy; and the Toyota Research Institute (TRI).
Linfeng Zhang, Jiequn Han, Han Wang, Roberto Car, and EJPRL Weinan. Deep potential molecular dynamics: a scalable model with the accuracy of quantum mechanics. Physical review letters, 120 (14):143001, 2018.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-589, 2021.
Kenneth Atz, Francesca Grisoni, and Gisbert Schneider. Geometric deep learning on molecular representations. Nature Machine Intelligence, 3(12):1023-1032, 2021.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):e2016239118, 2021. doi: 10.1073/pnas.2016239118. URL https://www.pnas.org/doi/full/10.1073/pnas.2016239118. bioRxiv 10.1101/622803.
Raphael JL Townshend, Stephan Eismann, Andrew M Watkins, Ramya Rangan, Maria Karelina, Rhiju Das, and Ron O Dror. Geometric deep learning ofrna structure. Science, 373(6558): 1047-1051, 2021.
Benjamin Sanchez-Lengeling and Alán Aspuru-Guzik. Inverse molecular design using machine learning: Generative models for matter engineering. Science, 361(6400):360-365, 2018.
Carla P Gomes, Daniel Fink, R Bruce van Dover, and John M Gregoire. Computational sustainability meets materials science. Nature Reviews Materials, 6(8):645-647, 2021.
Jonathan Schmidt, Mário RG Marques, Silvana Botti, and Miguel AL Marques. Recent advances and applications of machine learning in solid-state materials science. npj Computational Materials, 5 (1):83, 2019.
Han Wang, Linfeng Zhang, Jiequn Han, and E Weinan. Deepmd-kit: A deep learning package for many-body potential energy representation and molecular dynamics. Computer Physics Communications, 228:178-184, 2018.
Ben Blaiszik, Logan Ward, Marcus Schwarting, Jonathon Gaff, Ryan Chard, Daniel Pike, Kyle Chard, and Ian Foster. A data ecosystem to support machine learning in materials science. MRS Communications, 9(4):1125-1133, 2019.
Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain. Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm. npj Computational Materials, 6(1):138, 2020.
Conrad L Clement, Steven K Kauwe, and Taylor D Sparks. Benchmark aflow data sets for machine learning. Integrating Materials and Manufacturing Innovation, 9:153-156, 2020.
Faiza Qayyum, Do-Hyeun Kim, Seon-Jong Bong, Su-Young Chi, and Yo-Han Choi. A survey of datasets, preprocessing, modeling mechanisms, and simulation tools based on ai for material analysis and discovery. Materials, 15(4):1428, 2022.
Stefano Curtarolo, Wahyu Setyawan, Gus L.W. Hart, Michal Jahnatek, Roman V. Chepulskii, Richard H. Taylor, Shidong Wang, Junkai Xue, Kesong Yang, Ohad Levy, Michael J. Mehl, Harold T. Stokes, Denis O. Demchenko, and Dane Morgan. Aflow: An automatic framework for high-throughput materials discovery. Computational Materials Science, 58:218-226, 2012. ISSN 0927-0256. doi: https://doi.org/10.1016/j.commatsci.2012.02.005. URL https://www.sciencedirect.com/science/article/pii/S0927025612000717.
Appendix for  $\mathbf{M}^2\mathbf{Hub}$
A Additional Related Work 16
A.1 Materials Synthesis 16
A.2Experiment Control with Machine Learning 16
B Additional Data Sources 17
C Additional Experimental Results 17
D Dataset Statistics 17
E Oracle Function Details 18
F Experimental Details 19
F.1 Hyperparameters 19
As mentioned in Sec. 5, our main focus in the current version is virtual screening, inverse design and molecular simulation while ignoring tasks related to experiments where problem formulation, dataset curation, evaluation and benchmarking are much more challenging. In this section, we introduce additional related work that leverage machine learning to assist in the experiment phase of materials discovery.
A.1 Materials Synthesis
In addition to designing materials computationally, new materials have to be synthesized with experiments. Even though computational methods have been deployed to simulate or predict material properties, it is much harder to predict how materials can be synthesized. Thus, it is a nontrivial problem to study material synthesis. Moreover, the synthesis process is challenging to predict as well. Material scientists have to post-process the synthesized material to identify the synthesized structures. Specifically, X-ray Diffraction (XRD) is a commonly used technology to detect crystal structures. Chen et al. [2021] combines deep learning with reasoning module to incorporate physical constraints and identify crystal structure phase compositions from the experimental results with XRD patterns.
A.2 Experiment Control with Machine Learning
Experiment is an indispensable step to validate the properties of the materials. However, experiments are usually very expensive and even inaccessible in real scenarios. In addition, current experiment highly relies on human expert design which may be suboptimal. Therefore, automated experiment design becomes an urgent yet challenging problem. One way to improve the efficiency of experiment design is to leverage machine learning models for uncertainty estimation. Specifically, active learning can be utilized to construct a loop of decision and feedback. Machine learning models suggest the next experiment and the experiment provides feedback to improve machine learning models Ament et al. [2021]. Another promising direction is to leverage reinforcement learning methods which have an agent attempting to achieve some goal by the feedback provided by the environment Degrave et al. [2022].
B Additional Data Sources
In addition to the datasets included in our current version (Sec. 3), there are other available data sources which may be used in different purposes and we will consider to add in our future version.