SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Materials Platform for Data Science (MPDS) Blokhin and Villars [2019] (no license) is a data repository that collects experimental and computational materials data through data mining from the scientific publications. There, around half a million articles were manually processed and systematized, covering a broad spectrum of physical sciences, such as physics, chemistry, materials science, environmental science, engineering, and geology.
- Crystallography Open Database (COD) Gražulis et al. [2012] (license: CC0 1.0): COD is a open-access database containing crystallographic data on inorganic and organic compounds. It includes experimentally determined crystal structures along with associated metadata for organic, inorganic, metal-organic compounds and minerals.
- Open Quantum Materials Database (OQMD) Saal et al. [2013] (license: CC-BY 4.0): OQMD is a database that focuses on quantum-mechanical calculations of materials properties. It contains calculated data on crystal structures, electronic structures, formation energies, and other material properties for a wide range of inorganic compounds.
- Inorganic Crystal Structure Database (ICSD) Hellenbrandt [2004] (commercial): ICSD is a comprehensive database that compiles  $>281,000$  experimentally determined crystal structures of inorganic compounds. To ensure the high quality of structures in ICSD, a structure has to be fully characterized and passed thorough quality checks by its expert editorial team before inclusion. The information in ICSD is updated biannually.
- Cambridge Structural Database (CSD) Groom et al. [2016] (commercial): CSD contains over 1.1M accurate 3D experimentally crystalized structures with data from X-ray and neutron diffraction analyses. It contains diverse types of organic crystal structure (drug, pigment, etc.) and metal-organic crystals (transition metal complex, metal-organic framework, etc.).
- Novel Materials Discovery (NOMAD) Draxl and Scheffler [2019] (license: CC-BY 4.0): NOMAD is a data management platform for materials science data where users can share data freely. Here, NOMAD is a web-application and database that allows to centrally publish data. But you can also use the its utilities to build your own local database.
- Materials Cloud (MC) Talirz et al. [2020] (license: CC0 1.0): MC is built to enable the seamless sharing and dissemination of resources in computational materials science, offering educational, research, and archiving tools; simulation software and services; and curated and raw data. These underpin published results and empower data-based discovery, compliant with data management plans and the FAIR principles Scheffler et al. [2022]. In addition to database, MC also provides lectures for computational materials science, various visualization and simulation tools.
- AFLOW Curtarolo et al. [2012] (MIT license) Similar to MC, AFLOW is a composite platform includes materials database, search and visualization, simulation, and machine learning models.
In addition to our benchmark results, two popular benchmarking datasets, OC20 and QM9 have been extensively tested in previous work. We directly take the experimental results from Liao and Smidt for OC20 (Table.5) and Du et al. [2023] for QM9 (Table.6) as a reference on performance of existing approaches. Note that there are two commonly used data splits for QM9 in previous literature and they are both reported.
D Dataset Statistics
In Table 7, we report the statistics of each dataset with number of samples and number of atoms in each material.
Table 5: Benchmark on machine learning forcefield (OC20 IS2RE test set) (results taken from Liao and Smidt). (The best results are bolded.)
Energy MAE	EwT
Methods	ID	OOD Ads	OOD Cat	OOD Both	Average	ID	OOD Ads	OOD Cat	OOD Both	Average
CGCNN	0.6149	0.9155	0.6219	0.8511	0.7509	3.40	1.93	3.10	2.00	2.61
SchNet	0.6387	0.7342	0.6616	0.7037	0.6846	2.96	2.33	2.94	2.21	2.61
DimeNet++	0.5621	0.7252	0.5756	0.6613	0.6311	4.25	2.07	4.10	2.41	3.21
Equiformer	0.5037	0.6881	0.5213	0.6301	0.5858	5.14	2.41	4.67	2.69	3.73
Table 6: Benchmark on molecular property prediction (QM9) (results taken from Du et al. [2023]). (The best results are bolded.)
Task Units	α bohr3	Δε meV	εHOMO meV	εLUMO meV	μ D	Cν cal/mol K	G meV	H meV	R2 bohr3	U meV	U0 meV	ZPVE meV
EGNN	.071	48	29	25	.029	.031	12	12	.106	12	11	1.55
Equiformer	.056	33	17	16	.014	.025	10	10	.227	11	10	1.32
LEFTNet	.048	40	24	18	.012	.023	7	6	.109	7	6	1.33
SchNet	.235	63	41	34	.033	.033	14	14	.073	19	14	1.70
DimeNet++	.044	33	25	20	.030	.023	8	7	.331	6	6	1.21
LEFTNet	.039	39	23	18	.011	.022	6	5	.094	5	5	1.19
Table 7: Dataset statistics (number of samples in each dataset and size of systems in each dataset.
Datasets	df2d	edos	pdos	qmod bandgap	omdb bandgap	dielectric	log_rvrh	log_kvrh	e_form	mp bandgap	is_metal	perovskites	phonons
number of samples	636	55,659	14,244	20,425	12,500	4,764	10,987	10,987	132,752	106,113	106,113	18,928	1,265
number of atoms	7.19±4.35	10.08±9.06	7.23±5.46	113.67±68.86	82.29±26.55	16.89±14.67	8.63±8.66	8.63±8.66	29.15±30.1	30.02±29.94	30.02±29.94	5.00±0.00	7.63±3.74
E Oracle Function Details
FP-based oracle function This method generates desired properties for any specific input materials. The core concept is SCM/MagPie featurization and machine learning prediction. The oracle function first reads CIF files to extract the structures or compositions of the materials. Then, the function preprocess and prepares the appropriate data format based on the task at hand. Before training, the materials data are transformed by the Sine Coulomb Matrix and MagPie featurization algorithms to convert the raw data into a form that can be used by the machine learning model. After that, a standard machine learning pipeline is built to predict the target property for these materials. The pipeline uses random forest as the machine learning model. Finally, the method saves the predictions and returns them for user convenience.
Structure-based oracle function This method is used to match a given film with a list of substrates. The function first reads the input film and substrates to get their structures. Then, the SubstrateAnalyzer from pymatgen Ong et al. [2013] is called to calculate possible matches between the film and each substrate. It finds the best matches (with the smallest matching area) for each orientation of the substrate. Finally, all the match information is stored and returned for users. The match information includes the substrate's formula, orientations of the substrate and film, the matching area, and optionally, the elastic energy and strain. This substrate matching process can be useful in thin film deposition processes, where you want to match the crystal structure of a thin film material to a substrate material to ensure good adhesion and minimize defects.
Table 8: Hyparometers for benchmarked machine learning models.
CGCNN	ALIGNN	SchNet	EGNN	DimeNet++	GemNet	Equiformer	LEFTNet
cutoff	6.0	8.0	6.0	6.0	6.0	6.0	6.0	6.0
max_neighbors	N/A	12	N/A	N/A	N/A	50	500	N/A
num_layers	5	3	6	4	3	3	6	4
hidden_depth	256	128	128	128	192	128	512	128
learning_rate	1e-4	1e-3	1e-4	1e-4	1e-4	5e-4	1e-3	5e-4
optimizer	AdamW	AdamW	AdamW	Adam	AdamW	AdamW	AdamW	AdamW
schedule	N/A	N/A	N/A	N/A	N/A	ReduceLROOnPlateau	Cosine	N/A
training_epochs	500	500	500	500	500	500	500	500
F Experimental Details
We report the general hyperparameters shared across models in Table 8. For model-specific parameters, we report in https://github.com/yuanqidu/M2Hub/config.
Table 9: Detailed system size statistics for each task in each dataset.
Data Name	Task	Mean	std
Carbon24	carbon24	9.21	3.58
DFT2D	jdft2d	7.19	4.35
DFT3D	avg_elec_mass	8.58	6.70
DFT3D	avg_hole_mass	8.58	6.70
DFT3D	bulk_modulus_kv	6.71	5.11
DFT3D	dfpt_piezo_max_dielectric_ionic	9.83	6.41
DFT3D	dfpt_piezo_max_dij	7.99	4.96
DFT3D	dfpt_piezo_max_eij	9.83	6.41
DFT3D	ehull	10.34	8.84
DFT3D	encut	10.33	8.83
DFT3D	epsx	8.57	7.15
DFT3D	epsy	8.57	7.15
DFT3D	epsz	8.57	7.15
DFT3D	exfoliation_energy	9.42	6.79
DFT3D	formation_energy_peratom	10.34	8.84
DFT3D	kpoint_length_unit	10.33	8.83
DFT3D	magmom_outcar	10.21	8.72
DFT3D	max_cfg	5.79	3.53
DFT3D	mbj_bandgap	8.01	6.37
DFT3D	mepsx	8.06	6.40
DFT3D	mepsy	8.06	6.40
DFT3D	mepsz	8.06	6.40
DFT3D	n-Seebeck	10.92	8.59
DFT3D	n-powerfact	10.92	8.59
DFT3D	optb88vdw_bandgap	10.34	8.84
DFT3D	optb88vdw_total_energy	10.34	8.84
DFT3D	p-Seebeck	10.92	8.59
DFT3D	p-powerfact	10.92	8.59
DFT3D	shear_modulus_gv	6.71	5.11
DFT3D	slme	10.98	7.28
DFT3D	spillage	7.58	5.74
EDOS-PDOS	edos_up	10.08	9.06
EDOS-PDOS	pdos_elast	7.23	5.46
MatBench	dielectric	16.89	14.67
MatBench	log_gvrh	8.63	8.66
MatBench	log_kvrh	8.63	8.66
MatBench	mp_e_form	29.15	30.10
MatBench	mp_gap	30.02	29.94
MatBench	mp_is_metal	30.02	29.94
MatBench	perovskites	5.00	0.00
MatBench	phonons	7.53	3.74
Perov5	perov5	5.00	0.00
OC20	S2EF	73.25	30.96
OC20	IS2RE	77.75	31.40
OC20	IS2RS	77.75	31.40
OMDB	band_gap	82.29	26.55
QM9	all	18.02	2.94
QMOF	bandgap	113.67	68.86
tmQM	all	65.99	27.01