The Open MatSci ML Toolkit: A Flexible Framework for Machine Learning in Materials Science
Authors: Santiago Miret
Kin Long Kelvin Lee
Carmelo Gonzales
Marcel Nassar
Matthew Spellings
Venue: Accepted by TMLR ()

Abstract:   We present the Open MatSci ML Toolkit: a flexible, self-contained, and scalable Python-based framework to apply deep learning models and methods on scientific data with a specific focus on materials science and the OpenCatalyst Dataset. Our toolkit provides: 1. A scalable machine learning workflow for materials science leveraging PyTorch Lightning, which enables seamless scaling across different computation capabilities (laptop, server, cluster) and hardware platforms (CPU, GPU, XPU). 2. Deep Graph Library (DGL) support for rapid graph neural network prototyping and development. By publishing and sharing this toolkit with the research community via open-source release, we hope to: 1. Lower the entry barrier for new machine learning researchers and practitioners that want to get started with the OpenCatalyst dataset, which presently comprises the largest computational materials science dataset. 2. Enable the scientific community to apply advanced machine learning tools to high-impact scientific challenges, such as modeling of materials behavior for clean energy applications. We demonstrate the capabilities of our framework by enabling three new equivariant neural network models for multiple OpenCatalyst tasks and arrive at promising results for compute scaling and model performance. The code of the framework and experiments presented in this is paper are publicly available at https://github.com/IntelLabs/matsciml.

Published in Transactions on Machine Learning Research (07/2023)
The Open MatSci ML Toolkit: A Flexible Framework for Machine Learning in Materials Science
Santiago Miret Intel Labs
cantiagoo.miret@inte...
Kin Long Kelvin Lec Intel Accelerated Computing Systems & Graphics
Stn.lomg.kclvin.icc.
Carmelo Gonzales
cormelo.gonsalee@@meee.ccom
Intel Labs
Marecl Nassar
wuarell.aassrrOnnteel.
Intel Labs
Matthew Spellings
mspells @nveellil@weectoonnstttttteee
Vector Institute
Reviewed on OpenReview: https: openreview. net/ forum? id= QBMyDZsPMd
Abstract
flexible, self-contained, and scalable Python- based framework to apply deep learning models and methods on scientific data with specific focus on materials science and the OpenCatalyst Dataset. Our toolkit provides: 1. scalable machine lcarning workflow for matorials science leveraging PyTorch Lightning, which enables seamless scaling across different computation capabilities (laptop, server, cluster) and hardware platforms (CPU, GPU, XPU). 2. Deep Graph Library (DGL) support for rapid graph neural network prototyping and development. By publishing and sharing this toolkit with the research community via open-source release, hope to: 1. Lower the entry barrier for new machine learning researchers and practitioners that want to get started with the OpenCatalyst dataset, which presently comprises the largest computational matcrials science datasct. Enable the scientific community to apply advanced machine learning tools to high-impact scientific challenges, such as modeling of materials behavior for clean energy applications. We demonstrate the capabilities of our framework by enabling three new equivariant neural network models for multiple OpenCatalyst tasks and arrive at promising results for compute scaling and model performance. The code of the framework and experiments presented in this is paper are publicly available at s tiitthu.... com/ IntelLabs/natscinn
We present the Open MatSci ML Toolkit:
1 Introduction
Recent years have great advances in applying advanced machine learning methods, especially novel deep learning methods, to scientific challenges that rely computational modeling in the development of new physical systems (Krenn el al.] 2022 Axelrod et al. 2022 Junper et al. 2021). Modern deep neural networks trained to reproduce physical calculations, which are used to understand and optimize the behavior of real systems, can operate with high accuracy and are often orders of magnitude faster compared to methods
based solely on human-derived calculations (Friederich et al. 2021 Chen Ong 2022
Fung et al. 2021
Equal contribution.

---

Published in Transactions on Machine Learning Research (07/2023)
However, these representational capabilities usually come with significant engincering costs: The large variety of physical systems studied in chemistry, as well as the methods to represent those physical systems along with the data-driven paradigms used to predict their behavior, can make it cumbersome, time-consuming and error-prone to adapt models and tools to new datasets and applications. To mitigate these frequent challenges, we present the Open MatSci MT. Toolkit, a flexible framework develop deep learning techniques for chemistry and materials science applications.
Catalysts essential components of chemical processes that help accelerate the rate of various chemical reactions. Catalytic materials design, especially the design of low-cost catalysts, remains an ongoing challenge that will continue to become more and more important for variety of applications, including renewable energy and sustainable agriculture. The OpenCatalyst Project, jointly developed by Fundamental Al Research (FAIR) at Meta AI and Carnegie Mellon University's Department Chemical Engineering, encompasses one of the first large-scale datasets to enable the application of machine learning (ML) techniques, with the full dataset containing over 1.3 million molccular relaxations of 82 adsorbates on 55 different catalytic surfaces [Chanussot* et. al. 2021). The original release from 2019 has also been supplemented by subsequent. updates in 2020 and 2022, along with active leaderboard and annual competition (Tann et al. 2022
The notable effort of providing high-quality data for catalytic materials is a major step forward in enabling ML researchers and practitioners to innovate on materials design challenges at great computational scales, and has already enabled the development of new geometric deep learning architectures ( (Klicpera et al. 2020b) Gasteiger ct al.) 2021 , some of which have been trained with nearly a billion parameters (Sriram et al. 2022
Open MatSci ML Toolkit
DGL Powered
PyTorch Lightning Powered Computation Scaling
Graph Neural Networks
While the toolkit of the original OpenCatalyst. repository is very powerful, it incorporates significant amount of complexity due to various interacting pieces of software: model definitions, functions for distributed training, and task abstractions. These components are often not self-contained, which makes it difficult for new MT. researchers to navigate and interact with the repository, create new architectures or modeling methods, and run experiments on the dataset. To address these usability challenges, the Open MatSci ML Toolkit, flexible and casy- to-scale framework for deep learning on the Open Catalyst Dataset. We designed the Open MatSci ML Toolkit with the following basic principles and compelling features:
Hardware Agnostic Experiment Scaling
Besearcher Friendly GNN Design
Experiments Quickly
Leverage DGL Abstractions
Easily Design New GNNs
by Adding More Hardware Devices Integrated support for distributed XPU, precision. hyperparameter
Data Pipelines Engineering to
DGL Compatible
optimization and more
Open MatSci ML Toolkit enables researchers to create and perform scientific machine lcarning on the OpenCatalyst in a scalable manner leveraging DGL and PyTorch Lightning.
Figure 1:
introduce
Ease of use for new ML researchers and practitioners that want to get started with the OpenCatalyst. dataset.
Scalable compulation leveraging PyTorch Lightning Falcon & The PyTorc Lightning Team 2019 across different computation capabilities (laptop, server, cluster) and hardware platforms (i.e. CPU, GPU, XPU) without sacrificing the scientific abstraction.
Support for DGL WWngg et al. 2019) for rapid GNN development to complement the original repository's usage of PyTorch Geometric (Fey & Lenssen 2019 In this work we showcase set of models built our toolkit for OCP-20 described in Section 4
2

---

Published in Transactions on Machine Learning Research (07/2023)
The examples outlined in this study and its associated open-source repository show how to get started with the Open MatSci ML. Toolkit, using simple Python script, Jupyter notebook, or the PyTorch Lightning CLI for a simple training instance on a subset of the original dataset (referred to as a development or dev-set) shipped with the repository that can be run on a laptop without neoding to download and preprocess the minimal canonical dataset. Subsequently, we scale our example Python script to large compute systems with advanced equivariant doep learning models, including multi-GPU training on a single compute node, to distributed training across multiple nodes and GPUs in a computing cluster. Leveraging the capabilities of both PyTTorch Lightning and DGL, enable compute and experiment scaling with minimal additional overheard and complexity.
2 Background & Related Work
Geometric Deep Learning (GDL) generalizes neural networks to non-Euclidean domains such as graphs and manifolds (Bronstein et al. 2017 2021 Such domains are increasingly used to model systems in scientific applications, such as molecular and crystal structures like those found in the OpenCatalyst project. In particular, GDL provides way to represent entity interactions and various invariances and equivariances under geometric transformations-ber point cloud dataaare are the most common representations of such molecule-catalyst systems with different GDL methods operating on inputs with various inductive biases.
vital for modeling molecules and catalyst interactions. Graphs and
Graph Neural Networks, one of the earliest GDL applications to atomic systems, process graph representation of molecular and solid-state material systems (Gilmer et al. 2017). These neural networks are able to take advantage of the natural representation of molecules as graphs, whereby message passing between nodes (aloms) resembles atomic interactions. Despile this, vanilla message passing GNNs are unable to distinguish between certain motifs and are bounded by the performance of the L-Weisfeller-Lehmaa (WL) isomorphism test [Xu et al. 2019). Even with recent advances in pushing GNNs beyond the WL-test (Morris) et al. 2019), such networks do not utilize any of the known physical symmetries, thereby reducing their effectiveness on molecular data beyond their theoretical limitations.
Equivariant Neural Networks restrict the space of functions learnable by the network to obey symmetrics found in the input data. Symmetries are powerful inductive bias that drastically narrow the solution space that needs to be explored by the neural network, thereby making learning easier and improving generalization. Physical systems often exhibit many symmetries, either in their overall structure (e.g. water within the C2v point group), within structural motifs (e.g. shree-fold rotation symmetry in methyl rotors), and, when nuclear dynamics are considered, complete nuclear permutation groups (Bunker & Jenson| 1998 Williams & Eisfeld 2020) (c.g. ethanc and the G36 group (Mellor ct al., 2019t). Composing many layers that individually respect symmetries relevant a variely of problems has proven to be a powerful method to design deep neural network architectures for range of modeling challenges. Exploiting such symmetries has emerged as a driving force for designing neural networks for physics models, for example as equivariance under the SO(3) group; that is, equivariance under translations and rotations in 3D space. Various works, such as Thomas et al. (2018); Anderson et al. (2019); Fuchs et al. (2020), leverage group representations to achieve full equivariance. While equivariant models can be more parameter efficient, they can also be computationally
expensive or constrained in expressivity due to the model basis.
Equivariant/Directionna /Directional Graph Neural Networks merge both graph and point cloud representations to leverage the benefits of both representations. One of these equivariant GNNs (E(n)-GNN by [Satorras et al. (202)) achieves this by separating node features into spatial components that are equivariant and invariant atomic features. These are then mixed together in a series of aggregation steps that preserve the spatial equivariance and feature invariance. Other GNNs take a more direct approach to equivariance by embedding angles, distance and dihedral angles in classic basis sets such as Legendre polynomials and spherical harmonics (GemNet |Gasteiger et al., 2021) and DimeNet (Klicpera et al. 2020a]), which possess well-known symmetry properties. These models can be viewed as directional GNNs since they typically expand their receptive field beyond the 1-hop neighborhood utilizing angular data to direct the message propagation. Directional
httpp:: ttttps://gtthbb... catluuu..oom///eee ham///nnellaba//ttcinii AA collection of atomic particles in three-dimensional space, which unlike graphs do not have explicit connectivity.
3

---

Published in Transactions on Machine Learning Research (07/2023)
models have high representational power; but come at significant computational and memory cost. The performance of these models depends on the task and complexity of the data: EGNN can perform on-par with DimeNet and GemNet on simple classification tasks (Satorras et al., 2021); however, directional models
typically perform better in more advanced applications such as conformer search (Ganea et al., 2021 Jing
et al. 2022).
Short-Range Equivariant Networks restrict the ability of signals to travel long distances as layers are composed. In typical graph neural network, signals from one node are able to travel additional neighbor hop away in the graph for every message passing layer in the network; in contrast, short-range networks avoid transmitting these signals. These short-range restrictions can help promote localized reprosentations that are more efficient to learn. One such architecture uses geometric algebra or Clifford algebra to formulate rotation-invariant and -equivariant combinations of input vectors, which are transmitted in a permutation- equivariant way vía an attention mechanism (Spellings 2021p. The Allegro framework (Musaclian al.) 2022) learns produce localized representations from a series of cquivariant tensor products, rather than using geometric algebra to achieve rolation equivariance.
3 Software Framework
The Open MatSci ML Toolkit software framework designed with great emphasis on abstraction and inheritance in order to maximize reusability and agility for machine learning researchers. These ideas are achieved in part by present-day best practices in Python as object-oriented language, and through modern, specialized frameworks such as PyTTorc Lightning and DGL. We believe these design choices make it significantly easier to apply novel model architectures and training techniques to scientific data, in particular the OpenCatalyst. dataset. In addition to enabling researcher to construct graph neural networks with DGL as described in Section 3.2 we also enable support for point cloud represontations, which are described in Section 3.3 These capabilities significantly the options for research to design diverse sets of model architectures that can leverage both graph and point cloud representations for atomistic data drawing upon some of the approaches outline in Section 2 In the following sections, we will discuss changes in abstractions and refactoring steps from the original OpenCatalyst implementation.
3.1 PyTorch Lightning Refactor
In modern AI/ML workflows, the concept of "MLOps" comprises the lifecycle from model conception and implementation, training and testing in a variety of software software/hardware harddare environments, to drawing inferences on new data, and all of the iterative cycles in between. Thus, a non-negligible amount of time spent by researchers for new workloads is typically in engineering: interfacing data with new architectures, metric logging, performance profiling, and ensuring consistent functionality through the development process from testing locally laptop to distributed training multiple computing nodes and across different types of accelerators. Because of the grand scale that OpenCatalyst aims for and successfully achieves, a large amount of the original codebase corresponds to performance and functionality; meaning that complexity is necessary to be able to take advantage of data parallelism, to perform hyperparameter optimization, and to support the various catalyst prediction tasks. This lays a significant amount of responsibility on both developers and users: the former must create a comprehensive suite of tests and rely heavily on CT/CD to ensure functionality, and the latter must navigate a maze of software dependencics and documentation, which are also maintained by the developer.
One half of the conceptual changes in the Open MatSci ML Toolkit-the other half being the primary graph framework-is to offload MLOps related components to a well-designed and maintained framework: PyTorc Lightning FFaconn & The PyToorch Lightning Team 2019). By reusing dataset and framework components from OpenCatalyst and relying on the pipeline abstractions from PyTorch Lightning, we can maintain a more compact codebase while providing more flexibility, extendibility, transparency, and functionality. Figure 2 illustrates the end-to-end pipeline/directed directed acyelic graph for the Open MatSci ML Toolkit, whose elements should be relatively familiar to those who have used OpenCatalyst and/or PyTor Lightning.
4

---

Published in Transactions on Machine Learning Research (07/ 2223))
Lightning data module
D2EaFDG..DttModudle
User loop control
Lightning Trainer
Launch
network backbone
Lightning module
EGNN
S2EFLilModule
Callbacks
Script/ / Lightning CLI Slurm
Logger
Accelerator
Environment
Illustration of the Open MatSei ML Toolkit pipeline with concrete components using the Structure to Energy & Forces ("S2EF") task. Dataset/tae task splits and configurations are specified through LightniingDataMotulee.. Task specific LightningModules encode the logic for training, metric logging, and how data passed from dataset to an underlying abstract deep learning model. The Trainer interface provides an the ability to control feedback (e.g. logging, progress bars), training flow (Callbacks), and XPU usage without the need to modify the pipeline source code.
Figure 2:
3.1.1 Data abstraction
In order to support future neural network research, we expanded the scope of the original OpenCatalyst dataset to support graph and non-graph data structures, as well as implemented a number of quality of life improvements to the general developer workflow. We refer the reader to the Appendix (i.e. Figure A 5) for more details pertaining to the changes, and here we only briefly highlight the core differences in user experience.
One of the core goals of the Open MatSci ML Toolkit to have continuity from developing and testing on local environments such laptops using the pipeline in high performance computing environments. In terms of data pipeline abstraction, on the one end the Open MatSci ML Toolkit provides preprocessed, miniature (~100 graphs) development or "devset"s: this circumvents the need to download, extract, and preprocess the data on personal computers constrained by storage and by computational power, while allowing researchers to protolype on the full pipeline. The development sets are created by laking random subsplits of the smaller data splits from the OpenCatalyst dataset. We also provide a mechanism for creating other splits not included with the Open MatSei ML Toolkit, facilitating further rescarch into data efficiency. To use the devsets for development, there is convenient mechanism for retrieving the DGL version of each task:
from ocpmodela. lightning Pata_utile import 32EFDeLDataModule 1S2REDGLDataNodue default settings optimized for local development: small batch, no parallel loaders devset_modul. S2EFDGLDataModulee www.mddwseseeseeeee
On the other end of the spectrum, where one wishes to distribute the dataset across multiple workers on multiple compute nodes, users can leverage the same data modules as the miniature case: the DistribiteeDattaarrllll data sampling and loading is offloaded to PyTorch Lightning internals. We will discuss this further in Section 3.1.
As alluded earlier in this section, the Open MatSci ML Toolkit data pipeline abstraction is designed to facilitate exploration of other data representations of materials systems, including algorithms less strongly linked to graph-based message passing. This includes recent approaches based on geometric algebra or Clifford algebras for point clouds by Spellings 2021), as well as methods based on tensor products, such as Musaelian et al. 2022) These algorithuss operate locally on the bonds within a point cloud without typical message passing from one atom to another, while retaining the data efficiency imparted by model equivariance.
3.1.2 Model abstraction
The model abstraction, as seen in the bottom left nodes in Figure 2 comprises a neural network backbone and lask-specific LightningModule. In the concrete example described in Section 4 the EGNN model represents a subclass of an Abstracttnnerg/Moell: a model that takes arbitrary imput, and predicts the energy. For instance, a graph-based model will process nodes and perform some readout operation to reduce node and graph level features to a scalar value for the energy. At a higher level, the task specific S2EFLitModule is instantiated by passing instance of EGNN, and implements the logic for training (i.e. worward-backward
5

---

Published in Transactions on Machine Learning Research (07/2023)
passes), validation and testing, as well as logging. By conceptually separating the model (i.e. the neural network itself) from the training mechanism, researchers only need to focus architecture development by subelassing AbstractEnergy/Modell as the rest of the pipeline stays the same barring changes in which kind of data is required by the model.
3.1.3 Training loop
For typical regression tasks such as the initial structure relaxed energy ('IS2RE"), the training loop takes advantage of aulomatic oplimization in PyTTrch Lightning, meaning that the backpropagation and optimization steps are abstracted away from the user. In cases with specialized autograd steps, such as the force component of "S2EF", and in many physics inspired neural network architectures, backpropagation is manually performed. While this highlights a gap in coverage by MLOps frameworks such as Py Torch Lightning, it also illustrates their open-endedness: one can maintain the flexibility necessary for research, while keeping the many other benefits and best practices guided by PyTorch Lightning.
With regards to changes in the behavior of the pipeline, users can rely on modular PyTorch Lightning components that batch, at the end of the validation epoch spoch-with
trigger certain parts of the training loop-such at the beginning of training with the Callbacks abstraction. The scope of such events include, but is not limited to: early stopping according to some validation metric, model checkpointing based on training intervals or metrics, as well as more specialized and flexible tasks like code profiling targeted to specific pipeline components-25 all without the need to inspect and modify the core source code.
Regarding logging/ /epprn/gexprimentt tracking, the Open MatSci ML Toolkit relies on PyTorch Lightning Loggers, which are able to log locally CSV or Tensorboard format, as well as hosted services such as Weights and Biases and SigOpt; the latter of which can also be used for hyperparamcter optimization (HPO), which makes up valuable tool in researcher's toolkit. Users are not confined to a single choice of logging HPO platform, providing flexibility in tooling that fits a given researcher's workflow.
The final component relevant to the training process is the PyTorch Lightning Trainer class, which orchostrates the components mentioned above and execules training, validation, lesting, and inference loops. The Trainer interface also configures more performance oriented settings such as accelerator usage, distributed compute, and mixed precision shown in the example configuration below:
trainer Trainer
max max_epochs epochssc callbacks= accelerator=" "pp"" use XPUs; precision="bi1"" new data types; strategy="ddp", devices=8, aun_nodes=4
pipeline behavior;
8 workers per 4 nodes;
The main advantage is being able to seamlessly navigate between development and training cycles: the core pipeline remains unchanged, yet with a simple change in configuration, the user is able to take advantage of computational resources as they become available. Under the hood, the Lightning abstractions handle data movement. lo devices, aulocasting correct contexts, and orchestrating workers.
3.2 Deep Graph Library (DGL) Refactor
The original OpenCatalyst repo leverages Pytorch Geometric (PyG) for implementing various neural networks described in Section [ In the Open MatSci ML Toolkit, we chose to complement the original implementation by building on top of the Deep Graph Library (DGL). While both PyG and DGL are highly performant libraries for graph neural works and the decision to choose one over the other often subjective, we motivate our choice of DGL for this library as follows:
Graph Abstraction: The DGL graph data structure AgL.DGLGraph offers more flexibility for storing diverse molecular data the PyG structure torch_ barch_geometric.om data Data. This abstraction

---

Published in Transactions on Machine Learning Research (07 (07/2023)
allows for more general data pipelines amicable to experimentation and ablation studies; two qualities that are important in scientific exploration.
Cross-platform Optimization: While both DGL and PyG well-optimized for single-node GPU deployment, DGL also supports additional platforms, in particular efficient sparse matrix algorithms for CPUs with documented compute benefits datacenter CPUs.
Support for Hampling-Basede and Distributed Training beyond Data-Parallel: many applications involve large graph data that does not fit onto single GPU. Such cases require specialized sampling techniques to either shrink the graph size or distribute the storage across multiple nodes. While both PyG and DGL. support sampling-basee baaedd training, DGL is more mature when comes to sampling graphs and running distributed training of GNNs (Mostafa,
2021
Zheng et al. 2021
Next, we discuss each of the above points in more detail.
3.2.1 Graph Abstraction
PyG's Data has a graph data structure composed of the following fixed attributes: node feature matrix x, edges edge_ index, edge attributes edge_atta _age_attr, labels y, and node position matrix pos. Additional attributes can set using keyword argument collection in the constructor. contrast, DGL's DGLGraph provides a dictionary-style access to graph data through DGLGraph ndata (node features) and DGLGraph. edata (edge features). While, these dictionaries do not impose any restrictions feature names number, the DGLGraph object will enforce that the feature dimension match the number of nodes and edges.
While PyG's Data covers large part of the use cases in the GNN world, we argue that using the flexible DGLGraph as example representation decouples the dataset from the model-specific data pipeline, while retaining data consistency such as matching the number of features and edges. This lends itself to a wide variely of customized pipelines that can explore various model explorations and ablation studies. For example, consider molecular data structure that contains various features on both atoms and bonds: QM9 Ramakrishnan et al. 20014); common molecular property prediction benchmark, includes atom features detailing atom coordinates x, atom type tp, atomic number 2, number of hydrogens nH, and hybridization Gilmer et al. 2017). A given set of molecular features would then contain the following fields:
class Example (NamedTuple):
x: [flaat
tp: List [bool]
z: int
nH: int
bonds: List (Tuple [int, int]]
Assuming all the above fields have been cast into their appropriate Tensor types, converting Example to PyG's Data can be done as follows
example_ dict example. axxampee_aaddcctt asdict()
src, dst zip dict pop pop''ondss') edge_index torch. hatack ((are, dat))
pos example_dict.pop pop(("))
assign to fixed attributes embed_tp, embed_z, embed_nH embed (exampe)))))
feat torch cat (Aembed__ enbed. embed_nH) data Data(pos-pa) 10 assign to generic container
x=feat, edge__ndex-edge_Btte index)
11 data = Data tataspos=pp,, edge bbge__indr==dge.. index #*exampee_dict)
while DGL's DGLGraph this would look like
dst zip (*example dict dico..pop'"oods"" dgl_graph dgl graph ((are, dst)) ('bonds"))
dgl_graph.. ndata asdict

---

Published in Transactions on Machine Learning Research (07/2023)
The above example demonstrates that the DGLGraph can accommodate diverse data without having to perform any special model-specific preprocessing while retaining graph consistency across assigned nodes and features. PyG Data, on the other hand, either requires feature preprocessing and assignment to the field or generically attaching the features to the data object.
3.2.2 Posss-Paatfom Optimization
GPU acceleration plays a pivotal role in modern neural network training in general, including for GNNs. Recently, there has been an increasing interest in deploying GNNs, particularly in distributed computing settings, on other a diverse hardware platforms including datacenter CPUs and custom deep learning accelerators from nascent hardware vendors. While both DGL and PyG are well optimized for CUDA, DGL implements highly efficient kernels for sparse matrix operations central to graph processing (hrough libxsmm, providing more flexibility in the type of hardware used for training and inference (Avancha et al. 2020
3.2.3 Sampling and Distributed Training Beyond PatalParallel
Increasingly, single device training of GNNs is reaching its limits as graphs and models scale in size and memory requirements. A common way to address these limits is to shrink the size of the graphs by sub- sampling the graph so that it fits into single device memory Hamilton et al. (017)). Yet, this lype of sampling can lead to performance degradation due to the discarded neighborhood information, prompting a surge in new distributed training for GNNs (Hamilton et al. 2017 Md et al.) 2021 Mostafa 2021 Zheng et al.) 2021). DGL already supports various forms of graph distributed training for both sampled and full batch training, and this capability could prove invaluable for molecular design datasets as both model and data complexity grows. While these types of distributed training might not be directly relevant OpenCatalyst, they could be useful in future applications with large system sizes, such as large-scale molecular dynamics with billions
of atoms (Musaelian et al. 2022 Guo et al. 2022 Shibuta et al. 2017).
3.3 Point Cloud Representation
Up to this point, our discussion in Section 3 has mainly highlighted graph representations of atomic data, and subsequently, graph neural network architectures. While graphs are natural way to represent systems of bonded atoms, we also aim to support alternative data representations in our framework, in order to enable development of diverse deep learning architecturos. One such example, which is already implemented in Open MatSSi ML. Toolkit, are alom-centered point cloud representations that are used to achieve translation and rotation equivariance in several recent architectures, including NequlP (Batzner et al.), Geometric Algebra Attention Networks (Spellings 2021 and Allegro (Musaclian et al. 2022
Atom-centered point cloud representations often incur high memory cost, leading us to build our procedure on top of the data abstractions discussed earlier, while only constructing neighborhood point clouds for the most relevant atoms. In the case of OpenCatalyst, this means that we extract the molecular adsorbate (typically a few atoms) and the surface, which-according to prior chemical knowledge should contribute the most to describing the system as a whole. Furthermore, each catalyst + molecule system, we randomly sample a number of additional bulk crystal atoms below the surface layer that constitute the substrate for a more holistic description. We construct atomic features using symmetric one-hot embeddings based on the corresponding atomic numbers, which seeks to mirror the treatment for the positions and embed information about the central atom, as well as cach non-central atom in the point cloud. Finally, the point cloud data are batched by padding molecule and substrate dimensions with zeros that can subsequently be masked during computation. Given that the point cloud representation shares the same core components as graph datasots, the same pipeline can be used with minimal code changes, largely owing to inheritance patterns (see Figure 3 in Appendix C The same also applies for different methods of constructing atomic point cloud representation that may be different. from atom-centered point clouds we apply in our experiments.
8

---

Published in Transactions on Machine Learning Research (07 2022))
4 Experiments & Testing
We designed our experiments to showcase the capabilitics of our toolkit highlighted in Section We perform all our experiments on TitanVV GPUs with the number of GPUs and experimental runtimes detailed in each individual experimental section. Concretely, our experiments demonstrate:
Seamless Compute Scaling: We show the benefits of compute scaling across multiple devices in a single node and multi-node setting for different OpenCatalyst tasks described in Section 4.1
Model Enablement: We implement three new model architectures and show generally reasonable
results on OCP-20 tasks described in Section 4.2 Concretely, we provide training results for the following architectures:
F(n))GNN by Satorras et al.
2021 a rotation-equivvaiian graph neural network.
MegNet by Chen et al. (2019) a domain specific GNN for materials science applications.
Geometric ALgebra Attention Network (GALA) by Spellings (2021) a short-range cquivariant
network based on Clifford algebra that applies point cloud representations.
As shown in Table 1 the success of the new models relies on our compute scaling capabilities and our flexible MLOps data and model pipelines.
All of the models presented have the inductive bias of rotation equivariance; in general, equivariance for functions f()). (() for an entity I is defined as: f(o(1) (((()). Intuitively this means that the features of an entity transform equally with a given manipulation, such as a rotation. This is particularly useful for property prediction in material compounds, such as the IS2RE and S2EF tasks, where rotation of the entire compound itself does not affect the properties of the compound. Each of the models, however, impart equivariance in different ways through their architecture. We refer the reader to Section and the original papers for further details.
4.1 Hardware Scaling Capabilities
Model
Number GPUs
Epoch Time (h)
Experiment Time (h)
Task
Size
IS2RE (All)
E(n)-GNN
72k
24
0.063
2.84
MegNet
986k
0.108
Gala
1.5M
24
1.32
59.6
S2EF (2M)
B(n)-GNN
72k
32
0.404
18.2
MegNet
1.2M
40
1.36
61.3
Table 1: Compute details for our experiments on IS2RF-All (~500K Training & 255 Validation) and S2EF-2M (2M Training & 1M Validation) indicating the compute requirements to perform effective model training OCP-20 in reasonable wall-clock time. Both model architecture (Gala > MegNet > E(n)-GNN) and task complexity (S2EF IS2RE) influence compule needs.
As an illustrative example, deploy E(n)-GNN using the Open MatSci ML Toolkit to the OCP-20 S2EF task with 200K training samples and 1M validation samples, which is amendable studying the compute performance one can achieve using our framework. We apply distributed data-paralla training based on the Torch library (Li et al. 2020) to scale training across multiple devices on single GPU compute node, as well as multiple GPU compute nodes. As seen in Figure 3 single node scaling to multiple GPUs for the S2EF and IS2RE tasks shows a docreasing benefit as more GPUs are added, asymptotically approaching the apparent limit of parallelizable computation benefits for this particular setting. Epoch training throughput in the multi-node setting for two nodes suggests close linear scaling. While the overall benefits of compute
9

---

Published in Transactions on Machine Learning Research (07 2023)
S2EF Training (200K)
500
Train (200K)
IS2RE Training
S2EF Validation (1M)
Val (1M)
IS2RE Validetion (25K)
250
1500
200
2000
350
200
Number GPUS
Number GPUS
(a) Single Node Scaling on S2EF (b) Multi-Node Node Scaling on S2EF (c) Single Node Scaling on IS2RE
Figure 3: Time per Epoch (s) for Multiple Devices on Single Node and 8-Device Multi-Node Sctting for the S2EF Task on 200K and 1M Validation Data Split. and Single Node Scaling for IS2RE Task on 10K and 255 Validation Data Set.
scaling increase with more intensive task, the overarching compute time, both in core-time and in wall-clock time, also increases making the overall experiment more costly.
4.2 OCP-20 Task Performance
We leverage our toolkit to perform training experiments two large-scale tasks within OCP-20:
IS2RF-AlI: Tnitial Structure to Relaxed Energy for all data contained OCP-20 (~500K training samples). IS2RE is trained on MAE (LI) loss: C 2 1 model parameterized by yi being the energy labels and z; being the input data features. The primary aim of this task is to model the absorption energy of the catalyst adsorbate in the relaxed state, which is generally most relevant for catalytic design applications.
" Σ 1// J f fenergy (z:)l with fo being the
S2EF-2M: Structure Energy Forces prediction for 2 million training samples. S2EF is also trained on the L1 loss with separate term for energy prediction and force labels: _ C L L Lforce energy
D 19; - fo fenergy (2:)1 + Ei= foforee_ fforce (z:)l. The forces obtained by taking the (x) with being the atomic positions in the system. This requires significantly larger memory and compute compared to the IS2RE task, as well as support for multiple passes of automatic differentiation. The S2EF tasks aims to model the prediction of energy and forces on a single frame thereby training the model to provide a more general approximation the original chemistry simulation.
gradient of energy predictions with respect to atomic positions: fforce - _ df
We perform the IS2RF-AII task for all three models and the S2EF tasks for E(n)-GNN and MegNet. We did not perform S2EF with Gala because the overall memory requirements were too large to fit onto GPU, mainly due to the significant additional memory cost of force prodictions. Gala training on IS2RE was already strained memory utilization given that we applied a batch size of point cloud and performed gradient accumulation across batches to compensate for the small batch size. The full list of relevant. hyperparameters can be found in Appendix
4.2.1 IS2RE-AII Task Performance
The IS2RE-AII training results in Table 2 and Figures 4 Ap & 4 show that Gala performs best on training loss and MegNet performs best on in-distribution validation loss. E(n)-GNN performs worst across all in-distribution and out-of-distributin tasks for total task loss. The trend of the training loss, whose deviation corresponds to the loss across various batches in given epoch, indicates downwards slope with recurring
10

---

Published in Transactions on Machine Learning Research (07 2022)
1.0
10
1.0
E(n)-GNN: IS2RE Train
MegNet: IS2RE Train
Gala: IS2RE
E(n)-GNN: (ID)
MegNet: IS2RE (ID)
(ID)
0.8
0.6
0.6
LOSS
Loo5
0.2
0.2
0.0
0.0
Training Epach
Training Epoch
Training Step
(a) E(n)-GNN IS2RE-AII Training
(b) MegNet IS2RE-AII Training
(c) Gala IS2RE-AII Training
1.0
1.0
E(n)-GNN: S2EF Train
MegNet: S2EF 2M Train
E(n)-GNN: S2EF Val (ID)
MegNet: S2EF Val (ID)
0.8
0.8
0.6
0.6
Loss
Loss
0.2
0.2
15
20
25
30
45
15 20 25
45
Training Epoch
Training Epoch
(d) E(n)-GNN S2EF-2M Training
(e) MegNet S2EF-2M Training
Figure 4: Training and in-distribution validation loss curves for the IS2RE-AII Task on E(n)-GNN, MegNet and Gala (top row) and the S2FF-2M Task for E(n)-GNN and MegNet (bottom row). In-distribution validation loss generally follows the training loss for all tasks with a somewhat steeper decrease for S2EF than IS2RE. Full results, including
validation losses, are shown in Table 2 and Table
Training
Total Loss
Energy MAE (eV)
Energy wishin Thresbold (%)
Model
Cat
OOD
IS2RE Validation Dataset
R(n]-GNN
0.239
0.521
0.271
0.710
0.892
0.745
3.15
3.19
1.95
MegNet
0.233
0.216
0.314
0.250
0.656
0.634
0
0,500
3.83
3.99
1.90
GALA
0.153
0.240
0.217
0.276
0.236
1.508
1.565
1.469
1.216
1.02
1.11
1.00
1.19
B(n)-GNN
0.206
0,710
0,803
0.790
3.19
3.06
1.94
1.99
MegNot
0.653
0.668
1.60
GALA
1.505
1.06
Table 2: Task results for IS2RE (All) with MAE training loss for in-distribution (ID), out-of-distributioo catalysts (OOD-Cat), out-of-distribution adsorbates (OOD-Ads), and out-of-distribution catalysts and adsorbates (OOD-Both) for total loss, energy MAE (eV) and energy within threshold (EwT)) We highlight the best performing model for each metric. s_" indicates settings that are not applicable for the test data. While GALA mostly has the best performance measured by the loss, MegNet and E(n)-GNN perform better among the energy MAE and EwT.
deviation pattern. This suggests that the networks may find particular data samples more challenging than others as it cycles through the same batches over different epochs. The learning curves for all models also show that the in-distribution validation loss generally follows the training loss, which is expected for this
11

---

Published in Transactions on Machine Learning Research (07/2023)
setting. In the case of out-of-distribution tasks, MegNet performs slightly better than Gala OOD-Catalyst with Gala performing best on 00D-Adsorbatt and OOD-Both. Gala's relatively consistent performance across ID and OOD tasks suggests that Gala may have greater generalization ability compared to E(n)-GNN and McgNet. It is also worth noting that all three models have the most difficulty on the OOD-Adsorbates lask, indicating that is more difficult. to generalize for new molecules the calalyst-+ system compared to changing the underlying catalyst crystal structure. For the Energy MAE (cV) and EwT metrics MegNet outperforms E(n)-GNN in ID and OOD-Cat, while E(n)-GNN purporrsms MegNet in OOD-Ads and OOD-Both. This trends holds across both validation and training data splits.
4.2.2 S2EF Task Performance
4d and 1e include errors from both the energy prediction The total loss values shown in Table 3 and Figures and the force prediction components. Compared to the IS2RE results in Figures +. & 4 the S2EF results in Figures Ad and 4e indicate stronger downwards slope in both the training and in-distributton validation losses, particularly for MegNet. MegNet outperforms E(n)-GNN across all tasks, including in- distribution and out-of-ciitribbutin settings. For non-loss metries (Energy MAE, EFwT & Force MAE), E(n)-GNN outperforms MegNet for validation data, while MegNet outperforms E(n)-GNN for test data. MegNet's 's generally stronger performance compared to E(n)-GNN could be related to two factors: 1. Greater representation capacity due a higher number of model parameters; 2. MegNet's domain specific feature design, including graph-level variables, may be more useful in resolving the greater diversity of data found in the S2EF tasks. The results shown in Table 1 also show the greater compute cost associated with the S2EF-2M lask, both in the amount of hardware used as well as the related wall-clock time, driven by larger dataset size and greater task complexity.
Model
Training
Farray (A))
Birrgy
Thoeshold (IK)
MAR (#V/A)
000
0OD
Buch
OOD
OOD Ads
0OD
SIPP Validarion Datsert
Roo)-GNN
0.322
0:333
0.331
9888
9.88
1A0E
1,923
1,77
1291
1000
(0990)
0158)
01043
Maghie
0399
0384
030
6:006
1788
1.733
116
0:18
0.133
6.101
611
R(o)-GNN
1.79
1.091
0/391
0.14
MogNot
01582
Table 3: Task results for S2EF (2M) with MAE training loss for in-distribution (ID), out-of-distribution catalysts (OOD-Cat), out-o--distribution adsorbates (OOD-Ads), and out-of-distribution catalysts and adsorbates (OOD-Both) for total loss, energy MAE (eV), forces MAE (eV/A), and energy within threshold (EwT) as percentage of the dataset. We highlight the best performing model for each metric. "_" indicates settings that are not applicable for the test data. MegNet outperforms E(n)-GNN for total and for all test data metrics, while E(n)-GNN performs better for validation data on energy MAE, forces MAE and EFwT.
5 Discussion
5.1
Comparisons with existing models
One of the primary motivations of this paper, and the future work we hope to enable, is the search for new neural network architectures to accelerate materials discovery. We hence contextualize the results in Table 2 and Table 3 by making caveated comparisons with published Open Catalyst results, such as DimeNet+ KKiicpera al.) 2020a.) and GemNet-XL (Sriram et al.) 2022). In the case of IS2RE, the task measures mean absolute deviation in adsorption energies (in units of eV) calculated with the RPBE density functional [Chanussot** et al. (021), which estimate the thermodynamic stability of molccules adhering to catalytic surfaces. While direct comparisons are not possible as validation results were not included in earlier papers, the IS2RE MAE results for each model shown in Table 2 are on the order of ~022 eV across multiple validation distributions. The reported test errors for DimeNet++ (0.56 <V, (Chanussot* * et al.) (202]) ) and more recently GemNet-XI. (0.38 eV, (Sriram et al. 2022)), ) suggest that our reported results generally perform worse in Energy MAF.
12

---

Published in Transactions on Machine Learning Research (07/2023)
In the case of S2EF, our comparisons have additional limitations given that our training set of 2M samples is at least an order of magnitude smaller than those applied in other works (Chanussot* et al. 2021: Sriram et al., 2022). Nevertheless, E(n)-GNN and MegNet both present somewhat favorable performance in total loss when compared to GemNet-XL trained on the full S2EF training set (~~13M) by Sriram et al. (2022), who reported OOD-Both joint (sum of energy and lorces) test. errors of 0.38, compared with 0.43 E[n))GNN] and 0.38 (MegNet). We emphasize that, while these are not directly comparable, we believe that our results highlight the potential in discovering novel architectures particularly parameter efficient ones like E(n)-GNN and data representations, such the atom-centered point cloud representation described in Section 33 which are facilitated by the Open MatSci MT. Toolkit. As the framework matures, we look forward towards scaling experiments further and enabling direct comparisons with the OCP-20 leaderboard, while also continuing to grow the application of machine learning models for materials discovery applications (Friederich et al., 2021
Chen & Ong 2022
5.2
Future Work
In this paper, we introduced the Open MatSci ML. Toolkit and demonstrated how it can be applied to train advanced geometric deep learning models on different tasks within the OpenCatalyst dataset through seamless compute scaling and automated MLOps. While the primary aim of this paper was to showcase the capabilities of the framework, the challenge of training better and more effective machine learning models on the OpenCatalyst dataset remains. We hope that our results provide convincing starting point to continuc to build top of the current version of the framework and enable future research in geometric deep learning applied to materials science. Moreover, while the OpenCataly Catalyst remains the largest for materials science, further relevant datasets and benchmarks (Dunn et al. 2020 Jain et al. 2013 Kirklin et al., 2015 Calderon ctt al., 2015) could be integrated into the Open MatSci ML Toolkit leveraging our flexible and extensible Tramework to accelerate the development of advanced machine learning tools for materials science.
13

---

Published in Transactions on Machine Learning Research (07 2022))
References
Anderson, B., Hy, T.-., and Kondor, R. Cormorant: Covariant molecular neural networks, 2019, doi: URL
Avancha, S., Md, V., Misra, S., and Mohanty, R. Deep graph library optimizations for intel(r) x86 architecture. CoRR, abs/2007.06354 2020. URL httts: wttp://www..rgg/a////0000000
Axelrod, S., Schwalbe-Koda, D., Mohapatra, S., Damewood, J., Greenman, K. P., and K6meezBommarrlll R. Learning matter: Materials design with machine learning and atomistic simulations, Accounts of Materials Research, 3(3):343 357, 2022.
Batzner, S., Musaclian, A., Sun, L., Geiger, M., Mailoa, J. P., Kornbluth, M., Molinari, N., Smidt, T. E., and Kozinsky, B. Equuiequu graph neural networks for data-efficient and accurate interatomic potentials. Nature Communications, 13(1):2453 ISSN 2041-1723. doi: 10.1038/s41467-000 467-022-29999-55 29939-5. URL
http://www. http://wwwatu..../a natura. www.nnatttee.eraraao httpeeee//m/rrieeee////1553
Bronstein, M. M., Bruna, J., LeCun, Y., Szlam, A., and Vandergheynst, P. Geometric deep learning: Going beyond Euclidean data. IEEE Signal Processing Magazine, 34(4):18 42, July 2017. ISSN 1053-5888, 1558-0792. doi: 10. 10.1109/MSP... 2.17.......00 2693418.
Bronstein, M. M., Bruna, J., Cohen, T., and Veličković, P. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges, May 2021.
Bunker, P. R. and Jensen, P. Molecular Symmetry and Spectroscopy. NRC Research Press, Ottawa, 2. ed edition, 1998. ISBN 978-0-660-17555 16519-5-
Calderon, C. E., Plata, J. J., Toher, C., Oses, C.,, Levy, O., Fornari, M., Natan, A., Mehl, M. J., Hart, G., Nardelli, M. B., The aflow standard for high-throughput materials science calculations. Computational Materials Science, 108:233 238, 2015.
Chanussot L., Das*, A., Goyal*, S., Lavril*, T., Shuaibi*, M., Riviere, M., Tran, K., Heras- Deraas-Domingo, J., Ho, C., Hu, W., Palizhati, A., Sriram, A., Wood, B., Yoon, J., Parikh, D., Zitnick, C. L., and Ulissi, Z. Open catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 2021. doi: 10.1021 hatp1///ccaaal.0044500
Chen, C. and Ong, P. universal graph deep learning interatomic potential for the periodic table. arXiv preprint arXtv:2202.02450 2022.
Chen, C., Ye, W., Zuo, Y., Zheng, C., and Ong, S. P. Graph networks universal machine learning framework for molecules and crystals. Chemistry of Materials, 31)333356433372 2019.
Dunn, A., Wang, Q., Ganose, A., Dopp, D., and Jain, A. Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm. npj Computational Materials, 6(1): 1 1-10, 2020.
Falcon, W. and The PyTorch Lightning Team. PyTorrc Lightning, 3 2019. URL https://github.com Lightning AI/lightning
Fey, M. and Lenssen, E. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Friederich, P., Häse, F., Proppe, J., and Aspuru-Guzik, A. Machine-learned potentials for next-generation matter simulations. Nature Materials, 20(6):750 761, 2021.
Fuchs, F. B., Worrall, D. E., Fischer, V., and Welling, M. Se(3)-transformnn 3d roto-translation equivariant attention networks, 2020. URL https::/aaxit: www.xivvvrrrreoooo http:/bbs/20....0 10503
Fung, V., Zhang, J., Juarez, E., and Sumpter, G. Benchmarking graph neural networks for materials chemistry. npj Computational Materials, 7(1):188, 2021.
14

---

Published in Transactions on Machine Learning Research (07/2023)
Ganea, O., Pattanaik, L., Coley, C., Barzilay, R.,, Jensen, K., Green, W., and Jaakkola, T. Geomol: Torsional geometric generation of molecular 3d conformer ensembles. Advonces in Neural Information Processing Systems, 34:13757 14-3777-13769, 2021.
Gasteiger, J., Becker, F., and Günnemann, S. Gemmet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34:6790-6802 2021.
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. Neural message passing for quantum chernistry, In Proceedings of the 34th International Conference on Machine Learning Volume 70, JCML'17, pp. 1263- 1262-1272. IMLR.org. 2017.
Guo, Z., Lu, D., Yan, Y., Hu, S., Liu, R., Tan, G., Sun, N., Jiang, W., Liu, L., Chen, Y., et al. Extending the limit of molecular dynamics with ab initio accuracy to 10 billion atoms. In Proceedings of the 27th ACM SIGPLAN Symposium Principles and Practice of Parallel Programming, pp. 205-218, 2022.
Hamilton, W. L., Ying, R.,, and Leskovec, J. Inductive representation learning large graphs. In Proceedings of the 31st International Conference on Newral Information Processing Systems, NTPS'17, pp. 1025- 1035, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Jain, A., Ong, S. P., Hautier, G., Chen, W., Richards, W. D., Dacek, S., Cholia, S., Gunter, D., Skinner, D., Ceder, G., et al. Commentary: The materials project: A materials genome approach to accelerating materials innovation, APL materials, 111)0011002, 2013.
Jing, B., Corso, G., Chang, J., Barzilay, R., and Jaakkola, T. Torsional diffusion for molecular conformer generation. arXiv preprint ar Xix:22206..0 01 0299, 2022.
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunkkool, K., Bates, R., Zidek, A., Potapenko, A., et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583 589, 2021.
Kirklin, S., Saal, J. E., Merodig, B., Thompson, A., Doak, J. W., Aykol, M., Rühl, S., and Wolverton, C. The open quantum materials database (oqmd): assessing the accuracy of dft formation energies. nyj Computational Materials, 1(1):1-15, 15, 2015.
Klicpera, J., Giri, S., Margraf, J. T., and Günnemann, S. Fast and Uncertainty-Awar Directional Message Passing for -on-Equilibriun Molecules. arXiv:2011.14115 [physics], December 2020a.
Klicpera, J., GroB, J., and Günnemann, S. Directional message passing for molecular graphs. arXiv preprinl arXiv:2003.033 031223, 2020b.
Krenn, M., Pollice, R., Guo, S. Y., Aldeghi, M., Cervera-Lierta, A., Friederich, P., dos Passos Gomes, G., Häse, F., Jinich, A., Nigam, A., al. On scientific understanding with artificial intelligence. Nature Reviews Physics, pp. 1 1-9, 2022.
Li, S., Zhao, Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., Paszke, A., Smith, J., Vaughan, B., Damania, P., et al. Pytorch distributed: Experiences accelerating data parallel training. arXiv preprint Xiv:2000 15700, 2020.
Md, V., Misra, S., Ma, G., Mohanty, R., Georganas, E., Heinecke, A., Kalamkar, D., Ahmed, N. K., and Avancha, S. Distgnn: Scalable distributed training for large-scale graph neural networks. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384421. doi: 0111455/44888734455555 URL htttps://dol. http::////ooo.o/1...o. 10. 1145/3458817 3480856
Mellor, T. M., Yurchenko, S. N., Mant, B. P., and Jensen, Transformation Propertics under the Operations of the Molecular Symmetry Groups G36 and G36(EM) of Ethane H3CCH3. Symmetry, 11(7):862, July 2019. ISSN 2073-8994. doi: 10.3390/symm 10700622.
15

---

Published in Transactions on Machine Learning Research (07/2023)
Morris, C., Ritzert, M., Fey, M., Hamilton, W. L., Lenssen, J. E., Rattan, G., and Grohe, M. Weisfeiler and leman go neural: Higher-order graph neural networks. AAAI, abs/1810.022444, 2019. URL http: 02204
Mostafa, H. Sequential aggregation and rematerializaaion: Distributed full-batch training of graph neural networks on large graphs. arXiv preprint arXiv:2111.0660 2021.
Musaelian, A., Batzner, S., Johansson, A., Sun, L., Owen, C. J., Kornbluth, M., and Kozinsky, B. Learning local equivariant representations for large-scale atomistic dynamics. arXie preprint arXiv:2204.02240, 2022.
Ramakrishnan, R., Dral, P. O., Rupp, M., and Von Lilienfeld, O. A. Quantum chemistry structures and properties of 134 kilo molecules. Scientific dala, 1(1):1-7, 2014.
Satorras, V. G., Hoogeboom, E., and Welling, M. E (n) equivariant graph neural networks. In International conference machine learning, pp. 9323 9032. PMLR, 2021.
Shibuta, Y., Sakane, S., Miyoshi, E., Okita, S., Takaki, T., and Ohno, M. Heterogeneity homogeneous mucleation from billion-atom molecular dynamics simulation of solidification of pure metal. Nature communications, 8(1):1-9, 2017.
Spellings, M. Geometric algebra attention networks for small point clouds. arXiv preprint arXiv:2110.0239. 2021.
Sriram, A., Das, A., Wood, B. M., Goyal, S., and Zitnick, C. L. Towards training billion parameter graph neural networks for atomic simulations. arXiv preprint ar Xrxi:2200..0000 0099997 2022.
Thomas, N., Smidt, T. E., Kearnes, S., Yang, L., Li, L., Kohlhoff, K., and Riley, P. Tensor field networks: Rotation- and translation-eeqiivrraat neural networks for 3d point clouds. CoRR, abs/1802.08219, 2018.
URL http: http///axvvv../ http::///wwxiv..ggab///////s//// 18028082121100
Tran, R., Lan, J., Shuaibi, M., Goyal, S., Wood, B. M., Das, A., Heras- Domingo, J., Kolluru, A., Rizvi, A., Shoghi, N., et al. The open catalyst 2022 (oc22) dataset and challenges for oxide clectrocatalysis. arXiv preprint ar (xXt:2200.08811, 2022.
Wang, M., Zheng, D., Ye, Z., Gan, Q., Li, M., Song, X., Zhou, J., Ma, C., Yu, L., Gai, Y., Xiao, T., He, T., Karypis, G., Li, J., and Zhang, Z. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXin: arXim:1909.00 190..0135, 2019.
Williams, D. M. G. and Fisfeld, W. Complete Nuclear Permutation Inversion Invariant Artificial Neural Network (CNPI-ANN) Diabatization for the Accurate Treatment of Vibronic Coupling Problems. The Journal of Physical Chemistry A, 124(37):7008-722 September 2020. ISSN 1089-5639. doi: 10.1021/acs jcca.9105991.
Xu, K., Hu, W., Leskovec, J., and Jegelka, S. How powerful are graph neural networks? ICLR, abs Abs/1810.00020 2019. URL http::/ www.aarwwvvv.ioaaaaaaaae httggabb/181000 00826
Zheng, D., Song, X., Yang, C., LaSalle, D., Su, Q., Wang, M., Ma, C., and Karypis, G. Distributed hybrid cpu and gpu training for graph neural networks on billion-scale graphs ar preprint or xix: 2021.
16

---

Published in Transactions on Machine Learning Research (07/202) 2023)
A
Hyperparameters
Example hyperparameters for E(n)-GNN
Table Hyperparameters for E(n)-GNN
Hyperparameter
Value
MLP hidden dim
32
MLP output dim
32
# of EGNN layers
Node MLP dim
[48, 48
Edge MLP dim
[16, 16
Atom position MLP dim
664, 64
MLP activation
ReLU
Graph read out
Sum
Node projection block depth
2
Node projection hidden dim
128
Node projection activation
ReLU
Output block depth
Output hiddem dim
64
Output activation
ReLU
Optimizer parameters
Learning Rate
0.003626
Gamma
0.6878
Batch Size
Example hyperparameters for MegNet
Table 5: Hyporparametors for MegNet
Hyperparamcter
Value
Edge MLP dim
2
Node MLP dim
5
Graph variable MLP dim
9
MLP projection dim
11
MegNet blocks
4
MLP hidden dims
[128, 64
MegNet convolution dims
[128,
128, 64
# of S2S layers
5
# of $2S iterations
4
Output projection dims
[64, 16]
Dropoul
0.1
Optimizer parameters
Learning Rate
0.0001
Gamma
0.2
Batch Size
Example hyperparameters for Gala.
17

---

Published in Transactions on Machine Learning Research (07/2023)
Table Hyperparameters for Gala
Hyperparameter
Value
Input dimension
200
Hidden dimension
100
Merge function
concat
Join function
concat
Rotation-invariant mode
full
Rotation-covariant mode
full
Rotation-invariant value norm Rotation-cquivariiat value norm Value function normalization
mmmentum
momentum layer
layer
Score function normalization
layer
Block-level normalization
layer
Optimizer parameters
Learning Rate
0.001
Gamma Batch Size
0.8
1
B Development Example
A self-contained python script running the full pipeline on one of our dev-sets is shown below:
wwn Sample Python Script Without Inports
.*.
Define Parameters
BATCH_SIZE NUM WORKERS
BATCH SIZE
REGRESS FORCES False
epochs
Model configuration for Megllet
model_config =
"edge_feat_dia": 2,
"node_foat_ddn""
"graph_attr_dim ": 9,
"dim":
"num_blocks";
17
"hiddens": [128,
18
"conv_hiddens": [128, 128, 64]
18
"#2s_nun_layers": 5,
20
"s2s_ "s2e_num_itere"" 4,
21
"output_hiddene"" [64, 16],
22
"'s_claassffcattoon"" False, "dropout": 0.1,
23
24
25
28 use default settings for NegNet
27
negnet Keglet (#=model_config)
the CNN in the LitNodule for all the logging. loss conputation, etc. model S2EFLitModul (mognet regress regreas_forces-RRee wtw.ffeeeRReeess_FRRROOOO 1r=1e-3, gamma=0.1 data_module SzEFDCLDataMocull. from from_devset( ( batch_ batch__a1z==AATHLLSIE aun_worker==NUX. VORKERS
34
35
18

---

Published in Transactions on Machine Learning Research (07 (07/2023)
alternatively,
you don't with validation, just 92EFD6LDataMoodule.
trom_devsed
37
data_module 82EFDGLDataModulee
38
39
ral_path==2ee_devsst
40
Batch_sizze=BATO_SSZZ
41
Bun_workere-HUX_MORRE
42
48
44
trainer = pl. Trainer (acceelerator=='ppu" strategy="ddp" devices=2, sax_epochs=epocha
45
trainer , fit (model, aetamodule=ddtt_noddul
Listing 1: Sell-Contained Example Script With Scalable Devices
As can be seen in the definition of trainer, this short script already performs training two GPUs with users being able to change the devices variable to adjust the numbers of GPUs they want to leverage for distributed training on single node.
19

---

Published in Transactions on Machine Learning Research (07 (07/202) /203)
C
Data pipeline abstraction
PyTorch Dataset
Implements .mdbb logic
BaseOCPDatasee
Base Dataset
Interprets .mdb graph representations
DGLDataset
PointCloudDataset
Extracts task specific keys and labels
IS2REDGLDatar
Tensor Abstraction
Graph Abstraction
User Interaction
IS2RED DataMMdulle
Wraps task specific datasets to extract point clouds
D2tFDGLDDaaaMddul
Coordinates splits, collating, and data loading
PointCloudDataMoouule
PyTorch Lightning Data Modules
Figure 5: Inheritance diagram for the data abstraction in Open MatSci ML. Toolkit. The main user interaction layer is presented at the bottom, corresponding to subclasses of LightningDataModte directional relationship between the classes; the dashed line indicates that the PointCloudDatases wraps the task specific datasets, whereby the user is provided with a sampled point cloud representation of the original graphs.
Arrows denote
20


Reviews:
[{"review_id": "oRuD9clqGr", "reviewer": "Reviewer_347i", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The framework enables to use both graph representation, point cloud representation and other (futuristic) representation of the data.\n\n2. The selection of the base frameworks (DGL and PyTorch Lighting) is well justified.\n\n\nWeaknesses \n\n1. From my understanding it seems that most of the work of model and data abstractions, scaling, logging and more is done by DGL and PyTorch Lightning. Please clearly state your exact contribution to the pipeline.\n\n2.  The experiments lack comparison to previous works. Please add timing comparisons between your framework to previous works.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduce the Open MatSci ML Toolkit: framework to apply deep learning models and methods on tasks from the field of materials science, and specifically from OpenCatalyst Dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Follow my answers in the previous section.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a0g1uJHMye", "reviewer": "Reviewer_6eMk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of this work include:\n\n-\tThe paper is well-written and the language is easy to understand \n\n-\tThe introduction provides brief summary of why catalysts and materials discovery using AI/ML is important. This makes this work appealing to readers outside the field of chemistry and materials science. \n\n-\tIt utilizes faster and more scalable deep learning frameworks such as PyTorch Lightning and DGL to enable cross-platform optimization and sampling-based training. This effort essentially lowers the barrier of entry to use the OCP20 dataset. The code could potentially run on a single-node GPU or data center CPUs with built-in data parallelism. \n\n-\tNew model architectures are explored for OCP20 tasks, and they show promising results. The experiments ran on the new framework were able to finish within a reasonable amount of time (shorter than 60 hours).\n\nThe weaknesses of this work include:\n\n-\tSome context and details of OCP20 tasks are missing or not well-elaborated in this paper. For example, the three tasks with the abbreviations of OOD Cat, OOD Ads, and OOD both were not explained in this paper. This omission makes it difficult to read this paper as a standalone study without referring to the original OCP20 paper.\n-  There is a lack of connection between the choice of methods and the gains in experimental results. For instance, it is unclear how the refactor in DGL and the point of cloud representation would result in any performance gains in modeling molecular graphs. Additionally, the authors should expand on the rationale behind the new model architecture, namely, why those models were chosen and why they are suitable for the OCP20.\n-\tIn the discussion part of task performance, the authors only briefly mention the models that perform well on certain tasks. However, they do not delve into the potential causes of these performance differences. For researchers in the catalysis field, understanding the reasons why some models perform well can be even more significant than achieving high scores in benchmarking tests. This knowledge can provide valuable insights into the underlying mechanisms of catalysis and lead to the development of more effective catalysts. Furthermore, understanding the reasons behind model performance can aid in improving the accuracy and reliability of predictions, leading to more efficient modeling approaches. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors have presented an open-source software toolkit named Open MatSci ML, which builds on top of the OpenCatalyst project (OCP20) and provides scalable deep learning solutions for various catalyst modeling tasks. The OpenCatalyst Dataset is currently the largest open-source dataset in the field of computational materials sciences. The original OCP20 paper demonstrated the use of state-of-the-art graph neural network models to solve three tasks indicative of day-to-day catalyst modeling. In this work, the authors refactored OCP20 in PyTorch Lightning and Deep Graph Library (DGL) to provide ease of use for researchers, scalable computation across devices, faster training, and better support for molecular graphs. They also explored a few new model architectures, including E(n)-GNN, MegNet, and Geometric Algebra Attention Network (Gala), which showed competitive results on OCP20 tasks. As an extension of the OpenCatalyst project, the graph deep learning framework presented in this work has the potential to accelerate materials discovery and enable catalyst modeling for clean energy applications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The authors could expand on the rationale behind choosing DGL and point cloud representations. Additionally, they could provide further discussions on how these representations enable new model architecture and lead to gains in prediction accuracy.\n2. The discussion in the task performance section could be expanded further. The authors should discuss the reasons that make these models suitable for deep graph neural networks and catalyst modeling.\n3. In Section 4.1, the author should mention the specifications of the GPU nodes used for the experiments\n4. Table 2 displays the in-distribution validation loss and out-of-distribution validation loss. I assume the latter is the independent test set. How does the test set performance compare to the original three models presented in the OCP20 paper? The original test set metrics should also be included in the table and compared to the metrics obtained in this paper. Additionally, each task in the OCP20 has multiple metrics representing the key structural and electronic properties of the catalysts. Does the new model predict other metrics as well? For instance, for the IS2RE task, how do the models perform on the EwT metric, going beyond the energy MAE loss?\n5. In the future work section, the author should discuss how easy it would be to extend the framework to other material science or catalyst datasets. What is the required data format?  The author could provide guidelines for expanding the OCP20 dataset. After all, the OCP20 dataset only consists of molecular adsorptions onto surfaces. Alloys, reactions, transition states, and so on are also of interest to material scientists.\n\n\nBy addressing these additional points, the authors can further strengthen their papers and make this work more appealing to its target audience, i.e., researchers in the AI, chemistry, and materials science community. \n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W0lXcjOCTP", "reviewer": "Reviewer_3e5y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=== Strengths === \n\nThe framework seems reasonably designed, and the authors give a good amount of attention to detailed considerations, like differences between relying on DGL vs PyG. While I don't work with Open Catalyst personally, the toolkit seems like a useful thing to have, especially for somebody just starting out. The paper is also very well-written and clear.\n\n \n\n=== Weaknesses === \n\nThe authors spend a lot of time outlining the MLOps improvements (like seamless transition from CPU to GPU), but most of these benefits just come from using `pytorch_lightning`. This is not a weakness in itself (using existing frameworks is good, we should not try to reinvent the wheel!), but examining things more closely I'm not sure if the \"volume\" of work done here is enough for a paper at TMLR.\n\n \n\n=== Other comments & questions === \n\n- (O1): Repo link does not work for me, as it claims the repository has expired, thus I couldn't take the code into account during my review. \n\n- (O2): In pseudocode on page 7, I was confused why `data` is assigned to twice. After reading the surrounding text I realized the pseudocode is simply showing two ways of creating a `Data` object, and is not supposed to be interpreted as a single chunk of code. Maybe modify the comments in that snipped to include something like \"Option 1:\" and \"Option 2:\" for clarity. \n\n \n\n=== Nitpicks === \n\n- In the caption of Figure 2, the font style of the last letter of \"LightningDataModules\" is not consistent with how this is handled in the other such cases. \n\n- \"grow the application of machine learning models for materials discovery applications\" - a bit repetitive. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present The Open MatSci ML Toolkit, which is a framework for developing ML models for materials, and in particular the Open Catalyst dataset. They give a detailed discussion of the considerations behind their design decisions, and how they make both starting and scale-up easy. The paper finishes with experiments comparing existing architectures to showcase what the framework can do. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As a start, I'd request the authors to fix the various small issues mentioned above, in particular the repo link. ", "broader_impact_concerns": "I see no unaddressed ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that the companion paper to the library is a useful contribution to the field. All reviewers were also satisfied with the answers provided by the authors. Authors should incorporate the additional performance metrics of the OpenCatalyst Dataset paper as discussed with reviewer 6eMk. The authors should also provide a link to the repo and could include the hardware used for the experiments.
Decision: Accept (poster)