dataesr/scientific-paragraphs-categorization huggingface dataset:
Machine learning plays an increasingly important role in many areas of chemistry and materials science, being used to predict materials properties, accelerate simulations, design new structures, and predict synthesis routes of new materials. Graph neural networks (GNNs) are one of the fastest growing classes of machine learning models. They are of particular relevance for chemistry and materials science, as they directly work on a graph or structural representation of molecules and materials and therefore have full access to all relevant information required to characterize materials. In this Review, we provide an overview of the basic principles of GNNs, widely used datasets, and state-of-the-art architectures, followed by a discussion of a wide range of recent applications of GNNs in chemistry and materials science, and concluding with a road-map for the further development and application of GNNs.
D ata science and machine learning have become an integral part of natural sciences, discussed as the fourth pillar in science, next to experiment, theory, and simulation 1. Machine learning methods are increasingly applied in all steps of the materials development cycle, from finding initial candidate materials using property prediction 2,3 , database screening 4,5 or even inverse materials design 6,7 , over the detailed analysis of materials in machine learning accelerated simulations  8, 9  , to the prediction of synthesis conditions  10, 11  and automated experimental data analysis  12, 13  and experimental planning 14. Machine learning models applied in chemistry and materials science cover a wide spectrum of methods, ranging from classical machine learning models such as decision tree ensembles to modern deep learning methods such as convolutional neural networks 15 and sequence models 16 originally developed for challenges in computer vision and natural language processing.
We will present and compare state-of-the-art architectures and benchmark datasets, as well as summarize initial efforts toward inverse materials design based on GNNs. The section 'Applications' covers a wide range of current application areas but also open challenges for GNNs in chemistry and materials science. In section 'Outlook' we conclude with a perspective on necessary and expected future developments and so far unused potential of GNNs in materials science. Graph neural networks in materials science and chemistry Basic principles. In the most general sense, graphs are used to describe abstract structures consisting of entities or objects represented as vertices (or nodes) and their connections, called edges. Formally, a graph is a tuple G = (V, E) of a set of vertices v ∈ V and a set of edges e v,w = (v, w) ∈ E, which defines the connection between vertices.
Consequently, GNNs can be applied for tasks such as drug design or material screening  54  , which require knowledge about functional groups, scaffolds  55  or the full chemical structure and its topology. In molecular applications, the chemical graph is often extracted from SMILES codes and augmented with features that can be readily obtained from cheminformatics software such as RDKit 56 or OpenBabel  57. Common features for atoms, bonds, and the molecule itself are listed in Table  1. Besides hand-crafted input features, learned embeddings of molecules and materials motivated by word embedding techniques in natural language processing have been explored which can be used for downstream tasks  48, [58] [59] [60]. For specific tasks in chemistry, the connectivity of atoms in molecules (i.
Under the term geometric deep learning, architectures and descriptors are summarized that focus on manifolds, geometric data, or structured data  77, 78. This includes the work on 3D point clouds  79  , which aims at learning segmentation and object detection of a large number of 3D points. In the case of PointNet++ 80 a graph is constructed which reduces the point set from learned descriptors using the points' features. Commonly, adjacency matrices are defined by using distance cutoffs between points in 3D clouds, while edges carry explicit information about distances between nodes, i. e. , points. Graph pooling or coarsening algorithms  81, 82  that reduce the input representation and condense structure information are also promising for GNNs to tackle larger molecules such as proteins or polymers.
For solid crystals and periodic structures, the periodicity and space group symmetries are additional symmetries to be added to the representation for GNNs. Periodic extensions of the crystal graph 2,53 of the unit cell have been introduced  61  and their representation builds on the idea of a k-point mesh of Monkhorst-Pack grid points to sample the Brillouin zone  88. State-of-the-art architectures and benchmarks. Different architectures have been proposed in the literature to improve the performance of GNNs on typical tasks arising in chemistry and materials sciences. Table  2  shows a list of popular benchmark datasets for different materials classes, i. e. , molecules  62, [89] [90] [91] [92] [93] [94] [95] [96] [97] [98] [99]  or crystals  [100] [101] [102]  , and respective supervised tasks, i.
e. , regression or classification. While some datasets contain experimental data, the largest datasets typically use computational methods to generate labels. Most datasets in this table can be downloaded from data collections such as TUDatasets  103  and MoleculeNet  95. While a large variety of datasets and tasks exist for chemistry, there are only a few large datasets for materials, limited to crystalline structures. Recent datasets were constructed by filtering the Materials Project (MP)  100  and Open Quantum Materials Database (OQMD)  101  for specific targets such as electronic band-gap or formation energy while removing incomplete data  88. In Section 'Basic principles', the message passing framework for GNNs has been illustrated. Here, we will discuss modified and extended GNN models, which are relevant for materials science and chemistry.
The addition of more complicated node aggregation functions such as gated recurrent units  104  or long short-term memories  105  has been employed by GraphSAGE for inductive learning  106. For graph embedding tasks, a state-or super-node  21  , which is connected to all nodes, extends the message passing framework to help extract global graph information, in addition to the final node aggregation step. A message passing neural network (MPNN) with edge features capturing bond information was applied to molecular graphs  21  and crystal graphs  53. A variant of the original MPNN involves directed edge embeddings and message passing between edges in D-MPNN  40. Known from models in natural language processing  107  , masked self-attention layers which attend over the node's neighborhood have been suggested for graph attention networks  108  and used explicitly for molecules in Attentive Fingerprint models  54.
More attentionbased architectures are listed in Table  3. Besides graph models which focus on the chemical graph, there is a large group of models explicitly designed for learning quantum properties. They commonly take atomic numbers and positions as input and train on data derived from (approximate) solutions of the steady-state Schrödinger equation. A popular benchmark dataset is the QM9 dataset 62 with 13 quantum properties of small molecules with up to nine atoms apart from hydrogen. The improvement of graph networks on QM9 property prediction over the past few years is highlighted in Fig. 1b. Among the first graph networks that reached chemical accuracy on QM9 is SchNet  22  , which makes use of convolutional filters for inter-atomic distances and applies skip connections between node updates.
g. , whether to add a node or which nodes to connect. Each module is a fully-connected neural network modeling probabilities of executing particular types of modifications. You et al. suggested a purely RL-based approach based on Graph Convolutional Policy Networks (GCPN)  154  (see Fig. 2b ). In this setting, the agent explores the chemical space and generates new molecular graphs by modifying the starting molecules according to the reward function, representing the molecular property to optimize. Atance et al. recently introduced a similar RL approach based on Gated GNNs  155  , outperforming other GNN-based approaches in molecular graph generation tasks  156. Another sequential approach based on conditional graph generative models has been used by Li et al. on drug design tasks  157. The previous two works inspired recent molecular graph generation frameworks such as GraphINVENT  156.
Applications After introducing the basic principles of GNNs as well as selected GNN architectures and benchmark datasets, we will provide a structured overview of GNN applications in chemistry and materials science. GNNs were successfully applied to a rich variety of different challenges, ranging from property prediction of molecules and materials over accelerated atomistic simulations to predicting reactivity and synthesis routes of molecules and materials. While other machine learning models such as densely connected neural networks were successfully applied to these tasks as well, state-of-the-art GNNs in many cases currently outperform other models. However, there exists a range of open challenges, including data requirements and data efficiency, as well as a lack of fully GNN-based generative models for molecular and materials design.
169. In recent years, one application focus were Covid 19 related challenges, where GNNs were used for e. g. , finding new drug candidates  170  or detecting infections in medical images  171, 172. Similar methods are also applicable to other challenges in drug design and medicine. Furthermore, GNNs were applied to predict electronic and optical properties of molecules. For many applications such as organic electronics, organic photovoltaics, and organic lightemitting diodes, the energy of the highest occupied molecular orbital (HOMO), the lowest unoccupied MO (LUMO), and the optical gap are of high importance for the device efficiency. These properties can therefore be found in numerous databases  62, 90, [173] [174] [175] [176] [177]. Related properties include (transition) dipole moment  62, 178  , ionization potential, and electron affinity  90.
In devices, these properties often depend on the molecular environment, which can be modeled and accounted for with GNNs (see Fig. 2a )  179. For certain applications such as opto-electronic materials  174, 175  , GNNs have been proposed to complement scalar molecular properties with spectroscopic properties  112. The aforementioned properties are often determined using computationally expensive simulation methods. Cheaper, mostly semi-empirical methods can provide fast but potentially less accurate estimates. GNNs have been used to represent Kohn-Sham wavefunctions and energy levels in a minimal basis representation  180  , as well as for delta-learning from semiempirical methods to DFT computed molecular properties  177. For applications where not enough data is available to train GNNs, the representations learned by GNNs on large generic datasets can also be transferred to supervised tasks with little data  20, 181, 182  , where they are used as input for other machine learning models such as gradient-boosted models  183.
In the GNN framework (GNNFF  190  ), a message passing step builds upon an embedding step in which node and edge features include atom type and interatomic distances respectively. Force magnitudes per atom are then calculated from the sum of the forces of the neighboring atoms. Evaluation on the ISO17 database reveals higher force accuracies compared to SchNet while being 1. 6 × faster. The approach is also shown to be scalable to larger systems. Due to the direct prediction of the forces, the energy of the system is however not necessarily preserved, making the model not suitable to predict energy-related properties. ForceNet  191  is based on an encoder-decoder architecture and tries to completely capture 3D geometry through a specific design of the message passing structure. In contrast to models such as SchNet and DimeNet, ForceNet encodes physical information without constraining the model architecture to enforce physical invariances.
Instead, physics-based data augmentation is performed on the data level to achieve rotational invariance. The evaluation was performed on the OC20 dataset which contains DFT calculated energies and per-atom forces of more than 200 million large atomic structures (20-200 atoms) including nonequilibrium structures from optimization trajectories. The resulting mean absolute force errors are comparable to DimeNet++, while being faster in training and prediction. A promising approach to encoding more physical information about a system is the design of equivariant models. Models that are based on equivariant message passing, e. g. , PaiNN 29 , NequIP  25  , NewtonNet  192  , are shown to significantly increase data efficiency and predictive performance compared to models that are based on invariant convolutions.
The Neural Equivariant Interatomic Potential (NequIP  25  ) predicts both energy and forces utilizing E(3)-equivariant convolutions over geometric tensors. Evaluated on the MD17 data set its accuracy exceeds those of existing models while needing up to three orders of magnitude less training data. Due to its data efficiency, it was also used with coupled cluster (CCSD(T)) based training data, showing great potential for applications where a prediction accuracy beyond density functional theory (DFT) is needed. In order to further improve data efficiency, GNNFF  190  and NewtonNet  192  introduce more physical priors in the form of latent force vectors as well as operators containing physical information. This leads to good prediction accuracy with higher computational efficiency at only 1-10% of the training data compared to other models.
196  replacing manual input features with a GNN architecture making the models transferable across different molecular systems such as small proteins. Excited states dynamics. GNNs were also shown as a very promising tool to tackle the challenging task of simulating excited state dynamics of complex systems  197. Unlike ground-state dynamics, multiple potential energy surfaces as well as their crossings and couplings must be considered, leading to a higher dimensionality and complexity of the problem. Furthermore, even the generation of reliable training data using quantum mechanical calculations is challenging. 2d )  199. The diabatic artificial neural network (DANN) is based on PaiNN combined with a physics-informed diabatic model for photodynamic simulations for virtual screening. The resulting machine-learned potential is transferable among azobenzene derivatives and estimated to be multiple orders of magnitude faster than the underlying quantum mechanical calculation method, even considering computational effort for transfer which required additional training data.
Crystalline and solid state systems. Compared to molecules, crystal structures and solid-state materials have some additional challenges, such as periodic boundary conditions for crystals and multiple kinds of disorder, either in form of perturbations in the crystal structure itself or in the (lack of) long-range ordering of atoms. We will present different recent applications of GNNs in solid state systems, from predicting global properties of crystal structures with and without disorder, over driving atomistic simulations, to the design of new materials aided by materials synthesis prediction, active learning, and inverse design. Table  4  gives an overview of datasets used in the following applications. We will also discuss approaches in which the trained GNN models have been analyzed to enable further insight into specific scientific questions, which is often equally important as accurate numerical predictions.
Materials property prediction and materials design. GNNs can be used to predict a multitude of materials properties, ranging from formation energies  24, [224] [225] [226] [227] [228]  and synthesizability prediction  229  over band-gaps  [230] [231] [232]  and other functional properties to mechanical properties  61, 233. The most straightforward application of property prediction models is the screening of large (unlabeled) crystal databases, where exhaustive screening using conventional simulation techniques (e. g. , DFT) is often not feasible. Screening using GNNs only requires labels from simulation or experiment for the training set while providing fast predictions on the full database -provided the model generalizes well. Wang et al. use a crystal GNN to predict methane adsorption volumes in metal-organic frameworks (MOFs)  234.
The pooling function leverages domain knowledge by additionally including structural properties (see Fig. 2f ), e. g. , the pore limiting diameter, achieving better performance than previous work  235. They apply the model to the screening of a hypothetical MOF database by Wilmer et al. 236  and find several high-performing candidates  234. Gu et al. use an ensemble of attention crystal GNNs to screen alloy catalysts for CO 2 reduction  237. Only bulk-relaxed structures without the adsorbate (e. g. , from the Materials Project (MP)  100  database) are needed as input, removing the costly DFT relaxation from the screening process. The performance approaches that of base-line models trained on fully relaxed structures  237. An interesting application of GNNs was presented by Goodall et al. , who use GNNs for representation learning of chemical formulas  238.
The chemical formula is represented as a dense weighted graph, in which each node corresponds to a chemical element weighted by the fraction of this element in the chemical formula. This graph representation was used to train a GNN in a supervised way to obtain a mapping from the chemical formula to an embedding. It was demonstrated that this representation has a better sample efficiency than other structure agnostic approaches. Schmidt et al. use a crystal graph attention network for stabilityscreening of non-relaxed hypothetical materials, predicting the convex hull distance  24. Only graph distances are included, making the model usable for hypothetical materials where the exact coordinates might be unknown. A vast database is used, combining MP, AFLOW  239  and group-internal datapoints.
Transfer learning then allows the screening of 15 million tetragonal perovskites of composition ABCD 2. Dai et al. use GNNs to predict magnetostriction of polycrystalline systems (see Fig. 2g )  240. Instead of using GNNs to model the atoms in the unit cell, individual grains and their interactions to neighboring grains are represented by nodes and edges in the GNN. This shows that GNNs can be used on different scales to predict materials properties. Typically, screening of materials databases using ML models is only possible if training data is available which covers the target materials distribution, i. e. , which adequately allows generalization to the rest of the database. Active learning offers a promising solution when the available data does not fulfill this criterion. Based on uncertainty quantification, the training dataset can then be iteratively extended to include previously uncertain data points and thereby efficiently explore the chemical space.
Lu et al. use active learning to search for 2D ferromagnetic materials with a custom crystal graph multilayer descriptor in a small dataset  241. Further applications of active learning based GNNs in materials science are promising and can be expected in the future. Disordered systems and defects. Disordered materials are a wide and particularly interesting field of study, as the effects of disorder can influence or even dominate material properties. Disorder ranges from weak disorder (defects, dislocations, grain boundaries) to strong disorder (e. g. , glassy systems and inhomogeneous systems such as porous materials) and includes topological/ structural disorder, orientational disorder (spins, dipoles), or substitutional disorder (chemical doping, compositional disorder)  242. Due to its inherent multi-scale nature, disorder poses severe challenges not only to materials modeling and simulation, but also to materials synthesis, characterization, and fundamental description.
Dataset Size AFLOW  239  -calculated properties of materials >3,400,000 Inorganic Crystal Structure Database (ICSD)  299  extensive and well curated experimental database ≈210,000 Pure carbon and C-H-N-O structures at different pressures 300 ≈200,000 Materials Project 100 -calculated properties of materials ≈145,000 Hypothetical MOF database  236  137,953 NREL Materials Database (NRELMatDB) 301computational materials database focused on renewable energy applications ≈60,000 CO and H surface binding energy dataset  302  ≈40,000 Inorganic materials synthesis recipes  303  19,488 Perovskite structures and energies  304  18,928 CoRE MOF database  305  -Experimental MOF database >14,000 bcc iron structures with energies and various kinds of defects  306  12,193 MOF methane adsorption volume of CoRE MOFs (from GCMC)  234.
10,102 Elemental boron structures with energies  307  5038 Computational 2D Materials Database (C2DB)  308, 309  ≈4000 DDEC MOF point charges  310  2932 methods such as DFT, crystalline systems with substitutional disorder can be modeled using larger, more representative unit cells or by defining atoms/nodes with mixed occupations. Furthermore, amorphous systems can be modeled explicitly by using representative simulation boxes and aggregating over learned atom representations characteristic of the properties of the amorphous system. In the following, we will present seminal work in that direction. One important challenge for machine learning models is how to represent substitutional disorder such as doping and fractional occupancies in the crystal graph. Chen et al. addressed this question with a MEGNet model with trainable 16-dimensional embeddings for each atom which were pre-trained on a dataset with only ordered structures  243.
Outlook GNNs became a very versatile and important tool very quickly. A lot has been achieved already, not only in terms of fundamental method development tailor-made for requirements of materials science and chemistry (see Section 'Graph neural networks in materials science and chemistry'), but also in terms of applications (see Section 'Applications'), where GNNs were successfully applied for materials simulation, screening, design, and synthesis. However, there is a wide range of open questions and challenges which need to be solved in order to leverage the full potential of GNNs in materials science and chemistry. Despite the growing amount of research and publications on GNN model development and application, GNN models remain expert tools, i. e. , they are comparably hard to implement, adapt, transfer, train, and apply to given datasets and applications.
Libraries such as PyTorch Geometric  259  , DGL  260  or the Keras based KGCNN  261  implement a selection of state-of-the-art GNN layers and models. However, the use of such libraries in many cases requires expert knowledge which goes beyond the knowledge needed for the successful application of more established machine learning models. One of the reasons for this is certainly related to the fast development of new GNN variants which are partially hard to compare. The widespread use and further development of common benchmarks (e. g. , the QM9 dataset) as well as open communication of models and training workflows, including open-source code, inspired by common practice in the machine learning community, are essential for a reliable quantitative comparison of future developments. However, non-optimal hyperparameters and the (prohibitively) high computational cost of hyperparameter optimization is an open challenge.