Grammar-Induced Geometry for Data-Efficient Molecular Property Prediction
Authors: Minghao Guo
Veronika Thost
Samuel Song
Adithya Balachandran
Payel Das
Jie Chen
Wojciech Matusik
Venue: Submitted to ICLR 2023 (2023)
Keywords: Molecular property prediction
Graph grammar
Data-efficient model

Abstract: The prediction of molecular properties is a crucial task in the field of material and drug discovery. The potential benefits of using deep learning techniques are reflected in the wealth of recent literature. Still, these techniques are faced with a common challenge in practice: Labeled data are limited by the cost of manual extraction from literature and laborious experimentation. In this work, we propose a data-efficient property predictor by utilizing a learnable hierarchical molecular grammar that can generate molecules from grammar production rules. Such a grammar induces an explicit geometry of the space of molecular graphs, which provides an informative prior on molecular structural similarity. The property prediction is performed using graph neural diffusion over the grammar-induced geometry. On both small and large datasets, our evaluation shows that this approach outperforms a wide spectrum of baselines, including supervised and pre-trained graph neural networks. We include a detailed ablation study and further analysis of our solution, showing its effectiveness in cases with extremely limited data (only ${\sim}100$ samples), and its extension to application in molecular generation.

Under review as a conference paper at ICLR 2023
GRAMMAR-INDUCEN GEOMETRY FOR DATA- EFFICIENT MOLECULAR PROPERTY PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
The prediction of molecular properties is a crucial task in the field of material and drug discovery. The potential benefits of using deep learning techniques are re- flected in the wealth of recent literature. Still, these techniques are faced with a common challenge practice: Labeled data are limited by the cost of manual ex- traction from literature and laborious experimentation. In this work, we propose a data-efficient property predictor by utilizing a Iearnable hierarchical molecu- lar grammar that can generate molecules from grammar production rules. Such a grammar induces an explicit geometry of the space of molecular graphs, which provides informative prior molecular structural similarity. The property pre- diction is performed using graph neural diffusion over the grammar-induced ge- ometry. On both small and large datasets, our evaluation shows that this approach outperforms a wide spectrum of baselines, including supervised and pre-trained graph neural networks. We include detailed ablation study and further analy- sis of our solution, showing its effectiveness in cases with extremely limited data (only 100 samples), and its extension application in molecular generation.
INTRODUCTION
Molecular property prediction an essential step in the discovery of novel materials and drugs, as it applies to both eigh-throughpur screening and molecule optimization. Recent advances in ma- chine learning, particularly deep learning, have made tremendous progress in predicting complex property values that are difficult to measure in reality due to the associated cost. Depending on the representation form of molecules, various methods have been proposed, including recurrent neural networks (RNN) for SMILES strings (Lusci et al., 2013 Goh et al., (017), feed-forward networks (FFN) for molecule fingerprints (Tao et al. 202 lb a), and, more dominantly, graph neural networks (GNN) for molecule graphs (Yang et al. 2019 Bevilacqua et al.) 2022 Aldeghi & Coley 2022 Yu & Gao 2022). They have been employed to predict biological and mechanical properties of both polymers and drug-like molecules. Typically, these methods learn deep neural network that maps the molecular input into an embedding space, where molecules are represented as latent features and then transformed into property values. Despite their promising performance common bench- marks, these deep learning-bai based approaches require large amount of training data in order to be effective (Audus & de Pablo 2017 Wieder et al. 2020).
In practice, however, scientists often have small datasets at their disposal, in which case deep learn- ing fails, particularly in the context of polymers (Subramanian al. 2016 Altae-Tran al. 2017). For example, due to the difficulty of generating and acquiring data which usually entails synthesis, wet-lab measurement, and mechanical testing- state-of-thh-as works on polymer property predic- tion using real data are limited to only few hundred samples (Menon et al. 2019 Chen et al., 202T) To compensate for the scarcity of experimental data, applied works often rely on labeled data gen- erated by simulations, such density functional theory and molecular dynamics (Aldeghi & Coley) 2022 Antoniuk et al. 20022). Yet, these techniques suffer from high computational costs, tedious pa- rameter optimization, and considerable discrepancy between simulations and experiments (Afzal
et al. 2020 Chen et al. 2021), which limit their applicability in practice.
Recent deep learning research has recognized the scarcity of molecular data in several domains and has developed methods handling small datasets, including self-supervised learning (Zhang et al.
2021 Rong et al. 2020 Wang et al. 2022 Ross et al. 2021), transfer learning (Hu et al. 2020

---

Under review as a conference paper at ICLR 2023
Meta Grammtrr
puto
pst
Molecalas Granmar
Grammar
Learning
pioco
Propont valore
Hierarchical Molecular Grammar Grammor-induced Goometry Graph Neural Diffusion
Figure 1: Overview. Given set of molecules, learn hierarchical molecular grammar that can generate molecules from production rules. The hierarchical molecular grammar induces an explicit geometry for the space of molecules, where structurally similar molecules are closer in distance along the geometry. Such a grammar-induced geometry provides an informative prior for data- efficient property prediction. We achieve this by using graph neural diffusion over the geometry.
and few-shot learning (Guo et al. 2021b Stanley al. 2021). These methods involve pre-training networks on large molecular datasets before being applied to domain-specific, small-scale target datasets. However, when applied to datasets of very small size (e.g.. 3000), most of these methods are prone to perform poorly and are statistically unstable (Hu et al., 2020). Moreover, as we will show in our experiments, these methods are less reliable when deployed on target datasets that contain significant domain gaps from the pre-trained dataset (e.g., inconsistency in molecule sizes).
As an alternative to pure deep learning- based methods, formal grammars over molecular structures offer an explicit, explainable representation for molecules and have shown their great potential in addressing molecular tasks in a data-efficient manner (Kajino 2019] Krenn et al. 2019 Guo et al.) 2021a 2022). A molecular grammar consists of set of production rules that can be chained to generate molecules. The production rules, which can either be manually defined (Krenn et al., 2019: Guo et al. a021ab or learned from data (Kajino 2019; Guo et al., 2022). encode necessary con- straints for generating valid molecular structures, such as valency restrictions. A molecular gram- mar has the combinatorial capacity represent large amount of molecules using relatively small number of production rules. It has thus been adapted as a data-efficient generative model (Kajino) 2019 Guo al. 2022). While molecular generation based on grammars has been widely studied and is relatively straightforward, extending the data- -ata-efficiency advantage of grammars to property prediction has not yet been well-explored.
Motivation. In this paper, we propose a framework for highly data-efficient property prediction based on a learnable molecular grammar. The intuition behind that the production rule sequences for molecule generation provide rich information regarding the similarity of molecular structures. For instance, two molecular structures that share common substructure would use the same se- quence of grammar production rules. As it is widely recognized in cheminformatics that molecules with similar structures have similar properties (Johnson & Maggiora 1990 Martin et al., 2002), grammar production sequences thus be used as strong structural prior to predict molecular properties. We aim to develop model that explicitly represents grammar production sequences and captures the structure-level similarity between molecules. Even from few molecules, this model is expected to reveal a wealth of information regarding the structural relationship between molecules, providing the key to a data efficient property predictor.
Framework. Figure D outlines our approach. At the heart of our method is a grammar-induced geometry (in the form of graph) for the space of molecules. In the geometry, every path trac- ing from the root to a leaf represents a grammar production sequence that generates a particular molecule. Such a geometry explicitly captures the intrinsic closeness between molecules, where structurally similar molccules are closer in distance along the geometry. In contrast to the embed- ding space used in most deep learning methods, our grammar-induced geometry is entirely explicit, which can be integrated with the property predictor by considering all involved molecules simul- taneously. To construct the geometry, we propose a hierarchical molecular grammar consisting of two parts: a pre-defined meta grammar at the top and a learnable molecular grammar at the bottom. We provide both theoretical and experimental evidence to demonstrate that the hierarchical molecu- lar grammar is compact yet complete. To predict properties, we exploit the structural prior captured by the grammar-indinn induced geometry using a graph neural diffusion model over the geometry. A joint optimization framework is proposed to simultaneously learn both the geometry and the diffusion.
2

---

Under review as a conference paper at ICLR 2023
Contributions. Our evaluation covers two small datasets of polymers (each containing ~300 sam- ples) and two large datasets of drug-like molecules (containing ~600 and ~=1 000 samples, re- spectively). Our approach outperforms both competitive state-of-the-ar GNN approaches and pre- trained methods by large margins. A further analysis shows that when trained on only a subset of the training data (~100 samples), our method can achieve performance comparable to pre-trained GNNs fine-tuned on the whole training set of the downstream prediction task, thus demonstrating the effectiveness of our method on extremely small datasets. We also show that our method can be integrated with molecular generation to guide the discovery of molecules with optimal properties.
Related Work. Our method is mainly related to three areas: 1) machine learning, particularly graph neural networks for molecular property prediction, in which we target the same problem but propose an entirely new approach: 2) grammars for molecular machine learning, which are used in existing works for molecular generation, while we use grammars to induce a geometry for molecular property prediction; and 3) geometric deep learning, where deep networks are applied to non-Euclidean domains. We briefly discuss the latter two in Section 2 as necessary background for our approach and refer to Appendix G Goor an in-depth discussion of related works.
2 PRELIMINARIES
Molecular Hypergraph Grammar (MHG). In MHGs, molecules are represented as hypergraphs H (V, where nodes represent chemical atoms and hyperedges represent chemical bonds or ring-like substructures. A MHG G (W, ,, P. X) contains set V of non-terminal nodes, a set Σ of terminal nodes representing chemical atoms, and a starting node X. describes how molecular hypergraphs are generated using set of production rules P P= {pali 1, k,....k k- of form p; LHS > RHS, where the left- and right-hand sides (LHS and RHS) are hypergraphs. Starting at X, a molecule is generated by iteratively selecting rule whose LHS matches a sub-hypergraph in the current hypergraph and replacing it with RHS until only terminal nodes remain. For each production rule, the LHS contains only non-terminal nodes and no terminal nodes, whereas the RHS can contain both non-terminal and terminal nodes. Both sides of the production rule have the same number of anchor nodes, which indicate correspondences when LHH is replaced by RHS in production step. For a formal definition, see Guo et al. (2022).
Graph Diffusion models the information propagation between nodes a graph using heat equa- tions. The node features are updated following a diffusion PDE as follows:
/"
T aU
au: a
Ur Uo= +
dt,
L, U.:
(1)
a
where matrix U, represents the features of all nodes in the graph time and the matrix L., which has the same sparsity structure as a graph Laplacian, represents the diffusivity defined on all edges in the graph. Lg is calculated using a(.,:; aa) parameterized by a, i.e. L,; a(U,)) UD)) a)) for all connected node pairs (i, For more details, see Chamberlain et al. (2021).
Notation. For graph grammar G E. X), say graph H can be derived from the grammar if there is a sequence of production rules from P that generates this graph, denoted by H.H. H1 &H2 denotes one-step grammar production that transforms graph H1 to graph H2 using rule As special forn of general graph, tree is denoted as T and the set of all possible trees is denoted as T. Note that all trees discussed in this paper are unrooted without order, i.c. connected acyclic undirected graphs (Bender & Williamson| 2010 A(T) denotes the maximal degree of T. (TD T₂) denotes the free edit distance between tree Ti and tree T2 as defined in Zhang (1996) and Paaßen (2018). |P| denotes the number of rules in a production rule set P and [T] denotes the size of tree T.
3 GRAMMAR-INDUCED GEOMETRY FOR PROPERTY PREDICTION
3.1 GRAMMAR-INDUCE) GEOMETRY
Problem Formulation. A molecular property predictor can be expressed as a function T(.) H. R that maps molecules formulated as hypergraphs H = (V, F.) H into scalar values. The mapping
3

---

Under review as a conference paper at ICLR 2023
Ps:
O
pat
pst
(a) Molecular hypergraph grummar
Stagle Rand
Onyacn Alesa Noak
Chaiie Ase Noda
teaiall Neda
Boaa Cather Nada
Navicmizzal Nodz
(b) Grammar-inducee geometry
Figure 2: (a) Example of a molecular hypergraph grammar and (b) its induced geometry. All possi- ble production sequences to generate three molecules are outlined in the geometry, where the nodes represent intermediate or molecular hypergraphs and the edges represent production steps. Struc- turally similar molecules have a smaller shortest path between them and are thus closer in distance along the geometry.
function π = f contains two components: an embedding function f(-) H. + R that maps input hypergraphs into an Euclidean latent feature space (also known as the embedding space), and a transformation function g(-) : IR" - R (usually a simple linear function) which maps latent features into property values. The embedding function is designed
capture the similarity between molecules, such that molecules with similar properties are closer in the embedding space in terms of Euclidean distance (Cayton 2005). The key reason why most supervised machine learning methods fail in data-sparse cases is that the embedded function fails to capture molecular similarity since it only has access to a limited number of samples with property labels. To address this issue, we propose an additional, explicit prior molecular similarity that does not depend on data labels, but provides valuable information for property prediction.
Grammar-induced Geometry. Our efforts motivated by the fact that structure-level similarity between molecules is known to be strongly correlated with their properties, e.g., organic molecules sharing functional group are proven to have similar chemical properties (Johnson & Maggiora, 1990 Martin et al.] al.] (200).. We explicitly model this structure-level similarity as an informative prior for data-efficient property prediction.
Inspired by the fact that grammars are generative models for molecules based on combinations of molecular substructures, we exploit the molecular hypergraph grammar capture the struc- tural similarity between molecules. Figure 2 outlines how a molecular hypergraph grammar G= GN Σ, P,X) can be used to construct a geometry for the space of molecules in the form of graph g = EV. The geometry enumerates all possible production sequences, where every pos- sible intermediate or molecular hypergraph is represented as node in V = {vl H, (V,, Ex)) }. and each production step is represented as an edge in & = [(s, 1) H. H,, P). In this geometry, every leaf node represents molecule. Every path tracing from the root Hroot = (X, Ø) to a Icaf' represents a production scquence that generates the corresponding molecule. Since all the molecules can be derived from the initial node X. any two leaves representing two molecules are connected in this geometry, where the path between them represents the sequence of transforming from one molecular structure to the other. More structurally similar molecules have a smaller shortest path as their molecular hypergraphs share common intermediate hypergraph. The distance between two molecules is defined as the shortest-path distance between them along the geometry with unit weight each edge. As result, structurally similar molecules are closer in distance along the geometry. We use this geometry as an additional input to the molecular property predictor: π' ('(:)) H. x 6 + R, where E is the -nnuued geometry. The geometry can be optimized in conjunction with the property predictor in order to minimize the prediction loss. As the geometry is determined by the grammar, the optimization of the geometry can be converted into the learning of production rules, where the latter can be achieved using the method described in Guo et al. (2022).
The crucial remaining question is how to construct this geometry G = (V,8) from a given molecular hypergraph grammar? A key characteristic of the grammar-induce, geometry is that each node in
'To distinguish the graph of the geometry from the graph of molecules, use hypergraphs to formulate the molecular graphs generated from grammar production.
4

---

Under review as a conference paper at ICLR 2023
the geometry represents unique intermediate hypergraph or molecular hypergraph. This ensures that the geometry includes the minimal path between two molecules. To satisfy this characteristic, use breadth-firs, first search (BFS) originating from the root Hoot = (X,, Ø) to iteratively expand the geometry. At each iteration when we visit a node e, we enumerate all the production rules that could be applied to the intermediate hypergraph He represented by the current node. Then we produce a new hypergraph H(() for each applicable rule p.: { H.
H hypergraph 11 if 11 new + He for all o' E V. we create new node v
H4 S
H1))
new Pi E P); For each new new?
H news add it to the node set V, and create new edge (v, 2 (ew)) ) that gets added to the edge set E of the geometry. If there is an isomorphic hypergraph in the existing node set, i.. Hg = H i V. we do not create any new node; instead, we add an edge (v, ú) between the matched node e and the currently visited node u. The algorithm terminates once all the molecules involved in the property prediction task have been visited in the geometry.
In practice, the construction of grammar-indues geometry very costly, and often computationally intractable for more complex grammars. By testing with random grammars learned from Guo et al. (0222), we find that it is infeasible to construct the geometry when there are more than ten production rules. The major bottleneck comes from the combinatorial complexity of production rules: The more rules a grammar has, the more intermediate hypergraphs it can generate. As the depth of the geometry increases, the number of intermediate hypergraphs increases exponentially. This incurs significant computational cost, as each newly expanded hypergraph requires pair-wise isomorphism tests. This cost also poses a serious obstacle to the optimization of the geometry, since the geometry must be rebuilt at each optimization iteration.
3.2 HIERARCHICAL MOLECULAR GRAMMARS
We propose hierarchical molecular grammar to address the computational challenge mentioned above. Our insight stems from the fact that every hypergraph can be represented as tree-structure object, i.e. a junction tree using tree decomposition (Diestell 2005). The tree representation brings two benefits: 1) It allows to measure the underlying structural similarity between molecules by reflecting the scaffold of the substructure components (Jin et al. 2018); and 2) as a more compact representation, the possible junction tree structures considerably fewer than molecular hyper- graphs, and therefore can be enumerated. By carefully designing the form of grammar rules, we divide the generation process of molecular hypergraph into two parts: first generating tree with homogeneous tree nodes, then converting the tree into a molecular hypergraph by specifying a sub- hypergraph per tree node. The grammar-induced geometry can thus be divided into two parts: a top part where geometry nodes represent only trees and bottom part where the leaves of the geometry represent molecular hypergraphs. Due this hierarchical decomposition, the top part of the geome- try is data-independent and can be computed offline to apply to any molecular datasets. At run-time, we only need to construct the bottom part of the geometry, which is formulated as byproduct of grammar learning in our approach. As a result of the geometry construction, each molecule is connected to one junction tree in the top part. Molecular structure similarity is determined by the distance between their corresponding junction trees along the geometry.
A hierarchical molecular grammar consists of two sub-gra uub-grammars that generate two parts of the ge- ometry respectively. We refer to the uub-grammar that only generates trees as meta grammar and the top-part geometry it constructs as a meta geometry. We provide formal definitions of meta grammar and demonstrate that despite the additional hierarchical constraint, our hierarchical molec- ular grammar is as expressive as the general molecular hypergraph grammar in [uoo et al. (2022) Figure 3 shows an overview of hierarchical molecular grammar and its induced geometry.
Meta Grammars. The formal definition of a meta grammar is as follows.
Definition 1. A meta grammar T = (W, Ø, P. X) is a hypergraph grammar, which only contains non-terminal nodes and only generates trees, i.e. Vu {w|X' p
w), w T.
A meta grammar G is k-degree if. for all trees T of maximal degree A(T) S k, we have X T.
A meta grammar G is edit-complete if. for any tree pair (T,T') with |T| < [T"), tree edit distance TED(T, = 1, and X P
X²T,T", T", there is a rule E Pc such that T ST.
A k-degree, edit-complete meta grammar G is minimal if there is no other such meta grammar G' with Poll < IPol.
5

---

Under review as a conference paper at ICLR 2023
P1
p1:
ps:
pe:
ps:
p
Meta Gennetry
(b) Meta grammar
Molecula Rale Leaming
PrP
Meta Molecular leaf'
Milecda Lourning
Makeake Rade
Leaviny
(a) Geometry induced by hiemrccicaal molecular grammar
(c) Molecular grammar
Figure 3: (a) Overview of the geometry induced by hierarchical molecular grammar. (b) A pre- defined meta grammar is used to pre-compute the meta geometry offline. (c) At run-time, molecular grammar is obtained using molecular rule learning. Each molecule is converted into junetion tree and is connected to meta tree node of the meta geometry that represents an isomorphic meta tree.
In Appendix H we provide the formal construction of meta grammars for generating trees with arbitrary degree and more elaboration the three additional attributes.
Proposition 1. A meta grammar G with meta rule set Pa constructed in Appendix H is of degree k. edit-complete, and minimal.
Appendix H also provides the proof that the three attributes are satisfied. Generally speaking, we construct meta rule sets of arbitrary degrees by induction from 1-degree meta grammar which con- sists of only one meta rule. practice, meta grammar with reasonable degree should be chosen to ensure the coverage of most molecules. In our experiments, we find it sufficient to use a meta rule set of degree 4, which contains 8 rules total.
Hierarchical Molecular Grammars. We define hierarchical molecular grammar follows,
Proposition 2. A hierarchical molecular grammar G = (G, Gmol) consisting of two hypergraph grammars: 1) a k-degree, edit-complete, and minimal meta grammar G defined in Definition 1 and 2) a molecular grammar Gool (N, Σ, Pmot, Ø) where the RHS of any molecular rule P only contains terminal nodes (i.e. atoms), is complete grammar.
More discussion on the proposition is in Appendix H The completeness of our hierarchical molec- ular grammar also shows that there is no loss of representation capacity compared general molecular hypergraph grammars.
Geometry Construction and Molecular Grammar Learning. The geometry induced by the hier- archical molecular grammar is constructed in two phases, as shown in Figure Bja the top for the meta grammar and the bottom for the molecular grammar. Since the meta grammar is pre-defined, the top meta geometry can be pre-computed offline following the BFS procedure described in Sec- tion with a given maximum BFS depth. Each node in the meta geometry represents a of non-terne non-terminal nodes. We call the tree generated using the meta grammar meta tree and the node in the meta geometry a meta tree node. We find it sufficient use maximum depth of 10 in practice. The bottom part of the geometry determines how each molecule is connected to the meta geometry as a molecular leaf through its junction tree structure. We obtain the junction trees of molecules simultaneously with the learning of molecular grammar rules. Figure (c) illustrates the process. Thus, the geometry optimization can be achieved by learning the molecular grammar.
We follow the grammar learning from Guo et al. (2022) but constrain the learned rules to contain only terminal nodes on the RHS. so as to satisfy the molecular grammar. Each molecule con- sidered as hypergraph, and set of hyperedges is iteratively sampled from the hypergraph until all hyperedges are visited. We construct the molecular rules along with the junction tree. At each iteration, molecular rule is constructed for each connected component, where the LHS contains

---

Under review as a conference paper at ICLR 2023
single non-terminal node and the RHS contains the connected component. For the junction tree, we add a tree node representing the connected component and create a tree edge between two nodes if their corresponding connected components share common hypergraph node. The iterative sam- pling follows an i.i.d. Bernoulli distribution based on a probability function Φ(e; 0) defined on each hyperedge with learnable parameters 0. At the conclusion of the sampling process, a junction tree of the molecular hypergraph along with set of molecular rules for converting this junction tree back the original hypergraph. The molecule is then connected the meta tree node in the meta geometry that represents the isomorphic meta tree to the junction tree. We provide a detailed formulation of the constructed molecular rule and the resulting junction tree in Appendix and more details of the grammar-induced geometry construction in Appendix
can obtain
3.3 GRAPH DIFFUSION OVER GRAMMAR GEOMETRY
Property prediction requires a model that can operate on our grammar-induced geometry and is suit- able for scarce data. We choose the graph neural diffusion model GRAND (Chamberlain et al. 2021] for its effectiveness in overcoming the oversmoothing that plagues most GNNs. Three learn- able components are used in a graph diffusion process: an encoder function yp defined on all the nodes in the grammar-induced geometry, decoder function & defined on molecular leaves, and graph diffusion process given in Equation Specifically, the encoder yp yields the initial state of the diffusion process Un- = (((n))) where Un is the matrix form of the input features of all the nodes. The decoder produces predicted property values of all molecular leaves ur = v O UT), where Ur is the node-feature matrix from the final diffusion state, M is a binary mask that masks out the rows corresponding to non- mol-mole nodes, @ is the Hadamard product, and uy is the resulting vector containing property values of all molecular leaves. Our optimization objective encodes the learning for both the grammar-induced geometry and the diffusion model:
min ((r, û) = min min l(ur, û); (.(1..%))
(2)
6 (0.00.)
where à represents the vector of ground-truth property values for all leaves and I(::)) )) is a regression loss. Recall that the geometry is only determined by the molecular rules, so the molecular grammar learning parameters 0 are the only parameters relevant for obtaining the geometry. Since 0 and (yp, s/, a) are two groups of independent variables, we exploit block coordinate descent (Wright, 2015) to optimize the objective. A detailed derivation is provided in Appendix K
4 EVALUATION
The experiments demonstrate the generality of our approach and answer the following questions:
Does our approach outperform existing methods on small datasets?
How well does our approach perform on large, widely-studied datasets?
To what extent is our approach effective on extremely small datasets?
.
Can our approach guide generative models toward molecules with optimal properties?
4.1
EXPERIMENT SETUP
Data. We evaluate our approach on eight datasets: CROW (a curated dataset from literature), Perme-
ability Yuan al. 2021), FreeSolv (Mobley & Guthrie 2014). Lipophilicity WWang et al. 2013
HOPV Lopez et al. 2016). DILI Ma et al. (2020). and PTC Xu et al. 2018p. These datasets
cover: 1) commonly used benchmark datasets including (FreeSot Lipophilicity. HOPV, Clin'oox) and TUDataset (PTC), 2) both classification (DILI, PTC, Clin'Tox and regression tasks (CROW, Permeability, FreeSolv, Lipophilicity, HOPV), and 3) sizes that are small (CROW, Perme- ability, HOPV, PTC) and large (FreeSoly, Lipophilicity, ClinTox). We report the mean absolute error (MAE) and the coefficient of determination (IZ²) for regression, and Accuracy and ROC-AUC scores for classification. See Appendix E for more details.
Baselines. We compare our approach with various approaches: Random Forest, FNN, WD-MPNN (D-MPNN), ESAN, HM-GNN, PN, and Pre-trained GIN. For descriptions, see Appendix F To show the generalizability of our pipeline (called Geo-DEG), we implement two versions, each with different diffusion encoder, GIN and MPNN. Appendix F provides the implementation details.
7

---

Under review as a conference paper at ICLR 2023
Table 1: Results on two small datasets of polymers (best result bolded, second-best underlined). Our
approach (Geo- DEG) outperforms state-of-the-a art GNNs and pre-trained methods by large margins.
CROW
Permeability
MAE
R² f
MAE |
R²
Random Forest
27.9 3.2
0.67 = 0.08
0.58 + 0.01
0.72 0.03
FFN
24.0
0.84 # 0.02
0.56 + 0.04
0.73 # 0.06
wD-MPNN ESAN
20.6 1.3
0.89 # 0.02
0.46 + 0.03
0.80 + 0.03
26.1 1.3
0.79 + 0.02
0.40
0.81 = 0.03
HM-GNN
30.8 1.8
0.76 = 0.01
=
0.68 0.01
PN (finetued)
21.1 1.3
0.89 0.01
0.48 + 0.04
0.70 = 0.03
Pre-trained GIN (finetued)
19.3 = 1.4
0.91 0.01
0.46 0.04
0.69 + 0.02
Geo-DEG (GIN)
17.0 + 1.4
0.92 + 0.01
0
0.84 0.02
Geo-DEG (MPNN)
18.5 + 1.2
0.91 = 0.01
0.32 + 0.03
0.03
0.83 0.02
4.2 RESULTS ON SMALL DATASETS
To answer the first question posted at the beginning, we conduct experiments five small datasets: CROW, Permeability, HOPV, DILI, and PTC. Jable[ssovy the results on the first two datasets, which contain polymers. Regarding MAE, the baselines ESAN and HM-GNN are only compara- ble with simple, non-deep learning methods (e.g., random forest), while pre-trained GIN and wD- MPNN perform reasonably well. Both variants of our method outperform all the other methods by a large margin. Results and discussion on the other datasets are given in Appendix A
Discussions. In general, CROW is more challenging dataset than Permeability, as it has fewer samples and a broader range of property values (with standard derivation of 86.8 versus 1.24 for Permeability). On CROW, traditional machine learning methods such as Random Forest and FNN are quite competitive and even outperform modern GNN architectures such as ESAN and HM-GNN. wD-MPNN achieves reasonable performance thanks to the special graph representation for polymer ensembles. PN and Pre-trained GIN perform exceptionally well on CROW due to their pre-training on large datasets. However, this pre-training does not help on Permeability, which has a much larger average molecule size (with an average molecular weight of 391.8 versus 153.8 for CROW). Thus, there is a domain gap between the dataset where PN and Pre-trained GIN are pre-trained and the Permeability dataset, resulting in poor performance of both methods. ESAN benefits from the subgraph selection scheme on large molecular graphs and performs well on Permeability. The poor performance of HM-GNN both datasets shows that is not as effective in regression tasks is in classification tasks. The overall results show that: 1) our method is capable of handling polymers with varying molecular sizes, and 2) the superior performance of our method when coupled with either GIN or MPNN confirms its generalizability to different feature extractors.
4.3 RESULTS ON LARGE DATASETS
To answer the second question, we conduct experiments on three large datasets: FreeSolv, Lipophilicity, and ClinTox. Tables and in Appendix [A] show the results. For FreeSolv and Lipophilicity, simple machine learning methods including random forest and FFN perform poorly. Pre-trained GNNs cannot perform as well as they do on small polymer datasets. The performance of advanced GNN architectures, especially D-MPNN, is reasonably good on both datasets. Our method equipped with MPNN diffusion encoder performs the best among all approaches. Results and discussion on the other datasets are given in Appendix A
Discussions. On large datasets, random forest and simple FFN become less competitive than on small datasets. This is reasonable since larger datasets require larger model capacities to represent the relationship between molecules and their properties. Pre-training does not significantly help on either dataset due to the inherent domain gap between datasets used for pre-training versus those used for fine-tuning. Among the three GNN-based baselines, D-MPNN performs the best and ex- hibits better stability than ESAN and HM-GNN. Our approach further improve the performance of D-MPNN and outperforms all the baselines. These results demonstrate that our method is scalable and competitive with ofte-of-the-ar methods for large molecule datasets.
8

---

Under review as a conference paper at ICLR 2023
Prosrained GIN
Protenal Solearayy
12%
30%
37%
Analysis extremely small datasets
Oun-GIX
OKn-MISS
100
Cr Paraciblity
10
CUW Fanaalii Paecbb Lipele
(a) Analysis combination with generative models
(c) Ablation study
grammar
Figure 4: (a) Two objectives from the Permeability dataset predicted using our method combined with the grammar based generative model. Molecules lying on the Pareto front are highlighted. (b) Performance of our method trained on subsets of CROW training data with different ratios. Even when the training set halved, our approach can still achieve competitive results compared to Pre- trained GIN fine-tuned on the whole training set. (c) Comparison of performance using different meta grammars, demonstrating the necessity of complete meta grammar.
4.4 ANALYSIS
This section examines the effectiveness our method on extremely small datasets, its combination with generative models that answer the last two questions posted at the beginning, and ablation study on the meta grammar.
Performance on Extremely Small Datasets. We conduct study on the minimum viable number of samples for our method. We randomly sample 20% of CROW as a fixed testing set. Using the remaining data, we construct eight randomly selected training sets, each consisting of portion of all remaining data, ranging from 20% to 100%. These eight training sets are used to train our method using GIN the feature extractor. Figure b)) shows the performance
the testing set. Even when the training set is halved (with only 94 samples), our approach still achieves results that comparable those of the Pre-trained GIN fine-tuned on the whole training set. Appendix B includes more detailed study for the effect of changing dataset training size on model performance.
Combination with Generative Models. In our optimization framework, grammar metries can be considered as additional objectives, allowing us to jointly optimize both generative models and property predictors. Following Guo et (2022), we use diversity and Retro" score as our grammar metrics and then perform joint optimization on the Permeability dataset. After training, we generate 400 molecules and predict their property values using our approach, including all six property types from Permeability. Figure 4[aa shows two out of six predicted property values with the Pareto front highlighted. Clearly, our approach can be combined with generative models to provide comprehensive pipeline for the discovery of optimal molecules. The use of Retro" also enables finding synthesis paths of generated molecules as shown in Appendix L
Ablation Study on Meta Grammar. In this study, examine the necessity of the meta rules in our hierarchical grammar. We remove two rules that have degree 4 on the LHS from the 4-degree meta grammar, resulting in meta geometry with same number of nodes but 10% fewer edges than the one used in our main experiments. With this modified meta grammar, we run the pipeline for all four datasets and compare the results with the original meta grammar in Figure Fc). All four datasets exhibit performance drop when using the modified meta grammar. The results provide experimental evidence for the necessity of complete meta grammar.
5 CONCLUSION
We propose data-efficient molecular property predictor based hierarchical molecular gram- mar. The grammar induces an explicit geometry describing the space molecular graphs, such that a graph neural diffusion on the geometry can be used to effectively predict property values of molecules on small training datasets. One avenue of future work is to extend our pipeline to model 3D molecular structures and to address general graph design problems.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mohammad Atif Faiz Afzal, Andrea R Browning, Alexander Goldberg, Mathew D Halls, Jacob L Gavartin, Tsuguo Morisato, Thomas F Hughes, David Giesen, and Joseph E Goose. High- throughput molecular dynamics simulations and validation of thermophysical properties of poly- mers for various applications. ACS Applied Polymer Materials, 3(2):620 630, 2020.
Matteo Aldeghi and Connor W Coley. A graph representation of molecular ensembles for polymer property prediction. arXiv preprint arXiv:2205.08611 2022.
Uri Alon and Eran Yahav, On the bottleneck of graph neural networks and its practical implications. arXiv preprint arXiv:2006.05203 2020.
Han Altae-Tran, Bharath Ramsundar, Aneesh S Pappu, and Vijay Pande. Low data drug discovery with one-shot learning. ACS central science, 3(4):283-293, 2017.
Evan R Antoniuk, Peggy Li, Bhavya Kailkhura, and Anna M Hiszpanski. Representing polymers as periodic graphs with learned descriptors for accurate polymer property predictions. arXiv preprint arXiv:2205. 13757, 2022.
Debra J Audus and Juan J de Pablo. Polymer informatics: Opportunities and challenges, ACS macro letters, 6(10);107--108, 2017.
Edward A Bender and S Gill Williamson. Lists, decisions and graphs. S. Gill Williamson, 2010.
Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M Bronstein, and Haggai Maron. Equivariant subgraph aggregation net- works. ICLR, 2022.
Derrick Blakely, Jack Lanchantin, and Yanjun Qi. Time and space complexity of graph convolutional networks. Accessed on: Dec. 31, 2021.
Cristian Bodnar, Francesco Di Giovanni, Benjamin Paul Chamberlain, Pietro Liò, and Michael M Bronstein. Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in gnns. arXiv preprint arXiv:2202.04577 2022.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geomet- ric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-44 2017.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021.
Wenming Cao, Zhiyue Yan, Zhiquan He, and Zhihai He. A comprehensive survey on geometric deep learning. IEEE Access, 8:35929-35949, 2020.
Lawrence Cayton. Algorithms for manifold learning. Univ. of California at San Diego Tech. Rep, 12(1-17):1, 2005.
Ben Chamberlain, James Rowbottom, Maria I Gorinova, Michael Bronstein, Stefan Webb, and Emanuele Rossi. Grand: Graph neural diffusion. In International Conference on Machine Learn- ing, pp. 1407-1418, PMLR, 2021.
Lihua Chen, Ghanshyam Pilania, Rohit Batra, Tran Doan Huan, Chiho Kim, Christopher Kuenneth, and Rampi Ramprasad. Polymer informatics: Current status and critical next steps. Materials Science and Engineering: R: Reports, 144:100595, 2021.
Luigi Pietro Cordella, Pasquale Foggia, Carlo Sansone, and Mario Vento. An improved algorithm for matching large graphs. In 3rd IAPR-TCI5 workshop on graph-based representations in pattern recognition, pp. 159-159. Citeseer, 2001.
Luca Cosmo, Anees Kazi, Seyed-Ahmad Ahmadi, Nassir Navab, and Michael Bronstein. Latent- graph learning for disease prediction. In International Conference on Medical Image Computing and Computer-Assissen Intervention, pp. 643-653. Springer, 2020
10

---

Under review as a conference paper at ICLR 2023
Calin Cruceru, Gary Bécigneul, and Octavian- Eugen Ganea. Computationally tractable riemannian manifolds for graph embeddings. In Proceedings of the AAAI Conference on Artificial Intelli- gence, 2021.
Hanjun Dai, Yingtao Tian, Bo Dai, Steven Skiena, and Song. Syntax-directed variational autoen- coder for structured data. arXiv preprint arXiv: 1.kiv:8802.0000 1802.08786, 2018.
Reinhard Diestel. Graph theory 3rd ed. Graduate texts in mathematics, 173:33, 2005.
Ahmed AA Elhag, Gabriele Corso, Hannes Stärk, and Michael M Bronstein. Graph anisotropic diffusion for molecules. In ICLR2022 Machine Learning for Drug Discovery, 2022.
Evan N Feinberg, Debnil Sur, Zhenqin Wu, Brooke Husic, Huanghao Mai, Yang Li, Saisai Sun, Jianyi Yang, Bharath Ramsundar, and Vijay S Pande. Potentialnet for molecular property predic- tion. ACS central science, 4(11):1520-23)
2018.
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. Advances in neural information processing systems, 31, 2018.
Garrett B Goh, Nathan O Hodas, Charles Siegel, and Abhinav Vishnu. Smiles2vec: An inter- pretable geweral- purpose deep neural network for predicting chemical properties. arXiv preprint arXiv:1712.02034, 2017.
Minghao Guo, Wan Shou, Liane Makatura, Timothy Erps, Michael Foshey, and Wojciech Ma- tusik. Polygrammar: Grammar for digital polymer representation and generation. arXiv preprint arXiv:2105.05278, 2021a.
Minghao Guo, Veronika Thost, Beichen Li, Payel Das, Jie Chen, and Wojeiech Matusik. Data- efficient graph grammar learning for molecular generation. arXiv preprint arXiv:2203.0803/ 2022.
Zhichun Guo, Chuxu Zhang, Wenhao Yu, John Herr, Olaf Wiest, Meng Jiang, and Nitesh V Chawla. Few-shot graph learning for molecular property prediction. In Proceedings of the Web Conference 2021, pp. 2559-2567, 2021b.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. ICLR, 2020.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. International conference on machine learning, pp. 2323-2332. PMLR, 2018.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In International conference on machine learning, pp. 4839-4848. PMLR, 2020.
Mark A Johnson and Gerald M Maggiora. Concepts and applications of molecular similarity. Wiley, 1990.
Chonghyo Joo, Hyundo Park, Jongkoo Lim, Hyungtae Cho, and Junghwan Kim. Development of physical property prediction models for polypropylenc composites with optimizing random forest hyperparameters. International Journal of Intelligent Systems, 37(6):3625-3653 2022.
Hiroshi Kajino. Molecular hypergraph grammar with its application to molecular optimization. In International Conference on Machine Learning, pp. 3183-3191. PMIR, 2019.
Mario Krenn, Florian Häse, A Nigam, Pascal Friederich, and Alán Aspuru-Guzik. Selfies: a robust representation of semantically constrained graphs with example application in chemistry. arXiv preprint arXiv:190..13741 2019.
Steven A Lopez, Edward O Pyzer-Knapp, Gregor N Simm, Trevor Lutzow, Kewei Li. Laszlo R Seress, Johannes Hachmann, and Alán Aspuru-Guza Guzik. The harvard organic photovoltaic dataset. Scientific data, 3(1):177, 2016.
11

---

Under review as a conference paper at ICLR 2023
Alessandro Lusci, Gianluca Pollastri, and Pierre Baldi. Deep architectures and deep learning in chemoinformatics: the prediction of aqueous solubility for drug-like molecules. Journal of chem- ical information and modeling, (117):1563-15777 2013.
Hehuan Ma, Weizhi An, Yuhong Wang. Hongmao Sun, Ruili Huang, and Junzhou Huang. Deep graph learning with property augmentation for predicting drug-induced liver injury. Chemical Research Toxicology, 34(2):495-506 2020.
Yvonne C Martin, James L Kofron, and Linda M Traphagen. Do structurally similar molecules have similar biological activity? Journal of medicinal chemistry, 45(19):430-43338 2002.
Krzysztof Maziarz, Henry Jackson-Flux, Pashmina Cameron, Finton Sirockin, Nadine Schneider, Nikolaus Stiefl, Marwin Segler, and Marc Brockschmidt. Learning to extend molecular scaffolds with structural motifs. arXiv preprint arXiv:2103003380 2021.
Aditya Menon, James A Thompson-Colón and Newell R Washburn. Hierarchical machine learn- ing model for mechanical property predictions of polyurethane elastomers from small datasets. Frontiers in Materials, 6:87, 2019.
David L Mobley and Peter Guthrie. Freesolv: database of experimental and calculated hydration free energies, with input files. Journal of computer-aided molecular design, 28(7):711 1-7000 2014.
AkshatKumar Nigam, Robert Pollice, Mario Krenn, Gabriel dos Passos Gomes, and Alan Aspuru- Guzík. Beyond generative models: superfast traversal, optimization, novelty, exploration and discovery (stoned) algorithm for molecules using selfies. Chemical science, 2021.
Benjamin Paaßenn. Revisiting the tree edit distance and its backtracing: A tutorial. arXiv preprint 1805.0180500666 2018.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems, 33:12559-125711 2020.
Jerret Ross, Brian Belgodere, Vijil Chenthamarakshan, Inkit Padhi, Youssef Mrouch, and Payel Das. Do large scale molecular language representations capture important structural information? arXiv preprint arXiv:2 2021.
Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borg- wardt. Weisfeiler-Iehman graph kernels. Journal of Machine Learning Research, 12(9), 2011.
Megan Stanley, John Bronskill, Krzysztof Maziarz, Hubert Misztela, Jessica Lanini, Marwin Segler, Nadine Schneider, and Mare Brockschmidt. Fs-mol: A few-shot learning dataset of molecules. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Govindan Subramanian, Bharath Ramsundar, Vijay Pande, and Rajiah Aldrin Denny. Computational modeling of f3-sccretase 1 (bace-1) 1) inhibitors using ligand based approaches. Journal of chemical information and modeling, 1936:)999991199 2016.
Lei Tao, Guang Chen, and Ying Li. Machine learning discovery of high-temperature polymers. Patterns, 2(4):100225, 2021a.
Lei Tao, Vikas Varshney, and Ying Li. Benchmarking machine learning models for polymer formatics: An example of glass transition temperature. Journal of Chemical Information and Modeling, 61(11):5395-5413 2021b.
Jian-Bing Wang, Dong- Cao, Min-Feng Zhu, Yong-Huan Yun, Nan Xiao, and Yi-Zeng Liang. In silico evaluation of logd7. and comparison with other prediction methods. Journal of Chemo- metrics, 29(7):389-398 2015.
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):279- 287, 2022.
12

---

Under review as a conference paper at ICLR 2023
Oliver Wieder, Stefan Kohlbacher, Mélaine Kuenemann, Arthur Garon, Pierre Ducrot, Thomas Sei- del, and Thierry Langer. A compact review of molecular property prediction with graph neural networks. Drug Discovery Today: Technologies, 37:1-12, 2020.
Ronald J Williams. Simple statistical gradient-folloingg algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-256, 1992.
Stephen J Wright. Coordinate descent algorithms. Mathematical Programming, 151(1):3-34, 2015.
Fang Wu, Qiang Zhang, Dragomir Radev, Jiyu Cui, Wen Zhang, Huabin Xing, Ningyu Zhang, and Huajun Chen. Molformer: Motif-based transformer on 3d heterogeneous molccular graphs. arXiv preprint /rXi::///0001111, 2021.
Chencheng Xu, Qiao Liu, Minlie Huang, and Tao Jiang. Reinforced molecular optimization with neighborhood-contron grammars. Advances in Neural Information Processing Systems, 33: 8366-8377, 2020.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv: 180.:0800 2018.
Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman- Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molecular repre- sentations for property prediction. Journal of chemical information and modeling, 59(8):3370- 3388, 2019.
Zhaoning Yu and Hongyang Gao. Molecular representation learning via heterogeneous motif graph neural networks. In International Conference on Machine Learning. pp. 25581-25594 PMLR, 2022.
Qi Yuan, Mariagiulia Longo, Aaron W Thornton, Neil B McKeown, Bibiana Comesana-Gandara Johannes Jansen, and Kim Jelfs. Imputation missing gas permeability data for polymer membranes using machine learning. Journal of Membrane Science, 627:1 19207, 2021.
Kaizhong Zhang. A constrained edit distance between unordered labeled trees. Algorithmica, 15 (3):205-222, 1996.
Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self- supervised learning for molecular property prediction. Advances Neural Information Process- ing Systems, 34:15870-15882. 2021.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Kc. Uni-mol: A universal 3d molecular representation learning framework. 2022.
13

---

Under review as a conference paper at ICLR 2023
A
ADDITIONAL RESULTS
Table 2: Results on FreeSolv and Lipophilicity (best bolded, second-best underlined).
FreeSolv
Lipophilicity
MAE +
R² f
MAE +
R² f
Random Forest
4.58 = 0.34
0.60 = 0.09
0.64 = 0.04
0.65 + 0.03
FFN
3.67 0.40
0.77 = 0.05
0.51 = 0.02
0.80 0.03
D-MPNN ESAN
0.54 0.08
0.04 0.00
0.90 + 0.02
0.67 0.07
0.00
0.46 0.04
0.90 + 0.01
HM-GNN
0.63 = 0.05
0.02
0.58 # 0.04
0.84 + 0.02
PN (finetued)
0.65 # 0.04
0.85 + 0.02
0.56 0.04
0.81 = 0.02
Pre-trained GIN (finetued)
1.01 0.11
0.74
0.52 0.03
0.89 0.01
Geo-DEG (GIN)
0.62 0.06
0.90 + 0.02
0.48 0.02
0.88 0.02
Geo-DEG (MPNN)
0.49 # 0.06
0.94 + 0.02
0.42 + 0.02
0.91 + 0.02
Table 3: Results on HOPV (best bolded, second-best underlined).
MAE (normalized)
Methods
R22 1
Random Forest
0.36 + 0.03
0.69 0.05
FFN
0.35 + 0.03
0.67 + 0.06
D-MPNN
0.36 + 0.03
0.69 + 0.04
ESAN
0.37 # 0.02
0.66 + 0.06
HM-GNN
0.40 + 0.02
0.65
PN (finetued)
0.42 + 0.02
0.65
Pre-trained GIN (finetued)
0.38 + 0.02
0.66 + 0.03
Geo-DEG (GIN)
0.32 + 0.03
0.70
Geo-DEG (MPNN)
0.30 + 0.02
0.74 + 0.03
Table 4: Results on DILI and PTC (best bolded, second- underlined).
DILI
PTC
Accuracy 1
ROC-AUCT
Accuracy 1
ROC-AUCT
Random Forest FFN
0.70 # 0.09
0.80 + 0.06
0.60 + 0.06
0.63 # 0.05
0.69 = 0.07
0.78 + 0.07
0.58 + 0.06
0.61 + 0.04
D-MPNN
0.75 # 0.10
0.83 + 0.07
0.67 # 0.06
0.70 # 0.05
ESAN
0.74 # 0.10 0.12 0.10
0.64 + 0.08
0.68 = 0.06
HM-GNN
0.76 0.08
0.83
0.66 + 0.07
0.69 0.06
PN (finetued)
0.75 + 0.10
0.83
0.61 + 0.08
0.65 + 0.07
Pre-trained GIN (finetued)
0.74 = 0.09
0.82
0.62 + 0.09
0.66 = 0.07
Geo-DEG (GIN)
0.76 + 0.09
0.84 + 0.05
0.64 + 0.09
0.68 + 0.06
Geo-DEG (MPNN)
0.78 0.08
0.86 + 0.06
0.69 + 0.07
0.71 # 0.07
Results & Discussion. For HOPV, since it is a small regression dataset, traditional methods (random forest and FFN) achieve competitive performance and are even better than several graph neural networks including ESAN and HM-GNN. Both variants of our method outperform other basclines by large margin, which aligns with the results the other two small regression datasets. In the two small classification datasets, DILI and PTC, there is no significant difference between baseline models regarding performance. The reason for this is that the test dataset is small, and the accuracy of the model can be limited by few hard examples that cannot be classified correctly. is therefore more informative to use the ROC-AUC score. Among the baselines, HM-GNN performs the best due to its motif-based representation. Our method can still outperform all other methods regarding both accuracy and ROC-AUC score. For ClinTox in Table 5 we compare our approach with a wide range of highly performant baseline methods from recent paper Zhou al. 2022 which
14

---

Under review as a conference paper at ICLR 2023
Table 5: Results on ClinTox (best bolded, second-best underlined).
Methods
ROC-AUC1
D-MPNN
90.6
AttentiveFP
84.7 0.3
N-Grampr
7..5
N-GramxGB
8.7
BROVERbase
81.2
GROVER Jurge
7.72
GraphMVP
7.81
MoICLR
9.52
GEM
90.1 1.3
Pre-trained GIN
72.6 1.5
Uni-Mol
91.9 + 1.8
Geo-DEG (GIN)
74.4 + 1.8
Geo-DEG (MPNN)
92.2 + 0.7
by itself proposes one of the SOTA methods on Clin'Tox. Note that these are very strong baseline methods, including many methods pre-trained using 3D molecular data. The overall results show that our method achieves significantly better performance on challenging small regression datasets and outperforms a wide spectrum of baselines on various common benchmarks.
B ANALYSIS ON TRAINING DATASET SIZE
We further extend our study in Section for the effect of changing dataset training size on model performance by including the analysis of all three small regression datasets with more baselines compared. Figure 5 illustrates the results of Pre-trained GIN, wD-MPNNN/D-MPNN, and two vari- ants of our model on CROW, Permeability, and HOPV. We report the performance of cach model trained using different numbers of training samples, randomly sampled from the original training dataset. Across different training dataset sizes, our proposed method consistently outperforms other baselines. A smaller training dataset leads to a larger performance improvement, demonstrating the data-efficiency of our model.
<de Fioca and
DAIPN
Go 200 XI1)
4G3 28330
MAE (oomaloo))
MAN
MAE
116
1X
0:31
1.30
219
(Tranung sumpies
Trissisa sample
sample (c) HEPP
(b) Permosbiiit
Figure 5: Analysis on training dataset size.
C ANALYSIS ON CONSTRUCTION COST REDUCTION OF HIERARCHICAL MOLECULAR GRAMMAR
In this section, we provide empirical evidence on the construction cost reduction of our hierarchical molecular grammar compared with non-hierarchical molecular grammar from Guo et al. 2022)) We conduct ten groups of experiments by randomly sampling five molecules from CROW dataset by ten times. For each group of experiment, we sample ten grammars for both hierarchical and non-hierarchical versions respectively and construct the geometry to cover the five molecules. The
non-hierarchical grammar is sampled using the algorithm from Guo et al.) (2022). The geometry
construction is achieved following the BFS procedure described in Section 3.1 for both kinds of
15

---

Under review as a conference paper at ICLR 2023
Awarge Number of Riilles
Non-hienschical grammer
[Geearhicaal grammar
Avenge Rurning Time (6)
Averge Numter of Itrenoden Grayts
Experiment
ID
Figure 6: Analysis construction cost reduction of hierarchical molecular grammar. For our hierar- chical molecular grammar, the number of intermediate graphs is the same as the number of molecule samples, which is five in the experiments.
grammar. In the experiments, we find that for non-hierarchical grammar, even for five molecules, occurs frequently that the size of the geometry grows extremely large but still cannot cover the five molecules. Therefore, we stop the construction when the size of the geometry reaches 2 10 iilluttalsss (a) the average running time, (b) the average number of production rules, and (c) the average number of intermediate graphs of the final constructed geometry. It can be noted that our hierarchical molecular grammar greatly reduces the running time of the geometry construction. For non-hierarchical grammars, it is always intractable to construct the geometry since the number of intermediate graphs enormous (some approaching 1.5x 109). Our hierarchical molecular grammar provides a practical framework for grammar-induced geometry with theoretical foundations.
D ANALYSIS ON COMPUTATIONAL COMPLEXITY
The computation of our approach consists of three parts: 1) the construction of grammar-induced geometry, 2) diffusion encoder, and 3) graph diffusion over the geometry. Following Blakely et al. 2021). suppose we have K samples of molecules, each with N nodes and |&| edges. To construct grammar-induced geometry, we need to sample all the edges in order to contract molecules into junction trees, so the computational complexity is O(K| ( ). For diffusion encoder, the complexity the same with general graph neural networks, which is O Ld {E)), where L is the number of layeres and d is the feature dimension. As indicated in Chamberlain et al. (202T). the complexity of graph diffusion is (((E')d)(E, + Ey). where E' the edge set rrammar-inducee geometry, and Eb, E are the numbers of function evaluations for forward and backward pass, respectively. Therefore, the overall computational cost of our method is OKK El K Ld( (Ndd(N++))) +E + |E")d(( E, + Ey)) O Ld( (Nd + (&|) E8' [dd[(E,, + E2)). Compared with general graph neural networks which have a computational complexity of O(KLd(Nd + [E))), our approach entails additional computations O1E" [d[ Es EE)), where [g'] the of the number of edges in the meta geometry |Emetaa and the number of samples K, since each molecule is connected to the meta geometry by an edge. In our experiments, we use 4-degree meta grammar with depth 10, which gives us meta geometry of 455 edges, i.c. 455 = EEettal < K|E). We also find in our experiments that E, + +E, < LNd. As result, the overall computational complexity our method the same as that of general graph neural networks in the big O sense. In practice, geometry can be constructed parallel, which reduces the running time of our entire method.
E DETAILS ON DATASETS
The Chemical Retrieval on the Web (CROW) polymer database curates database of thermo- physical data for over 250 different polymers, focusing primarily on the most commonly used plas- ties and resins in industry. CROW distinguishes from other digital polymer property databases due
htttps://polyneeee
16

---

Under review as a conference paper at ICLR 2023
to its heavy usage of real, experimental data individually sourced via extensive literature search. In the CROW polymer database. approximately 95% of reported polymers offer some experimental data from literature, with the remaining 5% derived purely from simulation. We only use those data with labels from real experiments in our evaluation. The CROW polymer database reports several properties including the glass transition temperature, Hildebrand solubility parameter, molar heat capacity, refractive index, and molar cohesive energy. Of these, we choose glass transition temper- ature as our benchmark performance comparison since it is one of the best documented polymer properties in the literature (Tao et al.) 2021b). For each dataset except Clin'To,, we randomly split the data into 4 1 training and testing sets and create five such splits using 5 random seeds. For ClinTox, we follow the settings in Zhou et al. (2022) to train our model and report the results of other baselines directly from the paper. Each of five splits is used to train and test a separate instance of each benchmark model. For the other three datasets, we refer the reader to the original papers: Permeability (Yuan et al., 2021). FreeSolv (Mobley & Guthrie 2014). Lipophilicity (Wang et al. 2013). HOPV (Lopez et al. 2016). DILI (Ma et al. 2020), and PTC (XC al. 2018).
F DETAILS ON THE IMPLEMENTATION
Baselines. We compare our approach with various baselines: 1) Random Forest and FFN, two of the best-performing machine learning models for polymer informatics benchmarking (Tao et al.) 2021b); 2) wD-MPNN, a state-of-the-an method specifically designed for polymer property pre- diction (Aldeghi & Coley 2022p: 3) ESAN. a general GNN architecture with enhanced expressive power (Bevilacqua et al. 2022;; 4) HM-GNN, motif-based GNN for molecular feature represen- tation (Yu & Gao 2022 5) PN and Pre-trained GIN, two pre-trained GNNs with ofate-of-the-art performance on shw-shot learning (Stanley et al.) 202T) and transfer learning (Hu et al.) 2020). As HM-GNN and Pre-trained GIN only provide code for graph classification, we modify the final layer of their networks and use 4 loss for training. For the other methods, we follow the same implemen- tation as their original papers. Since wD-MPNN cannot be depolyed general molecules other than polymers, we report the results of D-MPNN (Yang et al. 2019) instead for two large datasets.
Our System. For our approach, meta geometry contains all the meta trees whose size is smaller than 11, resulting in 149 nodes and 455 edges. For the molecular rule learning of 0, follow all the hyperparamters used in Guo al. (2022). For the graph diffusion, the input feature of each meta tree node is the Weisfeiler Lehman graph hashing feature (Shervashidze et 2011). The encoder for meta tree nodes is an embedding layer that maps hashing features into a 300-dimension continuous vector. For molecular leaves, we use two different encoders: GIN from Xu et al. (2018) and MPNN from Yang et al. (201)). both of which output feature vector of dimension 300. For the decoder, we use layer fully connected network with size 300. For the graph diffusion process, we follow Chamberlain et al. (022)) and use Dormand Prince adaptive step size scheme (DIOR15) with adjoint method. The diffusivity function a(;:;)) an attention function. We use Adam optimizer for the training of 0 and (4p, v/, ar), with learning rate 0.01 and 0.001, respectively. We train 0 for ten epochs. For each training epoch of 0, we train (1p, v,a) a) for 50 epochs.
use 4-degree meta grammar, which contains eight rules. The
G RELATED WORKS
Machine Learning for Molecular Property Prediction. The use of machine learning methods to predict molecular properties has a long history. Many early methods use SMILES strings or handcrafted fingerprints as input and rely on traditional machine learning methods, such as random forest and Gaussian processes, which are still competitive today in many applications (Tao et al., 2021b) Joo et al.) 2022). Recently, graph- based representations of molecules have gained increasing popularity with the development of GNNs (Feinberg et al. 2018 Xu et al. 2018 Wu et al. 2021 Bevilacqua et al.] [022] Yu & Gao. 2022, Aldeghi & Coley, 2022] Alon & Yahav 2020). We refer the reader to Wieder et al. (2020) for a detailed review of GNN-based property predictors. Current state-of-the-art GNN-based methods provide more advanced representations of molecules based on graphs. [Bevilaquaa et al. (2022) represents each individual graph as a set of subgraphs, increasing the expressive power of GNNs. Aldeghi & Coley (202) tailors molecular graphs by
adding stochastic edges and designs GNN
specifically for polymers based on Yang et al. (2019).
Yu & Gao (2022) and Zhang et al. (2021) leverage motifs to model molecular relationships. All
17

---

Under review as a conference paper at ICLR 2023
these methods require large training datasets to achieve reasonable performance. It is common to use self-supervised learning (Rong et al., 2020; Wang et al.) 2022) and transfer learning (Hu et al. 2020) to deal with sparse data, where neural networks fine-tuned on the target small dataset. Most of these methods, however, focus solely on molecular classification, while our approach can also address regression problems with extremely sparse data, which are considerably more challenging.
pre-trained large datasets and then
Molecular Grammars. As an interpretable and compact design model, grammar has recently gained increasing attention in the field of molecule discovery (Dai el al. 2018 Kajino 2019 Krenn
al. 2019 Xu et al. 2020: Nigam al. 202T Guo al., 2021a 2022) A molecular grammar can
be considered as a generative model that uses production rules to generate molecules. The rules of a
2019 Guo et al. 202 la or automatically learned from molecular datasets (Kajino 2019 Xu et al.
grammar can be either manually constructed (Dai et 2018 Krenn et al.
2020 Nigam et al. 2021
Guo et al., 2022). Recent works have demonstrated that a learnable hypergraph grammar can be
effective in capturing hard chemical constraints, such valency restrictions (Kajino 2019 Guo
ct al. 2022) Guo et al. (202)) further proposes a learning pipeline for constructing a hypergraph
grammar-based based generative model that can incorporate domain-specifie knowledge from very small datasets with dozens of samples. Despite the inherent advantages of molecular grammars, such as explicitness, explanatory power, and data efficiency, existing works mainly use them for molecular generation. Prediction and optimization of molecular properties can only be accomplished by using separate model based on the grammar representation in a latent space, which is learned individu- ally (Kajino 2019 Xu et al. 2020). We integrate molecular grammar into property prediction tasks by constructing a geometry molecular graphs based on the learnable grammar, which allows us to optimize both the generative model and the property predictor simultaneously. On extremely small datasets, our approach benefits especially from the data efficiency of grammar and achieves superior performance over existing property predictors.
Hierarchical Molecular Generation. The decomposition of molecular structures our hierarchi- cal molecular grammar is related to substructure-bare based methods in molecular generation (Tin et al.)
2018 2020 Maziarz et al. 202T Jin et al. (2018) employs encoders and decoders to generate a junction tree-structured scaffold and then combine chemical substructures into molecule. Jin et al. 2020 uses a hierarchical graph encoder-decoder to generate molecules in a coarse-to- manner, from atoms to connected motifs. |Mziarz al. (202]) integrates molecule fragments and atom- by-atom construction to generate new molecules. Our approach is fundamentally different from all these existing hierarchical representations in two respects: 1) Without the need for any training, our proposed meta grammar (the coarse level) can enumerate all possible junction tree structures by using a compact set of meta production rules, which can be theoretically guaranteed. Our method only requires learning at the fine level, which are molecular fragments determined by molecular rules, whereas existing methods require learning two models for both coarse and fine levels. 2) As opposed to existing methods that use latent spaces, our meta geometry induced by meta grammar explicitly models the similarity between molecules by using graph distance along the geometry. Due to the edit-completeness of meta grammar in Definition | the graph distance is an explainable measurement of minimal editing distance between molecular graphs, whereas the distance in latent spaces used in existing methods lacks explainability.
Geometric Deep Learning applies deep neural networks to non- -on-Euclidean domains such as graphs and manifolds with wide range of applications (Bronstein et al. 2017; Cao et al. 2020; Bron- stein et al., 2021). Related to our method, a series of recent works use graph neural diffusion by treating GNNs as a discretisation of underlying heat diffusion PDE (Chamberlain et al. 2021 Elhag et al. 2022 Bodnar et al., 2022). These methods work on large graph data where the graph connectivity of individual nodes provided. Additionally, there are also existing works on infer- ring the underlying geometry of graph data. Ganea et al. (0118) and Cruceru et al. (202T) embed graphs into non- -on-Euclidean manifolds such as hyperbolic and elliptical spaces and optimize their em- beddings within these Riemannian spaces. [Cosmo et al. (2020) learns a latent graph to model the underlying relationship between data samples and applies GNN to Different from these methods, our approach uses explicit and learnable intrinsic geometry based on graph grammar to model the relationship between molecular data. Since graph grammar is a generative model, our framework is capable of optimizing both molecular generation and property prediction simultaneously.
18

---

Under review as a conference paper at ICLR 2023
H MORE DETAILS OF HIERARCHICAL MOLECULAR GRAMMAR
Discussion on Hierarchical Molecular Grammar. The three additional attributes in Definition can be used to deduce several desirable properties of meta grammar. "Degree k" ensures the meta grammar is expressive and complete, covering all possible trees under simple tree degree con- straint. "Edit completeness" enables the explicit capture of transformations between two trees with edit distance one and therefore between two arbitrary trees with arbitrary distances. "Minimality" guarantees that the meta grammar is compact. Based these three attributes, we can construct generic meta grammar generating trees of non-terminal nodes by using only small but expressive set of production rules. Figure shows a 3-degree meta grammar. Each rule has one non-terminal node R2 on the LHS and two RR" s on the RHS. Depending on the number of anchor nodes on the LHH,, these rules can transform tree nodes of different degrees by attaching a new node using dif- ferent schemes indicated by the RHS. It is evident that we can generate any possible tree with a degree smaller than 4 by adopting a sequence rules from this 3-degree meta grammar. In Proposi- tion Mionl"Coeleetenee "Completeness" states that any molecular graph can be derived from a hierarchical molecular grammar with a set of appropriate molecular rules. This can be demonstrated by the fact that we can generate arbitrary molecule by first using the meta grammar to generate tree of non-terminal nodes (which has the same tree structure as a junction tree decomposed from the molecule), and then using the molecular grammar transform the tree into a molecular hypergraph by specifying a molecular fragment for each non-terminal node.
Meta Rules Construction. We refer to the production rule set of k-degree, edit-complete, and minimal meta grammar as a k-degree meta rule set. We visualize 4-degree meta rules in Figure (7a where {p1}. (P1. P2) and (p1, P2. P3. Pa.Ps) correspond meta rule sets of degree 3. respectively, 1-degree meta rule set p()) is constructed as
p() p(1) 1,
(L) (. - R I = ({X), 00), RAS = H(V
R.0 F
Pb
V/01 R20 = {Ri, R2), E = R5)
When using the meta grammar for production, we treat Ri. R2 as the same type of non-terminal node, i.e. R R2, i 1.2, despite the use of indices the above formulation.
For k- degree meta rule set p (k > 1), the construction is achieved by induction:
P (k) -plk-1) Up(k), p(k) -pp
(K) D
i=1
()) LHS - R* LHS = H(V ((M)) RHS(") = H(V E(L))
Po
n.0
P. LHS (k) + RM LHS = H(V!!) bl,)) RUS( = HVV (k)
R. ER = 1, 1
h
v200 (R') U
(Vanc.hh
E(A U [(RR, Vanas)),
1=1
k-1 (N) = (Ri R3) U (Vancj) Flyy ((Ri, Ri) U U
k-1 [(Vance.j Ri)),
R.0
j=1
j=1
k VL = (R') U
k
F(0) = U[(R', Vanes)),
j=1
5-1
k
V = (Ri, Ri} u (Vanes)) F/l. = ((Ri, Ri)) ([(Vanc))) Ri) U (Vann..j Ri)) j=it1
j=1
Specifically, the k-degree meta rule set pkk) contains all the rules from (k-1)-degree meta rule set pl(-1) as well as other newly introduced rules ((k), Each rule contains one non-terminal node R*
19

---

Under review as a conference paper at ICLR 2023
(R) has a degree of on the LHS and two non-terminal nodes Ri and R2 on the RHS. The LHS of Pb k-1 as it contains k-1 anchor nodes Vanc- The RHS of Po attaches one non-terminal node Ri to the other non-terminal node Ri which connects all the anchor nodes. Therefore, the RHS has degree of k which is larger than the degree of the LHS by one. In all the other rules, the LHS has k anchor nodes and is of degree k while the RHS keeps maximal degree of k. The anchor nodes on the RHS are combinatorially distributed to Ri and R2, i.e. if Ri is attached to i anchor nodes, R2 attached to the rest of k anchor nodes. Since we consider Ri and R. to be the same during grammar production, the range of {1..... 12 In total, there are 1+|2 k
rules in p(k)
Proof of Edit Completeness. According to the definition in Zhang 1996); Paaßen 2018), tree edit distance (T; T') between two trees 7 and T the minimum number of operations required to transform one tree into the other. There are three types of available operations: inserting, deleting, and relabeling. Consider two arbitrary trees T and T" that satisfy the definition of edit completeness (being derived from the meta grammar, TED T, T") = 1, and [T'| < [T). Since all the trees generated from the meta grammar have homogeneous tree node labels (i.e. R*). there is no rela- beling operation in TED calculation. Furthermore, the tree size relation between T and T' admits only one type of tree edit operation: a one-step inserting. Figure iblllusstates the one-step inserting operation. It operates on a certain node in T whose subtree is (v)) A new node v', which takes a subset of (v) as the children, is added as a new child of node v. The rest of Tana stays as the children of node v. Suppose the degree of node v in T is m and the degree related to subtree (V) is n. Without loss of generality, we suppose 0 < n S m/2. We can treat e in T as the RD on the LHS, v' and v in T" as Ri and Ri on the RHS, respectively. Then the one-step inserting operation can be achieved using the rule pl pi(m+1) p(m+1) from (m+1)-degred meta rule (k) that generates T Im +1) as constructed above. Since A(T") 2 m++1, the meta grammar PG set P
has k 2 m+1 So pn (m++1) E p(m+1) C P(()) thus finish the proof.
Proof of Degree k. For an arbitrary tree T with maximal
degree k we prove that meta grammar P(A) can generate T by finding a reverse path of production rule from the tree T to the initial node X. We first find all the nodes {u:)%= in T that have a degree of k. For each node, by edit completeness, serting and make the node to have a degree of k-1. It can be shown from the proof for edit completeness that each inverse operation deployment of production rule. By using n reverse pro- duction steps on all the k-degree nodes, we can obtain a T" with A(T') k-1. We continue the process until the tree is O-degree, i.c. the X: thus finish the proof.
P
X
Ost
pst
use the inverse operation of in-
pst
p
inserting corresponds to an inverse
p4:
pat
(a) dedgre meta rules.
Proof of Minimality. From edit completeness, we know each production rule in the meta grammar corresponds to least one case of one-step inserting operation. As con- strainted by degree k, arbitratry trees that are of degree smaller than i need to be covered by the meta grammar. We thus need to handle all the possible cases of one-step inserting operation when finding the reverse production path from the tree to X. Note that one-step inserting opeartion involves the division of a subtree, which has combinatorial ways. Then every combinatorial distribu- tion of anchor nodes corresponds to one case of subtree division. This proves each production rule in the meta rule set is necessary. Since there are no duplicate rules, it can be concluded that the meta grammar constructed above is minimal.
T
T"
(b) One-step inserting operation.
Figure 7: Illustration of inserting opera- tion and 4-degree meta grammar.
4-degree Meta Rules. Figure 7al shows the 4-degree meta grammar we use in approach. Since the edit completeness is the foundation to prove all the other attributes of the meta grammar, we use a tree edit distance toolki and verify its correctness for the 4-degree meta grammar.
http:://ppythnnbssree.orr//////
20

---

Under review as a conference paper at ICLR 2023
I MOLECULAR RULES CONSTRUCTION
Algorithm 1: Molecular Rules Construction.
Input : molecular hypergraph H (V, EH), probability function o(:;0)
Output: molecular rule set moo {p: [p:) junction tree T = (VT, ET) where each or E VT
is a subset of V
Initialize Pmot 2, VT = 2, Ey 2; Create a set for unvisited nodes Vann +- V. a set for unvisited hyperedges Eunv &- ERt
while Euny * Ø do
fx.--.xx],. E Eunu,i= i I,... K Bunel
X (ernoulli(@(e;0))
/+ Construct rule for each connected component
*/
for each Hub (Voubr Eaub) in ConnectedComoments (X, H) do
/+ Scparate the connected component
Voub + Veub. Esud Esub, Vone + Ø, Eane Ø;
for each r E Vsu do
V. GetNeighhors (v, H);
if Vn Vaud & then
if v Vunv InRing (u, H) then eub Voub \ {u];
Bzub + Erus's ((ss,))(sss ()) E Esut]:
Vano + Vano (vaner |Vanel + 1}:
Eane Eanc {(s, van))) v) E Esud. Vanc + 13:
else
Vano + Vanc (Vaner = |Vanel + A3]KknVoaull
17
Eane Eune {(s, (sn v) E Esud, Vanc + k=)==1
IVanVaull
/* Construct production rule
p= RuleConstruction (Vend, Eand Vanc Eanc); Pmol + Pmot Up);
19
20
/* Construct junction tree UT = (Voub): for each (1t E VTlu, nT Ø) do
21
22
Fr - ET U ((1,
24
VT + (
/+ update visited status nodes and hyperedges
Vanv +- Vunv Voubo Eumo &- Exno U Exubi
return Pmots T -Vr, ET);
To obtain the input for the molecular rule construction, first convert the molecule into molecular hypergraph H (V, E A node E represents an atom of the molecule. A hyperedge e Eμ corresponds to either bond that joins only nodes ring (including aromatic ones) that joins all nodes in the ring. An illustration is provided Figure 2 of Guo et al. (202)). The probability function db:; ()) is defined each hyperedge: ((c; a o(-Fo(((e)) where (.)) is the sigmoid function, Fo(-) is a two-layer fully connected network whose final output dimension is 1, and f(-) a feature extractor using a pre-trained GNN (Hu al. 2020
Algorithm illustrates the detailed process of constructing molecular rules for a single molecule. Note that in our approach, the molecular rule construction is performed simultaneously for all input molecules. For each molecule, we first perform an i.i.d. sampling on all the hyperedges following Bernoulli distribution which takes the value with probability indicated by Φ(e; 0). We then obtain binary vector X that indicates whether each hyperedge is sampled (line 4-5). Next, all connected components are extracted with respect to the sampled hyperedges (line 6). production
rule is constructed for each connected component Hsub (Vaud, Esus) (line 7-1)). Specifically,
production rule needs two components: hypergraph H (V.ut, Fesb) for RHS and anchor
21

---

Under review as a conference paper at ICLR 2023
nodes Vanc indicating the correspondence between the LHS and the RHS. Haub is obtained by removing visited nodes and in-ring nodes from Huub Vane contains nodes from that are connected to nodes from sub but do not appear in Huuh themselves. Bane the edges connecting anchor nodes to Hub following the same connectivity in the original graph H. Following Guo et al. 2022)) the function culeConstructiin (Veudr Erab. Vane, Bane) returns production rule LHS RHS constructed as
LHS H(VL, EL), VL U Vane. EL {(R', ( E Vanc):
(3)
RHS = H(R., ER), VR Vand U Vanc ER = Esub U Eane-
In contrast to Guo et al. (2022), our molecular rule construction does not replace the connected component with the non-terminal node R" every iteration since according to definition, the molecular rule does not contain any non-terminal nodes on the RHS.
The junction tree of the molecule is constructed along with the construction of molecular rules (line 21-24). We simply treat each connected component Vud sampled at each iteration as node UT E VT in the junction tree T (Vr, Er). Two nodes connected by edge er Er if their corresponding connected components share hypergraph nodes from H. Since each hyperedge in H only visited once, the constructed junction tree then contains all the hyperedges and nodes of H without redundancy (Kajino 2019).
J GRAMMAR-INDUGED GEOMETRY CONSTRUCTION
Algorithm 2: Grammar-induced Geometry Construction.
Input : meta production rules Pa = (p: } maximum BFS depth D, a set molecular M hypergraphs { { and their corresponding junction trees J = {
Output: geometry in the form of graph G (V, E) where each H, EV,, EV
represents a meta tree or a molecular hypergraph and & is the edge set of the geometry
/+ Add root of the geometry
Initialize Uroot = Hroot = (X', Ø), V = {Uroot J. E Ø;
Create queue data structure Q:
Qeequeue (Mroot):
/+ Breacth-first search for meta gecmetry, pre-computed offline
while Q is not empty and | < do
v = dod.dqqueee/ ():
/+ Expand the meta geometry
for each Ps E Pa do
if pt is applicable to H. then H H Hnewi
Unew Hnew;
if Unew V then
11
V+ V-VUR [unem). ((v, (new));
Q. equeue (Unew);
else
i Get rotoo (H V),E E ((u, i)]:
/* Connect molecular loaves during run-time
15 for each T, E J do for each v do
16
17
if Ts tosTooppppicc (H,, T,) then
0; = Hü; V+ Vu & & {(v,a)))
return G (V,E);
Algorithm illustrates the detailed algorithm to construct the grammar-induced geometry. It con- tains two parts: the construction of the meta geometry (line 4-14) and the construction of the molec- ular leaves (line 15-19). The meta geometry construction follows the standard breadth-fire search
22

---

Under review as a conference paper at ICLR 2023
(BFS) starting from the root Hoot (X, Ø) (line 1). Every time when visit node V in g. we find all the rules that are applicable to He from the meta rule set (line 6-7). Each applicable rule is applied to H in order to create
meta tree Hnew (line 8). Depending a node in V that represents a isomorphic tree to Hnew in the current geometry, we either create a new node Unew (line 9) find the existing matched node i (line 14). We then add edge between and Unews or between and è (line 11 and 14). The function Get I weeeeeerppirraapppee
whether there
(eew V) enumerates every node in V. checks if the tree represented by the node is isomorphic new and returns the matched node. We use the algorithm from Cordella et al. (2001) which is implemented in networks P for graph isomorphism test. To increase the speed of Get GstIsomorphn wwwoooorhiieeeo.o crraph news V), we use Weisfeiler Lehman graph hash (Shervashidze et al. 2011) and only perform isomor- phism test for graph pairs that share the same hashing code.
The molecular leaves are added during run-time (line 15-19). For each input molecular hypergraph, we check if there is meta tree node that isomorphic to its junction tree. so, we add an edge con- necting the matched meta tree node to the molecular hypergraph. The function IsIsomorphic (Hl,, TD is implemented using the same package for Iet www.:cmmmppirrraappp...o (Hnews V) and returns true if He and T, are isomorphic to each other.
K OPTIMIZATION
Figure 8 shows graphical model of depen- dency in our optimization problem. The two sets of parameters and (ye, vh, a:) are indepen- dent and thus can be iteratively optimized using block coordinate descent. Since the objective in Equation 2 is differentiable with respect to (vo, v/, a.) we can use gradient descent for the optimization iterations related to the graph neu- ral diffusion. The geometry learning, however, non-differentiable due to the fact that we perform sampling to construct molecular rules and construct the geometry. Hence, rewrite the objective for optimizing an expectation form and apply REINFORCE (Williams) 1992) to obtain a stochastic gradient, as done in Guo et al. 2002):
UT
1(.-))
0
I
Figure 8: A graphical model of dependency.
min (uTT û) mjjn Ex [(urrüū]
VøEx [(ur, i)}
{ -x
l(ur, úi (X)
Ex [(uT, ú) log(o(x))
1 Σ ((ff)) (n) V, log(p(X("))),
where X is concatenation of binary vectors indicating how hyperedges sampled in molecular rule construction in Appendix[[
L EXAMPLES OF RETRO-S RETRO-SYNTHEE SYNTHESIS PATHS
Figure shows the retro-synthesis paths of three molecules generated using our pipeline. Since Retro" score used one of the grammar metrics, approach is capable of retro-synthesis plan- ning for all the generated molecules, providing a complete pipeline for novel molecule discovery.
Isomorphic sm. html
23

---

Under review as a conference paper at ICLR 2023
OH
HD
Figure 9: Examples retro-synthesis paths.
24


Reviews:
[{"review_id": "q-0coIOk6V-", "reviewer": "Reviewer_CjiQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an efficient hierarchical molecular grammar learning algorithm for molecular property prediction. The similar property of same production rule motivates the study of data-efficient grammar-induced geometry. Specifically, the authors describes the process of building hierarchical molecular grammar by pre-defined meta grammar and learnable molecular grammar. Results on both small and large polymer datasets demonstrates the effectiveness of the method.", "summary_of_the_review": "This paper discusses an efficient training framework for molecular property prediction via grammar-induced geometry. However, I found the motivation and design of the paper is not well supported in the main body of the paper. ", "strength_and_weaknesses": "Strength:\n1. The molecular grammar hyper-graph construction is innovative.\n2. The illustration of grammar-induced geometry is clear and interesting. \n\nWeakness:\n1. This paper seems to focus more on grammar tree construction but the detail of how to integrate the prediction model (GRAND) in the paper and induced grammar is not very clear. For example, what if input polymer is not on the induced grammar tree, how do you guarantee every input can be represented on graph.\n2. The motivation of using GRAND is not clear. I am assuming it's either because it's performing better or fit the grammar tree. Because the diffusion PDE seems has nothing to do with the proposed framework.\n3. The experimental results are not very comprehensive. Admittedly, it shows great performance on two datasets in the paper, however, datasets like ZINC are more frequently used in the literature. Is the proposed method only used for polymers?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "p4urVZwpD-6", "reviewer": "Reviewer_Ahkk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a data-efficient property predictor by utilizing a learnable hierarchical molecular grammar that can generate molecules from grammar production rules. The grammar induces an explicit geometry describing the space of molecular graphs, such that a graph neural diffusion on the geometry can be used to effectively predict property values of molecules on small training datasets. On both small and large datasets, the evaluation shows that this approach outperforms a wide spectrum of baselines, including supervised and pre-trained graph neural networks.", "summary_of_the_review": "This paper proposes a framework for highly data-efficient property prediction based on a learnable molecular grammar. The intuition behind is that the production rule sequences for molecule generation provide rich information regarding the similarity of molecular structures. The proposed method combines previous existing methods [1, 2] and the novelty is incremental. The major concern is the weak evaluation of the proposed method. I would suggest the authors to include more experimental evaluation.", "strength_and_weaknesses": "Strength:\n\n- The proposed method represents grammar production sequences and captures the structure-level similarity between molecules. With a few molecules, this method can utilize the structural relationship between molecules to predict molecular property. It is especially effective for small dataset.\n\n- This paper improves previous molecular hypergraph grammar method [1] by integrating junction tree representation of molecules [2], and proposes a hierarchical molecular grammar to address the computational challenge in construction of grammar-induced geometry. \n\nWeaknesses:\n\n- The proposed method is limited to small dataset scenarios. While there are many pretraining /transfer learning methods work well on both small and large datasets. \n\n- Evaluation is weak. In table 1, the proposed method is only validated on two datasets, compared with limited baseline methods. DILI is a representative small dataset and the authors should at least include experiments on DILI datasets to show the effectiveness of the proposed method, which is claimed to be effective on small datasets. Please refer to [3] for experiments on DILI datasets. Pre-trained GIN is neither the SOTA methods. Please the authors include more high-performance baseline methods. Experiments on large datasets in Table 2 only included two datasets, where there are tens of datasets for evaluation.\n\n- Novelty is limited. The proposed method is an incremental improvement based on previous methods [1, 2].\n \n\n[1] Guo, Minghao, et al. \"Data-efficient graph grammar learning for molecular generation.\"ICLR (2022.\\\n[2] Jin, Wengong, Regina Barzilay, and Tommi Jaakkola. \"Junction tree variational autoencoder for molecular graph generation.\" ICML 2018.\\\n[3] Ma, Hehuan, et al. \"Deep graph learning with property augmentation for predicting drug-induced liver injury.\" Chemical Research in Toxicology 34.2 (2020): 495-506.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Z2FwrpHzpU", "reviewer": "Reviewer_aHZ7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of molecular property prediction, particularly focusing on the limited availability of labelled data in many domains. It notes the limitations of existing methods targeted towards this scenario, and presents a molecular hypergraph grammar to describe molecules, and this grammar is used to induce a geometry over molecular space to enable effective molecular similarity measurement. To combat the computational challenge of enumerating a full molecular grammar-induced geometry the authors propose a hierarchical grammar consisting of a junction tree meta-geometry and a lower assignment of tree nodes to true molecular graphs. The property prediction then uses a graph neural diffusion model operating on all nodes in the grammar-induced geometry. The performance of this method is evaluated on two small datasets of polymers, and two slightly larger small molecule property prediction datasets. ", "summary_of_the_review": "The paper makes a good contribution with a novel method for molecular property prediction that uses a hierarchical grammar to induce a geometry, then using this to make property predictions. The idea of using such a grammar in a property prediction setting is novel, and the additional hierarchical component is somewhat novel but bears some similarities to previous work on molecular substructures such as JT-VAE, hierVAE, and MoLER. In addition, the molecular grammar itself is very similar to the work of Guo et al. 2022. Therefore, much of the additional contribution comes from the application of these ideas in property prediction rather than molecule generation. This is a nice potential use of such a framework, and indeed could result in better cross-domain adaptation, but it was not clear the authors fully explored or supported claims made in the paper around this. In addition, the limitation of property prediction tasks to regression feels somewhat odd -- while regression is indeed more challenging, not addressing the many unsolved classification problems available in datasets such as MoleculeNet and FS-Mol prevents a strong comparison between this work and previous works being made. \n\nOverall, the ideas in this paper, while clearly strongly inspired by other works, are good. However, the paper in its current form is marginal because it would benefit from more time spent in experimentation and analysis to more strongly support the claims made. While the ideas are good it is not clear that the method is generally performing well, and whether it is more broadly applicable. If the method is truly very effective, and this can be demonstrated to be so on a wider range of commonly benchmarked tasks, then this would be distinctly impactful to the field. However, as it stands such conclusions are difficult to reach on the basis of the empirical exploration presented here. \n\n====update in response to authors comments====\nFollowing the response and revisions of the authors, in particular the inclusion of additional evaluation, a discussion of the computational complexity, and more clarification of the baselines, this review is being updated to reflect this. ", "strength_and_weaknesses": "Strengths: \n* This is a nice extension of molecular grammars to include a more computationally tractable method by inclusion of a hierarchical component.  \n* The use of graph neural diffusion to make use of the grammar-induced geometry for molecular property prediction is a good idea -- one would indeed hope that a representation using explicit molecular substructures would ultimately be the best for molecular comparison, and it is helpful that the authors explore this.\n* The description of the model is clear. \n* The level of detail given in the appendix regarding the molecular rules and geometry construction is helpful. \n\nWeaknesses:\n* The lack of reference to related work -- the section in the appendix only addresses very general molecular property prediction and not molecular grammars, which in the context of a number of other works approaching similar ideas seems lacking. Concretely, such a section would be highly valuable. In addition, while Guo et al. 2022 is referenced, it does not seem that the strong similarities are acknowledged. Is it possible to compare the performance of this model to one using a non-hierarchical grammar (i.e. does this particular hierarchical grammar improve the molecular property prediction)? In addition, reference is made to the superior computational performance; is it possible to quantify this further? Such comparisons would be very useful. Furthermore, there have been previous works taking a hierarchical approach to the decomposition of molecular structure, such as JT-VAE, and hierVAE. While these works are in molecular generation, the underlying principles bear strong similarity. Extension of the related work section (and possible inclusion in the main body of the paper) would be helpful. \n* The authors strictly address only regression. While this is certainly a more challenging task in general, it does a) make direct comparison to other methods more challenging b) limit the range of datasets on which the method can be evaluated. The small molecule datasets in use appear to be the more challenging regression ones in Molecule Net (i.e. excluding quantum mechanics). As the authors note in the appendix, most related work in molecular property prediction is in classification, so it would be helpful to be able to draw some direct comparison to the state-of-the-art methods there.\n* While the authors note that molecular grammars should improve the ability for a model to generalise outside of the domain, and give this as a reason for improved performance of their method over pre-trained GIN and PN, as these were trained on lower average molecular weight datasets than Permeability, they do not justify this with their results. In each of the CROW and Permeability cases their method was trained on the dataset at hand, and so does not have to demonstrate cross-domain adaptation. While GIN and PN are fine-tuned on the datasets, they come with strong intrinsic bias to their original pretraining datasets, and were pretrained for classification in addition. As both were pretrained on a set of small drug-molecule protein-binding data, it is actually remarkable they perform so well on a polymer dataset, where the relevant substructures are likely quite different. A concrete improvement would be demonstration of cross-domain transfer of the model.  \n* Overall, the empirical evaluation and discussion felt rushed, the paper would benefit from more expansion around the results. Given the explicit reference to four key questions in the evaluation section, it would be helpful to have these more clearly addressed in the main body of the text, and perhaps slightly less time in description of the model. \n* Section 4.4 \"Combination with generative models\" is not a strong comparison to the other work in molecular generation, it is not clear how this particular method really compares in performance to, for instance, Guo et al 2022. It is also generally unclear whether the method has made improvements, or this is simply demonstrating that this kind of generation can be done, which was already answered by previous works. \n* Figure 4(b) addresses the effect of changing dataset training size on model performance. \n  * This kind of plot has appeared with other datasets such as FS-Mol. Given that this exploration is very useful, it would be helpful if the authors included analysis of performance on other datasets. \n  * A straight line represents the performance of a Pretrained-GIN, which is therefore presumably not finetuned on training set data. This is an unfair comparison. The Pretrained-GIN should be finetuned on the same data that the author's model has access to, and those results reported, in line with the typical few-shot literature eg. Hu et al. 2020, Stanley et al. 2021. \n  * It would be helpful to see the performance of a few more of the benchmarked models as the training set size is changed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4qQ89CFCon", "reviewer": "Reviewer_EFAZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose the use of a learnable hierarchical molecular grammar that can be used for property prediction tasks in the context of drug/materials discovery. This is done by extending the work of Guo et al. (2022) on molecular grammars and adapting it for the supervised learning setting via the usage of a neural diffusion model.", "summary_of_the_review": "Overall, this is a good paper with a solid theoretical background that deserves credit. I mostly take issue with the (rather limited) result section, which I believe could use further work in terms of tested datasets. \n\n* The authors could consider testing their method on other well-known benchmarking molecular datasets, as available in the MoleculeNet sets to further improve their claims - or elaborate why they chose to test only on the CROW and Permeability datasets.\n* The performance of the GEO-DEG approach seems not to be as strong compared to simpler approaches such as D-MPNN when trained on larger datasets (Table 2, Appendix) - to the degree that in the Lipophilicity dataset results are within 1 std. Could the authors elaborate on why they believe this happens?\n* A point that seems to be somewhat absent in the study is the computational complexity required to train and evaluate the proposed learned grammar approach and graph diffusion model on new data. Could the authors provide some extra analyses so that the reader gets an idea on how expensive this methodology is compared to other baseline approaches (e.g. D-MPNN)\n* In the process for geometry construction, the authors claim that they find it sufficient to use a maximum depth of 10. Could they elaborate  on how that decision was reached?\n* The authors motivate hierarchical molecular graphs by claiming that the construction of grammar-induced geometries is costly. In fact, they mention that they find it infeasable to construct grammars when one considers more than ten production rules. Have the authors tried and compare their results with grammars constructed with only a few production rules?\n* For PN and the Pre-trained GIN models, how was the pretraining done? Were these trained on additional polymer data or were they pretrained on other molecular species? Additionally on Figure 4b, it is not entirely clear how the Pre-trained GIN baseline was computed. Did the authors use the remaining 80% of the CROW dataset and finetuned the model there? How was this finetuning performed?\n\n\nOther points:\n* What features were used for the training of the random forest baselines?\n* Figure 4b could ideally provide additional error bars", "strength_and_weaknesses": "Strengths:\n* Natural extension of the molecular grammar work proposed by Guo et al. (2022) for the supervised learning case\n* The methodology is well motivated and described\n* Good level of detail provided in the appendix\n\n\nWeaknesses:\n* Rushed experiment section. Specifically the authors limit themselves to a couple of relatively small datasets when they could test on more widely used benchmarks in molecular ML to demonstrate the superiority of their approach (see summary of the review).\n* Computational complexity not clear.\n* No code provided to reproduce results / test the existing approach", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject