Learning 3D Representations of Molecular Chirality with Invariance to Bond Rotations
Authors: Keir Adams
Lagnajit Pattanaik
Connor W. Coley
Venue: ICLR 2022 Poster (2022)
Keywords: geometric deep learning
equivariance
molecules

Abstract: Molecular chirality, a form of stereochemistry most often describing relative spatial arrangements of bonded neighbors around tetrahedral carbon centers, influences the set of 3D conformers accessible to the molecule without changing its 2D graph connectivity. Chirality can strongly alter (bio)chemical interactions, particularly protein-drug binding. Most 2D graph neural networks (GNNs) designed for molecular property prediction at best use atomic labels to naïvely treat chirality, while E(3)-invariant 3D GNNs are invariant to chirality altogether. To enable representation learning on molecules with defined stereochemistry, we design an SE(3)-invariant model that processes torsion angles of a 3D molecular conformer. We explicitly model conformational flexibility by integrating a novel type of invariance to rotations about internal molecular bonds into the architecture, mitigating the need for multi-conformer data augmentation. We test our model on four benchmarks: contrastive learning to distinguish conformers of different stereoisomers in a learned latent space, classification of chiral centers as R/S, prediction of how enantiomers rotate circularly polarized light, and ranking enantiomers by their docking scores in an enantiosensitive protein pocket. We compare our model, Chiral InterRoto-Invariant Neural Network (ChIRo), with 2D and 3D GNNs to demonstrate that our model achieves state of the art performance when learning chiral-sensitive functions from molecular structures.

Published as a conference paper at ICLR 2022
LEARNING 3D REPRESENTATIONS OF MOLECULAR CHIRALITY WITH INVARIANCE TO BOND ROTATIONS
Keir Adams' LLgnnjjt Pattanaik', Connor W. Coley 'Department of Chemical Engincering
DDpaarment of Electrical Engineering and Computer Science Massachusetts Institute of Technology, Cambridge, MA 02139, USA kkeir, lagnajit, ccoley 0mit @0itt.du
ABSTRACT
Molecular chirality, a form of stereochemistry most often describing relative spa- tial arrangements of bonded neighbors around tetrahedral carbon centers, influ- ences the set of 3D conformers accessible to the molecule without changing its 2D graph connectivity. Chirality can strongly alter (bio)chemical interactions, particularly protein-drug binding. Most 2D graph neural networks (GNNs) de- signed for molecular property prediction at best use atomic labels to naively treat chirality, while B(3)-innariant 3D GNNs are invariant to chirality altogether. To enable representation learning on molecules with defined stereachemistry, we de- sign an SE(3)-invariant model that processes torsion angles of a 3D molecular conformer. We explicitly model conformational flexibility by integrating a novel type of invariance to rotations about internal molecular bonds into the architec- ture, mitigating the need for multi-conformer data augmentation. We test our model on four benchmarks: contrastive learning to distinguish conformers of dif- ferent stereoisomers in learned latent space, classification of chiral centers R/S, prediction of how enantiomers rotate circularly polarized light, and ranking enantiomers by their docking scores in enantiosensitive protein pocket. We compare our model, Chiral InterRoto-Invariant Neural Network (ChIRo), with 2D and 3D GNNs demonstrate that our model achieves state of the art performance when learning bbiral-sensitive functions from molecular structures.
INTRODUCTION
Advances in graph neural networks (GNNs) have revolutionized molecular representation learning for (bio)chemical applications such as high-fidelity property prediction (Huang et al. [021 Chuang
al. 2020), accelerated conformer generation (Ganea et al. 202T Mansimov et al., (009)] Simm
& Hewnanddez 2020 Xu al. 2021: Pattanaik al. 2020b), and molecular optimization
(Elton et al. 2019 Brown et al. 2019). Fueling recent developments have been efforts to model shape-dependent physio-chemical properties by learning directly from molecular conformers (snap- shots of 3D molecular structures) or from 4D conformer ensembles, which better capture molecular flexibility. For instance, recent state-of-the-art (SOTA) GNNs feature message updates informed by bond distances, bond angles, and torsion angles of the conformer (Schütt et al. [2117] Klicpera et al.) 20206 Liu al. 202 Klicpera al. 202T). However, few studies have considered the expressiv- ity of GNNs when tasked with learning the nuanced effects of stereochemistry, which describes how the relative arrangement of atoms space differ for molecules with equivalent graph connectivity.
Tetrahedral (point) chirality is prevalent form of stereochemistry, and describes the spatial ar-
rangement of chemical substituents around the vertices of tetrahedron centered on a chiral center, typically carbon atom with four non-equivalent bonded neighbors. Two molecules differing only in the relative arrangements around their chiral centers are called stereoisomers, or enantiomers they can be interconverted through reflection across plane. Enantiomers are distinguished in chem- ical line drawings by a dashed or bold wedge indicating whether a bond to a chiral center is directed into out of the page (Figure [A). Although enantiomers share many properties such boiling points, electronic energies, and solubility in most solvents, enantiomers can display strikingly dif- ferent behavior when interacting with external chiral environments. Notably, chirality is critical for

---

Published as a conference paper at ICLR 2022
A
Enantiomers
images]
Conformers
Conformers
B
Achiral Graph
E(3)-invariance
SE(3)-nvarrance
(SE(3) InterRoto--Invarinnee
Separates
X
X
V
Enantiomers?
X
X
Invariant?
Schematic of how different conformers of enantiomers (panel A) separate in learned latent space (panel B) under 2D (achiral) graph model, E(3)- and SE(3)-invariant 3D models, and ChIRo, which invariant rotations of internal molecular bonds cnteeRoto-invarianeee
Figure 1:
drug design Nguyeen et al. 2006; Jamali et al. 1989), where protein- ligand interactions are influ- enced by ligand chirality, well for designing structure-directing agents for zcolite growth (Luis & Beatriz 2018) and for optimizing asymmetric catalysts (Pialtz & Drury 2004 Liao et al. 2018).
Chiral centers are inverted upon reflection through a mirror plane. Consequently, B(3)-invariant 3D GNNs that only consider pairwise atomic distances or bond angles in their message updates, such SchNet (Schtitt et al.) 2017) and DimeNe//DimeNet++ (Klicpera al. 2020a b), inherently limited in their ability to distinguish enantiomers (Figure BB). Although SE(3)-invariant 3D GNNs, such the recently proposed SphereNet (Liu al. 202 and GemNet (Klicpera et al. 202T). can
in theory learn chirality, their expressivity in this setting has not been explored.
Alongside the development of 3D GNNs, which process individual 3D conformers, there have been efforts to better represent conformational flexibility by encoding multiple conformers in 4D en- semble for property prediction (Zankov al., 2021 2019 Axelrod &
202
identifying important conformer poses (Chuang & Keiser 2020). and predicting solvation energies (Weinreich et al. 2021). Unless in the solid state, molecules are not rigid objects or static 2D graphs, but are flexible structures that rapidly interconvert through rotations of chemical bonds as well as through smaller perturbations such as bond stretching, bending, and wagging. Explicitly modeling this probability distribution cling of protein-drug interactions, where the most relevant active pose of the ligand is not known which depend
accessible conformer space has the potential to improve the mod-
priori, as well in the prediction of Boltzmann-average formers. One challenge with these methods selecting which conformers to include an ensemble: the space of accessible conformations combinatorially explodes with the number of rotatable bonds, and important poses are not known priori. Modeling flexibility with multi-instance methods thus requires explicit conformer enumeration, increasing the cost of training/inference without guaran- teeing performance gains. Apart from 2D methods, which ignore 3D information altogether, no studies have explicitly modeled conformational flexibility directly within a model architecture.
distribution of con-
To explicitly model tetrahedral chirality and conformational flexibility, we design a neural frame- work to augment 2D GNNs with processing of the SE(3)-invariant internal coordinates of con- former, namely bond distances, bond angles, and torsion angles. Our specific contributions are:
We design method for graph neural networks to learn the relative orientations sub- stituents around tetrahedral chiral centers directly from 3D torsion angles
We introduce novel invariance to internal rotations of rotatable bonds directly into a model architecture, potentially mitigating the need for 4D ensemble methods conformer-bas data augmentation to treat conformational flexibility of molecules
We propose a contrastive learning framework to probe the ability of SE(3)-invariant 3D graph neural networks to differentiate stereoisomers in learned latent space
2

---

Published as a conference paper at ICLR 2022
Through our ablation study, demonstrate that global node aggregation scheme, adapted from |Winter et al. (2021), which exploits subgraphs based on internal coordinate connectivity can provide simple way improve GNNs for chiral property prediction
We explore multiple tasks to benchmark the ability of our model to learn the effects of chirality. We do not consider common MoleculeNet (Wu et al. 2018)) benchmarks, as focus is tasks where the effects of chirality are more distinguishable from experimental noise. Our self-supervised contrastive learning task is the first of its kind applied clustering multiple 3D conformers of dif- ferent stereoisomers in a latent space. Following Pattanaik et al. (2020a). we also employ a toy R/S labeling task as necessary but not sufficient test of chiral recognition. For harder classification task, we follow Maaede et al. (202T) in predicting how enantiomers experimentally rotate circularly polarized light. Lastly, we create a dataset of simulated docking scores to rank small enantiomeric ligands by their binding affinities in a chirality-sensitiv protein pocket. We make our datasets for the contrastive learning, R/S classification, and docking tasks available to the public. Comparisons with 2D baselines and the SF(3)-invarial SphereNet demonstrate that our model, Chiral Inter Roto- Invariant Neural Network (ChIRo), achieves SOTA in 3D chiral molecular representation learning.
RELATED WORK
Message passing neural networks. Gilmer al. (2017) introduced a framework for using GNNs to embed molecules into continuous latent space for property prediction. In the typical 2D message passing scheme, a molecule is modeled as a discrete 2D graph G' with atoms as nodes and bonds as edges. Nodes and edges are initialized with features and Ciy to embed initial node states:
(1) where Ni) denotes the neighboring atoms of node i. In each layer t of message passing, node states are updated with aggregated messages from neighboring nodes. After T layers, graph feature vectors are constructed from some (potentially learnable) aggregation over the learned node states. readout phase then uses this graph embedding for downstream property prediction:
h Ux( {eg)))N())
h b+ Uc(h), +
m m++1_
Σ JEN(S)
(,/ e.y)
ÿ = Readout(g),
Agg( (b)
(3)
There exist many variations on this basic message passing framework (Duvenaud et al., 2015 Kearnes et al., 2016). In particular, Yang et al. (2019)'s 's directed message passing neural network (DMPNN, or ChemProp) based on Dai al. (2016) learns edge-based messages m and updates edge embeddings hy rather than node embeddings. The Graph Attention Network (Veličković et al.,
2018 constructs message updates m using attention pooling over local node states.
3D Message Passing and Euclidean Invariances. 3D GNNs differ in their message passing schemes by using molecular internal coordinates (distances, angles, torsions) to pass geometry- informed messages between nodes. It is important use 3D information that is least SE(3)- invariant, as molecular properties are invariant to global rotations or translations of the conformer.
SchNet (Schütt et al., 2017). a well-established network for learning quantum mechanical properties 3D conformers, updates node states using messages informed by radial basis function expansions of interatomic distances between neighboring nodes. DimeNet (Klicpera et al. 20206) and its newer variant DimeNet++ (Klicpera et al. 2020a) exploit additional molecular geometry by using spherical Bessel functions to embed angles Piyk between the edges formed between nodes and j and nodes j and ls * i in the directed message updates to node i.
SchNet and DimeNet are E(3)-invariant, as pairwise distances and angles formed between two edges are unchanged upon global rotations, translations, and reflections. Since enantiomers are mirror images, SchNet and DimeNet are therefore invariant to this form chirality. To be SE(3)-invariant, 3D GNNs must consider torsion angles, denoted viizyjj between the planes defined by angles diay and Φzyj, where i, JX. U. are four sequential nodes along a simple path. Torsion angles are negated upon reflection, and thus models considering all torsion angles should be implicitly sensitive to chirality. Shem-Shepherd et al.] (202I). [iu et al. (202I) (SphereNet). and Klicpera et al. (202T) (GemNet) introduce 3D GNNs that all embed torsions in their message updates.
3

---

Published as a conference paper at ICLR 2022
Using a complete set of torsion angles provides access to the full geometric information present in the conformer but does not guarantee expressivity when learning chiral-dependent functions. Torsions are negated upon reflection, but any given torsion can also be changed via simple rotations of a rotatable bond-which changes the conformation, but not the molecular identity (i.e., chirality does not change). Reflecting a non-chiral conformer will also negate its torsions, but the reflected conformer can be reverted to its original structure via rotations about internal bonds. To model chirality, neural networks must learn how coupled torsions, the set of torsions {Vizys (s.j) that share a bond between nodes r and y (with I or y being chiral), collectively differ between enantiomers.
E(3)- and SE(3)-Equivariant Neural Networks. Recent work has introduced equivariant layers into graph neural network architectures to explicitly model how global rotations, translations, and (in some cases) reflections of a 3D structure transform tensor properties, such as molecular dipoles or force vectors. -E(3)-qquivariant models (Fuchs et al. 2020; Thomas et al. 2018)) should be sensitive to chirality, while E(3)-equivariant models (Satorras et al., (021) will only be sensitive the output layer is not E(3)-invariant. Since we use SEv3)-invariant internal coordinates as our 3D representation, we only compare our model to other SE(3)- or E(3)-invariant 3D GNNs.
Explicit representations of chirality in machine learning models. A number of machine learning studies account for chirality through hand-crafiee molecular descriptors (Schneider et al., 2018
Golbraikh et al.) [001]] Kovatcheva et al., 2007; Valdés -Martii al. Mamede et al. 1021
A naïve but common method for making 2D GNNs sensitive to chirality is through the inclusion of
chiral tags as node features. Local chiral tags describe the orientation of substituents around chiral centers (CW or CCW) given an ordered list of neighbors. Global chiral tags use the Cahn-Ingold- Prelog (CIP) rules for labeling the handedness of chiral centers as R ("rectus") or S ("sinister"), is unclear whether (and how) models can suitably learn chiral-dependent functions when exposed to these tags as the only indication of chirality. Pattanaik et al. (2020a) propose changing the symmetric message aggregation function in 2D GNNs (sum/max/meent to an asymmetric function tailored to tetrahedral chiral centers, but this method does not learn chirality from 3D molecular geometries.
Chirality in 2D Vision and 3D Pose Estimation. Outside of molecular chirality, there has been work in the deep learning community to develop neural methods that learn chiral representations for 2D image recognition (Lin et al., 2020) and 3D human pose estimation (Yeh et al., 2019). In particular, Yeh et al. 2019) consider integrating equivariance to chiral transforms directly into neural architectures including feed forward layers, LSTMs/GRUs, and convolutional layers.
CHIRAL INTERROTO-IVVARAWT NEURAL NETWORK (CHIRO)
Our model uses a 2D GNN to embed node states based on graph connectivity, two feed-forward networks based on Winter et al. (202T) to encode 3D bond distances and bond angles, a specially de- signed torsion encoder inspired by Ganea et al. (2021) to explicitly encode chirality with invariance and readout phase for property prediction. Figure 2 visualizes how we encode torsion angles to achieve
to internal bond rotations (InterRoto-invariancee
ceteRRotoinvvaiiaa representation of chi- rality. The model is visualized in appendix [A.4 We implement our network with Pytorch Geometric
Fey & Lenssen
2019). Our code is available at /etttps://qithbte com/ kei raamams/ ChhRRo
2D Graph Encoder. Given a molecular conformer, we initialize a 2D graph G where nodes are atoms initialized with features x; and edges are bonds initialized with features eij- We treat all hydrogens implicitly, use undirected edges, and omit chiral atom tags and bond stereochemistry tags. x; include the atomic mass and one-hot encodings of atom type, formal charge, degree, number of hydrogens, and hybridization state, e,j include one-hot encodings of the bond type, whether the bond is conjugated, and whether the bond is in ring system. In select experiments, we include global and local chiral atom tags and bond stereochemistry tags for performance comparisons.
Any 2D GNN suffices to embed node states. Following Winter et al. (202I), ) we use (Simonossk & Komodakis 2017)> 's edge convolution (FConv) to embed the node features with the edge features:
h +x +
Σ Sa(e,)
JEN())
where θ is a learnable weight matrix and fe is multi-layer perceptron (MLP). Node states h th are then updated through T sequential Graph Attention Layers (GAT) (Veličković et al. 2018
h = GAT( (h. (hh ((-")))N())) t = T,...T
4

---

Published as a conference paper at ICLR 2022
OL (or (o-0----0 tw tu
R
C
(9)
Σ F0000 vil
C
stat8 (6) vil)
Figure Geometric visualization of how ChiRo learns chiral representations with InterRoto- invariance. Given two enantiomers with three coupled torsions tha Wh. and te involving chiral carbon center, rotating the shared carbon-carbon bond by E [0, 2T) rotates each torsion together, forming three periodic sinusoids (solid RGB curves). Learning phase shifts yp for each torsion produces shifted waves (dashed RGB curves) that differ between enantiomers. the shifted sinusoids results in different degrees of wave interference between enantiomers, yielding different amplitudes of the net waves. These amplitudes, visualized here as the different radii (r) of the circles formed by plotting the summed sines against the summed cosines, invariant the rotation of the internal carbon-carbon bond. See appendices A.I A.2 for further details.
weighted sum of
Bond Distance and Bond Angle Encoders. To embed 3D bond distance and bond angle informa- tion, we follow Winter et al. (202I) b by individually encoding each bond distance di (in Angstroms) and angle Digk into learned latent vectors. We then sum-pool these vectors get conformer-level latent embeddings Zd E I²= and Z E R=:
Zd Σ fa(b,, dey) + fa(h,. h dig)
(6)
(UJ)
25
Σ (K..E)EG
So hy, ha, cos digk, sin duyx)) + (hg, hj, h;, cos Oiyke sin difk)
cos
where fa and fe are MLPs and (...) denotes concatenation. We maintain invariance node ordering by encoding both (i,j) and (j,i) for each distance dij. and both (i, k) and (k,j, i) for each angle diyk- For experiments that mask all internal coordinates (i.e., set all bond lengths and angles to O), 2g and z represent aggregations of 2D node states based on subgraphs of local atomic connectivity.
Torsion Encoder. We specially encode torsion angles to achieve two desired properties: 1) an in- variance rotations about internal molecular bonds, and 2) the ability to learn molecular chirality. Both of these properties depend critically on how the model treats coupled torsion angles-torsions that cannot be independently rotated in the molecular conformer. For instance, torsions daa t/b, and de in Figure 2 are coupled, as rotating the internal carbon-carbon bond rotates each torsion simulta- neously. Symmetrically aggregating every redundant torsion does not respect this inherent coupling- However, encoding a set non-redundant torsions-a minimal set that completely define the struc- ture of the conformer-would make the model sensitive to which arbitrary subset was selected.
Ganea et al. 1 provide partial solution to this problem in their molecular conformer generator, which we add to below. Rather than encoding each torsion individually, they compute a weighted sum of the cosines and sines of each coupled torsion (which are redundant), where neural network predicts the weights"] Formally, given non-terminal (internal) bond between nodes z and y that yields set of coupled torsions (Vizyj (1.1) for i E N(x) {8)
and j E N N \ {z), they compute:
(aLex)
(197)
(1y) iEN()) (EN(
Σ
Σ
(oos()/agg)) sin sin(Wazy))
Ganca et al. 2021 untrained network with randomized parameters to predict the weighting coeffi cients. Instead, learn the parameters in fo and compare these approaches Appendix A.8
5

---

Published as a conference paper at ICLR 2022
(ry) 3g
= -o(((h hz bys hy) + (hú hy ba h.))
So
(zv) bounded to where Se is an MLP that maps to TR. The sigmoid activation o, which keeps each Gj (0, 1), is our addition. The following formulation is our novel addition to Ganea et al. (2021).
Because rotating the bond (x,y) rotates the torsions {Manys together, the sinusoids (sin(Wiews) (i.j) and (cos((rays))(s... have the same frequency. Therefore, the weighted sums of these cosines and sines are also sinusoids, which when plotted against each other as a function of rotating the bond (z, y), form a perfect circle (appendix [A.I). Critically, the radius of this circle, ||ato)) (22) (mn)
We therefore encode these radii in order to achieve invariance to how any bond in the conformer Is rotated.
ai ||. is invariant to the rotation of the bond (æ, y) (Figure 2).
The above formulation, as presented, is also invariant to chirality (appendix A.2 symmetry, we add learned phase shift, Piayjj to each torsion Wizzy:
To break this
(cos Piryj' sin (iayy)) =
norm (b, br, by h2) + (by, by h h.)))
(10)
(ryy (atoa))
(2w)
Σ Σ (20) coos(fuusss + Pixys); sin(thrys + przuss) (11)
sin
(x)
where J&p is an MLP that maps to IR² and l2 norm indicates L2-normalization. Because enantiomers (ay) and phase shifts Piayj are identical between enantiomers (if x; is initialized without chiral tags). However, because enantiomers have different spatial orientations of atoms around chiral centers, the relative values of coupled torsions around bonds involving those chiral centers also differ (Figure 2). As result, learning phase shifts creates different degrees of wave-interference when summing the weighted cosines and sines for inverted
have the same 2D graph, the learned coefficients bi
chiral centers. The amplitudes of the net waves (corresponding to radius | O'sin ID will differ between enantiomers, allowing our model encode different radii for different enantiomers.
(ry) (ary)
With this torsion aggregation scheme, we complete our torsion encoder by individually encoding each internal molecular bond, along with learned radius, into a latent vector and sum-pooling:
Za
Zazy'
Zazy
= fa heg hy,
(ar)) Oen
|) + fa hy: hg |
(zv)
d' l)) (12)
(ay)
(E.))EG
Readout. For property prediction, we concatenate the sum-pooled node states with the conformer embedding components Z.i 2g. and 2.. capturing embeddings of bond lengths, angles, and torsions, respectively. This concatenated representation is then fed through a feed-forward neural network:
ÿ == four Σ hi, Zdd Zed,, Zec
(13)
NEG
explores the option of propagating the learned 3D representations of chirality to node states prior to sum-pooling via additional message passing, treating each Zo.s as a vector of edge- attributes. This stage of chiral message passing (CMP) is designed to help propagate local chiral representations across the molecular graph, and to provide an alternative method of augmenting 2D GNNs which include chiral tags as atom features. However, CMP does not significantly affect ChIRo's performance across the tasks considered and is thus not included in our default model.
Appendix A.3
4 EXPERIMENTS
We evaluate the ability of our model to learn chirality with four distinct tasks: contrastive learning to distinguish between 3D conformers of different stereoisomers in learned latent space, classifying enantiomer chiral centers R/S, predicting how enantiomers rotate circularly polarized light, and ranking enantiomers by their docking scores in chiral protein environment. We compare our model with 2D baselines including DMPNN with chiral atom tags (Yang et al. 2019). and with and without chiral atom tags (Pattanaik et al. 2020a). We also compare our model to 3D GNN baselines, including the Bi3)-inn SchNet and DimeNet++, and the SE(3)-invariant Net.

---

Published as a conference paper at ICLR 2022
ChiRo
SchNet
DimeNet**
SphereNet
Original OMEGA Conformers
Original Reflected Conformers
OH
Conformers with Rotated Bonds
Figure 3: Visualization of how conformers of two enantiomers in the contrastive learning cluster in the learned latent space for ChiRo, SchNet, DimeNet++, and SphereNet (top row) using the original OMEGA-generaate adds points with inverted chirality (opposite color); and (bottom row) upon rotating internal bonds involving the chiral center. Unlike SchNet, DimeNet++, and even the SE(3)-invariant SehereNet, ChIRo maintains perfect separation between stereoisomers across these conformer transformations.
conformers; (middle row) upon reflecting the conformers, which
Contrastive Learning Distinguish Stereoisomers, For contrastive learning, we use subset of the PubChem3D dataset, which consists of multiple OMEGA -eeeGr-ggeeeetedd conformations of organic molecules with up to 50 heavy atoms and 15 rotatable bonds (Bolton et al., 2011 Hawkins al.) 2010). Our subset consists of 2.98M conformers of 598K stereoisomers of 257K 2D graphs. Each stereoisomer has at least two conformers, and each graph has at least two stereoisomers. We create 70/15/15 craiiingvalidattionness splits, keeping conformers corresponding to the same 2D graphs in the same data partition. See appendix A.5 for full training details.
We formulate this task to answer the following question: can the model learn to cluster conformers sharing the same (chiral) molecular identity in a learned latent space, while distinguishing clusters belonging to different stereoisomers of a shared 2D graph? We train each model with triplet margin loss (Balntas et al., 2016) with a normalized Euclidean distance metric:
22
Lriplet max(0. d(za: zp) d(zu, 2n) + 1). d(z), z2)
(14)
22
where Za. Zp. Zn are learned latent vectors of anchor, positive, and negative examples. For each triplet, we randomly sample a conformer a of stereoisomer as the anchor, conformer * a of stereoisomer as the positive, and conformer of stereoisomer j share the same 2D graph. For ChiRo, we use z = Za- For SchNet, DimeNet++, and SphereNet, aggregations of node states (out_channels) z. We set R2 to visualize the latent space.
as the negative, where and
Figure 3 visualizes how conformers of two enantiomers in the test set separate in the latent space for ChiRo, SchNet, DimeNet++, and SphereNet. We explore how this separation affected upon reflecting the conformers (which inverts chirality) and rotating bonds around the chiral center. By design, ChIRo maintains perfect separation across these transforms. While SchNet and DimeNet++ may seem to superficially separate the original conformers, the separation collapses upon reflection and rotation due their -i3a-innar.. SphereNet learns good separation that persists through reflection, but the clusters overlap upon rotation of internal bonds. This emphasizes that 3D GNNs have limited chiral expressivity when not explicitly considering coupled torsions message updates.
Classifying Tetrahedral Chiral Centers as R/S. As a toy test of chiral perception for SchNet, DimeNet++, SphereNet and ChIRo, we test whether these 3D models can classify tetrahedral chiral centers as R/S, simple indication of chirality that can be easily encoded into the initial node fea- tures. We use a subset of PubChem3D containing 466K conformers of 78K enantiomers with one tetrahedral chiral center. We create 70/15/15 train/validation/test splits, keeping conformers corre- sponding to the same 2D graphs in the same partition. We train with a cross-entrppy loss and sample one conformer per enantiomer in each batch. See Appendix fo5 full training details.
7

---

Published as a conference paper at ICLR 2022
Table I reports classification accuracies when evaluat- ing on all conformers in the test set, without conformer- based averaging. The SE(3)-invariant ChiRo and SphereNet both surpass 98% accuracy, whereas the E(3)-invariant SchNet and DimeNet++ fail learn this simplest indication of chirality. We emphasize that this classification task, which RDKit trivially solves, is nec- essary but not sufficient to demonstrate that a model can meaningfully learn chiral representations of molecules.
Table 1: R S classification accuracy on the test set for ChiRo and 3D GNN base- lines. Mcan and standard deviations are reported across three trials.
Model
R/ S Accuracy (%) 1
ChIRo
98.5
SchNet
54.5
DimeNet++ SphereNet
65.7 2.9
Predicting Enantiomers' Signs of Optical Rotation. Enantiomers rotate circularly polarized light in differ- ent directions, and the sign of rotation designates the enantiomer as / (levorotatory, -) or d (dextrorotatory, +). Optical activity is an experimental prop- Following Mamede
98.2 + 0.2
erty, and has no correlation with R/S classifications (Table appendix cct al. (202T) who used hand-crafted descriptors to predict d labels, we extract 30K enantiomers (15K pairs) with their optical activity in chloroform solvent from the commercial Reaxys database (Reaxys) Accessed Aug. 6-9, 2021), and generate 5 conformers per enantiomer using the ETKDG algorithm in RDKit (Landrum (01O). Appendix A.7 describes the filtering procedure. We eval- uate with 5-fold cross validation, where each pair of enantiomers are randomly assigned to one of the five folds for testing. In each fold, we randomly split the remaining dataset into 87.5/12.5 splits, assigning enantiomers to the same data partition. We train with a cross- entropy loss. To characterize how conformer data augmentation affects the performance of ChIRo and SphereNet, we employ two training schemes: the first randomly samples a conformer per enan- tiomer in each batch; the second uses one pre-sampled conformer per enantiomer. In both schemes, we evaluate on all conformers in the test splits. Appendix A.5 specifies full training details.
Table reeoorsss the classification accuracies for each model. Notably, ChiRo outperforms SphereNet by a significant 14%. Also, whereas ChiRo's performance does not suffer when only one conformer per enantiomer is used, SphereNet's performance drops by 10%. This may be due to SphereNet confusing differences in conformational structure versus chiral identities when evaluating pairs of enantiomers (appendix 10p. The consistent performance of ChIRo emphasizes its inherent model- ing of conformational flexibility without need for data augmentation. ChiRo also offers significant improvement over DMPNN, demonstrating that our torsion encoder yields a more expressive repre- sentation of chirality than simply including chiral tags in node features. The (smaller) improvements over Tetra-DMPNN, which was specially designed to treat tetrahedral chirality in 2D GNNs, suggest that using 3D information encode chirality is more powerful than 2D representations of chirality.
Ranking Enantiomers by Binding Affinity in Chiral Protein Pocket. In silico docking sim- ulations are widely used to screen drug-ligand interactions, especially high-throughput virtual screens. However, docking scores can be highly sensitive to the conformation of the docked lig-
Table 2: Comparisons of ChIRo with baselines when classifying enantiomers as / / d and ranking enantiomers by their docking scores. Mean accuracies and standard deviations are reported on the test sets across 5 folds (1 d) or three trials (enantiomer ranking).
Model
Accuracy (%)
l/d
Enantiomer Ranking
ChiRo
79.3 + 0.4
72.0 + 0.5
ChiRo (1 (-Cooformmr)
79.1 0.5
71.9 + 0.3
Net
65.5 2.4
68.6 + 0.3
SphereNet (1 (1-Conformer)
5..2
63.0 + 0.1
DMPNN with Chiral Tags
7..4
$5.9
Tetra- DotrN (permute)
7.72
6.3
Tetra-DMPNN (concatenate)
7.76
6.3
Tetra-DMPNN (permute) with Chiral Tags
6.66
Tetra-DMPNN (concatenate) with Chiral Tags
76.5 + 0.8
70.1 0.5
8

---

Published as a conference paper at ICLR 2022
Table 3: Effects of ablating components of ChiRo on test-set accuracies for the / classification and enantiomer docking score ranking tasks. Mean and standard deviations are reported across 5
folds (1 / d) or 3 repeated trials (enantiomer ranking). The first row indicates the original ChIRo.
Model Components
Accuracy (%)
Tags
(daj. Pijk. Viryj)
(R.ds Zep. na))
1/d
Enantiomer Ranking
79.3 + 0.4
72.0 + 0.5
77.7 + 0.8
71.3 + 0.7
76.2 + 0.6
68.4 + 0.9
X
70.0 - 0.6
63.7 + 0.2
X
V
50.0 + 0.0
49.6 + 0.3
and, which creates noisy background from which to extract meaningful differences in docking scores between enantiomers. To partially control for this stochasticity, we source conformers of 200K pairs of small (MW 225, # of rotatable bonds < S) enantiomers with only chiral center from PubChem3D (Bolton et al. 201T). We use AutoDock Vina (Trott & Olson (010) to dock each enantiomer in small docking box (PDB ID: 4JBV) three times, and select pairs for which both enantiomers have a range of docking scores < 0.1 kcal/mol across the three trials. Finally, we select pairs of enantiomers whose difference (best) scores 2 0.3 kcal/mol to form a dataset of enantiosensitive docking scores. Each conformer for an enantiomer is assigned the same (best) score. This treats docking score as a stereoisomer-level property. Our final dataset includes 335K conformers of 69K enantiomers (34.5K pairs), which we split into 70/15/15 training/valien sainiing/vlidaaionnteess splits, keeping enantiomers in the same data partition. We train with a mean squared error loss and either randomly sample a conformer per enantiomer in each batch per enantiomer. Appendix AA.5] specifies full training details. Note that we evaluate the ranking ac- curacy of the predicted conformer-aavrrgged docking scores between enantiomers in the test
one pre-selected conformer
ChiRo outperforms SphereNet in ranking enantiomer docking scores, achieving an accuracy of 72% (Table 2) This performance is fairly consistent when evaluating ChIRo subsets of enantiomers which have various differences in their ground-truth scores (Figure 12, appendix A.9). SphereNet once again suffers drop in performance without conformer data augmentation, whereas ChIRo's performance remains high without such augmentation. ChIRo also outperforms the 2D baselines, with performance gains similar those in the // d classification task.
Ablation Study. To investigate how the components of ChIRo contribute to its performance on the / / d classification and ranking tasks, we ablate various components including whether stereochemical tags are included in node/edge intialization, whether internal coordinates are masked-out (set to zero), and whether the conformer latent vectors Zd, Zg. and 2 are used readout (Table Overall, when internal coordinates are masked, including (za, Za. Z,) in readout improves performance by ~5% on both tasks. This suggests that if chiral tags are used as the only indication of chirality (i.e.. 3D geometry is excluded), using the learned node aggregation scheme provides a simple way to improve the ability of 2D GNNs to learn chiral functions. Notably, adding chiral tags to the unablated ChIRo does not improve performance, which is expected given ChiRo's improvements over 2D baselines (Table 2). Omitting chiral tags and 3D geometry yields an achiral 2D GNN.
5 CONCLUSION
In this work we design a method for improving representations of molecular stereochemistry in graph neural networks through encoding 3D bond distances, angles, and especially torsions. Our torsion encoder learns expressive representations of tetrahedral chirality by processing coupled tor- sion angles that share common bond while simultaneously achieving invariance to rotations about internal molecular bonds, diminishing the need for conformer-based data augmentation to cap- ture conformational flexibility. Comparisons SphereNet variety of tasks (one self-supervised, three supervised) dependent on molecular chirality demonstrate that ChIRo achieves state-of-the-ar in learning chiral representa- tions from 3D molecular structures, while also outperforming 2D GNNs. We leave consideration of other types of stereoisomerism, particularly cis/trans isomerism and atropisomerism, to future work.
the E(3)-invariant SchNet and DimeNet++ and the

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
Advancing representations of molecular chirality and conformational flexibility has the potential accelerate pharmaceutical drug design, renewable energy development, and progress towards energy-efficient and waste-reducing catalysts, among other areas of scientific research. However, principle, the same advances could also be used for harmful biological, chemical, or materials research and applications thereof.
REPRODUCIBILITY STATEMENT
Care has been taken ensure reproducibility of all models and experiments this paper. In addition to detailing exact model architectures and training details in the appendix, we make source code with experimental setups, model implementations, and random seeds available htttt::://// /tttps://ithub com/ kei radams/ CIRR Our GitHub site also contains links to the exact datasets and splits used each experiment for the contrastive learning, R/S classification, and ranking enantiomers by docking score tasks. Although copyright restrictions prevent us from releasing the dataset and splits for the / d classification task, we detail our data filtering and processing steps in appendix A.7
ACKNOWLEDGMENTS
This research was supported by the Office of Naval Research under grant number N00014-21-1 2195. L.P. thanks the MIT-Takeda fellowship program for financial support. The authors acknowl- edge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC re- sources that have contributed the research results reported within this paper. The authors thank Octavian Ganea and John Bradshaw for providing helpful suggestions regarding the content and orgamization of this paper.
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
Simon Axelrod and Rafael Gombar-Bombarell. Molecular machine learning with conformer ensem- bles. arXiv preprint arXiv:2012.08455 2021.
Vassilcios Balntas, Edgar Riba, Daniel Ponsa, and Krystian Mikolajezyk. Learning local feature descriptors with triplets and shallow convolutional neural networks. In Edwin R. Hancock Richard C. Wilson and William P. Smith (eds.), Proceedings of the British Machine Vision Conference (BMVC), pp. 1191-1199.11 BMVA Press, September 2016. ISBN 1-901725-59-6 doi: 10.5244/ 1.9. URL https:tt///.. //x doi wrg/10.gg/ooo.ooooooooooo . 5244/2.22. 30.
Evan E. Bolton, Jie Chen, Sunghwan Kim, Lianyi Han, Siqian He, Wenyao Shi, Vahan Simonyan, Yan Sun, Paul A. Thiessen, Jiyao Wang, Bo Yu, Jian Zhang, and Stephen H. Bryant. PubChem3D: new resource for scientists. Journal of Cheminformatics, 3, 2011.
Nathan Brown, Marco Fiscato, Marwin H.S. Segler, and Alain C. Vaucher. GuacaMol: Benchmark- ing models for de novo molecular design. Journal of Chemical Information and Modeling, 59(3): 1096-1108, 2019.
Kangway V. Chuang and Michael J. Keiser. Attention-basse based learning molecular ensembles. arXiv preprint arXiv:2011.12820 2020.
Kangway V. Chuang, Laura M. Gunsalus, and Michael J. Keiser. Learning molecular representations for medicinal chemistry. Journal of Medicinal Chemistry, 63:8705-8722 2020.
Hanjun Dai. Bo Dai, and Le Song. Discriminative embeddings of latent variable models for struc- tured data. In Proceedings of the 33rd International Conference on Machine Learning, JMLR: WCP, volume 48, 2016.
10

---

Published as a conference paper at ICLR 2022
David Duvenaud, Dougal Maclaurin, Jorge aguilera-Iparraguinn Rafacl Gómez-Bombarelli, Tim- othy Hirzel, Alán Aspuru-Guzik, and Ryan Adams. Convolutional networks on graphs for learning molecular fingerprints. In Proceedings of Advances in Neural Information Processing Systems, volume 28, 2015.
Daniel C. Elton, Zois Boukouvalas, Mark D. Fuge, and Peter W. Chung. Deep learning for molecular design- review of the state of the art. Molecular Systems Design & Engineering, 2019.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with PyTorch Geometric. ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Daniel Flam-Shepherd, Tony Wu, Pascal Friederich, and Alan Aspuru-Guzik. Neural message pass- ing on high order paths. Machine Learning: Science and Technology, 2(4), 2021.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. SE(3)-transformers: 3D roto- translation equivariant attention networks. 34th Conference on Neural Information Processing Systems (NeurIPS), 2020.
Octavian-Fugen Ganea, Lagnajit Pattanaik, Connor W. Coley, Regina Barzilay, Klavs Jensen, William H. Green, and Tommi S. Jaakola. GeoMol: Torsional geometric generation of molecular 3D conformer ensembles. 35th Conference on Neural Information Processing Systems (NeurIPS), 2021.
Justin Gilmer, Samuel S. Schoenholz, Patrick E. Riley, Oriol Vinyals, and George F. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 1263-1272, 2017.
Alexander Golbraikh, Danail Bonchev, and Alexander Tropsha. Novel chirality descriptors derived from molecular topology. J. Chem. Inf. Comput. Sci., 41:147-158, 2001.
Paul C. D. Hawkins, A. Geoffrey Skillman, Gregory L. Warren, Benjamin A. Ellingson, and Matthew Stahl. Conformer generation with OMEGA: algorithm and validation using high quality structures from the protein databank and cambridge structural database. Journal of Chem- ical Information and Modeling, 50(4):572-5844 2010.
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W. Co- ley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeuties Data Commons: Machine learning datasets and tasks for drug discovery and development. In Proceedings of Neural Information Processing Systems, NeurlPS Datasets and Benchmarks, 2021.
Jamali, R. Mehvar, and F.M. Pasutto. Enantioselective aspects of drug action and disposition: therapeutic pitfalls. Journal of Pharmaceutical Sciences, 78(9):695-755, 1989.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Ziclinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Se- bastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596:583-558 5898 2021.
Steven Kearnes, Kevin McCloskey, Mare Berndl, Vijay Pande, and Patrick Riley. Molecular graph convolutions: Moving beyond fingerprints. Journal of Computer-Aided Molecular Design, 30: 595-608, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Johannes Klicpera, Shankari Giri, Johannes T. Margraf, and Stephan Günnemann. Fast and uncertainty- directional message passing for non-equilibrium molecules. Machine Learning for Molecules Workshop at NeurlPS, 2020a.

---

Published as a conference paper at ICLR 2022
Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molec- ular graphs. International Conference on Learning Representations (ICLR), 2020b.
Johannes Klicpera, Florian Becker, and Stephan Günnemann. GemNet: Universal directional graph
neural networks for molecules. 35th Conference on Neural Information Processing Systems
(NeurIPS), 2021.
Assia Kovatcheva, Alexander Golbraikh, S. Oloff, J. Feng. W. Zheng, and Alexander Tropsha QSAR modeling of datasets with enantioselective compounds using chirality sensitive molecu- lar descriptors. SAR and QSAR in Environmental Research, 16(1 (6(1-)):33-11 102, 2007.
Greg Landrum. RDKit: Open-source cheminformatics. 2010. URL https://wwes rakit. org/
Kuangbiao Liao, Yun-Fang Yang, Yingzi Li, Jacob N. Sanders, K.N. Houk, Djamaladdin G. Musaev, and Huw M.I.. Davies. Design of catalysts for site-selective and enantioselective functionalization of non-activated primary C-H bonds. Nature Chemistry, 10:1048-1055 2018.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, and lon Sto- ica. Tune: research platform for distributed model selection and training. arXiv preprint arXiv: 18, 2018.
Zhiqiu Lin, Jin Sun, Abe Davis, and Noah Snavely, Visual chirality. Computer Vision and Pattern Recognition (CVPR), 2020.
Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Ji Shuiwang. Spherical message passing for 3D graph networks. arXiv preprint arXiv:2102.05011 2021.
Gómezz-ortigüüela Luis and Bernardo-Maestrr Beatriz. Chiral Organic Structure-Directing Agents. In: Gómez-Hortigüela, Luis (eds) Insights into the Chemistry of Organic Prructure-Directioe Agents in the Synthesis of Zeolitic Materials. Structure and Bonding, volume 175. Springer Inter- national Publishing, 2018
Rafael Mamede, Bruno Simões de Almeida, Mengyao Chen, Qingyou Zhang, and Joao Aires-de Sousa. Machine learning classification of one-chiral-center organic molecules according to optical rotation. Journal of Chemical Information and Modeling, 61(1):67-75, 2021.
Elman Mansimov, Omar Mahmood, Scokho Kang, and Kyunghyun Cho. Molecular geometry pre- diction using a deep generative graph neural network. Scientific Reports, 9, 2019.
Lien Ai Nguyen, Hua He, and Chuong Pham-Huy. Chiral drugs: An overview. International Journal of Biomedical Science, 2(2):85-100, 100, 2006.
Lagnajit Pattanaik, Octavian Eugen Ganea, Ian Coley, Klavs F. Jensen, William H. Green, and Con- nor W. Coley. Message passing networks for molecules with tetrahedral chirality. arXiv preprint arXiv:2012.00094, 2020a.
Lagnajit Pattanaik, John Ingraham, Colin A Grambow, and William H Green. Generating transi- tion states of isomerization reactions with deep learning. Physical Chemistry Chemical Physics, 22(41)) 23626,, 2020b.
Andreas Pfaltz and William J. III Drury. Design of chiral ligands for asymmetric catalysis: From C2-symmetric P.P- and N.N-ligands sterically and electronically nonsymmetrical P.N-ligands. PNAS, (775555553 2004.
Reaxys. Reaxys and the Reaxys trademark are owned and protected by Reed Elsevier Properties SA and used under license. Accessed Aug. 6-9, 2021. URL httthtttt:://ee reaxys com/
Victor Garcia Satorras, Emiel Hoogehoom, and Max Welling. E(n) equivariant graph neural net- works. In Proceedings of the 38th International Conference on Machine Learning. PMLR, volume 139, pp. 9332, 2021.
Nadine Schneider. Richard A. Lewis, Nikolas Fechner, and Peter Ertl. Chiral cliffs: Investigating the influence of chirality binding affinity. ChemMedChem, 13(13):315--1324, 2018.
12

---

Published as a conference paper at ICLR 2022
Kristof T. Schütt, Pieter-Jan Kindermans, Huziel E. Sauceda, Stefan Chmiela, Alexandre Tkatchenko, and (LLus-RRbber Müller. SchNet: A continuous-iile convolutional neural net- work for modeling quantum interactions. In Proceedings of Advances in Neural Information Processing Systems, volume 30, pp. 992-1002, 2017.
Gregor Simm and Jose Miguel Hermandez-Lobato A generative model for molecular distance geom- etry. In Proceedings of the 37th International Conference on Machine Learning, PMLR, volume 119, pp. 8949-8995 2020.
Martin Simonovsky and Nikos Komodakis. Dynamic edge-conditioned filters in convolutional neu- ral networks on graphs. Computer Vision and Puttern Recognition (CVPR), 2017.
Nathaniel Thomas, Tess Smidt, Steven M. Kearne,, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-equivarian neural networks for 3D point clouds. arXiv preprint arXiv:/802.088) 2018.
Oleg Trott and Arthur J. Olson. AutoDock Vina: improving the speed and accuracy of docking with a new scoring function, efficient optimization and multithreading. Journal of Computational Chemistry, (1()):455 461, 2010.
José R. Haldés-Martiní, Yovani Marrero-Ponn PoneroePonce, César R. García-Jacas, Karina Martinez- Mayyora Stephen Barigye, Yasser Silveira Vaz d' Almeida, Facundo Pérez-Giménes and Carlos A. Morell. QuBiLS-MAS, open source multi-platform software for atom-and bond- aadd-asedd topological (2D) and chiral (2.5 D) algebraic molecular descriptors computations. Journal of Cheminformar- ics, ((1):1-26, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph attention networks. International Conference on Learning Representations (ICLR), 2018.
Jan Weinreich, Nicholas J. Browning, and O. Anatole von Lilienfeld. Machine learning of free energies in chemical compound space using ensemble representations: Reaching experimental uncertainty for solvation. The Journal of Chemical Physics, 154(13), 2021.
Robin Winter, Frank Noé, and Djork-Aré Clevert. Auto-encoding molecular conformations. arXiv preprint w=Xii:2101.01611 2021.
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. MoleculeNet: A benchmark for molecular machine learning. Chem Sci., ((2):113-530) 2018.
Minkai Xu, Shitong Luo, Yoshua Bengio, Jian Peng, and Jian Tang. Learning neural generative dynamics for molecular conformation generation. International Conference Learning Repre- sentations (ICIR), 2021.
Kevin Yang, Kyle Swanson, Wengong Jin, Connor W. Coley, Philipp Eiden, Hua Gao, Angel Guzman- Perez, Timothy Hopper, Brian Kelley. Miriam Mathea, Andrew Palmer, Volker Settels, Tommi Jaakola, Klavs F. Jensen, and Regina Barzilay. Analyzing learned molecular representa- tions for property prediction. Journal of Chemical Information and Modeling, 397003337-3338 2019.
Raymond A. Yeh, Yuan-Ting Hu, and Alexander G. Schwing. Chirality nets for human pose regres- sion. 33rd Conference on Neural Information Processing Systems (NeurIPS), 2019.
Dmitry V. Zankov, Maxim D. Shevelev, Aleksandra V. Nikoenko, Pavel G. Polishchuk, Asima Rakhimbekova, and Timur I. Madzhidov. Multi-instance learning for structure-activity modeling for molecular properties. International Conference on Analysis of Images, Social Networks and Texts, pp. 62-71, 2019.
Dmitry V. Zankov, Mariia Matveieva, Aleksandra Nikoenko, Ramil Nugmanov, Igor Baskin, Alexandre Varnek, Pavel Polishchuk, and Timur Madzhidov. QSAR modeling based on con- formation ensembles using a multi-instance learning approach. Journal of Chemical Information and Modeling, 2021.
13

---

Published as a conference paper at ICLR 2022
A
APPENDIX
CONTENTS
Appendix
14
A.1
Why does plotting asn sin vs. Cleos
(29)
(ay) yield a perfect circle?
15
A.2
Why do need phase shifts?
16
A.3
Chiral Message Passing
17
A.4
Model Architectures
17
A.4.1
ChiRo
17
A.4.2 3D Baselines
20
A.4.3 2D Baselines
22
A.5
Training Details
23
A.5.I
DimeNet++ Initializations
23
A.5.2
SphereNet Data Processing Errors
23
A.5.3
Auxiliary Torsion Loss when Training ChiRo
23
A.5.4
Contrastive Learning
23
A.5.5
R/S Classification
24
A.5.6
d Classification
24
A.5.7
Ranking Enantiomers by Docking Scores
................................
24
A.6
Hyperparameter Optimizations
26
A.6.1
ChiRo
26
................................
A.6.2
SphereNet
27
A.6.3 DMPNNI
27
................................
A.7
Datasets
28
A.7.I
Contrastive Learning and R/S Classification Datasets
28
A.7.2
Classificaaoio Datasel
30
A.7.3
Ranking Enantiomers by Docking Scores
32
A.8
Additional Ablations
34
A.9
Additional Evaluation of Ranking Enantiomers by Docking Scores
34
A.10
Analyzing Invariatn 3D GNNs without InterRoto Interkoto-Invariance
14

---

Published as a conference paper at ICLR 2022
(ay)
(ay) VS. Ocoo
A.1
WHY DOES PLOTTING OSSIN
YIELD PERFECT CIRCLE?
is not immediately obvious that plotting sin versus Oceos
(my)
(zy) forms perfect circle with a constant
radius, opposed to ellipse other shape with non-constant radii. Yet, this result follows from simple trigonometry.
From Equation
have:
C coo
(2r4)
Σ
E
(1v) Oj
cos(Warys + PPryy))
iEN(Z) JEN(M)(
(ay) sin
Σ E
(zu) Ci
sin sin(Wergs + Paayy)
for internal molecular bond between nodes (z, y) that forms set of coupled torsions {Vizys (i.)) where E N (a) {y} and j E N2y) \ {x}. For clarity, we drop the (zy) notation and index each torsion this set of coupled torsions by single index k such that (Vien (t. [vu
We want show that plotting Σx=1 Cę sin(R+ the +(p)) versus Ex= C cos R t/u +.p)). where R E [0, 2T) is rotation of the bond (zr, y). yields a circle with a constant radius invariant to the rotation R. simple case, consider
2. We then have:
acoo = cos R + 1/1 + (P)) + C2 cos R + 12 + P2) cos R + 01) + cos + 82)
Olsin 6j sin (R + 1/1 + pp)) + c2 sin R + 1 sp2) = e₁ sin 01) + 02 sin R 02)
Ocos can be expanded and simplified as: ei cos( R + 01) + 22 cos( 02)
cos(R) cos(O,) sin [t) sin(0i) + +22 cos(RR cos((2)) - sin(R) sin cos cos(R) b binn(R) d cos (R e)
where
cos ci cos cos(61) + 02 cos(02)
b dsin c₁ sin(01) C2 sin)
Similarly, Osgin can be expanded and simplified as: ei sin R 01) + 02 sin + 02)
sin (sin(R) cos( + cos( sin(81)) 02 sin (R) cos(O2) cos( cos(R) sin(B2)
a' d'sin( R) b' cos R) d' sin(R +6)
where
a' d' cos e' ci (01) 02 cos(O2)
b' = d' sin e' = sin( ( + (in
Since = d' and e e', we have that aleos = cos R +e) and asin = dsin(R follows that in the general case (n 2 2):
Ck cos the + (px) r roo(R R
Oceos
k=
Ogin
C₁ sin(wt + th + (p)) sin R
for some coefficients and s, where R [0, 2TT) represents rotation about the internal bond. It immediately obvious that plotting Osgin versus Oroo for all E [0, 2T) yields a perfect circle with radius
15

---

Published as a conference paper at ICLR 2022
A.2 WHY DO WE NEED PHASE SHIFTS?
At first glance, is unclear as to why Equations 8 are insufficient to learn chirality, e.g. why we need add phase shifts distinguish enantiomers. To understand why this is the case, consider an arbitrary stereoisomer containing an internal molecular bond between nodes æ and (with x or being chiral) that forms set of n coupled torsions (Wizy) (6.5)
(w (m)
H indexed by k. Direct
(xy) (2x)
computation of the norm a
asin
= |[kcoss Osin|| yields:
Ocoss Osinl
C'eos + Otgin"
where we have dropped the (zy) notation for clarity. Expanding and squaring Equation 3 gives:
aloos
o
cos
Uk
200e cos the cos 1/1
1=
Ossin
sin 1/
Σ
Σ
20 sin the sin t/z
1=11=k+1
Now consider reflecting this stereoisomer across a plane to generate its enantiomer with inverted chiral centers. Reflecting a conformer negates its torsions, and thus the reflected enantiomer's in- ternal bond between (reflected) nodes x' and y' will form set of n negated torsions (Wlay
(k) (Ak) -1/y for each k. Because enantiomers have the same 2D graph con- (vayy Jk=1 where Way
(())
nectivity, the learned set of coefficients {cr, } (which depend only on the permutation invariant node
features, Equation 5 will be identical between enantiomers. It immediately follows from the identities sin(-
(i)( and cos(-z:) cos(x) that a' 2 a and a a Because
1oooss Osin is invariant to any rotation of the bond (x, y), no matter how we rotate (æ, y) or (x', y'),
(ccoos Otsin
| d'in!
16

---

Published as a conference paper at ICLR 2022
Table Effects of ablating components of ChIRo on test-set accuracies for the d classification and enantiomer docking score ranking tasks, when chiral message passing is included prior to readout. Mean and standard deviations are reported across 5 folds / d) or repeated trials (enantiomer ranking). The first row indicates the original ChIRo without CMP.
Model Components
Accuracy (%) t
CMP
Tags
(dej. Pigke Press)
(Zeds Do 2o.)
1/d
Enantiomer Ranking
X
X
79.3 0.4
72.0 + 0.5
X
78.5 0.5
71.5 + 0.5
V
V
77.0 0.5
71.7 + 0.6
V
V
X
75.1 0.3
69.8 = 0.6
V
V
X
X
75.0 +.9 0.9
68.9
V
X
X
50.0 + 0.0
49.9 + 0.3
A.3 CHIRAL MESSAGE PASSING
Tetrahedral chirality fundamentally a node-level property. Yet, we have treated chirality through the lens of torsions and internal bonds involving chiral centers. To propagate the learned representa- tions of chirality to node states, we may perform an (optional) additional phase of message passing, treating each Zazy information across the graph, i.e., help learn the global effects of local chirality. This phase may be included prior to readout, and uses single EConv layer followed by sequence of TeMP GAT layers:
vector of edge-attributes. This might allow the model to propagate chiral
(CMM h
h CMP
GAT ... GAT CMP
Σ hy -fCMP(Zag)
(15)
JEN())
During readout, these updated node states h CMP are summed rather than the (non-chiral) node states.
Table 4 reports the effects of including chiral message passing ChiRo's performance the / classification task and the ranking enantiomers by docking scores task, along with the effects of ablating other components of ChIRo. Although CMP does not improve performance (and marginally hurts) the unablated ChiRo, using CMP in place of (z,,, Zg, Za) during readout provides another option of augmenting 2D GNNs that solely use chiral tags the only indication of chirality.
A.4 MODEL ARCHITECTURES
A.4.1 CHIRO
Figure Seture|viia visualizes the full architecture of ChIRo. Table 5iseciffese the hyperparameters chosen for each task. See appendix A.6 for details on hyperparameter optimizations.
17

---

Published as a conference paper at ICLR 2022
Internal Coordinates
dis
2D Message Passing
Torsion Encoder
Edge/Node Fealurization
fulb, h2 by by)
fa (b,, Nr, by: By)
(mut
D:Os, by, Iu)
La by. Br, by)
Edge Convolution
O
L2-nomalization
GAT Layers (h.)
Σ
20 (conthhos: Wouth sinthhu woul?
Bond Distance Encoder
(alte) oe Doaa)
fx(b,, by, dgy) felb,, b,, die)
La lb, |lofoot (/2211))
(Ra,)oo.
fa(h, hy Booleo (1
Sum-Pooling
( 1o.s)
(b.)
Edge Convolution
Sum-Pooling
%.
Angle Encoder
fe (b, by, ha, cos chyie sin (hoe)
GAT Layers
(b CAP
fo (ba, hy, hu, con degs, sin desu)
Readout
Sum-Pooling
mmp Buc)
Figure 4: Architecture of ChiRo
18

---

Published as a conference paper at ICLR 2022
Table 5: Hyperparameterss optimized for ChiRo each task. Parameters for chiral message passing
are shown, although chiral message passing is omitted in the default version of ChIRo.
Hy Hyperparameters
Task
Contrastive
R/S
11 d
Ranking
Node Features Dimension
52
Edge Features Dimension
14
All MLP Hidden Activations
LeakyReLU
All MLP Output Activations
Identity
EConv MLP Hidden Layer Size
32
64
32
EConv MLP # Hidden Layers
2
1
2
hg. hy, h CMP Dimension
64
32
32
64
hg. .....
Dimension
64
64
32
64
# GAT Layers
2
3
2
2
GAT Heads
4
4
4
fa Hidden Layer Size
128
32
64
fa # Hidden Layers
2
2
La Hidden Layer Size
128
32
64
Je Hidden Layers
2
2
fa Hidden Layer Size
64
128
32
64
fa Hidden Layers
2
2
2
fo Hidden Layer Size
64
128
32
64
fe # Hidden Layers
2
2
2
f Hidden Layer Size
64
256
256
256
F # Hidden Layers
2
2
3
2
four Hidden Layer Size
64
64
128
four # Hidden Layers
2
2
2
24. Zd. 2n Dimension
2
64
8
CMP BConv Hidden Layer Size
32
256
256
CMP EConv # Hidden Layers
1
2
# CMP GAT Layers
3
3
CMP GAT Heads
2
2
%aux Learning Rate
8.25e-4
6.86e-3
1.86e-3
8.25e-4
6.06e-4
5.69e-4
1.28e-4
6.06e-4
Batch Size
32
16
16
32
# Epochs
50
100
100
150
19

---

Published as a conference paper at ICLR 2022
Table 6: Hyperparameters selected for SphereNet each task. See appendix A.6 for hyperparame- ter optimizations.
Hyperparameters
Task
Contrastive
R/S
l/d
Ranking
Readout MLP Hidden Activations
LeakyReLU
Readout MLP Output Activation
Identity
Readout MLP Hidden Size
256
64
64
Readout MLP # of Hidden Layers
4
2
2
hiiden_channer
128
256
64
256
out_channels cutoff num_layers int emb size basis emb size dist basis_emb_size_ange basis_emb_sizze _izeetorsion out_emb chnb_channels opm-rcherical num_radial
2
64
64
32
5.0
5.0
5.0
5.0
4
4
4
5
64
32
128
64
8
8
8
8
8
8
8
8
256
128
64
32
7
7
7
6
6
envelope exponent num before skip num hon_aferrskkip num_output layers
5
5
1
1
2
2
3
3
3
3
Learning Rate
le-4
1.54e-4
4.79e-4
1.40c-4
Batch Size
32
64
16
32
A.4.2 3D BASELINES
To adapt SchNet, DimeNet++, and SphereNet for our tasks, we increase the dimensionality of their respective aggregated node embeddings ("out_channell") and use this aggregation as a latent vector cither for direct use in Equation| 14 (for self-supervised contrastive learning), or for input to a feed- forward readout MLP for downstream regression/cassificattinn For all three 3D GNNs, we use their default node featurizations, using the atomic number as the only node feature.
Table lists the hyperparameters used for SphereNet all four tasks. Tables and 8 list the hyperparameters used for SchNet and DimeNet++ on the contrastive learning and R classification tasks.
20

---

Published as a conference paper at ICLR 2022
Table 7: Hyperparameters selected for SchNet each task. Apart from the number of out channels and the inclusion a readout MLP, the default SchNet architecture was used. Note that we used the same readout MLP architecture as in the optimized SphereNet.
Hyperparameters
Task
Contrastive
R/S
Readout MLP Hidden Activations
LeakyReLU Identity
Readout MLP Output Activation
Readout MLP Hidden Size
64
Readout MLP # of Hidden Layers
hidden channels out channels
128
128
num_fil num_filters num interrrcaatsons num num-gaussinn gaussians cutoff
128
128
50
50
10.0
10.0
max neighbors
32
32
Learning Rate Batch Size
le-4
le4
32
32
Table 8: Hyperparameters selected for DimeNet++ on each task. Apart from the number of out channels and the inclusion of readout MLP, the default DimeNet++ architecture was used. Note that we used the same readout MLP architecture as in the optimized SphereNet.
Hyperparameters
Task
Contrastive
R/S
Readout MLP Hidden Activations
LeakyReLU Identity
Readout MLP Output Activation
Readout MLP Hidden Size
64
Readout MIP # of Hidden Layers
Lidden_channeis
128
128
out_channels num blocks cutoff int_emb size basis_emt saris_em__iiee size out_emb channels num_spherical num_radial
32
5.0
5.0
64
8
256
256
7
envelope exponent
5
num before skip
num_output_layy layers
Learning Rate
le-4
le-4
Batch Size
32
32
21

---

Published as a conference paper at ICLR 2022
Table 9: Hyperparameters selected for DMPNN each task. Following Pattanaik et al.] only optimize hyperparameters for the baseline sum aggregator and extend these hyperparameters to the TetraDMPNN models.
Hyperparameters
Task
Contrastive
R/S
hidden_siee
300
300
depth
3
dropout
0.2
0.2
Max Learning Rate
le-4
le-4
Batch Size
50
50
A.4.3 2D BASELINES
To optimize hyperparameters for all 2D baselines, we directy use the code provided by Pattanaik al. We do not make any modifications to the architectures, since train similar tasks as the original work.
22

---

Published as a conference paper at ICLR 2022
A.5
TRAINING DETAILS
This section describes the full training protocols for each task considered in this paper.
A.5.1 DIMENET++ INITIALIZATIONS
The default parameter initializations for the output blocks of DimeNet++ make DimeNet++ unable to break symmetry between different output channels when the number of output channels is set > 1. Thus, we remove the default initializations for the output blocks when training DimeNet++ on the contrastive learning task (out channels 2) and the R/S classification task coutnelss 32).
A.5.2 SPHERENET DATA PROCESSING ERRORS
When training SphereNet, occasionally the publicly-available implementation of SphereNet failed to process select conformers. Because this occurred for only a tiny fraction of the conformers in the overall datasets, we removed 2D graphs whose stereoisomers contained problematic conformers from the R/S and 1 / d datasets for SphereNet only. We emphasize that this filtering step did not meaningfully change the size of the datasets: only 40 2D graphs (out of 39,256) were removed for the R/S task, and only 6 2DD graphs (out of 15,038) were removed for the // d task. No molecules had to be removed for the ranking enantiomers by docking scores task. For the contrastive learning task, we simply skipped the rare batch during training/inference that contained problematic conformers. Only 452 2D graphs out of 257,743 caused processing errors in the contrastive learning dataset.
A.5.3 AUXILIARY TORSION LOSS WHEN TRAINING CHIRO
In Equation 10 we predict angular phase shift yp by using fy to predict cos y and sin y separately with a linear output activation, and then use &g normalization on the vector [cos yp, sin yp] to ensure that these sin and cos (and thus the angle (o) have correct circular properties, namely that ip (0,2m)) We have chosen to predict angles using sin and cos rather than directly predicting a scalar xp [0, 2T) (c.g., through a scaled sigmoid activation) in order avoid biasing the predicted phase shifts toward 0. However, the E2 normalization can also cause the predicted phase shift to be biased toward 0, x æ, and 3x/ /2 if one of the unnormalized cos & or sin y blows up. To prevent this, we follow JumperJuprrrrr et al. (202I) I) in adding an auxiliary loss during training to encourage the unnormalized [cos (p, sin gp| to fall close to the unit circle:
Lax Yaux (1 || cos sp sin (lll
(16)
where Taux is a small scalar that is tuned during hyperparameter optimization.
A.5.4 CONTRASTIVE LEARNING
For contrastive learning, we train with triplet margin loss with a normalized Euclidean distance metric and a margin of (Equation [4]. In each training epoch, we randomly partition the train- ing data into N/6 minibatches, where N is the number of distinct stereoisomers (not conformers) in the training set and b is the batch size. Before triplet sampling, each minibatch therefore con- b) in the minibatch, we then generate a
tains b stereoisomers. For each stereoisomer = triplet (a. p, n),, where a (anchor) and p (positive) are two randomly selected (without replacement) conformers of stereoisomer i, and n (negative) is a randomly selected conformer of stereoisomer j i, where and j share the same 2D graph connectivity. If stereoisomer has multiple differ- ent stereoisomers j, k... (all sharing the same 2D graph) present in the training set, one of these stereoisomers is randomly chosen. Each anchor, positive, and negative each triplet are processed independently by the network to generate b triplets of latent vectors (Za: Rp. 2,,), which are then fed into the triplet margin loss function. Loss contributions from each triplet are averaged to form a loss for the entire batch. In the case of ChIRo, we add the mean auxiliary torsion loss across all conformers in the batch to the batch triplet loss.
We use the Adam optimizer (Kingma & Bal 2014) with flat learning rate throughout training, but employ gradient clipping with a maximum gradient L2-norm of 10. We train each model for a maximum of 50 epochs, and use the averagee triplet loss (without the auxiliary torsion loss contributions) on the validation set to select the model with the best estimated generalization perfor- mance across the 50 epochs. We do not employ dropout other forms of explicit regularization.
23

---

Published as a conference paper at ICLR 2022
A.5.5 R R/S CLASSIHICATION
For R classification, we train each 3D model with binary cross-entropy loss. In each training epoch, we randomly partition the training data into N/b minibatches, where N is the number of enantiomers in the training set and is the batch size. For each enantiomer in the batch, we randomly sample a conformer for that enantiomer and a conformer for its opposite enantiomer with the same 2D graph. This ensures that minibatches contain both enantiomers of the enantiomeric pair such that the stochastic gradient steps consider contributions from both enantiomers. We average the loss contributions for all conformers in the batch. In the case of ChiRo, the mean auxiliary torsion loss across all conformers in the batch is added to the batch cross-entropy loss.
We use the Adam optimizer with a flat learning rate throughout training, but employ gradient clip- ping with a maximum gradient L.2norm of 10. We train each model for maximum of 100 epochs, and use the batch-averaged classification accuracy on the validation set to select the model with the best estimated generalization performance across the 100 epochs. We do not employ dropout or other forms of explicit regularization.
For testing, we evaluate the classification accuracy on all conformers in the test set, and do not use any form of conformer-base averaging or voting.
A.5.6 1/ d CLASSIFICATION
For 1 / d classification, we train each model with a binary cross-entropy loss. In each training epoch, we randomly partition the training data into N/6 minibatches, where N is the number of enantiomers in the training set and b is the batch size. For each enantiomer in the batch, we sample either 1) random conformer or 2) a pre-selected conformer for that enantiomer, and either 1) a random conformer or 2) a pre-selected conformer conformer for its opposite enantiomer with the same 2D graph. This ensures that minibatches contain both enantiomers of the enantiomeric pair such that the stochastic gradient steps consider contributions from both enantiomers. In both cases, average the loss contributions for all conformers in the batch. In the case of ChIRo, we add the mean auxiliary torsion loss across all conformers in the batch to the batch cross-entropy loss.
We use the Adam optimizer with a flat learning rate throughout training, but employ gradient clip- ping with a maximum gradient L.2-norm of 10. We train each model for a maximum of 100 epochs, and use the batch-averaged classification accuracy on the validation set to select the model with the best estimated generalization performance across the 100 epochs. We do not employ dropout or other forms of explicit regularization.
For testing, we evaluate the classification accuracy on all conformers in the test set, and do not use any form of conformer-based averaging or voting.
A.5.7 RANKING ENANTIOMERS BY DOCKING SCORES
For ranking enantiomers by their docking scores, we train each model with a mean squared error (MSE) loss, with the target values being the ground-truth docking scores for each enantiomer. Note that although we train the models with an MSE loss, we are more concerned with the ability of each model to correctly rank enantiomers by their docking scores than with the absolute performance of each model as a surrogate model for docking score prediction. Thus, we evaluate this task using the ranking accuracy, defined as whether or not the model correctly predicts a lower docking score for the enantiomer with the lower ground-truth score.
In each training epoch, we randomly partition the training data into N/6 minibatches, where N is the number of enantiomers in the training set and b is the batch size. For each enantiomer in the batch, we sample either 1) random conformer or 2) a pre-selected conformer for that enantiomer, and either 1) a random conformer or 2) pre-selected conformer conformer for its opposite enantiomer with the same 2D graph. This ensures that minibatches contain both enantiomers of the enantiomeric pair such that the stochastic gradient steps consider contributions from both enantiomers. In both cases, we average the loss contributions for all conformers in the batch. In the case of ChIRo, we add the mean auxiliary torsion loss across all conformers in the batch to the batch MSE loss.
We use the Adam optimizer with a flat learning rate throughout training, but employ gradient clip- ping with maximum gradient 1.2-norm of 10. We train each model for maximum of 150 epochs,
24

---

Published as a conference paper at ICLR 2022
and use the batch-averaged ranking accuracy the validation set to select the model with the best estimated generalization performance across the 150 epochs. We do not employ dropout or other forms of regularization.
For testing. we first predict the docking score for all conformers in the test set. We then average the predicted docking scores for each conformer of each enantiomer, yielding mean predicted score for each enantiomer. Finally, we compute the ranking accuracy using these mean predicted scores.
25

---

Published as a conference paper at ICLR 2022
Table 10: Hyperparameter search space for ChiRo
Hyperparameter
Search Space
EConv MLP Hidden Layer Size
[32, 64, 128, 256]
BConv MI. # Hidden Layers
[1,2]
ho. hr, h Dimension
[8, 16, 32, 64]
h,, t 1, ..... T Dimension
[16, 32, 64]
# GAT Layers
[2,3,4]
# GAT Heads
[1,2,4,8]
fa. for fa, fc Hidden Layer Size
[32, 64, 128, 256]
la Je. Jon Jr. # Hidden Layers
[1,2,3.4]
Se Hidden Layer Size
[32, 64, 128, 256]
& # Hidden Layers
[1,2,3,4]
f oor Hidden Layer Size
[32. 64, 128, 256]
fou # Hidden Layers
[1,2,3,4]
Zd. ZΦ: 2a Dimension
[8, 16, 32, 64)
CMP EConv Hidden Layer Size
[32, 64, 128, 256]
CMP EConv Hidden Layers
[1,2,3,4]
# CMP GAT Layers
2, 3, 4]
# CMP GAT Heads
[1. 4, 8]
Yaux
log uniform (Ie-4, 1c-2) log uniform (5e-5, 5e-3) [16, 32, 64, 128, 256]
Learning Rate
Batch Size
A.6 HYPERPARAMETER OPTIMIZATIONS
A.6.1 CHIRO
Hyperparameters were tuned for ChIRo on the R / S. // d, and ranking enantiomers by docking score tasks, using the Raytune (Liaw et al. 2018) Python package with the HyperOpt plug-in. For the R / S classification and ranking enantiomers by docking score tasks, we tuned hyperparameters by training with the training set and evaluating model accuracy the validation set. For the / d classification task, we used the training and validation splits of the first cross validatoi fold to tune hyperparameters, evaluating model accuracy on the validation split. The optimal hyperparameters were then held constant for the remaining four folds.
For each task below, we used the HyperOptSearch search algorithm in Raytune, which employs Tree-structured Parzen Estimators, to search for optimal hyperparameters over the search space specified in TO
R/S S Classificattinn We trained a total of 100 models with different hyperparameter combinations for a maximum of 50 epochs, using the Asyne Hyperband Scheduler (grace period = 5, reduction factor = 3, brackets = 1) for aggressive early stopping.
I //2 d Classification We performed two stages of optimization. We first trained 100 models with different hyperparameter combinations for maximum of 50 epochs, using the Async Hyperband Scheduler (grace period 5, reduction factor = 3, brackets = 1) for aggressive early stopping. We then re-ran the optimization for 50 parameter combinations over maximum of 100 epochs, using the best five models from the first optimization as starting (seed) configurations. In this second stage, again used HyperOptSearch but changed the Asyne Hyperband Scheduler to use parameters (grace period = 10, reduction factor = 4, brackets = 1).
Ranking enantiomers by docking score. We trained total of 100 models with different hyperpa- rameter combinations for a maximum of 50 epochs, using the Async Hyperband Scheduler (grace period 5, reduction factor 3, brackets 1) for aggressive early stopping.
26

---

Published as a conference paper at ICLR 2022
Table 11: Hyperparameter search space for SphereNet
Hyperparameter
Search Space
hidden channels
[64, 128, 256]
out_channels
[16, 32, 64, 128, 256] [5.0, 10.0] [3,4.5]
cutoff nam_rayers int ent_emb.... size basis emb sizee.dddeee dist basis_ ses___mb_iii_e_____ size angle basis cat.emmb_siiee size_torsion out_emb_channed num num.spherical num num_radial
[32, 64, 128]
8
8
[64, 128, 256]
7
6
envelope exponent
5
num. before skip
num sum_after_stt after_skip camm_ouupu_lyyya Readout MLP Hidden Size
2
[32, 64, 128, 256]
Readout MLP of Hidden Layers
[1.2,3.4]
Learning Rate
log uniform (5e-5, 1e-2)
Batch Size
[16, 32, 64, 128, 256]
Table 12: Hyperparameter search space for DMPNN
Hyperparameter
Search Space
hidden size
[300, 600, 900, 1200]
depth dropout
[2, 3, 4, 5, 6]
[0. 0.2, 0.4, 0.6, 0.8, 1] log uniform (Ie-5, 1e-3)
Max Learning Rate
Batch Size
[25, 50, 100]
A.6.2 SPHERENET
Hyperparameters were also tuned for SphereNet on the / // d, and ranking enantiomers by docking score tasks, using the same optimization methodologies as when tuming ChiRo. The hy- perparameter search space is specified in Table IT] In each of these tasks, models with different hyperparameter combinations for a maximum of 50 epochs, using the Hyper- OptSearch algorithm and the Asyne Hyperband Scheduler (grace period = 5, reduction factor = 3, brackets = 1) for aggressive early stopping.
trained total of 50
A.6.3 DMPNN
We tune hyperparameters for the DMPNN on the / d classification and ranking enantiomers by docking score tasks using the original code provided by Patttanaa et al.,, which employs the Optuna hyperoptimization framework (Akiba et al. 2019). We train a total of 100 models using the Hyper- bandPruner algorithm prune unpromising trials and the CmalsSammler sample new trials. Note that we only optimize hyperparameters for the sum aggregation baseline model, and we extend these hyperparameterss to the TetraDMPNN models, following the work of Pattanaik et al. Because the TetraDMPNN models use permutation- aggregation function, the runtime of these models is much slower, which renders a full hyperparameter optimization infeasible.
27

---

Published as a conference paper at ICLR 2022
A.7 DATASETS
A.7.1 CONTRASTIVE LEARNING AND R/S CLASSIFICATION DATASETS
To create the datasets for the contrastive learning and R/S classification tasks, we first randomly selected 50 sdf files from the "10 conformers per compound" directory from the PubChem3D FTP site /ttp://fe htt:/fftp.ncke ncbi nlm.m.mmm....mmm.mmmm...m nih h. gov gwv hwoo///pbchemmmmm Compound__ 3D/] out of 6199 available. After extracting the conformers each sdf file, we filtered the data to only include conformers 2D graphs for which least 2 stereoisomers appear in the data, and where each of these stereoisomers has at least 2 conformers in the data. We further filtered the data to only include 2D graphs whose conformers contain at least torsion, recognized by RDKit. This dataset was separated into 80/20 partitions, with conformers corresponding to the same 2D graph being assigned to the same data partition. The larger (80%) subset used as-is for contrastive learning. The smaller (20%) subset was further filtered to only include pairs of enantiomers with only chiral center, excluding 2D graphs that contained other (cis/trans) stereoisomers of these enantiomer pairs in the dataset. Note that this extra step ensures that there are only two stereoisomers (which are enantiomers) per 2D graph. The resultant subset was used as the R/S classification dataset.
Figures F S the distributions of the number of conformers per stereoisomer and the number of stereoisomers per 2D graph in the full contrastive learning dataset. Figure shows the distribution of the number of conformers per enantiomer in the full R/S classification dataset. Table 13 indicates the distribution of R/S labels the R/S dataset.
The full contrastive learning dataset was split into 70/15/15 sets for training, validation, and testing, respectively, with conformers corresponding to the same 2D graphs being assigned to the same data partition. The training set contains 2,088,008 conformers of 418,922 stereoisomers of 180,426 distinct 2D graphs. The validation set contains 450,726 conformers of 89,786 stercoisomers of 38,658 distinct 2D graphs. The test set contains 448,017 conformers of 89,914 stereoisomers of 38,659 distinct 2D graphs.
similarly separated into 70/15/15 sets, with pairs of enantiomers be- ing assigned to the same data partition. The training set contains 326,865 conformers of 55,084 stereoisomers (27,542 pairs of enantiomers). The validation contains 70,099 conformers of 1,, 1,748 stereoisomers (5874 pairs of enantiomers). The test set contains 69,719 conformers of 11,680 stereoisomers (5840 pairs of enantiomers).
The full R/S dataset
10a
Stereolsomers
3.0
2.0
10
10
Conformers
Figure 5: Histogram of the number of conformers per stereoisomer in the contrastive learning dataset.
28

---

Published as a conference paper at ICLR 2022
10"
10a
# 2D Graphs
10a
of Stereolsomers
Figure 6: Histogram of the number of stercoisomers per 2D graph in the contrastive learning dataset.
10a
of Stereolsomers
2.0
10
31
12
13
15
16
17
18
19
20
of Conformers
Figure 7: Histogram of the number of conformers per enantiomer in the R/S classification dataset.
Table 13: Balance of R/S labels in the R/S classification dataset.
R / S Label
of Conformers # of Enantiomers
R 236222
39256
S
230461
39256
29

---

Published as a conference paper at ICLR 2022
A.7.2 1/ d CLASSIFICATION DATASET
Figure ennumerttss the data filtering steps used to extract and filter the / / classification dataset from the experimental Reaxys database, In Reaxys, we queried non-fragmented molecules with molecular weights I 564 which had optical rotatory power reported at 18-30 "C and a wavelength of 589 nm. We further filtered these molecules to include those with SMILES strings that were val- idated by RDKit, and those which contained only chiral center. Of these molecules, we removed duplicate entries if they were reported with different signs of optical rotation (which indicates an experimental measurement error). If duplicates had consistent signs of optical rotation, we ran- domly selected one entry. We then removed entries that did not have full stereochemistry specified in their SMILES strings, since we would later need to generate conformers for each molecule. We also removed molecules whose non-enantiomeric (e.g., cis/trans) stereoisomers also appeared in the dataset. We then checked if pairs of enantiomers were both reported in the dataset, and excluded such pairs if they were reported with the same sign of optical rotation (which indicates an experi- mental measurement error). For molecules whose opposite enantiomers were not in the dataset, we artificially generated the opposite enantiomer and assigned it the opposite sign of optical rotation. This ensures a balanced dataset. We then selected pairs of enantiomers which were reported with 2 95% enantiomeric excess order reduce the risk of experimental noise causing labeling errors. Lastly, we used RDKit to generate 5 conformers for each enantiomer in the filtered dataset. If RD- Kit failed to generate a conformer for any pair of enantiomers, both enantiomers in the pair were removed from the dataset.
Table reports the distribution of // labels the final dataset, which contains 150,380 conformers of 30,076 enantiomers (15,038 pairs of enantiomers). We split this dataset into 5 folds for cross- validation, randomly assigning each pair of enantiomers (with their conformers) to a test set in one of the five folds. For each fold, we randomly split the remaining (i.e., non-testing) pairs of enantiomers into 87.5/12.5 training/validationn sets. Note that this ensures each fold has 70/10/20 training/validationnesst splits, with pairs of enantiomers being assigned to the same data partition within each fold, and that each pair of enantiomers only appears in one test set across the five folds.
Reaxys Database
Select (vallid) molecules with chiral center only. MW < 564, containing organic atoms with optical rotatory power reported 18-30 "C and 589
chloroform solvent.
Remove duplicate entries if reported with different signs of optical rotation. Among valid duplicates, randomly select one entry.
Remove entries do not have full stereochemistry (e-g., cisitrans stereolsomerism) specified. Remove molecules whose non-enantiomeric stereoisomers also appear in the dataset.
enantiomar pairs both appear the dataset but are reported with the same sign of rotation, remove both onantiomers. a molecule's enantiomer
appear dataset, artificially generate the opposite enantiomer and assign it opposite sign optical rotation.
Select enantiomer onantiomeric excess.
are reported 96%
Generate conformers each enantiomer with RDKit. RDKit cannot generate conformer, remove the pair enantiomers.
Figure 8: Data extraction, filtering, and generation steps for the / d classification task.
30

---

Published as a conference paper at ICLR 2022
Table 14: Balance // d labels in dataset.
1/d Label # of Conformers
# of Enantiomers
75190
15038
75190
15038
Table 15; Frequency of R/S labels amongst enantiomers in the / / d dataset that rotate light in the positive (d) or negative (0) direction. The balance R/S labels indicates the lack of empirical correlation between these two classification schemes.
R
7316
7722
S
7722
7316
31

---

Published as a conference paper at ICLR 2022
A.7.3 RANKING ENANTIOMERS BY DOCKING SCORES
To create the dataset for the ranking enantiomers by their docking scores task, we randomly se- lected 750 sdf files from the "10 conformers per compound" directory from the PubChem3D FTP site Etp: ftp..cb................ n nlm. nih. gwv/ppbcceeeteeeeee Compound 3D/] out of the 6199 avail- able, but excluding the 50 sdf files used previously for the contrastive learning and R/S classification datasets. After extracting the conformers in each sdf file, we filtered the data to only include con- formers of 2D graphs for which only 2 stereoisomers (corresponding to pairs of enantiomers with chiral center) appeared in the dataset. As before, we only included enantiomers whose conform- contain at least torsion, recognized by RDKit. We then filtered enantiomers which have molecular weight < 225 and < rotatable bonds (as computed by RDKit). This step was designed to intentionally select small enantiomers which had few rotational degrees of freedom such that the docking simulations would be less stochastic. We then docked each pair enantiomers three times against the protein (PDB ID: 4JBV) in the docking box centered at [10, 16, 61] with (x.y.z) radii of [10, 14, 12]. In each docking simulation, we increased the exhaustiveness parameter to 24 to help reduce noise in the resultant docking scores. As a final control for stochasticity in the docking simu- lations, we also removed pairs of enantiomers for which one or both of the enantiomers had range of (top) docking scores 0.1 kcal/mol across the three simulation trials. Finally, in order to select pairs of enantiomers which exhibit meaningful differences in docking scores, we filtered the dataset to only include pairs of enantiomers which had a difference their best docking scores (across the three trials) of at least 0.3 kcal/mol. These enantiomers and their PubChem3D conformers were used as the final dataset. Figure visualizes the full filtering procedure.
Figure TO plots the distribution of the number of conformers per enantiomer in the docking dataset. Figure II plots the distribution of the difference in docking scores between pairs of enantiomers in the docking dataset. We split the full dataset into 70/15/15 training/valian vaddiin//alidaaionnesst sets, assigning pairs of enantiomers (with their conformers) to the same data partition. The training set contains 234,622 conformers of 48,384 enantiomers (24,192 pairs of enantiomers). The validation set tains 49,878 conformers of 10,368 enantiomers (5 84 pairs of enantiomers). The test set contains 50,571 conformers of 10,368 enantiomers (5184 pairs of enantiomers).
PubChem3D Conformers
pairs enantiomers only chiral and which have conformers each
Select enantiomers with M,,
Dock each enantiomer and select pairs where both anantiomers have a range of scores: 5 0.1 kcal/mol
Select enantiomer pairs which difference docking
kcalimol
Figure 9: Data generation and filtering steps for the ranking enantiomers by docking scores task.
32

---

Published as a conference paper at ICLR 2022
10a
of Stereolsomerg
10"
2.0
109
10
12 13 14 of Conformers
22
Figure 10: Histogram of the number of conformers per enantiomer in the docking dataset.
of Palts of Enantormers
20200
00 03
1. 3. 24 27 Difference Docking Scores (kcalmmol
30
Figure 11: Distribution of the differences in (best) docking scores between pairs of enantiomers in the docking dataset.
33

---

Published as a conference paper at ICLR 2022
A.8 ADDITIONAL ABLATIONS
It is not strictly necessary that ChiRo learn the weight coefficients (0w) in order to distinguish enantiomers. ChiRo can still learn chiral representations and preserve invariance to internal bond rotations if each C3j is not the case for the phase shifts Piayy- which must be learned for good performance. To demon- strate this, we perform additional ablations where the feed-forward network Se is not trained, but rather keep its initial random parameters. We perform the same analysis for Jop to also evaluate whether ChiRo needs to learn the phase shifts Piayyj or if these can also be generated by a random network. Table [o]repors the results of not training these MLPs on the otherwise unchanged ChIRo on the 1/ d classification task and ranking enantiomers by docking scores task. Overall, learning the parameters in fa leads to only small (if any) performance gains versus keeping the randomly initial- ized parameters. On the other hand, learning the parameters in fúp leads to considerable performance gains versus not learning fe. This emphasizes that in our case, relying on random noise to break symmetry between embeddings of enantiomers is insufficient to learn expressive chiral representa- tions. Rather, ChIRo best models the effects of chirality by learning to break the symmetry through learned phase shifts in task-specific manner.
Cj
(xx) is generated by network with random, untrained parameters. However, this
Table 16: Effects of not learning the parameters in fc and fo ChIRo's performance the / classification and the ranking enantiomers by docking score tasks. For the original, unablated ChiRo (first row), mean accuracy and standard deviations the test sets reported across folds (1 / d) or three repeated trials (enantiomer ranking). To better account for the impact of random network initializations, for the ablated models we report the mean accuracy and standard deviations on the test sets across 3 repeated trials (enantiomer ranking) or the mean, fold-averaged accuracy (e.g., mean of means) and standard error of this mean across three repeated 5-fold CV trials (1 (1/6 classification).
Model Components
Accuracy (%)
Learned fe Learned f&p
1/d
Enantiomer Ranking
79.3 + 0.4
72.0 + 0.5
79.4 + 0.3
71.7 : 0.2
X
53.7 + 1.2
68.4 + 0.7
X
X
50.8 + 0.8
66.8 + 0.3
A.9 ADDITIONAL EVALUATION OF RANKING ENANTIOMERS BY DOCKING SCORES
Because some enantiomers have larger differences in their ground truth docking scores than other enantiomers, a single ranking accuracy metric may not fully describe the ability of the models to learn the enantioselectivity of the protein pocket. To evaluate model performance more thoroughly, we compute ranking accuracies on various subsets of the test set. Figure 12 plots the ranking accu- racies of ChiRo, SphereNet, DMINN (concatenate) with chiral tags, and DMPNN with chiral tags when evaluated on subsets of the test data where the difference in ground truth docking scores (upper left) greater equal margin, (upper right) less than equal to margin, or (bottom) exactly equal to a margin. The plots suggest that while ChIRo is superior in correctly ranking enan- tiomers which have relatively small differences in their true docking scores, the differences between each model become less distinct when ranking enantiomers with larger differences in their ground- truth docking scores. This may be due to the fact that the docking dataset is imbalanced, skewed heavily toward enantiomers that have smaller differences in their true docking scores (Figure II).
34

---

Published as a conference paper at ICLR 2022
CHRR
with
rendom guess
rendom guess
Panking Accuraty
0.8
0.6
Ronking
65
0.5
03
04
05
0.6
0.7
0.8
0.4
1.
1.1
1.2
1.3
14
1.5
15
Margin (kcal/mal)
Margin (kca//ma))
ChiRo
Tetra-DWeNN (cencati with
0.9
Spheretiet
Ranking Accuracy
0.7
0.6
03
0.4
05
0.6
677
0.8
09
3.0
3.1
32
33 244 355
Margin (kcal/mol)
Figure 12: Ranking accuracy of the predicted conformer-averaged docking scores when evaluated on subsections of the test in which the ground truth difference in best docking scores between enantiomers is (upper left) greater than or equal to the specified margin, (upper right) less than or equal the specified margin, or (bottom) equal to the specified margin. Error bars for the models represent standard deviations in ranking accuracy across three random training/Inferenee trials. Error bars for the random baseline correspond 0.5, N), where N is the number of enantiomer pairs in the subset.
standard deviation of a binomial distribution B(p =
35

---

Published as a conference paper at ICLR 2022
A.10 ANALYZING
3D GNNS WITHOUT
INVARIANCE
that an SE(3)-invariant 3D GNN without InterRoto- invariance, such SphereNet, will get confused between differences in molecular chirality (e.g.. inverted chiral centers) with differences in conformational structure (e.g., bond rotations) when learning chiral-dependent functions. To support this hypothesis with quantitative evidence, Fig- ure 13 shows the distribution of the fraction of conformers per enantiomer in the test sets of the 1/d classification task which were predicted have the same sign of optical rotation by SphereNet and ChiRo, irrespective of the actual accuracy of the predicted class labels. SphereNet predicts the same label for each of the RDKit-generated conformers per enantiomer for only ~32% of the enantiomers in the test sets, compared to ~93% for ChIRo. When only trained 1 conformer per enantiomer, SphereNet's labeling consistency drops to 14%, whereas ChIRo's labeling consis- tency remains roughly the same at ~922% ChiRo's dramatically increased consistency in labeling conformers sharing the same chiral molecular identity compared to SphereNet is direct result of Note that ChIRo's consistency is not perfectly 100% because the
The qualitative results shown in
ChiRo's coerRoto-invarianee RDKit-generated conformers have conformational differences not associated with simple bond ro- tations. For instance, two conformers can slightly differ their bond distances and bond angles, perturbations to which ChIRo is not invariant.
Moreover, manually rotating bonds near the chiral center (similar the bottom in Figure 3 causes SphereNet to get confused when predicting signs of optical rotation for these rotated con- formers, even for enantiomers (in the test-set) whose conformers were all originally classified cor- rectly by SphereNet (Figure 14). Since ChiRo is invariant to such bond rotations (Figure 3), it is not susceptible to this particular type of confusion, and therefore has increased ability generalize to unseen chiral conformers.
SphereNet
ChiRo
1.0
1.0
0.9-
0.9
Fraction Enantiomers
Fraction Enantiomers
0.8
0.6-
0.6
0.5
05
0.4
0.4
03
0.1
0.1
0.0
0.0
3/5
4/5
5/5
3/5
4/5
Fraction Conformers
Labels
Fraction Conformers
Same Predicted Labels
SphereNet Conformer
ChiRo 1 Conformer
Fraction Enantioners
0.8-
Enantioners
0.8
0.7-
0.7
0.6
05
0.4
0.4
03
03
02
02
0.0
3/5
4/5
5/5
3/5
4/5
5/5
Fraction of Conformers with the Same Predicted Labels
Fraction of Conformers with the Same Predicted Labels
Figure 13: Distribution of the fraction of conformers per enantiomer predicted to have the same class label for the l/d classification task across the test sets in each of the five folds, for both SphereNet and ChiRo when trained on all conformers in the training set (top row) on single conformer per enantiomer (bottom row). Note that for the 1/ classification task, there are exactly five conformers per enantiomer in the test splits of each fold.
36

---

Published as a conference paper at ICLR 2022
120
4000
6000
100
110
40
3718
782
3000
6667
1773
5000
The Label
Thu Label
4000
2000
127
149
4351
1764
6876
2000
1000
Prodicted
Presicted Label
Prodicted Label
OH *
HN
*
Figure 14: Confusion matrices indicating SphereNet's (mis)classificatiion of conformers three pairs of enantiomers in the 1/d classification task's test (of the first fold) whose highlighted bonds are each rotated in increments of 12 (left and center) or 30 (right). These three pairs of enantiomers were specifically chosen for this confusion analysis because SphereNet correctly classifies all of their original (non-rotated) conformers.
37


Reviews:
[{"review_id": "jT6-5SX8wP6", "reviewer": "Reviewer_pDCs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the model sounds reasonable and the results on multiple tasks are promising. I have several minor questions about empirical studies.\n- “ Although SE(3)-invariant 3D GNNs, such as the recently proposed SphereNet (Liu et al., 2021) and GemNet (Klicpera et al., 2021), can\nin theory learn chirality, their expressivity in this setting has not been explored.” Give that both SphereNet and GemNet can learn chirality, why only SphereNet is compared in experiments? It is better to compare with both methods.\n- “ We emphasize that we view this classification task as necessary but not suf￾ficient to demonstrate that a model can learn a meaning￾ful representation of chiral molecules.” Can you establish a bit more about this? Why not sufficient? \n- SphereNet is comparable to Chiro in Table 1, but much worse in Table 2. Is this mainly because the differences between two tasks?  I’d like to see some analysis here. Why is Chiro much better in the second task? It would be great if you can show several case studies to support Chiro.", "summary_of_the_paper": "This work focuses on representation learning for molecules with stereochemistry and designs an SE(3)-invariant model that processes torsion angles of a 3D molecular conformer. In particular, the authors explicitly model conformational flexibility by integrating a novel\ntype of invariance to rotations about internal molecular bonds into the architecture, mitigating the need for multi-conformer data augmentation. ", "summary_of_the_review": "Overall it is a nice work, but I’d like to see some deep analysis on experiments, e.g., why the proposed model is better, and in which cases, etc. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UIxLPvdCQc", "reviewer": "Reviewer_KqtQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Novelty]\n\nFrom my point of view, if a molecular representation model wants to achieve good expressivity, it should:\n\n1. perceive more specific molecular properties, such as chirality utilized to separate enantiomers;\n2. be invariant to abundant information for better generalization.\n\nTraditional models, such as 2D-GNNs, E(3)-Invariant ones and SE(3)-Invariant ones, can't meet the requirements above , so their expressivity is limited. Contrastively, ChIRo well tackles these two issues by defining and implementing (SE(3)+InterRoto)-Invariance and is proved to be more expressive.\n\n[Theory]\n\nChIRo uses a series of trigonometric functions to encode the torsion angles. To demonstrate that such encoding is invariant to bond torsion but sensitive to chirality, this paper gives a proof in Appendix. This proof is clear but Figure 2 makes me confused before reading the proof. (This will be elaborated in Cons.[Illustration].) Despite this flaw, the theoretical foundation of this paper is very solid.\n\n[Reproduction]\n\nIn Section 3, it gives a very clear description on the structure of ChIRo. And implementation details are presented in Appendix.\n\n[Representation]\n\nThis paper is very clear and easy to understand.\n\n\n\nCons:\n\n[Model Design]\n\nAs far as we know, most molecular representation models considering 3D structure use the spatial information when passing messages among the atoms and bonds, such as 3DGCN, DimeNet and PhysChem(https://openreview.net/forum?id=Uxi7X1EqywV). However, ChIRo consists of a 2D-GNN and an individual NN to embedding D/Phi/Psi found in molecular conformation. We think the way ConfAE (the method ChIRo follow) processes the spatial information will lose graph topology, because D/Phi/Psi are detached from message passing and treated as \"bags of words\". This reason of adopting such structure is \"to be invariant to the order of presenting the atoms\", but actually the output of MPNN is invariant to input atom order if only order-independent aggregation functions are used (such as sum/mean pooling). Therefore, we think the structure design of ChIRo can be further improved.\n\n[Illustration]\n\nIn Figure 2, the contents in 3rd and 4th columns are confusing before the readers fully understand the method and its demonstration (e.g. what's the relationship between the solid sinusoids and dotted ones, what does \"r\" mean and so on). We hope the illustrations can be better arranged.\n\nIn Figure 3, the illustrations of \"Original OMEGA Conformers\" and \"Original+Reflected Conformers\" are almost the same, we can't figure out what the difference is between them.\n\n[Experiment]\n\nAlthough ChIRo is evaluated in 4 tasks, its application value remains suspicious.\n\nFirstly, in the first 3 tasks (Contrastive Learning to Distinguish Stereoisomers, Classifying Tetrahedral Chiral Centers as R/S and Predicting Enantiomers’ Signs of Optical Rotation), the targets are strongly correlated to the chirality of enantiomers. So, these tasks seem more like sensitivity tests of \"if ChIRo can actually perceive chirality information\" rather than \"if ChIRo can handle existing tasks\". \n\nSecondly, ChIRo is labeled as a \"molecular representation model\", so it's necessary to demonstrate its effectiveness on various popular tasks on molecules, such as QM7/8/9 with given conformations (see MoleculeNet(https://pubs.rsc.org/en/content/articlelanding/2018/sc/c7sc02664a)), or other molecular property prediction tasks (FreeSolv, TOX21, ...) assisted with RDKit-generated conformations (see HamNet(https://openreview.net/forum?id=q-cnWaaoUTH)). Besides the 3 tasks mentioned in last paragraph, this paper only evaluate ChIRo on one dataset of molecule-protein binding. Therefore, we think ChIRo has not yet been proved to be a SOTA molecular representation model.\n\nThirdly, in Ablation Study, we see \"adding chiral tags to our unablated model does not improve performance\". We think that's caused by overfitting to abundant tags (as chirality information has been well embedded thru torsion encoder), and we hope a validation of it can be given in this paper rather than just a statement of this abnormal phenomenon.", "summary_of_the_paper": "\nThis paper proposes a 3D-GNN molecular representation model  (ChIRo) which:\n\n1. can perceive chirality information to improve expressivity;\n2. is invariant to bond rotations and mitigates the need for data augmentation.", "summary_of_the_review": "This paper proposes ChIRo to improve the expressivity in molecular representation by well tackling chirality. It has a solid theoretical foundation and a clear and relatively well-designed model structure. Although the experiment is not yet sufficient to demonstrate its application value, we believe that ChIRo's effectiveness can be claimed after being evaluated on diverse tasks on molecular representation, such as molecular property prediction datasets in MoleculeNet.\n\nIn summary, it's a good piece of work but needs some improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MMFdhBiRYG6", "reviewer": "Reviewer_Viqv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I liked the paper's idea. It is well written, easy to follow, and the provided list of related work was relevant. Even though technically it might be incremental, it might be able to provide an interesting insight and informative approach to see. \n\nHowever, a big concern is: it is unclear which parts are technically novel, and actually brings any new flexibility because the paper combines several existing ideas. I understand providing a new effective combination of existing ideas is also an important contribution, but for this paper, I'm still not sure on this point of the significance of the provided combinations. The current form just seemed a minor straightforward modification by combining existing GNN design patterns to achieve SE(3) invariance. It is also unclear why other referred models such as SphereNet (Liu+, 2021), GemNet (Klicpera+ 2021), GeoMol (Ganea+ 2021) are insufficient for this targeted problem.\n\nTechnically speaking, the components are (1) 2D graph encoders, (2) bond distance and bond angle encoders, and (3) torsion encoders. \n\n- (1) are a standard design pattern by EdgeConv and GAT layers\n- (2) largely relies on the idea by Winter et al (2021)\n- (3) largely relies on the idea by GeoMol (Ganea+ 2021) that considers torsional geometric handlings. \n\n(1) are standard design patterns by EdgeConv + GAT layers, and (2) follows Winter et al (2021) to drop out any information on atom-atom distances and the like. To handle the chirality, the most important aspect would be torsion angles of (3), but comparisons to the original GeoMol (Ganea+ 2021) are presented only in \"A.8 Additional Ablations\" with a confusing statement of \"It is not strictly necessary that ChIRo learn the weight coefficients or phase shifts in order to distinguish enantiomers\".\n\nSo, the presented model seemed a minor modification of existing ideas to achieve SE(3)-invariance. The novelty is somewhat still unclear. Several natural questions are\n\n- For example, can we include GeoMol (Ganea+ 2021) in Table 2?\n- If (2) is from Winter et al (2021), and (3) is mainly from Ganea+ (2021), can we consider a direct combination of these two as a basline in all experiments? OR this work is exactly doing this...? If this won't work, why? \n- Achieving SE(3)-invariance itself is the purpose? I understood SchNet and DimeNet will fail, but cited existing models such as SphereNet (Liu+, 2021), GemNet (Klicpera+ 2021), GeoMol (Ganea+ 2021) are also insufficient for some reasons? How they are different from the presented work?\n\n\n*Remarks*: For ML practitioners who try to apply GNNs to life science problems or chemistry problems, the chirality of molecules is very important issue but also quite a big technical hurdle. When a molecule takes a 3D shape, we have a degree of freedom to rotate every C-C single bond. However, we have enantiomers, that is, non-superposable isomers by these operations but indistinguishable by 2D chemical formulas. These mirror-image molecules are very important both in life science (many drugs are enantiomers because interacting molecules in living cells also have chiral centers) and chemistry (to produce such drug compounds efficiently we need asymmetric synthesis producing only one of such enantiomers). \n", "summary_of_the_paper": "This paper presents a novel SE(3)-invariant GNN model for predicting 3D geometry-dependent physicochemical properties of molecules. In particular, this method focuses on an important issue of how to handle the chirality of molecules in molecular GNNs. When a molecule takes a 3D shape, we have a degree of freedom to rotate every C-C single bond. To distinguish enantiomers (mirror-image molecule pairs), we also need to see whether two molecule pairs are superposable by these single-bond rotation operations. The developed method proposed a torsion-angle encoder having 1) an invariance to rotations about internal molecular bonds and 2) the ability to learn molecular chirality. Several empirical results are also reported.\n", "summary_of_the_review": "The idea and the focused problem of the paper is very interesting and can be impactful. However, technically speaking, the current form seemed quite incremental, and a minor modification of existing ideas to achieve SE(3)-invariance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2TiGcPNkxAy", "reviewer": "Reviewer_3jRF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n**Strength**\n\n1. Good efforts towards incorporating inductive bias from science perspectives into model designs, have a quite lot insights.\n\n**Weakness**\n\n1. Rationale of the problem setting. Base on my experiments (using the models and data from [1]), using \"chiral atom tags\" with the vanilla GIN model will reach 100% test accuracy. So it would be nice to elaborate why \"omit chiral atom tags and bond stereochemistry tags\"(page 4) is a rational setting.\n2. For the last part of \"Torsion Encoder\", I understand that it is complete to include necessary independent torsion angles, but how would the model perform if we feed it more, redundant and self-consistent features? It would be helpful to check on this :)\n\n\n**Comments**\n\n1. Why the scales of the embeddings are quite different in Figure 3 (1e^2 to 1e^8)?\n2. I feel the term \"contrastive learning\" more close to \"instance discriminative\" instead of triplet loss, it is a bit appropritate forr the term usage. If you have some literatures to back this, please let me know.\n\n**Open Questions**\n\nNot addressing the following comments won't have negative effects on my rating. But it would be nice if the authors can make a few clarifications, which also helps with my understandings!\n\n1. Have you tried SE(3)-equivariant models (Fuchs et al., 2020; Thomas et al., 2018)? It is a bit sceptical that they can actually distinguish this geometry chirality (i.e, considering coupled torsions in the message updates).\n2. Can it be incorporated into other models as a simple add-on/module?\n3. Do improving model's capacity on chirality identification have positive impacts on the performance on benchmarks such as QM9?\n\n**Reference**:\n\n[1] \"Message passing networks for molecules with tetrahedral chirality\"\n[2] ", "summary_of_the_paper": "The paper focuses on improving the capacity of GNN models, using chirality identification as the case study. As an important character to represent the geometry in molecules, chirality has a fundamental impact on the molecule properties and downstream applications. It is a major extension of [1]. The distinguishing features are mainly based on the sinusoid transformations of torsion angles, which are invariant to the bond rotations.", "summary_of_the_review": "This paper is well written and focuses on a simple yet important problem. I tend to accept and am happy to adjust my assessment based on the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents ChIRo, a method that incorporates 3D torsion angles of a molecular conformer to specifically handle chirality. Specifically ChIRo uses trigonometric functions to encode the torsion angles, which are invariant to bond torsion but sensitive to chirality, thus capable of distinguishing between enantiomers.  Overall, although not groundbreaking, we found the idea presented in the paper to be sufficiently novel and its ability to handle chirality to be of significant practical values.
Decision: Accept (Poster)