Learning from Protein Structure with Geometric Vector Perceptrons
Authors: Bowen Jing
Stephan Eismann
Patricia Suriana
Raphael John Lamarre Townshend
Ron Dror
Venue: ICLR 2021 Spotlight (2021)
Keywords: structural biology
graph neural networks
proteins
geometric deep learning

Abstract: Learning on 3D structures of large biomolecules is emerging as a distinct area in machine learning, but there has yet to emerge a unifying network architecture that simultaneously leverages the geometric and relational aspects of the problem domain. To address this gap, we introduce geometric vector perceptrons, which extend standard dense layers to operate on collections of Euclidean vectors. Graph neural networks equipped with such layers are able to perform both geometric and relational reasoning on efficient representations of macromolecules. We demonstrate our approach on two important problems in learning from protein structure: model quality assessment and computational protein design. Our approach improves over existing classes of architectures on both problems, including state-of-the-art convolutional neural networks and graph neural networks. We release our code at https://github.com/drorlab/gvp.

Published as a conference paper at ICLR 2021
LEARNING FROM PROTEIN STRUCTURE WITH GEOMETRIC VECTOR PERCEPTRONS
Bowen Jing. Stephan Eismann," Patricia Suriana, Raphael J.L. Townshend, Ron O. Dron Stanford University (bjing,
candrarrecs.stanfordd.. psuriana, raphael, condrer)@cs stan ford. edu
ABSTRACT
Learning on 3D structures of large biomolecules is emerging a distinct area in machine learning, but there has yet to emerge unifying network architecture that simultaneously leverages the geometric and relational aspects of the problem domain. To address this gap, we introduce geometric vector perceptrons, which extend standard dense layers to operate on collections of Euclidean vectors. Graph neural networks equipped with such layers are able to perform both geometric and relational reasoning on efficient representations of macromolccules. We demon- strate our approach on two important problems in learning from protein structure: model quality assessment and computational protein design. Our approach im- proves over existing classes of architectures the-art convolutional neural networks and graph neural networks. We release our code at /https:///gtthiie wtt.tuu.occmmaaaaaaaeaaeeeeeeeee com www.oo//ddooaaoeoeooooooooeoooee lab/
both problems, including state-of-
INTRODUCTION
Many efforts in structural biology aim to predict, or derive insights from, the structure of a macro- molecule (such a protein, RNA, DNA), represented set of positions associated with atoms or groups of atoms in 3D Euclidean space. These problems can often be framed as functions map- ping the input domain of structures to some property of interest-for example, predicting the qual- ity of a structural model or determining whether two molecules will bind in particular geometry. Thanks to their importance and difficulty, such problems, which broadly refer as learning from structure. have recently developed into an exciting and promising application area for deep learning
(Graves al. 2020. Ingraham et al. 2019 Pereira et al. 2016 Townshend al. 2019 Won al., 2019).
Successful applications of deep learning are often driven by techniques that leverage the problem structure of the domain-forr example, convolutions in computer vision (Cohen & Shashua 2017) and attention in natural language processing (Vaswani et al. 2017). What are the relevant consider- ations in the domain of learning from structure? Using proteins the most common example, have on the one hand the arrangement and orientation of the amino acid residues in space, which govern the dynamics and function of the molecule (Berg al. 2002). On the other hand, proteins also possess relational structure in terms of their amino-acid sequence and the residue-residue inter- actions that mediate the aforementioned protein properties (Hammes-Schiffer & Benkovic 2006). We refer to these as the geometric and relational aspects of the problem domain, respectively.
Recent state-of-the-art methods for learning from structure leverage one of these two aspects. Com- monly, such methods employ either graph neural networks (GNNs), which are expressive in terms of relational reasoning (Battaglia et al. (0188). or convolutional neural networks (CNNs), which op- erate directly on the geometry of the structure. Here, we present a unifying architecture that bridges these two families of methods to leverage both aspects of the problem domain.
We do so by introducing geometric vector perceptrons (GVPs), drop-in replacement for standard multi-layer perceptrons (MLPs) in aggregation and feed-forward layers of GNNs. GVPs operate directly both scalar and geometric features-features that transform a vector under rotation of spatial coordinates. GVPs therefore allow for the embedding of geometric information at nodes and
"qqual contribution

---

Published as a conference paper at ICLR 2021
edges without reducing such information to scalars that may not fully capture complex geometry. We postulate that our approach makes easier for a GNN to learn functions whose significant features are both geometric and relational.
Our method (GVP-GNN) can be applied to any problem where the imput domain is a structure of single macromolecule of molecules bound one another. In this work, specifically demonstrate our approach on two problems connected to protein structure: computational protein design and model quality assessment. Computational protein design (CPD) is the conceptual inverse of protein structure prediction, aiming to infer an amino acid sequence that will fold into given structure. Model quality assessment (MQA) aims select the best structural model of protein from large pool of candidate structures and is an important step in structure prediction (Cheng et al. 2019). Our method outperforms existing methods on both tasks.
2 RELATED WORK
ML methods for learning from protein structure largely fall into one of three types, operating on sequential, voxelized, or graph-structured representations of proteins. We briefly discuss each type and introduce state-of-the-art examples for MQA and CPD to set the stage for our experiments later.
Sequential representations In traditional models of learning from protein structure, each amino acid is represented as feature vector using hand-crafted representations of the 3D structural en- vironment. These representations include residue contacts (Olechnovič & Venclovas 2017). ori- entations or positions collectively projected to local coordinates (Karasikov et al.) 2019), physics- inspired energy terms (O (O'Conlell al., 2018 Uziela et al. 2017), context-free grammars of protein topology (Greener et al. 2018). The structure is then viewed as a sequence or collection of such features which can be fed into ID convolutional network, RNN, or dense feedforward network. Although these methods only indirectly represent the full 3D structure of the protein, number them, such ProQ4 (Hurtado al., 2018), VoroMQA (Olechnovic Venclovas 2017). and SBROD (Karasikov et al. 2019), are competitive in assessments of MQA.
Voxelized representations In lieu of hand-crafted representations of structure, 3D convolutional neural networks (CNNs) can operate directly on the positions of atoms in space, encoded as oceu- pancy maps in a voxelized 3D volume. The hierarchical convolutions of such networks are easily compatible with the detection of structural motifs, binding pockets, and the specific shapes of other important structural features, leveraging the geometric aspect of the domain. A number of CPD methods (Anand et al. 2020 Zhang et al., 2019 Shroff et al. 2019) and the MQA methods 3DCNN (Derevyanko et al. 2018) and Ornate (Pagès et al. 2019) exemplify the power of this approach.
Graph-structured representations A protein structure also be represented graph over amino acid nodes, reducing the challenge of representing a collective structural neigh- borhood in single feature vector that of representing individual edges. Graph neural networks (GNNs) can then perform complex relational reasoning over structures (Battaglia et af., 2018) example, identifying key relationships among amino acids, flexible structural motifs described as a connectivity pattern rather than a rigid shape. Recent state-of-the-art GNNs include Structured Transformer [Ingraham al., 2019 on CPD, ProteinSolver (Strokach et al. (020) on CPD and mu- tation stability prediction, and GraphQA (Baldassarre et al. 2020) on MQA. These methods vary in their representation of geometry: while some, such as ProteinSolver and GraphQA, represent edges as function of their length, others, such as Structured Transformer, indirectly encode the 3D geometry of the proximity graph terms of relative orientations and other scalar features.
proximity
3 METHODS
Our architecture seeks to combine the strengths of CNN and GNN methods in learning from biomolecular structure by improving the latter's 's ability to reason geometrically. The GNNs de- scribed in the previous section encode the 3D geometry of the protein by encoding vector features (such as node orientations and edge directions) in terms of rotation- intation-invariant scalars, often by defin- ing local coordinate system each node. We instead propose that these features be directly
2

---

Published as a conference paper at ICLR 2021
A
Geometric Vector Perceptron
protein design
VER MR3
V.E
hx3
V
VER
VER
sequence
TIGOMPATSS
W
W
o l
predkt
Ourput 6M
Input (s.)
backbone structure
vector channel
I'lla
s_E
S'ERR
quality assessment
Score
$ S,R
sequence
W
predict
SER"
scalar channel
Figure 1: (A) Schematic of the geometric vector perceptron illustrating Algorithm Given tuple of scalar and vector input features (s, V), the perceptron computes an updated tuple (s', V') s' is function of both and V. (B) Illustration of the structure- based prediction tasks. In computational protein design (top), the goal is to predict an amino acid sequence that would fold into given protein backbone structure. Individual atoms are represented as colored spheres. In model quality assessment (bottom), the goal is to predict the quality score of candidate structure, which measures the similarity of the candidate with respect the experimentally determined structure (in gray).
represented geometric spatial coordinates- at steps of graph propagation.
in IR which transform appropriately under a change of
This conceptual shift has two important ramifications. First, the input representation is more effi- cient: instead of encoding the orientation a node by its relative orientation with all of its neighbors, only have to represent one absolute orientation per node. Second, it standardizes global coor- dinate system across the entire structure, which allows geometric features to be directly propagated without transforming between local coordinates. For example, representations of arbitrary positions in space-including points that are not themselves nodes-can be easily propagated across the graph by Euclidean vector addition. We postulate this allows the GNN more easily access global geo- metric properties of the structure. The key challenge with this representation, however, is to perform graph propagation in a way that simultancously preserves the full expressive power of the original GNN while maintaining the rotation invariance provided by the scalar representations. We do so by introducing a new module, the geometric vector perceptron, replace dense layers GNN.
3.1 GEOMETRIC VECTOR PERCEPTRONS
The geometric vector perceptron is a simple module for learning vector-valued and valler- functions over geometric vectors and scalars. That is, given tuple (s. V) of sealar features R and vector features V E R=x3 we compute new features (s', V') E Iim. x Rux3 The computation illustrated in Figure and formally described in Algorithm
At its core, the GVP consists of two separate linear transformations W WA for the scalar and vector features, followed by nonlinearities a, at However, before the scalar features are trans- formed, we concatenate the Lg norm of the transformed vector features VA this allows us to extract invttiinnivvaiintt information from the input vectors V. An additional linear transformation W, is inserted just before the vector nonlinearity to control the output dimensionality independently of the number of norms extracted.
The GVP is conceptually simple, yet provably possesses the desired properties of invari- ance/equivariance and expressiveness. First, the vector and scalar outputs of the GVP are equivariant and invariant, respectively, with respect to arbitrary composition It of rotations and reflections in 3D Euclidean space i.e., GVP(s, V) = (s', V') then
GVP(s, R(V)) (5,R(("))
3

---

Published as a conference paper at ICLR 2021
Algorithm 1 Geometric vector perceptron
Input: Scalar and vector features (s, V) E R x R2x3
Output: Scalar and vector features (s', V*) E R x Rax3
h max (v, p1)
GVP:
Va + Wr
1R1x3
V₁ f- Vn
E R2x3
[Vall2 (row-wise)
E R
Va < [Vul2 (row-wise)
R
Sh+n f concat (Sh.
E IR 4++
Sm Sn + b
E
R
s' f o (Sm) E 1mm
V t o+ (Vn) @ Vu (row-wise multiplication)
e IRxx3
return (s', V")
This is due to the fact that the only operations on vector-valued inputs are scalar multiplication, linear combination, and the L2 norm We include formal proof in Appendix A
In addition, the GVP architecture can approximate any continuous rotation- and reflection-invarianl scalar-valucd function of V. More precisely, let G, be GVP defined with n, 0-that is, one which transforms vector features to scalar features. Then for any function f [R*x3 + R invariant with respect to rotations and reflections in 3D, there exists functional form G, able to e-approximate f, given mild assumptions.
Theorem. Let R describe arbitrary rotation and/or reflection in RS, For 2 3 let Q C R2xx be the set of all V [V1, .... vuj²" E Rux3 such that V1. V2, V3 are linearly independent and O < 1ill2 & b for all i and some finite b > O. Then for any continuous F Q + IR. such that F (((())
F(V) and for any c > 0, there exists a form f(V) w G.G such that
[F(V) f(V)] for all V E Q!"
We include formal proof in Appendix A approximate similarly-defing functions over the full input domain R" x R=*3
As a corollary, GVP with nonzero is also able to
In addition to the GVP layer itself, random (as opposed to coordinates within vector channels). We also introduce layer normalization for the vector features
version of dropout that drops entire vector channels at
V V V-V/V I
E Æ
(2)
That is, scale the row vectors of V such that their root-mean mean-square norm is one. This vector layer norm has no trainable parameters, but we continue to use normal layer normalization on scalar channels with trainable parameters 7,
We study our hypothesis that GVPs augment the geometric reasoning ability of GNNs on a synthetic dataset (Appendix B). The synthetic dataset allows to control the function underlying the ground- truth label in order to explicitly separate geometric and relational aspects in different tasks. The GVP-augmented GNN (or GVP-GNN) matches CNN on a geometric task and standard GNN relational task. However, when we combine the two tasks in one objective, the GVP-GNN does significantly better than either GNN or a CNN.
3.2 REPRESENTATIONS OF PROTEINS
The main empirical validation of our architecture is its performance putational protein design (CPD) and model quality assessment (MQA). These tasks, as illustrated in Figure B and described in detail in Section 4 are complementary in that one (CPD) predicts property for each amino acid while the other (MQA) predicts global property.
real-world tasks: com-
Thhe nonlinearity at is scaling by o² applied the Ta norm.
4

---

Published as a conference paper at ICLR 2021
We represent protein structure imput proximity graph with minimal number of scalar and vector features to specify the 3D structure of the molecule. A protein structure is a sequence of amino acids, where each amino acid consists of four backbone atom and set of sidechain atoms located in 3D Fuclidean space. We represent only the backbone because the sidechains are unknown in CPD, and our MQA benchmark corresponds to the assessment of backhone structure only.
Let X; be the position of atom X in the ith amino acid (e.g. N₂ is the position of the nitrogen atom in the ith amino acid). We represent backbone structure graph 9 (V,E) where each node b, E V corresponds to an amino acid and has embedding h with the following features:
Scalar features (sin, cos J {ő, eh, w,w,, where 6, v. are the dihedral angles computed from C,-1, N₁, Cos, C₁ and N;+1-
The forward and reverse unit vectors in the directions of Cos+1 Cos, and Ca Coi. respectively.
The unit vector the imputed direction of C33, - Ca This is computed by assuming tetrahedral geometry and normalizing
c) c/12 -2 (n+ cl22
where n = N; - Co: and C. Ca:; This vector, along with the forward and reverse unit vectors, unambiguously define the orientation of each amino acid residue.
A one-hot representation of amino acid identity, when available.
The set of edges is & {ee--)ees for all i.j where D is among the k: nearest neighbors of v, as measured by the distance between their Co atoms. Each edge has embedding h ()-+)) with the following features:
The unit vector in the direction of Cay Ca..
The encoding of the distance ||Cay Caa ||2 in terms of Gaussian radial basis functions D
A sinusoidal encoding of j i as described in Vaswani et al. (007)) representing distance along the backbone.
In our notation, each feature vector a concatenation of scalar and vector features as described above. Collectively, these features are sufficient for complete description of the protein backbone.
3.3 NETWORK ARCHITECTURE
Our architecture (GVP-GNN) leverages message passing (Gilmer et al. 2017) in which messages from neighboring nodes and edges are used to update node embeddings at each graph propagation step. More explicitly, the architecture takes as input the protein graph defined above and performs graph propagation steps according to:
(J-+))
9 (concat (n,",, (0p,nb'- m)/-+))
(3)
h (i)
+- LayerNorm he
Dropout
Σ
(4)
Here, g is sequence of three GVPs. h." and h. (f+1) are the embeddings of the node i and edge
(j - i) as above, and (i represents the message passed from node j node ke' is the number of incoming messages, which is equal to & unless the protein contains fewer than & amino acid residues. Between graph propagation steps, we also use a feed-forward point-wise layer to update
the node embeddings at all nodes i:
h + Dropout (g (h!")
h LayerNorm
(5)
CCo, C, N, and O. The alpha carbon Ca is the central carbon atom each amino acid residue.
*CB is the second carbon from the carboxyl carbon C.
WWe use 16 Gaussian radial basis functions with centers evenly spaced between 0 and 20 angstroms.
5

---

Published as a conference paper at ICLR 2021
where g is a sequence of two GVPs. These graph propagation and feed-forward steps update the vector features at each node in addition to its scalar features.
In computational protein design, the network learns generative model over the space of protein sequences conditioned on the given backbone structure. Following [ngraham Inggrahamm et al. (2019), we frame
this as autoregressive task and use masked encoder-decoder architecture to capture the joint dis- tribution over all positions: for each i, the network models the distribution at i based on the complete structure graph, as well as the sequence information at positions j i. The encoder first performs three graph propagation steps on the structural information only. Then, sequence information is added to the graph, and the decoder performs three further graph propagation steps where incoming messages h h()+))) for j 2 are computed only with the encoder embeddings. Finally, we use one last GVP with 20-way scalar softmax output to predict the probability of the amino acids.
In model quality assessment, we use three graph propagation steps and perform regression against the true quality score of a candidate structure, a global scalar property. To obtain single global representation, we apply a node-wise GVP to reduce all node embeddings to scalars. We then average the representations across all nodes and apply a final dense feed-forward network to output the network's 's prediction.
Further details regarding training and hyperparameters can be found in Appendix D
4 EVALUATION METRICS AND DATASETS
Protein design Computational protein design (CPD) is the conceptual inverse of protein structure prediction, aiming to infer an amino acid sequence that will fold into a given structure. CPD is difficult to directly benchmark, as some structures may correspond to a large space of sequences and others may correspond to none at all. Therefore, the proxy metric of native sequence recovery inferring native sequences given their experimentally determined structures-is often used (Li et al,, Connell O'Conell et al.) 2018 Wang et al. (2018). Drawing an analogy between sequence design and language modelling, Ingraham et al. (2019) also evaluate the model perplexity on held-out native sequences. Both metrics rest on the implicit assumption that native sequences are optimized for their structures (Kuhlman & Baker 2000) and should be assigned high probabilities.
To best approximate real-world applications that may require design of novel structures, the held-out evaluation set should bear minimal similarity to the training structures. We use the CATH 4.2 dataset curated by Ingraham et al. (2019) in which all available structures with 40% nonredudancy are par- titioned by their CATH (class, architecture, topology/fold, homologous superfamily) classification. The training, validation, and test splits consist of 18204, 608, and 1120 structures, respectively.
We also report results on TS50, an older test set of 50 native structures first introduced by Li et al. (2014). The smaller size of this benehmark also allows a comparison to the computationally expen- sive physics-based calculations of the f xbb protocol in Rosetta, a software suite well-established in the structural biology community (Das & Baker 2008). No canonical training and validation sets exist for TS50. To evaluate on TS50, we filter the CATH 4.2 training and validation sets for sequences with less than 30% similarity (as computed by PSIBLAST) to any sequence in TS50.
Model quality assessment Model quality assessment (MQA) aims to select the best structural model of a protein from large pool of candidate structures The performance of different MQA methods is evaluated every two years in the community- Critical Assessment of Structure Pre- diction (CASP) (Cheng et al.) 2019). For number of recently solved but unreleased structures, called targets, structure generation programs produce a large number of candidate structures. MQA methods are evaluated by how well they predict the GDT-TS score of candidate structure compared to the experimentally solved structure for that target. GDT-TS is a scalar measure of how similar two protein backbones are after global alignment (Zemla et al. 2001).
In addition to accurately predicting the absolute quality of a candidate structure, a good MQA method should also be able to accurately assess the relative model qualities among pool of candi- dates for a given target so that the best ones can be selected, perhaps for further refinement. There-
We refer models of protein structure "structural models" or "candidate structures" to avoid confusion with the term "model" used in the MI. community.

---

Published as a conference paper at ICLR 2021
Table I: GVP-GNN outperforms Structured Transformer and sets a new state-of-the art on the CATH 4.2 protein design test set (and its short and single-chain subsets) in terms of per-residue perplexity (lower better) and recovery (higher is better). Recovery is reported as the median (over all structures) of the average % of residues correctly recovered in 100 sampled sequences.
Perplexity
Recovery %
Method
Type
Short
Single-chain
All
Short
Single-chain
All
GVP-GNN
GNN
7.10
7.44
5.29
32.1
32.0
40.2
Structured GNN
GNN
8.31
8.88
6.55
28.4
28.1
37.3
Structured Transformer
GNN
8.54
9.03
6.85
28.3
27.6
36.4
fore, MQA methods are commonly evaluated on two metrics: a global correlation between the predicted and ground truth scores, pooled across all targets, and the average per-larget correlation among only the candidate structures for specific target (Cao & Cheng. 2016; Derevyanko et al., 2018 Pagès et al. 2019). We follow this convention in our experiments.
We train and validate on 79200 candidate structures for 528 targets submitted to CASP 5-10. We then test GVP-GNN on two MQA datasets. First, we score 20880 stage 1 and stage 2 candidate structures from CASP 11 (84 targets) and 12 (40 targets). This benchmark was first established by Karasikov et al. (2019) and has been used by many recently published methods. Second, to compare with larger number of methods more recent structural data, we also score 1472 stage candidate structures from CASP 13 (20 targets). We add the CASP 1 11-12 structures to our training set to evaluate on CASP 13. Further details the MQA datasets can be found in Appendix C
EXPERIMENTS
Protein design GVP-GNN achieves state-of-the-art performance on CATH 4.2, representing a substantial improvement both in terms of perplexity and sequence recovery over Structured Trans- former (Ingraham et al., 2019). a GNN method which was trained using the same training and validation sets (Table I.. Following Ingrahhm et al. (2019). report evaluation short (100 or fewer amino acid residues) and single-chain subsets of the CATH 4.2 test set, containing 94 and 103 proteins, respectively, in addition the full test Although Structured Transformer leverages an attention mechanism on top of graph-structured representation of proteins, the authors note in ablation studies that removing attention appeared increase performance. We therefore retrain and compare against a version of Structured Transformer with the attention layers replaced with standard graph propagation operations (Structured GNN). Our method also improves upon this model.
On the smaller test set TS50, we achieve 44.9% recovery compared to Rosetta's 's 30% and outperform methods based on each of the three classes of structural representations. Overall, we place 2nd out of 9 methods in terms of recovery (see Appendix E). However, the results for this test set should be taken with grain of salt, given that the different methods did not use canonical training datasets.
Model quality assessment We compare GVP-GNN against other single-structure, structure-only methods on the CASP 11-12 test set in Table 2 These include the CNN methods 3DCNN (Derevyanko al.) 2018) and Ornate (Pagés et al.] 2019), the GNN method GraphQA (Baldas- (Olechnoviž
sarre et al. 2020). and three methods that use sequential representations--Nnne & Venclovas 2017 7). SBROD (Karasikov al.) 2019). and ProQ3D (Uziela et al.) (017). All of these methods learn solely from protein structure with the exception of ProQ3D, which in addition uses sequence profiles based on alignments. We include ProQ3D because it is an improved version of the best single-model method in CASP 11 and CASP 12 (Uziela et al. 2017). GVP-GNN outperforms all other structural methods in both global and per-target correlation, and even performs better than ProQ3D on all but one benchmark. We also train and evaluate DimeNet, recent 3D-aware GNN architecture which achieves state-of-the-art on many small-molecule tasks (Klicpera et al. 2019 on CASP 1-12. DimeNet does not outperform any of the models in Table (see Appendix
"We obtained the data for the comparisons from Baldassarre et al. 2019)
Thhere are two versions of GraphQA; we compare against the one using only structure information.
7

---

Published as a conference paper at ICLR 2021
Table GVP-GNN improves over other single-structure. structure-only methods on CASP 11 and 12 in terms of global (Glob) and mean per-target (Per) Pearson correlation coefficients (higher is better). Each method is classified as one of the three types discussed in Section [ ProQ3D is set aside as the only method shown which additionally uses sequence-based profiles. For each metric, the top performing structure-only method is in bold, as is the top method overall (if different).
CASP
CASP 12
Stage
Stage 2
Stage
Stage
Method
Type
Glob
Per
Glob
Per
Glob
Per
Glob
Per
GVP-GNN
GNN
0.84
0.66
0.87
0.45
0.79
0.73
0.82
0.62
3DCNN
CNN
0.59
0.52
0.64
0.40
0.49
0.44
0.61
0.51
Ornate
CNN
0.64
0.47
0.63
0.39
0.55
0.57
0.67
0.49
GraphQ.A
GNN
0.83
0.63
0.82
0.38
0.72
0.68
0.81
0.61
VoroMQA
Seq
0.69
0.62
0.65
0.42
0.46
0.61
0.61
0.56
SBROD
Seq
0.58
0.65
0.55
0.43
0.37
0.64
0.47
0.61
ProQ3D
Seq
0.80
0.69
0.77
0.44
0.67
0.71
0.81
0.60
Table 3: GVP-GNN improves over single-structure methods participating in CASP 13 the 20
evaluated targets. The seven top methods highlighted by the CASP organizers are shown. GVP- GNN is the top structure-only method and the top method overall in terms of global correlation.
Method
Global
Per-target
GVP-GNN
0.888
0.671
SASHAN
0,840
0.633
FaeNNz
0.810
0.650
VoroMQA-A
0.744
0.595
VoroMQA-B
0.726
0.586
ProQ3D
0.847
0.660
MULTICOM-NOVEL.
0.652
0.551
ProQ4
0.604
0.691
We compare GVP-GNN with all 23 single-structure MQA methods participating in CASP 13 for which complete predictions on the 20 evaluation targets are available. Seven of these methods were highlighted as best-performing by the CASP organizers in Cheng al. (019)) and are shown along with GVP-GNN in Table These include four methods learning solely from structural features and three also using sequence profiles. SASHAN learns a linear model over secondary structure and contact-based features (Cheng et al. 2019). FaeNN, 3 (Studer et al. (020), ProQ3D (Uziela jet al., 2017), and VoroMQAT statistical potential top of such structural features. Finally, MULTICOM-NOVEL (Hou et al, 2019) and ProQ4 (Hurtado et al. 2018) employ noeddimensional deep convolutional networks on top of sequential representations. GVP-GNN outperforms all methods in terms of global correlation and outperforms all structure-only methods in per-target correlation. See Appendix E for full results.
(Olechnovič ((le((AnoOeo
& Venclovas 2017) learn multi-layer perceptron or
Finally, because our architecture updates vector features along with scalar features at each node embedding, it is possible to visualize learned vector features the intermediate layers of the trained MQA network. We show and discuss the interpretability of such features in Appendix F
Ablation studies The methods we have compared against include number of GNNs (Structured Transformer/GNN, ProteinSolver, GraphQA). We train a number of ablated models for CPD and MQA to identify the aspects of the GVP which most contribute to our performance improvement over these GNNs 4T) Replacing the GVP with a vanilla MIP layer or propagating only scalar features both remove direct access geometric information, forcing the model learn scalar-
*FacNNz is also known as QMEANDisCo VQrroMQAAA and VoroMQA-B the same except that the former relaxes the sidechains before scoring.
8

---

Published as a conference paper at ICLR 2021
Table 4: Ablations of the GVP architecture decrease performance on CPD and MQA. We include Structured GNN and GraphQA state-of-the-ae GNN references for CPD and MQA, respectively. Metrics are defined the same way as in Tables 1 (CPD) and (MQA)
CPD
MQA
CATH 4.2 All
CASP 11 Stage 2
CASP 12 Stage 2
Modification
Perplexity
Recovery
Global
Per-target
Global
Per-target
None
5.29
40.2
0.87
0.45
0.82
0.62
MLP layer
7.76
30.6
0.84
0,36
0.79
0,59
Only scalars
7.31
32.4
0.84
0.38
0.83
0.59
Only vectors
11.05
23.2
0.56
0.16
0.57
0.39
No W
5.85
37.1
0.86
0.41
0.81
0.60
Structured GNN
6.55
37.3
GraphQA
0.82
0.38
0.81
0.61
valued, indirect representations of geometry. These modifications result in considerable decreases in performance, underscoring the importance of direct access to geometric information. Propagating only the vector features results in features (such as torsion angles and amino acid identity) and the part of the GVP with approximation guarantees. Therefore, the dual scalar/vector design of the GVP is essential: without either, the best ablated model falls short of Structured GNN on CPD and only matches GraphQA on MQA. Finally, eliminating the second vector transformation W, y results in slight decrease performance. Therefore, all architectural elements contributed to our improvement over state-of tatte-of-theaare trtthhe-tt..
larger decrease it both eliminates important scalar input
6 CONCLUSION
In this work, we developed the first architecture designed specifically for learning on dual relational and geometric representations of 3D macromolecular structure. At core, our method, GVP-GNN, augments graph neural networks with computationally simple layers that perform expressive geo- metric reasoning over Euclidean vector features. Our method possesses desirable theoretical prop- erties and empirically outperforms existing architectures on learning quality scores and sequence designs, respectively, from protein structure.
The equivariance of GVP layers with respect 3D translations and rotations also highlights a sim- ilarity to methods that leverage irreducible representations SO(3) to define equivariant convo- lutions point clouds (Thomas et al. 2018 Anderson et al. 2019). These methods allow for cquivariant representations cost, their applications have until recently been limited to small molecules (Eismann et al., 2020). Our architecture presents an alternative, relatively lightweight approach to equivariance that is well- suited for large biomolecules and biomolecular complexes.
higher-order tensors, but due to their complexity and computational
In further work, we hope to apply our architecture to other important structural biology problem areas, including protein complexes, RNA structure, and protein-ligand interactions.
ACKNOWLEDGEMENTS
We acknowledge support from the U.S. Department of Energy, Office of Science, Office of Ad- vanced Scientific Computing Research, Scientific Discovery through Advanced Computing (Sci- DAC) program, and Intel Corporation. SE supported by a Stanford Bio-X Bowes fellowship. RILT is supported by the U.S. Department of Energy, Office of Science Graduate Student Research (SCGSR) program. We thank Tri Dao. Trenton Chang, and all members of the Dror group for feedback and discussions.

---

Published as a conference paper at ICLR 2021
REFERENCES
Namrata Anand, Raphael Ryuichi Eguchi, Alexander Derry, Russ B Altman, and Possu Huang. Protein sequence design with a learned potential. bioRxiv, 2020.
Brandon Anderson, Truong-Son Hy. and Risi Kondor. Cormorant: Covariant molecular neural networks. arXiv preprint arXiv: 100iv:1900.000 04015, 2019.
Federico Baldassarre, David Menéndez Hurtado, Arne Elofsson, and Hossein Azizpour. Graphqa: Protein model quality assessment using graph convolutional network. 2019.
Federico Baldassarre, David Menndez Hurtado, Arne Elofsson, and Hossein Azizpour. GraphQA: protein model quality assessment using graph convolutional networks. Bioinformaties, 2020.
Peter Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 1806601666.1, 2018.
Jeremy M Berg, John I. Tymoczko, and Lubert Stryer. Biochemistry, W.H. Freeman and Company, 2002.
Renzhi Cao and Jianlin Cheng. Protein single-model quality assessment by feature-based probability density functions. Scientific reports, 6:23990, 2016.
Sheng Chen, Zhe Sun, Lihua Lin, Zifeng Liu, Xun Liu, Yutian Chong, Yutong Lu, Huiying Zhao, and Yuedong Yang. To improve protein sequence profile prediction through image captioning on pairwise residue distance map. Journal of Chemical Information and Modeling, 2019.
Jianlin Cheng, Myong-Ho Choe, Ame Elofsson, Kun-Sop Han, Jie Hou, Ali HA Maghrabi, Liam McGuffin, David Menéndez-Hurtado. Kliment Olechnovië, Torsten Schwede, et Estimation of model accuracy in 13 Proteins: Structure, Function, and Bioinformaties, 87(12):1361-1377 2019.
Nadav Cohen and Amnon Shashua. Inductive bias of deep convolutional networks through pooling geometry. In International Conference on Learning Representations, 2017.
George Cybenko. Approximation by superpositions sigmoidal function. Mathematics control, signals and systems, 2(4):303-314, 1989.
Rhiju Das and David Baker. Macromolecular modeling with rosetta. Annu. Rev. Biochem., 77: 363-382, 2008.
Georgy Derevyanko, Sergei Grudinin, Yoshua Bengio, and Guillaume Lamoureux. Deep convolu- tional networks for quality assessment of protein folds. Bioinformatics, 34(23):4044-4000 2018.
Stephan Eismann, Raphael JL Townshend, Nathaniel Thomas, Milind Jagota, Bowen Jing, and Ron O Dror. Hierarchical, rotation-equivariant neural networks to select structural models of protein complexes. Proteins: Structure, Function, and Bioinformatics, 2020.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1263-1272, 2017.
Jordan Graves, Jacob Byerly, Eduardo Priego. Naren Makkapati, S Vince Parish, Brenda Medellin, and Monica Berrondo. A review of deep learning methods for antibodies. Antibodies, 9(2): 2020.
Joe Greener, Lewis Moffat, and David T Jones. Design of metalloproteins and novel protein folds using variational autoencoders. Scientific reports, 8(1):1-12. 2018.
Sharon Hammes- Schifef-rreffrrrrrrreee and Stephen J Benkovic. Relating protein motion to catalysis. Annu. Rev. Biochem., 75:519-541, 2006.
10

---

Published as a conference paper at ICLR 2021
Jie Hou, Renzhi Cao, and Jianlin Cheng. Deep convolutional neural networks for predicting the quality of single protein structural models. bioRxiv, pp. 590620, 2019.
David Menéndez Hurtado, Karolis Uziela, and Arne Elofsson. Deep transfer learning in the assess- ment of the quality of protein models. arXiv preprint FrXivv/804..6220 2018.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. In Advances in Neural Information Processing Systems, pp. 15794-15805, 2019.
Mikhail Karasikov, Guillaume Pagès, and Sergei Grudinin. Smooth orientationndeepedden scoring function for coarse-grained protein quality assessment. Bioinformatiis, 35(16))2801-280) 2019.
Johannes Klicpera, Janek GroB, and Stephan Günnemann. Directional message passing for molec- ular graphs. In International Conference Learning Representations, 2019.
Brian Kuhlman and David Baker. Native protein sequences are close to optimal for their structures. Proceedings of the National Academy of Sciences, 97(11):10383-100 2000.
Zhixiu Li, Yuedong Yang. Eshel Faraggi, Jian Zhan, and Yaoqi Zhou. Direct prediction of profiles of sequences compatible with protein structure by neural networks with fragment-based local and energy-based nonlocal profiles. Proteins: Structure, Function, and Bioinformatics, 82(10): 2565-2573. 2014.
James O' Connelll Zhixiu Li, Jack Hanson, Rhys Heffernan, James Lyons, Kuldip Paliwal, Abdollah Dehzangi. Yuedong Yang, and Yaoqi Zhou. Spin2: Predicting sequence profiles from protein structures using deep neural networks. Proteins: Structure, Function, and Bioinformatics, 86(6); 629-633, 2018.
Kliment Olechnovië and Česlovas Venclovas. Voromqa:
Assessment of protein structure quality using interatomic contact areas. Proteins: Structure, Function, and Bioinformatics, 85(6):1131- 1145, 2017.
Guillaume Pagès, Benoit Charmettant, and Sergei Grudinin. Protein model quality assessment using 3d oriented convolutional neural networks. Bioinformatics,
2019.
Janaina Cruz Pereira, Ernesto Raul Caffarena, and Cicero Nogueira dos Santos. Boosting docking- based virtual screening with deep learning. Journal of chemical information and modeling, 56 (12):2495-2500) 2016.
Yifei Qi and John ZH Zhang. Denseopd: Improving the accuracy of neural-network-basee compu- tational protein sequence design with densenet. Journal of Chemical Information and Modeling, 6053):1245-1252, 2020.
Raghav Shroff, Austin Cole, Barrett R Morrow, Daniel J Diaz, Isaac Donnell, Jimmy Gollihar, Andrew D Ellington, and Ross Thyer. A structure-based deep learning framework for protein engineering. bioRxiv, pp. 833905, 2019.
Alexey Strokach, David Becerra, Carles Corbi-Verge, Albert Perez-Riba, and Philip M Kim. Fast and flexible protein design using deep graph neural networks. Cell Systems, 11(4):002-411, 2020.
Gabriel Studer, Christine Rempfer, Andrew M Waterhouse, Rafal Gumienny, Juergen Haas, and Torsten Schwede. Qmeandiscodissancce constraints applied on model quality estimation. Bioin- formatics, 36(6):1765-1177 2020.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and and cunslatton-equivaraann neural networks for 3d point clouds. arXiv preprint arXiv:1802.082/9, 2018.
Raphael Townshend, Rishi Bedi, Patricia Suriana. and Ron Dror. End-to-end learning on 3d protein structure for interface prediction. In Advances in Neural Information Processing Systems, pp. 15616-15625. 2019.
11

---

Published as a conference paper at ICLR 2021
Karolis Uziela, David Menéndez Hurtado, Nanjiang Shu, Bjöm Wallner, and Ame Elofsson. Proq3d: improved model quality assessments using deep learning. Bioinformatics, 33(10) 1580, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Jingxue Wang, Huali Cao, John ZH Zhang, and Yifei Qi. Computational protein design with deep learning neural networks. Scientific reports, 8()):199.001 2018.
Jonghun Won, Minkyung Baek, Bohdan Monastyrskyy, Andriy Kryshtafovych, and Chaok Seok. Assessment of protein model structure accuracy estimation in casp Challenges the era of deep learning. Proteins: Structure, Function, and Bioinformatics, 87(12):3351-1360 2019.
Adam Zemla, Česlovas Venclovas, John predictions in casp4. Proteins: Structure, Function, and Bioinformatics, 45($5):13-21, 2001.
John Moult, and Krzysztof Fidelis. Processing and evaluation of
Yuan Zhang, Yang Chen, Chenran Wang. Chun-Chao Lo, Xiuwen Liu, Wei Wu, and Jinfeng Zhang. Prodconn: Protein design using convolutional neural network. Proteins: Structure, Function, and Bioinformatics, 2019.
12

---

Published as a conference paper at ICLR 2021
A PROPERTIES OF GEOMETRIC VECTOR PERCEPTRONS
A.1 EQUIVARIANCE AND INVARIANCE
The vector and scalar outputs the GVP are equivariant and invariant, respectively, with respect to an arbitrary composition of rotations and reflections in 3D Euclidean space described by R iee.
RVP((s,R(V)) = R(V'))
Proof. We can write the transformation described by R as multiplying V with a unitary matrix U 3 from the right. The Lo-norm, scalar multiplications, and nonlinearities are defined row- wise as in Algorithm 1. We consider scalar and vector outputs separately. The scalar output, as function of the inputs, is
[ Vil2
=
W
+b)
Since | VU A Vll 2. we conclude s' is invariant. Similarly the vector output
V = + (I VIl) @ W,W.V
The row-wise scaling can also be viewed meftipuutiplicatioon by diagonal matrix D. [w, W. Vil2 = IW W VU D is invariant. Since
Since
DW W& = (DW W
we conclude that V' equivariant.
A.2 APPROXIMATION OF
FUNCTIONS
The GVP inherits an analogue of the Universal Approximation property Cybenko 1989) of standard dense layers. If R describes an arbitrary rotation or reflection in 3D Euclidean space, we show that the GVP architecture can approximate arbitrary scalar-valued functions invariant under R and defined Q" Rx the bounded subset of |R*3 whose elements can be canonically oriented based on three linearly independent vector entries. Without loss of generality, we assume the first three vector entries can be used.
The machinery corresponding to such approximations corresponds to GVP G, with only vector inputs, only scalar outputs, and a sigmoidal nonlinearity a; followed by a dense layer. This can also be viewed as the sequence of matrix multiplication with W,. taking the 12-norm, and dense network with one hidden layer. Such machinery can be extracted from any two consecutive GVPs (assuming sigmoidal or).
We restate the theorem from the main text:
Theorem. Let R describe an arbitrary rotation and/or reflection in RR, For V 2 3 let Q C R be the set of all V [V], .... vz)" Rux3 such that vı, v2 Vs linearly independent and O < |well2 < for all and some finite > 0. Then for any continuous
Q2 - IR. such that (()(V) = F(V) and for any > 0, there exists a form S(V) w=GG(VV such that [F(V) S(V) for all V E Ω
Proof. The idea is to write F' as composition F = Fow and w = hoy. We show that multiplication with Wn, and and taking the L2-norm can compute y, and that the dense network with one hidden layer can approximate F h.
Call an element V E Ω2 oriented if 0j 2i x2e, + y and V3 øge, + y3e, 23 with L1, 8/2 23 > Define w Ω² + 3 to be the orientation function that orients its input and then extracts the vector of 3v coefficients, [an 22, 1/2 Fa, V3 23, ... . Ti, Dis 2i, ......... These T
13

---

Published as a conference paper at ICLR 2021
elements can be written as
|211l2
Fa
Us 01
01/01,
i2
122122 -
Y: (v, 02 12/22//22.
i2 3
23 = V' 2 z3 -3
Z; (u; Uy I223 veys) 23: 124
and are invariant under rotation and reflection, because they are defined using only the norms and inner products of the Vi. Then F F o w, where F: [-b, 1j3w-3 R.
The key insight is that if construct Wt, such that the rows of Wt A V are the original vectors vi, Vi and all differences v; - v,, vi, j S min(i, (in(i,)), 3), then we can compute w(V) from the row-wise norms of W V. That is, w application of the cosine law. The GVP precisely computes y (V) a o (W my + b). It remains to show that there exists a form f(y) w [o my m b)] that e-approximates F h [-26, 20) R. Up translation and uniform scaling of the hypercube, this is the result of the Universal Approximation Theorem (Cybenko) 1989)
o where y = v(V = 1l-12 o (W V) E IR and is an
intermediate step:
write
B SYNTHETIC TASKS
We perform controlled experiments on synthetic dataset in order to analyze the benefits of the GVP architecture and determine if it indeed improves the geometric and joint geometric-relational reasoning abilities of GNNs.
Dataset The synthetic dataset is designed to mimic the essential qualities of the domain of protein structures. Each "structure" consists of n 100 random points in R3, distributed uniformly in the ball radius r = 10, with the constraint that no two points are less than distance d = 2 apart. Each position is also associated with random unit vector "sidechain") Three points are randomly chosen and are labelled as "special"; these will be used to define the learning tasks. We generate 20k "structures" and split them 80% train 10% validation 10% test.
endow it with orientation.
In the voxelized representation of a data point, the volume is voxelized into unit cubes. Each point is partitioned into neighborhood of eight voxels by trilinear interpolation, such that exact coordinate information is retained. Separate channels are used for the special and non-special points. The "sidechains" are represented with a of 100 points located at the ends of the unit vectors and mapped into a third channel.
In the graph-structure representation of a data point, a proximity graph is drawn with 10 nearest neighbors. Each node is labelled with a one-hot encoding of its type ("special" or "non- special")) and each edge with its Euclidean length. In the vanilla GNN, orientation information encoded by additionally including all three dot products in each edge embedding. In the GVP-GNN. each node embedding contains the node's "sidechain" vector, and each edge embedding contains unit vector indicating the direction of the edge.
Tasks We identify two regression tasks to exemplify geometric and relational reasoning, respec- tively. the "Off-center" task, the network predicts the distance from the centroid of the three special points to the centroid of the entire structure. In the "Perimeter" task, the network predicts the perimeter the triangle defined by the three special points. We characterize the former as primarily geometric, points in space that are not themselves nodes, and the latter as primarily relational, as it involves distances between three specific pairs of nodes. Finally, to represent problem with geometric and relational aspects, in the "Combined" task we attempt to predict the difference of the (normalized) off-center and perimeter objectives.
it requires reasoning about global properties of the 3D shape, in particular
14

---

Published as a conference paper at ICLR 2021
Models We train 3-layer shallow CNN and 3-layer GNN with single-layer feed-forward net- work. These are compared against a GVP-GNN that is otherwise identical to the standard GNN. To reflect the spirit of the synthetic experiment, all models have the same intermediate dimensionality of 32 (4 vector and 20 scalar channels in the GVP-GNN), we use the same training procedure for all models, and hyperparameter tuning architecture search is performed.
Results The results of the synthetic experiments are shown in Table 5 The vanilla GNN sig- nificantly outperforms the CNN on the perimeter task, while the CNN significantly outperforms the GNN on the off-center task, supporting our conceptual framework of the relative strengths of the two architectures. However, the GVP-GNN matches (and even outperforms) the CNN on the geometric task while maintaining the GNN's performance on the relational task. It additionally significantly outperforms both models on the combined task. On the basis of these results, the GVP appears successful in combining the strengths of the CNN and GNN into single architecture.
Table 5: Performance of the three compared model architectures the off-center (geometric), perimeter (relational), and combined objectives. The MSE losses are standardized such that predict- ing constant value (i.e. the mean) would result unit loss. Results are reported as the mean + S.D. over k = 5 random splits, where the best of three random seeds is taken for each split.
Model
Parameters
Off-center (geometric)
Perimeter (relational)
Combined
CNN
59k
0.319 + 0.014
0.532 + 0.028
0.522 + 0.016
GNN
40k
0.871 0.045
0.128 + 0.009
0.421 + 0.025
GVP-GNN
22k
0.206 + 0.024
0.106 + 0.006
0.155 + 0.024
C MQA DATASETS: FURTHER DETAILS
The MQA training and validation dataset includes 528 targets from CASP 5-10 and 150 candidate structures per target. These targets are partitioned at random into 480 training targets and 48 vali- dation targets. We include native structures for training and validation make of the greatest range of GDT-TS scores. We do not include native structures for testing in order to mimic CASP and real-world applications and because other methods were not tested native structures.
In the CASP assessments, stage refers to a set of 20 candidate structures per target and stage 2 to set of 150 candidate structures per target (5 from each structure prediction server). Both sets are pre-designated by the CASP organizers.
There has been slight inconsistency in the literature with regards the exact composition of the CASP 11 and 12 test sets. We use the list established by Karasikov et al. (019) because nearly all recent methods have been benchmarked on this set at some point. The CASP 13 test set cludes 1472 stage 2 candidate structures from the following 20 targets: T0950, T0951, T0953s1, TO953s2, T0954, T0955, T0957s1, T0957s2, T0958, T0960, T0963, T0966, T0968s1, T0968s2, T1003, T1005, T1008, T1009, T1011, T1016. These were the targets for which candidate struc- tures, submitted predictions, and ground scores were publicly available (obtained as described by Baldassarre et al. (2020)) the time of writing. The exact numbers of targets and structures in each set can be found in Table
Table 6: MQA datasets
Dataset
# Targets
# Structures
Includes natives?
Training
480
72000
Yes
Validation
48
7200
Yes
CASP 11 stage 1
84
1680
No
CASP 11 stage 2
83
12450
No
CASP 12 stage 1
40
800
No
CASP 12 stage 2
40
5950
No
CASP 13 stage 2
20
1472
No
15

---

Published as a conference paper at ICLR 2021
D TRAINING AND HYPERPARAMETERS
To train the MQA model to perform regression against the model quality score, we use a sum of an absolute loss and pairwise loss. That is, for each training step intake pairs i, where i, are candidate structures for the same target and compute
== = HH g(0) + H(y g()) (0)) - ((0)
(6) where H is the Huber loss. When reshuffling at the beginning of each epoch, we also randomly pair up the candidate structures for each target. Interestingly, adding the pairwise term also improves global correlation, likely because the much larger number of possible pairs makes it more difficult overfit.
To train the CPD model to perform classification / discrete generative modelling, we use the cross- entropy negative log likelihood loss.
For both the MQA and CPD model, we use node and hidden embeddings with vector and 100 scalar channels and edge embeddings with 1 vector and 32 scalar channels. The input node and edge features are first transformed by a sequence of GVPs to these dimensionalities before graph prop- agation. In all training runs, use the Adam optimizer Batches are constructed by grouping structures of similar size to have maximum of 1800 residues per batch for CPD and 3000 residues per batch for MQA. We also tune the following hyperparame- ters over a total of 70 training runs:
perform mini-batch gradient descent.
Learning rate in the range of 10-4 to 10
Dropout probability in the range of 10 -4-4 to 10-1
Number of graph propagation layers in the range of 3 to 6
Relative weight of the MQA pairwise loss in the range to
All models are implemented in TensorFlow 2.1 and trained for maximum of 100 epochs. This takes around two days for both models on a single Titan GPU. However, we note that the GPU memory, not compute power, is the bottleneck when training, based on the the average volatile GPU usage. We therefore anticipate that the runtime can be further optimized.
E ADDITIONAL RESULTS
E.1 DIMENET ON MQA
DimeNet Kliippera et al. 2019) is a recent GNN architecture designed to incorporate the 3D geome- try of small molecule graphs by encoding relative edge orientations in local spherical Bessel basis. DimeNet and our architecture are similar in that both seek to leverage geometric aspects of problem domain top of graph-structun votation-invariant features to indirectly encode geometry into its message-passing operations. Ad- ditionally, it updates edge embeddings by propagating messages between each pair of neighboring edges. While this paradigm appears well-suited for the domain of lcarning from small molecules, does not scale well to large protein structure graphs. In evaluating DimeNet on MQA, we could only extend the distance cutoff to 7.5 angstroms, while 30 neighbors corresponds to roughly 13 angstroms. DimeNet does not perform comparably to our model, or previous GNNs designed for learning from structure such as GraphQA (Table
representations. However, unlike architecture, Dimenet uses
E.2 MQA: RESULTS ON CASP 13
We report results for all 23 single-structure methods assessed in CASP 13 for which scores on all 20 targets are available (Table R). The following 7 methods were excluded because they do not report results for some targets: LamoureuxLab, SBROD-server, SBROD, 3DCNN, MESHI-server. SBROD-plus. FALCON-QA. and Grudinin. We do include comparisons with Lamoureux (pre- viously 3DCNN), 3DCNN (previously Ornate), and SBROD on CASP 11-12. All of the methods highlighted cop-performin by the CASP organizers in Cheng et al. (2019) are in compari- son for CASP 13. All predictions were obtained from the CASP download center as described by Baldassarre al. (2020).
16

---

Published as a conference paper at ICLR 2021
Table 7: Comparison of our GVP architecture, DimeNet, and the GraphQA, another GNN-based MQA method, on CASP 11-12. As in the main text, the global and mean per-target Pearson corre- lations are shown. DimeNet does not perform comparably to either GVP-GNN or GraphQA.
CASP 11 Stage 2
CASP 12 Stage 2
Method
Global
Per-target
Global
Per-target
GVP-GNN
0.87
0.45
0.82
0.62
GraphQA
0.82
0.38
0.81
0.61
DimeNet
0.61
0.30
0.62
0.47
Table 8: Comparison of GVP-GNN against all 23 available single-structure MQA methods in CASP 13 sorted by global correlation. The total number of predictions is shown, which may be less than 1472 even though they include all 20 targets. GVP-GNN is the best-performing method in terms of global correlation. In terms of per-target correlation, GVP-GNN outperforms all other structure-ond onlly methods and also all methods using sequence profiles except for ProQ4 and two ProQ3D variants.
Method
Global
Per-target
Predictions
Structure only?
GVP-GNN
0.888
0.671
1472
Yes
ProQ3D
0.847
0,660
1467
No
SASHAN
0,840
0.633
1472
Yes
MESHI-conT-eerver
0,838
0.651
1472
Yes
ProQ3
0,822
0.576
1468
No
MESHI
0.813
0.666
1472
Yes
MESHI-enrich-seerver
0.813
0.666
1472
Yes
FaeNNz
0,810
0.650
1472
Yes
ProQ3D-CAD
0,803
0.673
1468
No
ProQ3D-IDDT
0,803
0.687
1467
No
ProQ2
0,802
0.577
1472
No
ProQ3D-TM
0.791
0.654
1467
No
MASSI
0.776
0.582
1472
Yes
VoroMQA-A
0.744
0.595
1472
Yes
VoroMQA-B
0.726
0.586
1472
Yes
MASS2
0.689
0.584
1472
Yes
MULTICOM-NOVEL
0.652
0.551
1472
No
ProQ4
0.604
0.691
1472
No
PLU-AngularQA
0.577
0,460
1472
Yes
Bhattacharya-Server
0.577
0.501
1452
No
Bhattacharya-Singd
0.498
0.525
1452
No
Kiharalab
0.375
0.565
1472
No
PLU-TopQA
0.239
0,049
1472
Yes
lagodzinski-Cao-Q0
0.180
0.341
1472
Yes
E.3 CPD: RESULTS ON TS50
We compare against a number of recent CPD methods on the TS50 test set in Table These include two CNNs (ProDCoNN and DenseCPD), distance-map method (SBROF), and sequential representation methods (Wang's model and SPIN2). We also evaluate the GNN ProteinSolver on TS50 by sampling 100 sequences with temperature 1 (the default setting) for each structure using the public web server. No canonical training and validation sets exist for TS50. Therefore, order to evaluate TS50, remove sequences with more than 30% similarity from the CATH 4.2 training and validation sets and retrain our model. We outperform all other methods with the exception of DenseCPD, CNN method with canonical orientations. Interestingly, DenseCPD leverages the same underlying representation as ProDCoNN, yet achieves remarkably better performance. The main
"'Ecept for ProteinSolver, data for other methods obtained from Qi & 7hang 2020
Li al. 2014
17

---

Published as a conference paper at ICLR 2021
difference between the two methods is that ProDCoNN has convolutional layers and DenseCPD has 21 layers organized into dense residual blocks.
Table 9: Sequence recovery on TS50. Recovery for GVP-GNN and ProteinSolver is as defined in Table [ recovery for other methods, which model residues independently, is just classification accuracy. GVP-GNN is the second best-performing method, behind the CNN method DenseCPD. There is canonical training and validation for methods evaluated on TS50.
Method
Recovery %
GVP-GNN
44.9
DenseCPD Qi & Zhang 2020)
50.7
ProDCoNN Zhang
2019
40.7
SBROF (Chnn el al. 2019)
39.2
SPIN2 O'Conelll ef 2018
33.6
Wang's s model (Wang el |
33.0
ProteinSolver (Stroaach et al. 2020)
30.8
SPIN (Li et al. 2014)
30.3
Rosetta
30.0
VISUALIZATION AND INTERPRETATION OF LEARNED FEATURES
The geometric vector perceptron updates the vector features, in addition scalar features, at node embeddings during graph propagation. Therefore, while the input vector channels represent the forward and reverse directions each amino acid, the intermediate layers represent learned vec- tor features. Could some of these features correspond interpretable properties of the structure? Among total of 64 intermediate vector channels learned by the MQA model, few appeared visu- ally interpretable and shown selected structures in Figure 2 We caution against generalizing from the necessarily small number of images that could be manually inspected, but find these pre- liminary visualizations intriguing.
A
B
D
Figure 2: Four different learned vector channels of the MQA model are visualized on four separate structures. The backbone represented as a chain points, and each vector rooted the position of the amino acid node to which it belongs. From left to right: the vectors appear to A) point in the direction of motion that would make the protein more compact; B) point along the central axis of the alpha helix; C) point outwards from the structure; and D) point inwards into the structure.
18


Reviews:
[{"review_id": "neWf4mx0ewW", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to leverage the geometric and relational aspects of the 3D structure\nsimultaneously with the proposed GVP layer. The approach extends standard dense layers. The\nauthors claim that GNNs with such layers can perform both geometric and relational reasoning\non representations of macromolecules. The experiments on two problems in protein structure\nlearning show improvement over CNN and GNN. However, some details are missing, and the\nintuition of the methods is not clear. The draft needs to be refined and organized well.\nPros:\n1. The proposed GVP can learn geometric features from protein structure and is\nrotation/reflection-invariant.\n2. The learned vector features in the intermediate layer are interpretable.\n\nCons:\n1. Fig.1 A is not clear. The intermediate layer and weight matrices are all represented with\nblocks. The schematic diagram needs to be refined.\n2. Some equations are not clear. In equation (2) and (3), the subscripts j, m, e, v are not\ndefined.\n3. The writing and format of the paper are hard to read. In the 3.2 section, lots of symbols\nare used without definition (e.g. C, $alpha$, RBF). In the 3.3 section, h_v^j, h_v^i is not\nclear. some schematic diagrams will be helpful for illustrating the network structure.\n4. The intuition and motivation of the methods are not clear. For example, why is it easier\nfor GNN to access the global geometric properties of protein structure with GVP?", "extra_scores": {}, "extra_text": {}}, {"review_id": "h8f5rtbTAEV", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The challenge of predicting the structure of biological macro-molecules is widely relevant in many applications and difficult to address. This paper divides the types of approaches taken to address this challenge into those that use \"geometric\" information (i.e. positions of molecules in space), and those that utilize \"relational methods\" mainly through graphs (how different parts of molecule relate). This study is an attempt to integrate the two source of information by a novel network architecture. They introduce geometric vector perceptrons as a way of summarizing geometric information for graph layers without loss of information as it happens in dense layers. They evaluate the performance of these architectures on MQA and CPD tasks, both relevant and standard benchmarks in the field. \n\nPositives:\n\n- The paper well-written and sets up the problem and how it supposed to be solved very well. \n\n- The method seems to be high-performing on well-established benchmarks. \n\n- The conceptual contribution of GVP seems promising, intuitively relevant and usable.\n\n- Ablation and hyper-parameter studies are well-scoped. \n\nAreas for improvement: \n\n- The authors advertise GVPs as a \"drop-in replacement\" for MLP layers, however, looking at their ablation studies, while the MLP layer is less performant than GVP, it still is outperforming many SOTA methods. For this reason I suggest including the MLP results in the main text, so that the performance improvement is clear. It is also relevant to discuss what drives the performance of the method such  that even with MLP, it outperforms sophisticated architectures introduced before.\n\n- For MQA tasks, A discussion and ideally quantitative comparison with Ingraham 2019 ICLR (Differentiable Simulator), and AlphaFold (Senior et al 2020 Nature) seems pertinent. At least to me it is not immediately obvious if learning seq -> structure is a sufficiently different task not to be considered/discussed here. Presumably sequence data is more widely available than the geometry.  \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "X8pIs3B6o2", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors introduce a novel procedure to predict or acquire insights from the structure of a macromolecule (such as a protein, RNA, or DNA), represented as a set of positions associated with atoms or groups of atoms in 3D Euclidean space. Their approach, called GVP-GNN, can be applied to any problem where the input domain is a structure of a single macromolecule or molecules bound to one another. Their approach is divided into two steps: model quality assessment and computational protein design. \nThe paper addresses a notable problem in computational biology: learning on 3D structures of large biomolecules.  I have noticed various industry-based studies struggling to find the right solution. \nThe paper is powerfully written, and the approach is novel. The steps are described explicitly, which helps the reader to understand them. \n\nI have a few suggestions for the Authors. Please grammar proof the paper thoroughly to avoid small grammatical mistakes such as articles, or typos. \nIn the results, please refer to the suggested approach as GVP-GNN. Sometimes, you refer to your method as \"ours\". Please be uniform. \nThe availability of a dataset, especially the synthetic dataset, will allow the paper to be reproducible by others. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DF-7-BQcrs0", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work uses a graph representation of the protein backbone and a GNN\nfor model quality assessment (MQA) and protein design.\n\nStrengths:\nThe GNN architecture proposed has the important property that the vector and scalar outputs\nare equivariant and invariant with respect to composition of 3D rotations and reflections.\n\nThe submission is clear and correct, and the experiments are reproducible.\n\nWeaknesses:\nMost datasets used in this work and methods compared with are from 2016 or earlier (CASP12 or earlier, and for example ProQ3D);\nwhereas a more relevant comparison would use datasets from CASP13 (or even 14) and recent methods (for example ProQ4).\nNearly none of the methods in CASP 12 (2016) and earlier CASPs used deep learning\nsince Tensorflow was released in 11.2015 and PyTorch in 9.2016.\nOnly in CASP13 (2018) did the majority of groups use deep learning. \nThere is an inconsistency between the text describing the dataset used for MQA in section 4 and Table 6 in Appendix C, which may be clarified in the text by mentioning CASP13.\n\nWhile the work does compare against several GNNs, a comparison with important recent work on protein design using GNNs \nsuch as ProteinSolver [1] or a reference would improve the paper.\nOther missing references on protein design are methods using 3D CNNs [2] conditional VAE [3] and conditional GAN [4],\nas well as work on MQA [5]. Therefore, the unequivocal conclusion that the method \"empirically outperforms existing architectures on learning quality scores and sequence designs, respectively, from protein structure\" is partially accurate and may be rephrased to improve the paper.\n\n[1] Fast and Flexible Protein Design Using Deep Graph Neural Networks, Strokach et al, 2020.\nhttp://design.k8s.proteinsolver.org\n[2] A structure-based deep learning framework for protein engineering, Shroff et al, 2019.\n[3] Design of metalloproteins and novel protein folds using variational autoencoders, Greener et al, 2018.\n[4] Structural bioinformatics de novo protein design for novel folds using guided conditional Wasserstein generative adversarial networks, Karimi et al, 2019.\n[5] Deep transfer learning in the assessment of the quality of protein models, Hurtado et al, 2018.\nhttps://github.com/ElofssonLab/ProQ4", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work uses a graph representation of the protein backbone and a GNN for model quality assessment (MQA) and protein design. The proposed GNN has the property that the vector and scalar outputs are equivariant and invariant with respect to composition of 3D rotations and reflections. Overall speaking, the reviewers like this paper very much (especially its technical novelty), and provide quite positive comments. On the other hand, there are also some concerns being mentioned:

1)	The datasets used in the experiments are a little old – experiments on CASP 13 are preferred.
2)	Some technical details are not very clear and the paper writing needs improvements
3)	Experimental comparison with some recent baselines is missing.

The authors did a good job in their rebuttal and paper revision. Most of the above concerns have been addressed. Therefore, we think the current version of the paper is clearly beyond the bar of ICLR.

Decision: Accept (Spotlight)