Orientation-Aware Graph Neural Networks for Protein Structure Representation Learning
Authors: Jiahan Li
Shitong Luo
Congyue Deng
Chaoran Cheng
Jiaqi Guan
Leonidas Guibas
Jianzhu Ma
Jian Peng
Venue: Submitted to ICLR 2023 (2023)
Keywords: geometric learning
representation learning
structural biology

Abstract: By folding to particular 3D structures, proteins play a key role in living beings. To learn meaningful representation from a protein structure for downstream tasks, not only the global backbone topology but the local fine-grained orientational relations between amino acids should also be considered. In this work, we propose the Orientation-Aware Graph Neural Networks (OAGNNs) to better sense the geometric characteristics in protein structure (e.g. inner-residue torsion angles, inter-residue orientations). Extending a single weight from a scalar to a 3D vector, we construct a rich set of geometric-meaningful operations to process both the classical and SO(3) representations of a given structure. To plug our designed perceptron unit into existing Graph Neural Networks, we further introduce an equivariant message passing paradigm, showing superior versatility in maintaining SO(3)-equivariance at the global scale. Experiments have shown that our OAGNNs have a remarkable ability to sense geometric orientational features compared to classical networks. OAGNNs have also achieved state-of-the-art performance on various computational biology applications related to protein 3D structures.



Under review as a conference paper at ICLR 2023
PROTEIN STRUCTURE REPRESENTATION LEARNING THROUGH CIIENTTTIONAWARE GRAPH NEURAL NETWORKS
Anonymous authors Paper under double-bling review
ABSTRACT
By folding to particular 3D structures, proteins play a key role in living beings. To learn meaningful representation from protein structure for downstream tasks, not only the global backbone topology but the local fine-grained orientational re- lations between amino acids should also be considered. In this work, pro- pose the Orientation-Awe Aware Graph Neural Networks (OAGNNs) to better sense the geometric characteristics in protein structure (e.g. inner-residue torsion gles, inter-residue orientations). Extending a single weight from a scalar to a 3D vector, we construct a rich set of seometric-meaninge cess both the classical and SO(3) representations of a given structure. To plug our designed perceptron unit into existing Graph Neural Networks, we further introduce an equivariant message passing paradigm, showing superior versatility the global scale. Experiments have shown that our OAGNNs have remarkable ability to sense geometric orientational fea- tures compared to classical networks. OAGNNs have also achieved state-of-the- art performance on various computational biology applications related protein 3D structures.
operations to pro-
maintaining (3)
INTRODUCTION
Built from sequence of amino-acid residues, protein performs its biological functions by folding particular conformation in 3D space. Therefore, untilizing such 3D structures accurately the key for downstream analysis. While we have witnessed remarkable progress in protein structure predictions (Rohl et al. 2004 Källberg et al., 2012 Back et al. 2021 Jumper et al. 202I). another thread of tasks with protein 3D structures input starts to draw great interest, such function prediction (Hermosilla et al. 2020: Gligorijević et al. 2021). decoy ranking (Lundström et al. 2001 Kwon et al.) |021 Wang al. 20221), protein docking (Duhovny et al. [0022] Shulman- Peleg al.
2004
Gainza et al. (2200) Sverrisson et al.
2021) and driver mutation identification (Lefèvre et al.
1997
Antikainen & Martin 2005 et al.
2020 Jankauskaité al. 2019).
Most existing works in modeling protein structures directly borrow models designed for other ap- plications, including 3D-CNNs T al., 2012) in computer vision, Transformers (Vaswani et al.) 2017h) from natural language processing, and GNNs (Kipf & Welling 2016) in data mining. Though compatible with general objects, these models have overlooked the subtleties in the fine-grained ge- ometries, which are much more essential in protein structures. For instance, given the protein structure, as shown in Figure [ the locations of four backbone atoms (carbon, nitrogen, and oxygen) determine local skeleton, and different residues interact with each other through per- forming specific orientations between their local frames, either of which have important impacts on the protein structure and its function (Nelson et al. 2008
amino acid in
Recent attempts in building geometric-aware neural networks mainly focus on baking 3D rigid transformations into network operations, leading to the area of SO(3)-invariant and equivariant networks. One representative work is the Vector Neuron Network (VNN) (Deng et al. 2021). which achieves SO(3) equivariance on point clouds by generalizing scalar neurons to 3D vectors. An- other work is the GVP-GNN (Ting et al., (02T) that similarly vectorizes hidden neurons in GNN and demonstrates better prediction accuracy on protein design and quality evaluation tasks. However,

---

Under review as a conference paper at ICLR 2023
(a)
(6)
NN
Irput Proten
Reddual
Graphe Lavel Prececson
Node-Lvvl Prdiction
yroda
Overview (a) Each amino acid has its own rigid backbone with four heavy atoms, and can be represented by both lists of scalar and vector features. (b) Tasks associated with the protein 3D structure. Graph-level tasks consider the whole protein structures, and Node-level tasks operate on specific residues.
Figure 1:
these models can only adopt linear combinations of input vectors, which significantly limits their modeling capability. A simple example is that, given two input vector features dj and v2. the outputs W1 V1 + W202 through one linear layer constrained in the 2D plane spanned by v1, U2 even after applying their scalar-product non-lincarities. That is. VNN-based models are limited in perceiving orientational features, which have been proven crucial for proteins to perform their func- tions and interact with other partners (e.g. inner-residue torsion angles, inter-residue orientations)
(Nelson al., 2008 Voet&
Voet 2010 Xu & Berger 2006) Alfford et al. 2017).
To achieve more sensitive geometric orientation awareness, we propose Directed Weight (W) per- ceptrons by extending not only the hidden neurons but also the weights from scalars to 3D vectors, naturally saturating the entire network with 3D structure information in the Euclidean space. Di- rected weights support a set of ccometric-meaningg list features) and the classical (scalar-lish list) latent features, and perform flexible non-linear integration of the hybrid scalar-vector features. As protein structures are naturally attributed proximity graphs, we introduce a new Equivariant Message Passing Paradigm on protein graphs, connect the W- perceptrons with the graph learning models by using rigid backbone transformations for each amino acid, which provides a versatile framework for bringing the biological suitability and flexibility of the GNN architecture.
operations both the vector neurons (vector-
To summarize, our key contributions include:
We propose network unit based on the Directed Weights for capturing fine-grained geometric relations, especially for the subtle orientational details proteins.
We construct Equivariant Message Passing paradigm based on protein graphs
Our overall framework, the Orientation-Aware Graph Neural Networks, is versatile in terms of compatibility with existing deep graph learning models, making them biologically suitable with minimal modifications existing GNN models.
2 RELATED WORK
Representation learning on protein 3D structure. Early approaches rely on hand-crafted fea-
extracted and statistical methods to predict function annotations (Schaap al.) 2001 Zhang & Zhang 2010). Deep learning has been found to achieve success then. 3D CNNs are first proposed to process protein 3D structures by scanning atom-level features relying on multiple 3D voxels. One of the representative works (Derevyanko et al. 20188) adopts a 3D CNN-based model for assessing the quality of the predicted structures. 3D) CNNs also shed light on other tasks such as interface predic- tion (Toonsshen et al. 2019 Amidi et al. 2018). People also extend them to spherical convolutions
GGanza et al./ 2020) Sverrisson et al. 2202 Hermosilla Casajus et al.) 2021). to the Fourier space Zhemchuzhnikov et al. 2022) and the 3D Voronoi Tessellation space (Igashov et al. 2021 Graph
Convolutional Networks (Kipf & Welling, 2016) have also been adopted to capture geometric and
biochemical interactions between residues Ying et al. 2018 Gao & Ji 2019 Fout 2017). and have
been shown to achieve great performance on function prediction (Li et al. 2021), protein design
(Strokach et al. 2020) and binding prediction (Vecchio et al. 2021 Recently, transformer-based
2

---

Under review as a conference paper at ICLR 2023
methods (Vaswani et al. 2017) have trend to replace conventional methods in other bioinformatics
tasks Ingrraham et al., 2019 Back et al. 2021 Jumper et al.
2021 Cao et al. 2021).
Equivariant neural networks. Eguivariance is important property, for generalizing to unseen
conditions in geometric learning. (Cohen & Welling 2016 Weiler et al. 2018) Köhler et al. 2020
Satorras et al. 2021 Thomas et 2018 Fuchs al. 2020 Anderson al., 2019 Gasteiger al.
2019b Batzner et al. 2021 Eismann et (2.22.22.... Methods such as Tensor Field Network (Thomas)
al. 2018) and Cormorant (Anderson et al. 2019), have been developed to generate irreducible representations for achieving rotation equivariance in 3D. In comparison to Tensor Filed Network with complex tensor products, the Vector Neuron Network (VNN) achieves rotation equivariance in much simpler way, which generalizes the values of hidden neurons from scalars to 3D vectors DDng et al. 2021). GVP-GNN has also been introduced for learning protein representations by featuring geometric factors as vectors (Jing et al. 2020). Message passing in GNNs for vector representations using equivariant features have also been explored (Schütt al.) 2021 Luo et al.) 2022). However, guarantee rotation equivariance, they only linearly combine 3D vectors, in essence, limiting their geometric representing capacity. Another line methods condition filters on invariant scalar features for maitaining equivariance (Schütt et al.
2017
Gasteiger et al. 2019a
et al. 2021).
3 DIRECTED WEIGHT PERCEPTRON
Directed Lineer
(b) Non-Linearity
(c) Directed Interaction
Wector Nenture
Wy's
W.
V
R
V' x
Resturen
Aeddio Annturme
Outsul Pantoer
Wi arrva
E
RelUT(
Raukar Natkees
a- Wy
Vecti Vatial
Maliil
Figure 2: Model Details. A I-layer DWP consists of three following modules. Both the input, hidden and output are tuples, (a) Directed Linear Module applies multiple geometric operations to update scalar and vector features in four different ways with normal and directed weights. (b) Non-Linearity Module employs ReLU and sigmoid functions for scalar and vector features. (c)
Directed Interaction Module updates the features by using one another as updating parameters after non-linearity in another way.
A protein is modeled as KNN-graph G = =V,E) where each node V corresponds to one amino acid, characterized by a scalar-vector tuple hy = (s., Vv) with s, E RC Va E Cx3 and the edges are constructed spatially by querying its k-nearest neighbours in the space. The edge features & = {ety lizj, which represent edge connected node and j. are also multi-channel scalar- vector tuples. For simplicity, we set the channel numbers for scalar and vector features as the same, but they can be different. In practice, scalar, vector features are constructed based on a given protein structure and are kept separate. To make them compatible, the channel numbers of scalars are larger than the vectors. Details can be found in the Appendix Section B.I
3.1 DIRECTED WEIGHTS
Classical neural networks only consider scalar-list features of the form 8 E R with each layer transforming them with a weight matrix W RC''
ppc'xx XC and a bias term b E RC¹;
Ws+b
(I)
Although has been proved to be universal approximator, such a layer has intrinsically no geometric interpretation in the 3D shape space, and even the simplest 3D rigid transformation on the input can result in unpredictable changes of network outputs. Recent attempts have lifted neuron features from
3

---

Under review as a conference paper at ICLR 2023
scalar-lists to vector-lists lists V RCx3 with linear operation mapping to V RC'x3 Deng al. Jing et al. 2020):
2021
V'= WV
(2)
Under this construction, SO(3) in the latent space are simply matrix multiplications, which establishes clear 3D Euclidean structure. However, to maintain input-output consistency under rigid transformations (e.g. rotation and translation), their operations are still limited to classical lin- combinations weighted by W. To define operations that more adaptive to the geometrically meaningful features, we introduce the Directed Weights, that we can define any tensor with the last dimension of 3 as directed weight matrix, for example, W E RCC xxx3 seen as stack of geo- metric meaningful vectors lying the 3D Euclidean Space, and can be acted by SE(3) group from the right.
3.2 W W-OPERATORS
With weights and features both equipped with 3D vector representations, design set of geometric operators O (e.g. x) with learnable W parameters in neural networks, which can operate on both scalar-list features E ee and vector-list features V E RCx3
W
WOV
Geometric vector operations. Let W E RC'C be conventional scalar weight matrix, WI xC'xx33 and W₂ Rx3 be directed weight tensors. Beyond linear combinations, We can define two operations that leverage geometric information:
dot (V; W1) W1
CRO
(V; W, W2) (( 2
RC'x
(5)
Here s'axx transforms C vector features C" scalars using dot-product with directed weights, de- tailedly, s Ej Whihkssi which explicitly measures angles between vectors, and this opera- tor brings network the ability to accurately sense the orientational features. In Virg a vector crosses a directed weight before being projected onto the hidden space. While the output of plain linear com- binations between two vectors ii and the can only lie the plane W t1 + W202, cross-product in Equation Saattiionnrreeteee creates new direction outside the plane, which is crucial in 31D modelling. For instance, the side-chain angles of a given residue could be largely determined by its C₂ and C3 vectors, but may lie on the direction perpendicular the space constructed by the two vectors.
Scalar lifting. In addition, directed weight W Cxx3 can lift scalars to vectors by adopting the following operation, the ij th entry of sW 8, Wi,
Vin W) sW
RC3
(6)
This maps each scalar to particular vector, enabling inverse transformations or information bottle- necks from P. to 3 An intuition example is that, we can map scalar representing the distance between two amino acids to a vector pointing from one amino acid to the other, leading to more biological meaningful representations for the protein fragment.
Linear combinations. In the end, we keep the linear combination operations with sealar weights for both scalar and vector features:
1/n (8: W) Ws
E C
V (V: W) = WV
E C²
(8)
While these operators enables more flexible network design, equivariance to rigid transformations broken if considering more complex functions beyond linear combinations. We will introduce globally equivariant paradigm to tackle this issue in Section 4
3.3
W. PERCEPPROOO UNIT
Now we combine all the Woppertors together, assembling them into what we call Directed Weight Perceptrons (W-perceptrons). A W-perceptron unit is function mapping from (s, V) to
4

---

Under review as a conference paper at ICLR 2023
another vector tuple x' (s', V'), which can be stacked as network layers to form multi- layer perceptrons.
A single unit comprises three modules in sequential way: the Directed Linear module, the Non- Linearity module, and the Directed Interaction module (Figure 2
Directed linear module. The output hidden representations derived from the set of operations in- troduced previously are concatenated and projected into another hidden space of scalars and vectors separately:
s Wa(sdat sim
(9)
V Wv(Vess ors
Vifto Vinl
RR'x
(10)
Here W, E [²'(((++0)) and Wy +C++ are scalar weight matrices. In other word, the five separate updating functions allow transformation from scalar to scalar (Equation 7 scalar vector (Equation[6) (). vector scalar (Equation 4 and vector vector (Equation X Equation (
boosting the model's capability to reason in 3D space.
Non-linearity module. We then apply the non-linearity module to the hidden representation (s": V*). Specifically, we apply standard ReLU non-linearity (Nair & Hinton 2010) to the scalar components. For the vector representations, following (Weiler et al. [018:] Jing et al. 2020 Schütt al., 2021), compute sigmoid activation on the L2-norm of each vector and multiply back to the vector entries accordingly:
g < ReLU(s"), ofj + v.j sigmoid(])e/1la)
(11)
where wnth E R$ are the vector columns in 4 and on are their entries.
Directed interaction module. Finally we introduce the Directed Interaction module, integrating the hidden features d and 4 into the output tuple (s', V'
= V. V
E ERC
(12)
V = ('='WWa V E CCx33
(13)
Here Wv, Wa are directed weight matrices with sizes C' C" x and C' 3, respectively, 3 de- notes element-wiss multiplication for two matrices. This module establishes a connection between the scalar and vector feature components, facilitating feature blending. Specifically, Equation [2 dynamically determines how much the output should rely on scalar and vector representations, and Equation weights list of vectors using the scalar features as attention scores.
4 ORIENTATION-AWARE GRAPH NEURAL NETWORKS
To achieve SO(3) equivarianne without the loss of modeling capacity, we introduce an equivariant message passing paradigm protein graphs, to easily plug versatile W-Perceptron into any existing graph learning framework, which makes network architectures free from equivariant con- straints (Section 4..). The integrated models, called Orientation-Aware Graph Neural Networks, can not only accurately model the essential orientational features but also maintain the rotation cquivariance efficiently. We also design multiple variants in analogy other graph neural networks Section
4.1 EQUIVARIANT MESSAGE PASSING PROTEINS
A function f R R3 is SO(3) -equiaaauaaai (rotation-equivariiat) if any rotation matrix R E x3x3 applied to the input vector æ leads to the same transformation on the output f f()):
Rf(x) = f(Rx)
(14)
Such equivariant property can be achieved on a protein graph with local orientations, which is nat- urally defined from its biological structure (Ingraham et al. 2019) Specifically, each amino acid
5

---

Under review as a conference paper at ICLR 2023
node U1 V has four backbone heavy atom (Ca, C , O²), defining local frame Ou
No
Ca
RR
Yu
= C
C E R
(15)
O
u
Yo
Bv
yv
R3x3
(16)
||2
y 2
1
The local frame O2 is rotation matrix that maps 3D vector from the local the global coor- dinate system. An equivariant message passing paradigm then emerges through transforming node features back and forth between adjacent local frames. Formally, give an amino acid with hidden representation h (s, V), let fi and fg be transformations on the vector feature V from and to the global coordinate system:
O,) : (s, vo.). Oh, == (s, VO..)
(17)
The message passing update for node from layer / to layer + 1 is performed in the following steps, the notations indicates the concatenation of different hidden representations along the channel- wise dimension:
Transform neighbourhood vector representations of u into its local coordinate system Ou-
2. For each adjacent node compute the message m on edge (u, v) E & with an multi- layer W W-pereeptron J to the aggregated node and edge feature [hi, h eque] (Equation 18
3. Update node feature h1 with another multi-layer W-perceptron H. and transform back to the global coordinate system (Equation 19).
This paradigm achieves SO(3) -eq)v--avvva The formal proof of equivariance is presented in the Appendix Section
very general sense with no constraints on F, H.
m
++1
Σ F(fi(hh hl, VEN,
(18)
1 fy(H( my (+1), O,)
(19)
4.2 VARIANTS OF ORIENTATION- AWARE GRAPH NEURAL NETWORKS
By integrating the W-Perceptrons with equivariant message passing, propose multiple variants of entire Orientation-Aus Graph Neural Networks to boost the design space for different tasks.
OA-GCN. This is the one described and implemented in the previous subsection with Equation 18
and Equation 19
OA-GIN. We adopt graph isomorphism operator Xu et al. e for tunable skip connections,
2018) with learnable weighing parameter
I+1 m = (1 (1++) O.) + E F(f(hh,, Ow)) oEN.
(20)
OA-GAT. In comparison GCN and GIN, it is not trivial incorporate with the Graph Attention Network (GAT) (Veličković et al., 2018), as residues interact with each other unevenly in proteins. In partieular, define separate attentions for the scalar and vector representations
(s) ns Vil = f( O.)
(21)
8/2 Σ
V
Σ
a V.
(22)
h+1 (V)((((s.V')))..
(23)
The attention scores a², ad are the softmax values over the inner products of all neighboring source- target pairs defined as the follows:
exp ((8,,
exp (vr(V, V.))
a Σ EwENNUEut exp ((s,, sw))²
(24)
EwEN, exp (tr(V. V.))
The product for scalars and vectors standard inner product and Frobenius product, respectively.

---

Under review as a conference paper at ICLR 2023
5 EXPERIMENTS
To demonstrate the basic rationale in perceiving angular features of our W-Perceptron, we design a synthetic task (Section 5.1). We then conduct experiments on multiple benchmarks, including node-level tasks: Residue Identification (RES) (Section 5.2). Computational Protein Design (CPD) (Section 5.3), and graph-level tasks: Model Quality Assessment (MQA) (Section 5.4). Ablation studies are also included (Section 5.5p. More details are documented in the Appendix Section
5.1 SYNTHETIC TASK
a).
b).
somplex
Poullive
Negative
64
OFF
100
202
330
420
502
30C
700
ECO
Figure 3: a). The synthetic study. The vector in red is the anchor used to calculate the ground truth labels. Vectors falling into the cone are positive samples, whereas outside points are negative. b). The validation Binary Cross Entropy loss of compared methods.
Datasets. We test the performance of different perceptron designs on synthetic binary classifica- tion dataset with angular. We sample a random unit vector um E R as the anchor vector. Then each vector 2; in the space is labeled positive if its angle between the anchor vector ami $ /4, and negative otherwise. We generate 100, 000 random vectors with 50% positive samples and 50% negative samples. For fair comparison, restrict the number of parameters and hyperparameters of different models to approximately the same.
Results. As shown in Figure in general, our W P--rrceptron (DWP) converges faster than other models and achieves the lowest validation loss. We also notice that the Geometric Vector Perceptron (GVP) cannot converge given sufficient number of epochs due to the poor capability of perceiving angulars by only using linear combinations of vector features. In comparison to GVP, Vector Neural Network (VNN) performs better based its non-linear ReLU operation for vector features. Sim- ple Multi-Layer Perceptron (MLP) can also capture useful information according to its universal approximation ability. The superior performance demonstrates that our W-Perceptron has better ability on perceiving potential geometric features in space.
5.2 RESIDUE IDENTIFICATION
Datasets. Residue Identification (RES) aims to predict the identity of particular masked amino acid based on its local surrounding structure (Torgg & Altman| 2017). We download 1000, 000 (10%) substructures from the ATOM3D project (Townshend al. 2021). which are originally derived from 574 proteins from the PDB (Berman et al. 2000). The entire dataset is split into training, validation, and testing datasets with the ratio 80%, 10%, and 10%. There are no two proteins with similar structures in the test and non-test datasets based on the CATH 4.2 topology class at the domain level Orengo et al. 1997).
Metrics. We use classification accuracy to evaluate model performance.
Baselines. The 3D-CNN encodes the positions of the atoms a voxelized 3D volume. We also compare GNN variants such GCN GIN and GAT with our OA-GCN, OA-GIN and OA-GAT models. We also include GVP-GNN as well.
Results As shown in Table T GCN, GIN and GAT almost cannot accurately identify the type of amino acids, suggesting the lack ability of conventional GNN models to capture 3D geometric information. 3D-CNN performs better due to the power of the voxelization technique. Our models

---

Under review as a conference paper at ICLR 2023
outperform other models, suggesting OA-GNNs could help represent protein substructure geome-
tries better.
Table 1: Results of different RES methods on ATOM3D.
Table 2: Ablation studies.
Model
3D-CNN
GCN
GIN
GAT
GVP-GNN
OA-GCN
OA-GIN
OA-GAT
Model
No DW
No Int
No Equi
Acc %
45.1
8.2
9.1
12.4
48.2
50.2
50.8
49.2
Acc %
47,3
47.7
33.0
5.3
COMPUTATIONAL PROTEIN DESIGN
Datasets. Computational Protein Design (CPD) predicts the native protein sequence of a given backbone structure. Specifically, we focus on two databases: CATH 4.2 (Ingraham al. 2019) organizes proteins in hierarchical structure (Orengo et al. 1997). Following prior works, there are 18, 024 protein chains in the training set, 608 in the validation set, and 1, 120 in the test. TS50 dataset (Li et al. 2014) is relatively old benchmark consisting of only 50 protein structures widely used in biology communities.
Metries. Native Sequence Recovery Rate is adopted to evaluate the prediction performance, which compares the predicted sequence with the ground truth sequence at each position and calculates the proportion of the correctly recovered amino acids (Li et al. 2014). The perplexity score (Je-)
linek et al., 1977) evaluates whether a model could assign a high likelihood to the test sequences (Ingraham et al. 2019).
Table 3: Results of different CPD methods on the
Table 4: Results of MQA models on CASP 11 stage 2.
CATH 4.2.
Model
Perplexity J
Recovery % f
Model
Average 1
Global
Short
Single
All
Short
Single
All
p
St-Tranffrrmer
8.54
9.03
6,85
28.3
27.6
36.4
VoraMQA
0.42
0.41
0.29
0.65
0.69
0,51
St-GCN
8,31
8.88
6,55
28.4
28.1
37.3
RWplus
0.17
0.19
0.06
0.01
St-GIN
8.03
8.52
6.15
27.7
28.4
38.1
SBROD
0.43
0.41
0.29
0.55
0.57
0.39
St. GAT
10.86
10.67
9.89
26.2
26.8
35.2
Prog3D
0.44
0.43
0.30
0.77
0.80
0.59
GVP-GNN
7.10
7.44
5.29
32.1
32.0
40.2
3DCNN
0.49
0.39
0.27
0.64
0.67
0.48
OA-GCN
5.42
5.69
3.94
39.6
47.5
Omate
0.39
0.37
0.26
0.63
0.67
0.48
OA-GAT
OA DA-GIN
5.92
5.84
5.39
4.13
3.85
37.5
38.8
40.1
46.7
47.8
GVP-GNN
GraphiAA
DimeNet
0.58
0.30
0.48
0.33
0.35
0.40
0.46
0.28
0.42
0.80
0.75
0.61
0.72
0.61
0,62
0.81
0.43
0.74
No DW
6.52
6.79
5.28
36.7
35.0
42.9
OA-GCN
0.62
0.37
0.51
0.84
0.65
0.85
No Int
6.45
6.36
4.87
37.2
36.3
44.9
OA-GIN
0.63
0.36
0.52
0.86
0.67
0.88
No Equi
6.21
6.04
4.64
36,9
37.1
45.8
OA-GAT
0.65
0.42
0.53
0.83
0.69
0.86
Table 5: Results of structure biology methods on the TS50.
Model
Rosetta
SPIN
ProteinSolver
Wang's
SPIN2
SBROF
ProDCoNN
St-Trans
Reev %
30.0
30.3
30.8
33.0
33.6
39.2
40.7
42.3
Model
DenseCPD
GVP-GNN
OA-GCN
OA-GAT
OA-GIN
No DW
No Int
No Equi
Recv %
50.7
44.9
53.8
54.5
52.7
46.8
48.7
49.5
Baselines. For the CATH 4.2 dataset, we compare our models with other wate-of-the-aa models, including GVP-GNN, Structured Transformer and Structured GNN (Ingraham et al.) 2019). We compare multiple machine learning approaches in the TS50 dataset, including 3D CNN-basee meth- ods (ProDConn (Zhang et al., (2200), DenseCPD (Qi & Zhang 2020)). sequential models (Wang's model Wang et al. (201 8), SPIN (Li al. 2014), SPIN2 (O' Connell et al. 2018)) and GNN-based method (Strokach et al. 2020). We also compare with classical statistic methods (Schaap et al.
2001
Cheng et al. 2019).
Results. As shown in Table 3 our models achieve significant improvement over other methods on the CATH dataset. This result also suggests that the the short-chain and single-chain subsets more challenging, which is consistent with the result in (Ingraham et al. 2019). As shown in Table [) our models also gain superior performance. These results suggest that adopting OA-GNNs is more effective way of designing valid protein sequences.
5.4 MODEL QUALITY ASSESSMENT
Datasets. Model quality assessment (MQA) evaluates the quality of predicted protein structures K et al. 2021). The goal is to fit a model approximating the numerical metric used to compare
As there is no canonical training set for TS50, we follow prior methods to use protein sequences with less
than 30% similarity with the TS50 test set from the CATH training set for training (Jing et al. 2020 Qi &
Zhang 2020)
8

---

Under review as a conference paper at ICLR 2023
the predicted 3D structure with the native 3D structure. For this task, We randomly split the targets in CASPS-CASPI0 (Moult et al.; 2014) and sample 50 decoys for each target for generating the training and validation sets and use the CASP11 stage 2 proteins as the test set to ensure no similar structures are involved. In total, there are 508 proteins for training, 56 for validation, and 85 targets with 150 decoys for the test.
Metrics. We regress the global GDT TS score (Zemla 2003) which evaluates the quality of decoys.
We adopt the correlation between the predicted and real GDT_TS values measure the performance. We use three statistical correlation metrics: Pearson's correlation r, Spearman's p. and Kendall's т. We calculate the correlation for each target and average all the correlations over all the targets. We also calculate the Global correlations by taking the union of all decoy sets without considering the targets (Pages et al., 2019
Baselines. VoroMQA (Olechnovič & Venclovas, 2017) leverages potential model with protein contact maps, while RWplus (Zhang & Zhangg 2010) relies on physical energy terms. SBROD (Karaiikvv et al. (209)) uses hand-crafted features. Proq3D (Uziela et al.) 2017) employs FCNs for regression. 3DCNN (Derevyanko et al. (018)) and Ornate (Pagès et al. 2019) apply 3D CNNs for extracting meaningful features. GraphiA (Baldassarre et al. 2021), Dime Net (Gasteiger al.) 2019b) and GVP-GNN are the most similar models as ours which adopted GNN-baser methods on protein graphs.
Results. As shown in table our models outperform other methods on both average and global metrics except for Spearman's s correlation p where achieve the second-best demonstrating models can not only evaluate the quality for the same protein but also works well across different proteins in more general way.
5.5 ABLATION STUDIES
In the end, conduct ablation experiments to study the contribution of different modules of our model on the tasks of CPD 3 and RES (ta More specifically, based on OA-GCN, we replace all the directed weights with the regular linear layer with scalar weights (No DW). or remove the Interaction Module (No Int). or break the equivariance by canceling coordinate transformations in the step of message passing (No Equi).
In general, all three components of our model are important since removing each of them results in significant performance drop. According to the performance gain, the directed weights are the most important module for the CPD task, the orientational features are essential for the protein to fold to a given structure. For Residue Identity, however, equivariance message passing is the most important component, indicating the necessity of rotation equivariance in learning the representation of local protein structures.
CONCLUSION
For the first time, we generalize the scalar weights in neural networks to 3D directed vectors for better perceiving geometric features like orientations and angles, which are essential in learning rep- resentations from protein 3D structures. Based on the directed weights W, provide a toolbox of operators as well a W W-Perceptron Unit encouraging efficient scalar-vector feature interac- tions. We also enforce global -qqiiviquiea orientations naturally defined by the rigid property of each amino acid residue, which can be used for designing more powerful networks. Our integrated Orientation-Awar Graph Neural Networks achieve comparably better performances on multiple biological tasks in comparison with existing Mfae-oofthh-aat art methods.
a message passing paradigm using the local
Limitations and future work. While our method shows better awareness to fine geometric details, there remains huge space for non-linee non-lincarity designs upon geometric representations. In the future, we will extend our framework to other 3D learning tasks like small molecules or point clouds by learning local rotation transformations and using directed vector weights.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rebecca F Alford, Andrew Leaver-Fay, Jeliazko R Jeliazkov, Matthew J O O'Meara, Frank P DiMaio, Hahnbeom Park, Maxim V Shapovalov, Douglas Renfrew, Vikram K Mulligan, Kalli Kappel, et al. The rosetta all-atom energy function for macromolecular modeling and design. Journal of chemical theory and computation, (16):3031-3045 2017.
Ethan C Alley Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with requence-based deep representation learning. Nature methods, 16(12):1315-133) 2019.
Afshine Amidi, Shervine Amidi, Dimitrios Vlachakis, Vasileios Megalooikonomou, Nikos Para- gios, and Evangelia I Zacharaki. Enzynet: enzyme classification using 3d convolutional neural networks on spatial representation. PeerJ, 6:e4750, 2018.
Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. Advances in neural information processing systems, 32, 2019.
Nina M Antikainen and Stephen Martin. Altering protein specificity: techniques and applications. Bioorganic & medicinal chemistry, 13(8):2701-2716, 2005.
Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee. Jue Wang. Qian Cong. Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):871- 876, 2021.
Federico Baldassarre, David Menéndez Hurtado, Arne Elofsson, and Hossein Azizpour. Graphqa: protein model quality assessment using graph convolutional networks. Bioinformaties, 37(3): 360-366, 2021.
Simon Batzner, Tess E Smidt, Lixin Sun, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, and Boris Kozinsky. Se (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. arXiv preprint arXiv:2 2021.
Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. In International Conference Learning Representations, 2018.
Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip E Bourne. The protein data bank. Nucleic acids research, 28(1): 235-242, 2000.
Yue Cao, Payel Das, Vijil Chenthamarakshan. Pin- Yu Chen, Igor Melnyk, and Yang Shen. Fold2seq: A joint sequence (Id)-fold (3d) embedding-based generative model for protein design. In Inter- national Conference on Machine Learning, pp. 1261-1271 PMLR, 2021.
Jianlin Cheng. Myong-Hi Ho Choe, Ame Elofsson, Kun-Sop Han, Jie Hou, Ali HA Maghrabi, Liam J McGuffin, David Hanéndez-Hurtadde Kliment Olechnovič, Torsten Schwede, et al. Estimation of model accuracy in casp13. Proteins: Structure, Function, and Bioinformaties, 87112):131--1377, 2019.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- ence on machine learning, pp. 2990-2999. PMLR, 2016.
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas. Vector neurons: A general framework for so ee)-equiaa networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12200-12209, 2021.
Georgy Derevyanko, Sergei Grudinin, Yoshua Bengio, and Guillaume Lamoureux. Deep convolu- tional networks for quality assessment of protein folds. Bioinformatics, 34(23):4046-4055 2018.
Frederik Diehl.
Edge contraction pooling for graph neural networks.
arXiv preprint
arXiv: 10990, 2019.
10

---

Under review as a conference paper at ICLR 2023
Dina Duhovny, Ruth Nussinov, and Haim J Wolfson. Efficient unbound docking of rigid molecules. In International workshop on algorithms in bioinformatics, pp. 185-200. Springer, 2002.
Stephan Eismann, Raphael JL Townshend, Nathaniel Thomas, Milind Jagota, Bowen Jing, and Ron O Dror. Hierarchical, rotation-equivarin neural networks to select structural models of protein complexes. Proteins: Structure, Function, and Bioinformatics, 89(5):493- 50193--01,-00 2021.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: towards crack- ing the language of life's s code through supe-supe deep learning and high performance com- puting, arXiv preprint
200::2007.062255 2020.
Alex M Fout. Protein interface prediction using graph convolutional networks. PhD thesis, Colorado State University, 2017.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d roto- translation equivariant attention networks. Advances in Neural Information Processing Systems, 13:1970-1981, 2020.
Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, D Boscaini, MM Bronstein, and BE Correia. Deciphering interaction fingerprints from protein molecular surfaces using geo- 2020.
metric deep learning. Nature Methods,
Octavian-Eugee Ganea, Xinyuan Huang, Charlotte Bunne, Yatao Bian, Regina Barzilay, Tommi Jaakkola, and Andreas Krause. Independent se (3)-euuivrriant models for end-to-end rigid protein docking. In International Conference on Learning Representations, 2021.
Hongyang Gao and Shuiwang Ji. Graph u-nets. In international conference on machine learning, pp. 2083-2092. PMLR, 2019.
Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molec- ular graphs. In International Conference on Learning Representations, 2019a.
Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molec- ular graphs. In International Conference on Learning Representations, 2019b.
Vladimir Gligorijević, P Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn C Taylor, Ian M Fisk, Hera Vlamakis, al. Structure- based protein function prediction using graph convolutional networks. Nature communications, 12(1):1-14, 2021.
Pedro Hermosilla, Marco Schäfer, Matej Lang, Gloria Fackelmann, Pere-Pau Vázquez, Barbora Ko- zlikova, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinss convolution and pooling for learning on 3d protein structures. International Conference on Learning Represen- lations, 2020.
Pedro Hermosilla Casajus, Marco Schäfer, Matej Lang, Gloria Fackelmann, Pere Pau Vázguez Alco- cer, Barbora Kozliková. Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsic convolution and pooling for learning on 3d protein structures. In International Conference on Learning Representations, ICLR 2021: Vienna, Austria, May 04 2021, pp. 1-16. OpenReview. net, 2021.
Jie Hou, Badri Adhikari, and Jianlin Cheng. Deepsf: deep convolutional neural network for mapping protein sequences to folds. Bioinformatics, 3295):1295-1303 2018.
Ilia Igashov, Kliment Olechnovič, Maria Kadukova, Česlovas Venclovas, and Sergei Grudinin. Voroenn: deep convolutional neural network built on 3d voronoi tessellation of protein structures. Bioinformatics, 37(16)2332-23333 2021.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. Advances neural information processing systems, 32, 2019.

---

Under review as a conference paper at ICLR 2023
Justina Jankauskaité, Brian Aiénez--Garche Justas Dapkunas, Juan Fernánee-Reccio and lain H Moal. Skempi 2.0: an updated benchmark of changes in pootein-protedn binding energy, kinetics and thermodynamies upon mutation. Bioinformatics, 35(3):462-44) 469,,469,220 2019.
Fred Jelinek, Robert Mercer, Lalit R Bahl, and James K Baker. Perplexity a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1): S63-S63, 1977.
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence, 35(1)2221-231, 2012.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2020.
Bowen Jing, Stephan Fismann, Pratham N Soni, and Ron O Dror. Equivariant graph neural networks for 3d macromolecular structure. ICML 2021 CompBio Workshop, 2021.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583 589, 2021.
Morten Källberg, Haipeng Wang, Sheng Wang, Jian Peng, Zhiyong Wang, Hui Lu, and Jinbo Xu. Template-based protein structure modeling using the raptorx web server. Nature protocols, 7(8): 1511-1522, 2012.
Mikhail Karasikov, Guillaume Pagès, and Sergei Grudinin. Smooth orientation-dependdet scoring function for coarse-grained protein quality assessment. Bioinformatics, 35(16):2801-281)) 2019.
Jacob Devlin Ming-Wo Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. In Proceedings of NAACL-HL7, pp. 4171- 4186, 2019.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 1609.00000 2016.
Jonas Köhler, Lcon Klein, and Frank Noć. Equivariant flows: exact likelihood generative learn-
ing for symmetric densities. In International Conference on Machine Learning, pp. 5361-5370. PMLR, 2020.
Sohee Kwon, Jonghun Won, Andriy Kryshtafovych, and Chaok Scok. Assessment of protein model structure accuracy estimation casp14: Old and new challenges. Proteins: Structure, Function, and Bioinformaties, 2021.
Fabrice Lefevre, Maric-Hélène Rémy, and Jean-Michel Masson. Alanine-stretch scanning muta- genesis: simple and efficient method probe protein structure and function. Nucleic acids research, 25(2):447-443 1997.
Bian Li, Yucheng T. Yang. John A. Capra, and Mark B. Gerstein. Predicting changes in protein ther- modynamic stability upon point mutation with deep 3d convolutional neural networks. bioRxiv, 2020. doi: 10. 0.00022000.22.2.959987.
Shuangli Li, Jingbo Zhou, Tong Xu, Liang Huang, Fan Wang, Haoyi Xiong, Weili Huang, Dejing Dou, and Hui Xiong. Structure aware interactive graph neural networks for the prediction of protein-ligand binding affinity. In Proceedings of the 27th ACM SIGKDD Conference on Knowl- edge Discovery & Data Mining, pp. 975-985, 2021.
Zhixiu Li, Yuedong Yang. Eshel Faraggi, Jian Zhan, and Yaoqi Zhou. Direct prediction profiles of sequences compatible with protein structure by neural networks with fragment-based local and energy-based nonlocal profiles. Proteins: Structure, Function, and Bioinformatics, 82(10): 2565-2575 2014.
12

---

Under review as a conference paper at ICLR 2023
Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks. 2021.
Jesper Lundström, Leszek Rychlewski, Janusz Bujnicki, and Ame Elofsson. Pcons: A neural- network- based consensus predictor that improves fold recognition. Protein science, 10(11):2354- 2362, 2001.
Shitong Luo, Jiahan Li, Jiaqi Guan, Yufeng Su, Chaoran Cheng, Jian Peng, and Jianzhu Ma. Equiv- ariant point cloud analysis via learning orientations for message passing. In Proceedings of the JEEF/CVF Conference on Computer Vision and Puttern Recognition, pp- 19932-18941, 2022.
John Moult, Krzysztof Fidelis, Andriy Kryshtafovych, Torsten Schwede, and Anna Tramontano. Critical assessment of methods of protein structure prediction (casp)-round x. Proteins: Struc- fure, Function, and Bioinformatics, 82:1-6, 2014.
Alexey G Murzin, Steven E Brenner, Tim Hubbard, and Cyrus Chothia. Scop: a structural classifi- cation of proteins database for the investigation of sequences and structures. Journal molecular biology, 247(4):536-540) 1995.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Icml, 2010.
David L Nelson, Albert Lehninger, and Michael M Cox. Lehninger principles of biochemistry. Macmillan, 2008.
James O'Connell, Zhixiu Li, Jack Hanson, Rhys Heffernan, James Lyons, Kuldip Paliwal, Abdollah Dehzangi, Yuedong Yang, and Yaoqi Zhou. Spin2: Predicting sequence profiles from protein structures using deep neural networks. Proteins: Structure, Function, and Bioinformatics, 86(6): 629-633. 2018.
Kliment Olechnovič and Česlovas Venclovas. Voromqa: Assessment of protein structure quality using interatomic contact areas. Proteins: Structure, Function, and Bioinformatics, 85(6):1131 1145, 2017.
Christine A Orengo, Alex D Michie, Susan Jones, David T Jones, Mark B Swindells, and Janet M Thortton. Cath-a hierarchic classification of protein domain structures. Structure, 508):1093- 1109, 1997.
Guillaume Pagès, Benoit Charmettant, and Sergei Grudinin. Protein model quality assessment using 3d oriented convolutional neural networks. Bioinformatics, 35(18) 2019.
Alioscia Petrelli and Luigi Di Stefano. On the repeatability of the local reference frame for partial shape matching. In 2011 International Conference on Computer Vision, pp. 2244-2251. IEEE, 2011.
Yifei Qi and John ZH Zhang. Densecpd:
improving the accuracy of neural-network- compu- tational protein sequence design with densenet. Journal of chemical information and modeling, 125- 2020.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S Song. Evaluating protein transfer learning with tape. Advances in neural information processing systems, 32:9689, 2019.
Carol A Rohl, Charlie EM Strauss, Kira MS Misura, and David Baker. Protein structure prediction using rosetta. Methods in enzymology, 383:66-8 0066699,2222 2004.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural net- works. In International conference on machine learning, pp. 9323-9323333 PMLR, 2021.
Marcel G Schaap, Feike J Leij, and Martinus Th Van Genuchten. Rosetta: A computer program for estimating soil hydraulic parameters with hicrarchical pedotransfer functions. Journal of hydrol- 2001.
ogy,
13

---

Under review as a conference paper at ICLR 2023
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Rober Müller. Schnet: continuous filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning, pp. 9377-9388. PMLR, 2021.
Alexandra Shulman-Peleg, Ruth Nussinov, and Haim J Wolfson. Recognition of functional sites in protein structures. Journal of molecular biology, (39(3):607-66)) 2004.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-155 1958, 2014.
Nils Strodthoff, Patrick Wagner, Markus Wenzel, and Wojeiech Samek. Udsmprot: universal deep sequence models for protein classification. Bioinformatics, 36(8):2401-249) 2020.
Alexey Strokach, David Becerra, Carles Corbi-Verge, Albert Ribaz-Riba, and Philip M Kim. Fast and flexible protein design using deep graph neural networks. Cell Systems, 11(4):002-44 2020.
Freyr Sverrisson, Jean Feydy, Bruno E Correia, and Michael M Bronstein. Fast end-to-ent learning on protein surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15272-15281, 2021.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariann neural networks for 3d point clouds. arXiv preprint arXiv:1802.082/// 2018.
Wen Torng and Russ B Altman. 3d deep convolutional neural networks for amino acid environment similarity analysis. BMC bioinformatics, 18(1):1-23, 2017.
Raphael Townshend, Rishi Bedi, Patricia Suriana, and Ron Dror. End-to-end learning on 3d protein structure for interface prediction. Advances in Neural Information Processing Systems, 32:15642. 15651, 2019.
Raphael John Lamarre Townshend, Martin Vögele, Patricia Adriana Suriana, Alexander Derry, Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Bowen Jing, Brandon M Ander- son, Stephan Eismann, et al. Atom3d: Tasks on molecules three dimensions. In Thirty-fitth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Karolis Uziela, David Menendez Hurtado, Nanjiang Shu, Bjöm Wallner, and Ame Elofsson. Proq3d: improved model quality assessments using deep learning. Bioinformatics, 33(1)):1578- 1580, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. $998-6008,70 2017.
AD Vecchio, A Deac, Liò, and Veličković. Neural message passing for joint paratope-epitope prediction. 2021.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
Donald Voet and Judith Voet. Biochemistry. John Wiley & Sons, 2010.
Jingxue Wang, Huali Cao, John ZH Zhang, and Yifei Qi. Computational protein design with deep learning neural networks. Scientific reports, 8(1):1-9, 2018.
Wei Wang, Yan Huang. Yizhou Wang, and Liang Wang. Generalized autoencoder: A neural network framework for dimensionality reduction. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 490-497, 2014.
14

---

Under review as a conference paper at ICLR 2023
Xiao Wang, Sean T Flannery, and Daisuke Kihara. Protein docking model evaluation by graph neural networks. Frontiers in Molecular Biosciences, 8:402, 2021.
Edwin C Webb et al. Enzyme nomenclature 1992. Recommendations of the Nomenclature Commit- of the International Union of Biochemistry and Molecular Biology on the Nomenclature and Classification of Enzymes. Academic Press, 1992.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Informa- fion Processing Systems, 31, 2018.
Jinbo Xu and Bonnie Berger. Fast and accurate algorithms for protein side-chain packing. Journal of the ACM (JACM), 53(4):533-557, 2006.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? International Conference on Learning Representations. 2018.
Jiaqi Yang, Yang Xiao, and Zhiguo Cao. Toward repeatability and robustness of the local refer- ence frame for 3d shape matching: An evaluation. IEEE Transactions on Image Processing, 27 (8):3766-3781, 2018.
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hier- archical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31, 2018.
Adam Zemla. Lga: method for finding 3d similarities in protein structures. Nucleic acids research, 31(13):3370-3372 2003.
Jian Zhang and Yang Zhang. novel side-chain orientation dependent potential derived from random-walk reference state for protein fold selection and structure prediction. PloS one, 5(10): e15386, 2010.
Yuan Zhang, Yang Chen, Chenran Wang, Chun-Chao Lo, Xiuwen Liu, Wei Wu, and Jinfeng Zhang. Prodconn: Protein design using convolutional neural network. Proteins: Structure, Function, and Bioinformatics, 88(7):819-829, 2020.
Dmitrii Zhemchuzhnikov, Ilia Igashov, and Sergei Grudinin. 6denn with roto-translational convolu- tion filters for volumetric data processing. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4707-4715 2022.
15

---

Under review as a conference paper at ICLR 2023
A PROOF OF ROTATION EQUIVARIANCE
A function f taking a 3D vector a E It as input is rotation equivariance, if applying ant rotation matrix R 12x3 on æ leads the same transformations of the output f(x). Formally, R R is rotation equivariance by fulfilling:
R(f(=)) = f(R))
(25)
For notation consistency, we consider each row of the matrix to be an individual 3D vector and use right-hand matrix multiplication here. When performing equivariant message passing protein graph for node with local rotation matrix Og we first transform the vector representations of its neighbors from global reference its local reference, that is, for a particular neighbor node j with vector feature Vi E RRx3 apply our DWNN layers f. and transform the updated features back the global reference. We only one neighbor node uj for u, for simplicity. The updated vector representation V, for node i is
V
[f(
(26)
If we apply global rotation matrix R to all vectors in the global frame, the local rotation matrix O will be transformed to O O,R, and Vj to be Vj R, now the output V; is
V ffV, 0 )o;
(27)
[f(V J
(28)
= -O,,,
(29)
And if instead, we directly apply the rotation matrix original output Vi, we got
V. = J(,,
(30)
V,R
(31)
In other words, rotating the input leads to the same transformations to output, so we can preserve rotation equivariance for vector representations in this way. Note that scalar features remain invariant when applying global rotation, so our pesssag-ppassing paradigm with global and local coordinate transformations is rotation equivariance.
B EXPERIMENT DETAILS
We represent a protein 3D structure as an attributed graph, with each node and edge attached with scalar and vector features that are geometric-aware. We implement our DWNN in the equivariant message passing manner, with 3 layer Directed Weight Perceptrons for all tasks.
B. PROTEIN FEATURES
In this paper, we use attributed graph g = (V,E) to represent the protein structure, where each node corresponds to one particular amino acid in the protein with edges connecting its k-nearest neighbors. Here we set k = 30. The node features V (oI..., uN } and edge features & = (ey Jige are both multi-channel scalar-vector tuples with scalar features like distances and dihedral angles and vector features like unit vectors representing particular orientations.
node i/ represents the i-th residue the protein with scalar and vector features describing its geometric and chemical properties if available. Therefore, node in this graph may have multi- channel scalar-vector tuples (st, Vi), 8, E R6 or 26 V. E p3x3 as its initial features.
Scalar Feature. The (sin, cos} (1/, (,,,., Φ). Here {1b, ws )) are dihedral angles computed from its four backbone atom positions, Co;-1, N., Cos, Ni+1-
Scalar Feature. A one-hot representation of residue if the identity is available.
Vector Feature. The unit vectors in the directions of Co - Cos and Co 1 Cos. Vector Feature. The unit vector in the direction of C33-- -Ca, corresponds to the side-chain directions.
16

---

Under review as a conference paper at ICLR 2023
The edge cty connecting the i-th residue and the j-th residue also has multi-channel scalar-vector tuples as its feature (Sij, Vij). Sij E TR31, Vi E I
Scalar Feature. The encoding of the distance |[aa, CCay-Cap Ca.ll using 16 Gaussian radial basis functions with centers spaced between 0 to 20 angstroms.
Scalar Feature. The positional encoding of j corresponding the relative position in the protein sequence.
Scalar Feature:
The contact signal describes if the residues contact in the space, if |Cay - Cayll < 8 and 0 otherwise.
Scalar Feature. The H-bond signal describes if there may be a H-bond between the two nodes calculated by backbone distance.
Vector Feature. The unit vector in the direction of Caj Cas.
B.2 MODEL DETAILS
Synthetic Task. We uniformly sample points on the sphere and the ratio of positive and negative samples is 1:1, We consider the length of each vector as channel scalar feature E R and the vector itself as one channel vector feature E TR 1x3 The VNN and GVP models in this experiment are set layers. And our is only 1 layer, consisting I Directed Linear Module, 1 NonLinear Module, and Directed Interaction Module. We also train a 3-layer MLP by concatenating the scalar and vector feature as four-channel scalar feature E IR as input. Specifically, the parameter counts of VNN, GVP, MLP, DWP is 24,28,24 and 24. We use Binary CrossEntropy Loss for this two-class classification task. For MLP, the scalar and vector features are concatenated along the channel, which means the feature of each point is vector in R.A. For GVP and VNN, the input scalar and vector are represented by separate neural network components.
For all models we trained in the protein-related task, we use 128 scalar channels and 32 vector channels for each node's 's hidden representations, 64 scalar channels, and 16 vector channels for each edge's 's hidden representations. There is 4 message passing updations in implemented models, where each passing layer consists of 3 stacked DWPs. total there are about 2.9 million parameters of models.
CPD. We train our model a BERT-style recovery target (Strokach et al. 2020), more specifically, dynamically mask 85% residues of the whole protein and let the model recover them using structure information from its neighbors using CrossEntropy loss. During testing, we mask all the residues and recover the whole protein sequence in one forward pass, also try to iteratively predict one residue per forwarding pass.
MQA. Because MQA is regression task, we train model using MSE loss. When testing, compute the different types of correlations between predicted scores and ground truth.
RES. Residue identity requires the model to classify the center residue from local substructure of the protein. We construct a subgraph of each local substructure according to Section B. To guarantee equitable comparison, we retrain all the methods based on our protein graph from scratch.
B.3 TRAINING DETAILS
For the synthetic task, train each model for 1000 epochs with a learning rate of le 3 and plot the training loss for evaluation.
We train models with learning rate 3e - 4 for CPD and 2e 4 for MQA, dropout rate of 10% Srivastava et al. 2014), and Layernorm paradigm. We train all the models on NVIDIA Tesla V100 for 100 epochs for each task.
Our model uses 4-layer message passing, where each message passing consists of 3-layer DWP, The total model consists of about 2.9 million parameters.
17

---

Under review as a conference paper at ICLR 2023
B.4 BASELINE DETAILS
We obtain most of our baseline numerical results from other benchmark papers. But because some models haven't been evaluated on some of the datasets we used, we reimplement them following the same model complexity and training hyperparameters as our models.
In the synthetic dataset, we download the source code from (Deng et al. 2021) and (Jing et al, 2021), both of which use CrossEntropy loss. The number of parameters of VNN, GVP, MLP, and DWP 24, 28, 24, and 24, respectively.
For the RES task Table 1, we implement GCN, GIN, and GAT models, which are trained and evaluated with the same settings as our models. Following the souree code from (Townshend al. 202]]ingg et al. 202I), we also retrain 3D CNN and GVP-GNN. To make fair comparison, we keep the model complexity (e.g. hidden dimensions, layer numbers) and hyperparameters (epochs, random seed) of these baselines the same as our models.
For the CPD task in Table 3, we report the result of St-Transformer from (Ingraham et al., 2019), St-GCN, and GVP-GNN from (Jing et al.) 2020). We modify the source code from Structed- Transformer((ngrahan et al., 2019) to get the result St-GAT and St-GIN by replacing the original attention layer in the encoder and decoder module with Graph Attention Network and Graph Iso- morphism Network. The hidden dimensions and layer numbers are kept the same as our models. In Table 5, adopt most of the results from (Jing al. 2020) except running the St- St-rrasffomer model on the TS50 dataset.
For MQA in Table 4, merge the results of the same task from (Jig al.. 2020 and Structed- Transformer (Ingraham et al. 2019). As GVP-GNN only reports three of six of our metrics, so we reimplement GVP-GNN and DimeNet on our benchmark. Same as the above model, the hidden dimensions and layer numbers are kept the same as our models.
For ablation models, we concatenate scalar and vector features as input and replace all the directed weights with the regular linear layer with scalar weights to get the No DW model. By removing the Interaction Module lying in the final layer of each perceptron, we obtain the No Int model. We also replace local frames with unity matrices to break the equivariance in the message passing part (No equivariance).
In the FOLD and REACT tasks. Kipf et al. and Diehl et al. construct the input graph using the protein's 's contact map and use wisr distances of residues as edge features. Baldassarre et use the spatial features including the dihedral angles, surface accessibility, and secondary structure type of each residue's node feature. They also use distances, sequence distances, and bond types for edge features. Hermosilla et al. consider more fine-grained atom-level spatial information, e.g. covalent radius, van der Waals radius, and atom mass, which leads to better performance. Gligorijevi et al. refer different types of contact maps and geometric distances for the input of GNN. All GNN and CNN baselines make use of 3D information in proteins.
C ADDITIONAL RESULTS
C.I MULTIPLE RUNS ON RES TASK
We conduct different runs RES dataset for both baseline and our models. As shown in Table 6 we report the mean and std results of each method across five different random seeds, our models are still doing best among other methods and have stable, robust performance with few deviations.
Table 6: Results across five different random seeds of models on RES dataset (Mean Std).
Model 3D CNN
GCN
GIN
GAT
GVP-GNN
OA-GCN
OA-GIN
OA-GAT 49.2+0.3
Acc % 45.0+0.4
8.2+0.3
9.240.5
12.4+0.2
48,4+0.4
50.2+0.2
50.8+0.1
C.2 ITERATIVE PREDICTION ON CPD TASK
We tested the sequential decoding of the prediction from the N-side to the C-side and reported the native sequence recovery as shown in Table 7
18

---

Under review as a conference paper at ICLR 2023
Table 7: Iterative prediction results of models both CATH (Short, Single, All) and TS50 dataset.
Model
Short
Single
All
TS50
OA-GCN (iter)
34.1
35.7
45.3
50.9
OA-GIN (iter)
34.4
37.2
45.7
52.2
OA-GAT (iter)
36,0
36,5
36.5
49.6
The results have shown that the sequential decoding approach basically caused degradation of the results, but because we introduced the directed weight and geometric operations, our results are still better than other benchmarks. We have added these results in the supplementary material.
C.3 PROTEIN FUNCTION CLASSIFICATION
Datasets. Fold Classification (FOLD) predicts the structure category of given protein structure. We choose the SCOP v1.75 dataset collected by Murzin et al. (1995) that organizes structures into 3-layer hierarchical classes. In total, there 16, 712 proteins covering 7 major structural types with 1, 195 identified folds. We adopt a reduced dataset from Hou et al. (2018), and remove homologous sequences between and training data sets Family (Fam), Superfamily (Sup) or Fold levels, resulting in three different classification tasks.
The Cizyme-Catallzae Reaction Classification (REACT) study is very similar task as it requires to classify the EC number Webb et al. (1992) of a catalyzed enzyme based on the protein structures. Therefore, we put these two tasks byde-by-side
evaluate the performance of different methods. We use the dataset collected by (Hermosilla Casajus er al.) 2021), containing 37, 428 proteins from 384 types of -aayyyyyee classes and we split them into training, validation set, test set and
ensure sequence or structure overlaps across different sets. In total, there are 29, 215 structures for training, 2, 562 for validation, and 5, 651 for test.
Metrics. Following standard benchmarks Diehl (2019); Hermosilla Casajus al. 202 we use the classification accuracy for evaluating the prediction performance.
Baselines. We compare with CNN-based models and GNN-based models which learn the protein annotations using 3D structures from scratch. For fair comparison, we also do not include LSTM- based transformer-based methods, they all pre-train their models using millions of protein sequences and only fine-tune their models on 3D structures |Bppler & Berger (2018); Alley et al. (2099) Rao et al. (2019): Stroothooff et al. (2020): Elnaggar et al. (2020).
Results. As shown in Table 8 our models demonstrate comparable prediction performance both tasks across different levels of the hierarchy. Classifying the fold and enzyme classes based on the protein structure is one of the key problems in structural biology and our proposed models can be used new tools to conduct function annotations for new proteins.
For FOLD and REACT tasks, all GNN-based baselines use 3D structure information:
Kipf & Welling (016) and Diehl (2019) construct the input graph using the protein's 's con- tact map and use pair-wise distances of residues as edge features.
Baldassarre et al. (2021) use the spatial features including the dihedral angles, surface accessibility, and secondary structure type of each residue's node feature. They also use distances, sequence distances, and bond types for edge features.
Hermosilla Casajus et al. (202T) consider more fine-grained atom-level spatial informa- tion, e.g. covalent radius, der Waals radius, and atom mass, which leads better performance.
Gligorijević al. (2021) refer different types of contact maps and geometric distances for the input of GNN.
Despite not using special training techniques, pre-training tasks, and fine- tuning of hyperparameters, models demonstrate comparable prediction performance in both tasks across different levels of the hierarchy. Especially with the introduction our directed weight with equivariant message passing, it is a big improvement over the normal GNN.
19

---

Under review as a conference paper at ICLR 2023
Table 8: Results Fold and React classification
FOLD
Architecture
REACT
Fold
Sup
Fam
Hou al. (2018)
ID ResNet
17.0%
31.0%
77.0%
70.9%
Derevyanko et al. 2018
3D CNN
31.6%
45.4%
92.5%
78.8%
Kipf & Welling (0166
GCN
16.8%
21.3%
82.8%
67.3%
Diehl 20T9
GCN
12.9%
16.3%
72.5%
57.9%
Hermosilla Casajus ci al. (2021
GCN
45.0%
69.7%
98.9%
87.2%
Baldassarre ct al. (2021)
GCN
23.7%
32.5%
84.4%
60.8%
Gligorieevi et
(202T
LSTM+GCN
15.3%
20.6%
73.2%
63.3%
Our method
OA-GCN
31.2%
39.8%
84.8%
76.0%
Our method
OA-GIN
32.8%
38.3%
85.2%
76.7%
Our method
OA-GAT
29.9%
36.6%
79.0%
77.2%
D COMPARISONS TO PRIOR WORKS
D.1 COMPARISON TO ANGULAR ENGULAR-EXPLICIT MODELS
There are several works that explicitly model angles/torsion angles in GNNs. DimeNet(Gasteiger al., 2019b) uses physical-informee informedd RBF and SBF functions represent distances and angular information. SphereNet Liu et al.] (2021) further employs 3D meaningful spherical coordinate system to represent molecular torsional angles. However, these methods rely on invariant sealar input features for remaining equivariance, and during convolution, these scalar features (distances, angles) are just used for updating hidden representations and stay the same in the next layer updating.
On the technical level, our model also has several differences from other methods when compared with existing studies to consider backbone torsion angles (DimeNet, SpehreNet, etc.), as follows:
RBF-embedded distances and Trigonometric-embeadeee angles are adopted the input fea- ture engineering part (see supplementary material). To better represent the 3D geometries, we also introduce several normal vectors as input node features and edge characteristics. The scalar representations invariant and vectors don't 't consider vector representations and only focus on invariant scalar features.
equivariant. But these methods
2. We capture fine-grained geometries with the help of vector representations and directed weight perceptrons. And representations updated layer-by-layer. Our directed weights are learnable to be expandable for complex reasoning capabilities. But these methods only embed angles and distances into edge features during message updating and lack geometry meaningful operations. They also don't 't have directed weights, which are key contributions to our work.
3. We maintain equivariance with the help local coordinate transformations on vector rep- resentations equipped with rigid residue positions. But these methods don't ton'te consider equiv- ariant vectors, they only condition on invariant scalars. SphereNet does use local spherical systems, but they used for calculating relative locations of atoms, seen feature engineering part.
4. We aim to design new type of vector neurons for geometric learning and a general mes- sage passing paradigm for 3D graphs. We have shown our versatility any existing graph learning frameworks. However, these methods are designed specifically for molecular graphs and are hard extend for other 3D point sets.
D.2 COMPARISON TO OTHER EGNN WORKS
Local-global frame transformation is an important technique for 3D object learning (Petrelli &
Di Stefano 2011 Yang et al. 2018 Luo al. 2022) and is inevitable in the discussions of 31>
20

---

Under review as a conference paper at ICLR 2023
equivariance. Our proposed framework focuses on its computation in the latent space together with our pector-based feature representations, while many existing works only apply rotations to the primal space of explicit geometric/physical properties (e.g. simple cartesian coordinates). It also provides hetter flexibility over message-passing operators by eliminating the requirements for spe- eific feature-update arithmetics this also serves as an adaptor for plugging our designed feature representations into many existing GNN models while guaranteeing equivariance.
Among the EGNNs, we could not agree that our model is simplified model from others. Specif- ically, some of these EGNNS could be considered a special case of our framework. For instance, E(n) GNN (Satorras et al., 202I) combines node coordinates weighted by hidden representations, which could be recognized as linearly updating of simple vector features (coordinate) and unit ma- trix transformations (in analogy to our back and forth transformations).
Additionally, to the best of our knowledge, there is no previous work that integrates vector-based features and directed weighted neural networks into local-global transformation frameworks. We ap- ply transformations on hidden scalar-vector features back and forth for better representations under equivariance, which turns out to show decent performances the protein learning tasks.
D.3 COMPARISON TO OTHER BACKBONE-ORIENTATION MODELS
Using sin, ... Φ. 1. features popular techniques in protein learning tasks (Ingraham et al. 2019 Jing et al. 2021 Ganea et al. 2021), but our models can have a more powerful ability to make of these features. The meaning of Orientation-Aware GNNs is not to use torsional angles as the node and edge features, but for reasoning on them. Our real contribution is to design new type of neural network, which has good capability in sensing these geometric features (e.g. angles, orientations) as well as capturing complex interactions. The synthetic experiment gives an intuitive example our claims, where our designed directed weight perceptrons have more powerful in sens- ing and handling angular-related characteristics. Here we share an even simpler example: Assume the input contains two 3D vectors i1 and 22 described by their coordinates [21, y1, ø1, 22, y2, 22), the problem we solved is that how should we let the model know the angle between n1 and 02. The most straightforward intuition is that the weights are directed vectors, then we could count on the inner products between weights and input vectors to capture the angle information. Furthermore, one wants to retrieve more complex geometric objects, e.g. normal vectors, dihedral angles, or easily solve
parallel surfaces based on input coordinates, employed cross-produs operation that. As we use scalar-vector tuple hidden representations for nodes and edges, and directed weights with coometric-meaningee metric reasoning ability and modeling capability beyond angles, especially approximating nonlinear functions on vector representations.
operators learnable, our models could perform more complex geo-
The most straightforward intuition is that if the weights are directed vectors, then we could count on the inner products between weights and input vectors to capture the angle information. Furthermore, if one wants to retrieve more complex geometric objects, e.g. normal vectors, dihedral angles, or parallel surfaces based on input coordinates, our employed gross-produce operation can easily solve that. As we use scalar-vector tuple hidden representations for nodes and edges, and directed weights with ccometric-meaning
operators are learnable, our models could perform more complex geo- metric reasoning ability and modeling capability beyond angles, especially approximating nonlinear functions on vector representations.
The motivation and function of residue orientations are also different from prior works. Instead of using the angles (orientations) as input features, residue backbone orientations define the coordinate frame of the whole system and used in the equivariant message passing part for coordinate trans- formation. Rigid orientations play the role of bridging local and global coordinates for equivariance in our message passing part.
21

---

Under review as a conference paper at ICLR 2023
E
EXPLANATIONS TO SOME MODULES
E.1
ABOUT THE d'dot
For the s' can be interpreted the summation of the channel-wise dot product be- tween the input features (channels of vectors) and the directed weights (a matrix of vectors, so 3-dimensional tensor). Let us consider the i-th number in the output feature, can be formulated as:
wisk j
Notice that both are 3D vectors, **the dot product can essentially reflect the cosine value between them after normalization Essentially, our method utilizes the dot product operation between the directed weights and the input coordinate features, and more importantly, the weight vectors are learnable. We also recommend you pay attention other operations we introduced for reasoning in complex geometric relations and encouraging efficient scalar-vector feature interactions, For instance, the cross-product operation can create new orthogonal orientations; interaction modules can facilitate feature blending.
E.2 ABOUT THE
PART
The initial motivation of scalar lifting is to enable scalar-vector interactions, which are not well addressed in previous works (for instance, VNN (Deng et al. 202T) and GVP (Ting et al. 2020) can only perform vector-to-scalee mappings with very limited choice of operations). Unprojecting from lower dimensions higher dimensions is indeed bijeeicct on the test data themselves, but is widely adopted approach like the diluted convolution or the transposed convolution that succeeds the projection onto higher dimensional features. Mapping from high dimensions to lower space and then projecting back is also popular technique in FCNs (Kenton & Toutanoval 2019 Jumper jet al. I. Generative Models (Wang et al. (2014), and Manifold Learning, which can be regarded as feature compression and information bottleneck. Therefore, though not unique, the learnable directed weights endow the model with the expressiveness to recover useful information hopefully after training on many data. More intuitively. the scalar lifting operator generates fixed directional vectors with different lengths during inference, which can potentially provide more information during the next round of feature processing and facilitate information flow between scalar and vector features.
The scalar lifting module serves pass scalar information to the vector representation our model. Though we give a geometric interpretation of the intuition, our directed weight is learnable, the operation of the model is complex and its final geometric meaning is non-linear, which is difficult to define simply. The function of the mapping is learned according to the needs of the downstream task and is not limited to reconstruction or compression. From the experimental results, the network designed does achieve excellent performance, demonstrating our strong geometric perception and representation capabilities.
22


Reviews:
[{"review_id": "zeJhhWAZKW7", "reviewer": "Reviewer_XTJW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper designs a directed weight operation to bake the SO(3)-action into the neural network by adding an extra dimension in the weight matrix. Except for the directed weight perceptrons, the paper proposes an equivariant message passing neural network combing the DWP module. The synthetic experiment shows the effectiveness of the DWP, and the experiments on protein 3D structures achieve good performance. Overall, though the experiments are able to show performance enhancement using such a directed weight matrix, the paper has a limited illustration of the DWP module. This paper also contains some typos, need to check more carefully.", "summary_of_the_review": "The paper proposed a DWP plus SO(3)-equivariant GNNs for protein engineering. The merit of the design compared to the existing GVP and other equivariant message passing is not significant and experiments are not sufficient for illustrating the new model's effectiveness.", "strength_and_weaknesses": "**Strength:**\n1. The DWP is novel to combine the SO(3)-rotation and the linear transformation, unlike the VNN model, the authors also consider extracting orientation information with the weight matrix. Adding an extra dimension on the weight matrix is quite similar to the 3D convolution filters that extract 3D information from the 3D image data.\n2. The synthetic experiments are a bonus to show the effectiveness of DWP and beat other related methods.\n3. The proposed DWP module is compatible with other GNN models, which is quite interesting to investigate in many applications that require more orientation information.\n \n**Weaknesses:**\n1. The number of learning parameters will triple the size of the linear transform matrix and the complexity might be too big.\n2. The ablation experiments should also consider keeping the model parameter almost the same as your proposed model since the DWP brings more parameters to the model than the other methods.\n3. More illustrations or experiments to directly research the directed weight matrix compared with the undirected one.\n4. More experiments are needed to verify the quality of the model.\n\n**Question:**\n1. The proposed method (even the illustrative figure) is similar to GVP. What is the merit of using DWP compared to GVP?\n2. Is there any constraint act on the DWP to preserve some property? If we regard the directed weight matrix as the same as the normal weight matrix, we are not sure whether the orientation information in the data is extracted by the DWP module.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ywXin33IqnU", "reviewer": "Reviewer_kS63", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a class of SO(3)-equivariant neural networks called Orientation-Aware Graph Neural Networks (OAGNNs). \nThe model is based on specific layers acting on scalar and vector features, designed to ensure better expressive power (compared to previous works) while maintaining SO(3) equivariance. \n\nCompared to previous related work, the main novelty of the proposed method is to have order-3 tensors of learnable weights acting on the features, effectively expanding classical perceptron weights from scalars to oriented vectors in 3d. \n\nIn the experimental section, the authors focus primarily on tasks related to structural biology. \nThis is a natural playground for SO(3) equivariant models, since many important characteristics of proteins can be described in terms of local frames irrespective of the global orientation of the molecules. \n\nAll results show that the proposed method is significantly better than the tested baselines, on almost all tasks and metrics considered. ", "summary_of_the_review": "There are no major concerns with the paper, but there is margin for improvement. Extending the comparison with previous work is particularly important, since it is not clear whether the proposed method is simply making use of the extra parameters compared to VNN.\n\nI have recommended a weak acceptance, conditional on the authors addressing my concerns above. ", "strength_and_weaknesses": "**Strengths**: \n\n- The paper is interesting and tackles an important problem. I expect that the paper could be of interest to the GNN, computer vision, and computational biology communities. \n- The results are strong and there are no concerns in terms of significance (although confidence intervals are not reported). The design of the model is validated also through ablation studies. \n\n**Weaknesses**: \n\n- Equation (6) implies that $c \\vec W \\in \\mathbb{R}^{C \\times 3}$ for $s \\in \\mathbb{R}^C$ and $\\vec W \\in \\mathbb{R}^{C \\times 3}$.  How is the product $c\\vec W$ defined here?\n- A lot of the geometrical intuition behind the paper is never formally stated. For example: \n\t- Why is equivariance to rigid transformations broken in Equations 7,8?\n\t- What does it mean that the network can \"sense the orientational features\"?\n\t- What set of transformations is the proposed method equivariant to, that VNNs weren't before? \n\t- What does it mean that the proposed method is \"more adaptive to the geometrically meaningful features?\"\n- In Experiment 5.1, do all models have a comparable number of parameters? DWP uses significantly more parameters than VNN, so this should be factored into the comparison. \n- Why is there no comparison with VNN in Experiment 5.2? Is there some reason that makes it impossible to design a GNN with vector neurons? This is important because GVP is the worst-performing baseline in Experiment 5.1, so the more interesting comparison is between OA and VNN. \n- Typos: \n\t- Missing parenthesis in the numerator of Eq. 24, right.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eH7IYZAIL3Y", "reviewer": "Reviewer_EZjF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a graph neural network architecture that uses vectorized neurons, extensive hidden feature mixing operations, and global/local reference frame transformations to produce protein embeddings that are sensitive to or \"aware of\" residue-specific orientations within the larger macromolecule. They first introduce Directed Weights, combinations of multidimensional tensors that generalize scalars to vectors and vectors to higher-dimensional tensors. They then describe the Directed Weight Perceptron module, which performs numerous scalar and vector operations such as linear multiplication, dot products, and cross products. Armed with networks that more naturally describe data and relationships in 3D space, the authors next describe their SO(3) equivariant message passing paradigm. They then describe a few variants of their model adapted to mimic the architectures of recent graph models such as GCNs, GINs, and GATs. Finally, the authors demonstrate their models' performance on a number of protein-related tasks.", "summary_of_the_review": "An interesting method, but the presentation lacks clarity and several design decisions are not well justified.\n\nWhat would improve my score: add clarifications and additional details.", "strength_and_weaknesses": "This paper introduces a novel concept, but the presentation, design choices, and testing methodology could all be much clearer. Moving some of the information from the appendix to the text would clarify the role of various mechanisms presented. For example, discussing more directly the scalar and vector feature constructions earlier in the text and not leaving them to the appendix would justify the use of the separate scalar features in the first place, which may otherwise seem extraneous when compared to feature vectors that could be magnitudes larger. Second, it isn't clear why the Directed Interaction model is necessary. Its stated purpose is to establish a connection between the scalar and vector features, which seems redundant given the operations blending scalar and vector features in the Directed Linear module. Clarification on what this module provides that the latter does not would be helpful.\n\nThe experimental results generally seem reliable and convincing, but it is not immediately clear what configurations of models are being compared to each other. It even seems in some places that conflicting information is given i.e. in B.2 the DWP model is 1 layer for the Synthetic Task but B.3 \"Our model uses 4-layer message passing, where each message passing consists of 3-layer DWP\". The latter further seems redundant with \"there is 4 message passing...3-layer networks.\" One can tease these statements/cases apart, but more concise wording and clearer statement of model architecture/hyperparameters would be beneficial. Further, the synthetic task does not seem convincing without more information about how the points on a sphere were sampled, and statistics such as the variances of the positive and negative samples. Finally, it seems rather astounding that an MLP can match the performance of the most sophisticated models with a comparable number of weights on this task. The universal approximation theorem is relevant for arbitrarily deep and/or wide networks, so this is not sufficient justification. Similarly, it appears that GVP fails completely on this synthetic task, but is tested in later tasks whereas VNN performs well and is excluded without explanation.\n\nThe visualizations are informative but the table layout (tables 3-5) is confusing. There are also many citations to the arxiv version of papers that have been published in ML conference proceedings or other venues. The references should be carefully checked and corrected.\n\nSome specific questions:\n1. Why was GVP chosen over VNN for testing on benchmarks?\n2. What applications outside of protein embedding is this work relevant to?\n3. Why is the Directed Interaction module necessary when feature blending already occurs in the Directed Linear modules?\n4. How are the k-nearest neighbors computed? Is the full N x N distance matrix available?\n5. How could MLPs produce similar results as more efficient graph models with the same number of parameters?\n6. Is there rigorous rationale for averaging Pearson r, Spearman rho, and Kendall tau? Was there consideration of weighting the factors? Is there any redundancy between the metrics?\n7. How exactly were the synthetic task vectors sampled? Were the positive and negative areas of equal area? was the negative area less densely sampled?\n8. Will code and trained models be released?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "c9oEnWC7Vsu", "reviewer": "Reviewer_6r4X", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This is a work on rotationally equivariant feed forward networks for applications in protein folding and related tasks.\nA number of equivariant operations which appear useful for these applications are introduced, such as a cross product operator.\nExperiments are done on several useful tasks using protein structure databases, and significantly improved results are reported.", "summary_of_the_review": "Valuable improvements to ML for protein folding.", "strength_and_weaknesses": "Strengths: simple ideas, broad experimental testing, good performance improvement.\n\nWeaknesses: very empirical, leaves much for future work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject