Learning Hierarchical Protein Representations via Complete 3D Graph Networks
Authors: Limei Wang
Haoran Liu
Yi Liu
Jerry Kurtin
Shuiwang Ji
Venue: ICLR 2023 poster (2023)

Abstract: We consider representation learning for proteins with 3D structures. We build 3D graphs based on protein structures and develop graph networks to learn their representations. Depending on the levels of details that we wish to capture, protein representations can be computed at different levels, \emph{e.g.}, the amino acid, backbone, or all-atom levels. Importantly, there exist hierarchical relations among different levels. In this work, we propose to develop a novel hierarchical graph network, known as ProNet, to capture the relations. Our ProNet is very flexible and can be used to compute protein representations at different levels of granularity. By treating each amino acid as a node in graph modeling as well as harnessing the inherent hierarchies, our ProNet is more effective and efficient than existing methods. We also show that, given a base 3D graph network that is complete, our ProNet representations are also complete at all levels. Experimental results show that ProNet outperforms recent methods on most datasets. In addition, results indicate that different downstream tasks may require representations at different levels. Our code is publicly available as part of the DIG library (\url{https://github.com/divelab/DIG}).

Published as a conference paper at ICLR 2023
LEARNING HIERARCHICAL PROTEIN REPRESENTA- TIONS VIA COMPLETE 3D GRAPH NETWORKS
Limei Wang', Haoran Liu'r, Liu Jerry Kurtin', Shuiwang Ji
Texas A&M University, College Station, TX
2 Florida State University, Tallahassee, FL
(limei, liuhr99, jkurtin, sji] (eil@tamu.eet edu, liuy0cs, fsu.cc..f.ueele
ABSTRACT
We consider representation learning for proteins with 3D structures. We build 3D graphs based on protein structures and develop graph networks to learn their representations. Depending the levels of details that wish to capture, protein representations can be computed at different levels, e.g., the amino acid, back- bone, or all-atom levels. Importantly, there exist hierarchical relations among different levels. In this work, we propose to develop a novel hierarchical graph network, known ProNet, to capture the relations. Our ProNet is very flexible and can be used to compute protein representations at different levels of gran- ularity. By treating each amino acid as a node graph modeling as well as harnessing the inherent hierarchies, our ProNet is more effective and efficient than existing methods. We also show that, given a base 3D graph network that is complete, our ProNet representations are also complete at all levels. Experi- mental results show that ProNet outperforms recent methods on most datasets. addition, results indicate that different downstream tasks may require representa- tions at different levels. Our code is publicly available as part of the DIG library (https: http//gtthhubutee com wwt.mm/divlllbbbbb...
INTRODUCTION
Proteins consist of one or more amino acid chains and perform various functions by folding into 3D conformations. Learning representations of proteins with 3D structures crucial for wide range
of tasks (Cao et al., 2021] [Strokach et al.] 22200) Wu et al.) 2021 Yang et al.) 2019 Ganea et al.
2022, Stärk et al. [022] Morehead et al. [022a by Liu et al. 2020). In machine learning, molecules,
2017 Jumper et al.) 2021 Gao et al. 2021 Gao & Ji 2019 Yan et al. 2022: Wang et al. 2022b Yu et 2022 Xie et al.
proteins, etc. are usually modeled as graphs (L.iu et al.) [022 Fout et al.
2022a b Gui et al. 2022 Luo et al. 2022). With the advances of deep learning. 3D graph neural
networks (GNNs) have been developed to learn from 3D graph data (Liu et al., 2022 Jumper et al.
2021
Xie & Grossman 2018 Liu al. 2222 Joshi et al. 2023). In this work, we build 3D graphs
based on protein structures and develop 3D GNNs to learn protein representations.
Depending on the levels of granularity wish to capture, we construct protein graphs different levels, including the amino acid, backbone, and all-atom levels, as shown in Fig. Specifically, each node in constructed graphs represents amino acid, and each amino acid possesses internal structures at different levels. Importantly, there exist hierarchical relations among different levels. Existing methods for protein representation learning either ignore hierarchical relations within proteins (Ting et al. 2021b ZZang et al. 2023). or suffer from excessive computational complexity (Jing et al. 2021a Hermosilla al.]| (021) as shown in Table In this work, we propose novel hierarchical graph network, known ProNet, to learn protein representations at different levels. Our ProNet effectively captures the hierarchical relations naturally present proteins.
By constructing representations at different levels, our ProNet effectively integrates the inherent hierarchical relations of proteins, resulting in a more rational protein learning scheme. Building on novel hierarchical fashion, our method can achieve great efficiency, even at the most complex
Equal contributions
I Equal senior contributions

---

Published as a conference paper at ICLR 2023
Ser
Met
Glu
Amino acid level
Backbone level
4 ácids
All-atom level
Figure I: Illustration of hierarchical representations of proteins. Different colors indicate different types of amino acids. The filled circles are Ca atoms, and non-filled are other atoms. Each amino acid has different levels of inner structures. From coarse- coarse-grained to fine- une-grained levels, we can use Ca coordinates, backbone atom coordinates, or all-atom coordinates represent the protein structure. Note that we treat each amino acid as a node in the graph modeling despite different levels. The actual atoms are in 3D, and this illustration uses 2D for simplicity.
all-atom level. In addition, completeness at all levels enable models to generate informative and discriminative representations. Practically, ProNet possesses great flexibility for different data and downstream tasks. Users can easily choose the level of granularity at which the model should operate based their data and tasks. We conduct experiments on multiple downstream tasks, including protein fold and function prediction, protein-ligand binding affinity prediction, and protein-protein interaction prediction. Results show that ProNet outperforms recent methods most datasets. We also show that different data and tasks may require representations at different levels.
BACKGROUND
Representation learning of small molecules with 3D structures has been studied recently (Schütt et al. 2017 Klicpera al.) 2020; Liu et al. [022]] Wang et al. 2022a), and existing methods can fully determine 3D structures of molecules (Wang et al. 2022a). However, representation learning of proteins with 3D structures is still challenging due the large number of atoms and special hierarchies that naturally present in protein structures, Existing methods for protein representation learning either ignore hierarchical relations within proteins, or suffer from excessive computational complexity, as explained in Table T Detailed related work is listed in Sec. [ In this section, we first introduce hierarchical structures of proteins in See. [2.]] which inspires us to design hierarchical representations of proteins in Sec. [
We then introduce existing complete 3D graph networks in Sec. 2.2 which can be used to capture protein structures completely.
2.1
HIERARCHICAL PROTEIN STRUCTURES
Proteins are macromolecules consisting of one or more chains of amino acids. Each chain may contain up to hundreds or even thousands amino acids. An amino acid consists of an amino (-NH2) group, a carboxyl (-COOH) group, and side chain that is unique each amino acid. The functional groups are all attached to the alpha carbon (C,)) atom. The Ca atoms, together with the corresponding amino group and carboxyl group. form the backbone of protein. As shown in Fig. [ we can Ca coordinates, backbone atom coordinates, or all-atom coordinates to represent protein structures, leading to three levels of representations. Note that protein structures are traditionally organized into primary, secondary, tertiary, and quaternary levels, and our categorization of levels is different. Next, we can use complete 3D graph networks fully capture protein structures at three levels.
Table Comparisons of existing protein learning methods. Firstly, treating atoms instead of amino acids as nodes leads to high complexity. Here N, and & denote the number of amino acids, the number of atoms, and the average degree a 3ID protein graph, and N > n. In addition, most existing methods only capture one level of protein structures, and only IEConv considers hierarchical relations of proteins using several pooling layers. Our method can learn hierarchical representations at three levels. Lastly, our method can capture 3D structures completely at all levels.
Method
Node
Complexity
Hierarchical
Hierarchical Relations
Complete not
Level
GearNer Zhang at
2025
Amino Acid
O(uk)
Amino Acid
X
GVP WVP-NNN etc. [ing at al. 202 Ingraham et
2019
Amino Acid
O(uk)
Backbone
x
d
vector-gated GVP.NNN Tine 2021a
Atom
Ni) O(Nk)
All-Atom
/
Hemmosila
Atom
All-Atam
x
Amino Acid
/
Ours
Amino Acid
O(nk)
Backhone
/
5 All-Atom
/
2

---

Published as a conference paper at ICLR 2023
2.2 COMPLETE 3D GRAPH NETWORKS
3D Graphs. Many real-world data can be modeled as 3D graphs. A 3D graph can be represented as G G= (V.EE,P)) Here, V {vi)i=1...... is the set of node features, where each V; E R denotes the the set of edge features, where eij R& denotes the edge feature vector for edge ij. P im...., is the set of position matrices, where P. RR denotes the position matrix for node i. k; can be different for different applications. For example, if we treat each atom in a molecule as a node, then h₂- = 1 for each node i. For a protein, if we treat each amino acid as a node, then k; is the number of atoms in amino acid i. In our method, we represent proteins as 3D graphs and learn hierarchical representations of proteins in Sec.
feature vector for node: i. & = {eyy (ejyh.il......
Complete Geometric Representations. As defined in ComENet WWng et al. 2022a). a geometric transformation F(-) is complete if for two 3D graphs G = E") and G (V. E2)), the geometric representations F(G²) F(G²) > 3R E SE(3), for = ,,..., l...., n, P. R R(PP).
Here SE(3) the Special Euclidean group that includes all rotations and translations in 3D. Based on the definition, ComENet proposes complete representation for small molecules with torsion angles and spherical coordinates.
Complete Message Passing Scheme. By incorporating complete geometric representations to the commonly-use message passing framework (Gilmer et al. 2017), we achieve a complete message passing scheme
= UPDATE
MESSAGE (v', Sy F()))
JEA;
where Mf denotes the set of node i's neighbors, and UPDATE and MASSAGE functions are usually implemented by neural networks or mathematical operations.
HIERARCHICAL REPRESENTATION OF PROTEIN STRUCTURES
Notations. To learn representations of proteins with 3D structures, we first model protein a 3D graph G = (V.E, P)) as introduced in Sec. 2.2] Specifically, we treat each amino acid as node and define edges between nodes using cutoff radius. That is, if the distance between two nodes is less than predefined radius, there is an edge between these two nodes. For node i, the node feature V; is the one-hot embedding of the amino acid type. For an edge ij, the edge feature C, an embedding of the sequential distance j - i, following existing studies (Ingraham al. 2019 Zhang jet al. 2023). In addition, the position matrix P. for node includes the coordinates of all atoms in the amino acid if available. Note that the rows in P. are given in fixed atom order. For example, for the amino acid alanine, the atom order in the position matrix is N, C.. C, O, and C3.
Considering the hierarchical structures of amino acids and proteins introduced in Sec. 2.1 propose to learn protein representations at different levels, including the amino acid, backbone, and all-atom levels, as shown in Fig. [ In addition, we aim to capture protein structures completely at each level. Therefore, for levels from top to bottom in Fig. [ we design complete geometric representations as F(G)) Jasse. F(G))aa and F(G),, respectively. By incorporating the designed complete geometric representations into Eq. I we can fully capture protein structures at all levels.
3.1 AMINO ACID LEVEL REPRESENTATIONS
At the amino acid level, we treat each amino acid as node and use C. coordinates as the position of the node. This leads to the most coarse- -rasineg representation of the protein. By ignoring the detailed inner structures like the backbone and side chain of amino acids, methods designed for small molecules can be applied directly to learn amino acid level representations of proteins. To achieve complete representations,
design the geometric representation FFG) hase at the amino acid level as {(d,, 0j., Φ5 Tya) ]=:)), JEN, following ComENet (Wang et al. 2022a). Here (dja, Oj dys)) is the spherical coordinate of node in the local coordinate system of node to determine the relative position of j, and Tj: is the rotation angle of edge ji to capture the remaining degree of freedom.
ComENet is used to obtain complete representations the amino acid level in our study. However, our method is significantly different from ComENet. Firstly and most importantly, we study protein representation learning based the unique structural properties of proteins. As result,
3

---

Published as a conference paper at ICLR 2023
Yj
Amino Acid
Vi
X
C,
Amino Acid
(a)
Figure 2: Illustrations of our proposed backbone-level representations, Construction of the backbone coordinate system for amino acid i. (b) Computation of the three Euler angles between the backbone coordinate systems for two amino acids and [c) Illustrations of determining the relative rotation between amino acids i and for existing methods and our proposed method. The purple dashed line indicates how existing methods determine the relative rotation between and by computing all backbone dihedral angles along the chain. The yellow arrow shows how our method determines the relative rotation between amino acids and j using only three Euler angles.
design hierarchical protein learning framework, which can incorporate the inherent hierarchies in protein structures and can largely advance protein representation learning. ComENet is designed for small molecules whose structures are less complicated than protein structures. Secondly, we seek to obtain complete representations at each hierarchical level, and ComENet can only be applied to learn complete representations at the amino acid level. Given F(a) at the amino acid level, we further design F(G))b bb and F(G) an based on the unique structural properties of proteins to learn complete representations at all levels. Thirdly and technically, even at the amino acid level, we use a different strategy to define the local coordinate system (LCS) for each amino acid, not directly analogizing amino acids in methods to atoms in ComENet. The LCS is used to compute (djz, 0fc. 3e.. Tjs)) for each edge ji. Our strategy is also developed based on protein structures. Specifically, we define the LCS for node i based on nodes i and i following existing protein learning studies (Ingraham iet al. 2019). ComENet defines it based on i-th nearest neighbor ft and second nearest neighbor 8t, which requires extra computation sort the neighbors and find the nearest two. Therefore, our method can reduce the computation of ComENet and is more efficient.
3.2 BACKBONE LEVEL REPRESENTATION WITH EULER ANGLES
Building on the proposed amino acid level representation, we further consider all backbone atoms for each amino acid to derive finer-grate finer-grained protein representations. Since can fully capture C. coordinates via the complete geometric representation F(G) at the amino acid level, the remaining degree of freedom at the backbone level is the rotation between two backbone planes. This is because, with such rotation, we can easily determine the coordinates of other backbone atoms besides C. atoms based rigid bond lengths and bond angles (Jumper et al. (021). Therefore, we propose to use Euler angles to capture such rotation. Specifically, we first define the local coordinate system for an amino acid i as y; = r.V C. t, r. r 2; t, x yi. and x; = y; x 2;, as shown in Fig. We then compute three Euler angles Tay Tay and Tif between two backbone coordinate systems as shown Fig. b) Here, n = 2, x zj is the intersection of two local system, Tey the signed angle between n and X; T is the angle between 2; and Zj. and T3 is the angle from n to xj- By considering these three Euler angles, the orientation between any two backbone planes can be determined, thereby fully capturing backbone structures of proteins. Thus, the complete geometric representation this level is F(G) bo F(G)) base U ( 5 -=...., JEN.
7
Advantages of using Euler angles. Most existing approaches directly integrate backbone information into amino acid features. Specifically. they compute three backbone dihedral angles wi! wf. and wi (Ingrabam et al. [011 Jing et al. 202 Li ct al. 2022) for each amino acid based on N,, Ca.- C. and N++1 atoms as shown in Fig.
Then the sin and cos values for the three angles are part of the node features of amino acid For any two amino acids i and j. if we safely assume > i, the relative rotation of these two backbone triangles is determined by all the amino acids between i and along the protein chain. Thus, the relative rotation is determined by all the (j - 1) x 3 bond rotation angles {w],, w w ) as shown in Fig. c) However, our proposed backbone level method can determine the relative rotation for any two amino acids i and j by only three Euler angles T/, T/ and T,. regardless of the sequential distance j - along the protein chain. Hence, our method can significantly improve the efficiency of representation learning this level.
4

---

Published as a conference paper at ICLR 2023
3.3 ALL-ATOM LEVEL REPRESENTATION WITH SIDE CHAIN TORSION ANGLES
To obtain the most fine-grate ane-grained representations of proteins, we consider all atoms in each amino acid. As introduced in Sec. 2.1 an amino acid consists of backbone atoms and side chain atoms. Therefore, building on our backhone level representation, we further incorporate side chain information, leading to the all-atom level representation as shown in Fig. [ We assume all bond lengths and bond angles in each amino acid are fully rigid (Jumper et al. (022). then the degree of freedom we need to consider is torsion angles in side chains @umper et al. 2021). There are at most five torsion angles for any amino acid. For example, as shown in Fig. S in Appendix[A] the alanine has zero side chain torsion angle, the cysteine has only one, and the leucine has two. Note that only the amino acid arginine has five side chain torsion angles, and the fifth angle is close to O. Therefore, we only consider the first four torsion angles for efficiency, denoted as x² xx2 x² x². _ We list the atoms used to compute side chain torsion angles for each amino acid in Table 7 in Appendix [A] With such side chain torsion angles, we can determine the side chain structure for each amino acid. Based the backbone level representation. the geometric representation at this level is F(G) aul = F(G)abb U [(x], x² X². x x=) Note that although side chain torsion angles are important properties of protein structures, it is only used in recent studies (Jumper et al. 2021) for all-atom coordinates prediction, and none of the existing protein representation learning methods use capture protein structure information. Here we incorporate it for protein representation learning, leading to our all-atom level representation.
Differences with existing all-atom level methods. Several existing stud- ies also consider all-atom information of proteins (Hermosilla et al., 202T Jing et al., 2021a), but our method is significantly different and possesses unique advantages, as illustrated in Fig. Specifically, The vector-gater
Pooling
w/o hierarchies
Mainstream hierarchical
Ours Ours-All-An All-Atom
Figure 3: Illustrations of three kind of all-atom level methods.
GVP-GNN (Jing al., 2021a) belongs to the "w/o hier- archies" methods in Fig. It treats each atom as a node and uses an equivariant GNN to update node features. However, the important hierarchical information is not considered. IEConv (Hermosilla et al. 2021) follows the "mainstream hierarchical" " methods in Fig. 3 It treats each atom as a node and employs several pooling layers to obtain representations different levels. For each level, it needs several intrinsic-extriniic convolution layers for message passing, By treating an atom as a node and employing a large model with more than ten layers, IEConv induces excessive computing costs. Our proposed method effectively preserves hierarchical relations of proteins. In addition, by treating each amino acid as a node and integrating side chain torsion angles as node features, our method has much fewer nodes in constructed graphs, resulting in much more efficient learning scheme. We provide mathematical expressions and explanations in Table We also conduct experiments in Sec. 6.5 to show the efficiency and effectiveness of our all-atom level method.
3.4 COMPLETENESS ANALYSIS
In this study, our primary objective is to develop hierarchical representation learning framework for proteins based on their unique structural properties. After constructing a 3D graph G= (V,E, P) for a protein from the framework, we further achieve complete representations three hierarchical levels. Intuitively, complete representations can capture all details of 3D protein structures and enable our method to generate distinct representations for different 3D graphs, subject to rigid transformations such as rotation and translation. Based on the definition of completeness in Sec. 2.2 and Wang et al.) (2022a), we rigorously show how our method can achieve completeness at all three levels. We also summarize whether existing methods are complete or not in Table []
Completeness of the Amino Acid Level. At the amino acid level, we only consider C. coordinate of each amino acid. The methods designed for small molecules can be applied directly at this level. As shown in Sec. 3.1] we design our geometric representation F(G') of a 3D protein graph based on ComENet (Wang et al. 2022a). Since ComENet is provably complete, and the different definition of a local coordinate system in our method does not affect the completeness proof of ComENet, our method can naturally achieve completeness at this level.
Completeness of the Backbone Level. Given a complete F(G) hiver we rigorously show the geo- metric representation the backbone level F(G) F(G) U {(T)) T3o 7 Dim bi jeN; is
5

---

Published as a conference paper at ICLR 2023
Note that our proof is based on the assumption that all bond lengths and bond angles are fully rigid in amino acids, and this assumption is widely accepted (Jumper et al.) 202I). Intuitively, complete geometric representation at the backbone level can capture all 3D infor- mation of the backhone structure. As protein backbones largely determine protein functions (Lopez) & Mohiuddin 2020 Nelson et al. 2008), capturing fine details of them can benefit various tasks.
complete in Appendix B.1
Completeness of the All-Atom Level. Given a complete hh)) we rigorously show the geometric representation the all-atom level F(G)aa F(G))b U {Xx), Xi, x²+ x=m) is complete in Appendix B.2 With the complete geometric representation at this level, our method can fully capture 3D information of all atoms in a protein. Therefore, our method can distinguish any two distinct protein structures in nature. Especially, our all-atom method can capture side chain structures compared with the backbone level method. Side chains are important for proteins (Spassov et al.. 2007). The tertiary and quaternary structures of a protein are determined by interactions between side chains and environment O''oonoo al., 2010). In addition, interactions between side chains also play crucial role in protein-protein and protein-ligand interactions (Tanaka & Scheraga 1976 Berka ct al. 2009), Overall, the all-atom method can capture information for both inter- and intra-protein interactions, leading to better performances various tasks.
PRONET
Based on the message pass- ing scheme Eq. and the hierarchical geometric rep- resentations in Sec. 3 we propose our ProNet for hi- erarchical protein represen- tation learning as shown in Fig. E The inputs to ProNet are node features, edge features, and geomet- Fol- lowing existing graph neu- ral networks (Schütt et al.) 2017 Wang et al.) 2022a), our ProNet contains several interaction blocks to update node features and one Readout function obtain graph-level represen- tations. The Readout function includes a summation function and several fully-comnecte layers. Specifically, the interaction block, we design our novel hierarchical message passing layer Hier- Geom-MP to learn protein representations based on node features, edge features, and either one level of geometric representations. Hier-GGem-MM is specially designed for protein learning and can effectively capture hierarchical relations of proteins. The Conv is adapted from GraphConv (Morris) cet al., 2019) and is used to update node features based on edge features. detailed description of the model architecture is provided in Appendix C Note that there are three levels of geometric represen- tations, and our model only takes one as input. The three levels of geometric representations result in three levels of ProNet, namely ProNet-Amino Acid, ProNet-Backbong can easily adapt the framework to different downstream tasks by specifying the level of geometric representations. Our experiment results in Sec. also show that different data and tasks may require representations at different levels.
% peay
6 Yar
F0G
/EM
j M
A.
Interaction
4.0.0.c
Internction
Coay
Hler-Geem-MP
0.
(G)
Internction
ML.
MLP
Cony
Cony
MLP
Interaction
Hire-Gron-MP
Ouiput
An illustration of ProNet. || denotes concatenation. Cony denotes a graph convolution layer update node features. Hier-Geom- MP denotes the proposed hierarchical message passing layer. BF denotes basis functions to embed distances and angles. Details of the model architecture and basis functions are provided in Appendix C22
Figure 4:
ric representations.
and ProNet-All-Atom. Users
5 RELATED WORK
Learning protein representations is essential to variety of tasks in protein engineering. Existing
methods for protein learning consider different kinds of protein information, including amino acid
sequences (Öztürk al. 2018 Bepler & Berger, 2019 Rao
Rao et al. 2019 Elnaggar et al. 2021 Bileschi
et al. 2022), protein surfaces (Gainza et al.) 2020 Sverrisson et al. 2021 Dai & Bailey Kellogg
2021 Somnath al. (02I), and protein 3D structures (Fout al. [0117] Gligorijevćć et al. 2021
Baldassarre et al. 2021: et al. 2021a; Cha et al.
2022). Due to recent advances in protein
structure prediction (eenio et al. 2020 Jumper al.
2021 Varadi et al. 2022
Baek et al. 2021

---

Published as a conference paper at ICLR 2023
structures of many proteins becoming available with high accuracy. In addition, protein structures are crucial for protein functions. In this work, we focus on representation learning of proteins with 3D structures. Earlier studies formulate proteins 3D grid-like data and employ 3D CNNs for learning (Derevyanko et al. 2018 Townshend et al. 2021). ). However, the grid-like data is extremely sparse, leading expensive learning cost and unsatisfactory performance. Hence, recent studies model proteins as 3D graphs and use 3D GNNs to learn representations (Hermosilla et al. 2021
Hermosilla & Ropinski 2022 Jing al. 2021b Ingraham et al. 2019 Zhang et al. 2023 Li et al.
2022). Based on the analysis in Sec. 3 and Table previous methods on 3D protein graphs can be
categorized into three levels, including amino acid level, backbone level, and all-atom level. For example, GearNet (Zhang et al., 2023) treats an amino acid node and uses amino acid types as node features, thus it is categorized as an amino acid level method. GVP-GNN (Jing et al. 202 lb) represents protein backhone structures with backbone dihedral angles computed from backbone atoms. Thus is a backbone level method. IEConv (Hermosilla et al.) 2021) treats each atom as a node in protein graphs, therefore, it is an all-atom level method. The differences between existing methods and our ProNet are illustrated mathematically in Table I and explained in details at the end of Sec. 3.1 Sec. 3.2 and Sec. 3.3
6 EXPERIMENTS
We evaluate our ProNet on various protein tasks, including protein fold and reaction prediction, protein-ligand binding affinity prediction, and protein-protein interaction prediction. Detailed de- scriptions of the datasets are provided in Appendix [D We also conduct ablation study on the design of our all-atom method in Sec. 6.5] showing the efficiency and effectiveness of our method. Detailed experimental setup and optimal hyperparameters are provided in Appendix
F Additional experimen-
tal results provided in Appendix F The code is integrated in the 3 part of DIG library (Liu
iet al. 2021) and available at https::/// /ttgt/gtttaa/ com/ http//dvvellabbbaeaa
6.1
FOLD CLASSIFICATION
Protein fold classification Hou et al. 2018 Levitt & Chothia 1976) is
Table 2: Accuracy (%) on fold and reaction elassification tasks. The top two results are highlighted as 1st and 2nd.
crucial to capture protein structure- function relations and protein evolu- tion. Following the dataset and exper- imental settings in Hou al. (2018)
Methud
React
Fold
Fold
Sup-
Fam.
Avg.
GCN (Kpf Welling 2017
67.3
16.8
21.3
82.8
40.3
DeepSTF (Hog et of 2018
70.9
17.0
31.0
77.0
41.7
and (Hermosilla et al. (2021), we eval-
GVP-GNN Ding z021b
65.5
40.8
uate our methods on the fold classifi-
IBConv Helmosil e El 2uzl
87.2
45.0
69.7
98.9
71.2
New IECony [eermonnl Rupanski 2012
87.2
47.6
70.2
99.2
72.3
cation task. A detailed description of
Sommani Rig
78.9
the data is provided in Appendix D In total, this dataset contains 16,712 pro-
DWNN Lre Bi Aizz)
76.7
31.8
37.8
85.2
51.5
GearNer Zhang di 2023
79.4
28.4
42.6
95.3
55.4
GearNet-1BBoon Zilinn cl 2025
83.7
42.3
64.1
99.1
68.5
teins from 1,195 folds. There are three test sets, namely Fold, Superfamily, and Family. We report the accuracies on the three test sets and the average of the three accuracy values in Table 2
GearNet-Edge (Zhing cruu 2114)
86.6
44.0
66.7
99.1
69.9
GearNet-Edge--EECa (Zhing 2023
72.7
ProNct-Amino Acid
86.0
$1.5
69.9
99.0
73.5
ProNet- Backbome PreNet-All-Ato
86.4
52.7
70.3
99.3
74,1
85.6
52.1
69.0
99.0
73.4
The results for baseline methods are taken from original
papers (Hermosilla et al. 202T Somnath et al. 2021 Zhang al. 2023 Li et al.) 2002).
Table shows that our methods can achieve the best results on two of
Table 3: Comparisons between ProNet and other methods
in terms of computational cost on the Fold dataset using the same Nvidia GeForce RTX 2080 Ti 11GB GPU.
the three test sets and the best av- erage value. For Superfamily and Family, our methods outperform all of the baseline methods and achieve similar performance as GearNet-Edge- IICony
Method
Hierarchical
Time (sec.)
Comverge
Level
Train
Inference
Time
Gear Edgg (Zhngg et all 2013
Amino Acid
00M
GearNet lidge cuy Zingg ar 2023
Amino Acid
0OM
IEConv. But uses edge message passing scheme, which is more computationally expen- sive than the node message passing
GVP-4INN dinn Narmo
RackIvite
35
-9hh
IECom [Temoomm
All-Atom
165
22
24
ProNet-
ProNes Hockhone
Bickhone
ProNat-Al-Alor
All-Atoe
~9h

---

Published as a conference paper at ICLR 2023
scheme in method, as discussed in Liu al. (2022). In addition, as shown in Table GearNet- Edge-IEConv can not be trained using one Nvidia GeForce RTX 2080 Ti 11GB GPU due to its high complexity. For Fold, the most difficult one among the three test sets, all of our methods levels can significantly outperform baseline methods, and ProNet-backbone improves the accuracy from 48.3% to 52.7%, demonstrating the good generalization ability of our methods. Our methods also set the new state of the art for the average value.
three
6.2 REACTION CLASSIFICATION
Enzymes are proteins that act biological catalysts. They can be classified with enzyme commission
(EC) numbers which groups enzymes based on the reactions they catalyze [Webb] 1992 Omelchenko
et al. 2010). We follow the dataset and experiment settings in |Hermosilla et al. (2021) to evaluate
methods on this task. In total, this dataset contains 37,428 proteins from 384 EC numbers (Berman
ct al. 2000 Dana et al. 2019). Comparison results are summarized in Table 2 where the results for
baseline methods are taken from original papers (Hermosilla et al., 2021 Hermosilla & Ropinski
2022 Li et al.| 2022)] Zhang al., 2023). Our methods achieve better or comparable results compared
with previous methods. Note that IEConv methods (Hermosilla et al. [202]] Hermosilla & Ropinski
2022) are complicated and have larger numbers of parameters. Specifically, the numbers of parameters
for IEConv methods are about 10M and 20M, while that of our methods are less than 2M.
6.3 LIGAND BINDING AFFINITY
Computational prediction of pootein-ligand binding affinity (LBA) is essential for many downstream tasks in drug discovery as it mitigates the cost of wet-lab experiments and accelerates virtual screening (Huang al., 202Tp. In this task, use the dataset curated from PDBbind (Wang et al.) 2004; Liu et al. 2015) and experiment settings in Sommath et al. (202T). We adopt dataset split with 30% and 60% sequence identity thresholds to verify the generalization ability of our models for unseen proteins. In terms of experiment settings, we employ the two-branch network following (Somnath jet al. (202T) for fair comparison. We use the same ligand network as Holoprot (Somnath et al. (021) and use our ProNet as the protein network. Detailed experimental setup is provided in Appendix
Comparison results are summarized in Table 4 where the baseline results are taken from (Somnath et al. 2)22T) and Townshend et al.] (202T). Results are reported for 3 experimental runs. The detailed standard deviation of experiment results are provided in Appendix F Note that methods in Atom3l different experiment setting than other methods. Therefore, it is not fair to compare our results with Atom3D methods. However, we still include their results in Table case readers are interested in their setting. Specifically, the models in Atom3D the task less challenging. This is because the binding affinity would be highly related to binding structure (Lu et al. 2022), the models that take binding pockets as input incorporate prior information on the binding site, binding pose, and the interaction between protein and ligand. Other baseline methods do not consider such prior information in the input. The results show that our methods achieve either best or second best results on both splits and obtain significantly better results than previous state-of -h-the- methods on the sequence identity 60% split. For our methods at different
trained with binding pockets only, making
Table 4: Results on pootein-ligane binding affinity prediction task. The top two results are highlighted as 1st and 2nd. denotes methods trained with the complex binding pockets only, which provides prior information on the interaction between protein and ligand and makes the task less challenging.
Sequence Identity 30%
Sequence Identity 60%
Method
RMSE J
Pearson
Spearman
RMSE 1
Pearson
Spearman
Arom3D-1DCNN (Townshend et al. 2021
1.416
0.550
0.553
1.621
0.608
0.615
FNNN Towwsshend er 2021
1,568
0.389
0.408
1.620
0.623
0,633
Atom3D GNN- Townshend et al. 2021
1.601
0.545
0.533
1.408
0.743
0.743
DeepDTA Oxttlrk et
2018
1,866
0.472
0.471
1.762
0.666
0.663
Bepler and Berger (211))
Hepler & Berger 2019
1.985
0.165
0.152
1,891
0.249
0.275
TAPE Rao 2019
1,890
0.338
0.286
ProfTrans [Plmap Par al.
2021
1.544
0.438
0.434
1.641
0.595
0.588
MaSII Guinza 2020
1.484
0.467
0.455
1.426
0.709
0.701
IBComv Hermosslla 2 dm
2021
1,554
0.414
0.428
1.473
0.667
0.675
Holoproe-Putt Surbace Sommant
202
1,464
0.509
0.500
1,365
0,749
0,742
HloloproeSSperpiis (Sommah el
1.491
0.491
0.482
1.416
0.724
0.715
ProNet-Amino Acid
1.455
0.536
0.526
1.397
0.741
0.734
ProNet-Backboog
1,458
0.546
0.550
1.349
0.
0.759
All-Atm-l
1.463
0.551
0.551
1.343
0.765
0.761
8

---

Published as a conference paper at ICLR 2023
levels, the all-atom one is best on 5 out of 6 metrics. As the binding affinity may correlate to the chemical reactions on the side chain of protein, the results may imply that the all-atom method can capture more information for both inter- and intra- protein interaction.
6.4 PROTEIN PROTEIN INTERACTION
Protein- prrtiin interactions (PPI) are involved in most cellular pro-
Table 5: Results on the PPI task. The top two results are high- lighted as 1st and 2nd.
cesses and essential for biological applications (Ganea et al. 2022).
For example, antibody proteins bind to antigens to recognize dis-
eases (Townshend et al.] 202T). Following the dataset (Townshend
AUROC
ict al., [019|/revee et al., 2015) and experiment settings in Town-
AoondD 3DCNX Towasheed out 2031
shend et (202I), predict whether two amino acids contact when the two proteins bind. The evaluation metric is AUROC. Re-
AlemnDD-OSS CSN [ [emmmee
6,699
GVP-GNN (ink
0,866
PioNect Anine Acid
0.557
sults in SShw that our all- atom level method outperforms
PioNee Aloo
all previous methods, improving the result from 0.866 to 0.871. In addition, the results for three levels may imply that our all-atom representation can capture more details from side chains on both interacting proteins and thus benefits the binding site prediction.
6.5 OBSERVATIONS AND ABLATION STUDIES
different downstream tasks may require methods at different levels. As shown in Table our ProNet- ProNet-backbone backbone outperforms the methods of the other two levels on function prediction tasks, including fold and reaction classification tasks. This indicates the backbone- can capture details from the folding structure of proteins, rendering better predictions for protein functions. In contrast to function prediction tasks, as shown in Table and T our ProNet-all- atom outperforms the methods of the other levels on most metrics of interaction prediction tasks, namely LBA and PPI tasks. This observation implies that our all-atom level method is able to capture fine-grained side chain structure information, eventually contributing the predictions of binding affinity and binding sites for interacted proteins.
Observations:
method
Ablation studies on all-atom level. As discussed in Sec. 3.3 we con- duct experiments on three all-atom level methods to show the advan- tages of our proposed all-atom level method. We adopt the same base model (Wang et al.) 2022a) for fair comparison. The results are shown in T The first baseline is "w/o hierarchies", where each atom is treated as a node in a 3D protein graph. The performance of this method unsatisfying, possibly due the lack of hierarchical infor- mation in graph modeling. In addition, it takes a much longer time for both training and inference. The second baseline "Mainstream hierarchical" with one hierarchical pooling layer, which naturally requires a two-level architecture. The first level follows the "w/o hierarchies" method, and the second level treats each amino acid node with features obtained by aggregating representations of atoms in the corresponding amino acid. The computational cost is high since two base models are involved for two levels. Our proposed ProNet-All-Are achieves the best performance among the three methods with less computing time. Overall, our method is both efficient and effective.
Table Comparison of three all-atom methods on the Fold dataset. The best results are highlighted in the table. All the models are trained using the same computing infrastructure (Nvidia GeForce RTX 2080 TI IIGB) for fair comparison. The training time is the average time per epoch, and the three methods use similar epochs converge.
Method
Time (sec.)
Accuracy (%)
Train
Inference
Fold
Sup.
Fam.
Avg.
who hierarchies
181.2
18.1
36.9
49.5
94.2
60.2
Mainstream hicrarchical
148.7
17.7
51.5
68.7
99.0
73.1
ProNet-All All--Al---tt Atom
32.1
6,3
52.1
69.0
99.0
73.4
CONCLUSION
Protein structures are crucial for protein functions and can be represented at different levels, including the amino acid, backbone, and all-atom levels. We propose ProNet to capture hierarchical relations among different levels and learn protein representations. Particularly, ProNet is complete at all levels, leading to informative and discriminative representations. Results show that ProNet outperforms previous methods on most datasets, and different tasks may require representations different levels.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported part by National Science Foundation grant IIS IIS-200686] and National Institutes of Health grant U01AG070112.
REFERENCES
Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov. Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):8718776 2021.
Federico Baldassarre, David Menéndez Hurtado, Amne Elofsson, and Hossein Azizpour. GraphQA: protein model quality assessment using graph convolutional networks. Bioinformatics, 37(3): 360-366, 2021.
Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. In International Conference Learning Representations, 2019. URL https://
ggenreview.net/
Karel Berka, Roman Laskowski, Kevin E Riley, Pavel Hobza, and Jiri Vondrasek. Representative amino acid side chain interactions in proteins. comparison of highly accurate correlated ab initio quantum chemical and empirical potential procedures. Journal of Chemical Theory and Computation, 5(4):982-992, 2009.
Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip E Bourne. The protein data bank. Nucleic acids research, 28(1): 2000.
Maxwell L Bileschi, David Belanger, Drew H Bryant, Theo Sanderson, Brandon Carter, D Sculley, Alex Bateman, Mark A DePristo, and Lucy J Colwell. Using deep learning to annotate the protein universe. Nature Biotechnology, pp. 1-6, 2022.
Yue Cao, Payel Das, Vijil Chenthamarakshan, Pin- Chen, Igor Melnyk, and Yang Shen. Fold2Seq: A joint sequence (1D)-fold (3D) embedding-basee baaed generative model for protein design. In International Conference on Machine Learning, pp. 1261-1271. PMLR, 2021.
Minjeong Cha, Emine Sumeyra Turali Emre, Xiongye Xiao, Ji- Young Kim, Paul Bogdan, J Scott VanEpps, Angela Violi, and Nicholas Kotov. Unifying structural descriptors for biological and bioinspired nanoscale complexes. Nature Computational Science, 2(4):243-252, 2022.
Bowen Dai and Chris Bailey-Kellogg. Protein interaction interface region prediction by geo- metric deep learning. Bioinformatics, 37(17))2580-25881 03 2021. ISSN 1367-4803. doi: 10. h0.11993iiinnoommtiichhhaabbbbb URL |tt.://taaa/.aaaa...ro/////////a
btab154
Jose M Dana, Aleksandras Gutmanas, Nidhi Tyagi, Guoying Qi, Claire O'Donovan, Maria Martin, and Sameer Velankar. SIFTS: updated structure integration with function, taxonomy and sequences resource allows 40-fold increase in coverage of structure- annotations for proteins. Nucleic acids research, 47(D1):D482 D489, 2019.
Georgy Derevyanko, Sergei Grudinin, Yoshua Bengio, and Guillaume Lamoureux. Deep convo- lutional networks for quality assessment of protein folds. Bioinformatics, 34123)-4046-4053, 2018.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: Towards cracking the language of lifes code through self-supervised deep learning and high performance computing. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop Representation Learning Graphs and Manifolds, 2019.
10

---

Published as a conference paper at ICLR 2023
Alex Fout, Jonathon Byrd, Basir Shariat, and Asa Ben-Hur. Protein interface prediction using graph convolutional networks. Advances in neural information processing systems, 30, 2017.
Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, D Boscaini, MM Bronstein, and BE Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, 17(2):184-192 2020.
Octavian-Fugen Ganea, Xinyuan Huang, Charlotte Bunne, Yatao Bian, Regina Barzilay, Tommi S. Jaakkola, and Andreas Krause. Independent SE(()eqqivariial models for end-to-end rigid protein docking. In International Conference on Learning Representations, 2022. URL https:// openreview www. (tt.:n/ttoouummdd-G@Gaaa 9mLet]
Hongyang Gao and Shuiwang Graph U-nets. international conference on machine learning, pp. 2083-2092. PMLR, 2019.
Hongyang Gao, Yi and Shuiwang Ji. Topology graph pooling networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. (5(12):4512-44 -518, 2021.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1263-1272. JMLR. org, 2017.
Vladimir Gligorijević, P Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn C Taylor, Ian M Fisk, Hera Vlamakis, et al. Structure- based protein function prediction using graph convolutional networks. Nature communications, 12 (1):1-14, 2021.
Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. GOOD: A graph out-of-distribution benchmark. In The 36th Annual Conference on Neural Information Processing Systems (Track on Datasets and Benchmarks), 2022.
Pedro Hermosilla and Timo Ropinski. Contrastive representation learning for 3D protein structures, www.tee/fofrre
2022. http://atp..eeooeeeeeeeeeeee.eea
IM6
Pedro Hermosilla, Marco Schäfer, Matej Lang, Gloria Fackelmann, Pere-Pau Vázquez, Barbora Kozlikova, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsic convolution and pooling for learning on 3D protein structures. In International Conference on Learning Representations, 2021. URL wttpt::t////m httttp:://ppnneeiieee .et. www..ee/oooounid-1nnsSs
Jie Hou, Badri Adhikari, and Jianlin Cheng. DeepSF: deep convolutional neural network for mapping protein sequences to folds. Bioinformatics, 1295):1295-1303 2018.
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. Advances in neural information processing systems, 2021.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. Advances in Neural Information Processing Systems, 32, 2019.
Bowen Jing, Stephan Eismann, Pratham N Soni, and Ron O Dror. Equivariant graph neural networks for 3D macromolecular structure. In The ICML Workshop on Computational Biology, 2021a.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2021b. URL W.R.httpss///ppeeeeei htt.:/www...e.....eeeeeeeeeo foumm?
1L=IYLJDVSxxJJ
Chaitanya K Joshi, Cristian Bodnar, Simon V Mathis, Taco Cohen, and Pietro Liò. On the expressive power of geometric graph neural networks. arXiv preprint arXiv:2301.09300 2023.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunkosu Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873)5583-555)))
2021.
11

---

Published as a conference paper at ICLR 2023
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representation, 2014.
2014.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations (ICLR), 2020.
Michael Levitt and Cyrus Chothia Structural patterns in globular proteins. Nature, 261(5561): 552-558, 1976.
Jiahan Li, Shitong Luo, Congyue Deng, Chaoran Cheng, Jiaqi Guan, Leonidas Guibas, Jian Peng, and Jianzhu Ma. Directed weight neural networks for protein structure representation learning. arXiv preprint arXiv:2201. 132999 2022.
Meng Liu, Youzhi Luo, Limei Wang. Yaochen Xie, Hao Yuan, Shurai Gui, Haiyang Yu, Zhao Xu, Jingtun Zhang, Yi Liu, et al. DIG: a turnkey library for diving into graph deep learning research. Journal of Machine Learning Research, 22(240):1-9, 2021.
Yi Liu, Hao Yuan, Lei Cai, and Shuiwang Ji. Deep learning of high-order interactions for protein interface prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 679-687 2020.
Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3D molecular graphs. In International Conference Learning Representa- lions, 2022. URL pttptt:::///eenveiieee
net wtte://foommmiiiiisssire
9r
Zhihai Liu, Yan Li, Li Han, Jie Li, Jie Liu, Zhixiong Zhao, Wei Nie, Yuchen Liu, and Renxiao Wang. PDB-wide collection of binding data: current status of the PDBbind database. Bioinformatics, 31 (3):405-412, 2015.
Michael Lopez and Shamim S Mohiuddin. Biochemistry, essential amino acids. 2020.
Wei Lu. Qifeng Wu, Jixian Zhang, Jiahua Rao, Chengtao Li, and Shuangjia Zheng. Tankbind: Trigonometry-wwan neural networks for drug-protein binding structure prediction. bioRxiv, 2022.
Shengjie Luo, Tianlang Chen, Yixian Xu, Shuxin Zheng, Tie- Yan Liu, Liwei Wang, and Di He. One transformer understand both 2D & 3D molecular data. arXiv preprint 1rXiv:2210.01765, 2022.
Alex Morehead, Chen Chen, and Jianlin Cheng. Geometric transformers for protein interface contact prediction. In International Conference on Learning Representations, 2022a. URL
https::/openrev/
net teeee//frum21d=5446555
Alex Morehead, Xiao Chen, Tianqi Wu, Jian Liu, and Jianlin Cheng. EGR: Equivariant graph refinement and assessment of 3D protein complex structures. arXiv preprint 1033
2022b.
Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 1602-4600 2019.
David L Nelson, Albert L Lehninger, and Michael M Cox. Lehninger principles of biochemistry. Macmillan, 2008
Marina V Omelchenko, Michael Y Galperin, Yuri I Wolf, and Eugene V Koonin. Non-homologous isofunctional enzymes: systematic analysis of alternative solutions in enzyme evolution. Biology direct, 5(1):1-20, 2010.
Hakime Öxtürk, Arzucan Özgür, and Elif Ozkirimli. DeepDTA: deep drug-targed binding affinity prediction. Bioinformatics, 34(17)i121-i822 2018.
Clare M Connorr, Jill U Adams, and Jennifer Fairman. Essentials cell biology. Cambridge, MA: NPG Education, 1:54, 2010.
12

---

Published as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. Advances in neural information processing systems, 32, 2019.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: A continuous-filet convolutional neural network for modeling quantum interactions. In Advances in neural information processing systems, pp- 991-1001, 2017.
Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Židek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature, 577(7792):70--710, 2020.
Vignesh Ram Somnath, Charlotte Bunne, and Andreas Krause. Multi-scale representation learning on proteins. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:://pperevi..eet net.:/ewweeeeteeeeeeeeeeeeeeeeee forum?
1d==xEk433 E06
Velin Z Spassov, Lisa Yan, and Paul K Flook. The dominant role of side-chain backhone interactions in structural realization of amino acid code, chirotor: A side-chain prediction algorithm based on side-chain backbone interactions. Protein Science, 16(3):994-506, 2007.
Hannes Stärk, Octavian Ganea, Lagnajit Pattanaik, Regina Barzilay, and Tommi Jaakkola. EquiBind: Geometric deep learning for drug binding structure prediction. In International Conference on Machine Learning, pp. 20503 205221. PMLR, 2022.
Alexey Strokach, David Becerra, Carles Corbi-Verge, Albert Perez-Riba, and Philip M Kim. Fast and flexible protein design using deep graph neural networks. Cell systems, 11(4):402-411 2020.
Freyr Sverrisson, Jean Feydy, Bruno E Corrcia, and Michael M Bronstein. Fast end-to-end learning protein surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12272-1221.. 2021.
Seiji Tanaka and Harold A Scheraga. Medium-and long-range interaction parameters between amino acids for predicting three-dimensional structures of proteins. Macromolecules, 96):945-9550 1976.
Raphael Townshend, Rishi Bedi, Patricia Suriana, and Ron Dror. End-to-end learning on 3D protein structure for interface prediction. Advances in Neural Information Processing Systems, 32:15642. 15651, 2019.
Raphael John Lamarre Townshend, Martin Vögele, Patricia Adriana Suriana, Alexander Derry, Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Bowen Jing, Brandon M. Anderson, Stephan Eismann, Risi Kondor, Russ Altman, and Ron O. Dror. ATOM3D: Tasks on molecules in three dimensions. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL FkDZLpK1M12
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, al. AlphaFold Protein Structure Database: Massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic acids research,
2022.
Thom Vreven, Iain H Moal, Anna Vangone, Brian G Pierce, Panagiotis L Kastritis, Mieczyslaw al.
Torchala, Raphael Chaleil, Brian Jiménez-Garche Paul A Bates, Juan Fernandez-Ren Updates to the integrated pootein-protend interaction benchmarks: docking benchmark version and affinity benchmark version 2. Journal of molecular biology, (7919333311-041 2015.
13

---

Published as a conference paper at ICLR 2023
Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. ComENet: Towards complete and efficient message passing for 3D molecular graphs. In The 36th Anmual Conference on Neural Information Processing Systems, 2022a. URL https:: http://ppnneeeee.eeeeeeeeeeee
nCZMqeWSFJ
Renxiao Wang, Xueliang Fang, Yipin Lu, and Shaomeng Wang. The PDBbind database: Collection of binding affinities for protein- ligand complexes with known three-dimensional structures. Journal of medicinal chemistry, 47(12):9977-2988 2004.
Zhengyang Wang, Meng Liu, Youzhi Luo, Zhao Xu, Yaochen Xie, Limei Wang, Lei Cai, Qi Qi, Zhuoning Yuan, Tianbao Yang, and Shuiwang Ji. Advanced graph and sequence neural networks for molecular property prediction and drug discovery. Bioinformatics, 58(9):2579-2588) 2022b.
Edwin C Webb. Enzyme nomenclature 1992 recommendations of the Nomenclature Committee of the International Union of Biochemistry and Molecular Biology on the nomenclature and classification of enzymes. Academic Press, 1992.
Zachary Wu, Kadina E Johnston, Frances H Arnold, and Kevin Yang. Protein sequence design with deep generative models. Current opinion in chemical biology, 65:18-27, 2021.
Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for accurate and interpretable prediction of material properties. Physical review letters, 120(14):145301 2018.
Yaochen Xie, Sumeet Katariya, Xianfeng Tang, Edward Huang, Nikhil Rao, Karthik Subbian, and Shuiwang Ji. Task-agnostic graph explanations. In The 36th Annual Conference Neural Information Processing Systems, 2022a.
Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: unified review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022b.
Keqiang Yan, Yi Liu, Yuchao Lin, and Shuiwang Ji. Periodic graph transformers for crystal material property prediction. In The 36th Annual Conference on Neural Information Processing Systems, 2022.
Kevin K Yang. Zachary Wu, and Frances H Arnold. Machine-leaming-guidel directed evolution for protein engineering, Nature methods, 16(8):687-694, 2019.
Haiyang Yu, Limei Wang, Bokun Wang. Meng Liu, Tianbao Yang, and Shuiwang Ji. GraphFM: Im- proving large-scale GNN training via feature momentum. In Proceedings of The 39th International Conference on Machine Learning, pp. 25684-25701, 2022.
Zuobai Zhang, Minghao Xu, Arian Rokkum Jamasb, Vijil Chenthamarakshan. Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. In International Conference on Learning Representations, 2023, URL httpt::t///ppppep //openrevieiii net peet/Eorum??l--to3@BBtBt
14

---

Published as a conference paper at ICLR 2023
Appendix
A SIDE CHAIN TORSION ANGLES
HO
HO
HO
O
O
NH
NH,
x
2H X2
2
SH
Alanine
Cysteine
Leucine
Figure 5: Illustration of amino acid structures. The red circles are side chain torsion angles.
Table 7: Atoms for computing the side chain torsion angles for each amino acid.
x"
ALA
ARG
N, Ca. Ca, C.,,
C., Ca, Cyyy Cs C., Cg. C.. Os1
Ca, Cs,
Cz, Co,
Cs, No, Cc,
ASN
N. Ca. Cs, Ci,
ASP
N, Cas Ca, Cit
Ca Ca, Cr, O6i
CYS
N. Ca. Cz. S
GLN
N. Ca. Cs. C.,
C.. Cy, Cs C.. Cyp Cs
Cg. Cs. Oel CR C.py Ca, Oct
GLU
N, C.n Ca C.y
GLY
HIS
N. Cay Ca. C.
C.,, Ca,Ca, Cr, No1
ILE
N, C, Ca. C
C., Cp. C,, Cs
LEU
N. Cas Cs. C.,
Ca. Cg, C.. Ca1
LYS
N, Cas Ca
Can Ca, Ca,y Cs
Cg. Cr C. C3. Ss.
Ca, No
MET
N. Ca. Ca.Co. Ci,
Ca. Cp, C,,
PHE
N. Cas Cs. C.,
Ca. Cg. C.. Co1
PRO
N, Cas Ca, C.y
Cas Co. Cs
SER
N, Ca. Cz. Ox
THR
N, C., Ca. Ow
TRP
N. Cas Ca, C.,
C.. Cy,, Că Ca. Cg. Cy. C61
TYR
N. Ca Cs. C., N.C.n Cp, Cn1
VAL
We list atoms used to compute side chain torsion angles for each amino acid in Table Note that AlphaFold2 (Jumper et al.) 2021) also considers alternative side chain torsion angles. This because some side chains parts are 180 -otttrrtaiii--ymmeeet same physical structure with the internal atom names changed. But in our method, atom names are given, therefore, do not need to consider the alternative side chain torsion angles.
and the torsion angle x and x result in the
B PROOFS
As defined in Wang et al. (2022a) and discussed in Sec. 2.2 the formal definition of completeness for 3D protein graphs shown below.
Definition 1 (Completeness), For protein graphs G (V,E, R and G (V,ER? R²). where R. |= and R = {R2)= i respectively, a geometric representation F(G) is considered complete if
=(G))=F(G') = hE E SE(3), for 1,..., n, R. T(R?).
15

---

Published as a conference paper at ICLR 2023
Here SE(3) is the Special Euclidean group that includes all rotations and translations in 3D. To show whether F(G) is complete, we need to prove in both directions. We first need to show Eq. 2 holds from right to left, which is obvious. This is because our proposed F(G) is based on relative information such as distances and angles, thus it is naturally SE(3) invariant. Secondly, we need to show that Eq. 22-2holds from left right. Essentially, we need to show that 3D structure can be uniquely determined by F(G). In this section, we rigorously show that our proposed F(G) and a(l) are complete.
B.1 PROOF OF THE COMPLETENESS FOR BACKBONE LEVEL REPRESENTATION
Proof. Based Def. D need to prove that the coordinates of all backbone atoms in each amino acid can be uniquely determined given F(b))b- As F(G) buse is complete for amino acid level, the positions of all Ca atoms in a 3D protein graph are determined remaining degree freedom at the backbone level is the rotation between two backbone planes. This because, with such rotation, besides C. atoms based on the rigid bond lengths and bond angles, Hence, building on the amino acid level, we only need to prove that the local coordinate system for each backbone plane can be uniquely determined.
stated in Sec. 3.4 Thus the
can easily determine the coordinates of other backbone atoms
We prove this by induction. First, we denote n as the number of nodes, i.e., amino acids, in 3D protein graph. Apparently, the case representation F(G))b is complete, thus, the locations of all the l backbone planes are uniquely determined. Then we need to prove the proposition holds for the = k + case. Without losing generality, we denote node as the (k + 1)-th node, which is connected to node among existing nodes, forming connected graph G of size (k: + 1). We then prove that the local coordinate system of the (A: + 1)-th backbone plane is uniquely determined by the proposed Euler angles (T); T 7J.). As illustrated in Fig. [b) we use unit vectors (x,, Yt, z:) and (X,+ Yj, 2ÿ) to denote the backbone coordinate axes of node and node j, respectively. The intersection vector between plane x;y; and x is denoted as n = Zi X Zj. Given the Euler angles TJJ, TJ.. Te we have
holds. Assume the case n = k holds that the geometric
x; cos J
X; X n Z; = sin jar
(4)
7: 2j = Co8 Jj+-
(5)
X; co8 Tj+-
n X Xj Zj = sin Tjr-
Then we sequentially prove by contradiction that vectors Zy. Xj. and yy can be uniquely determined by the Euler angles.
Assume the coordination system for the backbone of node j is not unique, i.e., there are alternative unit vectors (x), yy, 2/) satisfying Eq. 3 And the alternative intersection vector is denoted as n' Z, zzy.
Step 1: Prove the intersection n is unique.
Substituting n' into Eq. and Eq. and subtracting the derived equations with Eq. and Eq. respectively, we can derive that
(n- n') 0, (n n') .. 0. Since vectors x; and (n n') are on the same plane perpendicular to 2,, there exist À * 0 such that
(8)
x nn- n' Az,-
(9)
Then we can derive that
Az; 2, = O.
(10)
Since À + and z; is unit vector, Eq. 10 creates contradiction. Therefore, such n' does not exist. The intersection vector n of the planes x:y; and x,y, is uniquely determined by the Euler angle T].
Step 2: Prove 2j is unique.
16

---

compute the unique rotation matrix corresponding to the Published as a conference paper at ICLR 2023
Substituting z into Eq. 5 and subtracting the derived equation with Eq.
derive that
2i (zj zzj) = 0. Besides, based on the proof in Step 1, we have
(11)
n Z; X Zj:
(12)
Z; X z)j.
By subtracting the above equations on both sides, we have
Z; (z) z/) 0.
(13)
Eq. and Eq. are contradicted since the non-zero vector (z, 13y-2, 22)) both parallel and perpendic- ular to the unit vector 2,- Thus, 2j is uniquely determined by 7). and 7J6-
Step 3: Prove xj is unique.
Substituting x3, into Eq. and Eq. and subtracting the derived equations with Eq. and Eq. respectively, we can derive that
n (x) x²) = 0, n x (x, x .2j = O. As (x) - -x)) and n are on the same plane perpendicular to zy. n (x) xf) puj holds for some * 0. Thus, we can derive that
(14)
pzz 2ッ = 0,
(15)
which contradicted to the fact that f + and 2 is a unit vector. Therefore, x, can not have alternative solutions, i.e., x, uniquely determined by TJes TJa TJe
Step 4: Prove y, is unique.
Since z, and xj are unique. y, Zj x, also uniquely determined by the Euler angles.
The geometric representation F(G) F(G)) buse U ((T TJ 7 h= i.E........ JeN, on backbone level provides unique representation for different protein backbone structures. Thus, the backbone level representation F(G) hb is complete.
With the complete representation, three static Fuler angles J&b Tja T7. as
=2
M = MrMs
(16)
where
cos T sin Tj
sin 7 cos Jj
cos
sin 3
Mi
M2
0
COS T
sin cos Tj
Ma
sin
0
0
sin Jj
Thus, given the unit vectors (x;, Vi, zi) of node i, we can derive the backbone coordinate system of node j as
X,
x;
yj
M
y: Z;
(17)
Zj
B.2 PROOF OF THE COMPLETENESS FOR ALL-ATOM LEVEL REPRESENTATION
Proof. To prove completeness at the all-atom level, based on Def. we need to prove the positions of all atoms in each amino acid are uniquely determined with F(G) Since the geometric representation at the backbone level is complete, the backbone structure of given protein can uniquely determined by F( Therefore, for each amino acid i, we only need to prove that all the atoms of the side chain are uniquely determined by four side chain torsion angles. Note that all bond lengths and bond angles in each amino acid are fully rigid, thus only consider the unit vector between atoms
17

---

Published as a conference paper at ICLR 2023
in amino acid. Here we provide rigorous proof for the amino acid cysteine. The proof can be generalized to other types of amino acids.
A eysteine has six atoms, including N, Ca, C,O,Cs, and S.. Firstly, the positions of N, Cas C, and are determined at the backbone level. We can easily further determine the position of C3 since the atoms N, C.. C, and C3 are in rigid group shown in Table 2 of (Jumper et al. [02T). Therefore, we only need to prove that the position of atom S, is uniquely determined. For an amino acid cysteine with node index
P. p and p denote the unit vectors of r.." -re ; F i The unit vectors of p x P and p x P are denoted as a, and b, Given the side chain torsion angle X! we have
Cp r and
a; b; cos X
(18)
a; x b; P = sin sin
Assume the position of atom Sy is not uniquely determined by x3. then there is an alternative position of S., satisfying Eq. 18 The new unit vector from C3 to S, denoted as p and bj P p
Substituting b into Eq. 18 and subtracting the derived equations with Eq. 18 we can derive that
(b; b)) = 0,
(19)
(b; bj) pp. O.
Since vectors a, and b; - by are perpendicular p`. a, (b; - by) ppp holds for some p * Then derive that
a, x (b; b;)
pp² p =
0.
(20)
Since o * 0 and P is unit vector, Eq. 20 creates contradiction. Therefore, such p does not exist, and the position of atom S.y is uniquely determined by X;
C PRONET
In this section, we provide details about the geometric representations and the model architecture.
C.1 GEOMETRIC REPRESENTATIONS
The geometric representation at the amino acid level is F(G)) (djji, Oji, Ojú, Tj:)) jeN, as introduced in Sec. 3.1 For each edge ji, we need to compute four geometries based on the positions of nodes j, 2 i, + 1,j and 1. We use P. P Paj. p/. P denote the unit vectors r;-1 rj rs+1 r;, - r,, r- rj and r - ry- Then the four geometries for edge jú are computed based on
0j: arccos (P) Pij)
n1 d p n2 = p Pij.
6j = atan2 (nı n2, n2 X n2),
(21)
[ ifj=i-1 Dj
P ifi 1
p; [p] otherwise
p otherwise
n3 = Pij X Pi, n1 = - X Pjr
Tj atan2 (nz n4, n2 n4).
As introduced in Sec. 3.2 the geometric representation at the backbone level is
F(G),. = F(G) U U(TT (2)) J=..........
{(dji fyt Ojss T=e))==........ meM. [T]) The h heN.
(22)
{(dzi, Ozt, Djs TJss 7/ 7/ ;= JEM.
=
The steps compute the Fuler angles +1 - 3 provided in Sec. 3.2 and Fig.
18

---

Published as a conference paper at ICLR 2023
As introduced in Sec. 3.3 the geometric representation at the all-atom level is F(G)an = F())bb U ((x) xi. xxxxxxxx)== x=))e=
(23)
= {(dzi, Oze, Øyx; Tye T/a Ths H=,,.,, JEN; U ((x) xi: X x2)
The atoms used to compute the side chain torsion angles x² x² x3 x² are provided in Table
C.2 MODEL ARCHITECTURE
Overall architecture. As shown in Fig. 4 the architecture of ProNet contains several interaction layers and output layer. Each of the interaction layers updates node features based the message passing scheme in Eq. [ Specifically, for an interaction layer, the inputs are node features, edge features, and geometric representations, and the outputs are updated node features. Given the inputs, we firstly construct two intermediate updated node features. The first one is obtained by the Conv layer, and the second one is obtained by the Hier-Geom-MP layer. Then we concatenate these intermediate updated node features and use several fully- cully-connected layers to obtain the final output of this interaction layer. Following interaction blocks, the final protein representation g is obtained with the output layer, which is implemented with a READOUT function:
g READOUT ((t/..))...)))
(24)
Here, v indicates the feature vector of node i at the last layer. Specifically, the Readout function includes a summation function and several fully-connected layers.
Basis function. We use basis functions to embed our proposed geometric representations. Specifically, we use spherical harmonics to encode distance d and angles 0, 0, T, T 72, , following Liu al. (202)). Formally, (d, 0, Φ) is encoded with je (Itm d Y (0, o). and (d, r). (d, (d, 72), (d, 79)
are encoded with je [ee d) (a). Here a can be 11, 72, or je(-) is spherical Bessel function of order f, T is spherical harmonic function of degree and order e, is the cutoff, and 8en is the n-th root of the Bessel function of order In addition, we use sin and cos functions embed the side chain torsion angles x xxxxxx"xx* x²
Conv block. The Conv block is adapted from the GraphConv layer (Morris et al. 2019) to update node features. Specifically, given input node features {v:Ji=l,.... and edge attributes Liijii the updating function is m Wiv! + W₂ Jee, v o (Wafe)). Here @ denotes element-wise multiplication, f.j can be edge features Cij encoded geometric representations.
D DATASET DESCRIPTION
Fold Dataset. We use the same dataset as in Hou et al. (2018) and Hermosilla et al. (2021). In total, this dataset contains 16,292 proteins from 1,195 folds. There are three test sets used to evaluate the generalization ability, namely Fold in which proteins from the same superfamily are unseen during training. Superfamily in which proteins from the same family unseen during training, and Family in which proteins from the same family are present during training. Among the three test sets, Fold is the most difficult one since this test set differs the most from the training set. this task, 12,3 12 proteins are used for training, 736 for validation, 718 for Fold, 1,254 for Superfamily, and 1,272 for Family.
Reaction Dataset. For reaction classification, the 3D structure for 37,428 proteins representing 384 EC numbers are collected from PDB (Berman al. 2000), and EC annotations for each protein are downloaded from the SIFTS database (Dana et al. 2019). The dataset is split into 29,215 proteins for training, 2,562 for validation, and 5,651 for testing. Every EC number represented all splits, and protein chains with more than 50% similarity are grouped together.
LBA Dataset. Following Sommath et al. (222I) and Townshend et al. (021). we perform ligand binding affinity predictions on subset of the commonly-used PDBbind refined set (Wang et al. 2004; Liu et al. (013). The curated dataset of 3,507 complexes is split into train/val/test splits based on 30% or 60% sequence identity threshold to verify the model generalization ability for unseen proteins. For a protein-ligand complex, we predict the negative tog-tranfforeed binding affinity pK -logoo K) in Molar units.
19

---

Published as a conference paper at ICLR 2023
Table 8: Model and training hyperparameters for our method on different tasks.
Values/Search Space
Hyperparameter
Fold
Reaction
LBA
PPI
Number of layers
3. 5,
4,
3, 4. 64, 128, 256
3,4,
Hidden dim
64, 128, 256
64, 128, 256
64, 128, 256
Cutoff
8,
6, 10
10
30
Dropout
0.2, 0.3, 0.5 1000
0.2, 0.3, 0.5
0.2, 0.3
Epochs
400
300
20
Batch size
16, 32
16, 32
8, 16, 32, 64
8, 16, 32
Learning rate
le-4, 2e-4, 5e-4
le-4, 2e-4, 5e-4
le-5, 5e-5, le-4, 5e-4
le-4, 2e-4, 5e-4 0.5
Learning rate decay factor
0.5
0.5
0.5
Learning rate decay epochs
100, 150, 200
50, 60, 70, 80
50, 100
4, 8, 10
PPI Dataset. Following the dataset and experiment settings in TTwnshend et al. (2021) we predict whether two amino acids contact when the two proteins bind. We use the Database of Interacting Protein Structures (DIPS) (Townshend et al. 2019) for training and make prediction on the Docking Benchmark 5 (DB5) (Vreven et al. 2013). The split of protein complexes ensures that no protein in
the training dataset has more than 30% sequence identity with any protein in the DIPS test set or the DB5 dataset.
E EXPERIMENTAL SETUP
This section describes the full experiment setup for each task considered in this paper. The imple- mentation of our methods is based on the PyTorch (Paszke et al.) 2019) and Pytorch Geometric (Fey Lenssen 2019). and all models are trained with the Adam optimizer (Kingma & Ba 2014). All experiments conducted for model and training hyperparameters are listed in Table g Note that we select hyperparameters at the amino acid, backbone, and all-atom levels by the same search space, and optimal hyperparameters are chosen by the performance on the validation set.
single NVIDIA GeForce RTX 2080 Ti 1 IIGB GPU, The search space
Fold and Reaction dataset. Similar to Hermosilla al. 2021). apply data augmentation techniques to increase data on fold and reaction classification tasks. Specifically, for the input data, we apply Gaussian noise with a standard deviation 0.1 and anisotropic scaling in the range [0.9, 1.1] for amino acid coordinates. The same noise is added to the atomic coordinates within the same amino acid, ensuring that the internal structure of each amino acid is not changed. We also mask the amino acid type with probability of 0.1 or 0.2. For each interaction layer, we employ Gaussian noise with a standard deviation of 0.025 to both features and Fuler angles further enhance the robustness of our models. We also find that warmup can further improve the performance on reaction classification.
LBA dataset. We follow the experiment settings Somnath et al., (2021) for LBA tasks. Since our proposed methods focus protein representation learning. we employ a two-branch network for fair comparison. One branch of the network provides the representations for protein structures using our methods and the other branch generates the representations for ligands with a graph convolutional network. We employ the same architecture for the ligand branch as in SSomath et al. (2021) and use models the protein branch. A few fulle-connected layers then applied to the concatenations of protein and ligand representations to obtain the final representation of the corresponding complex.
F ADDITIONAL EXPERIMENTAL RESULTS
F.1 RESULTS ON LBA
Results on LBA Dataset with standard deviation are listed in Table and Table I0
F.2 RESULTS ON ADDITIONAL DATASETS FROM ATOM3D
We also conduct experiments on additional datasets from Atom3D (Townshend et al., (022). specifi- cally on Protein Structure Ranking (PSR) and Ligand Efficacy Prediction (LEP) datasets. Detailed descriptions of these datasets are provided below.
20

---

Published as a conference paper at ICLR 2023
Table 9: Results with standard deviation LBA dataset split by sequence identity 30%. The top two results are highlighted as 1st and 2nd. denotes methods trained with binding pockets only.
Sequence Identity 30%
Method
RMSE
Pearson
Spearman
AtoN3D-3DCNN* Townshend al 2021
1,466 0.021
0,550 + 0.021
0.553 =0,009
Atom3D-ENN Townshend of ml. 2121
1.568 + 0.012
0.389
0.024
0.408 + 0.021
Atom3D-GNN Townshend
1,011 0.048
0.545
0.027
0.533 0.033
DeepDTA Oztürk al.
2018
1866 0.080
0.472
0.022
0.471 = 0.024
Bepler and Berger [201
[Bepler & Berger 2019
1,985 = 0.006
0.165 = 0.006
0.152 = 0,024
TAPE (Ran et al. 2019
1890 0.035
0.338 0.044
0.286 + 0.124
ProfTrans [Binuggir
2021
1,544 0.015
0,438 0.05
0,44 0.058
MaSIF (Ganza el ml. 40201
1.884 + 0.018
0.467 + 0.020
0.455 + 0.014
GVP* di 20/8
1.594 = 0.073
IEConv [Hemmsi
2021
1.554 # 0.016
0.414 + 0.053
0.428 + 0.032
Holoprot-Full Surince bomtiant et al.
2021
1.444 = 0.006
0.509 0.002
0,500 = 0.005
Holoprot-Superpixel (Somath al
2021
1.491 0.004
0.491 0.014
0.822 - 0.032
ProNet-Amino Acid
L455 0.009
0.536
0.012
0.526 0.012
ProNet-Backhone
1,458 = 0.003
0,546
0.007
0.550 = 0.008
ProNet-All All-Atomm
1.463 0.001
0.551 0.005
0.551 + 0.008
Table 10: Results with standard deviation on LBA dataset split by sequence identity 60%. The top two results are highlighted as 1st and 2nd. denotes methods trained with binding pockets only.
Sequence Identity 60%
Method
RMSE
Pearson
Spearman
Atom3D-3DD D (Towushend et al. 2021
1.621 +: 0.025
0.608 +: 0.020
0.615 : 0.028
Alom3D-ENN' (Townshend et al. 2021
1.620 + 0.049
0.623 + 0.015
0.633 + 0.021
Atom3D-GNN* Townstenn
1.408 0.069
0.743 : 0.022
0.743 0.027
DeepDTI (Ottirk et 2018
1.762 0.261
0.666 0.012
0.663 0.015
Bepler and Berger (2019)
Bepler & Berger 2019
1.891 + 0.004
0.249 + 0.006
0,275 + 0.008
TAPE (Rag et al. 2019
1.633 +: 0.016
0.568 + 0.033
0.571 + 0.021
Pro[Trans [Emaggee
2021
0.016
0.595
0.014
0,588 0.009
MaSIF (Gainz et al 2020
1.017
0.709 + 0,008
0,701 + 0.001
IEConv [Hermoslla ct 2021
0.024
0.667 1 0.011
0.675 + 0.019
Holopre:Fll Surface (Somnith al.
2021
1.365 0.038
0.749
+ 0.014
0.742 + 0.011
Holoprit Holopiet-Sperppxell Sommalh el al. 2021
1.416 -:: 0.022
0,724 + 0.011
0,715 +: 0,006
ProNet-Amino Acid
1.397 0.018
0.741 + 0.008
0,734 + 0.009
ProNet- Backhone
1.349 0.019
0.764
0.006
0.759 0.001
ProNet-All-Aton
1.343 + 0.025
0.765 + 0.009
0.761 : 0.003
PSR (Protein Structure Ranking). This task aims to predict the global distance test (GDT_TS) for each pro- tein and is formulated as a regres- sion task. In terms of the evaluation metrics, Rg is Spearman correlation. Mean Rs measures the correlation for structures corresponding to the same biopolymer, whereas global Rs mea- sures the correlation across all biopolymers. The results are listed in Table [T] As shown in the table, our methods can outperform all baseline methods and significantly improve mean Rs results.
Table 11: Results on the PSR task. The top two results highlighted as 1st and 2nd.
Method
Mean Rs f
Global Rg 1
Mom3D-33DNN (Townshend et al 2021
0.431
0.789
Atom3D-GNN (Towashend ctal. 2021
0.411
0.750
GVP-GNN (Jing ar 20218
0.511
0.845
ProNet-Amino Acid ProNet-All-Aton
0.621
0.795
0.638
0.845
0.632
0.849
LEP (Ligand Efficacy Prediction). This task aims to predict whether a molecule bound to the structures will be an activator of the protein's 's func- tion or not. This task is formulated as binary classification task, and the evaluation metric is AUROC. The re- sults are listed in Table 12 As shown in the table, our methods can outper- form all baseline methods.
Table 12: Results on the LEP task. The top two results are highlighted as 1st and 2nd.
Method
AUROC
Atom3D-3DCNN (Townshend et al. 2021
0,589
Atom3D-GNN Townshend et al. 202
0.681
Atom3D-ENN (lownshend el al. 2021
0.663
GVP-GNN (Jing et. 2021a
0.628
ProNet-Amino Acid
0.646
ProNet-Backbone
0.687
ProNet-All-Atom
0.692
21


Reviews:
[{"review_id": "w7j9K6QXXT", "reviewer": "Reviewer_X8gX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a hierarchical graph neural network for protein property prediction. It models a protein at three levels -- amino acids, its backbone atoms, and its side chain atoms. Each level is modeled by a ComENet layer that learns a complete representation of the geometric structure. The method is evaluated on multiple protein classification tasks: fold classification, reaction classification, and ligand binding affinity prediction and compared with a variety of existing methods. ", "summary_of_the_review": "Overall, I vote for a weak acceptance of the paper because it provides a completeness proof for the proposed architecture, which can be valuable to the community.", "strength_and_weaknesses": "Strength:\n1. The evaluation is fairly comprehensive. It included most of the state-of-the-art methods published recently.\n2. Authors provide completeness proof for each layers, which is important for understanding the power of these geometric neural networks.\n\nWeakness:\n1. The improvement of the proposed method over previous baselines are relatively small. In particular, on the ligand binding affinity prediction task, the model underperforms the 3DCNN model (which is not equivariant) under 30% sequence identity cutoff. The 30% identity cutoff is a harder setup that requires more generalization to novel protein structures and therefore more important than 60% identity cutoff. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "uRGJYFSf9O", "reviewer": "Reviewer_S9jc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This concerns learning the hierarchical representation (e.g., a.a., backbone, all-atom) for proteins with 3D structures. A novel hierarchical graph network, termed ProNet, was developed. ProNet is flexible, efficient and effective. It is shown that ProNet representations are complete at all levels if the base 3D graph network is complete. The experimental result shows the proposed model is able to achieve state-of-the-art for multiple tasks. It also shows that different downstream tasks my require different levels of representations.", "summary_of_the_review": "This paper provides an effective way to represent proteins. Its idea is straightforward, and the results are promising. There are a few places that need more explanations.", "strength_and_weaknesses": "Strength: The hierarchical design of the representation improves the computational cost and model performance at the same time. Representations at different level can be used to solve different tasks.\n\nWeaknesses: The all-atom level representation share $F(G)_{bb}$ with the backbone level representation. The backbone level representation share $F(G)_{base}$ with the amino acid level representation. The importance of such redundant design is unclear.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_OiV2eEiPxZ", "reviewer": "Reviewer_KnXi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a complete protein learning method based on hierarchical representations: amino acids, backbone, and all-atom level. ", "summary_of_the_review": "The authors provide a new protein representation learning method. But I am mainly concerned about its novelty and significance. ", "strength_and_weaknesses": "This paper showcased a hierarchical graph network, \"ProtNet\", which computes the protein representations at different hierarchical levels. The authors investigated and integrated the physical protein (amino acid chains) conformational properties into their methods. Their experiments indicated the current method is largely on par with other SOTA methods, with ${maybe}$ marginal gain in some metrics. \n\n1. The whole method is largely based on a very recent 2022 Neurips paper ComENet. Of course, ProtNet exploits unique properties of proteins and further adapted the ComENet to specialize for proteins. But the argument in 3.1 about \"significantly different from ComENet\" is not convincing enough and I would like to see the authors could further elaborate on the innovations and novelties in the current work.\n 1.a) The first reason is ProNet is \"based on the unique structural properties of protein\". Thus, it is a special case of ComENet. Certainly, protein is a polymer of amino acids, thus providing more structural information for leveraging than general molecules (in which case we can also argue for greater generalization of the latter though). All properties (e.g., dihedral angles, side chain torsion) the authors discussed in the paper are long-known physical measures in protein characterization and have been exploited in various ML methods. \n 1.b) Regarding ProtNet can represent backbones and all-atoms, this again has been seen in many previous ML methods for protein representations and while torsion angles can uniquely identify the conformation, I would like to understand better how simply adding them as node attributes archives \"all-atom representation\" \n 1. c) The local coordinate system that is based on neighbor amino acids (they are linked by bonds) is a natural property of chains/sequences/polymers/proteins. The correct analogy should be atoms linked by chains in ComENet. \n2. Moreover, I did not find any new discoveries, insights or learning opportunities in this work. The authors might be able to strength the paper by providing deeper analyses and understandings of the methods, and fundamentals ...\n3. As the experiments largely followed (Townshend et al., 2021), it would be more convincing if we do all protein tasks exhaustively (e.g., RES, MSP, PSR, LEP are missing). \n4. Additionally, in table 4, the 60% similarity cutoff is higher than the original 30% setup. (a) is there a reason we see higher RMSE for Atom3D? (b), doesn't it mean ProNet has less generalization power on more similar proteins?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "JzmmvE8U0Aj", "reviewer": "Reviewer_hFzJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a framework, ProNet that integrated different level of protein information for protein related tasks. It showed improved performance on available datasets. ProNet is thus consisted by multiple components. The author also noticed that different tasks may require different level of representations.", "summary_of_the_review": "I regard this paper is quite complete. But I find the contribution is quite marginal and lack some more in-depth analysis. ", "strength_and_weaknesses": "Strength:\nThis paper is well written. It also provide complete analysis of complete graph.\nThey conducted lots experiment to show the performance difference compared with previous methods.\nThey method also focus on amino acid rather than atoms that make both the data and the model to be light.\n\nWeakness:\nThe method aims to integrate all level of protein information. But however, the performance increase is rather marginal compared with existing methods. That raised some question, whether such different level of information is needed. If so, how to integrate them. However, I do not think the author address these questions. For example, how about different integration methods.\nThis method has many components and lots of hyperparameters. I would recommend to conduct a more through ablation study. It could also bring some insights on why integrating different level of information.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster