Intrinsic-Extrinsic Convolution and Pooling for Learning on 3D Protein Structures
Authors: Pedro Hermosilla
Marco Schäfer
Matej Lang
Gloria Fackelmann
Pere-Pau Vázquez
Barbora Kozlikova
Michael Krone
Tobias Ritschel
Timo Ropinski
Venue: ICLR 2021 Poster (2021)
Keywords: classification
bioinformatics

Abstract: Proteins perform a large variety of functions in living organisms and thus play a key role in biology. However, commonly used algorithms in protein representation learning were not specifically designed for protein data, and are therefore not able to capture all relevant structural levels of a protein during learning. To fill this gap, we propose two new learning operators, specifically designed to process protein structures. First, we introduce a novel convolution operator that considers the primary, secondary, and tertiary structure of a protein by using $n$-D convolutions defined on both the Euclidean distance, as well as multiple geodesic distances between the atoms in a multi-graph. Second, we introduce a set of hierarchical pooling operators that enable multi-scale protein analysis. We further evaluate the accuracy of our algorithms on common downstream tasks, where we outperform state-of-the-art protein learning algorithms.

Published as a conference paper at ICLR 2021
INTRINSIC-EXTRINSIC CONVOLUTION AND POOLING FOR LEARNING ON 3D PROTEIN STRUCTURES
Pedro Hermosilla Ulm University
Marco Schäfer Tübingen University
Matëj Lang
Masaryk University
Gloria Fackelmann Ulm University
Pere Pau Vázquez University of Catalonia
Barbora Kozlíková
Masaryk University
Michael Krone
Tobias Ritschel University College London
Timo Ropinski
Tübingen University
Ulm University
ABSTRACT
Proteins perform a large variety of functions in living organisms and thus play key role biology. However, commonly used algorithms protein learning were not specifically designed for protein data, and are therefore not able to capture all relevant structural levels of protein during learning. To fill this gap, we propose two new learning operators, specifically designed to process protein structures. First, introduce novel convolution operator that considers the primary, secondary, and tertiary structure of protein by using n-D convolutions defined on both the Euclidean distance, as well as multiple geodesic distances between the atoms in multi-gras multi-graph. Second, introduce set of hierarchical pooling operators that enable multi-scale protein analysis. We further evaluate the accuracy of our algorithms common downstream tasks, where outperform state-of-the-an protein learning algorithms.
INTRODUCTION
Proteins perform specific biological func- tions essential for all living organisms and hence play key role when investigating the most fundamental questions in the life sciences. These biomolecules are com- posed of one several chains of amino acids, which fold into specific conforma- tions to enable various biological function- alities. Proteins can be defined using a multi-level structure:: The primary structure is given by the sequence of amino acids that connected through covalent bonds and form the protein backbone. Hydrogen bonds between distant amino acids in the chain form the secondary structure, which defines substructures such as a-helices and f-sheets. The tertiary structure results from protein folding and expresses the 3D spatial arrange- ment of the secondary structures. Lastly, the quarternary structure given by the interaction of multiple amino acid chains.
Geometry similar Topology different
Geometry different Topology similar
Figure 1: Invariances present in protein structures.
Considering only one subset of these levels can lead misinterpretations due ambiguities. As shown by Alexander et al. (2009), proteins with almost identical primary structure, i.e., only contain- ing a few different amino acids, can fold into entirely different conformations. Conversely, proteins from SH3 and OB folds have similar tertiary structures, but their primary and secondary structures differ significantly (Agrawal & Kishan 2001) (Fig. []] To avoid misinterpretations arising from these observations, capturing the invariances with respect to primary, secondary, and tertiary structures is of key importance when studying proteins and their functions."
Previously, the SOTA dominated by methods based on hand-crafted features, usually extracted from multi-sequence alignment tools (Altschul et al. 1990) or annotated databases (El-Gebali et al. 2019). In recent years, these have been outperformed by protein learning algorithms in different

---

Published as a conference paper at ICLR 2021
protein modeling tasks such protein fold classification (Hou et al. 2018 Rao et al. 2019 Bepler
& Berger 2019 Alley et al. [2119] Min et al. |2020] or protein function prediction (Strodthoff et al.
2020 Gligonjeviic et al.] 2019: Kulmanov | [017 Kulmanov & Hochndorf] 2019]AAiii et al. 2017). This can be attributed to the ability of machine learning algorithms to learn meaningful representations of proteins directly from the raw data. However, most of these techniques only consider a subset of the relevant structural levels of proteins and thus can only create a representation from partial information. For instance, due to the high amount of available protein sequence data, most techniques solely rely on protein sequence data as input, and apply learning algorithms from
the field of natural language processing (Rao et al.) 2019 Alley et al., 2019 Min al. [2020]
Strodthoff et al., 2020). ID convolutional neural networks (Kulmanov et al., 2017 Kulmanov &
Hoehndorf, 2019). or use structural information during training (Bepler & Berger 2019). Other
methods have solely used 3D atomic coordinates as an input, and applied 3D convolutional neural networks (3DCNN) (Amidi et al. 2017 Derevyanko et al., 2018) or graph convolutional neural networks (GCNN) (Kipf & Welling, 2017). While few attempts have been made to consider more than one structural level of proteins in the network architecture (Gligorijevic et al.) 2019), none of these hybrid methods incorporate all structural levels of proteins simultaneously. In contrast, common approach is to process one structural level with the network architecture and the others indirectly as input Peatures(Baldassaae et al. (2020) Hou et al. (2018))
In this paper, we introduce a novel end-to-end protein learning algorithm, that is able to explicitly incorporate the multi-level structure of proteins and captures the resulting different invariances. We show how a multi-graph data structure can represent the primary and secondary structures effectively by considering covalent and hydrogen bonds, while the tertiary structure can be represented by the spatial 3D coordinates of the atoms (Sec. 3. By borrowing terminology from differential geometry of surfaces, we define a new convolution operator that uses both intrinsic (primary and secondary structures) and extrinsic (tertiary and quaternary structures) distances (Sec. 4 Moreover, since protein sizes range from less than one hundred to tens of thousands amino acids (Brocchieri & Karlin 2005), we propose protein-specific pooling operations that allow hierarchical grouping of such a wide range of sizes, cnabling the detection of features at different scales (Sec. (5). Lastly, demonstrate, that by considering all mentioned protein structure levels, we can significantly outperform recent SOTA methods on protein tasks, such as protein fold and enzyme classification.
Code and data of our approach is available at htttp:://githu..com// IEConV prrtovvprrotiss
2 RELATED WORK
Early works on learning protein representations (Asgari & Mofrad 2015 Yang et al. embedding algorithms (Mikolov et al. 2013), as employed in Natural Language Processing (NLP). Other approaches have used ID convolutional neural networks (CNN) to learn protein representations directly from an amino acid sequence, for tasks such as protein function prediction (Kulmanov et al.,
2018 used word
2017, Kulmanov & Hoehndorf 2019), protein-compound interaction TTuuaai et al. 2018), protein fold classification (Hou et al. 2018). Recently, researchers have applied complex NL. moddls trained unsupervised on millions of unlabeled protein sequences and fine-tune them for different downstream
tasks RRao et al. 2019; Alley et al. 2019 Min et al. 22000; Stroothooff et al. 2020: Bepler & Berger 2019). While representing proteins as amino acid sequences during learning, is helpful when only
sequence data is available, it does not leverage the full potential of spatial protein representations that
become more and more available with modern imaging and reconstruction techniques.
To learn beyond sequences, approaches have been developed. that consider the 3D structure of proteins. range of methods has sampled protein structures to regular volumetric 3D representations and assessed the quality of the structure (Derevyanko et al. (2018), classified proteins in enzymes classes (Amidi et al. (2017), predicted the protein-ligand binding affinity (Ragoza et al., 2017) and the binding site (Jiménez et al. 2017), as well as the contact region between two proteins (Townshend et al. 2019). While this is attractive, as 3D grids allow for unleashing the benefits of all approaches developed for 2D images, such as pooling and multi-resolution techniques, unfortunately, grids do not scale well to fine structures or many atoms, and even more importantly, they do not consider the primary and secondary structure of proteins.
2

---

Published as a conference paper at ICLR 2021
Another approach that makes use of protein's 3D structure, is representing proteins as graphs and applying GCNNs (Kipf & Welling 2017; Hamilton et al. 2017). Works based on this technique represent each amino acid node the graph, while edges between them created if they are at a certain Fuclidean distance. This approach has been successfully applied to different problems. Classification of protein graphs into enzymes, for example, have become part of the standard data sets used to compare GCNN architectures (Gao & Ji. 2019; Ying et al. 2018). Moreover, other works with similar architectures have predicted protein interfaces (Fout et al., 2017), or protein structure quality (Baldassarre et al., 2020). However, GCNN approaches suffer from over- ver-smoothing, i. e.e indistinguishable node representations after stacking several layers, which limits the maximum depth usable for such architectures (Cai & Wang 2020).
is also worth noticing that some of the aforementioned works on GCNN have considered different levels of protein structures indirectly by providing secondary structure type or distance along the sequence as initial node or edge features. However, these are not part of the network architecture and can be blended out due to the over-smoothing problem of GCNN, On the other hand, recent protein function prediction method proposed by Gligorijevic et al. (2019) uses Long-Short Term Memory cells (LSTM) to encode the primary structure and then apply GCNNs to capture the tertiary structure. Also, the recent work from Ingraham et al.] (2019) proposes an amino acid encoder that can capture primary and tertiary structures in the context of protein generative models. Unfortunately, none of these previous methods can incorporate all structural protein levels within the network architecture.
3 MULTI-GRAPH PROTEIN REPRESENTATION
To simultancously take into account the primary, secondary, and tertiary protein structure during learning, we propose represent proteins as a multi-graph g = (N, A,A. 13). In this graph, atoms are represented as nodes associated with their 3D coordinates, / E Rrix and associated features, F E _ being the number of atoms, and the number of features. Moreover, A E gax* and B E R=** are two different adjacency matrices representing the connectivity of the graph. Elements of matrix A are defined Au if there is a covalent bond between atom and atom j. and A,j = 0 otherwise. Similarly, the elements of matrix B are defined as Bij = 1 if there is a covalent or hydrogen bond between atom and atom j. and B, otherwise.
3.1 EXIRRIICEXTTINSIC DISTANCES
Differential geometry of surfaces (Pogorelov 1973) defines intrin- sic geometric properties as those, that are invariant under isometric mappings, i.e., under deformations preserving the length of curves on a surface. On the other hand, extrinsic geometric properties dependent on the embedding of the surfaces into the Euclidean space. Analogously, in our protein multi-graph we define intrinsic geometric properties as those that are invariant under deformations preserving the length of paths along the graph, i.e., deformations that preserve the connectivity of the protein. Additionally, we define extrinsic geometric properties as those, that depend on the embed- ding of the protein into the Euclidean space, i.e., on the 3D protein conformation. Using this terminology, we define three distances in our multi-graph, one extrinsic and two intrinsic (see Fig. 2}. The extrinsic distance Te is defined by the protein conformation in Euclidean space, therefore we use the Euclidean distance between atoms, which enables us to capture the tertiary and quaternary structures of the protein. The intrinsic distances are inherent of the protein and independent of the actual 3D conformation. For the first intrinsic distance Ti we use the shortest path between two atoms along the adjacency matrix A of the graph, capturing the primary structure. The second intrinsic distance Tie is defined as the shortest path between two atoms along the adjacency matrix B, capturing thus the secondary structure.
Extrinsic dist. 1,
Intrinsic dist. T,
Covalent bonds
Intrinsic dist. Fo
Hydrogen bonds
Figure 2: Distances between atoms in our protein graph.
4 ONTRINSIC-EXTRING
PROTEIN CONVOLUTION
The key idea of our work is to take into account the multiple invariances described in Sec. during learning. Therefore, based on the success of convolutional neural networks for images (Krizhevsky
3

---

Published as a conference paper at ICLR 2021
Convix,)
Figure 3: Intrinsic-extrinsicc convolution our multi-graph for atom A. First, we detect the neighbor- ing atoms involved in the convolution using a ball query (a function returning all nodes closer than to point x). For each atom, the extrinsic (pink) and two intrinsic (blue and green) distances are input into the kernel and the result is multiplied by the atom's features. Lastly, all contributions from neighboring atoms summed up.
ct al. 2012) and point clouds (Qi et al. 2017a), in this paper we define a convolution operator for proteins which is able to capture these invariances effectively. To this end, we propose convolution on 3D protein structures, which is inspired by conventional convolutions as used to learn on structured images. First, define the neighborhood of an atom all atoms Euclidean distance smaller than me- Moreover, we define our convolution kernels as single Multi Layer Perceptron (MLP) which takes as input the three distances defined in Sec. 3.1] one extrinsic and two intrinsic, and outputs the values all kernels. This enables the convolution to learn kernels based on one or multiple structural levels of the protein. Thus, the proposed convolution operator is defined as:
Tx(X,Xx) me
Tii (x, Tie Tia X:)
* F)x())
Σ Σ (EN( 1=1
Fly Kj,k
m2
where (xx are the atoms at Euclidean distance d < me from x, Fi, is the input feature j of atom x,, KJ,k is the aforementioned kernel that maps R R, xx) is the Euclidean distance between atom and atom Xg, Ti and Ti2 are the two intrinsic distances, and m., mi. and m2 the maximum distances allowed (m, = m2 = hops in all experiments while m. is layer-dependent). All normalized distances clamped to [0, 1]. The convolution, performed independently for every atom, is illustrated on a model protein in Fig. 3 where the distances between neighboring atoms shown. This operation has all properties of the standard convolution, locality and translational invariance, and at the same time rotational invariant.
While we state our operation as an unstructured convolution as done in 3D point cloud learning (Hermosilla et al. 2018 Groh et al. 2018 Wu et al. 2019: Xiong et al. 2019 Thomas et al. 2019 can also understood states are atoms, edges are formed based nearby atoms and messages between atoms. The key difference standard GCNNs that we have edges multiple kinds of bonds, use hierarchy of graphs, and that the message passing function is learned.
message passing framework (Kipf Welling 2017) where hidden
HIERARCHICAL PROTEIN POOLING
We consider proteins at atomic resolutions. This allows us to identify the spatial configuration of the amino acid side chains. something of key importance for active site detection. However, proteins can have large number of atoms, preventing the usage of large receptive fields in our convolutions (computational restriction) and limiting the number of learned features per atom (memory restriction).
Pooling is technique commonly used in convolutional neural networks, as it hierarchically reduces the dimensionality of the data by aggregating local information (Krizhevsky et al. 2012). While is able to overcome computation and memory restrictions when learning images, unfortunately, it cannot be directly applied to proteins, as conventional pooling methods rely on discrete sampling. While, the other hand, some of the techniques developed for unstructured point cloud data (Qi jet al. 2017b) (0176//Heme Hermosilla et al. 2018) could be applied learn on the 3D coordinates of atoms, is unclear what the edges of that graph representation would be.
4

---

Published as a conference paper at ICLR 2021
Global features
(e)
(c)
Figure 4: Hierarchical protein pooling: We segment the protein into amino acids (blue, orange, green). First, (a), we apply spectral clustering each independent amino acid graph. Then, (b), each resulting amino acid is pooled to its a-carbon. After that, we apply two backbone pooling operations, (c) and (d). Lastly, we apply the symmetric operation average, (e), to obtain the final feature vector.
Therefore, we define a set of operations that successively reduce the number of nodes in our protein graph. First, iteratively reduce the number of atoms per amino acid to its alpha carbon. Afterward, we reduce the number of nodes along the backbone. Fig. illlstrrtte this process, which we describe in detail in the following paragraphs.
Amino acid pooling Simplified representations of amino acids have been previously used in Molecu- lar Dynamics (MD) simulations, order get the number of computations down manageable level. Nevertheless, in contrast to our approach, these techniques do not use a uniform pooling pattern. Rather, while the atoms belonging the backbone are not simplified, most of these approaches (Si mons et al. 1997) simplify all atoms of the side chains to single node. Other more conservative approaches, such as PRIMO (Kar et al. 2013) or Klein's models (De et al. 2009), represent side chains with a variable number of nodes using lookup table. However, they do not perform uniform simplification, resulting in a high variance the number of atoms per cluster. Moreover, these methods need to manually update the lookup table to incorporate rare amino acids.
In this work, we decided instead to follow the common practice CNN for images where uniform pooling is used over the whole image. We propose method that is able to reduce the number of nodes in the protein graph by half. To this end, we generate an independent graph for each amino acid using the covalent bonds edges and apply spectral clustering (von Luxburg 2007) to reduce by half the number of nodes. Since the number of amino acids is finite (20 appearing in the genetic code), these pooling matrices reused among all proteins (see Fig. (a)). However, since the method only requires a graph as input, it can be directly applied to synthetic or rare amino acids.
From the amino acid pooling matrices, we create protein pooling matrix P RR=x, where n is the number of input atoms in the protein and m is the number of resulting clusters in the simplified protein. This matrix is defined as P.j = 1 if the atom collapses into cluster j. and Pij 0 otherwise. Using P we can create a simplified protein graph, G' = =(N, F', A', B'), with the following equations:
N' D D-''P
D'=D-PP A' = PAPT
B' PBPT
where E Rmxx a diagonal matrix with Djj = Σi Pij. Note, that the resulting matrices A' and 1 might not be binary adjacency matrices, i. c., non-zero diagonal values or values greater than one. Therefore, we assign zeros to the diagonals and clamp edge values one. These matrices are computed using sparse representations in order to reduce the memory footprint.
Alpha carbon pooling a second pooling step, we simplify the protein graph to a backbone representation, whereby we cluster all nodes from the same amino acid to a single node. We define P accordingly, and the number of clusters m equal to the number of amino acids, while Pig = node belongs to amino acid j, and Pú = 0 otherwise. We use Eq. 2 to compute F A', and B However, ' is defined as the alpha carbon positions of each amino acid since they better represent the backbone and the secondary structures.
Backbone pooling The last pooling steps simplifications of the backbone chain. In each pooling step, every two consecutive amino acids the chain are clustered together, effectively reducing by half the number of amino acids. Therefore, to compute N,,, F', A', and B", we define P accordingly and use Equation 2 For single-chain proteins, for example, Piş 1 if [i/2] = j. otherwise.
5

---

Published as a conference paper at ICLR 2021
Atom embedding
Pooling
Pooling (6. 128)
Reshar (8. 200)
Pooling
ReeNve x2 (12)
Pooling
ReeNie x2 (16. 1024
Pooling
Global festures
B
ResiNet
ReeNie
RissNee
Conr
Alom feelures
MLP
Las
Int Eaat Conv
dl
Heerarhy Level
Probabliiles
Comy
Figure The architecture our model. The imput is composed of atom features and an atom embedding learned together with the network. Each layer is composed of two ResNet (He et al. 2016) bottleneck blocks, for which the radius in angstrom and the number of features are indicated in parentheses, followed by pooling operation. An illustration of single ResNet bottleneck block presented in the right for D input features (before each convolution we use batch normalization and Leaky ReLU). The global protein features are processed by MLP which computes the final probabilities. Protein graphs used in each level are indicated at the bottom.
6 EVALUATION
To evaluate our proposed protein learning approach, we specify deep architecture (Sec. 6. which we compare to SOTA methods (Sec. (.2) as well as different ablations of our approach (Sec. 6.3)
During the entire evaluation, we focus on two downstream tasks in Sec. 66.4 and Sec. 6.5
6.1 OUR ARCHITECTURE
By facilitating protein convolution and pooling, learning on complex structures. In particular, the proposed architecture encodes an input protein into latent representation which is later used in different downstream tasks.
able to realize deep architecture that enables
The input of our network protein graph at atom level with a 6D input feature vector per each atom. The atom features comprise 1) covalent radius, 2) der Waals radius, 3) atom mass, and 4-6) are the features of an atom type embedding learned together with the network. The input is then processed by 5 layers which iteratively increases the number of features, each composed of two ResNet (He et al. 2016) bottleneck blocks followed by pooling operation (see Fig. 5 The respective receptive fields are [3, 6, 8, 12, 16] angstroms (Å) using [64, 128, 256, 512, 1024] features. The size of the receptive fields has been chosen to include reduced number of nodes in it. As several proteins come without hydrogen bonds, we compute them using DSSP (Kabsch & Sander 1983).
As the architecture is fully-convolutionaa, it can process proteins of arbitrary size, but after finitely many steps, it obtains an intermediate result of varying size. Hence, to reduce this to one final result vector, a symmetric aggregation operation, average, is used. Lastly, a single layer MLP with 1024 hidden neurons is used to predict the final probabilities.
6.2 OTHER ARCHITECTURES
We compare our architecture, as described above, to SOTA learners designed for similar downstream tasks. First, we compare to the latest sequence-based methods pre-trained unsupervised on millions of sequences: Rao et al. Bepler & Berger (2019); Alley et al. (2019): Strodthoff et al. (2020); Elnaggar al. (2020). Second, we compare to different methods that take only the 3D protein structure into account: Kipf & Welling (2017); Diehl (2019); Derevyanko et al. (2018). Lastly, we also compare to two recent hybrid methods: Gligorijevic et al. 2019), who process primary structure with several LSTM cells first and then apply GCNN, and Baldassarre et al. (2020), who indirectly process primary and secondary structures by using as input distances along the backbone as edge features and secondary structure type node features in GCNN setup. When available, we provide the results reported in the original papers, while more details of the training procedures are provided in Appendix D

---

Published as a conference paper at ICLR 2021
Table 1: Comparison of network to other methods the tasks (protein fold and enzyme catalytic reaction classification) measured as mean accuracy, where we outperform all methods.
Architecture
# params
FOLD
REACT
Fold
Super.
Fam.
HHSuite
17.5
69.2 %
98.6 %
82.6
et al.
2018)
CNN
1.0 M
40.9
50.7 %
76.2 %
ID ResNet
41.7 M
17.0%
31.0 %
77.0 %
70.9 %
Rao et al.
2019
LSTM
4
26.0 %
43.0 %
92.0 %
79.9
Transformer
38.4
21.0% %
34.0 %
88.0 %
69.8 %
Bepler & Berger
2019
LSTM
31.7
17.0%
20.0 %
79.0 %
74.3
Bepler Berger
2019
LSTM
31.7
36.6%
62.7 %
95.2 %
66.7
Alley ct al. (2019)
mLSTM
1
23.0%
38.0 %
87.0 %
72.9 %
Strodthoff ct
2020
L.TM
22.7 M
14.9% %
21.5 %
83.6 %
73.9
Elnaggar al.
2020
Transformer
M
26.6 %
55.8 %
97.6 %
72.2 %
Kipf & Welling 2017
GCNN
1.0M
16.8 %
21.3 %
82.8 %
67.3 %
DichI
GCNN
1.0M
12.9% %
16.3
72.5 %
57.9
Derevyanko et al.
2018)
3D CNN
6.0
31.6%
45.4 %
92.5 %
78.8 %
Gligorijevic
2019
LSTM+GCNN GCNN
6.2
15.3
20.6 %
73.2 %
63.3
Baldassarre et al.
2020
1.3 M
23.7 %
32.5 %
84.4 %
60.8 %
Ours
9.8 M
45.0 %
69.7 %
98.9 %
87.2 %
"Pre-trained unsupervised 10-31 million protein sequences. Pre-trained several supervised tasks with structural information.
6.3 ABLATIONS OF OUR METHOD
We study four axes of ablation: convolution, neighborhood, pooling, and representation. When moving along one axis, all other axes are fixed to Ours (+).
Convolution ablation. We consider four different ablations of convolutions: GCNN ()) (Kipf & Welling 2017); ExConv (.), kernels defined in 3D space only (Hermosilla et al. 2018):
InConvC uses only intrinsic distance 711: InConvH (+) uses only intrinsic distance Ti2; InConvCH makes use of both intrinsic distances at the same time; Ours (o) uses both
intrinsic distances plus distances along the three spatial dimensions; and lastly, Ours (+). that refers to our proposed convolution which uses both geodesics plus Fuclidean distance.
Neighborhood ablation. We compare several methods to define our receptive field: CovNe gh (+). which uses the intrinsic distance Tı the graph; HyNeigh (o); which uses the intrinsic distance 72, as well as Ours which uses the Euclidean distance.
Pooling ablation. We consider six options: NoPool (o), which does not use any pooling operation: GridPool (+) overlays the protein with increasingly coarse grids and pools all atoms into one cell (Thomas et al.) 2019); TopKPool (o) learns per-node importance score which is used to
drop nodes (Gao & Ji 2019): EdgePool (+), which learns a per-edge importance score to collapse
edges (Diehl 2 a (o), which uses the centroid simplification method used by the Rosetta software Molecular Dynamics simulation (Simons et al.] 1997); and our pooling, Ours (+).
Representation ablation. Lastly, evaluate the granularity of the input, considering the protein at the amino acid level, AminoGraph (+), or at atomic level, Ours (+).
6.4 TASK 1: FOLD CLASSIFICATION (FOLD)
Task.
Protein fold classification of key importance in the study of the relationship between protein structure and function, and protein evolution. The different fold classes group proteins with similar secondary structure compositions, orientations, and connection orders. In this task, given a protein, predict the fold class, whereby the performance is measured mean accuracy.

---

Published as a conference paper at ICLR 2021
Table 2: Study of ablations (rows) for the FOLDD and REACTION tasks (columns).
FOLD
REACT
Fold
Super.
Fam.
50
Fold task
GCNN
25.7 %
46.5 %
95.9 %
84.9 %
ExConv
30.1 %
46.3 %
92.0% %
85.0%
20
InConvC
37.6 %
65.1 %
98.7 %
85.4%
88
Reaction task
Conv.
InConvH
62.0 % 40.8 %
98.4 %
85.5 %
InConvCH
43.5 %
66.7 %
98.7
85.2%
Curs3DCH
40.7 %
62.2 %
98.1% %
85.8
Ours
45.0 %
69.7 %
98.9
87.2 %
84
Fold
Reaction
CovNeigh
27.2 %
41.5 %
92.3
41.6 %
47
Neighbors
HyNeigh
33.3 %
50.6 %
96.9 %
56.9 %
25
40
Curs
45.0 %
69.7 %
98.9
87.2
Fold Task
NoPool
37.1 %
59.8 %
97.6% %
84.7 %
GridPocl
28.6 %
41.8 %
91.8 %
86.1 %
TopKPool
40.7 %
65.4 %
98.4% %
84.5 %
Pool
Reaction Task
EdgePool
44.4 %
69.6 %
99.0% %
86.9
88
RosettaCEN
41.7 %
66.5 %
98.9 %
86.5 %
Curs
45.0 %
69.7 %
98.9
87.2 %
84
AminoGraph
39.6 %
64.7 %
99.1
85.3 %
Reaction
Repr.
Ours
45.0 %
69.7 %
98.9 %
87.2 %
88
88
85
85
Data set. We the training/validatiioneess splits of the SCOPe 1.75 data set Hou al. (2018) )). This data set consolidated 16, 712 proteins from 1, 195 folds. We obtained the 3D structures of the proteins from the SCOPe 1.75 database (Murzin al. 1955 The data set provides three different test sets: Fold, in which proteins from the same superfamily are not present during training; Superfamily, in which proteins from the same family are not seen during training; and Family, in which proteins of the same family are present during training.
Results. Results, as compared to other methods, are reported Tbl. 1 We find our method to perform better by a large margin without using additional features as input pre-trained on additional data. Tbl. 2 shows how the tested ablations perform on this task. Overall we see that Curs (o) performs better than any ablation, indicating that our protein-specific representation, convolution, and pooling are all important ingredients. Lastly, we compare in top Tbl. to the state art results reported in the paper (Hou et al. 2018) the results obtained using multiple sequence alignment, where we outperform all.
6.5 TASK 2: ENZYME- ENZYME-CATALYZEI CATALYZED REACTION CLASSIFICATION (REACT)
Task. For this task, elassify proteins based the enzyme-catalyzed reaction according all four levels of the Enzyme Commission (EC) number (Webb) 1992). The performance is again evaluated as mean accuracy.
Data set. We collected a total of 428 proteins from 384 EC numbers. The data was then split into 29, 215 instances for training, 562 instances for validation, and 5, 651 for testing. Note that all proteins have less than 50% sequence-sin similarit in-between splits. A full description of the data set provided in Appendix
Results. Again, our method outperforms previous works this task (Tbl. D Ablations of our
method (Tbl. 2 give further indication that all our components are relevant also for other tasks.
6.6
TASK 3: ONE-SHOT FOLD CLASSIFICATION (ONESHOT)
Task. The objective of this task is to give the binary answer to the question if pair of proteins, which are both not observed during training, are in the same fold or not. This also known as one-shot learning and adopt the siamese architecture from Koch al. (2015) using protein encoder.
8

---

Published as a conference paper at ICLR 2021
Data. We used the same dataset for classification (Sec.(6.4) but withheld random subset of 50 folds out of the original 1195 folds during training. During testing, we compare all proteins contained in the test set to all proteins contained in reference set that comprises of all proteins of the original training set, and thus represents all 1995 folds. From each test protein, we select the pair with higher probability predict the fold. To analyze how accuracy is affected by folds unseen during training, we divided the Fold test set from Task I in two different test sets. Our seen test set contains 419 proteins from 86 folds seen during training, while our unseen test set contains 299 proteins from the 50 folds not seen during training.
Results. When analyzing how many proteins could be predicted correctly, we obtained an accuracy of 39.0% for seen and an accuracy of 31.9% for unseen. This indicates that our method is able to generalize to folds that have not been seen during training. It also shows that our method is flexible enough to take concepts like the one from Koch et al. (2055), which uses hierarchical image convolutions, and generalize them to proteins. To view the resulting accuracies in the context of the other values reported in paper, we also have trained the architecture for standard classification of Sec. 6.4 with the training set of this task composed of 1145 folds. We obtained an accuracy of 46.5 5 when testing against the 86 folds compared to the 39 % of the one-shot training.
7 CONCLUSIONS
Based on the multi-level structure of proteins, we have proposed neural network architecture to pro- cess protein structures. The presented architecture takes advantage of primary, secondary, and tertiary protein structures to learn a convolution operator that works with intrinsic and extrinsic distances as input. Moreover, we have presented set of pooling operations that enable the dimensionality reduction of the input mimicking the designs used by convolutional neural networks on images. Lastly, our evaluation has shown that by incorporating all the structural levels of a protein in our designs we are able to outperform existing SOTA protein learning algorithms on two downstream tasks, protein fold and enzyme classification.
Despite the reported success achieved in protein learning, some limitations apply to our idea. The main one being, that it requires the 3D structure of each protein, whilst other methods only make use of the protein sequence. This, however, may be alleviated by the advances in protein structure determination (Callaway] 2020) and prediction (Senior et al.] 2020). Moreover, like other commonly used approaches such as GCNN, changes. This could be solved by incorporating directional information into the kernel's 's input. We leave this improvement for future work.
convolution operator invariant rotations but also to chirality
ACKNOWLEDGEMENTS
This work was partially funded by the Deutsche Forschungseemeinschaft (DFG), grant RO 3408/2-1 1 (ProLint), the project 1-7778855----1--I1- and by 839 FEDER (EU) funds.
from the Spanish Ministerio de Economía y Competitividad,
REFERENCES
Vishal Agrawal and Radha KV Kishan. Functional evolution of two subtly different (similar) folds, BMC Structural Biology, 2001.
Patrick A. Alexander, Yanan Hc, Yihong Chen, John Orban, and Philip N. Bryan. A minimal sequence code for switching protein structure and function. Proceedings of the National Academy of Sciences, 2009.
Ethan C. Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M. Church. Unified rational protein engineering with sequence-based deep representation learning. Nature Methods, 2019.
S.F. Altschul, W. Gish, W. Miller, E.W. Myers, and DJ Lipman. Basic local alignment search tool, J Molecular Biology, 215(3):403-10, 1990.
A. Amidi, S. Amidi, D. Vlachakis, Megalooikonomou, N. Paragios, and E. Zacharaki. EnzyNet: enzyme classification using 3D convolutional neural networks on spatial representation. arXiv: 2017.
E. Asgari and M.R.K. Mofrad. Continuous distributed representation of hiological sequences for deep proteomics and genomics. PLoS ONE, 2015.

---

Published as a conference paper at ICLR 2021
F. Baldassarre, D. M. Hurtado, A. Elofsson, and H. Azizpour. GraphQA: Protein Model Quality Assessment using Graph Convolutional Networks. Bioinformaties, 2020.
Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. International Conference on Learning Representations, 2019.
H. M. Berman, Westbrook, Z. Feng, G. Gilliland, T. N. Bhat, H. Weissig, I. N. Shindyalov, and P. E. Bourn.. The Protein Data Bank. Nucleic Acids Res, 2000.
Luciano Brocchieri and Samuel Karlin. Protein length in eukaryotic and prokaryotic proteomes. Nucleic Acids Res, 33( (3100)33390-00, 2005.
Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. JCML 2020 Graph Representa- tion Learning workshop, 2020.
2020
E. Callaway. Revolutionary cryo-em is taking over structural biology. Nature News, 2020.
Papers With Code. URL on 1.10.2020.
on-od Accessed
Dana, Gutmanas, N. Tyagi. Qi. C. O'Donovan, M. Martin, and S. Velankar. SIFTS: updated Structure Integration with Function, Taxonomy and Sequences resource allows 40-fold increase in coverage of structure-based annotations for proteins. Nucleic Acids Research, 2018.
Georgy Derevyanko, Sergei Grudinin, Yoshua Bengio, and Guillaume Lamourcux. Deep convolutional networks for quality assessment of protein folds. Bioinformatics, 34(23):4046-53, 2018.
R. Vane, W. Shinoda, P. B. Moore, and M. L.. Klein. Transferable coarse grain nonbonded interaction model for amino acids. Journal of Chemical Theory and Computation, 2009.
Frederik Diehl. Edge contraction pooling for graph neural networks. 1rx3:v:905.11099 2019.
Paul D Dobson and Andrew Doig. Distinguishing enzyme structures from non-enzymes without alignments. J Molecular Biology, 330(4):771-78) 2003.
Sara El-Gebali, Jaina Mistry, Alex Bateman, Sean Eddy, Aurélien Luciani, Simon C Potter, Matloob Qureshi, Lorna Richardson, Gustavo Salazar, Alfredo Smart, Erik L Sonnhammer, Layla Hirsh, Lisanna Paladin, Damiano Piovesan, Silvio C E Tosatto, and Robert D Finn. The Pfam protein families database in 2019. Nucleic Acids Research, 2019.
A. Elnaggar, M. Heinzinger, C. Dallago, G. Rihawi, Y. Wang, L. Jones, T. Gibbs, T. Feher, C. Angerer, M. Steinegger, D. Bhowmik, and Rost. Prottrans: Towards cracking the language of life's code through supervisdd deep learning and high performance computing. 2020.
A. Fout, Byrd, B. Shariat, and A. Ben-Hur. Protein interface prediction using graph convolutional networks. In Advances in Neural Information Processing Systems 30. 2017.
Hongyang Gao and Shuiwang Ji. Graph U-nets. In ICML, pp.
2019.
Gligorijevic, P. D. Renfrew, T. Kosciolek, J. Leman, K. Cho, T. Vatanen, D. Berenberg, B. Taylor, I. M. Fisk, R. J. Xavier, R. Knight, and R. Bonncau. Structure-basse function prediction using graph convolutional networks. bioRxiv, 2019.
Fabian Groh, Patrick Wieschollek, and Hendrik Lensch. Flex-convolution (million-scale point-cloud learning beyond grid-worlds). arXiv: /803.0729, 2018.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. In NIPS, pp- 1024-34. 2017.
He, X. Zhang, Ren, Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Hermosilla, Ritschel, P-P Vazquez, A. Vinacua, and T. Ropinski. Monte carlo convolution for learning on non-unin non-uniformly sampled point clouds. ACM Trans. Graph. (Proc. SIGGRAPH PH Asia), 37(6), 2018
J. Hou, B. Adhikari, and J. Cheng, Deepsf:: Deep convolutional neural network for mapping protein sequences to folds. In Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, 2018
10

---

Published as a conference paper at ICLR 2021
John Ingraham, Vikas Garg. Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. In NeurlPS, pp. 15820- 15831, 2019.
J Jiménez, S Doem, G Martinez-Rosell. and A S Rose. DeepSite: protein-binding site predictor using 3D- convolutional neural networks. Bioinformatics, 33(19)3003-300 30426 2017.
Wolfgang Kabsch and Christian Sander. Dictionary of protein secondary structure: pattern recognition of hydrogen-bondee and geometrical features. Biopolymers: Original Research on Biomoleculex, 22(12): 2577-37, 1983.
Kar, S. M. Gopal, Y.-M. Cheng, A. Predeus, and Feig. Primo: transfcrable coarse-grained force field for proteins. Journal of Chemicul Theory and Computation, 2013.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ICML, 2017.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML Deep Learning Workshop, volume 2, 2015.
A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
Maxat Kulmanov and Robert Hoehndorf. DeepGOPlus: improved protein function prediction from sequence. Bioinformaticx, 36(2):422-429 2019.
Maxat Kulmanov, Mohammed Asif Khan, and Robert Hachndorf. DeepGO: predicting protein functions from sequence and interactions using deep ontology-awand classifier. Bioinformatics, 34(4):660 6616666668,,,, 2017.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint
2013.
Sconwoo Min, Scunghyun Park, Siwon Kim, Hyun-Soo Choi, and Sungroh Yoon. Pre-training of deep bidirectional protein sequenee representations with structural information. Bioinformatics, 2020.
A.G. Murzin, S.E. Brenner. T. Hubbard, and C. Chothia. SCOP: structural classification of proteins database for the investigation of sequences and structures. J Molecular Biology, 1955.
D. Q Nguyen, T. D. Nguyen, and D. Phung. Universal self-attention network for graph classification. 2020.
A. V. Pogorelov. Extrinsic geometry of convex surfaces, volume 35. American Mathematical Soc., 1973.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. CVPR, 2017a.
Charles R Qi, Li Yi, Hao Su, and Leonidas Guibas. PointNet++: Deep hierarchical feature learning on point sets in metric space. NIPS, 2017b.
Matthew Ragoza, Joshua Hochuli, Elisa Idrobo. Jocelyn Sunseri, and David Ryan Koes. Protein ligand scoring with convolutional neural networks. J Chemical Information and Modeling, 57(4):942 957, 2017.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S. Song. Evaluating protein transfer learning with tape. Advances in Neural Information Processing Systems, 2019.
A. W. Senior, Fvans, Jumper, Kirkpatrick, L.. Sifre, T. Green, C. Qin, A. Zidek, W. R. Nelson, Bridg- land, Penedones. Petersen, K. Simonyan, Crossan, P. Kohli, Jones. Silver. Kavukeuoglu. and D. Hassabis. Improved protein structure prediction using potentials from deep learning. Nature, 2020.
K. T. Simons, Kooperberg, F. Huang, and Baker. Assembly protein tertiary structures from fragments with similar local sequences using simulated annealing and bayesian scoring functions. J Molecular Biology, 1997.
Nils Strodthofff, Patrick Wagner, Markus Wenzel, and Wojciech Samek. Udsmprot: universal deep sequence models for protein classification. Bioinformatics, 2020.
Hugues Thomas, Charles R Qi. -mamnuulu Deschaud, Beatriz Marcotegui, François Goulette, and I Leonidas J Guibas. Kpconw: Flexible and deformable convolution for point clouds. In ICCV, pp. 641 1-20. 2019.
11

---

Published as a conference paper at ICLR 2021
Matteo Togninalli, Elisabetta Ghisu, Felipe Llinares-López. Bastian Rieck, and Karsten Borgwardt. Wasserstein weisfeiler-Ichman graph kemells, In Advances Neural Information Processing Systems 32. 2019.
Raphael Townshend, Rishi Bedi, Patricia Suriana, and Ron Dror. End-to-end learning 3D protein structure interface prediction. NeurIPS, pp. 15642-15651, 2019.
Masashi Tsubaki, Kentaro Tomii, and Jun Sese. Compound-protein interaction prediction with end-to-end learning of neural networks for graphs and sequences. Bioinformatics, 35(2):309 318, 2018.
L. van der Maaten and G. E. Hinton. Visualizing high-dimensional data using t-sne. J Machine Learning Reseurch, 2008
Ulrike Luxburg. A tutorial spectral clustering. Statistics and computing, 17(4):395-416, 2007.
Edwin C. Webb, Enzyme Nomenclature 1992. Academic Press, 1992.
Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9621 1621-30, 2019.
Yuwen Xiong, Mengye Ren, Renjie Liao, Kelvin Wong, and Raquel Urtasun. Deformable filter convolution for point cloud reasoning, arXiv: 13079, 2019.
K. Yang, 7. Wu, C. N Bedbrook, and F. H Arnold. Learned protein embeddings for machine learning. Bioinformatics, 2018.
Zhitao Ying, Jiaxuan You, Christopher Morri,, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. In NIPS, pp. 4800-4810, 2018.
Muhan Zhang, Zhicheng Cui, Marion Ncumann, and Yixin Chen, An end-to-end deep learning architecture for graph classification. In Proc. AAAI Conference Artificial Intelligence, pp. 4438 4445, 2018.
Zhen Zhang, Jiajun Bu, Martin Ester, Jianfeng Zhang. Chengwei Yao, Zhi Yu, and Can Wang. Hierarchical graph pooling with structure learning, arXiv: /tt///9111000 1.05954, 2019.
Qi Zhao and Yusu Wang. Learning metrics for persistence- based summaries and applications for graph classification. In Advances in Neural Information Processing Systems 32, 2019.
12

---

Published as a conference paper at ICLR 2021
A
ADDITIONAL EXPERIMENTS
A.1 ENZYME-NO-ENZYN CLASSIFICATION
Task. This is a binary classification of a protein into either being an enzyme not. Even though this task is rather simple compared other tasks, have included it, it has been extensively used as benchmark for graph classification methods (Code), where is referred as D&D, and allows direct comparison with several methods.
Table 3: Results for the D&D data set (Dobson & Doig 2003).
Gao & Ji 2019
82.4 %
Ying 2018
82.1 %
Zhao & Wang
(2019)
82.0
Data set. We downloaded the 1,178 proteins of the data set defined by
Nguyen ct al.
20200
81.2 %
Zhang ct al. (209
81.0% %
Dobson & Doig (2033) from the Protein Data Bank (PDB) (Berman
Iogmiaali al. (2019)
79.7 %
et al. 2000). Due to the low number of proteins, performance measure using
Zhang et al.
2018
79.4 %
10-fold cross-vas validatiin.....
Ours
85.5 %
Results. We compared to the results reported for methods that did not use extra training data and they were trained supervised method. We found method to outperform all recent algorithms.
A.2 LATENT SPACE VISUALIZATION
Fold Test set
Superfamily Test set
Family Test set
8: Alpha proteins
b: Beta proteins
c: Alpha and beta (a/b) Alpha and beta (a+b)
e: Mutti-domann proteins
f: Membrane and cell surface
Small proteins
Figure 6: Visualization of the learned representations for the proteins of the different test sets of the SCOPe 1.75 data set. The data points are colored based on the higher level in the SCOPe hierarchy, showing that the network learned latent space in which folds from the same class are clustered together.
order further validate that the network learned mcaningful protein representations, we used t-SNE (van der Maaten & Hinton] (008) to visualize the gigh-dimnnsionl space the representation Iearned for cach protein on the different test sets of the SCOPe 1.75 data set (Hou et al. 2018), see Fig. 6 We use protein representation the global feature vector before is fed to the final MLP. We color-coded each data point with the label of the higher level in the SCOPe hierarchy, protein class. We this lahel, the network learned a representation in which folds are clustered by their upper class the SCOPe hierarchy. In particular, the four major classes (a. b, c, and d) and the class of small proteins can be easily differentiate in the latent space. However, classes and are more difficult to identify. This could be due to the tuning of t-SNE, or, in the case of class of more than one protein domain, each belonging to one the other classes.
that, despite the network
trained using
the fact that proteins from this class composed
B AMINO ACID CLUSTERING MATRICES
Fig. illustrates clustering matrices generated by the spectral clustering algorithm (von Luxburg 2007) for the 20 amino acids appearing in the genetic code.
C ENZYME REACTION DATA SET
Here we describe how the Enzyme Reaction data set was obtained and processed for the task described in
6.5 We downloaded ECC annotations from the SIFTS database (Dana et al. (0188). This database contains
ECC annotations for entries the Protein Data Bank (PDB) (Berman al. 2000) From the annotated enzymes,
13

---

sequence similarity Published as a conference paper at ICLR 2021
bedlacinn
Nightte
Goaami and
Clycion
Cysteine
Threomine
Jyen
Nonine
Thyotophan
Phanrjidorin
Clutamine
Protie
Methionne
Histdine
Asparagine
Figure 7: Visualization of the amino acid clustering matrices obtained with the spectral clustering algorithm
(von Luxburg 2007).
we cluster their protein chains using 50 % sequence similarity threshold, provided by PDB (Berman al. 2000). We selected the unique complete EC numbers (e.g. EC 3.4.11.4) for what five more unique clusters were annotated with
each cluster, we selected among them) as representatives. Note that even if the proteins are similar, selecting several chains per cluster can be understood as type of data augmentation. In total, we collected 37, 428 annotated protein chains. Due to the low number of annotations for some of the enzyme types, the resulting data set contains highly unbalanced number of protein chains per number.
different proteins (less than
sets, training, validation, and testing, ensuring that every EC number was represented in each set and all protein chains belonging to the same cluster are in the same set. This resulted in 29, 215 protein chains for training, 2, 562 protein chains for validation, and 5, 651 for testing.
Lastly, we split data set into
D TRAINING AND HYPERPARAMETERS
D.1 OUR ARCHITECTURE
We trained model with Momentum optimizer with momentum equal 0.98 for 600 epochs for the FOLD task and 300 epochs
REACTION. used initial learning rate 0.001, which was multiplied by 0.5 every 50 epochs with the allowed minimum learning rate of 1e 6. Moreover, we used L2 regularization sealed by 0.001 and we clipped the norm of the gradients to 10.0. We used batch size equal to eight for both tasks. We represented the convolution kernel with single layer MLP with 16 hidden neurons for the FOLD task and 32 for the REACTION task. further regularize our model, we applied dropout with prohability of 0.2 hefore each 1 1 convolution, and 0.5 in the final MLP. Moreover, we set to zero all features of an atom before the Intrinsic Extrinsic convolution with probability of 0.05 for the FOLD task and 0.0 for the REACTION task. Lastly, added Gaussian noise to the features before each convolution with standard deviation of 0.025.
the FOLDD task, we increased the data synthetically by applying data augmentation techniques. Before feeding the protein the network, we applied random Gaussian noise to the atom coordinates with a standard deviation of 0.1, randomly rotated the protein, and randomly scaled each axis in the range [0.9, 1.1].
D.2 RAO ET AL. 2019, BEPLER & BERGER 2019, ALLEY ET AL. 2019
For the FOLD task, used the reported numbers in the pa- per (Rao et al. 2019) for the same data set. For the REACTION task, we downloaded the code and pre-trained models from the official repository of our data with their pipeline. We used the same training pro- cedure the used by the authors: The weights of the
Table 4: Optimal batch size for the models from Rao al. (2019), Bepler & Berger (201)). ) and Alley et al. (209))
al. (2019) and pre-processed
Model
Batch
ResNet
153
2019
LSTM
221
Transf.
90
14
Bepler & Berger 2019
240
Alley et all
2019
221

---

Published as a conference paper at ICLR 2021
work were initialized with the values of a pre-trained model on unsupervised tasks, and they were fine-tuned for the task. We used a learning rate of le 4 with a warm-up of 1000 steps. In order to find the optimal batch size for cach model, we trained each model increasing/deerrasing the suggested optimal batch size by mulliplying/dividingg by 2 until no improvement achieved. Thl. presents the optimal selected batch sizes.
D.3 BEPLER & BERGER 2019 MULTITASK
We download the code and pre-trained models on the multitask method described in the paper Bepler Berger (2019) from the official repository. Since this model was designed to predict feature embeddings for amino acids, we added an attention- basee pooling layer which computes weighted aggregation of the amino acid embeddings a protein descriptor. This is then process by 2 layer MLP to finally classify the protein. This model was then trained by selecting the best hyper-parameters from different runs. We increased the batch size by multiplying it by 2 and the best performance was achieved with a batch size 64. The learning rate was selected by multiplying it by 0.1, and the best performance was achieved by a learning rate of 0.001 for the classification head and attention- based pooling layers and 0.00001 for the pre-trained model. During the first epoch only the classification head and pooling layers where updated.
D.4 STRODTHOFF ET AL. 2020
We downloaded the code and pre-trained models from the official repository (Strodthoff al., 2020) and pre-processed the data sets with their pre-processing pipeline. We used the same training procedure followed by the authors, an initial learning rate of 0.001, which was decreased every time a new layer unfreezed for fine-tuning. To evaluate the model. we averaged the performance of the forward and backward pre-trained models. We experimented with different batch sizes and last layer sizes since in the original work the last layers were composed of only 50 neurons. The best performance was obtained for batch size of 32, and hidden layer of 1024 for the FOLD task and 512 for the REACTION task.
model was
D.5 ELNAGGAR ET AL. 2020
We downloaded the code and pre-trained model for the ProtBert-BFD network from the official repository (EL naggar et al. 2020) and followed their code example for fine-tuning the network. We use the default parameters for classification tasks and optimized the batch size by increasing and decreasing it multiplying by 2. The best performance was achieved by batch size of 32.
D.6 KIPF& & WELLING 2017
We implemented a GCNN architecture using the message passing defined by Kipf & Welling (017)) We used the standard architecture of three GCNN which outputs concatenated and give input to global pooling ReadOut layer. The global features were processed by a single layer MI.P with 1024 hidden layers. The protein graph was defined using the contact map of the amino acid sequence with threshold of 8 A, as this common practice in GCNN literature (Gao & Ji 2019). The initial amino acid features were learned with 16D embedding layer that was trained together with the model. We trained the model until convergence with momentum optimizer, batch size of 8, and an initial learning rate of 0.001 which was multiplied by 0.5 every 50 epochs with an allowed minimum learning rate of le 6. We also use dropout with probability equal to 0.5 in the final MLP and L2 regularization scaled by 5e 4. All parameters were selected for optimal performance using the same procedure in our ablations.
D.7 DIBIIL 2019
The Diehl (2019) model was implemented hy applying the pooling algorithm after each GCNN layer in the model of Kipf & Welling (017). The same hyperparameters used train this model.
D.8 DEREVYANKO ET AL. 2018
We implemented the 3DCNN model of the paper Derevyanko al. 2018) following the same number of layers, features, and input resolution, 11 density volumes with a resolution 120 x 120 x 120. We trained the model with momentum optimizer, a batch size of 16, and an initial learning rate of 0.005 which was multiplied by 0.5 every 50 epochs with an allowed minimum learning rate of le 6. The model was trained until convergence. For regularization, we used dropout layers with prohability of 0.5 in the final MI. and 1.2 regularization scaled by 4. All parameters were selected for optimal performance using the same procedure in our ablations.
15

---

Published as a conference paper at ICLR 2021
D.9 GLIGORUEVIC AL. 2019
We downloaded code from the official repository and pre-processed both data sets using their pipeline. downloaded pre-trained language model and run several experiments to select the best hyperparameters. We used a dropout rate of 0.5, learning rate of 2e - 4, and a batch size of 16. The model was trained until convergence.
D.10 BALDASSARRE ET AL. 2020
We downloaded the code from the official repository and pre-processed both data sets using their pipeline. Since this method was designed perform predictions protein and amino acid level, adapted the architecture by only using the global feature vector
searching for the best architecture best model had three GCNN layers, 128 out node features, 32 out edge features, and 512 out global features. We used learning rate of 0.005 scaled by 0.8 every 50 epochs, weight decay of le 5, and batch size of 16. The model was trained until convergence.
predictions
our tasks. The
16


Reviews:
[{"review_id": "yau7nsQ6n7P", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "__Summary__\nThe authors describe a method to transform 3D protein structures for supervised machine learning. Their method introduces a convolution operation that considers both the intrinsic distances between atoms as defined by their bond structure and the extrinsic distances as defined by 3D proximity. They also introduce interpretable pooling operations developed using known biology of the amino acids. Overall, the method is effective and straightforward to follow due to having avoided unnecessary complexity. The figures greatly aid the reader.\n\nThe authors’ method outperforms a variety of competitive alternatives on protein fold and function classification tasks. These are important problems for which the authors’ model has achieved a significant performance boost. I don’t see why this model wouldn’t work well for any 3D protein structure labels that can be collected. They also perform a through ablation analysis to establish the contribution of the various components of their method.\n\n__Major comments__\n* I wasn’t able to understand what the “neighborhood” ablations represent and how they differ from “convolution” ablations. Are the neighbors used for anything other than the convolutions? For example, “CovNeigh” uses only the intrinsic distances, similarly to “InConvC”. What makes these different?\n\n__Minor comments__\n* On page 7, a Table 4 is mentioned that doesn’t appear to exist. I think they mean Table 3.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lvFug2S-F5I", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents new convolutional and pooling operators for protein structures. These components are used to design an architecture that shows strong performance on several downstream tasks.\n\nThe main strength of the paper is the presentation of new ideas for modeling protein structures. The proposed operators leverage the intuition behind convolutional networks but extend them for the protein case, e.g. by introducing rotational invariance in addition to translational invariance. The ideas themselves are interesting to machine learning researchers and useful to those working proteins. Due to the complexity of the model, I recommend that the authors release their code so that other researchers could evaluate these ideas on additional problems. The writing and presentation is clear.\n\nWeaknesses:\n- More updated baselines should be used. For example, for the sequence-only baselines, the authors should compare to ProTrans [Elnaggar, et al. 2020] or [Rives, et al. 2019] which show better results than the baselines used here. On the structural side, the authors should compare to the architectures proposed by [Du, et al. 2019] or [Anand, et al. 2020]. \n- A key sequence baseline is missing: multiple sequence alignments.\n- The only tasks considered are classification tasks. The paper could be improved by evaluating on more practical tasks, such as protein design, e.g. the tasks in [Du, et al. 2019] or [Anand, et al. 2020]. The architecture described here could be very useful in those settings.\n- The authors compare to Bepler, et al. (2019) which is a great baseline since it uses both sequence and structural information. However, it appears from the text that the authors used the version of this model provided by Rao, et al. However, Rao et al. simply used the architecture from Bepler, et al. and re-trained it on sequence data only. Therefore, I recommend that the authors retrieve the weights from Bepler, et al. directly.\n- On the fold classification task, the hardest test set considered is \"Fold, in which proteins from the same superfamily are not present during training.\" It would be interesting to evaluate the model on a harder generalization setting in which proteins from the same fold are also not present during training. The delta between this model and DeepSF decreases when the sets go from family -> superfamily -> fold. To complete the picture, it would be important to go one step further.\n- Relatedly, the authors have not demonstrated that the models can generalize to novel folds. Without demonstrating this, the model cannot be used for important tasks, such as protein design. The paper would be much more compelling if the authors could show that their architecture generalizes better than prior work. To accomplish this, the authors would need to move beyond a classification framework toward a clustering framework because it's impossible for a classifier to predict novel folds.\n- The names of the test splits on the fold classification task is non-standard. Generally, \"fold split\" means that proteins from the same fold are not included in the same set; \"superfamily split\" means that proteins from the same superfamily are not included in the same set, etc. What the authors call the family split (\"in which proteins of the same family are present during training\") is usually not included as overfitting to / memorizing the training set could still result in good performance (perhaps this is why the proposed model scores 98.9% here).\n\nTo summarize the weaknesses: more work is needed on the baselines and metrics. Additional evidence is also needed to support that the model can generalize to unseen folds. \n\nOverall, this paper is a great start and the proposed model architecture could be interesting to ML researchers and practitioners in the biology space. In its current state, this is a borderline paper because it is missing a critical component of generalization of novel folds, which is necessary for this model to have significant impact in the field. If the authors can resolve my concerns during the rebuttal period, I am willing to raise my score.\n\nUpdate: The authors have included an additional experiment around fold generation in Sec 6.6. However, no baselines are included, so it is difficult to understand the result in context and understand how this method generalizes compared to existing methods. The authors have also included two additional baselines: Bepler, et al. and MSAs. More analysis is needed to compare this with SOTA in representation learning. The authors compare to \"Elnaggar et al. (2020)\" but it isn't clear which model was used. Elnaggar et al. (2020) have released a series of different models. The authors should clarify this in the camera-ready and ensure they used the best models released by Elnaggar et al. I have increased my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_z76ad0_t6Z", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n\n- I think the paper is exceptionally well-written and the figures are very carefully designed. Applaud!\n\n- Thank you for proper train/test/validation splits! Glad there are varying degrees of difficulty with proper held-out sequences.\n\n- I very much appreciate proper comparison to other methods. Very thorough.\n\n- Less important, but the model also performs better at these two tasks than any other approach. ( I say this because I believe the field shouldn't always require SoA if there is a significant technical advancement.)\n\nCons:\n\n- The authors site \"over-smoothing\" for why their convolution operator performs better, but provide no direct evidence that this is the case. It needs to be noted that this is either a hypothesis, or more concrete evaluation of this needs to be performed to make this claim.\n\n- Are there any replicates for standard error and ablation studies?\n\n- Table 3 BLAST comparison is weak. JackHMMER or HMMER based tools are more appropriate than BLAST.\n\nNeutral:\n\n- What defines a hydrogen bond? This definition is clear to me in secondary structure, but seems more loose in tertiary structure.\n\n- In your figures, it looks like only carbons, oxygens, and nitrogens are defined. What about hydrogens? If hydrogens aren't parameterized, how do you define hydrogen bonds? This may be good to clarify.\n\n- In Table 2, does the modification of the architecture change the number of parameters?\n\n- Definition of a \"ball query\" might be helpful.\n\n- Are there any sequences with post-translational modifications in the dataset? If so, how are those handled?", "extra_scores": {}, "extra_text": {}}, {"review_id": "IFYd1JRAH3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a deep learning architecture for representing and performing classifications on protein structures.  The representation involves three different distances: Euclidean distance and the shorted path between two atoms, where edges are either along covalent bonds or also include hydrogen bonds.  Each atom has a vector of associated features, and convolution is accomplished by defining a kernel on all three distances and then summing the features of each neighboring atom, weighted by the kernel value.  The paper also proposes three protein-specific pooling operations to cope with the large input size when representing all atoms in a protein.\n\nOverall, this is an extremely clear paper, and the core ideas appear to be sound.  Furthermore, the experimental validation is quite extensive, and the results are impressively good.  Some positive points are that the authors consider several different tasks, and numerous state-of-the-art methods are included in the comparison.  I particularly appreciated the careful ablation study, demonstrating not just that the entire system works end-to-end but that the various pieces each contribute to its behavior.\n\nThe experimental setup appears to be valid. There is always the chance that these results could be optimistic due to (presumably unintentional) model selection happening during development of the proposed method, or because of a mismatch between the training data used for the published models and the test set used here.  But I can't see how the authors could have done a better job to guard against such issues, other than the obvious step of making their code and trained models publicly available.  It is unfortunate that the manuscript makes no mention of this. \n\nOne drawback to this work is its focus on recent literature.  I found it strange that the earliest citation in the related work section is from 2013.  The tasks being solved here have been the focus of extensive research going back 25 years or more.\n\nThe manuscript is up front about the fact that a drawback of the method is its requirement that the input proteins have known 3D structure.  However, another potential drawback is that the input does not take into account homology information drawn from, e.g., a sequence similarity search over a large protein database.  This information is typically represented as a PSSM column for each observed amino acid.  I would like to have seen this acknowledged, since it seems like a potentially valuable source of additional information.\n\nA minor point: the introduction states that the model captures primary, seconary and tertiary structure, and then says that \"As chain bindings affect the tertiary structure, the quaternary structure is captured implicitly.\"  But of course, this argument could apply to any of the other levels: amino acid sequence implicitly captures secondary and tertiary structure.\n\nIncidentally, the Murzin cite has an incorrect year (1955).\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9TRw3pMaZdM", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a graph neural network architecture that operates on the atoms in a protein structure.\nIt proposes a specific multigraph and pooling model structure, constructed using Euclidian distance and 3 types of edges (Euclidian, covalent, and hydrogen+cov).\nThere are three consecutive levels of granularity, with nodes corresponding to: (1) atoms (2) amino acids and (3) grouped amino acids.\nThe model is used to make a global prediction for a protein, and results are presented on the taks of Fold Classification and Reaction classification.\n\nI recommend rejection for the paper in its current form, based on the concerns about the relevance of this method for fold classification (1/2 experiments), its framing as representation learning, and its framing as convolution vs graph neural networks.\n\n## Strenghts:\n* The two key model choices feel like a powerful choices for a graph neural network with an interesting domain-motivated set of architectural choices:\n  - construction of the hypergraph with shortest-distance edges of 3 types\n  - custom pooling of the graph from atom level nodes to groups of amino acids\n* The paper has helpful visualizations and well-written, however see below for concerns around framing.\n* Excellent ablation study in Table 2.\n\n## Weaknesses:\n* The key weakness is that the protein structure has to be provided as input to the network\n    - ~~therefore fold classification is a flawed experiment as the full atomic coordinates is all that's needed for perfect assignment to the folds. Specifically comparisons to sequence-only classification (TAPE, Unirep, etc) are misleading.~~\n    - as the authors point out, the amount of available data is tremendously less than sequence-only models. In fact the framing as \"representation learning\" is odd in this context, as there is no way to leverage unlabeled data, and no self-supervised objective is proposed.\n* I find the framing of the method somewhat misleading, on a few counts:\n    a) ~~representation learning~~ (see remarks abvove, no self-supervised + transfer of features)\n    b) naming the core layer of the model a \"convolution on 3D protein structures\" is off. A crucial element of standard convolution is the regularity of the domain, while this is intrinsically graph structured data. Furthermore I believe the method still fits in the \"neural message passing\" framework (see bullet below). Therefore the proposed architecture seems to be much better summarized as \"message passing graph neural network on a hierarchy of multi-graphs (hierarchy through protein pooling), with 3 types of graph edges defined by bonds and euclidian distances\".\n    c) after pooling, when vertices don't correspond to atoms but to clusters, the proposed convolution/GCN does not directly apply anymore. What are the edge connections at this stage?\n* I disagree with the phrase \"Although this operation could appear similar to message passing algorithms, they differ significantly\". I believe the method fits in the MPNN framework, roughly as follows (notation following Gilmer et al 2017, renaming x, xi to v, w):\n    - hidden state (per node) F_v with v the node (either atom, group of atoms, amino acid, cluster of amino acids depending on stage in the hierarchy)\n    - edge introduced if euclidian distance d(vw) < m_e\n    - edge features: 3 distances clamped [0,1]\n    - learned message function $M = \\kappa(e_{vw}) .  F_w$\n    - $h_v^{t+1} = m_v^{t+1}$ - or possibly including batchnorm and relu.\nThe above re-formulation is quite close to GCN from Kipf & Welling (2016) but with more complex learned message function function of the 3 distances.\n\n* Writing clarity: for eq (1) the notation needs to be introduced with dimensions (x, $\\kappa$, F). Specifically for kernel $\\kappa$ it needs to be made clear that $\\kappa_j$ is a function from $R^3 \\to R$ (?)\n* Comments on experimental results:\n    - ~~as mentioned above, I think fold classification is not an appropriate benchmark for this model~~\n    - for enzyme reaction classification, a sota method on this problem should be included as benchmark. Ryu et al 2019 (DeepEC) seems relevant here, or a method based on HMM profiles.\n\n-----\n### Edit: reply to author's response and updated paper \n(also see strikethroughs in the original review above)\n* Fold classification: let me withdraw my concern here, and will defer to other reviewers & AC judgement if this task makes sense with protein structure as input -- indeed it may not to be a trivial task. \n* Framing as (a) representation learning: improved in the updated paper, (b) convolution: still stands - the point cloud convs are not a very good comparison, since there is no graph structure there. (c) pooled coarsened graph stages: thanks for the pointer to end of Sec5.\n* Positioning wrt message passing: the paragraph is a big improvement, removing some claims about over-smoothing. However re: \"the message passing function is learned\": this is still very much within the default MPNN framework from Gilmer et al. Altogether, the whole method would still be much better framed as a graph-based network, rather than shoehorning this into a description of a single \"convolutional operator\".  This will allow a proper discussion of what is currently the end of Sec5, where the graph does not correspond to an atom-level graph anymore, rather they now correspond to amino acid or coarser level graphs - it is confusing that this coarser graph stages are so briefly glossed over.\n-- The citation to \"can also be understood in a message passing framework (Kipf & Welling, 2017)\" is off, should be \"Gilmer et al., 2017\" https://arxiv.org/abs/1704.01212 \n\nIn conclusion, I am raising my score from 4->5, leaning towards 6. There is a lot of good work in this paper, and I would consider the paper a clear accept with the same method and same results, if it were thoroughly rewritten based on graph neural networks. Requiring full atomic structure as input to the method is the major limitation to the application and impact of the method.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Protein molecule structure analysis is an important problem in biology that has recently become of increasing interest in the ML field. The paper proposes a new architecture using a new type of convolution and pooling both on Euclidean as well as intrinsic representations of the proteins, and applies it to several standard tasks in the field. 

Overall the reviews were strong, with the reviewers commending the authors for an important result on the intersection of biology and ML. The reviewers raised the points of:
- weak baselines (The authors responded with adding suggested comparison, which were not completely satisfactory)
- focus mostly on recent protein literature
- the reliance of the method on the 3D structure. The AC however does not find this as a weakness, as there are multiple problems that rely on 3D structure, which with recent methods can be predicted computationally rather than experimentally. 

We believe this to be an important paper and thus our recommendation is Accept. As the AC happens to have expertise in both 3D geometric ML and structural biology, he/she would strongly encourage the authors to better do their homework as there have been multiple recent works on convolutional operators on point clouds, as well as intrinsic representation-based ML methods for proteins. 
Decision: Accept (Poster)