Protein Representation Learning by Geometric Structure Pretraining
Authors: Zuobai Zhang
Minghao Xu
Arian Rokkum Jamasb
Vijil Chenthamarakshan
Aurelie Lozano
Payel Das
Jian Tang
Venue: ICLR 2023 poster (2023)
Keywords: Protein representation learning
self-supervised learning

Abstract: Learning effective protein representations is critical in a variety of tasks in biology such as predicting protein function or structure. Existing approaches usually pretrain protein language models on a large number of unlabeled amino acid sequences and then finetune the models with some labeled data in downstream tasks. Despite the effectiveness of sequence-based approaches, the power of pretraining on known protein structures, which are available in smaller numbers only, has not been explored for protein property prediction, though protein structures are known to be determinants of protein function. In this paper, we propose to pretrain protein representations according to their 3D structures. We first present a simple yet effective encoder to learn the geometric features of a protein. We pretrain the protein graph encoder by leveraging multiview contrastive learning and different self-prediction tasks. Experimental results on both function prediction and fold classification tasks show that our proposed pretraining methods outperform or are on par with the state-of-the-art sequence-based methods, while using much less pretraining data. Our implementation is available at https://github.com/DeepGraphLearning/GearNet.

Published as a conference paper at ICLR 2023
PROTEIN REPRESENTATION LEARNING BY GEOMETRIC STRUCTURE PRETRAINING
Zuobai Zhang Minghao Xu Arian Jamasb', Vijil Chenthamarakshan' Aurélie Lozano', Payel Das Jian Tang! Mila- Québec Al Institute', Université de Montréal", University of Cambridge IBM Research', HEC Montréal", CIFAR AI Chair
Izuobai. zhang, minghao. xu ) @mila. quebec, ar j390 cam ac. uk (ecvijil, aelooano, daspa pa/ @us. i.m. com, jian. tang@ anng@he.... ca
ABSTRACT
Learning effective protein representations is critical variety of tasks in biology such predicting protein function or structure. Existing approaches usually pretrain protein language models on a large number of unlabeled amino acid sequences and then finetune the models with some labeled data in downstream tasks. Despite the effectiveness of sequence- basenceee-eedddeeeee approaches, the power pretraining known protein structures, which available in smaller numbers only, has not been explored for protein property prediction, though protein structures are known be determinants of protein function. In this paper, we propose to pretrain protein representations according to their 3D structures. We first present simple yet effective encoder to learn the geometrie features of protein. We pretrain the protein graph encoder by leveraging multiview contrastive learning and different self seff-predittion tasks. Experimental results on both function prediction and fold classification tasks show that our proposed pretraining methods outperform or are on par with the state-of-the-arr sequence-based methods, while using much less pretraining data. Our implementation is available littp:: ////ttttua com/ DeepsraplleernnnngGGaaale
INTRODUCTION
Proteins are workhorses of the cell and are implicated in broad range of applications ranging from therapeutics to material. They consist a linear chain of amino acids (residues) which fold into specific conformations. Due to the advent of low cost sequencing technologies (Ma & Johnson 2012 Ma 2015), in recent years a massive volume of protein sequences have been newly discovered. As functional annotation of protein sequence remains costly and consuuinu,, accurate and efficient in silico protein function annotation methods are needed to bridge the existing sequence-funee function gap.
Since a large number of protein functions are governed by their folded structures, several data- driven approaches rely on learning representations of the protein structures, which then can be used for a variety of tasks such as protein design (Ingraham et al. 2019 Strokach et al. 2020 Cao al. 202T Jing al. (02T), structure classification (Hermosilla
2021). model quality assessment Baldassarre et al. 2021 Derevyanko et al. 2018), and function prediction (Gligorievvć experimental protein structure determination, the number of reported protein structures is orders of magnitude lower than the size of datasets in other machine learning application domains. For example, there are 1 182K experimentalllyddtermmii
al. (021)) ). Due the challenge
structures in the Protein Data Bank (PDB) (Berman et al. 2000) yy 47M protein sequences in Pfam (Mistry et al.) 2021 and vs 10M annotated images in ImageNet (Russakovsky et al. 2015).
To address this gap, recent works have leveraged the large volume of unlabeled protein sequence data to learn an effective representation of known proteins (Bepler & Berger 2019 Rives et al.) 2021 Elnaggar et al. [221)) A number of studies have pretrained protein encoders millions of sequences via self-supervised learning. However, these methods neither explicitly capture nor leverage the available protein structural information that is known to be the determinants of protein functions.

---

Published as a conference paper at ICLR 2023
To better utilize structural information, several structure-based protein encoders (Hermosilaa et al. 2021, Hermosilla & Ropinski 2022 Wang et al.] 2022a) have been proposed. However, these models have not explicitly captured the interactions between edges, which are critical in protein structure modeling (Jumper et al. 2021). Besides, very few attempts (Hermosilla & Ropinski 2022 Chen et al. 2022 Guo al. (022) have been made until recently to develop pretraining methods that exploit unlabeled 3D structures due to the scareity of experimenttlly-deeermiii
protein structures. Thanks to recent advances in highly accurate deep learning-basee protein structure prediction methods (Back) it al.; 2021] Jumper et al. (02T), it is now possible to efficiently predict structures for large number of protein sequences with reasonable confidence.
Motivated by this development, we develop protein encoder pretrained on the largest possible number of protein structures that is able to generalize to a variety of property prediction tasks. We propose a simple yet effective structure-based encoder called GeooEEry-AAarr Relational Graph Neural Network (GearNet), which encodes spatial information by adding different types of sequential structural edges and then performs relational message passing on protein residue graphs. Inspired by the design of triangle attention in Evoformer (Jumper et al. 202I), we propose a sparse edge message passing mechanism to enhance the protein structure encoder, which is the first attempt to incorporate edge-level message passing on GNNs for protein structure encoding.
We further introduce geometric pretraining method to learn the protein structure encoder based on the popular contrastive learning framework (Chen et al. 2020). We propose novel augmentation functions discover biologically correlated protein substructures that co-occur in proteins (Ponting) & Russell 2002) and aim to maximize the similarity between the learned representations of substruc- tures from the same protein, while minimizing the similarity between those from different proteins. Simultaeneously, we propose a suite of straightforward baselines based on self-prediction (Devlin (t al.] 2018). These pretraining tasks perform masked prediction of different geometric or physico- chemical attributes, such as residue types, Euclidean distances, angles and dihedral angles. Through extensively benchmarking these pretraining techniques on diverse downstream property prediction tasks, we set up solid starting point for pretraining protein structure representations.
Extensive experiments on several benchmarks, including Enzyme Commission number predic- tion (Glioorijevćć et al.) 2021), Gene Ontology term prediction (Gligorijević et al.] 2021). fold classification (Hou et al. 2018) and reaction classification (Hermosilla et al.) (021)) verify our Gear- Net augmented with edge message passing can consistently outperform existing protein encoders on most tasks in a supervised setting. Further, by employing the proposed pretraining method, our model trained on fewer than million samples achieves comparable even better results than the Hlateooftthe-at sequence-base based encoders pretrained on million- or billion-scale datasets.
RELATED WORK
Previous works seek to learn protein representations based on different modalities of proteins, including amino acid sequences (Rao et al. al, 2019 Elnaggar et al. 2021 Rives et al., 2021). multiple sequence alignments (MSAs) (Rao et al. 2021 Biswas et al.) 2021 Meier et al. [021] and protein structures (Hermosilla et al., 2021 [2Iioriioo et al. [021] Somnath et al. (021). These works share the common goal of learning informative protein representations that can benefit various downstream
applications, like predicting protein function (Rives et al.) (021) and protein- protein-protein interaction (Wang
et al. 2019). as well as designing protein sequences (Biswas et al. 2021).
Compared with eqquence-based methods, mauuctur--bsedd methods should be, in principle, better solution to learning an informative protein representation, as the function of protein is determined by its structure. This line of works seeks to encode spatial information in protein structures by 3D CNNs (Derevyanko et al., 2018) or graph neural networks (GNNs) (Gligorijević et al. 2021 Baldassarre et al.) 2021 Jing et al., 2021; Wang et al.] 2022a Aykent & Xia 2022). Among these methods, IEConv (Hermosilla et al. 2021) tries to fit the inductive bias of protein structure modeling, which introduced graph convolution layer incorporating intrinsic and extrinsic distances between nodes. Another potential direction is to extract features from protein surfaces (Gainza et al. 2020; [Sverrisonn et al.) 2021 Dai & Kelloogg
2021). (ommath et al. (2021) combined the advantages of both worlds and proposed a parameter-efficient multi-scale model. Besides, there are
We use AlphaFoldDB v1 and v2 Varadi et al. 2021 for pretraining, which is the largest protein structure datahase before March, 2022.
2

---

Published as a conference paper at ICLR 2023
also works that enhance pretrained sequence- eqqeence-based models by incorporating structural information in the pretraining stage (Bepler & Berger 2021 or finetuning stage (Wang et al.) (022b).
Despite progress in the design of structure-based encoders, there are few works focusing structure- based pretraining for proteins. To the best of our knowledge, the only attempt is three concurrent
works (Hermosilla & Ropinski] [022] Chen et al.] 2022] Guo al. (022), which apply contrastive learning, self-prediction and denoising score matching methods on a small set of tasks, respectively. Compared with these existing works, our proposed encoder conceptually simpler and more effective on many different tasks, thanks to the proposed relational graph convolutional layer and edge message passing layer, which are able to efficiently capture both the sequential and structural information. Furthermore, we introduce a contrastive learning framework with novel augmentation functions to discover substructures in different proteins and four different self- seff-prediction tasks, which can serve as solid starting point for enabling self-supervised learning on protein structures.
STRUCTURE-BASTA PROTEIN ENCODER
Existing protein encoders are either designed for specific tasks or cumbersome for pretraining due to the dependency on computationally expensive convolutions. In contrast, here we propose simple yet effective protein structure encoder, named GeomEtry-Aware Relational Graph Neural Network (GearNet). We utilize sparse edge message passing to enhance the effectiveness of GearNet, which is novel and crucial in the field of protein structure modeling, whereas previous works (Hermosilla al. 2021 Somnath et al. 202I) only consider message passing among residues or atoms.
3.1
GEOMETRY-AWARD RELATIONAL GRAPH NEURAL NETWORK
Given protein structures, our model aims to learn representations encoding their spatial and chemical information. These representations should be invariant under translations, rotations and reflections in 3D space. To achieve this requirement, we first construct our protein graph based on spatial features invariant under these transformations.
Protein graph construction. We represent the structure a protein as a residue-level relational graph 9 = (V. E, R), where V and & denotes the set of nodes and edges respectively, and R is the set of edge types. We use (i.j. r) to denote the edge from node i to node j with type r. We use n and m to denote the number of nodes and edges, respectively. In this work, each node in the protein graph represents the alpha carbon of a residue with the 3D coordinates of all nodes x E R=x3 We use fü and f(i.j.)) to denote the feature for node and edge (i. r), respectively, in which reside types, sequential and spatial distances are considered.
Then, we add three different types of directed edges into our graphs: sequential edges, radius edges and K-nearest neighbor edges. Among these, sequential edges will be further divided into 5 types of edges based on the relative sequential distance we add sequential edges only between the nodes within the sequential distance of 2. These edge types reflect different geometric properties, which all together yield a comprehensive featurization of proteins. More details of the graph and feature construction process can be found in Appendix C.
{-2, 1, 0, 1,2] between two end nodes, where
Relational graph convolutional layer. Upon the protein graphs defined above, we utilize a GNN to derive per-residue and shole-protenn representations. One simple example of GNNs is the GCN (Kipf & Welling 2017), where messages are computed by multiplying node features with a convolutional kernel matrix shared among all edges. To increase the capacity in protein structure modeling, IEConv (Hermosilla et al. 2021) proposed to apply a learnable kernel function on edge features. In this way, different kernel matrices be applied different edges, which achieves good performance but induces high memory costs.
To balance model capacity and memory cost, work (Schlichtkrull et al. (018)) to learn graph representations, where convolutional kernel matrix shared within each edge type and there are |R| different kernel matrices in total. Formally, the relational graph convolutional layer used in our model is defined as
relational graph convolutional neural net-
h (0) fū, u,
(BN [BN (Ere W,
n. ")"
h.0 h.-1)
1) + u (1)
3

---

Published as a conference paper at ICLR 2023
Specifically, we use node features f. as initial representations, Then, given the node representation h, for node at the 1-th layer, we compute updated node representation un!) by aggregating features from neighboring nodes (i). where (i) = {j E V (j, i, r) E} denotes the neighborhood of
node i with the edge type r, and W, denotes the learnable convolutional kernel matrix for edge type ReLU function the activation a (-))
Here BN denotes batch normalization layer and
Finally, we update f.) with u ()) and add a residual connection from last layer.
3.2 EDGE MESSAGE PASSING LAYER
As in the literature of molecular representation learning, many geometric encoders show benefits from explicitly modeling interactions between edges. For example, DimeNet (Klicpera et al.) 2020) 2D spherical Fourier-Bessel basis function to represent angles between two edges and pass messages between edges. AlphaFold2 (Jumper et al. 2021) leverages the triangle attention designed for transformers to model pair representations. Inspired by this observation, we propose variant of GearNet enhanced with an edge message passing layer, named as GearNet-Edge. The edge message passing layer can be seen as a sparse version of the pair representation update designed for graph neural networks. The main objective is to model the dependency between different interactions of residue with other sequentially or spatially adjacent residues.
Formally, we first construct relational graph G' = -V" R') among edges, which is also known as line graph in the literature (Harary Norman 1960). Each node in the graph G' corresponds to an edge in the original graph. G' links edge (i. r1) in the original graph edge (w, k. r2) if and only j = w and i A k. The type of this edge is determined by the angle between (i, j, f1) and (w, k, r2). The angular information reflects the relative position between two edges that determines the strength of their interaction. For example, edges with smaller angles point to closer directions and thus are likely to share stronger interactions. To save memory costs for computing large number of kernel matrices, we discretize the range [0, π] into 8 bins and use the index of the bin as the edge type.
Then, we apply similar relational graph convolutional network on the graph g' to obtain the message function for each edge. Formally, the edge message passing layer is defined as
m (fJimL) f(ii.))) m (bddr)) = BN Σ
TER' W ((
(w.krrr) ))-
(1-1)
(K.J.r.) denote the message function for edge (i,j,r) ) in the I-th layer. Similar as Eq. (T). the message function for edge (i,j.r1) will be updated by aggregating features from its neighbors (V(((i,;;,,) where NN, j.)) {(w, k, Vl((w, k. r2), (1,j,r1),r) E E'} denotes the set of incoming edges of (1. J. r1) with relation type r in graph g'.
Here use m (!)
Finally, we replace the aggregation function Eq. (I) in the original graph with the following one:
(1) u
a (BN EEER W, (o (n(-))
+ FC(m()))))
where FC(-) denotes a linear transformation upon the message function.
Notably, it is novel idea to relational message passing model different spatial interactions among residues. In addition, to the best of our knowledge, this is one of the first works that explore edge message passing for macromolecular representation learning (see concurrent work (Morehead et al., 2022) for protein-protein interactions). Compared with the triangle attention in AlphaFold2. our method considers angular information to model different types of interactions between edges, which are more efficient for sparse edge message passing.
Invariance of GearNet and GearNet-Edge. The graph construction process and input features of GearNet and GearNet-Edge only rely on features (distances and angles) invariant to translation, rotation and reflection. Besides, the message passing layers use pre-defined edge types and other 3D information invariantly. Therefore, GearNet and GearNet-Edge can achieve E(3)-invariance.
4 GEOMETRIC PRETRAINING METHODS
In this section, we study how to boost protein representation learning via self-supervised pretraining on massive collection of unlabeled protein structures. Despite the efficacy of self-supervised
4

---

Published as a conference paper at ICLR 2023
Protein Structure
Contrastive Learning
1/ = (ili E iil
gtr
Subsequence cropping
Random edge masking
Subspace cropping
Identity
x
Gy
(apaer) Pd
{1]l
V. |.
xplla d)
9pd
(space) (space)
9pd
x
Sequental cdge (6-1)
edge (d-1) K-nesrost neighour odgs
g
Rodius edge
Figure I: Demonstration of multiview contrastive learning. For each protein, we first construct residue graph G based the structural information (some edges are omitted to save space). Next, two views 92 and G, of the protein are generated by randomly choosing the sampling scheme and noise function. For Gz. first extract subsequence and then perform random edge masking with dash lines indicating masked edges. For Gy. we perform subspace cropping and then keep the subspace graph p.d g unchanged. Finally, contrastive learning loss optimized maximize the similarity between 9N and Gy in the latent space while minimizing its similarity with a negative sample g'.
pretraining in many domains, applying it to protein representation learning is nontrivial due the difficulty of capturing both biochemical and spatial information in protein structures. To address the challenge, first introduce our multiview contrastive learning method with novel augmentation functions to discover correlated co-occurrence of protein substructures and align their representations in the latent space. Furthermore, four self-prediction baselines are also proposed for pretraining structure-basc based encoders.
4.1 MULTIVIEW CONTRASTIVE LEARNING
is known that structural motifs within folded protein structures are biologically related (Mackenzie & Grigoryan 2017) and protein substructures can reflect the evolution history and functions of proteins (Ponting Russell, 2002). Inspired by recent contrastive learning methods (Chen 2020; He et al. 2020), framework aims to preserve the similarity between these correlated substructures before and after mapping to a low-dimensional latent space. Specifically, using a similarity measurement defined in the latent space, biologically-related substructures embedded close to each other while unrelated ones are mapped far apart. Figure [
illustrates the high-level idea.
Constructing views that reflect protein substructures Given a protein graph G. consider two different sampling schemes for constructing views. The first one is subsequence cropping, which randomly samples left residue I and right residue and takes all residues ranging from / This sampling scheme aims to capture protein domains, consecutive protein subsequences that reoccur in different proteins and indicate their functions (Ponting & Russell 2002). However, simply sampling protein subsequences cannot fully utilize the 3D structural information in protein data. Therefore, we further introduce subspace cropping scheme to discover spatially correlated structural motifs. We randomly sample residue as the center and select all residues within a Euclidean ball with a predefined radius d. For the two sampling schemes, we take the corresponding subgraphs from the protein residue graph G = (V,E, R). In specific, the subsequence graph ((ve)) and the subspace graph ((space) can be written as:
(seg)
{ili E V.lsisrs
e( ((,.)))) E EEiE (180)) .j E v,(80a) }.
(space) p.d
{ili E V, ||, 22 S d}.
(space) p.d
((i,((()) (1.5)) e E,i (sspace)
pd .j E (space))
5

---

Published as a conference paper at ICLR 2023
where g/ (V(6e0) _((0))) 1,r R) and G(spnce)
e(space) pp.d .R).
After sampling the substructures, following the common practice in self-supervised learning (Chen) ct al. 2020), we apply a noise function to generate more diverse views and thus benefit the learned representations. Here we consider two noise functions: identity that applies transformation and random edge masking that randomly masks each edge with a fixed probability 0.15.
Contrastive learning. We follow SimCLR (Chen al. 2020) optimize a contrastive loss function and thus maximize the mutual information between these biologically correlated views. For each protein G, sample two views GN and Gy by first randomly choosing one sampling scheme for extracting substructures and then randomly selecting one of the two noise functions with equal probability. We compute the graph representations h and hy of two views using our structure- based encoder. Then, a two-layer MLP projection head is applied to map the representations to lnver-dimensional space, denoted as 22 and Zy. Finally, an InfoNCE loss function defined by distinguishing views from the same or different proteins using their similarities (Oord et al. 20188). 8). For positive pair x and y, we treat views from other proteins in the same mini-batch as negative pairs. Mathematically, the loss function for positive pair of views x and can be written as:
exp(sim(Ze:: zy)/T)
Cu. = log
2B Ek=1 [[tta] exp(sim(zz, (s), =/T)
where B, denotes the batch size and temperature, 1 [(gta]] E {0. 1} is an indicator function that is equal to iff lo * x. The function sim( u, v) is defined by the cosine similarity between and v.
Sizes of sampled substructures. The design of our random sampling scheme aims to extract biologically meaningful substructures for contrastive learning. To attain this goal, it is of critical importance to determine the sampling length - for subsequence cropping and radius d for subspace cropping. On the one hand, we need sufficiently long subsequences and large subspaces to ensure that the sampled substructures can meaningfully reflect the whole protein structure and thus share high mutual information with each other. On the other hand, if the sampled substructures are too large, the generated views will be so similar that the contrastive learning problem becomes trivial. Furthermore, large substructures limit the batch size used in contrastive learning, which has been shown to be harmful for the performance (Chen et al. 2020).
To study the effect of sizes of sampled substructures, we show experimental results on Enzyme Commission (abbr: EC, details in Sec. (.TT) using Multiview Contrast with the subsequence and substructure cropping function, respectively. To prevent the influence of noise functions, we only consider identity transformation in both settings. The results plotted in Figure 2 and subspace cropping, as the sampled substructures become larger, the results first rise and then drop from certain threshold. This phenomenon agrees with our analysis above. In practice, we set 50 residues the length for subsequence and 15 as the radius for subspace cropping, which are large enough to capture most structural motifs according statistics in previous studies (Tateno et al. 1997).
Spoltal cstall
0.90
Saboe queece- Ldeerity
Saloqacc likanitty
0.53
For both subsequence
0.90
280
Figure 2: Fmax on EC E.S. cropping sequence length (# residues) for Subsequence Cropping and spatial cutoff (A) for Subspace Cropping.
In Appendix J we visualize the representations of the pretrained model with this method and assign familial labels based on domain annotations. We can observe clear separation based on the familial classification of the proteins in the dataset, which proves the effectiveness of our pretraining method.
4.2 STRAIGHTFFOWWRR BASELINES: SELFF PREDICTIO METHODS
Another line of model pre-training research is based on the recent progress of self-predicti methods in natural language processing (Devlin et al., 20188] Brown et al., 2020). Along that line, given a protein, our objective can be formulated as predicting the structure. Here, we propose four self-supervised tasks based on physicochemical or geometric properties: residue types, distances, angles and dihedrals.
part of the protein given the remainder of
The four methods perform masked prediction on single residues, residue pairs, triplets and quadru- ples, respectively. Masked residue type prediction is widely used for pretraining protein language

---

Published as a conference paper at ICLR 2023
models (Bepler & Berger 2021 and also known masked inverse folding in the protein commu- nity (Yang et al. 2022). J. Distances, angles and dihedrals have been shown to be important features that reflect the relative position between residues (Klicpera et al. 2020 For angle and dihedral prediction, we sample adjacent edges to better capture local structual information. Since angular values are more sensitive errors in protein structures than distances, use discretized values for prediction. The
four objectives are summarized in Table I and details will be discussed in Appendix. D
Method
Loss function
Sampled items
Residue Type Prediction
Ci CB(fresitae (h.). fi)
Single residuc
Distance Prediction
((ga)) (fam (Pi, H() [a,; 2t,1l2)
Single edge
Angle Prediction
(uir)). CB( CBi mighl (b, h2, k(), bin( Zijk)) (esaen)kkkka))kkkkr)) Jan (h., hi, 2 k(), bin( ((j())))))
Adjacent edge pairs
Dihedral Prediction
Adjacent edge triplets
Table I: seff-prediction methods. We use Fresidue (-), fasu ). fangle() fan (.) to denote the MLP head in each task. CE(-) denotes the cross entropy loss and bin(-) is used to discretize the angle. f. and æ; denote the residue type feature and coordinate of node i, respectively. hy denotes the representation of node after masking the corresponding sampled items in each task.
5 EXPERIMENTS
In this section, we first introduce our experimental setup for pretraining and then evaluate our models on four standard downstream tasks including Enzyme Commission number prediction, Gene Ontology term prediction, fold and reaction classification. More analysis can be found in Appendix F K
5.1 EXPERIMENTAL. SETUP
Pretraining datasets. We the AlphaFold protein structure database (CC-BY 4.0 License) (Varadi eet al,, (2021) for pretraining. This database contains protein structures predicted by AlphaFold2, and we employ both 365K proteome-wide predictions and 440K Swiss-Prot (Consortium) 2021 predictions. In Appendix F. we further report the results of pretraining on different datasets.
Downstream tasks. We adopt two tasks proposed in Gligorijevié et al. (202T) and two tasks used in Hermosilla et al. (202T) for downstream evaluation. Enzyme Commission (EC) number prediction seeks to predict the EC numbers of different proteins, which describe their catalysis of biochemical reactions. The EC numbers are selected from the third and fourth levels of the EC tree (Webb et al., 1992), forming 538 binary classification tasks. Gene Ontology (GO) term prediction aims to predict whether a protein belongs to some GO terms. These terms classify proteins into hierarchically related functional classes organized into three ontologies: molecular function (MF), biological process (BP) and cellular component (CC). Fold classification is first proposed in Hou et al. (2018), ) with the goal to predict the fold class label given a protein. Reaction classification aims to predict the onzyme-catalyzer reaction class of a protein, in which all four levels of the EC number are employed to depict the reaction class. Although this task is essentially the same with EC prediction, we include it make fair comparison with the baselines in Hermosilla et al. (2021)
Dataset splits. For EC and GO prediction, we follow the multi-cutoff split methods in Gligorijevi al. (2021) to ensure that the test set only contains PDB chains with sequence identity no more than 95% to the training set as used in Wang et al. (2022b) (See Appendix. for results at lower identity cutoffs). For fold classification, [Hou et al. (2018) provides three different test sets: Fold, in which proteins from the same superfamily are unseen during training, Superfamily. in which proteins from the same family are not present during training: and Family, in which proteins from the same family are present during training. For reaction classification, we adopt dataset splits proposed in Hermosilla et al. (02T), where proteins have less than 50% sequence similarity in-between splits.
Baselines. Following Wang et al. (202bb) and Hermosilla et al. (2021), we compare our encoders with many existing protein representation learning methods, including four sequence-based en-
coders (CNN (Shanehsazzadeh et al. 2020). ResNet (Rao et al.] 2019), LSTM (Rao et al. 2019)
encoders (GCN (Kipf & WWeling 2017). and Transformer (Rao et al. 2019 ). six structure-
GAT (Veličković et al. 2018). GVP (Jin et al.) 20 3DCNN MQA (Derevyanko et al. 2018),
GraphQA (Baldassarre et al. 2021 and New IEConv (Hermosilla & Ropinski 2022)). We also
include two models pretrained on large-scale sequence datasets (ProtBHRT-BB-D (Elnaggar et al.
7

---

Published as a conference paper at ICLR 2023
Table 2: Fmax on EC and GO prediction and Accuracy (%) on fold and reaction classification. [t] denotes results taken from Wang et al. (2022b) and [*] denotes results taken from Hermosilla jtt al. (202T) and Hermosilla & Ropinski (2022). Bold numbers indicate the best results under w/o pretraining and w/ pretraining settings. For pretraining, we select the model with the best performance when training from scratch, i.e. GearNet-Edge for EC, GO, Reaction and CaarNee-EEge-IECCCE for Fold Classification. We use the pretraining methods to name our pretrained models.
Method
Dataset (Size)
Pretraining
EC
BP
MP
GO
CC
Fold
Fold Closification
Super.
Fam.
Aog.
Reuction
CNN
Stanchaarzneeh et el 2020
0.545
0.244
0.354
0.287
11.3
13.4
53.4
26.0
51.7
Rerver
0.605
0.280
0.405
0.304
10.1
7.21
23.5
13.6
24.1
LSTM
0.425
0.225
0.321
0.283
4.33
18.1
11.0
Transformer
0,238
0.264
0.211
0.405
40.4
weo pretraining
Kix Wellins
0.320
0.252
0.195
0.329
16.8"
21.3"
52.89
GAT
NeNKoss 2.3
2
0.368
0.281
0.317
0.385
12.4
16.5
72.7
33.8
55.5
GVP
Tn 2l AU4
0:489
0.526
0.426'
0.420'
16.0
22.5
83.8
40.7
65.5
SDCNN_MD MIAA
Dxtevv Bino elal 2018
0.077
0,240
0.147
0,305
31.6*
45.44
92.5"
56.5"
72.2
GaspiQA
B Binrssmn
0.509
0.308
0.529
0.413
23.7*
32.5*
54. 4
46 16.9
60.8*
New IECone
0,735
0.374
0.544
0,664
47.6*
70.2*
99.2*
72.3*
87.2*
GearNet
0.730
0.356
0.503
0.414
28.4
12.6
95.3
55.4
79.4
GearNeetIECoo
0.800
0
0,422
42.3
64,1
GearNet-Edgee
0810
0.403
0,500
0,450
44.0
06.7
M6.6
0,100
0.400
0.581
0.430
48.3
70.3
99.5
85.3
DiepFRR
Ghrortijsii et al 2021
Piam (10M)
0.63
0.399
0.465
0.460
15.34
20.6*
63.34
ESM 15
pones 2044
UniRet50 (24M))
0.864
0.452
0.657
0.477
26.8
60.1
97.8
61.5
83.1
w/ pretraining
ProBBEXT-BPT
Peraee 2121
HFD (2.18)
03338
0.27%
0.456
0.408
26.67
55.8*
97.66
60.0"
72.2
LM-GVP
Wing F0220
UniRet100 (216M)
0.664
0.417
0.5457
New IFCony
2023
PDR (476K)
$0.6*
76.9*
Residue Type Predaction
AlpbaFoldDa (805K)
0.845
0.430
0.465
48.8
71.0
99.4
Distince Prediction
AleaFaaMDR (805K)
0.339
0.448
0.66
0.464
50.9
73.5
09.4
74.6
N7.5
Angle Prediction
AlpeFFoldD (805K)
0.853
0.458
0.625
0.473
56.5
76.3
99.6
77.4
86.8
Dihedral Prediction
AlpaaFoodDD (805K)
0.859
0.458
0.026
0.465
51.8
77.8
99.6
75.9
87.0
Multiview Contrast
AljiaFoldD88 (905K)
0.874
0.490
0.654
0.488
54.1
30.5
99.9
78.1
87.5
202Tp. ESM-1b (Rives et al. 202T) and two models combining pretrained sequence- based encoders with structural information RDeepFRI (Gligorijevćć et al. [ and LM-GVP (Wang et al., 2)02bb)) For LM-GVP and New IEConv, we only include results reported in the original paper due to the computational burden and the lack of codes. We do not include MSA-based baselines. since these evolution- based methods require a lot of resources for the computation and storage of MSAs but have been shown to be inferior ESM-1b 1b on function prediction tasks in Hu et al. (002)).
Training. On the four downstream tasks, we train GearNet and GearNet-Edge from scratch. As we find that the IEConv layer is important for predicting fold labels, we also enhance model by incorporating this as an additional layer (see Appendix C.2). These models are referred as GearNet- IBConv and Ceeareet-Edge-BBCony respectively. Following Wang al. (2022b) and Hermosilla & Ropinski (2022). the models are trained for 200 epochs on EC and GO prediction and for 300 epochs on fold and reaction classification. For pretraining, the models with the best performance when trained from scratch are selected, i.e., GearNet-Edde -dge for EC, GO, Reaction and MoarNeet-dge-IEEe Fold Classification. The models are pretrained on the AlphaFold database with our proposed five methods for 50 epochs. All these models are trained on 4 Tesla A100 GPUs (see Appendix E.3)
for
Evaluation. For EC and GO prediction, we evaluate the performance with the protein maximum F-score Fmax. which is commonly used in the CAFA challenges (Radivojac et al.) (013) (See Appendix [E.2 for details). For fold and reaction classification, the performance is measured with the mean accuracy. Models with the best performance on validation sets are selected for evaluation.
5.2 RESULTS
We report results for four downstream tasks in Table 2 including all models with and without pretraining. The following conclusions can be drawn from the results:
Our structure-based encoders outperform all baselines without pretraining on of 8 datasets. By comparing the first three blocks, we find that GearNet can obtain competitive results against other baselines on three function prediction tasks (EC, GO, Reaction). After adding the edge message passing mechanism, GearNet-Edge significantly outperforms other baselines on EC, GO-BP and GO-MF and is competitive on GO-CC. Although no clear improvements are observed on function prediction (EC, GO, Reaction) by adding IEConv layers, CearNee-Edge-IECC. results on fold classification. This can be understood since fold classification requires the encoder to capture sufficient structural information for determining the fold labels. Compared with GearNet-
achieve the best
8

---

Published as a conference paper at ICLR 2023
Edge, which only includes the distance and angle information features, the IEConv layer is better at capturing structural details by applying different kernel matrices dependent on relative positional features. These strong performance demonstrates the advantages of our structure-based encoders.
Structure- encoders benefit a lot from pretraining with unlabeled structures. Comparing the results in the third and last two blocks, it can be observed that models with all proposed pretrain- ing methods show large improvements over models trained from scratch. Among these methods, Multiview Contrast the best on 7 of 8 datasets and achieve the state-of-the-art results on EC, GO-BP, GO-MF, Fold and Reaction tasks. This proves the effectiveness of our pretraining strategies.
Pretrained structure- based encoders perform on par with even better than sequence-based encoders pretrained with much more data. The last three blocks show the comparision between pretrained requence-based and structure-baset models. It should be noted that our models are pretrained on dataset with fewer than one million structures, whereas all sequence-based pretraining baselines are pretrained on million- or billion-sete scallen-scale sequence databases. Though pretrained with an order of magnitude less data, our model can achieve comparable or even better results against these sequence-basee based models. Besides, our model is the only one that can achieve good performance on all four tasks, given that sequence-based models do not perform well on fold classification. This again shows the potential of structure- pretraining for learning protein representations.
5.3 ABLATION STUDIES
Method
layers
params.
EC
Method
FC
GO-BP
GO-MF
GO-CC
GearNet-Edge
6
42M
0.810
Multiview Contrast
0.874
0.490
0.654
0.488
w/o rel. conv.
6
23M
0.752
subsequence ideatity
0,866
0.477
0.627
0.473
-uhsuuce identity
0.480
0.640
wwo rel. conv.
8
39M
0.754
subsequence + random edge masking
0.869
0.484
0.641
0.471
w/o rel. conv.
10
60M
0.744
subspace random edge masking
0.876
0,481
0.645
0.470
Table 3: Ablation studies of GearNet-Fdge and Multiview Contrast. Rel. conv. is the abbreviation for relational graph convolution. The numbers of layers and parameters in GearNet-Edge are reported.
To analyze the contribution of different components in our proposed methods, we perform ablation studies on function prediction task. The results are shown in Table
Relational graph convolutional layers. To show the effects relational convolutional layers, we replace it with graph convolutional layers that share single kernel matrix among all edges. To make the number of learnable parameters comparable, we run the baselines with different number layers. As reported in the table, results can be significantly improved by using relational convolution, which suggests the importance of treating edges as different types.
Edge message passing layers. We also compare the results of GearNet with and without edge message passing layers, the results of which are shown in Table It can be observed that the performance consistently increases after performing edge message passing. This demonstrates the effectiveness of our proposed mechanism.
Different augmentations in Multiview Contrast. We investigate the contribution of each augmenta- tion operation proposed in the Multiview Contrast method. Instead of randomly sampling cropping and noise functions, we pretrain our model with four deterministic combinations of augmentations, respectively. As shown in Table 3 all the four combinations can yield good results, which suggests that arbitrary combinations of the proposed cropping and noise schemes can yield informative partial views of proteins. Besides, by randomly choosing cropping and noise functions, we can generate more diverse views and thus benefit contrastive learning, as demonstrated in the table.
CONCLUSIONS AND FUTURE WORK
In this work, we propose a simple yet effective structure-based encoder for protein representation learning, which performs relational message passing on protein residue graphs. novel edge message passing mechanism is introduced to explicitly model interactions between edges, which show consistent improvements. Moreover, five self-supervised pretraining methods are proposed following two standard frameworks: contrastive learning and self-prediction methods. Comprehensive experiments over multiple benchmark tasks verify that our model outperforms previous encoders when trained from scratch and achieve comparable or baselines while pretraining with much less data. We believe that our work is an important step towards adopting seff-supervised learning methods on protein structure understanding.
better results than the state-of-the- arrt

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The authors would like to thank Meng Qu, Zhaocheng Zhu, Shengchao Liu, Chence Shi, Minkai Xu and Huiyu Cai for their helpful discussions and comments.
This project is supported by AIHN IBM-MILA partnership program, the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR AT Chair Program, collaboration grants between Microsoft Research and Mila, Samsung Electronics Co., Ltd., Amazon Faculty Research Award, Tencent AI Lab Rhino-Bird Gift Fund, NRC Collaborative R&D Project (A14D-CORE-06) as well as the IVADO Fundamental Research Project grant PRF-2019-35831139722
REFERENCES
Mehmet Akdel, Douglas EV Pires, Eduard Porta Pardo, Jürgen Jänes, Arthur O Zalevsky, Bálint Mészáros, Patrick Bryant, Lydia I. Good, Roman A Laskowski, Gabriele Pozzati, al. structural biology community assessment of alphafold 2 applications. bioRxiv, 2021.
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence-based deep representation learning. Nature methods, 16(12):1315-133 1322- 2019.
Stephen F. Altschul, Thomas L.. Madden, Alejandro A. Schäffer, J Zhang, Zhang, Webb Miller, and David J. Lipman. Gapped blast and psi-blast: a new generation of protein database search programs. Nucleic acids research, 25 17:3389-4022 1997.
Sarp Aykent and Tian Xia. Gbpnet: Universal geometrie representation learning on protein structures. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.
Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using a three-track neural network. Science, 373(6557) 2021.
Federico Baldassarre, David Menéndez Hurtado, Ame Elofsson, and Hossein Azizpour. Graphqa: protein model quality assessment using graph convolutional networks. Bioinformaties, 37(3): 360-366, 2021.
Albert P Bartók, Mike C Payne, Risi Kondor, and Gábor Csányi. Gaussian approximation potentials: The accuracy of quantum mechanics, without the electrons. Physical review letters, 104(13): 136403, 2010.
Albert P Bartók, Risi Kondor, and Gábor Csányi. On representing chemical environments. Physical Review B, 87(18):184115, 2013.
Jörg Behler and Michele Parrinello. Generalized neural-network representation of high-dimensional potential-energy surfaces. Physical review letters, 98( 14)401, 2007.
Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. arXiv preprini arXiv: 1902.0 2019.
Tristan Bepler and Bonnie Berger. Learning the protein language: Evolution, structure, and function.
Cell Systems, 12(6):654-669, 2021.
Helen M Berman, John Westbrook. Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip E Bourne. The protein data bank. Nucleic acids research, 28(1): 235-242. 2000.
Surojit Biswas, Grigory Khimulya, Ethan Alley, Kevin M Esvelt, and George M Church. Low-n protein engineering with data-efficient deep learning. Nature Methods, 18(4):389-398 2021.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14 165, 2020.
10

---

Published as a conference paper at ICLR 2023
Yue Cao, Payel Das. Vijil Chenthamarakshan, Pin- Yu Chen, Igor Melnyk, and Yang Shen. Fold2seq: A joint sequence(ld) foldd embedding- based generative model for protein design. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Learning, volume 139 of Proceedings of Machine Learning Research, pp. 1261-1271. PMLR, 18 24 Jul 2021. wtt.ttps:: httttppp://prrceeee: ngs ml wtt....sss//13///a22222 htt.eeaeenn........errr////////
Machine
html
Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-aware protein self-supervised learning. arXiv preprint arXiv:2204.04213, 2022.
Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. Graph networks as universal machine learning framework for molecules and crystals. Chemistry of Materials, (1(9):3564-3577 2019.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020.
Stefan Chmiela, Alexandre Tkatchenko, Huziel E Sauceda, Igor Poltavsky, Kristof Schütt, and Klaus-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science advances, 3(5):e1633015, 2017.
The UniProt Consortium. Uniprot: the universal protein knowledgebase in 2021. Nucleic Acids Research, 49400))4400- 2021.
Bowen Dai and Chris Kellog. Protein interaction interface region prediction by geometric deep learning. Bioinformatics, 2021.
Georgy Derevyanko, Sergei Grudinin, Yoshua Bengio, and Guillaume Lamoureux. Deep convo- lutional networks for quality assessment of protein folds. Bioinformatics, 34(23):4046-0533
2018.
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/810.04400 010000 2018.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Wang Yu, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Towards cracking the language of lifes code through self superviedd deep learning and high performance computing. JEEE Transactions Pattern Analysis and Machine Intelligence, pp. 1-1, 2021. doi: 10. 10.100//AAM....21330355
Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, D Boscaini, MM Bronstein, and BE Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, 17(2):184-192 2020.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference machine learning, pp. 1263-1272. PMLR, 2017.
Vladimir Gligorijević, P Douglas Renfrew, Tomasz Kosciolek. Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn C Taylor, Ian M Fisk, Hera Vlamakis, et al. Structure- based protein function prediction using graph convolutional networks. Nature communications, 12 (1):1-14, 2021.
Yuzhi Guo, Jiaxiang Wu, Hehuan Ma, and Junzhou Huang. Self-supervised pre-training for protein embeddings using tertiary structures. In AAAI, 2022.
William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning large graphs In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1025-1035, 2017.
Frank Harary and Robert Z Norman. Some propertics of line digraphs. Rendiconti del circolo matematico di palermo, 9(2)1611-168 1960.

---

Published as a conference paper at ICLR 2023
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning graphs. In International Conference on Machine Learning, pp. 41144111111 PMLR, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Liang He, Shizhuo Zhang, Lijun Wu, Huanhuan Xia, Fusong Ju, He Zhang, Siyuan Liu, Yingee Xia, Jianwei Zhu, Pan Deng, et al. Pre-training co-evolutionary protein representation via pairwise masked language model. arXiv preprint arXiv:21 15527, 2021.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprini arXiv: 06.616006011111 15, 2016.
Pedro Hermosilla and Timo Ropinski. Contrastive representation learning for 3d protein structures. In Submitted to The Tenth International Conference on Learning Representations, 2022. URL
https::/openneviiet net wow.mmidddiiiii..a.
Pedro Hermosilla, Marco Schäfer, Matëj Lang. Gloria Fackelmann, Pere Pau Vázquez, Barbora Kozliková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsie convolu- tion and pooling for learning on 3d protein structures. International Conference on Learning Representations, 2021.
Liisa Holm. Benchmarking fold detection by dalilite v.5. Bioinformatics, 2019.
Jie Hou, Badri Adhikari, and Jianlin Cheng. Deepsf: deep convolutional neural network for mapping protein sequences to folds. Bioinformatics, 34(5):1295-1300) 2018.
Min Hu, Fajie Yuan, Kevin Kaichuang Yang, Fusong Ju, Jingyu Su, Hongya Wang, Fei Yang, and Qiuyang Ding. Exploring ioalution-basedde & -free protein language models as protein function predictors. ArXiv, abs/2206.06583. 2022.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. 12265, 2019.
Strategies for pre-training graph neural networks. arXiv preprint
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. Advances in Neural Information Processing Systems, 32:15820-15831 2019.
Bowen Jing, Stephan Eismann, Pratham N. Soni, and Ron O. Dror. Learning from protein structure with geometric vector perceptrons. In International Conference Learning Representations, 2021.
Peter Bjørn Jørgensen, Karsten Wedel Jacobsen, and Mikkel N Schmidt. Neural message passing with edge updates for predicting properties of molecules and materials. arXiv preprint 106.0380600334666 2018.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasu Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-5888 2021.
Thomas Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07300 2016.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference Learning Representations (ICLR), 2020.
Johannes Klicpera, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. arXiv preprint arXiv:2 106008903, 2021.
12

---

Published as a conference paper at ICLR 2023
Yi Liu, Limei Wang, Meng Liu, Xuan Zhang. Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks. arXiv preprint arXiv:2102.05013, 2021.
Amy X Lu, Haoran Zhang, Marzyeh Ghassemi, and Alan M Moses. Self-supervised contrastive learning of protein representations by mutual information maximization. BioRxiv, 2020.
Bin Ma. Novor: real-time peptide de novo sequencing software. Journal of the American Society for Mass Spectrometry, 26(1 (6(11))1885 18:18-8-1844 2015.
Bin Ma and Richard Johnson. De novo sequencing and homology searching. Molecular cellular proteomics, 11(2), 2012.
Craig O. Mackenzie and Gevorg Grigoryan. Protein structural motif's in prediction and design. Current opinion in structural biology, 44:161-167, 2017.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 10Xf::180..03346, 2018.
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alexander Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. bioRxiv, 2021.
Jaina Mistry, Sara Chuguransky, Lowri Williams, Matloob Qureshi, Gustavo Salazar, Erik LL Sonnhammer, Silvio CE Tosatto. Lisanna Paladin, Shriya Raj, Lorna J Richardson, al. Pfam: The protein families database 2021. Nucleic Acids Research, 49(D1):D412-D419, 2021.
Bhaskar Mitra and Nick Craswell. Neural models for information retrieval. ArXiv, abs/1705.01509. 2017.
Alex Morehead, Chen Chen, and Jianlin Cheng. Geometric transformers for protein interface contact prediction. In International Conference on Learning Representation, 2022.
Alexey Murzin, Steven E Brenner, Tim Hubbard, and Cyrus Chothia. Scop: a structural classifica- tion of proteins database for the investigation of sequences and structures. Journal of molecular biology, 247(4):536-540, 1995.
Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena-Hurtade Aidan N. Gomez, Debora Marks, and Yarin Gal. Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval. In ICML, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1xXiv:8807003745 2018.
Chris Paul Ponting and Robert R Russell. The natural history of protein domains. Annual review of biophysics and biomolecular structure, 31 :4541111 2002.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gec: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1150-11600 2020.
Predrag Radivojac, Wyatt T Clark, Tal Ronnen Oron, Alexandra M Schnoes, Tobias Wittkop, Artem Sokolov, Kiley Graim, Christopher Funk, Karin Verspoor, Asa Ben-Hur, et al. A large-scale evaluation of computational protein function prediction. Nature methods, 10(3):221-225 2013.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. In Advances in Neural Information Processing Systems, 2019.
Roshan Rao, Jason Liu, Robert Verkuil, Joshua Meier, John F Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. bioRxiv, 2021.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15), 2021.
13

---

Published as a conference paper at ICLR 2023
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. arXiv preprint orXiv:2007.02835 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 2015.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. arXiv preprint arXiv:2102.09944 2021.
Michael Schlichtkrull, Thomas N Kipf. Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference, pp. 593-607. Springer, 2018.
Kristof Schütt, Farhad Arbabzadah, Stefan Chmiela, Klaus R Müller, and Alexandre Tkatchenko. Quantum-chemical insights from deep tensor neural networks. Nature communications, 8(1):1-8, 2017a.
Kristof T Schütt, Pieter-Jan Kindermans, Huziel E Sauceda, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: continuous-fiter convolutional neural network for modeling 2017b.
quantum interactions. arXiv preprint
Amir Shanehsazzadeh, David Belanger, and David Dohan. Is transfer learning necessary for protein landscape prediction? arXiv preprint 1xXi:2011.034444 2020.
Iyy N. Shindyalov and Philip E. Bourne. Protein structure alignment by incremental combinatorial extension (ce) of the optimal path. Protein engineering, 11 9:739-47, 1998.
Vignesh Ram Somnath, Charlotte Bunne, and Andreas Krause. Multi-scale representation learning on proteins. Advances in Neural Information Processing Systems, 34, 2021.
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature Biotechnology. 35:1026-100 1028,, 2017.
Martin Steinegger and Johannes Söding. Clustering huge protein sequence sets in linear time. Nature communications, 911):188, 2018.
Martin Steinegger, Markus Meier, Milot Mirdita, Harald Vochringer, Stephan J. Haunsberger, and Johannes Soeding. Hh-suite3 for fast remote homology detection and deep protein annotation. BMC Bioinformatics, 20, 2019.
Alexey Strokach, David Becerra, Carles Corbi- Verge, Albert Perez-Riba, and Philip M Kim. Fast and flexible protein design using deep graph neural networks. Cell Systems, 11(4):00--41, 2020.
Haitian Sun, Tania Bedrax-Waar and William W. Cohen. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text. EMNLP, abs/1904.00537, 2019.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pp. 3319-3328. PMLR, 2017.
Baris Suzek, Hongzhan Huang, Peter McGarvey, Raja Mazumder, and Cathy H Wu. Uniref: comprehensive and non-redundant uniprot reference clusters. Bioinformatics, 13(10):1222-1288 2007.
Freyr Sverrisson, Jean Feydy, Bruno E Correia, and Michael M Bronstein. Fast end-to-end learning on protein surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12272-15211, 2021.
Tateno, Ikco, T Imanishi, Н Watanabe, T Endo, Yamaguchi, Y Suzuki, K Takahashi, K Tsunoyama, M Kawai, et al. Evolutionary motif and its biological and structural significance. Journal of molecular evolution, 44(1):538-$4 $43, 1997.
Michel van Kempen, Stephanic Kim, Charlotte Tumescheit, Milot Mirdita, Johannes Soeding, and Martin Steinegger. Foldseek: fast and accurate protein structure search. bioRxiv, 2022.
14

---

Published as a conference paper at ICLR 2023
Mihaly Varadi, Stephen Anyango. Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. Alphafold protein structure database: massively expanding the structural coverage of potein-sequeene space with high-aceuracy models. Nucleic acids research, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
URL
net forum? iicimiidd-rre 1d=rJXMpikks accepted poster.
Limei Wang, Haoran Liu, Yi Liu, Jemry Kurtin, and Shuiwang Ji. Learning protein representations via complete 3d graph networks. ArXiv, abs/2207. 12600, 2022a.
Yanbin Wang. Zhu-Hong You, Shan Yang, Xiao Li, Tong-Hat Hai Jiang, and Xi Zhou. A high efficient biological language model for predicting mootein-proteni interactions. Cells, 8(2):122, 2019.
Zichen Wang. Steven A Combs, Ryan Brand, Miguel Romero Calvo, Panpan Xu, George Price, Nataliya Golovach, Emmanuel O Salawu, Colby J Wise, Sri Priya Ponnapalli, et al. Lm-gyp: an extensible sequence and structure informed deep learning framework for protein property prediction. Scientific reports, 12(1):1-12, 2022b.
Oren F Webb, Tommy Phelps, Paul R Bienkowski, Philip M Digrazia, David C White, and Gary Sayler. Enzyme nomenclature. 1992.
Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. arXiv preprint 0r6iv:2106.04/1 2021.
Jinn-Moon Yang and Chi-Hua Tung. Protein structure database search and evolutionary classification. Nucleic Acids Research, 34:3646 3659, 2006.
Kevin Kaichuang Yang. Niccol'o Zanichelli, and Hugh Yeh. Masked inverse folding with sequence transfer for protein representation learning. bioRxiv, 2022.
Yuning You and Yang Shen. Cross-modality and self-supervised protein embedding for compound- protein affinity and contact prediction. Bioinformatics, 3Spupplemmett_22 2):1168
1eripemeeet_):iiii 1i74, 2022.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33: 5812-5823, 2020.
Yang Zhang and Jeffrey Skolnick. Tm-align: a protein structure alignment algorithm based on the tm-score. Nucleic Acids Research, 33:2302 2309, 2005.
Mengyao Zhao, Wan-Ping Lee, Erik P Garrison, and Gabor Marth. Ssw library: An simd c/c++ library for use in genomic applications. PLoS ONE, 8, 2013.
Zhaocheng Zhu, Chence Shi, Zuobai Zhang, Shengehao Liu, Minghao Xu, Xinyu Yuan, Yangtian Zhang, Junkun Chen, Huiyu Cai, Jiarui Lu, et al. Torchdrug: A powerful and flexible machine learning platform for drug discovery. arXiv preprint 21X1002002.08320, 2022.
15

---

Published as a conference paper at ICLR 2023
A
MORE RELATED WORK
A.1 SEQUENCE-BANE BASED METHODS FOR PROTEIN REPRESENTATION LEARNING
Sequence-based protein representation learning is mainly inspired by the methods of modeling natural language sequences. Recent methods aim capture the biochemical and co-evolutionary knowledge underlying a large-scale protein sequence corpus by self-supervised pretraining. and such knowledge is then transferred to specific downstream tasks by finetuning. Typical pretraining objectives explored
in existing methods include next amino acid prediction (Alley et al. 2019] Elnaggar et al.
2021
masked language modeling (MLM) (Rao al. 2019 Elnaggar al. 2021; Rives et al. 202
pairwise MLM (He ct al. 2021) and contrastive predictive coding (CPC) (Lu et al. 2020 Compared
to sequence- based approaches that learn the whole sequence space, MSA-based methods (Rao et al., 2021; Biiwas et al., 2021 Meier et al.) (021) leverage the sequences within protein family to
capture the conserved and variable regions of homologous sequences, which imply specific structures and functions of the protein family.
A.2 STRUCTURE-BASED METHOSS FOR BIOLOGICAL MOLECULES
Following the early efforts (Behler& & Parrinello 2007 Bartók et al. 2010 2013 Chmiela et al. 2017 of building machine learning systems for molecules by hand-crafted atomic features, recent works exploited end-to-end message passing neural networks (MPNNs) (Gimer et al. 2017) to encode the structures of small molecules and macromolecules like proteins. Specifically, existing methods employed node/atom message passing (Gilmer et al. 2017 |Schitt et al. [2017a b), edge/bond message passing (Jørgensen et al.) 2018 Chen et al. 2019) and directional information KKlicpera et al. 2020
Liu et al. 2021 Klicpera et al. 2021) to encode 2D or 3D molecular graphs.
Compared to small molecules, structural representations of proteins are more diverse, including residue-level, atom-level graphs and protein surfaces. There are recent models designed for residue- level graphs (Hermosilla et al. 2021 Hermosilla & Ropinskii (022) and protein surfaces (Gainza
al. 2020 Sverrisson et al. 2021). which achieved impressive results on various tasks.
A.3
PRETRAINING GRAPH NEURAL NETWORKS
Our work is also related to the recent efforts of pretraining graph neural networks (GNNs), which sought to Iearn graph representations in self-supervised fashion. In this domain, various self- supervised pretext tasks, like edge prediction (Kipf & Welling 2016 Hamilton al., 2017). , context
prediction (Hu et al. 2019: Rong et al. (020)) node/edge attribute reconstruction ( (Hu et al. 2019) and contrastive learning (Hassani Khasahmadi] 2020 Qiu et al. 2020) You et al. 2020] Xu et al.) 2021), are designed to acquire knowledge from unlabeled graphs. Besides, there is one recent work applying self-supervised learning for protein-ligand affinity prediction (You & Shen 2022). this work, focus on learning representations of residue-les level graphs of proteins in a superviid way. To attain this goal, we design novel protein-specife pretraining methods to learn the proposed structure-bar based encoder.
B POTENTIAL NEGATIVE IMPACT
This research project focuses on learning effective protein representations via pretraining with large number of unlabeled protein structures. Compared to the conventional sequence-based pretraining methods, our approach is able to leverage structural information and thus provide better representations. This merit enables more in-depth analysis of protein research and can potentially benefit many real-world applications, like protein function prediction and sequence design.
Limitations. In this paper, we only use 805K protein structures for pretraining. As the AlphaFold Protein Structure Database now covers over 100 million proteins, it would be possible train huge and more advanced structure- based models on larger datasets in the future. Scalability, however, should be kept in mind. Thanks to their simplicity, datasets, while more cumbersome procedures might not. Besides, we only consider function and fold prediction tasks. Another promising direction is apply our proposed methods more tasks, e.g.,
models could readily accommodate larger
16

---

Published as a conference paper at ICLR 2023
Protein Graph Construction
Spatial Relational Graph Neural Network
Edge Message Passing
(6,4)
(4.7)
(5.4)
(2:4)
(h-))
(4,7
Sequcatial
Sequential (dm-))
K-acured neighhor cdlge
Radius edge
13.00
Figure 3: The pipeline for GearNet and GearNet-edge. First, construct relational protein residue graph with sequential, radius and knn edges (some edges are omitted in the figure to save space). Then, relational graph convolutional layer is applied. Similar message passing layers be applied on the edge graph to improve the model capacity. This figure shows the update iteration for node and edge (4, 7, red). respectively.
protein-protein interaction modeling and pootein-guided ligand molecule design, which underpins many important biological processes and applications.
cannot be denied that some harmful activities could augmented by powerful pretrained models, e.g., designing harmful drugs. We expect future studies will mitigate these issues.
C MORE DETAILS OF GEARNET
In this section, describe more details about the implementation of our GearNet. The whole pipeline of our structure-based encoder is depicted Figure
C.I PROTEIN GRAPH CONSTRUCTION
For graph construction,
three different ways to add edges:
1. Sequential edges. The i-th residue and the j-th residue will linked by an edge the
sequential distance between them is below predefined threshold deeq. i.e., l/ il dseq- The type of each sequential edge is determined by their relative position = j - - in the sequence. Hence, there are 2d types of sequential edges.
2. Radius edges. Following previous works, we also add edges between two nodes and when the Euclidean distance between them is smaller than threshold dradis-
3. K-nearest neighbor edges. Since the scales of spatial coordinates may vary among different proteins, a node will be also connected to its k-nearest neighbors based on the Euclidean distance. In this way, the density of spatial edges guaranteed be comparable among different protein graphs.
Since we not interested in spatial edges between residues close with each other in the sequence, further add a filter to the latter two kinds of edges. Specifically, for radius or KNN edge connecting the i-th residue and j-th residue, it will be removed if the sequential distance between them lower than long range interaction cutoff drang i.e., i jl < diong-
In this paper, we set the sequential distance threshold dsey the radius d'uiius 10.0, the number of neighbors k 10 and the long range interaction cutoff dioan 5. By regarding radius edges and KNN edges as two separate edge types, there will totally 2dhen = 7 different types edges.
17

---

Published as a conference paper at ICLR 2023
Necessity of spatial edges. Here explain the necessity of radius and KNN edges by statistics and intuitions. These two kinds of edges result in very different degree distributions. In Figure we plot the average degree distribution over all proteins in AlphaFold Database v1. If we only consider KNN edges, the node degrees in protein graphs are close to con- stant, which makes difficult to capture those areas with dense interactions between residues. If we only consider radius edges, then there will be about 45,000 proteins with average degrees lower than two. In these sparse graphs, pretraining cannot cap- ture structural information effectively, e.g., Angle Prediction with limited edge pairs and Dihedral Prediction with limited edge triplets. Such sparsity can be hard to overcome by tuning radius cutoff, for the various scales of average distance on different proteins. By simply combining two kinds of edges, we can overcome these issues.
Radiuo edges
KXN edges
Redius KNN edges
troteins
30000
5000
Average degrees
Figure 4: The average degree distri- bution on AlphaFold Database.
Node and edge features. Most previous structure-based encoders designed for biological molecules (Baldassarre et al. 202T Hermosilla et al. 202T) used many chemical and spatial features, some of which are difficult obtain or tone-consu to calculate. In contrast, we only use the one-hot encoding of residue types with one additional dimension for unknown types as node features, denoted as f E {0, 1] nx21, which is enough to learn good representation as shown in our experiments.
The feature f(.j.)) for an edge (i. r) is the concatenation of the node features of two end nodes, the one-hot encoding of the edge type, and the sequential and spatial distances between them:
f(a..)) = Cat (fi, fj, onehot (r), V - -). || æz l12),
where Cat(.) denotes the concatenation operation.
C.2 ENHANCE GEARNET WITH IECONV LAYERS
In our experiments, we find that IEConv layers are useful for predicting fold labels in spite of their relatively poor performance on function prediction tasks. Therefore, we enhance our models by adding simplified IEConv layer an additional layer, which achieve better results than the original IEConv. Next, we describe how to simplify the IEConv layer and how to combine it with our model.
Simplify the IEConv layer. The original IEConv layer relies on intrinsic and extrinsic distances between two nodes, which are computationally expensive. Hence, we follow the modifications pro- posed in Hermosilla & Ropinski (2022), which show improvements as reported in their experiments. We briefly describe these modifications for completeness.
In the IEConv layer, we keep the edges in our graph G and use (1) to denote the hidden representation for node in the (-th layer. The update equation for node is defined as:
E ko(f(G,,,)) h (1-))
(5)
where N(i) is the set of neighbors of i, i,) is the edge feature between and and k.)-) an MLP mapping the feature to a kernel matrix. Instead of intrinsic and extrinsic distances in the original IEConv layer. follow New IEConv, which adopts three relative positional features proposed in Ingraham et al. (2019) and further augments them with additional input functions.
We aim to apply this layer original paper. Therefore, we simply remove the dynamically changed receptive fields, pooling layer and smoothing tricks in our setting.
constructed protein residue graph instead of the radius graph in the
Combine 1EConv with GearNet. Our model is very flexible to incorporate other message passing layers. To incorporate IEConv layers, we use our graph and hidden representations input and replace the update equation Eq. ① with
h
h (1-1)
u (1) + k
(6)
18

---

Published as a conference paper at ICLR 2023
D SELF-PREDICTION METHODS
The high-level ideas of the four self-prediction methods are demonstrated in Figure 5
Residue Type Prediction is based on the masked language modeling objective, which has been widely used in pretraining large protein language models (Bepler & Berger 202I). For each protein, we randomly mask node features of some residues and then predict these masked residue types via gaueture-based encoders. This method is also known Attribute Masking in the literature of molecules (Hu et al. 2019) and Masked Inverse Folding in the protein community (Yang et al. 2022).
Distance Prediction aims to learn local spatial structures by predicting the Euclidean distance between two nodes connected in the protein graph. A fixed number of edges are randomly selected and masked from the original graph. Then, the representations of end nodes will be used to predict the distances between them.
Self- Ceff-Prediction Learning
Raides Type Prodiction
Dittasee Prediction
Besides, angles and dihedrals between adjacent edges are also important features that reflect the relative position between residues Klicpera et al. (2021). Similarly, we can define the masked geometric losses Angle Prediction and Dihedral Prediction by randomly selecting and masking adjacent edge pairs and triplets. Here discretize the angles and diheddras by cutting the range [0, œ into 8 bins. Then, the representations of end nodes in the masked graph will be used to predict which bin the angles between them will belong
05
Angle Profiction
Dibatral Prediction
Figure 5: Illustration of self seff-prediction methods.
E
EXPERIMENTAL DETAILS
E.
DATASET STATISTICS
Table 4: Dataset statistics for downstream tasks.
# Proteins
Dataset
Train
# Validation
Test
Enzyme Commission
15,550
1,729
1,919
Gene Ontology
29.898
3,322
3,415
Fold Classification Fold
12,312
736
718
Fold Classification Superfamily
12,312
736
1,254
Fold Classification Family
12.312
736
1.272
Reaction Classification
29.215
2,562
5,651
Dataset statistics of downstream tasks are summarized in Table 4 Details are introduced as follows.
Enzyme Commission and Gene Ontology. Following DeepFRI (Gligorijević et al., (021). the EC numbers are selected from the third and fourth levels of the EC tree, forming 538 binary classification tasks, while the GO terms with at least 50 and more than 5000 training samples are selected. The non-redundant sets are partitioned into training, validation and test sets according to the sequence identity. We retrieve all protein chains from PDB with the code in their codebase and remove those with obsolete pdb ids, so the statistics will be slightly different from that in the original paper.
Fold Classification. We directly use the dataset in Hermosilla al. (2021), 1, which consolidated 16,712 proteins with 1,195 different folds from the SCOPe 1.75 database (Murzin et al. 1995).
Reaction Classification. The dataset comprises 37,428 proteins categorized into 384 reaction classes. The split methods are described in Hermosilla et al. (202T). where they cluster protein chains via sequence similarities and ensure that protein chains from the same cluster in the same split.
19

---

Published as a conference paper at ICLR 2023
E.2 EVALUATION METRICS
Now we introduce the details of evaluation metrics for EC and GO prediction. These two tasks aim to answer the question: whether protein has some particular functions, which can be multiple binary classification tasks.
The first metric, protein-centric maximum F-score Fmax defined by first calculating the precision and recall for each protein and then taking the average score over all proteins. More specifically, for given target protein and decision threshold & [O, 1J, the precision and recall are computed as:
precision,() = E, 1/ P.(
E, e P.(t)
and
E, EISER(() E
recall, (t)
(8)
where f is function term in the ontology, T3 is a set of experimentally determined function terms for protein (t().) denotes the set of predicted terms for protein with scores greater than or equal to t and 1 (0, }} is indicator function that is equal to 1 iff the condition is true.
Then, the average precision and recall over all proteins at threshold defined as:
precision(t)
precision,
(9)
((t
and
recall(1) N Σ (t).
(10)
where we use N denote the number of proteins and (t) to denote the number of proteins on which at least one prediction was made above threshold 1, i.e., (P:(1) > 0.
Combining these two measures, the maximum F-score is defined as the maximum value of F-measure over all thresholds. That is,
precision(t) recall(t) precision(() + recall(I) 5
Fmax
max
(1I)
The second metric, pair-centric under precision-recall curve AUPRpair- is defined the average precision scores for all protein-function pairs, which is exactly the micro average precision score for multiple binary classification.
E.3 IMPLEMENTATION DETAILS
In this subsection, describe implementation details of all baselines and our methods. For all models, the outputs will be fed into a three-layer MLP to make final prediction. The dimension of hidden layers in the MLP is equal to the dimension of model outputs.
CNN Shanehsazzadeh et al. 2020). Following the finding in Shanehsazzadeh et al. 2020))
we employ a convolutional neural network (CNN) encode protein sequences. Specifically, convolutional layers with 1024 hidden dimensions and kernel size 5 constitute this baseline model.
also adopt deep CNN model, i.e., the ResNet for protein sequences proposed by Rao et al. 2019). in our benchmark. This model is with 12 residual blocks and 512 hidden dimensions, and uses the GELU (Hendrycks & Gimpel 2016) activation function.
ResNet (Rao
al. 2019)
LSTM (Rao et al. 2019). The bidirectional LSTM model proposed by Rao et al. 2019 is another baseline for protein sequence encoding. It is composed of three bidirectional LSTM layers with 640 hidden dimensions.
20

---

Published as a conference paper at ICLR 2023
Transformer (Rao et al., 2019). The -tttenttiinnbasee Transformer encoder (Vaswani et al. 2017) is a strong model in natural language processing (NLP), Rao et al. (2019) adapts this model into the field of protein sequence modeling. We also adopt one of our baselines. This model has a comparable size with BERT-Small (Devlin et al. 2018), which contains Transformer blocks with
5 512 hidden dimensions and attention heads, with activation GELU (Hendryeks & Gimpell 2016).
GCN (Kipf& & Welling, 2017). We take GCN as a baseline to encode the residue graph derived by
our graph construction scheme. We adopt the implementation in TorchDrug (Zhu et al.) 2022), where 6 GCN layers with the hidden dimension of 512 are used. We run the results of GCN on EC and GO by ourselves and take its results on Fold and Reaction classification from Hermosilla et al. (202T)
GAT Veličković et al., 2018 We adopt another popular graph neural network, GAT, as structure- based baseline. We follow the implementation in TorchDrug and use 6 GAT layers with the hidden dimension of 512 and I attention head per layer for encoding. The results on EC, Fold and Reaction classification are based on our runs, and the results on GO are taken from Wang et al. (2022b).
GVP (Jing et al. 2021). The GVP model (Jing al. 2021 is decent protein structure encoder. iteratively updates the scalar and vector representations of a protein, and these representations possess the merit of invariance and equivariance. In our benchmark, evaluate this baseline method following the official source code. In specific, 3 GVP layers with 32 feature dimensions (20 scalar and 4 vector channels) constitute the GVP model.
3DCNN LOCNN_MQA (Derevyanko et al., 2018). We implement the 3DCNN model from the pa- per (Dervvyanko al. (0188) with box width of 40.0 and input resolution of 120 120 x 120. The model has 6 residual blocks and 128 hidden dimensions with ELU activation function.
GraphQA (Baldassarre et al. 2021). Following hyperparameters in the original paper, we con- struct residue graphs based on bond and spatial information and re-implement the graph neural network. The best model has 4 layers with 128 node features, 32 edge features and 512 global features.
New IEConv (Hermosilla Ropinski 2022). Since the code for New IEConv has not been made public when the paper is written, we reproduce the method according to the description in the paper and achieve similar results on Fold and Reaction classification tasks. Then, we evaluate the method on EC and GO prediction tasks with the default hyperparameters reported in the original paper and follow the standard training procedure on these two tasks.
DeepFRI GGlioriijevi et al. 2021). We also evaluate DeepFRI (Gliorijjevi et al. 202 1 in benchmark, which is popular structure-ber baased encoder for protein function prediction. DeepFRI employs an LSTM model to extract residue features and further constructs a residue graph to propagate messages among residues, in which a 3-layer graph convolutional network (GCN) (Kipf & Welling 2017) is used. We directly utilize the official model checkpoint for baseline evaluation.
ESM-1b (Rives et al. 2021). Besides the from-scratch sequence encoders above, we also compare with two state-of-the-ar pretrained protein language models. EBM-Ib (Rives et al. (021) is huge Transformer encoder model whose size is larger than BERT-Large (Devlin et al., 2018). and it is pretrained on 24 million protein sequences from UniRef50 (Suzek et al. 2007) by masked language modeling (MLM) (Devlin et al., 2018). In our evaluation, we finetune the ESM-1b model with the learning rate that is one-tenth of that of the MLP prediction head.
PRotBERT-BFD (Elnaggar et al., 2021). , The other protein language model evaluated in our benchmark ProtBERT-BFD (Elnaggar et al. 202I) whose size also excesses BERT-Large (Devlin ct al.) 2018). This model is pretrained on 2.1 billion protein sequences from BFD (Steinegger &
Söding 2018) by MLM (Devlin et al. 2018). The evaluation ProIBERT-BFD uses the same
learning rate configuration as ESM-Ib.
LM-GVP (Wang et al. 2022b). To further enhance the effectiveness of GVP (Jig et al. 2021] )
Wang et al. (2022b) proposed to prepend a protein language model, i.e. ProtBERT (Elnaggar et al.
21

---

Published as a conference paper at ICLR 2023
Table 5: Hyperparameter configurations of our model on different datasets. The batch size reported in the table refers to the batch size on each GPU. All the hyperparameters are chosen by the performance on the validation set.
Hyperparameter
EC
GO
Fold
Reaction
#layer
6
6
GNN
hidden dim.
512
512
512
512
dropout
0.1
0.1
0,2
0.2
optimizer
Adam
Adam
SGD
SGD
learning rate
le-4
le-4
1e-3
le-3
Learning
weight decay
0
5e-4
Se-4
batch size
2
2
2
2
# epuch
200
200
300
300
Table 6: Fmax on EC and GO tasks under diffferent sequence cutoff's (30% 40% / 50% / 70% 95%%..
Method
BC
GO-BP
GO-ME
GO-CE
CNN
0.366/0.30 0.36
0.22 0.429
0, 0. 199 FO, 1977 //
0:243/0.233
0.292 70.356
0.258 25.25 / 2N3 0.387
RoeNet
0449//441///
00526
0.230/0
0:283/
0.321 273
LSTM
0.547/0
19 105
0/22X
0.276
0.263
2703
Transormer
0.167/01 8:178/11////
/0.262 0022//033//0.66
0.184/00187
0.195 2.204 70.211
0.37%
0.345/
0.246
0.251
/0.24K /0.3//0002025555
0.180/0.118
00.87
0.318
GearNee-rdge
/..04/0.777
/0.377/0.000
DespFRI
0.361
0.374
3633633/ (0445
/0.444 (0-55)
ESM-1b
0127//0..///77771 /0.854
0,394
/0.42/00.455
0.546
11.625
/0.465 (1465)
Multiview Conteast
0.744 0.808 0..54/0..874
(L.42/ 0.449 (0.47) /8.49
0.533 0.544 /0.5///0///112223
0.459 (.466
202T), before GVP to additionally utilize protein sequence representations. We also adopt this hybrid model as one of our baselines, and its implementation follows the official source code.
Our methods. For pretraining, we use Adam optimizer with learning rate 0.001 and train a model for 50 epochs. Then, the pretrained model will be finetuned on downstream datasets.
For Multiview Contrast, we set the cropping length of subsequence operation as 50, the radius of subspace operation as 15, the mask rate of random edge masking operation as 0.15. The temperature T in the InfoNCE loss function set as 0.07. When pretraining GearNet-Fdge and GearNet-Fdge- IECony, we use 96 and 24 as batch sizes, respectively.
For Distance Prediction, we set the number of sampled residue pairs as 256. The batch size will be set as 128 and 32 for GearNet-Edge and GearNet-Edge-IIEe IEConv, respectively. For Residue Type, Angle and Dihedral Prediction, we set the number sampled residues, residue triplets and residue quadrants as 512. The batch size will be set as 96 and 32 for GearNet-Edge and GearNet- GearNet-Edge-IEo Edge-IECooy, respectively.
For downstream evaluation, the hidden representations in each layer of GearNet will be concatenated for the final prediction. Table 5 lists the hyperparameter configurations for different downstream tasks. For the four tasks, we use the same optimizer and number of epochs in the original papers to make fair comparison. For EC and GO prediction, we use ReduceLRRnnlaate
scheduler with factor 0.6 and patience 5, while use StepLR scheduler with step size 50 and gamma 0.5 for fold and reaction classification.
F ADDITIONAL EXPERIMENTAL RESULTS ON EC AND GO PREDICTION
Results under different sequence identity cutoffs. Besides the experiments in Section 5 where 95% is used as the sequence identity cutoff for EC and GO dataset splitting, we also test our models and several important baselines under four lower sequence identity cutoffs and show the experimental results in Table 6 The aim of this experiment is to test the robustness of different models under different hold-out test sets, with lowering cutoff indicating lower similarity between training and test sets. It can be observed that, at lower cutoffs, our model can still achieve the best performance among models without pretraining and get comparable better results against ESM-1b lb after pretraining.
22

---

Published as a conference paper at ICLR 2023
Table 7: AUPR on EC and GO prediction. [+] denotes results taken from Wang et al. (202bb) For pretraining, we select the model with the best performance when training from scratch, i.e., GearNet-Edge. We omit the model name and use pretraining methods to name our pretrained models.
Method
Pretraining Dataset (Size)
EC
GO
BP
MF
CC
CNN Shamebsazzadee el al 2020
0.526
0.159
0.351
0.204
ResNer Kio a 2019
0.590
0.205
0.434
0.214
LSTM (Ra0
2017
0414
0.156
0.334
0.192
Transformr R30 ml
2019
0.218
0.156
0.177
0.210
w/o pretrsining
Kipf Welling
0.319
0.136
0.147
0.175
GAT Vellloovic al
2018
0.320
0.171
0.329
0.249
GVP Jiny al. 2021
0.482
0.224
0.458'
0.279
3DCNN_ Derev vanko al
2018
0,029
0.132
0.075
0.144
GraphQA [Barcssse & PUP
0.543
0.199
0.347
0.265
New IE5Comv (ermoosilaa & Kopinski
2022
0.775
0.273
0.572
0.316
GearNet GearNet-IECony
0.251
0.211
0.490
0.276
0.835
0.231
0.547
0.259
GearNet-Edge
0.251
GeaarNttEEdgeIIE.C.
0.843
0.244
0.561
0.284
DeepfRR (Glizorievii al.
(10M)
0.282
0.363
pretraining
ESM-IB (Kives et ar 2021
UniRet50 (24M)
0.889
0,332
0.639
0.324
ProtBERT-BBD (Elnastar
BFD (2. IB)
0.859
0.188
0.234
L.MGVP
Wing et al. 2122h
UniRef (216M)
0.710
0.302'
0.580'
0.423'
Residue Type Prodiction
AlphaFoldDB (805K)
0.870
0.267
0.583
0.311
Distance Prediction
AlphaFoldDB (805K)
0,863
0,274
0.586
0.527
Angle Prediction
AlphaFoldDR (805K)
0.880
0.291
0.603
0.311
Dihedral Prediction
AlphaFoldDB (805K)
0.881
0.304
0.603
0.338
Multiview Contrast
Alphal'oldDB (805K)
0.292
0.596
AUPR on EC and GO prediction. We have reported experimental results on EC and GO prediction with Froxx as the metric in Section 5 Here we report another popular metric AUPR in Table Note that we still use the best model selected by Fmax on validation sets. It can be observed that our model can still achieve the best performance on EC prediction in both from scratch and pretrained settings. However, there are still non-trive tnivirivial gaps between our models with the state-of-the probably is because of the inconsistency between the two evaluation metrics. It would be interesting to study the relationship between these two metrics and develop a model good at both in future works.
results. This
Sampling schemes in self- relf-rrediction methods. Different sampling schemes may lead to different results for self- eeff-prediction methods. We study the effects of sampling schemes using Dihedral Prediction as an example. Instead of sampling dihedral angles formed by three consecutive edges, we try to predict the dihedrals formed by four randomly sampled nodes. We observe that the Fmux decreases from 0.859 to 0.821. This suggests that it is better of learning residue representations to capture local spatial information instcad of global information. The change of sampling schemes will make self- peef-credittion tasks more difficult to solve, which even brings negative effects after pretraining.
Pretraining on different datasets. We use the AlphaFold protein structure database as our pre- training database, because it contains the largest number of protein structures and is planned to cover over 100 million proteins. However, the structures in this database are not experimentally determined but predicted by AlphaFold2. Therefore, it is interesting to see the performance of our methods when pretraining on different datasets.
To study the effects of the choice of pretraining dataset, we build another dataset using structures extracted from Protein Data Bank (PDB) (Berman et al., 2000). Specifically, we extract 123,505 experimentally-de determind protein structures from PDB whose resolutions are between 0.0 and 2.5 angstroms, and we further extract 305,265 chains from these proteins to construct the final dataset.
Next, we pretrain our five methods on AlphaFold Database v1 proteeomewiiee structure predictions), AlphaFold Database v2 (Swiss-Pae -rott structure predictions) and Protein Data Bank and then evaluate the pretrained models on the EC prediction task. The results reported in Table g As can be seen in the table, our methods can achieve comparable performance on different pretraining datasets. Consequently, our methods are robust to the choice of pretraining datasets.
23

---

Published as a conference paper at ICLR 2023
Table 8: Results of GearNet-Edge Edge pretrained on different pretraining datasets with different methods Models are evaluated on the EC prediction task.
Restuce Type Prediction
Prediction
Augle
Diltedrral
AU
AUUPR,
Feus
A
AL1P%
Alpl Dutaalose (s] 1 +2+
904.72
0.891
0.874
0.870
0.834
0.563
0.530
0.590
0.553
0.581
0.659
AlphuaFold Dutabous (x1)
365,195 198
0,890
0.874
0.869
0.642
0.871
0.343
0.879
0.854
0.877
0.632
Alghairold Dutabaaee (x2)
439.674
0,800
0.874
0.888
(1836
11358
03386
03881
01353
0.81
9.991
Protcin Data Bank
305,265
0.881
0.859
0,870
0.841
0.865
0.847
0.580
0.857
0.586
0.856
G STRUCTURE PRETRAINING ON EGNN
To verify the effectiveness of proposed methods, we choose another common backbone model, equivariant graph neural network (EGNN) (Satorras et al., 2021). for pretraining We follow the experimental setup in Appendix E.3 for pretraining and finetuning the model. The results on the EC dataset are reported in Table 9 It can be seen that the performance of EGNN is improved by large margin with the five pretraining methods. Among them, Distance Prediction and Multiview Contrast are the top two methods. The ranks of these five methods quite different from the ranks in the main paper. This is probably because the capacity of EGNN limits its performance some pretraining tasks and thus reduces their benefits.
Method
F Fmax
EGNN
0.640
Residue Type Prediction
0.729
Distance Prediction
0.761
Angle Prediction
0.718
Dihedral Prediction
0.662
Multiview Contrast
0.752
Table 9: Pretraining results on EC with EGNN as backbone models.
H COMBINE SEQUENCE- AND STRUCTURE-BASE ENCODERS
In the main paper, we compare the pretrained sequence- and structure-based encoder and show that geometric structure pretraining can achieve compet- itive results with much less data. To further show the benefit of incorporating structural information, here we choose ESM-1b as a baseline and build a structure- base encoder based on its output. Specif- ically, we replace the raw node features GearNet with pretrained sequence representations. We train the model on EC with the same configurations for GearNet and finetune the ESM-1b lb model with learn- ing rate le-5. The results are shown in Table [Õ] It can be observed that the ESM-1 Net without structure-based pretraining, which suggests the
GO
Method
EC
BP
MF
CC
GeurNet
0.730
0.356
0.503
0.414
GearNet-Egge -w/ Multiview Contrast
0,810
0.403
0,580
0.450
0.874
0.490
0.654
0.488
ESM-
0.864
0.452
0.657
0.477
ESM-Ib+GearNet
0.883
0.491
0.677
0.501
Table 10:
Results (Fmax) for combining seugence- and structure-based encoders.
model can achieve SOTA performance importance of utilizing protein structures. Also, it is promising to explore pretraining methods on the combined encoder. We leave this direction for future work.
I COMBINE NEURAL AND RETRIEVAL- -BASED METHODS
Searching database to retrieve similar proteins is popular method used in the biological community when predicting properties target protein, e.g., searching multiple sequence alignments and templates for structure prediction (Jumper et al.) (021). There have been large amount of tools proposed for aligning protein sequences (Altschul et al. 997, [Steinegger & Söding [017]] Steinegger et al. 2019) and structures (Yang & Tung 2006]] Zhang & Skolnick 2005] van Kempen et al. 2022 Holm 2019). this section, we first compare our neural representation based method with retrieva-bbsedd methods on function and fold classification tasks and then showcase the potential of our proposed methods on structure-bait based search tasks.
Comparison with retrieval-basee methods. We select a structure alignment tool, Foldseek (van Kempen al. (2022), as retrieval-based baseline. The method trains VQ-VAE on SCOPe40 to discretize structural units into an alphabet twenty 3Di states and then transforms the problem to 3Di sequence alignment, which is done by MMseqs2 (Steinegger Söding 2017). When using
24

---

Published as a conference paper at ICLR 2023
GO-BP
GO-FF
GO-DC
Fold Clesification
A
A
AU
AUPR
Fold
Super.
Foldheck (vo Kempen et al.
0.888
0.462
7.57
GearNee-Bdgee-IECC
Mallivien Contraat
0.490
0.596
0.48M
54.1
90.5
Muttiview Cantrant Foldbeek
0,908
0.983
0.314
0.500
0.615
0.673
0.319
0.467
Table 11: Comparison between neural and retrieval-based methods on BC, GO and fold classification tasks. As in T we use GearNet-Edge on EC and GO prediction and SeaaNeet-Ege--EECon on fold classification as backbone models. The results w/o and w/ pretraining and those ensembled with Foldseek are reported on EC and GO. We omit the ensemble results on fold classification due to the poor performance of Foldseek.
Foldseek, follow the parameters provided in their github repo For each protein in the test set of our benchmark tasks, we use Foldseek to retrieve the most similar protein in the training set, the label of which will be used for prediction.
We report the results of retrieval-based and our proposed neural methods in Table 11 First, we find that Foldseek achieves very good performance on EC and GO prediction. In terms of Fmax. it is better than GearNet-Edge on all tasks and competitive with the pretrained GearNet-Edge on EC and GO-MF. The accuracy of Foldseek makes it strong baseline for proteins with similar structures in the training set. However, when the dataset is split by structural similarities, e.g., fold classification, the structural alignment tool fails to retrieve similar proteins and get accurate prediction as shown in the table. This can be attributed to the inherent limitation of retrieval-based methods, i.e., the lack of generalization ability to novel data points.
Furthermore, to utilize the advantages from both worlds, we combine neural and retrieval- meth- ods via ensemble. As shown in the last row of Table [1 both metrics are significantly improved on EC, GO-BP, GO-MF compared with the separate neural and retrieval-based methods. Consequently, would be interesting to explore the combination of these two kinds of methods the future, as have done in many machine learning tasks (Mitra & Craswell] 2017 Sun al. 2019 Notin et al. 2022))
Results on structure-based search tasks. To show the potential of structure-based modeling for biological applications, we test our model on the SCOPe40 benchmark proposed in van Kempen al. (2022). The authors cluster the SCOPe 2.0198 at 40% sequence identity and obtain 11,211 non rend-aadndant protein sequences. They perform an all-versus-all search on the dataset and test the ability of structure alignment tools for finding proteins of the same SCOPe family, superfamily, and fold. For each query, they measure the fraction of TPs (true positive matches) out of all correct matches until the first (false positive) that matches a different fold. The sensitivity is calculated by the area under the curve of the cumulative ROC curve up to the first FP.
Table 12:
Sensitivity of searching proteins of the same family, superfamily and fold on SCOPe40. Results are evaluated with the scripts and predictions provided in (van Kempen et al. 2022).
Method
Fold
Super.
Fam.
Avg.
MMseqs2 (Steinegger & Söding 2017
0.001
0.082
0.542
0.208
3D-BLAST Yang & lung 2006
0.009
0.126
0.572
0.235
CLE-SW (Zhao et al. 2013
0.021
0.293
0.763
0.359
CE (Shindyalov & Bourne 1998)
0.131
0.529
0.885
0.515
TMalign- (Zhang & SkolnIck
2005
0.188
0.618
0.906
0.571
TMalign ZZhang & Skolnick 2005
0.188
0.610
0.901
0.566
Foldseek vvan Kempen et al.
2022
0.155
0.593
0.914
0.554
DALI Holm 2019)
0.310
0.751
0.942
0.667
GearNet-EEge-IECoon
0.474
0.722
0.936
0.710
GearNee--Edg--IECoov + DALI
0.481
0.779
0.963
0.741
25

---

Published as a conference paper at ICLR 2023
AlTar aperiandly
family
Cytochrome P450 family
faciitatrr
Restidase
Small GTPare sapertenity
TRAPIC class
-ommts ATfasse supertamtly
Perocdass famty
Kruspsel C2H2 amc-iinge prosem family
Mathyttranieeeee superiam
Closs like SAM tinding netlytrandierse superfanily
AAA AlPese family
Troteto euperfundly
Mincchondeel zamer 2.429) tamiy
ABC cancporter supefanilyy
GDSL pelytic encyme family
Figure 6: Latent space visualization of GearNet-Edge (Multiview Contrast) on AlphaFold Database vl.
We report the results of seven structure alignment tools and sequence search tool evaluated in van Kempen al.) (2022). For comparison, use the GearNet-Edge-ICCon model trained fold classification to extract representations for proteins and use the cosine similarity between representa- tions to retrieve similar proteins. The sensitivity of these methods reported in Table It can be observed that method achieves the best performance with DALI, though finding fewer proteins of the same family and superfamily, our method can achieve higher sensitivity at the fold level. This can be explained by the better generalization ability to novel structures of neural methods. With the ensemble of neural and retrieval- based methods, we achieve the SOTA performance at all levels. This again demonstrates the effectiveness of our proposed method and the potential of combining neural and retrieval-based methods.
average among all baselines. Compared
J LATENT SPACE VISUALIZATION
For qualitatively evaluating the quality of the protein embeddings learned by our pretraining method, we visualize the latent space of the GearNet-Edes model pretrained by Multiview Contrast. Specif- ically, utilize the pretrained model to extract the embeddings of all the proteins in AlphaFold Database vl, and these embeddings are mapped to the two-dimensional space by UMAP (McInnes) al. 2018) for visualization. Following Akdel al. (202)). highlight the 20 most common super-
26

---

Published as a conference paper at ICLR 2023
INTR-A Binding)
1B85-A (Heme Binding)
Normalized Attributton
Figure 7: Identification of active sites on proteins responsible for binding based on attribution scores. Two proteins binding to specific targets are selected for illustration (INYR-A for ATP binding and 1B85-A for Heme binding). For these two complexes, ligands are shown in yellow spheres while the residues of the receptors are colored based on attribution scores. Residues with higher attribution scores are colored in red while those with lower scores are colored in blue.
families within the database by different colors. The visualization results shown in Fig. It can be observed that our pretrained model tends to group the proteins from the same superfamily together and divide the ones from different superfamilies apart. In particular, it succeeds in clearly separating three superfamilies, i.e., Protein kinase superfamily. Cytochrome P450 family and TRAFAC class myosin-kinesin ATPase superfamily. Such decent capability of discriminating protein superfamilies, to some degree, interprets our model's 's superior performance on Fold Classification.
K RESIDUE-LEVEL EXPLANATION
Protein functions are often reflected by specific regions on the 3D protein structures. For example, the binding ability of protein ligand is highly related the binding interface between them. Hence, to better interpret our prediction, we apply Integrated Gradients (IG) (Sundararajan et al. 2017). a model-agnostic attribution method, on our model to obtain residue-level interpretation. Specifically, we first select two molecular functions, ATP binding (GO:0005524) and Heme binding (GO:0020037), from GO terms that are related ligand binding. For each functional term, we pick one protein and feed it into the best model trained on the GO-MF dataset. Then, we use IG to generate the feature attribution scores for each protein. The method will integrate the gradient along straight-line path between baseline input and the original input. Here the original input and baseline input are the node feature f and a zero vector, respectively. The final attribution score for each protein will be obtained by summing over the feature dimension. The normalized score distribution over all residues are visualized in Figure 7 As can be seen, our model is able to identify the active sites around the ligand, which are likely be responsible for binding. Note that these attributions are directly generated from our model without any supervision, which suggests the decent interpretability of our model.
27


Reviews:
[{"review_id": "jOA2uyQJ74", "reviewer": "Reviewer_swLM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors propose a 3D structure-aware pre-training method for protein representation learning. Two novel structure-based protein encoders, GearNet and GearNet-Edge, are proposed to derive per-residue and whole-protein representations. The multi-view contrastive loss is adopted to maximize the similarity between different sub-structures of the same protein, while minimizing those from different proteins. Empirical evaluation is conducted on four down-stream tasks to verify the effectiveness of the proposed protein encoder and pre-training learning mechanism.", "summary_of_the_review": "The overall quality is satisfying, but there are still a few unresolved concerns (as listed in the “Cons” section) that may need further discussion and clarification.", "strength_and_weaknesses": "Pros:\n1. The construction of protein graph with multiple edge types, each associated with a GCN kernel matrix is novel. Additionally, as shown in Table 3, there is a significant performance boost after introducing edge-type-specific convolution (GearNet-Edge vs. w/o relational convolution).\n2. Different from commonly used self-supervised learning objectives that randomly masking out certain residues and/or coordinates, the proposed multi-view contrastive learning looks interesting and is reasonable to some extent.\n3. The empirical evaluation is extensive, which includes four down-stream tasks and various network architectures and pre-trained models. The ablation study also reflects the importance of some key components of the proposed method.\n\nCons:\n1. The proposed protein encoder only considers C-alpha atoms, which may be insufficient for certain down-stream tasks that require more fine-grained formulation of protein structures. For instance, to predict the binding affinity between protein and ligands, protein side-chain conformation should be considered.\n2. The ablation study on the necessity of relation convolution in Table 3 is not entirely convincing. By replacing edge-type-specific convolutional kernels with a single shared one, the number of learnable model parameters is greatly reduced, which may limit the model capacity. How about a deeper network with shared convolutional kernel?\n3. The proposed protein encoder is invariant to arbitrary translations, rotations, and reflections. I am not sure whether reflection-invariant is indeed necessary for protein representations since native L-amino-acid and unnatural D-amino-acid (only differ in the handedness) may have different physicochemical and biological properties.\n4. For self-prediction baselines, the constructed protein graphs may have permanently lost detailed 3D coordinate information in the input node features, which may limit the self-prediction learning performance. How about those GNN models that explicitly consider node-wise 3D coordinates, e.g., EGNN?\n5. In Table 3, it seems that “subspace + random edge masking” works better, even outperforming “GearNet-Edge (Multiview Contrast)”. So, why not directly using this for sampling sub-structures, rather than alternating between different sampling schemes?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EUI0_BDjFt7", "reviewer": "Reviewer_86an", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "While a large part of the literature focuses on learning protein representation from their amino acid sequences (allowing pre-training from huge existing database of known protein sequences), the authors suggest a novel approach to learn representation from their 3D structure.\nThe first contribution lies in the definition of a novel encoder called GearNet, based on a relational Graph Neural Network.\nThe protein at hand is initially represented as a graph, where each node (i.e. the alpha carbon of a residue) is characterized by its 3D position in its observed or predicted structure.\nThe edges are added based on closeness in the sequence or in an Euclidean sense (nodes within a certain radius and K-nearest neighbor nodes), if they are not close in a sequence sense.\nTo obtain the protein representation, the aforementioned graph is passed through a relational GNN.\nThe authors suggest a second contribution to explicitly model interactions between edges of the graph: they design an edge message passing layer, coined GearNet-Edge.\nIt is based on a relational graph where the nodes are the edges of the aforementioned graph and the edges in this new graph represent the (discretized) angles between the adjacent edges in the previous graph.\nA straightforward message passing layer is defined over that new graph and the aggregation function in GearNet-Edge combines the aggregation functions defined over both graphs.\nAs another contribution, the authors suggest to pre-train the resulting encoders on large collections of unlabeled protein structures.\nTo do so, they suggest a procedure inspired from SimCLR where different, random, views of a protein graph are created by cropping their full graph and randomly masking some of the edges.\nPairs of views are generated and labeled as positive if they originate form the same initial protein.\nThey also suggest more straightforward baselines to pre-train their encoders based on self-prediction where the encoders are used to perform masked predictions (residue type, distances etc.) on single, pair, triplets or quadruplets of residues.\nFinally, the authors validate the approach by comparing their (pre-trained or not) method with different baselines and sequence and structure-based approaches from the literature, on 4 different downstream prediction tasks: enzyme commission number, gene ontology term, fold classification, reaction classification.\nThey also perform an ablation study to validate the different contributions more individually.", "summary_of_the_review": "Overall, the authors suggest a novel method to learn efficient representations of proteins, leveraging their 3D structure.\nFollowing the aftermath of the recent release of AlphaFold2 and their prediction of the structure of a huge amount of new data, this contribution seems well-timed and seem to result in good performance compared to other very recent methods.\nI hope that this review process will offer the authors the opportunity to especially improve the clarity of the current manuscript.\nThat being said, I believe that time is of the essence as I suspect that a handful of related methods are currently being developed and published.\nGiven the general quality, I recommend that this paper be accepted.", "strength_and_weaknesses": "Strength:\n- Protein modeling / representation is indeed a very important topic that has a high impact both inside and outside biomedical applications. By tackling this problem in a relatively generic way (the representations are not tied to a specific downstream task), this paper definitely has a very high potential impact.\n- The suggested approach seems to be well \"rooted\" in the current trends in the field. Recent breakthroughs offered by AlphaFold2 to (mostly) accurately predict protein structure definitely offer new opportunities for designing methods that leverage 3D structure information.\n- To my knowledge, the related work section is well documented and it is relatively clear where the authors' contributions are.\n- The experimental section of the paper is quite elaborate with GearNet* being evaluated on various tasks, datasets and against a relatively large number of competing methods and baselines.\n- The amount of work seems substantial, to the point where the authors provide 10 pages of supplementary material with a lot of interesting comments and results.\n\nWeaknesses:\n- While the idea of having a better general protein representation (as opposed to one tailored for a specific downstream task) is of course seducing, it makes for a much stronger claim that is hard to completely substantiate. The results seem good on various downstream tasks but it is of course not clear how that would carry over other relevant tasks (for instance anything pertaining to predicting protein-protein interactions).\n- While the paper reads well overall, I find that some parts lack clarity. In general, many choices of modeling could be better justified and/or offer more insights.\n- Notably, from reading the \"Protein graph construction\" subsection, I was confused about which type of features where used for the nodes. Only the spatial coordinates are mentioned, while not actually used and I had to go deep into the supplements to really understand that the type of residue, e.g. the actual sequence information was directly used.\n- I find Section 3.2 to also lack clarity. Can the authors offer a bit more insight on what kind of information the angles, that determine the edge types, actually give? Can you describe more clearly how it's different (and why a better choice) from the representation used in AlphaFold2?\n- Why are those angles (in section 3.2, but also in 4.2) discretized? Would it not be a more straightforward option to keep the angles as float values (and perform regression in 4.2)?\n- Despite some efforts, I do not really understand the assumption underlying the contrastive learning approach. Especially for proteins with longer and less redundant sequences, why should different views, describing potentially radically different regions of the protein, be more related to each other than views taken from different proteins?\n- The construction of the views, mixing subspace and subsequence sampling, along with the addition of masking or not, seems a bit convoluted to me. From reading the manuscript, I don't get a strong intuition why this mixture is meaningful and the results reported in Table 3 seem to indicate that adding the subsequence sampling and identity transformation does not yield better performance.\n- I am not sure I really like the argument that the proposed method pre-trains with much less data than sequence-based approaches. While the number of proteins used for pre-training is indeed much lower, much more data per protein is being used when incorporating 3D structure.\n- Related to the previous point, it would be nice to see some runtime and memory usage analysis of the different methods used.\n- Although it is qualitative, and not entirely compelling because the analysis has been performed only for the author's model, and it could be that they are hand picked, I find the results presented in Appendix H to be very interesting. It would be nice to at least put a reference to it in the main text.\n\nMore open question:\nThe current model includes mostly structural information about the backbone of the protein, which seems to be sufficient for the chosen downstream tasks. For tasks like complex interaction prediction, that probably will be a bottleneck.\nDo the authors have suggestion as to how to extend the current model to include side chain information?\n\nMinor:\n- Enzyme Commission acronym EC is used in section 4.1 but only defined later in 5.1.\n- There's a typo just before 4.2 \"subsequecne\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6saYRuB9i3", "reviewer": "Reviewer_roE6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a simple SE(3) invariant model and pretrains it on large dataset of alphafold predicted structures. This is then finetuned on a handful of downstream tasks, where it achieves better results than some prior methods.", "summary_of_the_review": "The paper is technically sound and is likely an improvement on the prior methods they benchmark against (e.g. Hermosilla et al). However, the paper does not add major novel insights, and does not have strong results on actually interesting downstream tasks.\n\nI recommend the authors look into areas where structure-based modeling is actually used in practice and seeing how their model may be applied.\n\n*********** After Author Response ***********\n\nThe authors have performed significant additional work, including adding the baseline I requested (kNN-1 search via Foldseek) and a benchmark on structure search as a task. The results on this are very promising since fast structure-based search is actually a current pressing issue in the field. Combining accurate cosine-similarity based search with tools like FAISS is potentially a highly valuable tool for the community, and something I would encourage the authors to pursue.\n\nOverall, I now believe this paper is a good candidate for acceptance at ICLR and have revised my score.", "strength_and_weaknesses": "Strengths:\n* The paper is technically sound, and the method presented is relatively simple and scalable, so could be extended to training on larger datasets relatively easily.\n* The paper is well written and easy to follow + understand\n* Baselines chosen are reasonable and convincing that this method performs well on the chosen downstream tasks.\n\nWeaknesses:\n\n1) The downstream evaluations chosen are just not that convincing to me. This set of tasks has become somewhat standard among a subset of the geometric modeling community, but these tasks are fairly contrived, and the results are not so impressive.\n    - 3D structure-based models are *provided the structure as input*. This is a huge amount of information relative to sequence-based models (as the authors state in the paper). So I am simply confused as to why they perform only slightly better than sequence based models such as ESM-1b. I would expect that, given the structure, it would be very simple to outperform models such as ESM-1b for many function-prediction based tasks, especially since ESM-1b likely contains less information than a multiple sequence alignment.\n    - Before breaking out a neural network predictor, the approach I would take to classify functions based on structure would be to use a structure search tool, such as foldseek. A simple baseline therefore would be a 1-NN algorithm: take the test set structure, use foldseek to search the training set, and return the label of the structure with best similarity. Is the approach proposed here better than that baseline?\n    - The one task where structure-based methods show a clear advantage is the fold classification task. However, the construction of this task deliberately designed to advantage structure-based models over sequence-based models. The dataset splits were constructed such that the goal was to match inputs with very different sequences, but similar structures (i.e. remote homology detection). When the structure is provided, however, it is not clear why this task is interesting. I would expect the baseline I suggested above to perform very well.\n\n2) There are two application areas to my knowledge where structure-based modeling has shown a great deal of promise. The first is inverse folding (e.g. Dauparas et al. 2022, https://www.science.org/doi/10.1126/science.add2187), which has enabled breakthroughs in de novo protein design. The second is structure-based search (van Kempen et al. 2022, https://www.biorxiv.org/content/10.1101/2022.02.07.479398v4) which uses a small neural network to speed up structural searches, an important task given the development of large scale protein structure databases. \n\nThe second task (structure-based search) seems like a very promising application for the model, and would be much more convincing than those shown in the paper. Given the contrastive nature of the training method, structural similarity search seems like a natural application (although care would have to be taken to ensure generalization to novel structures outside the method's training set).\n\n3) The inability to model side chains seems like a potential drawback for structure-based methods. Especially in regards to function / binding affinity prediction, this may hinder the ability to model the output better than sequence-based models can.\n\nMinor Comment: How much does the use predicted structures affect the model's capabilities? What is the relationship e.g. between AlphaFold pLDDT and model accuracy if predicted structures are used?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1jeFfY0DmA", "reviewer": "Reviewer_pcxa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a method to leverage unlabeled protein structural data to improve the prediction of various protein properties.  The authors present two main contributions: (1) a geometric encoder for protein structures, and (2) a multiview contrastive learning pretraining strategy, which together lead to state-of-the-art performance on several protein property prediction tasks.  The geometric encoder makes use of edge messaging passing, which the authors claim had not been used for macromolecular representation learning.  The use of different kinds of edges (sequential, radius, KNN) is also claimed to be novel for this task.", "summary_of_the_review": "I recommend weak acceptance.  In my opinion, the empirical results and overall clarity of the paper outweigh the lack of algorithmic novelty, though the lack of code prevents better assessing reproducibility.", "strength_and_weaknesses": "The empirical results are compelling, demonstrating that the new wealth of structures can be successfully used to improve sequence-based models.  The proposed architecture is conceptually straightforward and does not include any particularly novel algorithmic contributions, though several components have not been used in the macromolecular representation learning area.  The ablation studies are a very nice addition as well.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster