Multi-Scale Representation Learning on Proteins
Authors: Vignesh Ram Somnath
Charlotte Bunne
Andreas Krause
Venue: NeurIPS 2021 Poster (2021)
Keywords: Proteins
Representation Learning
Graph-Neural Networks

Abstract: Proteins are fundamental biological entities mediating key roles in cellular function and disease. This paper introduces a multi-scale graph construction of a protein –HoloProt– connecting surface to structure and sequence. The surface captures coarser details of the protein, while sequence as primary component and structure –comprising secondary and tertiary components– capture finer details. Our graph encoder then learns a multi-scale representation by allowing each level to integrate the encoding from level(s) below with the graph at that level. We test the learned representation on different tasks, (i.) ligand binding affinity (regression), and (ii.) protein function prediction (classification).
On the regression task, contrary to previous methods, our model performs consistently and reliably across different dataset splits, outperforming all baselines on most splits. On the classification task, it achieves a performance close to the top-performing model while using 10x fewer parameters. To improve the memory efficiency of our construction, we segment the multiplex protein surface manifold into molecular superpixels and substitute the surface with these superpixels at little to no performance loss.

Multi-Scale Representation Learning on Proteins
Vignesh Ram Somnath' Dept. Computer Science
Charlotte Bunne"
Andreas Krause
Dept. of Computer Science ETH Zurich bunnec@ethz.co
Dept. of Computer Science ETH Zurich
ETH Zurich vsommath@ethz.co
krauseaßethz.c ch
Abstract
Proteins are fundamental biological entities mediating key roles in cellular function and disease. This paper introduces multi-scale graph construction of protein HOLOPROT - connecting surface to structure and sequence. The surface captures coarser details of the protein, while sequence as primary component and structure comprising secondary and tertiary components capture finer details. Our graph encoder then learns multi-scale representation by allowing each level to integrate the encoding from level(s) below with the graph at that level. We test the learned representation different tasks. (i.) ligand binding affinity (regression), and (ii.) protein function prediction (classification). On the regression task, contrary to previous methods, our model performs consistently and reliably across different dataset splits, outperforming all baselines on most splits. On the classification task, it achieves performance close the po-ppeformmng model while using 10x fewer parameters. To improve the memory efficiency of our construction, we segment the multiplex protein surface manifold into molecular superpixels and substitute the surface with these superpixels at little to no performance loss.
Introduction
Protein design and engineering has become a crucial component of pharmaceutical research and development, finding application in wide variety of diagnostic and industrial settings. Besides understanding the design principles determining structure and function of proteins, current efforts seek further enhance or discover proteins with properties useful for technological or therapeutic applications. To efficiently guide the search in the vast design space of functional proteins, need to be able to robustly predict properties of a candidate protein [Yang et al., 2019]. Moreover, understanding role and function of proteins is crucial to study causes and mechanism of human disease [Fessenden, 2017].
To achieve this, representations incorporating the complex nature of proteins are required. Proteins consist of amino acids, organic molecules linked by peptide bonds forming linear sequence. Each of the twenty amino acids carries a unique side chain, giving rise to incomprehensibly large combinatorial space possible protein sequences. The primary sequence drives the folding of polymers spontaneous process guided by hydrophobic interactions, formation of intramolecular hydrogen bonds, and van der Waals forces into unique three-dimensional structure. The resulting shape and surface manifold with rich physiochemical properties carry essential information for understanding function and potential molecular interactions.
Previous methods typically only consider an individual subset within these scales, focusing on either sequence (Öztürk al., 2018, Hou al., 2018], three- dire-diimnsional structure [Hermosilla et al., 2021, Derevyanko et al., 2018] or surface [Gainza et al., 2020]. Two proteins with similar sequences can fold into entirely different conformations. While these proteins might catalyze the same type of
Equul contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(Section 3.1.1)
surface graph 9s Es)
structure graph (Section 3.1.2) Ser = (Va. Ea)
surfac
nolecular superpixel (Section 4)
protein ligand binding affinity
secondary sertiary
(Section 5.1)
peinary edge
cuzyme-cutalyzed reaction classificution
untenctiion between structure and sariace
(Section 5.2)
Figure 1: Overview of HOLOPROT Our multi-scale protein representation algorithm integrates primary, secondary and tertiary elements of protein structures and connects them the surface. We extract higher-level protein motifs by introducing molecular superpixels. Both structure and surface represented graphs G8 and Gs, respectively. The method is evaluated on two representative tasks, protein-ligand binding affinity and enzyme-catalyzed reaction classification.
reactions, their behavior to specific inhibiting drugs might be divergent. Interaction between proteins and ligands, on the other hand, is controlled by molecular surface contacts [Gainza et al., 2020]. Molecular surfaces, determined by subjacent amino acids, are fingerprinted with patterns of geometric and chemical properties, and thus their integration in protein representations is crucial.
In this work, we present a novel multi-scale graph representation which integrates and connects the complex nature of proteins across all levels of information. HOLOPROT consists of a surface and structure layer (both represented as graphs) with explicit edges between the layers. Our construction is guided by the intuition that propagating information from surface to structure would allow each residue to learn encodings reflective of not just its immediate residue neighborhood, but also the higher-level geometric and chemical properties that arise from interactions between residue and its neighborhood. The associated multi-scale encoder then learns representations by integrating the encoding from the layer below, with the graph at that layer (Section 3). Such multi-scale representations have been previously used in molecular graph generation [Jin et al., 2020] with impressive results.
We further improve the memory efficiency of our construction by segmenting the large and rich protein surface into molecular "superpixels" ", summarizing higher-level fingerprint features and motifs of proteins. Substituting the surface layer with these superpixels results in little no performance degradation across the evaluated tasks. The concept of molecular superpixels might be of interest beyond our model (Section 4).
The multi-objective and multi-task nature of protein engineering poses challenge for current methods, often designed and evaluated only on specific subtasks of protein design. By incorporating the biology of proteins, strong representations exhibit robust performance across tasks. We demonstrate our model's versatility and range of applications by deploying it tasks of rather distinct nature, including a regression task, e.g., inference of protein ligand binding affinity, and classification tasks, i.c. manzyee-cttllyan reaction classification (Section 5).
2 Related Work
Protein Representation Learning With increasing availability of sequence and structure data, the field of protein representation learning has advanced rapidly, with methods falling largely in one of the following categories:
Sequence-base based methods. One-dimensiona amino acid sequences continue to be the simplest, most abundant source of protein data and various methods have been developed that borrow architectures developed in natural language processing (NLP). One- -ie-dimensional convolutional neural networks have been used to classify a protein sequence into folds and enzyme function [Hou et al., 2018, Dalkiran et al., 2018), and to predict their binding affinity to ligands [Özüürk al., 2018]. Furthermore, methods have applied complex NI.P models trained unsupervised on millions of unlabeled protein sequences and fine-tuned them different downstream tasks [Rao

---

et al., 2019, Elnaggar et al., 2020, Bepler and Berger, 2019]. Despite being advantageous when only the sequence is available, these methods ignore the full spatial complexity of proteins.
Structure-bused methods. To learn beyond sequences, approaches have been developed, that consider the 3D structure of proteins. 3D convolutional neural networks have been utilized for protein quality assessment [Derevyanko et al., 2018]. protein contact prediction [Townshend et al., 2019] and pootein-ligand binding affinity tasks [Ragoza et al., 2017, Jiménez et al., 2018, Townshend et al., 2020]. An alternate representation treats proteins as graphs, applying graph neural networks for enzyme classification [Dobson and Doig. 2005], interface prediction [Fout et al., 2017], and protein structure quality prediction [Baldassarre et al., 2021]. Gligorijevic et al. [2021] use a long short term memory cell (LSTM) to encode the sequence, followed by graph convolutional network (GCN) [Kipf and Welling, 2017] to capture the tertiary structure, and apply this to the function prediction task. Hermosilla et al. [2021] propose convolutional operator that learns to adapt filters based on the primary, secondary, and tertiary structure of a protein, showing strong performance on reaction and fold class prediction.
Surface-based methods. Taking a different viewpoint, Gainza et al. [2020] hypothesize that the protein surface displays patterns of chemical and geometric features that fingerprint protein's 's interaction with other biomolecules. They utilize geodesic convolutions, which are extensions of convolutions surfaces, and learn fingerprint vectors, showing improved performance across binding pocket and protein interface prediction tasks.
Protein Motif Detection Protein motifs have largely been synonymous with common and con- served patterns in a protein's sequence or structure influencing protein function, e.g., the helix- turn-helix motif binds DNA. Understanding these fragments is essential for 3D structure prediction, modeling, and drug design. While reliably detecting evolutionary motifs, existing tools [Golovin and Henrick, 2008] do not provide a full segmentation of the protein surface manifold. Our work takes a different viewpoint, by looking at protein motifs from the context of a protein surface. Previous meth- ods developed in this context either only consider geometric information rather than physiological properties [Cantoni et al., 2010], are computationally expensive [Cantoni et al., 2011], or designed for particular downstream tasks [Stepniewska Srepriewsska-zzibbinskcc
al., 2020]. Our molecular superpixel approach provides lask-independent segmentation utilizing both geometric and chemical features, while also being computationally efficient.
3 Multi-Scale Protein Representation
In this section, we describe our multi-scale graph construction and the associated encoder. Figure 1 illustrates the main principles of HOLOPROT. We represent protein P graph Gp with two layers capturing different scales:
(i.) Surface layer. This layer captures the coarser representation details of a protein. The protein surface is generated using the triangulation software MSMS [Connolly, 1983, Sanner et al., 1996]. We represent this layer vector fug denoting its charge, hydrophobicity and local curvature [Gainza et al., 2020]. Two surface nodes (us, us) have an edge if they are part of triangulation. Each surface node additionally has a residue identifier r, indicating the amino acid residue corresponds to. Multiple surface nodes can have the same residue identifier.
graph Gs. where each surface node us has feature
(ii.) Structure layer. This layer captures the finer representation details of protein. A protein typically has four structural levels: (i.) primary structure (sequence), (ii.) secondary structure (o-helices and f-sheets), (ii..) tertiary structure (3D structure) and (iv.) quaternary structure (complexes) [Fout et al., 2017]. We represent this layer as a graph Gg. where each node us corresponds to residue r. Two nodes (ug, UB) have an edge in GB if the C., atoms of the two nodes occur within a certain distance of each other. Distance based thresholding ensures that different structural levels are implicitly captured in the neighborhood of node ug.
We further introduce edges from the surface layer to the structure layer in order to propagate information between them. Specifically, we introduce a directed edge between surface node us and backbone node ug if they both have the same residue identifier r. Typically, we have between 20-40 surface nodes {us} that map to the same structure node ug. This gives the multi-scale
3

---

graph which is then encoded by our multi-scale message passing network. Details on the features used for both the structure and surface layer can be found Appendix ??.
3.1 Multi-Scale Encoder
Our multi-scale message passing network uses one message passing neural network (MPN) for each layer in the multi-scale graph [Lei al., 2017, Gilmer et al., 2017]. This allows to learn structured representations of each scale, which can then be tied together through connections between the scales. Before detailing the remainder of the architecture, For simplicity, we denote the MPN encoding process as MPNa(.) with parameters O. We denote MLPp((, (x( y) for multi-layer perceptron (MIP) with parameters 0, whose input is the concatenation of x and y. and MLPa(x) when the input is only x. We also denote the residue identifier of a node with id and the neighbors of a node N (u). The details of the MPN architecture listed in the Appendix ??.
introduce some notational preliminaries.
3.1.1 Surface Message Passing Network
We first encode the surface layer 9s of the multi-scale protein graph Gp. The inputs the MPN are node features fug and edge features fusus of 9s. For more details on the input features used for surface nodes and edges, refer Appendix ??. The MPN (with parameters 0s) propagates messages between the nodes for K iterations, and outputs representation lug for each surface node us.
[hug} } = MPNN, (Gs, (fug {Fuses losEN(uss)
3.1.2 Structure Message Passing Network
For each node ug in the structure layer Gs. we first prepare the input to the MPN (with parameters 0s) by using an MLP (with parameters 0) on the concatenated version of its initial features fug and the mean of the surface node vectors with the same residue identifier S = (h.g (i((us) = id(us)}
Xus MLPa Es hus/1st).
Given the edge features fugug. we then K iterations of message passing, compute the represen- tations bug for each structure node ws.
[Lug] MPNg, (9B, {Xug) }, (fueve (ug))
The graph representation cyp an aggregation of structure node representations,
CGp
Σ
hug
MB MBE98
3.2
Task Specific Training
This multi-scale encoding allows to learn structured representation of a protein tying different scales together, which can then be utilized for any downstream task. In this work, we evaluate our method two rather distinct tasks (i.) protein-ligand binding affinity regression, and (ii.) enzyme- catalyzed reaction classification. The architectural details for both downstream tasks are described below. These modules can be adapted and modified in order to utilize HOLOPROT for other use cases.
3.2.1 Protein-Liigand Binding Affinity
Protein-ligand binding affinity prediction depends on the interaction of a protein, encoded using the HoLoPROT framework, and a corresponding ligand, in most cases small molecules. To encode the ligand represented as a graph Gz, we use another MPN (with parameters 0z) and aggregate its node representations to obtain graph representation cg.- We concatenate the graph representations Cop (Equation I) of the protein and cg. of the ligand, and use that as input to MLP (with parameters Φ) obtain predictions,
MLP (Cgg Cg2)
(2)
The model is trained by minimizing the mean squared error.

---

a.
=
molecular molecularsspprpixxlls
hydropathy
shape index
free electrons
Figure 2: Molecular Superpixels and Surface Features of the HIV-1 Protease (PDB ID: 2AVQ). Molecular superpixels, indicated by different colors (A: 20), and the corresponding surface features, i.e., b. hydropathy, e. shape index, and free electrons. As highlighted, molecular superpixels are spatially compact and overlap with surface regions dominated by single features such as hydrophobic patches while capturing coherent areas across all surface features. The protein complex contains 198 residues.
3.2.2 Enzyme-Catalyzed Reaction Classification
To predict the enzyme-catalyzed reaction class, we use the graph representation cyp of the protein obtained via HoLoPRoT as the input to MLP (with parameters () to obtain the prediction logits,
Pk MLP (cg)
The model is trained by minimizing the cross-entropy loss.
4 Superpixels Molecular Surfaces
Protein surface manifolds are complex and represented via large meshes. In order to improve the computational and memory efficiency of construction, introduce the notion of molecular superpixels. Originally developed in computer vision [Ren and Malik, 2003, Mori et al., 2004, Kohli al., 2009], superpixels are defined perceptually uniform regions in the image. In the molecular context, we refer to superpixels as segments on the protein surface capturing higher-level fingerprint features and protein motifs such hydrophobic binding sites.
In order to apply the segmentation principle to dinendimmnsional molecular surfaces, we employ graph-based superpixel algorithms triangulated surface meshes. The superpixel representation of the protein surface needs to satisfy several requirements, as (i.) molecular superpixels should not reduce the overall achievable performance of HOLOPROT, and (ii.) molecular superpixels need to form geometrically compact clusters, and overlap with surface regions that are coherent in physiological surface properties, e.g., capture hydrophobic binding sides highly charged areas. Popular graph-based segmentation tools such as Felzenszwalb and Huttenlocher [2004, FH], mean shift [Comanieiu and Meer, 2002], and watershed [Vincent and Soille, 1991], however, produce non-compact superpixels of irregular sizes and shapes. By posing the segmentation task as a maximization problem
graph maximizing over (i.) the entropy rate of the random walk the surface graph Gs (Vs, Es) favoring the formation of compact and homogeneous clusters, and (i.) a balancing term encouraging clusters with similar sizes, the entropy rate superpixel (ERS) segmentation algorithm [Liu et al., 2011 I] outperforms previous methods across different tasks [Stutz al., 2018] and achieves the desired properties of molecular superpixels.
In order to incorporate geometric and chemical features of the surface Fs. we extend the surface graph Gs (Vs. Es) with non non-negative similarity measure w, given Wij Eters [f,, fe, for nodes v and by if connected by edge Gij. We simulate random walk X {X. E T, E Vs) on a protein surface mesh, where the transition probability between two nodes Y; and is defined as Pty P(X Uy XX, v4) wss/un ws. where wi E EES Wik. stationary distributions of nodes Vs are given by
W.s.The The corresponding
4 = (2. (IDI))
-=
W1
W2
W
WT
WT
WT
5

---

Molecular superpixels are then defined by subset of edges M S Es such that the resulting graph, Is (Vs. MM) contains exactly & connected subgraphs. Computing molecular superpixels is achieved via optimizing the objective function with respect to the edge set M
max- M
Σ 14 L Pis(M) log (Pis(M))
Pzm (i) log (pz (i) RM
(i. entropy rate
(ii.) balancing function
s.t. M and n.M 2 k,
where n.M is the number of connected components in the graph, PMM denotes the distribution of cluster memberships ZM. and 2 O is the weight of the balancing term. Both terms satisfy monotonicity and submodularity and can thus be efficiently optimized based techniques from submodular optimization [Nemhauser et al., 1978J. For further details on the entropy rate superpixel algorithm, see Liu et al. [2011].
molecular superpixel m comprising & surface vertices is then given as fm = -fe, (fe....... fog) for all f E Fs. We summarize the feature representation of each molecular superpixel via the graph 9M (VM,EM) where each node m E VM represented via (mean(fm), std(f,,) max(f,.), min(f,.) for all f E Fs and an edge E EM via the Wasserstein distance between neighboring superpixels.
Figure 2 demonstrates molecular superpixels for the enzyme HIV-I protease [Brik and Wong, 2003]. Besides being spatially compact, superpixels overlap with surface regions dominated by single features such as hydrophobic patches, while capturing coherent areas across all surface features. Further examples of superpixels are displayed in Appendix ??.
5 Evaluation
Successful protein engineering requires optimization of multiple objectives. When searching for protein with desired functionality, auxiliary but crucial properties such as stability measured in terms of free energy folding also need to be satisfied. Furthermore, the field also subject to plethora of potential tasks and applications. In order to capture the multi-objective and multi-task nature of protein enginecring, we evaluate our method on two representative tasks: regression the binding affinity between proteins and their ligands, and classification of enzyme proteins based the type of reaction they catalyze.
5.1 Protein-Ligand Binding Affinity Prediction
Studying the interaction between proteins and small molecules is crucial for many downstream tasks, e.g.. accelerating virtual screening for potential candidates in drug discovery or protein design to improve the output of an enzyme-catalyzed reaction. The architecture of the regression module is described in Equation 2.
Dataset. The PDBBIND database (version 2019) [Liu et al., 2017] is a collection of the experimen- tally measured binding affinity data for all types of biomolecular complexes deposited in the Protein Data Bank [Berman et al., 2000]. After quality filtering for resolution and surface construction, the refined subset comprises total of 4, 709 biomolecular complexes. The binding affinity provided in PDBBIND is experimentally determined and expressed molar units of the inhibition constant (K,) or dissociation constant (Ka). Similar previous methods [Öztürk et al., 2018, Townshend et al., 2020], we do not distinguish both constants and predict negative log- reg-transformed binding affinity pKa/pK,, We split the dataset into training, test and validation splits based on the scaffolds of the corresponding ligands (scaffold), or a 30% and a 60% sequence identity threshold (identity 30%, identity 60%) to limit homologous ligands or proteins appearing in both train and test sets.
Baselines. For evaluating the overall performance on the regression task, we compare HOLOPROT against several baselines including current state-of-ther art methods on both tasks. This comprises sequence- based methods [Öztürk et al., 2018, Rao et al., 2019, Bepler and Berger, 2019, Elnaggar et al., 2020] well as methods based on the three-dimensional structure of proteins [TTwnsheend et al., 2020, Hermosilla et al., 2021], and recent methods using geometric deep learning on protein molecular surfaces [Gainza al., 2020].

---

Table Protein-Ligand Binding Affinity Prediction Results Comparison predictive performance of ligand binding affinity using the PDBbind dataset [Liu et al., 2017] of HOLOPROT against other methods. Results are reported for 3 experimental runs.
Model
Params
Sequence Identity %)
Sequence Identity %)
RMSE
Pearson
Spearmin
RMSE
Peurson
Spearman
Sequence- Methods
Oztüikk [2018]
1.9
1.86 = 0.080
0.472
0.022
0:471 + 00.024
1,702 + 0.261
0.666
0.012
0.663 = 0.015
Repler and Berger [2019]
48.8 M
1.985
0.006
0.165
0.006
0.152 + -.0024
1.891 + 0.004
0.249
0.006
0.275 + 0.008
Rao al. [2019]
93.0 M
1,890
1 0.035
0.338
0.044
0.286
1. 0.124
1,633 1 0.016
0.568
0.033
0.571 1 0.021
Elnaggar et al. [2020]
2.4M²
1.544
0.0155
0.438
0.053
0.434 + 0.058
1.641 + 0.016
0.595
0.014
0.588 + 0.009
Surface-based Methods
Gainza et al. [2020]
0.62
1.484 = 0.018
0.467
0.020
0.455 = 0.014
1,426 4: 0.017
0.709 i 0.008
0.701 = 0.011
Structure-bssed Methods
Townshend et al. 12020)
1.429 = 0.042
0.541
0.029
0.532 + 0.033
1,450 4: 0.024
0.716 i 0.008
0.714 = 0.009
Townshend [2020)
1,936 1 0.120
0,581
0,039
0.647 1. 0,071
1,493 1 0.010
0.669 0.013
0.691 1 0,010
Hermosilla et al. [2021]
5.80
1.554 = 0,016
0.414
0.053
0,428 = 0.032
1.473 +: 0.024
0.667 : 0.011
0.675 = 0.019
HOLOPRII
M 1,76 M
1.464
0.006
0.509
0.002
0.500 + 0.005
1.365 0.038
0.749
0.742 + 0.011
HOLOPROT (+)
1.491
1: 0.004
0.491 0.014
0.482
0.017
1.416 1 0.022
0.724 0.011
0.715 1. 0.006
Model
Params
Scaffold
RMSE
Peasson
Spearman
Sequence-hased Methods
Oztürk et al [2018]
1.93 M
1.908
0.145
0.384 1 0.014
0.387 0.016
Bepler and Berger [2019]
48.8 M
18864
0.009
0.269
+ 0.002
0,285 + 0.019
al. [2019]
9
1.680
0.055
0.487
+ 0.029
0.462 0.051
Elaaggar et al. [2020)
2.4M
1,592
0.009
0.398 + 0.027
0,409 0.029
Surface-based Methods
Gainza et al. (2020)
0.62 M
1,583
0.132
0.416 1. 0.111
0.412 0.126
Structure- Methods
Hermosilla et al. [2021]
5.80 M
1,592 = 0.012
0.365 + 0.024
0.373 + 0.019
HotLoPRoT (a)
1.44 M
1.523
0.028
0.489
+ 0.019
0.491 0.020
HoLOPROT
1.28 M
1,516
0.014
0.491 1 0.016
0.493 0.014
full surface
molecular superpixels
Evaluation metrics. For evaluating different methods, we use three metrics - root mean squared error (RMSE), Pearson correlation coefficient, and Spearman correlation coefficient. We also include the mean and standard deviation across experimental runs.
Results. Table I displays the results on motein--liaane binding affinity. HOLOPROT (o. ) performs consistently well across different tasks and dataset splits, outperforming all methods on the splits scaffold and identity 60%. On identity 30%, our method outperforms most baselines, while having lower variability across the evaluated metries. HOLOPROT with molecular superpixels (+) performs similar to HOLOPROT on the entire surface, with no or little performance loss, suggesting that molecular superpixels capture meaningful biological motifs. We include the models from (TTownshend et al., 2020) for completeness, but note that these models were trained only using the protein binding pocket. Binding sites on proteins are often structurally highly conserved regions [Panjkovich and Daura, 2010]. Considering only binding pockets, which vary less between the train and test splits, provides an additional simplification making the task less challenging, All other baselines were tested on the full proteins.
5.2 Mazzye-Catalyaa Reaction Classification
Predicting the reaction class of enzymes without the use of sequence similarity allows for efficient screening of de novo proteins. i.e., macromolecules without evolutionary homologs, for catalytic properties [des Jardins al., 1997]. The architecture of the classification module is described in Equation 3).
'The embeddings obtained via Elnaggar et al. [2020] were saved to disk, instead of finetuning the entire pretrained model.
"Equivariant neural network (ENN) on binding pocket only.
Graph neural network (GNN) on binding pocket only.
7

---

Table 2: Enzyme- Catallyed Reaction Classification Results Comparison of classification accuracy of HOLOPROT against other methods.
Model
Parameters
Reaction Class
Accuracy
Sequence-based Methods
Hou et al. [2018]
41.7 M
70,9 %
Bepler and Berger [2019]
31.7 M
66.7 %
Rao et al. [2019] (Transformer)
38.4 M
69.8 %
Ellaggar al. [2020]
420.0 M
72.2 %
Structure-based Methods
Kipf Welling [2017]
1.0M
67.3 %
Derevyanko et al. [2018]
6.0 M
78.8 %
Hermosilla et al. [2021]
9.8 M
87.2 %
HOLOPROT
0.64 M
77.8 %
HOLOPROT
0.64 M
78.9 %
full surface
molccular superpixels
Dataset. Enzyme Commission (EC) numbers constitute an ontological system with the purpose of defining and organizing enzyme functions [Webb, 1992]. The four digits of an EC number are related in functional hierarchy, where the first level annotates the main enzymatic classes, while the next levels constitute subclasses, e.g. the EC number of the HIV-1 protease is 3.4.23.16 This task aims at predicting the enzyme- anzyme-catalyzed reaction class of protein based on according to all four levels of the EC number. We use the same dataset and splits provided by [Hermosilla et al., 2021], comprising 37, 428 proteins from 384 EC numbers, with 29, 215 instances for training, 562 instances for validation, and 5, 651 for testing. For more details on dataset construction, we refer to Hermosilla et al. [2021, Appendix CJ.
Baselines. For the classification task, we again compare HOLOPROT against several baselines including sequence-based methods [Hou et al., 2018], methods partially pretrained on millions of sequences [Rao et al., 2019, Bepler and Berger, 2019, Elnaggar et al., 2020] well as methods utilizing principles geometric deep learning [Kipf and Welling, 2017, Derevyanko et al., 2018, Henmosilla et al., 2021]. The values for different baselines taken from [Hermosilla et al., 2021].
Evaluation metric.
Model performance measured via the mean accuracy score.
Results. We report the results of enzyme-catalyzer reaction classification in Table 2. While our method (o, ) is unable to outperform the current state-of-the-e method [Hermosilla et al., 2021], achieve equivalent, if not better results to other methods at fraction of the parameters used. Molecular superpixels also capture biologically meaningful protein surface motifs, as evidenced by small increase in the overall classification performance.
5.3 Ablation Studies
To further evaluate the contribution of HOLOPROT to learning multi-scale protein representations, we conduct several ablation studies. First, we analyze if the performance of the multi-scale model outperforms its isolated components, i.c. when using only structure or surface representation for subsequent downstream tasks. The second ablation axis analyzes the construction of molecular superpixel representations. Besides computing summary features for each molecular superpixel as described in Section 4, we learn patch representations via MPN on the superpixel graph. The ablation study were conducted on both tasks, ligand binding affinity (Section 5.1) and enzyme catalytic function classification (Section 5.2).
As displayed in Table HOLOPROT with (+) and without molecular superpixels (0) improve over the performance of structure and surface representations. Further, the results of the ablation study clearly show that different protein scales are more relevant for particular downstream tasks, e.g., predicting the -attaleed reaction class from surface only results in poor performance. We further

---

Table 3: Ablation Studies Results Evaluation of architectural design choices of HOLOPROT by analyzing the performance of its individual components as well as feature summarization of molecular superpixels.
Ligand Binding Affinity
Enzyme Class
Model
Sequence Identity (30 %)
RMSE
Pearson
Spearman
Accuracy
Structure
1.476 + 0.027
0.51
0.029
0.503 0.027
74.2
Surface
1.482 + 0.015
0,512
0.022
0.505 0.017
28.6
HOLOPROT
1.464 = 0.006
0.509
0.002
0.500 0.005
77.8
HOLOPROT
1.491 = 0.004
0.491 = 0.014
0.482
0.017
78,9
HOLOPROT
1.491 = 0.027
0.503 = 0.005
0.492 0.004
75.7
full surface
molecular superpixels
molecular superpixel with MPN
improvement in applying a MPN within a molecular superpixel (m) over using summary features (+).
Further ablation studies presented in Appendix ??.
5.4 Limitations
Despite the reported success HOLOPROT, our method faces some limitations. First, HOLOPROT relies on existing protein structures and the corresponding generated surface manifolds. However, protein sequence data still remains the most abundant data source, and in protein design, confor- unknown. This limitation could however be partly remedied,
mations of mutated macromolecules (i.) by the recent advancements protein structure prediction [Senior et al., 2020, Jumper et al., 2021, AlphaFold] [Back al., 2021, [oAFll) and protein structure determination methods such as cryo-electron microscopy [Callaway, 2020], and (ii.) by utilizing homology modeling algorithms available wild type structures for mutant analysis [Schymkowitz al., 2005]. Second, our method requires precomputed surface meshes, resulting in an additional preprocessing step before deploying HOLOPROT the desired application. This bottleneck be bypassed by utilizing techniques developed in the concurrent work by Sverrisson et al. [2020], which allow computation and sampling of the molecular surface on-the-fly.
6 Conclusion
In this work, we present a novel multi-scale protein graph construction, HOLOPROT, which integrates finer and coarser representation details of protein by connecting sequence and structure with surface. We further establish molecular superpixels, which capture higher-level fingerprint motifs on the protein surface, improving the memory efficiency our construction without reducing the overall performance. We validate HOLOPROT's effectiveness and versatility through representative tasks on protein-ligand binding affinity and anzyme-catalyzer reaction class prediction. While being significantly more parameter-efficient, HOLOPROT performs consistently well across different tasks and dataset splits, partly outperforming current state-of-the-art methods. This will potentially of great benefit and advantage when working with datasets of reduced size, e.g., comprising experiments on mutational fitness of proteins, thus opening up new possibilities within protein engineering and design, which we leave for future work.
Acknowledgments
This project received funding from the Swiss National Science Foundation under the National Center of Competence in Research (NCCR) Catalysis under grant agreement 51 NF40 180544. Moreover, we thank Mojmír Mutný and Clemens Isert for their valuable feedback.
References
M. Baek, DiMaio, Anishchenko, Dauparas, S. Ovehinnikov, G. Lee, Wang, Q. Cong, L. N. Kinch, R. D. Schaeffer, al. Accurate prediction of protein Structures and interactions using

---

three-track neural network. Science, 373(6557), 2021.
F. Baldassarre, D. Menéndez Hurtado, A. Elofsson, and H. Azizpour. GraphQA: protein model quality assessment using graph convolutional networks. Bioinformatics, 37(3), 2021.
T. Bepler and B. Berger. Learning Protein Sequence Embeddings using Information From Structure. In International Conference on Learning Representations (ICLR), 2019.
H. M. Berman, J. Westbrook, Z. Feng, G. Gilliland, T. N. Bhat, H. Weissig, I. N. Shindyalov, and P. E. Bourne. The Protein Data Bank. Nucleic Acids Research, 28(1), 2000.
A. Brik and C.-H. Wong. HIV-1 protease: mechanism and drug discovery. Organic & -Biomolecular Chemistry, I(1), 2003.
E. Callaway. 'It will change everything': DeepMind's Al makes gigantic leap in solving protein structures. Nature, 2020.
V. Cantoni, R. Gatti, and L.. Lombardi. Segmentation of SES for Protein Structure Analysis. In Bioinformatics, 2010.
V. Cantoni, R. Gatti, and L. Lombardi. 3D Protein Surface Segmentation through Mathematical Mor- phology. In International Joint Conference on Biomedical Engineering Systems and Technologies. Springer, 2011.
D. Comaniciu and P. Meer. Mean Shift: A Robust Approach Toward Feature Space Analysis. IEEE Transactions Putlern Analysis and Machine Intelligence, 24(5), 2002.
M. L. Connolly, Solvent-accessible surfaces of proteins and nucleic acids. Science, 221(4612), 1983.
A. Dalkiran, S. Rifaioglu, M. J. Martin, R. Cetin-Atalay, V. Atalay, and T. Dogan. ECPred: a tool for the prediction of the enzymatic functions of protein sequences based on the EC nomenclature. BMC Bioinformatics, 19(1), 2018
G. Derevyanko, Grudinin, Y. Bengio, and Lamoureux. Deep convolutional networks for quality assessment of protein folds. Bioinformatics, 34(23), 2018.
M. des Jardins, P. D. Karp, M. Krummenacker, T. J. Lee, and C. A. Ouzounis. Prediction of enzyme classification from protein sequence without the use of sequence similarity. In Proceedings. International Conference
Intelligent Systems for Molecular Biology, volume 5, 1997.
D. Dobson and A. Doig, Predicting Enzyme Class From Protein Structure Without Alignments. Journal of Molecular Biology, 345(1), 2005.
A. Elnaggar, M. Heinzinger, C. Dallago, G. Rihawi, Y. Wang, L. Jones, T. Gibbs, T. Feher, C. Angerer, D. Bhowmik, et al. ProtTrans: Towards Cracking the Language of Life's Code Through Self- Supervised Deep Learning and High Performance Computing. arXiv Preprint, 2020.
P. F. Felzenszwalb and D. P. Huttenlocher. Efficient Graph-Based Image Segmentation. International Journal of Computer Vision, 59(2). 2004.
M. Fessenden. Protein maps chart the causes of disease. Nature, 549(7671), 2017.
A. Fout, Byrd, B. Shariat, and A. Ben-Hur. Protein Interface Prediction using Graph Convolutional Networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017.
P. Gainza, F. Sverrisson, F. Monti, F. Rodola, D. Boscaini, M. Bronstein, and B. Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, 17(2), 2020.
Gilmer, S. Schoenholz, F. Riley, O. Vinyals, and G. Dahl. Neural Message Passing for Quantum Chemistry. In International Conference on Machine Learning (ICML), 2017.
V. Gligorijevic, D. Renfrew, T. Kosciolek, J. K. Leman, K. Cho, T. Vatanen, D. Berenberg, B. Taylor, M. Fisk, R. J. Xavier, R. Knight, and R. Bonneau. Structure-Ban Based Function Prediction using Graph Convolutional Networks. Nature Communications, 12(1), 2021.
10

---

A. Golovin and K. Henrick. MSDmotif: exploring protein sites and motifs. BMC Bioinformaties, 9 (I), 2008.
Hermosilla, M. Schäfer, M. Lang, G. Fackelmann, P.-P. Vázquez, B. Kozlikova, M. Krone, T. Ritschel, and T. Ropinski. Atrinsic-Extrinsie Convolution and Pooling for Learning on 3D Protein Structures. In International Conference on Learning Representations (ICLR), 2021.
Hou, B. Adhikari, and Cheng. DeepSF: deep convolutional neural network for mapping protein sequences to folds. Bioinformatics, 34(8), 2018.
J. Jiménez, M. Skalic, G. Martinez-Rosell, and G. De Fabritiis. KDEEP: Protein-Ligand Abso- lute Binding Affinity Prediction via 3D-Convolutional Neural Networks. Journal of Chemical Information and Modeling, 58(2), 2018.
W. Jin, R. Barzilay, and T. Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In International Conference on Machine Learning, pages
PMLR, 2020.
Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Zidek, A. Potapenko, et al. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873), 2021.
T. N. Kipf and M. Welling, Semi-Supervised Classification with Graph Convolutional Networks, In International Conference on Learning Representations (ICLR), 2017.
Kohli, P. H. Torr, et al. Robust Higher Order Potentials for Enforcing Label Consistency, Interna- tional Conference on Computer Vision (ICCV), 82(3), 2009.
T. Lei, W. Jin, R. Barzilay, and T. Jaakkola. Deriving Neural Architectures from Sequence and Graph Kernels. In International Conference on Machine Learning (ICML), 2017.
Miu, O. Tuzel, S. Ramalingam, and R. Chellappa. Entropy Rate Superpixel Segmentation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2011.
Z. Liu, M. Su, L. Han, J. Liu, Q. Yang, Y. Li, and R. Wang. Forging the Basis for Developing Protein Ligand Interaction Scoring Functions. Accounts of Chemical Research, 50(2), 2017.
G. Mori, X. Ren, A. A. Efros, and J. Malik. Recovering Human Body Configurations: Combining Segmentation and Recognition. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), volume 2, 2004.
G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. An analysis of approximations for maximizing submodular functions. Mathematical programming, 14(1), 1978.
H. Özüürk, A. Özgür, and E. Ozkirimli. DeepDTA: deep drug-target binding affinity prediction. Bioinformatics, 34(17), 2018
A. Panjkovich and X. Daura. Assessing the structural conservation of protein pockets to study functional and allosteric sites: implications for drug discovery. BMC structural biology, 10(1): 1-14, 2010.
M. Ragoza, J. Hochuli, E. Idrobo, J. Sunseri, and D. R. Koes. Protein-ligand scoring with convolu- tional neural networks. Journal of chemical information and modeling. 57(4): 2017.
R. Rao, N. Bhattacharya, N. Thomas, Y. Duan, X. Chen, J. Canny, P. Abbeel, and Y. S. Song. Evaluating Protein Transfer Learning with TAPE. In Advances in Neural Informtion Processing Systems (NeurIPS), 2019.
X. Ren and J. Malik. Learning a Classification Model for Segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, 2003.
M. F. Sanner, A. J. Olson, and J.-C. Spehner. Reduced Surface: An Efficient Way to Compute Molecular Surfaces. Biopolymers, 38(3), 1996.
Schymkowitz, J. Borg, F. Stricher, R. Nys, F. Rousseau, and Serrano. The FoldX web server: an online force field. Nucleic Acids Research, 33, 2005.

---

A. W. Senior, R. Evans, J. Jumper, J. Kirkpatrick, L. Sifre, T. Green, C. Qin, A. Zidek. A. W. Nelson, A. Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature, 577(7792), 2020.
M. M. Saapniewssa-Dziubinkee P. Zielenkiewicz, and P. Siedlecki. Improving detection of protein- ligand binding sites with 3d segmentation. Scientific Reports, 10(1), 2020.
D. Stutz, A. Hermans, and B. Leibe. Superpixels: An evaluation of the state-of-the-an Computer Vision and Image Understanding, 166, 2018.
Sverrisson, J. Feydy, B. Correia, and M. Bronstein. Fast end-to-end learning on protein surfaces. bioRxiv, 2020.
R. Townshend, R. Bedi, P. Suriana, and R. Dror. End-to-End Learning on 3D Protein Structure for Interface Prediction. Advances Neural Information Processing Systems (NeurIPS), 32, 2019.
R. Townshend, M. Vögele, Suriana, A. Derry, A. Powers, Laloudakis, S. Balachandar, B. Anderson, S. Eismann, R. Kondor, et al. ATOM3D: Tasks On Molecules in Three Dimensions.
NeurIPS Workshop of Learning Meaningful Representations of Life (LMRL), 2020.
L. Vincent and P. Soille. Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations. IEEE Computer Architecture Letters, 13(06), 1991.
E. C. Webb. Enzyme Nomenclature 1992. Recommendations of the Nomenclature Committee of the In- ternational Union of Biochemistry and Molecular Biology on the Nomenclature and Classification of Enzymes. Academic Press, 1992.
K. K. Yang, Z. Wu, and Arnold. Machine-leaming--uuidee directed evolution for protein engineering. Nature Methods, 16(8), 2019.
12


Reviews:
[{"review_id": "qXha_TXEutE", "reviewer": "Reviewer_TKtD", "summary": "The authors consider the problem of learning protein representations which jointly leverage features at multiple scales: sequence, structure, and surface. To accomplish this, a graph construction is proposed with interconnected structure and surface layers, and physicochemical node/edge features. A proposed model encodes the graph sublayers into a unified representation of the protein.  Computed representations are evaluated on two tasks: prediction of (1) ligand binding affinity and (2) enzyme class.  Finally, the authors propose the notion of “molecular superpixels” - adapting an image segmentation algorithm to partition molecular surfaces into compact, geometrically and chemically homogenous zones.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe paper presents a new approach to multi-scale protein representation learning, with a creative adaptation of superpixels for partitioning molecular surfaces into compact, homogenous regions.\n\nMain Strengths:\n\n\n\n* The multi-scale graph construction - uniting sequence, structure, and surface - is well motivated, conceptually simple, and a novel contribution.\n* The application of superpixels to biomolecules is a novel contribution, and is compelling for: (1) its connection to a successful method in an outside field, and (2) its general applicability to any feature set.\n\nMajor Weaknesses:\n\n\n\n* Final result impact is diluted by underperformance relative to (Hermosilla et al, 2021) on enzyme class task, despite Holoprot’s additional inclusion of surface features.\n* Results in Tables 1 and 2 are hard to parse without indicators of model type and input data (sequences, structures), which are important to contextualize results.  The results are less impressive when ignoring the sequence-only rows and including other strong baselines in the Hermosilla baseline source: HHSuite (82.6%) and Rao - LSTM (79.9%).\n\nBased on the novelty + conceptual simplicity of the proposed method, but with the weakness of experimental results limiting the impact: I recommend 6: Marginally above acceptance threshold.\n\n\n### Setup, approach, originality\n\n\n\n* Problem setup is good (forward-looking and impactful): structure-based representation + surface graph, evaluated on reasonable tasks.\n    * [-]  But, rather than tasks relying on fixed-size embeddings of the full protein, predicting protein surface interaction location like the tasks in Gainza et al 2020 seems more promising and appropriate for this model class - was this considered?\n* The multi-scale graph construction is conceptually simple, and well motivated.\n* The application of superpixels to biomolecules is a novel contribution, and is compelling for its simplicity and generality.\n* Though, as the author’s allude to in section [4 - Superpixels on Molecular Surfaces] there is a long history of methods for identifying regions on three-dimensional molecular surfaces.  Molecular superpixels are stated to be advantageous due to better geometric compactness relative to some baselines, but no quantitative analysis, nor claim is made of general superiority over alternative algorithms - fair.\n\n\n### Quality\n\nMajor:\n\n\n\n* [+] Ablations were informative.  It is nice to see that more powerful pooling on molecular superpixels was unnecessary.  And it is interesting that the marginal benefit of the structure / surface layers is so different across tasks.\n* [+] [5.4 - Limitations]  Important call-out that this method requires structures is appreciated.\n* [?] [Abstract]  It is claimed that Holoprot is more _consistent_ than other methods. How is this reflected in the results? The level of variation looks comparable to that of Hermosilla/Gainza/others.\n* [-] The list of features in the surface and structure layers node/edge features seem extensive.\n    * What is the reasoning behind the node/edge features chosen in the structure layer?\n    * It would be informative to know how helpful the 7 edge features in the surface network (Hanocka et al 2019) are.  Since it seems Gainza did not use these. (e.g. Are features or the model the driver of the boost over Gainza?) \n\nMinor:\n\n\n\n* [Table 1] Townshend et al. (2020) results are missing for the scaffold-level split?\n* Line 540: “structure” is a typo.\n\n### Clarity\n\n\nMajor:\n\n\n* [+] Paper + supplement are well organized.  Explanations are clear, and the appendix concisely adds much useful detail.\n* [Table 1 + 2]\n    * The tables would benefit highly from organization by model / input-data type, as opposed to the current organization by chronology.\n        * Partial example: Table 1 of the baseline results source (Hermosilla, et al., 2021), which specifies architecture + special notes on data sources.\n    * [-] Other top baselines from that paper should not be omitted: HHSuite (82.6%) and Rao - LSTM (79.9%).\n* The superpixel ERS method is hard to follow, could probably be expanded in appendix, and a high level intuitive explanation be provided in the maintext.\n\n### Significance\n\n* Multi-scale graph construction and representation algorithms are conceptually clean and well motivated.  The focus on explicitly incorporating protein surface features seems a useful and forward-looking approach.\n* The superpixel algorithm is not quantitatively shown as superior to alternative algorithms, but is compelling for its simplicity and general applicability.  Future researchers may find this to be a useful primitive.\n* The final result impact is diluted by underperformance relative to (Hermosilla et al, 2021) on the enzyme class task, despite Holoprot’s additional inclusion of surface features.\n* Low data regime argument is interesting [Conclusion], but could be improved:\n    * Is it possible to scale the current model, for better performance?\n    * Is it possible to evaluate on a task that is more in the low data regime?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.  We appreciate the acknowledgement in [section 5.4 - Limitations] and [section 2 - Related Work] that baseline models were trained with very divergent input data modalities, and that this method explicitly requires structures.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}, {"review_id": "QI6ZgPjVvqg", "reviewer": "Reviewer_hzxn", "summary": "The authors introduce a representation learning approach to protein structures which encompasses primary sequence, secondary structure, inter-residue distances, and surface features captured by triangulations. These different scales are encoded using two linked graph representations, and message passing networks are trained to learn geometric features from this system. Applications to protein-ligand binding and enzyme reaction classification provide evidence the model can learn effectively with few parameters. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper pursues an interesting and novel direction in protein representation learning, seeking to integrate multiple existing approaches to feature extraction in a single architecture. Given the plethora of sequence and structure-based models, as well as recent work on surface representation, this is a natural line of investigation to pursue. The execution is simple and clean, making the paper easy to follow. Evaluations and empirical results are good but improvements or further analysis will strengthen the overall work.\n\nStrengths:\n1. The features chosen at each input scale are well-chosen. The graph structures representing each scale and the model choices for linking these scales are also reasonable.\n2. Applying superpixels to protein surfaces is an interesting idea in its own right, and the authors systematically evaluate its impact on performance throughout the paper. The particular choice of superpixel generation algorithm is motivated well by the authors.\n3. Dataset choices were good and proper attention was paid to splitting of train/eval/test sets. \n4. The ablation study in Table 3 is thorough and provides context for model performance.\n\nWeaknesses (elaborated on in Limitations):\ni. The language model baseline for protein-ligand binding affinity needs to be improved. Given that ProtBERT-BFD (Elnaggar et al 2020) is included in Table 2, this would be a natural choice to include for Table 1 as well. This model has proven substantially better than previous language models (such as Rao et al or Bepler and Berger), so it is necessary to get an informative comparison to pretrained sequence models.\nii. A number of analysis questions should be explored further, as they are central to the story. Major ones include: why is structure-only in Table 3 comparable to Holoprot across all tasks? why does Holoprot do worse on 30% sequence identity splits but better on 60% and scaffold-based splits in Table 1?\niii. The authors argue their model is more parameter efficient in settings where it loses to the Hermosilla et al baseline. Is it instead possible for them to increase the parameters of Holoprot by an order of magnitude and see if this improves performance? \n\nMinor comments:\n- The formatting of Table 2 could be improved to highlight which methods are sequence-only, which are structure-only, etc.\n\nThings that would raise my score:\n1) Inclusion of ProtBERT-BFD for the protein-ligand binding results.\n2) Expanded analyses identifying successes or challenges of the model. Some are suggested in this review, but these are just suggestions. This is particularly valuable for future work from the authors or others in the community.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Choice of language model in Table 1 could be improved. Either use of ProtBERT-BFD (Elnaggar et al 2020) or ESM (Rives et al 2020) are appropriate. Given the use of Elnaggar et al in Table 2, this might be the simpler choice.\n2. The change in performance from structure-only to Holoprot across Table 3 is favorable to Holoprot but not drastic. This seems to suggest that the addition of surface-level representation is improving performance in specific regimes or subproblems. Understanding this phenomenon would be very useful for future multiscale representation learning. Currently the authors claim \"HOLOPROT with and without molecular superpixels improve over the performance of structure and surface representations\" but this claim seems a bit strong, especially given the numbers on protein-ligand binding.\n3. The authors do not provide any suggestion for why they lose to Townshend et al at 30% sequence identity splits but beat them otherwise. Is this due to over-reliance on sequence features by Holoprot? Again, some analysis would be illuminating here.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "NcrbioAA8s", "reviewer": "Reviewer_tk1T", "summary": "The paper proposes a novel model incorporating sequence-level, structure-level, and surface-level information from a protein. This information is combined through an MPNN with a hierarchical structure. The authors additionally propose a molecular superpixel representation of the protein surface in order to lower computational/memory costs, with no loss of performance. The model achieves generally good results while using a very small number of parameters on both binding affinity and enzyme-catalyzed reaction classification.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting paper in an important emerging field. Given recent advances from CASP, I expect that structure-based prediction / representation learning will play an increasingly large role in protein property prediction and protein design. This paper has two major contributions - the architecture combining structure- and surface-based features, and the use of molecular superpixels. These ideas are likely to be generally useful to the structure representation community. The evaluations are reasonable and the model achieves decent performance using relatively few parameters. The ablations also suggest both structure and surface information are useful, although the effect seems minimal in the case of ligand binding affinity.\n\nHere are some suggestions for improvements to the paper:\n\n- The method proposed here uses a one-hot representation of amino acid identity as the sequence level feature for each residue. Why not instead use the output of a protein language model, such as ESM-1b or ProtBERT-BFD? These feature vectors are known to contain a good deal of biologically relevant information (including e.g. secondary structure) so it seems like this is an easy way to get the benefits of a large scale language model incorporated into your model.\n- In the ligand binding affinity task you do not benchmark against a recent sequence-based method (e.g. ESM-1b, ProtBERT-BFD, ProTrans-T5). It would be nice to have this evaluation.\n- How was the model size for HoloProt chosen? It seem as though the number of parameters is much smaller, and you make an argument for the parameter efficiency. However, could the method be scaled up to achieve SOTA performance? In other words, was the model size chosen due to computational constraints or is there a more fundamental limitation preventing scaling the model and achieving better results?\n- The paper introduces molecular superpixels as a method to reduce computational / memory costs, but the improvements are not benchmarked.\n- I am hesitant to suggest additional experiments since I understand they are likely infeasible in the author response period. However, if time permits or for future publications it may be worth looking into the Atom3D suite of tasks (Townshend 2020).\n\nMinor issue:\n\n- The citation provided for AlphaFold2 is the Senior 2020 Nature paper describing AlphaFold1. The correct citation is found at the end of [this blog post](https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology). \n\nFinally, I note that you describe training several of these models takes a significant amount of time (40 hours). For some of these models (e.g. the pretrained language models) a better option is to embed each sequence and save the features to disk (assuming you are not fine-tuning the entire model). Then during training, simply loading the features and training the MLP will likely allow you to train the model in minutes. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss limitations of the work in the paper, and some additional suggestions are provided in my review above.\n\nThis paper is a basic computational biology work and has no immediate social impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a very interesting approach for protein representation learning. The AC and reviewers greatly appreciated the author feedback and we urge the authors to incorporate their points into the manuscript.

In particular it would be important to include the comparison with ProtBERT-BFD  and the extended analysis results. 

If the authors cannot include results for larger number of parameters as promised, it is critical that they include comments on limited resources, where the bottlenecks are and path to resolve these.

Decision: Accept (Poster)