Multi-Scale Representation Learning on Proteins

1 Introduction . Protein design and engineering has become a crucial component of pharmaceutical research and development , finding application in a wide variety of diagnostic and industrial settings . Besides understanding the design principles determining structure and function of proteins , current efforts seek to further enhance or discover proteins with properties useful for technological or therapeutic applications . To efficiently guide the search in the vast design space of functional proteins , we need to be able to robustly predict properties of a candidate protein [ Yang et al. , 2019 ] . Moreover , understanding role and function of proteins is crucial to study causes and mechanism of human disease [ Fessenden , 2017 ] . To achieve this , representations incorporating the complex nature of proteins are required . Proteins consist of amino acids , organic molecules linked by peptide bonds forming a linear sequence . Each of the twenty amino acids carries a unique side chain , giving rise to an incomprehensibly large combinatorial space of possible protein sequences . The primary sequence drives the folding of polymers – a spontaneous process guided by hydrophobic interactions , formation of intramolecular hydrogen bonds , and van der Waals forces into a unique three-dimensional structure . The resulting shape and surface manifold with rich physiochemical properties carry essential information for understanding function and potential molecular interactions . Previous methods typically only consider an individual subset within these scales , focusing on either sequence [ Öztürk et al. , 2018 , Hou et al. , 2018 ] , three-dimensional structure [ Hermosilla et al. , 2021 , Derevyanko et al. , 2018 ] or surface [ Gainza et al. , 2020 ] . Two proteins with similar sequences can fold into entirely different conformations . While these proteins might catalyze the same type of ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . i.fr n ya ou sa fin oo se finir oo sa reactions , their behavior to specific inhibiting drugs might be divergent . Interaction between proteins and ligands , on the other hand , is controlled by molecular surface contacts [ Gainza et al. , 2020 ] . Molecular surfaces , determined by subjacent amino acids , are fingerprinted with patterns of geometric and chemical properties , and thus their integration in protein representations is crucial . In this work , we present a novel multi-scale graph representation which integrates and connects the complex nature of proteins across all levels of information . HOLOPROT consists of a surface and structure layer ( both represented as graphs ) with explicit edges between the layers . Our construction is guided by the intuition that propagating information from surface to structure would allow each residue to learn encodings reflective of not just its immediate residue neighborhood , but also the higher-level geometric and chemical properties that arise from interactions between a residue and its neighborhood . The associated multi-scale encoder then learns representations by integrating the encoding from the layer below , with the graph at that layer ( Section 3 ) . Such multi-scale representations have been previously used in molecular graph generation [ Jin et al. , 2020 ] with impressive results . We further improve the memory efficiency of our construction by segmenting the large and rich protein surface into molecular “ superpixels ” , summarizing higher-level fingerprint features and motifs of proteins . Substituting the surface layer with these superpixels results in little to no performance degradation across the evaluated tasks . The concept of molecular superpixels might be of interest beyond our model ( Section 4 ) . The multi-objective and multi-task nature of protein engineering poses a challenge for current methods , often designed and evaluated only on specific subtasks of protein design . By incorporating the biology of proteins , strong representations exhibit robust performance across tasks . We demonstrate our model ’ s versatility and range of applications by deploying it to tasks of rather distinct nature , including a regression task , e.g. , inference of protein ligand binding affinity , and classification tasks , i.e. , enzyme-catalyzed reaction classification ( Section 5 ) . 2 Related Work . Protein Representation Learning With increasing availability of sequence and structure data , the field of protein representation learning has advanced rapidly , with methods falling largely in one of the following categories : Sequence-based methods . One-dimensional amino acid sequences continue to be the simplest , most abundant source of protein data and various methods have been developed that borrow architectures developed in natural language processing ( NLP ) . One-dimensional convolutional neural networks have been used to classify a protein sequence into folds and enzyme function [ Hou et al. , 2018 , Dalkiran et al. , 2018 ] , and to predict their binding affinity to ligands [ Öztürk et al. , 2018 ] . Furthermore , methods have applied complex NLP models trained unsupervised on millions of unlabeled protein sequences and fine-tuned them on different downstream tasks [ Rao et al. , 2019 , Elnaggar et al. , 2020 , Bepler and Berger , 2019 ] . Despite being advantageous when only the sequence is available , these methods ignore the full spatial complexity of proteins . Structure-based methods . To learn beyond sequences , approaches have been developed , that consider the 3D structure of proteins . 3D convolutional neural networks have been utilized for protein quality assessment [ Derevyanko et al. , 2018 ] , protein contact prediction [ Townshend et al. , 2019 ] and protein-ligand binding affinity tasks [ Ragoza et al. , 2017 , Jiménez et al. , 2018 , Townshend et al. , 2020 ] . An alternate representation treats proteins as graphs , applying graph neural networks for enzyme classification [ Dobson and Doig , 2005 ] , interface prediction [ Fout et al. , 2017 ] , and protein structure quality prediction [ Baldassarre et al. , 2021 ] . Gligorijevic et al . [ 2021 ] use a long short term memory cell ( LSTM ) to encode the sequence , followed by a graph convolutional network ( GCN ) [ Kipf and Welling , 2017 ] to capture the tertiary structure , and apply this to the function prediction task . Hermosilla et al . [ 2021 ] propose a convolutional operator that learns to adapt filters based on the primary , secondary , and tertiary structure of a protein , showing strong performance on reaction and fold class prediction . Surface-based methods . Taking a different viewpoint , Gainza et al . [ 2020 ] hypothesize that the protein surface displays patterns of chemical and geometric features that fingerprint a protein ’ s interaction with other biomolecules . They utilize geodesic convolutions , which are extensions of convolutions on surfaces , and learn fingerprint vectors , showing improved performance across binding pocket and protein interface prediction tasks . Protein Motif Detection Protein motifs have largely been synonymous with common and conserved patterns in a protein ’ s sequence or structure influencing protein function , e.g. , the helixturn-helix motif binds DNA . Understanding these fragments is essential for 3D structure prediction , modeling , and drug design . While reliably detecting evolutionary motifs , existing tools [ Golovin and Henrick , 2008 ] do not provide a full segmentation of the protein surface manifold . Our work takes a different viewpoint , by looking at protein motifs from the context of a protein surface . Previous methods developed in this context either only consider geometric information rather than physiological properties [ Cantoni et al. , 2010 ] , are computationally expensive [ Cantoni et al. , 2011 ] , or designed for particular downstream tasks [ Stepniewska-Dziubinska et al. , 2020 ] . Our molecular superpixel approach provides a task-independent segmentation utilizing both geometric and chemical features , while also being computationally efficient . 3 Multi-Scale Protein Representation . In this section , we describe our multi-scale graph construction and the associated encoder . Figure 1 illustrates the main principles of HOLOPROT . We represent a protein P as a graph GP with two layers capturing different scales : ( i . ) Surface layer . This layer captures the coarser representation details of a protein . The protein surface is generated using the triangulation software MSMS [ Connolly , 1983 , Sanner et al. , 1996 ] . We represent this layer as a graph GS , where each surface node uS has a feature vector fuS denoting its charge , hydrophobicity and local curvature [ Gainza et al. , 2020 ] . Two surface nodes ( uS , vS ) have an edge if they are part of a triangulation . Each surface node additionally has a residue identifier r , indicating the amino acid residue it corresponds to . Multiple surface nodes can have the same residue identifier . ( ii . ) Structure layer . This layer captures the finer representation details of a protein . A protein typically has four structural levels : ( i . ) primary structure ( sequence ) , ( ii . ) secondary structure ( α-helices and β-sheets ) , ( iii . ) tertiary structure ( 3D structure ) and ( iv . ) quaternary structure ( complexes ) [ Fout et al. , 2017 ] . We represent this layer as a graph GB , where each node uB corresponds to a residue r. Two nodes ( uB , vB ) have an edge in GB if the Cα atoms of the two nodes occur within a certain distance of each other . Distance based thresholding ensures that different structural levels are implicitly captured in the neighborhood of a node uB . We further introduce edges from the surface layer to the structure layer in order to propagate information between them . Specifically , we introduce a directed edge between a surface node uS and a backbone node uB if they both have the same residue identifier r. Typically , we have between 20-40 surface nodes { uS } that map to the same structure node uB . This gives us the multi-scale graph which is then encoded by our multi-scale message passing network . Details on the features used for both the structure and surface layer can be found in Appendix ? ? . 3.1 Multi-Scale Encoder . Our multi-scale message passing network uses one message passing neural network ( MPN ) for each layer in the multi-scale graph [ Lei et al. , 2017 , Gilmer et al. , 2017 ] . This allows us to learn structured representations of each scale , which can then be tied together through connections between the scales . Before detailing the remainder of the architecture , we introduce some notational preliminaries . For simplicity , we denote the MPN encoding process as MPNθ ( · ) with parameters θ . We denote MLPθ ( x , y ) for a multi-layer perceptron ( MLP ) with parameters θ , whose input is the concatenation of x and y , and MLPθ ( x ) when the input is only x . We also denote the residue identifier of a node u with id ( u ) , and the neighbors of a node u as N ( u ) . The details of the MPN architecture are listed in the Appendix ? ? . 3.1.1 Surface Message Passing Network . We first encode the surface layer GS of the multi-scale protein graph GP . The inputs to the MPN are node features fuS and edge features fuSvS of GS . For more details on the input features used for surface nodes and edges , refer to Appendix ? ? . The MPN ( with parameters θS ) propagates messages between the nodes for K iterations , and outputs a representation huS for each surface node uS , { huS } = MPNθS ( GS , { fuS } , { fuSvS } vS∈N ( uS ) ) .

Summary: The paper proposes a novel model incorporating sequence-level, structure-level, and surface-level information from a protein. This information is combined through an MPNN with a hierarchical structure. The authors additionally propose a molecular superpixel representation of the protein surface in order to lower computational/memory costs, with no loss of performance. The model achieves generally good results while using a very small number of parameters on both binding affinity and enzyme-catalyzed reaction classification.