Multi-Scale Representation Learning on Proteins

1 Introduction . Protein design and engineering has become a crucial component of pharmaceutical research and development , finding application in a wide variety of diagnostic and industrial settings . Besides understanding the design principles determining structure and function of proteins , current efforts seek to further enhance or discover proteins with properties useful for technological or therapeutic applications . To efficiently guide the search in the vast design space of functional proteins , we need to be able to robustly predict properties of a candidate protein [ Yang et al. , 2019 ] . Moreover , understanding role and function of proteins is crucial to study causes and mechanism of human disease [ Fessenden , 2017 ] . To achieve this , representations incorporating the complex nature of proteins are required . Proteins consist of amino acids , organic molecules linked by peptide bonds forming a linear sequence . Each of the twenty amino acids carries a unique side chain , giving rise to an incomprehensibly large combinatorial space of possible protein sequences . The primary sequence drives the folding of polymers – a spontaneous process guided by hydrophobic interactions , formation of intramolecular hydrogen bonds , and van der Waals forces into a unique three-dimensional structure . The resulting shape and surface manifold with rich physiochemical properties carry essential information for understanding function and potential molecular interactions . Previous methods typically only consider an individual subset within these scales , focusing on either sequence [ Öztürk et al. , 2018 , Hou et al. , 2018 ] , three-dimensional structure [ Hermosilla et al. , 2021 , Derevyanko et al. , 2018 ] or surface [ Gainza et al. , 2020 ] . Two proteins with similar sequences can fold into entirely different conformations . While these proteins might catalyze the same type of ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . i.fr n ya ou sa fin oo se finir oo sa reactions , their behavior to specific inhibiting drugs might be divergent . Interaction between proteins and ligands , on the other hand , is controlled by molecular surface contacts [ Gainza et al. , 2020 ] . Molecular surfaces , determined by subjacent amino acids , are fingerprinted with patterns of geometric and chemical properties , and thus their integration in protein representations is crucial . In this work , we present a novel multi-scale graph representation which integrates and connects the complex nature of proteins across all levels of information . HOLOPROT consists of a surface and structure layer ( both represented as graphs ) with explicit edges between the layers . Our construction is guided by the intuition that propagating information from surface to structure would allow each residue to learn encodings reflective of not just its immediate residue neighborhood , but also the higher-level geometric and chemical properties that arise from interactions between a residue and its neighborhood . The associated multi-scale encoder then learns representations by integrating the encoding from the layer below , with the graph at that layer ( Section 3 ) . Such multi-scale representations have been previously used in molecular graph generation [ Jin et al. , 2020 ] with impressive results . We further improve the memory efficiency of our construction by segmenting the large and rich protein surface into molecular “ superpixels ” , summarizing higher-level fingerprint features and motifs of proteins . Substituting the surface layer with these superpixels results in little to no performance degradation across the evaluated tasks . The concept of molecular superpixels might be of interest beyond our model ( Section 4 ) . The multi-objective and multi-task nature of protein engineering poses a challenge for current methods , often designed and evaluated only on specific subtasks of protein design . By incorporating the biology of proteins , strong representations exhibit robust performance across tasks . We demonstrate our model ’ s versatility and range of applications by deploying it to tasks of rather distinct nature , including a regression task , e.g. , inference of protein ligand binding affinity , and classification tasks , i.e. , enzyme-catalyzed reaction classification ( Section 5 ) . 2 Related Work . Protein Representation Learning With increasing availability of sequence and structure data , the field of protein representation learning has advanced rapidly , with methods falling largely in one of the following categories : Sequence-based methods . One-dimensional amino acid sequences continue to be the simplest , most abundant source of protein data and various methods have been developed that borrow architectures developed in natural language processing ( NLP ) . One-dimensional convolutional neural networks have been used to classify a protein sequence into folds and enzyme function [ Hou et al. , 2018 , Dalkiran et al. , 2018 ] , and to predict their binding affinity to ligands [ Öztürk et al. , 2018 ] . Furthermore , methods have applied complex NLP models trained unsupervised on millions of unlabeled protein sequences and fine-tuned them on different downstream tasks [ Rao et al. , 2019 , Elnaggar et al. , 2020 , Bepler and Berger , 2019 ] . Despite being advantageous when only the sequence is available , these methods ignore the full spatial complexity of proteins . Structure-based methods . To learn beyond sequences , approaches have been developed , that consider the 3D structure of proteins . 3D convolutional neural networks have been utilized for protein quality assessment [ Derevyanko et al. , 2018 ] , protein contact prediction [ Townshend et al. , 2019 ] and protein-ligand binding affinity tasks [ Ragoza et al. , 2017 , Jiménez et al. , 2018 , Townshend et al. , 2020 ] . An alternate representation treats proteins as graphs , applying graph neural networks for enzyme classification [ Dobson and Doig , 2005 ] , interface prediction [ Fout et al. , 2017 ] , and protein structure quality prediction [ Baldassarre et al. , 2021 ] . Gligorijevic et al . [ 2021 ] use a long short term memory cell ( LSTM ) to encode the sequence , followed by a graph convolutional network ( GCN ) [ Kipf and Welling , 2017 ] to capture the tertiary structure , and apply this to the function prediction task . Hermosilla et al . [ 2021 ] propose a convolutional operator that learns to adapt filters based on the primary , secondary , and tertiary structure of a protein , showing strong performance on reaction and fold class prediction . Surface-based methods . Taking a different viewpoint , Gainza et al . [ 2020 ] hypothesize that the protein surface displays patterns of chemical and geometric features that fingerprint a protein ’ s interaction with other biomolecules . They utilize geodesic convolutions , which are extensions of convolutions on surfaces , and learn fingerprint vectors , showing improved performance across binding pocket and protein interface prediction tasks . Protein Motif Detection Protein motifs have largely been synonymous with common and conserved patterns in a protein ’ s sequence or structure influencing protein function , e.g. , the helixturn-helix motif binds DNA . Understanding these fragments is essential for 3D structure prediction , modeling , and drug design . While reliably detecting evolutionary motifs , existing tools [ Golovin and Henrick , 2008 ] do not provide a full segmentation of the protein surface manifold . Our work takes a different viewpoint , by looking at protein motifs from the context of a protein surface . Previous methods developed in this context either only consider geometric information rather than physiological properties [ Cantoni et al. , 2010 ] , are computationally expensive [ Cantoni et al. , 2011 ] , or designed for particular downstream tasks [ Stepniewska-Dziubinska et al. , 2020 ] . Our molecular superpixel approach provides a task-independent segmentation utilizing both geometric and chemical features , while also being computationally efficient . 3 Multi-Scale Protein Representation . In this section , we describe our multi-scale graph construction and the associated encoder . Figure 1 illustrates the main principles of HOLOPROT . We represent a protein P as a graph GP with two layers capturing different scales : ( i . ) Surface layer . This layer captures the coarser representation details of a protein . The protein surface is generated using the triangulation software MSMS [ Connolly , 1983 , Sanner et al. , 1996 ] . We represent this layer as a graph GS , where each surface node uS has a feature vector fuS denoting its charge , hydrophobicity and local curvature [ Gainza et al. , 2020 ] . Two surface nodes ( uS , vS ) have an edge if they are part of a triangulation . Each surface node additionally has a residue identifier r , indicating the amino acid residue it corresponds to . Multiple surface nodes can have the same residue identifier . ( ii . ) Structure layer . This layer captures the finer representation details of a protein . A protein typically has four structural levels : ( i . ) primary structure ( sequence ) , ( ii . ) secondary structure ( α-helices and β-sheets ) , ( iii . ) tertiary structure ( 3D structure ) and ( iv . ) quaternary structure ( complexes ) [ Fout et al. , 2017 ] . We represent this layer as a graph GB , where each node uB corresponds to a residue r. Two nodes ( uB , vB ) have an edge in GB if the Cα atoms of the two nodes occur within a certain distance of each other . Distance based thresholding ensures that different structural levels are implicitly captured in the neighborhood of a node uB . We further introduce edges from the surface layer to the structure layer in order to propagate information between them . Specifically , we introduce a directed edge between a surface node uS and a backbone node uB if they both have the same residue identifier r. Typically , we have between 20-40 surface nodes { uS } that map to the same structure node uB . This gives us the multi-scale graph which is then encoded by our multi-scale message passing network . Details on the features used for both the structure and surface layer can be found in Appendix ? ? . 3.1 Multi-Scale Encoder . Our multi-scale message passing network uses one message passing neural network ( MPN ) for each layer in the multi-scale graph [ Lei et al. , 2017 , Gilmer et al. , 2017 ] . This allows us to learn structured representations of each scale , which can then be tied together through connections between the scales . Before detailing the remainder of the architecture , we introduce some notational preliminaries . For simplicity , we denote the MPN encoding process as MPNθ ( · ) with parameters θ . We denote MLPθ ( x , y ) for a multi-layer perceptron ( MLP ) with parameters θ , whose input is the concatenation of x and y , and MLPθ ( x ) when the input is only x . We also denote the residue identifier of a node u with id ( u ) , and the neighbors of a node u as N ( u ) . The details of the MPN architecture are listed in the Appendix ? ? . 3.1.1 Surface Message Passing Network . We first encode the surface layer GS of the multi-scale protein graph GP . The inputs to the MPN are node features fuS and edge features fuSvS of GS . For more details on the input features used for surface nodes and edges , refer to Appendix ? ? . The MPN ( with parameters θS ) propagates messages between the nodes for K iterations , and outputs a representation huS for each surface node uS , { huS } = MPNθS ( GS , { fuS } , { fuSvS } vS∈N ( uS ) ) .

Summary: The authors introduce a representation learning approach to protein structures which encompasses primary sequence, secondary structure, inter-residue distances, and surface features captured by triangulations. These different scales are encoded using two linked graph representations, and message passing networks are trained to learn geometric features from this system. Applications to protein-ligand binding and enzyme reaction classification provide evidence the model can learn effectively with few parameters.