Intrinsic-Extrinsic Convolution and Pooling for Learning on 3D Protein Structures

1 INTRODUCTION Proteins perform specific biological functions essential for all living organisms and hence play a key role when investigating the most fundamental questions in the life sciences . These biomolecules are composed of one or several chains of amino acids , which fold into specific conformations to enable various biological functionalities . Proteins can be defined using a multi-level structure : : The primary structure is given by the sequence of amino acids that are connected through covalent bonds and form the protein backbone . Hydrogen bonds between distant amino acids in the chain form the secondary structure , which defines substructures such as α-helices and β-sheets . The tertiary structure results from protein folding and expresses the 3D spatial arrangement of the secondary structures . Lastly , the quarternary structure is given by the interaction of multiple amino acid chains . Considering only one subset of these levels can lead to misinterpretations due to ambiguities . As shown by Alexander et al . ( 2009 ) , proteins with almost identical primary structure , i.e. , only containing a few different amino acids , can fold into entirely different conformations . Conversely , proteins from SH3 and OB folds have similar tertiary structures , but their primary and secondary structures differ significantly ( Agrawal & Kishan , 2001 ) ( Fig . 1 ) . To avoid misinterpretations arising from these observations , capturing the invariances with respect to primary , secondary , and tertiary structures is of key importance when studying proteins and their functions . Previously , the SOTA was dominated by methods based on hand-crafted features , usually extracted from multi-sequence alignment tools ( Altschul et al. , 1990 ) or annotated databases ( El-Gebali et al. , 2019 ) . In recent years , these have been outperformed by protein learning algorithms in different protein modeling tasks such as protein fold classification ( Hou et al. , 2018 ; Rao et al. , 2019 ; Bepler & Berger , 2019 ; Alley et al. , 2019 ; Min et al. , 2020 ) or protein function prediction ( Strodthoff et al. , 2020 ; Gligorijevic et al. , 2019 ; Kulmanov et al. , 2017 ; Kulmanov & Hoehndorf , 2019 ; Amidi et al. , 2017 ) . This can be attributed to the ability of machine learning algorithms to learn meaningful representations of proteins directly from the raw data . However , most of these techniques only consider a subset of the relevant structural levels of proteins and thus can only create a representation from partial information . For instance , due to the high amount of available protein sequence data , most techniques solely rely on protein sequence data as input , and apply learning algorithms from the field of natural language processing ( Rao et al. , 2019 ; Alley et al. , 2019 ; Min et al. , 2020 ; Strodthoff et al. , 2020 ) , 1D convolutional neural networks ( Kulmanov et al. , 2017 ; Kulmanov & Hoehndorf , 2019 ) , or use structural information during training ( Bepler & Berger , 2019 ) . Other methods have solely used 3D atomic coordinates as an input , and applied 3D convolutional neural networks ( 3DCNN ) ( Amidi et al. , 2017 ; Derevyanko et al. , 2018 ) or graph convolutional neural networks ( GCNN ) ( Kipf & Welling , 2017 ) . While few attempts have been made to consider more than one structural level of proteins in the network architecture ( Gligorijevic et al. , 2019 ) , none of these hybrid methods incorporate all structural levels of proteins simultaneously . In contrast , a common approach is to process one structural level with the network architecture and the others indirectly as input features ( Baldassarre et al . ( 2020 ) or Hou et al . ( 2018 ) ) . In this paper , we introduce a novel end-to-end protein learning algorithm , that is able to explicitly incorporate the multi-level structure of proteins and captures the resulting different invariances . We show how a multi-graph data structure can represent the primary and secondary structures effectively by considering covalent and hydrogen bonds , while the tertiary structure can be represented by the spatial 3D coordinates of the atoms ( Sec . 3 ) . By borrowing terminology from differential geometry of surfaces , we define a new convolution operator that uses both intrinsic ( primary and secondary structures ) and extrinsic ( tertiary and quaternary structures ) distances ( Sec . 4 ) . Moreover , since protein sizes range from less than one hundred to tens of thousands of amino acids ( Brocchieri & Karlin , 2005 ) , we propose protein-specific pooling operations that allow hierarchical grouping of such a wide range of sizes , enabling the detection of features at different scales ( Sec . 5 ) . Lastly , we demonstrate , that by considering all mentioned protein structure levels , we can significantly outperform recent SOTA methods on protein tasks , such as protein fold and enzyme classification . Code and data of our approach is available at https : //github.com/phermosilla/ IEConv_proteins . 2 RELATED WORK . Early works on learning protein representations ( Asgari & Mofrad , 2015 ; Yang et al. , 2018 ) used word embedding algorithms ( Mikolov et al. , 2013 ) , as employed in Natural Language Processing ( NLP ) . Other approaches have used 1D convolutional neural networks ( CNN ) to learn protein representations directly from an amino acid sequence , for tasks such as protein function prediction ( Kulmanov et al. , 2017 ; Kulmanov & Hoehndorf , 2019 ) , protein-compound interaction ( Tsubaki et al. , 2018 ) , or protein fold classification ( Hou et al. , 2018 ) . Recently , researchers have applied complex NLP models trained unsupervised on millions of unlabeled protein sequences and fine-tune them for different downstream tasks ( Rao et al. , 2019 ; Alley et al. , 2019 ; Min et al. , 2020 ; Strodthoff et al. , 2020 ; Bepler & Berger , 2019 ) . While representing proteins as amino acid sequences during learning , is helpful when only sequence data is available , it does not leverage the full potential of spatial protein representations that become more and more available with modern imaging and reconstruction techniques . To learn beyond sequences , approaches have been developed , that consider the 3D structure of proteins . A range of methods has sampled protein structures to regular volumetric 3D representations and assessed the quality of the structure ( Derevyanko et al. , 2018 ) , classified proteins in enzymes classes ( Amidi et al. , 2017 ) , predicted the protein-ligand binding affinity ( Ragoza et al. , 2017 ) and the binding site ( Jiménez et al. , 2017 ) , as well as the contact region between two proteins ( Townshend et al. , 2019 ) . While this is attractive , as 3D grids allow for unleashing the benefits of all approaches developed for 2D images , such as pooling and multi-resolution techniques , unfortunately , grids do not scale well to fine structures or many atoms , and even more importantly , they do not consider the primary and secondary structure of proteins . Another approach that makes use of a protein ’ s 3D structure , is representing proteins as graphs and applying GCNNs ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ) . Works based on this technique represent each amino acid as a node in the graph , while edges between them are created if they are at a certain Euclidean distance . This approach has been successfully applied to different problems . Classification of protein graphs into enzymes , for example , have become part of the standard data sets used to compare GCNN architectures ( Gao & Ji , 2019 ; Ying et al. , 2018 ) . Moreover , other works with similar architectures have predicted protein interfaces ( Fout et al. , 2017 ) , or protein structure quality ( Baldassarre et al. , 2020 ) . However , GCNN approaches suffer from over-smoothing , i. e. , indistinguishable node representations after stacking several layers , which limits the maximum depth usable for such architectures ( Cai & Wang , 2020 ) . It is also worth noticing that some of the aforementioned works on GCNN have considered different levels of protein structures indirectly by providing secondary structure type or distance along the sequence as initial node or edge features . However , these are not part of the network architecture and can be blended out due to the over-smoothing problem of GCNN . On the other hand , a recent protein function prediction method proposed by Gligorijevic et al . ( 2019 ) uses Long-Short Term Memory cells ( LSTM ) to encode the primary structure and then apply GCNNs to capture the tertiary structure . Also , the recent work from Ingraham et al . ( 2019 ) proposes an amino acid encoder that can capture primary and tertiary structures in the context of protein generative models . Unfortunately , none of these previous methods can incorporate all structural protein levels within the network architecture . 3 MULTI-GRAPH PROTEIN REPRESENTATION . To simultaneously take into account the primary , secondary , and tertiary protein structure during learning , we propose to represent proteins as a multi-graph G = ( N , F , A , B ) . In this graph , atoms are represented as nodes associated with their 3D coordinates , N ∈ Rn×3 , and associated features , F ∈ Rn×t , n being the number of atoms , and t the number of features . Moreover , A ∈ Rn×n and B ∈ Rn×n are two different adjacency matrices representing the connectivity of the graph . Elements of matrix A are defined as Aij = 1 if there is a covalent bond between atom i and atom j , and Aij = 0 otherwise . Similarly , the elements of matrix B are defined as Bij = 1 if there is a covalent or hydrogen bond between atom i and atom j , and Bij = 0 otherwise . 3.1 INTRINSIC-EXTRINSIC DISTANCES Differential geometry of surfaces ( Pogorelov , 1973 ) defines intrinsic geometric properties as those , that are invariant under isometric mappings , i.e. , under deformations preserving the length of curves on a surface . On the other hand , extrinsic geometric properties are dependent on the embedding of the surfaces into the Euclidean space . Analogously , in our protein multi-graph , we define intrinsic geometric properties as those that are invariant under deformations preserving the length of paths along the graph , i.e. , deformations that preserve the connectivity of the protein . Additionally , we define extrinsic geometric properties as those , that depend on the embedding of the protein into the Euclidean space , i.e. , on the 3D protein conformation . Using this terminology , we define three distances in our multi-graph , one extrinsic and two intrinsic ( see Fig . 2 ) . The extrinsic distance τe is defined by the protein conformation in Euclidean space , therefore we use the Euclidean distance between atoms , which enables us to capture the tertiary and quaternary structures of the protein . The intrinsic distances are inherent of the protein and independent of the actual 3D conformation . For the first intrinsic distance τi1 we use the shortest path between two atoms along the adjacency matrix A of the graph , capturing the primary structure . The second intrinsic distance τi2 is defined as the shortest path between two atoms along the adjacency matrix B , capturing thus the secondary structure .

Summary: This paper describes a deep learning architecture for representing and performing classifications on protein structures.  The representation involves three different distances: Euclidean distance and the shorted path between two atoms, where edges are either along covalent bonds or also include hydrogen bonds.  Each atom has a vector of associated features, and convolution is accomplished by defining a kernel on all three distances and then summing the features of each neighboring atom, weighted by the kernel value.  The paper also proposes three protein-specific pooling operations to cope with the large input size when representing all atoms in a protein.