Fold2Seq: A Joint Sequence(1D)-Fold(3D) Embedding-based Generative Model for Protein Design

1 INTRODUCTION . Computational protein design is the conceptual inverse of the protein structure prediction problem , and aims to infer an amino acid sequence that will fold into a given 3D structure . Designing protein sequences that will fold into a desired structure has a broad range of applications , from therapeutics to materials ( Kraemer-Pecore et al. , 2001 ) . Despite significant advancements in methodologies as well as in computing power , inverse protein design still remains challenging , primarily due to the vast size of the sequence space - and the difficulty of learning a function that maps from the 3D structure space to the sequence space . Earlier works rely mostly on energy minimization-based approaches ( Koga et al. , 2012 ; Rocklin et al. , 2017 ; Huang et al. , 2011 ) , which follow a scoring function ( force fields , statistical potentials , or machine learning ( ML ) models , ) and sample both sequence and conformational space . Such methods often suffer from drawbacks such as low accuracy of energy functions or force-fields ( Khan & Vihinen , 2010 ) and low efficiency in sequence and conformational search ( Koga et al. , 2012 ) . Recently , as the data on both protein sequences ( hundreds of millions ) and structures ( a few hundreds of thousands ) is quickly accumulating , data-driven approaches for inverse protein design are rapidly emerging ( Greener et al. , 2018 ; Karimi et al. , 2020 ; Ingraham et al. , 2019 ) . Generally , data-driven protein design , attempts to model the probability distribution over sequences conditioned on the structures : P ( x|y ) , where x and y are protein sequences and structures , respectively . Two key challenges remain : ( 1 ) defining a good representation ( y ) of the protein structure and ( 2 ) modelling the sequence generation process conditioned on y . Current protein design methods use protein backbone information from a single protein structure ( fixed backbone ) or from a set of backbone structures consistent with a single fold ( flexible backbone ) . In earlier studies , the protein structures are represented as a 1D string ( Greener et al. , 2018 ) , a 1D vector ( Karimi et al. , 2020 ) , a 2D image ( Strokach et al. , 2020 ) , or a graph ( Ingraham et al. , 2019 ) . The sequence generation methods used in the protein design studies can be classified as non-autoregressive ( Karimi et al. , 2020 ; Greener et al. , 2018 ; Strokach et al. , 2020 ) and autoregeressive ( Ingraham et al. , 2019 ; Madani et al. , 2020 ) . In non-autoregerssive methods , y is usually concatenated with a Gaussian random noise z ( which is the latent vector of the sequence ) please check here to be the input to a sequence generator P ( x|y ) = fg ( y , z ) , while in autoregressive methods , P ( x|y ) is decomposed through the chain rule : P ( x|y ) =∏n i=1 P ( xi|x1 , x2 , ... , xi−1 , y ) , where x = ( x1 , x2 , ... , xn ) . In this paper , we focus on designing sequences conditioned on a protein fold . A protein fold is defined as the arrangement ( or topology ) of the secondary structure elements of the protein relative to each other ( Hou et al. , 2003 ) . A secondary structural element can be defined as the three dimensional form of local segments of a protein sequence . Protein folds are therefore necessarily based on sets of structural elements that distinguish domains . As protein structure is inherently hierarchical , the complete native structure can have multiple folds and a fold can be present in many protein structures . A single structure ( fixed backbone ) or an ensemble of structures ( flexible backbone ) can be used as representatives of a fold . The ensemble representation is often a better choice , as it captures the protein dynamics . Despite the recent progress in using ML models for protein design , significant gaps still remain in addressing both aforementioned challenges ( fold representation and conditional sequence generation ) . First , the current fold representation methods are either hand-designed , or constrained and do not capture the complete original fold space ( See Sec . 2.2 for details ) , resulting in low generalization capacity or efficiency . Second , the sequence encoding and the fold encoding are learned separately in previous methods , which makes two latent domains heterogeneous . Such heterogeneity across two domains actually increases the difficulty of learning the complex sequence–fold relationship . To fill the aforementioned gaps , the main contributions of this work are as follows : • We propose a novel fold representation , through first representing the 3D structure by the voxels of the density of secondary structures elements ( SSEs ) , and then learning the fold representation through a transformer-based structure encoder . Compared to previous fold representations , this representation has several advantages : first , it preserves the entire spatial information of SSEs . Second , it does not need any pre-defined rules , so that the paramterized fold space is not neither limited or biased toward any particular fold . Third , the representation can be automatically extracted from a given protein structure . Lastly , the density model also loosens the rigidity of structures so that the structural variation and lack of structural information of the protein is better handled . • We employ a novel joint sequence-fold embedding learning framework into the transformer-based auto-encoder model . By learning a joint latent space between sequences and folds , our model , Fold2Seq , mitigates the heterogeneity between two different domains and is able to better capture the sequence-fold relationship , as reflected in the results . • Experiments on standard test sets demonstrate that Fold2Seq has superior performance on perplexity , native sequence recovery rate , and native structure recovery accuracy , when compared to competing methods including the state-of-the-art RosettaDesign and other neural net models . Ablation study shows that this superior performance is directly attributed to our algorithmic innovations . Experiments on real-world test sets further demonstrates the unique practical utility and versatility of Fold2Seq compared to the structure-based baselines . 2 RELATED WORK . Data-driven Protein Design A significant surge of protein design studies that deeply exploit the data through modern artificial intelligence algorithms has been witnessed in the last two years . Greener et al . ( 2018 ) used the conditional variational autoencoder for generating protein sequences conditioned on a given fold . Karimi et al . ( 2020 ) developed a guided conditional Wasserstein Generative Adversarial Networks ( gcWGAN ) also for inverse fold design . Madani et al . ( 2020 ) trained an extreme large ( 1.2B parameters ) language model conditioned on taxonomic and keyword tags such as molecular functions for generating protein sequences . Ingraham et al . ( 2019 ) developed a graph-based transformer for generating protein sequences conditioned on a fixed backbone . Lastly , Strokach et al . ( 2020 ) formulated the inverse protein design as a constraint satisfaction problem ( CSP ) and applied the graph neural networks for generating protein sequences conditioned on the residue-residue distance map that is a static representation of the structure . Protein Fold Representation For an extensive overview of molecular representations , including those of proteins , please see David et al . ( 2020 ) . Murzin et al . ( 1995 ) and Orengo et al . ( 1997 ) manually classified protein structures in a hierarchical manner based on their structural similarity . These classifications can be regarded as one-hot encoding of the fold representations . Taylor ( 2002 ) represents a protein fold using a “ periodic table ” . This representation was later used for inverse fold design ( Greener et al. , 2018 ) . However , it only considers three pre-defined folds ( ( layer , layer and barrel ) for a set of structures , which significantly limits the spatial information content of the fold . Hou et al . ( 2003 ) chose hundreds of representative proteins and calculated the similarity scores among them . This similarity matrix was then used for kernel Principle Component Analysis ( kPCA ) . A similar idea was used in Karimi et al . ( 2020 ) for inverse protein design . This representation needs a pre-defined set ( ( alpha only , beta only , and alpha+beta ) of structures along with a similarity metric . Such representation could lead to biased or constrained representations of the fold space and also may not preserve the detailed spatial information of the fold . Finally , Koga et al . ( 2012 ) summarized three rules that describe the junctions between adjacent secondary structure elements for a specific fold . Again , these rules are hand designed for a subset of structures , which makes the representation restricted to a small part of the fold space and offers limited generalizability during conditional sequence generation . Joint Embedding Learning Joint embedding learning across multiple different data modalities was first proposed by Ngiam et al . ( 2011 ) on audio and video signals . Since then , such approaches have been then widely used in cross modal retrieval or captioning tasks ( Arandjelovic & Zisserman , 2018 ; Gu et al. , 2018 ; Peng & Qi , 2019 ; Chen et al. , 2018 ; Wang et al. , 2013 ; Dognin et al. , 2019 ) . In few/zero-shot learning , joint feature-label embedding was used for predicting the label of instances belonging to unseen classes ( Zhang & Saligrama , 2016 ; Socher et al. , 2013 ) . Several papers have demonstrated that learning joint embedding is useful for the single modal classification tasks ( Ngiam et al. , 2011 ; Wang et al. , 2018 ; Toutanova et al. , 2015 ) . Moreover , Chen et al . ( 2018 ) used joint embedding learning for text to shape generation . Lastly , Joint sequence-label embedding is also applied for molecular prediction/generation ( Cao & Shen , 2020 ; Das et al. , 2018 ) . 3 METHODS . 3.1 BACKGROUND AND NOTATION . A protein is formed by a linear chain of amino acids ( residues ) that defines its 1D sequence . Chemical nature , as well as physical and chemical interactions with neighboring residues drive the folding of a sequence into different secondary structure elements or SSEs ( helix , beta-sheet , loop , etc. , see Fig 1 ( a ) ) , that eventually forms a complete native 3D structure . A protein fold captures the topology and the composition of secondary structure elements , thus serving as an intermediate between the 1D sequence and the full 3D structure . 3.2 STRUCTURE REPRESENTATION THROUGH 3D VOXELS OF THE DENSITY OF SSES . In this subsection , we describe how we represent the 3D structure to explicitly capture the fold information . We denote the position ( 3D coordinate ) of each residue by its alpha carbon . For a given protein with length N , we first translate the structure to match its center of mass with the origin of the coordinate system . We then rotate the protein around the origin to let the first residue be on the negative side of z-axis . We denote the resulting residue coordinates as c1 , c2 , ... , cN . We assign the secondary structure label to each residue based on their SSE assignment ( Kabsch & Sander , 1983 ) in Protein Data Bank ( Berman et al. , 2000 ) . We consider 4 types of secondary structure labels : helix , beta strand , loop and bend/turn . In order to consider the distribution of different secondary structure labels in the 3D space , we discretize the 3D space into voxels , as shown in Fig 1 . A technical challenge here is that the sizes of different proteins vary drastically . As we are only considering the arrangement of SSEs , not their exact coordinates , we here rescale the original structure , so that it fits into a fixed-size cubic box . Based on the distribution of sizes of single-chain proteins in the CATH database ( Sillitoe et al. , 2019 ) , we choose a 40Å × 40Å × 40Å box with each voxel of size 2Å× 2Å× 2Å . We denote the scaling ratio as r ∈ R3 . For voxel i , we denote the coordinates of its center as vi . We assume that the contribution of residues j to voxel i follows the Gaussian form : yij = exp ( − ||cj r − vi||22 σ2 ) · tj , ( 1 ) where tj ∈ { 0 , 1 } 4 is the one-hot encoding of the secondary structure label of amino acid j . The standard deviation is chosen to be 2Å . We sum up all residues together to obtain the final features of the voxel i : yi = ∑N j=1 yij . The structure representation y is the vector of yi ’ s over all voxels . In the next subsection , we will describe how we encode y into the latent space and how we learn a joint sequence-fold embedding in order for generating sequences consistent with a desired 3D structure .

Summary: This paper tackle the challenge of designing protein sequences that are consistent with a given 3D fold. To address this challenge, the authors propose a transformer-based generative framework that designs protein sequences conditioned on a given fold. There are two central contributions - the first is a novel fold representation, in which the 3D structure is represented by the voxels of secondary structure elements, and then a fold representation is learned via a transformer-based structure encoder. The second is a joint sequence-fold embedding learning framework. The authors use ablation studies to show that learning a joint latent space between sequences and folds enables the model to better capture the the sequence-fold relationship, improving experimental results. The authors provide a good summary of related work.