Fold2Seq: A Joint Sequence(1D)-Fold(3D) Embedding-based Generative Model for Protein Design

1 INTRODUCTION . Computational protein design is the conceptual inverse of the protein structure prediction problem , and aims to infer an amino acid sequence that will fold into a given 3D structure . Designing protein sequences that will fold into a desired structure has a broad range of applications , from therapeutics to materials ( Kraemer-Pecore et al. , 2001 ) . Despite significant advancements in methodologies as well as in computing power , inverse protein design still remains challenging , primarily due to the vast size of the sequence space - and the difficulty of learning a function that maps from the 3D structure space to the sequence space . Earlier works rely mostly on energy minimization-based approaches ( Koga et al. , 2012 ; Rocklin et al. , 2017 ; Huang et al. , 2011 ) , which follow a scoring function ( force fields , statistical potentials , or machine learning ( ML ) models , ) and sample both sequence and conformational space . Such methods often suffer from drawbacks such as low accuracy of energy functions or force-fields ( Khan & Vihinen , 2010 ) and low efficiency in sequence and conformational search ( Koga et al. , 2012 ) . Recently , as the data on both protein sequences ( hundreds of millions ) and structures ( a few hundreds of thousands ) is quickly accumulating , data-driven approaches for inverse protein design are rapidly emerging ( Greener et al. , 2018 ; Karimi et al. , 2020 ; Ingraham et al. , 2019 ) . Generally , data-driven protein design , attempts to model the probability distribution over sequences conditioned on the structures : P ( x|y ) , where x and y are protein sequences and structures , respectively . Two key challenges remain : ( 1 ) defining a good representation ( y ) of the protein structure and ( 2 ) modelling the sequence generation process conditioned on y . Current protein design methods use protein backbone information from a single protein structure ( fixed backbone ) or from a set of backbone structures consistent with a single fold ( flexible backbone ) . In earlier studies , the protein structures are represented as a 1D string ( Greener et al. , 2018 ) , a 1D vector ( Karimi et al. , 2020 ) , a 2D image ( Strokach et al. , 2020 ) , or a graph ( Ingraham et al. , 2019 ) . The sequence generation methods used in the protein design studies can be classified as non-autoregressive ( Karimi et al. , 2020 ; Greener et al. , 2018 ; Strokach et al. , 2020 ) and autoregeressive ( Ingraham et al. , 2019 ; Madani et al. , 2020 ) . In non-autoregerssive methods , y is usually concatenated with a Gaussian random noise z ( which is the latent vector of the sequence ) please check here to be the input to a sequence generator P ( x|y ) = fg ( y , z ) , while in autoregressive methods , P ( x|y ) is decomposed through the chain rule : P ( x|y ) =∏n i=1 P ( xi|x1 , x2 , ... , xi−1 , y ) , where x = ( x1 , x2 , ... , xn ) . In this paper , we focus on designing sequences conditioned on a protein fold . A protein fold is defined as the arrangement ( or topology ) of the secondary structure elements of the protein relative to each other ( Hou et al. , 2003 ) . A secondary structural element can be defined as the three dimensional form of local segments of a protein sequence . Protein folds are therefore necessarily based on sets of structural elements that distinguish domains . As protein structure is inherently hierarchical , the complete native structure can have multiple folds and a fold can be present in many protein structures . A single structure ( fixed backbone ) or an ensemble of structures ( flexible backbone ) can be used as representatives of a fold . The ensemble representation is often a better choice , as it captures the protein dynamics . Despite the recent progress in using ML models for protein design , significant gaps still remain in addressing both aforementioned challenges ( fold representation and conditional sequence generation ) . First , the current fold representation methods are either hand-designed , or constrained and do not capture the complete original fold space ( See Sec . 2.2 for details ) , resulting in low generalization capacity or efficiency . Second , the sequence encoding and the fold encoding are learned separately in previous methods , which makes two latent domains heterogeneous . Such heterogeneity across two domains actually increases the difficulty of learning the complex sequence–fold relationship . To fill the aforementioned gaps , the main contributions of this work are as follows : • We propose a novel fold representation , through first representing the 3D structure by the voxels of the density of secondary structures elements ( SSEs ) , and then learning the fold representation through a transformer-based structure encoder . Compared to previous fold representations , this representation has several advantages : first , it preserves the entire spatial information of SSEs . Second , it does not need any pre-defined rules , so that the paramterized fold space is not neither limited or biased toward any particular fold . Third , the representation can be automatically extracted from a given protein structure . Lastly , the density model also loosens the rigidity of structures so that the structural variation and lack of structural information of the protein is better handled . • We employ a novel joint sequence-fold embedding learning framework into the transformer-based auto-encoder model . By learning a joint latent space between sequences and folds , our model , Fold2Seq , mitigates the heterogeneity between two different domains and is able to better capture the sequence-fold relationship , as reflected in the results . • Experiments on standard test sets demonstrate that Fold2Seq has superior performance on perplexity , native sequence recovery rate , and native structure recovery accuracy , when compared to competing methods including the state-of-the-art RosettaDesign and other neural net models . Ablation study shows that this superior performance is directly attributed to our algorithmic innovations . Experiments on real-world test sets further demonstrates the unique practical utility and versatility of Fold2Seq compared to the structure-based baselines . 2 RELATED WORK . Data-driven Protein Design A significant surge of protein design studies that deeply exploit the data through modern artificial intelligence algorithms has been witnessed in the last two years . Greener et al . ( 2018 ) used the conditional variational autoencoder for generating protein sequences conditioned on a given fold . Karimi et al . ( 2020 ) developed a guided conditional Wasserstein Generative Adversarial Networks ( gcWGAN ) also for inverse fold design . Madani et al . ( 2020 ) trained an extreme large ( 1.2B parameters ) language model conditioned on taxonomic and keyword tags such as molecular functions for generating protein sequences . Ingraham et al . ( 2019 ) developed a graph-based transformer for generating protein sequences conditioned on a fixed backbone . Lastly , Strokach et al . ( 2020 ) formulated the inverse protein design as a constraint satisfaction problem ( CSP ) and applied the graph neural networks for generating protein sequences conditioned on the residue-residue distance map that is a static representation of the structure . Protein Fold Representation For an extensive overview of molecular representations , including those of proteins , please see David et al . ( 2020 ) . Murzin et al . ( 1995 ) and Orengo et al . ( 1997 ) manually classified protein structures in a hierarchical manner based on their structural similarity . These classifications can be regarded as one-hot encoding of the fold representations . Taylor ( 2002 ) represents a protein fold using a “ periodic table ” . This representation was later used for inverse fold design ( Greener et al. , 2018 ) . However , it only considers three pre-defined folds ( ( layer , layer and barrel ) for a set of structures , which significantly limits the spatial information content of the fold . Hou et al . ( 2003 ) chose hundreds of representative proteins and calculated the similarity scores among them . This similarity matrix was then used for kernel Principle Component Analysis ( kPCA ) . A similar idea was used in Karimi et al . ( 2020 ) for inverse protein design . This representation needs a pre-defined set ( ( alpha only , beta only , and alpha+beta ) of structures along with a similarity metric . Such representation could lead to biased or constrained representations of the fold space and also may not preserve the detailed spatial information of the fold . Finally , Koga et al . ( 2012 ) summarized three rules that describe the junctions between adjacent secondary structure elements for a specific fold . Again , these rules are hand designed for a subset of structures , which makes the representation restricted to a small part of the fold space and offers limited generalizability during conditional sequence generation . Joint Embedding Learning Joint embedding learning across multiple different data modalities was first proposed by Ngiam et al . ( 2011 ) on audio and video signals . Since then , such approaches have been then widely used in cross modal retrieval or captioning tasks ( Arandjelovic & Zisserman , 2018 ; Gu et al. , 2018 ; Peng & Qi , 2019 ; Chen et al. , 2018 ; Wang et al. , 2013 ; Dognin et al. , 2019 ) . In few/zero-shot learning , joint feature-label embedding was used for predicting the label of instances belonging to unseen classes ( Zhang & Saligrama , 2016 ; Socher et al. , 2013 ) . Several papers have demonstrated that learning joint embedding is useful for the single modal classification tasks ( Ngiam et al. , 2011 ; Wang et al. , 2018 ; Toutanova et al. , 2015 ) . Moreover , Chen et al . ( 2018 ) used joint embedding learning for text to shape generation . Lastly , Joint sequence-label embedding is also applied for molecular prediction/generation ( Cao & Shen , 2020 ; Das et al. , 2018 ) . 3 METHODS . 3.1 BACKGROUND AND NOTATION . A protein is formed by a linear chain of amino acids ( residues ) that defines its 1D sequence . Chemical nature , as well as physical and chemical interactions with neighboring residues drive the folding of a sequence into different secondary structure elements or SSEs ( helix , beta-sheet , loop , etc. , see Fig 1 ( a ) ) , that eventually forms a complete native 3D structure . A protein fold captures the topology and the composition of secondary structure elements , thus serving as an intermediate between the 1D sequence and the full 3D structure . 3.2 STRUCTURE REPRESENTATION THROUGH 3D VOXELS OF THE DENSITY OF SSES . In this subsection , we describe how we represent the 3D structure to explicitly capture the fold information . We denote the position ( 3D coordinate ) of each residue by its alpha carbon . For a given protein with length N , we first translate the structure to match its center of mass with the origin of the coordinate system . We then rotate the protein around the origin to let the first residue be on the negative side of z-axis . We denote the resulting residue coordinates as c1 , c2 , ... , cN . We assign the secondary structure label to each residue based on their SSE assignment ( Kabsch & Sander , 1983 ) in Protein Data Bank ( Berman et al. , 2000 ) . We consider 4 types of secondary structure labels : helix , beta strand , loop and bend/turn . In order to consider the distribution of different secondary structure labels in the 3D space , we discretize the 3D space into voxels , as shown in Fig 1 . A technical challenge here is that the sizes of different proteins vary drastically . As we are only considering the arrangement of SSEs , not their exact coordinates , we here rescale the original structure , so that it fits into a fixed-size cubic box . Based on the distribution of sizes of single-chain proteins in the CATH database ( Sillitoe et al. , 2019 ) , we choose a 40Å × 40Å × 40Å box with each voxel of size 2Å× 2Å× 2Å . We denote the scaling ratio as r ∈ R3 . For voxel i , we denote the coordinates of its center as vi . We assume that the contribution of residues j to voxel i follows the Gaussian form : yij = exp ( − ||cj r − vi||22 σ2 ) · tj , ( 1 ) where tj ∈ { 0 , 1 } 4 is the one-hot encoding of the secondary structure label of amino acid j . The standard deviation is chosen to be 2Å . We sum up all residues together to obtain the final features of the voxel i : yi = ∑N j=1 yij . The structure representation y is the vector of yi ’ s over all voxels . In the next subsection , we will describe how we encode y into the latent space and how we learn a joint sequence-fold embedding in order for generating sequences consistent with a desired 3D structure .

Summary: This manuscript presents a method for generating protein sequences conditioned on protein structures. The core idea is to represented protein structures by their secondary structures in 3D space. This voxel grid is then encoded into a vector representation and decoded to a distribution over sequences. The authors propose to learn this model jointly with a sequence encoder, combining the sequence and structure representations to decode the sequence during training. For inference, the sequence encoder component is not used. Learning to generate protein sequence conditioned on structure is an interesting and important problem and has been attracting increasing attention from the ML community. Representing structures as voxel grids is an approach worth exploring and flexible structure representations could be promising. However, it isn’t clear to me that this work achieves those goals and comparisons against key baselines (namely Ingraham 2019) are missing. Furthermore, the authors make many unsupported and unsubstantiated claims about their method. Specific comments and questions follow below.