Design in the Dark: Learning Deep Generative Models for De Novo Protein Design

1 INTRODUCTION . Generative modelling is beginning to be used for the task of designing protein molecules , a problem which offers potential solutions to a vast number of medical ( Chevalier et al. , 2017 ; Silva et al. , 2019 ) and industrial challenges ( King et al. , 2012 ; Wang et al. , 2021 ) . Computational protein design methods look to efficiently generate large numbers of candidate sequences , prior to laboratory validation , that are confidently predicted to have stable and ordered structures together with prespecified structural and functional attributes . Recent generative modelling studies have looked to improve on the high time and computational cost of contemporary design methods , which rely on Monte Carlo sampling and energy function-based physics simulations ( Huang et al. , 2016 ) . Here , our aim is to develop a tool , based on a deep generative model , that can rapidly generate sequences with stable and ordered structures . Furthermore , we build this tool for the important task of de novo protein design . Computational de novo protein design is arguably the most promising and general approach to protein design , but also the most difficult ( Huang et al. , 2016 ; Korendovych & DeGrado , 2020 ) . As an approach , it looks to design proteins unlike those seen in nature , with desired structural or functional attributes . Crucially , this is done without using any information from a preexisting protein as a starting point , scaffold , or guide.1 Instead , the designer must rely on general design methods with little , if any , provided information to design suitable sequences with suitable predicted structures ( Woolfson , 2021 ; Marcos et al. , 2018 ; Vorobieva et al. , 2021 ) . This presents a remarkably challenging design setting , evident in that de novo designs have yet to be successfully generated for a number of common but biotechnologically important folds , like immunoglobins . A tool that quickly generates sequences , unlike those seen in nature , with a diverse range of ordered structures is of great value as it can be used to both rapidly generate candidates directly for design tasks as well as provide a basis for developing further novel design methods . In this work , we develop deep generative sequence models that we show as satisfying this specification . Unlike previous generative protein design approaches that typically rely on natural sequences and structurebased conditioning information , we build an approach for de novo design by developing a framework for training unconditional autoregressive language models on synthetic sequences . We call 1The opposite of such an approach is protein engineering , an important but distinctly different field we do not cover here . However , for clarity , an example of protein engineering in a machine learning context would be fine-tuning a pre-trained language model on existing proteins with the desired attributes . this framework Design in Areas of Restricted Knowledge ( DARK ) 2 . DARK iteratively generates large datasets of synthetic sequences that are optimized in an unsupervised way to have unspecified ordered structures that are highly likely under a supervised protein structure predictor ( Figure 1 ) . We build this tool as an unconditional model to avoid limiting its potential applications and also to show that it is possible to learn general relationships between sequence and structure without the aid of conditioning information . We evaluate our approach by first demonstrating that DARK models satisfy our criteria for a de novo design tool . In particular , we show that the final model , DARK3 , generates novel sequences with ordered structures as judged by state-of-the-art structure predictor AlphaFold ( V2 ; Jumper et al . ( 2021 ) ) . We also test the structural generalization of our approach using a stringent unseen structure-based test set and find DARK models performs well . We demonstrate the applications of DARK3 as our tool with an example of designing a sequence with fold commonly used as a de novo scaffold for grafting functional sites . Finally , we show that DARK models enable the development of new methods by developing AlphaFold refinement , a novel and efficient approach for producing high confidence design candidates with AlphaFold . We also use it to de novo design a sequence with a high confidence immunoglobin fold . In summary , we make the following contributions : ( 1 ) We show that unconditional generative models of protein sequences can learn distributions that capture general structure information by learning from synthetic sequences . ( 2 ) We propose a novel framework , DARK , for efficiently training deep generative models on synthetic protein sequences . ( 3 ) We provide the final DARK3 model as a flexible and fast tool for de novo design that generates novel sequences with diverse ordered structures . ( 4 ) We demonstrate AlphaFold refinement , a novel way to efficiently de novo design high confidence sequence candidates using AlphaFold and DARK . 2 RELATED WORK . Here , we discuss related work applying generative models to protein design as well as those that have focused on de novo design , and broader contemporary protein design methods . Generative sequence modelling in protein design There have been a variety of machine learning approaches ( Killoran et al. , 2017 ; Wang et al. , 2018 ; Norn et al. , 2021 ) including generative modelling ( Anand & Huang , 2018 ; Sabban & Markovsky , 2020 ; Linder et al. , 2020 ) in protein design and protein engineering ( Yang et al. , 2018 ; Sinai & Kelsic , 2020 ) . Unconditional and conditional generative models have been explored in a variety of problems adjacent to protein design , such as variant prediction ( Riesselman et al. , 2018 ; Shin et al. , 2021 ) . Other examples are recent work using 2This is inspired by the huge ‘ dark ’ areas of the sequence space that are unexplored by nature ( Taylor et al. , 2009 ; Perdigão et al. , 2015 ) . language models to learn representations for downstream supervised tasks ( Alley et al. , 2019 ; Bepler & Berger , 2019 ; Elnaggar et al. , 2020 ; Rao et al. , 2021 ) . Most relevant to the work presented here is the set of models that have been developed to explore design using generative models conditioned on structural information . Although a range of architectures have been used , these models vary primarily in the representations of structural information used for conditioning . Amongst others , models have conditioned on low information descriptions of protein folds ( weak conditioning ) ( Greener et al. , 2018 ; Karimi et al. , 2020 ) , secondary structure labels ( Singer et al. , 2021 ) , and course descriptions of protein backbones ( Ingraham et al. , 2019 ; Strokach et al. , 2020 ; Cao et al. , 2021 ) .Some methods train proteins of one structural or functional type , effectively conditioning on that specific type ( Yu & Buehler , 2020 ; Shin et al. , 2021 ) . Conditioning out structure provides a way to address an inherent challenge to working with natural protein sequences . The vast majority of known natural proteins have either a weak signal for structure or it is mixed in with other information , such as functional constraints and evolutionary drift . Unconditional generative models , trained on natural sequences , fail to generalize to sequence with structures beyond those represented by their training sets because of this lack of structure signal ( Ingraham et al. , 2019 ) . This is also well understood when viewed from the lens of protein structure prediction . Even highly accurate structure predictions models like AlphaFold ( Jumper et al. , 2021 ) rely on the information contained in aligned sets of sequences similar to the query sequence to achieve accurate predictions for natural sequences . With only individual natural sequences , performance tends to be poor ( Moffat & Jones , 2021 ; Xu et al. , 2021 ) . Generative Models and de novo protein design A generative sequence model for de novo design needs to be highly general , learning general sequence and structure relationships . All considered , we find it unsurprising that there has been no clear and unambiguous demonstration of a deep generative model , without any aid from conditioning , that generates sequences unlike natural sequences and , crucially , with confidently predicted ordered structures . Here , we demonstrate such a model . We achieve this by learning from syntheticsequences with a strong structure signal . Previously de novo designed sequences are known to have extremely strong structure signals , but just over 100 de novo designed sequences ( Woolfson , 2021 ) have lab-determined structures in the Protein Data Bank ( PDB ) ( Burley et al. , 2021 ) . Thus , we find an alternative approach which is to generate synthetic sequences by leveraging a sequence optimization objective from the recent trDesign method ( Anishchenko et al. , 2020 ) . Like contemporary approaches ( Leaver-Fay et al. , 2011 ) , it is primarily a Monte Carlo sampling method , however in contrast , it slowly samples from a trained supervised structure predictor , using simulated annealing to maximize the structure likelihood of random sequences . The final sequences were shown to be extremely similar to pre-existing de novo designed sequences in their attributes and to have a strong structure signal . Contemporary de novo protein design Although recent nonparametric methods have been developed ( Zhou et al. , 2020 ) , by far the most historically successful de novo design approach is the Rosetta framework ( Leaver-Fay et al. , 2011 ) , which is the recognized gold-standard ( Huang et al. , 2016 ) . The Rosetta approach constitutes a complex computational pipeline that uses a variety of different protocols to iteratively sample viable structures and sequences using physics-based force-field simulations and Monte Carlo search procedures ( Vorobieva et al. , 2021 ; Marcos et al. , 2018 ) making it lengthy and computationally expensive . New techniques that can rapidly produce quality samples , such as the work presented here , offer the potential to greatly reduce these costs . For further discussion regarding de novo design , we point the reader to the following reviews and retrospectives by Huang et al . ( 2016 ) , Korendovych & DeGrado ( 2020 ) , and ( Woolfson , 2021 ) . 3 LEARNING DARK MODELS . 3.1 BACKGROUND . Let x ∈ V L be a protein sequence of length L over a vocabulary V of discrete amino acids ( |V | = 20 ) . Each amino acid is textually represented by a one-letter code . Each sequence x maps to a corresponding molecular structure y in three-dimensional space . Protein structure is described by the 3D coordinates of each atom , however more coarse-grained representations are commonly used . Here , we use the discrete representation of structure known as distograms ( Senior et al. , 2020 ) . When using AlphaFold we use its predicted distograms and atomic coordinates , depending on the task ( Jumper et al. , 2021 ) . Distograms represent a structure as a [ L , L , D ] tensor , comprising Ddimensional one-hot encodings of a discretized distance measure between residue pairs . We do not consider the amino acid Cysteine for experimental reasons when generating synthetic sequences3 , however , for consistency we allow all trained models to predict it . For simplicity , we assume a fixed length of L = 100 , however our approach easily generalizes to variable sequence lengths as it is based on autoregressively factorized probability models . We are interested in the set of sequences that are unlike those in nature xd ∈ V L with structures that are folded in some ordered and regular state yd . We would like to sample from their joint distribution , which is convenient to break down , by chain rule , to p ( xd , yd ) = p ( yd|xd ) p ( xd ) , a common way to consider the protein design problem . The first term p ( yd|xd ) is often thought of as protein structure prediction and so it is convenient to model it with some accurate proxy regression model pβ ( yd|xd ) , being a trained supervised structure predictor , with parameters β ∈ B . We refer to this proxy regression model as an oracle . In this work , we focus on modeling the prior , p ( xd ) as the tool we desire , which , when combined with AlphaFold , provides a rapid means to produce high quality sequence and structure samples for de novo design problems .

Summary: The authors propose a way to generate novel protein sequences that are likely to have folded structures. To do so, they start with a pre-trained sequence-to-structure predictor, and use simulated annealing to generate a set of initial sequences that are likely to have a structure. These seed sequences are used to train an unsupervised sequence model (an autoregressive transformer). More sequences are generated from this sequence model, and they are optimized to maximize the likelihood they have structures (using the pre-trained sequence-to-structure model), and are added to the growing set of training sequences. This process is repeated for several iterations to finetune the unsupervised sequence model. The authors show that sequences generated from this sequence model are likely to be structured as predicted by AlphaFold.