Design in the Dark: Learning Deep Generative Models for De Novo Protein Design

1 INTRODUCTION . Generative modelling is beginning to be used for the task of designing protein molecules , a problem which offers potential solutions to a vast number of medical ( Chevalier et al. , 2017 ; Silva et al. , 2019 ) and industrial challenges ( King et al. , 2012 ; Wang et al. , 2021 ) . Computational protein design methods look to efficiently generate large numbers of candidate sequences , prior to laboratory validation , that are confidently predicted to have stable and ordered structures together with prespecified structural and functional attributes . Recent generative modelling studies have looked to improve on the high time and computational cost of contemporary design methods , which rely on Monte Carlo sampling and energy function-based physics simulations ( Huang et al. , 2016 ) . Here , our aim is to develop a tool , based on a deep generative model , that can rapidly generate sequences with stable and ordered structures . Furthermore , we build this tool for the important task of de novo protein design . Computational de novo protein design is arguably the most promising and general approach to protein design , but also the most difficult ( Huang et al. , 2016 ; Korendovych & DeGrado , 2020 ) . As an approach , it looks to design proteins unlike those seen in nature , with desired structural or functional attributes . Crucially , this is done without using any information from a preexisting protein as a starting point , scaffold , or guide.1 Instead , the designer must rely on general design methods with little , if any , provided information to design suitable sequences with suitable predicted structures ( Woolfson , 2021 ; Marcos et al. , 2018 ; Vorobieva et al. , 2021 ) . This presents a remarkably challenging design setting , evident in that de novo designs have yet to be successfully generated for a number of common but biotechnologically important folds , like immunoglobins . A tool that quickly generates sequences , unlike those seen in nature , with a diverse range of ordered structures is of great value as it can be used to both rapidly generate candidates directly for design tasks as well as provide a basis for developing further novel design methods . In this work , we develop deep generative sequence models that we show as satisfying this specification . Unlike previous generative protein design approaches that typically rely on natural sequences and structurebased conditioning information , we build an approach for de novo design by developing a framework for training unconditional autoregressive language models on synthetic sequences . We call 1The opposite of such an approach is protein engineering , an important but distinctly different field we do not cover here . However , for clarity , an example of protein engineering in a machine learning context would be fine-tuning a pre-trained language model on existing proteins with the desired attributes . this framework Design in Areas of Restricted Knowledge ( DARK ) 2 . DARK iteratively generates large datasets of synthetic sequences that are optimized in an unsupervised way to have unspecified ordered structures that are highly likely under a supervised protein structure predictor ( Figure 1 ) . We build this tool as an unconditional model to avoid limiting its potential applications and also to show that it is possible to learn general relationships between sequence and structure without the aid of conditioning information . We evaluate our approach by first demonstrating that DARK models satisfy our criteria for a de novo design tool . In particular , we show that the final model , DARK3 , generates novel sequences with ordered structures as judged by state-of-the-art structure predictor AlphaFold ( V2 ; Jumper et al . ( 2021 ) ) . We also test the structural generalization of our approach using a stringent unseen structure-based test set and find DARK models performs well . We demonstrate the applications of DARK3 as our tool with an example of designing a sequence with fold commonly used as a de novo scaffold for grafting functional sites . Finally , we show that DARK models enable the development of new methods by developing AlphaFold refinement , a novel and efficient approach for producing high confidence design candidates with AlphaFold . We also use it to de novo design a sequence with a high confidence immunoglobin fold . In summary , we make the following contributions : ( 1 ) We show that unconditional generative models of protein sequences can learn distributions that capture general structure information by learning from synthetic sequences . ( 2 ) We propose a novel framework , DARK , for efficiently training deep generative models on synthetic protein sequences . ( 3 ) We provide the final DARK3 model as a flexible and fast tool for de novo design that generates novel sequences with diverse ordered structures . ( 4 ) We demonstrate AlphaFold refinement , a novel way to efficiently de novo design high confidence sequence candidates using AlphaFold and DARK . 2 RELATED WORK . Here , we discuss related work applying generative models to protein design as well as those that have focused on de novo design , and broader contemporary protein design methods . Generative sequence modelling in protein design There have been a variety of machine learning approaches ( Killoran et al. , 2017 ; Wang et al. , 2018 ; Norn et al. , 2021 ) including generative modelling ( Anand & Huang , 2018 ; Sabban & Markovsky , 2020 ; Linder et al. , 2020 ) in protein design and protein engineering ( Yang et al. , 2018 ; Sinai & Kelsic , 2020 ) . Unconditional and conditional generative models have been explored in a variety of problems adjacent to protein design , such as variant prediction ( Riesselman et al. , 2018 ; Shin et al. , 2021 ) . Other examples are recent work using 2This is inspired by the huge ‘ dark ’ areas of the sequence space that are unexplored by nature ( Taylor et al. , 2009 ; Perdigão et al. , 2015 ) . language models to learn representations for downstream supervised tasks ( Alley et al. , 2019 ; Bepler & Berger , 2019 ; Elnaggar et al. , 2020 ; Rao et al. , 2021 ) . Most relevant to the work presented here is the set of models that have been developed to explore design using generative models conditioned on structural information . Although a range of architectures have been used , these models vary primarily in the representations of structural information used for conditioning . Amongst others , models have conditioned on low information descriptions of protein folds ( weak conditioning ) ( Greener et al. , 2018 ; Karimi et al. , 2020 ) , secondary structure labels ( Singer et al. , 2021 ) , and course descriptions of protein backbones ( Ingraham et al. , 2019 ; Strokach et al. , 2020 ; Cao et al. , 2021 ) .Some methods train proteins of one structural or functional type , effectively conditioning on that specific type ( Yu & Buehler , 2020 ; Shin et al. , 2021 ) . Conditioning out structure provides a way to address an inherent challenge to working with natural protein sequences . The vast majority of known natural proteins have either a weak signal for structure or it is mixed in with other information , such as functional constraints and evolutionary drift . Unconditional generative models , trained on natural sequences , fail to generalize to sequence with structures beyond those represented by their training sets because of this lack of structure signal ( Ingraham et al. , 2019 ) . This is also well understood when viewed from the lens of protein structure prediction . Even highly accurate structure predictions models like AlphaFold ( Jumper et al. , 2021 ) rely on the information contained in aligned sets of sequences similar to the query sequence to achieve accurate predictions for natural sequences . With only individual natural sequences , performance tends to be poor ( Moffat & Jones , 2021 ; Xu et al. , 2021 ) . Generative Models and de novo protein design A generative sequence model for de novo design needs to be highly general , learning general sequence and structure relationships . All considered , we find it unsurprising that there has been no clear and unambiguous demonstration of a deep generative model , without any aid from conditioning , that generates sequences unlike natural sequences and , crucially , with confidently predicted ordered structures . Here , we demonstrate such a model . We achieve this by learning from syntheticsequences with a strong structure signal . Previously de novo designed sequences are known to have extremely strong structure signals , but just over 100 de novo designed sequences ( Woolfson , 2021 ) have lab-determined structures in the Protein Data Bank ( PDB ) ( Burley et al. , 2021 ) . Thus , we find an alternative approach which is to generate synthetic sequences by leveraging a sequence optimization objective from the recent trDesign method ( Anishchenko et al. , 2020 ) . Like contemporary approaches ( Leaver-Fay et al. , 2011 ) , it is primarily a Monte Carlo sampling method , however in contrast , it slowly samples from a trained supervised structure predictor , using simulated annealing to maximize the structure likelihood of random sequences . The final sequences were shown to be extremely similar to pre-existing de novo designed sequences in their attributes and to have a strong structure signal . Contemporary de novo protein design Although recent nonparametric methods have been developed ( Zhou et al. , 2020 ) , by far the most historically successful de novo design approach is the Rosetta framework ( Leaver-Fay et al. , 2011 ) , which is the recognized gold-standard ( Huang et al. , 2016 ) . The Rosetta approach constitutes a complex computational pipeline that uses a variety of different protocols to iteratively sample viable structures and sequences using physics-based force-field simulations and Monte Carlo search procedures ( Vorobieva et al. , 2021 ; Marcos et al. , 2018 ) making it lengthy and computationally expensive . New techniques that can rapidly produce quality samples , such as the work presented here , offer the potential to greatly reduce these costs . For further discussion regarding de novo design , we point the reader to the following reviews and retrospectives by Huang et al . ( 2016 ) , Korendovych & DeGrado ( 2020 ) , and ( Woolfson , 2021 ) . 3 LEARNING DARK MODELS . 3.1 BACKGROUND . Let x ∈ V L be a protein sequence of length L over a vocabulary V of discrete amino acids ( |V | = 20 ) . Each amino acid is textually represented by a one-letter code . Each sequence x maps to a corresponding molecular structure y in three-dimensional space . Protein structure is described by the 3D coordinates of each atom , however more coarse-grained representations are commonly used . Here , we use the discrete representation of structure known as distograms ( Senior et al. , 2020 ) . When using AlphaFold we use its predicted distograms and atomic coordinates , depending on the task ( Jumper et al. , 2021 ) . Distograms represent a structure as a [ L , L , D ] tensor , comprising Ddimensional one-hot encodings of a discretized distance measure between residue pairs . We do not consider the amino acid Cysteine for experimental reasons when generating synthetic sequences3 , however , for consistency we allow all trained models to predict it . For simplicity , we assume a fixed length of L = 100 , however our approach easily generalizes to variable sequence lengths as it is based on autoregressively factorized probability models . We are interested in the set of sequences that are unlike those in nature xd ∈ V L with structures that are folded in some ordered and regular state yd . We would like to sample from their joint distribution , which is convenient to break down , by chain rule , to p ( xd , yd ) = p ( yd|xd ) p ( xd ) , a common way to consider the protein design problem . The first term p ( yd|xd ) is often thought of as protein structure prediction and so it is convenient to model it with some accurate proxy regression model pβ ( yd|xd ) , being a trained supervised structure predictor , with parameters β ∈ B . We refer to this proxy regression model as an oracle . In this work , we focus on modeling the prior , p ( xd ) as the tool we desire , which , when combined with AlphaFold , provides a rapid means to produce high quality sequence and structure samples for de novo design problems .

Summary: I think it helps to start by summarizing [Anischenko et al. 2020, Protein design by deep network hallucination], and then explain the differences in the submission.  [Anischenko et al. 2020] aim to generate AA sequences that are diverse but that still reflect the properties of training structures. * Their goal is to generate protein sequences whose structures tend to be unlike those of random sequences. Since random sequences have structures that tend to be unordered, they incidentally generate sequences having ordered structures. * They use the deep resnet from rtRosetta (similar to AlphaFold's) to predict interresidue geometry maps from sequence, and that is how structure is represented and compared. * Their design goal is to design samples from p(structure | sequence) p(sequence). * To represent p(structure | sequence) they use a proxy objective: the KL divergence between the predicted structures and the structures of random sequences; in other words, good sequences are those whose distance maps are very different from the distance maps of random sequences. * To represent p(sequence) they just use -KL(AA frequences of predicted sequences || AA frequences of PDB sequences). * Sequences are designed by starting from random and then annealing according to the proxy objective. * Designed sequences are then clustered by template modeling score (TM-score) and selected (from each cluster) based on  a measure of prediction 'consistency'.   Now, the submitted manuscript (DARK) is the same objective and proxy objective as [Anischenko et al. 2020], but with following methodological changes: 1. For the sequence->structure oracle, instead of using rtRosetta's interresidue geometry maps as the structure representation, here distograms [Senior et al. 2020] are used, and are output from a modified version of DMPfold2. 2. Rather than relying solely on one round of "random initialization -> annealing", DARK uses an iterative self-training scheme. First they do one pass of [Anischenko et al. 2020] to get 1st round of annealed sequences. The annealed sequences from 1st round are used to train an autoregressive generative model (Transformer-based). That model is sampled from to generate additional "random initializations" (which are now guided, not purely random). These guided samples are then optimized (just hill-climbing now instead of annealing, for speed), and added to the training set for subsequent rounds, etc.  There are other differences from [Anischenko et al. 2020], like using pLDDT score (orderedness) as a measure of "success", but the above two seem to be the main differences. The first of these two is worth trying but is not a conceptual contribution. The main idea of the paper is the second: self-training to generate diversified 'intermediate' initializations for further refinement. Basically it's using the initial refined sequences as a self-prior for exploring new sequences to refine.