Design in the Dark: Learning Deep Generative Models for De Novo Protein Design
Authors: Lewis Moffat
Shaun M. Kandathil
David T. Jones
Venue: ICLR 2022 Submitted (2022)
Keywords: generative models
sequence design
language models
proteins

Abstract: The design of novel protein sequences is providing paths towards the development of novel therapeutics and materials. 
Generative modelling approaches to design are emerging and to date have required conditioning on 3D protein structure-derived information, and unconditional models of protein sequences have so far performed poorly.
Thus, it is unknown if unconditional generative models can learn a distribution of sequences that captures structure information without it being explicitly provided, and so be of use in important tasks like de novo protein sequence design, where it is not possible to condition on structure.
Here, we demonstrate that it is possible to use unconditioned generative models to produce realistic samples of protein sequences. 
We progressively grow a dataset of over half a million synthetic sequences for training autoregressive language models, using an iterative framework we call DARK.
It begins by training an autoregressive model on an initial sample of synthetic sequences, sampling from it, and refining the samples thus generated, which are then used for subsequent rounds of training.
Using the confidence measures provided by AlphaFold and other measures of sample quality, we show that our approach matches or exceeds the performance of prior methods that use weak conditioning on explicit structural information, and improves after each iteration of DARK.
Crucially, the DARK framework and the trained models are entirely unsupervised; strong structural signal is an objective, but no model is ever conditioned on any specific structural state.
The trained model indirectly learns to incorporate a structural signal into its learned sequence distribution, as this signal is strongly represented in the makeup of the training set at each step.
Our work demonstrates a way of unconditionally sampling sequences and structures jointly, and in an unsupervised way.

Under review as a conference paper at ICLR 2022
DESIGN IN THE DARK: LEARNING DEEP GENERATIVE MODELS FOR DE Novo PROTEIN DESIGN
Anonymous authors Paper under double-blind review
ABSTRACT
The design of novel protein sequences providing paths towards the development of novel therapeutics and materials. At the forefront is the challenging field of de novo protein design, which looks to design protein sequences unlike those found in nature using general design methodologies. In this work, we develop tool for novo design, based on a deep generative sequence model, that rapidly samples novel protein sequences with diverse and ordered structures. To build this tool we develop a framework, called DARK, that trains the underlying generative model iteratively expanding set of synthetic sequences. The resulting model gen- eralizes where models trained on natural sequences struggle and greatly improves the efficiency of comparable sampling-basee based approaches. We further show how it can generate high quality candidates for de novo design problems and aid in the development of further novel design methods, in all, providing another step, amongst others, towards truly automated and intelligent protein design.
INTRODUCTION
Generative modelling is beginning to be used for the task of designing protein molecules, a problem which offers potential solutions vast number of medical (Chevalier al.) 2017; Silva et al.) 2019) and industrial challenges (King et al., 2012: Wang et al. 2021). Computational protein de- sign methods look efficiently generate large numbers of candidate sequences, prior to laboratory validation, that are confidently predicted to have stable and ordered structures together with pre- specified structural and functional attributes. Recent generative modelling studies have looked to improve on the high time and computational cost of contemporary design methods, which rely on Monte Carlo sampling and energy function-based physics simulations (Huang et al. 2016).
Here, our aim is to develop tool, based on a deep generative model, that can rapidly generate sequences with stable and ordered structures. Furthermore, we build this tool for the important task of de novo protein design. Computational de novo protein design is arguably the most promising and general approach to protein design, but also the most difficult (Huang et al., [016)] Korendovych & DeGrado 2020). As approach, it looks to design proteins unlike those structural or functional attributes. Crucially, this is done without using any information from a pre- existing protein as a starting point, scaffold, guide !
in nature, with desired
Instead, the designer must rely general design methods with little, if any, provided information to design suitable sequences with suitable predicted structures (Woolfson) 2021 |Mrcos et al. 2018 Vorobieva al. 202T). This presents remarkably challenging design setting, evident in that de novo designs have yet to be successfully generated for a number of common but biotechnologically important folds, like immunoglobins.
A tool that quickly generates sequences, unlike those seen in nature, with a diverse range of or- dered structures is of great value it can be used to both rapidly generate candidates directly for design tasks as well as provide basis for developing further novel design methods. In this work, we develop deep generative sequence models that we show as satisfying this specification. Unlike previous generative protein design approaches that typically rely on natural sequences and structure- based conditioning information, we build an approach for de novo design by developing a frame- work for training unconditional autoregressive language models on synthetic sequences. We call
'The opposite of such an approach is protein engineering, an important but distinctly different field we do not cover here. However, for clarity, example of protein engineering in machine learning context would be fine-tuning pre-trained language model on existing proteins with the desired attributes.

---

Under review as a conference paper at ICLR 2022
Refine
orgama Pyls (td)
Refine
argaaax Poly (yy)
Sample
Net
Sample
Sample
AlphaFold Buly Is,)
Train
Train
Prelua
Model
PAS
Sequence
Samples
Samples
Samples
Sample
N=1
2
Figure I: Growing a synthetic dataset and training unconditional language models with DARK. An illus- trative overview of the procedure used by DARK to iteratively grow training of synthetic sequences that used to train increasingly performant generative models. The final structure generation with AlphaFold for sampled sequences also included.
this framework Design Areas of Restricted Knowledge (DARK) DARK iteratively generates large datasets of synthetic sequences that are optimized in an unsupervised way to have unspecified ordered structures that are highly likely under supervised protein structure predictor (Figure [ unconditional model to avoid limiting its potential applications and also to
We build this tool show that it is possible to learn general relationships between sequence and structure without the aid of conditioning information.
We evaluate our approach by first demonstrating that DARK models satisfy our criteria for de design tool. In particular, with ordered structures as judged by state-of-the-ar structure predictor AlphaFold (V2; Jumper al. (202T)) We also test the structural generalization of our approach using stringent unseen structure-based test set and find DARK models performs well. We demonstrate the applications of DARKa as our tool with an example of designing sequence with fold commonly used as de novo scaffold for grafting functional sites. Finally, we show that DARK models enable the development of new methods by developing AlphaFold refinement, novel and efficient approach for producing high confidence design candidates with AlphaFold. We also use it to de novo design sequence with a high confidence immunoglobin fold.
show that the final model, DARK3, generates novel sequences
In summary, we make the following contributions: (1) We show that unconditional generative mod- els of protein sequences can learn distributions that capture general structure information by learning (2) We propose novel framework, DARK, for efficiently training deep generative models on synthetic protein sequences. (3) We provide the final DARK, model as a flex- ible and fast tool for de novo design that generates novel sequences with diverse ordered structures. (4) We demonstrate AlphaFold refinement, novel way to efficiently de novo design high confidence sequence candidates using AlphaFold and DARK.
from synthetic sequences.
RELATED WORK
Here, we discuss related work applying generative models to protein design as well as those that have focused on de novo design, and broader contemporary protein design methods.
Generative sequence modelling in protein design There have been variety of machine learning
approaches (Killoran et al., 2017 Wang et al. [2018) Norn et al. 2021) including generative mod- elling (Anand & Huang 2018 Sabban & Markovsky 2020 Linder et al. 2020) in protein design and protein engineering Yang et al. 20188 Sinai & Kelsic 2020). Unconditional and conditional generative models have been explored
variety of problems adjacent protein design, such as
variant prediction (Riesselman et al. 2018 Shin et al. 2021). Other examples are recent work using
2This is inspired by the huge 'dark' areas of the sequence space that are unexplored by nature Taylor et al. Perdigão al. 2015).
2009
2

---

Under review as a conference paper at ICLR 2022
A
Rapid sequence genecation Alphafisd refinement
Grudy
Geoaly hill casden sequeiic
Drech
Detmising Alphelliit Koseor
Sinalaad hauniity
X6 the prifrasee mo druclar
DARSS Apgaatill
Si,, Arotaling
Stmulated aronaiing ol candoon
AlphaFood
Uptinizing Alpholioad
sequetee
jorfetsacee No dfriceofatttctt
DARK
AlphaFoid
DARE3
Canady hill clib DSM
Optinizing Cross Entroy of Diaogaana
Good perpl some Clabe struntses
5006
1000
2000
anaple
C
Exomple:
g
Method
Ang. LiphaFeid score
Sies Arnsoiing
1.26
LAARCS lefite
2.30
DA RKS AT Roline
Ang Nesporece lcentryy
12.6%
Ang eLO07
1.M
2
03e0 A
094 A
$22
2.6-A
0198
Aug cLDO7 of Rocest Dasgree Stap.
Afve
88.1
Bldtrar AST Relleeeen
Allvv
Figure 2: Generating de novo designed sequences with AlphaFold refinement and DARK;- (A) Overview of our AlphaFold refinement approach. (B) Comparisons of the method and two benchmarks in C. (C) Plot showing the design trajectories for 10 DARK samples with AlphaFold refinement (10K steps) compared to 10 runs the Simulated Annealing benchmark and runs of the Greedy benchmark (both 20K st)ps).(D) Structures and metrics of the 10 AlphaFold refinement examples in C before and after refinement. (E) Averaged metries of the benchmarked methods, with more detailed metries for AlphaFold refinement.
language models to learn representations for downstream supervised tasks (Alley al. 2019 Bepler & Berger 2019 Elnaggar et al. 2020 Rao et al. 2021).
Most relevant the work presented here is the set of models that have been developed explore design using generative models conditioned on structural information. Although range of archi- tectures have been used, these models vary primarily in the representations of structural information used for conditioning. Amongst others, models have conditioned on low information descriptions of protein folds (weak conditioning) (Greener et al., 2018 Karimi et al., 2020), secondary structure labels (Singer et al. 2021), and course descriptions of protein backbones (Ingraham et al. 2019 Strokach et al. 2020; Cao et al. 202T).Some methods train proteins of one structural or functional type, effectively conditioning that specific type Yuu & Buehler 2020 Shin et al. 2021).
Conditioning out structure provides a way to address an inherent challenge to working with natural protein sequences. The vast majority of known natural proteins have either weak signal for struc- ture or it is mixed in with other information, such as functional constraints and evolutionary drift. Unconditional generative models, trained natural sequences, fail to generalize to sequence with structures beyond those represented by their training sets because of this lack of structure signal (Ingraham et al. 2019). This is also well understood when viewed from the lens of protein struc- ture prediction. Even highly accurate structure predictions models like AlphaFold (Jumper et al.) 2021) rely on the information contained in aligned sets of sequences similar to the query sequence to achieve accurate predictions for natural sequences. With only individual natural sequences, per- formance tends to be poor (Moffat & Jones 2021 Xu et al. 2021).
Generative Models and de novo protein design A generative sequence model for de novo design needs be highly general, learning general sequence and structure relationships. All considered, we find it unsurprising that there has been no clear and unambiguous demonstration of a deep generative model, without any aid from conditioning, that generates sequences unlike natural sequences and, crucially. with confidently predicted ordered structures. Here, we demonstrate such a model. We achieve this by learning from syntheticsequences with strong structure signal.
Previously de novo designed sequences are known to have extremely strong structure signals, but just over 100 de novo designed sequences (Woolfson) 2021) have lab-determined structures in the Protein Data Bank (PDB) (Burley et al., 2021). Thus, we find an alternative approach which is to generate synthetic sequences by leveraging a sequence optimization objective from the recent trDe- sign method (Anishchenko et al. 2020). Like contemporary approaches (Leaver-Fay et al.) 2011 1, it is primarily a Monte Carlo sampling method, however in contrast, it slowly samples from a trained supervised structure predictor, using simulated annealing to maximize the structure likelihood of random sequences. The final sequences were shown to be extremely similar to pre-existing de novo designed sequences in their attributes and to have strong structure signal.
3

---

Under review as a conference paper at ICLR 2022
Contemporary de novo protein design Although recent nonparametric methods have been de- veloped (hou et al. 2020), by far the most historically successful de novo design approach is the Rosetta framework (Leaver-Fay et al. 20IT), which is the recognized gold-standard (Huang et al. 2016). The Rosetta approach constitutes a complex computational pipeline that uses a variety of dif- ferent protocols iteratively sample viable structures and sequences using physics-basd based force-field simulations and Monte Carlo search procedures (Vorobieva et al. [021] Marcos et al. (0118) making lengthy and computationally expensive. New techniques that can rapidly produce quality samples, such as the work presented here, offer the potential to greatly reduce these costs. For further discus- sion regarding de novo design, we point the reader to the following reviews and retrospectives by Huang et al. (2016), [Korndovyych & DeGrado (2020), and (Woolfson) (021).
3 LEARNING DARK MODELS
3.1 BACKGROUND
Let x E VL be a protein sequence of length over a vocabulary of discrete amino acids (VV) = 20). Each amino acid is textually represented by one-letter code. Each sequence maps corresponding molecular structure y in three-dimensional space. Protein structure is described by the 3D coordinates of each atom, however more goarse-grained representations are commonly used. Here, we use the discrete representation of structure known as distograms (Senior et al. 2020). When using AlphaFold we use its predicted distograms and atomic coordinates, depending on the task (Jumper et al. 2021). Distograms represent a structure as [L, L, DJ tensor, comprising D- dimensional one-hoo hot encodings of a discretized distance measure between residue pairs. We do not consider the amino acid Cysteine for experimental reasons when generating synthetic sequenced however, for consistency we allow all trained models to predict it. For simplicity, we assume fixed length of L = 100, however approach easily generalizes to variable sequence lengths as it is based on autoregressively factorized probability models.
We are interested in the set of sequences that are unlike those in nature va E VL with structures that are folded in some ordered and regular state Ya- We would like to sample from their joint distribution, which is convenient to break down, by chain rule, to p(zs, ya) common way to consider the protein design problem. The first term 22)) is often thought as protein structure prediction and it is convenient model it with some accurate proxy regression model pee pe(ya(za), being a trained supervised structure predictor, with parameters /3 E B. We refer to this proxy regression model as an oracle. In this work, we focus on modeling the prior, p(ra) as the tool we desire, which, when combined with AlphaFold, provides a rapid means to produce high quality sequence and structure samples for de novo design problems.
3.2 AN UNSUPERVISED DESIGN LOSS FUNCTION
Recently, Anishchenko et al. (2000) demonstrated a method of sampling sequences for de novo design which, using our own notation, also looks to maximize p(yal x2)p( using the trRosetta structure predictor as an oracle (Yang al.] (2020). This samples sequences using simulated an- nealing of random sequences to optimize a combined proxy objective for both terms, ultimately generating sequences with the structure signal we are interested in. Here we describe the proxy objective in our own notation and then how we use it. First, p(yulza) is approximated using the Kullback- Leibllr divergence (DKL) between an oracle and a trained background network p. (ya) which approximates a prior over ordered structures.
arg max log (a (a) ss arg max DKL (ps(valra))lee. at
(1)
The background network is trained by taking model with the same architecture and dataset as (a( and training with the sequence inputs replaced by Gaussian noise") (Appendix A..
The final objective also includes term for approximating weak sequence prior by taking the
This is common in design studies, in part because it significantly complicates otherwise straightforward recombinant protein expression in bacteria, the most relevant factor for us (See Appendix AA7].
'n practice, for a given length, the background model distograms can he calculated once, taking average over arbitrary number of predictions (c.g. 100), and then saved be used for all IG-score calculations.
4

---

Under review as a conference paper at ICLR 2022
negative DKL between the sequence's normalized unigram frequencies p(a""r)) and the unigram frequencies of proteins with high resolution crystal structures in the PDB, p(TPDB). For simplicity, refer to this objective the Information Gain score (IG-score):
IG-score = arg max
DKL (ps(ual
DKL (p(=7 (
(2)
Zd
own version of trDesign, termed DARK,, and correspondingly our own version of the IG-score, using a version of the recent DMPfold2 model, chosen for its fast inference, for the oracle and background network (Kandathil et al. (020), which is modified so that it produces distograms instead of atomic coordinates (Appendix A.I). DARK, is slow, it takes an average of 11.4 minutes to generates a sample. Given the end point of protein design in the lab, performant design model needs to provide as many high confidence candidates as possible to justify the cost of validation as well as combat very high attrition rates. A method like trDesign is too slow, and has less potential future adaptions, to be used as the tool we desire. Instead we show how we can use IG-score to learn generative models that sample similar sequences orders of magnitude faster with many potential upplications. To kickstart the approach described in the next section, use DARKo (40,000 simulated annealing steps) to sample small number of sequences, 15K, that we call the seed samples. It is also used to sample two sets of 950 sequences to be used as validation and test set for models in the next section (See Appendix [B.1 for further details). After this DARK, is discarded as approach, and the only aspect of trDesign that we build on our own version of the IG-score; the rest of what is described here is our own construction.
Here, we implement
3.3 BUILDING THE DARK FRAMEWORK FOR DESIGN
Algorithm
Design in Areas of Restricted Knowledge (DARK)
Input: Oracle model pa (val(a))
Input: Iterations N
Input: Sample sizes at each iteration Mo.N {mo. m,N
1: Fgo +- Generate my seed samples from Pa (Val
2: for 1, .... N do
3:
(.. f- p., (2g(2)m1)) Learn a new sequence model
4: Fz. Ancestral sampling of p-y" (J6.)
m, new samples
5: Tz- Rapid optimization of xg- IG-score using the oracle
6: Tza [ -gn-1] Combine optimized samples with existing samples
7:
end for
DARK (Algorithm]) I tackles the primary limitation on effectively learning models, which is having suitable sequences to train and test against. The first step is to take some initial set of samples 2,, to train an initial sampling model p,, (2) with parameters 71 T. Here we use the aforementioned seed samples, but this could be random sequences sequences from DARKo-Gradd an approach we developed that backpropagates the negative IG-score to generate samples (Appendix (A.2). in fashion similar to (Killoran al. 2017). We the seed samples out of convenience they have the highest score, and we reasoned that they would require less iterations by comparison. The sampling models are all generative sequence models that can be sampled. After training py, (x.0). ). it is used to generate a large set of samples T... Instead of then training a new sampling model on T.n. each sequence in x... quickly optimized to improve the IG-score.
This is done by a quick greedy hill-climb on the sequence for a set number of steps (Line 5 in Algorithm [p. which we refer to refinement. improves the IG-score then it is kept, otherwise it is discarded. We use 3000 steps, fraction of the steps done in DARKs, as this was found improve the average IG-score of xss to approximately equal to the IG-score of aga. For simplicity, this is fixed for all steps. The result of optimizing J.s is then combined with H,50 to make x,,. In the second iteration, x2z is used to train a new sampling model pya (x,1). After some number of iterations N. the final sampling model is viewed as a strong prior over Za, also making it a powerful generative model that can be used for de novo design tasks. Here we perform 3 iterations with training set sizes of 15K, 100K, and 500K, where the difference is sampled and refined between iterations. These sample sizes are arbitrary and they were entirely dictated by computational resources available at the time. Three iterations were performed
each step, a random position is mutated.
5

---

Under review as a conference paper at ICLR 2022
Table 1: Perplexities (PPLs), IG-scores, & Sampling Speed for protein language models. Lower PPL is better and higher IG-score is better. IG-scores are calculated from 1000 sequence samples except the 15K DARKg-Seee Samples. IG-scores for refined (Ref) samples are included for DARK models. Sampling Speed is for a single example (See Appendix for for further details).
Language Model
PPL Test +
IG-score 1 IG-score (Ref) 1
Sample Speed (s)
Unigram
17.69
0.27
Pfam HMM profiles*
11.64*
UF50
12.11
0.21
0.0524
LSTM
9.94
0.92
0.0210
DARK,
6.33
1.38
1.77
0.0234
6,26
1.33
0.0234
DARK2
5.73
1.53
1.81
0.0524
DARKE-STRICT
6.50
1.48
0.0524
DARK3 3
5.20
1.63
1.84
0.0524
DARK -eed Samples
1.76
1.76
685
DARKo-
0.97
15.1
"Values reproduced from rrommingraham et (2019) conditional model conditioned on specific sequence position of specific family.
as we believed it sufficient to demonstrate DARK's effectiveness. We note that the iterative aspect of DARK shares similarities to methods like Estimation of Distribution Algorithms (EDAs), a common approach in model based optimization (MBO) (Bengoetxea et al.) 2001]] Larrañaga & Lozano 2001 Brookes et al., 2020). Viewed through this lens, it suggests a number of ways that DARK can be adapted and potentially improved on, which we leave for future work (Kumar & Levine 2020
Trabucco et al. 202T).
Deep generative architectures In DARK, we use deep autoregressive language models for learn- ing protein sequence distributions. Specifically, we use standard Transformer decoder architecture (Vaswani et al. 2017] Radford et al. 2019) which has been used extensively to train state-of-the-ee language models capable of generating high- synthetic sequence samples. In the first iteration of DARK we use small decoder model, termed DARK1, with 4 layers, 4 heads, and a feed-forward size of H = 128. To affirm the choice of self-attention based architecture, we compare it to a 1- layer LSTM with a hidden dimension H 128. We also include results for variant of DARKi referred to as DARKk-Adversariaa that uses DMPfold2 as an adversarial regularizer during training (Appendix(A.3 A.3). After the first iteration, the second and third iteration models, DARK, and DARK, respectively. use a decoder with 12 layers, 12 heads, and H = 768. The parameters are increased to coincide with the order of magnitude change in the size of the training set from DARK, to DARK2- To provide a even comparison to models trained on natural sequences, we train a model, termed UF50, on 50M natural sequences from the UniRef50 sequence database (The UniProt Consortium, 2021) using the same architecture as DARK; (Appendix A.55). For all trained models we perform stopping and a small amount of hyper-paramett optimization on the validation set.
Quantitatively measuring sample quality We define high-quality samples as those that are di- verse, have sequences unlike any natural sequence, and predicted to have stable and ordered structure. In de novo design, the most important test of a candidate sequence is that it is confi- dently predicted have a stable and ordered atomistic structure (Huang al., 2016). We provide direct measure of both confidence and order jointly using AlphaFold's pLDDT score, being its con- fidence metric, which ranges from 100 (Jumper et al. (021). This is a measure of confidence, and an indirect measure of order. It was found that low scores (pLDDT< 50) strongly suggest disorder and vice versa (Tunyasuvunakool et al. 2021). Thus, we use the proportion of samples with a pLDDT> 70 as measure of high quality for the predicted structures, which we refer to as Good+ pLDDT, indicating "good or better" quality according to criteria in Tunyasuvunakool et al. 22021) When discussing results we use 'ordered" to mean 'confidently predicted and ordered'. Us- ing AlphaFold also provides a check to ensure that the model has not learned to produce adversarial examples against the original DMPfold2 oracle. We also measure the diversity of sequences and predicted structures using the estimated number of clusters in samples of sequences, and samples of
predicted structures (See Appendix B.7 & B.8

---

Under review as a conference paper at ICLR 2022
Table 2: DARK; produces samples with confidently predicted structures by AlphaFold. Good+ pLDDT
scores from AlphaFold (higher better) are shown for unconditional language models. See Table B for addi- tional benchmarks.
Language Model
Good+ pLDDT (%) f
Training Data
Random
1.8
UF50
15.9
Natural
DARKs
32.3
Synthetic
DARK Seed Samples
40.8
Synthetic
Measuring the structural generality of DARK in a strict setting We take two key approaches to providing stringent measure of how well DARK generalizes with regards to structure. Both are enabled by using AlphaFold to generate all-atom structure predictions for all 100,000 sequences contained in the 2 training set, which covers the 15,000 seed samples and the 85,000 samples from the first iteration of DARK. We then assign each sample's ' structure to a topology. This process the repeated for the 950 sequence validation set; we ignore the 950 sequence test set for what is described here. The first approach we take is to carefully construct a sfric/ training and test set split based on structure, from the 100K samples (See Appendix [B.2 for exhaustive details). This has been recently suggested for machine learning-basee protein design studies (Ingraham et al. 2011)) and is considered a gold-standard approach in protein structure prediction (Chothia & Lesk 1986; Söding & Remmert 2011). Our strict test set contains sequences with overlapping topologies in either the training set or validation set. We also remove any sequences from the validation and training set that are detected be similar those in the test set with the MMseqs2 search tool (Steinegger & Söding 2017). We train and evaluate model on these sets, termed DARK2-STRICT. In the limit of working with synthetic, we believe this constitutes a stringent test of generality. The second approach we take is to compare the number of unique topologies represented by the 15,000 seed samples and those represented by the 85,000 sequences from the first iteration of DARK to show that DARK iterations generate sequences with unseen structures.
3.4 USING DARK; FOR DE NOVO DESIGN
Rapid de novo design with DARK We provide example of how DARK can be used in spe- cific design task by setting the challenge of de novo designing 4-helix bundle, which is commonly seen in design studies and in nature (Orengo et al. 1997). 4-helix bundles are also of interest as they are being used as de novo scaffolds for grafting in functional sites (Woolfson 2021 To design a sequence, we use DARK and AlphaFold generate a database of samples and predicted structures. We take the highest PLDDT 4-helix bundle from the dataset and then validate its structure prediction by predicting its structure using the Rosetta Abinitio Relax protocol without any homology informa- tion (Leaver-Fay et al.) 2011) (this took 433 CPU hours). We generate 10,000 structure predictions (decoys), and rank them by lowest Rosetta all-atom score. For this example, we use the set of 1000 unrefined sequences from DARK; as if it were structure database, but this can be expanded to an arbitrary size in practice. 1000 sequences with predicted structures is equivalent to generating 28 sequences with an approach like trDesign by time. Due to its speed, DARK is able to provide large number of potential candidates sequences for different design tasks like this one in an off-the off-the-shelf way.
An novel but simple approach to de novo designing with AlphaFold To demonstrate that DARK models can be used in method development, we use DARK; to develop simple design method we call AlphaFold refinement (Figure A & (BB). This provides a solution the problem of having candidate sequence with structure of interest to designer but it has low confidence. AlphaFold refinement functions by bootstrapping the initial predicted structure as target for per- forming hill-climb based optimization of the DARK; sampled sequence. More specifically, we minimize the Cross-Entropy between the AlphaFold predicted distogram and the one-hot distogram of the bootstrapped target structure. To provide a comparison AlphaFold refinement, we construct a trDesign-basse based approach that optimizes an AlphaFold IG-score score) using simulated an- nealing. AIG-score also provides measure of confidence. As example of its practical use, use AlphaFold refinement to de novo design a sequence with high confidence immunoglobin fold (Figure 3 (See Appendix for exhaustive details).
7

---

Under review as a conference paper at ICLR 2022
4 RESULTS
Here, we assess DARK3 as a tool and provide demonstrations of its use.
4.1 EVALUATING DARK MODELS AS A TOOL FOR DE NOVO PROTEIN DESIGN
Comparing DARK to a simulated annealing approach We use DARKs without refinement as our final design tool and we find that it produces similar samples by IG-score DARK, (seed sam- ples), our version of trDesign, but does so four orders of magnitude faster, 52 ms compared to 11 mins per example (Table []. We also find that the iterative approach to DARK effective for improv- ing performance between iterations and significantly reduces the required resources. Rounding up to the nearest day, if we were to re-perform DARK from nothing to having a trained DARK, would take 12 days when parallelized across ten V100 GPUs. Of that time, model training constitutes just over days and only requires 1 GPU. To generate 500,000 examples with DARK, would take ~88 days across ten V100 GPUs Compared to methods like trDesign, the sample speed of DARK mod- els allows for the generation of large quantities of candidate sequences for different design tasks. With an approach like DARK's, that relies on generative neural networks, there are also clear ways to introduce controlled generation (Dathathri et al. 2019 Keskar et al. 2019).
DARK generates samples with ordered structures We find that DARK3 reliably generates sam- ples with ordered predicted structures (Table 2). Without any refining conditioning, 32.3% of DARK, samples have Good+ pLDDT scores. Given the IG-score of unrefined DARK; samples is slightly lower than that of the DARK,, seed samples [T. it is unsurprising that similar differ- ence is present with the Good+ pLDDT (Table 2 Compared to an approach like trDesign, for a small decrease in Good+ PLDDT, DARK3 is able generate sequences orders of magnitude faster, also giving it much greater flexibility for integrating and developing future approaches.
DARK samples are diverse We find that samples from DARK models have diverse sequences and predicted structures (Table 3 & 4). Across all iterations, clustering results suggest that only small number of samples have similar sequences and structures. That said, clustering is not a very robust measure as it doesn't ' indicate from DARK; and predict their structures to gain a further qualitative insight into the diversity of the structures represented in the samples. There appears to be preference for all-o: and all-8 structures, but a & 3 proteins are still common. We include examples in Figure 5 Looking forward, future investigation is required to properly quantify and examine the structure distribution of sequences from the final model. That said, we do find sequences of design interest in the samples that we have generated. For example, sequences with topologies that have yet to be successfully de novo designed, like immunoglobin folds.
exact degree of diversity. We also generate 5000 sequences
DARK samples are unlike natural sequences We find that samples from the DARK; model are distinct from sequences in nature. A sequence search, using the MMseqs2 search tool (Steinegger & Söding
2017) with high sensitivity parameter (-s 7), against UniRef100 (The UniProt Con- sortium 2021) of the 1000 evaluated sequence samples returned no hits for all sequences except one, which had 5 weak hits. Repeating the sequence search with different tool, HHblits (Remmert let al. 2012), no natural sequences were found that met the HHblits default threshold value, being an E-value below x 10-3 (See Appendix B.7 for further details). This strongly suggests that DARK; generates sequences that are unlike those found in nature. We find the opposite results for the nat- ural sequence model. Repeating the MMseqs2 search with the 1000 samples from the UF50 model results in 41 samples having a combined total of 1.76 million unique hits. Given the UF50 model is trained on natural sequences this is unsurprising. All considered, training synthetic sequences appears to provide a simple means to generate sequences unlike those in nature.
Learning from natural sequences leads to poor performance The UF50 model provides a direct comparison to having trained on natural sequences instead of synthetic sequences, and is similar to protein engineering focused approaches like ProGEN (Madani et al. 2020) and UniRep (Alley et al.) 2019). As expected, it performs poorly by both perplexity and IG-score (Table MD and underperforms compared to DARK, by Good+ PLDDT (Table Especially given the test set comprises synthetic sequences, from naive perspective we would expect UF50 to assign high likelihoods to the test set.
8

---

Under review as a conference paper at ICLR 2022
Not only is its training set two orders of magnitude larger than DARK,'s 's the test set sequences have a clear structure signal, It is also hyperparameter optimized on the synthetic validation set. For the Good+ PLDDT, it is also advantaged many of the sequences in the AlphaFold training set are in its own training set. Ultimately, this result is entirely expected (See Section 2
DARK models generalize to sequences with unseen structures We find that DARK general- izes to sequences with unseen structures. DARK2-S7RIC7, tested on on the strict test set (Section 3.3 & Appendix [..2). performs well and achieves a perplexity of 6.50, similar to the other DARK models and similarly outperforming the baselines (Table D. In assigning high likelihoods to the sequences in the strict test set, DARK2-STRICT provides direct evidence that DARK models are learning general relationships between sequences and structure. This is further supported by how unseen topologies are sampled between DARK iterations (Section [.3)). Within the 100K predicted structures of the DARK2, we find that the initial 15K seed samples contains 421 unique topologies and the 85K set from the first iteration of DARK contains 1011 unique topologies, with an inter- section of 373 between the two. As such, there are 638 topologies sampled during the first DARK iteration that were not present in the 15K training set. This accounts for 1476 sequences, of which 225 sequences (15.2% have 2LDDT> 70. This provides direct evidence that the DARK iterations are expanding the training sets to include sequences with unseen and ordered structures, suggesting a general approach.
4.2 USING DARK2 FOR DE NOVO DESIGN
An example of de nove design with DARK For the task of designing 4-helix bundle (Section
3.4. we find that D successfully produces a candidate, effectively off-the-shelf. The predicted structure of the chosen sequence has high PLDDT of 91.4, similar to the pLDDTT of existing de novo proteins (Appendix [B.4]. We also find that its AlphaFold structure is in agreement with the best ranked predicted structure by Rosetta, having low not-mean-square deviation of 3.11Â (by a-carbons) compared to AlphaFold's prediction (Figure x (). In all, this provides a demonstration of how models like DARK3, paired with AlphaFold, can rapidly provide de novo candidates for tasks like generating scaffolds for grafting in functional sites.
Generating de novo designed sequences with AlphaFold refinement and DARK We find Al- phaFold refinement can reliably generate high confidence (pLDDT 91.0) sequence candidates, while maintaining their structures,. This is slightly better than the confidence (pLDDT 88.1 1) with which AlphaFold predicts the structures of recently released de novo designed sequences in the PDB (Fig- ure 2 & 2)). We also find that AlphaFold refinement is efficient, typically converging by 7500 steps which is 10 hours on consumer-grade graphics card (RTX 2080 Ti). This significantly more efficient than approaches like de novo design with Rosetta which can take thousands of CPU hours. In contrast, we found the simulated annealing approach produced what we consider an ad- versarial attack on AlphaFold (Figure 2 & 4. For the task of de novo designing sequence with an immunoglobin fold, we find that DARK with AlphaFold refinement is also successful, producing a final predicted structure with high PLDDT of 93.1 (Figure 3), demonstrating how DARK can be used to generate candidates for important but yet to be de novo designed folds.
5 CONCLUSION
In this work, we show that it is possible to build a tool, in the form of DARK3, that can rapidly generate sequences with diverse and ordered structures. Looking forward, building a variable length training set and smoothing the topology distribution between iterations are clear first targets as well as exploring related approaches in MBO. Controlled generation is an especially useful route to explore and there is clear way to approach it: AlphaFold structure predictions themselves can be used as pseudo-labels for training future conditional versions of DARK entirely based on synthetic sequences. The AlphaFold refinement approach can also be easily extended to the sub-problem of fixed-bee badd-bbckbone protein design. Looking forward, we hope that DARK, and many others, are continuing provide steps towards intelligent and truly automatic protein design.

---

Under review as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
Left blank for anonymity during double-bling blind peer review.
REPRODUCIRILLIY STATEMENT
In order to guarantee reproduciblity of this work, we provide a detailed appendix with specific implementation and methodological details which is sufficient to reproduce the results in this work. Following release of the paper, we will release both a code repository containing the source code for the DARK models well as the associated scoring functions and analysis scripts. The training data and model parameters will be made available from our servers due to GitHub file size limits.
REFERENCES
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence-based deep representation learning. Nature methods, (6(12):135--122) 2019.
Namrata Anand and Po-Ssu Huang. Generative modeling for protein structures. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pp. 7505-7516, 2018.
Antonina Andreeva, Eugene Kulesha, Julian Gough, and Alexey G Murzin. The scop database in 2020: expanded classification of representative family and superfamily domains of known protein structures. Nucleic acids research, 48(DD))1D37--D332 2020.
Ivan Anishchenko, Tamuka Martin Chidyausiku, Sergey Ovchinnikov, Samuel Pellock, and David Baker. De novo protein design by deep network hallucination. bioRxiv, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.066500 2016.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 13X..v:330.334:: 2013.
Endika Bengoetxea, Pedro Larranaga, Isabelle Bloch, and Aymeric Perchant. Estimation of distri- bution algorithms: A new evolutionary computation approach for graph matching problems. In International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recog- nition, pp. 454-469. Springer, 2001.
Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. arXiv preprint arXiv: 00222,086..2220 2019.
Gaurav Bhardwaj. Vikram Khipple Mulligan, Christopher D Bahl, Jason M Gilmore, Peta J Harvey, Olivier Cheneval, Garry W Buchko, Surya VSRK Pulavarti. Quentin Kaas, Alexander Eletsky, et al. Accurate de novo design of hyperstable constrained peptides. Nature, 538(7625):329-333, 2016.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkir. O' O'Reilly Media, Inc.". 2009.
David Brookes, Akosua Busia, Clara Fannjiang, Kevin Murphy, and Jennifer Listgarten. A view of estimation of distribution algorithms through the lens of expectation-maximzaaiio ings of the 2020 Genetic and Evolutionary Computation Conference Companion, pp. 189-190, 2020.
Proceed-
Stephen K Burley, Charmi Bhikadiya, Chunxiao Bi, Sebastian Bittrich, Li Chen, Gregg V Crichlow, Cole H Christie, Kenneth Dalenberg, Luigi Di Costanzo, Jose Duarte, et al. Resb protein data bank: powerful new tools for exploring 3d structures of biological macromolecules for basic and applied research and education in fundamental biology, biomedicine, biotechnology. bioengineer- ing and energy sciences. Nucleic acids research, 49(D1):D43--D451 2021.
10

---

Under review as a conference paper at ICLR 2022
Yue Cao, Payel Das, Vijil Chenthamarakshan, Pin- Yu Chen, Igor Melnyk. and Yang Shen. Fold2seq: A joint sequence (Id)-fold (3d) embedding babeedding-based generative model for protein design. In Inter- national Conference Machine Learning, pp. 1261-1271. PMLR, 2021.
Aaron Chevalier, Adriel-Adriano Silva, Gabriel J Rocklin, Derrick R Hicks, Renan Vergara, Pa- tience Murapa, Steffen M Bernard, Lu Zhang, Kwok-Ho Lam, Guorui Yao, et al. Massively parallel de novo protein design for targeted therapeutics. Nature, 550(7674):74-79, 2017.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014.
Cyrus Chothia and Arthur M Lesk. The relation between the divergence of sequence and structure in proteins. The EMBO journal, 5(4):823-826, 1986.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosin- ski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv: 1912.02164, 2019.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: towards crack- ing the language of life's s code through self supervisd deep learning and high performance com- 2020.
puting. arXiv preprint arXiv:2007.0622
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 12873-12883, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farke Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Joe Greener, Lewis Moffat, and David T Jones. Design of metalloproteins and novel protein folds using variational autoencoders. Scientific reports, 8(1):1-12, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Po-Ssu Huang, Scott E Boyken, and David Baker. The coming of age of de novo protein design. Nature, (37(()00))3230--33333 327, 2016.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. Advances in Neural Information Processing Systems, 32:15820-15831, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztiio with gumbel-softmax. arXiv preprint arXiv:1611.01 144, 2016.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature. 596(7873):583-58, 2021.
Shaun M Kandathil, Joe Greener, Andy M Lau, and David T Jones. Deep learning- predic- tion of protein structure using learned representations of multiple sequence alignments (version 1). Biorxiv, pp. 2020-11, 2020.
Mostafa Karimi, Shaowen Zhu, Yue Cao. and Yang Shen. De novo protein design for novel folds using guided conditional wasserstein generative adversarial networks. Journal of Chemical Infor- mation and Modeling, 60(12):567756611 2020.
A. Katharopoulos. A. Vyas, N. Pappas, and F. Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In Proceedings of the International Conference on Machine Learning (ICMI), 2020. 2020.
11

---

Under review as a conference paper at ICLR 2022
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint 199.:::9099 2019. 2019.
Nathan Killoran, Leo J Lee, Andrew Delong, David Duvenaud, and Brendan Frey. Generating and designing dna with deep generative models. arXiv preprint arXiv:1712.06144, 2017.
Neil P King, William Sheffler, Michael R Sawaya, Breanna Vollmar, John P Sumida, Ingemar André, Tamir Gonen, Todd O Yeates, and David Baker. Computational design of self-assembling protein nanomaterials with atomic level accuracy. Science, 336(6085): 2012.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6991 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv: 1312.611 2013.
arXiv preprint
Ivan V Korendovych and William F DeGrado. De novo protein design, retrospective. reviews of biophysics, 53, 2020.
Quarterly
Aviral Kumar and Sergey Levine. Model inversion networks for model-based optimization. Ad- vances in Neural Information Processing Systems, 33, 2020.
Pedro Larrañaga and Jose A Lozano. Estimation of distribution algorithms: new tool for evolu- lionary computation, volume 2. Springer Science & Business Media, 2001.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using a learned similarity metric. In International conference on machine learning, pp. 3558-5566 PMLR, 2016.
Andrew Leaver-Fay. Michael Tyka, Steven M Lewis, Oliver F Lange, James Thompson, Ron Jacak, Kristian W Kaufman, P Douglas Renfrew, Colin A Smith, Will Sheffler, et al. Rosetta3: an object- oriented software suite for the simulation and design of macromolecules. Methods in enzymology, 487:545-574 574, 2011.
Johannes Linder, Nicholas Bogard, Alexander B Rosenberg, and Georg Scelig. A generative neural network for maximizing fitness and diversity of synthetic dna and protein sequences. Cell Systems, 11(1):49-62 2020.
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint ar arXiv:2004.03499 2020.
Enrique Marcos, Tamuka M Chidyausiku, Andrew C McShan, Thomas Evangelidis, Santrupti Nerli, Lauren Carter, Lucas G Nivón, Audrey Davis, Gustav Oberdorfer, Konstantinos Tripsianes, et al. De novo design of non-local ß-sheet protein with high stability and accuracy. Nature structural & molecular biology, (5(11))0028-100 2018.
Milot Mirdita, Lars von den Driesch, Clovis Galiez, Maria J Martin, Johannes Söding, and Martin Steinegger. Uniclust databases of clustered and deeply annotated protein sequences and align- ments. Nucleic acids research, 45(DI):D170-D -176. 2017.
Lewis Moffat and David Jones. Increasing the accuracy of single sequence prediction methods using deep semi- supervisd learning framework. Bioinformatics. 37(21):37443375) 2021.
Christoffer Norn Basile IM Wicky, David Juergens, Sirui Liu, David Kim, Doug Tischer, Brian Koepnick, Ivan Anishchenko, David Baker, and Sergey Ovchinnikov. Protein sequence design by conformational landscape optimization. Proceedings of the National Academy of Sciences, 118 (11), 2021.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation leamn ing. arXiv preprint 1 17.00077 2017.
12

---

Under review as a conference paper at ICLR 2022
Christine A Orengo, Alex D Michie, Susan Jones, David T Jones, Mark B Swindells, and Janet M Thorton. Cath-a hierarchic classification of protein domain structures. Structure, 508):1093- 1109, 1997.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32: 8026-8037, 2019.
Nelson Perdigão, Julian Heinrich, Christian Stolte, Kenneth S Sabir, Michael J Buckley, Bruce Tabor, Beth Signal, Brian S Gloss, Christopher J Hammang, Burkhard Rost, et al. Unexpected features of the dark proteome. Proceedings of the National Academy of Sciences, 112(52):15898- 15903, 2015.
Eric F Pettersen, Thomas D Goddard, Conrad C Huang, Elaine C Meng, Gregory S Couch, Tris- tan Croll, John H Morris, and Thomas E Ferrin. Uesf chimerax: Structure visualization for researchers, educators, and developers. Protein Science, 30(1):70- 82, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, lly Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Roshan Rao, Jason Liu, Robert Verkuil, Joshua Meier, John F Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. bioRxiv, 2021.
Michael Remmert, Andreas Biegert, Andreas Hauser, and Johannes Söding. Hhblits: lightning-fast iterative protein sequence searching by hmm-hmm alignment. Nature methods, 9(2):173-175, 2012.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286 PMLR, 2014.
Adam J Riesselman, John B Ingraham, and Debora S Marks. Deep generative models of genetic variation capture the effects of mutations. Nature methods, 15(10)8816 822, 2018.
Sari Sabban and Mikhail Markovsky. Ramanet: Computational de novo helical protein backbone design using long short-term memory generative neural network. F1000Research, 9(298)2298, 2020.
Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Žídek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature, 577(7792):706-710) 2020.
Jung-Eun Shin, Adam J Riesselman, Aaron W Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew Kruse, and Debora Marks. Protein design and variant prediction using autoregressive generative models. Nature communications, 12(1):1-11, 2021.
Danicl-Adriano Silva, Shawn Yu, Umut Ulge, Jamie Spangler, Kevin M Jude, Carlos Labão- Almeida, Lestat R Ali, Alfredo Quijano-Rubio, Mikel Ruterbusch, Isabel Leung, et al. De novo design of potent and selective mimics of il-2 and il-15. Nature, 1657777))) 2019.
Sam Sinai and Eric D Kelsic. A primer on model-guided exploration of fitness landscapes for biological sequence design. arXiv preprint arXiv:2010. 10614, 2020.
Jedediah M Singer, Scott Novotney, Devin Strickland, Hugh K Haddox, Nicholas Leiby, Gabriel Rocklin, Cameron M Chow, Anindya Roy, Asim K Bera, Francis C Motta, et al. Large-scale design and refinement of stable proteins using sequence-only models. bioRxiv, 2021.
Johannes Söding and Michael Remmert. Protein sequence comparison and fold recognition: progress and ppacticee benchmarking. Current opinion in structural biology, 21(3):404-44 411, 2011.
13

---

Under review as a conference paper at ICLR 2022
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, 1lya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1955 2014.
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology. 15(11)2606--1 2017.
Alexey Strokach, David Becerra, Carles Corbi-Verge Albert Perez-Riba, and Philip M Kim. Fast and flexible design of novel proteins using graph neural networks. BioRxiv, pp. 868935, 2020.
William R Taylor, Vijayalakshmi Chelliah, Siv Midtun Hollup, James T MacDonald, and Inge Jonassen. Probing the "dark matter" of protein fold space. Structure, ((9)):2244-252, 2009.
The UniProt Consortium. Uniprot: the universal protein knowledgebase 2021. Nucleic Acids Research, 49(D1)) 2021.
Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective mod- els for effective offline model-basee optimization. In International Conference Machine Learn- ing, pp. 10358-10368. PMLR, 2021.
Kathryn Tunyasuvunakool, Jonas Adler, Zachary Wu, Tim Green, Michal Zielinski, Augustin Židek, Alex Bridgland, Andrew Cowie, Clemens Meyer, Agata Laydon, al. Highly accurate protein structure prediction for the human proteome. Nature, 596(7873):590-596, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 1998-6008, 2017.
Anastassia Vorobieva, Paul White, Binyong Liang, Jim E Horne, Asim K Bera, Cameron M Chow, Stacey Gerben, Sinduja Marx, Alex Kang, Alyssa Q Stiving, et al. De novo design of transmembrane barrels. Science, 371(6531), 2021.
Fenghin Wang, Ordy Gnewou, Charles Modlin, Leticia C Beltran, Chunfu Xu, Zhangli Su, Puneet Juneja, Gevorg Grigoryan, Edward H Egelman, and Vincent Conticello. Structural analysis of cross a-helical nanotubes provides insight into the designability of filamentous peptide nanoma- terials. Nature communicationns, 12(1):1-14, 2021.
Jingxue Wang, Huali Cao. John ZH Zhang, and Yifei Qi. Computational protein design with deep learning neural networks. Scientific reports, 8(1):1-9, 2018.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. arXiv preprint TTt.:I9000043999 2019.
Derek N Woolfson. A brief history of de novo protein design: minimal, rational, and computational. Journal of Molecular Biology, pp. 167160, 2021.
Jinbo Xu, Matthew Mcpartion, and Jin Li. Improved protein structure prediction by deep learning irrespective of co-evolution information. Nature Machine Intelligence, pp. 1-9, 2021.
Janyy Yang, Ivan Anishchenko, Hahnbeom Park, Zhenling Peng, Sergey Ovchinnikov, and David Baker. Improved protein structure prediction using predicted interresidue orientations. Proceed- ings of the National Academy of Sciences, 117(3):1496-) 2020.
Kevin K Yang, Zachary Wu, and Frances H Arnold. Machine learning in protein engineering, arXiv preprint
2018
Chi-Hua Yu and Markus J Buchler. Sonification based de novo protein design using artificial intel- ligence, structure prediction, and analysis using molecular modeling. API. bioengineering, 4(1): 016108, 2020.
Yang Zhang and Jeffrey Skolnick. Tm-align: a protein structure alignment algorithm based on the tm-score. Nucleic acids research, 33(7):2302-2309 2005.
Jianfu Zhou, Alexandra E Panaitiu, and Gevorg Grigoryan. A general-purpor protein design frame- work based on mining sequence-structure relationships in known protein structures. Proceedings of the National Academy of Sciences, 1059))) 2020.
14

---

Under review as a conference paper at ICLR 2022
APPENDIX
A IMPLEMENTATION AND TRAINING DETAILS
A.1 ORACLE MODEL AND BACKGROUND MODEL
We use the DMPfold2 model for the Pss (y(z)) oracle (Kandathil et al. 2020). We received the trained model as well as the training and validation data described in Kandathil et al. (2220) via correspon- dence with the authors. We note that the code and pre-print for more recent but non-probabilistic version appears to be available but we refer here to the CASP14 version of DMPfold2 described in the first pre- pre-print (Kandathil et al. 2020), which outputs distograms.
DMPfold2 was chosen as an oracle as it reports favourable performance and, importantly, its ability to perform fast inference directly from input alignment, or in case, single sequence. It uses a combination of two GRUs (Cho et al. (2014) followed by residual network to predict distograms from multiple sequence alignment. The input is of shape [A, L, 22] where the A dimension the number of sequences in the alignment. First, a stack of unidirectional GRUs reduces A to a fixed size, resulting in an output array sized [T., 512), that is fed as input to stack of bidirectional GRUs that produces the final representation of the input alignment. This representation is then fed to convolutional ResNet, which produces the final outputs (see below).
We refactored the code for the DMPfold2 model in order to perform batched inference on individ- ual sequences. We find that the IG-score for a single sequence (L 100) can be calculated in approximately 0.06s. For batch of 100 sequences this reduces to 0.01s per sequence.
Distogram outputs from DMPfold2 From a single input sequence of length L, DMPfold2 pro- duces an output tensor of size [L, L, 104]. The 104D third axis is split into four separate discrete distributions which are individually softmaxed, being sized [L, L, and three of [L, 34]. The output tensor contains predictions of residue-residue hydrogen bonding, residue-residue f-carbon (CB) distance distributions (distograms), and backbone and torsion angle distributions. See Kandathil et al. (2020) for further details.
DMPfold2 Background Model To implement the IG-score we train a background network, Pe Pe(a), as per Anishchenko al. (2020), using the DMPfold2 architecture and author-provided training data. This model is distinguished with the subscript Training the background model constituted replacing the input sequences during training with 64D Gaussian noise, and altering the input layer accordingly, but otherwise following the training procedure described in the DMPfold2 paper. We pre-calculate the background values for a given sequence length by feeding noise to the network 100 times and taking the average of the predictions, leaving a final tensor of size [L, L, 104] containing the resulting predictions. Although this can be calculated on the fly, for simplicity we pre-calculate and save them once. As we focus on = 100, this amounts to saved array of size [100, 100, 104].
Using DMPfold2 to calculate the IG-score The IG-score is calculated by first predicting the output tensor for sequence with DMPfold2. Next, DKL is calculated between each of the four distograms in the tensor and their corresponding pre-calculated background versions. This results in four [T,, T., 1] tensors, one for each distogram, containing the scalar DKL. values. For each tensor, the mean DEL. is taken across both dimensions, resulting in four scalars. These four DKL. values are added together with equal weighting. Finally, the IG-score this value minus the DKL between the unigram frequency distribution in the predicted sequence and the corresponding distribution in high resolution protein structures from the PDB. This results a single scalar DKL. The latter values are sourced from the publicly available trDesign code repository
15

---

Under review as a conference paper at ICLR 2022
A.2 DARKo- SAMPLING SEQUENCES FROM DMPFOLD2 WITH GRADIENT DESCENT
We wished to investigate whether the seed sequences we use in DARK could be generated by a minimizing the negative IG-loss predicted by DMPfold2 with respect to the input sequence. We do so implementing and testing an approach similar to those shown by Killoran et al. (2017) and Norn al. (.021). but instead using DMPfold2. We begin with an input of shape [B, L. 22 where the first dimension is the number of sequences in the minibatch. To keep compute time results comparable to the other methods presented here we use sequence of L = 100. The version of DMPfold2 we use is altered to predict distograms for batches of sequences under the constraint that the alignment is always of size I, which is not a constraint for us here.
100. The input itself is a one-hot encoded random
To begin optimization, we compute the distograms from the batch of input sequences, backpropagate the negative IG-score our loss, and then update the original input [B, L, 22] vector. This is one step. As we are back propagating through a discrete node, and the updated input vector is no longer discrete, we do the following in all optimization steps to make the operation differentiable. We softmax the updated input vector (we refer to this now as a profile) in the final dimension and then sample sequences from the softmaxed residue distributions using a straight-through estimator (Bengio et al. 2013).
We perform 200 of the above steps as we found it sufficient for the negative IG-scores to converge. Instead of sampling final sequences from the final profiles, we track and keep the sampled sequence that achieves the highest IG-score during the 200 steps. We explored a variety of optimizers, profile regularization and normalization approaches, and other variants with the aim of improving the av- erage final IG-score. Despite this, we generally found Adam (Kingma & Ba 2014) with a learning rate of I, default 3 parameters, and temperature of 1.0 on the sotumax to be the most consistently effective approach. To calculate the average IG-score of this method, and the average sample time, we generate 1000 sequences from the described approach. These resulting metrics can be seen in Table
A.3 DARK
As part of our investigations we were motivated to assess the effect of incorporating structure pre- dictor directly into the training of our autoregressive DARK models. There has been a range of recent work (Larsen et al. 2016; Esser et al. 2021) 1 exploring the combination of learned and fixed dis- criminators, like the learned descriminator in generative adversarial networks (GANs) (Goodfellow) let al. 2014), with generative autoencoding models like variational autoencoders (VAEs) (Kingma & Welling 2013: Rezende et al. 2014), vector quantized VAFS (VQ-VAFs) (Oord et al.] (017). and many others. Although this topic has mostly been explore with latent variable models, we wishes to investigate whether such an approach would benefit our autoregressive approach. We note however that there are potentially problems with this approach which we discuss below.
To explore this model, we began with the full training procedure and architecture of the DARK, model. We then incorporated DMPfold2 by, during a forward pass in training. taking the output probabilities for the batch in question and sampling one-hot sequences combined with a straight- through estimator like in Appendix A.2 The negative IG-score is then calculated as a loss, LIG- We see using DMPfold2 in this way as a kind of adversarial regularizer, thus we termed it DARK,- Adversarial. It must be noted that the DMPfold2 parameters are not being updated in this procedure, the model is effectively frozen. We calculate the final loss L as,
Lyimal = LAR + 7LIG
(3)
where LAR is the original autoregressive cross-entropy loss and y a tunable hyperparameter. We performed a small search of y values and found = 0.5 to perform well however this was not rigorously optimized. As can be seen in Table [ the approach does not perform significantly differ- ently from the original DARK, model. We in part suspect that this is due to the DMPfold2 model simply reinforcing the loss signal from the cross-entropy against the original sequence; the original sequence is already optimized to have a high IG-score. A less charitable view is that the gradient returning to the decoder model from DMPfold2 is simply contributing noise. As mentioned, there is also potential issue with the method itself. being that by backpropagating through DMPfold2, which sees the full sampled sequence, we are potentially exposing the decoder to gradient infor- mation from the full sequence, breaking the autoregressive nature of the model. Given DMPfold2
16

---

Under review as a conference paper at ICLR 2022
is frozen, and did not model collapse during training, we do not believe this is occurring, however, we highlight it as necessitating further empirical investigation.
A.4 AUTOREGRESSIVE MODELS
We build the autoregressive neural network models used in this work with PyTorch (Paszke al. 2019). We also use several utilities from the NLTK (Bird et al., (2099) package. For all neural network architectures we do mini-batch gradient descent against negative log likelihood (NLL) loss. We use the Adam optimizer (Kingma & Bal 2014) with default / values of 0.9 & 0.999. Early stopping performed using the validation set. batch size of 64 is used with learning rate of 1 x 10-3 for the RNN baseline, and 1 x 10- for self-attention models. The learning rate was not changed according to a schedule as was not found to provide significant benefit, and so was left static. All training was done with automatic mixed precision as it brought significant improvement in computational efficiency. Below we describe the architectures for the best RNN baseline model and for the self-attention models.
Inputs For all language models trained on synthetic data, the input is a batch of B sequences in an array sized [B, L). This contains 22 integers corresponding to 20 amino acids, a start token, and a stop token. The length L is the length of the protein sequence with a start token placed at the beginning of the sequence. As is standard for autoregressive models, the target sequence used for calculating the loss is the protein sequence with stop token placed at the end. All neural networks use embedding layers that use these as integer indices for embedding vectors learned during training.
RNN architecture The best RNN model uses an embedding layer, learned during training, of size 64. The embedded sequence of size [B, L, 64 is fed into an LSTM model with a hidden size H of 128 and layer (Hochreiter & Schmidhuber 1997). The hidden vector output at each time step of the LSTM then fed to a linear layer (H 28) - ReLU activation + linear layer (H = 22). The resulting final array, sized [B, L, 22], is then log log-softmaxed.
The Big and Small Xmfr models use a common architecture but with different numbers of parameters. We describe the architecture using the sizes of the Big model first. An embedding layer of size 384 is concatenated with a positional encoding of size 384 (Vaswani et al. 2017), resulting in an array of size [B, L, 784]. The positional encoding uses maximum length of 101. In comparison, the Small model uses an embedding size of 64, so the input to the following decoder has 128 dimensions in the final axis. We use standard decoder architecture for autoregres- sive multi-headed self-attention with causal masking ( (Vaswani et al., [017] Radford et al.) 2019). This includes 10% Dropout (Srivastava et al.,20014 and the use of Layer Normalization (Ba al.) 2016). We use the Fast Transformers library (Katharopoulos et al., (020) for implementing models. The Big model uses 12 layers with H = 784 and 12 heads, and the Small model uses layers with H = 128 and 4 heads. For both models the output of the decoder goes into linear layer (!f 128) + ReLU activation > linear layer (H- = 22). The output is produced is also a [B, L, 22] array that rof-softm
Xfmr architecture
For both RNN language models and self- language models, additional decoding strategies beyond simple autoregressive sampling like Top-K sampling (Welleck al., 2019) were not found to significantly improve results. We mask out cysteine when sampling from DARK models, however we include it in the sizing of the input embeddings and output size to maintain consistency with other approaches. The output array of size 22 accounts for the 20 amino acids, a start, and stop token. Further investigation of more complex decoding schemes, as well as the use of different positional encoding methods and types of attention, are potential avenues for future research.
A.5 TRAINING AN UNCONDITIONAL MODEL WITH NATURAL SEQUENCES
Here we describe further architecture and training details of the 'UF50' model used to benchmark the DARK models produced in this work. This model provides a direct comparison of training with natural sequences and training with synthetic proteins from DARK. UF50 is an unconditional language model that uses the same training procedure and architecture the DARK, model, with some modifications, and trained on sequences from nature. Specifically, the model is trained on the representative sequences for each cluster in the UniRef50 database (The UniProt Consortium,
17

---

Under review as a conference paper at ICLR 2022
yielding training set of 50,106,395 sequences. UF50 provides a benchmark for approach similar to the ProGEN model (Madani et al. 2020). if ProGEN was an unconditional model. Pro- GEN is conditioned on a variety of evolutionary and functional information for the task of protein engineering, making it not directly comparable to the work performed here.
2021),
We now describe the modifications used in the UF50 model. During training, missing amino acids and non-canonical amino acids are mapped to an additional 'UNK' token which does not contribute the loss. Thus, the final vocabulary size is 24. Batches are dynamically padded to the length of the longest sequence, and padded positions are masked for the loss calculation. To reduce the GPU memory required for long sequences, any sequences longer than 100 amino acids (start/stop not included) are clipped to 100. For the target sequence, instead of placing a stop token at position 101, the amino acid sequence in position 101 from the original non-clipped sequence is used. this way, due to the causal masking, this is exactly the same information available to the model as if it was not using clipped sequences, within the range of 101 amino acids. As all evaluated sequences are of length 100, the clipping has no effect on any conclusions drawn. To provide a level playing field, UF50 uses the same validation set as DARK models. We also use rejection sampling to sample 1000 sequences of length 100 for evaluation and comparison. Additional decoding strategies like Top-K K sampling and placing a temperature factor on the next-token softmax were explored but not found to aid performance.
A.6 FURTHER COMPUTATIONAL EFFICIENCY AND HARDWARE DETAILS
We report all timing estimations, unless stated otherwise, based on using an RTX 2080 Ti graphics card on the same single machine. We report all sample speed timings in seconds per example. Sample timing estimations were calculated with batches of 100 sequences, and calculated on the same single machine. The only example were this was not the case was for DARKo-Grad which was performed in batches of 10 due to memory constraints. Model training and sample refinement was performed with resources available at the time but these typically consisted of V100 graphics cards. As such, we report timing for training, refinement, and simulated annealing experiments as they would take on V100 graphics cards. The average time taken for refinement is 51.3 seconds.
A.7 EXCLUSION OF CYSTEINE
Cysteines are special case with regards to the other amino acids as they are primarily present in proteins to form disulfide bonds. Disulfide bonds are very strong bonds that form between two cysteine residues. They are, for the most part, present only in proteins that are secreted by the cell or exist on the outward facing surface of the cell membrane. This is due to the fact that the disulfide bond cannot form in the reducing environment of the cytosol of cells. We note that for all the statements, as always with biology, there are exceptions to the rules. In de novo design studies, the inclusion of cysteine is understandably usually framed the inclusion of disulfide bonds. Marcos ct al. (2018) and Bhardwaj et al. (2016) provide recent examples of de novo designing with the intent to use disulfide bonds or simply explore if their inclusion provides benefit. Disulfide bonds can be very useful for adding stability to a protein but complicate protein expression and depending on the advantageous. From structural perspective,
aim of the design study, it may be disadvantageous a disulfide bond can very much 'make or break' a candidate so it requires consideration. Its for this reason that the inclusion of cysteine and disulfide bonds is typically addressed in methodologically specific way. Bhardwaj et al. (2016) being direct example of that. This is relevant in our case because it clearly indicates how the default stance is generally to not include cysteine and disulfide bonds unless there is reason to. Ultimately, its exclusion does not make the problems we tackle in this work any easier and it were to be included it would be unlikely to add any significant value.
B SEQUENCE AND STRUCTURE ANALYSIS
B.1 VALIDATING THE VALIDATION AND TEST SET
Ideally each seed sample from the black-box oracle could be viewed as identical and independently distributed (i.. However, given the sometimes unexpected pathologies (c.g. adversarial exam- ples) that can be present in neural networks, we directly measure the diversity of the training set
18

---

Under review as a conference paper at ICLR 2022
sequences well as look for any direct overlap of the validation and test sets with the training set. As can be seen in Table 4 the training set sequences begin to form clusters at around 30% sequence identity.
As a method of comparing generative models between architectures and iterations of DARK, a test set and a validation set of 950 samples each are generated, using the simulated annealing approach of DDssgg.n We check that the two sets do not share extremely similar or identical sequence samples with the seed set, producing artificially high performance metrics. As such, validation and test set sequences are separately compared to all 15,000 sequences in the seed sample set. This is done using the MMseqs2 (Steinegger & Söding, 2017) search tool with a sensitivity parameter of 7 and one iteration. Given the clustering results of the seed samples (Table we expect a high proportion of the sequences to share some small similarity to sequences in the seed sample set. This was found to be the case; from the validation and test set, 854 out of the 950 sequences in each set were found to have some similarity to sequences in the training set. The mean sequence identity between the similar sequences for both sets was low, being 31% for both. The highest sequence identity found in the validation set was 68%, and for the test set 60%. These results are as would be expected; they suggest that both sets are sampled from the same distribution the seed set while not containing any near-identical copies. This is acceptable, given that this is an unsupervised task using synthetic samples, where the aim is to learn that distribution. We make the distinction had natural sequences been used a more stringent test and validation splitting based on structure and evolutionary relationships would be required.
As part of investigating the effects of DARK iterations, we also performed this search process with the test set for the 500,000 examples of the third iteration's training set. The number of examples in the test set that had any detected hits increased a small amount, from 854 (N = 1) to 922. The mean sequence identity of the hits increased slightly from 31% to 33% and the highest identity hit increased from 60% to 66%. Again, these values are acceptable, and even potentially a positive sign. One likely cause for the increase in the performance of models across iterations of DARK that the area of sequence space covered by the training set is progressively growing in an unsupervised manner. Even from a pessimistic perspective, if the N = 3 model has better likelihood estimates on the test set because the previous iterations generated training samples closer in space to the test set, then it suggests that the iterations are indeed expanding and filling in unseen parts of the distribution.
B.2 THE STRICT STRUCTURE- BASED TEST SET AND DARK -STRICT
We take two key approaches to providing a stringent measure of how well DARK generalizes. Both are enabled by first using AlphaFold to generate all-atom structure predictions, and pLDDT scores, for all 100,000 sequences contained in the DARK, training set. This set covers the 15,000 initial seed sequences and the 85,000 sequences from the first iteration of DARK. We also assign each structure a topology in the form of a topology string. This describes the topology by the secondary structure elements going from the N-terminus to C-terminus. For example, if this had been done for the de novo designed "Top7' protein (PDB ID: IQYS) the topology string would be 333330333' or 'bbababb'. We also repeat this procedure for the 950 sequence validation set.
The first way we use this 100K set is to group sequences by their topologies and show that the first iteration of DARK generates sequences with structures unseen in the initial 15K seed set. The seed set contains 421 unique topologies and the 85K set contains 1011 unique topologies, with an intersection of 373 between the two. As such, there are 638 topologies in the 85K set that were unseen in the training set. This accounts for 1476 sequences, of which 225 sequences (15.2 (1..%) have a pLDDT> 70. This provides strong evidence that the DARK iterations are expanding the training sets to include sequences with unseen and, importantly, ordered structures.
The second way we use this set to construct, and train a model on, new training and test set split based on structure, as is considered the gold-standard in evaluating structure prediction methods (Chthiaa & Lesk| 1986, Söding & Remmert 201I). We refer to the model trained and tested on this new split as DARK,-STRIC, gies, approximately 10%, from the 1059 unique topologies in the 100K set, ignoring the 100 most common topologies and uninteresting topologies, being single helix (131 sequences) and only coil (3 sequences). This 101 topologies accounts for 413 sequences which are removed from the 100K set, and become the strict sequence test set (See Appendix B.2 for the test topologies and further
We construct the test set by first randomly sampling 101 topolo-
19

---

Under review as a conference paper at ICLR 2022
details). We also remove and discard 4 sequences from the 950 example validation set that have strict test set topology. The original 950 example test set is not used here.
Next we use the MMseqs2 search tool (Steinegger & Söding, 2017) with the maximum sensitiv- ity parameter (-a 5) to aggressively search for potential similarity between the strict test set sequences and sequences in the remaining 99587 training examples and 946 validation examples. We then discard any training and validation sequences that MMseqs2 returns as hit, meaning it is potentially similar to a test set sequence. This reduces the size of the training and validation sets to 87551 and 621 examples respectively. We then use the 87551 sequence training set, 621 sequence validation set, and 413 sequence strict test set to train and evaluate the DARK2-STRICE model.
We note that this approach to assigning structures to topologies is an aggressive one as it does not consider the different packing arrangements that the secondary structure elements can have. Thus, our measures of the number of topologies present in the set is likely a lower bound on the number of topologies that would be assigned by a more intricate approach like that done for CATH (Orengo eet al., 1997). This may make the test set we have constructed more difficult however we wish to take a conservative approach and challenge the model, so we find this acceptable.
The 101 randomly selected topologies, and their sequence counts, making up the STRICT test set The 'a' is for alpha helix and 'b' is for beta sheet. The 101 topologies cover a total of 413 sequences:
bbabbbabbbb (7), abaaaab (2), baaababab (1), auabbbabb (1), babb (3), bbbbbbbbbbbub (6),
bababaaa (1), abbaabbbb (2). bbbabbababb (3), bbabbbaabb (3). bbabbbababbab (1). bbbbb-
bababb (3), aaabbb (5), babbabbbba (2), abbbbbbbabbab (1), aabbaa (32), bbaababa (1), bbbbbb-
baab (4), bbabbbababb (1), abaaabbaa (1), abbaaabbaa (I), bbubbabibibubbb (1), abbaabba
(7), babbbabab (3), babbabbabb ( (19), bbbbahbaabba (1), aabaaabbaa (1), babbbbbabbbb (2), bb-
baabbba (2), bababbbbbbb (2), bbabbabbabbb (6). babbbaab (2), bbbababbb (18), babbbabba
(3), bbbaabaa (1), babaabb (14), abaabbabbab (1), bbubabbbb (19), baaaab (9), bbaaaabb (I),
abaabaabaabaa (1), bbaabab (4), abbbababbbbb (1), bbbbaabbbb (1), bbbbbbbbaab (1), baab-
baab (2), bbababababab (1), bbabbabbabbab (2), aabbbab (4), bbaabbabbb (1), bbbbbbbabab
(5). bbaababaab (I). bbabbbbab (29), abbbbabb (22). aaaabab (2), babbbbbabbb (4), bababbabb
(6), babuabbbaub (1), bbabbabbbbbbb (1), bbbbabbbabbb (2), bubbabbabab (1), bbabaabbb (2),
buabbbbbb (5), auabuabaab (1), bbbuabb (7), abbbbabbabb (1), bbaabbbbaab (1), ababana (1),
babaabab (7), bbabbbbaa (3), aabbaabb (4), abbababb (5), abbbabbabbb (1). bbaabaabb (1), bbb-
baab (7), baababb (8), aaabaabba (1), abababaa (1), bbbabbbbbbabbab (1), bbabbbbabbbb (4),
abbaaaaa (3), abbbbababb (1), bbbbabbbaab (1), abaabbabba (1), bbbbabbbbabbb (2), bbabbb-
babbbabb (1). babbbba (26), adabbabbb (1), abbbabbbbbbb (1), babbbbabab (3), bbaabbabbba
(1), bbbbbabbabbb (1), abbabbabbb (2), bbbaabbhabba (I), bbabbbbbbbba (1), babbababb (8),
abb (5), bbaaabbbbba (1), bababaab (2), bbbbabbbbbbbb (2), babbbbabba (3)
B.3 PREDICTING STRUCTURE WITH ALPHAFOLD
We use the publicly available version of AlphaFold for predicting structures and as part of a quality metric. In both cases, we use the standard Alpharold inference pipeline, meaning we predict with the five 'CASP14' models, take the predicted structure with the highest pLDDT as the final result, and perform relaxation. We use adapted inference scripts to accommodate for predicting with only a single sequence and without using the template pipeline or the multiple sequence alignment (MSA) pipeline. This consisted of setting all models to use the same default configuration, as they all use variants on the default, and then setting all to not use templates. The MSA clusters are reduced to I sequence, being the single input sequence, and the deterministic flag is set to true. The number of recycles is set to 4. Following structure prediction, the standard relaxation is performed for the best final structure. We note that relaxation is not necessary to produce final structure, thus it can be said that a trained language model, like those we present, with AlphaFold provides a direct way to sample from joint distribution of sequence and structure, and without additional conditioning information. Predicting structure with AlphaFold provides the pLDDT scores at each
20

---

Under review as a conference paper at ICLR 2022
residue position. We take the average of these for the pLDDT score of given sequence. Structures are visualized with ChimeraX (Pettersen et al. 2021).
B.4 ALPHAAOLL PREDICTION CONFIDENCE AND DARK3 FOR RECENTLY RELEASED DE NOVO DESIGNED STRUCTURES IN THE PDB
As we use the PLDDT from AlphaFold as a metric for measuring structural order, we wish to double- check that AlphaFold assigns good or better pLDDT scores to de novo designed sequences in the PDB that were not used as a part of its training set. We manually curate a list of 26 of these sequences from the PDB. The PDB IDs containing these are 6MRR, 6MRS, 6NUK, 6MSP, 6XWI, 6XXV, 6VTW, 7KBQ, 6DOT, 7JH6, 6XEH, 6X9Z, 6WVS, 6E5C, 6DS9, STX8, 5W9F, 6YWD, 6Z35, 6W6X, 6W90, 6W15, 6W3W, 6VGB, 6W70, 6YQY.
This is not a comprehensive list as, amongst other reasons, some sequences were not included they were very sequence-similar to an already selected structure or if they are large landem repeat structure (e.g. 'armadillo repeats') ') to avoid biasing the average. This is still a biased set of structures as they are the product of whichever goal given study was investigating, but the pLDDT scores are still informative. We note that although the PDB has 'DE NOVO PROTEIN' classification, this includes engineered, designed, and de novo designed proteins. The average calculated pLDDT of all sequences in the list is 88.1 (o- 4.41), a good score, as is expected.
We also calculate the likelihood of these sequences under DARK; compared to random sequences of length 100. We ignore the 5 sequences that contain cysteine. To account for DARK having been trained on static length proteins we ignore the prediction of the 'stop token' and ignore all residues in a sequence above position 100 if they exist. We calculate the bits per character (BPC), where characters are residues, across 100 random sequences and the 21 designed sequences. The random sequences have mean BPC of 5.88 bits and the designed sequences have a mean BPC of 4.96 bits. In having a lower BPC suggests that the DARK models have learned a distribution of sequences that incorporates structure and is capable of producing de novo designed sequences more than random. It is also unlikely that by random chance the DARK models have generated and learned from sequences in the same parts of the dark sequence space
B.5 BENCHMARKING WEAKLY CONDITIONED METHODS
We provide results for benchmarking with Good+ PLDDT proportions on two recent weakly condi- tioned generative protein design models, which we refer as gram VAE (Greener al., 2018)) and geWGAN (Karimi et al. 2020). These both use a fold level description that we call weak as they are far removed from the explicit structural information provided in fixed backbone models (Ingraham et al. 2019 Strokach et al. 2020 Cao et al.) 2021). We stress that these models are not directly comparable the unconditional models explored in this work. Furthermore, we provide these only to gain small degree of insight but this is significantly muddied by both models being able to 'cheat'. If either of them reproduce a sequence from their training set, then is extremely likely that the same sequence and its structure are in the AlphaFold training seeand and so likely to be assigned a higher PLDDT than would otherwise be expected. Nonetheless, we provide these benchmarks as they provide some small degree of context. Even with extra information from conditioning variables and this ability to cheat, they are still outperformed by DARKs (Table B8. We speculative that this difference can even be seen visually in the predicted structures shown by both models (Figure 5c of Karimi et al. (2020) and Figure 6 of Greener et al. (2018) which contain clear disordered regions.
The gram V uses context-free grammar of SCOP protein folds (Andreeva et al. 2020) conditioning variable for a Variational AutoEncoder (VAE). Both encoder and decoder use non- maximum length of 140. The geWGAN is condi- tional Wasserstein Generative Adversarial Network that also pads length to use fixed-sized networks (Krimii et al. 2020). It conditions on a small latent representation of folds, learned from perform- ing kernel PCA on an all-vs-all distance matrix between all folds in the dataset, where the kernel used is symmetric TM-score (Zhang & Skolnick 2005) between two folds. The publicly available
linear layers and sequences are padded
Allpaaoold makes the blanket statement that all structures from the PDB with maximum release date of 30 April 2018 are used in its training.
21

---

Under review as a conference paper at ICLR 2022
implementation provides 600 sequences generated for the 6 folds (100 sequence each) in its test set using the gcWGAN and the gram in a head to head comparison. Being publicly available these provided sequences to calculate Good+ pLDDT proportions
10
and direct comparison, for both models. The gramVAR achieves 21. .0% and the geWGAN achieves 2.83%.
We note that the majority of the folds in the geWGAN test set are in the gram training set, which likely explains part of the performance difference. We are not concerned by this as the DARK models still outperform the gram VAE with an artificial boost. We also note that the gcWGAN model does not discretize the output of the generator and use any approaches to backpropagating through the discrete output variable (Bengio et al. 2013 Jang et al., (016). This provides the discriminator a trivial learning scheme: place the decision boundaries at the edge of the probability simplex of tokens at any position. This may explain part of the geWGAN's poor performance. In both cases, we recognize that these models are not ideally suited for sequence problems and have reported poor performance elsewhere (Cao et al. (02T). We expect future weak conditioning methods will provide more competitive benchmarks. Ideally, we hope that future investigations into novel unconditional models will provide direct benchmarks, establishing the best approach to unconditional modelling.
B.6 DARK MODEL SAMPLES ARE DISTINCT FROM NATURAL PROTEINS
We find that samples from the DARK, model are distinct from sequences in nature. A sequence search against UniRef100 of the 1000 evaluated sequence samples (without refinement) returned no hits for all sequences except one, which had 5 weak hits. Two pairs of these 5 sequences were extremely sequence-similar, Repeating the sequence search with different tool, HHblits (Remmert al. 2012), no natural sequences were found that met the HHblits default threshold value (an E-value below 1 x 10 10-3).
B.7 SEQUENCE SEARCHING AND CLUSTERING
We use the MMseqs2 package (Steinegger & Söding 2017 to perform sequence searches and se- quence clustering. For clustering, the default clustering pipeline and settings are used with sequence identity as the threshold. If not specified, this is assumed to be a threshold of 30% sequence identity. For sequence searching we the default pipeline of MMseqs2 and high sensitivity parameter of 7.0 with iteration against UniRef100 (03-2021 release) (The UniProt Consortium, 2021). Any sequence searching done with HHblits (Remment al. 2012) uses E-value cut-off of 1 x 10-3 with iteration against UniRef30 (06-2021 release) (Mirdita et al. 2017). In this work, we call any two sequences similar if MMseqs2 (or HHblits) returns a hit between them, meaning it has detected similarity. Sequences that have a, for example, 40% sequence identity or above are also defacto assumed as similar.
B.8 STRUCTURE CLUSTERING
To cluster predicted protein structures we perform single-linkage agglomerative clustering using TM-score (Zhang & Skolnick (005) as a similarity metric, meaning an all-against-all matrix of TM-scores is calculated for each structure in the set of 1000 samples. We TM-score of 0.4 as the cutoff for selecting best cluster centroids and a TM-score of 0.65 for single-linkage clustering around the centroids.
B.9 GENERATING DE NOVO DESIGNS WITH DARK AND ALPHAFOLD REFINEMENT
Background To give a concrete demonstration of how DARK3, and models like it, can be used in developing novel methods, we develop simple but novel design method. The method we develop, termed AlphaFold refinement, tackles what could be considered the next step for using model like DARK. Although we have used AlphaFold primarily evaluate samples, and by extension models, it can also be used to improve candidates. An obvious first step is that AlphaFold produces both distogram based predictions and an all-atom structure prediction, so a simulated annealing approach like trDesign's (Anishchenko et al. 2020) could be used to sample sequences. We note that the
22

---

Under review as a conference paper at ICLR 2022
all-atom prediction can be considered point estimate of the distogram, but the all-atom model is ultimately the output of the most interest to us for design.
The challenges of a simulated annealing approach with AlphaFold Unfortunately, this ap- proach poses two problems. The first is that, at its fastest, meaning running AlphaFoold with no recycling (Jumper et al. (02T), each forward pass takes approximately 5 seconds. This is in contrast to 0.064 seconds with DMPfold2. Performing simulated annealing run with the same schedule as we use would take 2.3 days which is prohibitively long. The second problem is one we find in extending the approach itself. Directly optimizing on the AlphaFold IG-score (AIG-score) itself appears to produce, what amounts to adversarial attacks on AlphaFold. This can be seen in Fig- ure ZA & 4 and is particularly evident in the low AIG-scores (low compared to our approach) but extremely high pLDDT scores. The visual confirmation of this is evident in Figure where all struc- tures produced are very unrealistic single long helix. We implement the AIG-score as the IG-score but using AlphaFold to calculate the distogram prediction from which the p(y (z) term is calculated. As we do not have the resources that were use to train AlphaFold (Jumper et al., (00T)) we calculate a proxy background distribution by predicting the structure of 1000 random sequences of L. 100 and then averaging their resulting distogram predictions together into one vector.
Using DARK3 to enable AlphaFold design Introducing DARK3 into this problem brings several benefits. First and foremost it allows us to generate diverse and high confidence samples from Al- phaFold without them being adversarial attacks. also reduces the number of design steps such an approach would take, and adds measure of control into the design process. We call this method AlphaFold refinement as it is similar in concept to the refinement done with DMPfold2 between iterations of DARK. DARK, allows us to generate large sequence databases, with predicted struc- tures, that have a diverse range of topologies. These include structures with topologies that have yet to be successfully de novo designed, and many with high confidence stable structures that can be used as de novo scaffolds into which functional sites are grafted (Woolfson) 2021). In short, DARKa produces sequences that have structures that are of design interest.
AlphaFold refinment AlphaFold refinement begins with sequence sample from DARK and its AlphaFold predicted structure, chosen by the designer. This is how control is introduced into the design process. For a graphic displaying the full AlphaFold refinement procedure see Figure 2 The next step is to take the predicted all-atom structure and calculate its one-hot encoded distogram representation (Jumper et al. 202I). We then bootstrap this representation as a fixed optimization target. The actual optimization that occurs is greedy hill-climb on the sequence against the Cross- Entropy between the predicted distogram and the bootstrapped target. As we use all five parameter sets when we use AlphaFold, we treat the distogram prediction at each step as if it were an ensemble of five AlphaFolds; we average the five predicted distograms as one final predicted distogram with which to calculate the Cross-entropy. We run this for Alphaloold refinement for 10000 steps, as can be seen in Figure 22, however we found comparable results could be achieved in 7500 steps, being 10 hours per sequence a single consumer- -aguudee--rreeeeee GPU (RTX 2080 Ti). This is significantly more efficient than an approach like using Rosetta which can take thousands of CPU hours. In summary, mutate the sequence and keep mutations that minimize the Cross-Entropy to the original struc- ture. The underlying reason for undertaking this approach is that the structure and sequence we are interested in will, more often than not, have lower PLDDT than is possible with similar sequence that folds to the same structure. In short, there is always room for improvement, and AlphaFold with DARKs provides a way to address this.
This provides two benefits. The first is that it forces the objective to keep the original structure we are interested in. We find that this also provides a shorter and, we speculate, more direct optimization path to a high confidence structure. There stands to reason as we giving fixed objective, much easier task, rather than letting the random walk find arbitrary minima. This evident in the average AIG-score and pLDDT score improving vastly while the final sequences remain very similar from a sequence identity perspective. What is particularly notable is that the average PLDDT score of the 10 examples explored is very similar to that of recently released de novo designed structures in the PDB (Figure 2D & 2). The 10 examples we take are chosen from the 5000 DARK;, samples previously generated in semi-random the exact same topology. accounting for packing, and any single sheet (CATH 2.20; Orengo et al. (197]) topologies simply because they are uninteresting. The 10 examples cover 4 & structures,
fashion. We chose them randomly but rejected any with
23

---

Under review as a conference paper at ICLR 2022
3 structures, and 3 structures. This rather even spread of examples was entirely coincidental but convenient.
Another aspect of using the distogram is that it provides softer means of keeping the original structure. In essence, by working in probabilistic space, AlphaFold can perform small rearrange- ments the original structure to improve its confidence while not straying far. This is difficult to quantify but can be seen in Figure 2p Example 8, a complex a & 3 topology where several of the sheets are brought together in the lower half of the structure (as visualized) to improve packing. To summarize, in the limit of AlphaFold's prodigious accuracy, AlphaFold refinement provides way to refine design candidates of interest and improve the confidence (and order). Another way of viewing this approach is as an entirely neural network driven approach flexible-bace baibbe- design, here used as part of overall de novo design method. In that vein, we note that this approach could design.
be extended very easily to other tasks in protein design like
Designing novel immunoglobin As an example of using this approach we take a immunoglobin from the 5000 DARK, samples that previously generated and then AlphaFold refine it. We take the first example of an immunoglobin in the set and find that we successfully refine its structure to high PLDDT and AIG-score while not altering the original approach. The results of this design can be seen in Figure 3 This provides concrete example of using DARK; and AlphaFold to design high confidence candidate for topology of high scientific interest that traditional approaches like Rosetta are known to struggle with.
24

---

Under review as a conference paper at ICLR 2022
C ADDITIONAL FIGURES AND TABLES
Table 3: Samples from DARK models have diverse sequences and predicted structures. Sequence and structure diversity shown with the number of clusters (of 1000 samples) estimated by sequence clustering and structure clustering respectively. Change in the Good+ pLDDT between iterations is also included.
DARK Model
Good+ pLDDT (%) f Sequence Clusters
Structure Clusters
DARK,
28.0
1000
829
DARK2
31.0
973
835
DARK3
32.3
962
822
Table 4: Diversity of the sequence examples in the training set. Diversity at each iteration is measured by number of sequence clusters found by clustering with four different sequence identity (Seq. ID) thresholds. The number training examples (Train Ex.) each iteration included.
Iteration
99% Seq. ID
90% Seq. ID 50% Seq. ID 30% Seq. ID
Train Ex.
N
15,000
15,000
14,994
11,076
15,000
2
100,000
100,000
99,942
66,509
100,000
N = 3
500,000
500,000
498,077
231,425
500,000
Table 5: Showing baseline results of refining random sequences against the AlphaFold IG-score using greedy hill-climb. Sequence identity between sequence before refining and final sequence produced by refining. This was ran for 20,000 steps.
Best AlphaFold IG-score
Sequence Identity
Example
t
to Starting Sequence
0,865
0.16
0.861
0.14
0.849
0.12
1.210
0.24
0,837
0.15
0.855
0.18
1.227
0.22
0.859
0.13
0.852
0.21
10
0.834
0.14
Average
0.925
0.169
25

---

Under review as a conference paper at ICLR 2022
Table 6: Showing baseline results of using trDesign-like approach with AlphaFold. It refines random quences against the AlphaFold IG-score using simulated annealing, Sequence identity is between sequence before refining and the final sequence produced by refining. This ran for 20,000 steps.
Best AlphaFold IG-score
Sequence Identity
Example
to Starting Sequence
1.204
0.03
1.198
0.07
1.210
0.07
1.214
0.02
2.009
0.05
1.224
0.06
1.204
0.05
1.196
0.03
1.205
0.10
10
1.204
0.04
Average 1.287
0.052
Table 7: Showing results of performing AlphaFold refinement with DARK3. refines DARK; sequences
against the Cross-Entropy between AlphaFold's initial step distogram and the current sequence's distogram, indirectly improving the AlphaFold IG-score, using greedy hill-climb. Sequence identity is between quence before refining and the final sequence produced by refining. This for 10,000 steps it sufficient convergence and stopped as not waste resources.
Best AlphaFold IG-score
Sequence Identity to Starting Sequence
Example
2.387
0.33
2.190
0.39
1.992
0.38
2.160
0.36
2.008
0.32
2.145
0.39
2.197
0.42
1.316
0.43
2.317
0,40
10
2.329
0.33
Average
2.104
0.375
Table Good+ pLDDT scores from AlphaFold (higher better) are shown for unconditional language mod- els, uniformly random sequences, and two recent conditional models that use non-specific (weak) structure information for conditioning.
Language Model
Good+ PLDDT (%)
Conditioning
Training Data
Random
1.8
geWGAN Karimi et al.
2020 gram VAE Greener al.
2.8
Structure Structure
Natural
2018
21.0
Natural
UF50
15.9
Natural
-Avee-Advrsaarial
28,0
Synthetic
DARK,
28.0
Synthetic
DARK,
31.0
Synthetic
PARK2-STRICT
29.2
Synthetic
DARK3
32.3
Synthetic
DARK3 (Ref)
37.3
Synthetic
DARK Seed Samples
40.8
Synthetic
26

---

Under review as a conference paper at ICLR 2022
Example: De Novo Design of an Immunoglobin Fold Using DARK3 and AlphaFold Refining
pLDDT After: 93
Ca-RMSD: 1.317A
Sequence Identity: 26%
pLDDT Before:
Refore AF Refinement
After
Figure 3: Showing the structure of DARK sampled sequence predicted have immunoglobin fold by Al- phaFold. The structure of sequence shown after AlphaFold refinement along with change in pLDDT, the Ca-RMSD difference between the two structures, and the sequence identity between the two sequences. The structures are visualized with ChimeraX (Pettersen et al. 2021
Advensarial Examples Produced by Simulated Annealing of the AlphaFold IG-score
Average PLDDT: 97.1
Figure 4: The superimpased structures of 10 examples produced by simulated annealing of the AlphaFold IG- score, in essence, trDesign approach with AlphaFold. All 10 examples achieve low IG-score of 1.28 (our method achieves 2.10) but extremely high mean pLDDT of 97. This direct approach to maximizing the AlphaFold IG-score effectively generates adversarial examples. evident the same single long helix. The structures are visualized with ChimeraX (Pettersen el al. 2021
samples being predicted to have
AlphaFFold
a a&g
all-
ryhal
Figure 5: DARK; paired with AlphaFold unconditionally samples diverse predicted structures. We show
varied examples protein structures, predicted by AlphaFold, for sequences sampled directly from DARK; without any refinement. These structures have variety of different folds, and are visualized with ChimeraX (Pettersen et al. 2021 For clarity, we split the rows into examples of all-a, a & B, and all-3 structures.
27

---

Under review as a conference paper at ICLR 2022
Ca-RMSD 3.1A
Rosetta AbinitioRelax
AlphaFold
Figure 6: Agreement between Rosetta and AlphaFold structure prediction. Predicted structures of quence sample (ID 209a) using AlphaFold and Rosetta AbinitioRelax. Both approaches in agreement, pro- ducing close to the same structure. For latter, the best predicted structure with the lowest Rosetta all-atom score is used, being -279. The root-mean-square deviation between a carhons in the two different predictions is included.
Rumplex
= Rellnod Samples
0.35
0.00
32.3% >70 pLDET
37.3% PLDET
6.25
1.88 PLDDI
6.000
0.100
0.088
(Alppatioo))
Figure 7: DARK models generate samples with confidently predicted structures. A Normalized histograms of the DDDDT scores produced by AlphaFold when predicting structures for three sets of sequence samples, each containing 1000 sequences. A skew-normal is fit each and plotted for clarity. Random refers uniformly random sequences of 100. The Good+ PLDDT (pLDDT> 70) proportion also included. N 3 refers to the DARK, model.
28

---

Under review as a conference paper at ICLR 2022
-31
-20
-25
-20
-65
-200
-200
RMSD
Figure 8: Rosetta predicts correct fold and potential alternative conformation. The top 1000 predicted structures (by Rosetta score) for sampled sequence. of 10000 structures, produced by the Rosetta AbinitioRe- lax protocol. Rosetta
expressed in Rosetta Energy Units (REU; lower values better). RMSD root-mean-square deviation (RMSD) Á from AlphaFold structure prediction using o-carbon atoms. The hest model is bottom left but second peak suggests an alternative conformation may be possible (bottom right). We note similarity to the multi-peat peak distributions presented in Figures 2B & 2C No al. 202I
sampled sequence
29


Reviews:
[{"review_id": "WtQojQ-vKv", "reviewer": "Reviewer_g4tX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Generated sequences are highly likely to be extremely biased toward a specific subset of sequences\nThe way these sequences are generated is that they start with some seed sequences optimized out of DMPfold2. Then, the sampling model $p_{\\gamma^{n}}$ is fit to this distribution, and then used to generate sequences from this distribution. These sequences are then further optimized using the same DMPfold2 model to generate more training examples. Throughout this procedure, $p_{\\gamma^{n}}$ is almost certainly going to learn a very similar distribution to the initial seeds, and is unlikely to generate novel examples beyond these seeds which are significantly different in any way. Furthermore, these seeds are generated from a single model (DMPfold2). This already biases these sequences toward the sequences used to train DMPfold2.\n\nOverall, this iterative process (i.e. sampling sequences and maximizing IG-score) doesn't seem like it would help at all with generalizing to novel sequences, and only serves to focus the sequence model on a specific sub-distribution. I expect any improvements in structural likelihood that stems from iteration to just be refining the structural likelihood of the initial seed sequences, which were selected suboptimally (via simulated annealing).\n\nThe number of sequence clusters (as a measure of the diversity among generated sequences) was shown for the three DARK iterations, but it is not clear how to interpret numbers of 1000, 973, and 962. Is that big or small? How does that compare to the baselines? Also, having 1000 clusters for 1000 sequences may suggest that this metric is not very reasonable to begin with. With that many sequences, one would expect at least a few clusters of size greater than 1 just by random chance (usually).\n\nFurthermore, the authors claim that the 1000 samples drawn from DARK have no strong similarity to natural proteins in UniProt. This needs to be quantified.\n\n### The model is still conditioned on structure\nEven though the paper claims that the DARK method is unconditional, it still depends very heavily on structure. Structural information is coming in from the conditional model DMPfold2, which has an extremely central role in the algorithm. Thus, even though the final model $p_{\\gamma^{n}}$ only considers sequences, it has structural information built into it. Specifically, it is _conditioned_ on the initial seed samples and DMPfold2.\n\n### There is no real validation of the generated sequences' structures\nUsing AlphaFold is unfortunately not a convincing validation of these sequences' structural likelihoods. Although it is technically a different neural network from DMPfold2, it is still only an estimate with no theoretical guarantees. That is to say, it is not \"real\". Without the ability to synthesize and test these generated sequences for structure experimentally, one could use existing structured proteins and use the sequence model to score these sequences. The model should give structured proteins a higher score than disordered proteins.\n\n### Outperforming natural sequences is concerning\nThe fact that the model doesn't seem to do so well with natural sequences is very concerning, because natural sequences (excluding those that are disordered) are typically highly thermostable and thus structured. In fact, natural sequences in many ways should be the upper bound of how well the generating model performs (unless the claim is that these generated sequences are _more_ structured than natural sequences).\n\n### Missing some baselines\nPrevious works have also attempted to generate _de novo_ proteins. For example, Alley, et. al., 2019 (UniRep) can be used to generate sequences that can also be scored by AlphaFold. Anand & Huang 2018 also generate novel proteins that are structured. Notably, UniRep is truly unconditioned on structure (i.e. there is no notion of structure in their model), whereas DARK still conditions on structure. In many ways, DARK lies on the spectrum between these two previous works (in terms of reliance on structure).\n\nIt should also be noted that UniRep is capable of generating sequences that are structured, which is shown in the paper.\n\n### Some choices in the algorithm are unclear or of questionable soundness\nThe choice of the loss function is not quite clear. For example, why use the background model $p_{\\epsilon}(y_{d})$? Why not just maximize $p_{\\beta}(y_{d}\\vert x_{d})$? Additionally, is the sequence prior needed? What happens if it is removed? Is there a weight $\\lambda$ for these two halves of the loss function?\n\nIt is also not clear why examples are concatenated (step 6 of the algorithm). Do old examples need to be present even though larger and larger sets of sequences are being generated?\n\nThe use of a different model architecture for each iteration of DARK is concerning. This suggests that the developed method may be extremely overfit to this specific task/dataset. There doesn't seem to be a justification for the different architectures.\n\nAlso why are Cys residues removed? Given that disulfide bridges are a critical part in maintaining tertiary structure in many proteins, a model that does not consider their existence raises some red flags.\n", "summary_of_the_paper": "The authors propose a way to generate novel protein sequences that are likely to have folded structures. To do so, they start with a pre-trained sequence-to-structure predictor, and use simulated annealing to generate a set of initial sequences that are likely to have a structure. These seed sequences are used to train an unsupervised sequence model (an autoregressive transformer). More sequences are generated from this sequence model, and they are optimized to maximize the likelihood they have structures (using the pre-trained sequence-to-structure model), and are added to the growing set of training sequences. This process is repeated for several iterations to finetune the unsupervised sequence model. The authors show that sequences generated from this sequence model are likely to be structured as predicted by AlphaFold.\n", "summary_of_the_review": "The problem of generating novel protein sequences is a popular one. Although this submission develops a somewhat novel approach to solving this problem, the method's success is dubious, if not absent. The overarching claim that the method is unconditional is not quite true, and the sequences generated are almost certainly highly biased and limited. There is also no real validation of the generated sequences, and minimal comparisons to existing work in the same space. For these major methodological and technical issues, I unfortunately cannot support its acceptance.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "PI2fI7Lj5EW", "reviewer": "Reviewer_Jhfe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Author's goal: Create a ML model that generates de novo sequences that fold.\n- That fold could be anything at all. Simply \"a folding predictor is confident that it folded my sequence accurately\".\n\nI think this is a fun idea, but too flawed. in application as it is a little circular  I am curious what the generated sequences look like... Not because the are likely to fold, rather because they are ones that \"fool\" AlphaFold.\n\nCritique in summary:\nThe work boils down to an adversarial attack on DMSfold2, which proves to enrich for high scores in AlphaFold. But, adversarial attacks commonly create nonsense examples that exploit biases in the target model.\nAlphaFold is not convincing validation, because it is too similar to the oracle used for optimization.\nBoth DMSfold2 and AlphaFold are misused. They've been trained & tested for multiple sequence alignments of natural sequences. There's very very little evidence that they accurately predict the fold of individual non-natural sequences.\n\nThe method and my problems with it:\n1. Sequence generation = autoregressive sequence model. First, the AR model is trained on seed sequences. It then generates new sequences.\nAs a seed, the authors use sequences generated from trDesign (Baker lab 'hallucinations' model). Effectively, these are sequences predicted to fold according to Rosetta.\n\nProblem = They've started with the answer, Rosetta designs.\n- In the output, 922 / 950 seqs were ~33% identity to seed sequences.\n- Need authors to include more detailed sequence comparison.\n- That 33% might be enough 'cheating' to get high 'validation' scores, in which case they've just used Rosetta.\n- They need to report the validation scores for the trDesign seed sequences too.\n- Are the predicted folds for final output the same as for seed sequences?\n\n2. Generated sequences are then mutated to optimize \"folding score\" = KL divergence of distograms (binned pair distances) according to DeepMetaPsiCov fold 2.\nThis should mean \"optimize sequences so that predicted contact maps are highly non-random\". With each iteration, the AR model is then trained on the optimized sequences plus the previous iterations' sequences.\n\nProblem = They are optimizing a sequence generator to exploit a model OUTSIDE of its use-case. This is like an adversarial attack. I would expect to get sequences that exploit false biases. \n\n3. For validation the final output sequences are run through AlphaFold prediction & scoring = pLDDT (predicted Local Distance Test score). Authors claim AlphaFold is independent validation from the loss function they optimized against.\n\nProblem =\nAlphaFold is not an independent validation, because it is virtually the same model, trained on the same data as DMSfold2, and outputs the same sort of score.\n- Trains on MSAs\n- Trains on natural sequences\n- Very similar architectures\n- Scores, AlphaFold pLDDT vs DMSfold2 confidence. Both are predictions of distogram measurements, LDDT and TM, respectively. And trained on PDB structures of natural sequences.\nAlso a misuse of AlphaFold the same way as in #1. AlphaFold uses MSAs, and is trained/validated 99% on natural sequences.\n\n\nResults, boil down to 2 weak observations:\n1. The basic training task is possible: training the AR model indeed creates sequences that improve KL Divergence of DMPfold2 distograms.\n- Non-result.. should be guaranteed that the ML training will improve objective score.\n\n2. The output sequences have higher AlphaFold predicted LDDT than random sequences, sequences output by an AR trained on UniRef50, or two other models.\n- Comparative advantage is versus other unproven models, but not e.g. natural proteins, or Rosetta designs.\n- Advantage is all but guaranteed by similarity to DMPfold2, which was optimized for.\n- Unclear that the AlphaFold pLDDT score is meaningful for non-natural sequences.\n- Unclear if higher scores are byproduct of using trDesign seed sequences.\n\n", "summary_of_the_paper": "Dear Program Chair(s)\nAfter reading the back and forth on this paper I feel it might help if i send comments _even though they are very very late \n", "summary_of_the_review": "Author's goal: Create a ML model that generates de novo sequences that fold.\n- That fold could be anything at all. Simply \"a folding predictor is confident that it folded my sequence accurately\".\n\nI think this is a fun idea, but far too flawed. I am curious what the generated sequences look like... Not because the are likely to fold, rather because they are ones that \"fool\" AlphaFold.\n\nCritique in summary:\nThe work boils down to an adversarial attack on DMSfold2, which proves to enrich for high scores in AlphaFold. But, adversarial attacks commonly create nonsense examples that exploit biases in the target model.\nAlphaFold is not convincing validation, because it is too similar to the oracle used for optimization.\nBoth DMSfold2 and AlphaFold are misused. They've been trained & tested for multiple sequence alignments of natural sequences. There's very very little evidence that they accurately predict the fold of individual non-natural sequences.\n\nThe method and my problems with it:\n1. Sequence generation = autoregressive sequence model. First, the AR model is trained on seed sequences. It then generates new sequences.\nAs a seed, the authors use sequences generated from trDesign (Baker lab 'hallucinations' model). Effectively, these are sequences predicted to fold according to Rosetta.\n\nProblem = They've started with the answer, Rosetta designs.\n- In the output, 922 / 950 seqs were ~33% identity to seed sequences.\n- Need authors to include more detailed sequence comparison.\n- That 33% might be enough 'cheating' to get high 'validation' scores, in which case they've just used Rosetta.\n- They need to report the validation scores for the trDesign seed sequences too.\n- Are the predicted folds for final output the same as for seed sequences?\n\n2. Generated sequences are then mutated to optimize \"folding score\" = KL divergence of distograms (binned pair distances) according to DeepMetaPsiCov fold 2.\nThis should mean \"optimize sequences so that predicted contact maps are highly non-random\". With each iteration, the AR model is then trained on the optimized sequences plus the previous iterations' sequences.\n\nProblem = They are optimizing a sequence generator to exploit a model OUTSIDE of its use-case. This is like an adversarial attack. I would expect to get sequences that exploit false biases. \n\n3. For validation the final output sequences are run through AlphaFold prediction & scoring = pLDDT (predicted Local Distance Test score). Authors claim AlphaFold is independent validation from the loss function they optimized against.\n\nProblem =\nAlphaFold is not an independent validation, because it is virtually the same model, trained on the same data as DMSfold2, and outputs the same sort of score.\n- Trains on MSAs\n- Trains on natural sequences\n- Very similar architectures\n- Scores, AlphaFold pLDDT vs DMSfold2 confidence. Both are predictions of distogram measurements, LDDT and TM, respectively. And trained on PDB structures of natural sequences.\nAlso a misuse of AlphaFold the same way as in #1. AlphaFold uses MSAs, and is trained/validated 99% on natural sequences.\n\n\nResults, boil down to 2 weak observations:\n1. The basic training task is possible: training the AR model indeed creates sequences that improve KL Divergence of DMPfold2 distograms.\n- Non-result.. should be guaranteed that the ML training will improve objective score.\n\n2. The output sequences have higher AlphaFold predicted LDDT than random sequences, sequences output by an AR trained on UniRef50, or two other models.\n- Comparative advantage is versus other unproven models, but not e.g. natural proteins, or Rosetta designs.\n- Advantage is all but guaranteed by similarity to DMPfold2, which was optimized for.\n- Unclear that the AlphaFold pLDDT score is meaningful for non-natural sequences.\n- Unclear if higher scores are byproduct of using trDesign seed sequences.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JqpCv6xg3Q4", "reviewer": "Reviewer_sSh9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The larger methodological message of the paper seems to be that, if your goal is to generate samples of sequences in proportion to their score, rather than \"start from N random sequences and anneal them to your objective\", you can start by annealing a small number of random sequences, then that first batch to guide the generation of new (non-random) initial sequences to anneal. Eventually you can work your way up to N total annealed sequences, but most of them started with better initializations, so the hope is that the overall scores are better.\n\nThat's a reasonable thing to propose and to try in the context of protein sequence design. But I have moderate concerns about how this was depicted, because the paper doesn't really try to connect or contrast this scheme with larger ideas from sampling (e.g., \"estimation of distribution\" methods) or from machine learning (e.g., self-training). Even if those connections were made clear, ultimately I also find the choice of baselines confusing, because they are all so different from they don't get to the heart of what's different about this method, i.e., too many things change between the compared methods.\n\n\n\n### Comments\n\nI think the high-level methodology here can be viewed conceptually a kind of self-training within an EDA (\"estimation of distribution\") framework [see Figure 3.2 in Larranaga and Lozano 2002, Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation]. The paper would benefit if it were framed that way, and if it discussed success / failure modes of self-training in scenarios like sequence generation [He et al. 2020, Revisiting Self-Training for Neural Sequence Generation], classification, etc. \n\nViewing DARK as an EDA/self-training of trDesign, I think also \"stDesign\" for \"self-training design\" would have been a more fitting name, attributionally, than something completely new like DARK. But that's OK.\n\nThe paper acknowledges multiple times that it builds upon Anischenko, and that's great. But the way the paper is written it often still fails to make clear what is new and what is borrowed. That is especially true of section 3.4, which is describing Anischenko's objective and proxy objective, but in different notation, and without making this clear.\n\nOne of the listed contributions is \"By pairing DARK with AlphaFold, we show a fully neural way to unconditionally sample both protein sequences and structures jointly.\" Just some feedback on that: I think it's a bit of a stretch to call this a contribution. First, spart from self-training, the overall joint sampling scheme is essentially taken from [Anishchenko et al 2020]. [Anishchenko et al 2020] also mention that annealing could in principle be replaced with gradient ascent by backpropagating to the sequence, akin to which others like [Killoran et al 2017] attempted in simpler settings, which seems more \"neural\" than what the submission is doing.\n\nThe first listed contribution says that this is the first paper to \"show that unconditional generative models... can capture the important long range dependencies that relate protein sequences with their structures.\" As far as I cans ee from Figure 2, it recapitulates alpha helices and beta sheets, of varying lengths; that's good, and you can probably argue that these are indeed long range (not purely pairwise, for example). But, you don't need deep learning to sample from the space of protein sequences that are broken up into stretches of beta sheets and/or alpha helices. Could the authors please comment on whether other structures were observed in sampling.\n\nIn most protein design situations, there is obviously a design criterion (affinity to a target, fluorescence, etc.), and that design criteron may call for conditioning on a family or families of proteins, or on desirable features of the protein sequence. Randomly sampling proteins that have \"order\" casts a relatively wide net, so I think it'd be helpful for the authors to talk about how they envision an unconditional sampler being used with respect to realistic design criteria. Are the authors envisioning to simply reject the ones that fail the criterion, from a huge pool of randomly sampled (but ordered) proteins? [Anishchenko et al 2020] go to the trouble of experimentally validating the secondary structures of predicted sequences, but they also do not express a clear vision in this regard.\n\nBy analogy to \\beta-VAEs [Higgins et al 2017, ICLR], there might be a missed opportunity to tune the sample quality by introducing a simple hyperparameter that controls the weighting between the two KL terms.\n\nThe paper repeatedly talks of producing \"high-quality structures\" when it effectively means something more like \"highly ordered structures\" or \"structures with high AlphaFold confidence.\" Whenever there's a specific notion of quality being pursued, the wording should reflect that, and it's a more fair way to characterize the situation when comparing to other generative methods that had different goals and therefore different notions of 'quality'.\n\nDo the samples depicted in Figure 2 represent, proportionally, the kind of samples generated?\n\nI am surprised that the paper does not include a \"DARK_0\" baseline, which would essentially be just the first round of [Anischenko et al. 2020] using the alternate representations.\n\nSince DARK includes multiple rounds, and each round includes both training and hill-climbing, it seems fair to ask whether applying the same computational budget to \"annealing from purely random seeds\" baseline would produce comparable results and diversity. On some level, that is the most natural baseline (and other variations of it that would exhibit different diversity/score tradeoffs, e.g., different rejection thresholds, or pure hill-climbing). If the authors could please comment on this.\n\n\n\n\n\n### Minor comments\n\npage2: \"to unconditional sample\" -> \"to unconditionally sample\"\n\npage2: please remove \"rigorously\"; the results in this paper are suggestive in nature\n\npage3: \"that does considers\" -> \"that does consider\"\n\npage4: \"by Bayes\" -> \"by chain rule\"\n\npage4: Saying that one of these quantities \n\npage4: Symbol \\mathcal{X} seems described as a \"sequence space\" (which V^L is) and is then treated as a set of training points. I think the sentence introducing \\mathcal{X} and \\mathcal{Y} can be tidied up. \n\npage6: \"p_(y_d\" -> \"p(y_d\"\n\npage6: \"distills\" now has a specific technical meaning in deep learning so just be careful to avoid confusion.\n\nFigure 2 is not referenced anywhere.", "summary_of_the_paper": "I think it helps to start by summarizing [Anischenko et al. 2020, Protein design by deep network hallucination], and then explain the differences in the submission.\n\n[Anischenko et al. 2020] aim to generate AA sequences that are diverse but that still reflect the properties of training structures.\n* Their goal is to generate protein sequences whose structures tend to be unlike those of random sequences. Since random sequences have structures that tend to be unordered, they incidentally generate sequences having ordered structures.\n* They use the deep resnet from rtRosetta (similar to AlphaFold's) to predict interresidue geometry maps from sequence, and that is how structure is represented and compared.\n* Their design goal is to design samples from p(structure | sequence) p(sequence).\n* To represent p(structure | sequence) they use a proxy objective: the KL divergence between the predicted structures and the structures of random sequences; in other words, good sequences are those whose distance maps are very different from the distance maps of random sequences.\n* To represent p(sequence) they just use -KL(AA frequences of predicted sequences || AA frequences of PDB sequences).\n* Sequences are designed by starting from random and then annealing according to the proxy objective.\n* Designed sequences are then clustered by template modeling score (TM-score) and selected (from each cluster) based on  a measure of prediction 'consistency'.\n\n\nNow, the submitted manuscript (DARK) is the same objective and proxy objective as [Anischenko et al. 2020], but with following methodological changes:\n1. For the sequence->structure oracle, instead of using rtRosetta's interresidue geometry maps as the structure representation, here distograms [Senior et al. 2020] are used, and are output from a modified version of DMPfold2.\n2. Rather than relying solely on one round of \"random initialization -> annealing\", DARK uses an iterative self-training scheme. First they do one pass of [Anischenko et al. 2020] to get 1st round of annealed sequences. The annealed sequences from 1st round are used to train an autoregressive generative model (Transformer-based). That model is sampled from to generate additional \"random initializations\" (which are now guided, not purely random). These guided samples are then optimized (just hill-climbing now instead of annealing, for speed), and added to the training set for subsequent rounds, etc.\n\nThere are other differences from [Anischenko et al. 2020], like using pLDDT score (orderedness) as a measure of \"success\", but the above two seem to be the main differences. The first of these two is worth trying but is not a conceptual contribution. The main idea of the paper is the second: self-training to generate diversified 'intermediate' initializations for further refinement. Basically it's using the initial refined sequences as a self-prior for exploring new sequences to refine.", "summary_of_the_review": "The core methodological idea is worth exploring, and ultimately publishing, in the context of sampling protein sequences. But, the novel part of the methodology is conceptually similar to other population-based sampling strategies like EDA (estimation of distribution) and to self-training, and those connections are not mentioned or contrasted. Importantly, the baselines are different in ways that aren't relevant to the core novelty, making it hard to draw strong conclusions about the core idea itself (versus the influence of all the other maybe-helpful-but-not-methodologically-interesting changes in the system proposed). I'm open to changing my rating, but this is my take.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "J7LhCO5BfNw", "reviewer": "Reviewer_jJy9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well written and easy to follow. The main ideas are clearly explained, the contributions and the empirical evaluation protocols and results are well presented.\n\nOn the strenghts side, the presented method make use of strong state-of-the-art components that are assembled to provide an end-to-end approach for de novo protein sequences and structures design. The comparison with similar approaches is well motivated and seems to be fairly adressed.\n\nOn the weakness, I would be happy to have feedback of the authors on the following questions:\n1) You state in section 3.1 that you do not consider Cysteine for experimental reasons? Why? Can you elaborate on the implied limitations of this choice?\n2) You present your work as a faster alternative to the trDesign approach. Why is there no computational times reported? How long does it take to your approach to match the perplexity or IG-Score of the seed sample generated by trDesign?\n3) Why do you mesure the IG-score on 1000 samples only?\n4) Section 4.2: this sentence is really difficult to understand without having to look to the appendices \"For further context, we calculated the mean pLDDT scores of recently released de novo designed sequences out of AlphaFold’s training set and found it to be high at 88.1 (σ = 4.41). This suggests that Good+ pLDDT is a good measure of a model’s ability to produce samples with a quality sufficient for design tasks.\"\n\nTypos:\n- Page 3: \"develped\"\n- Page 8: \"with with\"", "summary_of_the_paper": "The paper present a method to sample jointly de novo protein sequences and structures without relying on any structure conditioning. The main contribution is a new iterative procedure to generate a dataset of increasing size in an unsupervised fashion. This dataset is finally used to learn a generative model that captures the distribution over the \"unnatural\" sequence space, structures are predicted by using AlphaFold on the generated sequences. The authors provide empirical evidence of the quality (diversity and structural stability) of the de novo designs.", "summary_of_the_review": "The paper is clearly of interest. The approach is, up to my knowledge, a novel and wise way of assembling building blocks from various recent articles. However, in the current state of the article, there are still a few points that need to be clarified in order to be fully equiped to assess the potential of the approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C1ryAbZL12V", "reviewer": "Reviewer_mSTZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper provides an effective technique for generating sequences that are predicted to fold into a stable structure without any conditioning. The authors focus on de novo protein design which is left without a precise definition by the authors but appears to refer to designing sequences that fold into structures and are not \"similar\" to natural sequences. No further details about what similarity means or how it is measured are given. I personally find the problem a bit underspecified and it is not clear what a practical application of a method that generates sequences that are predicted to fold but are otherwise completely random is. Can the authors please explain further what the concrete goal is and what the application is?\n\nThe primary issue is that there is a lack of available training data. The authors therefore apply a technique used in trDesign to generate synthetic training data that they hope fit their criterion of folding to a stable structure. The final method iteratively samples from the generative model, optimizes the sequences such that they are predicted to fold using the trDesign approach, and then retrains the model on the generated sequences. \n\nWhile the method works, it appears to mostly be bootstrapping on the method proposed by trDesign. To put it another way trDesign is used to generate lots of sequences and then the authors train a language model on those sequences and demonstrate the resulting sequences are able predicted fold ~30% of the time by AlphaFold and are reasonably diverse. Most of the technical contribution seems to come from trDesign. There is nothing unique to the approach that encourages diversity or even consistency with respect to the distribution that they are trying to learn. It appears mostly to iteratively retrain itself on inputs generated by trDesign. \n\nWhile DARK improves over two baseline models, gcWGAN and gramVAE, according to their new pLDDT metric, these methods use different oracle functions for structure prediction. It is possible that the entire difference can be explained by using DMPfold2 rather than the structure prediction method in gcWGAN. As the paper currently stands, I do not find these comparisons compelling. \n\nIn order to improve the paper the authors should\n1) Provide a more concrete definition of de novo design and more clearly state what the goals of this paper are and why trDesign does not achieve those goals. Why do we need an unconditional model to generate sequences? What application does this solve that trDesign could not already solve? \n\n2) Better baselines could be performed by implementing other methods to use the same structure prediction oracle. \n\n", "summary_of_the_paper": "This paper presents an unconditional generative model that can produce sequences predicted to fold into stable structures.", "summary_of_the_review": "Overall, the authors present an interesting method that clearly achieves their goal of generating sequences predicted by AlphaFold to fold. I am left feeling that this problem is not clearly motivated and that the solution is not significantly novel from an ML perspective. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Despite a lively discussion and author explanation and revision, this paper remains below the bar for publication at ICLR. The technical exposition and goals remain poorly explained. The technical contribution is not sufficient. And the utility of the empirical results remain in question. The strong consensus among the reviewers who submitted reviews in a timely manner is that the paper is not suitable for publication.  The 5th and final review, was submitted too late, well beyond the end of the discussion period, and hence was not considered in this decision.
Decision: Reject