Question: You present your work as a faster alternative to the trDesign approach. Why is there no computational times reported? How long does it take to your approach to match the perplexity or IG-Score of the seed sample generated by trDesign?
Evidence: x s * ← Rapid optimization of x s * IG-score using the oracle 6:
x s n ← [x s * , x s n−1 ] Combine optimized samples with existing samples 7: end for DARK (Algorithm 1) tackles the primary limitation on effectively learning models, which is having suitable sequences to train and test against. The first step is to take some initial set of samples x s 0 to train an initial sampling model p γ 1 (x s 0 ) with parameters γ 1 ∈ Γ. Here we use the aforementioned seed samples, but this could be random sequences or sequences from DARK 0 -Grad, an approach we developed that backpropagates the negative IG-score to generate samples (Appendix A.2), in a fashion similar to. We use the seed samples out of convenience as they have the highest score, and we reasoned that they would require less iterations by comparison. The sampling models are all generative sequence models that can be sampled. After training p γ 1 (x s 0 ), it is used to generate a large set of samples x s * . Instead of then training a new sampling model on x s * , each sequence in x s * is quickly optimized to improve the IG-score. This is done by a quick greedy hill-climb on the sequence for a set number of steps (Line 5 in Algorithm 1), which we refer to as refinement. In each step, a random position is mutated. If it improves the IG-score then it is kept, otherwise it is discarded. We use 3000 steps, a fraction of the steps done in DARK 0 , as this was found to improve the average IG-score of x s * to approximately equal to the IG-score of x s 0 . For simplicity, this is fixed for all steps. The result of optimizing x s * is then combined with x s 0 to make x s 1 . In the second iteration, x s 1 is used to train a new sampling model p γ 2 (x s 1 ). After some number of iterations N , the final sampling model is viewed as a strong prior over x d , also making it a powerful generative model that can be used for de novo design tasks. Here we perform 3 iterations with training set sizes of 15K, 100K, and 500K, where the difference is sampled and refined between iterations. These sample sizes are arbitrary and they were entirely dictated by computational resources available at the time. Three iterations were performed as we believed it sufficient to demonstrate DARK's effectiveness. We note that the iterative aspect of DARK shares similarities to methods like Estimation of Distribution Algorithms (EDAs), a common approach in model based optimization (MBO). Viewed through this lens, it suggests a number of ways that DARK can be adapted and potentially improved on, which we leave for future work.
In DARK, we use deep autoregressive language models for learning protein sequence distributions. Specifically, we use a standard Transformer decoder architecture which has been used extensively to train state-of-the-art language models capable of generating high-quality synthetic sequence samples. In the first iteration of DARK we use a small decoder model, termed DARK 1 , with 4 layers, 4 heads, and a feed-forward size of H = 128. To affirm the choice of self-attention based architecture, we compare it to a 1layer LSTM with a hidden dimension H = 128. We also include results for a variant of DARK 1 referred to as DARK 1 -Adversarial, that uses DMPfold2 as an adversarial regularizer during training (Appendix A.3). After the first iteration, the second and third iteration models, DARK 2 and DARK 3 respectively, use a decoder with 12 layers, 12 heads, and H = 768. The parameters are increased to coincide with the order of magnitude change in the size of the training set from DARK 1 to DARK 2 .
To provide a even comparison to models trained on natural sequences, we train a model, termed UF50, on 50M natural sequences from the UniRef50 sequence database (The UniProt Consortium, 2021) using the same architecture as DARK 3 (Appendix A.5). For all trained models we perform early stopping and a small amount of hyper-parameter optimization on the validation set.
Quantitatively measuring sample quality We define high-quality samples as those that are diverse, have sequences unlike any natural sequence, and are predicted to have a stable and ordered structure. In de novo design, the most important test of a candidate sequence is that it is confidently predicted to have a stable and ordered atomistic structure. We provide a direct measure of both confidence and order jointly using AlphaFold's pLDDT score, being its confidence metric, which ranges from 0 to 100. This is a measure of confidence, and an indirect measure of order. It was found that low scores (pLDDT< 50) strongly suggest disorder and vice versa. Thus, we use the proportion of samples with a pLDDT> 70 as a measure of high quality for the predicted structures, which we refer to as Good+ pLDDT, indicating "good or better" quality according to criteria in. When discussing results we use 'ordered' to mean 'confidently predicted and ordered'. Using AlphaFold also provides a check to ensure that the model has not learned to produce adversarial examples against the original DMPfold2 oracle. We also measure the diversity of sequences and predicted structures using the estimated number of clusters in samples of sequences, and samples of predicted structures (See Appendix B.7 & B.8). Measuring the structural generality of DARK in a strict setting We take two key approaches to providing a stringent measure of how well DARK generalizes with regards to structure. Both are enabled by using AlphaFold to generate all-atom structure predictions for all 100,000 sequences contained in the DARK 2 training set, which covers the 15,000 seed samples and the 85,000 samples from the first iteration of DARK. We then assign each sample's structure to a topology. This process is the repeated for the 950 sequence validation set; we ignore the 950 sequence test set for what is described here. The first approach we take is to carefully construct a strict training and test set split based on structure, from the 100K samples (See Appendix B.2 for exhaustive details). This has been recently suggested for machine learning-based protein design studies and is considered a gold-standard approach in protein structure prediction. Our strict test set contains no sequences with overlapping topologies in either the training set or validation set. We also remove any sequences from the validation and training set that are detected to be similar to those in the test set with the MMseqs2 search tool . We train and evaluate a model on these sets, termed DARK 2 -STRICT. In the limit of working with synthetic, we believe this constitutes a stringent test of generality. The second approach we take is to compare the number of unique topologies represented by the 15,000 seed samples and those represented by the 85,000 sequences from the first iteration of DARK to show that DARK iterations generate sequences with unseen structures.
Table 1: Perplexities (PPLs), IG-scores, & Sampling Speed for protein language models. Lower PPL is better and higher IG-score is better. IG-scores are calculated from 1000 sequence samples except the 15K DARK0-Seed Samples. IG-scores for refined (Ref) samples are included for DARK models. Sampling Speed is for a single example (See Appendix A.6 for further details).
Answer: Thank you, we agree this should have been a part of the text. We have added significant timing information throughout and we believe it now forms a part of our argument. DARK3 without refinement can produce samples with similar IG-scores to our version of trDesign (DARK0 ) in 0.05 seconds. In contrast, DARK0 takes over 11 minutes (Table 1).