Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Dear Program Chair(s)
After reading the back and forth on this paper I feel it might help if i send comments _even though they are very very late 

main_review: Author's goal: Create a ML model that generates de novo sequences that fold.
- That fold could be anything at all. Simply "a folding predictor is confident that it folded my sequence accurately".

I think this is a fun idea, but too flawed. in application as it is a little circular  I am curious what the generated sequences look like... Not because the are likely to fold, rather because they are ones that "fool" AlphaFold.

Critique in summary:
The work boils down to an adversarial attack on DMSfold2, which proves to enrich for high scores in AlphaFold. But, adversarial attacks commonly create nonsense examples that exploit biases in the target model.
AlphaFold is not convincing validation, because it is too similar to the oracle used for optimization.
Both DMSfold2 and AlphaFold are misused. They've been trained & tested for multiple sequence alignments of natural sequences. There's very very little evidence that they accurately predict the fold of individual non-natural sequences.

The method and my problems with it:
1. Sequence generation = autoregressive sequence model. First, the AR model is trained on seed sequences. It then generates new sequences.
As a seed, the authors use sequences generated from trDesign (Baker lab 'hallucinations' model). Effectively, these are sequences predicted to fold according to Rosetta.

Problem = They've started with the answer, Rosetta designs.
- In the output, 922 / 950 seqs were ~33% identity to seed sequences.
- Need authors to include more detailed sequence comparison.
- That 33% might be enough 'cheating' to get high 'validation' scores, in which case they've just used Rosetta.
- They need to report the validation scores for the trDesign seed sequences too.
- Are the predicted folds for final output the same as for seed sequences?

2. Generated sequences are then mutated to optimize "folding score" = KL divergence of distograms (binned pair distances) according to DeepMetaPsiCov fold 2.
This should mean "optimize sequences so that predicted contact maps are highly non-random". With each iteration, the AR model is then trained on the optimized sequences plus the previous iterations' sequences.

Problem = They are optimizing a sequence generator to exploit a model OUTSIDE of its use-case. This is like an adversarial attack. I would expect to get sequences that exploit false biases. 

3. For validation the final output sequences are run through AlphaFold prediction & scoring = pLDDT (predicted Local Distance Test score). Authors claim AlphaFold is independent validation from the loss function they optimized against.

Problem =
AlphaFold is not an independent validation, because it is virtually the same model, trained on the same data as DMSfold2, and outputs the same sort of score.
- Trains on MSAs
- Trains on natural sequences
- Very similar architectures
- Scores, AlphaFold pLDDT vs DMSfold2 confidence. Both are predictions of distogram measurements, LDDT and TM, respectively. And trained on PDB structures of natural sequences.
Also a misuse of AlphaFold the same way as in #1. AlphaFold uses MSAs, and is trained/validated 99% on natural sequences.


Results, boil down to 2 weak observations:
1. The basic training task is possible: training the AR model indeed creates sequences that improve KL Divergence of DMPfold2 distograms.
- Non-result.. should be guaranteed that the ML training will improve objective score.

2. The output sequences have higher AlphaFold predicted LDDT than random sequences, sequences output by an AR trained on UniRef50, or two other models.
- Comparative advantage is versus other unproven models, but not e.g. natural proteins, or Rosetta designs.
- Advantage is all but guaranteed by similarity to DMPfold2, which was optimized for.
- Unclear that the AlphaFold pLDDT score is meaningful for non-natural sequences.
- Unclear if higher scores are byproduct of using trDesign seed sequences.


summary_of_the_review: Author's goal: Create a ML model that generates de novo sequences that fold.
- That fold could be anything at all. Simply "a folding predictor is confident that it folded my sequence accurately".

I think this is a fun idea, but far too flawed. I am curious what the generated sequences look like... Not because the are likely to fold, rather because they are ones that "fool" AlphaFold.

Critique in summary:
The work boils down to an adversarial attack on DMSfold2, which proves to enrich for high scores in AlphaFold. But, adversarial attacks commonly create nonsense examples that exploit biases in the target model.
AlphaFold is not convincing validation, because it is too similar to the oracle used for optimization.
Both DMSfold2 and AlphaFold are misused. They've been trained & tested for multiple sequence alignments of natural sequences. There's very very little evidence that they accurately predict the fold of individual non-natural sequences.

The method and my problems with it:
1. Sequence generation = autoregressive sequence model. First, the AR model is trained on seed sequences. It then generates new sequences.
As a seed, the authors use sequences generated from trDesign (Baker lab 'hallucinations' model). Effectively, these are sequences predicted to fold according to Rosetta.

Problem = They've started with the answer, Rosetta designs.
- In the output, 922 / 950 seqs were ~33% identity to seed sequences.
- Need authors to include more detailed sequence comparison.
- That 33% might be enough 'cheating' to get high 'validation' scores, in which case they've just used Rosetta.
- They need to report the validation scores for the trDesign seed sequences too.
- Are the predicted folds for final output the same as for seed sequences?

2. Generated sequences are then mutated to optimize "folding score" = KL divergence of distograms (binned pair distances) according to DeepMetaPsiCov fold 2.
This should mean "optimize sequences so that predicted contact maps are highly non-random". With each iteration, the AR model is then trained on the optimized sequences plus the previous iterations' sequences.

Problem = They are optimizing a sequence generator to exploit a model OUTSIDE of its use-case. This is like an adversarial attack. I would expect to get sequences that exploit false biases. 

3. For validation the final output sequences are run through AlphaFold prediction & scoring = pLDDT (predicted Local Distance Test score). Authors claim AlphaFold is independent validation from the loss function they optimized against.

Problem =
AlphaFold is not an independent validation, because it is virtually the same model, trained on the same data as DMSfold2, and outputs the same sort of score.
- Trains on MSAs
- Trains on natural sequences
- Very similar architectures
- Scores, AlphaFold pLDDT vs DMSfold2 confidence. Both are predictions of distogram measurements, LDDT and TM, respectively. And trained on PDB structures of natural sequences.
Also a misuse of AlphaFold the same way as in #1. AlphaFold uses MSAs, and is trained/validated 99% on natural sequences.


Results, boil down to 2 weak observations:
1. The basic training task is possible: training the AR model indeed creates sequences that improve KL Divergence of DMPfold2 distograms.
- Non-result.. should be guaranteed that the ML training will improve objective score.

2. The output sequences have higher AlphaFold predicted LDDT than random sequences, sequences output by an AR trained on UniRef50, or two other models.
- Comparative advantage is versus other unproven models, but not e.g. natural proteins, or Rosetta designs.
- Advantage is all but guaranteed by similarity to DMPfold2, which was optimized for.
- Unclear that the AlphaFold pLDDT score is meaningful for non-natural sequences.
- Unclear if higher scores are byproduct of using trDesign seed sequences.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: I think it helps to start by summarizing [Anischenko et al. 2020, Protein design by deep network hallucination], and then explain the differences in the submission.

[Anischenko et al. 2020] aim to generate AA sequences that are diverse but that still reflect the properties of training structures.
* Their goal is to generate protein sequences whose structures tend to be unlike those of random sequences. Since random sequences have structures that tend to be unordered, they incidentally generate sequences having ordered structures.
* They use the deep resnet from rtRosetta (similar to AlphaFold's) to predict interresidue geometry maps from sequence, and that is how structure is represented and compared.
* Their design goal is to design samples from p(structure | sequence) p(sequence).
* To represent p(structure | sequence) they use a proxy objective: the KL divergence between the predicted structures and the structures of random sequences; in other words, good sequences are those whose distance maps are very different from the distance maps of random sequences.
* To represent p(sequence) they just use -KL(AA frequences of predicted sequences || AA frequences of PDB sequences).
* Sequences are designed by starting from random and then annealing according to the proxy objective.
* Designed sequences are then clustered by template modeling score (TM-score) and selected (from each cluster) based on  a measure of prediction 'consistency'.


Now, the submitted manuscript (DARK) is the same objective and proxy objective as [Anischenko et al. 2020], but with following methodological changes:
1. For the sequence->structure oracle, instead of using rtRosetta's interresidue geometry maps as the structure representation, here distograms [Senior et al. 2020] are used, and are output from a modified version of DMPfold2.
2. Rather than relying solely on one round of "random initialization -> annealing", DARK uses an iterative self-training scheme. First they do one pass of [Anischenko et al. 2020] to get 1st round of annealed sequences. The annealed sequences from 1st round are used to train an autoregressive generative model (Transformer-based). That model is sampled from to generate additional "random initializations" (which are now guided, not purely random). These guided samples are then optimized (just hill-climbing now instead of annealing, for speed), and added to the training set for subsequent rounds, etc.

There are other differences from [Anischenko et al. 2020], like using pLDDT score (orderedness) as a measure of "success", but the above two seem to be the main differences. The first of these two is worth trying but is not a conceptual contribution. The main idea of the paper is the second: self-training to generate diversified 'intermediate' initializations for further refinement. Basically it's using the initial refined sequences as a self-prior for exploring new sequences to refine.
main_review: The larger methodological message of the paper seems to be that, if your goal is to generate samples of sequences in proportion to their score, rather than "start from N random sequences and anneal them to your objective", you can start by annealing a small number of random sequences, then that first batch to guide the generation of new (non-random) initial sequences to anneal. Eventually you can work your way up to N total annealed sequences, but most of them started with better initializations, so the hope is that the overall scores are better.

That's a reasonable thing to propose and to try in the context of protein sequence design. But I have moderate concerns about how this was depicted, because the paper doesn't really try to connect or contrast this scheme with larger ideas from sampling (e.g., "estimation of distribution" methods) or from machine learning (e.g., self-training). Even if those connections were made clear, ultimately I also find the choice of baselines confusing, because they are all so different from they don't get to the heart of what's different about this method, i.e., too many things change between the compared methods.



### Comments

I think the high-level methodology here can be viewed conceptually a kind of self-training within an EDA ("estimation of distribution") framework [see Figure 3.2 in Larranaga and Lozano 2002, Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation]. The paper would benefit if it were framed that way, and if it discussed success / failure modes of self-training in scenarios like sequence generation [He et al. 2020, Revisiting Self-Training for Neural Sequence Generation], classification, etc. 

Viewing DARK as an EDA/self-training of trDesign, I think also "stDesign" for "self-training design" would have been a more fitting name, attributionally, than something completely new like DARK. But that's OK.

The paper acknowledges multiple times that it builds upon Anischenko, and that's great. But the way the paper is written it often still fails to make clear what is new and what is borrowed. That is especially true of section 3.4, which is describing Anischenko's objective and proxy objective, but in different notation, and without making this clear.

One of the listed contributions is "By pairing DARK with AlphaFold, we show a fully neural way to unconditionally sample both protein sequences and structures jointly." Just some feedback on that: I think it's a bit of a stretch to call this a contribution. First, spart from self-training, the overall joint sampling scheme is essentially taken from [Anishchenko et al 2020]. [Anishchenko et al 2020] also mention that annealing could in principle be replaced with gradient ascent by backpropagating to the sequence, akin to which others like [Killoran et al 2017] attempted in simpler settings, which seems more "neural" than what the submission is doing.

The first listed contribution says that this is the first paper to "show that unconditional generative models... can capture the important long range dependencies that relate protein sequences with their structures." As far as I cans ee from Figure 2, it recapitulates alpha helices and beta sheets, of varying lengths; that's good, and you can probably argue that these are indeed long range (not purely pairwise, for example). But, you don't need deep learning to sample from the space of protein sequences that are broken up into stretches of beta sheets and/or alpha helices. Could the authors please comment on whether other structures were observed in sampling.

In most protein design situations, there is obviously a design criterion (affinity to a target, fluorescence, etc.), and that design criteron may call for conditioning on a family or families of proteins, or on desirable features of the protein sequence. Randomly sampling proteins that have "order" casts a relatively wide net, so I think it'd be helpful for the authors to talk about how they envision an unconditional sampler being used with respect to realistic design criteria. Are the authors envisioning to simply reject the ones that fail the criterion, from a huge pool of randomly sampled (but ordered) proteins? [Anishchenko et al 2020] go to the trouble of experimentally validating the secondary structures of predicted sequences, but they also do not express a clear vision in this regard.

By analogy to \beta-VAEs [Higgins et al 2017, ICLR], there might be a missed opportunity to tune the sample quality by introducing a simple hyperparameter that controls the weighting between the two KL terms.

The paper repeatedly talks of producing "high-quality structures" when it effectively means something more like "highly ordered structures" or "structures with high AlphaFold confidence." Whenever there's a specific notion of quality being pursued, the wording should reflect that, and it's a more fair way to characterize the situation when comparing to other generative methods that had different goals and therefore different notions of 'quality'.

Do the samples depicted in Figure 2 represent, proportionally, the kind of samples generated?

I am surprised that the paper does not include a "DARK_0" baseline, which would essentially be just the first round of [Anischenko et al. 2020] using the alternate representations.

Since DARK includes multiple rounds, and each round includes both training and hill-climbing, it seems fair to ask whether applying the same computational budget to "annealing from purely random seeds" baseline would produce comparable results and diversity. On some level, that is the most natural baseline (and other variations of it that would exhibit different diversity/score tradeoffs, e.g., different rejection thresholds, or pure hill-climbing). If the authors could please comment on this.





### Minor comments

page2: "to unconditional sample" -> "to unconditionally sample"

page2: please remove "rigorously"; the results in this paper are suggestive in nature

page3: "that does considers" -> "that does consider"

page4: "by Bayes" -> "by chain rule"

page4: Saying that one of these quantities 

page4: Symbol \mathcal{X} seems described as a "sequence space" (which V^L is) and is then treated as a set of training points. I think the sentence introducing \mathcal{X} and \mathcal{Y} can be tidied up. 

page6: "p_(y_d" -> "p(y_d"

page6: "distills" now has a specific technical meaning in deep learning so just be careful to avoid confusion.

Figure 2 is not referenced anywhere.
summary_of_the_review: The core methodological idea is worth exploring, and ultimately publishing, in the context of sampling protein sequences. But, the novel part of the methodology is conceptually similar to other population-based sampling strategies like EDA (estimation of distribution) and to self-training, and those connections are not mentioned or contrasted. Importantly, the baselines are different in ways that aren't relevant to the core novelty, making it hard to draw strong conclusions about the core idea itself (versus the influence of all the other maybe-helpful-but-not-methodologically-interesting changes in the system proposed). I'm open to changing my rating, but this is my take.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an unconditional generative model that can produce sequences predicted to fold into stable structures.
main_review: The paper provides an effective technique for generating sequences that are predicted to fold into a stable structure without any conditioning. The authors focus on de novo protein design which is left without a precise definition by the authors but appears to refer to designing sequences that fold into structures and are not "similar" to natural sequences. No further details about what similarity means or how it is measured are given. I personally find the problem a bit underspecified and it is not clear what a practical application of a method that generates sequences that are predicted to fold but are otherwise completely random is. Can the authors please explain further what the concrete goal is and what the application is?

The primary issue is that there is a lack of available training data. The authors therefore apply a technique used in trDesign to generate synthetic training data that they hope fit their criterion of folding to a stable structure. The final method iteratively samples from the generative model, optimizes the sequences such that they are predicted to fold using the trDesign approach, and then retrains the model on the generated sequences. 

While the method works, it appears to mostly be bootstrapping on the method proposed by trDesign. To put it another way trDesign is used to generate lots of sequences and then the authors train a language model on those sequences and demonstrate the resulting sequences are able predicted fold ~30% of the time by AlphaFold and are reasonably diverse. Most of the technical contribution seems to come from trDesign. There is nothing unique to the approach that encourages diversity or even consistency with respect to the distribution that they are trying to learn. It appears mostly to iteratively retrain itself on inputs generated by trDesign. 

While DARK improves over two baseline models, gcWGAN and gramVAE, according to their new pLDDT metric, these methods use different oracle functions for structure prediction. It is possible that the entire difference can be explained by using DMPfold2 rather than the structure prediction method in gcWGAN. As the paper currently stands, I do not find these comparisons compelling. 

In order to improve the paper the authors should
1) Provide a more concrete definition of de novo design and more clearly state what the goals of this paper are and why trDesign does not achieve those goals. Why do we need an unconditional model to generate sequences? What application does this solve that trDesign could not already solve? 

2) Better baselines could be performed by implementing other methods to use the same structure prediction oracle. 


summary_of_the_review: Overall, the authors present an interesting method that clearly achieves their goal of generating sequences predicted by AlphaFold to fold. I am left feeling that this problem is not clearly motivated and that the solution is not significantly novel from an ML perspective.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper present a method to sample jointly de novo protein sequences and structures without relying on any structure conditioning. The main contribution is a new iterative procedure to generate a dataset of increasing size in an unsupervised fashion. This dataset is finally used to learn a generative model that captures the distribution over the "unnatural" sequence space, structures are predicted by using AlphaFold on the generated sequences. The authors provide empirical evidence of the quality (diversity and structural stability) of the de novo designs.
main_review: Overall, the paper is well written and easy to follow. The main ideas are clearly explained, the contributions and the empirical evaluation protocols and results are well presented.

On the strenghts side, the presented method make use of strong state-of-the-art components that are assembled to provide an end-to-end approach for de novo protein sequences and structures design. The comparison with similar approaches is well motivated and seems to be fairly adressed.

On the weakness, I would be happy to have feedback of the authors on the following questions:
1) You state in section 3.1 that you do not consider Cysteine for experimental reasons? Why? Can you elaborate on the implied limitations of this choice?
2) You present your work as a faster alternative to the trDesign approach. Why is there no computational times reported? How long does it take to your approach to match the perplexity or IG-Score of the seed sample generated by trDesign?
3) Why do you mesure the IG-score on 1000 samples only?
4) Section 4.2: this sentence is really difficult to understand without having to look to the appendices "For further context, we calculated the mean pLDDT scores of recently released de novo designed sequences out of AlphaFold’s training set and found it to be high at 88.1 (σ = 4.41). This suggests that Good+ pLDDT is a good measure of a model’s ability to produce samples with a quality sufficient for design tasks."

Typos:
- Page 3: "develped"
- Page 8: "with with"
summary_of_the_review: The paper is clearly of interest. The approach is, up to my knowledge, a novel and wise way of assembling building blocks from various recent articles. However, in the current state of the article, there are still a few points that need to be clarified in order to be fully equiped to assess the potential of the approach.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a way to generate novel protein sequences that are likely to have folded structures. To do so, they start with a pre-trained sequence-to-structure predictor, and use simulated annealing to generate a set of initial sequences that are likely to have a structure. These seed sequences are used to train an unsupervised sequence model (an autoregressive transformer). More sequences are generated from this sequence model, and they are optimized to maximize the likelihood they have structures (using the pre-trained sequence-to-structure model), and are added to the growing set of training sequences. This process is repeated for several iterations to finetune the unsupervised sequence model. The authors show that sequences generated from this sequence model are likely to be structured as predicted by AlphaFold.

main_review: ### Generated sequences are highly likely to be extremely biased toward a specific subset of sequences
The way these sequences are generated is that they start with some seed sequences optimized out of DMPfold2. Then, the sampling model $p_{\gamma^{n}}$ is fit to this distribution, and then used to generate sequences from this distribution. These sequences are then further optimized using the same DMPfold2 model to generate more training examples. Throughout this procedure, $p_{\gamma^{n}}$ is almost certainly going to learn a very similar distribution to the initial seeds, and is unlikely to generate novel examples beyond these seeds which are significantly different in any way. Furthermore, these seeds are generated from a single model (DMPfold2). This already biases these sequences toward the sequences used to train DMPfold2.

Overall, this iterative process (i.e. sampling sequences and maximizing IG-score) doesn't seem like it would help at all with generalizing to novel sequences, and only serves to focus the sequence model on a specific sub-distribution. I expect any improvements in structural likelihood that stems from iteration to just be refining the structural likelihood of the initial seed sequences, which were selected suboptimally (via simulated annealing).

The number of sequence clusters (as a measure of the diversity among generated sequences) was shown for the three DARK iterations, but it is not clear how to interpret numbers of 1000, 973, and 962. Is that big or small? How does that compare to the baselines? Also, having 1000 clusters for 1000 sequences may suggest that this metric is not very reasonable to begin with. With that many sequences, one would expect at least a few clusters of size greater than 1 just by random chance (usually).

Furthermore, the authors claim that the 1000 samples drawn from DARK have no strong similarity to natural proteins in UniProt. This needs to be quantified.

### The model is still conditioned on structure
Even though the paper claims that the DARK method is unconditional, it still depends very heavily on structure. Structural information is coming in from the conditional model DMPfold2, which has an extremely central role in the algorithm. Thus, even though the final model $p_{\gamma^{n}}$ only considers sequences, it has structural information built into it. Specifically, it is _conditioned_ on the initial seed samples and DMPfold2.

### There is no real validation of the generated sequences' structures
Using AlphaFold is unfortunately not a convincing validation of these sequences' structural likelihoods. Although it is technically a different neural network from DMPfold2, it is still only an estimate with no theoretical guarantees. That is to say, it is not "real". Without the ability to synthesize and test these generated sequences for structure experimentally, one could use existing structured proteins and use the sequence model to score these sequences. The model should give structured proteins a higher score than disordered proteins.

### Outperforming natural sequences is concerning
The fact that the model doesn't seem to do so well with natural sequences is very concerning, because natural sequences (excluding those that are disordered) are typically highly thermostable and thus structured. In fact, natural sequences in many ways should be the upper bound of how well the generating model performs (unless the claim is that these generated sequences are _more_ structured than natural sequences).

### Missing some baselines
Previous works have also attempted to generate _de novo_ proteins. For example, Alley, et. al., 2019 (UniRep) can be used to generate sequences that can also be scored by AlphaFold. Anand & Huang 2018 also generate novel proteins that are structured. Notably, UniRep is truly unconditioned on structure (i.e. there is no notion of structure in their model), whereas DARK still conditions on structure. In many ways, DARK lies on the spectrum between these two previous works (in terms of reliance on structure).

It should also be noted that UniRep is capable of generating sequences that are structured, which is shown in the paper.

### Some choices in the algorithm are unclear or of questionable soundness
The choice of the loss function is not quite clear. For example, why use the background model $p_{\epsilon}(y_{d})$? Why not just maximize $p_{\beta}(y_{d}\vert x_{d})$? Additionally, is the sequence prior needed? What happens if it is removed? Is there a weight $\lambda$ for these two halves of the loss function?

It is also not clear why examples are concatenated (step 6 of the algorithm). Do old examples need to be present even though larger and larger sets of sequences are being generated?

The use of a different model architecture for each iteration of DARK is concerning. This suggests that the developed method may be extremely overfit to this specific task/dataset. There doesn't seem to be a justification for the different architectures.

Also why are Cys residues removed? Given that disulfide bridges are a critical part in maintaining tertiary structure in many proteins, a model that does not consider their existence raises some red flags.

summary_of_the_review: The problem of generating novel protein sequences is a popular one. Although this submission develops a somewhat novel approach to solving this problem, the method's success is dubious, if not absent. The overarching claim that the method is unconditional is not quite true, and the sequences generated are almost certainly highly biased and limited. There is also no real validation of the generated sequences, and minimal comparisons to existing work in the same space. For these major methodological and technical issues, I unfortunately cannot support its acceptance.
review_title: null

Meta-review: Despite a lively discussion and author explanation and revision, this paper remains below the bar for publication at ICLR. The technical exposition and goals remain poorly explained. The technical contribution is not sufficient. And the utility of the empirical results remain in question. The strong consensus among the reviewers who submitted reviews in a timely manner is that the paper is not suitable for publication.  The 5th and final review, was submitted too late, well beyond the end of the discussion period, and hence was not considered in this decision.
Decision: Reject