Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors implement an open-source simulation environment FLEXS to emulate complex biological landscapes (TF binding landscapes) and to train and evaluate (RNA or protein) sequence design problems. ‘Swampland’ landscapes with large areas of no gradient, are also studied.
They also propose a simple greedy-search evolutionary algorithm, Adapt-with-the-leader (ADALEAD) that is shown to perform better than its competitors and to optimize in challenging settings. 

The paper is well written and motivated and previous, relevant literature is discussed. 

Comments: 
>> An important component of the environment and for ADALEAD is phi^prime. The paper mentions that this is fully supervised. How is one to come up with the appropriate	phi_prime for a problem at hand. 

>> For the experiments, are the Ensembles always a 3-CNN model? Also, are the CNN architectures always the same?

>> Can non-neural net architectures be used as ensembles? 

>> In the Figure 1A, it seems that the schema is iterative between the 4 layers shown by the arrows. Is this the case? If so, the iterative aspect is not mentioned clearly in the paper. 

>> What is EI in the Bayesian optimization subsection?
review_title: ADALEAD+FLEXS for sequence design problems

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this work, the authors propose a greedy search approach for designing biological sequences in an active learning, batch setting. The algorithm is a fairly standard evolutionary algorithm which identifies the set of candidates at each iteration by adapting the best candidates from the previous iteration, and then evaluating those adaptations using a surrogate model. A set of experiments (using an evaluation sandbox also proposed by the authors for this work) suggest the proposed approach finds more high-quality solutions than competing approaches; however, especially in the experiments using the realistic/empirical surrogate model (an ensemble of 3 CNNs), the quality of the best solutions found by several approaches are statistically similar.

As promised by the authors, the proposed approach indeed appears rather straightforward to implement. Further, Table 1 does show that the proposed approach identifies more sequences which meet the specified criteria. To the best of my knowledge, the paper also places the work into context with respect to existing work.

As a non-expert approaching this paper, I find it very difficult to gauge the impact of the choice of the abstract and null models in the experiments. In particular, all of the results (and especially Table 1 and Figure 2B) suggest that the proposed approach is much better than the competition when using these models. On the other hand, when the ensemble is used as the surrogate model (e.g., Figures 2C(bottom) and 2D(left)), the performance of several methods become very similar.

Since the methodological contribution of this work is rather limited (again, a pretty standard evolutionary algorithm), it is concerning to me that the strongest empirical results only arise in the settings based on these choices.

Concerning the diversity of solutions found, in Table 1, it is not clear to me how meaningful the diversity measure (number of local optima with a score above the given threshold) actually is. Especially in cases where many solutions are found, it would be useful to also characterize the diversity of the sequences themselves, for example, by using BLOSUM or some other sequence similarity measure.

Of course, actual wetlab validation demonstrating the superiority of the proposed approach would significantly strengthen the contribution.

Minor comments
----------------

The references are not consistently formatted.

Table 1 should include some measure of variance.

When introducing “optimization”, the authors rightly point out that, e.g., moderate binding may be preferable to stronger binding in some cases. However, they also suggest that the stronger binder has a better fitness. To me, this suggests the model for fitness is wrong and fits better with the “robustness” theme (since an accurate fitness model would have a parabola shape or some such in this example). It is also not clear to me whether any of the objective functions for the simulations capture this phenomenon.

I found the description of the RNA ground truth simulation unclear.
review_title: Limited methodological contribution, unclear impact of experimental design choices

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this work. the authors addressed a critical issue on biological sequence design. It is an important task in biomedical research. The authors proposed an evaluation/simulation framework along with an evolutionary algorithm and showed the simple evolutionary algorithm "AdaLead" outperformed several existing methods. The paper is clearly written. 

pros: 
1. Addressing an important issue. Sequence design and optimization are critical for many real-world problems, such as antibody optimization, etc. 
2. The authors have surveyed and tested many existing algorithms. They have also considered many settings such as DNA, RNA, and protein as well as "swampland" where the optimization landscape is flat. Experiments are done thoroughly.  
3. Offered an open-sourced sandbox for model comparisons and proposed and evaluated a simple yet strong-performing evolutionary algorithm that should be used as a baseline in further works in this space.

cons:
1. First of all, although I appreciate a thorough investigation and comparison between existing methods and implementing an evolutionary algorithm as a strong baseline, it is, however, unclear to me the ML significance of this work to the ICLR community. It might be more suitable for a specific compbio venue. 
2. The authors proposed an evaluation framework where a noisy version of the oracle learned from sampling, $\phi^{'}$, could be obtained. Although this is a valid and reasonable framework, it is not fully in line with other algorithms' assumptions. For example, DbAS and CbAS are more conservative and exploring locally by generating a new batch of sequences that are similar to the best performing one in the previous batch. This can be also a valid approach as the extrapolation might be very bad. Moreover, the authors tuned AdaLead on their proposed evaluation framework and claimed it is not sensitive to parameters, I do not see any results regarding this matter.  In short, I suspect the evaluation framework proposed here is biased towards a simple evolutionary algorithm that does not even use much information (as AdaLead outperforms other more complicated evolutionary algorithms in fig A1B). 
3. Related to 2, there is a lack of exploration to understand why AdaLead is outperforming. What could we learn from this result? 

minor:
1. The authors talked about batch size issues in the paper. Could they give more details or some results on this? 
2. Some of the figure labels can be improved. There are some acronyms likely not familiar to computer scientists with little biological background left unexplained, such as "w.t." in fig 1B.
review_title: An empirical study on sequence design&optimization that might not be suitable for ICLR

final_score: rating: 3: Clear rejection
review_content: The authors introduce a new algorithm AdaLead to solve the problem of efficient design of biological sequences and FLEXS, an open-source simulation environment for sequence design. 

The structure of this paper is somewhat unconventional, the paper would benefit from more rigor and structure. There are 4 sections, the method section which is the section that contains the novel part (section 2) is a fairly short section (less than 2 pages). The experimental setting is crammed together with the results. So, the details of the outcome of the experiments are lost in the various implementation details. The experimental setting can be moved in a section where the authors talk about the benchmarks alone. Or perhaps in the appendix.  The authors talk about the contributions at the very end of the paper. A reader wants to know what the contributions of the paper are in the introduction section. It wasn't very clear for example that FLEXS was the main contribution of this paper until the very last section. FLEXS was barely mentioned at the beginning of the paper. 

The second contribution of the paper according to the authors is the novel algorithm AdaLead. This evolutionary algorithm is poorly explained. How are the recombine and mutate operators defined for example? In addition, this algorithm seems simplistic and I'm not sure we can call this algorithm a novel contribution. The authors should make more explicit why is this evolutionary algorithm a novelty, how does that improve wrt EAs? 

The authors dismiss the use of Bayesian optimization by saying that BO doesn't perform well on high-dimensional space and citing a tutorial by Frazier 2018 that says that usually BO is used on problems that have less than 20 dimensions. I'm not entirely sure why this is a problem for this work where Table 1 shows that the applications considered have a few variables only, up to 14 if I understand correctly. In addition, some BO frameworks such as SMAC (https://www.automl.org/automated-algorithm-design/algorithm-configuration/smac/)  were used on problems with more than 100 variables. 

The authors compare against BO by implementing a BO algorithm based on EI and an evolutionary sequence generator. While it is not clear what this means, this is surely an uncommon BO algorithm. Why didn't the authors use one of the standard BO packages available? The experimental results are unsatisfactory without a proper BO comparison. 

Finally, I question the interest of the ICLR audience for this type of study. While the paper would follow in the track: "applications in audio, speech, robotics, neuroscience, computational biology, or any other field" I found the algorithm being a specific application of a simplistic search algorithm to the domain of sequence design. What are the lessons learned? Is there any general insight that became available from this study? The improvement wrt to SOTA is shaky. 

Minor remarks: 
- Figure 2 is not cited in the text. 
- CMA-ES is not introduced - even if the CMA-ES is a popular algorithm it should be properly introduced.
- Model-free evolution in Figure 2 is not introduced. What is that algorithm about? 
- Figure 2, I'm not sure what is the y-axis in the performance plot. In these types of papers, we usually show simple regret or similar performance quantities. It seems that the y-axis is the higher the better and that 1 is the max achievable? 
- Figure 2 should give a better understanding of what the figure means and what are the methods that perform best by writing about that in the text of the paper. 
- Not clear to me why the queries to the surrogate model are sample-restricted. This is an uncommon setting in Bayesian optimization where usually the budget is restricted for the evaluation of the latent black-box function.  
- Not entirely clear what the role of the cardinality |S| in the optimization is. Is this an optimization with unknown feasibility constraints problem formulation? 


review_title: The authors introduce a new algorithm AdaLead to solve the problem of efficient design of biological sequences and FLEXS, an open-source simulation environment for sequence design. 

Meta-review: This paper considers the problem of (biological) sequence design and optimization. The authors made an interesting yet important case that in certain sequence design tasks, a simple evolutionary greedy algorithm could be competitive with the increasingly complex contemporary black-box optimization models.

Most reviewers appreciate the design of the open-source simulation environment in benchmarking AdaLead (and other competing algorithms) in a number of biological sequence design tasks (e.g. TF binding, RNA, and protein). However, there is a common concern that the experimental results are not fine-grained enough to explain the outperforming results of the proposed algorithm. There are also unresolved comments on missing important BO baselines in the empirical study. As a purely empirical work, these appear to be important concerns. While these results appear to be useful for the domain of biological sequence design, the reviewers are unconvinced that the proposed algorithm is significantly novel, or the results are sufficiently compelling to merit an acceptance to this venue.

Decision: Reject