Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this work the authors propose a framework for combinatorial optimisation problems in the conditions that the measurements are expensive. The basic idea is to make an approximation of the reward function and then train the policy using the simulated environment based on the approximated reward function. The applications are shown in a set of biological tasks, which shows that the model performs well compared to the baselines. 

The idea of learning the models of environment (or reward) and simulating the model to train the policy is not novel (e.g., https://arxiv.org/pdf/1903.00374.pdf). Similarly, in terms of formulating the discrete search problem as a reinforcement-learning problem, again there are similar works in the past, which are cited in the paper, but the combination of these two is novel to my knowledge; having said this the paper should discuss relevant works such as the one above. 

The experiments seem convincing to me overall, however, I have the following concerns: 

- The performance of the model seems similar to PPO in the large state-spaces (section 4.3), which somehow is disappointing.

- The performance of the model seems very sensitive to the choice of \tau (Figure 6 right), which is set to 0.5, but it is not mentioned how this parameter is chosen (or at least I couldn’t find it) and how much the performance of the model in the other experiments is affected by the choice of this parameter.  


review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: Designing new discrete sequences satisfying desirable properties is an important problem in molecular biology. This is a difficult combinatorial optimization problem because of the difficulty in optimizing over a combinatorially large space. The authors propose a RL based framework for this black box optimization problem.

The paper is well written, but I have questions about the efficacy of the method, particularly because I think some of these results are against weak baselines. For example, the authors don't compare against many better performing protein design methods (See: Ingraham et. al, GENERATIVE   MODELS   FOR   GRAPH-BASED   PROTEIN DESIGN, Sabban et. al, RamaNet: Computational De Novo Protein Design using a Long Short-Term Memory Generative Adversarial Neural Network). VAE based methods have worked well for designing sequences like SMILES strings, but the authors dismiss them claiming that they are better modelled as molecular graphs. While it could be true that molecules are better modeled as molecular graphs, it is not clear why methods that have worked well in a sequence based modeling using SMILES strings will not work for Protein Design. For AMP Design, again they compare with a weak baseline and don't compare with VAE based methods (like for example: Das et al. PepCVAE: Semi-Supervised Targeted Design of Antimicrobial Peptide Sequences)
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Contribution
This paper apply a model-based RL algorithm, DyNA-PPO for designing biological sequences. By being model-based, this algorithm is sample efficiency compared to model-free RL algorithms. This advantage is attractive and important in the context of biological sequence design since the designed is constrained to be done in the large batch / low round settings. To further improves model efficiency, the authors reduce learning bias by quantifying the reliability and automatically selecting models of appropriate complexity via cross validation. To encourage diversity in the target distribution they also penalize the reward using a visitation-based strategy.


Clarity
Overall, the paper is well written, well motivated and well structured. The technical content is also very clear and good.


Novelty
The novelty in this work seems to be more on the applicative side (RL to optimizing DNA and protein sequences) than the method itself. I agree with the authors that most existing optimization methods are ill equipped for the large batch / low round settings and as sample efficiency becomes critically important as the number of round gets lower and their method is a good solution in such settings. 

The technical novelty seems incremental as cross-validating and using a set of models under particular performance constraints does not constitutes a novel contribution. Penalizing the reward if the same sequence is seen multiple times seems decent and works well compared to the entropy regularization but it is still questionable if it is the best solution for biological sequences. Showing results when the reward is penalized using the hamming loss or the biological similarity  with previous sequences could go a long way to convince of your choice.


Experiments:
The experiments are overall well presented and seems robust given the number of replicates that was made each time. 
Analyzing the model performances across different metrics: diversity, fraction of optimals, cumulative maximum helped to  understand the method and its advantages. 

However, I would like to see other RL algorithms that were shown in the appendix for all those comparisons.
Including MCMC methods in the experiments will also allow to see how RL methods compared to  the sota in bioinformatics.
For the Icing dataset, you mention that it is a contribution but do not provide enough details regarding it to allow further research with it.

In a real life biological setting, the data obtained at each batch will be more likely different both in term of sequences but also in term of labels. How all your method (and the others) perform with changing distribution of data from one batch to another. Ex (e.g. 0 < y batch 1 < 100, 100 <= y batch 2 < 500, etc) ?

You tried two values for R^2 in one experiment (one positive and one negative). What happens for any other positive values (e.g 0.1. 0.2, 0.3, etc) ?


Points of improvement
Given the applicative nature of the paper and the proposed method there are few small experiments that could have been done to strengthen the manuscript (see questions and comments above). 

Preliminary rating:
* weak accept *
review_title: Official Blind Review #2

Meta-review: The paper proposes a model based proximal policy optimization reinforcement learning algorithm for designing biological sequences. The policy of for a new round is trained on data generated by a simulator. The paper presents empirical results on designing sequences for transcription factor binding sites, antimicrobial proteins, and Ising model protein structures.

Two of the reviewers are happy to accept the paper, and the third reviewer was not confident. The paper has improved significantly during the discussion period, and the authors have updated the approach as well as improved the presented results in response to comments raised by the reviewers. This is a good example of how an open review process with a long discussion period can improve the quality of accepted papers.

A new method, several nice applications, based on a combination of two ideas (simulating a model to train a policy RL method, and discrete space search as RL). This is a good addition to the ICLR literature.
Decision: Accept (Poster)