Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The manuscript presents a procedure for protein engineering using a model-based reinforcement approach building on the ESM2 language model. The authors demonstrate that the high dimensional ESM2 representation can be mapped to a lower dimensional representation space which is suitable for optimization, and that full amino acid sequences can be reconstructed from this reduced representation with meaningful accuracy. They then propose a off-policy reinforcement learning procedure to learn how to make updates in the representation space. Finally, the authors evaluate the method on two protein engineering datasets, and report convincing results.
strength_and_weaknesses: The paper is well written, the method well described, and the results are convincing. Please see below for detailed comments to specific parts of the paper.
clarity,_quality,_novelty_and_reproducibility: ## Clarity
The paper is clearly written, with excellent figures to support the story.

## Quality
The quality of the paper is high.

## Reproducibility
The authors have not made source code available as part of their submission, making it difficult to fully assess reproducibility. The method is, however, presented at a level of detail that should make it possible to reproduce the results. I strongly encourage the authors to share their source code upon acceptance of their paper.

## Detailed comments

Page 1. "To tackle this problem, in this paper we propose..."
This paragraph, and especially this sentence, suggests that latent space optimization of proteins is new, ignoring recent work on Bayesian Optimization of proteins on latent spaces, such as "Accelerating Bayesian Optimization for Biological Sequence Design with Denoising Autoencoders" (ICML 2022). This paragraph should therefore be rephrased.
This same ICML2022 reference should also be added to the "Biological sequence design" subsection on the next page, and ideally compared to in the results section. I am not affiliated with this paper in any way - but merely suggesting it because the methods are closely related, and both are contestants to constitute the current state of the art - so comparing them head-on would be relevant to the community.

Page 3: "k is predicted from a representation q with dimensions (L, E), and". Page 4: "As a result of performing the action a_t, the agent receives the reward r_t.". Page 5: "a dense reward, defined as r_t= f(s_t)". 
The last two statements seem to be at odds with the first statement. Originally, f is described as acting on a q representation of size (L,E), but later, the reward is calculated based on s_t, which is only E-dimensional. Please clarify.

Page 4. "The dataset proposed in (Sarkisyan et al., 2016) is used to train the GFP encoder-decoder and its functionality predictor."
It would be helpful if you could discuss somewhere in the paper how many parameters are involved in estimating the task-specific projection to lower dimensions (I assume that this is just ExR) and whether this estimation becomes problematic for smaller datasets that the one you studied here. Likewise for the oracle - although one could presumably use a general oracle instead of a task specific one in this case.

Page 4. "Datasets were split into train and test sets." How? Just uniformly?

Page 5. "We compare with four optimization methods"
I assume these are all optimized on the same oracle(?). As mentioned earlier, it would be beneficial to the community if you compared directly to the ICML2022 method here as well - if at all possible.

Page 6. "The sequence alone cannot easily convey structural information about a protein’s functional sites, making it unsuitable for guiding the optimization process."
I'm surprised by these results. In Table 2, you show that Random mutations work well as an optimization strategy on His3. Why does it fail completely in the experiment in Table 3? Wouldn’t you at least expect to recover the most important sites in the protein? Does this result perhaps primarily indicate a weakness in your policy optimization rather than the representation itself?

Page 7. "...is compared to random perturbation and BO,"
When writing “BO”, are you referring to a general Bayesian Optimization procedure - or the specific one that you cite earlier (Swersky et al., 2020)? Since you are using a continuous latent space here, I assume this is now a different BO - in which case you should make the distinction clear - and explain what the setup is - is it a standard GP-based BO with a expected improvement acquisition function? I would also suggest that the authors make it clear that this is just one particular (and perhaps particularly simple) choice of BO.

Page 7. "Fig. 2 shows that the functionality predictor trained without negative examples incorrectly predicts a high value for non-functional sequences (mean=4.002)"
Isn’t it odd that the oracle predicts a higher average functional value than any value it has observed during training? Does this indicate something is flawed with the oracle?

Page 7. "Fig. 3(d) presents optimization steps taken by the trained policy, which now shows that large (optimistic) perturbations taken by the policy often lead to failure in optimization."
It was not quite clear to me what this figure is meant to illustrate. Are the large steps failure modes of the learned policy? (i.e. should it have learned to prevent such steps?)

Page 7. Discussion
It would be helpful if the authors somewhere in the paper discussed the source of improvement over e.g. Bayesian Optimization. The BO and reinforcement learning literatures have quite different terminology and it can be a bit difficult to see exactly which components make a difference in practice. 
Is it the fact that a policy is learned vs the fixed acquisition function typically used in a BO setting? 


### Minor

Page 1. "The first cause is that the optimization process is usually performed by generating candidate sequences through amino acid substitutions"
Slightly odd statement, since any method (including the one proposed here), will ultimately use amino acid substitutions (or insertions/deletion). Perhaps writing "sequences *directly* through amino acid substitutions" would be better?

Page 3. "a 2-dimensional vector representation q ∈ Q of dimensions (L, E)"
Slightly confusion that the representation is both 2 dimensional and has dimension L,E. Consider rephrasing.

Page 4. "Negative examples are defined as random sequences with a zero functionality value."
Do you standardize the functionality values in any way? Otherwise, 0 seems like an arbitrary value.

Page 5. "We set three experimental rewards". At this point in the text it is not clear whether these losses will be used simultaneously or as alternatives. Perhaps add "alternative" here to make this clear.

Page 5. "The performance evaluation metric is calculated as the mean log-fluorescence intensity from the top K generated sequences."
For clarify, perhaps make it clear that this is according to the oracle, and not the ground truth values.

Page 5. "CbAsoptimize"
Missing space
summary_of_the_review: The paper proposes a new method for protein engineering. It is well-written, carefully documents its claims, and demonstrates convincing results. I have only minor suggestions for edits to the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a model-based reinforcement learning method for sequence design. The main innovation is that design is performed in the latent space, which should allow easier optimization given the combinatorial search space. Furthermore, the authors point out the flaw of using oracles trained on experimental data as the sole source of truth, as they give high functional scores to random sequences. They demonstrate that adding random sequences to the training set alleviates this issue.
strength_and_weaknesses: Strength:
* The paper is theoretically sound. The issues identified with current approaches are real, and the proposed solutions are sensible.

Weaknesses:
* The paper is not carefully written and it was hard for me to find information about architecture, training setup, and experimental results. Examples:
    - The paper says the CLS token is used for an embedding of a sequence. Further it says average pooling is applied to this vector to reduce dimensionality. I am not familiar with average pooling being used for dimensionality reduction. The method should be explained or a citation provided.
    - While the paper says CLS tokens are used for dimensionality reduction, this is not clear from Figure 5. 
    - The architecture of the MLP is not described in section A.1.
    - In section A.1, the paper states that encoder weights are not updated during fine-tuning, but the next sentence states that the encoder and decoder are jointly finetuned.
    - The decoder is not introduced in section A.1.
    - "Note that the information on mutated positions such as masks is not provided to the model." - I don't understand this statement. It would be nice to have a clearer explanation of the training procedure.
    - Details about training set split are not provided.
    - It was unclear how many total sequences the model was trained on and how many total sequences were seen in a given optimization round.
    - In section 4.3, it states that "only the latent vector [...] can simultaneously optimize GFP and His3 tasks." I'm not sure what this means -  how are the tasks being simultaneously optimized?
* The proposed method is initialized from sequences with relatively high functionality. The method proposed is able to optimize further from there. They also show a random mutation baseline that is similarly initialized. However, it does not seem that other methods are provided this initialization. This should be clearly discussed and some effort should be made to make this initialization.
* It's not clear what the highest functional value in the training set is. Is it higher than the values the model ultimately achieves?
* Is the encoder/decoder trained only on the training set or on the whole dataset?
* Figure 1 may benefit from providing a more detailed view of the process (what are the components trained on, how does sampling occur, etc.)

Minor Comments:
* The CLS token used for a protein-level vector but in ESM-2 it has no special meaning. Would averaging the final layers provide a better representation?
* Abstract: "GPF" -> GFP
* Page 5: "CbAsoptimize" -> "CbAs optimize"
* In tables, I personally dislike the use of "Baseline" as an entry. Especially given that there are other methods you compare against, it would be nice to clearly define what this is in the table.

clarity,_quality,_novelty_and_reproducibility: Clarity: There are major issues with the clarity of writing. It is difficult to judge other components as details about experimental setup, training procedures, etc. are missing. This needs to be addressed before publication.
summary_of_the_review: The paper proposes a theoretically sound procedure, but many small details are missing that make it difficult to assess the results. Clearer explanations of all aspects of training, model architecture, and experiments are needed. Additionally, it is possible the experimental setting is unfairly advantaging the proposed method by initializing from high-function sequences. Some attempts should be made to provide similar initialization to other methods, or it should be explained why this is not possible.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to generate protein sequences with high functionality and cellular fitness. The authors apply a reinforcement learning framework to walk through the space of latent representations of a pre-trained protein language model, using a functionality predictor as a reward function. This approach is tested for its ability to design examples of proteins GFP and His3 with predicted functionality, novelty, and diversity. The paper also tests improving the functionality predictor for use in reinforcement learning by including negative examples in training.

strength_and_weaknesses: Strengths:The approach combines existing models in a potentially interesting way : learning from the universe of natural proteins and labeled data from a large dataset of synthetic proteins with negative examples 

Weaknesses:  
 I believe the paper has a couple of  fundamental weaknesses as it stands that should be addressed before it could be accepted at ICLR. 1. While the evaluation is conceptually laudable  (performance, novelty, diversity), the metrics do not seem to capture the concepts well. Performance is evaluated irrespective of the distance from training.  Since ‘Novelty” is evaluated as binary- was in the training set or not. (I think - see equation in 3.1), the metric seems to have no measure of distance from training beyond a single mutation. (It is now well appreciated that predicting the effect of single mutations is relatively successful with even baseline methods such as conservation, Potts models or VAEs with alignments and transformers without alignments.) Therefore - for this piece of work to be evaluated I suggest it’s important to show sequence generation as a function of the distance from training data. A fundamental challenge in protein design is being able to generate sequences with a given function that have sequences different from natural or training examples. As one moves away from known sequences (in eg Hamming distance) - the harder it gets. For sequences that are only one  mutation away is relatively easy. ( many papers have shown this).  The performance results shown in Tables 1 indicate that their method is only 1% better than a  random single mutation for eg GFP, Table1, suggesting the metrics and/or the model is poor. Although the authors note this point , they do not follow up by addressing the reasons. 
2. The use of he Oracle twice is circular - therefore invalidates the claims of performance; there are some ways around this that the authors could try 
More minor weaknesses: 
3. the reference used to justify the evaluation metrics is Hoffman et al 2022 - but this paper is about optimising small molecules - which are v different in "seq distance to function" relationships -  this is especially important in relation to the point about the Novelty measure above.  
4. AlphaFold is not at all appropriate to support the claim of functional sequence optimisation - there are may mutations that will cause a protein to unfold that Alpha fold will predict as having almost exactly the same structure as it will align etc - therefore it proves nothing ( From their own FAQ page "AlphaFold has not been validated for predicting the effect of mutations. In particular, AlphaFold is not expected to produce an unfolded protein structure given a sequence containing a destabilising point mutation." And there are papers writing about this  eg Pak et al 2021 

clarity,_quality,_novelty_and_reproducibility: The paper contains some nice detail in the models but it would be great to have an anonymous github, more detail about test/train splits, more detail on the datasets used would be useful for non domain experts. The pipeline needs to be better articulated too - I am not sure of soem of the notation
Related work: some key work seems to be missing eg Gomez-Bombarelli et al (2018) property prediction on the latent space for design, Notin 2022 Tranception - and results on GFP and HIs3, 
summary_of_the_review: The paper has an interesting and somewhat novel approach but I am not convinced by the Performance and Novelty evaluation (as it is explained here) and I am uncertain about the AlphaFold result being informative as this would be the result of any single mutation even when known to destabilise the protein. This point about evaluation is non-trivial as papers like these should be setting the bar for future work; train test splits that are commensurate with the claims of the work and avoidance of circularity are really critical.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper combines ideas from latent space optimization and reinforcement learning to propose a method for model-based optimization for biological sequences. They also introduce a nice idea for using random samples as negative sampling (data augmentation) which seems to significantly drive performance. They then apply their results to GFP and His3 and demonstrate improved performance when tested against a prediction oracle.
strength_and_weaknesses: Strengths:
- There are a nice set of ideas presented in this work. Combining latent space optimization and reinforcement learning is an interesting set of ideas.
- Negative sampling-based data augmentation is a great idea that is likely a huge driver in improvements of this method as well as can be applied to most other MBO methods
- Sanity checking with AF2 is a nice 

Weaknesses:
- As stated in Evaluation Metrics section, its problematic to evaluate performance with the same oracle that you use to do property prediction. I suggest following Kolli et al (2022)[1] to use an oracle trained on the full dataset for evaluation and using a smaller dataset for train/test split for functionality predictors.
- Are the random mutations truly random? If so, a stronger baseline would be to compete against random mutations ranked by an oracle.
- How were hyperparameters chosen on-line? Such as the action space, f(s_t) threshold, part of representation space that can't be trusted, the number of negative samples, the data distribution of negative samples, etc. Often methods in the Model-based optimization space can't be trusted if they have many hyperparameters without a methodology as to how to tune them off-line.
- It seems that the performance of some of these baseline methods largely departs from other instances such as in Jain et al (2022) perhaps due to implementation details or differences in value K. Can this be commented on (either via an ablation over K or validating or commenting on the irreproducibility of experiments in prior work)? This would have a huge impact on ranking of these MBO algorithms seems rather poor despite using the same benchmark.
- Why was the subset of two proteins chosen? We see in other works large variability in method ranking across methods and so they attempt to combat this by testing across 5+ benchmark tasks (occasionally in Notin et al (2022) [2] it has risen up to 87 DMS datasets for prediction tasks).
- The analysis in Fig 3 would be more convincing if a held-out oracle was used rather than a similar evaluation vs prediction oracle.


[1] Kolli et al (2022). "Data-Driven Optimization for Protein Design: Workflows, Algorithms and Metrics"
[2] Notin et al (2022). "Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval"
clarity,_quality,_novelty_and_reproducibility: The paper contains some novel ideas combining latent space optimization and RL. Unfortunately, the description of the method is rather unclear and the Appendix does not provide much clarity. After multiple readings of the method, I'm not 100% sure I understand the exact sequence of steps in their method architecture due to ambiguous language and contradictory statements. The reproducibility across a diverse set of tasks is a minor weakness in this paper compared to other MBO methods in the literature. This includes the modification of the benchmark in the literature which seems to drastically change the qualitative results compared to those previously seen.

Clarity:
- Include in the related work: Gomez-Bombarelli et al (2018) as using property prediction on the latent space for design was first explored by them.
- "2-dimensional vector representation" usually implies a 2x1 vector rather than a L x E matrix which is perhaps a bit confusing.
- "We recover the amino acid sequence from the reduced representation of size (R,). Using a linear
layer, the size of the representation is expanded to (E,). To recover the L dimension, we concatenate the reduced representation with the wild-type representation of the pre-trained encoder of size (L, E)," Is the reduced representation in this case (R,) here as stated in the first sentence? Otherwise, its not clear how you concat shape (L,E) with (R,), so I assume its the lifted representation (E,)? Then the following linear layer has weights (L+1, L)? This bit is confusing and seemingly the crux of the method.
- "Given an input sequence x, the predictor returns a value k ∈ R, where k is the functionality metric to be maximized and R is the set of real numbers representing possible values for this metric. k is predicted from a representation q with dimensions (L, E), and the functionality predictor f(q)." Figure 1 seems to imply that the input to the property predictor is in X (one-hot encoded) space since it feeds the input of the encoder into the functionality predictor but the text implies that the output of the encoder with representation (L,E) is the input to the functionality predictor.
- " The next state st+1 is sampled from a transition probability function p(st+1|st, at).... This function calculates the elementwise sum of st and at so that st+1 = st + at." So that its not really sampled? This seems like a confusing
- "Both st and at have E dimensions....The latent representation space is set to R = 8, which sets the size of state and action vectors" Are these vectors in E space or R space then?
- For the negative samples, how was the evaluation for Fig 2 done? How was the train/test split chosen since a random one likely doesn't model OOD effects? 

Typos:
- "CbAsoptimize"-> "CbAS optimize" on Page 5
summary_of_the_review: This paper presents some nice ideas(RL, latent space optimization, negative sampling) for an exciting problem settings. They have some interesting evaluations(qualitative analysis of climbing the landscape, sanity check against AF2). However, the primary weaknesses of this paper are twofold: (1) the exposition surrounding the method is ambiguous(ie seemingly inconsistent language) and the experimental section largely deprioritizes the explaining the core details that give someone working in this space confidence that the method is effective (eg hyper-parameter optimization, how train/test splits are performed, inconsistent results from the literature) and (2) the experiments don't follow best practices in the field that allow the reader to interpret the results at face value. Luckily, I think both of these issues can be rectified and are not detrimental to the idea itself.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work proposed an approach for protein sequence design and engineering, using a framework extending from existing language model, which generate a low-dimensional latent space that the authors could leverage for sequence optimization. The use an off-policy RL method to help update designs in the representation space. And the authors demonstrate the method's performance using two protein variant datasets and reported good performance compared to baseline and existing methods. Overall the ideas proposed in this work are very good directions to help inspire better methods in the area of protein engineering and sequence design, as noted by several reviewers. In addition, the presentation of the work and results are overall clear and help to showcase the importance of main ideas. The main weaknesses are that: without a broader and deeper benchmarking and metric justification, it would be not straightforward to evaluate the improvement of the proposed method in a convincing fashion.
justification_for_why_not_higher_score: Overall the reviewers are reasonably consistent in terms of their assessment on the strengths and weaknesses of this work. It is very commendable that the authors have improved their work during the discussion, but major fundamental concerns, particularly as noted by reviewers f1s7, KJ1Y, 62rv, remain. The authors also mentioned that they are expanding the datasets for benchmarking and it would be very helpful for authors to take these steps to enhance the results and robustness of their evaluation/comparison. These improvements will make this work a strong candidate for a future submission.
Decision: Reject