Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper's aim is sequence-based protein engineering based on protein language models (PLMs). For this purpose, it uses a recently developed self-supervised in-filling language model. The model rearranges the middle (to be infilled) part of a sequence and move it to the end of the sequence which enables the use of standard forward prediction. The paper then tests these PLM models for identifying other possible residue sequences in the middle of a protein sequence that preserves the general 3D structure of the protein (secondary structural type).  
strength_and_weaknesses: - the paper provides a novel method for PLM especially suitable for sequence-based protein optimization.
- despite being a language model, the proposed architecture is relatively small in number of parameters which enables efficient use at test time and enables research for protein-engineering with limited computational resources.
- the additional experiments regarding the quality and the general analysis of the learnt representations are interesting, encouraging and informative.

*Weaknesses*:
- For an ML conference there seems to be no technical novelty, neither fundamental, nor incremental. The model is almost an exact copy of the standard LM with the addition of the structure constraint.
- The number of available protein (and non protein) language models are vast. The paper only compares with one method while it could (and should have) compared with other PLMs but also possibly other language models with the same trick (moving the missing part to the end).
- the main results that is the goal of the work for protein engineering seem quite comparable with the only baseline that is used (ProGen).
clarity,_quality,_novelty_and_reproducibility: The paper can be improved in clarity in general. While the idea of the paper is straightforward and thus easy to understand. The organization of the content, the notation, and figures can be revised to improve the flow. For instance, the introduction section can become sharper and more concise while the method section can benefit from more formal and structured content of the general problem, possibly the existing variants and the proposed approach.


summary_of_the_review: The paper uses a novel method for protein language models that is quite efficient in number of parameters and shows the results are comparable for protein sequence engineering compared to one other recent PLM baseline. However, considering that the results are clearly positive compared to the possible baselines, the lack of technical novelty (even in form of an increment on existing methods), the lack of a thorough set of baselines, and presentation of the work, I believe the paper is not ready for publication, at least at a ML venue. More developments on the method and/or a more thorough empirical evidence will increase the quality of the paper in another revision.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a new protein language model ProtFIM, and a new protein sequence design framework via ProtFIM. 
By comparing the performance with previous models via their new evaluation scheme ( SEIFER), ProtFIM achieved similar performance but with less parameters.
strength_and_weaknesses: Strengths: 
1.	The article is well written and easy to understand.

Weaknesses: 

 1. There are too few metrics for experimental evaluation. Only P@k and R@k are included in evaluating infilling.  Some metrics provided in Progen2 can be used (e.g. complexity).

 2. The method is not throughly evaluated.  The number of comparison methods is too small. There are still a lot of related work on generative PLMs.   The performance of the model is similar to Progen-large, and the main advantage is that the model size is smaller. Is it possible to add other models with different scales provided in Progen, especially small-scale models?  Then we can see whether ProtFIM still has an advantage over the models sharing similar scales.

3. The evaluation requires the consistence of secondary structure between the generated protein and the original protein. However, a new protein with better performance may have a different secondary structure. Whether asking the generated protein is completely similar to the original one in secondary structure will affect the innovation of the model?

4. It’s better to explain what (a) and (b) are in the caption of Figures 3 and 4. In addition, Figure 4 (a), (b) are both “Precision”. 

clarity,_quality,_novelty_and_reproducibility:  The article is easy to understand. The novelty is marginal.  ProtFIM achieves similar performance to Progen-large, but ProFIM has less parameters.  
summary_of_the_review: The idea of this paper is easy to understand, but not novel enough.  The performance of ProtFIM is similar to Progen-large, but ProtFIM has less parameters. More competing methods and some other metrics should be included.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a protein language model (pLM) that is trained on a fill-in-middle task that has seen application in the NLP domain. This is motivated by the fact that existing pLMs that are used for protein generation are autoregressive, and as such cannot condition on bidirectional context, which would actually be the most prominent case when re-engineering a region of a protein, as many residues relevant to function are found in the middle of the sequence rather than at its ends. The performance of the infilling model is evaluated by predicting the secondary structure of the filled in region and comparing it to the ground truth.
strength_and_weaknesses: Strengths
- The FIM paradigm was not applied to pLMs so far.

Weaknesses
- No clear performance improvement over the ProGen baseline. 
- The comparison to ProGen-large does not prove that FIM is more efficient per parameter. You should compare to an autogressive pLM of equal size without the FIM task to show that.
- If I understand correcty, also XLNet-style models should be capable of generating sequences with bidirectional context. Why is the ProtTrans XLNet model not considered?
- ESM-1b is not SOTA for zero-shot fitness prediction. Also, this evaluation again fails to prove that the parameter efficiency can be attributed to the FIM task.

Questions
- The definition of the @k metrics is hard to follow. Could it be expressed in a more straightforward way that directly explains what TP, FP, TN, FN are in the context?
- What is the reason for choosing a constructed secondary structure metric over perplexity? Perplexity can directly measure how well the model captures the masked span.
- Does the alpha helix bias mean that the FIM residue SS does not really depend much on the surrounding context?

Additional comments
In the introductory session, describing directed evolution as "random guessing or brute-force search" doesn't capture its essence, as it is actually meant to avoid doing just that.


clarity,_quality,_novelty_and_reproducibility: Clarity
The presentation is clear, but the choice of baselines prevents a clear understanding of the benefits of the presented approach.
Quality 
The paper evaluates on appropriate datasets. 
Novelty
Applying FIM for pLMs is novel. 
Reproducibility
Evaluation was done on established benchmark sets and the work should be relatively easy to reproduce.

summary_of_the_review: The paper introduces a FIM-trained pLM and argues in favor of it's performance based on the fact that it has a far lower number of parameters than the ProGen model used for comparison. However, this leaves it unclear whether the model is truly more parameter efficient because of the FIM task - an AR pLM of equal size would be required to show that. Also, additional baseline methods should be considered to convincingly show that FIM boosts performance for the investigated tasks.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduced a new type of protein language model capable of filling in the middle of a sequence (which wasn't easily doable with autoregressive or with masked language models). 
strength_and_weaknesses: Strengths:
+ The "fill in the middle" use case is very common in protein engineering, and such a model could indeed be very useful.

Weaknesses:
+ The training setup choices seem somewhat limited: For example, it only allows one masked region in the middle. Often, in protein engineering, there might be multiple segments for joint design. There is a much more general version of this (e.g. see CM3 https://arxiv.org/pdf/2201.07520.pdf).
+ The empirical results on fitness prediction seem worse than existing methods.
+ I'm unconvinced that secondary structure conservation is the best evaluation scheme. What about sequence recovery or perplexity on the "middle" sequence?
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is generally clear, although the writing could be improved at places.

Quality: The empirical results are not the most convincing. Secondary structure conservation seems like a relatively easy task. What about sequence recovery or perplexity on the "middle" sequence? Fitness prediction results are also not showing improvements.

Originality: New training objective is the main new contribution, although I'm not sure if this is the best way to frame the training objective. 


summary_of_the_review: Important direction for supporting protein infilling with language models, but limited technical or empirical contributions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper describes a protein language model to support sequence-based protein engineering. It uses a self-supervised in-filling language model that rearranges the middle (to be infilled) part of a sequence and moves it to the end of the sequence, enabling standard forward prediction. The paper then tests these models to identify other possible residue sequences in the middle of a protein sequence that preserves the general 3D structure of the protein (secondary structural type).

**Strengths:**

* The article is well-written and easy to understand.
* Experiments regarding the quality and the general analysis of the learned representations are interesting and informative.

**Weaknesses:**

* Reviewers agree there seems to be very little technical novelty and that the current approach is neither fundamental nor incremental. The model is almost an exact copy of the standard LM with the addition of the structure constraint.
* The method is not thoroughly evaluated. The number of comparison methods is too small. Few performance metrics are considered in experimental evaluation. 
* There is also consensus that the approach does not provide any clear performance improvement over the ProGen baseline. The empirical results on fitness prediction seem worse than existing methods.
justification_for_why_not_higher_score: The number of available protein (and nonprotein) language models is vast. The paper only compares the new method to one existing approach while it could (and should have) compared with other protein language models and possibly other language models with the same trick (moving the missing part to the end).
Decision: Reject