Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this work, the authors reappropriate the term “fine tuning” and call it “reprogramming”. The main aim of the work is to propose new CDR loops as an infilling approach Some experimental results suggest that the proposed approach outperforms the state of the art.

strength_and_weaknesses: To me, the main weakness of this work is the lack of reference to the “fine tuning” literature.
clarity,_quality,_novelty_and_reproducibility: Novelty

For me, it’s difficult to gauge the novelty since the authors completely missed the “find tuning” literature. From my perspective so far, there is no novelty.

Quality

The technical work lacks context, as already mentioned.

Clarity

Aside from the lack of context, the work is clear.

Reproducibility

I didn't seen any code or data, so I guess this isn't very reproducible.


summary_of_the_review: This paper misses a lot of relevant context, so it’s difficult to put in context. I would recommend the authors find relevant “fine tuning” papers and start from there.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a framework for the antibody sequence design task. The main idea is to use the pre-trained model for sequence prediction. The difference is that they use a pre-trained English BERT model with some modifications. The results show improvements of the sequence prediction. 

Overall speaking, this is an easy work. 

strength_and_weaknesses: Strength:
1. The writing is clear and the authors give a clear motivation for the work.
2. The results show improvements in the work. 

Weakness:
1. Novelty is really limited. The authors do not give enough modifications to the model, while it is interesting to use the English BERT model, but the method seems lack enough motivation when comparing the MR method with protein pre-trained model.
2. Similar to the above one, the results are interesting that the MR is better than protein pre-trained model, but the reason is not explained and the study is not enough. This strong lacks enough content for this paper. 
3. More studies and results are expected for the later version. 
4. Code is not released, though the results are promising.
clarity,_quality,_novelty_and_reproducibility: novelty is not enough.
summary_of_the_review: See above.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduced "Reprogramming for Protein Sequence Infilling", for the antibody CDR design problem. Starting from a more general protein language model, the model is "reprogrammed" for the infilling task (infilling for antibody CDR loops). The method is then empirically evaluated on antibodies in the SAbDab database.
strength_and_weaknesses: Strengths:
+ Reprogramming is an interesting idea and hasn't been previously applied to protein design.

Weaknesses:
+ Limited empirical improvements compared to ProtBert.
+ Limited in scope. The infilling problem for protein sequences could be of more general interest on other classes of proteins, although this paper is focused on the specific subarea of antibody CDR design.
+ Can the model capture interactions between the CDR loops?
clarity,_quality,_novelty_and_reproducibility: Quality: Limited empirical improvements from reprogramming, when compared to ProtBert without reprogramming.

Clarity: Overall easy to read, with clear tables and figures. It would help to be more clear on the motivation for the problem setup. In particular, typically CDR1 and CDR2 are largely dependent on the V gene (also reflected in framework regions). What's the motivation for only designing the CDRs when the V gene is already known?

Originality: Limited technical novelty.
summary_of_the_review: Limited technical novelty and application scope. This approach is interesting and probably could be useful in practical application, but I'm not sure how useful it is to the broader community.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The work proposes to adapt pretrained language models for CDR loop infilling. Essentially, this is a task that requires the model to predict the identity of a span of masked out tokens in a sequence of amino acid tokens. The work proposes to do the adaptation using a model reprogramming approach, where two linear layers are used to map from the (task_input, pretrained_output) domains to the (pretrained_input, task_output) domains, thereby learning a lightweight network that casts the task at hand into something that can be solved by the pretrained model.
This is a novel idea to practitioners in the protein deep learning field, where typically no benefit is expected from using models pretrained on human language. The paper claims that the method achieves good infilling (=reconstruction) performance while also having high sample diversity.

strength_and_weaknesses: +The idea is novel
- The performance compared to the baselines is not convincing. The comparably strong performance of EnglishBert, which should be expected to have no knowledge of the problem whatsoever, is especially worrying.
- Not clear how useful sequence diversity is as a quality metric of the model
- It is not clear if the baseline models are fair, as they have fewer learnable parameters.
- Discussion not addressing the implications of the results.

Questions
How do the authors explain the strong performance of EnglishBERT, which has no knowledge about the antibody inpainting problem?
What is the meaning of ProGen perplexity for CDR loops? Do native CDRs have lower perplexity?
Why was EnglishBERT chosen for reprogramming, when ProtBERT is pretrained on a domain much closer to the task?
Why are BERT models pretrained on antibody sequences not considered?

Comments
Figure 1 is confusing. Showing the 3D structure and the light chain sequence on the input side of the figure is distracting, and only upon reading it becomes clear that those are not part of the model input.
clarity,_quality,_novelty_and_reproducibility: It is a novel and original approach that has seen no prior application in biological sequence design. The paper lacks clarity as to why exactly model reprogramming should be the right choice for this biological problem.
summary_of_the_review: I am not convinced that this method works well. Essentially, the paper claims that the task of antibody sequence infilling can be solved by reprogramming (=adding two linear layers) a BERT model that is pretrained on english language and that this works better than a) a BERT model pretrained on proteins and b) antibody infilling models that additionally use structural information. This would be a very unexpected result that shows that the task of antibody sequence infilling is actually surprisingly trivial and can be solved with minimal knowledge of the problem. However, the paper fails to discuss this adequately and just presents the results as they are without further discussion of these implications.

The authors base their claim on a) good sequence recovery performance and b) highly diverse samples. While recovery is easily interpretable as it is the task that is optimized for, diversity as such has no inherent meaning, as long as it cannot be validated that the diverse samples are indeed biologically plausible. ProGen perplexity as it is presented is not conclusive of the biological plausibility, as it is not clear what perplexity behaviour can be expected for CDR loops which are highly variable and comparably unstructured.

EnglishBert should be understood as an almost-random baseline, as it has no knowledge about antibody structure aside from what can be incorporated from learning new amino acid embeddings in the presented setup. Yet, on CDR3, according to the presented metrics, it performs better than Refine-GNN. Why is that?

In the reprogramming framework, two linear layers with a large number of parameters are introduced to map from the task domain to the pretrained domain. For the baseline methods, this is not the case. How is ProtBERT considered to be a fair baseline method for the task, if it was never optimized for it and has no learnable parameters?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work studies the application of pretrained language models to antibody sequence prediction. Finetuning / domain adaptation of pre-trained language models is a valuable technique and it is interesting to apply this to antibody sequence prediction. The reviewers did not find this work sufficiently novel, or the results convincingly strong to recommend publication. The work could be strengthened by acknowledging existing finetuning literature, bolstering performance comparisons.
justification_for_why_not_higher_score: Quoting: The reviewers did not find this work sufficiently novel, or the results convincingly strong to recommend publication. The work could be strengthened by acknowledging existing finetuning literature, bolstering performance comparisons.
Decision: Reject