Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the different pre-training models for the antibody understanding tasks, propose new methods with biological information for pre-training, and a new antibody understanding benchmark is created. With different study experiments, the authors conclude several observations from different perspectives. 
strength_and_weaknesses: Strength:
1. The authors study the antibody understanding tasks, where antibody is the main and crucial element in drug discovery. The authors propose a new benchmark for the antibody understanding tasks, which contain four specific applications. 
2. The authors propose new biological information involved antibody pre-training methods, which improve the understanding of antibody. 
3. The authors study different pre-training models for antibody and they have several conclusions. 
4. The paper is clear and easy to follow. 

Weaknesses:
1. The authors described about the evolution information about the antibody. In their words, the antibody mutation is targeted at the specific objectives, for example to target on the specific antigen. This is somehow questionable, which is a result driven conclusion. Indeed, protein is also randomly mutated, while the last kept ones have specific structures, functions and so on. The differences between antibody mutation and protein mutation is hard to be convinced. 
2. The authors propose two new biological information (evolution) based pre-training objectives, which are actually straightforward. Though they are reasonable, as far as I see, the results are hard to say that these two are effective enough. In terms of these pre-training, different reasons may cause the performance change. I would like the authors to provide more details about the pre-training. For example, how to evaluate the pre-training performances. Indeed, the current ways are like multi-task pre-training. This is a little not enough. 
3. As for the created benchmark, one question is about the data, the authors mentioned the different specificities of these antibodies. I feel good about this, but the datasets seem not be so good enough. The first two tasks are from the same dataset, also the first affinity prediction is much like the last lack, only specific to covid. Besides, the performances on some tasks are already 0.8-0.9, which seem to be somehow good enough. That's what doubted me about the importance of these tasks. 
clarity,_quality,_novelty_and_reproducibility: See above. Novelty is incremental. 
summary_of_the_review: See above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides a comprehensive analysis of Pre-trained Protein Language Models (PPLM) and specific Pre-trained Antibody Language Models on the predictions of different antibody tasks and introduces a new pre-trained method that better utilizes antibody-specific information to achieve a pre-trained antibody language model.
strength_and_weaknesses: Strength:
  - This paper is really well-written and easy to follow. The authors provide essential biological and technical backgrounds and clearly state the status, problems, methods, and empirical results.
  - The problem it tries to solve is important, and the authors provide great insights into this problem.
  - The provided benchmark could be helpful for future studies.

Weaknesses:
  - Besides the analysis and insights, the contribution may not be significant enough.
  - From the modeling perspective, this paper just introduced two new training targets besides MLM that leads to slightly better performance compared to baselines such as *Ablang-H*.
  - From the benchmark perspective, providing new datasets or incorporating more existing datasets would make this contribution much more significant. 


clarity,_quality,_novelty_and_reproducibility: Clarity: Great

Quality: Good

Novelty: Good

Reproducibility: Easy to reproduce.
summary_of_the_review: Overall, this paper is of high quality. Considering its technical novelty and empirical performance, I would recommend a weak acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a first-of-its-kind suite of benchmarking tasks for antibody-specific language models and provides some interesting observations about the behavior of general protein models and antibody-specific models on these tasks. It also introduces a new antibody-specific pretraining objective based on the unique evolutionary process of antibodies. 

strength_and_weaknesses: =strengths=
Important application
Well written
Provides a first-of-its kind set of benchmarks for antibody ML
Contributes a new interesting antibody-specific LM model

=Weaknesses=
Some of the tasks in the benchmark are based on small datasets, such that reliably computing differences between ML systems may be difficult.
The covid-19 antibody discovery experiments seem to be a bit forced (see below).

clarity,_quality,_novelty_and_reproducibility: I really appreciated how the paper was written. It provides lots of basic background information on antibodies and discusses these complex topics well. I also really appreciated how much of the exposition was structured in terms of whether tasks are antibody-specific or more general to proteins.

In general, I am a big supporter of papers that contribute new benchmarking setups. These can be used to drive methods research for years. This paper appears to be the first setup for antibody-specific benchmarking.

summary_of_the_review: The paper introduces (1) a new set of benchmarking tasks, (2) benchmarks a number of models from recent papers, and (3) introduces a new antibody-specific model. I feel that (1) and (2) should be adequate for acceptance. A paper that introduces a new benchmark shouldn't be required to introduce a novel model that achieves SOTA on this benchmark. However, it appears that (3) performs slightly better than prior work.

The EATLM model is interesting. It adds two new modeling ideas on top of a baseline non-antibody model. It would have been helpful to provide an ablation that shows how much each of these contributes.

Overall, the performance improvement from the proposed EATLM model is positive, but small. It was hard for me to tell if it was actually significant. How did you obtain the error bars in Table 2? I'm concerned that the test sets are small, yet the error bars are small. I recommend obtaining error bars using bootstrap sampling on the test set.  Similarly, in Fig 4 the y axis is small. How do we know that the differences between the lines aren't just due to chance?

Fig 5 seems like a basic sanity check, not a groundbreaking result. Couldn't you achieve something similar, for example, by doing UMAP on basic sequence alignment distances between pairs of sequences in the dataset?

I didn't fully understand the 'Antibody Discovery' section, as this is far outside of my expertise area. As far as I understand, a classifier was trained on a dataset containing functional  and non-functional antibodies against covid. Then, this model was used to screen a list of candidate antibodies. The top-ranked ones were then labeled as true-positives simply if they have high sequence identity to true known positives. Wouldn't any sort of nearest-neighbor classifier be guaranteed to get extremely high performance on this task, by construction? I don't understand why the results are impressive.


Fine tuning language models for downstream tasks is quite challenging, as there are tons of design choices and hyper-parameters. How do you know that what you did provides a fair comparison across models? Is it a standard approach?
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper describes 1) five antibody prediction benchmark tasks, and 2)  two loss functions for pre-training antibody language proteins to incorporate the evolutionary relationship of antibodies during pre-training. 
strength_and_weaknesses: ## Strengths
* I am not aware of an existing benchmark specifically for antibodies
* The described loss functions for incorporating the evolutionary relationship of antibodies during pre-training is interesting and new as far as I know

## Weaknesses
* The paper is not written clearly enough. The lack of technical details, unclear definitions such as "Task specificity", and spelling errors make it hard to understand the paper.
* Performance improvements are overall small
* The benchmark contains only five tasks, train/test splits are not justified, and it is unclear if it will be open-sourced. It also does not allow splitting datasets in alternative ways, e.g. by varying the size of the training set or distance to a wildtype sequence.

1) The definition of "task specificity" is unclear and needs to be assessed quantitatively. As a consequence, the conclusion that the proposed loss functions improve performance most on the "most specific" tasks is vague.

2) Please describe the "Evolution-aware antibody pretraining method" more formally by using equations. Phrases such as "The model is made to distinguish the ancestor germline of the antibody by capturing the shared features" are insufficient for understanding the necessary technical details to reimplement the loss function.

3) Please correct spelling and grammatical errors throughout the paper.

4) Please describe how and which hyper-parameters of the proposed model and baseline models were tuned?

5) Please describe how models were fine-tuned and if they were all fine-tuned in the same way.

6) Please compare the number of parameters of baseline models and EATLM (w/o AGP, w/o MPP, AGP & MPP) in table 1. Performance improvements can be due to different numbers of parameters rather than differences in the loss function.

7) Please justify how datasets were split into train/test/eval splits. Sequences of the train and test set can be very similar if, e.g., datasets are split randomly. What does "training/validation/test split of 15,128/3,242/3,242", for example, mean?

8) The benchmark lacks regression tasks to assess the performance of, e.g., continuous binding affinities (10.48550/arXiv.2210.02881).

9) Please cite Li et al (10.48550/arXiv.2210.02881) in the related work section, who recently proposed an antibody benchmark with two tasks.

10) Please describe whether benchmark datasets and baseline models will be open-sourced

11) Table 2: Please separate metrics of different tasks by vertical lines. It is hard to follow which metrics belong to which tasks.

12) Figure 3: The caption is unclear. Does it show a confusion matrix of model predictions vs. ground-truth labels? The performance of which model is shown? How do per-class performances vary across models? Which class is hardest to predict?

13) Figure 4: Also quantify performances by reporting the AUC and alternative ranking metrics such as Spearman's R or NDCG score.
clarity,_quality,_novelty_and_reproducibility: The paper is not written clearly enough, lacks technical details, and contains many spelling error. It is unclear if the proposed benchmark and methods will be open-sourced.
summary_of_the_review: I suggest rejecting the paper since it is not written clearly enough and lacks technical details, which make it hard to understand the proposed methodology and assess benchmark results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: I found this paper to be quite interesting and thought that it can be a nice contribution to the ICLR community (although this is my opinion as a researcher and AC not too attached to the community focused on protein modeling using LM techniques).  The paper seeks to answer the following research questions, per the authors' text: (1) How do pre-trained language models perform in antibody tasks with different specificity? (2) How many benefits will the model gain if we introduce the specific biological mechanism to the pre-training process? (3) Do the learned antibody pre-trained representations make sense in real-world antibody problems, like drug discovery and immune process
understanding?  In addition, the authors present a new benchmark to facilitate a study on these questions titled AnTibody Understanding Evaluation (ATUE).

Strengths:  The authors study an antibody understanding task using pretrained LMs, they check if specific biological information, when added to the pretraining methods can improve understanding and they also work on a benchmark as mentioned above.  The reviewers agree that the authors put in a lot of work in making this line of work accessible to the readers.  The authors present extensive experiments on several tasks.

Weaknesses:  There is some criticism from the reviewers that the presented methods may not be very novel.  Several other criticisms regarding data quality etc. were brought up that the authors discussed during the review period.  Adding new datasets to the benchmark could also have been more impactful.

While I agree with the weaknesses, I do think that this work can result in a nice connection with the NLP community who are building LMs for general language tasks and the benchmark could be suitable for a new fork in LM research.  Hence, I think it will add value to the ICLR conference.
justification_for_why_not_higher_score: Due to the weaknesses pointed out by the reviewers, I think the paper could have been stronger.  Hence, poster is the right decision.
justification_for_why_not_lower_score: Please see above.  I believe that the paper should be accepted.
Decision: Accept: poster