Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: null
review_content: In this manuscript, the authors present PEER, a collection of protein ML-related tasks with pre-defined splits, covering five critical categories to test ML models’ understanding of proteins. The authors also benchmarked multiple protein language models (LMs) and reported performances on the PEER tasks. 

Overall, I recommend PEER add more novelty to benchmarking design. 

The current version is an amalgamation of a few popular protein ML benchmarks, including TAPE, FLIP, and PDBBind. I recommend the authors add more original contributions to improving the practical impact of those tasks. 

A few more specific comments:

1. Related work “Protein modeling benchmarks” is missing some important work:

- Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development

- Add TDA https://arxiv.org/abs/2102.09548: contains several related protein tasks such as PPI and protein-ligand interactions.

- Atom3d

2. Comments regarding curated tasks:

- Metrics such as accuracy and RMSE for PPI and PLI are inappropriate. In practical use cases such as protein design and virtual screening, ranking metrics such as precision@k are more appropriate.

- It is not stated in the manuscript how the negative pairs are produced in some of the PPI and PLI datasets. This is important since our current knowledge about positive PPI/PLI pairs are partial observation of the collection of ground truths. 

Resources like http://mips.helmholtz-muenchen.de/proj/ppi/negatome/ would be helpful.

- Formuating “Fold classification” task as a 1,195-class categorical classification is probably not appropriate as the SCOP classification of protein fold has hierarchical structure.

- Homology-related tasks are missing from the collection. This is important to validate the promise that protein LMs can learn evolutionary information.
 
review_title: need a bit more novelty in the  benchmarking design

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This paper compiles a list of 14 tasks across various protein datasets to serve as a potential benchmark in training sequence-based protein models. These tasks focus on a range of biological properties of interest. The authors aggregate these tasks into five branches: function, localization, structure, protein-protein interaction (PPI), and protein-ligand interaction (PLI). 

The authors explore this collection of datasets by testing a variety of different model architectures on single-task and a novel multi-task framework. Their multi-task results include structure-based auxiliary tasks, under the assumption that "structure influences function" (which is a biologically appropriate assumption). 

Finally, the authors provide a work-in-progress repo that should provide both pre-processing to the datasets, and a training script that allows users to verify their own model's performance against these datapoints.
strengths: The datasets they have chosen in their benchmark cover tasks on different biophysical length-scales and potentially time-scales. A standardized resource to access these multiple dataset types could be convenient for community usage. 

The authors also try a panel of different architectures on all tasks, and in tandem. They include feature-engineering models based on biological properties; this is a good inclusion as there are many research groups that use expert-crafted features.

It is interesting to see the MTL results across all 14 datasets, although this was was not done with all 14 tasks simultaneously.
weaknesses: (1) Several of the function datasets are not generalizable. For example, fluorescence of GFP proteins is not a property relevant to many other protein types. Good performance on this dataset (which can be attained by even lightweight models) may not indicate any knowledge of a broad sequence-to-function relationship. Such criticisms have been brought up before in dedicated fitness benchmarks such as FLIP (Dallago et al.). It is surprising the datasets proposed by FLIP are not included in these tasks.

(2) In general, the metrics for many tasks may not actually reflect a model that learns well. For example, there are notable class imbalances in some of these datasets that can create models that demonstrate strong predictive performance without understanding the sequence-to-output relationship. For example, in the subcellular localization prediction task (DeepLoc), not all classes are equally represented in the data; hence it may be possible to attain a model that is highly accurate on the dominant classes, but unable to distinguish the others. The metric for this task is "accuracy" - something that can be trivially high despite the model truly only "learning" a subset of the classes (if at all). The authors should consider redefining balanced subsets for train/val/test. Alternatively, reporting AUPRC/sensitivity or other metrics that explicitly consider imbalance can alleviate concerns on "high" performance inappropriately reflected.

(3) PLI-type tasks may be just as sensitive to the ligand representation as the sequence representation. How would you account for the protein-sequence contribution in this type of task?

(4) The authors do not clearly state the rationale for choosing these datasets of the many protein datasets that exist. Many of these datasets already exist as benchmarks on their own, but why does this unique set of 14 tasks confer the appropriate way to explore the representational strength of protein modeling?

(5) The MTL results are overstated; improvements, when present, are quite marginal over the baseline. Moreover, the authors define the auxiliary task with a tradeoff parameter generally set to 1. It is not clear whether this is an appropriate choice and the authors have not provided the loss over training for each task. It is possible the auxiliary task loss is dwarfed by the central loss. Supp Figure 1C shows a dramatic influence of the tradeoff parameter on beta-lac performance. Would the authors be willing to show losses across each task and the corresponding aux task in the supplement?

(6) Size effects influence model predictivity. Supp Fig 1A indicates on different architectures that longer truncation lengths enable improved predictive performance. How do the authors suggest comparing performance given variable length inputs? 
correctness: N/A
clarity: There is a minor typo on 5.3 under the bullet point "MTL less benefits..."; this might be "least benefits".

Impact of each of these datasets should be stated clearly. The authors explain what the datasets are, and what the underlying task is, but do not illustrate their rationale for why they included this task over other comparable datasets. Other such dataset or benchmarking papers often include an "impact" or "motivation" as to what attributes within the data make it attractive to include in a benchmark. 
relation to prior work: As mentioned in weaknesses, there is no novelty introduced in the processing of the tasks. The benchmarks also do not include the SOTA models performances that currently exist for some of the provided tasks. This is an important point of comparison.
additional feedback: 1) When performing computations for PPI affinity or PLI, how do the authors account for charge states and other nuances sensitive to the environment that the protein is in? Protonation states and subtle changes to the molecular representation can have vastly different implications for whether there will be an interaction. Notes on how this is handled is important in all the benchmark tasks, as it will standardize how users would represent the input data (unless you are offering pre-processing in your code base?).

2) How did the authors populate feature vectors for the GIN? I could not find explicit details on how the ligands were input to the GIN for a representation.

3) It is curious that the DDE-based predictor did so well on certain tasks (namely the function tasks in Table 2). Would the authors be willing to address their hypothesis as to why? This representation seems to do reasonably well on almost all tasks except localization.

4) Table 3 indicates an unusual trend; the inclusion of a structure auxiliary task seems to diminish performance on a structure central task in more cases than it helps. Do the authors have an intuition why this is the case? This is actually a very interesting result, as one should expect these tasks to be beneficial to one another. 

5) The MTL analysis does open a very interesting line of inquiry around what specific classes/improvements the auxiliary task confers in prediction. It would be nice to see (perhaps in follow up work) what classes do better/see reduced uncertainty in inference given the inclusion of the structural information. For some models, attention analysis etc. may open a new line of inquiry.
review_title: Insufficient novelty in benchmark design; counterintuitive MTL results may open interesting research questions with more analyses

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: PEER proposes a Benchmark for protein models on a variety of tasks, including protein function, localisation, structure, PPI, and ligand affinity. It evaluates a variety of methods, including feature engineering, deep learning (CNN, LSTM, Transformer, ResNet), pre-trained protein models (frozen VS unfrozen).

It is a much-needed paper in the community and will help advance the field of protein understanding. It brings a novel contributions on many aspects. However, the paper is a bit cluttered and it is difficult to get a global picture of what's going on and what are the different tasks. What does the paper need? Better Tables! And more fluidity in the text to know where it is going, not a simple enumeration. 

1. Sections 4.1.1, 4.1.2 and 4.1.3 can be summarized in a single table with only the most important info to compare the architectures.

2. Table 1 is missing many information. Perhaps flip the page into landscape mode and do a full-page table? 
    
    a. The acronym to correspond the entries of Table 1 with Table 2.

    b. A short description of the task

    c. The number of unique proteins

    d. Average and std of the protein sequence length

    e. The train/val/test split method

    f. Other info specific to the dataset: Number of unique ligands for for BindingDB. Mean and std for the regression tasks. The data imbalance for the classification tasks. 

3. The color in Table 2 is not informative and makes it very hard to read. Some values can be very close to each other, yet the strong color difference suggest that they are far. Instead, I would highly suggest simply doing a color-scale or "conditional formatting" for each row. The best in dark, the worst in white. That way, one can easily find which column is usually the best method. Relative comparison within a group will still be possible. Also, I suggest using a color such as green instead of gray to make it easier to read the table.

4. Table 3 could add a row at the bottom of average relative improvement, and another column at the right of each group for the same relative improvement. For example, +10% indicates that, in average, metrics improve by a relative 10% compared to the original. Again, this is more informative than a single blue/red color, and allow to see which task benefits the most, and which model benefits the most.

5. Analysis of the correlation between performance VS size of the training set VS length of proteins. This should be done for at least one model per category (feature engineer, deep learning, pre-trained models), and will help to see if they play a role in the model's performance.

In general, a good paper and good benchmark! Willing to upgrade my score if concerns are answered since they are mostly about presentation of the results.
strengths: Great diversity in benchmark, in models, in training setup, etc. Benchmark is much needed in the field.

Haven't looked at the code in details, but looks good at first glance.
weaknesses: The presentation is too cluttered. Difficult to gather information quickly from the text and the tables. Things can be structured and summarized better using the right structures.

Some information about the datasets is missing such as a number of unique proteins, the split used, the protein lengths, etc.

Some more studies could be done to understand the importance of language models.
correctness: Looks good to me.
clarity: Could be better. Too cluttered and difficult to find the information needed.

That being said, writing is okay.
relation to prior work: Looks good to me.
additional feedback: See above comments.
review_title: A much needed benchmark for fast-tracking protein understanding

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This paper is well written and understandable. The authors present a comprehensive benchmark for several protein sequence based prediction tasks, such as fitness prediction, ppi prediction, structural predictions. As for baselines, they choose some typical NLP models, such as LSTM  CNN and Transformer. In general this paper has limited novelty and contributions：  

First, most of thest tasks the paper present have already had well-established benchmarks. For example, for these fitness prediction tasks, they have occured in TAPE （Raoshan2019） and FLIP （Yang 2021）bechmarks. I did not see the necessarities introducing this benchmark again. Another example is structure prediction, there are also various beanchmarks, such as TAPE again. More importantly. the proposed structure prediction tasks are not important to the community now given that AlphaFold2 can predict extremely high accuracy for 3D structure, why we still need secondary and contact predict tasks?   Second, for these PPI prediction task, I would say for most biologists they would not care the accuracy of such binary prediction tasks.  They care more about which part or which postions there are interactions, particularly from 3D structure perspective, not a binary value yes or no. This paper is very AI-style without deeply considering  the key challenges in biology field.

In addition, they present only four types of baselines, LSTM transformer, cnn and resnet. What about some typical compuational baselines  such as Gremlin and more advanced baselines  such MSA transformer and Evoformer (the language model of AlphaFold).

In summary, I think this paper is not good enough for NeuralPS. I suggest rejection.

strengths: (1)A benchmark covers many tasks
(2)relevance to the compuational biology community
weaknesses: (1) most tasks, datasets and baselines were already existed in literature 
(2) contact prediction and secondary structure prediction tasks are no longer interesting given the remarkable results in 3D structure prediction by alphaFold2
(3) tasks such as PPI  prediction is not interesting from biology perspective.
(4)A lack of more advanced baselines such as MSA transformer and Evoformer.
correctness: This part is fine
clarity: yes
relation to prior work: They have presented most relevant literature but such papers are too much in literature.
additional feedback: no
review_title: A benchmark for protein sequence understandings with limited research contributions

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This work proposes PEER, a benchmark for protein sequence representation learning, including curated datasets and implemented baseline methods. Comprehensive experiments are conducted to evaluate the performance of multiple protein sequence models and multi-task learning.
strengths: (+) This work proposes the PEER benchmark datasets, covering more comprehensive protein sequence representation tasks and data than previous benchmark datasets. The creation of new and large benchmark will provide chances for researchers to more comprehensively and systematically evaluate their proposed new methods.  
(+) The implementations of multiple popular protein sequence models are provided in the proposed benchmark, making it more convenient to reproduce or compare with the results of baseline methods.  
(+) This work proposes a novel and insightful idea of improving protein sequence models by multi-task learning, which is very useful and enlightening for researchers in the protein representation learning area.
weaknesses: (-) Currently, the PEER benchmark focuses on protein sequence representation learning, but some recent methods have been proposed to do protein structure representation learning. I hope the PEER benchmark can offer 3D structures of proteins in the dataset if possible, and even include some structure representation baseline methods in the future.
correctness: Both the dataset collection and baseline evaluation are sound in this work. The experiments are appropriately designed and all claims are supported by solid experiment results.
clarity: The writing of this paper is good and well-organized.
relation to prior work: This paper clearly discussed the difference between the proposed benchmark and previous benchmarks.
additional feedback: N/A
review_title: A very excellent and useful benchmark providing much convenience for researchers interested in protein sequence representation learning

Meta-review: metareview: This work presents a diverse benchmark for multiple proteomic tasks, including comprehensive results for both single-task and multi-task learning. The paper is well written, and the benchmark is essential to the field. The mean reviewer's score is near acceptance. Perhaps most importantly, the reviewer's comments have been addressed during the rebuttal, and the code has been made publicly available. Therefore, I decide to accept this paper.

An additional helpful baseline would include a generic implementation of a system of neural networks that performs multiple tasks: consisting of (i) encoder networks for each input; (ii) a bottle-neck fusion network with fully connected layers that learns how to merge the encoded representations; and (iii) decoder networks for each output.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept