Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: null
review_content: &nbsp;

Just a quick note to double-check that the authors have not yet submitted a response? On occasion, the *Readers* box may be left blank which renders the response invisible to the reviewers.

&nbsp;
review_title: Author Response not yet Visible

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors present FLIP, a benchmarking tool that aims to evaluate models that take as input a sequence of amino acids and aim to predict how well a protein will accomplish a given function (given only the sequence). The authors process data for 3 different functinoal "landscapes" and aggregate them in FLIP: 
* GB1 - asses the effects of high order interactions between mutations
* AAV - Asses the effects of a subset of mutations wrt binding interfaces
* Themorstability - How do mutations affect the stability of a protein across a range of temperatures.

The authors present a benchmark comprised of data for the tasks above and they propose splits to use with the data in addition to training multiple models to serve as baselines. 
strengths: This work aggregates data from multiple sources into an easy to use benchmark, which while not the most glamorous of work is indeed useful, necessary and far from trivial since due to the _massive_ space of possible proteins and the limited amount of data that can be collected, generating sound experiments and splits for ML can be quite difficult. ML for protein engineering is becoming a larger field both at ML methods become more sophisticated and the amount of data available grows, so this work definitely has a relevant niche. The authors do a fairly good job at motivating why the tasks they're using are of importance and presenting the data in a mostly clear way. 
weaknesses: 

The authors do not provide a validation set, which does hurts the accessibility of this work as the authors themselves note determining splits takes some effort to do correctly. The authors note that choosing splits for this work can be difficult, "random splits are notoriously misleading .... as protein sequences are not sampled I.I.D" but they do not provide a validation set for use with each task even they the authors explicitly encourage researchers to do so: "We encourage users to create validation sets from the train sets for hyperparameter optimization". If random sampling from the train set is sufficient to generate a train set due to how the authors generated the splits, this point is moot but it isn't clear to me that this is the case. 

Apart from that, there are some specific technical points that I have some worries about, but on the whole I think this work is relevant and has the capacity to be useful for the broader community.
correctness: To this reviewers knowledge, all the claims made in this paper are correct but there are a few design decisions that I am a bit skeptical about.

1. This may be a bit on the pedantic side, but the authors I don't know that I agree that each split for a given landscape counts as it's own "task". For each landscape, the end goal is the same and the only difference is how the splits are generated. As such, I don't know that I would agree with calling every individual split it's own "task" without some further elaboration as to how exactly the landscape changes w.r.t the splits that were used. 
2. Still on the topic of splits, while the authors did a good job of establishing that random splits may not necessarily be the best idea but:
  a. They randomly sample for GB1 regardless (they do some random sampling in AAV too but with some more constraints)
  b. I don't exactly understand exactly why each proposed split avoids the problems presented by random splits, if the authors could clarify with greater clarity what exactly they are trying to avoid when choosing their splits and how their proposed splitting methods mitigate the those effects that would be helpful.

clarity: In this author's opinion, there are a few points that detract from the overall clarity of paper.

1. The tables with the performance metrics (tables 4-6) do not actually indicate what metric they are measuring. One could reasonably infer that the authors are referring to the spearman correlation coefficient (and this is confirmed by looking at the supplemental tables, where the authors present multiple metrics and hence provide labels), but including this explicitly in the table would help clarity greatly for little added text.
2. While the authors do a good job at motivating why the landscapes included are relevant to protein engineering work, they actually do not really explain what exactly the tasks are. For example, the authors never define "higher-order" interactions for the GB1 landscape, so what exactly is being measured? If the reader has read the previous work or has a background in biochemistry, this wouldn't be too big an issue. However, given the venue, I think the authors should err on the side of overexplainig what exactly the tasks are and what they are trying to measure as this may lead to a better understanding of what exactly the failure mode for each task may look like. 
3. Personally, it took me a little while to really understand what was going on in figure 1. I think I get what the authors were going for, a visual representation of the landscapes that the authors propose to evaluate models on and a visual aid for how the splits were determined, but looking at the figure without reading the paper just caused confused me and looking at the figure after reading the paper, figure 1 didn't really add or clarify anything further than what I had already read. Perhaps splitting this figure up into two different figures (one to showcase what the tasks actually are, perhaps actually showcasing the biochemical task beignet solved instead of an abstract task landscape) and another figure to visually represent what exactly the spits were may aid a reader in comprehension. 
relation to prior work: The authors provide an adequate summary of existing work in the space and explain how their approach is different, in it's aims and goals. 
additional feedback: Overall I'm super positive about the work! If the authors can address all my points in the clarity category that would be sufficient to move to a 6 in my opinion, although higher would require a bit more discussion with my current understanding.

I have a couple of questions for the authors:
1. What types of mutations are sampled in each of the landscapes? For example, are all the single mutants in GB1 modifications charged AA -> Non-polar AAs? What types of mutations are the ones that are present in the data?
2. A single metric is provided for evaluation, but is this sufficient for a benchmark? This may leads to various failure modes being left unobserved (for example, two models get the same score but they fail to generalize in different ways). Are there any plans to flesh out existing tasks with more metrics/visualizations/probes to better evaluate what failure modes a model may be falling into?
3. Is all the data available? I know the authors mentioned that they downsampled the GB1 data(which may in itself be a problem), but is it possible to download all the raw data so that other groups may try different methods of accounting for the class imbalance? It wasn't entirely clear to me if this was possible from what I could currently see. 


## Update
Thank you authors for your response. I'm updating the original review instead of following up due to not being able to get to this until after the discussion period ended. The authors response has been overall satisfactory and I have updated my score to reflect this. More details to come at a later time.
review_title: Promising Work in Benchmarks for Protein Function

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The authors introduce FLIP, a benchmark for protein function prediction motivated by current shortcomings in datasets for protein engineering. FLIP comprises three tasks covering protein thermostability prediction, a mutational screen along an AAV binding interface, and a combinatorial mutational screen on GB1 assessing binding fitness. The authors formulate multiple tasks with the datasets using well-principled splits which are a good contribution to the ML-assisted protein engineering field. The authors train baseline models for the tasks, using a variety of modelling approaches.
strengths: * The need for a meaningful benchmark in this context is timely, well-motivated and has the potential to be highly impactful as it is complementary to emerging benchmarks in computational protein science.

* The work presents a collection of tasks assessing fitness in a range of biological contexts/experimental setups covering the main relevant problem scenarios. 

* The authors provide well-motivated splitting strategies for the datasets.

* The authors make the data curation open and accessible.

Overall I am positive about the direction of the work and its central premise. I am open to changing my score based on dialogue with the authors.
weaknesses: ## Major
The majority of these comments stem from the view that benchmarks should be authoritative and complete if they are to serve as the basis of assessment by the community. From this perspective, I believe there is a little more work to be done to achieve this for FLIP. I reiterate that I am positive about FLIP and would like to see it adopted by the community but I think it falls short of this standard in its current form.

Hyperparameter tuning is not performed for baselines. The authors state they expect absolute and relative performance are likely to change following a hyperparameter search which makes it hard justify accepting the empirical results presented. The authors justify this as being a meaningful comparison because "applied users often accept defaults". This seems like a weak justification as a different set of hyperparameters could alter the findings and thus the direction of follow-on research if users were to simply accept defaults.  The purpose of benchmarks is to understand the relative merits of different modelling approaches on a relevant problem space and at present the authors do not present this. Furthermore, I think as a benchmark the authors should provide principled validation splits as this would be highly desirable by users of the benchmark.

I'm not sure I'm convinced that the inclusion of the meltome dataset is especially congruent with the other tasks in its current form. The other tasks focus on mutations along a single starting sequence and effectively sample large portions of the sequence space within a small number of edits. This allows for rich and meaningful characterisation of the fitness landscape. I appreciate the goal of including the meltome task is to increase the diversity of tasks to include a landscape over a diverse population of sequences but I believe this particular dataset introduces important considerations as a more complex dataset. At the "local" level stability can be affected by cellular/environmental conditions (eg pH), species-specific factors such as glycosylation so I'm not sure this is indicative of a model's ability to learn useful sequence-based representations of similar sequence variants across species/contexts. Stability is also influeced strongly by factors such as size, disordered regions, AA & SS composition and complexing with other proteins which complicates assessment at the "global" sequence level. These sorts of signals might indicate why the pretrained models are so much more successful. I also suspect that solubility may be a confounder of the measurement but accept this would be a limitation of the data collection method rather than the authors' work. Furthermore, measurements are given across a variety of cell lines within and between species (across which measurements can vary) and I believe there are replicates present in the dataset. It's not clear to me how these are addressed and I think more careful splitting and curation strategies might be required such as based on CATH/SCOP classifications or splitting the data into several tasks (eg. species/cell-line specific, human, human-secreted, eukaryotic etc) to account for the different contexts and eliminate non-sequence-based confounders as much as possible. In its current form it's not clear what insight good or bad performance on this task would give me into my model  (or its biological relevance) and its strengths/limitations due to the many externalities. At the very least I strongly believe there should be more discussion of these limitations /nuances as a warning to practitioners & to prevent inadvertent misuse. 

## Minor
The authors present well-motivated splitting strategies that are informed by domain knowledge and show empirically that this is a well-motivated problem to be addressed in a meaningful benchmark. I believe the authors can consider additional splitting strategies. For instance, as I'm sure the authors are aware, not all amino acid substitutions are created equal (eg similar sized/charged substitutions are more likely to have similar impacts on fitness). As an example I would propose clustering substitutions by the properties of the introducted amino acid(s) and creating splits based on these clusters as an additional task. This is a suggestion rather than a strict concern & would be interested in hearing the authors' thoughts on this.

Do the authors expect to add tasks to the benchmark as datasets become available. If so, can they indicate which types of additional datasets will be complementary to the tasks they have constructed?


I believe the addition of datasheets would be useful to summarise the datasets.
correctness: The work seems correct other than the points mentioned above.
clarity: The paper is well-written and accessible. The introduction motivates the creation of the benchmark and contextualises the problem for a lay ML audience.
relation to prior work: Related work is addressed well.
additional feedback: I would like to commend the authors for their interesting work.
review_title: A benchmark addressing a meaningful problem in ML-assisted protein biology

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: &nbsp;

The paper presents Fitness Landscape Inference for Proteins (FLIP), a benchmark for protein function prediction. Catering for relevant real-world protein engineering settings such as low-resource makes the benchmark especially appealing given the scarcity of high-quality experimental data in general in the natural sciences.

I believe the dataset is an interesting and novel contribution to the machine learning community and as such I lean on the side of acceptance. In particular, I think it is important to introduce machine learning practitioners to novel applications such as protein engineering. There is a synergistic interaction between new data formats and ML methodology and such benchmarks are liable to pave the way for advances in core ML, a theme very much in keeping with the directive of the Datasets and Benchmarks track. 

I have a few concerns which, if addressed in the rebuttal, may warrant an increased score.

&nbsp;

strengths: &nbsp;

1. The benchmark is **carefully designed** leveraging domain knowledge of protein engineering e.g. biologically relevant splits and choice of fitness landscapes for inclusion in the benchmark. The attention given to low-N tasks i.e. low-resource tasks is particularly appealing, a situation which arises in many areas of the experimental sciences [1] with few exceptions [2]!

2. The benchmark is **complementary to well-known benchmarks** in the ML community such as CASP which focus on protein structure prediction. In particular, the paper also serves as an accessible guide to ML researchers interested in protein engineering applications.

&nbsp;

weaknesses: &nbsp;

## __MAJOR POINTS__

&nbsp;

1. The model hyper-parameters for the benchmark are not tuned. I appreciate that some models require a lot of compute and the purpose of this study is to demonstrate how the benchmark might be used, however I would suggest **modifying some of the claims made in the paper where applicable**. In particular, the absence of hyper-parameter tuning should preclude the use of appropriate statistical hypothesis tests for validating claims. I highlight some of the statements: 

2. *pretraining offered by ESM-1b [34] or ESM-1v [16] does not help much when sufficient supervised data is available.* Is this statement a defensible conclusion if the models have not been tuned? 

3. Regarding the statement in the caption of Table 5, *This suggests that the large sizes of training sets are past the threshold where pretraining improves performance*. Is this a valid conclusion if the models (including ridge regression presumably) have not been tuned? 

4. Regarding the statement in the conclusion, *However, given that applied users often accept defaults, this is a meaningful comparison.* I contend that this is a valid statement. I think it should be removed and the conclusions from the baseline comparison should be amended.

5. Perhaps some **guidance on how practitioners would use the benchmark** in the scope of a real-world pipeline would be desirable i.e. how do ML models actually benefit laboratory scientists in the context of designing a protein with a desired function? The authors should have an extra two pages for the camera-ready version (the submitted version page limit is actually 9 pages, instead of 8, and there is an extra page made available for the camera-ready version!).
 
6. A **glossary of terms** defining concepts such as epistatic, tree of life, single amino acid mutations, wild-type to a lay ML audience may be desirable.

7. It would be nice to **introduce the task names before they are used in Table 2.**

8. It would be beneficial if the **codebase documentation were improved** e.g. the models.py module contains the home directory of one of the authors. It would be nice if the README.md contained an example of evaluating a custom model on the benchmark task.

&nbsp;

## __MINOR POINTS__

&nbsp;

1. "account" instread of "accounting" in the Related Work section on page 3.

2. The results tables are challenging to interpret because of a lack of boldface in the results figures e.g. in Table 1 - what exactly is the conclusion for the table?

3. Table 4 and 5 captions, it would be nice if the reported metric was given here.

4. It may be an irrelevant point but is there any benefit to representing proteins as graphs in the benchmark? [3].

&nbsp;

## __REFERENCES__

&nbsp;

1. Thawani et al. The photoswitch dataset: A molecular machine learning benchmark for the advancement of synthetic chemistry. arXiv. 2020.

2. Hey et al. Machine learning and big scientific data. Philosophical Transactions of the Royal Society A. 2020.

3. Jamasb et al. Graphein - A Python library for geometric deep learning and network analysis on protein structures. bioRxiv. 2020.

4. Gebru et al. Datasheets for datasets. arXiv. 2018.

&nbsp;
correctness: &nbsp;

To the best of my knowledge the claims made by the authors are correct save for those identified in the comments above.

&nbsp;
clarity: &nbsp;

The paper is well written and in general accessible to a lay machine learning audience. Improvements suggested above.

&nbsp;
relation to prior work: &nbsp;

Adequately addressed.

&nbsp;
additional feedback: &nbsp;

All points covered in the main response.

&nbsp;
review_title: Interesting Benchmark Addressing a Niche in Protein Function Prediction

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: FLIP curates transparent datasets, tasks, and benchmarks for protein fitness prediction. In addition, the authors include a keen analysis of which representation learning algorithms may be the best way forward towards increased performance for each fitness prediction task.
strengths: FLIP takes a great leap forward towards standardizing protein fitness prediction for the machine learning community via its biologically-informed tasks and cross-validation dataset splits. By incorporating important biological insights into the design of their datasets and benchmarks, the authors have developed research infrastructure for accelerated representation learning of protein sequences. Having more clearly defined benchmarks for protein fitness could enable more directed efforts into engineering new proteins to, for example, help address climate change or target existing diseases.
weaknesses: The main weakness this reviewer can identify is in some of the decision statements made by the authors.

For example, in Section 3 (Landscape and Tasks), the authors say "We prioritized complex landscapes rather than single amino acid variants, allowing for a larger potential sequence space as well as functional diversity to ensure model generalization for broad applicability". Are there any references to support this decision/conclusion?

Another example comes from Section 6 (Discussion): "It is important to note that we do not perform hyperparameter search for any baseline method. However, given that applied users often accept defaults, this is a meaningful comparison." To this reviewer, this claim seems a bit overly general. Do the authors have any specific instances/examples to give in which certain applied users (e.g., biologists) may be comfortable accepting default model arguments (as opposed to instances in which they may not)?
correctness: This reviewer cannot identify any improperly designed benchmarks or incorrect claims made by the authors. The datasets themselves also appear to be properly formatted for users' benefit.

In terms of small typing errors in the manuscript, an unnecessary space can be found in Section 7 (Conclusion) in the phrase "With the introduction of FLIP , we...".
clarity: This reviewer believes the manuscript is well written.
relation to prior work: This reviewer believes the authors clearly outline the position of their work in relation to previous contributions.
additional feedback: This reviewer would like to thank the authors for their diligent efforts in standardizing protein fitness prediction for the machine learning community. This reviewer believes such work may have great broader impacts resulting from it, such as accelerated biological and representation learning research for climate change.
review_title: FLIP Standardizes Protein Fitness Prediction for the Machine Learning Community

Meta-review: The reviewers feel that the dataset is useful and complementary to existing benchmarks, but also raised some issues including but not limited to: (1) three reviewers pointed that the hyperparameter tuning was not performed, therefore, some claims may not be convincing and (2) there is no official validation set provided. As a result, the contributions of this work are greatly weakened. The authors' response clarified some important points. In view of that, the authors are strongly invited to take the feedback on board for the final version.
Decision: Accept