Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors present the DIPS-PLUS dataset, which builds on the DIPS dataset to enable prediction of protein-protein interfaces. 
strengths: **Significance:** understanding protein-protein interface regions could aid in discovering new drugs and engineered proteins

**Relevance:** this work is relevant to those working at the intersection of ML and biology

**Accessibility:** this work makes the existing DIPS dataset more accessible to ML researchers
weaknesses: I do not know enough about the area to assess the weaknesses.
correctness: I do not know enough about the area to assess the correctness, though the work is presented clearly.
clarity: The paper is clear and well written.
relation to prior work: Yes, the authors discuss both a previous dataset (DB5,DB5-PLUS) along with the original DIPS dataset in detail.
additional feedback: The authors use unprofessional negative language such as "paltry" to refer to the DB5 dataset. This seems unnecessary and should be removed. That language is not needed to emphasize the benefits of the DIPS dataset over the DB5 dataset.
review_title: Clear contribution for facilitating prediction of protein-protein interface based on the DIPS dataset

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: Here, the authors provide a dataset of features, calculated on a per-residue level, for 42,112 protein complexes. This dataset is intended to contribute towards the problem of interface prediction between putative protein interaction partners. While previous datasets of atom-level features exist on a similar scale, their dataset is currently the largest for classic engineered residue-level features, improving the scale of these datasets by two orders of magnitude. 
strengths: As a computational biologist, I can definitely appreciate the utility of this dataset, and the challenges in producing it. Calculating these features for so many complexes involves a great deal of computational resources - the authors efforts in making this a public dataset will make these features accessible to many groups that would not otherwise have the time or hardware. I also think that the scale of this dataset goes very well in hand with current machine learning research for proteins: while there has been some success in unsupervised representation learning research directly from sequence using large-scale protein databases like Uniprot or Pfam, to my knowledge, large-scale models have not been trained on residue-level features, which I suspect is due to the lack of accessibility of large enough datasets of this nature. Hence, this dataset could potentially spur new methods.
weaknesses: My main issue with the paper is that it is currently not written in a way that a general machine learning audience will appreciate the dataset. The summary of the problem is okay, but the authors do not review prior literature sufficiently to establish an argument for why this dataset is useful, and how they anticipate it will be used. For a non-biologist, it is not clear what the difference between atom-level or residue-level features is. Even for a more specialist computational biologist like myself, I'm not entirely clear on what kind of difference the authors expect providing residue-level features, as opposed to atom-level features or just direct sequence, will make for current machine learning methods - is this simply an argument from principle, or is there any evidence that these features truly provide enough additional features for models to "generalize" well? This seems to be a really major premise of the paper - I would like the authors to better review the actual methods that have been previously built on DB5-Plus to convince the reader that residue-level features are a big deal. 

Additionally, the authors have not done enough to contextualize their work in respect to current feature learning approaches on proteins. The authors touch upon how deep learning has established itself as an effective means of automatically learning features from data - but they seem to have completely missed the major representation learning advances using deep learning to learn a mapping from raw sequence to feature representations. Many of these previous works have shown that these learned features correlate with many of the more classic bioinformatics features that the authors have described here like secondary structure. I'd like a better discussion of the trade-offs between classic features that the authors provide here versus the learned features: is there any reason to believe that these classic features are more conductive to performance than recent unsupervised protein features, or are the advantages more on other axes like interpretability? I know in many bioinformatics disciplines, e.g. especially bioimaging, even unsupervised or self-supervised deep learning representations have blown away expert-designed knowledge-based features on classification and regression tasks.

Finally, there are weaknesses from a structural biology perspective (see feedback). The authors spend a lot of time delving deep into the features they use, which are rather routine, so the paper reads like a manual. My impression is that the key challenge in this dataset is not the feature production, but the quality of the PDB structures given the sheer volume of data, not all of it reliable. From what I can tell, there has not been quality checks on the data on the level of basic aspects like clashes or filters for resolution (or at least the authors do not explain this), so I am uncertain about the overall quality of the data as a whole.
correctness: Yes - the features calculated are standard bioinformatics features, and I see no major issues with their definition. 
clarity: The language itself is fine, but the paper is not written with sufficient context, motivation, or background for a general machine learning audience to understand the contributions of the paper. 
relation to prior work: No, this is not sufficient: see weaknesses.
additional feedback: 1. The authors go into depth on explaining their features, but they spend no time addressing the known errors, biases, and artifacts in the structures themselves, which will inevitably influence the quality of the computed features. Did the authors perform any quality checks on these structures?  i.e. What is the resolution limit of the structure of a complex for entry in their dataset? What side-chain resolution do they tolerate? Are positions of side-chains in the structures checked for clashes, and if those are encountered, fixed by e.g. a molecular dynamics protocol? There is a lot more I could state, but including even just these basic things would help. 

2. How exactly do their features address the physics of the interactions between main-chain and side-chain groups? Physicochemical features of different amino-acids should be most informative for learning -- those I guess these are implicitly embedded in features such as protrusion index and residue depth - but more discussion on this would nonetheless be useful.
review_title: Review of DIPS-Plus: The Enhanced Database of Interacting Protein Structures for Interface Prediction

final_score: rating: 3: Clear rejection
review_content: summary and contributions: The authors add some residue-level features to an existing DIPS dataset and analyze potential impact and challenges.
strengths: The task is meaningful in biomolecular modeling.

The methods they mentioned are machine learning methods, which are relevant to the NeurIPS community.
weaknesses: The dataset is not novel, it’s just DIPS data with some additional features. And the features are some common features that can be calculated from raw atom coordinates with some open-source tools.

There is no discussion about data quality, such as the resolution of structures from PDB.

There is no evidence that the additional features are vital.

No benchmarking of existing methods, either machine learning or domain methods.

correctness: The statement of DB5 is not correct, please see the relation to the previous work section for more detail.

The author stated the best results for interface prediction without discussing any traditional domain methods, such as global docking. 

clarity: The authors explained all the features they added, but didn’t explain why they chose those and if those features help.
relation to prior work: The authors didn’t search the field thoroughly. I understand DB5 was widely used in previous machine learning research in this field, but more datasets exist indeed. PDBBind (https://doi.org/10.1093/bioinformatics/btu626) is a PDB-wide curation of protein binding data (same approach as DIPS, just earlier) released in 2004, including 5,341 structures of protein-protein complexes up to 2015. In the interface prediction task of MaSIF (https://www.nature.com/articles/s41592-019-0666-6), they used non-duplicated data from PRISM, PDBBind, SAbDab, and ZDock, comprising 12k data in total. Thus the statement of “DIPS is 200 times larger than previous datasets” is overstated, even DIPS is not the work of authors.
additional feedback: The term can be protein-protein binding site prediction or such.
review_title: The author just added some features to an existing dataset, and those features can be obtained from the original data

Meta-review: The authors present a dataset of features for 42,112 protein complexes that augment the DIPS dataset. The reviewers noted importance of the task and the possible applications for this dataset, but raise significant concerns about the novelty and the clarity of this work. While they have noted the improved new version submitted by authors, the reviewers still recommend against accepting the paper after the authors' response.
Decision: Reject