Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides a python library on modeling protein-protein interaction for the field of target discovery and drug design.
It includes commonly used database on both structural data and interaction data. Also, for neural network and modeling, it provides interfaces on different scale level for atom level to protein/DNA/RNA level. The code is open-access, which could be helpful for the drug community.
strengths_and_weaknesses: - Strengths: this paper gives a brief overview of widely used databases and modeling interface for protein interface and mesh. The python code is open access. This tools could be beneficial for communities from both target discovery and drug design.

- Weaknesses: it could be better if this toolbox could provide common methods and make benchmarks on these standard databases.
questions: - It is well-known that some databases could be extremely large. Could you provides the performance running on large-scale databases.

- do you plan to provide AI baselines and benchmark different methods on the open databases?
limitations: - my major concern is the scale and efficiency for this toolbox, especially for large-scale data.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents "Graphein", a tool for the pre-processing and overall data wrangling of biological networks for machine learning tasks.
strengths_and_weaknesses: The tool is very well documented and the code seem to be professional-grade. 

While the documentation in it's current version is very useful, it feels incomplete. This is mitigated by the authors providing a set of easy to follow and high quality set of examples, provided as python notebooks.

The single "weakness" of this package in my opinion is the lack of a manual for the "advanced API" (although this reasonable, considering the audience of such and API are expected to understand the inner workings of Graphein to use it adequately).

I've managed to run ALL the examples successfully in a couple of days, and had minimial issues that could be resolved with a quick search on the documentation or reading the next section of the manual. 
questions: - The documentation seems to be generated from python notebooks, and the output includes a lot of debug information. Could this be made "toggleable"? It will improve with initial reads of the reference, as (at a first glance) the output looks like an error.
limitations: Considering the quality of Graphein and the existing quality of other well-established pieces of code oriented for computational biology, I consider the limitations in the documentation a minimal issue for Graphein.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work introduces a Python library, Graphein, for transforming raw data including biomolecular structures and interactions from widely used bioinformatics databases into the datasets for geometric deep learning. Graphein supports a broad selection of data sources including small molecules, proteins, RNA, and biological interaction networks. Besides, Graphein provides several machine learning utilities for users to better work with the data.
strengths_and_weaknesses: - Originality: Although there are existing tools for building biomolecular graphs, they all exhibit limited utilities for geometric deep learning. The proposed tool covers a broader selection of bioinformatics data and provides more utilities for the transformed graph-structured data. 
- Quality: The proposed library provides convenient structure retrieval and graph creation tools. More empirical results based on the library need to be collected and shown.
- Clarity: The paper is well written and the proposed tool is clearly introduced.
- Significance: Geometric deep learning in the field of bioinformatics is getting more and more popular these years. Although there are several tools for transforming bioinformatics data to graph-structured data, they only support a small selection of data sources, and the transformed data can only be directly used by limited geometric deep learning libraries. Given that the proposed tool provides wider support for both data sources and downstream libraries, many researchers would be interested in using this tool for preparing graph-structured data for various tasks. 

questions: Currently, only a limited number of datasets are constructed in Graphein. Will more datasets that better cover each of the supported data types/sources be designed in the future?
limitations: To attract more researchers to use Graphein in their work, the results of baseline models can be collected and provided together with the datasets made by Graphein for comparison.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Graphein is a library for converting graph information contained in widely used bioinformatics databases to a machine learning friendly form. This allows machine learning researchers consistent and flexible preprocessing steps for structural and interaction graph data. As compared to previous widely used ML benchmarks and libraries which extract and collect fixed benchmark datasets, Graphein allows more customization to add new features as needed. This provides more low-level control for researchers to build and manipulate their own datasets beyond the standard datasets already supplied in tools like pytorch geometric, torch drug, and there are undoubtably researchers who are finding and will find Graphein useful for preparing biological structure and network data.
strengths_and_weaknesses: Strengths:
* Flexible preprocessing pipeline that abstracts away some of the implementation details of bioinformatics datasets.
* Excellent and clearly written documentation and notebooks giving a high-level introduction to the library.
* Good visualization tools, particularly for protein structure graphs at various levels of granularity.
* Easy to install and provides docker containers for portability. 
	
Weaknesses:
* The paper is mostly a summary of what the library is. I’m not sure I would ever recommend reading the paper over the documentation (of course this means the documentation is already excellent).
* There is limited comparison to the features provided in other related libraries. There are a number of tools that load and process geometric datasets, this paper could provide more information as to where Graphein fits in this larger ecosystem, what features it provides that other libraries do not, and why someone should use Graphein rather over custom preprocessing steps.
* There is no use / validation of datasets provided in the paper so it is difficult to evaluate how useful these particular datasets are in benchmarking methods. 
* As this paper presents a python library for curating, visualizing, and preprocessing datasets, there is nothing novel in the traditional ML sense of new ideas or architectures to evaluate in this work.


questions: Isn't this paper better suited for the datasets and benchmarks track? This paper is difficult to evaluate on the typical dimensions of NeurIPS main track: originality, quality, clarity, and significance. 

Consistent and repeatable datasets with train and test splits have proven extremely useful in the broader ML field. Using these benchmarks, methods can be directly and easily compared with minimal task-specific knowledge. These benchmarks are relatively easy to standardize and agree upon in image and natural language tasks. However, in biology there are vastly greater possible tasks and more heterogenous data sources. Graphein provides a very flexible pre-processing pipeline. I wonder if the authors have any thoughts as to how to ensure consistency between as data sources and data pipelines change? 


Minor comments:
Line 332: pre-rocessing  --> pre-processing

----------------------------
Post Response update:

Thank you to the authors for your detailed response. My concerns have been addressed and I have updated my score accordingly 4 --> 6. I believe additional discussion comparing Graphein to other molecule processing libraries would be helpful to answer the question "Why Graphein?" for ML practitioners, but overall a good contribution to ML infrastructure for biological graph dataset processing.
limitations: Yes
review_title: null

Meta-review: metareview: The paper describes a python library called Graphein, for working with biomolecules. The manuscript provides a bridge between life scientists and machine learners, describing the high level concepts that will facilitate a meaningful interaction between the two fields by suitable use of the Graphein library. The library provides programmatic interfaces to query bioinformatics databases, and also enables the use of popular geometric deep learning libraries.

Four reviewers carefully considered the paper, and the associated library, and they unanimously agree that the work provides an excellent contribution to the machine learning literature. It gives me great pleasure that the new directions of software in the NeurIPS community has attracted high quality submissions such as this. Therefore I recommend this paper for acceptance at NeurIPS 2022. Congratulations!
confidence: Certain
Decision: Accept