Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper advocates 3D representation for solving a suite of prediction tasks related to small molecules and/or biopolymers. The author curated either existing or new dataset for each task with a focus on reducing similarity between train and test sequences. Simple yet robust implementation of 3 classes of 3D models are benchmarked against state-of-the-art models with 1D or 2D representation, and show better performance over lower-dim counterparts. The author provides observation and insights about how the choice of model class depends on the task at hand and the many-body hypothesis as a design guideline. 
By providing benchmarking datasets/tasks and baseline models, this work will help lower the entry barrier for ML people to get into comp bio / molecule design.   



strengths: - This paper is well motivated and clearly presented. 
- Provides a landscape of important tasks/datasets, and state-of-the-art results. It will be a good start point for ML people interested to work on molecular learning. 
- The dataset and models are well documented and open sourced. Also dataset creation and models are detailed in supp materials. 
- The limitation of the current work and open questions are also discussed.
weaknesses: - The implementation of benchmark 3D models are indeed simple, but not sure they are "strong". It is not clear for a reader outside the community to know how strong the performance is if they are not familiar with the metrics. e.g. how far we are to solve a task. Also there are no 1d/2d models are in 2020?
- Compute / model sizes / inference time are not compared between models. I guess overall this domain are more data limited rather than compute limited.
- Lack of deeper insight regarding why 3D representation is better, e.g. what type of high-level structural pattern those 3D models can learn while 1D or 2D can't.
correctness: - The claim that 3D representation is better than 1D/2D repr for tasks related to structure/shape is quite intuitive, as they provide more complete info. I don't doubt this. The challenge is probably we don't always have those 3D structures available.
- I like the way to split training/test data with 30% seq identity. The authors provide sufficient details on source, preprocessing and dataset code.
- For benchmark models, for some models e.g. Cormorant, reduced dataset are used, in this case how to compare with 1D/2D baseline, with the same training/test data?
clarity: yes
relation to prior work: not sure if related work section is complete. e.g. related work on protein folding.
additional feedback: 
- may provide a leaderboard for those tasks in the future
- Figure 1: PPI --> PIP
- L739: HDF5 --> LMDB
- L854 - 862: not clear to me after going through several times
review_title: comprehensive work on dataset and benchmark for 3D molecular learning

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper introduces code and benchmarks that facilitate the use of state-of-the-art machine learning tools like convolutional neural networks and graph neural networks in the prediction of 3D molecular structures. Accurately modeling molecules in 3D is critical to mapping structure to function and has major implications to elucidate molecular mechanisms in biology and for drug discovery. The authors here provide a suite of readily accessible benchmark datasets and open-source code to help accelerate research and comparisons of computational models.
strengths: -- This dataset will be useful to accelerate research in a very important area of biology/biophysics.
-- This reviewer understands that the data and code are readily and openly accessible.
-- Investigators have been trying to predict 3D structure for decades. A large fraction of these efforts have been based on “basic physics principles” or even protocols that engage large numbers of humans. More recently, many heavy players in machine learning including DeepMind have become interested in this problem. The availability of benchmarks can provide a major boost to the field. 

weaknesses: -- I am not sure that this is a serious weakness but it seems that the field of 3D molecules is extremely large and heterogeneous. Predicting the structure of a 300 aminoacid protein is rather different from predicting the structure of a 20-atom ligand and the principles behind how chains of aminoacids folds can also be very different from those underlying folding of RNA molecules. The authors briefly allude to different subproblems (e.g. small molecule, protein, residue, etc.). I cannot help but wonder whether the field would benefit more from multiple separate datasets for different types of problems.  
correctness: The evaluation methods and design of the datasets seems to be sound and solid
clarity: The paper is very well written.
relation to prior work: This is the main weakness of this reviewer (not the authors or the paper). I am not an expert in all the datasets for 3D molecular structures. The ones I know about are cited here, but I cannot claim completeness. Other reviewers may have more to say about this important point.
additional feedback: None
review_title: Exciting push forward in the map from low-dimensional to 3D modeling of molecular structures

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper presents ATOM3D, a collection of both novel and existing benchmark datasets spanning several key classes of biomolecules. It experiments extensively with 3D molecular learning methods on various tasks presented in ATOM3D, and concludes a set of best practices for the field of 3D molecular learning, and reveals a number of insights related to 3D molecular learning. More notably, it makes 3D molecular learning extremely accessible by integrating everything into the atom3d Python packages. 
strengths: + The presented ATOM3D benchmark is a useful resource to the community. Notably, it makes 3D molecular learning extremely accessible by integrating everything into the atom3d Python packages. It is valuable in accelerating research into this field.
+ There are a number of insights about the benefits of using 3DCNN and GNN over conventional 2D and 1D methods on multiple tasks. These findings and insights are useful to know.
+ It identifies opportunities in several under-explored molecular tasks. 
weaknesses: - I think some of the phrasing in the paper is obscure and needs improvement from a machine learning practitioner's standpoint. For example, it is not very clear if line 173 is describing the metric or the task. "We predict the root-mean-squared deviation (RMSD) from the ground truth structure".  RMSD seems to be a metric, but how do you predict a metric? I think what it meant was to "evaluate the predictions on the RMSD metric". 

- It is not clear what we can learn from some of the results presented in table 6. For example, it is not clear what is the difference between 3DCNN and the SotA of [Alford,2017], although both methods are 3D. Simply showing one is better does not tells us much about why one outperforms another. 

- I think it helps to give more stats about each dataset presented in the benchmarks, e.g. number of samples etc.
correctness: Yes.
clarity: Overall clear but could be improved (see above comments about phrasing)
relation to prior work: Yes
additional feedback: ==============================
Post-rebuttal: 
I appreciate the authors for addressing my concerns. I am raising my rating.
review_title: A useful resource that is accessible by machine learning practitioners

Meta-review: The authors propose datasets and benchmarks for developing machine learning novel algorithms on 3D representations of molecules. The reviewers note the importance of the problem, the clarity of the presentation, and the scope of the work. They strongly recommend acceptance.
Decision: Accept