Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper is concerned with 3D molecule learning. They propose a collection of existing and new datasets (curated from existing datasets). They show that a lot of existing tasks can do well when 3D structure is considered. 

The main contribution of the paper is in curating the datasets into a well defined framework with consistent splits and evaluation metrics. This would allow the community members to easily benchmark their approaches against a variety of tasks. Also, the tasks encompass a range of modalities including RNA, Proteins and Small molecules. Some tasks like Ligand Binding Affinity (LBA) and Ligand Efficacy Prediction (LEP) requires modeling representations of both proteins and small molecules. I believe this benchmark dataset will be useful to the community.

I have some complaints with the experimental evaluations. For SMPs, a variety of unsupervised feature extractions methods have been proposed, which can be applied on 1D (SMILES representations) as well as graphs. For example see [1] and [2]. In QM9, N-Gram XGB (See table 2 from [2]), performs very well (top 1 performance on 9 out of 12 tasks). Did adding 3D information improve on this result? It is not clear. 

Similarly, for binding affinity prediction, the authors compare with DeepDTA. However, DeepAffinity[3] has shown promising results just using 1D representations. Are 3D models better than this? What happens if we use the same models on PDBBind data?

[1]  Rong et al: GROVER: Self-supervised Message Passing Transformer on Large-scale Molecular Data
 https://arxiv.org/pdf/2007.02835.pdf
[2] Liu et al: N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Molecules
[3] Karimi et al: DeepAffinity: Interpretable Deep Learning of Compound-Protein Affinity through Unified Recurrent and Convolutional Neural Networks

review_title: Main contribution is the dataset curation

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors introduce a repository of datasets for several atomistic learning tasks. These datasets are processed into a simple and standardized format. A systematic benchmark with atomistic learning methods is presented, showcasing the value of using 3D atom-level data instead of 1D or 2D features. The authors argue that these datasets will serve as a stepping stone for machine learning researchers interested in developing methods for atomistic learning and rapidly advance this field. The paper also presents the best practices for each of the tasks, as well as the splitting and filtering criteria to ensure generalizability and reproducibility.

Strong points of the paper
- The authors present a systematic evaluation of atomistic learning across multiple tasks and show that 3D data consistently yields better performance than 1D and 2D methods.
- The experiments are thorough and the literature for every task has been covered extensively. The performance evaluation for each task is fair, respecting the corresponding metrics.
- Novel molecular tasks and their corresponding datasets are introduced. This is a strong point for this paper. The availability of standard curated datasets helps to advance the field by making it easier to develop and compare new methods for these tasks.

Weak points of the paper
- The idea of using atomistic learning or at least 3D derived features have already been implemented or at least contemplated in many of the presented tasks (Gilmer et al [1], Wu et al [2], Townshend et al. [3]). In fact, these methods are often SOTA in their fields. Because of this, I feel that there is not much novelty with atomistic learning. 
- If the authors are selecting tasks where there are SOTA 3D methods, it would be also interesting to evaluate the performance against these methods (Townshend et al. [3] for instance). Or at least explain how their 3D approach is different.
- I feel that the selection of multiple tasks is limiting the authors in the amount of information that they can fit in the actual paper. As it is now, the paper relies heavily on the Appendix as important information is described there and not in the main paper.

My overall recommendation is a weak accept. This is because the methods used and the idea of atomistic learning are not novel. The novelty of the paper lies in the introduction of the curated datasets and the novel tasks defined by the authors. By creating a standardized set of prediction tasks and associated data sets, the authors have presented a resource that may help the community to compare 3D atomistic methods quickly and fairly. However, I personally feel that this work could be a better fit for a more biologically inclined venue.


Questions for the authors
- For the MSP task, could you explain why the problem was framed as a binary classification problem? There seems to be a fair amount of papers tackling the non-binary problem (Montanucci et al. [4], Cao et al. [5], Rodrigues et al. [6] to name a few).
- Is PPI the appropriate name for this task? Townshend et al., which as stated in the paper is a motivation for the approach, frames the problem as Protein Interface Prediction. The same goes for the baseline (Sanchez-Garcia et al. [7]). Although related, the name PPI could generate confusion as it spans a whole different literature. 

Suggestions
- I feel that it would be interesting to have a more detailed discussion on why the atom-level data improves performance on each task.
- If what the authors want is to show that 3D features outperform 2D or 1D for these molecular tasks, then a systematic evaluation of the same methods but with different types of features would be more significant. For instance 3D CNNs vs 2D CNNs (where possible).
 
References
[1] Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., & Dahl, G. E. (2017). Neural message passing for quantum chemistry. arXiv preprint arXiv:1704.01212.
[2] Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., ... & Pande, V. (2018). MoleculeNet: a benchmark for molecular machine learning. Chemical science, 9(2), 513-530.
[3] Townshend, R., Bedi, R., Suriana, P., & Dror, R. (2019). End-to-end learning on 3d protein structure for interface prediction. In Advances in Neural Information Processing Systems (pp. 15642-15651).
[4] Montanucci, L., Capriotti, E., Frank, Y., Ben-Tal, N., & Fariselli, P. (2019). DDGun: an untrained method for the prediction of protein stability changes upon single and multiple point variations. BMC bioinformatics, 20(14), 335.
[5] Cao, H., Wang, J., He, L., Qi, Y., & Zhang, J. Z. (2019). DeepDDG: predicting the stability change of protein point mutations using neural networks. Journal of chemical information and modeling, 59(4), 1508-1514.
[6] Rodrigues, C. H., Pires, D. E., & Ascher, D. B. (2018). DynaMut: predicting the impact of mutations on protein conformation, flexibility and stability. Nucleic acids research, 46(W1), W350-W355.
[7] Sanchez-Garcia, R., Sorzano, C. O. S., Carazo, J. M., & Segura, J. (2019). BIPSPI: a method for the prediction of partner-specific protein–protein interfaces. Bioinformatics, 35(3), 470-477.

review_title: Review for: ATOM3D: Tasks On Molecules in Three Dimensions

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a large benchmark of machine learning tasks for molecules represented by the 3D coordinates of their atoms. The benchmark is a combination of existing data sets and newly created ones, and covers a variety of applications and tasks, from small molecules to RNA or protein structures, and including classification, regression and ranking tasks. In addition, three deep-learning algorithms are implemented and evaluated on these benchmarks, and compared to state-of-the-art methods that do not use 3D information, and empirically demonstrate the benefit of incorporating 3D information in the networks.

The vast majority of machine learning methods that have been developed for molecules use either 1D or 2D information. Therefore, this resource and the empirical demonstration that using 3D information can improve performance can be very valuable to the community. However, I have a number of concerns about this paper:

1. I am assuming this has not been incorporated to the paper for anonymization reasons, but could the authors confirm that they are indeed planning to make both the data sets and the code used to produce the results (in particular, the three proposed neural networks architecture) available? This is obviously essential to the paper and I would feel more comfortable accepting a version of the paper that includes this information (possibly with URLs withdrawn if there is a concern about maintaining the review process blind).

2. The paper does not discuss the nature of the 3D information further than "By representing a molecule's atoms and their 3D positions". However, molecules do not have a fixed 3D structure, but rather multiple conformations driven in particular by rotatable bonds. Determining the multiple possible conformations of drug-like molecules is still an ongoing research topic (see for example the review of Hawkins (2017)), not to mention the determination of the 3D structure of proteins, which is indeed the topic of one of the data sets provided. What information is provided in the different data sets (a single conformation? multiple conformations?) and how does this affect both algorithms (if several conformations are used) and prediction performance? 

3. I would really refrain from using "atomistic learning" to describe what the community has been referring to as "learning from 3D molecular representations" for decades. 

Actually, the abstract (and, more generally, the paper) reads as if neural networks were the only kind of machine learning algorithms that could be applied to molecules and that very little work has been done in the past to incorporate 3D information in chemoinformatics. While it is true that most current techniques rely mostly on 2D (for small molecules) or 1D (for large molecules) representations, it is not for lack of trying to incorporate 3D information, but because 1) this information was either lacking or incomplete (in the sense that a single conformation gives somewhat limited information; for example, you may have the crystal structure of a protein, but that doesn't inform you directly on the pose of its pocket when binding a specific small molecule) and 2) earlier attempts at making use of 3D information have often found that it did not improve performance (see Swamidass et al. (2005) or Azencott et al. (2007)), either because of the aforementioned incompleteness or because the methods were not up to par. The framing of the paper ignores decades of work in chemoinformatics, in particular (but not limited to) around kernel methods. I am listing a few examples of such papers below, not because I think they should all be included in this paper, but because in my opinion the paper would benefit from considering this body of work.

In addition, although some authors have already used "atomistic machine learning" in the context of chemoinformatics (see Schütt et al. (2018)), the term "atomistic learning" is already often used in opposition to "holistic learning" in education.

4. Finally, in Section 4, the paper could benefit from stating very explicitly what is novel and what is not novel in the three proposed 3D architectures.

Axen, Seth D., et al. "A simple representation of three-dimensional molecular structure." Journal of medicinal chemistry 60.17 (2017): 7393-7409.

Azencott, C.-A., et al. "One-to four-dimensional kernels for virtual screening and the prediction of physical, chemical, and biological properties." Journal of chemical information and modeling 47.3 (2007): 965-974.

Gaüzere, B., Brun, L., and Villemin D,. "Two new graphs kernels in chemoinformatics." Pattern Recognition Letters 33.15 (2012): 2038-2047.

Hawkins, Paul C. D. "Conformation generation: the state of the art." Journal of Chemical Information and Modeling 57.8 (2017): 1747-1756.

Mahé, P., et al. "Graph kernels for molecular structure− activity relationship analysis with support vector machines." Journal of chemical information and modeling 45.4 (2005): 939-951.

Mohr, J. A., Jain, B. J., and Obermayer, K. "Molecule kernels: a descriptor-and alignment-free quantitative structure–activity relationship approach." Journal of chemical information and modeling 48.9 (2008): 1868-1881.

Nettles, J. H. et al. Bridging chemical and biological space: “target fishing” using 2D and 3D molecular descriptors. J. Medicinal Chem. 49, 6802–6810 (2006).

Rhodes, N., Clark, D. E. & Willett, P. Similarity searching in databases of flexible 3d structures using autocorrelation vectors derived from smoothed bounded distance matrices. J. Chem. Info. Mod. 46, 615–619 (2006).

Schütt, K. T. et al. SchNet - a deep learning architecture for molecules and materials. The Journal of Chemical Physics 148(24), 241722 (2018)

Swamidass, S. J., et al. "Kernels for small molecules and the prediction of mutagenicity, toxicity and anti-cancer activity." Bioinformatics 21.suppl_1 (2005): i359-i368.
review_title: An interesting resource but needs a bit more context

Meta-review: For many problems such as ligand-protein binding, quantitative structure activity prediction (QSAR), predicting protein function from structure, etc., the 3D geometry of the molecules is of great importance.  One way to represent this is simply to assign locations to all atoms in 3-dimensional space.  If using graph convolutional kernels or other relational representations such that aligning molecules is not necessary, these approaches with 3D geometry can be efficient and far more effective than 1D or 2D representations.  The contribution of the paper is to make this point and to produce a resource with this kind of 3D data.  Such a resource would be of high value.  Nevertheless, reviewers feel provision of such a resource is perhaps not a major contribution to the ICLR and ML communities.  There is a sense that more innovative and substantial contribution would come from addressing also the challenge that 3D geometry can changes and that there may be multiple low-energy conformations of biomolecules that should be considered.  The authors contend that unlike ligands which are small and may have many low-energy conformations, large biomolecules have a much more constrained conformational space.

This meta-reviewer is sympathetic to the authors' point and appreciates the importance of the resource.  Nevertheless, even large biomolecules often have some portions of flexible conformation and high 3D structure variation that should be considered.  And indeed addressing the kind of multiple instance problem that arises by considering multiple conformations of large molecules or of ligands binding to large molecules would certainly require and likely yield bigger ICLR/ML innovations.  In the end the paper contributes a useful resource but does not excite the reviewers substantially enough, without those extensions or others, for a recommendation of acceptance at this time.
Decision: Reject