Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The work presents a platform for supporting machine learning research on therapeutics. It consists of easily accessible datasets (66 in total) for a variety of machine learning tasks (instances of graph representation learning, domain generalization, and generative modeling), and support tools for model building and evaluation. The authors curate a diverse set of tasks in context of drug development which challenges the community to tackle cutting edge machine learning problems. Progress on algorithms on these tasks has the potential to enable scientific discovery in this high-impact domain. A comprehensive suite of benchmarks is developed to measure progress on the tasks along with code for easily performing the evaluation. With a well-documented set of support tools, the platform has the potential for wide adoption by the machine learning and domain scientists working on drug development, and thus, may accelerate progress in the field.

----
After author response

Thanks for responding in detail on unseen test sets and independent evaluation. I would encourage authors to collate references to laboratory studies for the tasks to give users an idea of acceptable performance on benchmarks.
strengths: The main strength is that the platform includes a diverse collection of machine learning tasks and accompanying datasets which simultaneously are of interest to drug development and machine learning communities. The platform also standardizes evaluation on the tasks through leaderboards and implementations of data processing and evaluation methods. Experiments with state-of-the-art methods are reported and areas for new algorithm development are highlighted.
weaknesses: The main weakness is the lack of discussion on (1) provisions to reduce overfitting of approaches to the chosen datasets and tasks (e.g. are the test sets hidden); (2) how can the submitted test results be independently validated by the platform; and (3) whether the platform will collate information on well-performing machine learning models, e.g. by enabling users to share model checkpoints or code.
correctness: Details of datasets curation and motivating applications for each are given. Evaluation metrics for the different machine learning tasks seem to be comprehensive. Experiment design follows recommended machine learning practices, e.g. reporting cross validation performance.
clarity: The authors meticulously detail each dataset and task; and their context in the drug development pipeline. The paper is quite well written. The experiment setup, baselines, and results are summarized effectively.
relation to prior work: The platform differentiates from prior work by focusing on tasks related to drug development, and providing datasets, benchmarks, and accompanying code for evaluation.
additional feedback: I am intrigued by the outlined research (line 66 in main paper) that the platform can help advance, e.g. data efficient learning and robust learning against distribution shifts for multi-modal data. I am excited about the developments to be enabled by the platform.

Minor questions

Can you make provisions for domain scientists to share guidelines on acceptable performance numbers to be met by the machine learning products on the benchmarks that will support taking them to the laboratories for further testing?

What is the rationale for not supporting model sharing or easy integration of existing model libraries, e.g. OpenChem [72], with the benchmarks?

The appendix mentions that “TDC does not involve human subjects research”. However, one proposed uses of TDC, mentioned in the main paper (line 66), is causal inference to quantify “treatment response across patient groups”. I am confused by the two statements. Is the claim that future research on TDC does not involve human subject whereas included datasets might have?
review_title: An extensive compilation of datasets, models, and benchmarks with a potential to accelerate machine learning for drug development

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper presents an overview of the Therapeutic Data Commons, a resource for datasets, algorithms, benchmarks, and leaderboards that all address machine learning problems related to drug discovery. 
strengths: The TDC is extensive, encompassing dozens of datasets, algorithms, and benchmarks. The infrastructure includes support for public leaderboards to incentive and track development over time. Everything is tied nicely together by a Python package that makes TDC accessible.

The included cases studies are useful to demonstrate the capabilities of the TDC.
weaknesses: Although the authors are not responsible for producing any of the datasets, only curating them, I would have appreciated some discussion of the ethics of collection of these datasets, i.e., were they collected with patient consent? Are there any latent HIPAA issues?
correctness: No concerns. 
clarity: The paper was clear to me. 
relation to prior work: The TDC seems go far beyond existing resources in this domain.
additional feedback: Thank you for the response and clarifications.
review_title: A high-quality, well-documented, extensive data availability and benchmarking resource

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: * TDC provides a data science ready dataset hub that allows the users to explore and retrieve drug discovery and development datasets.
* A TDC Python library was created for data preprocessing and method evaluation to help the users get started quickly.
* Some open source ML methods were applied on the datasets to solve drug discovery and development problems, and their performance were compared with the old conventional methods.

* Rating modified on July 19th, 2021 from 4 to 6.
strengths: * The authors have collected a good number drug discovery and development datasets that have been previously published.
* The authors created website and GitHub repositories for user to get access to the data and code.
weaknesses: * No novel dataset contribution to the community. All the datasets were collected from the previous publications or resources. No novel problems and tasks proposed.

* The authors were emphasizing what's different of their datasets from the other biochemical and chemical repositories are their datasets are data science ready. But it seems that many datasets were just tabular CSV datasets which are easy to preprocess, and the author did not mention how they make those datasets data science ready and whether it is easy or hard.

* TDC has to be maintained manually. There has to be people collecting new datasets from new publications and resources and arranging the new datasets. There is no easy way to consume new datasets automatically. I also did not find an interface to contribute new datasets on the TDC website, and all the dataset contributions have to go through the Slack channel. There is a lack of dataset contribution standards as well.

* I am not sure if the leaderboard can be truly trusted and useful. Instead of hiding the test set and hosting a server to evaluate the submissions from the users, TDC releases test sets and accepts user reported results via forms, although only the ones whose implementation is public and reproducible will be considered. This prevents the users, who do not want to release their code, from submitting their good results. I have to admit because of the nature of TDC that all the datasets were collected somewhere else and it is impossible to hide the test set. However, it does not make sense to serve a leaderboard unless TDC has its own private datasets and evaluation server for the leaderboard.

* Despite the TDC Python library has provided a couple of examples to get the user started, it does not have Sphinx API documentation or something equivalent, which prevents the user from using, extending, and contributing to the code. The code quality remains to be improved. The TDC Python library also implicitly assumes that the dataset size is small and can be handled by Pandas. In the age of data explosion, this assumption might not hold anymore. Other dataframe libraries, such as Rapids, might be more suitable for large datasets. Most of the data preprocessing and model evaluation metric implementation seems just trivial. 

* TDC ecosystem is a concern. Most of the MR contributions so far are from the major authors of the paper. Without the contribution momentum, TDC’s future will be a concern.

correctness: * I sort of disagree with some of the conclusions drawn from the benchmarks. 
* According to the authors, the default hyperparameters were used for the models and methods they obtained from the community, and the author did not try too much to optimize the model specifically for the datasets being used. For example, if the dataset is very small, but the number of parameters in the model is very large, without rigorous regularization or early stopping methods, the model will likely overfit to the training data and perform poor on the test data.
* In chapter 4.1, the authors claimed that "the ML SOTA models do not work well consistently for these novel realistic endpoints." However, close examination on the result Table 2 reveals that for the benchmarks that SOTA ML methods did not perform the best, the difference between the top curated methods and the top SOTA ML methods are not too large. Not to mention the hyperparameters and model architectures of the ML models were not well explored.
* In chapter 4.3,  the authors were evaluating the methods on the molecule generation tasks under the constraint of limited oracle calls. In practice, however, depending on the oracle algorithm, parallel computing implementation, and the distributed computing resources, the number of oracle calls could be almost "unlimited". Large pharmaceutical companies also have huge computing resources for oracle calls. If unlimited oracle calls is truly infeasible in practice, the author would have to mention in the paper why that is the case.


clarity: * Some of the abbreviations were not explained. For example, what is "GIN" (line 153)? 
* Some pretraining (line 153) details were not given either.
* Many tables in the papers were not referenced in the text. 
* How was the experiment repeated? What are the randomnesses of the runs?
* In chapter 4.2, the authors used temporal split, instead of other split methods, to split the drug-target interactions (DTI) dataset, without formulating the motivations clearly. The author should specifically mention that because there will be new chemical compounds being developed as drug and new proteins being investigated as targets, the model trained on the old dataset will not be valuable if it cannot perform well on unseen chemical compounds and proteins.

relation to prior work: * Overall, it is clearly discussed how this work differs from previous contributions.
* One thing I have to mention is that I think the author should have mentioned [DrugBank](https://go.drugbank.com/), which is highly related to TDC, in the related work chapter. The DrugBank provides APIs for the user to curate existing drug information. Most of the TDC dataset information might have been covered by DrugBank.

additional feedback: * TDC is a good collection of datasets used in the publications. However, it might not serve well for the people who wants to create custom datasets. 
* The TDC Python library is a good start for rookies, but it might not be too useful for experts.
* I was expecting novel datasets and/or well-investigated benchmarking on the existing datasets. But I did not see them in the paper.
review_title: A drug discovery and development dataset hub with some Python tools.

Meta-review: The paper makes a significant contribution in terms of datasets, models, and public infrastructure that has the potential to accelerate machine learning for drug discovery.
Decision: Accept