Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: - The paper proposes FS-Mol, which is a few-shot learning dataset of small molecules for QSAR. 
- The paper includes a benchmarking procedure on this dataset, and report baseline performance
strengths: - Few-shot learning is a relevant problem for early-stage drug discovery, and a standardized benchmark for the problem would be a useful resource. The dataset is carefully constructed. 
- The paper reports performance of 6 representative baselines. 
- The paper is well-written.
weaknesses: - My main question is why we only consider a small dataset size for training tasks. For the test task, it makes sense that there is a limited number of molecules since the setting considered is early-stage drug discovery. However, for training tasks, it seems reasonable to have activity data on many more molecules. In fact, to construct the dataset, the authors filter away tasks with lots of data points.
- I'm also wondering how tractable it is to do few-shot transfer, when the identity of the target protein is unknown. With just a few molecules and an unknown protein target, transfer seems challenging.
- The dataset is a curated form of a standard database. Careful curation is important, but the contributions are limited in my opinion. 
correctness: The paper is correct to my knowledge
clarity: The paper is written clearly
relation to prior work: The related work focuses on comparing with other datasets, and should be expanded to include prior work on few-shot learning for drug discovery. 
additional feedback: Please see comments above
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work provides a molecular dataset for modelling QSAR. Motivated by the real-world settings of limited datapoints and structuring reusability among molecules being experimented with, the work proposes a few-shot learning framework. Augmenting this, the work also provides a benchmarking procedure considering the datapoint per task as well as the number of tasks. 
strengths: 1. The crucial parts of the work: dataset preparation, splitting, task design, and benchmarking procedure, are very clearly described. 
2. The chosen baselines seem representative and their performance is analysed well with intuitions.
3. The provided code extensive and is well-structured 
weaknesses: Disclaimer: This work falls out of my area of expertise. Judging the work purely as a dataset and benchmarking work, I see very few weaknesses.

1. In Section 5.1 (Evaluation Methodology), the authors choose random splitting. However, does a five-fold stratification suffice to negate the noise due to 'lucky' splits? (In my experience in medical data, it does not...)
2.  "We used typical configurations for these classes of models based on the extensive industrial experience of some of the authors." (Line 243) could be replaced with the lessons that these authors can provide to the community. 
correctness: Yes. From a FS perspective, the dataset is constructed correctly and documented appropriately.
clarity: Yes, the paper is well written.
relation to prior work: Related work in Section 4 and Table 1 is well summarised. 
additional feedback: I really appreciate the attention given to the construction of the dataset and the content of the github repository.
review_title: Good dataset with sensible experiment and validation design

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper presents FS-Mol, a curated dataset for benchmarking few-shot learning methods and evaluating potential of such methods to improve performance on small labeled datasets. The authors generate a large set of tasks consisting of molecules and corresponding biological activity measurements from CHEMBL, and define standardized splits into training, validation, and testing sets. They benchmark several machine learning techniques, including few-shot and meta-learning techniques as well as single-task and self-supervised baselines. Results show that few-shot learning techniques improve over single-task and self-supervised baselines for tasks with small training data, but the benefit decreases as the amount of data for a new task increases.
strengths: - Standardized datasets and benchmarks for QSAR across a wide range of targets is a useful resource and contribution to the field
- Code and data is easily accessible on Github, along with processing code needed to adapt and generate new datasets, as well as update as the underlying CHEMBL data grows
- Ability to specify alternative featurizations for input molecules is useful and improves accessibility
- Benchmarking results provide useful context and comparison between techniques, particularly as performance varies with different dataset sizes
weaknesses: - Splitting the tasks into train/test by protein target makes sense, but there may still be highly related/homologous proteins that span between both sets. It would be good to see an assessment of how similar or different the protein topologies are between each of the splits, and how this impact generalization performance
- Also related to splitting, in Lines 138-139 it is unclear to me if the entire test set consists only of enzymes, or if they simply partitioned the subset of the test set by EC number. If the former, it would be good to have a wider distribution of protein families in the test set.
- For the self-supervised baseline, it seems the authors only fine-tuned on the support sets of the test tasks. A stronger baseline would be to fine-tune a self-supervised model as a feature extractor on the training set tasks before evaluating on the test set (similar to the procedure used for the multi-task baseline).
correctness: Datasets are well constructed and procedure is fully described in paper and code.
clarity: Yes
relation to prior work: Yes
additional feedback: None
review_title: A valuable dataset for few-shot learning in drug discovery

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper frames lead optimization in drug discovery as a few-shot learning problem, and presents a well-selected yet extensive subset of ChEMBL bioassays as an accessible starting point for further development of such methods. The dataset has distributional properties which differ from existing datasets and corresponds to an important stage in drug discovery where modern deep learning methods have yet to had a significant practical impact. The paper provides well-chosen baselines corresponding to classical ML methods, single-task deep learning methods, and deep learning methods leveraging few-shot learning.
strengths: The dataset provided by this paper is useful to both the few-shot learning and ML in chemistry communities.
* The dataset helps broadens the scope of few-shot learning beyond the fields where it is most established. The problem addressed is well-formulated, of clear importance, and involves an important but relatively less-explored data modality (graphs).
* Much recent work in deep learning for chemistry has focused on the large-data regime; such methods are rarely useful for practical problems where data is scarce. This paper helps draw attention to problems of greater relevance/impact to industry practitioners.
* Although the ChEMBL dataset is pre-existing, the authors collect, discretize, and split it in thoughtful manner which helps researchers ask high-level, interesting questions (RQ2, sub-split by EC, etc.) about the dataset.
* The baselines chosen by the authors are appropriate and interesting, spanning from traditional QSAR methods to atomic transformers.
* The results, especially broken down by support set size, are of significant interest to the ML in chemistry community.
weaknesses: * (RQ2) could be better formulated and explored.
    - (1) The progressive elimination of test-tasks as support size increases leads to artifacts in the results. The authors note this phenomenon in L318; however, ideally the results shown would adjust for this (in one of several possible ways).
    - (2) It would be very useful to see the dependency of few-shot learning methods on the number of train-tasks. I am sympathetic to the fact that it may not be feasible to evaluate all possible combinations for the number of train-tasks, $|\mathcal{T}_\text{support}|$ for train tasks, $|\mathcal{T}_\text{support}|$ for test tasks, etc. However, it would be useful to explicitly mention these variables, and the types of chemical application contexts they may correspond to.
correctness: * I am concerned by the size-bias in splitting (line 142). We don't get to see, for example, whether including a greater proportion of data-rich tasks in the train-task set would help meta-learning more (or less).
* Another very concerning possibility is that there is some covariate to dataset size that would skew the results. While this cannot be totally eliminated, it would be better to reconstruct train-test splits for each value of $|\mathcal{T}_\text{support}|$
clarity: The paper is generally well written. 
* However, Figures 1(a) and 1(b) could be misinterpreted as plotting cumulative number of compounds vs number of assays. I recommend showing the same data in histogram format, or otherwise emphasize that the assays are ranked by size (of the dataset).
* Details on the model selection / early stopping procedure during fine tuning are lacking.
* Details on the construction of the baselines (L243) are unclear. Are the hyperparameters chosen by the "extensive experience of some of the authors" or the grid search? If the latter (as the text suggests), then what was chosen based on experience?
* A slightly more detailed exposition of the meta-learning methods L284-296 would be helpful as context.
relation to prior work: While the dataset itself is not new, and is also quite similar to many existing datasets, authors make a convincing argument for its relative strengths, mainly in terms of number of tasks. I'm not convinced that the large number of molecules per task in the other datasets is a weakness of the previous datasets, though, as it's always possible to choose $|\mathcal{T}_\text{support}|$ to be as small as necessary. The class balance argument is similarly unconvincing for similar reasons.
additional feedback: Given that some of the EC categories had only one task in $\mathcal{D}_\text{test}$, Table 2 does not add much to the paper. The claim in L330 is also very broad and unconvincing without further elaboration/quantification.
review_title: Well-curated dataset and benchmarks for a high-impact application area for few-shot learning

Meta-review: This paper proposes FS-Mol, a molecular dataset and benchmarking system especially motivated by recent progress and work on few-shot learning approaches in the biological/molecular domain. Overall, reviewers found this to be a strong and valuable contribution to the community.
Decision: Accept