Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors proposed a few-shot learning model to tackle the lack-of-data problem in drug discovery, where they demonstrated the proposed MHNfs model on benchmark dataset FS-Mol and the Tox21 dataset, reportedly outperforming existing methods. I find this paper clearly written with key concepts well explained. The central idea behind the proposed method, which is to enrich molecular representations through querying from a massive reference space (i.e., the context) is reasonable, since it is what a human expert would do in this scenario. Technical details are concisely discussed in the main text as well as the supplementary information, where non-domain experts should be able to understand and appreciate their work.
strengths_and_weaknesses: **Strengths**

[*originality*] The proposed model architecture is novel for the few-shot learning task, compared to existing models listed in Table1.

[*quality*] The paper is well written with a clear story line and very few typos or grammar issues.

[*clarity*] The key idea is clearly explained and model architecture is accurately described.

[*significance*] The proposed method aims at solving a severe problem commonly encountered in drug discovery. The model seems to outperform all other existing models on the two prediction tasks demonstrated in the paper.

$\newline$

**Weakness**

[*originality*] The attention mechanism has been heavily exploited in the MHNfs model. Although the authors explained the difference between the proposed CM, CAM modules, they eventually both break down to the attention mechanism, which relies on computing the cosine similarity of linearly transformed key, query, and value embeddings. To that end, the novelty of the network architecture is not that significant in my opinion, regardless of the jargons and module names used to wrap these operations. That being said, the authors should probably spend more effort discussing and analyzing the significance of explicitly separating the support and context set, in comparison to other traditional methods. 

[*quality*] It would be ideal to have more discussions or insights on how the enrichment process benefits the predictive tasks (e.g., what did the model learn that made it better?), not just presenting tables with superior metrics. In addition, the authors used hand-crafted features (fingerprints and physical properties) for molecular embedding, I think it is widely accepted that there exists better molecular encoders, e.g., graph representations.

[*clarity*] No major concerns.

[*significance*] Regarding the quantitative model evaluation results, it would help the readers better appreciate the significance of this work by providing some sense of "a good model". Outperforming existing models is, of course, a significant contribution -- but readers who are not familiar with the specific predictive tasks may find it hard to judge whether the proposed method actually solved the proposed problem. For instance, in Table2, is AUC 0.679 good enough for the Tox21 task, or just a marginal performance gain compared to other methods?
questions: **Major concerns**

- For a real drug discovery problem, how should we choose the context dataset? The authors used part of the training set for the FS-Mol and Tox21 datasets, where the molecular data came from the same database, i.e., they are similar/related in some sense. However, when we only have access to, say, 16 known drug molecules for a particular disease, what would be the reasonable way to choose the context set? The authors should add more discussions about this problem. 

- A follow-up to the previous concern, is it possible to use a general molecular database, e.g., GEOM-drugs, as the context set, and use the same support and query set for FS-Mol and Tox21 predictions? If that works, then the proposed context-enrichment method really works.

- In the main text, line 82-84, "Those retrieved representations have amplified co-occurrences and covariance
83 structures, while peculiarities and spurious co-occurrences of the query and support set molecules are
84 averaged out." Is it possible to illustrate this with an example, or somehow support the claim in your case?

- In the main text, line 110-113, the authors discussed the limitations of standard supervised ML on data-scarce problems. I understand the point here, but I'm not sure if the statements here are accurate. Not all ML models will overfit the dataset, especially if somehow the model learns a proper similarity kernel function, which is exactly what the MHNfs model is doing.

- As far as I know attention-based models are data-hungry, and takes time and computation to extract meaningful hidden patterns. However, given the dimension of the hidden space (thousands) used in this work, the number of data points (thousands) in the context set, and the number of epochs (20), I was wondering if the model has been properly trained. Can you please show some learning curvs, if possible? I am also kind of curious to see if there is overfitting problem in this case.


**Minor concerns**
- In the main text, line 79, "as an associative memory".

- In the main text, line 92, "and a domain shift experiment".

- In the main text, line 122, "three consecutive modules that feed into each other". It is a bit confusing to me since the architecture is a simple feed-forward model, I don't see the modules feeding into *each other*.

- I am curious about the balancing strategy described in Equation 10. Naïvely, the kernel already provides a correlation between m and x, should we just attend to those with high correlation, and make some kind of vote from the highly correlated support set molecules, instead of using a weight based on the positive/negative ratio in the support set? Those molecules with small k(,) values may not contribute to the final voting. Any thoughts?

- In the main text, line 320, the authors claimed "significantly outperformed". From the numerical values, I get the "outperformed" part, but please justify the use of "significantly" in this case.




limitations: - The authors wrote in line 336 that "the model cannot be used for molecules such as RNA, DNA, or proteins". I do not agree with this statement. Given the fact that the proposed method essentially captures the similarity through an attention-based kernel function by referencing from the context set, regardless of the underlying molecular representation, this method should be applicable to macromolecules as well. (E.g., I personally find the MSA blocks in AlphaFold2 similar to the CM module here) In fact, this few-shot learning method is potentially applicable to other data-scarce problems, as long as we could define a kernel function for similarity assessment.

- In conclusion, I think the key problem that this work tried to address is evaluating molecular similarity in some embedding space, and use that information to facilitate model predictions with few support data. I would love to see how different molecular representations perform in terms of similarity assessment, and whether hand-crafted features outperform representations learned in a data-driven manner.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a context-enabled few shot learning method, which performs few shot classification by first looking at a group of context drug-like molecules, and then uses self-attention to distribute that information among the query and support set molecules. The authors show good empirical performance on FS-Mol, as well as ablations of the different parts of their model. Finally, the authors also present a FrequentHitters baseline, which performs very well despite ignoring the support set, yielding useful insight into why some of the FS-Mol baselines fail and how they could be improved. 
strengths_and_weaknesses: === Strengths === 

- (S1): The paper considers an important (and still somewhat underexplored) problem of few-shot drug design, proposing a method based on an interesting mechanism of context lookup. The model design is clear and reasonable, and empirical performance is encouraging. 

- (S2): Apart from outlining their main method, the authors also discuss a FrequentHitters baseline, which ignores the support set and yet performs very well, which is intriguing, and could inform improvements to existing methods (see (Q4) for further thoughts on this). 

- (S3): The paper is well-written, with clear explanations and visualizations (in particular, Figure 1 is quite nice). The discussion is well-structured, most design decisions are clearly explained and ablated. The text flows smoothly, with very few minor typos or grammar errors. 

 

=== Weaknesses === 

- (W1): The paper only reports results with support set size 16, ignoring {32, 64, 128, 256} which are also commonly evaluated (e.g. in the original FS-Mol paper [1], and some other papers reporting results on that benchmark like [2]). It would be very interesting to see a plot akin to Figure 2a in [1]. As a start, it would be interesting to see the result for at least one larger support set size e.g. 128 (at 256 some FS-Mol-specific quirks kick in, and this leads to some artifacts in the performance of some of the methods). 

- (W2): Despite most aspects of the work being well-explained, some things are still not clear to me (see the "Questions" section). Further clarifying these would help polish the paper further. In particular, addressing (Q5) would be helpful for understanding why MHNfs works well.

 

=== Summary === 

Overall, this paper proposes a clear and reasonable approach, that (despite its simplicity) works very well on an important benchmark. The authors also propose an interesting baseline, which performs surprisingly well, leading to some useful insights. Finally, the paper is pleasant to read and the results seem reproducible. I think the paper is in good shape and a useful contribution to NeurIPS, hence I vote for acceptance. 

 

=== Other comments === 

Here I include various other comments, which had only a small impact on my score. 

- (O1): Table 1 misses a (very recent) result of [2], which (looking at their Figure 1a) gets a very similar context-size-16 result to MHNfs. I'm curious as to what the comparison is for larger support set sizes. 

- (O2): The authors say they use MHNs, as opposed to attention, attributing this to MHNs being more general. As far as I understand (but please correct me if I'm wrong), the main part where MHNs are practically more general than cross attention is that the formulation of MHNs considers the possibility of sequentially updating the "attention weights" several times before returning the output (this is different from having several sets of parallel attention weights, i.e. heads). However, it seems to me many papers using MHNs don't make use of this sequential update, which reduces to standard cross attention. In particular, while I agree with the authors that Equation (9) is a special case of Equation (7), the latter looks like standard cross attention to me (i.e. it's obviously not self-attention, but it looks like the kind of attention a decoder would do over the outputs of an encoder in a translation model). In general, I'm not too strongly opposed to calling cross-attention MHNs to make a connection to some theoretical results or leave the possibility of doing the iterative update of attention weights, but if the flavour of MHNs used is synonymous to attention (which I understand is the case for this work), I think this should be made more clear. 

 

=== Nitpicks === 

Here I include some final nitpicks, which did not affect my score; they are here just to help improve the paper.

- Line 92: "an domain shift experiment" - "an" -> "a" 

- Line 108: "predicts y for a x" - "a" -> "an"; "the model that generalizes" - I'd say "a model" 

- Missing space in lines 149 and 237

- Line 241: "wich" -> "which" 

- Line 254: "dataset on related task and then adapting" -> "dataset of related tasks and then adapt" 

- Line 262: "support support" 

- Line 324: the last sentence of the paragraph reads a bit off, might be useful to rephrase it (even though it may technically be grammatically correct) 

- Line 346: "material science dataset" -> "material science datasets" 

- Line 363: "This, might" - drop the comma 

- Line 647 (appendix): "fingerprints Morgan" -> "Morgan fingerprints" 

 

=== References === 

[1] Stanley et al, "FS-Mol: A Few-Shot Learning Dataset of Molecules" 

[2] Chen et al, "Meta-learning Adaptive Deep Kernel Gaussian Processes for Molecular Property Prediction" 
questions: - (Q1): It's interesting that MHNfs works so well with a very simple encoder (just a single linear layer on fingerprints/descriptors). Did the authors try using a more complex (e.g. GNN-based) encoder? If training this end-to-end is not feasible due to overfitting, one approach could be to replace the rdkit-based input to the linear encoder with a frozen pretrained feature extractor, such as a GNN or a graph transformer. 

- (Q2): Is there any intuition behind the label balancing strategy in Section 3.4? 

- (Q3): It is not clear to me from Section 4.1 what the input to the linear encoder is: whether it's a fingerprint, descriptors from rdkit, or a concatenation of both. Appendix A.1.1 suggests a concatenation (although calling it "an additive combination"?), but that part of the paper is only about the FrequentHitters baseline. Generally, I think the input to the encoder should be clear from reading the main paper. 

- (Q4): The fact that some of the methods developed for FS-Mol are outperformed by FrequentHitters seems to point to one issue with how some of these methods are designed: there doesn't seem to be a direct way for similarity-based models to learn a "frequent hitter bias" that only depends on the query molecule. I wonder if combining a frequent hitter branch (which would directly predict the output from the embedding of the query) with some of the methods would further improve them. In particular, given that MHNfs is similarity-based, could such an addition improve results further? It's not obvious that it would, as MHNfs already performs better than FrequentHitters, but maybe FH succeeds in some of the places where MHNfs fails. 

- (Q5): Do the authors have any intuition as to what the context module is doing? What is it attending to? What kind of information can be extracted from looking at a seemingly random collection of drug-like molecules? Could it be focusing on some implicit signal, e.g. that active molecules may be more "popular" i.e. their variations may be also present in the context set as they were tested for activity against other targets?
limitations: The limitations of the method are discussed at length, and I have no concerns about that part of the paper. Perhaps one thing that would contribute to improving the understanding of the limitations is addressing (W1), as it would show whether MHNfs only works for very small support set sizes, or it can be considered a go-to method across a larger range.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Considering few-shot drug discovery, the paper proposes to enrich molecular representation by associating molecules from the support set and query set with additional context molecules. The proposed method consists of (i) context module designed upon Modern Hopfield  Network, (ii) cross attention module, and (iii) similarity module. 
The authors conduct experiments on FS-Mol dataset.


strengths_and_weaknesses: Strength

+ Figure 1 is nicely plotted, which conveys the flow of the proposed MHNfs.
+ The authors provide concise reproduction details in appendix. 


Weakness

- Motivation: The authors did not provide convincing motivations or necessity for using Modern Hopfield Networks. 
- Related Works: The paper lacks a related work section. It seems that the authors plan to describe this part in the introduction section. However, the literature review is not enough. In the past, several few-shot drug discovery works appear [a1,a2]. But the authors neglect them all. 
- Experiments: Existing works [a1,a2,a3] use classic molecular property prediction datasets (i.e., Tox21, MUV, SIDER). While the authors only consider FS-Mol. FS-Mol is a new dataset published in dataset track of NeurIPS 2021, which naturally has only been tested by naive baselines. To show the proposed MHNfs indeed works the best on few-shot drug discovery, the authors can either (i) directly compare with [a1,a2] on FS-Mol, or (ii) test the proposed MHNfs on Tox21, MUV and SIDER, such that the performance can be compared with [a1,a2,a3]. 
- Writing: The paper should be carefully proofread to correct title case, typos and grammar issues. In Table 1, the author only mark one result in bold while neglecting the others. 

[a1] Few-Shot Graph Learning for Molecular Property Prediction, WWW, 2021
[a2] Property-Aware Relation Networks for Few-Shot Molecular Property Prediction, NeurIPS, 2021
[a3] Low Data Drug Discovery with One-Shot Learning, ACS Cent. Sci., 2017



questions: Please reply to my concerns in weakness. 

Key questions: 
- There exists a number of works using external memory, such as [b1,b2]. Why consider Modern Hopfield Networks instead? Any strength?
- Please provide discussion and empirical comparison w.r.t [a1,a2]. 

[b1] Meta-learning with memory-augmented neural networks, ICML, 2017
[b2] Few-shot visual learning with contextual memory and fine-grained calibration, IJCAI, 2020

limitations: The authors provide discussion on limitations (including application domain and hyperparameter choices) and broader impact. The authors did not mention limitations of the proposed MHNfs from the perspective of methodology.
review_title: null

Meta-review: metareview: This paper proposes a context-enriched molecular representation approach for few-shot drug discovery. Specifically, they show that the proposed MHNfs outperform existing methods on standard benchmarks including FS-Mol and the Tox21.
The proposed approach is analogous to the retrieval-based approach or generation models by editing prototypes in the deep learning community, which enrich molecular representations through querying from a massive reference space, namely first looks at a group of context drug-like molecules, and then using self-attention to distribute that information among the query and support set molecules. 
This is a borderline paper and the main concern of this paper is the limited insights in result analysis (e.g., what did the model learn from the context). I recommend the authors to address the weaknesses above and resubmit to another venue.
confidence: Certain
Decision: Reject