Context-enriched molecule representations improve few-shot drug discovery
Authors: Johannes Schimunek
Philipp Seidl
Lukas Friedrich
Daniel Kuhn
Friedrich Rippmann
Sepp Hochreiter
Günter Klambauer
Venue: ICLR 2023 poster (2023)

Abstract: A central task in computational drug discovery is to construct models from known active molecules to find further promising molecules for subsequent screening. However, typically only very few active molecules are known. Therefore, few-shot learning methods have the potential to improve the effectiveness of this critical phase of the drug discovery process. We introduce a new method for few-shot drug discovery. Its main idea is to enrich a molecule representation by knowledge about known context or reference molecules. Our novel concept for molecule representation enrichment is to associate molecules from both the support set and the query set with a large set of reference (context) molecules through a modern Hopfield network. Intuitively, this enrichment step is analogous to a human expert who would associate a given molecule with familiar molecules whose properties are known. The enrichment step reinforces and amplifies the covariance structure of the data, while simultaneously removing spurious correlations arising from the decoration of molecules. Our approach is compared with other few-shot methods for drug discovery on the FS-Mol benchmark dataset. On FS-Mol, our approach outperforms all compared methods and therefore sets a new state-of-the art for few-shot learning in drug discovery. An ablation study shows that the enrichment step of our method is the key to improve the predictive quality. In a domain shift experiment, we further demonstrate the robustness of our method.

Published as a conference paper at ICLR 2023
MOLECULE REPRESENTATIONS IMPROVE FEW-SHOT DRUG DISCOVERY
Johannes Schimunek', , Philipp Seidl', Lukas Friedrich Daniel Kuhn', Friedrich Rippmann', Sepp Hochreiter', and Günter Klambauer*
ELLIS Unit Linz and LIT AT Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria
schimunekeml. jku. at
2 Computational Chemistry & Biologics, Merck Healtheare, Darmstadt, Germany
ABSTRACT
A central task in computational drug discovery is to construct models from known active molecules to find further promising molecules for subsequent screening. However, typically only very few active molecules are known. Therefore, few-shot learning methods have the potential to improve the effectiveness of this critical phase of the drug discovery process. We introduce new method for few-shot drug discovery. Its main idea is to enrich a molecule representation by knowledge about known context reference molecules. Our novel concept for molecule representation enrichment is to associate molecules from both the support set and the query set with a large set of reference (context) molecules through modern Hopfield network. Intuitively, this enrichment step is analogous to a human expert who would associate given molecule with familiar molecules whose properties are known. The enrichment step reinforces and amplifies the covariance structure of the data, while simultaneously removing spurious correlations arising from the decoration of molecules. Our approach is compared with other few-shot methods for drug discovery on the FS-Mol benchmark dataset. On FS-Mol, our approach outperforms all compared methods and therefore sets new state-of-the art for few-shot learning in drug discovery. An ablation study shows that the enrichment step of our method is the key to improve the predictive quality. In domain shift experiment, we further demonstrate the robustness of our method. Code is available at https::/// / /i thub. com/m /mlmmmm---- -jku/ M.
INTRODUCTION
To improve human health, combat diseases, and tackle pandemies there is steady need of discovering new drugs in a fast and efficient way. However, the drug discovery process is time-consuming and cost-intensive (Arrowsmith, 201 1). Deep learning methods have been shown to reduce time and costs of this process (Chen et al., 2018; Walters and Barzilay, 2021). They diminish the required number of both wet-lab measurements and molecules that must be synthesized (Merk et al., 2018; Schneider et al., 2020). However, as of now, deep learning approaches use only the molecular information about the ligands after being trained on a large training set. At inference time, they yield highly accurate property and activity prediction (Mayr et al., 2018: Yang et al., 2019), generative (Segler et al., 201 Sa; G6mez-Bombarelli et al., 2018), or synthesis models (Segler al., 2018b; Seidl al., 2022).
Deep learning methods in drug discovery usually require large amounts of biological measure- ments. To train deep learning-ban based activity and property prediction models with high predictive per- formance, hundreds or thousands of data points per task are required. For example, well- perl-performing predictive models for activity prediction tasks of ChEMBL have been trained with an average 3,621 activity points per task - i.e., drug target - by Mayr et al. (2018). The EXCAPE-DB dataset provides on average 42,501 measurements per task (Sun et al., 2017; Sturm et al., 2020). Wu et al. (2018) pub- lished large scale benchmark for molecular machine learning, including prediction models for the SIDER dataset (Kuhn et al., 2016) with an average of 5,187 data points, Tox21 (Huang et al., 201 Mayr al., 2016) with on average 9,031, and ClinTox (Wu et al., 201 8) with 1,491 measurements

---

Published as a conference paper at ICLR 2023
per task. However, for typical drug design projects, the amount of available measurements is very limited (Stanley et al., 2021; Waring et al., 2015; Hochreiter et al., 2018), since vitro experiments expensive and time-consuming, Therefore, methods that need only few measurements to build precise prediction models are desirable. This problem - i.e., the challenge of learning from few data points is the focus of machine learning areas like meta-learning (Schmidhuber, 1987; Bengio et al., 1991; Hochreiter et al., 2001) and few-shot learning (Miller et al., 2000; Bendre et al., 2020; Wang al., 2020).
Few-shot learning tackles the low-data problem that is ubiquitous in drug discovery. Few-shot learning methods have been predominantly developed and tested on image datasets (Bendre et al., 2020; Wang et al., 2020) and have recently been adapted to drug discovery problems (Altae-Tran al., 2017; Guo al., 2021; Wang et al., 2021; Stanley et al., 2021; Chen al., 2022). They are usually categorized into three groups according to their main approach (Bendre et al., 2020; Wang al., 2020; Adler et al., 2020). a) Data- Data-augmentation based approaches augment the available samples and generate new, more diverse data points (Chen et al., 2020; Zhao et al., 2019; Antoniou and Storkey, 2019). b) Embedding-bee representations. Predictive models then be constructed from only few data points by comparing these embeddings. For example, in Matching Networks (Vinyals et al., 2016) an attention mechanism that relies embeddings is the basis for the predictions. Prototypical Networks (Snell al., 2017) create prototype representations for each class using the above mentioned representations in the embedding space. c) Optimization-bann on efficiently navigating the parameter space. For example, with MAML the meta-optimizer learns initial weights that can be adapted novel task by few optimization steps (Finn al., 2017).
and nearest neighbour approaches learn embedding space
or fine-tuning methods utilize a meta-optimizer that focuses
Most of these approaches have already been applied to few-shot drug discovery (see Section 4). Surprisingly, almost all these few-shot learning methods drug discovery are worse than a naive baseline, which does not even use the support set (see Section 5). We hypothesize that the under- performance of these methods stems from disregarding the context both in terms of similar molecules and similar activities. Therefore, we propose a method that informs the representations of the query and support set with a large number of context molccules covering the chemical space.
Enriching molecule representations with context using associative memories. In data-scarce situations, humans extract co-occurrences and covariances by associating current perceptions with memories (Bonner and Epstein, 2021; Potter, 2012). When we show small set of active molecules to a human expert in drug discovery, the expert associates them with known molecules to suggest further active molecules (Gomez, 2018; He et al., 2021). In an analogous manner, our novel concept for few-shot learning uses associative memories to extract co-occurrences and the covariance structure of the original data and to amplify them in the representations (Fürst al., 2022). We use Modern Hopfield Networks (MHNs) as an associative memory, since they can store large set of context molecule representations (Ramsauer et al., 2021, Theorem 3). The representations that are retrieved from the MHNs replace the original representations of the query and support set molecules. Those retrieved representations have amplified co-occurrences and covariance structures, while peculiarities and spurious co-occurrences of the query and support set molecules are averaged out.
In this work, our contributions are the following:
We propose a new architecture MHNfs for few-shot learning in drug discovery.
We achieve new state-of-the-art on the benchmarking dataset FS-Mol.
We introduce novel concept to enrich the molecule representations with context by associ- ating them with a large set of context molecules.
We add naive baseline to the FS-Mol benchmark that yields better results than almost all other published few-shot learning methods.
We provide results of an ablation study and domain shift experiment to further demonstrate the effectiveness of our new method.
PROBLEM SETTING
Drug discovery projects revolve around models g(m) that predict molecular property or activity ÿ. given a representation m of an input molecule from a chemical space M. We consider machine learning models ý = Iwo (m) with parameters ur that have been selected using training set. Typically,
2

---

Published as a conference paper at ICLR 2023
Prediction
Similarity Module
Final Representations
Context-enriched Representations Acthvty Encoding
Associations
Uneer Projection
Context Module (Hopfiell Layer)
w=
We
We
Norm
Initial Embedding
Lincar Projection
Molecule Encoder
Structure
co
Compounds
Label
Query
Support
Module
Cross Attention Module
Figure 1: Schematic overview of our architecture. Left: All molecules are fed through a shared molecule encoder obtain embeddings. Then, the context module (CM) enriches the representations by associating them with context molecules. The cross-attention module (CAM) enriches representa- tions by mutually associating the query and support set molecules. Finally, the similarity module computes the prediction for the query molecule. Right: Detailed depiction of the operations in the CM and the CAM.
deep learning based property prediction uses a molecule encoder fMF .M - Rd. The molecule encoder process different symbolic low-level representations of molecules, such molecular descriptors (Bender et al., 2004; Unterthiner et al., 2014; Mayr et al., 2016). SMILES (Weininger, 1988; Mayr al., 2018; Winter al., 2019; Segler et al., 2018a), molecular graphs (Merkwirth and Lengauer, 2005; Kearnes et al., 2016; Yang et al., 2019; Jiang et al., 2021) I and can be pre-trained on related property prediction tasks.
For few-shot learning, the goal is to select a high-quality predictive model based on small set of molecules {æI,.... NN with associated measurements y - {y1, ..... y..]. The measurements are usually assumed to be binary E [-,,1), corresponding to the molecule being inactive or active. The set {(>,, yn. ) called the support set that contains samples from prediction task and N is
the support sef size. The goal is to construct a model that correctly predicts y for an that is not in the support set other words, model that generalizes well.
Standard supervised machine learning approaches typically just show limited predictive power at this task (Stanley et al., 2021) since they tend to overfit on the support set due to small number of training samples. These approaches learn the parameters of the model In from the support set in a supervised manner. However, they heavily overfit to the support set when N small. Therefore, few-shot learning methods necessary to construct models from the support that generalize well to new data.
MHNFS: HOPFIELD-BASED MOLECULAR CONTEXT ENRICHMENT FOR FEW-SHOT DRUG DISCOVERY
We aim at increasing the generalization capabilities of few-shot learning methods in drug discovery by enriching the molecule representations with molecular context. In comparison to the support set, which encodes information about the task, the context set i.e. large set of molecules includes information about large chemical space. The query the support set molecules perform retrieval from the context set and thereby enrich their representations. We detail this in the following.
3.1 MODEL ARCHITECTURE
We propose an architecture which consists of three consecutive modules. The first module a) the context module CCM enriches molecule representations by retrieving from large set of molecules.
3

---

Published as a conference paper at ICLR 2023
The second module b) the cross-atiention module fCAM (Hou et al., 2019; Chen al., 2021) - enables the effective exchange of information between the query molecule and the support set molecules. Finally the prediction for the query molccule is computed by using the usual c) similarity module f&M (Koch et al., 2015; Altae-Tran et al., 2017):
context module:
m' = poM(M C)
X' (XM((,C))
(1)
cross-attention module:
(m", x = (CAM((m') X').
(2)
similarity module:
1 = pm" ,x" y):
(3)
where E IR is molecule embedding from trainable or fixed molecule encoder, and m' and m² are enriched versions of it. Similarly, X ÆdxN contains the stacked embeddings of the support set molecules and X' and X" their enriched versions. C RdxM is large set of stacked molecule embeddings, y are the support set labels, and Ø is the prediction for the query molecule. Square brackets indicate concatenation, for example [m' X') is matrix with N I columns. The modules fCM rCAM, and fSM are detailed in the paragraphs below. An overview of our architecture is given in Figure 1. The architecture also includes skip connections bypassing (CM(., (.,-) and FCAM() and layer normalization (Ba et al., 2016), which are not shown in Figure
shared molecule encoder f creates embeddings for the query molecule support set molecules v, { and the context molecules Cm. = (cm There are many possible choices for fixed or adaptive molecule encoders (see Section 2), of which we use descriptor- based fully-come connected networks because of their computational efficiency and good accuracy (Dahl al., 201 4; Mayr al., 2016; 2018). For notational clarity we denote the course of the representations through the architecture:
fME (m)). the
PEB
CM
CCMM
m symbolic or low-level repr.
m molecule embedding
772' context repr.
m " similarity repr.
(4)
ME
CM ac' context repr.
CAM
Ty symbolic or low-level repr.
F malecule embedding
ar' similarity repr.
(5)
3.2 CONTEXT MODULE (CM)
The context module associates the query and support set molecules with large set of context molecules, and represents them as weighted average of context molecule embeddings. The context module is realised by continuous Modern Hopfield Network (MHN) (Ramsauer et al., 2021). An MHN a content-addressable associative memory which can be built into deep learning architectures. There exists analogy between the energy update of MHNs and the attention mechanism of Transformers (Vaswani et al., 2017; Ramsauer et al., 2021). MHNs are capable of storing and retrieving patterns from memory M gex given a state pattern & E Re that represents the query. The retrieved pattern guew E IR is obtained by
new M M softmax (S
where p is called the vector of associations and 3 is scaling factor or inverse temperature. Modern Hopfield Networks have been successfully applied to chemistry and computational immunology (Seidl et al., 2022; Widrich et al., 2020).
We use this mechanism the form a Hopfteld layer, which first maps raw patterns to an associative space using linear transformations, and uses multiple simultaneous queries E E dx
Hopfield(E, (E,
C) softmax (3 (WoC)² (W=E)),
where WE E dxd and We, W E x are trainable parameters of the Hopfield layer, softmax applied column-wise, and is hyperparameter. Note that in principle the =\t and C could have
4

---

Published as a conference paper at ICLR 2023
different second dimension long the linear transformations map the same dimension Note that all embeddings that enter this module are first layer normalized (Ba et al., 2016). Several of these Hopfield layers (Vaswani et al., 2017).
in parallel and refer to them as "heads" in analogy to Transformers
The context module of our architecture uses Hopfield layer, where the query patterns the embeddings of the query molecule m and the support set molecules X. The memory is composed of embeddings of a large of M molecules from chemical space, for example reference molecules, here called context molecules C. Then the original embeddings m and X are replaced by the retrieved embeddings, which are weighted averages of context molecule embeddings:
m' Hopfield(m, C) and x Hopfield( XX, C)).
(8)
This retrieval step reinforces the covariance structure of the retrieved representations (see Ap- pendix A.8), which usually enhances robustness of the models (Fürst et al., 2022) by removing noise. Note that the embeddings of the query and the support set molecules have not yet influenced each other. These updated representations m', XX' are passed the cross-attention module. Exemplary retrievals from the context module are included in Appendix A.7.
3.3 CROSS-ATTENTION MODULE (CAM)
For embedding- based few-shot learning methods the field drug discovery, Altae Tran al. (2017) showed that the representations of the molecules can be enriched, if the architecture allows information exchange between query and support set molecules. Altae-Tran et al. (2017) uses an attention- enhanced LSTM variant, which updates the query and the support set molecule representations in an iterative fashion being aware of each other. We further develop this idea and combine with the idea of using transformer encoder layer (Vaswani et al., 2017) as cross-attention module (Hou et al., 2019; Chen et al., 2021).
The cross- coss-attention module updates the query molecule representation m' and the support set molecule representations X' by mutually exchanging information, using the usual Transformer mechanism:
(m". x Hopfield(mm', X'), [m', X')),
where [m', ['] E (N+x)++1) is the concatenation of the representations of the query molecule m with the support set molecules X' and we exploited that the Transformer is special case of the Hopfield layer. Again, normalization is applied (Ba et al., 2016) and multiple Hopfield layers i.e.. heads can run in parallel, be stacked, and equipped with coineconnecso The representations m" and X" are passed to the similarity module.
3.4 SIMILARITY MODULE (SM)
In this module, pairwise similarity values k(m"" are computed between the representation of a query molecule m" " and each molecule æf the support set as done recently (Koch et al., 2015; Altae-Tran et al., 2017). Based on these similarity values, the activity for the query molecule is predicted, building a weighted mean over the support set labels:
N
1 =
Σ Nh k(m" ,."))
(10)
N
12
where our architecture employs dot product similarity of normalized representations k(m", [(mm"="")
=
m" n'l' x" (.) is the sigmoid function and is hyperparameter. Note that we use balancing
strategy for the labels v/a N/(V2NA) if Yn where N, is the number of actives and N, -N/(V2N))
else
the number of inactives of the support set.
3.5 ARCHITECTURE, HYPERPARAMETT SELECTION, AND TRAINING DETAILS
Hyperparameters. The main hyperparameters of our architecture are the number of heads, the embedding dimension, the dimension of the association space of the CAM and CM, the learning
5

---

Published as a conference paper at ICLR 2023
rate schedule, the scaling parameter B. and the molecule encoder. The following hyperparameters were selected by manual hyperparameter selection on the validation tasks. The molecule encoder consists of single layer with output size d 1024 and SELU activation (Klambauer et al., 2017). The CM consists of one Hopfield layer with 8 heads. The dimension e of the association space is set to 512 and B 1/ve. Since use skip connections between all modules the output dimension of the CM and CAM matches the input dimension. The CAM comprises one layer with 8 heads and an association- space dimension of 1088. For the input to the CAM, an activity encoding was added to the support set molecule representations to provide label information. The SM uses T = 32. For the context set, we randomly sample 5% from a large set of molecules i... the molecules in the FS-Mol training split for each batch. For inference, we used fixed set of 5% of training set molecules as the context set for each seed. We hypothesize that these choices about the context could be further improved (Section 6). We provide considered and selected hyperparameters: in Appendix A.
Loss function, regularization and optimization. We use the Adam optimizer (Kingma and Ba, 2014) to minimize the cross-entropy loss between the predicted and known activity labels. We use a learning rate scheduler which includes a warm up phase, followed by a section with a constant learning rate, which is 0.0001, and third phase in which the learning rate steadily decreases. As regularization strategy, for the CM and the CAM a dropout rate of 0.5 is used. The molecule encoder has dropout with rate 0.1 for the input and 0.5 elsewhere (see also Appendix A.
Compute time and resources. Training a single MHNIs model on the benchmarking dataset FS- Mol takes roughly 90 hours of wall-clock time A100 GPU. In total, roughly 15,000 GPU hours were consumed for this work.
4 RELATED WORK
Several approaches to few-shot learning in drug discovery have been suggested (Altae-Tree et al., 2017; Nguyen et al., 2020; Guo et al., 2021; Wang et al., 2021). I). Nguyen et al. (2020) evaluated the applicability of MAML and its variants to graph neural networks (GNNs) and (Guo et al., 2021) also combine GNNs and meta-learning. Altac- Tran-Tra&& et al. (2017) suggested an approach called Iterative Refinement Long Short-Term Memory, in which query and support set embeddings can share information and update their embeddings. Property-awan relation networks (PAR) (Wang al., 2021) an attention mechanism enrich representations from cluster centers and then learn a relation graph between molecules. Chen et al. (2022) propose to adaptively learn kernels and apply their method few-shot drug discovery with predictive performance for larger support set sizes. Recently, Stanley et al. (2021) generated a benchmark dataset for few-shot learning methods in drug discovery and provided some baseline results.
Many successful deep neural network architectures use external memories, such as the neural Turing machine (Graves et al., 2014), memory networks (Weston al., 2014), end-to-end memory networks (Sukhbaatar et al., 2015). Recently, the connection between continuous modern Hopfield networks (Ramsauer et al., 2021), which are content-addressall architectures (Vaswani et al., 2017) has been established. We refer to Le (2021) for an extensive overview of memory-based architectures. Architectures with external memories have also been used for meta-learning (Vinyals et al., 2016; Santoro et al., 2016) and few-shot learning (Munkhdalai and Yu, 2017: Ramalho and Garnelo, 2018; Ma et al., 2021).
associative memories, and Transformer
5
EXPERIMENTS
5.1 BENCHMARKING ON FS-MOL.
Experimental setup. Recently, the dataset FS-Mol (Stanley et al., 2021) was proposed benchmark few-shot learning methods in drug discovery. It was extracted from ChEMBL27 and comprises in total 489, 133 measurements, 233,786 compounds and 5,120 tasks. Per task, the mean number of data points is 94. The dataset is well balanced as the mean ratio of active and inactive molecules is close to 1. The FS-Mol benchmark dataset defines 4,938 training, 40 validation and 157 test tasks, guaranteeing disjoint task sets. Stanley et al. (2021) precomputed extended connectivity fingerprints (ECFP) (Rogers and Hahn, 2010) and key molecular physical descriptors, which were defined by RDKit (I (Landrum al., 2006). While methods would be allowed to use other representations of

---

natural exten- Published as a conference paper at ICLR 2023
Table I: Results on FS-MOL [AAUC-PR]. The best method is marked bold. Error bars represent standard errors across tasks according to Stanley et al. (2021). The metrics are also averaged across five training reruns and ten draws of support sets. In brackets the number of tasks per category is reported.
Method
All [157]
Kin. [125]
Hydrol. [20]
Oxid.[7]
GNN-ST (Stanley et al., 2021)
.029 .004
.027 = .004
.040 + .018
.020 + ,016
MAT (Maziarka et al., 2020)
.052 .005
.043 = .005
.095 .019
.062 + .024
Random Forest" (Breiman, 2001)
.092 = .007
.081 = .009
.158 .028
.080 .029
GNN-MT (Stanley al., 2021)
.093 .066
.093 + .006
.108 .025
.053 + .018
Similarity Search
.118 = .008
.109 + .008
.166 .029
.097 + .033
GNN-MAML" (Stanley et al., 2021)
.19 .009
.177 = .009
.105 .024
.054 = .028
PAR (Wang et al., 2021)
.164 = .008
.182 = .009
.109 - .020
.039 .008
Frequent Hitters
.12 .010
207 = .009
.098 .009
.041 .005
ProtoNet (Snell et al., 2017)
207 = .008
.255 = .009
.299 .030
.095 + .029
Siamese Networks (Koch et al., 2015)
223 .010
241 = .010
.178 778 .026
.082 + .025
IterReILSTM (Altae-Tran et al., 2017)
234 .110
.21 - .010
.199 .026
.098 + .027
ADKF-IFT (Chen et al., 2022)
234 .009
.28
1 .200
.217 + .017
.106 + .008
.HNffs (ours)
.241 .009
259
.010
.199 .027
.096 + .019
a metrics from Stanley et al. (2021).
results from Chen et al. (2022).
the input molecules, such as the molecular graph, we used a concatenation of these ECFPs and RDKit-based descriptors. We use the main benchmark setting of FS-Mol with support set size 16, which is close the 5- and 10-shot settings in computer vision, and stratified random split (Stanley et al., 2021, Table 2) for a fair method comparison (see also Section A.5).
Methods compared. Baselines for few-shot learning and our proposed method MHNIfs were com- pared against each other. The Frequent Hitters model is naive baseline that ignores the provided support set and therefore has learn to predict the average activity of a molecule. This method can potentially discriminate so-called frequent-hitte molecules (Stork et al., 2019) against molecules that are inactive across many tasks. We also added Similarity Search (Cereto-Massagul et al., 2015) as a baseline. Similarity search is a standard chemoinformatics technique, used in situations with single or few known actives. In the simplest case, the search finds similar molecules by computing of the molecules and using a similarity measure k(.,.)
fingerprint or bescriptoo-representtee such as Tanimoto Similarity (Tanimoto, 1960). Thus, Similarity Search, as used in chemoinfor- matics, can be formally written as ÿ 1/N En Yn k(m, æ,), where Z1...., Tn come from a fixed molecule encoder, such as chemical fingerprint or descriptor calculation. sion of Similarity Search with fixed chemical descriptors is Neural Similarity Search or Siamese networks (Koch al., 2015), which extend the classic similarity search by learning a molecule encoder: ú (--1 Em= V/2 (M))) I (ax)) Furthermore, re-implemented the IterRefLSTM (Altae-Tran et al., 2017) in PyTorch. The IterRefLSTM model consists of three modules. First, molecule encoder maps the query and support set molecules to its representations m and X. Second, an attention-enhann enhanced LSTM variant, the actual IeeRReLL iteratively updates the query and support set molecules, enabling information sharing between the molecules: (m', X'| TterReeT.STV M ([m, X)), where the hyperparameter T. controls the number of iteration Third, a similarity module computes attention weights based on the rep- softmax (k (m', X')). These representations are then used for the final prediction: 9 -= シー a:y. For further details, see Appendix A.1.5, The Random Forest bascline uses the chemical descriptors and is trained in standard supervised manner on the support set molecules for each task. The method GNN-ST is a graph neural network (Stanley et al., 2021; Gilmer al., 2017) that is trained from scratch for each task. The GNN-MT uses two step strategy: First, the model is pretrained on a large dataset on related tasks; second, an output layer is constructed to the few-shot task via linear probing (Stanley et al., 2021; Alain and Bengio, 2016). The Molecule Attention Transformer (MAT) is pre- trained in a sudervuueee fashion and fine- tne-tuning is performed for the few-shot task (Maziarka et al., 2020). GNN-MAML is based MAML (Finn et al., 2017), and uses a model-agnosti meta-learning strategy to find a general core model from which one can easily adapt to single tasks. Notably, GNN-MAMI. also can be seen proxy for Meta-MGNN (Guo
steps of the IterReiL resentations:
N
7

---

Published as a conference paper at ICLR 2023
0.250
0.745
0.245
MAUCPR
0.240
0.735
AUC
0.235
0.730
0.230
0.725
0.225
0.720
MHNTS (CM+CAM-SM)
MHNTS
MHNN -CM
#ICAM.IterRelLSTM)
SCM
RICAM.IeeRReLSSM
Figure 2: Results of the ablation study, The boxes show the median, mean and the variability of the average predictive performance of the methods across training reruns and draws of support sets. The performance significantly drops when the context module removed (light red bars), and when additionally the wross-attention module replaced with the IterRefl.ST .TT module (light blue bars). This indicates that our two newly introduced modules, CM and CAM, play a crucial role in MHNfs.
al., 2021), which enriches the gradient update step in the outer loop of the MAML-framework, by an attention mechanism and uses additional atom type prediction loss and a bond reconstruction loss. ProtoNet (Snell al., 2017) includes molecule encoder, which maps query and support set molecules to representations in embedding space. In this embedding space, prototypical representations of each class are built by taking the mean across related support set molecules for each class (details in Appendix 1.1.4) The PAR model ( (Wang et al., 2021) includes GNN which creates initial molecule embeddings. These molecule embeddings are then enriched by an attention mechanism. Finally, another GNN learns relations between support and query set molecules. The PAR model has shown good results for datasets which just include very few tasks such as Tox21 (Wang al., 2021). Chen al. (2022) suggest framework for learning deep kernels by interpolating between meta-learning and conventional deep kernels, which results in the ADKF-IK model. The model has exhibited especially high performance for large support set sizes. For all methods the most important hyperparameten were adjusted on the validation tasks of FS-Mol.
Training and evaluation. For the model implementations, we used PyTorch (Paszke al., 2019, BSD license). We used PyTorch Lightning (Falcon et al., 2019, Apache 2.0 license) as a framework for training and logic, hydra for config file handling (Yadan, 2019, Apache 2.0 license) and Weights & Biases (Biewald, 2020, MIT license) as experiment tracking tool. We performed five training reruns with different seeds for all methods, except Classic Similarity Search as there is no variability across seeds. Each model was evaluated ten times by drawing support sets with ten different seeds.
Results. The results in terms of area under precision-recall curve (AUC-PR) are presented in Table 1, where the difference to a random classifier is reported (AAUC-PR). The standard error reported across tasks. Surprisingly, the naive baseline Frequent Hitters, that neglects the support set, has outperformed most of the few-shot learning methods, except for the embedding based methods and ADKF-IFT. MHNfs has outperformed all other methods with respect AAUC-PR across all tasks, including the IterRefLSTM model (p-value 1.72e-7, paired Wilcoxon test), the ADKF-IFT model (p-value <1,0e-8, Wilcoxon test), and the PAR model (p-value <1.0e-8, paired Wilcoxon test).
5.2 ABLATION STUDY
MHNfs has two new main components compared to the most similar previous state-of-the-ee -rrt method HerRefLSTMM i) the context module, and ii) the cross- cross-attention module which replaces the LSTM- like module. To assess the effects of these components, we performed an ablation study. Therefore, we compared MHNfs to method that does not have the context module ("MHNfs -CM") and to method that does not have the context module and uses an LSTM-like module instead the CAM ("MHNfs For the ablation study, used all 5 training reruns and evaluated 10 times on the test set with different support sets. The results of this ablation steps are presented in Figure 2. Both removing the CM and exchanging the CAM with the Ieer STM module were
-CM (CAM.MteeRRRLLLTTT
8

---

Published as a conference paper at ICLR 2023
Table 2: Results of the domain shift experiment on Tox21 [AUC, AAUC-PRJ. The best method is marked bold. Error bars represent standard deviation across training reruns and draws of support sets
Method
AUC
AAUC-PR
Similarity Search (baseline)
.629 + .015
.061 = .008
IterRefLSTM (Altae-Tran et al., 2017)
.664 + .088
.067 = .008
MHNfs (ours)
.679 = .018
.073 = .008
detrimental for the performance of the method (p-value 0.002 and 1.72e-7, respectively; paired Wilcoxon test). The difference was even more pronounced under domain shift (see Appendix A.3.3). Appendix A.3.2 contains second ablation study that examines the overall effects of the context, the cross-attention, the similarity module, and the molecule encoder of MHNfs.
5.3 DOMAIN SHIFT EXPERIMENT
Experimental setup. The Tox21 dataset consists of 12,707 chemical compounds, for which mea- surements for up to 12 different toxic effects are reported (Mayr et al., 2016; Huang et al., 201 6a). was published with fixed training, validation and test split. State-of-the-ard supervised learning methods that have access to the full training set reach AUC performance values between 0.845 and 0.871 (Klambauer et al., 2017: Duvenaud et al., 2015: Li al., 2017: 2021; Zaslavskiy al., 2019; Alperstein et al., 2019). For our evaluation, we re-cast Tox21 as few-shot learning setting and draw small support sets from the tasks. The compared methods were pre-trained on FS-Mol and obtain small support sets from Tox21. Based on the support sets, the methods had to predict the activities of the Tox21 test set. Note that there strong domain shift from drug-like molecules of FS-Mol to environmental chemicals, pesticides, food additives of 1ox21. The domain shift also concerns the outputs where a shift from kinases, hydrolases, and oxidoreductases of FS-Mol to nuclear receptors and stress responses of Tox21 is present.
Methods compared. We compared MHNIs, the runner-up method IterRefLSTM, and Similarity Search-sinee it has been widely used for such purposes for decades (Cereto-Massagué et al., 2015).
Training and evaluation. We followed the procedure of Stanley et al. (2021) for data-cleaning. preprocessing and extraction of the fingerprints and descriptors used in FS-Mol. After running the cleanup step, 8,423 molecules remained for the domain shift experiments. From the training set, 8 active and 8 inactive molecules per task were randomly selected to build the support set. The test molecules were used as query molecules. The validation set molecules were not used at all. During test-time, support set was drawn ten times for each task. Then, the performance the models were evaluated for these support sets, using the area under precision-rece reccal curve (AUC-PR), analogously to random classifier (AAUC-PR).
the FS-Mol benchmarking experiment reported as the difference and the area under receiver operating characteristic curve (AUC) metrics. The performance values report the mean over all combinations regarding the training reruns and the support set sampling iterations. Error bars indicate the standard deviation.
Results. The Hopfield-based context retrieval method has significantly outperformed the IteRRefLSTM-batt model (PAAuC-PR-value 5. Pauc-value 2.5c-6, paired Wilcoxon test) and the Classic Similarity Search (PAAUC-PR-vallle 2.4c-9, vauc-value 7.6c-10, paired Wilcoxon test) and therefore showed robust performance on the toxicity domain, see Table 2. Notably, all models were trained on the FS-Mol dataset and then applied to the Tox21 dataset without adjusting any weight parameter.
CONCLUSION
We have introduced new architecture for few-shot learning in drug discovery, namely MHNfs, that based on the novel concept to enrich molecule representations with context. In a benchmarking experiment the architecture outperformed all other methods and in a domain shift study the robustness and transferability has been assessed. We envision that the context module can be applied to many different areas, enriching learned representations analogously our work. For discussion, A.9.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
The ELLIS Unit Linz, the LIT AT Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. IARAI is supported by Here Technologies. We thank Merck Healthcare KGaA for the collaboration. Further, we thank the projects AI-MOTION LLTT-2018-6-1 --UUU02-11))
DeepFlood (LIT-2019-8 8-UOU-213)) Medical Cognitive Computing Center (MC3), INCONTROL-RO (FFG-88 1064). PRIMAL (FFG-873979) S3AI (FFG-872172) DL for GranularFlow (FFG-871300 EPILEPSIA (FFG-892171), AIRI FG 9-N (FWF-36284, FWF-36235), ELISE -12020-ICT-2019- ID:: 951847), Stars4Waters Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FIL.I Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Verbund AG, GLS (Univ. Waterloo) Software Competence Center Hagenberg GmbH, TÜV Austria, Frauscher Sensonic and the NVIDIA Corporation.
We thank Audi.JKU Deep
REFERENCES
Adler, T., Brandstetter, J., Widrich, M., Mayr, A., Kreil, D., Kopp, M., Klambauer, G., and Hochreiter, S. (2020). Cross-domain few-shot learning by representation fusion. arXiv preprint 06940990
Alain, G. and Bengio, (2016). Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:/610.0164.
A Alperstein, 7., Cherkasov, A., and Rolfe, J. T. (2019). All smiles variational autoencoder. arXiv preprint arXiv:1905. 13343.
Altae-Tran, H., Ramsundar, B., Pappu, A. S., and Pande, (2017). Low data drug discovery with one-shot learning. ACS central science, 3(4):
Antoniou, A. and Storkey, A. (2019). Assume, augment and learn: Unsupervised few-shot meta- learning via random labels and data augmentation. arXiv preprint arXiv:1902.0988;
Arrowsmith, J. (2011). Phase ii failures: 2008-2010. Nature reviews drug discovery, 10(5).
Axelrod, S. and Gomez-Bombarelli, R. (2022). Geom, energy-annotated molecular conformations for property prediction and molecular generation. Scientific Data, 911):1114.
Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06455
Bender, A., Mussa, H. Y., Glen, R. C., and Reiling, (2004). Similarity searching of chemical
databases using atom environment descriptors (molprint 2d): evaluation of performance. Journal
of chemical information and computer sciences, 708-11788
Bendre, N., Marín, H. T., and Najafirad, (2020). Learning from few samples: A survey. arXiv preprint 1rXii:2007.1548.
Bengio, Y., Bengio, S., and Cloutier, J. (1991). Learning synaptic learning rule. In Seattle international joint conference on neural networks.
Biewald, I.. (2020). Experiment tracking with weights and biases. Software available from wandh.com.
Bonner, M. F. and Epstein, R. A. (2021). Object representations in the human brain reflect the co-occurrence statistics of vision and language. Nature Communications, 12(4081).
Breiman, L. (2001). Random forests. Machine learning. 45(1):5-32.
Cereto-Massagué, A., Ojeda, M. J., Valls, C., Mulero, M., Garcia-Vallvé, S., and Pujadas, G. (2015). Molecular fingerprint similarity search virtual screening. Methods, 71:58-63.
Chen, H., Engkvist, O., Wang, Y., Olivecrona, M., and Blaschke, T. (2018). The rise of deep learning in drug discovery. Drug discovery today, 121--22..
10

---

Published as a conference paper at ICLR 2023
Chen, H., Li, H., Li, Y., and Chen, C. (2021.. Sparse spatial transformers for few-shot learning. arXiv preprint arXiv:2109. 12932.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020). A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597- 1607. PMLR.
Chen, W. Tripp, A., and Hemández-Lobato, J. M. (2022). Meta-learning adaptive deep kernel gaussian processes for molecular property prediction. arXiv preprint 220.000000...708
Chollet, F. (2019). On the measure of intelligence. arXiv preprint arXiv: 1.01547.
Dahl, G. E., Jaitly, N., and Salakhutdinov, R. (2014). Multi-task neural networks for qsar predictions. arXiv preprint arXiv: 1231.
Duvenaud, D., Maclaurin, D., gguilera-Iparrgguirre. J., Gómez-Bombarelli, R., Hirzel, T., Aspuru- Guzik, A., and Adams, R. P. (2015). Convolutional networks on graphs for learning molecular fingerprints. arXiv preprint arXiv:1509.0929:
Eckert, H. and Bajorath, J. (2007). Molecular similarity analysis in virtual screening: foundations, limitations and novel approaches. Drug discovery today, 12(5-6):225-233,
Falcon, W. al. (2019). Pytorch lightning.
GitHuh.
Note: https://github
htmp:yyyoorkkiignnnnng//rrrhhhhh 3:6.
Finn, C.. Abbeel, P., and Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135, PMLR.
Fürst, A., Rumetshofer, E., Tran, V., Ramsauer, H., Tang, F., Lehner, J., Kreil, D., Kopp, M., Klambauer, G., Bitto-Nemling, A., al. (2022). Cloob: Modern hopfield networks with infoloob outperform clip, Advances in neural information processing systems.
Geppert, H., Horváth, T., Gärtner, T., Wrobel, S., and Bajorath, J. (2008). Support--ector-maahte based ranking significantly improves the effectiveness of similarity searching using fingerprints and multiple reference compounds. Journal of chemical information and modeling, 48(4):742-746.
Gilmer, J., Schoenholz, S. S., Riley, F., Vinyals, O., and Dahl, G. E. (2017). Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272. PMLR.
Gomez, L. (2018). Decision making in medicinal chemistry: The power of our intuition. ACS Medicinal Chemistry Letters, 9(10):56-958.
-.mez-Bombarelli, R., Wei, J. N., Duvenaud, D., Hernández-Lobato, J. M., Sánchez-Lengeling, J., Hirzel, T. D., Adams, P., and Aspuru-Guzi Guzik, A. (2018). Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268-276.
B., Sheberla, D., aguilera-Iparragun
Graves, A., Wayne, G., and Danihelka, I. (2014). Neural turing machines. AXXi:1410.5401.
arXiv preprint
Guo, Z., Zhang, C., Yu, W., Herr, J., Wiest, O., Jiang, M., and Chawla, N. V. (2021). Few-shot graph learning for molecular property prediction. In Proceedings of the web conference 2021, pages 2559-2567.
He, J., You, H., Sandström, E., Nittinger, E., Bjerrum, E. J., Tyrchan, C., Czechtizky, W., and Engkvist, O. (2021). Molecular optimization by capturing chemist's intuition using deep neural networks. Journal of cheminformatics, 13(1):1-17.
Hertz, T., Hillel, A. B., and Weinshall, D. (2006). Learning kernel function for classification with small training samples. In Proceedings of the 23rd international conference on machine learning, pages 401-408.
Hochreiter, S. (2022). Toward broad ai. Communications of the ACM, 65(4):56-57

---

Published as a conference paper at ICLR 2023
Hochreiter, S., Klambauer, G., and Rarey, M. (2018). Machine learning in drug discovery. Journal of Chemical Information and Modeling, 58(9): 1724.
Hochreiter, S., Younger, A. S., and Conwell, P. R. (2001). Learning to learn using gradient descent.
In International conference on artificial neural networks, pages 87-94. Springer.
Hou, R., Chang, H., Ma, B., Shan, S., and Chen, X. (2019). Cross attention network for few-shot classification. Advances in neural information processing systems 32.
Huang, R., Xia, M., Nguyen, D.-T., Zhao, T., Sakamuru, S., Zhao, J., Shahane, S. A., Rossoshek, A., and Simeonov, A. (2016a). Tox2 1ox2Ichallee challenge to build predictive models of nuclear receptor and stress response pathways as mediated by exposure to environmental chemicals and drugs. Frontiers in Environmental Science, 3:85.
Huang, R., Xia, M., Sakamuru, S., Zhao, J., Shahane, S. A., Attene-Ramos, M., Zhao, T., Austin, C. P., and Simeonov, A. (b016b). Modelling the tox21 10 chemical profiles for in vivo toxicity prediction and mechanism characterization. Nature communications, 7(1):1-10.
Jiang, D., Wu, Z., Hsieh, C.-Y., Chen, G., Liao, B., Wang, Z., Shen, C., Cao, D., Wu, J., and Hou, T. (2021). Could graph neural networks learn better molecular representation for drug discovery? comparison study of esscripto-basse and graph- based models. Journal of cheminformatics, 13(1):1-23.
Kearnes, S., McCloskey, K., Berndl, M., Pande, V., and Riley, P. (2016). Molecular graph convolu- tions: moving beyond fingerprints. Journal of computer-aide aided molecular design, 30(8):595-60
Kingma, D. P. and Ba, J. (2014), Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.690.
Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S. (2017). Self-normalizing neural networks In Advances neural information processing systems 30, pages 972-981.
Koch, G., Zemel, R., Salakhutdinov, R., et al. (2015). Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume Lille.
Kuhn, M., Letunic, I., Jensen, L. J., and Bork, (2016). The sider database drugs and side effects. Nucleic acids research, 4(D1))D1075-D107))
Landrum, G. et (2006). Rdkit: Open-source cheminformatics.
Le, H. (2021). Memory and attention in deep learning. arXiv preprint arXiv:2107.01390.
Li, J.. Cai, D... and He, X. (2017). Learning graph-level representation for drug discovery. arXiv preprint arXiv:1709.03:: 1709.0371.
Li, P., Li, Y., Hsieh, C.-Y., Zhang, S., Liu, X., Liu, H., Song, S., and Yao, X. (2021). Trimnet: learning molecular representation from triplet messages for biomedicine. Briefings in Bioinformatics, 22(4):bbaa266.
Ma, Y., Liu, W., Bai, S., Zhang, Q., Liu, A., Chen, W., and Liu, X. (2021). Few-shot visual learning with contextual memory and fine-grained calibration. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 811-817.
Mayr, A., Klambauer, G., Unterthiner, T., and Hochreiter, S. (2016). Deeptox: toxicity prediction using deep learning. Frontiers in environmental science, 3:80.
Mayr, A., Klambauer, G., Unterthiner, T., Steijaert, M., Wegner, J. K., Ceulemans, H., Clevert, D.-A., and Hochreiter, S. (2018). Large-scale comparison of machine learning methods for drug target prediction on chembl. Chemical science, :(44)5544-
Maziarka, L., Danel, T., Mucha, S., Rataj, K., Tabor. J., and Jastrzębski, S. (2020). Molecule attention transformer. arXiv preprint 200.0200002.02244..
12

---

Published as a conference paper at ICLR 2023
Merk, D., Friedrich, L., Grisoni, F., and Schneider, G. (2018). De novo design of bioactive small molecules by artificial intelligence. Molecular informatics. 37(1-2):1700153.
Merkwirth, C. and Lengauer, T. (2005). Automatic generation of complementary deseriptors with molecular graph networks. Journal of chemical information and modeling, 45(5):1 119-11688 168.
Miller, E. G., Matsakis, N. E., and Viola, P. A. (2000). Learning from one example through shared densities on transforms. In Proceedings ieee conference on computer vision and pattern recognition. cvpr 2000 (cat. no. PR00662), volume 1, pages 464-471.
Munkhdalai, T. and Yu, H. (2017). Meta networks. In International Conference Machine Learning, pages 2554-2563. PMLR.
Nguyen, C. Q. Kreatsoulas, C., and Branson, K. M. (2020). Meta-lcarning gnn initializations for low-resource molecular property prediction. arXiv preprint arXiv:2003.0500 05996.
Oord, A. v. Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv preprint arXiv:
Paszke, A., Gross, S.. Chintala, S.. Chanan, G.. Yang, E.. Vito, Z., Lin, Z., Desmaison, A., Antiga,
L.,, and Lerer, A. (2019). Automatic differentiation in pytorch. Conference on neural information processing systems.
Potter, M. (2012). Conceptual short term memory in perception and thought. Frontiers in Psychology, 3:113.
Ramalho, T. and Garnelo, M. (2018). Adaptive posterior learning: few-shot learning with a surprise- based memory module. In International Conference on Learning Representations.
Ramsauer, H., Schätl, B., Lehner, J., Seidl, P., Widrich, M., Gruber, L., Holzleitner, M., Adler, T., Kreil, D., Kopp, M. K., Klambauer, G., Brandstetter, J., and Hochreiter, S. (2021). Hopfield networks is all you need. In International conference learning representations.
Riniker, S. and Landrum, G. A. (2013). Open-source platform benchmark fingerprints for ligand- based virtual screening. Journal of cheminformatics, 5(1):1-17.
Rogers, D. and Hahn, M. (2010). Extendeed-connectvitty fingerprints. Journal of chemical information and modeling, 50(5):742-754.
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., and Lillicrap, T. (2016). Meta-learning with
memory-augmented neural networks. In International conference on machine learning, pages 1850. PMLR.
Schmidhuber, J. (1987). Evolutionary principles in self-referential learning.
Schneider. Walters, W. P., Plowright, A. T., Sieroka. N., Listgarten, J., Goodnow, A., Fisher, J., Jansen, J. M., Duca, J. S., Rush, T. S., et al. (2020). Rethinking drug design in the artificial intelligence era. Nature reviews drug discovery, 19(5):353-364
Segler, M. H., Kogej, T., Tyrcham, C., and Waller, M. (2018a). Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS central science, 4(1): (20):1200133 1211
Segler, M. H., Preuss, M., and Waller, M. P. (2018b). Planning chemical syntheses with deep neural networks and symbolic ai. Nature, 555(7699):604-611
Seidl, P., Renz, P., Dyubankova, N., Neves, P., Verhoeven, J., Wegner, K., Segler, M., Hochreiter, S., and Klambauer, G. (2022). Improving few-and zero-shot reaction template prediction using modern hopfield networks. Journal of chemical information and modeling, 62(9):2111-212)
Sheridan, R. P. and Kearsley, S. K. (2002). Why do we need so many chemical similarity search methods? Drug discovery today, 7(17):903911.
Simm, J., Klambauer, G., Arany, A., Steijaert, M., Wegner, J. K., Gustin, E.., Chupakhin, V., Chong, Y. T., Vialard, J., Buijnsters, P., et al. (2018). Repurposed tigh-throughput image assays enables biological activity prediction for drug discovery. Cell Chemical Biology, page 108399.
13

---

Published as a conference paper at ICLR 2023
Snell, J., Swersky, K., and Zemel, R. S. (2017). Prototypical networks for few-shot learning. arXiv preprint arXiv:1703.05111
Stanley, M., Bronskill, J. F., Maziarz, K., Misztela, H., Lanini, J., Segler, M., Schneider, N., and Brockschmidt, M. (2021). Fs-mol: A few-shot learning dataset of molecules. In Conference on neural information processing systems workshop.
Stork, C., Chen, Y., Sicho, M., and Kirchmair, J. (2019). Hit dexter 2.0: machine-learinng models for the prediction of frequent hitters. Journal of chemical information and modeling, 59(3):1030-1043,
Sturm, N., Mayr, A., Le Van, T., Chupakhin, V., Ceulemans, H., Wegner, J., Golib-Dzib, J.-F., Jeliazkova, N., Vandriessche, Y., Böhm, S., et al. (2020). Industry-scale application and evaluation of deep learning for drug target prediction. Journal of Cheminformatiis, 12(1):1-13.
Sukhbaatar, S., Weston, J., Fergus, R., et al. (2015). End-to-end memory networks. Advances in neural information processing systems, 28.
Sun, J., Jeliazkova, N., Chupakhin, V., Golib-Dzib, J.-F., Engkvist, O., Carlsson, L.., Wegner, J., Ceulemans. H., Georgiev. I., Jeliazkov, V., et al. (2017). Excape-db: an integrated large scale dataset facilitating big data analysis chemogenomics. Journal of cheminformatics, 9(1):1-9.
Tanimoto, T. (1960). Ibm type 704 medical diagnosis program. IRE transactions on medical electronics, (4):280-283.
Torres, L., Monteiro, N., Oliveira, J., Arrais, J., and Ribeiro, B. (2020). Exploring siamese neural network architecture for one-shot drug discovery. In 2020 ieee 20th international conference on bioinformatics and bioengineering (bibe), pages 168-175.
Triantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Evei, U., Xu, K., Goroshin, R., Gelada, C.,, Swersky, K., Manzagol, P.-A., et al. (2019). Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint 1xXiv:1903.03090
Unterthiner, T., Mayr, A., Klambauer, G., Steijaert, M., Wegner, K., Ceulemans, H., and Hochreiter, S. (2014). Deep learning as an opportunity in virtual screening. In Advances in neural information processing systems workshop.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al. (2016). Matching networks for one shot learning, Advances neural information processing systems, 29:3630-3633
Walters, W. P. and Barzilay, (2021). Critical assessment of ai in drug discovery. Expert opinion on drug discovery, pages 1-11.
Wang, X., Huan, J.. Smalter, A., and Lushington, G. H. (2010). Application of kemel functions for accurate similarity search large chemical databases. In BMC bioinformatics, volume 11, pages 1-14. BioMed Central.
Wang, Y., Abuduweili, A., Yao, Q. and Dou, D. (2021). Property-aware relation networks for few-shot molecular property prediction. Advances in Neural Information Processing Systems, 34:17441-17454.
Wang, Y. Yao, Q., Kwok, T., and Ni, L. M. (2020). Generalizing from few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34.
Waring, M. J., Arrowsmith, J., Leach, A. R., Leeson, P. D., Mandrell, S., Owen, R. M., Pairaudeau, G., Pennie, D., Pickett, S. D., Wang, J., al. (2015). analysis of the attrition of drug candidates from four major pharmaceutical companies. Nature reviews drug discovery, 14(7):4554486
Weininger, D. (1988). Smiles, chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31- 36.
14

---

Published as a conference paper at ICLR 2023
Weston, J., Chopra, S., and Bordes, A. (2014). Memory networks. arXiv preprint arXiv: arXiv:1410.3916.
Widrich, M., Schäft, B., Pavlović, M., Ramsauer, H., Gruber, L.., Holzleitner, M., Brandstetter, J., Sandve, G. K., Greiff, V., Hochreiter, S., et al. (2020). Modern hopfield networks and attention for immune repertoire classification. In Advances in neural information processing systems 33.
Willett, P. (2014). The calculation of molecular structural similarity: principles and practice. Molecu- lar informatics, 33(6-7):403-411
Winter, R., Montanari, F., Noé, F., and Clevert, D.-A. (2019). Learning continuous and data-driven molecular descriptors by translating equivalent chemical representations. Chemical science, 1066:1692-1701.
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., and Pande. V. (2018). Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):513-530.
Xia, J., Zhu, Y., Du, Y., Liu, Y., and Li, Z. (2022). A systematic survey of molecular pre-trained models. arXiv preprint arXiv:2210.16486
Yadan, O. (2019). Hydra framework for elegantly configuring complex applications. Github. Visited 2022-04-25.
Yang. K., Swanson, K., Jin, W., Coley, C., Eiden, P., Gao, H., Guzman-Perez, A., Hopper, T., Kelley, B., Mathea, M., al. (2019). Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling, 59(3):337-3388.
Ye, M. and Guo, Y. (2018). Deep triplet ranking networks for one-shot recognition. arXiv preprint 1rX.v.1804.07275.
Zaslavskiy, M., Jégou, S., Tramel, E. W., and Wainrib, G. (2019). Toxieblend: Virtual screening of toxic compounds with ensemble predictors. Computational Toxicology, 10:81-88.
Zhao, A., Balakrishnan, G., Durand, F,, Guttag, J. V., and Dalca, A. V. (2019). Data augmentation using learned transformations for one-shot medical image segmentation. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 8543-8553.
15

---

A
APPENDIX
Contents of the appendix
Appendix
16
A.I
Details on methods
16
A.1.1
Frequent hitters: details and hyperparameters
16
A.1.2
Classic similarity search: details and hyperparameters
17
A.1.3
Neural Similarity Search or Siamese networks: details and hyperparameters
18
A.1.4
ProtoNet: details and hyperparameters.....
18
A.1.5
IterRefLSTM: details and hyperparameters
19
A.1.6
MHNff: details and hyperparameters
20
A.1.7 PAR: details and hyperparameters
23
A.2
Details on the FS-Mol benchmarking experiment
23
A.3
Details on the ablation study
24
A.3.1 Ablation study comparison against IterRefL.STM
24
A.3.2
Ablation study B: all design elements
25
A.3.3 Ablation study C: Under domain shift on Tox21
25
A.4
Details on the domain shift experiments
25
A.5
Generalization different support set sizes
26
A.6
Generalization to different context sets
26
A.7
Details and insights on the context module
28
A.8
Reinforcing the covariance structure in the data using modern Hopfield networks
28
A.9
Discussion, limitations and broader inpact
30
A.1 DETAILS ON METHODS
Few-shot learning methods in drug discovery be described models with adaptive parameters that use a support set Z = { V1). (ÆN, NN)} as additional input to predict a label for molecule
i Iw (m, Z).
(A1)
Optimization-basee methods, such as MAML (Finn al., 2017), use the support set update the parameters w
= Iu(w: (m),
(A2)
where a(.) is function that adapts of g based Z for example via gradient-descent.
Embedding based methods use different approach and learn representations the support set molecules {æ1,.... (EN) sometimes written as stacked embeddings X R and the query molecule m, and some function that associates these two types of information with each other. We methods Similarity Search in Section A.1.2, Neural Similarity Search in Section 1.1.3, ProtoNet in Section A. 1.4, IterReiLSTM in Section A. A.1.5, PAR in Section A. A.1.7, and MHNí's in the main paper and details in Section A. 1.6. The "frequent hitters" baseline described in Section A.I.I.
describe the embedding-
A.1.1 FREQUENT HITTERS: DETAILS AND HYPERPARAMETERS
The "frequent hitters" model FH is baseline that we implemented and included in the method comparison. This method uses the usual training scheme of sampling query molecule with label y. having access to support set Z. contrast to the usual models of the type gw (m, Z), the frequent hitters model PH neglects the support set:
FH i = g (m).
(A3)
We use Z to denote the support set of already embedded molecules to keep the notation uncluttered. More correctly, the methods have access to the raw support set 7 = [[x1, v1),.... (#N, uN)), where J,, is symholic, such the molccular graph, low-level representation of the molecule.
16

---

Published as a conference paper at ICLR 2023
Table AI: Hyperparameter space considered for the Frequent Hitters model. The hyperparameters of the best configuration are marked bold.
Hyperparameter
Explored values
Number of hidden layers
1, 2, 4
Number of units per hidden layer
1024, 4096
Output dimension
512, 1024 ReLU 0.0001, 0.001 Adam,
Activation function
Learning rate
Optimizer
Weight decay
0, 0.01
Batch size
32, 128, 512, 2048, 4096 0, 0.1
Input Dropout
Dropout
0.1, 0.2, 0.3, 0.4, 0.5 False, True False, True dot product
Layer-normalization
Affine
Similarity function
Thus, during training for the same molecule m, the model might have to predict both y 1 and y = 1, since the molecule can be active in one task and inactive in another task. Therefore, the model tends to predict average activity of molecule to minimize the cross-entropy loss. We chose an additive combination of the Morgan fingerprints, RDKit fingerprints, and MACCS keys for the input representation to the MLP.
Hlpeepaaraeeerr search. We performed manual hyperparameter search on the validation set and report the explored hyperparameter space (Table A1). We use early-stoppii based on validation average- preccs--r,,s,on, a patience of 3 epochs and train for maximum of 20 epochs with a linear warm-up learning-rate schedule for the first epochs.
A.1.2 CLASSIC SIMILARITY SEARCH: DETAILS AND HYPERPARAMETERS
Similarity Search (Cereto-Massagun situations in which a single or few actives are known. In the simplest case, molecules that are similar to a given active molecule are searched by computing fingerprint of the molecules and using a similarity measure .), such as Tanimoto Cemilarity(Tanimot 1960). Thus, the Similarity Search as used in chemoinformatics can be formally written as:
al., 2015) is classic chemoinformatics technique used in
descriptor-representtiio fdese (m)
1/N Σ Yn f (m), pdese (Ix)))
(A4)
where the function fdese maps the molecule to its chemical descriptors or fingerprints and takes the role of both the molecule encoder and the support set encoder. Then, an association function, consisting of a) the similarity measure k(.,.) and b) mean pooling across molecules weighted by their similarity and activity, is used to compute the predictions.
Notably, there are many variants of Similarity Search (Cereto-Massagué et al., 2015; Wang et al., 2010; Eckert and Bajorath, 2007; Geppert et al., 2008; Willett, 2014; Sheridan and Kearsley, 2002; Riniker and Landrum, 2013) of which some correspond to recent few-shot learning methods with fixed molecule encoder. For example, (Geppert al., 2008) suggest use centroid molecules, i.c. prototypes averages of active molecules. This is equivalent to the idea of Prototypical Networks (Snell et al., 2017). Riniker and Landrum (2013) are aware of different fusion strategies for sets of active inactive molecules, which corresponds to different pooling strategies of the support set. Overall, the variants the classic Similarity Search are highly similar to embedding based few-shot learning methods except that they have fixed instead of learned molecule encoder.
Hy Hyperparameter search. For the Similarity Search, there were two decisions to make which was firstly the similarity metric and secondly the question whether should use a balancing strategy
17

---

Published as a conference paper at ICLR 2023
query molecule
FCDNN
LayerNorm
similarity function
molecule encoder
shared
weights
embedding space
prediction
support set 6og Qa 0t coa
FCDNN
LayerNomm
target
association function
Figure A1: Schematic overview of the implemented Neural Similarity Search variant
like shown in Section 3.4. We decided for the dot- dot-product as a similarity metric and for using the balancing strategy. These decisions were made by evaluating the models on the validation set.
A.1.3 NEURAL SIMILARITY SEARCH OR SIAMESE NETWORKS: DETAILS AND HYPERPARAMETERS
If the fixed encoder fdesc of the Classic Similarity Search is replaced by learned encoders fMB. Neural Similarity Search variants naturally arise.
lot of related work already
done (Koch al., 2015; Hertz al., 2006; Ye and Guo, 2018;
Torres et al., 2020). We adapted these ideas, such that a fully-connected deep neural network followed by a Layer Normalization (Ba et al., 2016) operation, fMÉ with adaptive parameters w, is used in Siamese fashion to compute the embeddings for the input molecule and the support set molccules. Within an association function block, pairwise similarity values for the query molecule and each support set molecule are computed, associating both embeddings via the dot product. Based on these similarity values, the activity for the query molecule is predicted, building the weighted mean over the support set molecule labels:
N
(m)) pME ( (1z)) (MF((..))
i = o
Σ Un ==
(AS)
N
where (-) is the sigmoid function and J is a hyperparameter in the range of vd. Note that this
method uses a balancing strategy for the labels 1/4, N/(V/NN if =1 where NA is the -N/(22N)) else
number of actives and N₂ is the number of inactives of the support set. Figure Al provides an schematic overview of the Neural Similarity Search variant.
We trained the model using the Adam optimizer (Kingma and Ba, 2014) to minimize binary cross- entropy loss.
Hyperparameter search. We performed manual hyperparameter search on the validation set. We report the explored hyperparameter space (Table A2). Bold values indicate the selected hyperparame- ters for the final model.
A.1.4 PROTONET: DETAILS AND HYPERPARAMETERS
Prototypical Networks (ProtoNet) (Snell al., 2017) learn prototype r for each class. Concretely, the support set Z is class-wise separated into Z+ == ((x, y) E 7 y = 1} and Z- == ((x, y) E y == -1). For the subsets 7+ and 7 prototypical representations t and r- can be computed by
Σ (=.)
ME
(A6)
Z+
18

---

Published as a conference paper at ICLR 2023
Table A2:
Hyperparameter space considered for the Neural Similarity Search model selection. The hyperparameters of the best configuration are marked bold.
Hyperparameter
Explored values
Number of hidden layers
1, 2,
Number of units per hidden layer
1024, 4096
Output dimension
512, 1024
Activation function
ReLU, SELU
Learning
0.0001, 0.001, 0.01
Optimizer
Adam 1000
Weight decay
Batch size
4096
Input Dropout
0.1
Dropout
0.5
Layer-normalization
False, True False
Affine
Similarity function
cosine similarity, dot product. MinMax similarity
and
1
Σ
ME (x).
(A7)
7
(Z
The prototypical representations r used to make the final prediction:
E R.d and the query molecule embedding m Rdd are then
exp(-d(m,r")
(A8)
exp(-d(m,, + exp( (o0(--((m--)
where d is distance metric.
Hyperparameter search. Hyperparameter search has been done by Stanley et al. (2021), to which refer here. ECFP fingerprints and descriptors created by GNN, which operates the molecular graph, are fed into a fully connected neural network, which maps the input into an embedding space with the dimension of 512. Stanley et al. (2021) use the Mahalanobis dis- tance to measure the similarity between query molecule and the prototypical representations in the embedding space. The learning rate is 0.001 and the batch size is 256. The implementation can be found here https : h/githtub com homm/miccooofftcc /PS- hhlllb/ll/////// bolob/main/ Es httoan/fffoo//////aaaaaaaooooooe mol/ /tt/////MM//bbbbbb/ee
protonet_train py and important byperparameters are chosen here https: / thub. http:///wwo/bboo//aaa/a///////o/
Connection to Siamese networks and contrastive learning with InfoNCE. If instead of the neg- ative distance -d(.,.) the dot product similarity measure with appropriate scaling used, ProtoNet for two classes becomes equivalent to Siamese Networks. Note that our study, another difference that ProtoNet uses GNN for the encoder, whereas the encoder of the Siamese Networks is descriptor-based fully-connected network. In case of dot product as similarity measure, the objective also becomes equivalent to contrastive learning with the InfoNCE objective (Oord et al., 2018).
A.1.5 ITERREFLSTM: DETAILS AND HYPERPARAMETERS
Altae-Tran al. (2017) modified the idea of Matching Networks (Vinyals al., 2016) by replacing the LSTM with their Iterative Refinement Long Short-Tene Memory (IterRefLSTM) The use of the IterRefLSTM empowers the architecture to update only the embeddings for the query molecule but also adjust the representations of the support set molecules.
For the IterRefLSSTM model, query molecule embedding m <EE (m) and support set molecule embeddings I, = fM (x,,) are created using two potentially different molecule encoders for the query molecule m and the support set molecules ,i,--., E.. The query and support molecule
19

---

Published as a conference paper at ICLR 2023
Table A3: Hyperparameter space considered for the IteReeLLTM model selection. The hyperparameters of the best configuration are marked bold.
Hyperparameter
Explored values
Molecule encoder
Number hidden layers
0, 1, 2, 4 1024, 4096
Number of units per hidden layer
Output dimension
512, 1024
Activation function
ReLU, SELU
Input dropout
0.1
Dropout
0.5
IterRef embedding layer
L
1,3
Similarity module:
Metric
cosine similarity, dot product, MinMax similarity
Similarity space dimension
512, 1024 False, True False, True
Layer-normalization
Affine
Training
Learning rate
0.0001, 0.001, 0.01 Adam, Adam 0, 0.0001 2048, 4096
Optimizer
Weight decay
Batch size
embeddings are then updated by LSTM-like module the actual IterRefLSTM:
[m', X'I = IterRefL.STM,,m,,
Here, m' and X' contain the updated representations for the query molecule and the support set molecules. The IterRefLSTM denotes the function which updates these representations. The main property of the IterRefLSTM module is that it is permutation-qquivarial tation x (.) of the input elements results in the permutation of output elements: =(mm', "imn''X' X') IterRefL.SMM. (= (m, X))). Therefore, the full architecture is invariant to permutations of the support set elements. For details, we refer to Altae-Trae et al. (2017). The hyperparameter I, E N controls the number of iteration steps of the IteReeLLSTM.
thus permu-
The IterRefLSTM also includes a similarity module which computes the predictions based on the updated representations mentioned above:
a softmax (k (mm,X')
On Un.
where I is the prediction for the query molecule. For the computation of the attention values a, the softmax function is used. & is similarity metric, such as the cosine similarity.
Hyperparameter search. All hyperparameters were selected based on manual tuning on the validation set. We report the explored hyperparameter space in Table A3. Bold values indicate the selected hyperparameters for the final model.
A.1.6 MHNES: DETAILS AND HYPPRPPAAMETTER
The MHNIf consists of a molecule encoder, the context module, the cross-attention-moddde and similarity module. The molecule encoder is fully-connected Neural Network, consisting of one layer with 1024 units. For the context module, a Hopfield layer with 8 heads used and also the cross- attention module include 8 heads. We chose concatenation of ECFPs and RDKit-based descriptors as the inputs for the MHNfs model. Notably, the RDKit-based descriptors were pre- propprceesed in a way that instead of values quantils, which were computed by comparing
value with the
20

---

Published as a conference paper at ICLR 2023
Table A4: Hyperparameter space considered for the MHNIS model selection. The hyperparameters of the best configuration are marked bold.
Hyperparameter
Explored values
Molecule encoder
Number hidden layers
0, I, 2, 4
Number of units per hidden layer
1024, 4096
Output dimension
512, 1024
Activation function
ReLU, SELU 0.1 0.5
Input dropout
Dropout
Context module (hopfield layer)
Heads
8, 16 512 (512:2048]
Association space dimension
Dropout
0.1, 0.5
Cross-attention module (transformer mechanism)
Heads
1, 1,8, 10, 16, 32, 64 567 [512; 4096]
Number units in the hidden feedforward layer
Association space dimension
1088 [512;2048] 0.1, 0.5, 0.6, 0.7 1,2,3
Dropout
Number layers:
Similarity module:
Metric
cosine similarity, dot product, MinMax similarity
Similarity space dimension
512, 1024
32 [20;45] False, True False, True
nayer-normallzztionn
Affine
Training
Learning rate
0.0001, 0,001, 0.01
Optimizer
Adam, AdamW 0. 0.0001
Weight decay
Batch size
4096
Warm-up phase (epochs)
Constant learning rate phase (epochs)
25, 35
Decay rate
0.994
Max. number of epochs
350
distributation of all FS-Mol training molecules, were used. All descriptors were normalized based the FS-Mol training data.
Hyperparameter search. All hyperparameters were selected based on manual tuning on the validation set. We report the explored hyperparameter space in Table A4. Bold values indicate the selected hyperparameters for the final model. Early stopping points for the different reruns are chosen based the AAUC-PR metric the validation set. For the five reruns the carly-stopping points, that were automatically chosen by their validation metrics, were the checkpoints at epoch 94, 192, 253, 253 and 309.
Model training. Figure A2 shows the learning curve of an exemplary training run of a MHNIs model on FS-Mol. The left plot shows the loss on the training set and the right plot shows the validation loss. The dashed line indicates the checkpoint of the model which was saved and then used for inference on the test set, whereas the stopping point was evaluated maximizing the AAUC-PR metric on the validation set.
Performance improvements in comparison to a naive baseline. Figure A3 shows a task-wise performance comparison between MHNfs and the Frequent Hitter model. Each point indicates task in the test set and colored according to their super-class membership. 132 cases the MHNfs outperforms the frequent hitter model. In 25 cases the frequent hitter model yields better performance.
21

---

Published as a conference paper at ICLR 2023
0.40
1.2
Loss on training 5et
1.1
0.30
Less on validaation
1.0
0.25
0.9
0.20
0.15
0.8
7/1
50
100
150
200
250
300
350
50
100
150
200
250
300
350
Epoch
Epoch
Figure A2: Exemplary MHNNS learning curve on FS-Mol. On the x-axis the number of epochs is displayed and on the y-axis the training loss (left) and the validation loss (right) is shown. The dashed line indicates the determined carly-stopping point which determined based on AAUC-PR on the validation set.
0.5
Oxidoreductase
Hydrolase
Lyase
0.4
Ligase
0.3
MHNfs [AAUC-PR]
0.2
0.1
0.0
-0.1
0.11
0.0
0.1
0.2
0.3
0.4
0.5
Frequent hitter [AAUC-PR]
Figure A3: Performance comparison of MHNIS with the frequent hitter model. Each point refers to task the test set. Dashed lines indicate variablility across training reruns and different test support sets. The most points are located above the dashed line, which indicates that MHNfs performs better than den FH baseline this task.
22

---

Published as a conference paper at ICLR 2023
Table A5: Hyperparameter space considered for the PAR model selection. The hyperparameters of the best configuration are marked bold.
Hyperparameter
Explored values
Training
Meta learning rate
1.0
10 10-06
05
10 14-04
1.0
10-1; 1.0 10-02
Inner learning rate
0.01, 0.1
Update step
1,2
Update step test
1,2
Weight decay
5.0 10
1.0- 10-08
Epochs
200000
Eval. steps
2000
Encoder
Use pre-trained GNN
yes, no
Attention-bast based module
Map dimension
128, 512
Map layer
2, 3
Pre fe layer
0,
Map dropout
0.1, 0.5
Context layer
2, 4
Relation graph
Hidden dimension
8, 128, 512
Number of layers
2,
Number of layers for relation edge update
3,
Batch norm
yes,
Relation dropout 1
0, 0.25, 0.5 0.2, 0.25, 0.5
Relation dropout 2
A.1.7
PAR: DETAILS AND HYPERPARAMETERS
The PAR model (Wang et al., 2021) includes pre-trained GNN encoder, which creates initial embeddings for the query and support set molecules. These embeddings are fed into an attention mechanism module which also uses activity information of the support set molecules to create enriched representations. Another GNN learns relations between query and support set molecules.
Hyperparameter search. For details we refer to Wang al. (2021) and https: / thub. com/tatal Met.:aaaaaalee//aaaeerrr2222222
ma main/parserrpp. All hyperparameters were se- lected based on manual tuning on the validation set. The hyperparameter choice for Tox21 (Wang et al., 2021) was used as starting point. We report the explored hyperparameter space in Table A5. Bold values indicate the selected hyperparameters for the final model. Notably, just report hyperparam- eter choices which were different from standard choices. We used training script provided by (Wang al., 2021), which can be found here https::////. /ttts::/ggiee httpt/gttttttttt/// coo/tatal P
A.2 DETAILS ON THE FS-MOL BENCIMARKING EXPERIMENT
This section provides additional information for the FS-Mol benchmarking experiment (see Section 5).
Memory-based baselines. The Classic Similarity Search can be considered as a method with associative memory, where the label retrieved from the memory. Notably, for this method, the associative memory is very limited since it is the support set. Siamese Networks, analogously to the Classic Similarity Search, retrieve the label from a memory, whereby the similarities are determined in learned space. Also, the (eeRReLLSTM-baa the LSTM controls storing and removing information from the training data by the input and the forget gate. In NLP. kNN-type memories currently used. Conceptually, they very similar to the Modern Hopfield Networks, setting the number of heads to one and choosing a suitable value for 3.
method can be seen having memory, whereby
23

---

Published as a conference paper at ICLR 2023
Table A6: Results on FS-Mol [AAUC-PR The error
bars represent standard deviation across training reruns.
Method
AAUC-PR
ADKF-IFT (Chen et al., 2022)
.234 = 0011
IterReILSTM (Altae-Tran al., 2017)
.234 = .002
MHNfs
.241 .005
35
6.5
244
04
MHNls &AUC .R
MHNls AAUC-PP
0.3
3.0
03
0.3 IterRefLSTM AAUC-PR
0.5 ADKF-IF IFF AAUC-PR
6.2
Figure A4: Task-wise model comparison. The left scatterplot shows a comparison between MHNfs and the IteR based method and the right scatterplot shows comparison between MHNfs and ADKF-IFT. Each dot refers to task in the test set. For tasks on which the MHNfs performs better related dots are colored blue; otherwise the dots are colored orange.
Results. The reported performance metrics comprise three different sources variation, namely variation across different tasks, variation across different support sets during test time, and variation across different training reruns. While error bars in Table report variation across tasks, error bars in Table A6 report variation across training reruns. For ADKF-IFT, the authors provided error bars for every single test task. Based on these error bars we sampled performance values to be able to compare ADKF-IFT with the MHNfs training reruns. Figure A4 shows task-wise model comparison between a) MHNI' and the IterRe1LSTM-base For a) MHNfs performs better 106 of 157 tasks and therefore significantly outperforms the IterRefLSTMM based method (binomial test p-value 6.8 10-9). 9). For b) MHNís performs better on 98 tasks and therefore significantly outperforms ADKF-IFT (binomial test p-value 0.001), too. Notably, ADKF-IFT performs better on non kinase-targets which can be seen in Table
method and b) MHNff and ADKF-IFT.
A.3 DETAILS ON THE ABLATION STUDY
The MHNfs has two new main elements compared to the most similar previous state-of-the art method ItT,,eLLTMM, which are the context module and the attentioon-moullee this ablation study we aim to investigate i) the importance of all design elements, which are the context module, the cross-attention module, and the similarity module, and ii) the superiority of the cross-attention module compared to the IterRefl STM module.
A.3.1 ABLATION STUDY A: COMPARISON AGAINST ITERREFLSTM
For fair comparison between the cross-attention module and the IterRefLSTM used pruned MHN version ("MHNfs -CM") which has no context module and compared it with the IterRefLSTM model. The evaluation includes five training reruns each and ten different support set samplings.
24

---

Published as a conference paper at ICLR 2023
Table A7: Results of the ablation study on FS-Mol [AUC, AAUC-PR J. The error bars represent standard deviation across training reruns and draws of support sets. The p-values indicate whether the difference between two models consecutive rows significant.
Method
AUC
AAUC-PR
PAUC
PAAUC-PR
MHNí' (CM+CAM+SM)
.739 + .055
.241 + .006
MHNffs -CM
.737 + .004
.240 .055
0.030
0.002
MHNfs -CM -CAM
.719 + .006
.223 = .006
< 1.0e-8
<1.0e-8
Similarity Search
.604 + .003
.113 + .004
<1.0e-8
1.0e-8
IterReLLSMM (Altae-Tran et al., 2017)
.730 = .005
.234 : .005
<1.0e-8
8.73e-7
a paired Wilcoxon rank sum test b IttrRefLSTM is compared to MHNIs -CM
The results, reported the across training reruns and support sets, can be seen in Table A7. We performed paired Wilcoxon rank sum test for both the AUC and the AAUC-PR metric. Both p-values indicate high significance.
A.3.2 ABLATION STUDY B: ALL DESIGN ELEMENTS
We evaluate the performance all main elements within the MHNfs, which are the context module, the cross-attention module, the similarity module and the molecule encoder. For this analysis, we start with the complete MHNfs which includes all modules and report AUC and A AUUC-PR performance values. Then. we iteratively omit the individual modules, measuring whether there is significant performance difference with and without the module. Table A7 shows the results, where performance values for the full MHNfs, a MHNfs model without the context module ("MHNfs -CM") and MHNfs module without the context and the cross-attenion module ("MHNfs -CM -CAM") is included. Notably, the model without the context module and without the cross-attention module just consists of learned molecule encoder and the similarity module. We evaluted the impact of the learned molecule encoder by replacing it with a fixed encoder, which maps a molecule to its descriptors. The model with the fixed encoder is classic chemoinformaties method which is called Similarity Search (Cereto-Ne -aseot-- et al., 2015).
For the evaluation, performed five training reruns for every model and sampled ten different support sets for every task. Table A7 shows the results in terms of AUC and AAUC-PR. We performed paired Wilcoxon rank tests both metrics, comparing table. The table shows that every module has a significant impact as omitting a module results in significant performance drop. The comparison between the MHNff version without the context module and without the cross-attention module with the Similarity Search showed a significant superiority of the learned molecule encoder comparison the fixed encoder.
methods in consecutive rows in the
A.3.3 ABLATION STUDY C: UNDER DOMAIN SHIFT ON Tox21
Referring to Section A.3.2, the context module and the cross-attention module showed their impor- tance for the global architecture. This importance gets even more pronounced for the domain shift experiment on Tox21 as one can see in Table A8.
Again, five training reruns and ten support draws used for evaluation. Including the context module makes clear and significant difference for both metrics AUC and AAUC-PR.
A.4 DETAILS ON THE DOMAIN SHIFT EXPERIMENTS
This section provides additional information for the Domain shift experminet on Tox21.
The reported performance metrics comprise three different sources variation, namely variation across different tasks, variation across different support sets during test time, and variation across different training reruns. While error bars in Table 2 report variation across both, drawn support sets and training reruns, bars in Table A9 just report variation across training reruns. Notably, for the Similarity Search, the performance values do not vary since the model does not include any trainable parameters.
Results.
25

---

Published as a conference paper at ICLR 2023
Table A8: Results of the ablation study on Tox21 [AUC, AAUC-PR J. The error bars represent standard deviation across training reruns and draws of support sets. The p-values indicate whether model is significantly different the MHNfs in terms of the AUC and AAUC-PR metric.
Method
AUC
AAUC-PR
PAUC
PAAUC-PR*
MHNfs (CM+CAM4SM)
.679 = .018
.073 = .008
MHNff -CM
.662 = .028
.069 = .012
6.28e-8
0.002
MHNff -CM -CAM
.640 + .018
.057 1 .009
<1.0e-8
<1.0e-8
Similarity Search
.699 + .015
.061 + .008
<1.0e-8
<1.0e-8
IterReiLSTM
.644 = .018
.067 .008
2.53e-6
3.38e-5
paired Wilcoxon rank sum test
Table A9: Results of the domain shift experiment on the Tox21 dataset [AUC, AAUC-PR). The best method is marked bold. Error bars represent standard deviation across training reruns
Method
AUC
AAUC-PR
Similarity Search (baseline)*
.629 + .000
.061 = .000
IterReILSTM (Altae-Tran et al., 2017)
.664 + .004
.067 = .001
MHNIs
.679 + .009
.073 = .003
"The Similarity Search does not include any learned parameters. There- fore, there is no variability across training reruns.
A.5 GENERALIZATION TO DIFFERENT SUPPORT SET SIZES
In the following section, we test the ability of MHNff to generalize to different support set sizes. During training in the FS-Mol benchmarking setting, the MHNfs model has access to support sets of size 16. However, at inference, the support set size might be different. Figure A5 provides performance estimates of the support-set-size-1 MHNfs models other support set sizes. Note that the estimates could be seen as approximate lower bounds of the predictive performance on settings with different support set sizes (y-axis labels). For a model used in production in a real-world drug discovery setting, MHNfs should be trained with varying support set sizes that resemble the distribution of real drug discovery projects.
Triantafillou et al. (2019) analysed the performance of different few-shot models across different support set sizes. Their analysis showed that in very-low-data settings embedding- methods, namely Prototypical Networks and fo-Proto-MAML, performed best. In contrast to that, finetuning- based method significantly profit from larger support set sizes (Triantafillou et al., 2019).
MHNí' is embedding-basee method and performs in accordance with the findings mentioned above (Triantafillou et al., 2019) - well for small support set sizes (see Table 1). Following Triantafil- lou et al. (2019), it is exactly the settings related to these smaller support set sizes, e.g. a support set size of 16, which are suitable for MHNfs. For large support set sizes, e.g. 64 or 128, we point to the work done by Chen al. (2022), in which the fine-tuning method ADKF-IFT achives AAUC-PR-score > 0.3 for large support set sizes.
A.6 GENERALIZATION TO DIFFERENT CONTEXT SETS
In this section, we test the ability of MHNI' to generalize to different context sets. While the FS-Mol training split is used as context during training, assessed whether model is robust to different context sets for inference. To this end we preprocessed the GEOM dataset (Axelrod and Gomez- Bombarelli, 2022) from which used 100,000 molecules that passed all pre-processing checks. From this set, we sample 10,000 molecules as context set for MHNfs. Because GEOM contains drug-like molecules, similar to FS-Mol the predictive performance remains stable (see Table A10).
26

---

Published as a conference paper at ICLR 2023
0.28
0.26
0.24
AAUC-RR (test set
0.20
0.18
0.14
64
Support
Figure A5: Performance of MHNfs for different support set sizes during inference time. The MHNfs models trained with support sets of the size 16.
Table A10: MHNIs performance for different context sets (AAUC- PR J. The emror bars represent standard deviation across training reruns and draws of support sets.
Dataset used as a context
AAUC-PR
FS-Mol (Stanley et al., 2021)
.2414 + .006 .2415 + .005
GEOM (Axelrod and Komez-Bombaree 2022)
27

---

Published as a conference paper at ICLR 2023
A.7 DETAILS AND INSIGHTS ON THE CONTEXT MODULE
The context module replaces the initial representations of query and support set molecules by retrieval from the context set. The context set a large set of molecules and covers a large chemical space. The context module learns how to replace the initial molecule embeddings such that the context-enriched representations are put in relation to this large chemical space and still contains all necessary information for the similarity-based prediction part. Figure A6 shows the effect of the context module for the MHNffs model. Extreme initial embeddings, such as the purple embedding on the right, are pulled more into the known chemical space, represented by the context molecules. Notably, the replacement described above is soft replacement, because also the initial embeddings contribute the context-enriched representations due skip-connections.
A.8 REINFORCING THE COVARIANCE STRUCTURE IN THE DATA USING MODERN HOPFIELD NETWORKS
We follow the argumentation of (Fürst al., 2022, Theorem A3) that retrieval from associative memory of a MHN reinforces the covariance structure.
Let us assume that we have molecule embedding from the query set Rd and molecule embedding from the support set æ E Rd and both have been enriched with the context module with memory C dx (ignoring linear mappings):
m' = C softmax(3C" (3C m) x' = C softmax(3C"" (3C x)
(A9)
(A10)
Then the similarity of the retrieved representations as measured by the dot product can expressed in terms of covariances:
28

---

Published as a conference paper at ICLR 2023
20
Initial Embedding Contexteenriched Representation
pc1
-15
-20
pco
Figure A6: PCA-downprojection plot of molecule embeddings. Each dot represents molecule embedding, of which the first two components are displayed on the x- and y-axis. Blue dots represent context molecules. Dark purple dots represent initial embeddings for some exemplary molecules, of which some exhibit extreme characteristics and are thus located away from the center. Arrows and light purple dots represent the enriched molecule embeddings after the retrieval step. Especially molecules from extreme positions are moved stronger to the center and thus are more similar to known molecules after retrieval.
m x' = softmax(3C m) m)" " x3 (c Cov(C, m)² m)" (a+
(A11)
x))(),))x).
(A12)
where è is the row mean of C and following the weighted covariances are used:
Cov(C, m) = (BCmm)
Cov(C, a:) CJJH (3C=)
(A13)
Jm R + M MMxMa is mean Jacobian function of the softmax (Fürst et al., 2022, (A122)))).
The Jacobian J of p softmax(Ba) is J(Ba) = (diag(p) -pp"".
bTJ(3a) = 3. (diag(p) - pP") = Eni Ds - (InL by
(A14)
this is the second moment minus the mean squared, which is the variance. Therefore, b JTI((a)b is B times the covariance of if component is drawn with probability p, of the multinomial distribution p. In our case the component is context sample c, Jm is the average of J(Aa) over X = to À = B.
Note that we can express the enriched representations using these covariance functions:
m' (ī + Cov(C, m) m)
(A15)
z' = (ī + Cov(C, (C)x)))))))
(A16)
which connects retrieval from MHNS with reinforcing the covariance structure of the data.
29

---

Published as a conference paper at ICLR 2023
A.9 DISCUSSION, LIMITATIONS AND BROADER INPACT
In benchmarking experiment, the architecture was assessed for its ability to learn accurate predictive models from small sets of labelled molecules and in this setting it outperformed all other methods. In a domain shift study, the robustness and transferability of the learned models has been assessed and again MHNfs exhibited the best performance. The resulting predictive models often reach an AUC larger than .70, which means that enrichment of active molecules is expected (Simm et al., 2018) when the models are used for virtual screening. has not escaped our notice that the specific context module we have proposed could immediately be used for few-shot learning tasks in computer vision, but might be hampered by computational constraints.
Effectively using the information stored in the training data for new tasks is not only key for our context-module but also for a lot of other few-shot strategies like pre-training or meta-learning. For pre-training and meta-learning based approaches, this information is stored in the model weights, while the context module directly has access to it via an external memory. We believe that accessing this information directly via an external memory is benefitial in this setting because a) pre-training for small molecule drug discovery is a promising approach, but still comes with its own challenges (Xia et al., 2022) and b) meta-learning approach, like MAML,, needs labeled data while Modern Hopfield Networks operate on unlabeled data and therefore might be able to give access to more comprehensive information in the data including unlabeled data points.
Limitations. In the FS-Mol benchmark experiment, the runner-up method ADKF-IFT (Chen et al., 2022) performed better on non kinase-tasks. We hypothesize that we could improve the MHNfs performance for non kinase tasks by upsampling the other task sub-groups. While the implementation of our method is currently limited to small, organic trug-like molecules as inputs, our conceptual approach can also be used for moleollol such as RNA, DNA or proteins. The output domain of our method comprises biological effects, such that the prediction must be understood in that domain. Our method demands higher computational costs and memory footprint as other embedding- based methods because of the calculations necessary for the context module. While we hypothesize that our approach could also be successful for similar data in the materials science domain, this has not been assessed. Our study is also constrained by a limited amount of hyperparameter search for all methods. Deep learning methods usually have large number of hyperparameters, such as hidden dimensions, number of layers, learning rates, of which we were only able to explore the most important ones. The composition and choice of the context set is also under-explored and might be improved by selecting reference molecules with an appropriate strategy.
Broader impact. Impact on machine learning and related scientific fields. We envision that with (a) the increasing availability of drug discovery and material science datasets, (b) further improved biotechnologies. and (c) accounting for characteristics of individuals, the drug and materials discovery process will be made more efficient. For machine learning and artificial intelligence, the novel way in which representations are enriched with context might strengthen the general research stream to include more context into deep learning systems. Our approach also shows that such a system is more robust against domain shifts, which could be a step towards Broad AI (Chollet, 2019; Hochreiter, 2022). Impact society. If the approach proves useful, it could lead to a faster and more cost- efficient drug discovery process. Especially the COVID- 19 pandemic has shown that it is crucial for humanity to speed up the drug discovery process to few years even months. We hope that this work contributes to this effort and eventually leads to safer drugs developed faster. Consequences of failures of the method. As common with methods in machine learning, potential danger lies in the possibility that users rely too much on our new approach and use it without reflecting on the outcomes. Failures of the proposed method would lead to unsuccessful wet lab validation and negative wet lab tests. Since the proposed algorithm does not directly suggest treatment or therapy, human beings are not directly at risk of being treated with harmful therapy. Wet lab and in vi-vi testing would indicate wrong decisions by the system. Leveraging of biases in the data and potential discrimination. As for almost all machine learning methods, confounding factors, lab or batch effects, could be used for classification. This might lead to biases in predictions or uneven predictive performance across different drug targets or bioassays.
30


Reviews:
[{"review_id": "XzAvtuGLmI", "reviewer": "Reviewer_b2MR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a novel few-shot learning approach (Hopfield-based molecular context enrichment for few-shot drug discovery or MHNfs) that exploits a context of other compounds outside the support and query set to improve the performance of the model. The model relies on the proposed context-module, made up of Hopfield layers, to associate the representations of the query and support set samples with other samples in the training data to enrich the representations. Afterwards, a cross-attention module is used to associate the representations within the query and support set. Experiments are conducted on FS-MOL and Tox21 datasets to show the proposed MHNfs is competitive versus state-of-the-art baselines and ablations experiments are conducted to show how the two modules improve the model performance.", "summary_of_the_review": "The paper proposed several novel contributions to the molecular few-shot learning literature, especially its context retrieval module. Though there are still some questions about the proposed context retrieval module such as its applications to other model architecture, scalability and intuition behind its benefit, the experimental results are convincing.", "strength_and_weaknesses": "Strengths:\nCompetitive few-shot learning performances\nThe approach to use the context of other training samples is novel and interesting.\n\nWeaknesses:\nThe paper could benefit from studies to show how context benefits the model’s performance. The current approach of sampling 5% of the training data does not give much intuition of how the proposed helps improve the performance.\nProposed approach relies on a specific model architecture, limiting its applications to other model architectures. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S9WscW8o35l", "reviewer": "Reviewer_XyoZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses molecular property prediction in the few-shot setting by suggesting a novel Modern Hopfield Network architecture and testing it on the molecule-specific few-shot dataset FS-Mol. The novel architecture seeks to make use of the context available in the wider molecular training set through means other than a pretraining step. ", "summary_of_the_review": "This is a simple but effective idea that beats previously suggested methods on this task. While the idea is simple, the work is well executed, described, and thoroughly explored, and thus makes a valuable contribution to this particular body of work. ", "strength_and_weaknesses": "Strengths\n* The paper is very clearly written and thorough; the model is transparently described. While a relatively simple idea, the execution is excellent and baseline comparisons thorough. \n* The ablation studies are informative. \n* The novel method proposed works well: the performance gains are significant in a challenging setting \n* The paper also addresses in detail a number of state-of-the-art baselines, and importantly brings to the attention of the machine learning community the importance of the Frequent Hitters naive baseline.\n* The appendices are similarly thorough.\n\nWeaknesses\n* It would be somewhat valuable to have more discussion around why the method is regularizing the covariance as the authors discuss and how this compares to/ why this is outperforming a pretraining or meta-learning strategy using the entire training set of tasks available in FS-Mol. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JHR0CrrveYj", "reviewer": "Reviewer_WgGe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors develop a deep learning method on molecules to better leverage training data and contextual information to make property predictions.", "summary_of_the_review": "I found the work to be interesting and somewhat complex (albeit convoluted). Though the work was well presented and in a generally interesting research area, it feels like this work, as demonstrated in the paper, is hitting diminishing returns, and could find additional evidence to help bolster and argument for the utility and necessity of this approach.", "strength_and_weaknesses": "I really appreciate the implementation of the baselines, as well as the authors’ interpretation of them (Frequent Hitters and Similarity Search). Upon release of code, it is useful to see those implementations as well.\n\nIn Section 5.3 - A domain shift experiment is performed. There 8 positive and 8 negative were randomly selected. How different are they? Can better sets of divergent molecules be made by clustering based on molecular fingerprints and/or Tanimoto distances?\n\nSection 5.3 - Are there better datasets than Tox21 for this analysis? It is difficult to really see your model outperform others in this challenge. Generally, I feel that this community is hitting diminishing returns with Tox21, so understanding if algorithmic improvements actually improve performance is challenging.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "95TEP8hxaj", "reviewer": "Reviewer_kPjE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To improve upon the current best performance of models that predict molecular properties in typical drug discovery scenarios (where the number of molecules with known properties are very small), the authors propose MHNfs, an embedding-based few-shot learner using Modern Hopfield Networks to provide learned associative memory for enriching a given molecular representation with those of reference molecules, in order that the properties of the given molecule are more accurately predicted. The authors show that 1) MHNfs outperform a variety of other few-shot learning methods on the activity prediction tasks in the FS-MOL dataset, and 2) that contextual enrichment is responsible for the improvement in performance over the next-best method. They also demonstrate that their method generalizes better to non-drug-like molecules than the next-best method.", "summary_of_the_review": "The use of associative memory for enriching molecular representations is novel, and the improvement of MHNfs on FS-MOL tasks over other methods appears significant, despite being within the standard error of the next-best method. The establishment of simple, rational baselines for the FS-MOL tasks is also an important contribution to understanding that benchmark and the source of performance gains (or losses).", "strength_and_weaknesses": "Strengths:\n- Proposes a novel few-shot mechanism based on comparisons to knowns molecules, which is loosely analogous to how medicinal chemists typically explore chemical space.\n- Demonstrates that the novel mechanism outperforms all models to which it was compared.\n- Shows that a very simple predictor based on average activity performs better than most previous few-shot learning methods.\n\nWeaknesses:\n- The performance of MHNfs on FS-MOL tasks and when generalizing to Tox21 is within the standard error of the next-best model. Moreover, MHNfs perform worse on non-kinase-related tasks. The authors should point these facts out, perhaps addressing the contributions of different factors to the standard errors.\n- No other comparable memory-based model is included for comparison. Comparison to a simple memory-based baseline would help understand the role of memory in the reported performance improvement.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster