A Strong Baseline for Molecular Few-Shot Learning
Authors: Philippe Formont
Hugo Jeannin
Pablo Piantanida
Ismail Ben Ayed
Venue: Accepted by TMLR ()

Abstract: Few-shot learning has recently attracted significant interest in drug discovery, with a recent, fast-growing literature mostly involving convoluted meta-learning strategies. We revisit the more straightforward fine-tuning approach for molecular data, and propose a regularized quadratic-probe loss based on the the Mahalanobis distance. We design a dedicated block-coordinate descent optimizer, which avoid the degenerate solutions of our loss. Interestingly, our simple fine-tuning approach achieves highly competitive performances in comparison to state-of-the-art methods, while being applicable to black-box settings and removing the need for specific episodic pre-training strategies. Furthermore, we introduce a new benchmark to assess the robustness of the competing methods to domain shifts. In this setting, our fine-tuning baseline obtains consistently better results than meta-learning methods.

Published in Transactions on Machine Learning Research (02/2025)
A Strong Baseline for Molecular Few-Shot Learning
Philippe Formont
International Laboratory on Learning Systems, Montreal, Canada Université Paris-Saclay, Ecole de technologie superieur, MILA
Hugo Jeannin
International Laboratory on Learning Systems, Montreal, Conada Ecole de technologie superieur
Pablo Piantanida
International Laboratory Learning Systems, Montreal, Canada Université Paris- Paris-Saclay, MILA Ismail Ben Ayed
International Laboratory on Learning Systems, Montreal, Canada Ecole de technologie supericur
Reviewed on OpenReview: https: // openreview. net/ forum? id= JQ0agisxny
Abstract
Few-shot learning has recently attracted significant interest in drug discovery, with recent, fast-growing literature mostly involving convoluted meta-learning strategies. We revisit the more straight forward fine-tuning approach for molecular data, and propose a regularized quadratic-pre probi loss based on the the Mahalanobis distance. We design a dedicated block- coordinate descent optimizer, which avoid the degenerate solutions of our loss. Interestingly, our simple fine-tuning approach achieves highly competitive performances in comparison to state-of-the-arl methods, while being applicable to black-box settings and removing the need for specific episodic pre-training strategies. Furthermore, to assess the robustness of the competing methods to domain shifts. In this setting, our fine-tuning baseline obtains consistently better results than meta- meta-learning methods.
introduce a new benchmark
1 Introduction
Drug discovery is a process that aims to answer the following question: "How do we design molecules with biological activity on a specific therapeutic target and other necessary properties become a drug?", To identify such compounds, scientists roly on combination of experimental and computational methods (Xu & Hagler 2002 Eckert & Bajorath 2007). This lengthy and costly process takes average 12 years and 1.8 billion dollars to bring new drug to the market (Pammolli et al. 2019 Arrowsmith, 2011). To reduce the cost associated with this process and to speed up the development of new drugs, machine learning methods are often used to guide the discovery of new compounds (Gilmer et al.) 2017 Chen et al. 2018 A common task for these models is to predict the activity of a compound on a given target exploiting the compound's 's structural or physio-chemical propertics, referred to as quantitative structure-activity relationship (QSAR) (Tropsha 2010). QSAR models can also be used oracles guide generative algorithms into generating compounds while optimizing their properties (Bengio et al.) 2021 Blaschke et al. 2020 Gao et al. 2022
Low data regime. The data exploited to train those models is searce and costly to generate, as it results from numerous wet. lab experiments. In practice the size of the datasets corresponding to a novel target fluctuates along each project. Often, only up to a few hundred molecules are available to train the models, For this reason, various approaches for few-shot-learning in drug discovery have been proposed, attempting to provide reliable predictions while using small amount of labelled data, where meta-laarning seems appear as the default and best solution (Schimunek et al. 2023 Chen et al. 2023 Altac-Tras et. al. 2017).

---

Published in Transactions on Machine Learning Research (02/2025)
Meta-learning. Meta-learning corresponds to family of methods pre-trained to 'learn to learn' from small set of labelled examples (support set), directly conditioning their predictions on this support set (Vinyals et al. 2016). Hence, these algorithms must pretrain a model specific to few-shot learning tasks. This implics that different pre-trained models must be maintained to perform few-shot and standard classification. Furthermore, most meta-learning algorithms use specific training procedures, as the training and test-Lime scenarios must be similar. Controlling the training procedure of these methods is hence necessary to use them. This can be problematic in a black-box setting, where only the predictions of the model can be queried through an API, and the model's weights cannot be accessed (Ji et al.) 2021 Ouali et al., 2023). Using these methods in black-box scenario would require imposing the training procedure on the server, which is not always possible, or would limit the number of models that could be used. Yet this black-box setting is especially valuable in drug discovery, where data is highly sensitive, and sharing a dalaset or pre-trained
model on a private dataset is not always possible (Heyndrickx et al. 2024 Pejo et al. 2022 Smajić et al., 2023
Competitive fine-tuning methods. Fine-tuning methods involve using a model pre-trained in a standard supervised setting and then fine-tuning it for few-shot classification. Unlike meta-learning, fine-tuming approaches utilize pre-trained models that can be applied to other tasks in standard machine learning, beyond few-shot classification, hence, a wider range of models can be used. In other domains, such as computer vision, meta-learning methods were shown to obtain only marginal improvements over more straightforward fine-tuning methods, using models pre-trained in a standard supervised setting (Chen et al. 2019 Luo et al, 2023). Some of these results were made possible thanks to the availability of foundational models pre-trained
large datasets available in these fields (Radford et al.; 2021 Brown al.] 2020). However, developing
such foundational models is non-trivial task in the context of drug discovery, and is still an active rescarch
field (Seidl et al.) 2023 Ahmad et al. 2022 Méndez-Lucio et al. 2022). In drug discovery, it was observed
that fine-tuning approaches yielded subpar performances (Stanley et al. 2021), a finding seemingly
at odds with existing results in vision applications.
Contributions. We re-visit fine-tuning black-box models using a simple Multitask backbone to perform QSAR few-shot learning. Our contributions could be summarized as follows:
We evaluate the standard linear probe, a baseline often overlooked in the molecular few-shot learning literature.
We generalize this standard cross-entropy baseline to a more expressive quadratic-probe loss based on the Mahalanobis distance and the class covariance matrices. We examine our loss from an optimization perspective, observe degenerate solutions and mitigate the issue with block-coordinate descent and dedicated regularization, which yields closed-form solutions at cach optimization step. Interestingly, our quadratic probe achieves highly compelitive performances as the size of the support set increases.
Finally, in addition to evaluations on the standard FS-mol benchmark, we built two out-of-domain evaluation setups to evaluate the models' robustness to domain shifts. Our fine-tuning baselines obtain consistently better results than meta-leariig algorithms.
2 Related Work
Notations. Assume that we are given a base dataset Dbase Wteteoeee where Thase is a set of tasks. For a task t, we have Disse= = (x,, yi Jusiga where D; is a molecule and ys E 10, denotes its label.
We then have access to a test datasot defined similarly: Dest (D) ) where Tnovel is the set of new tasks (T nTooeel n T base = 0). Finally, for a task 6, Diss is divided into two sets: S,, 2, n 0) defining a set of labelled data that we will use to adapt the model, ie., support set (0t, ye hes,, and an unlabelled set. whose labels are to be predicted, i.e. query set ( hee.
The goal of the fho- algorithms we study is to leverage the supervision information from the support samples to predict the labels within the query set.
2

---

Published in Transactions on Machine Learning Research (02/2025)
Meta-learning. Training a meta-learning model, referred to meta-training, differs from classical supervised procedure by the fact that the training set is divided into tasks. Fach task is divided into a support set and a query set that will be fed into the model to perform its training. This data processing is specific to meta-lcarning.
The work of Stanley Stanley et al. (2021) introduced a benchmark for molecular few-shot learning. FS-mol, and evaluated various methods. For instance, the authors trained an adaptation of Protonet (Snell al. 2017) with graph neural network (GNN). Besides, the authors pro-trained graph neural network (GNN) with the MAML algorithm (Finn et al. 2017), model aiming at learning good initialization of the model's weights, such that the model can be quickly adapted to new tasks. Building on the MAML paradigm, the property-aware relation network (PAR) (Wang et al.] 2021) builds similarity graph between the molecules of the support set and the queried molecule and extracts from this graph the molecule's predicted class, In recent work, Chen et al. (2023) proposed a method separating the parameters 6meta that are updated during meta-training, while the other parameters Badapt are only updated during the adaptation. The back-propagation on each lask is done by using Cauchy's implicit function theorem, which requires estimating the hessian of the loss a' Lumpposs on the support set. Finally, some work evaluated the effect of leveraging a "00"
large set of context molccules to construct similarity measures between the support set's molecules and the queried molecule using modern hopfield networks (Schimunek et al. 2023
Fine-tuning methods. Among the baselines presented by Stanley al. (Stanley et al. 2021). some of them used backbone trained without relying meta-training. First, the authors fine-tuned GNN trained with a multitask objective. This model consists in GNN common trunk followed by 2-layer task-specific head. At test time a new task-specific head is added corresponding to the new task to adapt on. Both the common-trunk and the task-specific parameters are fine-tuned on the support set. A second approach used the molecular attention transformer (MAT) [Maziarka et al.) 2019) and fine-tuned it on the support set of the novel task. Both methods achieved performances on FS-mol below the aforementioned meta-learning methods. Furthermore, they need to fine-tune the whole model, which can be computationally expensive and requires the user to access the model's weights at lest. time. Hence, these methods cannot be directly used in a black- box setting. In the recent years, significant effort was made to build foundational models able to handle molecular data (Méndez-Lucio et al. 2022: Beaini et al. 2024 Khan et al. 2024 Xia et al. 2023 but very few proposed meth assess the ability of these models to perform few-shot classification. Notably, the work of Seidl et al. (2023) proposed a multi-modal model leveraging both the molecular graph and a textual representation of the assays to make predictions, enabling the creation of zero-shot models. The authors evaluated the model on FS-mol with a linear probe. However, to train their model, the support set of all tasks was integrated into the pre-training set, which breaks the assumption T Trovet n Thase 0. This evaluation corresponds to assessment of the model's ability to generalize on previously seen tasks with few labels, not the model's ability to adapt to unseen tasks.
3 Methods
3.1
Model pre-training
Various methods exist to pre-train models for drug discovery (Hu* et al. 2020 which can usually be divided into supervised and self-supervised methods. Supervised pre-training refers to methods training a model on a dataset of molecules relying on the association between the input and labels. In contrast, self-supervised pre-training refers to methods training a model on a datasct of molocules without using or knowing the labels. While self-supervised pre-training enables the use of larger data sets, supervised pre-training tends to be more effective when the labels used during training correspond to assays similar to those of the downstream task Sun et al. 2022 In this work, we pursue the supervised setting via multitask model based on the training set of the I S- benchmark (Stanley et al. 2021). We focus our work on QSAR modeling, evaluating the ability of the pre-training backbone to adapt to new tasks. An evaluation of pretrained models originating from the molecular representation learning literature is available in Appendix A.6
The GNN backbone fo will take the molccular graph and the fingerprints as input, based on a principal neighborhood aggregation model (Corso et al. 2020). We add task-specific head gø to the backbone,
3

---

Published in Transactions on Machine Learning Research (02/2025)
one-layer binary classifier during the pre-training. Once the training is finished, only keep the parameters 0 of the backbone. The molecular embeddings passed to the few-shot classifier are obtained by
fa(#:)
Sa(9,, fi)
fo (x))
19 fill
with Gj denoting the molecular graph, f. the molecular fingerprint.
3.2 Multitask Linear Probing
As explained in the Section multitask models were shown be oulperformed by most other methods (Stanley et al. 2021). p. Still, we show that we can obtain better classification capabilities by following the paradigm of linear probing (Chen et al. 2019 Lincar probing corresponds model training the parameters uk, ||w = 1 for each class l E C-= (0, Relass) The predicted probability of molecule e to belong to
the class l E C is then given by:
exp (T(z,, un)) + bk)
Pik Exec exp (Tsz ww) + by)
(1)
where by is the model's bias, is temperature hyperparameter, and for the clarity of the notation, we omitted the dependency on the task t. The lincar probe's parameters are then optimized to minimize the cross-entropy loss on the support set.
3.3 Quadratic probing
Motivation. We propose an extension of the aforementioned linear probe, which is more competitive as the support set grows. Instead the cosine similarity used in the linear probe, we propose to use the Mahalanobis distance between the query points and the class prototypes:
Linear classifier
Mahalanobis classifier
( 2, -e-=:xllss 122 (z, - wx)) Σx' (z: wk).
(2)
where ΣĄ a positive semi-definite (PSD) matrix. For simplicity, will refer to M2 A Ex' the precision matrix of the distribution to simplify the notations.
The predicted probability of molecule æ; to belong to the class k E C can be then obtained as:
exp (-1|z; - wk Mr
Pi.k
(3)
Evecest exp (-1|z, ww Il.))
Figure 1: Values of the logits produced by two different classifiers: lincar probe (left) and a quadratic probe based on the Mahalanobis distance (right), in a two-dimensional feature space with two classes.
The Mahalanobis distance measures the distance between a point and a normal distribution of mean wa and covariance matrix Σg. This distance is more expressive than the cosine similarity, as it considers the covariance matrix of the class. In fact, one can see as gencralization of the lincar probe, which corresponds to choosing covariance matrix proportional to identity. Figure depicts an illustration of the difference between linear probe and our quadratic probe.
In the next section, we show that directly minimizing the cross-entropy loss over M2 yiclds degenerate solutions, in which the Frobenius norms the covariance matrices go to infinity. To mitigate this issue, we propose a surrogate objective, which corresponds approximately to augmenting the cross-ent cross-entropy loss with a regularization term based on the Frobinous norms of the covariance matrices, while yielding closod-form solutions at each optimization step w.r. Mg. We proceed with block-coordinats descent approach to optimize regularized quadratic-probe loss, updating the parameters (w)) (e(o))((.1) and (M& hE separately.
4

---

Published in Transactions on Machine Learning Research (02/2025)
Gradient descent. Optimizing the parameters of the quadratic probe is challenging task, the estimator's number of parameters is much larger than the linear probe's. The first greedy option would be to optimize M& by gradient descent on the support set. We propose to optimize D3 M (M& = = Dr = 25 parameter of the model to ensure that M2 is PSD.
However, by following this procedure, the model has a high chance of over-fitting. In particular, if the classes are linearly separable, the model will find a minimum of the cross-entropy loss, pushing M|M to become arbitrarily large.
Proposition 3.1. Let @ (wk, Ma and we will note pe z) = Pi.k as described Equation 3 (to highlight the dependency to the parmmeters O).
If the sumples from both classes are linearly separable, we can construct set of parameters, @ (op,MM()))ee(00 such that:
ViE E (O...., N), limx- Lee (z, Vi, (())) = 0,
Vb E {0, limx-+x0 M| = t00,
where (ze yr: +) = 1---log (e (2) is the point-wise cross-entropy loss w.r.t our model.
The proof of proposition 3.1 is relegated to Appendix A.2. This proposition relies on the assumption that the data points are linearly separable. In the few-shot learning setting, this assumption often verified. Indcod, as the data dimension much larger than the number of data points, Cover's theorem (Cover 1965) states that the points are almost certainly lincarly separable. Another argument supporting the assumption of linear separability comes from an approximation often made in high-dimensional spaces: if d > N, Vi,j E {O,.... ,0,...,.N3 NN² ziz,=10 = 0. Then, the vector Ei Viz4 (1 y;)zz defines a hyperplane separating data points in both classes.
Block coordinate optimization and modified loss. Instead of optimizing the parameters of the quadratic probe by gradient descent, we optimize the parameters of the model with a block coordinate descent, updating the parameters (w>Jke{o.1; and [Ma kefo.1 separately. In particular, we will perform one gradient descent. step on {w (o.)) using the cross-entropy loss and then minimize a modified loss w.r. (e that does not suffer from the issuc described in proposition 3.1 In the following section, (wx (0.1))) is considered fixed.
Decomposing the cross-entropy in two parts, we first identify the term pushing the parameters M, to be arbitrarily large. The cross-entropy loss of the quadratic probe can be divided into two terms:
Lee = [8|
Σ (Si (zi, Wr, 0) + (2,, O)),
(4)
:ES
where
fi (Zi, Bi: 0) 1 || - Wy, 11AA,
(5)
fa (zi, Q) e log
exp (-1|z: wx 11r.))
kE(0,13
Remark 3.2. For all E S, fi (zi, Vs. θ) is minimized when ||Mlllr - O, and fa (zi, Vi, Q) as | - +oc.
The function fa is hence responsible for the degenerate solutions obtained while minimizing the cross-entropy loss. We propose to replace f2 with a different function f2, playing the same role pushing | towards infinity, but with a less aggressive behaviour. The new loss obtained fi + fz admits a closed-form solution on Me, for a fixed wa, which corresponds to the estimator of the empirical covariance matrix's 's inverse. The expression of fg is then: ex
SR log det (M&)
f2 2
Σ
[S]
AE(0.1)
where S& E 8ly; k).
5

---

Published in Transactions on Machine Learning Research (02/2025)
Proposition 3.3. Both f2 and f2 optimized when the norms of (Malne(o.1) approach infinity. When for all h: E (0, 1), the eigenvalues of M& grow to infinily:
= Σ |M&lle);
kE(1,1)
(6)
and (2, 0
Σ
log Mx]r),
kE(0,1)
And the resulting loss fi + admits a closed--fomm solution, for E {0, 1]:
M2 = Syyl iES
(z) wk) (z, wkk wxk)²
(7)
Remark 3.4. Minimizing f1 + fz is equivalent to maximizing the log-likeihood of a multivariate Gaussian distribution with mean wg and covariance matrix Σg, explaining why the closed form solution we obtain corresponds to the estimator of the empirical covariance matrix's inverse. Empirical results displaying the evolution of the largest cigenvalue of M2 using gradient descent and the quadratic probe are displayed in Appendix
The function fe can hence be seen as an upper bound on f2 when |Male becomes large, for = 0, Indeed, f2 exerts less pressure expanding Mg to larger matrices. This replacement approximately be seen adding the regularization terms a: M || 3 log |M p to the loss (a and 3 are constants).
Finally, we train the quadratic probe by updating Mj using the closed-form solution obtained above (Equa- fion 7 and performing a gradient step on wy using the cross-entropy loss. Since the estimation of the covariance matrix can be unstable, we use a shrinkage parameter À E [0, 1] to stabilize the estimation of the covariance matrix (Ex _ (1 omppiiiiiia + AT). We consider as a hyperparameter of the model to be optimized (refer Appendix A.4 for further details). The algorithm of the training process of the quadratic probe be found in Appendix A.I.
4 Experimental Results
4.1 FS-mol benchmark
Dataset. We evaluate our methods on the F'Smmol benchmark (Stanley et al. 2021 This benchmark aims to classify molecules into active/inactive finactive binary classes for a given therapeutic target, divided into 5000 tasks (40 in the validation and 157 the test sets). The tasks the models are evaluated on are balanced, with on average 47% of active molecules, a value ranging from 30% to 50%.
Methods compared. We compare our methods to various meta- meta-learning algorithms: GNN-MAML (Finn et 2017), Protonet (Snell et al. 2017), ADKT-IFT (Chen et al. 2023). the Property Aware Relation network (PAR) Wang et al. 2021 and conditional neural process (CNP) Garnelo et al. 2018). MHNFS
(Schimunek al. 2023] We report the results of fine-tuning baselines such as fully fine-tuning of molecular attention transformer (MAT) (Maziarka et al.) 2019h and GNN-MT, their multitask fine-tuning approach.
We also re-evaluated the few-shot adaptation capabilities of CLAMP (Seidl et al. 2023) on the F'Smmol benchmark (without adding the support set of each task to the model's pre-training set). We also consider simple baselines: training a random forest classifier and a kNN on the support set's molecular fingerprint (Stanley et al. 2021). and Similarity Search, standard chemoinformatie method relying only on the Tanimoto
similarity between these fingerprints (Bender et al. 2004 Saaetto-Mssaauééé al. 2015).. The authors of FS-mol also propose to train a single-task GNN on the support set of task as baseline baseline (GNN-ST).
Evaluation procedure. To compare few-shot learning methods, all models were pre-trained on FS-mol's training set. Models are evaluated using the improvement of the average precision compared to a random
We used the author's scripts train the model while implementing the evalaaton scripts, which unavailable on the author's repository.

---

Published in Transactions on Machine Learning Research (02/2025)
Table 1: Results on FS-mml (AAUCPR) of each method for support set sizes of 16, 32, 64 and 128 (averaged over 10 runs). The best models, or models whose average performances are in the confidence interval of the best model, are highlighted in bold.
MODEL-TYPE
MODEL
[S] = 16
[S] = 32
[S] = 64
[S] = 128
KNN
0.051 o.0
0.0761
0.102.
0.139
GNN-ST
0.021 1: o.0no
0.027 1 0.00s
0.031 = 0.005
0.052 0.006
BASELINES
RF
0.093+
0.125+
0.163= 0,000
0,213
SIMSEARCH
0.113# du000
0.142 0.008
0.169= 0,008
0.210 c.00e
PAR SaodfSSur1
0.130# 0.009
0.140z 0.09
0.147 = 0.00
0.161 0.00
GNN-MAMI
0.160# 0.009
0.167# 0.009
0.174= 0,000
0.192 0.000
CNP
0.187 0.011
0.192 0:013
0,195 0,011
0,208 0.031
METAALEARNING
PROTONET NaVR182222
0.206 aunco
0.242 0o00
0.271 = 0,000
0.301 c.nca
MHNFS 1CLB
0.205 0.020
0.215+ 0000
0.219 0,000
0.231 0.000
ADKF-IFT IC 2010
0.231 + 0.808
0.263 0.010
0.287 1 0.010
0.31 8 0.010
MAT
0.052 + 0.065
0.069+ 0.005
0.092 0,007
0, 03166 0.000
GNN-MT'
0,112+
0.144 0.007
0.177 0,008
0,223
FINE-TUNING
CLAMP ICLR 205
0.202 0.000)
0. d 0.09
0.248 0.000
0.271 1 0.000
LINEAR PROBE
0.224: 0.010
0.252: n.to
0.273 0.010
0.301 i 0.000
QUADRATIC PROBE
0.227 0.010
0.255 0.010
0.279 0.010
0.310s 0.010
classifier (AUCPR)). Hyper-parameters are selected through hyper-parameter search on the validation set across all support set sizes. All hyperparameters are kept constant over each support set size, except for the optimal numbers of epochs for our fine-tuning baselines, which are fitted for each support set size on the validation set. We present the results for support set sizes 16, 32, 64 and 128 (larger support set sizes would discard some tasks from the test set).
Results. Table I presents the results of the different methods the FS-mol test set. First, evaluation of CLAMP achieves results similar to the ones reported in the original paper with 16 labelled points and obtains better results with 64 and 128 labelled points. On the other hand, we did not succeed in reproducing MHNFS's s results with our implementation.
The performances of both the linear and quadratic probes outperform the other fine-tuning baselines, Using a basic pro-traiming procodure, our finc-tumil fine-tuning methods also provide strong basclines for the few-shot adaptation of pre-trained models, to estimate the relevance of new pre-training procedure in this setup.
As expected, the quadratic probe obtains consis- tently slightly better results than the lincar probe,
and the difference becomes more significative as the amount of labeled data increases. This result is expected since the quadratic probe is a model fine- tuning more parameters than the linear probe, the cost of slightly more complex procedure (requiring to estimate M&). Additionally, the quadratic probe consistently reaches performances that are compet- itive with ADKT-IFT's results at each support set size.
0.30
Free-opt-reg
AAUCPR
Free-opt
I-probe
0.29
q-probe
0.28
0
25
50
75
100
epoch
Figure 2: Evolution of the model's performance on the validation set during the few-shot adaptation. Free- opt refers to performing a gradient descent on Mg, Free-opt-reg adds regularisation on the norm of Mg. ((S) 64)
4.2 Ablation on the Free optimisation of Σg
As explained in section 3 we optimize Σg by taking the empirical covariance matrix of the emboddings, considering Wj is the mean of the distribution. Figure highlights the difference between optimizing Eg with
7

---

Published in Transactions on Machine Learning Research (02/2025)
Table 2: Results on DTI tasks (AAUCPR) for support set sizes of 16 and on the library screening tasks (ranks).
KIBA
DAVIS
BINDINGDB
_D
LIBRARY SCREENING
MODEL
|||=
|S| = 16
[S] 32
[S] =
[S]
128
SIMILARITY SEARCH
17.3+ 0.
19.1+
17.3+ 13
3.97
3.69
3.19
2.78
PROTONET NrosiPS00m
25. 1 1.0
37.62
34.0=
3.66
3.57
3.32
3.17
ADKF-IFT ICLR 2021
31.0+ 1.8
36.5+ a.
34.++ 1.1
4.01
3.98
3.72
3.35
CLAMP ICLR 9013
26.1 1 1.
37.02
34.2=
3.42
3.19
3.28
3.42
LINEAR PROBE
29.0+
38.9+
36.8+
2.98
3.55
4.18
4.81
QUADRATIC PROBE 28.8 1.4
39.2# 1.8
37.1+ 1.0
2.96
3.02
3.31
3.46
method or using gradient descent steps to minimize the cross-entropy loss (Free-opt). Our quadratic probe (4-probe) outperforms the Pree-Optimisatine approach, and is less sensitive to the number of epochs the model should be optimized with. This sensitivity to the number of epochs becomes especially problematic under domain shifts, as this hyperparamcter fitted on a validation set that is not representative of the test set. Finally, by adding a regularization on the norm of M₃ to the Free-Opt approach, the model becomes less sensitive to the number of epochs but achieves lower performances than our proposed optimization approach.
4.3 Impact of Domain Shifts
To further investigate the robustness of the different methods, we wish to evaluate the models' performances when the tasks they are evaluated on differ from the distribution of the training set. We designed scenarios extracted from the therapeutic data commons platform (Huang et al. 2021) and the LIT-PCBA dataset. TTran-Nguyen et al. 20200, in which the tasks yield domain shift compared to the base dataset.
First, the FS-mol dataset mainly comprises balanced tasks, as explained in the previous section, while the tasks we will consider in this assessment are more imbalanced. The robustness of those different methods to class imbalance essential, as molecules are often inactive in practice.
In this section, we focus our work on the top-performing models on FS-mol, the linear and quadratic probes (l-probe, q-probe), protonet, ADKF-IFT (adkt), clamp, and the standard innno-iffrrmatic method: simscarch.
4.3.1 QSAR modelling with imbalanced class distribution
Datasets. In this section, we will consider the Drug-Targed Interaction datasets available on the Therapeutic Data Common (Huang al. 2021) platform. Drug Target interaction refers to family of tasks that aim at predicting the affinity of molecule to given target using the molecular graph of the molecule, and the sequence of the target (which we discarded in our experiments)
Performances tasks
0.4
Prior shift
AAUPRC
0.3
0.2
The datasets chosen to evaluate the models are the KIBA (Tang et al. 2014), DAVIS (Davis et al. 2011); and BindingDB Kd [Liu et al. 2006) datasets. These datasets are mainly composed of kinases, the most common target type in the FS-mol benchmark. Hence, this section will help evaluate the robustness of the different methods to prior shift (difference in the label's marginal distribution), keeping the larget of the task in-domain. The mean class imbalance, measured positive examples to
BindingDB
clamp
pyprobe
protonet
ppoobo
simsearch
Figure 3: Performances of various methods on the DTI tasks. The fine- tuning baselines outperform the meta- learning methods when the tasks be- come more imbalanced.
as the absolute deviation of the proportion a perfectly balancod scenario, is the lowest in the KIBA datasct (0.12), while the BindingDB Kd and DAVIS datasets possess more imbalanced tasks 19.19 and 0.21, respectively).
8

---

Published in Transactions on Machine Learning Research (02/2025)
Evaluation procedure. As some tasks are composed of few molecules (all tasks in the DAVIS dataset are composed of 68 molecules), we only considered support sets of 16 molecules in this evaluation. All models were trained on the FS-mol benchmark, and their hyperparameters were also chosen on this benchmark. We used the AAUCPR metric to evaluate our models.
Results. The results of the different methods on those datasets are presented in Figure 3 While the state- of-the-art meta-learning methods achieve competitive results on the KIBA dataset, our baselines outperform all other methods when the distribution of the labels becomes more imbalanced. The performances of the linear and quadratic probes are similar, which is expected we performed this evaluation on support set consisting of 16 molecules, a setup where both methods achieved similar results on the main benchmark. These results seem to indicate that the fine-tuning baselines are more robust to prior shifts.
On the other hand, experiments on the BindingDB Ki and IC50 datasets are presented in the Appendix |A.5.1 where our fine-tuning basclines do not outperform meta-learning models, but where no model significantly outperforms the similarity search baseline.
4.3.2 Library screening
Datasets. Finding potential hits (active molecules) in large library of compounds, referred to as library screening, is common step in drug discovery, where scientists can then buy the most promising compounds from a commercial molecular database (Gentile et al. 2022). In these tasks, the proportion of "hits" (active compounds) is usually very low (< 1%), yicld- ing a much more significant prior shift than in the previous section. Besides, on the opposite of the section above, the targets considered in the datasets we will use are not kinases (see Appendix A.5.2), including another type of domain shift.
|s| 166
|s| 32
2
Mean rank
3
4
|.|| 64
| 128
2
Mean rank
3
To perform this eval evaluation, we will use 6 HTS datasets from the therapeutic data commons platform (Huang et al.] 2021 and ten datasets from the LIT-PCBA database (Tran-Nguyen let al.] 2020); details on the task choice can be found in the Appendix A.5.2
4
10
15
1
10
15
top-k%
top-k%
clamp
I-probe adkt
protonet simsearch
Evaluation procedure. The experiments were run by con- sidering support sets of 16, 32, 64 and 128 molecules, with a proportion of hits of both 5% and 10% in the support set, leaving all remaining hits in the query set. We considered the top-k% hitrate (proportion of true positive in the predicted positive when selecting k-% of the library) as a metric. Taking the average hitrate over all datasets the average performance of the models largely varies from one dataset to the other. We hence decided to report the average rank of each model using autorank (Herbold 2020), over 20 runs.
q-probe
Figure 4: Average ranking performances of each method on the HTS tasks according to the percentage of the dataset selected. The quadratic probe obtains the best results when the support set's size is small, while the similarity scarch is the best model with larger support sets.
be problematic, as
Results. Figure 4 presents the average rank of each model on the HTS tasks over 20 runs, for the different. support set sizes, and proportion of molecules selected.
First, with limited number of known labels (support set size of 16 32), our method consistently outperforms all other basclines, trend observed across varying proportions of selected molecules. CLAMP also proves to be competitive in this low-data regime, surpassing the performance of meta-learning methods and the linear probe with larger support sets. Indeed, the linear probe exhibits strong performance with a support set size of 16, but its performance significantly diminishes as the support set size increases.
As the support set size increases, meta-learning methods demonstrate comparable results to the quadratic probe and CLAMP, until protonet out perforrmss the quadratic probe with a support set size of 128. Notably,
9

---

Published in Transactions on Machine Learning Research (02/2025)
the similarity search, employed as our baseline for these experiments, is outperformed by most other methods when the support set comprises only few molecules. However, it becomes the best model when the support set is extensive, exhibiting superior performances, especially when few molecules are selected in each dataset (1 to 5%).
The robustness of the similarity search in these tasks can be attributed the fact that it does not rely on any pre-training, and is hence not affected by the domain shift between the HTS and FS-mol datasets.
5 Summary and Concluding Remarks
While meta-learning methods consistently yield highly competitive results in molecular few-shot Iearning, it is essential to acknowledge that fine-tuning baselines have the potential to achieve comparable outcomes. In particular, we explored the intricacies of optimizing extension of the linear probe, quadratic probe. Both probing methods are capable of competing with state-of-the-art meta-learning methods while relying model pre-trained with a standard multitask objective.
Morcover, our findings demonstrate that these fino-tuning baselines exhibit heightened robustness when confronted with a shift in the label distribution. This resilience further enhances their practical utility in diverse scenarios. Although the quadratic probe requires an additional step cep-esstiaaiin covariance matrices-it consistently yields slightly better results than the linear probe, with notable improvements under certain domain shifts. Thus, both methods provide efficient few-shot classifiers with distinct strengths that can be evaluated based on the demands of specific real-world applications.
Regrettably, these baseline approaches are often overlooked in the existing literature in this field. Notably, these methods possess distinctive advantages, particularly in a black-box setting, where they can be readily benchmarked upon the availability of a pre-trained model.
The anonymized codebase used in this paper is available at https://ggtthbb.com//raaaow Brrong-Baellne--ooeeellrrrr
6 Statement of Broader Impact
This paper presents work that aims to advance the field of Machine Learning. Our work has many potential societal consequences. We hope our work be used to help drug discovery pipelines, help discover treatments, and accelerate the process. We did not identify any risk of harm specific to our work.
10

---

Published in Transactions on Machine Learning Research (02/2025)
References
Walid Ahmad, Elana Simon, Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta-2: Towards chemical foundation models, 2022.
Han Altae-Tran, Bharath Ramsundar, Aneesh S. Pappu, and Vijay Pande. Low data drug discovery with one-shot learning. ACS Central Science, 3(4):283 293, 2017. doi: 10.1021/accceentsii.6000 6000066 URL PMID: 28470045.
https://doi.co org/ 10. www.///accee/aaee.. 6b00006
John Arrowsmith. Phase failures: 2008 2010. Nature Reviews Drug Discovery, 10(5):328 329, May 2011.
ISSN 1474-1784. doi: 10.1038/nrd3409.
Simon Axelrod and Rafael Bombar-Bommarrlli. Geom, energy-annolated molecular conformations for properly prediction and molecular generation. Scientific Data, (1):1885, 2022. doi:
URL https::/dot. 1tt.://..rr/111..11/////o/// 100111//41557-0220228888
Dominique Beaini, Shenyang Huang, Joao Alex Cunha, Zhiyi Li, Gabriela Moiscscu-Pareja. Oleksandr Dymov, Samuel Maddrell-Mandder Callum McLean, Frederik Wenkel, Luis Müller, Jama Hussein Mohamud, Ali Parviz, Michael Craig, Michal Koziarski, Jiarui Lu, Zhaocheng Zhu, Cristian Gabellini, Kerstin Klaser, Josef Dean, Cas Wognum, Maciej Sypetkowski, Guillaume Rabusseau, Reihaneh Rabbany, Jian Tang, Christopher Morris, Mirco Ravanelli, Guy Wolf, Prudencio Tossou, Hadrien Mary, Therence Bois, Andrew W Fitzgibbon, Blazej Banaszewski, Chad Martin, and Dominic Masters. Towards foundational models for molccular learning on large-scale multi-task datasots. In The Twelfth International Conference on Learning Representations, 2024. URL htttp:://pperreiieee http:///wreeviieeeeeeeforr/////a met.//oorrmiiieeeecllcccuu
Andreas Bender, Hamse Y. Mussa, Robert C. Glen, and Stephan Reiling. Similarity Searching of Chemical Databases Using Atom Environment Descriptors (MOLPRINT 2D): Evaluation of Performance. Journal of Chemical Information and Computer Sciences, 44(5):1708 1718, September 2004. ISSN 0095-2338. doi: Publisher: American Chemical Society.
10.1021/ci0499719, URL
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advonces in Neural Information Processing Systems, volume 34, pp.
Curran Associates, Inc., 2021.
Thomas Blaschke, Josep Arús-Pous, Hongming Chen, Christian Margreitter, Christian Tyrchan, Ola Engkvist, Kostas Papadopoulos, and Atanas Patronov. Reinvent 2.0: An ai tool for de novo drug design. Journal of doi: 10.1021/acs.jei http//acccei...0000... 0c00915.
Chemical Information and Modeling, 60 60(12):5918 5922, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesso, Mark Chen, Eric Sigler, Matcusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. Tn H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877 1901. Curran Associates, Inc., 2020.
Mariusz Butkiewicz, Edward W. Lowe, Ralf Mueller, JefTrey L. Mendenhall, Pedro L. Teixeira, C. David Weaver, and Jens Meiler. Benchmarking Bgaand-Based Virtual High-Throughput Screening with the PubChem Database. Molecules, 18(1), January 2013. ISSN 1420-3049. doi: 00390//moleuuls11010001 URL https:/ hictps://ww.... http///wwwwmppp www. wttp:////00000/////////111111111
Adrià Cereto-Massagué, María José Ojeda, Cristina Valls, Miquel Mulero, Santiago Garcia- Garcia-Vallvé, and Gerard Pujadas. Molccular fingerprint similarity search virtual screening. Methods, 71:58 63, 2015. ISSN 1046-2012 2023. doi: http::///do..og////oo///////ym/
wwt.22014....000 005. URL http:///wwwwwciicccccccc.oo
science/ http:///aaaaaaa///////0000111010 Virtual Screening.
11

---

Published in Transactions on Machine Learning Research (02/2025)
Hongming Chen, Ola Engkvist, Yinhai Wang, Marcus Olivecrona, and Thomas Blaschke. The rise of deep learning in drug discovery. Drug Discovery Today, 23(6):1211-1250, 2018. ISSN 1359-6446. doi:
p11/813596646117333888
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2019.
Wenlin Chen, Austin Tripp, and José Miguel HernándezzLooatto Meta-learning adaptive deep kernel gaussian processes for molecular property prediction. In International Conference on Learning Representations, 2023.
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets. In Advances in Neural Information Processing Systems, 2020.
Thomas M. Cover. Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition. 1965.
Mindy I. Davis, Jeremy P. Hunt, Sanna Herrgard, Pietro Ciceri, Lisa M. Wodicka, Gabriel Pallares, Michael Hocker, Daniel K. Treiber, and Patrick P. Zarrinkar. Comprehensive analysis of kinase inhibitor selectivity. Nature Biotechnology, 29(11):046-1151, November 2011. ISSN 1546-1696. doi: 1990..9.... Number: 11 Publisher: Nature Publishing Group.
Hanna Eckert and Jürgen Bajorath. Molecular similarity analysis in virtual screening: foundations, limitations and novel approaches. Drug Discovery Today, 12(5):225-233: 2007. ISSN 1359-6446. doi: https:// doi.org/ 1rtp://00.000/...dddoo0000000000 $13596446070005299
URL http:://www..taeeeeccc.ccoo///cc
Benedek Fabian, Thomas Edlich, Héléna Gaspar, Marwin Segler, Joshua Meyers, Marco Fiscato, and Mohamed Ahmed. Molecular representation learning with language models and domain-relovant auxiliary tasks, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, 1CML'17, pp. 1126-1135, 1135. JMLR.org, 2017.
Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor W. Coley. Sample efficiency matters: A benchmark for practical molecular optimization. In Thirty-sitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https: /pperrve/ www.earreiieeeneetoouuiiuiiieeee med//EorumiiddyyZaadDoo
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neural processes. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1704-1713. PMLR, 10 Jul 2018. URL mlr.
hatp:///areeeeea...omm html
Francesco Gentile, Jean Charle Yaacoub, James Gleave, Michael Fernandez, Anh-Tien Ton, Fuqiang Ban, Abraham Stern, and Artem Cherkasov. Artificial intelligence enabled virtual screening of ultra-large chemical libraries with deep docking. Nature Protocols, 17(3):672 697, March 2022. ISSN 1750-2799. doi: 10.1038/ (0088//4159900020000655 URL
wwwp:w/waurr.coom
Number:
3 Publisher: Nature Publishing Group.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. CoRR, 2017.
Steffen Herbold. Autorank: A python package for automated ranking of classifiers. Journal of Open Source Software, 5(48):2173, 2020. doi: 10.21105//oss.02177 021 URL
02173
12

---

Published in Transactions on Machine Learning Research (02/2025)
Wouter Heyndrickx, Lewis Mervin, Tobias Morawietz, Noć Sturm, Lukas Friedrich, Adam Zalewski, Anastasia Pentina, Lina Humbeck, Martijn Oldenhof, Ritsuya Niwayama, Peter Schmidtke, Nikolas Fechner, Jaak Simm, Adam Arany, Nicolas Drizard, Rama Jabal, Arina Afanasyeva, Regis Loeb, Shlok Verma, Simon Harnqvist, Matthew Holmes, Balazs Pejo, Maria Telenezuk, Nicholas Holway, Arne Dieckmann, Nicola Rieke, Friederike Zumsande, Djork-Arné Clevert, Michael Krug, Christopher Luscombe, Darren Green, Peter Ertl, Peter Antal, David Marcus, Nicolas Do Hun, Hideyoshi Fuji, Stephen Pickett, Gergely Acs, Eric Boniface, Bernd Beck, Yax Sun, Arnaud Gohier, Friedrich Rippmann, Ola Engkvist, Andreas H. Göller, Yves Moreau, Mathieu N. Galtier, Ansgar Schuffenhauer, and Hugo Ceulemans. MELLODDY: Cross- pharma Federated Learning at Unprecedented Scale Unlocks Benefits in QSAR without Compromising Proprietary Information. Journal of Chemical Information and Modeling, 64(7))2331 2344, April 2024. ISSN 1549-9596. doi: 11001//cc..cinn.... Publisher: American Chemical Society.
URL httpp:///d// 102/10.. jcim.3000799
Weihua Hu*, Bowen Liu*, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations, 2020. URL https://
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai- Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2020.
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf H Roohani, Jure Leskovec, Connor W. Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. In Thirty-fifth Conference on Neural Information Processing Systems Dalasets and Benchmarks Track (Round 1), 2021.
Ruili Huang, Menghang Xia, Dac-Trung Nguyen, Tongan Zhao, Srilatha Sakamuru, Jinghua Zhao, Sampada A. Shahane, Anna Rossoshek, and Anton Simconov. Tox21challenge to build predictive models of nuclear receptor and stress response pathways as mediated by exposure to environmental chemicals and drugs. Frontiers in Environmental Science, 3, 2016. ISSN 2296-665X. doi: 10.3389 waw.s.2215.0000 URL
http:::/ww..rttt.rrrrs..... www.rontieerrrioooorrrrrrreaeeea 331.0..333fffe 2015.
Clemens Isert, Kenneth Atz, José Jiménez-Lune and Gisbert Schneider. Qmugs: Quantum mechanical propertics of drug-like molecules, 2021.
Disi Ji, Robert L. Logan, Padhraic Smyth, and Mark Steyvers. Active bayesian assessment of black-box classifiers. Proceedings of the AAAI Conference on Artificial Intelligence, 35(9):7935 May 2021. doi: n..1609/aaai.v35.3 16968. URL
Wasif Khan, Seowung Leem, Kyle B. See, Joshua K. Wong, Shaoting Zhang, and Ruogu Fang. A comprehensive survey of foundation models in medicine, 2024. URL httg://dwssssh///l//aaattttaa//s
10729
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3d geometry. In International Conference on Learning Representations 2022.
Tiqing Liu, Yuhmei Lin, Xin Wen, Robert N. Jorissen, and Michael K. Gilson. Bindingdb: a web-accessible database of experimentally determined protein protein-ligand binding affinities. Nucleic Acids Research, 35: D198-D201, 12 2006. ISSN 0305-1048. doi: 10. 10.1093/nar/ h/r////a/ggk999.
Xu Luo, Hao Wu, Ji Zhang, Lianli Gao, Jing Xu, and Jingkuan Song. A closer look at few-shot classification again. In International Conference on Machine Learning, 2023.
Lukasz Maziarka, Tomasz Danel, Slawomir Mucha, Krzysztof Rataj, Jacck Tabor, and Stanislaw Jastrzebski. Molecule allention transformer. In Graph Representation Learning Workshop at NewrIPS 2019, 2019.
Oscar Méndez-Lucin Christos Nicolaou, and Berton Earnshaw. Mole: molocular foundation model for drug discovery, 2022. URL
http://b//2211..222/
13

---

Published in Transactions on Machine Learning Research (02/2025)
Yassine Ouali, Adrian Bulat, Brais Matinez, and Georgios Trimiroponlos. Black box few-shot adaptation for vision-language models. In Proceedings of the IEEE/CVI CCF International Conference on Compuler Vision (ICCV), pp. 15534 15546, October 2023.
Fabio Pammolli, Lorenzo Righetto, Sergio Abrignani, Luca Pani, Pier Pelicci, and Emanuele Rabosio. The endless frontier? the recent upsurge of r&dd productivity in pharmaccuticals, 06 2019.
Balazs Pejo, Mina Remeli, Adam Arauy, Mathieu Galtier, and Gergely Acs. Collaborative Drug Discov- ery: Inferenco-level Data Protection Perspective, June 2022. URL http:///araav.or///b//226. 06506
arXiv:2205.06500.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self- supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems, 33, 2020.
Jerret Ross, Brian Belgodere, Vijil Chenthamarakshan, Inkit Padhi, Youssef Mrouch, and Payel Das. Large- scale chemical language representations capture molecular structure and properties. Nature Machine Intelligence, 4(12):1256 1264, 2022. doi:
Johannes Schimunek, Philipp Seidl, Lukas Friedrich, Daniel Kuhn, Friedrich Rippmann, Sepp Hochreiter, and Günter Klambauer. Context-enriched molecule representations improve few-shot drug discovery. In The Eleventh International Conference on Learning Represendations, 2023.
Philipp Seidl, Andreu Vall, Sepp Hochreiter, and Günter Klambauer. Enhancing activity prediction models in drug discovery with the ability to understand human language. Proceedings of the 40th International Conference on Machine Learning (ICML), July 2023.
Aljoša Smajić, Melanie Grandits, and Gerhard F. Ecker, Privacy-preserving techniques for decentralized and socure machine lcarning drug discovery. Drug Discovery Today, 28(12): December 2023. ISSN 1359-6446. doi: www./////...duiis... wtwuudis.222..11000 2023.103820. URL httpp://www..ccennddrrrccc wttp://iiiiettccoocciccccceeeeee articcee/pp//1139944422000333
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Megan Stanley, John F Bronskill, Krzysztof Maziarz, Hubert Misztela, Jessica Lanini, Marwin Segler, Nadine Schneider, and Marc Brockschmidt. FS-mol: A few-shot learning dataset of molecules. In thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günne- mann, and Pietro Liò. 3d infomax improves gnns for molecular property prediction. arXiv preprint arXiv:2110.04126. 2021.
Yan-YunSoe Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference Learning Representations, 2019.
Ruoxi Sun, Hanjun Dai, and Adams Wei Yu. Does gnn pretraining help molecular representation? In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advonces in Neural Information Processing Systems, volume 35, pp. 12096 12109. Curran Associates, Inc., 2022.
Jing Tang, Agnieszka Szwajda, Sushil Shakyawar, Tao Xu, Petteri Hintsanen, Krister Wennerberg, and Tero Aittokallio. Making sense of large-scale kinase inhibitor bioactivity data sets: A comparative and integrative analysis. Journal of Chemical Information and Modeling, 54(3):735 743, 2014. doi: 10.1021/ci400770
14

---

Published in Transactions on Machine Learning Research (02/2025)
Franck Touret, Magali Gilles, Karine Barral, Antoine Nougairède, Jacques van Helden, Etienne Decroly, Xavier de Lamballerie, and Bruno Coutard. In vitro screening of a FDA approved chemical library reveals potential inhibitors of SARS-CoV-2 replication. Scientific Reports, 10(1):13093, August 2020. ISSN 2045-2322. doi: (000000/41190-000000 100008//-15508-000000-4---444
Vict-Khoa Tran-Nguyen, Célien Jacquemard, and Didier Rognan. Lit-peba: An unbiased data set for machine learning and virtual screening. Journal of Chemical Information and Modeling, 60(9):4263 4273, 2020. doi:
(0.1021/acs.jeim.. 0c00150
Alexander Tropsha. Best practices for qsar model development, validation, and exploitation. Molec- ular Informatics, 29(6-7):476-488, 2010. doi:
http://.....0000////200000000000 2000000000...000 htw.://arww..lll....o/////////// ot//:///oo.o///////////////0//// 000.00/mmmm.2200000000 201000061
URL https
Oriol Vinyals, Charles Blundell, Tim Lillicaap, Daan Wierstra, et al. Matching networks for one shot learning.
In Advances in Neural Information Processing Systems, pp. 3630 3638, 2016.
Hongwei Wang, Weijiang Li, Xiaomeng Jin, Kyunghyun Cho, Heng Ji, Jiawei Han, and Martin D. Burke. Chemical--reaction-aae aware molecule representation learning. In International Conference on Learning Representations, 2022. URL http:p////peereeeeeeeeee/eeeeeee wtt.te//oormmm.c6aa3111.....
Yanli Wang, Stephen H. Bryant, Tiejun Cheng, Jiyao Wang, Asta Gindulyte, Benjamin A. Shoemaker, Paul A. Thiessen, Siqian He, and Jian Zhang. PubChem BioAssay: 2017 update. Nucleic Acids Research, 45(D1): D955-D963, 11 2016. ISSN 0305-1048. doi: 10.1093/nar//ee http///////....... URL https:///oi. org/1/...111.. 1093/n0r/// gkw1118
Yaqing Wang, ABULIKEMU ABUDUWEILI, quanming yao, and Dejing Dou. Property-aware relation networks for few-shot molecular property prediction. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (ods.), Advances in Neural Information Processing Systems, 2021.
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. MoleculeNet: A Benchmark for Molecular Machine Learning. October 2018. doi: 1188550/a2iiv1100....
URL http::/arxixi http:://arxiv.orr//b///77......
Jun Xia, Yanqiao Zhu, Yuanqi Du, and Stan Z. Li. A systematic survey of chemical pre-trained models, 2023.
URL https://arxiv.orrt http///axiiv.rrr//////21012/ abtpaab....aaa////a///a/aaaaa..a 11482.1111111411 16484
Jun Xu and Arnold Hagler. Chemoinformatics and drug discovery. Molecules, 7(8):566 600, 2002. ISSN 1420-3049. doi: 10.3390/70800560. URL htttp:://ww..w.t htt.a//ocoo//1000000///////6//
Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. arXiv preprint Xix:222006. 04 113, 2021.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangvang Wang, and Yang Shen. Graph contrastive learning with augmentations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5812- 5823. Curran Associates, Inc., 2020. URL Ste20008eeaa22r122///e55555aaaa2
15

---

Published in Transactions on Machine Learning Research (02/2025)
A Appendix
A.1 Algorithm
Procedure 1 Training of the quadratic probe
Input: A support set S = (7;; Vigg,, the model's weights (E learning rate and shrinkage cocfficient A.
for E for h {0, 1 do
) do
Sk 13: = kN
MR + (11 1) Tuul ExEse (z wx) (z (z-w))) +XI) {Compule the covariance matrices)
end for
+ (wk: Mxfoo((
(o) f [ ExNES Loe ()
for E {0, do
Wk W V w/ Lor (0) {Update the weights with gradient descent) }
end for end for Return +
A.2 Proofs
Proof of Proposition 3.1
Proof. The samples from both classes are linearly separable. As such, there exists a vector such that
VT zi >
if y = 1 yo 0,
vi (o,
(8)
8" 2; <b
where a constant, and we take v such that || = 1. First, by dividing the expression of pe (Y- z 2) by its numerator: Lze yr, θ) log(1 +
IIM 11 w li,,, 2)
(9)
Our goal, is to build set of parameters @()) such that limx Loo U/: O())) 0, and À is an eigenvalue of M, for all k E 1}.
Since ΣA is PSD, we can decompose it as M, = UK with Ug an orthogonal matrix, and Aş BiaggAA......A...d diagonal matrix containing the eigenvalues of Mg- The matrix Ux corresponds to a rotation the vector space to new basis corresponding to the eigenvectors of M2 (the rows of Ux).
We choose M₁ (A) that its first eigenvector is with cigenvalue of λς.o X, and we keep all other eigenvalues and eigenvector constant. We set wy
1)0 if (b + 1)
Wk
if h: 1.
First, let's find an equivalent of the Mahalanobis distance when A + -00:
z wu 1..
1
(z. - (xx=-x)) " A (zi - wk)
2
A
-;-wwk] vv (z, - w)
~x++00 2
2
(v 2i - T ww
~xto 2
16

---

Published in Transactions on Machine Learning Research (02/2025)
Let's start with the case y1 1:
X (1 +t²² v=22
||2 w1-v 1i--iliii Ii- 2
2, bb)
X (1 - (v² - b))
|| Wy. MM ~
Since we chose = 1, the assumption of linear separability in the equation states that v² bb> 0. As
result, (1 + 0" z, b)² =1 (1- 1-"ozzz b))² Hence, for all such that yr
(n" (z, w wi-w m) (x" (z, wyy))
(10)
The same reasoning applics for yi = 0, and we finally obtain for all E {O....,N}:
lim |2 - w1 wi-v-llll -ii-vill,... IM - || wy, li, = lim X-+00
A ( (v" (z: - wi- -vv" (zi - wx))))
(zi-),,))))
A-++00
2
(11)
+00.
Finally, using the equation 11 and the equation 9 we obtain:
lim A
Lee (Zi, Vo, (())) O.
(12)
Note that this proof holds because we could choose direction (an eigenvector of Ms) common all data points, obtained from the lincar scparability assumption the data.
Proof of Remark 3.2
Proof- It straightforward to that for given data point 2i, (ec yi: Q) fi (zz, Ue. θ) f2 (z,, θ).
Let's first consider ft. Since M₃ is PSD,
|| wx ll.. = DR (z+ w < 1 -w2112
(13)
Hence when [ 0 fi 0.
Then, let's consider fz.
mrg f2 (z,, 0)= arg min log M..1e (c.ke(0.1)
Σ exp (-1|z: wx (la.)
kE(0.1)
arg min Me kE(0.1)
Σ
exp (-1|z:
[ilzz-wwid - wall(a
arg min exp (-1|zi - wxll2a.) Mr
ke
Minimizing fa is then equivalent to maximizing || wallas. for all {0, 1}, and:
|| - w (z, 7 wxx 2 Ok.minll uw 11.
(14)
where Ok,min is the minimal cigenvalue of M
As a result, f2 is minimized when for all E (0, 1} ak,min + +oc, and hence when M 1 + +oc.
This decomposition illustrates the role of each term in the cross-entropy loss, the first trying to reduce the distance between each point belonging the same class, the other inereasing the distance between all points.
17

---

Published in Transactions on Machine Learning Research (02/2025)
Proof of Proposition 3.3
Proof. For molecular embedding 24, the expression of f2 (z1, θ) is:
fa (zi, 6) = loog
Σ
exp (-1|z; we liiee
kEf0,11
We use the following properly of the log-sum-exp function:
max {-1|z; we Ia I log
Σ
exp (-112; wxllire) max {-11z; wellir kE log
(e(1)1)
We then majorize the maximum function:
max {-1|z; - wx lil (E = min - - J
S 2
Σ
|2 - w
ke (0
Furthermore, || - 11zz-wllimn = =I (zi - -w>)11.
And |Dx (z: - -1 < | - ww 113, where ||z - wx constant w.r.t M
f2 (z =
Σ
1 =0
Σ
|
AE
AE(0,1)
Let's study the asymptotic behavior of fo (2)
S
1f2 0|1=
Σ KE 10.1
log det M2
[S]
Sx
log
[S]
kE (0.1)
jE(l....))
[S&]
< d Σ
max [o ok. (ksl)je(r...))
[S]
k
Where d is the dimension of the data, and oke is the j-th eigenvalue of Mg- Since all eigenvalues of M₃ for {0, 1} grow infinity, max {|log oh. (. O(log max {oks } jE sE(1...))) Finally, all precision matrices are PSD, hence for ls {0,13: v EyjE(L.nn) oh |M 2 max fok.J (E
As result. Sa (z,, θ) O - log MM
Proof of Remark 3.4
Proof. First the log-likelihood of a gaussian distribution of mean wy and of precision matrix MA, evaluated on z is:
Lwr.Me (z) = (log det M4 - -||2-
-walhan) -11=-2xlfm.) + c,
where constant.
18

---

Published in Transactions on Machine Learning Research (02/2025)
For given support set S, let's compute the modified loss:
1
|Sk log det (M&)
(fi(zz Vr. 6) f2(0)) =
1|
wy IM;
S
[|]
ES
iES
kE(0.1)
1
>
Wy. 1lsa,
|S log det (Mt)
[
|S|
kE(0,11
Σ
(1
wt llM log det (M&))
S
kE LES
For a class E {0, we are trying to minimize the negative log-likelihood of a Gaussian distribution evaluated on the support set Sk, with mean wy and precision matrix MІ.
A.3 Divergence M;
We study the evolution of the value of M2 when optimized with gradient descent (Free-Opt) and using the quadratic probe (q-probe). The results are reported in Figure 5 where we see that the maximal eigenvalue of Mg. which diverges with gradient descent, while remaining stable using the quadratic probe (we recall that the molecule's s embedding are distributed on the unit sphere of Rd).
[5| 16
[S] 128
Moximal aigevvalue of Ms
25
20
15
grabe
10
Free-opt
5
20
10
10
10
20
Epoch
Epoch
Epoch
Epoch
Figure 5: Evolution of the maximal eigenvalue of M2 along training when optimized with gradient descent (Free-Opt) or with the quadratic probe (q-probe).
A.4 Effect of À on the quadratic probe's performances
Estimating the covariance matrix of the data is challenging in the few- shot setting. To regularize the empirical covariance matrix, diagonal shift to make the inverse of the covariance matrix well-defined and more stable. The optimal value of À varies with the size of the support set, as shown in the Figure 6. We see that the optimal value of decreases with the size of the support set, which is expected as the empirical covariance matrix becomes more stable with more samples.
Optimal value Af foc support set size 0.4
All hyperparameters of the meta-learning methods constant over the support set sizes. In particular, all meta-learning methods are trained with fixed support set size (64 for Protonct, for instance) and evaluated on all support set sizes. We chose similarly to use single value of À = 0.2 across all support set sizes. However, while modifying a hyperparameter in meta-learning method usually requires retraining the model, only the few-shot adaptation step needs be re-run when changing À in the quadratic probe, for instance.
0.1
Support set size
Figure 6: Optimal values of Amaha found during the hyperparameter search on FS-mol's validation set. (computing Σg e (1- -- empiiriall AI)
19

---

Published in Transactions on Machine Learning Research (02/2025)
A.5 Out- Out-of-domain experiments details
Some few-shot learning methods for drug discovery proposed out-of-
domain scenarios to evaluate their models. Still, the domain shifts involved are often unrealistic, such evaluating model on Tox21 (Huang et al. 2016), datasct containing chemicals that are not drug-like (pesticides, food additives, ...), or evaluating models' performances on material design lasks. However, models trained and designed for activity prediction would marginally be used such drastically different scenarios, giving limited insights into the model's performances. To propose more realistic domain-shift experiments, we decided consider scenarios using molecular inputs that do drastically differ from the ones seen in training, well considering tasks related to activity prediction.
A.5.1 Drug-targen tang-tttagtiineracttonn details
Creation of Few-shot tasks
1.0
10
0.8
8
Proportion
Density
0.6
6
0.4
4
2
0.2
0
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
-10
-9
-8
-7
-6
-5
Distribution of the tasks' positive proportion in each dataset
Threshold cumulative distribution (pXC50) each dataset
DAVIS
BindingDB_KK
BindingDB Ki
BindingDB_ICS,
KIBA
Figure 7: Imbalance of the DTT datasets' task classes and distribution of the thresholds used to obtain the classification labels in each dataset.
As explained in Section 4.3.1 we introduce various new datasets on which we compared the various methods. To adapt the various datasets to the few-shot learning scenario, followed five steps:
Identifying tasks already in the FS-mol training set is not trivial, as the FS-mol dataset is large. To do so, removed every task in the DTT datasets that contained more than 30% of molecules already in the FS-mol training set.
We then removed every SMILES that cannot be processed by the RDKit library or ended up with a molecule containing an atom with valence greater than 6.
molecule in a given task can be associated with several different activity values on the same target. We decided to remove any molecule associated with multiple measurements in task.
We separated the molecules into active and inactive classes by computing the median of the activity values in the task, clipped between a maximum pXC50 of 9 and a minimum pXC50 of 5, or for BindingDB Ki and BindingDB_ICe IC50. We then removed molecules whose activity was equal to the threshold chosen in the previous step.
20

---

Published in Transactions on Machine Learning Research (02/2025)
5. Finally, dropped all tasks with less than 60 molecules, more than 5000, and removed all tasks whose label distribution was too imbalanced (a positive proportion outside of the [0.1, 0.6] range).
The resulting datasets provide different class imbalancos, illustrated in the Figure 7 inforred by the choice of the thresholds the activity numeric values for each task.
Complementary results and limitations
We provided in the paper the results on DAVIS, KIBA and BindingDB Kd, and we provide supplementary results on BindingDB IC50 and BBndingDB Ki in the Figure 8 We see that the fine-tuning methods presented do not obtain competitive results, which implies that pretraining procedure resulted in model that is still sensible to shifts in the target type. Finally, CLAMP's fine-tume ane-tuning outperforms our meth- ods, which rely on a standard multi-task model. This can explained by the fact that CLAMP is trained on lextual descriptions of the assays. Obtaining tex- tual information about the assays allows the model to assimilate knowledge that is sharcable and applica- ble across various (argel types, polentially leading to better robustness to this domain shift.
Performances on DTI tasks
0.250
AAUPRC
0.225
0.200
0.175
0.150
Binding Ki
BindingDB IC50
clamp
q-probe
protonet
I-probe
adkt
simsearch
Figure 8: Performances of the considered methods on the BindingDB Ki and BindingDB_ IC50 datasets. Few-shot models do not significantly outperform the similarity search, and CLAMP is the best fine-tuning method.
ADKT-IFT obtains competitive results, showing strong nosustnness to the lype of larget considered, but none of the methods achieved significantly better results than the similarity search. This underscores the significant challenge for the presented few-shot methods in effectively adapting to new targets.
A.5.2 Library screening details
Crcation of Few-shot tasks
This section provides more details on adapting the HTS datasets we used in our experiments. HTS datasets correspond to assays usually evaluated early in the drug discovery process. These tasks often comprise many compounds with very few active molecules to identify. We considered datasots available through the Therapeutic Data Commons (TDC) (Huang et al., 2021] plateform (Touret et al. 2020 Wu al. 2018 Butkiewicz et al. 2013), and various tasks from the LIT-PCBA dataset (Tran-Nguyen al. 2020), a database built on 149 dosc response PubChem bioassays (Wang et al. 2016).
Rumning these experiments on such large number of molecules would be too computationally expensive. As result, we down-sampled all datasets to 30000 molecules.
We processed each dataset as follows:
1. Only compounds whose SMILES can be processed by the RDKit library are kept, and we removed all molecules whose SMILES ended up with a molecule containing an atom with a valence greater than 6.
To create tasks of 30000 molecules, we first kept all active molecules if the resulting proportion of active molocules remained under 7%. Otherwise, we sampled the active molecules keep the same proportion of active molecules in the original dataset.
3. We then randomly sampled inactive molecules to obtain at most 30k molecules.
We removed all datasets initially composed of more than 300k compounds. We also did not include the dataset SARSCoV2 3CI.Pro Diamond dataset, it corresponds to fragment screen, using different.
21

---

Published in Transactions on Machine Learning Research (02/2025)
Table 3: Description of the datasets used for the library screening task.
BENCHMARK
DATASET NAME NAME
TARGET TYPE IF KNOWN
HIT PROPORTION
DATASET SIZE
TDC
SARSCOV2 VITRO TOURET
NOT KNOWN
5.9%
1500
HIV
NOT KNOWN
4.8%
30K
OREXINI RECEPTOR
GPCR
0.77%
30K
MI MUSARINIC RECEPTOR AGONIST
GPCR
0.62%
30K
MI MUSARINIC RECEPTOR ANTAGONIST CAV3 T-TYPR CALCIUM CHANNELS
GPCR
1.2%
30k
ION CHANNEL
4.8%
30k
ALDHI
OXIDOREDUCTASE
4.9%
30k
ESRI ANTAGONIST
NUCLEAR RECEPTOR
1.9%
5K
GBA
HYDROLASE
0.54%
30k
LIT-PCBA
MАPKТ
KINASE
1.0%
30k
MTORCI
PROTEIN COMPLEX
1.9%
30k
OPRKI
GPCR
0.066%
30k
РKM2
KINASE
1.8%
30k
PPARG
NUCLEAR RECEPTOR
0.44%
5K
TP53
ANTIGEN
1.8%
4k
VDR
NUCLEAR RECEPTOR
2.9%
30k
input type than the like compounds considered far. The Table 3 describes the datasets used for the library screening task. The datasets are diverse, with few datascts containing about 5% of hits (SARSCOV2 Vitro Touret, HIV, CAV3 T-lype Calcium channels, ALDH1), and the others having less than 2% of hits in general.
The Figure revcals how different the molccular structure between hits can be these datascts. In particular, computed the Tanimoto similarity between all active compounds in each dataset. Also, we measure the maximum similarity between
active compound and the other hits in the task. In particular, the hits structurally different in tasks such as PPARG Orexinl receptor. As result, retrieving active molecules using the information of the hits available in the support set can be challenging. In some other tasks, such as OPRK1 or HIV, some hits share similar structures. The heterogeneity of the tasks motivates the choice of using rankings as a metric, as the models' performances on each task can be drastically different.
Complete results
line plot displaying the evolution of the hitrate when the number of molecules selected increases. The datasets ordered according to the value 75% quantile of the maximum Tanimoto similarity between active compound and the other hits in the task. This can be interpreted as mild approximation of the task's difficulty (as it does not consider similarity to inactive compounds or the proportion of hits).
Results on all datascts presented in the figure 10 and 11
Interestingly, in the estimated 'hardest' tasks, when the support set size increases, the performances of some models (such as the linear probe) do not always increase (such as on the SARSCOV2 Vitro Touret the ALDH1 datasets). This can be explained by the fact that the hits are structurally different. Indeed, adding a new hit to the support set can be seen as adding noise to the support set, adding molecule whose structure does not help to find a common representation for all hits. This is especially the case on the SARSCOV2 Vitro Touret dataset.
22

---

Published in Transactions on Machine Learning Research (02/2025)
ALDH1
TP53
antagonists
SARSCOV2 Vitro
ESR1
MusC.
MAPK1
ESR1
MTORC1
VDR
orexin1 receptor
Musc. rec. ant.
MTORC1
GBA
PPARG
cav3 channels
PKM2
cav3
TP53
PKM2
HIV
SARSCoV2 Vitro
Musc. rec. agonists
OPRK1
GBA
OPRK1
similarity
neighbor between positive examples
Figure 9: Distribution of the Tanimoto similarity between active compound and its nearest active neighbour in the HTS datasets (left). Similarity matrices between the active compounds of the HTS datasets (right).
23

---

Published in Transactions on Machine Learning Research (02/2025)
[s] 32
[|] 64
[s]
128
[s] 16
[s|]
|st 128
Hitrate (1%
NTTORI Hitrate 196)
1P53
Musi: roc. antagonins Hitratt (X8)
OPRK] Hitrate (%)
Hitrate (96)
PPANG Hitrate 196)
HIV
20
10
oresint receator Hitrate (96
GBA Hitratt (%)
50
ESK1 Hitrate (99)
NAPKI Hirrate 199)
2
00
10
cavd channels Hitrote (%)
ALDRI Hitrott (%)
Musc rec, agonitts
PI04 Hitrate 186)
Hitrate (8)
N
Hitrate (%)
VOR
1.
54R5CO02 Hitrate
10
10
5
10
10
top-ke 1op-
top-k%
top-k%
top-k36
top-k5
top-k36
top-k%
top-k%
simsearch
protonet
adkt
proobe
Figure 10: Evolution of the hitrate with the percentage of the dataset selected, on each dataset for different support set sizes, when 5% of the support set are hits.
24

---

Published in Transactions on Machine Learning Research (02/2025)
[s]
[|] 64
[S]
128
[s] 16
[s|]
|st 128
Hitrate (1%)
ATTORI Hitrate 196)
TP53
Must: roc. antagoninss Hitrott (X))
Hitrate (%)
OPRK]
Hitrate (16)
PPANG Hitrate 196)
HIV
20
10
oresint receator Hitrate (96
GBA Hitratt (%)
ESK1 Hitrate (99)
MAPKI Hirrate 199)
00
10
cavd channels Hitrote (%)
ALDRI Hitrott (%)
Musc rec, agonitts
PI04 Hirrate 186)
Hitrate (8))
N
Hitrate (%)
10
VOR
1.
54RSCO02 Hitrate
10
10
5
10
10
top-ke top-k%%
top-k%%
top-k%
top-k36
top-k5
top-k36
top-k%
top-k%
simsearch
protonet
adkt
proobe
Figure 11: Evolution of the hitrate with the percentage of the dataset selected, on each dataset for different support set sizes, when 10% of the support set are hits.
25

---

Published in Transactions on Machine Learning Research (02/2025)
A.6 Performances of other molecular pretrained backbones
Table 4: Performance comparison of pretrained with different support set sizes trained with linear probe, compared to the linear probe applied to multi-task backbone.
Model
[ 16
[|]
32
[|]
64
[S] 128
MolBert (Fabian et al. 2020
0.045
0.058
0.074
0.105
Mttr (Sun et al. 2022
0.048 0.00G
0.060
0.00G
0.077
0.000
0.110
0.007
ChemBerla-MLM.. AAhad el al. 2022
0.049 1 0.005
0.063
0.006
0.080
0.005
0.113
1 0.007
ThreeDtnfomax (Säärk et al. 2021
0.050 + 0.004
0.067
0.005
0.081 + 0.000
0,108 + 0.006
Context (Sun et al. 2022
0.052 i: 0.005
0.067
0.005
0.082 + 0.005
0.113
+ 0.007
GPT-GNN (Hu al. 2020
0.056 =: 0.005
0.070
0.005
0.085 +: 0.005
0.114
A: 0.000
MolFormer Ross et a.. 2022
0.058 0.006
0.075
0.092
0,125
0.008
MAT (Maziarka et al. 2019
0.052 0.006
0.069
0.092
0.136
GROVER [Rng ct al. 2020
0.068
0.083
0.104
0.133
InfoGraph Sun et al.
2019
0.072 0.006
0.088
0.006
0.106
0.007
0.137
1 0.007
GraphLog Xu et al.
2021
0.071 + 0.006
0.089
0.006
0.111
0.005
0,138
+ 0.007
GraphCL You al., 2020
0.075 + 0.006
0.092
0.DO8
0.112 + 0.007
0.144
: 0.008
GraphMVP Liu et al.
2022
0.078 + 0.006
0.097
0.007
0.118 + 0.007
0.151
+ 0.008
MolR WWan et al.
2022)
0.094 i: 0.006
0.114
0.007
0.137 : 0.007
0.170
1: 0.008
ChemBerta-MTTH
Ahinad et al.
2022
0.098 0.00G
0.121
0.147
0.183
GNN-MT
0.112 0.00e
0.144
0.177 0.008
0.223
0.009
CLAMP (eidl et al.
2023
0.202 i 0.009
0..288
0.008
0.248 0.009
0.271
1 0.009
L-probe (ours)
0.224 + 0.010
0.252 + 0.010
0.273 0.010
0.301
0.D09
In the recent. years, several methods were introduced proposing different ways to pre-train models in drug discovery. However, most these models were not evaluated in a few-shot setting. We adapted several architectures on the tasks of FS-mol with the lincar probe, and compare the results the three methods already evaluated on this benchmark, that have been pretrained using the training set of FS-mol. The 14 backbones selected for these experiments et al.
pretrained on various datasets, such as PubChem (Wang
2016) GEOM (Axelrod & Gómez- Bonnbrelll 2022) and QMugs{Isert et al. 2021 )), USPTO (Wang ct al.
2022 and we used the weights shared publicly by the corresponding authors for each model.
We notice that none of the models enable competitive few-shot performances. This is most probably due to the fact that non of these approaches were using data originating from this benchmark there pre-training phase. Among these models, we notice that ChemBerta-MITP (Ahmad et al., 2022) achieves the best performances, model pretrained with supervised multitask pretraining objective.
26


Reviews:
[{"review_id": "sQP9YAmCL2", "reviewer": "Reviewer_WriN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* The motivation is clear and the topic important and relevant.\n\n* The work takes the black-box setting into account without access to model weights in downstream applications. In the context of large pre-trained models and the costs of serving such models, considering this setting is sensible.\n\n* The authors do not only include state-of-the-art baselines but further include classical baselines such as kNNs, random forests and simple GNN baseline. When working on tasks with very few data points, adding such baselines is a very good practice. In my opinion, this strengthens the impact of this work.\n\n**Weaknesses:**\n* A large portion of the paper is concerned with the quadratic probe. Using this approach appears much more involved as the linear probe. In particular, the fact that optimizing the quadratic probe with gradient descent leads to degenerate solutions indicates to me that this approach is less robust in real-world scenarios. And yet, the performance of the quadratic probe does not seem to lead to significant improvements over the linear probe. In Table 1 and Table 2 (QSAR datasets), all linear probe results are within one standard deviation of the quadratic probe. In light of these results, the authors should add more discussion on the benefits of the quadratic probe over the linear probe. I want to mention, however, that this does not invalidate the claim made by the authors that the quadratic probe performs better than the linear probe. I do recognize that the quadratic probe consistently leads to better results. I merely wonder whether the additional effort in optimizing the quadratic probe is justified by the improvements. There may not be a perfect answer to this question, but I do recommend to add a more nuanced discussion on this limitation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors demonstrate that a simple pre-training/fine-tuning strategy with probing leads to competitive results with far more involved meta-learning approaches. In addition, probing has the benefit of being applicable in black-box settings where the pre-trained model is only accessible through an API. The authors further develop an alternative to the standard linear probing, called quadratic probing, a generalization of linear probing that is more expressive in separating data and expected to perform better. As the authors show, a downside of quadratic probing is that it cannot be successfully optimized with gradient descent. To still make use of its benefits, the authors develop an approach with block coordinate optimization. Empirically, the block coordinate optimization leads to a more robust training behavior and to a competitive performance of quadratic probing when compared to state-of-the-art meta-learning approaches. In addition, both linear and quadratic probing perform strongly in a domain shift setting with unbalanced data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Important requested changes:**\n* Adding a more nuanced discussion on the benefits of the quadratic probe over the linear probe, considering that the linear probe performs very closely to the quadratic probe but is conceptually simpler and easier to optimize with standard procedures (e.g., gradient descent).\n\n**Minor remarks that would strengthen the work but are not critical to my recommendation:**\n* The authors show a clear understanding of drug discovery and the applications of machine learning therein. Since many ML researchers may be interested in contributing to this field but lack the necessary background to understand the broader task that drug discovery and ML-aided drug discovery tries to solve, the authors could add a background section (perhaps to the appendix), outlining the overall drug discovery process and indicate where ML approaches could make useful impact. In addition, such a section would provide space for defining some terminology that is common in drug discovery but not known to the broader ML community. I think this would greatly enhance the accessibility of the author's research.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cei1zgeR5l", "reviewer": "Reviewer_ANKt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The problem of interest, molecular few-shot learning, is an important topic in practical applications, e.g., drug discovery.\n\n2. The method achieves good (nearly state-of-the-art) performance on various molecular few-shot learning tasks.\n\n3. The authors performed useful analysis on the effectiveness of the proposed method.\n\nWeakness:\n\n1. My biggest question is \"how\" linear/quadratic probing is different from GNN-MT in the original FS-Mol paper. Linear/quadratic probing (in this paper) seems very similar to GNN-MT; the pre-training objective is the same, and the only difference is the fine-tuning scheme (this paper freezes the backbone while GNN-MT fine-tunes the backbone). However, the reported performance is dramatically different in Table 1. Could the authors provide an analysis on this phenomenon?\n\n2. Compared to linear probing, the improvement of quadratic probing is marginal. Also, it seems that quadratic probing requires more complexity than linear probing (due to block-coordinate descent and the estimation of covariance matrix's inverse).\n\n3. Empirical support of the claim on quadratic probing is weak. The authors claimed that linear probing yields \"degenerate solutions\", but I could not find the definition of degenerate solutions and how quadratic probing can resolve it. Also, the authors claimed that the proposed quadratic probing can prevent the divergence of the eigenvalues of M_k, but there is no empirical support.\n\n4. This paper is poorly formatted, e.g., \"))\" at the end of the reference and inconsistent \".\" at the end of the sub-subsection.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a simple yet effective baseline for molecular few-shot learning. The approach is very simple; it performs linear/quadratic probing of the pre-trained model. Experimental results show that the proposed approach achieves nearly state-of-the-art performance on molecular few-shot learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see Weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UZ33IAblpM", "reviewer": "Reviewer_gzkF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Although I know a bit about molecular property prediction and retrosynthesis, I am not entirely familiar with drug discovery tasks, meta-learning, or probing techniques. I tried to read the article as carefully as I could but I still might have missed or misunderstood some details.\n\n## Strengths\n- The proposed quadratic probing method seems solid and well-developed. Experiments also show its advantage over the linear probing method.\n- The experiments look solid in terms of tasks. The discussion seems reasonable.\n- The authors provide well-structured code which could facilitate further works.\n\n## Weaknesses\n- The narration of this article is not very fluent and could use some improvement. Please check the \"Requested Changes\" for details.\n- Some of the claims are not well-supported or potentially outdated.\n- In this article, the authors selected only a small scope of fine-tuning methods as the baseline. But there are some foundation models in the chemical domain released in recent years, such as those mentioned in [1, 2, 3]. The existence of such backbone models might impact the authors' claim \"a strong baseline\" in the title. Therefore it might be good for the authors to also use these models as baselines, or at least give a convincing discussion about whether these models are not applicable in the authors' setup.\n\n## Questions\n- Do the distributions of the support set and query set overlap? This information should be mentioned when these sets are introduced.\n- For equation (1), can you give a reference to this type of linear probing classifier? I personally have never seen people put the bias term outside of the temperature scaling function so I want to verify whether this is correct.\n- In section 3.3, \"Instead of the cosine similarity used in the linear probe\": I'm not sure whether the authors referred the \"cosine similarity\" to equation one. If so, I think it may not be accurate as temperature and bias terms are included.\n- I'm also confused by the statement in the abstract \"... and removing the need for specific episodic pre-training strategies\". It seems that the proposed methods still utilize a pre-trained model as the backbone?\n\n[1] Méndez-Lucio, Oscar, Christos Nicolaou, and Berton Earnshaw. \"MolE: a molecular foundation model for drug discovery.\" arXiv preprint arXiv:2211.02657 (2022).  \n[2] Xia, Jun, et al. \"A systematic survey of chemical pre-trained models.\" arXiv preprint arXiv:2210.16484 (2022).  \n[3] Zhou, Gengmo, et al. \"Uni-mol: A universal 3d molecular representation learning framework.\" (2023).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This article revisited the linear probing approach for drug discovery-related tasks, claiming that it is still a strong baseline and performs on par with the prevailing meta-learning-based techniques. In addition, the authors proposed a quadratic probing method, which substitutes the cosine-like similarity calculation in linear probing by the Mahalanobis distance. To resolve the issue of the Frobenius norm of the covariance matrices going to infinity when trained on gradient descent, the authors suggest block coordinate optimization, which breaks down the cross-entropy loss and updates the mean and covariance parameters iteratively. Experiments show that the probing methods have good performance on property prediction tasks as well as in the OOD settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Narration\n- In \"Introduction -> Meta-learning\", the second paragraph basically claims \"meta-learning is not applicable to black-box models as their weights are not accessible; but black-box models are valuable for drug discovery\". There are redundancies around the first point in the narration, and the second point is not supported by citations.\n\n- The \"Introduction\" section does not discuss enough why the black-box setting is essential for drug discovery. Relevant citations are missing.\n\n- Figure 1: Critical legends and explanations are missing. I'm not sure how to interpret this figure, nor can I gain any insights about the difference between these two classifiers.\n\n- It is advised to add equation (9) to the article body, probably around equation (2) or (3) for a clearer narration.\n\n- It also would be good if the authors could add a figure or algorithm illustrating the training process of the proposed quadratic probing method.\n\n- The authors should emphasize the difference between the meta-learning and fine-tuning approaches somewhere in the introduction.\n\n### Supports\n- In \"Introduction -> Low data regime\", it seems that none of the three cited papers (Schimunek et al. (2023); Chen et al. (2023); Altae-Tran et al. (2017)) supports the authors' claim \"where meta-learning seems to appear as the default and best solution\".\n\n- In \"Introduction -> Competitive fine-tuning methods\", the authors only cited Stanley et al. (2021) to support their claim \"fine-tuning approaches yielded subpar performances\". But the majority work of that paper was conducted in August 2021, which was a \"long\" while ago in terms of the development of foundation models. A quick search on Google gives more advanced foundation models for drug discovery, such as [1], which may invalidate the authors' claim.\n\n- Some related works such as [1] are missing. A recent survey [2] also mentioned some advanced techniques for drug discovery. If we widen the scope to molecular property prediction, then another survey [3] also showed a bunch of very related backbone models and their fine-tuning performance.\n\n### Format\n- It seems that the authors are using the citation format \"(\\citet{a}, \\citet{b})\". Instead, the authors should use \"\\citep{a,b}\".\n- In the LaTeX source code, quotation marks are \"``\" and \"''\" for left and right quotations.\n- \"Notations\" should not be included in the \"Related Work\" section.\n- The authors do not entirely follow the \"Default Notations\" defined in the TMLR & ICLR templates. Although it is not necessary, authors are suggested to follow those conventions to make readers' lives easier.\n- The authors should not re-use the same notation for different concepts, e.g., $x_i$ are included in both support and query sets.\n\n[1] Méndez-Lucio, Oscar, Christos Nicolaou, and Berton Earnshaw. \"MolE: a molecular foundation model for drug discovery.\" arXiv preprint arXiv:2211.02657 (2022).  \n[2] Khan, Wasif, et al. \"A Comprehensive Survey of Foundation Models in Medicine.\" arXiv preprint arXiv:2406.10729 (2024).  \n[3] Xia, Jun, et al. \"A systematic survey of chemical pre-trained models.\" arXiv preprint arXiv:2210.16484 (2022).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously recommend acceptance, particularly in light of the black-box setting where probing is a natural fit. For the camera ready version, the reviewers are encouraged to incorporate any relevant feedback from the reviewers in order to improve the paper.
Decision: Accept (poster)