Meta-learning Adaptive Deep Kernel Gaussian Processes for Molecular Property Prediction
Authors: Wenlin Chen
Austin Tripp
José Miguel Hernández-Lobato
Venue: ICLR 2023 poster (2023)
Keywords: meta-learning
few-shot learning
Gaussian processes
deep kernel learning
bilevel optimization
chemistry
molecules
drug discovery

Abstract: We propose Adaptive Deep Kernel Fitting with Implicit Function Theorem (ADKF-IFT), a novel framework for learning deep kernel Gaussian processes (GPs) by interpolating between meta-learning and conventional deep kernel learning. Our approach employs a bilevel optimization objective where we meta-learn generally useful feature representations across tasks, in the sense that task-specific GP models estimated on top of such features achieve the lowest possible predictive loss on average. We solve the resulting nested optimization problem using the implicit function theorem (IFT). We show that our ADKF-IFT framework contains previously proposed Deep Kernel Learning (DKL) and Deep Kernel Transfer (DKT) as special cases. Although ADKF-IFT is a completely general method, we argue that it is especially well-suited for drug discovery problems and demonstrate that it significantly outperforms previous state-of-the-art methods on a variety of real-world few-shot molecular property prediction tasks and out-of-domain molecular property prediction and optimization tasks.

Published as a conference paper at ICLR 2023
META- ADAPTIVE DEEP KERNEL GAUSSIAN PROCESSES FOR MOLECULAR PROPERTY PREDICTION
Wenlin Chen
Austin Tripp University of Cambridge j jtt212ccamm ac.uk
José Miguel Hernandez-Lobato University of Cambridge jmh2336 cam ac.
University of Cambridge MPI for Intelligent Systems wc33 70 cam. ac. uk
ABSTRACT
We propose Adaptive Deep Kernel Fitting with Implicit Function Theorem (ADKF- IFT), a novel framework for learning deep kernel Gaussian processes (GPs) by interpolating between meta-learning and conventional deep kernel learning. Our approach employs a bilevel optimization objective where we meta-learn generally useful feature representations across tasks, in the sense that task-specific GP models estimated on top of such features achieve the lowest possible predictive loss on average. We solve the resulting nested optimization problem using the implicit function theorem (IFT). We show that our ADKF-IFT framework contains previously proposed Deep Kernel Learning (DKL) and Deep Kernel Transfer (DKT) as special cases. Although ADKF-IFT is a completely general method, we argue that it is especially well-suited for drug discovery problems and demonstrate that it significantly outperforms previous state-of-the methods a variety of real-world few-shot molecular property prediction tasks and out-of-domain molecular property prediction and optimization tasks.
INTRODUCTION
Many real-world applications require machine learning algorithms to make robust predictions with well-calibrated uncertainty given very limited training data. One important example is drug discov- ery, where practitioners not only want models to accurately predict poochemicc/phhysccchhemee properties of molecules, but also want use models to guide the search for novel molecules with desirable properties, leveraging techniques such as Bayesian optimization (BO) which heavily rely on accurate uncertainty estimates (Frazier 2018). Despite the meteoric rise of neural networks over the past decade, their notoriously overconfident and unreliable uncertainty estimates (Szegedy et al., 2013) make them generally ineffective surrogate models for BO. Instead, most contemporary BO implementations use Gaussian processes (GPs) (Rasmussen & Williams 2006) surrogate models due to their analytically-tracee
and generally reliable uncertainty estimates, even on small datasets.
Traditionally, GPs are fit on hand-engineered features (e.g., molecular fingerprints), which can limit their predictive performance on complex, structured, high-dimensional data where designing informative features is challenging (e.g., molecules). Naturally, number of works have proposed to improve performance by instead fitting GPs on features learned by a deep neural network: family of models generally called Deep Kernel GPs. However, there is no clear consensus about how to train these models: maximizing the GP marginal likelihood (Hinton & Salakhutdinov 2007: Wilson al.) (0166b) has been shown overfit on small datasets (Ober et al. (021), 1), while meta-learning (Patacchiola et al. (2020) and mully-Bayesian approaches (Ober et al. 2021 avoid this at the cost of making strong, often unrealistic assumptions. This suggests that there is demand for new, better
techniques for training deep kernel GPs.
In this work, we present novel, general framework called Adaptive Deep Kernel Fitting with Implicit Function Theorem (A (ADKF-IFT) for training deep kemne GPs which we believe is especially well-suited to small datasets. ADKF-IFT essentially trains a subset the model parameters with a meta-learning loss, and separately adapts the remaining parameters marginal likelihood. In contrast to previous methods which use a single loss for all parameters, ADKF-IFT is able to utilize the implicit regularization of meta- meta-learning to prevent overfitting while avoiding the strong assumptions of a pure meta-leaminn approach which may lead to underfitting. The key contributions and outline of the paper follows:
each task using maximum

---

Published as a conference paper at ICLR 2023
1. As our main technical contribution, we present the general ADKF-IFT framework and its natural formulation as a bilevel optimization problem (Section B.I). then explain how the implicit function theorem (IFT) can be used to efficiently solve it with gradient- -asedd methods in a few-shot learning setting (Section 3.2).
2. We show how ADKF-IFT approaches based purely on single-task learning [Wilson et al. (216bb or purely on meta- learning (Patacchiola et al., 2020) for training deep kernel GPs (Section 3.3)
be viewed generalization and unification of previous
3. We propose specific practical instantiation of ADKF-IFT wherein all feature extractor parameters are meta-learned, which has clear interpretation and obviates the need for any Hessian approximations. We argue why this particular instantiation is well-suited to retain the best properties of previously proposed methods (Section 3.4).
4. Motivated by the general demand for better GP models in chemistry, we perform extensive empirical evaluation of ADKF-IFT on several chemical tasks, finding that it significantly improves upon previous state-of-the-art methods (Section 5).
2 BACKGROUND AND NOTATION
Gaussian Processes (GPs) are tools for specifying Bayesian priors over functions (Rasmussen & Williams, 2006). A GP(mm c)) is fully specified by a mean function ma(-) and symmetric positive-defiite covariance function co(:: .....)). The covariance function encodes the inductive bias (e.g.. smoothness) of a GP. One advantage of GPs that it is easy to perform principled model selection for its hyperparameters 0 E θ using the marginal likelihood p(y X, 0) evaluated on the training data (X.y) y) and to obtain closed-form probabilistic predictions p(y, X.,, X, y, 0) for the test data (X., y.); we refer the readers to Rasmussen & Williams (2006) for more details.
Deep Kernel Gaussian Processes are GPs whose covariance function is constructed by first using neural network feature extractor fs with parameters & & to create feature representations
h h= fx)x)) hn (x') of the input points x, x', then feeding these feature representations into standard base kernel (h, h') (e.g., RBF kernel) (Hinton & Salakhutdinov 2007 Wilson et al.
2016b a Bradshaw et al. 2017: Calandra et al. 2016). The complete covariance function is therefore ().
Ky x, x Go
(f (x),
with learnable parameters 1/5
Few-shot Learning refers to learning on many related tasks when each task has few labelled examples (Miller et al. 2000 Lake et al. 2011). In the standard problem setup, one is given a set of training tasks D = (i
(a meta-dataser) and some unseen test tasks D. = {T.). Each task [(x, yt) H NT is a set of points in the domain X' (e.g., space of molecules) with corresponding
labels (continuous, categorical, etc.), and is partitioned into support set ST C T for training and query set Q TV ST for testing. Typically, the total number of training tasks T = [D is large, while the size of each support set [T is small. Models for few-shot learning are typically trained to accurately predict Qr given Sy for T E DD during a meta-training phase, then evaluated by their prediction error on 2,, given Sy. for unseen test tasks T, D. during a meta-testing phase.
ADAPTIVE DEEP KERNEL FITTING WITH IMPLICIT FUNCTION THEOREM
3.1 THE GENERAL ADKF-IFT FRAMEWORK FOR LEARNING DEEP KERNEL GPs
Let Ae and Aş respectively be the sets of base kernel and feature extractor parameters for a deep kernel GP. Denote the set of all parameters by Ay Ae J Ag. The key idea of the general ADKF- IFT framework is that only a subset of the parameters AW,er Ay will be adapted to each individual task by minimizing a train loss Lr. with the remaining set of parameters Awn. = -Ay \ A meta-learned during meta-traine teeaitta phase to yield the best possible validation loss Lv average over many related training tasks (after Awam is separately adapted to each of these tasks). This can be naturally formalized the following bilevel optimization problem:
Vinta arg min Ep(Ti[ metaa Vidape (2b metas ST),T),
(1)
such that
Vattoge ( metus ST) arg min Lr Vadapts ST).
2

---

Published as a conference paper at ICLR 2023
Equations ① and (2) are most easily understood by separately considering the meta- meta-learned param- eters 44 meta and the task-specifie parameters 3 Vatun: For a given task 7 and an arbitrary value for the meta-learned parameters 3 meea: in Equation (2) the task-specific parameters 2b Batagg
chosen to minimize the frain loss LT evaluated on the task's support set Sy. That is, & Vadagt is adapted to the support set ST of the task T, with the aim of producing the best possible model on S for the given value of 1/ metus The result is a model with optimal task-specific parameters ab adups (4 meta: ST) for the given meta-learned parameters meeta and task T. The remaining question is how to choose value for the meta-learned parameters metaa knowing that e aauapp will be adapted separately to each task. In Equation ( we propose to choose meta to minimize the expected validation loss Ly over a distribution of training tasks p()). There are two reasons for this. First, on any given task T, the validation loss usually reflects the performance metric of interest on the query set QT of T (e.g.. the prediction error). Second, because the same value of v/2 meta will be used for all tasks, it makes sense to choose a value whose expected performance is good across many tasks drawn from p(T). ( That is, sb meta is chosen such that GP achieves the lowest possible average validation loss the query set Qr of a random training task T ~ P(T) after &5 ndapt is adapted to the task's ' support set Sy.
In practice, v/ metn would be optimized during meta-training phase using a set of training tasks D to approximate Equation (T). After meta- traitaigriiing (i.e., at meta-test time), we make predictions for each unseen test task T. using the joint GP posterior predictive distribution with optimal parameters 4/ nitiata and 2b adapt (//mmtta ST.):
p(O 10,,, 15 V adaape (1 metar ST.)).
Note that the description above does not specify a particular choice of A am: Av.gy: Lr, Lv. This is intentional, as there many reasonable choices for these quantities. Because of this, believe that ADKF-IFT should be considered a general framework, with a particular choice for these being an instantiaton of the ADKF-IFT framework. We give examples of this in Sections 3.3 and 3.4
3.2 EFFICIENT META-TRAINING ALGORITIM
In general, optimizing bilevel optimization objectives such as Equation 1 is computationally com- plex, mainly because each evaluation of the objective requires solving a separate inner optimization problem 2 Although calculating the hypergradient (i.e., total derivative) of the validation loss Ly w.r.t. the meta-learned parameters v/b optimization:
meta would allow Equation TT) to be solved with gradient-based
Lv
LL meta
8 Lv ① Da
(4)
d dbb mota
① 2v addap 2 2&b meta
Equation reveals that this requires calculating @ iinne vh munu i.e., how the optimal task-specific parameters v adgge (+) (0mnita:S S change with respect to the meta-learned parameters v/b metaa Calculating this naively with automatic differentiation platforms would require tracking the gradient through many iterations of the inner optimization (2), which in practice requires too much memory to be feasible. Fortunately, because 1/ diddup is an optimum of the train loss Lr, Cauchy's Implicit Function Theorem (IFT) provides a formula for calculating @ Wrinn/a oee for an arbitrary value of the meta-learned parameters Vi'meta and given task T':
① 2 atupi
02 Lr ( meta: 24 matupe ST.)
82 Lr (vb meta: 2b udugp STi)
(5)
① 2 meta Wara
a v/p adapt O u/p addgge
i 1/ adapt 2 up mett
Frs akgs.
meta: ST.). A full statement of the implicit function theorem in the context of ADKF-IFT can be found Appendix A The only potential problem with Equation (5) is the computation and inversion of the Hessian matrix 8² Parg. Paa a 0 Wame This computation can be done exactly if [A is small, which is the case considered in this paper (as will be discussed in Section 3.4). Otherwise, approximation to the inverse Hessian (e.g., Neumann approximation (Lorraine et al. 2020 Clarke et al.) 2022) could be used, which reduces both the memory and computational complexities to O([Aw)). Combining Equations (4) and (5). we have a recipe for computing the hypergradient / " um.. exactly for a single task, as summanized in Algorithm [ The meta-learned parameters th meta can then be updated with the expected hypergradient over p(7)
where & = 26 atapp (26(
3

---

Published as a conference paper at ICLR 2023
Algorithm Exact hypergradient computation in ADKF-IFT.
1: Input: training task T' and the current meta-learned parameters Vimeta
2: Solve Equation
2 to obtain Vaatag Byy
T Madup Vmena ST)
3: Compute 81
Patmpe Pamme
g2
Ø Lv (Wems Waten
by auto-diff.
Vines Valaa
Vaew 16
4: Compute the Hessian H Lz (o MatrssS
by auto-diff.
3 stags Vatspe
Veres Vaien
5: Solve the linear system v H = 82 for V.
6² C Waae Serge Paag set
6: Compute the mixed partial derivatives P
by auto-diff.
watt Stops
7: Output: the hypergradient dCv. Pesuu
81 P.P
Equations
4
and
3.3 ADKF-IFT AS UNIFICATION OF PREVIOUS METHODS
In prior work, the most common method used to train deep kernel GPs is to minimize the negative log marginal likelihood (NLML) on a single dataset (optionally with extra regularization terms). This commonly referred to as Deep Kernel Learning (DKL) (Wilson et al. (016b), and is stated explicitly in Equation (6) The most notable departure from DKL is Deep Kernel Transfer (DKT) (Patacchiola al. 2020), which instead proposes to train deep kernel GPs entirely using meta-learning, minimizing the expected NLML over distribution of training tasks, as stated explicitly Equation
2/ arg min NLML (1b, ST) 1
(6)
arg min E EP()) [NLML 7)
(7)
Interestingly, both DKL and DKT can be viewed as special cases of the general ADKF-IFT framework. is simple see that choosing the partition be Av 2, A Awang Ag and the train loss Lr to be the NLML in Equations (1) and (2) yields Equation (6): DKL is just ADKF-IFT if no parameters are meta-learned. Similarly, choosing the partition to Av.... = Ag, A Ø and the validation loss Cy to be the NLML in Equations ( and (2) yields Equation (7) DKT is just ADKF-IFT if all parameters are meta- meta-learned. This makes ADKF-IFT strictly more general than these two methods.
3.4 HIGHLIGHTED ADKF-IFT INSTANTIATION: META-LEARS Φ, ADAPT
Among the many possible variations of ADKF-IFT, we wish highlight the following instantation:
Aw = Aq. i.e., all feature extractor parameters & are meta-learned across tasks.
Avang Ao, i.e., all base kernel parameters (e.g.. noise, lengthscales, ett) are adapted.
The train loss Lr and validation loss Ly are the negative log GP marginal likelihood on ST and the negative log joint GP predictive posterior on QT given Sy, respectively. Please refer to Appendix B for equations of these loss functions.
There are several benefits to this choice. First, this particular choice loss functions has the advantage that the prediction procedure during meta-testing (as defined in Equation (3) exactly matches the meta-training procedure, thereby closely following the principle of learning to learn. Second, the partition of parameters can be intuitively understood as meta-learning generally useful feature extractor f such that it is possible on average to fit a low-loss GP to the feature representations extracted by f for each individual task. This is very similar to previous transfer learning approaches. Third, since most GP base kernels have only a handful of parameters, the Hessian in Equation (5) can be computed and inverted exactly during meta-training using Algorithm[] [T] this removes any need for Hessian approximations. Fourth, the inner optimization (2) for v addpp is computationally efficient, as it does not require backpropagating through the feature extractor fz.
More generally, we conjecture that adapting just the base kernel parameters will allow ADKF-IFT to achieve a better balance between overfitting and underfitting than either DKL or DKT. The relationship between these methods are visualized in Figure Panel (c) shows DKL, which trains separate deep kernel GP for each task. It is not hard to imagine that this can lead to severe overfitting for small datasets, which has been observed empirically by Ober al. (202)). Panel (b) shows DKT,
4

---

Published as a conference paper at ICLR 2023
update Paeeto using implicit function thosrem
train loss
agtivaite Vatue,
MMI
Expected validation Bosss
train loss
optirate Vacaal
Fx
train loss
oylinite Vactar
Vastage
(a) A daptive Deep Kernel Fitting with Implicit Function T Theorem (ADKF-IFT)
optimize Padapt
optimize Percio
M
train
optimize (Padage
Expected validation less
optimize Vaddep
train loss
Vadags
[0,
(b) Deep Kernel Transfer (DKT)
(c) Deep Kernel earning (DKL)
Figure 1: A contrastive diagram illustrating the training procedures of ADKF-IFT, DKT, and DKL..
which prevents overfitting by fitting one deep kernel GP for all tasks. However, this implicitly makes strong assumption that all tasks come from identical distribution over functions, including the same noise level, same amplitude, and same characteristic lengthscales, which is unlikely to hold in practice. Panel (a) shows ADKF-IFT, which allows these important parameters to be adapted, while still regularizing the feature extractor with meta-learning. We conjecture that adapting the base kernel parameters more appropriate given the expected differences between tasks: two related tasks more likely to have different noise levels or characteristic lengthscales than to require substantially different feature representations. We refer the readers Appendix [ppendix[H]oor more discussions.
RELATED WORK
ADKF-IFT part a growing body of literature of techniques train deep kernel GPs. As discussed in Section 3.3 ADKF-IFT generalizes DKL (Wilson et al.) (016b) and DKT (Patacchiola al.
2020). which exclusively use single-task learning and meta-learming, respectively. Liu et al. (2020) and van van Amersfoort et al. (2021) propose adding regularization terms to the loss of DKKL. order
to mitigate overfitting. These works are better viewed as complementary to ADKF-IFT rather than alternatives: their respective regularization terms could easily be added Lr in Equation (2) to improve performance. However, the regularization strategies in both of these papers are designed for continuous inputs only, limiting their applicability to structured data like molecules.
ADKF-IFT can also be viewed as a meta-learning algorithm comparable to many meeiously-prooooe methods (Lake al.) [001]
Vinyals et al.) 2016 Garnelo et al. 2018 Triantafilluu et al. [019 Park
& Oliva 2019: Tian et al. 2020; [Cenn et al, [201]] Liu et al. 2021 Wistuba & Grabocka 1021
Patacchiola al. 2022). One distinguishing feature of ADKF-IFT that is specially designed
for deep kernel GPs, whereas most methods from computer vision are designed exclusively for
neural network models, which as previously stated are unsuitable when reliable uncertainty estimates required. Furthermore, many of these algorithms such as ProtoNet (Snell et al.) (017) are designed principally or exclusively for classification, while ADKF-IFT is suited to both regression and classification. Compared to model-agnostic frameworks like MAML (Finn et al.) (017), ADKF. IFT does not require coarse approximations the hypergradient due to its use of the implicit function theorem. For further discussions of related works, please refer Appendix
5

---

Published as a conference paper at ICLR 2023
Table 1: Mean test performance (AUROC%) with standard deviations of all compared methods on MoleculeNet benchmark tasks at support set size 20 (i.e., 2-way 10-shot).
MoleculeNet benchmark task (#coompounds
Method
Tox21 (8,014)
SIDER (1,427)
MUV (93,127)
ToxCast (8,615)
Siamese
80.40 0.35
71.10 + 4.32
59.59 + 5.13
ProtoNet
74.98 0.32
64.54 - 0.89
65.88 + 4.11
63.70 = 1.26
MAML
80.21 0.24
70.43 0.76
2.28 63.90 2.28
66.79 = 0.85
TPN
76.05 0.24
67.84 0.95
65.22 5.82
62.74 1.45
EGNN
81.21 0.16
72.87 0.73
55.20=2.00
63.65 = 1.57
IterRefl STM
81.10+0.11
69.63 + 0.31
45.56 + 5.12
PAR
82.06 1.12
74.68 0.31
66.48 2.12
69.72 1 1.63
ADKF-IFT
82.43 0.60
1.21
98.18 3.05
72.07 0.81
Pre-GNN
82.14 0.08
73.96 +: 0.08
67.14 + 1.58
73.68 + 0.74
Meta-MGNN
82.97 0.10
68.99 - 1.84
Pre- Pre-PAR PAR
84.93 0.11
78.08 0.16
69.96 - 1.37
75.12 = 0.84
Pre-ADKF-IFT
86.06 0.35
70.95
95.74 0.37
76.22 + 0.13
5 EXPERIMENTAL EVALUATION ON MOLECULES
In this section, we evaluate the empirical performance of ADKF-IFT from Section 3.4 focus our experiments exclusively on molecular property prediction and optimization tasks because we believe this application would benefit greatly from better GP models: firstly because many existing methods struggle on small datasets of size ~ 102 which are ubiquitous in chemistry, and secondly because many tasks in chemistry require high-quality uncertainty estimates. First, we evaluate ADKF-IFT on four commonly used benchmark tasks from MoleculeNet (Wu et al., 2018), finding that ADKF-IFT achieves state-of-the-an results on most tasks (Section [.T). Second, we evaluate ADKF-IFT on the larger-scale FS-Mol benchmark (Stanley et al. 2021). 1) finding that ADKF-IFT is the best-performing method (Section 5.2). In particular, our results support the hypothesis from Section that ADKF-IFT achieves a better balance between overfitting and underfitting than DKL and DKT. Finally, we show that the ADKF-IFT feature representation is transferable to out-of-domain molecular property prediction and optimization tasks (Section 5.3). The general configurations of ADKF-IFT for all experiments considered in this paper are shown in Appendix B
We choose to
5.1 FEW-SHOT MOLECULAR PROPERTY PREDICTION ON THE MOLECULENET BENCHMARK
Benchmark and Baselines. We compare ADKF-IFT with two types of baselines on four few- shot molecular property classification benchmark tasks (Tox21, SIDER, MUV, and ToxCast) from MoleculeNet (Wu et al. 2018) (see Appendix for more details of MoleculeNet benchmark tasks):
1) methods with feature extractor trained from scratch: Siamese (Koch 2015). ProtoNet (Snell et al.
2017). MAML (Finn et al.] (017). TPN (Liu et al. (011)) EGNN (Kim et al. 2019)), Iter
Altae-Traan et al. 2017) and PAR (Wang et al., 2021); and 2) methods that fine-tune pretrained
feature extractor: Pre-GNN (Hu* et [l.00) Meta-MGNN (Guo et al. 202I) and Pre-PAR (Wang) al. 2021). Pre-ADKF-IEE refers ADKF-IFT starting from a pretrained feature extractor. All compared methods in this section use GIN (Xu et al. 2019) as their feature extractors. The pretrained weights for the methods of the second type are provided by Hu* et al. (2020).
Evaluation Procedure. We follow exactly the same evaluation procedure as that in Wang et al. (20TT: ; Hu x et al. (2000): Guo et al.] (202T). The task-level metric AUROC (area under the receiver operating characteristic curve). We report the averaged performance over ten runs with different random seeds for each compared method at the support set size 20 (i.e., 2-way 10-shot, as the support sets in MoleculeNet are balanced). We did not perform 1-shot learning, as it is unrealistic setting in real-world drug discovery tasks. All baseline results are taken from Wang et al. (2021).
Performance. Table shows that ADKF-IFT and PDK-ADFFIIFT achieve the best performance on Tox21, MUV, and ToxCast. In general, the larger the dataset is, the larger the performance gains of method over other baselines are, highlighting the scalability of our method. In particular, our method outperforms all baselines by a wide margin on MUV due to the relatively large amount of available compounds, but underperforms many baselines on SIDER due to lack of compounds.

---

Published as a conference paper at ICLR 2023
0.35
0.3
0.4
0.25
0.3
SAUPRC
0.20
0.2
0.15
0.1
ADKFIFTT
0.10
DKT
MAT
0.0
CNP
CNN-ST
ADKF-IFT
RF
GUN-MAM
-0.1
GNN-MI
GP-ST
DKL
CNP
MAT
0.00
128
-0.2
16
32
64
128
256
Support set size
Support size
(a) Classification (157 tasks).
(b) Regression (111 tasks).
Figure 2: Mean performance with standard errors of all compared methods all FS-Mol test tasks.
5.2 FEW-SHOO MOLECULAR PROPERTY PREDICTION ON THE FS-MOL BENCHIMRR
Benchmark. We further conduct our evaluation the FS-Mol benchmark (Stanley et al. 202 which contains a carefully constructed set of few-shot learning tasks for molecular property prediction. FS-Mol contains over 5,000 tasks with 233,786 unique compounds from ChEMBL27 (Mendez et al., 2019). split into training (4,938 tasks), validation (40 tasks), and test (157 tasks) sets. Each task associated with protein target. The original benchmark only considers binary classification of active/inactive compounds, but we include the regression task (for the actual numeric activity target IC50 or EC50) in our evaluation well, as it is desired and more preferred task to do in real-world drug discovery projects.
Baselines. We compare ADKF-IFT with four categories of baselines: 1) single-task methods:
Random Forest (RF), K-Nearest Neighbors (kNN), single-task GP with Tanimoto kernel (GP-ST)
(Ralaivola et al.) 2005), single-task GNN (GNN-ST) (Gilmer et al., 2017). Deep Kernel Learning
(DKL) (Wilson et al. 2016b); 2) multi-task pretraining: multi-task GNN (GNN-MT) (Corso et al.
2020 Gilmer et al. 2017); 7): 3) self-supervised pretraining: Molecule Attention Transformer (MAT)
Mazzirra et al. 2020); 4) meta-learning methods: Property-Aware Relation Networks (PAR) (Wang)
ct al. 2021). Prototypical Network with Mahalanobis distance (ProtoNet) (Snell et al. 2017), Model-
Agnostic Meta-Learning
(GNN-MAML) (Finn al. 2017). Conditional Neural Process (CNP)
(Ganelo et al. 2018). Deep Kernel Transfer (DKT) Patacchiola et al. 2020). The GNN feature extractor architecture f used for DKL, PAR. CNP, DKT, and ADKF-IFT the same that used for ProtoNet, GNN-ST, GNN-MT, and GNN-MAML. in |Stanley et al. (202T). T). All multi-task and meta- learning methods are trained from scratch FS-Mol training tasks. MAT is pretrained
million molecules sampled from the ZINC15 dataset (Sterling & Irwin 2015). The classification results for RF, KNN, GNN-ST, GNN-MT, MAT, ProtoNet, and GNN-MAML are reproduced according to Stanley et al. 202T). Detailed configurations of all compared methods can be found in Appendix D
Evaluation Procedure. The task-level metrics for binary classification and regression are AAUPRC (change in area under the precision-recit curve) and R (predictive/out-dl of -ff-epaat-t----leleeeee coefficient of determination), respectively. Details of these metrics can be found Appendix We follow exactly the same evaluation procedure that in Stanley et al. (202T), where the averaged performance over ten different stratified support/query random splits of every test task reported for each compared method. This evaluation process is performed for five different support 256. Note that the support sets are generally unbalanced for the classification task in FS-Mol, which natural the majority of the candidate molecules are inactive in drug discovery.
sizes 16, 32, 64, 128, and
Overall Performance. Figure 2 shows the overall test performance of all compared methods. Note that RF is strong baseline method, as it is widely used in real-world drug discovery projects and has comparable performance to many pretraining methods. The results indicate that ADKF-IFT outperforms all the other compared methods all considered support set sizes for the classification
7

---

Published as a conference paper at ICLR 2023
0.5
ADKF-IFT
ADKF
DAT+
0.975
0.290
0.555
0.200
16
30
178
216
Support set sine
Support
(a) Classification (157 tasks).
(b) Regression (111 tasks).
Figure 3: Mean performance with standard errors of ablation models on all FS-Mol test tasks. ADKF is like ADKF-IFT but assuming 00 20"/0 & O, i.e., updating & with the direct gradient Lv/o DKT+ like DKT but tuning the base kernel parameters 0 during meta-testing.
task. For the regression task, the performance gains of ADKF-IFT over the second best method, namely DKT, get larger as the support set size increases. In Appendix F.I we show that ADKF-IFT achieves the best mean rank for both classification and regression at all considered support set sizes.
F.I
Statistical Comparison. We perform two-sided Wilcoxon signed-rank tests (Wilcoxon) 1992) to compare the performance of ADKF-IFT and the next best method, namely DKT. The exact p-values from these statistical tests can be found in Appendix F22 The results indicate that ADKF-IFT significantly outperforms DKT for the classification task at all considered support sizes and for the regression task at support set sizes 64, 128, and 256 (at significance level a 0.05).
Ablation Study. To show that 1) the bilevel optimization objective for ADKF-IFT is essential for learning informative feature representations and 2) the performance gains of ADKF-IFT are not simply caused by tuning the base kernel parameters 0 at meta-test time, consider two ablation models: DKT- and ADKE. The test performance of these models are shown in Figure S
For ADKF, we follow the ADKF-IFT training scheme but assume 00" /a 4 i.e., updating the feature extractor parameters & with the direct gradient 3 Lv f rather than d d'v/dd. sb The results show that ADKF consistently underperforms ADKF-IFT, indicating that the hypergradient for the bilevel optimization objective has non-negligible contributions to learning better feature representations. For DKT+, we take a model trained by DKT and adapt the base kernel parameters O on each task at meta-test time. The results show that DKT does not improve upon DKT, indicating that tuning the base kernel parameters at meta-test time not sufficient for obtaining better test performance with DKT.
Sub-benchmark Performance. The tasks in FS-Mol can be partitioned into nub-benchmarks by Enzyme Commission number (Webb et al.) 1992). In Appendix [F.3 we show the test performance of top performing methods each sub- benchmark. The results indicate that, in addition to achieving best overall performance, ADKF-IFT achieves the hest performance on all hub-benchmarks for the regression task and more than half of the sub-henchmarks for the classification task.
5.3 OUT-OP-DOMAIN MOLECULAR PROPERTY PREDICTION AND OPTIMIZATION
Finally, we demonstrate that the feature representation learned by ADKF-IFT is useful not only for in-domain molecular property prediction tasks but also for out-of-domain molecular property prediction and optimization tasks. For this, we perform experiments involving finding molecules with best desired target properties within given out-of-domain datasets using Bayesian optimization (BO) with a GP surrogate model operating on top of compared feature representations. We use the expected improvement acquisition function (Jones al., 1998) with query-batch size All compared feature representations are extracted using the models trained on the FS-Mol dataset from scratch in Section 5.2) except for the pretrained MAT representation and fingerprint. We compare them on four representative molecular design tasks outside of FS-Mol. Detailed configuration of the GP and descriptions of the tasks can be found in Appendix We repeat each BO experiment 20 times, each time starting from 16 randomly sampled molecules from the worst 700 molecules within the dataset. Figure 4 shows that the ADKF-IFT representation enables fastest discovery of top performing molecules for the molecular docking, antibiotic discovery, and material design tasks. For
8

---

Published as a conference paper at ICLR 2023
Top- powerr conversion efficiencs
Topal binding score
retetve growen
0.
impresentation
M6t represetation
Glb VI represe station
represenaa
Prnnwer resessntation
4.5
rencaentaaion
6
12
queried
The number of mokecules querred
number mokerules queried
The number of molercees querted
(a) Molecular docking.
(b) Antibiotic discovery.
(c) Antiviral drug design.
(d) Material design.
Figure 4:
Mean top-1 target values with standard errors function of the number of molecules
queried for all compared feature representations on four out-of-domain molecular optimization tasks.
Table 2:
Mean predictive performance (test NLL) with standard errors of a GP operating top of each compared feature representation on the four out-of out-of-domain molecular design tasks.
Feature
Oft-of-domain molecular design task
representation
Molecular docking
Antibiotic discovery
Antiviral drug design
Material design
Fingerprint
1.138 1 0.014
1.669 1 0.075
4.601 1 0.086
1.091 0.011
PAR
1,270 + 0.019
2.185 + 0.115
4.840 + 0.086
1.283 0.017
MAT
1.528 + 0.028
2.390 = 0.104
4.797 = 0.088
2.198 0.063
GNN-MT
1.994 - 0.050
3.692 + 0.225
6.399 = 0.181
7.254 = 0.217
CNP
1.493 1 0.028
2.537 + 0.112
5.005 + 0.086
1.741 1: 0.043
ProtoNet
1.147 = 0.013
1.615 0.094
5.060 = 0.086
1.032 + 0.009
DKT
1.167 + 0.012
1.602 + 0.073
4.975 = 0.092
1.026 0.009
ADKF-IFT
1.137 = 0.011
1.496 + 0.043
4.781 + 0.087
0.996 0.007
the antiviral drug design task, although the ADKF-IFT representation underperforms the MAT and GNN-MT representations, it still achieves competitive performance compared to other baselines.
Table exxlicitly reports the regression predictive performance of GP operating top of each com- pared feature representation for these four out-of-domain molccular design tasks. The configuration of the GP is the same that in the BO experiments. We report test negative log likelihood (NLL) averaged over 200 support/query random splits (100 for each of the support set sizes 32 and 64). The results show that the ADKF-IFT representation has the best test NLL on the molecular docking, antibiotic discovery, and material design tasks, and ranks second on the antiviral drug design task.
CONCLUSION
We have proposed Adaptive Deep Kernel Fitting with Implicit Function Theorem (ADKF-IFT), a novel framework for fitting deep kernels that interpolates between meta-learning and conventional deep kernel learning. ADKF-IFT meta-leame a feature extractor across tasks such that the task- specific GP models estimated on top of the extracted feature representations can achieve the lowest possible prediction eтrоr on average. ADKF-IFT is implemented by solving a bilevel optimization objective via implicit differentiation. We have shown that ADKF-IFT is a unifying framework containing DKL. and DKT special cases. We have demonstrated that ADKF-IFT learns generally useful feature representations, achieving state-of-the-ar performance on a variety of real-world few-shot molecular property prediction tasks and on out-of-domain molecular property prediction and optimization tasks. We believe that ADKF-IFT could potentially be an important method to produce well-calibrated models for fully-automated eigh-throughhut experimentation in the future.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS AND DISCLOSURE FUNDING
We thank Massimiliano Patacchiola, John Bronskill, Marcin Sendera, and Richard E. Turner for helpful discussions and feedback. WC acknowledges funding via a Cambridge Trust Scholarship (supported by the Cambridge Trust) and a Cambridge University Engineering Department Studentship (under grant G105682 NMZR/089 supported by Huawei R&D UK). AT acknowledges funding via CT Taylor Cambridge International Scholarship. JMHI acknowledges support from a Turing Al Fellowship under grant EP/VO23756/1.
ETHICS STATEMENT
We believe that the ethical implications of this work are minimal: this research involves no human subjects, sensitive data where privacy is concern, no domains where discrriminationbbis//aaaesss concerning, and is unlikely to have a noticeable social impact. Optimistically, our hope is that in the future ADKF-IFT could be used in the drug discovery pipeline to create new beneficial medicines, giving it an overall positive ethical impact, However, as with most research in machine learning, new modelling techniques could be used by bad actors cause harm more effectively, but we do how ADKF-IFT is more concerning than any other method in this regard.
REPRODUCIBILITY STATEMENT
Our implementation and experimental results can be found at: https://githutt: com/
Wenlin Chen/ ADKF-IFT which is based on a forked from FS-Mol (Stanley et al. 202 1) and
PAR (Wang et al. 2021). Details for the setup of ADKF-IFT can be found in Appendix B while
details for other --M baselines can be found in Appendix D and details for other MoleculeNet
baselines can be found in Wang et al. 2202T] Appendix B).
The arXiv version of this paper be found at: http::///aax.v..oooaaa///////// which may be updated as needed.
REFERENCES
Han Altae-Tran, Bharath Ramsundar, Aneesh S. Pappu, and Vijay Pande. Low data drug discovery with one-shot learning. ACS Central Science, 3:283 - 293, 2017.
Yoshua Bengio. Gradient-based optimization of hyperparameters. Neural computation, 12(8): 1889-1900, 2000.
Edwin V Bonilla, Kian Chai, and Christopher Williams. Multi-task gaussian process prediction. Advances in neural information processing systems, 20, 2007.
John Bradshaw, Alexander G de G Matthews, and Zoubin Ghahramani. Adversarial examples, uncertainty, and transfer testing robustness in Gaussian process hybrid deep networks. arXiv preprint arXiv:1707.0244 02476, 2017.
Roberto Calandra. Jan Peters, Carl E Rasmussen, and Mare Peter Deisenroth. Manifold Gaussian processes for regression. In 2016 International Joint Conference on Neural Networks (IJCNN), 2016.
Da Chen, Yuefeng Chen, Yuhong Li, Feng Mao, Yuan He, and Hui Xue. Self-supervised learning for few-shot image classification. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1745-1749. IEEE, 2021.
Yutian Chen, Abram I. Friesen, Feryal Behbahani, Arnaud Doucet, David Budden, Matthew Hoffman, and Nando de Freitas. Modular meta-learning with shrinkage. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2858 2869. Curran Associates, Inc., 2020.
Ross M Clarke, Elre Talea Oldewage, and José Miguel Hermández-Lobato. Scalable one-pass optimisation of high-dimensional upiate hyperparameters by implicit differentiation. In International Conference Learning Representations, 2022.
10

---

Published as a conference paper at ICLR 2023
The COVID Moonshot Consortium, Hagit Achdout, Anthony Aimon, Elad Bar-David Haim Barr, Amir Ben-Shmuel, James Bennett, Vitaliy A. Bilenko, Vitaliy A. Bilenko, Melissa L. Boby, Bruce Borden, Gregory R. Bowman, Juliane Brun, Sarma BVNBS, Mark Calmiano, Anna Carbery. Daniel Carney, Emma Cattermole, Edcon Chang, Eugene Chernyshenko, John D. Chodera, Austin Clyde, Joseph F. Coffland, Galit Cohen, Jason Cole, Alessandro Contini, Lisa Cox, Milan Cvitkovic, Alex Dias, Kim Donckers, David L. Dotson, Alice Douangamath, Shirly Duberstein, Tim Dudgeon, Louise Dunnett, Peter K. Eastman, Noam Erez, Charles J. Eyermann, Mike Fairhead, Gwen Fate, Daren Fearon, Oleg Fedorov, Matteo Ferla, Rafaela S. Fernandes, Lori Ferrins, Richard Foster, Holly Foster, Ronen Gabizon, Adolfo Garcia-Sastre. Victor O. Gawriljuk, Paul Gehrtz, Carina Gileadi, Charline Giroud, William G. Glass, Robert Glen, Itai Glinert, Andre S. Godoy, Marian Gorichko, Tyler Ctorrie-Stone, Ed J. Griffen, Storm Hassell Hart, Jag Heer, Michael Henry, Michelle Hill, Sam Horrell, Victor D. Huliak, Matthew F.D. Hurley, Tomer Israely, Andrew Jajack, Jitske Jansen, Eric Jnoff, Dirk Jochmans, Tobias John, Steven De Jonghe, Anastassia L. Kantsadi, Peter W. Kenny, J. L. Kiappes, Serhii O. Kinakh, Lizbe Koekemoer, Boris Kovar, Tobias Krojer, Alpha Lee, Bruce A. Lefker, Haim Levy, Ivan G. Logvinenko, Nir London, Petra Lukacik, Han- nah Bruce Macdonald, Beth MacLean, Tika R. Malla, Tatiana Matviiuk, Willam McCorkindale, Briana L. McGovern, Sharon Melamed, Kostiantyn P. Melnykov, Oleg Michurin, Halina Mikolajek, Bruce F. Milne, Aaron Morris, Garrett M. Morris, Melody Jane Morwitzer, Demetri Moustakas, Aline M. Nakamura, Jose Brandao Neto, Johan Neyts, Luong Nguyen, Gabriela D. Noske, Vladas Oleinikovas, Glaucius Oliva, Gijs Overheul, David Owen, Ruby Pai, Jin Pan, Nir Paran, Ben- jamin Perry, Maneesh Pingle, Jakir Pinjari, Boaz Politi, Ailsa Powell, Vladimir Psenak. Reut Puni, Victor I.. Rangel, Rambabu N. Reddi, St Patrick Reid, Efrat Resnick, Emily Grace Ripka, Matthew C. Robinson, Ralph P. Robinson, Jaime Rodriguez-Guurra Romel Rosales, Dominic Rufa, Kadi Saar, Kumar Singh Saikatendu, Chris Schofield, Mikhail Shafeev, Aarif Shaikh, Jiye Shi, Khriesto Shurrush, Sukrit Singh, Assa Sittner, Rachael Skyner, Adam Smalley, Bart Smeets, Mihaela D. Smilova, Leonardo J. Solmesky, John Spencer, Claire Strain- Damerell,, Vishwanath Swamy, Hadas Tamir, Rachael Temnant, Warren Thompson, Andrew Thompson, Susana Tomasio, Igor S. Tsurupa, Anthony Tumber, Ioannis Vakonakis, Ronald P. van Rij, Laura Vangeel, Finny Varghese, Mariana Vaschetto, Einat B. Vitner, Vincent Voelz, Andrea Volkamer, Frank von Delft, Annette von Delft, Martin Walsh, Walter Ward, Charlie Weatherall, Shay Weiss, Kris M. White, Conor Francis Wild, Matthew Wittmann, Nathan Wright, Yfat Yahalom- Daniel Zaidmann, Hadeer Zidane, and Nicole Zitzmann. Open science discovery of oral non-covalent sars-cov-2 main protease inhibitor therapeutics. bioRxiv, 2022. doi: 10. 10.1100/2020.1 10.29.39317..
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 13260-13271 13271. Curran Associates, Inc., 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learing for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1126-1135. PMLR, 06-11 Aug 2017.
Alexander IJ Forrester, András Sóbester, and Andy J Keane. Multi-fidelity optimization via surrogate modelling. Proceedings of the royal society a: mathematical, physical and engineering sciences, 463(2088):3251-22 -32699,,, 2007.
Peter Frazier. A tutorial on Bayesian optimization. arXiv preprint arXiv:1807.0281 2018.
Miguel Carcia-Orteggó Gregor NC Simm, Austin J Tripp, José Miguel Hernández-Lobat Andreas Bender, and Sergio Bacallado. Dockstring: easy molecular docking yields better benchmarks for ligand design. arxiv preprint arXhv:2110.15486 2021.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neural processes. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1703-1713. PMLR, 10-15 Jul 2018.
Damien Garreau, Wittawat Jitkrittum, and Motonobu Kanagawa. Large sample analysis of the median heuristic. arXiv preprint arXiv: (707 2017.

---

Published as a conference paper at ICLR 2023
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley. Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Doina Precup and Yee Whye Teh ), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1263-1272. PMLR, 06-11 Aug 2017.
Zhichun Guo, Chuxu Zhang, Wenhao Yu, John Herr, Olaf Wiest, Meng Jiang, and Nitesh V Chawla. Few-shot graph learning for molecular property prediction. arXiv preprint arXiv:2102.07916. 2021.
Johannes Hachmann, Roberto Olivares-Amaya, Sule Atahan-EEvrenk, Carlos Amador-Bedolla, Roel S Leslie Vogt, Anna M Brockway, and Alán Aspuru-Guzik The harvard clean energy project: surge-scale computational screening and design of organic photovoltaics on the world community grid. The Journal of Physical Chemistry Letters, 2(17):
Sánchez- Sánchez-Carreea, Aryeh
2241-2251,201
Geoffrey E Hinton and Russ R Salakhutdinov. Using deep belief nets to learn covariance kernels for gaussian processes. J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007.
Weihua Hu*, Bowen Liu", Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations, 2020.
Donald R Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455-492, 1998.
Marc C Kennedy and Anthony O'Hagan. Predicting the output from a complex computer code when fast approximations are available. Biometrika, 87(1):1-13, 2000.
Jongmin Kim, Taesup Kim, Sungwoong Kim, and Chang Dong Yoo. Edge-labeling graph neural network for few-shot learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11-20, 2019.
Minyoung Kim and Timothy Hospedales. Gaussian process meta few-shot classifier learning via linear discriminant laplace approximation. arXiv preprini arXiv:2111.05392, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 2014.
Gregory R. Koch. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2, 2015.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedingy of the annual meeting of the cognitive science society, volume 33, 2011.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Dong C Liu and Jorge Nocedal. On the limited memory bigs method for large scale optimization. Mathematical programming, 45(1):503-523 1989.
Feng Liu, Wenkai Xu, Jie Lu, and Danica Sutherland. Meta two-sample testing: Learning kernels for testing with limited data. Advances Neural Information Processing Systems, 34:5848-5866 2021.
Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awarc- ness. Advances in Neural Information Processing Systems, 53:7498-7511 2020.
Yanbin Liu, Juho Lee, Minseop Park, Sachoon Kim, Eunho Yang. Sungju Hwang, and Yi Yang. Learn- ing to propagate labels: Transductive propagation network for few-shot learning. In International Conference Learning Representations, 2019.
12

---

Published as a conference paper at ICLR 2023
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 1540-1552. PMLR, 26-28 Aug 2020.
Jelena Luketina, Mathias Berglund, Klaus Greff, and Tapani Raiko. Scalable gradient- tuning of continuous regularization hyperparameters. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 2920, New York, New York, USA, 20-22 Jun 2016. PMLR.
tukasz Maziarka, Tomasz Danel, Stawomir Mucha, Krzysztof Rataj, Jacek Tabor, and Stanistaw Jastrzębski. Molecule attention transformer. arXiv preprint arXiv:2002.08224, 2020.
David Mendez, Anna Gaulton, A Patrícia Bento, Jon Chambers, Marleen De Veij. Eloy Félix, María Paula Magariñis,, Juan F Mosquera, Prudence Mutowo, Michal Nowotka, et al. Chembl: towards direct deposition of bioassay data. Nucleic acids research, 47(D1): -D430, 2019.
Erik G Miller, Nicholas E Matsakis, and Paul A Viola. Learning from one example through shared densities on transforms. In Proceedings IEEE Conference on Computer Vision and Putlern Recognition. CVPR 2000 (Cat. No. PR00662), volume 1, pp. 464-471. IEEE, 2000.
Radford M Neal. Bayesian Learning for Neural Networks. PhD thesis, University of Toronto, 1996.
Sebastian W. Ober, Carl E. Rasmussen, and Mark van der Wilk. The promises and pitfalls of deep kernel learning. In Cassio de Campos and Marloes H. Maathuis (eds.), Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pp. 1206-1216. PMLR, 27-30 Jul 2021.
Eunbyung Park and Junier Oliva, Meta-curvature. In H. Wallach, H. Larochelle, A. Beygelzimer, Advances in Neural Information Processing Systems,
F. d' d'Alché-Buc, E. Fox, and R. Garnett Advances volume 32. Curran Associates, Inc., 2019.
Massimiliano Patacchiola, Jack Turner, Elliot J. Crowley, Michael O' Boyle, and Amos J Storkey, Bayesian meta-learning for the few-shot setting via deep kernels. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 16108-6618. Curran Associates, Inc., 2020.
Massimiliano Patacchiola, John Bronskill, Aliaksandra Shysheya, Katja Hofmann, Sebastian Nowozin, and Richard E Tumnr. Contextual gqueeze-and-exxitaee for efficient few-shot image classification. arXiv preprint XXXx:220660098433 2022.
Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 737-746, New York, New York, USA. 20-22 Jun 2016. PMLR
Matthias Poloczek, Jialei Wang, and Peter Frazier. Multi-information source optimization. Advances in neural information processing systems, 30, 2017.
Lutz Prechelt. Early stopping-but when? In Neural Networks: Tricks of the trade, pp. 55-69. Springer, 1998.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, F. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Liva Ralaivola, Sanjay J Swamidass, Hiroto Saigo, and Pierre Baldi. Graph kernels for chemical informatics. Neural networks, 1898)1003- 2005.
Carl Edward Rasmussen and Christopher K. Williams. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, USA, January 2006.
13

---

Published as a conference paper at ICLR 2023
David Rogers and Mathew Hahn. Extended-connectin fingerprints. Journal of chemical information and modeling, 50(5):742-75,, 2010.
Warren S. Sarle. Stopped training and other remedies for overfitting. In Proceedings of the 27th Symposium on the Interface of Computing Science and Statistics, pp. 352-360, 1995.
Jake Snell and Richard Zemel. Bayesian few-shot classification with one-vs-each pólya-gamma augmented gaussian processes. arXiv preprint arXiv:2007.10417 2020.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In
I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Megan Stanley, John F Bronskill, Krzysztof Maziarz, Hubert Misztela, Jessica Lanini, Marwin Segler, Nadine Schneider, and Mare Brockschmidt. Fs-mol: A few-shot learning dataset of molecules. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Teague Sterling and John J Irwin. Zinc 15-ligand discovery for everyone. Journal of chemical information and modeling, 55(11):235)) 2337, 2015.
Jonathan M. Stokes, Kevin Yang. Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz, Nina M. Donghia, Craig R. MacNair, Shawn French, Lindsey A. Carfrae, Zohar Bloom-Ackermann, Victoria M. Tran, Anush Chiappino-Pepe, Ahmed H. Badran, Ian W. Andrews, Emma J. Chory, George M. Church, Eric D. Brown, Tommi S. Jaakkola, Regina Barzilay, and James J. Collins. A deep learning approach to antibiotic discovery. Cell, 180(4):688-7022e11) 2020. ISSN 0092-8674.
Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimization. Advances in neural information processing systems, 26, 2013.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 1rXiv:1312.66999 2013.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In European Conference on Computer Vision, pp. 266- Springer, 2020.
Petru Tighineanu, Kathrin Skubch, Paul Baireuther, Attila Reiss, Felix Berkenkamp, and Julia Vinogradska. Transfer learning with gaussian processes for bayesian optimization. In International Conference on Artificial Intelligence and Statistics, pp. 6152-6181. PMLR, 2022.
Prudencio Tossou, Basile Dura, Francois Laviolette, Mario Marchand, and Alexandre Lacoste. Adaptive deep kernel learning. arXiv preprint 1905 2019.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin. Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pitrre-Antoine Manzagol, et al. Meta-dataset: dataset of datasets for learning to learn from few examples. arXiv preprint arXiv: 000.:/90333990 2019.
Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal. On feature collapse and deep kernel learning for single forward pass uncertainty. arXiv preprint arXiv:2 / 2021.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gameet (ods.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
Yaqing Wang, Abulikemu Abuduweili, Quanming Yao, and Dejing Dou. Property- aware relation networks for few-shot molecular property prediction. Advances in Neural Information Processing Systems,34: 34:17441-17454 2021.
Oren F. Webb. Tommy J. Phelps, Paul R. Bienkowski, Philip M. Digrazia, David C. White, and Gary S. Sayler. Enzyme nomenclature, 1992.
14

---

Published as a conference paper at ICLR 2023
Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics, pp- 196-202, Springer, 1992.
Andrew G Wilson, Zhiting Hu, Russ R Salakhutdinov, and Eric P Xing. Stochastic variational deep
kernel learning. Advances in Neural Information Processing Systems, 2016a.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. In Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th International Conference on Artificial Intelligence and Statisties, volume 51 of Proceedings of Machine Learning Research, pp. 370-378, Cadiz, Spain, 09-11 May 2016b. PMLR.
Martin Wistuba and Josif Grabocka. Few-shot Bayesian optimization with deep kernel surrogates. arXiv preprint arXiv:2101.07600 2021.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 9()::13--330 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? ArXiv, 0tb//8800000/ 2019.
Miao Zhang, Steven W Su, Shirui Pan, Xiaojun Chang, Ehsan M Abbasnejad, and Reza Haffari. idarts: Differentiable architecture search with stochastic implicit gradients. In International Conference on Machine Learning, pp. 12557-12566. PMLR, 2021.
15

---

Published as a conference paper at ICLR 2023
A CAUCHY'S IMPLICIT FUNCTION THEOREM
We state Cauchy's s Implicit Function Theorem (IFT) in the context of ADKF-IFT in Theorem
Theorem (Implicit Function Theorem (IFT)) Let T' be any given task. Suppose for some V'era
and ap Wadops that 8 L Vadhge STi)
0. Suppose that 3 aiii (10 V Wadopr ST.)
aCr
E sdays
S Panne aiege
mema X V Madapp 7 V adapt a continuously differentiable function w.r.t. 4/ mena and 1/ adupe and the Hes-
sian 8 Lr Suunu-aassee SaigesSTi)
is invertibl. Then, there exists open set U meta containing
8 aaggo G aate
w Vi
adige
Wimto and a function Vs
adape
? motas STi)
meta
- V adapt such that Vattap = = Vaiaue (2 metas STi) U. Moreover, the rate at which
and 8 Cr Watoge S.T))
0, V Vab *ymeaa
Vaans When S
ab adupp' 1// meta: Sire) ) changing W.F.l. 1b meta for any o E U given by
a 2/2 addaee metar STD)
① 2gp meto
8 LT metas 1b adopps SiT))
82 CT metar 1b aadoprs STi)
① 02pp "daap ① ap udupt
2/2 udupe
2/ meta
W ii Spr)
B GENERAL CONFIGURATIONS OF ADKF-IFT FOR FEW-SHOT LEARNING EXPERIMENTS
Specifically, we In this paper, we consider the specific instantiation ADKF-IFT from Section 3.4
set Awaes Ag and Awan Ag. i.e., to meta-learn the feature extractor parameters & across tasks and to adapt the base kernel parameters I for each individual task. We choose the train loss Ly to be the negative log GP marginal likelihood evaluated on the support set ST, is common practice for choosing GP base kernel parameters:
Err(ueeta 3 aadaper ST) llg p(5 SS 2/ metaa aaaupp))
NST
S S5 S4) + log det(Kss) +
log log()
where Kss, = Kep Vam (St, ST) - INe, We choose the validation loss Ly to be the negative
log joint GP predictive posterior evaluated on the query set @T given the support set ST, also due to its common usage for making predictions with GPs:
Ly (Mmmetu vb addapt> T) = log p( 2 2 ST, metan V Wadnpe)
meta:
(9)
log (2 K Ke, ST KJ² S4. Ker - KK ST Kst Ks, @7);
where Ke, = Ki sap (OT, 27) + o² INa, and KSt or K KT ST = Heem (Sx
2+
We solve the inner optimization problem (2) using the L-BFGS optimizer (Liu & Nocedal 1989), )
since L-BFGS is the default choice for optimizing base kernel parameters in the GP literature. For the outer optimization problem (T). we approximate the expected hypergradient over (T) by averaging the hypergradients for a batch of K randomly sampled training tasks at each step, and update the meta-learned parameters 2/ meta with the averaged hypergradient using the Adam optimizer (Kingma & Ba 2014) with learning rate 10-3 3 for MoleculeNet and 10- 4 for FS-Mol. We set K 10 for MoleculeNet and K = 16 for FS-Mol. For all experiments on FS-Mol, we evaluate the performance of our model on small set of validation tasks during meta-training and use early stopping (Prechelt)
1998) to avoid overfitting of 2/ metaa
We use zero mean function and set Matérn52 without automatic relevance determination (ARD) NNeal 1996) as the base kernel in ADKF-IFT, since the typical sizes of the support sets in few-shot learning are too small to adjust a relatively large number of ARD lengthseales in ADKF-IFT. The
lengthscale in the base kernel of ADKF-IFT is initialized using the median heuristic (Garreau et al.)
2017) for each task, with log-normat prior centered at the initialization. Following Patacchiola et al.
(2020), treat binary classification +1 label regression for ADKF-IFT.
16

---

Published as a conference paper at ICLR 2023
Table 3: Statisties of four few-shot molecular property prediction benchmarks from MoleculeNet.
MoleculeNet benchmark task
Statistic
Tox21
SIDER
MUV
ToxCast
#compounds
8,014
1,427
93,127
8,615
#tasks
12
27
17
617
#training tasks
9
12
450
#test tasks
3
C DETAILS OF MOLECULENET BENCHMARK TASKS
In Table 3 we summarize the four few-shot molecular property classification benchmark tasks (Tox21, SIDER, MUV, and ToxCast) from MolcculeNet (Wu et al. 2018) considered in Section 5.1
D DETAILED CONFIGURATIONS OF ALL COMPARED METHODS FOR FS-Mol
Single-task Methods. Single-task methods (RF, KNN, GP-ST, GNN-ST, and DKL) are trained separately on the support set of each test task, without leveraging the knowledge contained in the training tasks. The implementations of RF, KNN, and GNN-ST are taken from Stanley et al. (2021). RF, KNN, and GP-ST operates top of manually curated features obtained using RDKit. RF and KNN use extended connectivity fingerprint (Rogers & Hahn 2010) (count-based fingerprint with radius 2 and size 2,048) and phys-chem descriptors (with size 42). GP-ST uses fingerprint (with radius and 2,048 bits based on count simulation). DKL. operates on top of a combination of extended connectivity fingerprint (Rogers & Hahn (010)) (count-basse fingerprint with radius 2 and size 2,048) and features extracted by a GNN. The base kernel used in DKL is the same as that used in ADKF-IFT. DKL trained for 50 epochs on the support of each test task. Hyperparameter search configurations for these methods are based on the extensive industrial experience from the authors of Stanley al. (202T). GNN-ST uses a GNN with hidden dimension of 128 and gated readout function Gilmer et al. (017), considering ~ 30 hyperparameter search configurations.
Multi-task Pretraining. The implementation of GNN-MT is taken from Stanley et (021)) GNN-MT shares a GNN with hidden dimension of 128 using principal neighborhood message aggregation (Corso et al., 2020) across tasks, and uses task-specific gated readout function Gilmer eet al. (017) and an MLP with one hidden layer on top for each individual task. The model is trained on the support sets of all training tasks with early stopping based on the validation performance on the validation tasks. The task-specific components of the model are fine-tuned for each test task.
Self-supervised Pretraining. The implementation of MAT is taken from Stanley et al. (202I). We the official pretrained model parameters (Maziarka et al.) 2020), which is pretrained 2 million molecules sampled from the ZINC15 dataset (Sterling & Irwin 2015). We fine-tuned it for each test task with hyperparameter search and early stopping based on 20% of the support set for each task.
Meta-learning Methods. Meta-learning methods (PAR, ProtoNet, GNN-MAML, CNP, DKT, and ADKF-IFT) enable knowledge transfer among related small datasets. The implementations of ProtoNet and GNN-MAMI. are taken from Stanley et al. (2021). The implementation of PAR is taken from its official implementation and integrated into the FS-Mol training and evaluation pipeline. PAR. ProtoNet, CNP, DKT, and ADKF-IFT operate on top of combination of extended connectivity fingerprint (Rogers& & Hahn 2010) (count-based fingerprint with radius and size 2,048) and features extracted by a GNN. The GNN feature extractor architecture used for DKL,, PAR, CNP, DKT, and ADKF-IFT'S is the same as that used for ProtoNet,
GNN-ST, and GNN-MT in (Stanley al.) 2221), with the size of the feature representation being tuned on the validation tasks. The base kernel used in DKT is the same as that used in ADKF-IFT.
E TASK-LEVEL EVALUATION METRICS FOR FS-MOL
Binary Classification. Following Stanley et al. (202T), , the task-level metric used for the binary classification task in FS-Mol is change in under the precision-recall curve (AAUPRC), which
17

---

Published as a conference paper at ICLR 2023
Table 4: Mean ranks of all compared methods in terms of their performance on all FS-Mol test tasks.
(a) Classification (157 tasks).
(b) Regression (111 tasks).
Support set size
Method
16
128
256
Support set
GNN-ST ST
11.29
11.53
11.75
11.85
12.19
Method
32
ANN
10.89
10.45
10.33
10.15
9.37
MAT
10.43
10.44
10.19
9.60
9.70
MAT
7.60
7.45
7.26
7.06
7.19
GNN-MT
6.61
6.40
6.15
5.95
5.68
GNN-MT
PAR
7:70
7.33
7.98
7.18
8.30
7.08
8.83
6.59
10.81
6.53
RF
5.00
4.47
4.16
3.72
3.56
DKL
7.28
8.21
4.42
5.63
6.35
GP-ST
6.71
6.37
6.28
6.18
5.14
GP-ST
4.23
4.14
3.87
3.37
3.07
GNN OAN-MAMI.
6.36
6.92
7.42
7.89
8.90
CNP
3.88
4.45
4.95
5.73
6.47
CNP
5.00
7.78
DKT
2.12
2.08
2.29
2.32
2.43
4.00
2.98
3.85
ADKE-IFT
2.12
1.86
1.68
1.74
1.36
DKT
3.44
3.19
2.99
2.90
2.67
ADKE-IFT
2.41
2.12
2.14
2.26
1.38
is sensitive to the balance of the two classes in the query sets and allows for a comparison to the performance of a random classifier:
Atageettarget classifier, 7) AUPRC(target classifier, 27) AUPRC(random classifier, 2r)
AUPRC(target classifier, Q7) #positive data points in º7
N
Regression. We propose to use the predictive/out-of-sunee coefficient of determination (R the task-level metric for the regression task in FS-Mol, which takes into account forecast errors:
Re (target regressor g. 7) E (x ) E (ym 22
Eymegy (ym Us, )2
where 7 NaT Eynesy ESy, Un is the mean target value in the support set Sy. This is different from the regular coefficient of determination (R²), wherein the total sum squares in the denominator computed using the mean target value jer Net Egm En in the query set Qr.
F FURTHER EXPERIMENTAL RESULTS ON FS-MOL
F.1 OVERALL PERFORMANCE
Table shows that ADKF-IFT achieves the best mcan rank for both classification and regression tasks all considered support set sizes. The trends these mean ranks are consistent those Figure Figures 6 and show the box plots for the classification and regression performances of all compared methods on FS-Mol test tasks, respectively. These plots are a disaggregated representation the results in Figure 2
F.2 STATISTICAL COMPARISON
Table shows shows the p-values from the two-sided Wilcoxon signed-rank test for statistical comparisons between ADKF-IFT and the next second method, namely DKT. The test results indicate that their median performance difference is nonzero (i.e., ADKF-IFT significantly outperforms DKT) for the classification task all considered support set sizes and for the regression task at support set sizes 64, 128, and 256 (at significance level 0.05). The p-values for statistical comparisons between ADKF-IFT and the two ablation models DKT+ and ADKF are also shown Table demonstrating that ADKF-IFT significantly outperforms these ablation models in most cases.
F.3 CUB-BENCHMARK PERFORMANCE
The tasks in FS-Mol can be partitioned into 7 sub- be-bbeencaarks by Enzyme Commission (EC) number (Webb et al. 1992), which enables sub-benchmark evaluation within the entire benchmark. Ideally,
18

---

Published as a conference paper at ICLR 2023
Table p-values from the two-sided Wilcoxon signed-rank test for statistical comparisons between ADKF-IFT and DKT/DKT+/ADKEF The null hypothesis is that the median of their performance differences on all FS-Mol test tasks is zero. The significance level is set a 0.05.
Support set sire
Compared
type
16
32
128
256
Classificttion
1.4 10
8.1 x 10
2.3 x 10
1.0 10
3.4 10
ADKFJIFT
Regression
x 10
9.6 x 10-2
8.7
10-7
Classification
8.2 x 10 13
7.0 x 10 16
2.3 10 18
1.2 x 9
1.6 10-
ADKF-IFT DKT+
Regression
10
4.2 10
8.4
5.2 10 10
1.7
1.1
4.8
ADKF-IFT ADKF
Regression
2.8 10
4.2 10100-
1.3 10
4.1 10
1.3 10-8
Table 6: Mean performance with standard enrors of top performing methods on FS-Mol test tasks within each sub-benchmark (broken down by EC category) at support set size 64 (the median of all considered support sizes). Note that class 2 is most common in the FS-Mol training set (~ 500 training tasks), whereas classes and are least common in the FS-Mol training set (< 50 training tasks each).
(a) Classification (AUUPCC).
FS Moll sub benchmark (EC casegory)
Method
Class
Description
Atasks
GP-ST
ProtoNet
DKT
ADKI- IFT
onidenduciases
0.156 0.044
0.152 0.040
0.137 + 0.037
0.145 0.040
0.160 + 0.045
Kinnes
125
0.152 = 0.008
0.161 = 0.009
0.285 : 0.010
0.282 4 0.010
0.299 : 0.010
hydrolaics
20
0.229 0.032
0.230
0.032
0.245 4 0.034
0.254 + 0.034
0.262 + 0.083
lyssses
ivomersses
0.166 0.040
0.212
0.052
0.172 + 0.044
0.204
0.068
0.198 + 0.046
lignecs
0.149 = 0.035
0.199 + 0.028
0.170 : 0.028
0,229 + 0.013
0.281 + 0.022
translcaaess
0.128 0. 0.039
0.109 0.049
0.089 0.028
0.122 :: 0.022
0.109 1 0.033
all anoymes
157
0.163 0.009
0.171 0.009
0.271 + 0.009
0,271 0.010
0,285 + 0.010
(b) Regression (R
ES Mal sub benchmark (EC categury)
Method
Class
Description
#usks
RF
GP-ST
CNP
DKT
ADKF-IFT IPT
oxidoreduttoses
0. 108 + 0.087
0.103 + 0.076
-0.012
10:2::000 0.011
0.098 = 0.078
0.116 1: 0.099
kinases
82
0.160 + 0.019
0.162
+ 0.022
0.127 0.017
0.343
0.022
0.368 1 0.024
hydrolases
0.256 + 0.058
0.061
0.014 1.01420.00
0.296
lystses
0.418 i 0.405
0.417
+ 0.416
0.100 0.068
0.440
0.418
0.442
0-403
ispmerses
0.125 + 0.077
0.086
+ 0.082
0.0122
0.209 0.113
0.228 1 0 0.068
ligases
0.182
+ 0.040
0.202
+ 0.079
0.002 0.004
0.277
0.035
0.279 + 0.043
all enzymes
111
0.178 + 0.019
0.181 + 0.021
0.097 0.014
D.321 0.021
0.340 + 0.022
the best method should be able to perform well across all nub-benchmarks. Table 6 shows the test performance of top performing methods on all sub-benchmarks at support set size 64 (the median of all considered support sizes) for both the classification and regression tasks. The results indicate that, in addition to achieving best overall performance, ADKF-IFT achieves the best performance on all sub-benchmarks for the regression task and on more than half of the sub-benchmarks for the classification task.
F.4 META-TESTING COSTS
Figure shows the meta-testing costs of all compared meta-learning methods terms of wall-clock time" on pre-defined set of FS-Mol classification tasks. These experiments are run a single NVIDIA GeForce RTX 2080 Ti. can be seen that ADKF-IFT is 2.5x slower than CNP, ProtoNet, and DKT, but still much faster than GNN-MAMI. We did not report the wall-clock time for PAR, because it is extremely memory intensive (PAR takes > 10x memory than ADKF-IFT does) and thus
We acknowledge that wall-clock time may not be the best metric for measuring the costs, since some meta-learning methods could be parallelized, which will reduce the wall-clock time accordingly. An alternative operation (MAC). However, it is difficult to obtain the accurate number of MACs
metric is multiply-accumul due the opaqueness of the GP modules used.
19

---

Published as a conference paper at ICLR 2023
Walkk time (seconds)
500
400
300
200
100
CNP
ProtoNet
DKT
ADKF-IFT
GNN-MAM
Figure 5: Wall-clock time consumed (with standard errors) when meta-testing pre-defined set of FS-Mol classification tasks using each of the compared meta- meta-learning methods.
Table 7: Descriptions of four out-of-domain molecular design tasks.
Molevalar design task
Daiu source
#compounss
Turget
Tirget source
Molocular docking (FSR3)
DOCHSTRING training set
2,312
hinding scate relative growth
AutoDack screesing experimental lub
Autibooric diseowary (E. coll BW25113)
Autitiotic training set
2,335
Antiviral drug design (SARS 2)V-2)
COVID Moonshot Harvad Energy Project
1,926
PICSO Floorsseence
Material design (Orgeiic Photovoltsic)
power convorsion efficiency
similation
cannot be GPU. We stress that this is not important metric for this paper, real-time adaptation not required in drug discovery applications, but could be of interest if ADKF-IFT were to be deployed in other settings.
G DETAILS OF THE OUT-OF-DOMAIN MOLECULAR OPTIMIZATION EXPERIMENTS
In Table 7 we summarize the four molecular design tasks considered in Section 5.3 Note that the datasets for the molecular docking and material design tasks are subsampled from the much larger datasets provided in DOCKSTRING (Garefa-Ortegol et al. 2021) and Harvard Clean Energy Project (Hachmann al. 2011), respectively. The datasets for the antibiotic discovery and antiviral drug design tasks are taken from the antibiotic training set and the COVID Moonshot dataset provided in
Stokes et (2020) and Consortium et al. (2022), respectively.
For the configuration of the GP, we use the Tanimoto kernel for fingerprint (with radius 2 and 2,048 bits based on count simulation) and Matérn52 kernel without ARD (with log-normal prior over the lengthscale, centered at the median heuristic initialization) for all the other compared feature representations. We re-fit the base kernel parameters using all available data points at the beginning of each BO iteration.
H DISCUSSIONS OF METHODS FOR LEARNING DEEP KERNEL GPs
H.1 THE OVERFITTING ISSUE IN DKL
Deep Kernel Learning (DKL) Wilson et al. 2016b) is a single-task method for fitting a deep kernel to dataset. DKL jointly fits both the feature extractor parameters and base kernel parameters by maximizing the GP marginal likelihood on single dataset (i.e., DKL essentially fits a neural network with GP "head" dataset).
is well known that neural networks will easily overfit to small datasets Saale 1995). This overfitting also happens in DKI,, despite the fact that it fits the neural network parameters using type-Il
20

---

Published as a conference paper at ICLR 2023
maximum likelihood approach: although early DKL papers suggested that the "model complexity" term (as measured by the log determinant of the kernel matrix) in the GP marginal likelihood objective would prevent this overfitting from happening (Wilson et al. 2016b), recent follow-up up work showed that this is not the case (Ober et al.) (021) deep-kernel GP can simultaneously overfit to the training data and appear have low "model complexity".
H.2 THE UNDERFITTING ISSUE IN DKT
Deep Kernel Transfer (DKT) (Patacchiola et al. 2020) is meta-learning method for fitting deep kernel to a distribution of datasets. DKT jointly fits both the feature extractor parameters & and base kernel parameters f by maximizing the expected GP marginal likelihood over distribution of datasets.
To mitigate the overfitting issue of DKL using meta-learning, DKT makes a very strong assumption that different tasks in the task distribution are drawn from an identical GP prior over functions. Explicitly, this means that the data generating process is assumed to have the same noise level, same amplitude, and same "characteristic lengthscale" for every task in the meta-dataset, which a very restrictive assumption violated by most real-world problems. For example, different datasets in meta-dataset may have
highly varying noise levels, so modelling all tasks with the same amount of observation noise will not be realistic;
different output ranges and units for regression: for example, one task might have data in the range 1-20 μM, while another might have 0-100% inhibition, meaning that a single signal variance (kernel amplitude) will not model the data well;
different "characteristic lengthscales": for some tasks, structurally similar molecules have very strongly correlated output labels, while for other tasks it is much weaker (i.e., there is much more variation in the labels of very similar molecules), suggesting that the "character- istic lengthscale" will be different.
Inevitably, trying to fit such a misspecified model will result in a set compromised base kernel parameters 0 which fit all datasets okay on average but do not fit each individual dataset very well. This is the underfitting issue of DKT.
H.3 THE ADVANTAGE OF ADKF-IFT
ADKF-IFT combines DKL and DKT in a way that can potentially inherit the strengths of both methods and the weaknesses of neither. By adapting the base kemnel parameters 0 specifically to each task, it prevents underfitting due to varying ranges, lengthscales, or noise levels between datasets. By meta-learning the feature extractor on many datasets, it prevents overfitting as observed by Ober et al.) (202I). This advantage is both theoretically principled (by solving bilevel optimization objective using the implicit function theorem) and empirically observable (we showed statistically significant performance improvement of ADKF-IFT over DKT in Section 5.2)
I EXTENDED RELATED WORK
I.1 MULTI-TASK GAUSSIAN PROCESSES
ADKF-IFT can be considered a method learn multi-task GPs. The dominant approach this problem in prior works is to learn shared kemnel for all data points across all tasks, transmitting information by explicitly modelling the covariance between data from different tasks (Kennedy &
O' O'Hagan 2000, Forrester et al.] 2007 Bonilla et al. (Sorskyyy et al. 2013) Poloczek al.) 2017). The main difference between methods in this family is the exact form of the kernel (Tighineanu et al. 2022), which is typically assumed to have particular structure (e.g. Kronecker product weighted sum of simpler kernels). ADKF-IFT cannot be naturally viewed in this way because the covariance between data points from separate tasks is always zero; information is instead transmitted between tasks via a shared set of kernel parameters. Therefore, we believe that ADKF-IFT a significant departure from the dominant paradigm in GP transfer learning.
21

---

Published as a conference paper at ICLR 2023
1.2 IMPLICIT FUNCTION THEOREM IN MACHINE LEARNING
The implicit function theorem employed in our work has been used in many previous machine learning
papers various contexts, e.g., neural architecture search (Zhang al. 202T), hyperparameteer-uniin
(Bengio 2000 Luketina et al. 2016, Pedregosa, 2016 Lorraine et al.
et al.) 2022). and
meta-learning RRaessarran et al. 2019; Lee et al. 2019 Chen al. 2020).
1.3
MODULAR META-LEARNING WITH SHRINKAGE (CHEN
2020
The method proposed by Chen et al. level it also divides model parameters into meta-learned and adapted parameten and optimizes the meta-learned parameters using the gradient of the validation loss after the adapted parameters have been adjusted to minimize the training loss using the implicit function theorem. The main differences between this work and ADKF-IFT are:
2020) shares many similarities with ADKK-IFFT at a high
1. Model: Chen et al. 2020) consider model where Φ are the means and variances of
Gaussian prior over model parameters, whereas in ADKF-IFT is subset of the parameters of an arbitrary deep kernel GP.
2. Hessian: Chen al. (2020) consider the case where θ is too large form the exact Hessian for the implicit function theorem. They instead use a conjugate gradient approximation. Although some instantiations of ADKF-IFT could require this, in our highlighted version the Hessian can be computed exactly, which we view as significant advantage.
3. Goal: The stated goal of Chen et al. (2020) is to decide which parameters should be meta- learned, while in ADKF-IFT this must be pre-specified, and give guidance for doing so in a way that results in transferable meta-learned features.
1.4 COMMENTS ON ADKL- GP (Tossou ET AL. 2019)
The preprint of Tossou al. 2019) proposes alternative adaptive deep kernel GP trained with meta-learning, where adaptation is performed by conditioning the feature extractor on an embedding of the entire support rather than adjusting a subset of the kernel parameters as in ADKF-IFT. In general their empirical results were not very strong, and in our opinion the method is very prone to overfitting, which we explain below.
The training objective for ADKL-GP is equivalent to the objective for DKT with an added contrastive loss, weighted by sensitive hyperparameter (see Equation (13) of Tossou et (2019)). y can be interpreted as balancing the degree of regularization between two extremes:
1. If y = O, there no regularization of the task encoding network, making significant overfitting to the meta-dataset possible. This is effectively equivalent to standard DKL WWllson et al. 2016b).
2. As n - oo, the regularization becomes infinitely strong, causing the task embeddings zt to collapse, and thereby preventing them from transmitting any information about specific datasets. With no information from 27 in this case, the objective is essentially the same as that of DKT (Patacchiola et al.) 2020).
For this method to be useful it would appear that y would need be carefully tuned to balance between these extremes. Tossou et al. (2019) perform a grid search over all hyperparameters including y (0, 0.01, 0.1} but find no consistent trend besides y > 0 being slightly helpful, although the differences in performance were small. This suggests that the method may be difficult to use in practice. ADKF-IFT however has such tunable hyperparameters, which we view significant strength. Instead, the balance between DKL and DKT is controlled by selecting which parameters are adapted and meta-learned, which is much more interpretable and makes easier to use in practice.
J FUTURE WORK
Some directions for future work follows:
Thhy the meta parameters and task-specific parameters instead.
22

---

Published as a conference paper at ICLR 2023
1. using ARD in the base kernel that feature selection for each individual task can be done by the GP model, with potential overfitting problems being reduced by assuming a sparse prior over lengthscales by learning now-dimensional manifold for them;
2. adapting the feature extractor to each task well by allowing small deviations across tasks according to meta-learned prior on the feature extractor parameters (e.g., as described in Chen et al. (2020));
3. adopting a more principled approximate inference strategy for few-shot GP classification (e.g., Pólya-Gamma data augmentation (Snell & Zemel 2020) or Laplace approximation (Kim & Hospedales) 202T));
4. injecting domain expertise in drug discovery into the base kemell with hand-curated features and kernel combinations.
23

---

Published as a conference paper at ICLR 2023
DKL
DKL
PAR
GNN-ST
MNN
KNN
9.00
MAT
MAT
RF
RF
GNN
GP-ST
CPSS
GNN MAML
GNN-MAMI
CNP
CNP
ProtoNes
DKT
ADKF-ITT
ADKF-IFT
0.0
0.2
0.3
AAUPRC
AAUPRC
(a) Ns..
(b) NST.
32.
DKL
DKL
GNN-ST
GNN-ST
KNN
000
KNN
MAT
MAT
RF
RF
MT
GP
GP-ST
GNN-MAML
GNN-MAMI
CNP
CNP
ProtoNet
ProtoNet
ADKEJITT
ADKF-IFT
0.3
AAUPRC
AAUPRC
(c) NsT.
64.
(d) Ns.. = 128.
DKL
GNN ST
KNN
MAT
GP-ST
MAML
CNP
ProtoNet
DKT
ADKF-IFT
AAUPRC
(c) NST. 256.
Figure 6: Box plots for the classification performance of all compared methods on 157 FS-Mol test tasks at different support set sizes.
24

---

Published as a conference paper at ICLR 2023
DKL-
DKL
GNN-MT
GNN MT
RF
RF
GP-ST-
GP-ST
CNP
CNP
DKT
DKT
-088
-05
-04
3.2
0.0
0.2
0.4
0.6
00.6
-044
-0.2
0.0
0.2
0.4
0..6
0.8
Rg
R2,
(a) NsT_
16.
(b) NST.
32.
DKL
DKL
MAT
MAT
MT
MT
RF
RF
GP-ST
GP.ST
CNP
CNP
DKT
DKT
ADKF-IFT
ADKFJIFT
-0.75
-0.50
-0.25
0.00
0.255
0.30
0.75
-2
-1
(c) NST.
64.
(d) Ns
128.
DKI
MAT
GNN-MT
RF
GP-ST
DKT
ADKF-IFT
0.0
0/2
0.4
0.6
0.8
RC.
(e) Ns..
256.
Figure 7: Box plots for the regression performance of all compared methods 111 FS-Mol test. tasks at different support set sizes.
25


Reviews:
[{"review_id": "SND64PGki8F", "reviewer": "Reviewer_sNf8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In order to fit molecular property prediction models that give good uncertainty estimates but also overcome issues associated with neural network methods that either overfit or underfit Gaussian Processes, the authors propose Adaptive Deep Kernel Fitting with Implicit Function Theorem (ADKF-IFT), a generalization of Deep Kernel Learning and Deep Kernel Transfer that meta-learns a subset of parameters (usually those associated with a feature extractor) across many tasks, and transfer learns the rest (usually those associated with the base kernels of Gaussian Processes). The authors propose a bilevel optimization method that avoids costly inner optimization gradient calculations by invoking Cauchy's Implicit Function Theorem. The authors demonstrate how ADKF-IFT outperforms a large variety of existing methods of molecular property prediction tasks in MoleculeNet and few-shot prediction tasks in FS-MOL. They also show how the feature representation learned by ADKF-IFT can be used with Gaussian Process surrogate models to provide optimal or near-optimal results in out-of-domain molecular optimization tasks like molecular docking and antiviral drug design. Using ablation studies, the authors also show that the bilevel optimization is what gives the reported performance gains.", "summary_of_the_review": "This paper makes a significant contribution to Gaussian Process training methods by proposing a novel bilevel optimization method that achieves state-of-the-art molecular property prediction. Such models could have significant impact on the work of researchers applying ML to drug discovery, or other related topics.", "strength_and_weaknesses": "Strengths:\n- ADKF-IFT provides the means to effective training of Gaussian Processes that overcomes the shortcomings of the two methods it generalizes, DKL and DKT while achieving state-of-the-art predictive power.\n- ADKF-IFT achieves state-of-the-art results on many molecular property prediction benchmarks, including few-shot tasks.\n- ADKF-IFT appears to learn informative molecular features that perform well for entirely different tasks.\n\nWeaknesses:\n- The authors do not include any examples that demonstrate that ADKF-IFT is neither over- or underfit, nor do they demonstrate that the uncertainty estimates derived from the learned GPs are better than those of GPs learned in other ways.\n- MoleculeNet is somewhat out-of-date - the authors should consider the superset of tasks proposed in the Therapeutics Data Commons (https://tdcommons.ai/) database.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RmuUpXufG43", "reviewer": "Reviewer_fnkh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new meta-learning framework formulated as a bilevel optimization problem. The implicit function theorem is utilized to efficiently compute the gradient of the bilevel optimization problem. It is shown that in the context of deep kernel learning the proposed framework encompasses DKL and DKT as extreme cases. In the wide range of models under this framework, the author proposes ADKF-IDF which meta-learns the parameters of the feature extractors and fine-tunes the parameters of the base kernel. This specific design choice avoids the computational hassle in the gradient computation and induces a natural interpretation of deep kernel learning in meta-learning. The ADKF-IDF is extensively compared to existing methods on few-shot learning tasks predicting molecular properties and on Bayesian optimization of molecular properties.", "summary_of_the_review": "The paper proposes a new meta-learning framework and a specific instance of it that is shown to be effective in few-short learning and out-of-sample prediction demonstrated via Bayesian optimization experiment. With a good combination of many technical tools under the new meta-learning framework, the paper has good empirical contributions. As long as my concerns mentioned in 'Weakness' can be addressed, I am willing to increase my score and support the acceptance of the paper.", "strength_and_weaknesses": "### Strengths\n- The paper proposed a meta-learning framework that is formulated via bilevel optimization and optimized using the implicit function theorem with general applicability.\n- In its application to deep kernel models, the authors propose a deep kernel model with a meta-learned feature extractor and fine-tunable GP parameters. This not only circumvents the computational hassle of the proposed but also gives a quite interpretable hierarchical structure. While including existing methods, DKL and DKT as special cases, this specific model is a balanced interpolation of both under the proposed meta-learning framework.\n- The proposed ADKF-IFT is extensively compared with existing methods on molecular property prediction few-shot learning. \nEspecially, in the experiments on FS-MOL benchmarks, it seems that much effort and resources have been put into the experiment, which provides also good baselines for others as the authors benefited from the baseline results in MoleculeNet experiments.\n- In addition to its strengths in meta-learning, the authors also conduct Bayesian optimization experiments which supports the good quality of learned representation in the proposed meta-learning framework and the well-calibrated uncertainty of ADKF-IFT.\n\n\n\n\n### Weaknesses\n- Is a random initialization of $\\psi_{adapt}$ enough to guarantee the below?\n$$\nIF(\\psi_{adapt} | \\psi_{meta}, S_T) = \\frac{\\partial L_{train} (\\psi_{meta}, \\psi_{adapt}, S_T)}{\\partial \\psi_{adapt}} = 0\n$$\n    - In Appendix B, it says that in ADKF-IFT, the fine-tuned parameter $\\psi_{adapt}$ is initialized using the median heuristic.\n    - It seems that such initialization does not guarantee $IF(\\psi_{adapt} | \\psi_{meta}, S_T) = 0$.\\\nDoes the median heuristic generate a point $\\psi_{adapt}$ such that $IF(\\psi_{adapt} | \\psi_{meta}, S_T) = 0$?\\\nIs there a mechanism that drives a point $\\psi_{adapt}$ such that $IF(\\psi_{adapt} | \\psi_{meta}, S_T) \\neq 0$ to a point such that $IF(\\psi_{adapt} | \\psi_{meta}, S_T) \\neq 0$?\n- How is it guaranteed that $\\psi_{adapt}^*$ is a minimum?\n    - From Thm.1, it seems that when the implicit function theorem is used, implicitly defining function is \n$$\nIF(\\psi_{adapt} | \\psi_{meta}, S_T) = \\frac{\\partial L_{train} (\\psi_{meta}, \\psi_{adapt}, S_T)}{\\partial \\psi_{adapt}} = 0\n$$\nThis implies that for a given $\\psi_{meta}$, $\\psi_{adapt}$ satisfying the above equation is a maximum, a minimum, or a saddle point.\n    - I was wondering how $\\psi_{adapt}$ is guaranteed to be a minimum.\\\nEven if the initialization guarantees $IF(\\psi_{adapt} | \\psi_{meta}, S_T) = 0$ or the gradient descent eventually makes the point to satisfy $IF(\\psi_{adapt} | \\psi_{meta}, S_T) = 0$, \n    - It seems unlikely that the minimization of upper validation loss is obtained with the maximization of lower train loss. \\\nThis may imply that some care is necessary for the choice of two losses in this bilevel meta-learning framework.\\\nI agree that the specific instantiation of ADKF-IFT seems reasonable also in this respect.\\\nStill, I am curious about the authors' thoughts and opinions on this.\n- Sensitivity to the initialization\n    - Even if it is guaranteed that $\\frac{\\partial L_{train} (\\psi_{meta}, \\psi_{adapt}, S_T)}{\\partial \\psi_{adapt}} = 0$, it is imaginable that, in some case, $\\psi_{adapt}$ is initialized to a point of a connected component of $\\\\{\\psi_{adapt} | \\frac{\\partial L_{train} (\\psi_{meta}, \\psi_{adapt}, S_T)}{\\partial \\psi_{adapt}} = 0 \\\\}$ which consists of maximum points.\nConsidering this case, I am curious about the sensitivity of the method to the initialization or the choice of hyperparameters (some hyperparameters may exclude such bad cases).\n\n\n\n\n\n### Minor points\n- It seems that this somewhat seemingly relevant work is missing\n    - Few-Shot Bayesian Optimization with Deep Kernel Surrogates; ICLR 2021\n    - I am not asking to include this as an additional baseline. \n    - The comparison with this seems to be able to further highlight the benefit of ADKF-IFT.\n- Significance of the differences in the ablation study.\n    - I can check visually the consistency of the superiority of ADKF-IFT in Figure 3. But in many cases, the confidence intervals (I guess they are from repeated experiments with some randomness, it is reasonable to call them confidence intervals) overlap. Maybe providing some simple statistical results may strengthen the argument of the ablation study.\n- Duplicated names\n    - In the paper, ADKF-IFT stands for two things, 1) meta-learning framework formulated by a bilevel optimization problem and trained using the implicit function theorem, 2) deep kernel GP model with meta-learned feature extractor and fine-tuned GP parameters using the proposed meta-learning framework\n    - Saving ADKF-IFT to the proposed specific deep kernel methods, it may be less confusing to use another name for the proposed meta-learning framework reflecting its generality, for example, Meta-BIFT (meta-learning formulated via BIlevel optimization and optimized via Implicit Function Theorem)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9X_Q3OcoiVV", "reviewer": "Reviewer_BY35", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents ADKF-IFT -- a new method to train deep kernel GPs using meta-learning. This method splits the parameters into two sets, where only one set is adapted to each task. This leads to a bi-level optimization problem, that the authors solve using the Implicit Function Theorem. The number of parameters which are adapted can help in balancing between over- and under-fitting, which helps the model perform well on tasks with small training datasets. The paper also shows that prior methods for training deep kernel GPs, DKT and DKL, are special cases of the proposed method.\n\nExperimental results from chemistry show that the model generally performs competitively on small data domains, and also perform well on OOD tasks.", "summary_of_the_review": "The paper presents a novel method for training deep kernel GPs which obtains strong results on small datasets. The proposed method is shown to be a generalization of existing methods. Overall, I find this work to be sufficiently novel and useful for many chemistry problems to justify acceptance.", "strength_and_weaknesses": "### Strengths\n* The paper presents a novel method for training deep kernel GPs that is competitive on small data domains.\n* The proposed method can help balance over- and under-fitting.\n* The paper demonstrates that the proposed method subsumes the existing DKT and DKL methods as special cases.\n\n### Weaknesses\n* It would be instructive to understand how the model performs on tasks with larger training datasets. I understand that is not the goal of the paper, but understanding the failure modes of the method (if there are any) can be useful.\n* How sensitive is the method to the choice of \\Psi_{adapt}, the parameters that are adapted to each task? Providing a discussion on this would be useful\n* Provide more details about the hyperparameters & architectures for reproducing the results.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster