MODELLING   BIOLOGICAL   ASSAYS   WITH ADAPTIVE DEEP KERNEL LEARNING
Authors: Prudencio Tossou
Basile Dura
Daniel Cohen
Mario Marchand
François Laviolette
Alexandre Lacoste
Venue:  (2020)
Keywords: few-shot learning
few-shot regression
deep kernel learning
biological assay modelling
drug discovery

Abstract: Due to the significant costs of data generation, many prediction tasks within drug discovery are by nature few-shot regression (FSR) problems, including accurate modelling of biological assays.  Although a number of few-shot classification and reinforcement learning methods exist for similar applications, we find relatively few FSR methods meeting the performance standards required for such tasks under real-world constraints. Inspired by deep kernel learning, we develop a novel FSR algorithm that is better suited to these settings. Our algorithm consists of learning a deep network in combination with a kernel function and a differentiable kernel algorithm. As the choice of the kernel is critical, our algorithm learns to find the appropriate one for each task during inference. It thus performs more effectively with complex task distributions, outperforming current state-of-the-art algorithms on both toy and novel, real-world benchmarks that we introduce herein. By introducing novel benchmarks derived from biological assays, we hope that the community will progress towards the development of FSR algorithms suitable for use in noisy and uncertain environments such as drug discovery.

Under review as a conference paper at ICLR 2020
MODELLING BIOLOGICAL ASSAYS WITH ADAPTIVE DEEP KERNEL LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Due to the significant costs of data generation, many prediction tasks within drug discovery are by nature few-shot regression (FSR) problems, including accurate modelling of biological assays. Although a number of few-shot classification and reinforcement learning methods exist for similar applications, we find relatively few FSR methods meeting the performance standards required for such tasks under real-world constraints. Inspired by deep kernel learning, we develop a novel FSR algorithm that is better suited to these settings. Our algorithm consists of learning a deep network in combination with kernel function and a differentiable kernel algorithm. As the choice of kernel is critical, our algorithm learns to find the appropriate kernel for each task during inference. thus performs more effectively with complex task distributions, outperforming current state-of-the-ard algorithms on both toy and novel, real-world benchmarks that we introduce herein. By introducing novel benchmarks derived from biological assays, we hope that the community will progress towards the development of FSR algorithms suitable for use in noisy and uncertain environments such drug discovery.
2
6
10
12
13
14
15
16
INTRODUCTION
Following breakthroughs in domains including computer vision, autonomous driving, and natural language processing, deep learning methods are now entering the domain of pharmaceutical R&D. Recent successes include the deconvolution of biological targets from -omics data (Min al., 2017), generation of drug-like compounds via de novo molecular design (Xu et al., 2019), chemical synthesis planning (Segler and Waller, 2017; Segler et al., 2017), and multi-modal image analysis for quantification of cellular response (Min et al., 2017). A common characteristic of these applications, however, is the availability of high quality, high quantity training data. Unfortunately, many critical prediction tasks in the drug discovery pipeline fail to satisfy these requirements, in part due to resource and cost constraints (Cherkasov et al., 2014).
18
19
20
21
22
23
25
We therefore focus this work modelling biological assays (bio-assays) relevant in the early stages of drug discovery, primarily binding and cellular readouts. Under the constraints of an active drug discovery program, the data from these assays, consisting of libraries of molecules and their associated real-valued activity scores, is often relatively small and noisy (refer to statistics in Section 5). In many contexts, it can be challenging build a training set of even a few dozen samples per individual assay. Modelling an assay is thus best viewed as few-shot regression (FSR) problem, with many variables (including experimental conditions, readouts, concentrations, and instrument configurations) accounting for the data distribution being generated. Practically, these variables make it infeasible to compare data collected across different assays, thereby making it difficult learn predictive models from molecular structures. Furthermore, bio-assay modelling is intended to be used for prioritizing molecules for subsequent evaluation (e.g. Bayesian optimization) and efficiently exploring the overall chemical space (c.g. active learning), accurate prediction and uncertainty estimation using few datapoints critical to successful application in drug discovery.
27
29
30
31
32
33
34
36
38
It is view that robust FSR algorithms are needed to tackle this challenge. Specifically, argue 40 that these algorithms should remain accurate in noisy environments, and also provide well-calibrated uncertainty estimates to inform efficient exploration of chemical space during molecular optimization. 42 Fortunately, recent advances in few-shot learning have led to new algorithms that learn efficiently and generalize adequately from small training data (Wang and Yao, 2019; Chen al., 2019). Most

---

Under review as a conference paper at ICLR 2020
have adopted the meta-learning paradigm (Thrun and Pratt, 1998; Vilalta and Drissi, 2002), where 45 some prior knowledge is learned across a large collection of tasks and then transferred to new tasks
46 in which there are limited amounts of data. Such algorithms tend to differ in two aspects: the nature
47 of the knowlege captured and the amount of adaptation performed at test-time for new tasks or datasets. The meta-knowledge refers to the domain specific prior needed to solve each task most effectively. Due to the size of the total chemical space accessible when modelling bio-assays (Bohacek al., 1996), there a particular need for the knowledge to be sufficiently rich as to allow for extrapolation and uncertainty estimation in unseen regions of chemical space at test- test-time (i.e. for new tasks). Given that the same molecule can behave differently across different assays, greater test-time adaptation is also required and must be accounted for during modelling.
49
50
51
52
53
54 In previous work, metric learning methods (Koch et al., 2015; Vinyals al., 2016; Snell et al., 2017; Garcia and Bruna, 2017; Bertinetto et al., 2018) accumulate meta- knowlegee in high capacity covariance/distance functions and use simple base-learners such as k-nearest neighbor (Snell et al., 2017; Vinyals et al., 2016) or low capacity neural networks (Garcia and Bruna, 2017) to produce adequate models for new tasks. However, they do not adapt the covariance functions nor the base- learners at test-time. Initialization- and pptimizatiio-bassed methods (Finn et al., 2017: Kim et al., 2018; Ravi and Larochelle, 2016) that learn the initialization points and update rules for gradient descent-based algorithms, respectively, allow for improved adaptation consuming and memory inefficient. We therefore argue that to ensure optimal performance when modelling bio-assays. is crucial to combine the strengths of both types of methods while also allowing for the incorporation of domain-specific knowledge when making predictions. We achieve this by framing FSR as a deep kernel learning (DKL) task, deriving novel algorithms that we apply to modelling specific assays and readouts.
55
56
58
60
61
new tasks but remain time
62
63
64
65
66
67
Our contributions are folde-fold. We first frame few-shot regression as a DKL problem and showcase its advantages relative to classical metric learning methods. We then derive the adaptive deep kernel learning (ADKL. framework by learning conditional kernel function that is task dependant, allowing for more test-time adaptation than the DKI. framework. Finally, we
Contributions:
68
69
70
71 introduce two real-world datasets for modelling biological assays using FSR. With this contribution, we hope to encourage the development of subsequent few-shot regression methods suitable for
73 real-world applications, as is the case for few-shot classification and reinforcement learning, each of
74 which have received comparatively greater attention in recent years (Wang and Yao, 2019).
75 2
DEEP KERNEL LEARNING
76 In this section, we describe the DKL framework introduced for single tasks by Wilson et al. (2016).
77 We then extend it to few-shot learning and discuss its advantages over metric learning algorithms.
78 Single Task DKL: Let Dí ((x, yi) C X x R. training dataset available for learning
79 the regression task t where X' is the input space and IK is the output space. A DKL algorithm aims to obtain non-linear embedding of inputs in the embedding space H. using deep neural network
81 Φg X + H of parameters 0. It then finds the minimal norm regressor h², in the reproducing kernel
82 Hilbert space (RKHS) R on H. that minimize an objective function such as
h = argmin X |l + E(h, D(m)
hER
83 where L is non-negative loss function that measures the loss of a regressor and À weighs the 84 importance of the norm minimization against the training loss. Following the representer theorem a5 (Scholkopf and Smola, 2001; Steinwart and Christmann, 2008), h² can be written as a finite linear 86 combination of kerel evaluations on training inputs, i.e.:
Σ (X..
(x)
where a' = (af..... o are the learned combination weights and kp: H x 7 R4 is chosen reproducing kernel of R with hyperparameters p. Candidate kernels include the radial basis, polynomial, and linear kernels. a' can be obtained by using a differentiable kernel method enabling 90 the computation of the gradients of the loss w.r.t. the parameters 0. Such methods include Gaussian 91 Process (GP), Kernel Ridge Regression (KRR), and Logistic Regression (LR).
88
2

---

Under review as a conference paper at ICLR 2020
As DKL inherits from deep learning and kernel methods, it follows that gradient descent algorithms
93 required to optimize the network parameters 0. The latter can be high dimensional and a substantial amount of training samples required to train DKL models and avoid overfitting. However, once 85 the latter condition is met, scalability of the kernel method can be limiting (running time in ((m)) for training samples) and approximations can be needed for scalability (see Williams and Seeger (2001); Wilson and Nickisch (2015)).
87
96
Few- Shot DKL:
In few-shot learning, one has access to a meta- meta- training collection Dmeta-tre
(Dir D'aa)) )" of 'I' tasks Each task t, has its own training (or support) set D and validation ;=1 (or query) set D A meta-testing collection Dmeta-tst tst is also available to assess the generalization performance of the few-shot algorithm across unseen tasks. To obtaín a Few-Shot DKL (FSDKL) can share the parameters of dg across all tasks, similar to metric method for FSR in such settings, learning algorithms. Hence, for a given task ty. the inputs are first transformed by the function do and then a kernel method is used to obtain the regressor h.' which will be evaluated on D Here, KRR and GP are explored as they are the state-of-the-art algorithms for kernel-based regression. The latter is used to allow models to provide accurate predictive uncertainty, which is useful when prioritizing molecules in the context of drug discovery.
96
101
103
105
107
Using the squared loss and the L2-norm compute | KRR gives the optimal regressor for a task t and its validation loss p.p as follows:
KRR:
(x) axx,tra with a
a=
==KKrm,,tr + A1) Vern
(3)
Copp=
E x.yr-Da.
(aK - -y
(4)
108 where ytm = (U1," "ID),...))" Ktrntrn is the matrix of kernel evaluations where entry i, is (x,)) 0 (x)) for pairs of examples in Dim An equivalent definition applies to Kx,rnn
GP: When using the negative log likelihood loss function, the GP algorithm gives a probabilistic regressor for which the predictive mean, covariance, and loss for a task t are:
C.
(y covch()).
(5)
E[h[] = Kval,tra (Kerntra + AI) AI)-"yytn ytrn.
(6)
coc (h,,) = K - Kwalten Keen,rrn + XT) -i-'Keee Keen,val
(7)
110 Finally, the parameters O of the neural network, along with X and the kernel hyperparameters p, are I1 optimized using the expected loss on all tasks:
argmin
Co
(8)
O.p.A to mw-.met--ee
112 To summarize, FSDKL finds representation common to all tasks such that the kernel method (in
113 case, GP and KRR) will generalize well from a small amount of samples. In doing so, this alleviates two of the main limitations of single task DKL:: i) the scalability of the kernel method is no longer an issue since we are in the few-shot learning regime', and ii) the parameters O (and p, A) are learned across potentially large amount of tasks and samples, providing the opportunity to learn rich representation without overfitting.
115
116
117
118
Despite shared characteristics with the metric learning framework, the FSDKL framework is more powerful and flexible. It provides better task-specifh adaptation due to the inference of the appropriate model using the kernel methods compared to shared model parameters in metric learning. After meta- training, any task-specific model also inherits the generalization guarantees of kernel-based models, and consequently increasing the number of shots for new tasks can only improve generalization performance. The incorporation of prior knowledge through user-specific kernel functions is also a major advantage of DKL over metric learning (e.g. use periodic kernels for periodic function regression tasks).
119
120
122
124
Even with several hundred samples, the computational cost of emhedding each example is usually higher than inverting the Gram matrix.
3

---

Under review as a conference paper at ICLR 2020
3
ADAPTIVE DEEP KERNEL LEARNING
127
In this section, we present a new algorithm, deemed adapative deep kernel learning (ADKL). Funda- mentally, it differs from FSDKL by having more flexibility in its kernel definition and by learning to produce task-specific kernel functions during the meta-training instead using one defined by the user. It does so by learning task representation using task encoding network 2/,, and leveraging it to build task-specific kernels using multi-modal neural network cp. More explicitly, given a task t,
129
131
ADKL first computes task embedding z, = n,,, (Dir,) using its support set Dern and then infers the adapted kernel with Cp. We describe in more detail both the task encoding network v,, and the
132
133
134
network Cp responsible for computing the task task-specific kernel below.
Kunt
D vol
h
Jj V)
Cp
C
D
KRR
Rows are GDt)
Kyrure
Vy
Yi
Figure 1: ADKL-KRR. The blue and orange colors show the procedure for a task during internal train and test, respectively. During training, ADKL first computes task embedding 2; 1/ (Di..) that is used with pseudo-representationn U by the network cp produce a the task-specific kernel function. The empirical kernel map of this kernel gives the function (-) ()) that evaluated for every training point to produce Kirntrr. The latter and the train targets are used by KRR (or GP) to produce the model lt; At evaluation, (-) is evaluated again for every test point to obtain Krad,trn. which is used to compute the predictions. The loss is then computed and used to update all parameters of ADKL.
135
3.1
TASK ENCODING
The challenge of the network 1b, is to capture complex dependencies in the training Dir,, to provide a useful task encoding z,- Furthermore, the task encoder should be invariant to permutations of the training set and be able to encode variable amount of samples. After exploring a variety of architectures, we found that those that are more complex, such as Transformers (Vaswani et al., 2017), tend to underperform. This possibly due to overfitting or the sensitivity of training such architectures.
136
137
136
139
140
142
Consequently, inspired by DeepSets (Zaheer et al., 2017), we propose a simple order invariant network that captures the first and second order statistics of regression datasets. Given dataset, this network first processes each of its samples individually as follows: a) extract input features using Φg (see section2), b) concatenate the input features with the target and embed the obtained vector using a simple fully connected neural network I, of parameters 17 2. then computes the first and the second order statistics of the obtained vectors for all samples of the dataset and concatenates them to produce the representation. More formally,
144
146
148
ab (Dirn) = [u', od], with jz
F, (x)))())) vi)), of (x.))
E
Var (x:()))())) vil) (x,g))
where [ the concatenation operator. As je' and of are invariant to permutations in Di follows that 2/, is also permutation invariant. Overall, & is simply the concatenation of the first and second moments of the sample representations, which were nonlinear transformations of the original inputs and targets.
149
150
151
152
To help the training of the parameters 17, we add a regularization term that maximizes the mutual information between Dir., and Dr This encourage the network to produce similar task encodings
These are the only parameters involved in the computation of the task encoding, which is why we also the notation ab,,.
4

---

Under review as a conference paper at ICLR 2020
when presented with different data partitions for given task. Concretely, we maximize the lower bound on the mutual information between the task representations given by the support and the query sets instead of the true mutual information (Belghazi et al., 2018). Using batch of b tasks and the cosine similarity as the similarity measure the between two task encodings, this lower bound I, is defined by Eq. (9) and is the regularizer that we used to a have better task encoder.
In def clbb, (D 3 (D,( In b(6-1j
(9)
153
3.2 TASK-SPECIRE KERNEL
Here, we describe how the task-specific kernels are inferred using the task representations described previously. In fact, they are all obtained using a multi-modal neural network cp of parameter Given any pair of input representations (ø(x) and of ((x)) and task encoding 2,, this network simply computes the input pair similarity under the condition given by the task encoding as follows:
155
156
157
cp(((x), 0(x'), ), z,) == MLP, p ([(((x) - o(x'))² zz)))
(10)
where (Ø(x) 2(x'))² the element-wiss L2 distance between the input representations, [:-]] is the concatenation operator and MLP is a fully connected neural network of parameters P which has single neuron at its last layer. It bears mentioning that Cp is symmetric and stationary with regard to (xx and ((x') their clement-wise L2 distances vector is received part of the input of the fully connected network. Further, by simply concatenating the task representation Z, to this distance vector at the input, Cp provides powerful approach produce task-specific kernels. However, these kernels are not positive semi- denindefinte (PSD) and cannot be directly used for KRR and GP. Therefore,
158
159
160
162
184
165
using the empirical kernel mapping technique (Schölkopf al., 1999) we computed the task-specific PSD kernel pp associated with a given task representation 2, obtained from Dimn- This kernel can be written the empirical kernel map of cp('s', ('';, 2t) with regard Dim i.e.:
186
167
kx, x') = C((x) (x(x), with
(11)
(x) = (cp(x, X1, 21,, ... Cp(x, Xm, 24)), and
(x.,) E D'mn Vi= = 1, ,
168
Using the empirical kernel map of Cp to compute kpp offers the opportunity to introduce pseudo-inpe representations (or
that could improve the kernel evaluations, specially in low data settings. More precisely, instead of computing the empirical kernel map with regard to
170
D Dim alone, we use (D UU) where U is the set of pseudo-representation.. The function Cr from
171
172
Eq. (11), becomes:
C,(x) = (c,(x, X1, 21), ... , Cx((, X 21), Cp(x, U1, 21), cp u 2:));
(12)
with i U = 1,,..,1U1 and(x;,)) D/ Vi = -,,... 1,...
173
The number of pseuuo-representttions is hyperparameter of ADKL (in our experiments we choose |U| [0, 50) and all pseudooreppresentaionn u, E H are learnable parameters that are shared by all tasks and learned during meta-training To prevent their collapse into a single point during the training and to ensure that they are well distributed in the feature space H. we add a regularization
174
175
176
177
term (D,,) to the training loss function. To introduce this regularization term, let's consider p and q to be the distributions that generate the true input representations and the pseuudoreprresentationn. respectively. We make the assumption that p and are both multivariate Gaussian distributions with diagonal covariance matrices and have respective parameters (ps. o%) and (Mu af)). The parameters of p are estimated using the running means and variances of all input representations computed over batches of tasks. Those of g are estimated using U. As, p and must be close, the training of the reeudo-rppresentttions is regularized by minimizing the KL distance Du between p and g, i.e.:
179
181
182
189
Du = KL o.) || [ o%))
(13)
Putting all together, the ADKL training objective is the following:
argmin E 0. t Comenn I
"Psseudo
(14)
184
with Ytask 2 as a tradeoff hypeeprrmtt for the regularization of the task-encoder and "pscudo 2 0
185
tradeoff hyperparameter for the regularization of the pseudo-inputs.
5

---

Under review as a conference paper at ICLR 2020
4 RELATED WORK
187
Across the spectrum of learning approaches, DKL methods lie between neural networks and kernel methods. While neural networks can learn from a very large amount of data without much prior knowledge, kernel methods learn from fewer data when given an appropriate covariance function that accounts for prior knowledge of the relevant task. In the first DKL. attempt, Wilson et al. (2016) combined GP with CNN to learn a covariance function adapted to a task from large amounts of data, though the large time and space complexity of kernel methods forced the approximation of the exact kernel using KISS- (Wilson and Nickisch, 2015). Dasgupta et al. (2018) have demonstrated that such approximation is not necessary using finite rank kernels. Here, we show that learning from collection of tasks (FSR mode) does not require any approximation when the covariance function is shared across tasks. This is an important distinction between our study and other existing studies in DKL, which learn their kernel for single task applications instead of multiple task collections.
188
189
190
191
192
195
195
196
197
On the spectrum between NNs and kernel methods we must also mention metric learning. Metric learning algorithms learn an input covariance function shared across tasks but rely only on the expressive power of DNNs. First, stochastic kernels are built out of shared feature extractors and simple pairwise metrics (e.g. cosine similarity (Vinyals et al., 2016), Euclidean distance (Snell et al., 2017)), or parametric functions (c.g. relation modules (Sung et al., 2018), graph neural networks
189
200
201
202
203
(Garcia and Bruna, 2017: Kim et al., 2019a)). Then, within tasks, the predictions are distance- weighted combinations of the training labels with the stochastic kernel evaluations no adaptation is done.
204
205
206
In connection with the test-time adaptation capabilities of our method, methods that combine metric learning with initialization-bast models are great competitors. In fact, Proto-MAML (Triantafillou et al., 2019), which captures the best of Prototypical Networks (Snell et al., 2017) and MAMI. (Finn et al., 2017), allows within-task adaptation using MAML top of a shared feature extractor. Similarly, Kim et al. (2018) have proposed a Bayesian version of MAML where a feature extractor is shared across tasks, while multiple MAML particles are used for the task-level adaptation. Bertinetto
208
210
212
et al. (2018) have also tackled the lack of adaptation for new tasks by using Ridge Regression and Logistic Regression to find the appropriate weighting of the training samples for classification tasks. This study can be considered as an instance of the FSDKL framework, though its contribution was limited to showing that simple differentiable learning algorithms can increase adaptation in the metric learning framework. Our work goes beyond by formalizing few-shot DKL and proposing ADKL: a nata-driven manner compute the correct kernel for a task.
213
214
215
218
217
218 Since ADKL-GP learns task-specific stochastic processes, it is related to neural processes (Garnelo et al., 2018a) and the ML-PIP framework (Gordon et al., 2018). Both propose a scalable alternative to learning regression functions by performing inference on stochastic processes. In these families of methods, both Conditional Neural Processes (CNP) (Garnelo et al., 2018b) and Attentive Neural Processes (ANP) (Kim et al., 2019b) learn conditional stochastic processes parameterized by task- specific conditions derived from the support sets, but CNP is the most related to ADKL-GP. CNP is an instance of ML-PIP when the task encoder gives a point estimate of the task parameters instead of a distribution. Finally, the main differences between ANP and CNP are the architecture of the task-encoder and the lack of mathematical guarantees associated with stochastic processes in CNP (as it does not impose any consistency with respect to prior process). By comparison, ADKL-GP
219
220
221
223
225
227
228
also learns conditional stochastic processes but has mathematical guarantees thanks to GP and PSD kernels.
229
DATASETS
230
Existing FSR methods have been mostly tested on 1D function regression and pixel-wise image completion tasks with MNIST and CelebA (Kim et al., 2018; Garnelo et al., 2018b;a). On one hand, the ID regression tasks are all relatively simple, almost noise-less, and homogeneous. On the other hand, methods have been successful for image completion tasks only outside the few-shot regime (i.e. when the number of samples is greater than 500) (Garneoo et al., 201 (0;a). For these reasons, introduce two task collections from a real-world context. Deemedd Binding and Antibacterial, these
232
234
236
237
task collections contain data from bio-assays that representative of real-world FSR tasks in drug

---

Under review as a conference paper at ICLR 2020
discovery. The pre-processeeeee processed versions of these collections and detailed statistics available here
(anonymized link).
239
Binding: All tasks in this collection aim to predict the binding affinity of small molecules to target protein. The characteristics of the proteins thus define different data distributions over the chemical space. The inputs and the targets for each task are molecules that have been tested in binding assay and the measured binding affinity of the molecule against a given protein. The task collection was extracted from the public database Bi and altered by removing bio-assays with targets correlated above 0.8 or those with less than 10 experimental measurements, leaving us with 5, 717 tasks.
241
243
244
245
248
247
Within this collection, the task is to predict the antimicrobial activity of small molecules against various bacteria. They are characterized by a bacterial strain whose resistance to drug-like molecules was being evaluated. The task collection was extracted from the public database PubChem. After also removing bio-assays with correlations above 0.8 and those with less than 10 samples, we obtain 3, 255 tasks.
Antibacterial:
248
249
250
252
Their meta-test partitions each contain 1000 tasks, with the remaining used in the meta-train and validatiin..... The molecules (represented as SMILES) are converted into vectors using routines available the RDKit software (more precisely into ECFP6 binary fingerprint vectors of 4,096 dimensions). These inputs were also processed in all methods using the same feature extractor architecture, which is a fully-connected network of 256 x 256 x 256. Due to the high noise-to-signal ratio, the targets are first scaled and then scaled linearly between 0 and to avoid scaling issues during training.
254
256
258
Fig. 2 highlights three aspects of the collections that make them complementary to existing bench- marks, but better suited for evaluating the readiness of FSR methods for real-world applications relative toy collections. First, the distributions of number of samples per task show that they naturally contain few samples, which we believe reflects the costs of acquiring labelled data in drug discovery setting, In comparison, the number of samples available per task is relatively large in previous benchmarks, with the few-shot regime being achieved artificially through sampling. Second, as illustrated by their noise-to-signal ratio, real-world tasks are inherently noisy, increasing the difficulties associated with few-shot learning. Finally, the input diversity within each task is reduced
280
282
263
264
265
266
267
relative to the total among tasks. Despite this diversity difference, good models should perform relatively well outside the input region they have seen in the support set. This situation challenges the methods to learn strong priors about the input space and to be able to generalize after seeing only a small fraction of it. These collections invite researchers to explore meta-learning with increasingly heterogeneous datasets and in noisy environments, as well as generalisation and extrapolation in large input spaces (such the drug-like chemical space, which is estimated to be approximately 1033 molecules (Polishchuk et al., 2013)).
289
271
272
273
To test our method in noise-less environment, also the Sinusoids collection introduced by Kim et al. (2018). This challenging few-shot regression benchmark consists of 5,000 tasks defined
275
276 by functions of the form: y A sin(wr + b) e with A 1.0), b [0.0, 2n and
277 We E [0.5, 2.0). Sampling inputs æ E [-5.0, 5.0] and observational noise E (0, (0.01A)²) and
278
computing y gives the samples for each task. Here, the meta-train, veta-validation, and meta-test
279
contain 56.25% 18.75% and 25% of all the tasks, respectively, and all methods use the same feature
280
extractor architecture, which is a fully-connected network of 40 40 40.
6
EXPERIMENTS
282
6.1
BENCHMARKING ANALYSIS
283
For all benchmarks, the performances of ADKL is compared against other meta-learning algorithms: R2-D2 (an instance of FSDKL Bertinetto et al. (2018)), CNP (Garnelo et al., 2018b), MAML (Finn et al., 2017). BMAML (Kim et al., 2018), and Learned Basis (Yi Loo, 2019) (all implementations are available here (anonymized link)). These algorithms have all proven to have efficient and effective test-time adaptation routines and therefore constitute strong baselines for benchmarking. However, for bioassay modelling benchmarks, we have also added two methods considered to be state-of-the- in chemoinformaties to assess performance relative to all meta-learning approaches. These methods
284
286
208
289
7

---

Under review as a conference paper at ICLR 2020
10"
Nambbr of samples per task
girwic melecuie sim laitty
2.8
10a
Noise to signal rato
2.6
0.4
10a
10
Bindnggl
32)
(Mean
Bindingeib
Antibactertel
Figure 2: Statistics on bio-assay modelling tasks. Left: Number of samples per task. Middle: Noisc-to-signal ratio. Right: Within-task
versus overall molecular diversity.
20
5
10
model
ADKL-GP
ADKL-KKR
0.0448
BMAML
0,0813
0.0571
0.046 1: 0.0071
0.0487
0.0012
BMAML
0.108
0.0021
0.1020 0.0029
0.4616
0,4210
CNP
0.9416-
0.0019
00003
+ 0.0030
D.0397
0.0027
CNP
0.10K3
0.0023
0.1239 = 0.021
0.1382
0.0049
BCFM--RR
0.0376
0.0012
0.052 1 0.0014
0.0317
0.0016
ECFMM-KRR
0.1166
0.0020
0.1003 1: 0.009
0.0956 : 0,0009
ECEPS+RE
0.0373
0.0012
0.8339
1: 0.6913
0.0311
0.0012
BCER4-RE
0.1129
0.0002
0.1016
0.0008
0.0970
0.0003
LeumediBasis
0.0761
0.0040
00751
0.0012
0.0615
0.0215
LeumedBaaiis
0.1274
0.0037
0.1308
0.0032
0.1329
0.0043
0.0015
0.460
0.0100
0.0012
RID2
0,0023
0.0962
0,0921
0.0010
Table 1: Average MSE on Binding
Table 2: Average MSE on Antibacterial
the Random Forest algorithm with ECFP4 (Extended Connectivity FingerPrints of diameter 4) as molecular input representation, and ECFP4 with KRR and tanimoto similarity as a kernel function OOlier et al., 2018). During meta-test, each task is partitioned into query and support sets, then the support set is used to generate a model which is evaluated on the query set to compute the MSE, This process is repeated 30 times per task and the average MSE over the repetitions per task and over all tasks is reported in Tables to 3.
281
292
294
296
For the Sinusoids collection, Table 3 shows that DKL-based methods significantly outperform all other methods despite their test-time adaptation capabilities. These results alone demonstrate the effectiveness of DKL-based methods in FSR relative to the current sate-of-the--e Furthermore, of all DKL-based methods, ADKL-KRR shows consistently stronger performance than others. This demonstrates that using ADKL increases test-time performance relative to FS-DKL (as R2-D2 and
298
300
ADKL-KRR only differ by the kernel definition). It also indicates that attempting capture the model uncertainty using GP in ADKL (instead of KRR) comes with a significant cost, especially in lower data regimes. This may be due the inability of GP to differentiate between the observational noise and the model uncertainty as the number of samples get smaller. Also, notice that all task encoding based methods significantly outperform the others. This shows that adequately capturing
302
304
305
308
the task representation is crucial for this task collection, and ADKL-KRR appears to be best equipped do so.
307
308
Tables and 2 show the performances of all methods on real-world datasets. As complements, Tables and show the p-value that assesses the statistical significance the difference between each model and ADKL-GP and ADKL-KRR, These p-values result from Wilcoxon ranked tests comparing the MSE per task of each algorithm
309
310
311
ADKLs. Combined together, these tables shows that ADKL methods significantly outperforms all other meta-learning methods (p-values < 0.05). They also outperformed the state-of the art in chemoinformatic for Antibacterial, but do not on Binding where those methods are significantly better than all meta-learning algorithms. Even if, ADKL is first step in the right direction, these results show that there remains much room to develop meta- leaanlea algorithms which are undoubtedly superior to methods in computational chemistry. It also worth noticing that ADKL methods are significantly better than R2-D2 for these collections also confirming that using task specific kernels useful and improve generalization.
313
315
317
8

---

Under review as a conference paper at ICLR 2020
m
5
10
20
ADKI. GP
ADKL KRR
ADKL KRR
model
ADKL-GP
1.21c-01
ADKL-GP
BMAMI.
2,042
1.371
0.844
CNP
0.00e+00
0.00c+00
CNP
1.776-114
CNP
1.616
0.392
0.117
ADKL.
1.21e.01
ADKI- KRK
Learned Basis
3.587
0.800
0.127
ECFP4
2.48e-78
3.69e-62
ECFP4*KRR
2.70c-03
MAML.
2.896
1.634
0.901
LeumedBasis
0.00=+00
0.00=+00
LeamedBassis
0.00=+00
ADKL-GP
1.178
0.084
0.007
BMAML
0.00++000
0.00c+00
BMAMI
0.00c+00
ADKI KRR
0.867
0.061
0.005
FSDKL (R2D2)
1.18
5.50c-15 15
FSDKL (R2D2)
1.76e-49
FSDKL (R2D22)
1.002
0.073
0.009
ECEPHRF
9.70e-81
2.20e-168
ECHM+RR
5.94g-01
Table 3: Average MSE on Sinusoidals
Table 4: Wilcoxon
Table 5: Wilcoxon
p-values BBndingDB
p-values BBnddnnDB
319
6.2
ACTIVE LEARNING
320
In this section, we report the results of active learning experiments. Our intent is to measure the effectiveness of the uncertainty captured by the predictive distribution of ADKL-GP for active learning, as critical to our drug discovery use-cases. CNP, comparison, serves to measure which of CNP and GP better captures the data uncertainty for improving FSR under active sample selection. For this purpose, we meta-train both algorithms using support and query sets of size 5. During meta-test time, five samples are randomly selected to constitute the support set Dorn and build the initial hypothesis for each task. Then, from a pool U of unlabeled data, we choose the input x" of maximum predictive entropy, i.e. x' argmaxxeu [log p(v)x, (//,, D-rn]]- The latter is removed from U' and added to Den with its predicted label. The within-task adaptation is performed
322
324
326
328
329
on the new support set to obtain a new hypothesis which is evaluated the query set Deat of the task. This process is repeated until we reach the allotted budget of 20 queries.
330
331
Fig. 3 illustrates, for all collections. the MSE after each sample acquisition iteration and under both random and active learning acquisition strategies. Under the active learning strategy, ADKL-GP consistently outperforms CNP. In particular, we observe that very few samples are queried by ADKL- GP to capture the data distribution whereas CNP performance remains far from optimal even when allowed the maximum number of queries. Further, since using the maximum predictive entropy strategy is better than querying samples at random for ADKL-GP (solid vs. dashed line), these results suggest that the predictive uncertainty obtained with GP is informative and more accurate than that of CNP. Moreover, when the number of queries is greater than 10, we observe performance degradation for CNP while ADKL-GP remains consistent. This observation highlights the generalization capacity of DKL methods, even outside the few-shot regime where they have been trained this same property does not hold true for CNP. We attribute this property of DKL methods to their use of kernel methods. In fact, their role in adaptation and generalization increases as move away from the few-shot training regime.
332
333
384
335
337
339
340
341
343
ADX wydettee
25-
CNP
0.00
SERCCC
0.
C5P
Figure Average MSE performance on the meta-test during active learning. The width of the shaded regions denotes the uncertainty over tive runs for the s sinusoida da collection. No uncertainty is shown for the real-world tasks as they were too time consuming.
344
6.3 ABLATION EXPERIMENTS
345 In our final set of experiments, we more closely evaluate the impact of the task encoder and the pseudo- 346 inputs on the generalization during meta-testing. We do so by training and evaluating ADKL. on
9

---

Under review as a conference paper at ICLR 2020
Sinusoids with different hyperparameter combinations. Figs. 4a 4d show the relative improvements (negative values) or setbacks (positive values) in the meta-test MSE compared to different baselines (but the joint impact of Yash and Ypseudo is only discussed in Appendix A.3).
348
350 First, Fig. 4a compares Mask E {0.01, 0.1} relative to Yaas = and consequently demonstrates that regularizing the task encoder by maximizing the mutual information between the support set and the query set significantly improves the generalization performance. This conclusion holds for all support sizes lested, shown in Appendix A. Combined with the results from Section 6. this figure shows the importance of good task encoders for generalization in few-shot learning and how using the regularization term that we introduced is a step forward in that direction.
352
353
354
355
358
Then, Fig. 4c measures the relative differences between Tpseudo {0.01,0.1 0.1} and "peeudo = 0 for different values of hyprparaameter combinations. It shows that improving the kernel map evaluations using pseudo-inph representations can significantly help with the generalization performance of ADKL. This conclusion also holds for all values tested for Drm see Appendix A.2). However, the improvements were more consistent for smaller support sets, which is not surprising improving the kernel map estimations in these cases is more critical.
357
359
361
Finally, Figs. 4b and 4d illustrate for ADKL-GP and ADKL-KRR, and different sizes of support sets, how the number of preudo-represennations (i.e UT) affects performance. The values for each cell relative performance using |U|E {20, 50} versus |U|= and have been averaged over different hy perparaametess and "peeado- In general, we can confirm that increasing the number of pscudo- representations increases the estimates of the kernel maps and improves generalization. However, the improvements are more prominent with KRR in comparison to GP, which may be due to the fact that GP attributes a part of the modelling noise the kernel evaluations, leading to more constraints on the optimization of the pseudooreepresentation parameters.
363
365
367
388
389
7eask 0.01
-27.3
-32.1
-27 3
-16.0
-16.0
-7.4
-46.9
-22.3
-44.1
-9.7
8
-30.0
-74.0
-50.1
5
0.1
-9.4
-33.9
-46.2
-13.7
-6.0
-43.3
-4.55
-52.5
-50.5
-7.7
-30.5
-75.0
-60.5
a
b
e
g
5
10
20
Configurations
[Drral
(a) Impact of the task encoder trade-off parameter.
(b) |U in ADKL-KRR
pyseudo
0.1 0.01
-6.1
-3.0
-12.8
-28.4
-28.8
-10.2
-39.2
-30.9
-20.2
-0.2
+-40.0
-31.2
-20.0
-8.0
+0.8
+1.0
-13.7
-7.7
-12.7
-39.1
27.7
21.77
-4.11
+19.6
-49.5
-2.22
a
b
C
g
h
j
5
10
20
Configurations
[De.ll
(c) Impact of the pseudo inputs trade-off parameter
(d) |0| in ADKL-GP
Figure 4: Relative decrcase/increase in the meta-test MSE compared different baselines. In (a) and (c) the baselines are Ytask = and "pseedo = 0, respectively. (b) and (d) the baselines are |U|=0
CONCLUSION
370
371
We investigate bio-assays modelling using FSR methods. Our proposed method, ADKL,, stores meta- keta--nnoweedge in kernel functions and adapts to new tasks using KRR GP. Our experiments provide evidence that the additional adaptation capacity at test-time provided by ADKL increases generalization significantly. Also, in Bayesian setup, ADKL provides better predictive uncertainty, increasing their utility in bioassay modelling. However, there is still room to improve ADKL and most meta-learning methods to be better than traditional chemoinformatic methods. We hope, by making our bio-assay task collections publicly available, that the community will leverage them to propose new competitive FSR methods.
372
373
375
377
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
380
Seonwoo Min, Byunghan Lee, and Sungroh Yoon. Deep learning in bioinformatics. Briefings in bioinformatics, 18(5):851-888 2017.
381
382
Youjun Xu, Kangjie Lin, Shiwei Wang, Lei Wang, Chenjing Cai, Chen Song, Luhua Lai, and Jianfeng Pei. Deep learning for molecular generation. Future medicinal chemistry, 11(6):567-597, 2019.
384
Marwin HS Segler and Mark Waller. Neural-symbolic machine learning for retrosynthesis and reaction prediction. Chemistry-A European Journal, 23(25):996-5571,, 2017.
386
Marwin HS Segler, Mike Preuss, and Mark P Waller. Learning to plan chemical syntheses. arXiv
387
preprint arXiv: 1rxivv/708.000 718.708.4440 04202, 2017.
Artem Cherkasov, Eugene N Muratov, Denis Fourches, Alexandre Varnek, Igor I Baskin, Mark Cronin, John Dearden, Paola Gramatica, Yvonne C Martin, Roberto Todeschini, et al. Qsar modeling: where have you been? where are you going to? Journal of medicinal chemistry, 57(12): 4977-5010, 2014.
389
391
382
Yaqing Wang and Quanming Yao. Few-shot learning: A survey. CoRR, abs1194.00046 2019. URL http:://axxiii org/ abs 9044 05046.
395
394
Wei-Y Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint arXiv: 04232, 2019.
395
396
Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn, pages 3-17. Springer, 1998.
398
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial intelligence review, 18(2):77-95, 2002.
389
Regine Bohacck, Colin McMartin, and Wayne C Guida. The art and practice of dtructurebbase drug design: a molecular modeling perspective. Medicinal research reviews, 16(1):3-50, 1996.
401
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICMI. deep learning workshop, volume 2, 2015.
403
404
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pages 3630-3638, 2016.
406
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Informaion Processing Systems, pages 4077-4087, 2017.
Victor Gareia and Joan Bruna. Few-shot learning with graph neural networks. arXiv preprint arXiv:1711.04043 2017.
409
Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. arXiv preprint arXiv: Sadv:/880 1805. 08130 2018.
411
412
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Vaie Volume 70, pages 1126-1135. JMLR. org, 2017.
413
415
Taesup Kim, Jaesik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning, arXiv preprint
416
2018.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. International Conference on Learning Representations, 2016.
418
419
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial Intelligence and Statistics, pages 370-378, 2016.
420
421
Bernhard Scholkopf and Alexander Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2001.
11

---

Under review as a conference paper at ICLR 2020
Ingo Steinwart and Andreas Christmann. Support vector machines. Springer Science & Business Media, 2008.
424
Christopher KI Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. In Advances in neural information processing systems, pages 682-688,70 2001.
426
Andrew Wilson and Hannes Nickisch. Kernel interpolation for scalable structured gaussian processes (kiss-gp). In International Conference on Machine Learning, pages 1775-1784, 2015.
428
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
430
432
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in neural information processing systems, pages 3391-3401, 2017.
433
434
435
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair. Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint (eXIv:/801.0400 2018.
437
Bernhard Schölkopf, Sehastian Mika, Chris JC Burges, Philipp Knirsch, Klaus-Rohert Müller, Gunnar Rätsch, and Alexander Smola. Input space versus feature space in kernel-based methods. IEEE transactions on neural networks, 10(5): 100001077, 1999.
439
441
Sambarta Dasgupta, Kumar Sricharan, and Ashok Srivastava. Finite rank deep kernel learning. 2018.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 1199-1208, 2018.
443
445
Jongmin Kim, Taesup Kim, Sungwoong Kim, and Chang D Yoo. Edge-labeling graph neural network for few-shot learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 11 11-20, 2019a.
447
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning learn from few examples. arXiv preprint arXiv: 03000, 2019.
449
450
451
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint 1rXiv:1807.01122. 2018a
452
453
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Meta- learning probabilistic inference for prediction. arXiv preprint 1rX1v:1805.0092. 2018.
454
455
Marta Garnelo, Dan Rosenbaum, Chris J Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Eslami. Conditional neural processes. arXiv preprint arXiv: 1007:01007 2018b.
456
458 Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprini 19Xiv:9901.07700 2019b.
460
Pavel Polishchuk, Timur Madzhidov, and Alexandre Varnek. Estimation the size of drug-like chemical space based on gdb- 17 data. Journal of computer-uided molecular design, 27(8):675-679 2013.
462
Gemma Roig Ngai-Man Cheung Yi Loo, Swee Kiat Lim. Few-shot regression via learned basis functions. open review Iidprint. 2019.
464
Ivan Olier, Noureddin Sadawi, G Richard Bickerton, Joaquin Vanschoren, Crina Grosan, Larisa Soldatova, and Ross D King. Meta-qsar: a large-scale application meta-learning to drug design and discovery. Machine Learning, 107(1):285-311 2018.
486
12

---

Under review as a conference paper at ICLR 2020
Appendices
488
A REGULARIZATION IMPACT
470
A.1 TASK REGULARIZATION
Table 6 presents the hyperparameter combinations used in the experiments to assess the impact of the trade-off parameter "task- We report the MSE performance obtained on the meta-test for each combination. To make reading this table easier, we also repeat the Fig. 5 showing the improvement 474 of the MSE relative to Ttosk (no regularization).
472
Table Effect of using task regularization (parameter Yaskk) on the MSE performance
Hask
0.00
0.01
0.10
algorithm
K
Ipseeuoo
Configuration
ADKL-KRR 20
0.01
0.0585
0.0327
0.0289
10
0.00
0.4051
0.2944
0.3671
0.10
0.4363
0.2964
0.2882
ADKL-GP
0.10
2.4920
2.2511
2.2994
ADKL-KRR
20
0.00
0,0574
0.0305
0.0302
ADKL-GP
5
0.01
f
2.5611
2.1511
2.2112
0.01
g
3.2933
2.7663
3.0971
10
0.01
h
0.7675
0.7105
0.4352
0.00
0.1201
0.0873
0.0646
ADKL-KRR
20
0.10
j
0.0575
0.0447
0.0273
Yrask 0011
.27.3
-32.1
-27.3
-16.0
-16.0
-7.4
-6.99
-22.3
-44.1
-9.7
0.1
-9.4
-33.9
.66.2
-13.7
-6.0
-43.3
-47.5
-52.5
-50
5
-7.7
g
j
Configurations
Figure Relative improvement of the MSE depending on the Yaas parameter
For more in-depth analysis, show below the similar tables and figures for different values of K (
476 5. 10 and 20). These results confirm that regularizing the task encoder is helpful for any value of K, even though the impact seems to become much more important K increases (observe that the
478 maximum improvement each figure increases with K).
For K
Ttaxk
0.00
0.01
0.10
al algorithm
"pocudo
ADKL-GP
3.2933
2.7663
3.0971
0.00
2,8528
3.1136
2.2801
Thout
0.1 0.01
16.0
2.5611
2.1511
2.2112
480
-6.0
-20.1
13.7
-7.7
22
-26
-27.8
8
+0.4
-20
-1.9
ADKL-KRR
0.10
2.4920
2.2511
1.7079
1,2088
2.2994
0.01
1.6344
1.6655
1.1974
7
9
0.10
1.6868
1.6532
1.2173
Contigurationss
0.00
1.1951
1.2129
1.1998
0.01
1.1655
1,1611
1.14
0.10
1.1658
1.1716
1.1442
For K 10
13

---

Under review as a conference paper at ICLR 2020
Toos Tpscudo
0.00
0.01
0.10
algorithm
K
10
ADKL-GP
0.00
0.6423
0.6556
0.6079
0.01
0.7675
0.7105
0.4352
0.3 0,00
12.1
-7/4
16.4
-14.1
-278
-19.2
-32.1
-39
-1.8
0.10
0.6182
0.6577
0.5244
482
-5.3
-43.
-1..2
-9.4
17.3
-33.
-211
ADKL-KRR
0.10
0.00
0.7326
0.4051
0.6294
0.2944
0.7663
0.3671
0.01
0.4386
0.3544
0.3628
9
0.10
0.4363
0.2964
0.2882
Configurations
0.00
0.3170
0,2967
0.2395
0.01
0.3070
0.2888
0.2299
0.10
0,3038
0,2893
0.2326
483
For = 20
Thooh
0.00
0.01
0.10
algorithm
ipecudo
K
20
ADKL-GP
0.00
0.1201
0.0873
0.0646
0.0958
0.0761
0.0952
Thoe
0.01
-27.3
-20.6
-6.1
-3.8
+1.6
-133 -14.4
-669
-44.1
-22.
0.10
0.0940
0.0882
0.1286
484
0.1
.49.2
.0.7
47.0
18.3
10.4
44.0
47.5
-005
-22.8
ADKL-KRR
0.00
0.01
0.1069
0.0526
0,1029
0.0535
0.1144
0.0430
0.01
0.0375
0,0325
0.0414
0.0380
0.0325
0.0395
Configgrations
0.00
0.0574
0,0305
0.0302
0.01
0.0585
0.0327
0.0289
0.10
0.0575
0,0447
0.0273
485
A.2
PIEUDO-INPUT REPRESENTATIONS
Table 7 presents the hyperparameter combinations used in the experiments to assess the impact of the trade-off parameter "ppeeuoo, which governs the penalty applied to the divergence between the distribution of learned represettpssssnssiioonss and the distribution of actual representations. We also repeat Fig. 6, the relative improvement of MSE compared to Ypseudo = 0 as shown in the main text.
487
489
Table 7: Effect of the pseudo-examples regularization (parameter "pseedo) on the MSE performance
Tpscudo Conf.
0.00
0.01
0.10
algorithm
K
Btakk
ADKL-GP
10
0.10
a 0.6079
0.4352
0.5244
20
0.01
b 0.0873
0.0761
0.0882
ADKL-KRR
20
0.00
0,0526
0.0375
0.0380
ADKL-GP
5
0.10
d 2.2801
2.2112
2.2994
ADKL-KRR
20
0.01
0.0535
0.0325
0.0325
ADKL-GP
5
0.01
f
2.9466
2.7663
2.7121
20
0.10
g 0.1147
0.1144
0.0870
0.00
h
0.1201
0.0958
0.0940
5
0.01
3.1136
2.1511
2.2511
0.00
2.8528
2.5611
2.4920
Ypsenalo
0.01
6..1
12.8
28.4
28.8
-10.2
-9.22
30.9
20.2
-0.2
0.1
-3.77
-12.7
-21.7
d
e
h
Configurations
Figure 6: Relative improvement of the MSE depending on the Ttask parameter
Once again, for more in-depth analysis, show below the same format of tables and figures for
492 different values of K, confirming again that regularizing using the
can be very helpful for any value of K. It is worth noticing here that the improvement gain is more consistent for
14

---

Under review as a conference paper at ICLR 2020
K compared to K {10, 20), supporting the fact that improving kernel maps evaluations using pseudo-rew renueettpptssnnaaioons is critical as size of the support set decreases.
496
For K 5
"pocudo
0.00
0.01
0.10
algurithm
Tiosh
ADKL-GP
2.9466
2,7663
2.7121
0.10
2.2801
2,2112
2.2994
Tpocuao
0.3 00001
-30
-46
-10.2
-43
-918
-2.6
ADKL-KRR
1.7123
1,6344
1,6688
487
-3.0
+0.8
-15
-2.
-1277
-16
1
-5.0
-27.7
-3.2
ADKL-GP
0.00
0.00
2,8528
1.1951
2.5611
1.1655
2.4920
1.1658
5
7
9
ADKI-KRR
0.10
0.01
1.2129
1.1998
1.1611
1.1416
1.1716
1.1442
Configuratione
0.10
1,2808
1.1974
1.2173
ADKL-GP
0.01
3.1136
2.1511
2.2511
ADKL-KRR
0.01
1.7079
1.6655
1.6532
For K 10
ipscudo
0.00
0.01
0.10
algorithm
Tbooh
ADKL-GP
0.01
0.7329
0.7907
0.6294
0.10
0.7479
0.7800
0.7663
Spouss
0.3 0.01
17.8
-3.2
19.5
-1.2
-2.7
-5.4
-28
-1.1
ADKL-KRR
0.00
0.3170
0.3070
0.3038
499
-1411
+2.5
-4.
-3.8
-11
-2.5
-0.3
+2.8
-13.7
-2.9
ADKL-GP
ADKL-KRR
0.10
0.00
0.6423
0.3671
0.7675
0.3628
0.6182
0.2882
7
9
ADKL-GP
0.01
0.01
0.2967
0.6556
0,7105
0,2888
0.6577
0.2893
Configurations
0.00
0.7145
0.6758
0.7326
0.10
0.6079
0,4352
0.5244
ADKL-KRR
0.10
0.2395
0.2299
0.2326
For K 20
Tpecudo
0.00
0.01
0.10
algorithm
Ttush:
ADKL-GP
0.1201
0.0958
0.0940
0.00
0.0794
0.1069
0.0702
Tperase
0.1 0.001
+7.1
+26
-31.2
0.01
0.0873
0.0761
0.0882
ADKL-KRR
0.01
0.0305
0.0327
0.0447
-21.7
-11.6
+10
+16.5
-
-9.4
+0.1
-21.1
-30.1
-8.1
0.00
0.0526
0.0375
0.0380
5
7
9
0.00
0.10
0.0574
0.0302
0.0585
0,0289
0.0575
0.0273
Contiporationss
ADKL-GP ADKL-KRR
0.10
0.1147
0.1144
0.0870
0.01
0.0535
0,0325
0.0325
0.10
0.0430
0.0414
0.0395
Overall, the effect of the regularization is beneficial, even though witness few pathological cases.
A.3
JOINT IMPACT OF "task AND Ypseudo
504
Since both Nask and "ppecudo have a high impact on the training and the generalization performance,
505
we need assess the relationship between the two. Fig. shows, for different values of K, the relative improvement of the test MSE compared to the case where no regularization is done, i.c.
508
507
and "pseudo 0. Overall, one can see that higher is better in both dimensions but there seems to be sweet spot on the grid for each value of K and therefore we can only advise the user to
Iaask
508
509
cross-validate on those hyperparameters.
K
K
10
K 20
0.0
++.00
-3.7
-1.
0.0
++0.0
+3.3
+2.5
0.0
++0.0
-1288
-13.2
Nhas
0.01
++0.5
-288
-28
0.01
.18.1
-1.99
18.9
0.01
-23.7
-0.8
29.8
0.1
-14.7
19.5
.1888
0.1
-16.0
-17.9
-27.9
0.1
.33.5
-36.1
.3.3
0.0
0.01
0.1
0.0
0.01
0.1
0.0
0.01
0.1
poeeddo
Tpacado
Figure 7: Average relative improvement of the MSE and joint impact of Taak and Ypsoudo-
15

---

Under review as a conference paper at ICLR 2020
B PREDICTION CURVES ON THE SINUSOIDS COLLECTION
511
Figure presents a visualization of the results obtained by each model on three tasks taken randomly from the meta-test We provide the model with ten examples from an unseen task consisting of a slightly noisy sine function (shown in blue), and present in orange the predictions made by the network based on these ten examples.
512
518
514
S0HAMEL 305:385
Predete
Orann mock
Probeae
Denoer Ir.h
Oomed
-10
-20
MAVIL MAR OUB
Doratt
ATCLAGE SEE 100
-19
1/50.
13:00 MBS
Figure 8: Meta-test time predictions on the Sinusoi ds collection
16

---

Under review as a conference paper at ICLR 2020
C SUPPLEMENTARY RESULTS FOR THE REAL-WORLD DATASETS
antibacterial,
baadingd: [K]
0.30
0.100
Pesyph M356
MSB
0.20
0.15
0.075
0.30
1000
0.055
ANKLOOP
ADRL RiiL-
IRMNN
Lomettlaah
AURL KRE
IRAMM
csp
BaTM
Learenllla
antibactarial, [
bindingth [K] - 10
0.100
0.30
0.105
0.00
Dicopla MSE
0,300
0.15-
0.075
0.00
0.005
0.00
1000
ADKL-CP
ADKL.RBB
IMAME
ONP
BOW
Lasmsdfaaas
ADKL-KIE
BMAMIL
Cap
BOYPA- Kill
LrumedfFail
ansibactrria,,
bindingds -
0.50
0:100
Pesopht MS3
0.23
0.90
0.100
0.10
0,000
14025
0.00
ADKL-OP
ADKL-KBR
BMAM
BCPP4-KmE
Lomeiilack
ADKL-KER
BBAMH
CNP
ROPPi-KiR
Hor.LR RF
Lawnettflao
Figure Distribution of the mean squared error (MSE) across the tasks
516
Figure shows the distribution over the random support/query sets generated at test time. Note that the results presented in the main paper estimate the influence of the initialisation by using multiple seeds and computing the standard deviation on the average MSE (averaged over the support/query splits).
518
519
520
The two pieces of information are important: the results presented above give us better idea of the "meta-gent "meetaggneraaliatton" capabilities of each algorithm, while those in the main paper assess the reproducibility and the statistical significance of the relative improvements.
521
17


Reviews:
[{"review_id": "rJxWx_3O5S", "reviewer": "AnonReviewer7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors of this work are attempting to solve the problem of few shot regression for drug discovery problems. This is an important problem in the field of deep drug discovery, since low data issues are very common. While previous papers have addressed the challenges of low data drug classification, there hasn't been much progress made so far on low data regression thus far.\n\nThe authors propose formulating the problem at hand as a few shot deep kernel regression problem (FSDKL). This framework has some similarities to past work deep metric learning, which has been used previously for few shot classification problems in drug discovery. In particular, the authors propose a new method, adaptive deep kernel learning to help improve task specific learning. ADKL is claimed to improve over FSDKL since it helps learn a task specific kernel function. There are a number of interesting methodological adaptations here, such as the use of an idea similar to DeepSets to encode order invariance in the training set (an important challenge when dealing with small \"support\" sets in training). The authors also incorporate some unlabeled data during training to improve on representation learning.\n\nTo test their contributions, the authors gather two new dataset collections, Binding and Antibacterial from publicly available sources. Some detail is provided about these collections, but a priori, it's not easy for me to judge the quality of these data sources since they haven't been benchmarked previously in the literature.\n\nIn the experimental section, the authors compare ADKL against a number of past low data learning methods. The benchmarks in section 6.1 would benefit from confidence intervals. In my experience, few-shot algorithms can be quite noisy, so statistical tests are important to distinguish architectural improvements from noise. Repeated trials could be taken over the choice of random seed for the experiments to gauge robustness. Without error bars, and on a new dataset, it's not possible to gauge if ADKL really provides an improvement. \n\nThe experiments having to do with active learning in section 6.2 are interesting. Why is only CNP measured though? Do other methods not make sense with active learning? \n\nIn conclusion, the authors consider an important problem for deep learning in drug discovery and offer a useful advance by adapting the framework of deep kernel learning to this space. However, the work could still use a good bit of polish to really shine. It's not clear to me that the suggested ADKL framework really makes an improvement over past few shot regression methods. Adding some statistical significance tests would be useful. It would also help if the authors benchmarked against datasets that were better known in this field, such as those from the moleculenet.ai suite. It's not easy to judge how easy/hard the new datasets the authors propose are, which makes it challenging to gauge the real improvement. I'm also not sure that this paper is a clear fit for ICLR. I think there's a real contribution to the field of deep drug discovery by the adaptation of the DKL framework to few shot drug discovery, but I'm not convinced that ADKL is superior to other DKL methods as a pure learning technique. \n\nAll that said, I'm comfortable marking the paper as a \"weak accept\" since I think there is a real scientific contribution here, but I encourage the authors to make a serious effort to improve their presentation and tighten-up their experiments.\n\nDetailed Suggestions:\n- Section 2 on Deep Kernel Learning and Section 4 on Related work should likely be merged together. Going back and forth between the literature and author contributions was a little confusing. \n- There are a lot of acronyms in the paper; I found myself having to refer back and forth to all the methods considered. It might be worthwhile to make a cleanup pass to make the discussions a little easier to read. \n- Figure 4 would benefit from a legend. At first glance, it wasn't clear what gamma_mine and gamma_pseudo mean here. The color plot is also a little difficult to make sense. Perhaps consider an alternative plot.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Bkxkgp7qqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an interesting paper that proposes the use of few shot regression to predict complicated experimental measurements such as protein-ligand binding affinity from very small, noisy real-world datasets. It is great that the authors make the effort to apply their approach to these important questions in drug-discovery. However, the paper as currently written is difficult to follow, and in particular it is hard to distinguish between places where existing methods are combined from novel contributions made by this paper. It would be helpful if the authors could explicitly delineate the novel contributions that they make. Moreover, there is a large body of work in the drug discovery literature that uses sparse experimental data on the interactions of multiple target proteins and multiple ligands to build models that predict the outcome of biological assays for held out protein targets, where this problem is known as drug-target interaction prediction, but these papers are not referenced in this work (e.g. reviewed in Chen et al. Molecules 23(9):1-15, 2018, Ezzat et al. 2017, 2018, 2019). \n\nIn addition, it is hard to understand the results of the experiments that the authors carry out in this paper using data from BindingDB and PubChem. I don't understand the scaling that is applied to the MSE metric for the binding or antibacterial datasets. Why are the reported MSEs so low for all the methods? What does this metric mean? If the targets are first log2-scaled, then scaled linearly, then how different are they after this process? Given the tiny amount of data available, it seems surprising to me that the MSE is so low. From the brief description given, these problems appear significantly harder than the artificial Sinusoids task, yet the reported MSEs are orders of magnitude smaller. How is molecule similarity measured in Figure 2c? It would be useful for the authors to visualize performance for the Binding and Antibacterial tasks - for example how different are the different protein targets? What are the molecular ligand structures in the train and test set in each case - could the authors provide some examples? Using random splits into train and test likely means that some test data points are very close to train data points - can the authors stratify their analysis to provide some insight into the performance beyond the average MSE? \n\nOverall this paper makes a potentially interesting contribution, but it is not well situated within the drug discovery literature, and the results are not explained in enough detail to be understandable by experts in the drug discovery field. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJltL5L55H", "reviewer": "AnonReviewer6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission is at the intersection of few-shot learning, kernel regression methods, and computational biology.\nThe main contributions are:\n  - A few-shot learning algorithm combining several ideas and features:\n    - Combining metric learning (shared across tasks) and kernel regression within each task\n    - Learning a task representation by maximizing an estimate of the mutual information between the train (support) and valid (query) sets of a task\n    - The addition of learned, synthetic \"pseudo-examples\"\n  - Two datasets for few-shot regression, from real-life biological assays\nThe proposed algorithm outperforms (or is competitive with) mainstream few-shot learning methods on a synthetic 1D regression dataset, as well as the two proposed datasets.\n\nThis paper should be accepted, because it significantly expands the field of few-shot learning by proposing both a novel problem to tackle (few-shot regression from a high-dimensional, noisy input), with public datasets, and novel algorithm to solve it (combining several recent advances in different sub-fields of machine learning).\n\nOverview\nThe overall problem is clearly stated, as well as its main challenges: noisiness of the data, different behaviors of the same input across tasks.\nDespite the complexity of the proposed algorithm, its different pieces and their motivations are clearly motivated, introduced, and tested in ablation studies. I liked the clarity of section 2.\nThe \"related work\" section is clear and presents a good overall picture of the field. Additional papers that may be of interest:\n  - Learned hallucination (Low-shot learning from imaginary data, Wang et al., CVPR 2018) seems to relate to the \"pseudo-representations\"\n  - TADAM (Task dependent adaptive metric for improved few-shot learning, Oreshkin et al., NeurIPS 2018) is another example of task representation used in conjunction with metric learning (in the context of classification, not regression, though)\nThe proposed datasets are an interesting new benchmarking task, that naturally requires learning a task description, I hope it will get traction.\n\nQuestions:\n  - The backpropagation through the kernel regressor (in order to train the parameters of the embedding function and other networks) seems unexpectedly straightforward. Is that only because there is a closed-form solution for the optimal regressor? Were the specific algorithms (KRR, GP) chosen because of that? Or could something like MetaOptNet (Meta-Learning with differentiable convex optimization, Lee et al., CVPR 2019) be used to relax that constraint?\n  - What \"generalization guarantees of kernel-based models\" (l. 122) would be applicable here? Do they hold even when the kernel is applied on top of a learned embedding ($\\phi_\\theta$), or even when it depends on a trained model itself ($C_t$)?\n  - What does \"correlations > 0.8\" mean exactly? (l. 258, 263)\n  - In the \"Binding\" dataset, is it possible that the same protein is used in different bio-assays? In that case, are the different experiments \"merged\" into the same task, or be considered different tasks? Could it be possible that tasks involving the same protein would be in different (meta-)splits, or has that been taken care of during the data collection?\n  - In the meta-test splits, how are examples split between the train/support and valid/query parts of each task?\n  - How were the specific architectures of the different neural networks designed, or selected? Between $\\phi$, v, r, $MLP_\\rho$, the space of hyper-parameters seems huge, and the effects of these choices might be drastic.\n\nAdditional feedback\n  - The caption of Figure 1 could (re-)introduce a definition of the notation. U and C_t for instance have not been introduced yet.\n  - I'm not sure I agree that FSDKL only \"share[s] characteristics with the metric learning framework\", I see it more as being in that framework, but incorporating other elements as well (like other methods do, e.g., TADAM or RelationNet).\n  - The horizontal axis of Figure 4 (a) and (c) are not clear until we see Table 4 of the appendix, and suggest an ordering of the different configurations, rather than 10 different categories. I'm not sure how to improve it though, maybe letters instead of numbers?\n\nOn notation:\n  - On l. 155, is $\\phi$ the same as $\\phi_\\theta$, or a different embedding function for x?\n  - In Eq. 11, should $D^t_{trn}$ be $D^{t_j}_{trn}$ instead in both terms? Similarly, $D^{t_j}_{val}$ in the first term?\n  - In Eq. 12, and l. 173, $\\phi_x'$ suggests it is a different embedding $\\phi'$ of the same $x$, if we want to convey that there are two inputs x and x' instead, $\\phi_{x'}$ may be clearer.\n  - In Fig. 4 (a) and 5, the vertical axis is labeled $\\gamma_{mine}$ instead of (I assume) $\\gamma_{task}$.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJgdYgX9qr", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To tackle data scarcity issue in biological assay modeling, this paper proposes a method to episodically train Deep Kernel Learning models such that at meta-test time the models requires less examples. \n\nThe proposed Adaptive Deep Kernel Learning extends Deep Kernel Learning (end-to-end train Gaussian Process/Kernel Ridge Regression using Neural Network's final layer embedding) by:\n\n1. using a learnable task-specific kernel generator network \n2. applying episodic few-shot/meta learning to train the system end-to-end\n\nHere are the major issues for rejecting:\n\nModel:\n- Section 3.1 and 3.2 about \"Task Specific Kernel\" are very hard to follow.\n  -- (line 170) equation 12, What is the size of the MLP that is used as the kernel network \n  -- Is there an ablation to show the contribution of using learnable kernel network vs hand-picked kernel for ADKL-GP and ADKL-KRR? \n  -- (line 181) \"where U is a set of unlabeled inputs\" Where does the unlabeled inputs come from?\n  -- Is there an ablation about |U|  <= 50 and its effect?\n\nExperiments and Datasets:\n\n- The ProtoMAML code provided by the authors has only a few lines of code. And in the comment, it says: \"It turns out that ProtoMAML is the same as the light version of BMAML with one particle.\" Such implementation detail should be mentioned in the main paper.\n\n- Section 5 lacks details. \n -- How each meta-train episode is formed? Did they stochasticall sample a subset of molecules for each episode?\n -- What is the total number of tasks in the meta-train and meta-test split respectively? Are there any overlapping between the two splits?\n\n\nPresentation Issues:\n- Section 2 (line 76) \"We extended it (Deep Kernel Learning) to few-shot learning and discuss its advantages over the metric learning framework\" This sounds like the author developed a brand new framework, while the rest of paper is about proposing a specific realization of few-shot learning: Few-shot Deep Kernel Learning, which meta-learn through a differentialble kernel learning process (with the task kernel adaptation network novelty). \n\n- Section 3 (Figure 1) \"The blue and orange colors show the procedure...\" However there are more than two colors (including different shades of blue and orange colors) in the figure, which makes the figure hard to parse .  \n\n- Section 4 (line 201) \"DKL methods lie between neural networks and kernel methods\". To me, at high-level, DKL adds kernel learning as a differentiable layer in the end of a neural network. See how a similar model is categorized in Bertinetto et al. ICLR 2019 paper \"Meta-learning with differentiable closed-form solvers\":  \"The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression,as part of its own internal model, enabling it to quickly adapt to novel data.\" The delta of the paper is adding 1) Gaussian Process and 2) task-specific adaptation. If Bertinetto et al. categorize their method as part of a deep network then I think so should this work.\n\n- Section 4 (line 239) \"Our work goes beyond ... by proposing ADKL: a data-driven manner for computing the correct kernel for a task.\" This is probably a grammar mistake. \"a data driven manner for\" sounds odd. \n\n- Section 5 (line 272) \"Fig 2 highlights three aspects of the collections that make them better benchmarks for evaluating the readiness of FSR methods for real-world applications relative to toy collections.\" What is the other benchmarks the authors are comparing with? Are they generic benchmarks for all types few-shot regression tasks/applications beyond biological assays (e.g. computer vision tasks like: object detection). Maybe defining the proposed benchmark as \"complimentary\" and \"bio-assay application focused\" would be more appropriate. \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1l2o9Jp5B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new framework for solving few-shot regression problems. The proposed framework is based on deep kernel learning, which lies in the intersection of neural networks and kernel methods. The authors introduced adaptive deep kernel learning, which learns kernel for multiple task collection and computes the correct kernel for a task in a data-driven manner. The method is evaluated on three tasks collections — Sinusoids (synthetic dataset), Binding (real dataset of 5717 task where each task represents a binding affinity of small molecules against a given protein) and Antibacterial (real dataset of 3255 tasks where each task represents antimicrobial activity against given bacterium).\n\n\nPros:\n1. The proposed framework introduces a new adaptive method for few-shot drug discovery regression problems. \n2. The paper addressed the question of uncertainty estimation for drug discovery tasks.\n\n\nCons:\n1. The authors only evaluate the performance of the proposed method against other DKL-based methods. They do not consider a comparison with widely used in computational chemistry classical machine learning methods such as random forest, gradient boosting, SVM (which is also a kernel method), etc.\n2. The experimental results in Tables 1-3 show only marginal improvement for real datasets. Also, it’s not clear if the numbers in the Tables 1-3 are on the original scale or on the transformed scale. To estimate how well the models perform it’s useful to transform the targets back to the original scale. \n3. Overall the paper is written in a somehow confusing manner and some details in the description of the experiments important for understanding are omitted.\n\nQuestions:\n1. How well the proposed method performs against classical machine learning methods?\n2. How was MSE in Tables 1-3 calculated? \n3. It would be useful to see the histogram of MSEs instead of just a single number.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work applies deep kernel learning to the problem of few shot regression for modeling biological assays. To deal with sparse data on new tasks, the authors propose to adapt the learned kernel to each task. Reviews were mixed about the method and experiments, some reviewers were satisfied with the author rebuttal while others did not support acceptance during the discussion period. Some reviewers ultimately felt that the experimental results were too weak to warrant publication. On the binding task the method is comparable with simpler baselines, and some felt that the gains on antibacterial were unconvincing. 
Other reviewers felt that there remained simpler baselines to compare with, for example ablating the affects of learning the kernel with simple hand picking one. While authors commented they tried this, there were no details given on the results or what exactly they tried. 

Based on the reviewer discussion, the work feels too preliminary in its current form to warrant publication in ICLR. However, given that there are clearly some interesting ideas proposed in this work, I recommend resubmitting with stronger experimental evidence that the method helps over baselines.
Decision: Reject