Task-similarity Aware Meta-learning through Nonparametric Kernel Regression
Authors: Arun Venkitaraman
Anders Hansson
Bo Wahlberg
Venue:  (2021)
Keywords: Task-similarity
Meta-learning
Kernel regression
Nonparametric regression
Task-descriptors

Abstract: This paper investigates the use of nonparametric kernel-regression to obtain a task- similarity aware meta-learning algorithm. Our hypothesis is that the use of task- similarity helps meta-learning when the available tasks are limited and may contain outlier/ dissimilar tasks. While existing meta-learning approaches implicitly assume the tasks as being similar, it is generally unclear how this task-similarity could be quantified and used in the learning. As a result, most popular meta- learning approaches do not actively use the similarity/dissimilarity between the tasks, but rely on availability of huge number of tasks for their working. Our contribution is a novel framework for meta-learning that explicitly uses task-similarity in the form of kernels and an associated meta-learning algorithm. We model the task-specific parameters to belong to a reproducing kernel Hilbert space where the kernel function captures the similarity across tasks. The proposed algorithm iteratively learns a meta-parameter which is used to assign a task-specific descriptor for every task. The task descriptors are then used to quantify the task-similarity through the kernel function. We show how our approach conceptually generalizes the popular meta-learning approaches of model-agnostic meta-learning (MAML) and Meta-stochastic gradient descent (Meta-SGD) approaches. Numerical experiments with regression and classification tasks show that our algorithm outperforms these approaches when the number of tasks is limited, even in the presence of out- lier or dissimilar tasks. This supports our hypothesis that task-similarity helps improve the meta-learning performance in task-limited and adverse settings.

Under review as a conference paper at ICLR 2021
TASK-SIMILARITY AWARE THROUGH NONPARAMETRIC KERNEL REGRESSION
META-LEGRNING
Anonymous authors Paper under double-blind review
ABSTRACT
This paper investigates the use of nonparametric kernel-regression to obtain a task- similarity aware meta-learning algorithm. Our hypothesis is that the use of task- similarity helps meta-learning when the available tasks are limited and may con- tain outlier/ dissimilar tasks. While existing meta-learning approaches implicitly assume the tasks as being similar, it is generally unclear how this sisilaiillar could be quantified and used in the learning. As result, most popular meta- learning approaches do not actively use the similarityddisiiilarit between the tasks, but rely availability of huge number of tasks for their working. Our con- tribution is a novel framework for meta- meta-learning that explicitly uses fusk-similarity in the form of kernels and associated meta-learning algorithm. We model the task-specific parameters to belong to a reproducing kernel Hilbert space where the kernel function captures the similarity across tasks. The proposed algorithm iter- atively learns a meta-parameter which is used to assign a task-specific descriptor for every task. The task descriptors are then used to quantify the task-similarity through the kernel function. We show how our approach conceptually generalizes the popular meta-learning approaches of model-agnostic meta-learning (MAML) and Meta-stochastic gradient descent (Meta-SGD) approaches. Numerical experi- ments with regression and classification tasks show that our algorithm outperforms these approaches when the number of tasks is limited, even in the presence of out- lier or dissimilar tasks. This supports our hypothesis that aisi-iimilyrrty helps improve the meta-learning performance in task-limited and adverse settings.
INTRODUCTION
Meta-learning seeks to abstract general learning rule applicable class of different learning problems or tasks, given the knowledge of set of training tasks from the class (Finn & Levine, 2018 Denevi et al., 2018 Hospedales et al., 2020 Grant et al., 2018 Yoon et al.) 2018). The setting such that the data available for solving each task is often severely limited, resulting in poor performance when the tasks are solved independently from each other. This also sets meta-lcarning apart from the transfer learning paradigm where the focus is to transfer a well-trained network from existing domain to another (Pan & Yang 2010). While existing meta-learning approaches implicitly assume the tasks as being similar, it is generally unelear how this task-similarity could be quantified and used in the learning. As a result, most popular meta-learning approaches do not actively use the similarity/dissiiilrrty between the tasks, but rely on availability of huge number of tasks for their working. In many practical applications, the number of tasks could be limited and the tasks may not always be very similar. There might even be 'outlier tasks' or out-of--he-diie the-distribution tasks' that are less similar or dissimilar from the rest of the tasks. Our conjecture is that the explicit incorporation or awareness of task similariyy helps improve meta-learning performance such task-limited and adverse settings.
The goal of this paper is to test this hypothesis by developing a task-similarity aware meta-learning algorithm using nonparametric kernel regression. Specifically, our contribution is a novel meta- learning algorithm called the Task-similarity Aware Nonparumetric Meeainn (TANML) that:
Explicitly employs similarity across the tasks to fast adapt the meta- meta-information to a given task, by using nonparametric kernel regression.

---

Under review as a conference paper at ICLR 2021
Models the parameters of task belonging to reproducing kernel Hilbert space (RKHS), obtained by viewing the popular meta-learning of MAML and Meta-SGD ap- proaches through the lens of linear/kernel regression.
Uses task-descriptors through a kernel function to quantify -al-smiiaaiyyyddismiiarttt
Offers a general framework for incorporating task-similarity in the meta-learning process. Though we pursued the algorithm with a specific choice of the deserrecrss,,ss the pro- posed RKHS task similaitilartty aware framework can be extended to other formulations.
We wish to emphasize that goal is not to propose another meta-learning algorithm that outper- forms the state-of-the-art, but rather to investigate if task-similarity can be explicitly incorporated and used to advantage in meaningful manner. We show how this is achieved as the consequence of viewing the popular MAML and Meta-SGD formulations through the lens of nonparametric ker- nel regression. In order keep the comparison fair on an apple-to-apple level, we compare the performance of TANML with that of MAML and Meta-SGD algorithms.
1.1 MATHEMATICAL OVERVIEW OF THE PROPOSED
AWARE FRAMEWORK
Given pairs of data (Tk, Mir) E TT= x IR' where k E 21,2.... K} generated by an unknown data source, we are interested in learning a predictor IR= x IRD e (x, 0) + f(x, 0) E RT* from the given data. For example, fxr 0) could be function defined by artificial neural network (ANN). We collect pairs of data in X = (I1. T2. .KK) and y = (31, 92. ... 9x) and define the loss function R km. x R Kn, x R (X. + V( E E which we then minimize with respect to 0. This constitutes the training of the predictor. In the case of a ANN, ((X' ) E R could be the mean-square loss the cross-entropy function. The data Y used for training is referred to as the training data. Let f denote the optimal value of I obtained from training. Given a new x: E IR"*, we use 9 = (z. 0) to predict y. The goodness of 0 is evaluated using y - i on a sequence of pairs of new data called the test data X, ) defined similarly as X' and y, but with K number of data pairs. The training of the predictor for the given data source is defined as a task.
Now, consider that we are interested in carrying out several such tasks for data coming from different but similar sources. Let X, X. D., i = 1, 1,... Tr. denote the data from T5r different data- sources, and defined similarly as J2 X, above. We refer the training of the predictor for data X,, 3/, X, }/; as the ith training task, and 0, is referred to as the parameter for the task. Meta- learning captures similarities across the tasks by learning common @ (which we denote by 0b) from the data of these Ter tasks (called the meta-training data), such that 0n can be quickly adapted to train predictor for data from new and different but similar data-sources. Depending on how @ is obtained from Pu. various meta-learning algorithms exist (Denevi et al.) 2018: Finn & Levine 2018 Allen et al.) 2019p. The performance of the meta-learning algorithm is evaluated on previously unseen data from several other similar sources X yY, Dr, = 1,... To (called the meta-test data) defined similarly to X Y, X this constitutes the meta-test phase. The training of the predictor for test data ,/r X'", i. y is referred to as the ith test task, 0: denotes the parameter for the ith test task. In the existing meta-learning approaches, both Og and 0: are obtained by adapting 0g using the gradient of C(X; V., 0) and L(X,", Dr, 0), respectively, evaluated at On-
In our work, we propose a meta-learning algorithm where 0: explicitly uses similarity between the ith training task and all the training tasks. Similarly, the parameters 0y for the test tasks also use explicitly a similarity between the ith test task and all the training tasks. As specified later, we define this task-similarity between two tasks through kernel regression, and our algorithm learns the kernel regression coefficients & as meta-parameters in addition to 00-
A motivating example Let us now consider specific loss function given by L(X,)),) =
ΣK 11 - f(x, I1. Training for tasks independently with limited training data will typically result in predictor that overfits X.Y, and generalizes poorly to X. MAML-type meta- learning approaches (Finn et al.) (017) solve this by inferring the information across tasks in the form of good initialization On- specialized/aaaateed to new task using the adaptation function RD x IR Kn. x RK,,, (0o, X') + IMAMI. (0o, X, 3/) E RD defined as:
IMAML (0n, x,)') - 8 a Vo, ((.
2

---

Under review as a conference paper at ICLR 2021
The parameters for the training and test tasks obtained through adaptation of 0
0, = 9MAML (0o, X; M:), i= 1,... Ter, and 0:: = -MMML (0o, X,,", yY"), - Ty The meta-parameter On is learnt by iteratively taking a gradient descent with respect to the test loss on the training tasks given by Σ L( Vú, IMAML. (0o. J:)). The parameters for a task are obtained directly from On and does not make use of any information from the other training tasks. As result, the common do learnt during the meta-training treats all tasks equally - the algorithm implicitly assumes similarity of all tasks, but is not able discern or quantify the degree of similarity or dissimilarity among the tasks. In contrast, our algorithm involves an adaptation function 9IANML (to be defined later) that explicitly uses notion of similarity between the tasks to predict parameters for a task. As a result, we expect that our approach helps train predictors even when the data-sources that are not very similar to each other. our numerical experiments in Section 4, we see that this indeed the case the sinusoidal function as the data source.
1.2 RELATED WORK
The structural characterization of tasks and use of task-dependent knowledge has gained interest in meta-learning recently. Edwards & Storkey (2017) proposed a variational autoencoder based approach to generate task/dataset statisties used measure similarity. Ruder & Plank (2017) con- sidered domain similarity and diversity measures in the context of transfer learning (Ruder & Plank] 2017). The study of how task properties affect the catastrophic forgetting in continual learning was pursued by Nguyen et al. (2019). [ee et al. (2020) proposed a task-adaptive meta-learning approach for classification that adaptively balances meta-learning and task-specific learning differently for ev- cry task and class. Bayesian approaches have been proposed to capture the similarity across tasks in the form of task hyperpriors (Yoon et al., 2018 Finn et al., 2018 Grant et al., 2018: Rothfuss al.) 2020). Pask-similarity defined through effective sample size has been used to develop new off- policy algorithm for meta-reinforcement learning (Fakoor et al. 2020). was shown by Oreshkin et al. (218)) that the performance few-shot learning shows significant improvements with the use of task-dependent metrics. While the use of kernels or similarity metrics is not new in meta-learning, they are typically seen in the context of defining relations between the classes samples within given task (Qiao et al.] (2188) Rusu et al. 2019| Vinyals et al. 2016 Snell et al. 2017 Oreshkin et al.) similarity metries in the activation space to predict parameters for novel categories in few-shot learning with zero-training. cotomattion-hhoreetee ideas have also been used in the study of the topology and the geometry of task spaces by [guuen et al. (2019); Achille et al. (2018). Achille et al. (2019) construct vector rep- resentations for tasks using partially trained probe networks, based on which task- siii-iiiily metries are developed. Task descriptors have been of interest specially in vision related tasks in the context of transfer learning (Zamir et al., 2018 Achille et al., 2019 Tran et al. 2019). Recently, neural tan- gent kernels were been proposed for asymptotic analysis of meta-learning for infinitely wide neural networks by considering gradient based kernels across tasks by Wang et al. (2020).
2018 Fortuin & Rätsch [019] Goo & Niekum 2020). Qiao et al. (018)
The work of Wang et al. (2020) is the closest in spirit to our work in that they consider kernels across meta-learning tasks. However, the premise of their work is very different from ours. The aim their work is to show how global convergence behaviour of popular MAML type task-specific adaptation can be assymptotically described using specific kernels, when every task involves training deep neural networks of asymptotically infinite very large widths. Our premise on the other hand is entirely different - we consider a task-specific adaptation that actively employs similarity of tasks in the form of valid kernel functions, in order to improve meta-learning performance. Our work does not make assumptions on the nature of the kernel, the structure of the learner, or its dimensions.
2 REVIEW OF MAML AND META-SGD
To facilitate our analysis, we first review MAML and Meta-SGD approaches and highlight the rel- evant aspects necessary for our discussion. We shall then show how these approaches lead to the definition of generalized meta-SGD and consequently, our TANML approach.
2.1 META AGNOSTIC
Model-agnostic meta-learning proceeds in two stages iteratively. As discussed in the motivating example, the parameter 0 for the ith training task X, V,, X,, Vi, i 1,.", Ter is obtained by
3

---

Under review as a conference paper at ICLR 2021
applying the adaptation function 9MAML to 8g 0; 9MAML 24, J.). This is called the inner update. Once 0g is obtained for all the training tasks, d is then updated by running one gradient descent step on the total test-loss given by >i=i L(, Vi 9MAML (00, N, ):). This is called the outer update. Each outer update involves Trr inner updates corresponding to the Tir training tasks. The outer updates are run for Nster iterations. This constitutes the meta- veta-training phase of MAML, described in Algorithm [ Once the meta- meta-training phase complete and 0g is learnt, the parameters for a new test task are obtained by applying the inner update on the training data of the test task. We note here that MAMI described Algorithm opposed to the general MAML where the inner update may contain several gradient descent steps. We note that when we talk of MAML in this paper, we always refer to the first-order MAML. A schematic of MAML is presented in Figure 1.
the first-order MAMI. (Finn et al. 2018), as
Algorithm 1: Model agnostic meta- vetaallarning (MAML)
Initialize 0, for Niter iterations do for i 1, Ter do 9MAMM(00 X,, (i
0 -o,,,
[Inner update]
end
0. 0n -Vo. En=
x4,))
[Outer update]
end
2.2 SCCA-STOCHASTIC GRADIENT DESCENT (META-SGD)
Meta stochastic gradient descent (eta-)SGD is variant of MAMI. that learns the component-wise step sizes for the inner update jointly with 0g. Let denote the vector of step-sizes for the different components of 0. As with MAML, the meta-training phase of Meta-SGD also involves an inner and outer update. The outer update computes the values of 0g and o; the inner update computes the parameter values 0, using the adaptation function RD x D x Kn= x Knn, e (0u, a, V4) + GMSGD a, X, V+)= RDD defined
fMSGD a, N. - 0 - a Vo. ((X,
where operator denotes the point-wise vector product. The outer update is rum for Nater iterations. The meta-training phase for Meta-SGD is described in Algorithm
Algorithm 2: Meta-stochastic gradient descent Initialize [f)
for Niter iterations do for i 1, Ter do
9MSGD O1 Vi) 0o a Vo, [Inner update]
end
(0. al = 1 al BV ΣTur ( 22,
[Outer update]
end
The predictor for the ith test task is then trained by applying the inner update on X,, Vr, using the values of 0g and a obtained in the meta-training phase.
We notice that the inner update is expressible as IMSGD (6. X. )b) = W (,0)) where
W I, [a] and (0,) A
[o Vo, C0 T
(1)
The matrix T denotes the transpose of W. denotes the identity matrix, and diag(a) denotes the diagonal matrix whose diagonal is equal to the vector We refer to 2,(0u) as the task descriptor of the ith training task. Thus, DDSGDD(p, a, X,, J/) takes the form of a linear predictor for 0 with (0,) the input and regression coefficients W. The adaptation 9MSGD(OU, a, X, J.) can be generalized to the case of W being a full matrix that to be learnt from the training tasks. This generalization results in the adaptation function R D x T3. Dx2D x TR Kn, x R RRKy (6o. W. X,Y) gGMSCD(O0 W, x,,' E DD given by
JGMSGD (0., W, W, W (0)) W 0. + W. Vo, C(2,
4

---

Under review as a conference paper at ICLR 2021
where W1, W2 R are the submatrices of W such that w [W1 W2]. Expressed in this manner, we notice how JGMSGD performs a parameter update similar to a second- order gradient update with W2 taking role similar to the Hessian matrix. We refer to the resulting meta-learning algorithm as the Generalized Meta-SGI) described Algorithm The second term ((W) in the outer loop cost function is regularization that ensures W bounded and avoids overfitting. On setting μ 0 and using W as defined in equation the Generalized Meta-SGD reduces the Meta-SGD. The Generalized Meta-SGD is thus more general form of MAML arrived at by viewing MAML/Meta-SGD as linear regression.
[
Algorithm 3: Generalized Meta-SGD
Initialize 0. W E 2DDxD] for Nater iterations do for i 1,, Ter do
(OM W, Di) = W (00) [Inner update]
end
10. W] (0. W] Wo.o.. EE 9ecMscooo W, W.K. + vol(W)
ICMSGD(O0.
end
3 TASK-SIMILARITY AWARE META-LEARNING
It is well known that the expressive power of linear regression is limited due to both its linear nature and the finite dimension of the input. Further, since the dimension of linear regression matrix W grows quadratically with the dimension of 0, large amount of training data would be necessary to estimate it. A transformation linear regression the form of "kernel substitution" or 'kernel trick' results in the more general nonparametric or kernel regression (Bishop] 2006; Schölkopf & Smola 2002). Kernel regression essentially performs linear regression in an infinite dimensional space making it simple yet powerful and effective nonlinear approach. This motivates to kernel regression model as the natural next step from the Generalized Meta-SGD developed in the earlier section. By generalizing the linear regression model in JGMSGD, we propose adaptation function RDD x R x kknz x kng e (0o, V X. Y) + STANML (0o, VI X, 3) E R² in the form of nonparametric kernel regression model given by
Pir
STANMI (0,, U X. = _ + k (0). T i(0..
(2)
j=1
where k [k (2, 21 (0n)), ...,, (z.) 2T, (0))] is the vector with kernel values between the ith training task and all the Ter training tasks, and V = (.b,,...., VT,..] is the matrix of kernel re- gression coefficients stacked along the columns. The kernel coefficient matrix v and the parameter On are learnt in the veta-training phase by iteratively performing an outer update as in the case of the Generalized Meta-SGD. The computed and 0, are then used train the predictor for new test task by applying the inner update on its training data. We call our approach Task-similarin Aware Nouparametric Letannn (TANML) since the kernel measures the similarity between tasks through the task-descriptors. As we show in the Appendix, TANML reduces to Meta-SGD and MAML for specific choices of the kernel and the regression coefficient matrix vb.
22D0 X 2D 1 R denotes a valid kernel function k(0.. i)
A
The kernel regression in equation
2 models 0, belonging to the space of functions defined as H:
Ter Σ kZz zy (oo),, b. E ERD 1' =' ER
H [
0::0
Th
(3)
The space is referred to as the reproducing kernel Hilbert space (RKHS) associated with the kernel k(:,)). We refer the reader to (Hofmann et al. 2008) and (Schölkopf & Smola 2002) for further reading on RKHS. The space H. has an important structure: each function in H. uses the
A valid kernel function is one that results in the kernel matrix evaluated for any number of datapoints to be symmetric and positive-semideeiiit cf. (Bishop 2006)
5

---

Under review as a conference paper at ICLR 2021
meta learnng
mela learning
task edaptation
acapeation
i
44
VLi
VC2
04
O
X2
V
C2
VLa
V
2
Z
0 RKHS
Figure 1:
Left: Schematic of MAML. Right: Schematic of the TANML. Only the computation of 0 shown to keep the diagram uncluttered.
information (the coefficients +b) from all the Tir training tasks weighted by the kernel that essentially quantifies similarity or correlation between the tasks through the task-descriptors defined earlier. While all meta-learning approaches use similarity implicitly, the number of works actively using similarity with training data at test time limited (Fakoor et al.) (020). Computing the optimal values of the kernel coefficients v and 0.. which forms the meta taaatttrannngg phase, is then equivalent to solving the functional minimization problem:
Σ C. +
arg min 0..EE
where the regularization term is the squared-nort the RKHS which promotes smoothness and controls overfitting, μ being the regularization constant. The squared- norm in an RKHS is defined as |10133 A ETn Σ Ex= & (.,, (z)), z. (0o)) = ((0,) w: and K(0,) I Te- is the matrix of kernels evaluated across all the training tasks. This novel connection between meta-lcarning and RKHS obtained from TANML could potentially help in the mathematical understanding of existing algorithms, and help develop new meta-learning algorithms in the light of the RKHS theory (Hofmann al.) 2008, Schölkopf & Smola 2002).
The meta meta-training training phase for TANML is described in Algorithm where we use (
V ((O,) V. In general, other regularizations such as l or le norms could also be used. We also note from equation 2 that the TANML approach is a general framework: any kernel and any task-descriptor which meaningfully captures the information in the task could be employed. What constitutes a meaningful descriptor for a task is an open question; while there have been studies on deriving features and metrics for understanding the notion of similarity between data sources or datasets, they have mostly been domain- specific and often require separate 'probe' neural networks for the extraction of features (Kim et al. (2019). The particular form of the task-descriptors used as a starting point, and follows natu- rally from analyzing them through the lens of linear and kernel regression. A schematic describ- ing the task-descriptor based TANML and the intuition behind its working is shown in Figure
in our derivation is the result of taking MAML/Meta-
Algorithm 4: Task- Aware Meta Learning
Initialize 10. V TR. for Nater iterations do for i 1, Ter do
TTANMI. Og V, X, V.) =M (K((0,))) [Inner update]
end
10. V update]
10. V
AV VV Ete L(X, Di, STANNI, v X.)) + vV((V) [Outer
end
On the choice of kernels and sequential training While the expressive power of kernels is im- mense, it is also known that the performance could vary depending on the choice of the kernel function (Shtlkoof & Smola 2002). The kernel function that works best for dataset is usually found by trial and error. A possible approach is to use multi-kernel regression where one lets the data decide which of the pre-specified set of kernels relevant (Sonnenburg & Schäfer 2005

---

Under review as a conference paper at ICLR 2021
Gönen & Alpaydin 2011).
comain-specifit knowledge may also be incorporated in the choice of kernels. In our analysis, we use two of the popular kernel functions: the Gaussian or the radial basis function (RBF) kernel, and the cosine kernel.
We note that since MAML and Met and similar approaches perform the inner update inde- pendently for every task, they naturally admit sequential batch based training. Since TANML uses an inner update in the form of a nonparametric kernel regression, it inherits one of the limita- tions of kernel-based approaches a result, the task losses and the associated gradients for all the training tasks are used at every inner update of TANML. One way overcome this limitation would be the use of online or sequential kernel regression techniques (Lu et al. [2016] Sahoo et al. 2019: Vermaak et al. 2003). We will pursue this in our future work.
that all training data is used simultaneously for every task. As
4 NUMERICAL EXPERIMENTS
We evaluate the performance of TANML and compare it with MAML and Meta-SGD on two syn- thesized regression datasets, five real-world time-series prediction problems from the Physionet 2012 Challenge dataset (Silva et al. 2012: Rothfuss et al. 2020), and on few-shot classification dataset Omniglot (Lake al. (0155). The synthesized regression tasks have been used previously by previous works (Denevi et al. 2018 Finn et al.) 2017| 2018)) in meta-learning as a baseline for evaluating the performance on regression tasks. In every experiment, the predictor f(x,, 0) is the output of fully-comnected four-layer feed-forwart neural network, with Rectified linear unit (ReLUU) as the activation function; 0 is the vector of all the weights and biases in the neural network the predicted output is a scalar i or vector y. for the regression and classification tasks, respec- tively. We consider two kernel functions for TANML: the Gaussian kernel k kzz,(0)) z. (0o)
exp (-lz,(0n) z, (0)))1//²)) and the cosine kernel k(z, (0o): (0o))
[2, [ In
(0.) (0.)
order that the structural similarities are better expressed, we update kernel regression for the pa-
rameters of the different layers separately. This is because using a single adaptation function all components of 0: might result in certain parameters dominating the kernel regression, specially when the dimension of the parameters becomes large. Hence, we perform the adaptation sepa- rately for components of 0; corresponding to the different layers I = 1, 1....... L using the adaptation functions IR x IR Dj xT Kn, x R KKn e (0u,1. 1 X, V) + ( Lr x, ))) E Rt for the parameter 0., belonging to the Ith network layer: 0,, 9TANMI., (0.,t, VI, J,,)
E/e & kiz (O.. 2 (On,)), = 1.... L. The NMSE performance on the meta-test set is obtained by averaging over 30 Monte Carlo realizations of tasks. For the regression tasks, the per- formance of the various meta-learning learning approaches are compared using the normalized mean-squared
2i Km (yk- Dx)²
TT K
error (NMSE) on the test tasks: NMSE
The numerical details of the ex- E Σk=1 22
K
periments not mentioned in the manuscript, such as the learning rate and other hyper-parametent given in the appendix for space constraints.
x. The task data pairs (z.y) E 16 x R are generated by linear model y B x + The regression coefficient vector ,3 for different tasks is randomly sampled with equal probability from two isotropic Gaussian distributions on B: with means 8g -4 and 3g = 4, where 4 denotes the vector of all fours. The additive noise e is assumed to be white and drawn from the standard normal distribution and uncorrelated with x, which is distributed as the multivariate normal distribution of size 16. We consider two cases of Ter = 32 and Tü, = 64 training tasks, Each task with K = 4 samples in training and test sets. We evaluate the performance of the MAML, Meta-SGD, and TANML on a test set of T, = 64 tasks. The NMSE test performance is reported in Table [T We observe that both MAML and Meta-SGD perform very poorly in comparison to TANMI: Meta-SGID performs slightly better than MAML. Further, we observe that TANML with the Cosine kernel performs the best among the four algorithms. The superior performance of TANML could be ascribed its the nonlinear nature with the gradients enter the estimation through the kernels. The adaptation function involves terms with products of different gradients acting in the spirit of a higher order method unlike MAML/Meta-SGD that use first order adaptation. This also corroborates with the findings of the recent theoretical work by Saunshi et al. (2020), where they show that MAML-type approaches can fail under convex settings (as is the case in this experiment). It is also interesting to
Experiment We consider the task of training linear predictors of the form f(z, 0) = T
7

---

Under review as a conference paper at ICLR 2021
Algorithm Tor 32
MAML
Meta-SGI
TANML-Gaussial
TANML-Cosine
0.95
0.91
0.185 0.15
0.079
To 64
0.91
0.86
0.070
Table 1: NMSE on test tasks for the regression experiment
Algorithm
Experiment 2a
Experiment 2a
Experiment 2b
Experiment 2b
10% outlier
20%
outlier
10% outlier
20%
outlier
Tir
256
512
256
512
256
512
256
512
MAML
0.83
0.77
0.75
0.74
0.89
0.81
0.83
0.76
Meta-SGI
0.92
1.04
0.81
0.93
1.5
0.92
1.06
0.93
GausiaGa
0.4
0.41
0.38
0.38
0.76
0.60
0.73
0.58
TANML-Cosine
0.37
0.35
0.30
0.26
0.44
0.38
0.47
0.33
Table 2: NMSE test tasks for regression experiment 2.
note that the value of 0g we obtain for TANMI almost coincides with O, which is the value for dg that theoretically minimizes the average error for this problem. We also find that both cosine and Gaussian kernels converge typically in about 5000 iterations, whereas MAML and Meta-SGD do show improvement in NMSE even after 30000 iterations.
Experiment In this experiment, we consider the task of training of non non-linear predictors which correspond to a fully connected ANN. We consider data pairs (x, E TR x RR generated from the sinusoidal data source y = sin (wx), where x drawn randomly from the interval [-1, 1]. and A and differ across tasks. We do not use the knowledge that the data comes from sinusoidal shots or data-pairs in each task. In order
source while training the predictors. We are given to illustrate the potential of TANML in using the similarity/ dissimilarity among tasks, we consider a fixed fraction of the tasks to be outliers, that is, generated from a non non-sinusoidal data source in both meta- meta-training and meta-test data, described next. We consider two different regression experiments:
(1) Experiment 2a Fixed frequency varying amplitude: The data for the different tasks generated from sinusoids with drawn randomly from (0, 1]. setting w 1. The outlier task data generated as y(x) Ar.
(2) Experiment 2b - Fixed amplitude varying frequency: The data for the different tasks gener- ated from sinusoids with w randomly drawn from [1, 1.5], setting A The outlier task data is generated as y(r) wx.
We perform the experiments with the number of meta-traine teetatttaiingg tasks equal Tir 256 and Túr 512. The NMSE performance on test tasks obtained by averaging over 100 Monte Carlo realizations of tasks is reported Table. We observe that TANML outperforms both MAML. and Meta-SGD in test prediction by a significant margin even when the fraction of the outlier tasks is 10% and 20%. This clearly supports our intuition that an explicit awareness or notion of similarity aids in the learning. specially when the number of training tasks is limited. We also observe that on an average TANML with the cosine kernel performs better than the Gaussian kernel. We note that the performance of the approaches in Experiment 2a is better than that in Experiment 2b. This is because there is higher variation among the tasks (changing frequency) than in Experiment 2a (changing amplitudes). We also observe that the performance improves slightly as T., increases.
Experiment 3 In this experiment, we consider the the regression task of time-series prediction on the ICU patient dataset from the Physionet 2012 challenge. The dataset consists of measurements of various vital characteristics of different patients monitored over a period of 48 hours, logged in at various non-uniform time instants. For a given vital characteristic, the time-series of every individual patient forms a task: the time of measurement and the measured value are the input and output, respectively. The goal of the experiment is to predict the time series for new unseen patients given the measurements taken during the first 24 hours. The experiment done in five different experiment settings corresponding to five different vital characteristics(V) to V5 ). Since our interest is in evaluating the performance in the limited task setting, we consider 100 tasks each for the meta- training. meta-valin vettavvaliation. and meta-testing. Further details of the implementation are given in the Appendix. The NMSE values obtained for the test tasks are shown Table 3 We observe that in all the experiments TANML with the Gaussian kernel results in the least NMSE, though we also note
8

---

Under review as a conference paper at ICLR 2021
Algorithm
V
1/
V/
V4
V/3
MAML
0.56
0.50
0.12
0.13
1.2
Meta-SGD
0.03
0.18
0.04
0.03
0.63
TANMI-Cosine
0.13
0.25
0.05
0.06
0.56
TANMI-Gaussian
0.03
0.12
0.04
0.02
0.42
Table 3: NMSE on test tasks for regression experiment on Physionet 2012 dataset.
Algorithm MAML
Experiment 4a: T₁, 50
Experiment 4b: T., 150
16
18.2
Meta-SGD
18.4
19.5
TANMI-Gaussian
19.6
22
TANMi-Cossie
23.2
24.1
Table 4: Test accuracy % on Omniglot dataset.
that the performance of Meta-SGD also coincides with that of TANML in some of the experiments. On comparison with results from Experiment
observe that while Meta-SGD shows significant improvement over MAML, it exhibits severe instability in the presence of outliers. On the other hand, we observe that TANML performs good prediction in both the experiments.
Ablation Study An ablation study of some aspects of TANML is given in Section B of the Ap- pendix. The study is made on the data from Experiment 3.
Few-shot learning We now consider the application of our approach to feo-shot learning on the character classification data from the Omniglot dataset. We consider the case of waw 1-shot learn- ing: that each task consists of a 5-class problem where each class is given two samples: one for training and the other for validation. The classes correspond different written characters /symbols of a language, the input is the image in a gray-scale value. Given one training image per class, the goal of each task is to build 5-way classifier and evaluate its performance on another set of five images. Unlike the state-of-the-art approaches where the goal is to perform few-shot learning over large number of training tasks (Li et al.] 2017). we restrict our experiments setting. We consider two settings, where we use Ter 50 and Tir = 100 meta- meta-trainie -raininiiig tasks. and evaluate the performance on a test set of 100 tasks. The cross-entropy loss used as the objective function for training. The test performance is measured in terms of the accuracy defined as the ratio of correctly classified samples to the total number of samples in each task and is reported in Table 4 We observe that TANML with both the cosine and the Gaussian kernel outperform MAML and Meta-SGD. We note that the accuracy values are low due to the limited task setting that we have considered. Nevertheless, the experiments demonstrate the potential of TANML improve few- shot learning in extremely data-limited scenarios. We predict a similar trend to be exhibited even when the number of training tasks is large.
the limited task
CONCLUSION
We proposed a task task-similarity aware meta-leaminn algorithm that explicitly quantifies and employs a similarity between tasks through nonparametric kernel regression. We showed how our approach brings novel connection between meta-learning and reproducing kernel Hilbert spaces. Our hy- pothesis was that an explicit incorporation of task-similarity helps improve the meta-learning per- formance in the task-limited setting with possible outlier tasks. Experiments with regression and classification tasks support our hypothesis, and our algorithm was shown to outperform the popular meta-learning algorithms by significant margin. The aim of the current contribution was to inves- tigate how task-similarity could be meaningfully employed and used to advantage in meta-learning. To that end, we wish to reiterate that the study is ongoing one and the experiments considered in this paper are in no way exhaustive. An important next step for our approach is also the use of online/sequential kernel regression techniques to run algorithm in sequential batch-based manner and scale to scenarios with large number of tasks. The nonparametric kernel regression framework also opens doors probablistic or Bayesian treatment of metanlcg that we plan to pursue in the recent future.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille, Glen Mbeng, and Stefano Soatto. Dynamics and reachability of learning tasks. 2018.
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji. Char- less C. Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta- learning. In 2019 IEEE/CVF International Conference on Computer Vision. ICCV 2019, Seoul, Korea (South). October 27 November 2, 2019, pp. 6429-6438. IEEE, 2019. doi: 10. 10111C/C.22099999 1.069. URL httphttpss:/o/o http:///i.orr//////////o 10w.1119//CCCC wt..///..11///1CC./ 201 9.00....00 00653
Kelsey R. Allen, Evan Shelhamer, Hanul Shin, and Joshua B. Tenenbaum. Infinite mixture proto- types for few-shot learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceed- ings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 232- 241. PMLR, 2019. URL
mlr.. ppwss/v97 aaleenl 9b. html
C. M. Bishop. Puttern Recognition and Muchine Learning (Information Science and Statistics). Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2006. ISBN 0387310738.
Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Learning to learn around A common mean. Advances in Neural Information Processing Systems 31: Annual Con- ference Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pp. 10190-10200, 2018. URL http:://ppper: nips. cc/paper/
8220 weeeeernnngtte learnearrnnneee www.annn--ood----o a common-mean
Harrison Edwards and Amos J. Storkey. Towards neural statistician. In 5th International Confer- ence on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL http:htts:////t /ttpps:://penrreiieer net forum?id
HJDBUF51e
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J. Smola. Meta-q-learning, 2020.
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. In 6th International Conference on Learn- ing Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. HyjC5yWCR
net htt.://wwwwe/eooom...... forum?id=
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 1126-1135, 2017. URL http:
//proceedings. . mlr www.spresssv.. /0/////00//E// fiffflin1ll html
Chelsea Finn, Kelvin Xu, and Sergey Levine Probabilistic model-agnostic meta- learning. In Advances in Neural Information Processing Systems 31: Annual Confer- ence on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018,
Montréal, Canada, pp. 9537-9548, 2018. URL http::/papee:: nips cc/paper/
Vincent Fortuin and Gunnar Rätsch. Deep mean functions for meta-lcaming in gaussian processes. CoRR, hass/190.00099. 2019. Utl.hbttp://rxx...org///e
08098
M. Gönen and E. Alpaydin. Multiple kernel learning algorithms. J. Mach. Learn. Res., 12:2211- 2268, July 2011.
Wonjoon Goo and Scott Niekum. Local nonparametric meta-learning. 2020.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning hierarchical bayes, 2018.
Thomas Hofmann, Bernhard Schölkopf, and Alexander J. Smola. Kernel methods in machine learning. Ann. Statist, 36(3) 06 2008 doi: 10. 10112440090333770000007.. URL https: //do:///oii. . 121 /tt.://...oooo.1111///0000000000
10

---

Under review as a conference paper at ICLR 2021
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-leamingg in neural networks: A survey, 2020.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, S. M. Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. CoRR, htss/9901.07611, 2019.
URL http://arxiv..
http:/bs/11000 05761
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266))1332--3338 2015. ISSN 0036-8075.
doi: 10.11///ciencc.aaabbb...... aab3050. URL httpss:/ www.::/www.siiiinnnn......... conto//nt//
350/ 2550/2560 6266/136663
Haebeom Lee, Hayeon Lee, Donghyun Na, Sachoon Kim, Minseop Park, Eunho Yang, and Sung Ju Hwang. Learning balance: Bayesian meta-learning for imbalanced and out-of-distribution tasks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview net/
Corum?ice www.mmmmdd-elllieee 1.d.-kkeeIBBYS JBY
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. CoRR, abs/1707.09835, 2017. URL
09:35
Jing Lu, Steven C. H. Hoi, Jialei Wang. Peilin Zhao, and Zhiyong Liu. Large scale online kernel learning. J. Mach. Learn. Res., 17:47:1-47:43, 2016. URL http::///mmm...o//ppperee.
217/14-148.1 html
Cuong V. Nguyen, Alessandro Achille. Michael Lam, Tal Hassner, Vijay Mahadevan, and Ste- fano Soatto. Toward understanding catastrophic forgetting in continual learning. CoRR, abs/1908.00 01001, 2019. httLttt:///xxxv..oo abs/ www./9/..............o.........e 01091]
Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: task de- pendent adaptive metric for improved few-shot learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicold Cesa-Bianchi, and Roman Gar- nett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pp. 719-729, 2018.
nips cc/paper/ URL wttp:p/_wee.ttttto---oooooooooe-
S. J. Pan and Q Yang. A survey transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10): 1345-11399 2010.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L. Yuille. Few-shot image recognition by predicting parameters from activations. In 2018 IEEE Conference Computer Vision and Pattern Recogni- tion, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 7229-7238, 2018. doi: 10.1 201.00055. / wptpaccsssccccccccpeppeeee.. thecvf wwt. content cvpr 2018/
ht.m
/ttp///aaFeeeeeeee Few- Shot Tmage_ Recognniion_CvVP
100118_pperrhttee httl
Jonas Rothfuss, Vincent Fortuin, and Andreas Krause. Pacoh: Bayes-optimal meta-learning with pac-guarantees, 2020.
Sebastian Ruder and Barbara Plank. Learning to select data for transfer learning with bayesian opti- mization. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.). Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 1111, 2017, pp. 372 1772-38..2 Association for Computational Linguistics, 2017. doi: 10.18653//1//11-11000 1038. URL https:: /o (0oooo..ooo0 www.:ooo.ooog..ooo
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In 7th International Con- ference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL httpttpss//pppeeeeieee..eee .et forum? id BJgKkhAcKT
Doyen Sahoo, Steven C. H. Hoi, and Bin Li. Large scale online multiple kemnel regression with application to time-series prediction. ACM Trans. Knowl. Discov. Data 13(1):9:1-9:33, 2019. doi: 10.1 10.1145/3299875. httpp::::: // .org/// 11415.1/33299.7.
11

---

Under review as a conference paper at ICLR 2021
Nikunj Saunshi, Yi Zhang, Mikhail Khodak, and Sanjeev Arora. A sample complexity separation between non-convex and convex meta-learning. ArXiv, abs/2002.11172. 2020.
Bernhard Schölkopf and Alexander Johannes Smola. Learning with Kernels: support vector ma- chines, regularization, optimization, and beyond. Adaptive computation and machine learning se- ries. MIT Press, 2002. ISBN 9780262194754, URLhttts: www. wor Ldcat... org/ oolc/
48970254
Ikaro Silva, George Moody, Daniel Scott, Leo Celi, and Roger Mark. Predicting in- hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. Com- puting in cardiology, 39:245-248, 2012. URL httpss:://ubmedddneo nlm.m.mm.... nih. gov
24678518
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learn- ing. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 4077-4087 Curran Associates, Inc., 2017. URL http: www.siipppsspp.ee..ee cc/paper/ 696-ppootttypiall--etttt wwwppppiiaa-eetotrreee.. for- few- shot wew.ninganinnnn pdf
G. Sonnenburg, S.and Rätsch and C. Schäfer. A general and efficient multiple kernel learning algo- rithm. Proc. Int, Conf. Neural Inf. Process. Syst. pp. 1273-1280, 2005.
Anh T. Tran, Cuong V. Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 1395-1405. IEEE, 2019. doi: 10. 10.1109//CCV.2219.00001
htLphttps:/
:/tp:s:///dot http//oooooog//////////o wro//o./...0 1101 RW..///ICC.... 2019.0011111
Jaco Vermaak, Simon J. Godsill, and Arnaud Doucet. Sequential bayesian kernel re- gression. In Sebastian Thrun, Lawrence K. Saul, and Bernhard Schölkopf (eds.), Ad- vances in Neural Information Processing Systems 16 [Neural Information Processing Sys- tems, NIPS 2003, December 8-13, 2003, Vancouver and Whistler, British Columbia, Canada), pp. 113-120. MIT Press, 2003. URL http::/papers: poppers.nis.... cc/paper/ 2362 3362-sequention ialeeetiiiibbbeiiannn.aaa -aw.aaabaaaasiin-eeeeeeeele keennn-eeeeeellll regression
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona. Spain, pp. 0630-3638, 2016. URL for one shot learning
hwtppppprssspss.....mm nips. cc/papex/
1985-maten matchiig-
Haoxiang Wang, Ruoyu Sun, and Bo Li. Global convergence and induced kernels of gradient-based meta-learning with neural nets, 2020.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems, volume 31, pp. 7332-7342. Curran Associates, Inc., 2018. URL httpps:/pprocedingg www.iis..eu....... wwweeisssccppppppee wttpc//p/pp//////11eeee 1BC21000111e20111400014441110111
pdf
Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In The IEEE Conference Com- puter Vision and Pattern Recognition (CVPR), June 2018
12

---

Under review as a conference paper at ICLR 2021
A CONNECTION OF TANML TO META-SGD AND MAML
We note that in the case when the kernel is the linear kernel (zz (fn), 2, (00)) = 2, (0)) (.00))=,,.00))
TANML becomes the special case of Generalized Meta-SGD proposed in Section 2.2. This is be- cause kernel regression and GMSGD further reduces to the Meta-SGD when the linear regression matrix W is constrained be of the form in equation and the regularization μ in the outer loop is set to zero. When regression coefficients of the GMSGD are fixed to W = II, -al, we obtain the MAML, which is a fixed linear transform acting on the task-descriptor. Thus, TANML reduces to the specific cases of Meta-SGD and MAMI,, when the kernel is the linear kernel and the regularization parameter y in the outer-loop is set to zero.
Let us again consider the adaptation rule of the TANML:
Ttr
gIANML L, V. = Σ + (,))( = v k000..
In the case linear kernel, we have that (z; (0.), (0)) - (0,) (00n)T2 2, (0o). Then, we have that k(0o. = Z (0.)z(0..))
where Z00) is the matrix of the task descriptors of all the training tasks arranged aallmmnwwie. This gives that
STANML ((,)) = V k(Oo. i) v Zz'z (00)=(0u,
In the case when V Z(0n) [I- diag(a)), we get that
fIANML (0. X, = Z (0))2((0): 1) [ diag(a)))((u. i) JMSGD
That is, when = I -diag(o:]] 9TANMI. = 9MSGD: where denotes the pseudo-inverse operation. In other words, TANMI with the kernel regression matrix constrained to be of the form
= [I diag(o)) and with no regularization or reduces to the Meta-SGD.
Further, by the same argument, have that in the case when TANML with V given by the fixed matrix V (Z(O))) al] gives
OTANML 9MSGD
This also helps gives an intuition on why behaviour of the TANML might perform better MAML and Meta-SGD are obtained by restricting the set of possible Vi constrained set. In contrast, TANML obtains ! through unconstrained search over R Tr. D
Thus, MAML and Meta-SGD are obtained as special cases of TANML. when the kernel the linear or the simple inner- mnoruttoduct kernel, and the regression matrix takes special forms.
B ABLATION STUDY
We consider the study of the influence of two aspects on TANMI::
B. REGULARIZATION PARAMETER l
As in the case of any regularized estimation or modelling approach, the motivation of including the regularization parameter is to avoid the overfitting for v specially in the task-l setting. Keeping all other parameters unchanged, we vary 14 and the resulting test NMSE is shown in Table 5
We observe that both a very large μ or a very small μ degrades the test performance. This is because unlike MAML and Meta-SGD, the since TANML estimates more number of parameters from the same training data, it tends to quickly overfit the training data to almost zero error. Hence, nonzero value of μ helps curtail the overfit. However, setting j to a large value biases the parameters such that very little learning takes place beyond the initial few meta-iterations resulting also in a poor test performance. Thus, the best test prediction being acheived with y set to 0.01.
13

---

Under review as a conference paper at ICLR 2021
J
V1
Vi Gaussian
V3
V3
V4
Cosine
Cosine
Gaussian
Cosine
Gaussian
0.63
0.56
0.39
0.31
0.52
0.63
0.1
0.34
0.08
0.06
0.04
0.05
0.05
0.01
0.13
0.03
0.05
0.04
0.06
0.02
0.001
0.32
0.08
0.1
0.07
0.08
0.06
Table 5: Effect of for regression experiment Physionet 2012 dataset for Ex1, Ex2, Ex3.
This in turn indicates that the regularization helps TANML perform better than MAML and Meta- SGD by avoiding overfitting the training data. This is perhaps why TANML exhibits a superior performance in comparison with the other two methods as we have seen in Section A of the Appendix, MAMI, and Meta-SGD are obtained when TANML uses linear kernel and zero regu- larization.
14

---

Under review as a conference paper at ICLR 2021
C NUMERICAL EXPERIMENTS
We compare four different approaches: MAML, Meta-SGD, TANML-Cosine, the algorithms were trained for 60000 meta-iterations, where each meta-iteration outer update uses the entire set of training tasks, and not stochastic gradient descent. All the experiments were performed on either NVIDIA Tesla K80 GPU on Microsoft Azure Platform.
D EXPERIMENT 1 HYPERPARAMETERS
In this experiment, task has four training input-output pairs and test input-output pairs, that is, K 4. Input æ E IR is drawn from (NO, I), the additive noise is white and drawn from from.V(0) (V 1).
linear predictor of dimension 16, same the input dimension. Each
D.1
MAML
Inner update learning rate: 0.01
Outer update learning rate: 10-4
Optimizer: Adam
D.2
META-SGD
Inner update learning rate a: learnt, initialized with values randomly drawn from (0.001, 0.01]
Outer update learning rate for 0:: 5 10-4
Outer update learning rate for 1 10-6
Optimizer: Adam
D.3
TANML-GAUSSIAN
Outer update learning rate for 0g: 1 10-3
Outer update learning rate for vr x 10-5
k = 0.1
a² = 0.5
Optimizer: Adam
D.4
TANML-COSINE
Outer update learning rate for 0, =0::
: 5 x 10-4
Outer update learning rate for V: 10-5
μ 0.1
Optimizer: Adam
D.5 EXPERIMENT 2 HYPER-
The hyper-parameters for the four approaches are listed below. The learning-rate parameters were chosen such that the training error converged without instability.
(1) Experiment 2a Fixed frequency varying amplitude: The data for the different tasks generated from sinusoids with A drawn randomly from (0, 1J. setting as yzx) = Ar.
1. The outlier task data generated
(2) Experiment 2b Fixed amplitude varying frequency: The data for the different tasks gener- ated from sinusoids with w randomly drawn from [1, 1.5), setting A The outlier task data
generated as (xx) wt.
15

---

Under review as a conference paper at ICLR 2021
D.6 MAML
Inner update learning rate: a: 0.01
Outer update learning rate: 10
Total ANN layers: with, 2 hidden layers
Non-linearity: ReLU
Optimizer: Adam
D.7
META-SGD
Inner update learning rate a: learnt, initialized with values randomly drawn from [0.001, 0.01]
Outer update learning rate for 0:: 10
Outer update learning rate for 1 10-6
Total ANN layers: with, 2 hidden layers
Non-linearity: Rel.U
Optimizer: Adam
D.8
TANML-GAUSSIAN
Outer update learning rate for 0g: 1 10-3
Outer update learning rate for Vr: 5 x 10-5
μ 0.1
0² 0.5
Total ANN layers: with, hidden layers
Non-linearity: ReLU
Optimizer: Adam
D.9
TANML-COSINE
Outer update learning rate for 0g = =0o: x 10-4
Outer update learning rate for V: 10-5
μ 0.1
Total ANN layers: with, 2 hidden layers
Non- Non-linearity: ReLU
Optimizer: Adam
E EXPERIMENT 3 HYPERPARAMETERS
For every task, we use half of the samples for training data, and the remaining half for the validation. In the case test tasks, the training set consists of samples from the first half of the time series-the goal is to predict the rest of the time series. In the case of training tasks, the training and validation sets are drawn randomly from the entire time-series. We consider only those patients which have at least four measurements. In order to minimize the dynamic range of the measurements, we divided all the true measurements by 150. The time was measured in minutes. The five vital characteristics considered (Silaa et al. 2012):
V1: Blood urea nitrogen
V2: Creatinine
V3: Invasive diastolic arterial blood pressure
V:: Urine
16

---

Under review as a conference paper at ICLR 2021
Vg: Heart-rate
The number of datapoints in each task varies from 4 to 48. The tasks with number of datapoints less than four are not considered. The following hyper-paramettee settings using in all the five vital characteristics.
E.1
MAML
Inner update learning rate: 10-3
Outer update learning rate: x 10-3
Total ANN layers: with, 2 hidden layers, neurons per layer
Non-linearity: ReLU
Optimizer: Adam
E.2
META-SGD
Inner update learning rate learnt, initialized with values randomly drawn from [0.001, 0.01]
Outer update learning rate for ou: 5 10-3
Outer update learning rate
10-5
Total ANN layers: with, 2 hidden layers, neurons per layer
Non-linearity: ReLU
Optimizer: Adam
E.3
TANML-GAUSSIAN
Outer update learning rate for 0o: 1 10-3
Outer update learning rate for v: 5 x 10-5
0.01
σ²
Total ANN layers: with, 2 hidden layers
Non-linearity: ReLU
Optimizer: Adam
E.4
TANML-COSINE
Outer update learning rate for 0, = =0: 5 x 10-4
Outer update learning rate for V: 1 x 10-5
0.01
Total ANN layers: with, 2 hidden layers
Non-linearity: ReLU
Optimizer: Adam
F EXPERIMENT HYPERPARAMETERS
We use two separate sets from training and test tasks as considered by Finn & Levine 2018 The images resized to 64x 64 dimensions.
17

---

Under review as a conference paper at ICLR 2021
F.1 MAML
Inner update learning rate: a: 10
Outer update learning rate: 10-3
Total ANN layers: with, 2 hidden layers, neurons per layer
Non-linearity ReLU
Optimizer: Adam
F.2
META-SGI
Inner update learning rate a: learnt, initialized with values randomly drawn from (0.001, 0.01]
Outer update learning rate for 0: 5 x 10-4
Outer update learning rate for a: x 10-5
Total ANN layers: with, hidden layers, neurons per layer
Non-linearity: ReLU
Optimizer: Adam
F.3
TANMIAGAUSSIAN
Outer update learning rate for 0g: x 10-3
Outer update learning rate for V: x 10-5
0.01
σ²
Total ANN layers: 4 with, 2 hidden layers
Non-linearity: ReLU
Optimizer: Adam
F.4
TANML-COSINE
Outer update learning rate for 0u = =0; 5 x 10-4
Outer update learning rate for V: 1 x 10-5
0.01
Total ANN layers: with, 2 hidden layers
Non-linearity: ReLU
Optimizer: Adam
18


Reviews:
[{"review_id": "mxnqg70QLyV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---- Update ----\n\nI thank the authors for clarifications. I trust that the suggestions of all reviewers, taken together, provide substantial avenues for improving the work. However, at this point I must keep my score and encourage the authors to continue the work with the valuable honest feedback provided here.\n\n---- Original Review ----\n\nSummary:\n\nThe paper aims to formalize “task similarity” in meta-learning settings by making use of nonparametric kernel regression techniques; such similarity information is then proposed as a means to alleviate some of the current issues with meta-learning algorithms such as MAML/Meta-SGD, namely reliance on large sets of similar meta-training tasks. Experiments focus on standard toy regression tasks with the added meta-training data scarcity.\n\n\n\nStrong points: \n- Principled approach to a challenging and current problem of interest for the sub-field of meta-learning and beyond. However, formalization of meta-learning approaches in terms of the NTK is recent but not novel [see reference Wang 2020 in paper].\n- Good work in progress, but the attempt to publish is premature.\n\n\n\nWeak points:\n- Very poor representation of relevant and conceptually similar recent work, see [1] for a comprehensive review, and specifically [2, 3, 4, 5, 6] for similar approaches.\n- Inaccurate claims of novelty are made; they must be made more specific and put into context. For example, the claim that task descriptors have not been used in the design of meta-learning algorithms is false, see [5, 6]. That said, the current approach could be used to analyze such SOTA approaches and perhaps explain their performance.\n- Meta-training data reuse across tasks at test time has been proposed previously, e.g. [7], so it is also not novel to this paper.\n- Very weak experimental evidence. Please use some of the few-shot image classification datasets, or standard RL tasks available since MAML was published.\n- Proposed method needs extensive approximations to scale up to more interesting problems.\n\n\n\n\nRecommendation and Rationale:\n\nI believe the paper should be rejected in current form, but I strongly encourage the authors to add more experimental data and submit to a workshop.\n\n\n\nReferences:\n[1] Meta-Learning in Neural Networks: A Survey\nTimothy Hospedales, Antreas Antoniou, Paul Micaelli, Amos Storkey. https://arxiv.org/pdf/2004.05439.pdf\n[2] Recasting Gradient-Based Meta-Learning as Hierarchical Bayes\nErin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, Thomas Griffiths. https://arxiv.org/abs/1801.08930\n[3] Bayesian Model-Agnostic Meta-Learning. Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, Sungjin Ahn. https://papers.nips.cc/paper/7963-bayesian-model-agnostic-meta-learning.pdf \n[4] Probabilistic Model-Agnostic Meta-Learning. Chelsea Finn, Kelvin Xu, Sergey Levine. http://papers.nips.cc/paper/8161-probabilistic-model-agnostic-meta-learning\n[5] Meta-Learning with Latent Embedding Optimization. Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, Raia Hadsell. https://arxiv.org/abs/1807.05960\n[6] Few-Shot Image Recognition by Predicting Parameters from Activations. Siyuan Qiao, Chenxi Liu, Wei Shen, Alan Yuille. https://arxiv.org/abs/1706.03466\n[7] Meta-Q-Learning. Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, Alexander J. Smola. https://arxiv.org/abs/1910.00125\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "dkQ2zIaS00h", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduced a meta-learning framework in which a kernel describing similarity between the tasks is used to construct an RKHS which is used to perform kernel regression. The framework is instantiated in a form of an algorithm: TANML which can be viewed as an extension to a popular Meta-SGD algorithm. The experiments on two regression tasks are presented to analyse the efficacy of the proposed method.\n\n1. I consider the method mathematically sound, ie. I don't see theoretical reasons which would make it obvious that it wouldn't work.\n2. The combination of using task-similarity and kernels is not present in the literature known to me, so the work under review contains (some elements of) novelty.\n3. However, both \"explicitly employing task-similarity\" (Achille et al. 2019) and \"using kernel methods\" (Vinyals et al. 2016) (separately) is well represented in past works.\n4. In my opinion moving kernels from space of images/classes (like in Vinyals and other kernel methods cited in the paper) to the space of tasks doesn't, on its own, demonstrate the level of novelty that is required by accepted papers. Comparison to and commentary on the previous metric-based meta-learning papers present in the work under review is unsatisfactory.\n5. The performed experiments are extremely toyish: the results are not sufficient to support the claims of the paper. This is further exacerbated by the fact that authors need to settle with optimization tricks to learn their TANML model.\n\nI recommend against publication at ICLR. The novelty of the authors' work is limited and the experiments are not convincing. I appreciate that the goal of the work is not to beat SOTA, and rather to introduce and investigate a small change to MAML, but I believe that reducing the scope of the research in this way grants an expectation of extensive experimental evidence which this paper is lacking.\n\nSuggestion:\nTo expand the research to higher-dimensional, few-shot classification tasks, one could hardcode the kernel based on the human understanding of the classes. This way, one could take a problem which Meta-SGD is known to be working with and see if an introduction of a kernel gives a measurable improvement.\n\nTechnical:\n1. \"Training for tasks individually will result in a predictor that overfits to $\\mathcal{X}$, $\\mathcal{Y}$, and generalizes poorly\": It's unclear what \"individually\" means here (is MAML with batch size = 1 training for tasks individually?). It's also far from obvious, in particular in the context of Raghu et al. (Rapid Learning or Feature Reuse?). I encourage authors to avoid using such statements without referring to argumentation behind them.\n2. I am also confused by the sentence \"while there have been studies on deriving features and metrics for understanding the notion of similarity between data sources or datasets, they have not been used in the actual design of meta-algorithms\": isn't the (cited by authors) Achille et al. (2019) one example of such work?\n3. Along the whole paper, \\citep is used, even when \\citet is appropriate. See when to use each one [here](https://www.reddit.com/r/LaTeX/comments/5g9kn1/whats_the_difference_between_cite_citep_and_citep/).\n4. TA*N*ML in Sec. 4.\n5. .. realizations of tasks is reported **in** Table **2**. In Sec. 4.2.", "extra_scores": {}, "extra_text": {}}, {"review_id": "EJv4um4rlZv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a theoretical formulation for meta-learning that uses task similarity based on task gradients, which helps learning in the presence of outlier tasks. The inner loop parameter update is given by linear kernel regression, where the kernel function computes similarity between gradients of different tasks. While the paper includes experiments that outperform MAML and Meta-SGD on estimating randomized linear predictors, and randomized sinusoids with outlier data-points, these are not sufficient to establish the efficacy of the approach. \n\nPros : \n1. This paper proposes a solution to the problem of meta-learning with dissimilar tasks, which is a central problem in meta-learning. The formulated approach is a generalization of MAML and Meta-SGD, as the update direction isn't necessarily the gradient direction, and there is an additional regulation term on meta-parameters in the outer loop. Tasks with similar gradients have a similar update in the meta-learning inner loop. \n\n2. The formulation involving linear kernel regression which enables including the task similarity in the kernel function seems novel and could provide the basis for subsequent work in the field for dealing with dissimilar tasks. The usage of similarity between task gradients to guide updates is similar in spirit to gradient projection techniques used in continual learning to avoid catastrophic forgetting. \n\n3. The included experiments do show much superior performance to MAML and Meta-SGD on the datasets considered, which included tests with outlier data points for sinusoid regression.\n\nCons :\n1. While the experiments show some promise for the method, these on simplistic datasets involving synthetic datasets for estimating randomized linear and sinusoid predictors. Given that the paper discusses MAML and Meta-SGD in some detail for setting up the new method, experiments on the Omniglot and MiniImagenet datasets considered in both those papers would help to better evaluate the proposed approach. \n\n2. The paper has no ablations or analysis for particular parts of their method, such as removing the gradient from the kernel function or removing the regularization term from the outer loop. Thus even on the simplistic datasets considered, it is hard to judge which aspects of the method make it work better. \n\nI am willing to increase my score if the authors include experiments on the datasets mentioned above, and include additional analysis/ablations of their method. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "D_I0_i7WbcD", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=== Summary ===\n\nThe paper proposes a meta-learning method based on a notion of task similarity/dissimilarity. In particular, the paper motivates its proposed method TANML through a generalization of Meta-SGD wherein the learnable parameter-wise learning rate in the inner update of Meta-SGD is replaced by a quadratic pre-conditioner matrix.\n\nThe proposed method TANML closely resembles gradient-based meta-learners in the outer update but replaces the inner update by the matrix-vector product of kernel regression coefficient matrix and task similarity vector based on a kernel function. In that, the kernel function effectively quantifies the similarity of the loss gradients of the different tasks, evaluated at a learnable parameter initialization. Overall, the coefficient matrix can be understood as a look-up matrix in which each row holds the learned parameter vector for one meta-training task, the final adapted parameters are a linear combination of these parameter vectors in, weighted by the kernel between current task and the meta-train tasks.\n\nIn two simple simulated experiments, the paper demonstrates that TANML is able to outperform MAML and Meta-SGD when the meta-train tasks are set up in a pathological way (e.g. by combining two dissimilar clusters of tasks or by adding outlier tasks).\n\n=== Reviewer’s main argument ===\n\nOverall, the idea of incorporating a notion of task-similarity into the meta-learner and the particular proposal to use the kernel between the task loss gradients to quantify such similarity is sound and is a valuable contribution in itself. \n\nUnfortunately, the relationship between Generalized Meta-SGD to TANML is unclear. Usually the connection between linear regression (c.f. Eq. 1 in the paper) and kernel regression (Eq. 2) is established through the particular form of the kernel regression coefficients. However, since the coefficient matrix is (meta-)learned in the paper, it is unclear how TANML relates to Meta-SGD. In fact, TANML seems more like a learned linear combination of task parameters which does not resemble much commonalities with MAML. Overall, the connection to MAML seems a bit set-up/artificial. Discussing the particular relationship between MAML/Meta-SGD and TANML would improve the storyline of the paper. For instance, if TANML is a generalization of MAML, it would be good to state with which particular choices of $\\theta_0$ and $\\Psi$, we can recover MAML.\n\nThe related work section is quite minimalistic. For instance, discussing how TANML is different from e.g. multi-task nonparametric methods (e.g. [1-2]) that also use a kernel between tasks, would better clarify how TANML relates to previous work.\n\nThe numerical experiments are very simple / limited and designed in a pathological way. Thus, it is not surprising that MAML/Meta-SGD perform worse than TANML. How applicable the experimental results are in more realistic meta-learning setups is unclear. Despite the simplicity of the experiments, there is not enough information to properly reproduce the experiment. For instance, how are A and $\\omega$ in experiment 2 sampled, how are the x in experiment 1 sampled and how many data points per task are used in experiment 1? The following would strengthen the experiment section:\n- A real-world use case in which we expect to see a meta-training set with e.g. outliers similar to experiment 2\n- Experiments with real world meta-learning datasets. For real-world & small-scale meta-learning environments for regression, see e.g. [3].\n- An additional meta-learning setup without outliers / clusters of meta-learning tasks. This way one can assess how the proposed method compares to MAML/Meta-SGD in standard setting\n- Adding missing details, e.g. to the appendix, which are necessary for reproducing the experiment.\n\n=== Overall assessment ===\n\nI vote for rejecting the paper. In the current state, the storyline from MAML to TANML provides little value to me as a reader. The proposed algorithm resembles a classical kernel-weighted linear combination of parameters and the pathological toy experiments provide little value for assessing the actual usefulness of TANML in realistic meta-learning scenarios. However, using the kernel between the task loss gradients as a similarity metrics of task is a nice idea and is a valuable contribution. I highly encourage the authors to further improve the paper. Overall, TANML has scientific merit - when introduced with a convincing storyline and properly supported by realistic experiments and relevant baseline comparisons, this would be a clear accept.\n\n=== Minor remarks ===\n\n- Section 2: Eq. 1: move the comma. It should be $[\\theta_0^\\top, \\nabla_{\\theta_0} \\mathcal{L}$ …\n- Section3: Either the $\\Psi$ should be a $T \\times D$ matrix, or there should be no transpose in Eq. 2\n- Section 3 Eq 2: The kernel in the sum should probably be between i and i’, not between i and i.\n- Section 4.1, 2nd paragraph: “... could be ascribed [to] its linear nature …”\n\n\n[1] Bonilla, Edwin V., Kian M. Chai, and Christopher Williams. \"Multi-task Gaussian process prediction.\" Advances in neural information processing systems. 2008.\n\n[2] Micchelli, Charles A., and Massimiliano Pontil. \"Kernels for Multi--task Learning.\" Advances in neural information processing systems. 2005.\n\n[3] Rothfuss, Jonas, Vincent Fortuin, and Andreas Krause. \"PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees.\" arXiv preprint arXiv:2002.05551 (2020).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses a method that incorporates the task-similarity (via task gradients) into the meta-learning. The inner loop update is done by kernel regression with the similarity between gradients of tasks considered, and the outer loop is the gradient update with a particular regularization. Without any doubt, it is a timely and important topic to develop a meta-learning method in the presence of outlier tasks. All reviewers criticized the experiments were done on only simple datasets without any ablation study. Authors revised their manuscript, to include more experiments, and tried to clarify the relation of their method to MetaSGD. Unfortunately, however, even after the author response, reviewers were not convinced that their concerns were resolved. In particular, it was claimed that the revised version still lacks comparisons to previous relevant work. 

Decision: Reject