Meta-Learning with Neural Tangent Kernels
Authors: Yufan Zhou
Zhenyi Wang
Jiayi Xian
Changyou Chen
Jinhui Xu
Venue: ICLR 2021 Poster (2021)
Keywords: meta-learning
neural tangent kernel

Abstract: Model Agnostic Meta-Learning (MAML) has emerged as a standard framework for meta-learning, where a meta-model is learned with the ability of fast adapting to new tasks. However, as a double-looped optimization problem, MAML needs to differentiate through the whole inner-loop optimization path for every outer-loop training step, which may lead to both computational inefficiency and sub-optimal solutions. In this paper, we generalize MAML to allow meta-learning to be defined in function spaces, and propose the first meta-learning paradigm in the Reproducing Kernel Hilbert Space (RKHS) induced by the meta-model's Neural Tangent Kernel (NTK). Within this paradigm, we introduce two meta-learning algorithms in the RKHS, which no longer need a sub-optimal iterative inner-loop adaptation as in the MAML framework. We achieve this goal by 1) replacing the adaptation with a fast-adaptive regularizer in the RKHS; and 2) solving the adaptation analytically based on the NTK theory. Extensive experimental studies demonstrate advantages of our paradigm in both efficiency and quality of solutions compared to related meta-learning algorithms. Another interesting feature of our proposed methods is that they are demonstrated to be more robust to adversarial attacks and out-of-distribution adaptation than popular baselines, as demonstrated in our experiments.

Published as a conference paper at ICLR 2021
META-LEARNING WITH NEURAL TANGENT KERNELS
Yufan Zhou; Zhenyi Wang,, Jiayi Xian, Changyou Chen, Jinhui Xu
Department of Computer Science and Engineering, State University of New York at Buffalo (yufanzho, zhenyiwe wwa, jxian, changyou, jinhui] ) @buffalo.ed
ABSTRACT
Model Agnostic Meta-Learning (MAML) has emerged as a standard framework for meta-learning, where meta-model is learned with the ability of fast adapting to new tasks. However, as double-looped optimization problem, MAML needs to differentiate through the whole inner-loop optimization path for every outer- luter-loop training step, which may lead to both computational inefficiency and sub-optimal solutions. this paper, we generalize MAML to allow meta-learning to be defined in function spaces, and propose the first meta-learning paradigm in the Reproducing Kernel Hilbert Space (RKHS) induced by the meta- meta-msdel's Neural Tangent Kernel (NTK). Within this paradigm, we introduce two meta- meta-learning algorithms in the RKHS, which no longer need a sub-optimal iterative inner-loop adaptation as in the MAML framework. We achieve this goal by 1) replacing the adaptation with a fast-adaptive regularizer in the RKHS; and 2) solving the adaptation analytically based on the NTK theory. Extensive experimental studies demonstrate advantages of our paradigm in both efficiency and quality of solutions compared to related meta-learning algorithms. Another interesting feature of our proposed methods is that they are demonstrated to be more robust to adversarial attacks and out-of- distribution adaptation than popular baselines, as demonstrated in our experiments.
INTRODUCTION
Meta-learning (Schmidhuber, 1987) has made tremendous progresses in the last few years. It aims to learn abstract knowledge from many related tasks so that fast adaption to new and unseen tasks becomes possible. For example, in few-shot learning, meta-learning corresponds to learning meta-model or meta-parameters so that they can fast adapt to new tasks with a limited number of data samples. Among all existing meta-lcarning methods, Model Agnostic Meta-Learning (MAML) (Finn et al. (017) is perhaps one of the most popular and flexible ones, with a number of follow-up up works such as (Nichol et al.] 2018 Finn al.) 2018 Yao et al.] 2019 Khodak et al.) 2019a b. Denevi et al. 2019; Fallah et al.) 20200 Lee et al. 2020 Tripuraneni et al. 2020). MAML adopts double-looped optimization framework, where adaptation is achieved by one or several gradient- descent steps in the inner-loop optimization. Such a framework could lead to some undesirable issues related computational inefficiency and sub-optimal solutions. The main reasons are that 1) is computationally expensive to paooaapre through a stochastic- stochastic-gradient--e -dedeennddddeeennt chain, and 2) is hard to tune the number of adaptation steps in the inner-loop as it can be different for both training and testing. Several previous works tried to address these issues, but they can only alleviate them to certain extents. For example, first order MAML (FOMAML) {Finn et al.) 2017) ignores the high-order terms of the standard MAML, which can speed up the training but may lead to deteriorated performance; MAML with Implicit Gradient (iMAML) (Rajeswaran et al. 2019) directly minimizes the objective of the outer-loop without performing the inner-loop optimization. But it still needs an iterative solver to estimate the meta grrddien..
To better address these issues, propose two algorithms that generalize meta-learning to the Reproducing Kernel Hilbert Space (RKHS) induced by the meta-model's Neural Tangent Kernel (NTK) (Jacot al.) 2018). In this RKHS, instead of using parameter adaptation, we propose to perform an implicit function adaptation. To this end, we introduce two algorithms to avoid explicit
"The first two authors contribute equally. Correspondence to Changyou Chen (changyou @ buffalo.edu 'he research of the first and fifth authors was supported in part by NSF through grants CCF-17164M0 and 199-191049..

---

Published as a conference paper at ICLR 2021
function adaptation: replaces the function adaptation step in the inner-loop with new meta- objective with a fast-adaptive regularizer inspired by MAML; the other solves the adaptation problem analytically based tools from NTK that the meta-objective can be directly evaluated on samples in a closed-form. When restricting the function space to be RKHS, the solutions to the proposed two algorithms become conveniently solvable. In addition, provide theoretical analysis our proposed algorithms in the cases of using fully-connected neural networks and convolutional neural networks as the meta-model. Our analysis shows close connections between our methods and the existing ones, Particularly, we prove that one of our algorithms is closely related to MAML with some high-order terms ignored in the meta-objective function, thus endowing effective optimization. In summary, our main contributions are:
We re-analyze the meta-learning problem and introduce two new algorithms for meta- learning in RKHS. Different from all existing meta-learning algorithms, our proposed methods can be solved efficiently without cumbersome chain-based adaptations.
We conduct theoretically analysis on the proposed algorithms, which suggests that our pro- posed algorithms are closely related the existing MAML methods when fully-connected neural networks and convolutional neural networks are used as the meta-model.
We conduct extensive experiments to validate our algorithms. Experimental results indicate the effectiveness of our proposed methods, through standard few-shot learning, robustness to adversarial attacks and of-disrriiuuiinn adaptation.
2
PRELIMINARIES
2.1 META-LEARNING
Meta-learning can be roughly categorized as black-box adaptation methods (Andrychowicz et al.
2016; Graves et al. 2014, Mishra et al.] 2018).
optimization- based methods (Finn et al. (017)).
non-parametric methods (Vinyals et al. 2016 Snell et al., 2017; Triantafillou et al.] 2020) and
Bayesian meta-learning methods (Finn et al. 2018 Yoon et al. 2018 Ravi Beatson 2019). In
this paper, we focus on the framework of Model Agnostic Meta- Learnin (MAML) (Finn al.
2017). J. which has two key components, meta initialization and fast adaptation. Specifically, MAML solves the meta- wetannee problem through houble-looped optimization procedure. In the inner-loop, MAML runs a specifcc adaptation procedure to transform a meta- perem-errreeeer 9, to a task-specific parameter, { for total of B different tasks. In the outer-loop. MAML minimizes a total loss of Σ L(.e... with respect to meta-parameter 0, where fo... is the model adapted on task m that is typically represented by deep neural network. It is worth noting that in MAML, one potential problem is to compute the meta-gradient Vo Σ L(Se.... ). It requires one to differentiate through the whole inner-loop optimization path, which could be very inefficient.
1 m=
2.2 GRADIENT FLOW
Our proposed method relies on the concept of gradient flow. Generally speaking, gradient flow is a continuous-time version of gradient descent. In the finite-dimensional parameter space, a gradient flow is defined by an ordinary differential equation (ODE), d0' /dt = e F F(0'), with a starting point 0 and function F: Rd + R. Gradient flow is also known as steepest descent curve.
One can generalize gradient flows to infinite-dimensional function spaces. Specifically, given a function space H. a functional F H R, and starting point fd E H. a gradient flow is similarly defined as the solution of dff /dt = V F(f'). This is a curve in the function space H. In this paper, we use notation F(f*) instead of Fz."(") denote the general function derivative of the energy functional FF with respect to function ft (Villani 2008
2.3 THE NEURAL TANGENT KERNEL
Neural Tangent Kernel (NTK) is a recently proposed technique for characterizing the dynamics of a neural network under gradient descent (Jacot et al.) 2018 Arora al. 2019 Lee et al. 2019). NTK allows one to analyze deep neural networks (DNNs) $) in RKHS induced by NTK. One immediate benefit of this is that the loss functional in the function space is often convex, even when it highly non-convex in the parameter space (Jacot et al. 2018) This property allows one to better understand the property of DNNs. Specifically, let Jo
DNN parameterized by The corresponding NTK θ
"Let H he the function space, F he the realization function for neural network defined in Section 3.2 Note even if functional loss (e.g., 1.2 loss) E H + R is convex on H. the composition E Fis in general not.
2

---

Published as a conference paper at ICLR 2021
(x1) T where X1, X2 are two data points. In our paper, we
is defined as: (x1, X2)
d0
30
will define meta-learning on an RKHS induced by such kernel.
META-LEARNING
RKHS
We first define the meta-learning problem in a general function space, and then restrict the function space to be an RKHS, where two frameworks will be proposed to make meta-learning feasible in RKHS, along with some theoretical analysis. For simplicity, in the following we will hide the superscript time unless necessary, egg,, when the analysis involves time-changing.
3.1 META-LEARNING IN FUNCTION SPACE
Given a function space H. distribution of tasks P(T), and loss function L, the goal of meta- learning is to find a meta function f E H. so that it performs well after simple adaptation on specific task. Let Dtr and Dinst be the training and testing sets, respectively, sampled from data distribution of task Tm The meta-learning problem on function space H. is defined as:
f arg migmin&ee E(), with {f) = ET. [ (Adapt(ff Dir),, D'est JEH
(1)
where Adapt denotes some adaptation algorithms, e.g., several steps of gradient descent; & H + R called energy functional, which is used evaluate the model represented by the function f.
In theory, solving equation Tis equivalent to solving the gradient flow equation df² /dd -V (.E(())
However, solving the gradient flow equation generally infeasible, since i) it is hard directly apply optimization methods in function space and ii) the energy functional & contains an adaptation algorithm Adapt, making the functional gradient infeasible. Thus, better way is to design a special energy functional so that it can be directly optimized without running the specific adaptation algorithm. In the following, we first specify the functional meta-learning problem in RKHS, and then propose two methods to derive efficient solutions for the problem.
3.2 META- IN RKHS
We consider a function f that is parameterized by 0 E IK.", denoted as fe. with P being the number of parameters. Define a realization function F R2. + 7 that maps parameters to a function. With these, we can then define energy function in the parameter space E E F Rº + R with being the composition operator. Consequently, with an initialized 6º, we can define the gradient flow of E(0') in parameter space d0' /dt V oz E(O'). In the following, equivalence between the gradient flow in RKHS and the gradient flow in the parameter space. We then propose two algorithms for meta-learnig the RKHS induced by NTK.
first establish an
Theorem Let H be an RKHS induced by the NTK O of fo- With fo = fgo, the gradient flow of E(ft) coincides with the fincction evolution of fer driven by the gradient flow of (0OO).
The proof of Theorem D relies on the property of NTK (Jacot et al. 2018). and is provided in the Appendix. Theorem T serves foundation of our proposed methods, which indicates that solving the meta-learning problem in RKHS can be done by some appropriate manipulations In the following, we describe two different approaches termed Meta- -KHa- and Reta-RKHS... -II, respectively.
3.3 META-RKHS-I: META-LEARNING IN RKHS WITHOUT ADAPTATION
Our goal is to design energy functional that has no adaptation component, but is capable of achieving fast adaptation. For this purpose, we first introduce two definitions: empirical loss function C(fe, Dm) and expected loss function L(fe). Let Dm {xm, is V Wm. In be set containing the data of regression task Tm. The empirical loss function L(Jo, D.n) and the expected loss function L.n (fo) can be defined as:
C(Se, Dm))
Σ (xxm,)) -ymell 1| L.n(fo) = Exmm ==1
[ (115(xmm)))) ym 112
[1
2n
3

---

Published as a conference paper at ICLR 2021
Our idea is to define regularized functional such that it endows the ability of fast adaptation in RKHS. Our solution based on some property of the standard MAML. We start from analyzing the meta-objective of MAML with a k-step gradient-descent adaptation, L.e, applying k gradient-descent steps in the inner-loop. The objective can be formulated as
R-
0 arg min BT.. [C(So, Dimstt] with & 0 -
Ve, ((o, (
==
where a the learning rate of the inner-loop, 0o = 0, and 0st1 0: aVo [(o; D Taylor expansion, we have
By
TT [[ Di ET...
[(So, Dia
Ve, [(Jo, V Dinatyr
(2)
Since Drr and Diest come from the same distribution, equation an unbiased estimator of mi
k-1
M = 1Cm (fo)
Σ B.l. where 83 (of
(3)
==
We focus on the case of 1, which MM BT., [Cmm - ( The first term on the RHS is the traditional multi-task loss evaluated at 0 for all tasks. The second term corresponds the negative gradient norm; minimizing means choosing a O with the maximum gradient norm. Intuitively, when 0 is not a stationary point of task, one should choose the steepest descent direction to reduce the loss maximally for a specific task, thus leading to fast adaptation.
aBT
The above understanding suggests us to propose the following regularized energy functional, Eg, for meta-learning in the RKHS induced with the NTK for fast function adaptation:
E(a, fo) = ET., (fo) - (mm))
where || 11x denotes the functional norm in H. and a is a hyper-parameter. The above objective is inspired by the Taylor expansion of the MAML objective, but is defined in the RKHS induced by the NTK. Its connection with MAML and some functional-space properties will be discussed later.
Solving the Function Optimization Problem To minimize equation 4 first derive Theorem to reduce the function optimization problem to a parameter optimization problem.
Theorem Let Sa be neural network with parameter O and H be the RKHS induced by the NTK θ of Ja. Then, the following are equivalent
E(a, fo) = =M1, and
Theorem is crucial to our approach as it indicates that solving problem equation[4ji no more difficult than the original parameter-based MAML, although it only considers one-step adaptation case. Next, we will show that multi-step adaptation the parameter space can also be apll-aaproximmated by our objective equation 4 but with scaled regularized parameter a. In the following, we consider the squared loss L. The case with the cross-entropy loss is discussed the Appendix. We assume that So is parameterized by either fully-connected or convolutional neural networks, and only consider the impact of number of hidden layers our theoretical results.
Theorem Let So be fully-connected neural network with L hidden layers and ReLU activation function, $1...., SL+1 be the spectral norm of the weight matrices, s max, Sh, and a be the learning min(1 (aan(1(LLs) (IsLL= L-111((+1) and min(s s),
rate of gradient descent. If < (ggr) with q
then the following holds
[M E(ka, S so (
Theorem Let fe be a convolutional neural network with L convolutional layers and I fully- connected layers and with ReLU activation function, and dg be the input dimension. Denote by Wh the parameter vector of the convolutional layer for S T. I, and the weight matrices of the fully connected layers for
+1 ||
means both the spectral norm of matrix
For ease of our later notation, write the gradient Vo, C. (thus the parameter as well) row vector.
4

---

Published as a conference paper at ICLR 2021
and the Euclidean norm of a vector. Define sh. dd [h if = L and | 1y11 2|2 if - ++ 1 1 h I + Let $ = max,, Sh, and a be the learning rate of gradient descent. If O(qr) with q min(L/(Lss), 1-1(+1)) and min(s"",ss) the following holds
MM 1 E(ka, fo)l s (
The above Theorems indicate that, for a meta-model with fully-con fully-connected and convolutional layers, the proposed
can be efficient approximation of MAML with bounded error.
Comparisons with Reptile and MAML Similar to Reptile and MAML, the testing stage of Meta- RKHS-I also requires gradient- based adaptation on meta-lest tasks. By Theorem [ we known that gradient flow of an energy functional can be approximated by gradient descent in a parameter space. Reptile with 1-step adaptation (Nichol et al., (018) is equivalent to the approximation of the gradient flow of {a, fe) with a O, which does not include the fast-adaptation regularization as in our method. For fairer comparison on the efficiency, we will discuss the computational complexity later.
From the equivalent parameter-optiiizzaton form indicated in Theorem 2 we know that our energy functional E(a, fo) is closely related to MAMI.. However, with this form, our method does not need the explicit adaptation steps in training (i.e., the inner-loop of MAML., leading to a simpler optimization problem. We will show that our proposed method leads to better results.
3.4 META-RKHS II: META- LEARNING IN RKHS WITH CLOSED-FOREE ADAPTATION
In this section, we present our second solution for meta-learning in RKHS by deriving a closed-form adaptation function, i.e., focus case where Adapt( [, D is analytically solvable using the theory of NTK. Specifically, we are given loss function L, tasks Tm with randomly split training set Dír ix i= i== and testing set Diest Let 0t and fm.ee denote the parameters and the corresponding function at time t adapted by task Tm from the meta parameter @ and meta function fe. respectively. From the NTK theory (Jacot et al. 2018 Arora et al. 2019 can write the ninction/parameter evolution as:
Lee et al. 2019). we
d0 dt Vo. C(f D and df d0 Of
OC(fmm D "2 Σ
Oxx"r
dt dt
00 m
afm, (xx"r,))
The above differential equation corresponds to the adaptation step, i.e., how to adapt the meta parameter/function for task m. By the NTK theory, we can show that this admits closed-form solutions. In our meta-learning settings, this indicates that no explicit udaptation steps are necessary.
the To see why this is the case, we first investigate the regression case, where the loss function squared loss. Let x E Diest be a test data point. As shown in Arora et al. (2019); Lee et al. (2019). with large enough neural network we can safely assume that NTK will not change too much during the training. In this case, we can have closed-fome solution for fm.e
(x) fx(x) + Hx,, X
X ( (e-III mi
(Jo -
(5)
where is the matrix exponential map, which can be approximated by Padé approximation (M.Arioli et al. 1996). H H m. X is an X kernel matrix with its (i,j) element being @(xm.i: Xm.),
H [x, X is 1 vector with its i-th element being O Xm.s). fo (x(t)) E R² is the predictions of all training data at the initialization, and y E Rn is the target value of the training data. Specifically, at time = oo, we have
f (x) fo(x) H(r X
X (X" fo((
(6)
The above results allow to directly define an energy functional by substituting Adapt DDM) in equation I with its closed-form solution fin_e. g. In other words, our new energy functional is
E(t, fo) ET. [Cm(
where fh is defined in equation 5 and Cm is the expectation of C c(1 (f Diast) For classification problems, we follow the same strategy as in in/oroa et al. (2019) to extend regression to classification. Mores details can be found in the Appendix, including the algorithm in Appendix A
5

---

Published as a conference paper at ICLR 2021
Table 1: Running time comparison per iteration with C1 dpp + Lp² and C2 dxp Ld
FOMAML
Reptile
Meta-RKHISI Meta-RKHs-II
Fully-connected O + +))C))
O(nkCi) O(nkC2)
O(1C1)
O(nC, +n²) O +
Convolutional
(n(k + 1)C2)
O(nC2)
On Potential Robustness of Meta-RKHS-II Our extensive empirical studies show that Meta- RKHS-II is more robust model than related baselines. We provide an intuitive explanation on the potential robustness of Meta-RKHS-IT as we find current theories of both robustness machine learning and NTK are insufficient for formal explanation. Our explanation is based some properties of both the meta-learning framework and NTK: 1) Strong initialization (meta model); For NTK to generalize well, we argue that it is necessary start the model with good initialization. This is automatically achieved in our meta-learning setting, where the meta model serves as the initialization for NTK predictions. Actually, this has been supported by recent research (Fort et al. 2020), which shows that there is a chaotic stage in the NTK prediction with finite neural networks, and the NTK regime can be reachable with a good initialization. 2) Low complex classification boundary: It is known that NTK is linear model in the NTK regime. Intuitively, generating adversarial samples with a lower complex model should be relatively harder because there is less data in the vicinity of the decision boundary compared to a more complex model, making the probability of the model being attacked smaller. Thus we argue that our model can be more robust than standard meta learning models. 3) Our NTK-based model is robust enough to adapt with different time steps. And these finite time steps can be more robust to adversarial attacks than that of the infinite-time limit partly due to the complexity of back- propppropaggting gradients. We note each of the individual factors might not be enough to ensure robustness. Instead, we argue it is the combination effect of these factors that lead robustness of our model. Formal analysis is out of the scope of this paper and left for future work.
Connection with Meta-RKHS-I The proposed two methods choose different strategies to avoid explicit adaptation in meta- meta-learning which seem to be two very different algorithms. We prove below theorem, which indicates that the difference of the underlying gradient flows of the two algorithms indeed increases w.r.t. both T and the depth I. of DNN (we only consider impacts of T and T.).
Theorem 5
Let fo be a neural network with L hidden layers, with each layer being either fully- connected or convolutional. Assume that [1 < co. Then, error(T) ( Se) - E(T, So)l is a non-decreasing function T. Furthermore, for arbitrary T > 0 we have error(T) (T) < (TTL+3))
Actually, Meta-RKHS II implicitly contains term of functional gradient norm because E(T,
Eyn (fo - _ || Voi oo fm Cmm dt The difference compared to Meta-RKHS-I mainly
comes from the fact that Meta-RKHS, I can be regarded as an approximation of time-discrete adaptation, while Meta-RKHS-II is based on time-continuous adaptation. In our experiments, we observe that RetS- is as fast as FOMAMI,, which means that it is more computationally efficient than the standard MAML. Mcanwhile Meta-RKHS-II is the more robust model in tasks of adversarial attack and -uist--dist adaptation.
Connection with iMAML Our proposed method is similar to the iMAML algorithm (Finn & Levine 2019) in the sense that both methods try to solve meta- meta-learning without executing the optimization path. Different from iMAML, which still relies on an iterative solver, our method only needs to solve a simpler optimization problem due to the closed-form adaptation.
3.5 TIME COMPLEXITY ANALYSIS
We compare the time complexity of our proposed methods with other first-order meta-learning methods. Without loss of generality, we analyze the complexity in the case of a L-layer MLP or L-layer convolutional neural networks. Recall that d, is the input dimension. Assume each layer has width (filter number) Op)). Let n be the data batch size, k the adaptation steps of inner-loop optimization. We summarize the time complexity in Table [ where we simply assume the complexity of multiplying matrices with sizes a x and b x c to be (abc). Note in the meta- meta-learning setting, n is typically small, indicating the efficiency of our proposed methods.

---

Published as a conference paper at ICLR 2021
Sampled
Sampaied bartue
Dabre
Balbre
ARW
-14
(a) Random Initialized
(b) Meta-RKHS-I
(c) Meta-RKHS-II
Figure Performance of random initialized network and our methods. The models before/after adaptation are shown in dotted/dashed lines, samples used for adaptation are also shown in the figure.
EXPERIMENTS
We conduct a set of experiments to evaluate the effectiveness of our proposed methods, including sine wave regression toy experiment, few-shot classification, robustness to adversarial attacks, out-of-distribution generalization and ablation study. Due to space limit, more results are provided in the Appendix. We compare our models with related baselines including MAML. (Finn al.) (017), the first order MAML (FOMAML) (Finn et al. (017). Reptile (Nichol et al., 2018 and iMAML (Rajeswaran al. 2019 Results are reported as mean and variance over three independent runs.
4.1 REGRESSION
Following Finn et al. (017); Nichol et (2018), we first test our proposed methods on the dimensional sine wave regression problem. This problem is instructive, where model is trained on many different sine waves with different amplitudes and phases, and tested by adapting the trained model new sine waves with only a few data points using fixed number of gradient-descen steps. Following Finn et al. (2017); Nichol et al. (2018), we use a fully-connecd neural network with hidden layers and the ReLU activation function. The results are shown in Figure
4.2 FEW-SHOT IMAGE CLASSIFICATION
For this experiment, we choose two popular datasets adopted for meta-learning: Mini-ImageNet while the
and FC-100 (Oreshkin et al. 2018). The cross-entropy loss is adopted for squared loss is used for Meta- SeetaRKKHS-IL following Arora al. (2019): Novak al. (2019). Similar to Finn et al. (0117). the model architecture is set to be a four-layer convolutional neural network with
ReLU activation. The filter number is set to 32. The Adam optimizer (Kingma & Bal 2015) is used to minimize the energy functional. Meta batch size is set to be 16 and learning rates are set to be 0.01 for Meta-RKHS-II.
The results are shown in Table Note the results of Reptile is different from those in Nichol et al. (2018), because we re-evaluate under the same set- ting as Finn et al. (0017). i.e., 10 steps of adaptation is applied during testing. Our results of iMAML is based the implementation of [Spigler((20) (2019). It observed that our proposed methods achieve better accuracy than different baselines. Interestingly, our Meta-RKHS: performs better than FOMAML (this also the case in other experiments), although they share a similar objective. We conjecture the reason is because our Mcta-RKHS-I restricts the function to be in an RKHS, making the functional space smaller thus easier to optimize compared to the unrestricted version of FOMAML. In terms of our two algorithms, there is not always a winner on all the tasks. We note that Meta-RKHS-I more efficient in training. However, we show below that Meta-RKHS-II is better in terms of robustness to adversarial attacks and out-of-distribttion generalization.
Table 2: Few-shot classification results on Mini- and FC-100.
MINI-IMANHENET
FC-100
03:11
1.96%
FOMAMI
1.75%
63.15
IMAML
61.89
0.9595
1.70%
RKHS-I
51.10 1.82%
1.90%%
61.47
0.86%
RKHS-II
0..04
05,
41.20 2.17%
4.3 ROBUSTNESS TO ADVERSARIAL ATTACKS
We now compare the adversarial robustness of our methods with other popular baselines. We adopt both white-box and black-box attacks in this experiment. For the white-box attacks, we adopt strong attacks including the PGD Attack (Madry al. 2017), BPDA attack (Athalye et al. 2018) and

---

Published as a conference paper at ICLR 2021
0.2
0.4
0.2
13
0.1
82
0.0
Quertes
Quertec
Sepdit
Vea 3K-51
MINL
Mets-R015
-... Net Ago
2103 PO
Hers rin0 PG?
MIN.
FOMMML
Figure 2: Black-box attack on Mini-ImageNet and FC-100. Mini-ImageNet 5-way 1-shot (left), FC-100 5-way 1-shot (middle) and Mini- Mini-ImageNet 5-way 5-shot (right).
L 1ifti nom BPD4 Artack
L Iithi by nom: BPD4 Attack
0.40
0:55
reggl
Hirp 41-2112
0:10
8:5
5055 1025 19255 23055 25253 20255 35055 ax055 45055
5255
16255 19:25 2305s
25253 20255 35055 46055 45055 Wited mage Ruds
Figure 3: BPDA attack on Mini- Mini-ImageNet 5-way 5-shot (left) and FC-100 5-way 5-shot (right).
iririty
iirinil
FORMAI
POPARL
13
13
Figure 4: SPSA attack on Mini-ImageNet 5-way 5-shot (left) and FC-100 5-way 5-shot (right).
2018 For PGD attack, we use foo norm and compare the results on Mini-imagenet and FC- 100. We compare the robust accuracy with different magnitude with 20-step attack with step size of 2 255.. For BPDA attack, we apply median smoothing, JPEGFilter and BitSqueezing as input transformation adapted from (Guo et al. 2018) as defense strategies. For SPSA attack, follow (Uesato et al. 2018) and the Adam learning rate 0.01, perturbation size 5 = 0.01. For Black-box attack, we adopt the strong query efficient attack method (Guo et al. 2019 Follow the setting Guo al. (2019), we use a fixed step size of 0.2.
SPSA attack (Uesato et al.
We consider both finite-time and infinite-time adaptation in this experiment. For finite-time adaptation, approximate the matrix exponential
the Padé approximation with P =Q are considered (Butcher & Chipman] 1992). We use Meta-RKKH--IL:1100 t100 PQ2 to denote methods using finite time t 100, P = Q = 1 or P Q 2, respectively. We observe other finite time t makes similar predictions, thus we only consider t = 100. The indicate the robustness of our Meta-RKHS-II. In fact, the gaps are significantly large, making it the only useful robust model in the adversarial-attact setting. Our Meta-RKHŠ-I is not as robust as Meta-RKHS-II, but still slightly outperforms other baselines. Regarding the white-box attack, results in Figure 3 and 5 again show that our proposed Meta-RKHS' II is significantly more robust than baselines under the three strong attacks. It is also interesting
and P Q
and MSta-RKHS-
results from the black-box attack in indicatee
that our Meta-RKHS- performs slightly better than Meta-RKHSS II in some rare cases, e.g., in the Mini- Imni-ImmeeNet 5-way 1-shot case when the attack magnitude is not too small. More results are presented in the Appendix.
8

---

Published as a conference paper at ICLR 2021
Limtnnty norm Altack
Infinity norm Attack
Infrnity
0.5
0.150
0.30
0.125
0.25
0.3
0.100
0.2
0.075
0.15
0.10
0.1
0.025
0.0
0.000
0.0
1255
2/25
3255
4/355
1/239
82355
92255
10225
1/285
2/259
3233
4235
6230
72339
8235
9233
10235
1/235
2755
31355
4235
57359
6255
7255
92255
10289
attack
magrib.de
Repile
Neta RceSal 0.00
Figure 5: fo norm PGD attack on Mini-ImageNet and FC-100. Mini- Mini-ImageNet 5-way 5-shot (left), Mini-ImageNet 5-way I-shot (middle) and FC-100 5-way 5-shot (right).
Table 4: Meta-RKHS-II with different time L.
TIME
t= 0.1
t=1
2 10
1= 100
( 00
MINI-MAGENET
WWY SHoT
49.67 1: 2.23%
48.27
2.23%
50.53
2.09%
49.13
2.1%
48.70
28%
5 WAY 5 SHOTs
61.51 + 0.93%
61.28
= 0.98%
65.
0.91%
61.21 = 1.06%
61.95 +: 0.96%
FC-100
5 WAY 5 SHOTS
WAY SHoT
36.50
48.35 +: 1.02%
2.10%
49.79 = 1.00%
2.32%
51.36
41:20
0.96%
2.17%
48 59 1.09%%
2.21%
37.60
49.48 + 0.996%
2.13%
4.4
OUT-OF-DISTRIBUTION GENERALIZATION
We adopt similar strategy in LLe et al. 2020) to test a model's ability of generalizing to out-of- distribution datasets. In this setting, the state of arts are achieved by Bayesian TAML (Lee et al.. 2020). Different from their setting that considers any-shot learning with maximum number of examples for each class being as large as 50, we only focus on the standard I or 5 shot learning. We thus modify their code to accommodate our standard setting. The CUB (Wah et al. (011T) and VGG Flower Nilsback & Zisserman (2008) are fine- amn-ggrinedd datasets used in this experiment, where all images are resized to 84 X 84. We follow Lee et al. (2020) to split these datasets into meta praining/validations
sets. We first train all the methods on Mini-ImageNet or FC-100 datasets, then conduet meta-testing on CUB and VGG Flower datasets. The results are shown in Table Again, our methods achieve the best results, outperforming the state-of-art method with our Meta-RKHSS indicating the robustness of our proposed methods. More results are presented in the Appendix.
4.5 ABLATION STUDY
Table Meta testing on different with model trained on Mini- Mini-ImageNet.
datasets
We conduct several ablation studies, including: comparing Reptile with Meta-RKHS-I under different adap- tation steps (results shown in the Ap-
WAY SHOT
WWY
5 SHOT
ALDORITHM
CUB
VGG FLOWER
CUB
VGG FLOWER
#1.52%
7.70%
0.94%
FOMAML
35.52 1.00%
53.86
1.00%
52,02
0.71%
68.55
1.16%
REFTIE
35/61
1.58%
53.57
1.59%
51
0.50%%
71.62-1
1..55
pendix), testing the impact of choos- ing different time / in Meta-RKHS-II (results shown in Table 4 and the im-
IMAME
40.55
00.61%
54.97
0.80%
46.31
2.00%
60.07
11.11%
BAYESIAN TAMUSSOTA)
41.37
0.50%5
58.56 1 0.00%
61.79
0..6%%
77.05 1. 0.40%
META-RKHS-
30.73
1.200%
54.79+111 61%
54.18- 0.79%
72.70 72.76+ 1.08%
MITA-RKHS-II
45.96
0.87%
60.80
1.0295
65.21
0.6%%
0.49%
pact of network architecture with dif-
ferent number of CNN feature channels (results shown in the Appendix). is interesting to see that a finite-time (around = 10) achieves the best accuracy, although the infinite-time case guarantees a stationary point. This indicates that a stationary point achieved by limited training data in the adaptation step is not always the best choice, because the limited training data might easily overfit the model, thus achieving worse test results.
5 CONCLUSION
We develop meta-learning in RKHS, and propose two practical algorithms allowing efficient adap- tation in the function space by avoiding some complicated adaptations in traditional methods. We show connections between our proposed methods and existing ones. Extensive experiments suggest that our methods are more effective, achieve better generalization and are more robust against adversarial attacks and out-of of-distribution generalization, compared to popular strong baselines.

---

Published as a conference paper at ICLR 2021
REFERENCES
Zeyuan Allen-Zhie Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via wvee-parameterization. volume Proceedings of Machine Learning Research, pp. 242-252, Long Beach, California, USA, 09-15 15 Jun 2019. PMLR. URL http::tt://///reeediii
mlr
press/v97/al zhul 9a html
Marcin Andrychowicz, Misha Denil, Sergio Gómez, Matthew W Hoffmmnn, David Plau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning learn by gradient descent by gradient descent. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, pp. 3981-3989. 2016.
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances Neural Information Processing Systems, 2019.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning, 2018.
C. Butcher and H. Chipman. Generalized padé approximations to the exponential function. BIT Numerical Mathematics, 32:1 18-110, 1992.
Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Learning-to-ler -o-lerm stochas- tic gradient descent with biased regularization. In http::/arrtiv.rrraa///99.... 103999 2019.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of prudient-basee model-agnostic meta-learning algorithms. In International Conference on Artificial Intelligence and Statistics, 2020.
Chelsea Finn and Sergey Levine. Meta-learning: from few-shot learning to rapid reinforcement learning. In ICML 2019 Meta-Learning Tutorial, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Advances in Neural Information Processing Systems. 2018.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy. and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. In Advances Neural Information Processing Systems, 2020.
Alex Graves, Greg Wayne, and Ivo Danihelka. https://ariioorrlds//111.... 2014.
Neural
turing machines. In
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. In International Conference on Learning Representations, 2018.
Chuan Guo, Jacob R. Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Q. Weinberger. Simple black-box adversarial attacks. In International Conference Machine Learning, 2019.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pp. 8571-8580, 2018.
Mikhail Khodak, Maria-Florina. Balcan, and Ameet Talwalkar, Provable guarantees for gradient-based meta-learning. In International Conference on Machine Learning, 2019a.
Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar, Adaptive gradient- based meta-learning methods. In Advances in Neural Information Processing Systems, 2019b.
10

---

Published as a conference paper at ICLR 2021
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Hae Beom Lee, Hayeon Lee, Donghyun Na, Sachoon Kim, Minseop Park, Eunho Yang, and Sung Ju Hwang. Learning to balance: Bayesian meta- meta-learning for imbalanced and outoo--distribution tasks. In International Conference on Learning Representations, 2020.
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems 32, pp. 8572-8558 Curran Associates, Inc., 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ArXiv, 0tss/170..060 2017.
M.Arioli, B.Codenotti, and C.Fassino. The Padé method for computing the matrix exponential.
Linear Algebra and its Applications, June 1996.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In International Conference on Learning Representations, 2018.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. In https://arxiv.orr/db///133..... 180.00999 2018.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Sixth Indian Conference on Computer Vision, Graphics and Image Processing. 2008.
Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Jiri Hron, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. In International Conference on Learning Representations, 2019.
Boris N. Oreshkin, Pau Rodriguez, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, 2018.
Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine, Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems. 2019.
Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning. International Conference on Learning Representations, 2019.
Filippo Santambrogio. Fuclidean, Metric, and Wasserstein gradient flows: an overview, 2016.
Jurgen Schmidhuber. Evolutionary principles in self-referential learning. Diploma thesis, Technische Universitat Munchen, Germany, 14 May 1987.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
Giacomo Spigler. Meta-learnt priors slow down catastrophic forgetting in neural networks. arXiv e-prints, art. 1rXivv1909.041700 Sep 2019.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- dataset: dataset of datasets for learning to learn from few examples. International Conference on Learning Representations, 2020.
Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning of linear representations. In https:/aarivvorggass//00./// 16844 2020.
Jonathan Uesato, Brendan O' O'Donoghue, Aaron van den Oord, and Pushmeet Kohli. Adversarial risk and the dangers of evaluating against weak attacks, 2018
Villani. Optimal transport Old and new, volume 338, pp. xxii+973. 01 2008. doi: 10.1007/
778-3-540-710500
11

---

Published as a conference paper at ICLR 2021
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In https://arxii.org/ffff/// 04080.pdff 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. In Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hierarchically structured meta-learning. In International Conference on Machine Learning, 2019.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-leaming. In Advances in Neural Information Processing Systems. 2018.
12

---

Published as a conference paper at ICLR 2021
A
ALGORITHMS
Our proposed algorithms for meta-learning in the RKHS are summarized Algorithm
Algorithm 1 Meta- Meta-Learning in RKHS
Require: p(T): distribution over tasks, randomly initialized neural network parameters O.
while not done do
Sample a batch of tasks (Tm Jm=1 P(T)
for all T. do
Sample a batch of data points Dm or Sample two batches of data points Dir Diest, end for
Evaluate the energy functional by equation 4 with {2m |m or Evaluate the energy functional by equation with {Di, D mm= Minimize the energy functional w.r.t O.
end while
B PROOF OF THEOREM 1
Theorem If fe a neural network with parameter E R and H. is the Reproducing Kernel Hilbert Space (RKHS) induced by O where @ is the Neural Tangent Kernel (NTK) of fo. then with initialization f = fe0, the gradient flow of ((f)) coincides with the function evolution of Jo induced by the gradient flow of E(
Proof Without loss of generality, we can rewrite E() = ET.,, [B m) (C([((x): m))) with some function ...,)..
For neural network fo with parameter 0 E RP, the gradient flow of E in R is
d0
dt
We have
d0'
V o
dt
ETy (x vn)
(V C(fo (Xm),
(x θ f0((2)m) 00
- ETT ((xmms )
0.8(xm)
We know that the dynamics of fer is
dfo- dl' 0.
dt dt a
(xm)) Of (xm)
fo
--ETm
E x m
0f (xm)
00
80
afo: (Xm)).
0. (x)
0f
}}
Eyn
E E(xxm
Ofo (Xm)
00
00
afoo (X5)).
ET
E (xc. m
(8)
of (Xm)
where O is the Neural Tangent Kernel of neural network for (Jacot et al. 2018
If Ht is the Reproducing Kernel Hilbert Space induced by kernel O' and Vx... H + R is the
evaluation functional X which is defined
Vx (J) f(xm):
13

---

Published as a conference paper at ICLR 2021
then for an arbitrary function g and small perturbation e, we have
V/x. (f Eg) Vxm (f)
(V,Vx.. Xm (f).) = lim
f(xm) + +g((xm) f f(xm)
(VyVx.. (f).g) lim
c-0
Vy.. (f).g) g(xm)
V Vxm (f).g) (0'mm .),9)
y Vx,, (f) O 3'(xm²))
fx(m) O' (Xm:")
With an initial function fo for H. the gradient flow of in dʃc
H is
VV
dt
We have
df
EET, (E (xm m
V FeC(( (xm),
dt
OC1 (x(()((mm)))) (om)),m..
-EE
I. (xm m
ye (Xx))
(Xm(
TO (Xm)). O (xm=
-EE
E (x ym
(9)
(xm)
We complete the proof by comparing equation 8 and equation 9
C PROOF OF THEOREM 2
Theorem If So is a neural network with parameter and is the Reproducing Kernel Hilbert Space (RKHS) induced by θ where θ is the Neural Tangent Kernel (NTK) of fe. then
Mi E(,, fo), and cllVoLLm([o)] v Lm (J)) (Jo)
Proof Without loss of generality, we rewrite L. (fo) Ex,, In regression task, we have
N [C(fo(xm)) vm)]
1
C(Jo(Xm)) -111100000000 ym
In classification task, we have
(xfo(x,)) ). ym)) = yy log(,f((Xm,))
where log is element-wise logarithm operation.
Wo (fo)ll
VoLm (So) "
VolExm.y.
[C(fo (xm. ymn)l V Volxm.N. ...
[C(fe(Xm): Fmn))
TaC (x((f()m)) (xm)
ym) (xmm)
[x)((mm) a0 (fo(X)).
Exm
Exm on
0fo( (xm)
00
00
2x(()))
OC (C(ff(xm)))))) (x))))))))
(fo(xm) 0fo(x(m) OC *C(ff((x)
-Exm.ym
Exe, W'nn
@fe(Xm)
00
00
0fe (xm)
@C(fo (xm)) )m
ac (
-Exm
Ryx S'm
(x xmm)
(xm)
(x(m))
14

---

Published as a conference paper at ICLR 2021
Exxm Mm
(f (Xm)), y.
O (xmm>) )
Ex,, v'
@C((o((X)) y'm)
O (x
(0((xm)
0So(x'm)
(C (Xm), y
TaC(Jo (x/) y'm)
Px on
V 1oJo
V (1
(fo(Xm))
0fo(xmm)
(V (Jo):
5o So (fo))
|| so Cm (f (fo)
where . inner product in Reproducing Kernel Hilbert Space (RKHS) H. the above equations, the definition of Neural Tangent Kernel (NTK), the property of inner product in RKHS, the definition of evaluation functional and its gradient in RKHS.
Recall that
E(a, fo) ETm [m - a C. (Jo)l
and
h-1
M₁ = BT Lm(ff) Σ
3
where B: V. (fo.) VoLm (So) and 0o 0, 0,0 0, Ve C(So,, Dir). The result is straightforward now.
D PROOF OF THEOREM
The proof techniques we use are similar to some previous works such as (Arora et al.) 2019 Allen-Zhu al., (011)). We summaries some of the differences. Different from previous works that typically assume a neural network is Gaussian initialized, we do not have such an assumption as we are trying to learn good meta-inie vetaainittallizatonn in the meta-learning setting. Previous works try to investigate the behavior of models during training, while we focus on revealing the connection between different meta-learning algorithms. Previous work focuses single-task regression/clasiifcatiin problems, while we focus on meta-learning problem.
Theorem Let be fully-connected neural network with T. hidden layers and ReLU activation function, $1.-., $L+1 be the spectral norm of the weight matrices, 8 = max, 8h, and a be the learning rate of gradient descent. If I O(gr) with min(1/(Ls")) L- 1/(1+1)) and r = min(s 1 s),
then the following holds
E(kaa, So) M 50 (
Proof We first prove the case of is 2, ie. applying two-step gradient descent adaptation in MAML.
We need prove the following theorem first.
Theorem 6 Let fo be fully-connected neural network with L hidden layers, and be data sample. Represent the neural network by fe(x) = (x (ooo(xW))...l)) W
where W1 ...., WL+1 denote the weight matrices, and is the ReLU activation function. Let $1,.., S1+1 be the spectral norm of weight matrices, and s = max, Sh. Let a be the learning rate of gradient descent, and f((x) be the resulting value after one step of gradient descent, and || be the Frobenius norm.
a O(gs-h, "), where q min(1/(Ls"",, -- then
of
0x)
0
00
00
SVL+1
Remark Theorem states that for a neural network with hidden layers. if the learning rate of gradient descent is bounded, then the of derivative w.r.t all the parameters will not change
15

---

Published as a conference paper at ICLR 2021
much, although there are (Lm²²) parameters, where denotes the maximum width of hidden layers. We use row vector instead of column vector for consistency, while it does not affect our results.
For simplicity, we will write (x) as gh. The bias terms in the neural network are introduced by adding an additional coordinate thus omitted in Theorem 6 Without loss of generality, we can assume | || I 1, which can be done by data normalization in pre- pre-processing.
Let g0(x) (x) o(o W D..,Wh-1)WW' be the activation at 4hhh hidden layer and g" x, g. ++1 [x().. Define diagonal matrices D where (1.1) 1(g" 2 0} and
ba -
Id (W
if = L ++
otherwise
where Id, is dy dy identity matrix. We first prove the following Lemma.
Lemma Given neural network stated in Theorem 6 let || ||2 denote the spectral norm,
AW" Wh Wh denote some perturbation on weight matrices, g"(x) (x) denote the resulting value
after perturbation, and AsS (x) g²(( (x). If s 2 1 and A W 1l2 S // L) for all h, then
1
11 s 0
Ls-h-
If s < 1 and || S 0(g) for all where q = min(1 (-V(Lt)))( and r max(q. 8).
then
||ag 0((h 9) {
0 k4t Oll h(-D))
f1/(1.8)) S 1--1/(141)
if1/(L8") > L 11((441)
Proof Proof of Lemma based on induction.
1
We first prove the case of s 1. Note that 9d xx thus Ag =0 0 0( For Ag', we have
L always holds.
|| = (x W - ox
W xW
due the property of ReLU activation
IllA l|
0 Lst
Thus, the hypothesis holds for Ag
Now, assume that the hypothesis holds for Ag" then have
Ag 1| [ - " I|l
< Wht g¹ wh ||. due the property of ReLU activation
S w + - 1 wwwphiiggwwtttt w
< 114g" + I1d" l/2
< 0(s)) + 11 + Al ll2
0ss)) +0* 1|2 114g "llllll 2|2
1
1
$ 0sss Loh- + 0(5 )
Ls ) + O Lsh-hot )0(
Ls
s 50 50 -oL-T)
The last three inequalities come from the fact that g² oV). < O(5a) and 1. Thus, we have proved the Lemma in the 1.
Now, we prove the first part the case of s < 1, i.c ||4 < 0(ph-1 19). Because Ag = 0, thus the hypothesis for Ag always holds.
16

---

Published as a conference paper at ICLR 2021
For Ag', we have
||
11o W
(x WIll
I x X -IW1I
x Illl 1/2
O
Thus, the hypothesis holds for Ag'.
Now, we assume that the hypothesis holds for Ag Then, we have
|Agh+11 1 00g wht)l|l
S 11g - g'
wht -wt - 1
|| ||@ + 11"'11//o*+|||
0s))11501||+ lg Ag" |||
01))0(()) 4) + 0(**))))))))))))))))))))) )4 + 90(1~~1))
O()
The last inequality comes from the fact that r max 8) and 5² <3
Next we consider the second part of the case of s < 1.
If 1/ & L- 1/(1+1), we know that = 1/(L and
1/(Ls*) < L (1((+11)
11(44) S Ls
L L-L(((+1) < st
- < g4+1
L-1s- S s,
which means g < 8, thus 8. Then, have
|ل0و4| 0( h-1 = 0(5 9)) =0
= =0 (
If 1f > L 1/(++1), we know that q = 1,11((++11 and > 8; then, r = and
| 0(r~-g)) = 0 -) 0(1") 0 O(L-h/(L+1)
Thus, we can conclude that Lemma 7 also holds for the case of s < 1, which completes the proof.
We now prove a similar Lemma for A
Lemma Given a neural network as stated Theorem 6 let || 1|2 denote the spectral norm, hw* = W wh denote some perturbation on weight matrices, 6 denote the resulting value after perturbation, and Ab -i¹ -h
If > 1 and | < O(s 1. LL) for all h. then
1
| S O
Ligh
If s < and |AW*|2 2 O(g) for all h. where q = min (Ls 1/(L+))) then
|A { OLL
f1/(Ls²)
L1-1/(Ct))
f1/(Ls4t) > -1111111)
17

---

Published as a conference paper at ICLR 2021
Proof Recall that
Id 6" = =0= geth (Wa-))rD*
if - ++1 otherwise
where LG is a dy dg identity matrix and (s.a)
1 1g^-1ww 2 0}. It is easy see that
||b*]| $ 0(s because ||D* S 1 and [ S
We first prove the case of 2 1. We know that A = 0 S 0(s-L-1/L) always holds.
For h < L. can re-write 6²
6a = Id, (W (D("" (..
Then, we have
b" Id, "
(10)
Because of the fact that
fo [+1 = W D'...... g' Dh+1 W.
and g" = D² Da (D We can re-write equation 10 as
(' fg.
Thus,
1T" b 150- - -Sol = 1
by Lemma] 7
Consequently, we have
- -
Since ||g* || - )s we know that
||aa||| S 0( Loh),
||AD||| s ((sb-ht1
always hold. Since L 2 1, 2 1, we simply have |4b4 0 1
Lihh
Now, we prove the case of s < 1. Similarly, we have
- Ilf - ffll fell= folg=11 5
Similarly, must have
I46 < 0
Lish |A < 0
T
where q min(1/(Ls"), L- 1///--111111 1/(+++)) and max(q. s) by Lemma If 1
< 11/(4+1) then s 1+1 > 1/L. We thus have
Ls Lsh- h+1
0 1
0 Lrh- O
O
2
L
8h
Lih
Hence, we get ||Ai|| < O Ligh
If 1/ > L 11(+()))), then 147 < 1/L. We have
0 L
- OLL- L1/(2(1)) S st s-h) = 0
Lsh
Thus, we get ||4b4|| < (h (1
18

---

Published as a conference paper at ICLR 2021
Lemma 9 Given a neural network stated in Theorem let - be the Frobenius norm, W WL+1 be the weight matrices the neural network, AW Wh-W"h be the perturbation on weight matrices, 9h be the parameter vector containing all the elements in Wh. 2 H³ O be the perturbation on parameter vectors, and f f((x) be the resulting value after perturbation.
If & 2 and IAW* I O for all for any weight matrices the following holds
0f (x)
(x)(x)
1 sL
S 50(
00h
agh
If 1 and |A < O(4) for all where q mimmmn1//// min(11(Ts*t)... 1-1/1(+)) for any weight matrices the following holds
0fg(x)
0x((x)
< 0 1
00h
d0h
Proof We first prove the case of d,, 1, i.e. the output of neural network 1-dimensional.
In this case, we know that
0fa(x)
(x
[f
05o(x) awh
11 = =|1
o5 (x) ow F
00h a
owh
and the derivative to W/t is
0fo(x) oWh
(
Then, we have
(x)(x) owh 1|- 11(0")/g~11 (B")Tg~~'11>
- + + 1111
- g6"
T0 11 -
Recall the fact that g* (ss) and 8h s (((*++1-))
When 1, from Lemma 7 and Lemma 8 we know that
1 || s 0
|Ag'l| S 0 (L
Lsh
Then, we have
fo((x) awh |, 5 00s0-1100
a
Lsh + 0(8 +
Lol-hhz) + 0( Ls!. Lsh
00-- s5
When < 1, from Lemma 7 and Lemma we know that
||584|| < O L OLL h((L+)))
i <L-L)(
if if 11((L*))L-11/A 1/(4tt)
and
O(L-1g-A) 8
if < -- 11(1441 if > L 11(1+1),
||4641| <
((
If 1/ S L 1/(++1) we have
A 0f((x) 9wh
00s Lagh
1 Tgb-1+2) (
) + 0( TTL-h44 )0( 1
Lish
19

---

Published as a conference paper at ICLR 2021
Since 1/(Ls*) s 1-1/(4+1) implies L- S s (from proof of Lemma 7,
have
Lsh S 8h-
Then we can conclude that
| [ S
5 87
1
ow"
If 1/1/(L8 1-1/(4+2), have
|| [5) own
||= -0100 +
Since 1/(Lsh) L 1/(41] implies --1 -stt gb+l (from proof of Lemma have
1(--I11/(((t) > sl-ht1
11b-1)/(44) > h-1
Then we have
|
[ OW
S <0 S 0
because s < 1.
sh
We have proved the Lemma for the case of dy = 1.
For the case of dy 1, we know that
|| @ fx)
dy
05 (x)
(xe,(())
< 0 dy
agh
d0h
agh
80h
where (x) the ith dimension of fo(x). The last inequality directly comes from the dimensional case. Since dy a constant, we ignore it. Then, we have
0f T 00h
S 5
a0h
which completes the proof.
Now we can prove Theorem 6 if Wh is obtained by one step gradient descent starting from Wh, is obtained by one step gradient descent starting from 0 and learning rate a. Then, for any weight matrix we have
AAW* 2|2 = V Wm. L(0)]l2
< | Wt. L(0))l=
lIa ((0))==
a Σn [fo(x;) y.l
(xx
T
n
a0h
1/2
du
( >>
ase (x:) 22
Σ
Owh
==
d -.
< <a0(s4),
where G [])());) y, are some constants.
20

---

Published as a conference paper at ICLR 2021
If 0(s (D) when 1, then for any weight matrix we have
|4 l2 - a0(s") - 0(s (/L)..
a 1 (((gs-)) where q = min(1/ L 1/(++)) when s < then for any weight matrix we have
|AW* S $00(5)=5000 O(()).
s
By Lemma 9 we can conclude that
0fo(x)
@fo(x)
s 50
Ow
OW
ss
Then, have
1/2
0fg(x)
(x)
0fp(x)
0fo(x)
12
00
00
80
30
1
S (
8V
WT++
When 1, know that
8 1. 2 11(+4))
Then, have
1 Lss T. < 1-1/((+1) Thus, we know 1/(Ls*) = min(1/(Ls*)) --1((++))) when 2
For the case of 2 1, we can rewrite S Oss (L) LL which completes the proof of Theorem 6
O(gs where q
Now, we prove Theorem with & = 2, i.c. two-step gradient descent adaptation. We know that
31 =
[V)m(()))l'
Thus, we have
|8 a Vo (fo)
=/aV (f) Vo (fo)T - (foCC
V om (Sa) VoLm (I
E2 X ml { [fo(Xm) yml 0fz(Xm)
(xo( - yml Oxo(xm) )T
[1
]] "1||| Vo
=0
00
00
[05o(xm)
(x((xm)
E (xxm)m)) [Jo(Xm) ym
+ofo(x)))
00
00
[Jo(Xm) mm 0fo(xm))
VoLm
00
=a Exxm Fm mi) { (x(m)
T55(xm)
(
00
d0
(xm))))
+ [5o(X - yml
Vo
(So) (m)))))
00
Sa
0 10 T.) 0 I
10(
0 SvL
1 /////
[oom
1
0
VoL. (fo)]| because L > 1
svI
0 gr
+ + or
(F)) where q mmn min(L/(Ls"), --1((+1)), min
VI
3vi
||| VI
VoL. (So)ll-
21

---

Published as a conference paper at ICLR 2021
In the case of dy = 1, we have
(x)
1,- (b"))gh-1 S O(s²), s((h),
owh
which has already been shown the proof of Lemma
Then, we have
VooCm (fo)]l (
L+1
Σ
0f((x)
1) =
Σ
L+1
0fo(x)
11:)
50(5'
a
owh
In the of d 2 1, the bound is simply scaled by constant of Vdy
Thus we have
B8 V So S O -
/ [[VoLm([[))] s (0(s5)) - 01
because 4= m
which completes the proof for the case of 2.
For the case of k > 2, we only need to make sure that the bound on learning rate always holds. Fortunately, since is finite constant, according what we have already showed in the proof of previous lemmas, every step of gradient descent will not change the spectral norm of the weight matrix too much: | ||2 S O(s O(s=L/L) LL) for all if 2 1, and |/W S Og)) for all if where q min(1/(La
- 1) Thus, we may assume that the bound on learning rate always holds during the adaptation. Using triangle inequality to generalize the results from k 2 to k > 2. iee. for S < is 1, we have
13. 1 F. (m) s (
Recall that
E(a, fe) ETm [Cm (fe) a:l v C. (fo)
and
k-1
M₁ ETm Cm((f) Σ 3
where B, Co VoC. and 0, = 0,,00 0,0,+1 = 0, aV C(fo, Dtr). The result is straightforward now.
E PROOF OF THEOREM 4
Theorem Let se be convolutional neural network with L convolutional layers and fully- connected layers and with ReLU activation function, and d, be the input dimension. Denote by Wh the parameter vector of the convolutional layer for h S L-1, 1, and the weight matrices of the fully connected layers for L --1+1 < h & L || 11/ means both the spectral norm of a matrix and the Euclidean of vector. Define
v d
Wh 112
1,.,,...,1 I < <h<
8/
2
f
and let max 8h and be the learning rate of gradient descent. If a S O(gr) with min(1/(Ls").. I I11/((+1)) and min(s-1, (i,)s, the following holds
MM E(ka, So)l 500 (
Proof We prove Theorem 4 by first transforming the convolutional neural network into an equivalent fully connected neural network and then applying Theorem
22

---

Published as a conference paper at ICLR 2021
First of all, assume that there O channels in hthh convolutional layer's output g"(x), where h = O..., represent the dimensionality of imput data by R Instead of using matrices, represent the output of every convolutional layer by a dg length vector g" = [g' f/ 92_ where every of [gh11 1.2 ... gho. is en length vector contains value of different channels at the same position.
For fully-connecttd layers, define CL-1 = CL+1 = 1. We may
We assume that for every element yhas of g its value is completely determined by elements of set Q where O contains kcn-1 elements with fixed positions in gd~1 for a given In other words, every element of the output of convolutional layer is determined by some elements with fixed positions from output of the previous layer. This is exactly how convolutional layer works in deep learning.
If we use a to represent the concatenation of s/a E Q. then A-1 is kca i length vector, 2
where k is the kernel size. Then we have
ot o h-1 U.!)
where U/y E Rkcs-1X0s, is a kca-1 ea matrix.
For notation simplicity, define matrix thhh Rd 1xd,0, where every column of t only has - non-zero elements, and it satisfies
g² 0
By the property of convolutional layer, we know the following facts:
One represent Thhh by th [V/ Vzh ..., Vh] where V/h E Rd.ca-1x0n is sub-matrix of Uhh:
Every V contains the same set of elements as Wh,, while these elements are located at different positions;
Every V,A can be obtained by any other V by swapping rows; Let's define UL-I = WL-1 ....,
"i.-I,..,LLtt, UL+1 = WL+1 for the fully-connected layer and out-
put layer. Then can represent the neural network just in Theorem 3 by fo(x) "...,x
and x E Rdars
Now let ta be the spectral norm of Uhh and max, t.e By Theorem we know that we want O(gr), where q = min(1/(Ls"),, L 1.-1((++1) min(s-
Because every V contains the same set elements, we know that every V has the same Frobenius norm. Because every V can be obtained by any other V by swapping rows, we know that every V has the same rank.
We know that
VF IIT S 11 S 10 S |10 VOLIVV Vd, Wh
where || ||= denotes Frobenius norm, denotes the rank of V/A. The last equality holds because matrix V and vector 4h have the same set of elements.
Let's define
Vd Wh 22, W
h 1, 1,..., I. 1
Sh
12.
I 1+1 hh
and max; Sh-
From above we know that l, O(sn), because sn/ /d/ S S sh. So we also have (s)) Then the conclusion is straightforward.
23

---

Published as a conference paper at ICLR 2021
F REVISION OF THEOREM 3 AND THEOREM 4 IN CLASSIFICATION CASE
now show how to obtain similar results Theorem 3 and Theorem 4 in classification problem, where cross-entropy loss is used instead of squared loss. We need two more restrictions in the classification case:
1. There exist matrix A and B such that gb A I softmax (g g for all data points, where softmax is the softmax operation at the last layer.
2. For any data point x whose belongs to e class, there exists a constant > such that (x) 2 e, i.c. the output of neural network has lower bound on the true class position.
The proof actually similar to the proof in regression case. We briefly talk about the differences here.
Firstly, in the classification case, softmax function used the last layer. By the first restriction, we can get rid of softmax function by introducing new matrices, which further leads to bound of the learning rate as in regression case.
Secondly, if the loss function is the cross-entropy loss, we have:
1
Ofo cm (Xm)
VeLm E (xm.
fe.cm (Xm)
00
where Gn denotes the class of Xm e.g. if Xm belongs the third class, then Cm 3. fo.c.. (xm) denotes the eth dimensional element of fo (xm). We want lower bound of fx,c(x) exists, so that the gradient VeLm (fo) can be further bounded.
Then we can prove similar theorems just follow the steps in regression case.
G PROOF OF THEOREM 5
Let fe be neural network with L hidden layers, with each layer being either fully- connected or convolutional. Assume that [| < co. Then, error(T)) = - fe) - E(T, is non-decreasing function T. Furthermore, for arbitrary T > 0 we have:
Theorem
error(T) S 2(T224))
Proof Recall that E1,, fo) is defined based on fh, which the resulting function whose parameters
dØ m
evolve according to the gradient flow dt
Vor L(f,, D
We actually have the following (Santambrogio 2016):
| | - 0'll - O(vt).
For simplicity and clearness, use A to denote the change of any vectors and matrices. Thus, know that
| < | | < | s O(vi).
Just like the proofs of Lemma
Lemma g and Lemma 9 we show that
|Agill S ((1"1)()1111)) |AM*| 0(1(L- T 00
1, SOUUTIITTTT
by mathematical inductions; we skip the details here. Note that different from some previous theorem, here focus time and thus hide the effect of the spectral norms by treating them as constants.
24

---

Published as a conference paper at ICLR 2021
Then, have
VeCm (fe)
oo fm (fm V m (fo)ll
B(xm m/) of (x)))()))))) yml
(5o(Xm)
}
- (xm) - -mm!
00:
00
(xm)
+ A @
ffo(xxm) T
B B(xm) mm) (x((xm)
[f)(Xmm) ym A
}
00
00
00
+O((+tyb++1)
Recall that:
E(T, EET Cmm(m.o))
(m(ff) (-" V.LmlIm
ETm
(m) + (' d'do" di or (m
TT
(m) [o (J (mm
RT
and
E(T, = Eyn. (fo) T fo Fm 1 ETm (fo) - -o
Because
E(T, f0) fo)
T
Vo Lm 0) dt - (o)m
''
VoLm (16) ( ( /|dte-- (1- VT
2V (fo) 4 (+lll\/VCm(((to)) dt,
have
++1 ++1121133 = ( 21.
error (T) Ë(T, - S 0
by simple calculation.
On the other hand, observe that
ET, fa) ET Cm((f) - 1-" Vo
ČTT fo) = ET., (fm((f) - r VoLm - (fo)
H
We let
G(r) I. | Vo F. ( e)
dt,
and assume that V Cf. is continuous at t = 0. Then, we have G'( = Vm (f6)12.
= |1 [/'" [o V
E(T, fo) ε(T, fo) f(T,
Vo
= Er (G(T) - TT GT))
25

---

Published as a conference paper at ICLR 2021
where (0) G(0) + (O) (note that G(O) 0) is first order approximation to G(T) T = O. When = 1, G(T) TG' can be taken as a local truncation error (i.e., the error that occurs in one step of numerical approximation). When T increases, the difference is no better than the global truncation error (in steps):
Q((T) - - (1- - (1-
Σ i=0
V Lm (fm - L. (m dt
i=
i+1
A4 (mmm)mmm))))
VLe V.m
where Aifm((m.o) Wo (fm,o) V Lm (Sm.mo as shown previously is the i-th time step, and (i) the gradient of G at time step Now we can see that ((T. E(T.
highly relates the difference between V L. (fim different time steps (i.e. A4 (fm, )) Ve Cm (f e and T The first two terms relate to how flat or sharp the hyperplane of C. (fm, is
near = 0. We can wrap it as a constant Co(L, = 0). Then, the error is at least 0).
For the hyperplane smooth enough, we can further get a first order approximation of (. and
yield C(C,t (T)( where C(C, t 0) can be analogized the second order derivative of
H SOME EXPERIMENTAL DETAILS
H.1 IMPLEMENTATION CLASSIFICATION FOR META-RKHS-II
As we mentioned earlier, our proposed energy functional with closed form adaptation can not be directly applied to classification problem. We handle this challenge following Arora et al. (2019). For dy class classification problem, every data x is associated with Rdd, one-hot vector y as its label. For C' classes classification problem, its encoding is C' dimensional vector and use -//C and (C - 1)/C as its correct and incorrect entries encoding. In the prediction, y is replaced by the encoding of training data. fe (x( is replaced by fe [1, 1] E Raxd, for dimension consistency. During the testing time, we compute the encoding of the test data point, and choose the position with largest value its predicted class.
I EXTRA EXPERIMENTAL RESULTS
1.1 COMPARISON WITH RBF KERNEL
One interesting question is, without introducing extra model components networks, what will the results of other kernel be? We provide the results using RBF (Gaussian) kernel here: 42.1 + 1.9 (5-way I-shot) and 54.9 1.1 (5-way 5-shot) Mini-ImageNee, 32.4 + 2.0 (5-way 1-shot) and 38.2 + 0.9 (5-way 5-shot) on FC-100, which are worse than the NTK based Meta-RKHS-II, showing the superiority of using NTK.
I.2 MORE RESULTS ON OUT-OF-DISTRIBUTION GENERALIZATION
We provide some more results out-of-distribution generalization experiments here. From the results we can find that the proposed methods is more robust and can generalize to different datasets better.
26

---

Published as a conference paper at ICLR 2021
Table 5: Meta testing on different out-of-distributioo datasets with model trained on FC-100.
5 WAY
SHOT
WAY 5 SHOT
ALGORITHM
CUB
VGG FLOWER
CUB
VGG FLOWER
MAML
31.58 + 1.89%
50.82 + 1.94%
41.72 1.29%
65.19 1.36%
FOMAML
32.34 + 1.57%
49.90 +: 1.78%
41.96 1.53%
66.87 1.45%
REPTILE
33.56 + 1.40%
46.77 1.81%
1.39911..89%
67.97 = 0.71%
IMAML
49.96 1.98%
3.622%
59.80 = 1.82%
BAYESIAN TAML(SOTA)
0.49%
49.58 0.55%
43.97 0.57%
67.36 0.53%
META-RKHS-
3..12%
48.81 1.89%
13.31
69.02 0..2%
META- SHEA-RKKS-II
36.35 = 1.07%
59.75 = 1.23%
49.92
76.32 0.58%
27

---

Published as a conference paper at ICLR 2021
1.3 MORE RESULTS ON ADVERSARIAL ATTACK
We now show some more extra results on adversarial attack in the following figures. Consistent to the results in main text, we can find that our proposed methods are more robust to adversarial attacks.
Lint PGD Attack
Readle
0.179
Neta-MKrSH
IMA
0.35
IMAML
Meto RKHS 1,100, PQ1
Peptlla
Meta-M0S-- PQR
accuracy
MAMI
FOMAYL
Mara WWW.--H-SSAAO T0000_0000 PQ]
0.075
0.000
0.05
0025-
20
30
60
70
Querles
1/33
2/35 w2b0
4/25 N255
8253
#255 W25
9223 10253
attack magretude spsiion
Figure 6: FC-100 5-way 5-shot Black-box attacks (left) and 5-way 1-shot PGD foc norm attack (right).
I.4 IMPACT OF GRADIENT NORM IN META-RKHS-L
In this experiment, compare between our proposed Meta-RKHS-I and Reptile. We evaluate the trained models with different adaptation steps in lesting-time. The comparison is shown in Figure 7 As we can see, our Meta-RKHS-L always gets better results than Reptile, which supports our idea that the learned function should be close to task-specific optimal and have large functional gradient norm. These two conditions together lead to the ability of fast adaptation.
1897
1554
1555
3.500
5337
1522
1830
1.0
0,00
1493
305
5.30
0.6
Mora R12--
Sesti
(a) Mini- Mini-ImageNet,
5 (b) Mini-ImageNet, Way Shots
5 (c) FC-100, 5 Way 1 (d) FC-100, 5 Way 5
Way Shots
Shots
Shots
Figure 7: Reptile (dashed) vs. Meta-RKHS-I (solid) with different testing adaptation steps (x-axis).
I.5 IMPACT OF NETWORK ARCHITRCTURE FOR DIFFERENT META-LFARNING MODELS
In this section, we compare different meta-learning models with feature channels of 100 and 200 of the CNN network structure with CNN layers respectively.
28

---

Published as a conference paper at ICLR 2021
Table Few-shot classification results on Mini-ImageNet with different number of feature channels of 4 convolution layers.
100
200
ALGORITHM
WAY
SHOT
WAY SHOTS
WAY SHOT
5 WAY 5 SHOTS
MAML
49.50 = 1.58%
64.31 = 1.07%
48.91 1.69%
63.96 0.82%
FOMAML
48.69 1.62%
63.73 0.76%
48.55 1.86%
63.18 0.96%
IMAML
49.30 1.94%
62.89 + 0.95%
48.23 1.58%
62.25 0.83%
REPTILE
50.20 + 1.69%
64.12 + 0.92%
48.72 + 1.97%
63.67 + 0.79%
META-RKHST
51.23 1.79%
66. 0.73%
51.54 64%%
65.92 .2.92 + 0.92%
META-RKHS-II
51.37 + 2.31%
66.97 + 0.98%
50.96 2.15%
65.21 0.87%
Table 7: Few-shot classification results on Mini-ImageNet with different number of feature channels of 5 convolution layers.
100
200
ALGORITHM
5 WAY SHOT
5 WAY 5 SHOTS
WAY SHOT
5 WAY 5 SHOTS
MAML
49.87 + 1.65%
65.78 + 1.18%
48.62 .82%
63.25 0.75%
FOMAML
48.93 + 1.71%
64.37 = 0.80%
48,27 + 1.74%
62.95 + 0.83%
1MAMI.
48.03 + 1.76%
62.15 0.83%
47.52 + 1.73%
61.77 + 0.89%
REPTILE
50.62 1.83%
64.53 + 0.97%
49.33 1.89%
63.26 0.70%
META-RKHST
52.45 + 1.88%
66.07 = 0.69%
51.37 + 1.92%
65.39 + 0.98%
META-RKHS-II
50.92 2.16%
66.45 + 0.91%
50.43 2.42%
64.17 = 1.06%
29


Reviews:
[{"review_id": "eFn5g3RtvPv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nIn the attempt to create an adaptation-free meta-learning method, authors construct an RKHS based on the NTK and explain how to do (gradient-based/MAML-style) meta-learning in this space (instead of parameter space). \nThey propose two energy functionals (first one based on maximizing the norm of the parameters and the second one making the adaptation in closed form, based on the NTK) that approximate the MAML's infeasible learning objective. Evaluation of the functionals doesn't require evaluating the function outside of \"current\" parameters $\\theta$, thus alleviating the need for explicit adaptation, which is a cause of technical problems in other methods.\nAuthors show theoretical arguments confirming their method is closely related to MAML as well as the results of extensive experiments in few-shot classification, regression and out-of-distribution generalization tasks. Authors claim the result of training their (second) method is robust to adversarial examples.\n\n### Good points\n1. The benefits of the presented method are obvious: problems associated with a long adaptation loop is a common yet unsolved problem contemporary methods struggle with.\n2. Experiments are extensive and their results give a convincing argument that the method works.\n3. I find the method I intuitive, as the \"high norm of the parameters\" coincides with parts of parameter space, from which one can adapt the most within few gradient steps (ie. MAML will be naturally seeking these places too).\n\n### Overclaims\n1. Authors claim their method is a first \"single-looped\" meta-learning algorithm. It's not clear from the paper what is formally meant by that; why is iMAML/Reptile/WarpGrad (Flennerhag et al. 2019, https://arxiv.org/abs/1909.00025) not \"single-looped\"? In general, a comment on WarpGrad, which learns a good-for-adaptation geometry through a gradient preconditioner, seems warranted.\n2. Authors claim their method is robust to adversarial attacks. They demonstrate it using just two attacks: a black- and a white-box one, which I don't consider extensive enough. There is a known bias of adversarial defences guarding only against particular attack methods and being susceptible to others (cf. Athalye et al. 2018: Obfuscated Gradients Give a False Sense of Security or Uesato et al. 2018: Adversarial Risk and the Dangers of Evaluating Against Weak Attacks).\nFurthermore, it's not clear that the chosen baselines are the best meta-learning methods out there in terms of adversarial attacks to compare to, which makes the claim of \"our methods are much more robust to adversarial attacks than existing approaches\" groundless.\n3. Results on classification/out-of-distribution generalization are ok-ish to support the claims of the paper, but for completeness I believe it would be preferable to note the SOTA performance on these tasks. Otherwise, a careless reader may be under the impression that the SOTA is held by the authors' methods.\n\n### Personal biases\nI believe short adaptation unrolls is a dead-end. As we get to meta-learn harder/wider task distributions, there won't be any place in the parameter space from which we could achieve good fine-tuning performance without an expressive adaptation procedure.\nThis is particularly clear in meta-RL, where without doing gradient steps, the data we will be obtaining won't fully describe the MDP we need to solve; one may imagine a task with multiple rooms where only after performing adaptation we will pass the correct door and observe the actual task.\n\n### Small\nIn Sec. 4.4 Nilsback & Zisserman (2008) should be a \\citep.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Tog3OhiiwzV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nIn this paper, the authors view MAML from the lens of Reproducing Hilbert Kernel Hilbert Spaces (RKHS) by applying tools from the theory of Neural Tangent Kernels (NTKs). Based on these insights, they develop two meta-learning algorithms that avoid gradient-based inner-loop adaptation. Their algorithms are theoretically grounded and exhibit improved empirical performance.  \n\nOverall, this is a solid contribution that should be of interest to the community. Thus, I recommend acceptance.\n\nStrengths\n\nThis paper is generally well written and proceeds to develop insights into gradient-based few-shot adaptation on first principles from NTK theory. In particular, they establish that parameter adaptation trajectories are equivalent to functional trajectories in some RKHS under the induced NTK, which allows us to bring to bear tools and analysis from that field. \n\nIn particular, the authors provide rigorous mathematical derivations and show that gradient-based few-shot adaptation of the initialisation can be approximated without inner-loop adaptation under certain assumptions on the NTK (that it induces a relatively linear adaptation space), from which they derive two algorithms that avoid gradient-based inner-loop adaptation. They demonstrate empirically that the proposed algorithms perform better than similar algorithms on the standard few-shot learning setup on miniImagenet, as well as that the method is significantly more robust to adversarial robustness and enjoys substantially generalisation to out-of-distribution task. \n\nI am particularly impressed by the second algorithm, Meta-RKHS-II, which derives a closed form-solution to gradient-based adaptation in RKHS that they then map back into parameter space via NTK. Meta-learning the closed-form solver can be thought of as learning a functional inference process for task adaptation. This provides a fresh new perspective and opens up for several new research directions; in particular, the authors mention that inference over the NTK can render the search for task models linear, which provides benefits both in terms of robustness and generalisation. They offer some empirical evidence that this is indeed the case.\n\nWeaknesses\n\nWhile I welcome the principled approach taken in this paper, it is somewhat underwhelming that the first algorithm (which avoids an n^3 complexity in the data) is derived from a first-order Taylor series expansion of the original MAML in parameter space. After all, Meta-RKHS-I can be motivated a first-order Taylor approximation to MAML without the need to involve RKHS. With that said, I do appreciate the effort taken to establish that this is equivalent to optimisation in function space. The gist of this algorithm is to convert MAML into a multi-task objective with a \"regulariser\" that tries to maximise the gradient norm at initialisation. While I believe the precise regulariser introduced here differs from other works, I am missing a discussion of similar works that also propose ways of regularising MAML updates [e.g. 1, 2, 3 and follow-ups]. Finally, I'm a bit unsure as to how Meta-RKHS-I behaves during meta-testing, given that it removes the adaptation loop at meta-training - does it simply use the meta-learner initialisation without adaptation or do you perform gradient descent on meta-test tasks?\n\n\nMinor comments\n\n- At times, the writing can be a bit aggressive. In the abstract, you claim 'superiority of your paradigm': improved performance on miniImagenet is not sufficient evidence to make such a strong claim.\n- if meta-RKHS-I does not do any adaptation on meta-test tasks, I do not think it is fair to say that it replaces the inner-adaptation process: it removes it in favor of a multi-task solution. This may not be a good strategy beyond few-shot learning.\n- You mention that NTKs can yield loss functionals that are convex, which seems appealing. I would suggest making this connection to your proposed algorithms stronger.\n- Eq. 2: is the transpose on the wrong gradient vector? On the line above it, you use \\nabla as a column vector.\n- Eq. 4: in the paragraph following it, you speak of 'solving' Eq 4 - but Eq 4 is a functional and not a problem: do you mean min Eq. 4?\n- Thm 2 is somewhat cryptic. What you mean to say is that the regulariser can be mapped into parameter space, so that E=M, correct?\n- Thms 3 and 4 could use some discussion. What is the take-away? How likely are these conditions to hold?\n- The coloured line in the robustness graphs are very hard to read - I initially thought that MAML achieved optimal robustness...\n\n+++ Post-rebuttal +++\nI've read the rebuttal and am content with the response. I'm maintain my recommendation to accept this paper.\n\nReferences\n[1]  Guiroy et. al.. Towards Understanding Generalization in Gradient-Based Meta-Learning. 2019.\n[2] Khodak et. al. Provable Guarantees for Gradient-Based Meta-Learning. 2019.\n[3] Zhou et. al. Efficient Meta Learning via Minibatch Proximal Update. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "FGQqZ62sqEX", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose two meta-learning algorithms in the reproducing kernel Hilbert space (RKHS) induced by the recently proposed Neural Tangent Kernels (NTK). The authors show how their algorithms obviate an explicit inner loop or task-adaptation step in the meta-learning training phase. In first algorithm, no explicit adaptation function is used, whereas in the second, a close form adaptation function which invokes the NTK is proposed - which is a simpler adaptation than that of MAML and hence, offers computational efficiency. The work is interesting and  supported by theory inspired from the NTK theory, and adds to the newly expanding literature in the use of kernels in meta-learning (unlike the authors’ claim in the introduction, theirs is not the first meta-learning paradigm in the RKHS cf (Wang et al 2020, Cerviño et al 2019)). The authors perform extensive experiments on regression and classification datasets and compare their results with other MAML-type algorithms. The experimental results do not show significant gains in terms of performance over the existing MAML approaches, except in the case of out-of-distribution datasets and adversarial attacks, where it is shown to outperform the others. The performance similarity to other methods is not surprising since the proposed approaches can be seen as an efficient approximations of the MAML.\n\nI give my detailed comments next:\n\n-Firstly, I believe the title of the paper could be changed to ‘Meta-Learning in the RKHS induced by the NTK’ or something more specific.  Currently, it comes off as rather broad and disproportionate to the work and existing work.\n\n-I had some issues with the claim that both the Meta-RKHS approaches do not need an explicit inner-loop adaptation: this does not seem true. In the case of MetaRKHS-I , an inner adaptation is not needed in the meta-training, but necessary just like the MAML during for a test task. In the case of MetaRKHS-II, the NTK gradient flow based adaptation in Eq(6) forms the inner-loop— just that it is a more efficient inner update than the MAML. The authors must consider rephrasing the claim to reflect these.\n\n-\tMeta-RKHS-I:\n1)\tFrom what I understand, the treatment is based on an approximation to the MAML with k-inner gradient descent steps through a Taylor expansion. To this extent, one can expect the performance to be similar to that of $k$-step MAML on using the step size $k\\alpha$ in equation (4) - indeed we see this in the experimental results.\n\n2)\tIt is unclear as to how once the meta-parameter $\\theta$ is learnt, the parameters for a new task are obtained from it— from what I see, it is obtained by $k$-steps of actual gradient descent just like the MAML. This was not mentioned anywhere clearly in the manuscript, I estimated this from the experiment plots in Figure 5 of the Appendix which mentions different inner steps.\n\n3) The Taylor expansion in of equation (2): Under what case is this expansion valid, does this again assume a high degree of similarity between the tasks?\n4) Before equation (3): ‘…equation 2 is an unbiased estimator of …’ This claim is not immediately evident to me, would be better if the authors could expand a bit here\n5) The connection to the NTK seems a bit weak and superficial — Based on eq (3), the authors propose the energy functional in eq(4) for learning the meta-parameter and this is where the RKHS comes in. However, in the very next paragraph in Theorem 2, the equivalence of the gradient in parameter space and the functional gradient is claimed. Here I do not see what properties of the NTK are being invoked. Eq (4) can be with the the parameter gradient instead of the gradient in the RKHS and the approach will continue to hold valid. The authors should bring out the connection to RKHS and NTK more clearly, at present, it seems to me that the approach does not explicitly have connections to an RKHS. Indeed, the proofs of Theorem 3 and 4 also do not seem to use the properties of RKHS or the NTK.\n\n-Meta-RKHS-II\nHere, the authors propose an adaptation function based on the NTK and the gradient flow. This is an interesting adaptation function that evokes the NTK and in the process can help approximate a $k$-order inner gradient and yet be free of the computational difficulties that come due to this in the standard MAML systems. Unlike Meta-RKHS-I, the connection to NTK is strong and clear here. The authors should consider expanding and emphasising this portion better. For example, what is the implication of Theorem 5? \n-Last paragraph on page 5: ‘Intuitively,…thus can be deem more robust.’ I do not follow this, could the authors expand here? I also found the robustness discussion to be needing clarity on the whole.\n\n-\tFigure I:  The axes and the legends are not readable.\n\n-Section 4.3 and 4.4: The Meta-RKHS methods significantly outperform other approaches in the case of adversarial attacks and out-of-the-distribution tasks. This is of merit since it is known that MAML type approaches are far too sensitive to the outlier tasks. Can this robustness be better explained or mathematically analysed in terms of the RKHS or the NTK? This would greatly strengthen the contribution.\n\nOverall, I feel that this is an interesting  and novel contribution, particularly in terms of mathematical concepts, though the approach does not necessarily outperform similar methods by a significant margin except in the case of out-of-the-distribution tasks or adversarial attacks. The connection to RKHS seems a bit weak and currently appears in the form of NTK and that too evidently only in MetaRKHS-II. A suggestion is also consider dataset cases where MAML necessarily requires multiple inner gradient steps to just one step inner gradient update ( the first order MAML) . (Currently all the examples in the paper show similar performance for both MAML and first order MAML). This could help verify if the Meta-RKHS approaches can indeed achieve similar performance as multi-step inner gradient MAML, while having much lower complexity.\n\n\nReferences:\n(Wang et al 2020)  Haoxiang Wang, Ruoyu Sun, and Bo Li. Global convergence and induced kernels of gradient-based meta-learning with neural nets, 2020. \n\n(Cerviño et al 2019) J. Cerviño, J. A. Bazerque, M. Calvo-Fullana and A. Ribeiro, \"Meta-Learning through Coupled Optimization in Reproducing Kernel Hilbert Spaces,\" 2019 American Control Conference (ACC), Philadelphia, PA, USA, 2019, pp. 4840-4846, doi: 10.23919/ACC.2019.8814419.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CLoUOYU6DJ2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper mainly deals with the computational issues of Model Agnostic Meta-Learning (MAML). Specifically, it proposes two meta-learning algorithms where the hypothesis class (i.e. the mapping function set) is defined in RKHS induced by NTK. Extensive experimental studies on many tasks (i.e. regression, few-shot image classification, robustness to adversarial attack, and out-of-distribution generalization) illustrate its superiority compared with other baselines.\n\n\n#############################################################################################\npros:\n1. Overall, this paper is well-written and organized.\n2. This work is based on recent solid theoretical results (i.e. NTK) from the deep learning theory.\n3. The proposed methods have promising performance empirically.\n\n#############################################################################################\ncons:\n1. For the proposed first algorithm (i.e. Meta-RKHS-1), what is the connection or difference between the introduced regularizer in Eq.(4) and some commonly used regularizers (e.g. $||f||_{\\mathcal{h}}^2$) in RKHS. Furthermore, does the objective function in Eq.(4) could become negative? Please give more explanation.\n2.  For the second algorithm (i.e. Meta-RKHS-2), the authors claim that it can have a closed-form adaptation function. In my opinion, this is mainly because the loss function is squared loss, just like kernel least squared regression. If the loss changes to other losses, such as cross-entropy loss, it cannot get a closed-form solution. If so, the algorithm also needs double-looped optimization. Please clarify it.\n3. Although this paper mainly focuses on the NTK-induced RKHS, other kernel functions (e.g. RBF) also can hold for these two algorithms. It is interesting to test whether NTK is better than RBF in the meta-learning setting, such as the regression task.\n4. The formal proofs for the theorems in this paper are mainly based on previous results. It is better to summarize the technical differences for clear theoretical contributions.\n ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers meta-learning based on MAML.  The authors use Neural Tangent Kernels (NTKs) to develop two meta-learning algorithms that avoid the inner-loop adaptation, which makes MAML computationally intensive.  Experimental results demonstrate favorable empirical performance over existing methods. 

The paper is generally well written and readable.  The proposed methods are well motivated and based on solid theoretical ground.  The emprirical performance shows advantages in efficiency and quality.   This work is worth acceptence in ICLR 2021. 
Decision: Accept (Poster)