Year: 2021

Abstract:
Model Agnostic Meta-Learning (MAML) has emerged as a standard framework for meta-learning, where a meta-model is learned with the ability of fast adapting to new tasks. However, as a double-looped optimization problem, MAML needs to differentiate through the whole inner-loop optimization path for every outer-loop training step, which may lead to both computational inefficiency and sub-optimal solutions. In this paper, we generalize MAML to allow meta-learning to be defined in function spaces, and propose the first meta-learning paradigm in the Reproducing Kernel Hilbert Space (RKHS) induced by the meta-model’s Neural Tangent Kernel (NTK). Within this paradigm, we introduce two meta-learning algorithms in the RKHS, which no longer need a sub-optimal iterative inner-loop adaptation as in the MAML framework. We achieve this goal by 1) replacing the adaptation with a fast-adaptive regularizer in the RKHS; and 2) solving the adaptation analytically based on the NTK theory. Extensive experimental studies demonstrate advantages of our paradigm in both efficiency and quality of solutions compared to related meta-learning algorithms. Another interesting feature of our proposed methods is that they are demonstrated to be more robust to adversarial attacks and out-ofdistribution adaptation than popular baselines, as demonstrated in our experiments.

heading: 1 INTRODUCTION
text: Meta-learning (Schmidhuber, 1987) has made tremendous progresses in the last few years. It aims to learn abstract knowledge from many related tasks so that fast adaption to new and unseen tasks becomes possible. For example, in few-shot learning, meta-learning corresponds to learning a meta-model or meta-parameters so that they can fast adapt to new tasks with a limited number of data samples. Among all existing meta-learning methods, Model Agnostic Meta-Learning (MAML) (Finn et al., 2017) is perhaps one of the most popular and flexible ones, with a number of follow-up works such as (Nichol et al., 2018; Finn et al., 2018; Yao et al., 2019; Khodak et al., 2019a;b; Denevi et al., 2019; Fallah et al., 2020; Lee et al., 2020; Tripuraneni et al., 2020). MAML adopts a double-looped optimization framework, where adaptation is achieved by one or several gradientdescent steps in the inner-loop optimization. Such a framework could lead to some undesirable issues related to computational inefficiency and sub-optimal solutions. The main reasons are that 1) it is computationally expensive to back-propagate through a stochastic-gradient-descent chain, and 2) it is hard to tune the number of adaptation steps in the inner-loop as it can be different for both training and testing. Several previous works tried to address these issues, but they can only alleviate them to certain extents. For example, first order MAML (FOMAML) (Finn et al., 2017) ignores the high-order terms of the standard MAML, which can speed up the training but may lead to deteriorated performance; MAML with Implicit Gradient (iMAML) (Rajeswaran et al., 2019) directly minimizes the objective of the outer-loop without performing the inner-loop optimization. But it still needs an iterative solver to estimate the meta-gradient.
To better address these issues, we propose two algorithms that generalize meta-learning to the Reproducing Kernel Hilbert Space (RKHS) induced by the meta-model’s Neural Tangent Kernel (NTK) (Jacot et al., 2018). In this RKHS, instead of using parameter adaptation, we propose to perform an implicit function adaptation. To this end, we introduce two algorithms to avoid explicit ∗The first two authors contribute equally. Correspondence to Changyou Chen (changyou@buffalo.edu). †The research of the first and fifth authors was supported in part by NSF through grants CCF-1716400 and IIS-1910492.
function adaptation: one replaces the function adaptation step in the inner-loop with a new metaobjective with a fast-adaptive regularizer inspired by MAML; the other solves the adaptation problem analytically based on tools from NTK so that the meta-objective can be directly evaluated on samples in a closed-form. When restricting the function space to be RKHS, the solutions to the proposed two algorithms become conveniently solvable. In addition, we provide theoretical analysis on our proposed algorithms in the cases of using fully-connected neural networks and convolutional neural networks as the meta-model. Our analysis shows close connections between our methods and the existing ones. Particularly, we prove that one of our algorithms is closely related to MAML with some high-order terms ignored in the meta-objective function, thus endowing effective optimization. In summary, our main contributions are:
• We re-analyze the meta-learning problem and introduce two new algorithms for metalearning in RKHS. Different from all existing meta-learning algorithms, our proposed methods can be solved efficiently without cumbersome chain-based adaptations. • We conduct theoretically analysis on the proposed algorithms, which suggests that our pro-
posed algorithms are closely related to the existing MAML methods when fully-connected neural networks and convolutional neural networks are used as the meta-model. • We conduct extensive experiments to validate our algorithms. Experimental results indicate
the effectiveness of our proposed methods, through standard few-shot learning, robustness to adversarial attacks and out-of-distribution adaptation.

heading: 2 PRELIMINARIES

heading: 2.1 META-LEARNING
text: Meta-learning can be roughly categorized as black-box adaptation methods (Andrychowicz et al., 2016; Graves et al., 2014; Mishra et al., 2018), optimization-based methods (Finn et al., 2017), non-parametric methods (Vinyals et al., 2016; Snell et al., 2017; Triantafillou et al., 2020) and Bayesian meta-learning methods (Finn et al., 2018; Yoon et al., 2018; Ravi & Beatson, 2019). In this paper, we focus on the framework of Model Agnostic Meta-Learning (MAML) (Finn et al., 2017), which has two key components, meta initialization and fast adaptation. Specifically, MAML solves the meta-learning problem through a double-looped optimization procedure. In the inner-loop, MAML runs a task-specific adaptation procedure to transform a meta-parameter, θ, to a task-specific parameter, {φm}Bm=1, for a total of B different tasks. In the outer-loop, MAML minimizes a total loss of ∑B m=1 L(fφm) with respect to meta-parameter θ, where fφm is the model adapted on task m that is typically represented by a deep neural network. It is worth noting that in MAML, one potential problem is to compute the meta-gradient∇θ ∑B m=1 L(fφm). It requires one to differentiate through the whole inner-loop optimization path, which could be very inefficient.

heading: 2.2 GRADIENT FLOW
text: Our proposed method relies on the concept of gradient flow. Generally speaking, gradient flow is a continuous-time version of gradient descent. In the finite-dimensional parameter space, a gradient flow is defined by an ordinary differential equation (ODE), dθt/dt = −∇θtF (θt), with a starting point θ0 and function F : Rd → R. Gradient flow is also known as steepest descent curve. One can generalize gradient flows to infinite-dimensional function spaces. Specifically, given a function spaceH, a functional F : H → R, and a starting point f0 ∈ H, a gradient flow is similarly defined as the solution of df t/dt = −∇ftF(f t). This is a curve in the function space H. In this paper, we use notation∇ftF(f t), instead of∇HF(f t), to denote the general function derivative of the energy functional F with respect to function f t (Villani, 2008).

heading: 2.3 THE NEURAL TANGENT KERNEL
text: Neural Tangent Kernel (NTK) is a recently proposed technique for characterizing the dynamics of a neural network under gradient descent (Jacot et al., 2018; Arora et al., 2019; Lee et al., 2019). NTK allows one to analyze deep neural networks (DNNs) in RKHS induced by NTK. One immediate benefit of this is that the loss functional in the function space is often convex, even when it is highly non-convex in the parameter space (Jacot et al., 2018) ∗. This property allows one to better understand the property of DNNs. Specifically, let fθ be a DNN parameterized by θ. The corresponding NTK Θ ∗LetH be the function space, F be the realization function for neural network defined in Section 3.2. Note even if a functional loss (e.g., L2 loss) E : H → R is convex onH, the composition E ◦ F is in general not.
is defined as: Θ(x1,x2) = ∂fθ(x1)
∂θ
∂fθ(x2)
∂θ
ᵀ
, where x1,x2 are two data points. In our paper, we will define meta-learning on an RKHS induced by such a kernel.

heading: 3 META-LEARNING IN RKHS
text: We first define the meta-learning problem in a general function space, and then restrict the function space to be an RKHS, where two frameworks will be proposed to make meta-learning feasible in RKHS, along with some theoretical analysis. For simplicity, in the following we will hide the superscript time t unless necessary, e.g., when the analysis involves time-changing.

heading: 3.1 META-LEARNING IN FUNCTION SPACE
text: Given a function space H, a distribution of tasks P (T ), and a loss function L, the goal of metalearning is to find a meta function f∗ ∈ H, so that it performs well after simple adaptation on a specific task. Let Dtrm and Dtestm be the training and testing sets, respectively, sampled from a data distribution of task Tm. The meta-learning problem on function spaceH is defined as:
f∗ = arg min f∈H
E(f), with E(f) = ETm [ L ( Adapt(f,Dtrm),Dtestm )]
(1)
where Adapt denotes some adaptation algorithms, e.g., several steps of gradient descent; E : H → R is called energy functional, which is used to evaluate the model represented by the function f .
In theory, solving equation 1 is equivalent to solving the gradient flow equation df t/dt = −∇ftE(f t). However, solving the gradient flow equation is generally infeasible, since i) it is hard to directly apply optimization methods in function space and ii) the energy functional E contains an adaptation algorithm Adapt, making the functional gradient infeasible. Thus, a better way is to design a special energy functional so that it can be directly optimized without running the specific adaptation algorithm. In the following, we first specify the functional meta-learning problem in RKHS, and then propose two methods to derive efficient solutions for the problem.

heading: 3.2 META-LEARNING IN RKHS
text: We consider a function f that is parameterized by θ ∈ RP , denoted as fθ , with P being the number of parameters. Define a realization function F : RP → H that maps parameters to a function. With these, we can then define an energy function in the parameter space as E , E ◦ F : RP → R with ◦ being the composition operator. Consequently, with an initialized θ0, we can define the gradient flow of E(θt) in parameter space as: dθt/dt = −∇θtE(θt). In the following, we first establish an equivalence between the gradient flow in RKHS and the gradient flow in the parameter space. We then propose two algorithms for meta-learning in the RKHS induced by NTK.
Theorem 1 Let H be an RKHS induced by the NTK Θ of fθ. With f0 = fθ0 , the gradient flow of E(f t) coincides with the function evolution of fθt driven by the gradient flow of E(θt).
The proof of Theorem 1 relies on the property of NTK (Jacot et al., 2018), and is provided in the Appendix. Theorem 1 serves as a foundation of our proposed methods, which indicates that solving the meta-learning problem in RKHS can be done by some appropriate manipulations. In the following, we describe two different approaches termed Meta-RKHS-I and Meta-RKHS-II, respectively.

heading: 3.3 META-RKHS-I: META-LEARNING IN RKHS WITHOUT ADAPTATION
text: Our goal is to design an energy functional that has no adaptation component, but is capable of achieving fast adaptation. For this purpose, we first introduce two definitions: empirical loss function L(fθ,Dm) and expected loss function L(fθ). Let Dm = {xm,i,ym,i}ni=1 be a set containing the data of a regression task Tm. The empirical loss function L(fθ,Dm) and the expected loss function Lm(fθ) can be defined as:
L(fθ,Dm) = 1
2n n∑ i=1 ∥∥f(xm,i)− ym,i ∥∥2, Lm(fθ) = Exm,ym [12∥∥f(xm)− ym ∥∥2 ] .
Our idea is to define a regularized functional such that it endows the ability of fast adaptation in RKHS. Our solution is based on some property of the standard MAML. We start from analyzing the meta-objective of MAML with a k-step gradient-descent adaptation, i.e., applying k gradient-descent steps in the inner-loop. The objective can be formulated as
θ∗ = arg min θ
ETm [ L(fφ ,Dtestm ) ] , with φ = θ − α k−1∑ i=0 ∇θiL(fθi ,Dtrm) ,
where α is the learning rate of the inner-loop, θ0 = θ, and θi+1 = θi − α∇θiL(fθi ,Dtrm)†. By Taylor expansion, we have
ETm [ L(fφ ,Dtestm ) ] ≈ ETm [ L(fθ,Dtestm )− α k−1∑ i=0 ∇θiL(fθi ,Dtrm)∇θL(fθ,Dtestm )ᵀ ] . (2)
Since Dtrm and Dtestm come from the same distribution, equation 2 is an unbiased estimator of
Mk = ETm [Lm(fθ)− k−1∑ i=0 βi], where βi = α∇θiLm(fθi)∇θLm(fθ)ᵀ. (3)
We focus on the case of k = 1, which is M1 = ETm [Lm(fθ)] − αETm [ ‖∇θLm(fθ)‖2 ] . The first term on the RHS is the traditional multi-task loss evaluated at θ for all tasks. The second term corresponds to the negative gradient norm; minimizing it means choosing a θ with the maximum gradient norm. Intuitively, when θ is not a stationary point of a task, one should choose the steepest descent direction to reduce the loss maximally for a specific task, thus leading to fast adaptation.
The above understanding suggests us to propose the following regularized energy functional, Ẽα, for meta-learning in the RKHS induced with the NTK for fast function adaptation:
Ẽ(α, fθ) = ETm [ Lm(fθ)− α‖∇fθLm(fθ)‖2H ] , (4)
where ‖ · ‖H denotes the functional norm inH, and α is a hyper-parameter. The above objective is inspired by the Taylor expansion of the MAML objective, but is defined in the RKHS induced by the NTK. Its connection with MAML and some functional-space properties will be discussed later.
Solving the Function Optimization Problem To minimize equation 4, we first derive Theorem 2 to reduce the function optimization problem to a parameter optimization problem.
Theorem 2 Let fθ be a neural network with parameter θ andH be the RKHS induced by the NTK Θ of fθ. Then, the following are equivalent
Ẽ(α, fθ) =M1, and ‖∇fθLm(fθ)‖2H = ‖∇θLm(fθ)‖2 .
Theorem 2 is crucial to our approach as it indicates that solving problem equation 4 is no more difficult than the original parameter-based MAML, although it only considers one-step adaptation case. Next, we will show that multi-step adaptation in the parameter space can also be well-approximated by our objective equation 4 but with a scaled regularized parameter α. In the following, we consider the squared loss L. The case with the cross-entropy loss is discussed in the Appendix. We assume that fθ is parameterized by either fully-connected or convolutional neural networks, and only consider the impact of number of hidden layers L in our theoretical results.
Theorem 3 Let fθ be a fully-connected neural network with L hidden layers and ReLU activation function, s1, ..., sL+1 be the spectral norm of the weight matrices, s = maxh sh, andα be the learning rate of gradient descent. If α ≤ O(qr) with q = min(1/(LsL), L−1/(L+1)) and r = min(s−L, s), then the following holds
|Mk − Ẽ(kα, fθ)| ≤ O ( 1 L ) .
Theorem 4 Let fθ be a convolutional neural network with L − l convolutional layers and l fullyconnected layers and with ReLU activation function, and dx be the input dimension. Denote by Wh the parameter vector of the convolutional layer for h ≤ L− l, and the weight matrices of the fully connected layers for L− l + 1 ≤ h ≤ L+ 1. ‖ · ‖2 means both the spectral norm of a matrix †For ease of our later notation, we write the gradient∇θiL (thus the parameter as well) as a row vector.
and the Euclidean norm of a vector. Define sh = √ dx‖Wh‖2 if h = 1, ..., L − l, and ‖Wh‖2 if L − l + 1 ≤ h ≤ L + 1. Let s = maxh sh and α be the learning rate of gradient descent. If α ≤ O(qr) with q = min(1/(LsL), L−1/(L+1)) and r = min(s−L, s), the following holds
|Mk − Ẽ(kα, fθ)| ≤ O ( 1 L ) .
The above Theorems indicate that, for a meta-model with fully-connected and convolutional layers, the proposed Meta-RKHS-I can be an efficient approximation of MAML with a bounded error. Comparisons with Reptile and MAML Similar to Reptile and MAML, the testing stage of MetaRKHS-I also requires gradient-based adaptation on meta-test tasks. By Theorem 1, we known that gradient flow of an energy functional can be approximated by gradient descent in a parameter space. Reptile with 1-step adaptation (Nichol et al., 2018) is equivalent to the approximation of the gradient flow of Ẽ(α, fθ) with α = 0, which does not include the fast-adaptation regularization as in our method. For a fairer comparison on the efficiency, we will discuss the computational complexity later.
From the equivalent parameter-optimization form indicated in Theorem 2, we know that our energy functional Ẽ(α, fθ) is closely related to MAML. However, with this form, our method does not need the explicit adaptation steps in training (i.e., the inner-loop of MAML), leading to a simpler optimization problem. We will show that our proposed method leads to better results.

heading: 3.4 META-RKHS-II: META-LEARNING IN RKHS WITH A CLOSED-FORM ADAPTATION
text: In this section, we present our second solution for meta-learning in RKHS by deriving a closed-form adaptation function, i.e., we focus on a case where Adapt(f,Dtrm) is analytically solvable using the theory of NTK. Specifically, we are given a loss function L, tasks Tm with randomly split training set Dtrm = {xtrm,i,ytrm,i}ni=1, and testing set Dtestm . Let θtm and f tm,θ denote the parameters and the corresponding function at time t adapted by task Tm from the meta parameter θ and meta function fθ, respectively. From the NTK theory (Jacot et al., 2018; Arora et al., 2019; Lee et al., 2019), we can write the function/parameter evolution as:
dθtm dt = −∇θtmL(f t m,θ,Dtrm), and df tm,θ dt = dθtm dt ∂f tm,θ ∂θtm
ᵀ
= n∑ i=1 ∂L(f tm,θ,Dtrm) ∂f tm,θ(x tr m,i) Θ(xtrm,i, ·).
The above differential equation corresponds to the adaptation step, i.e., how to adapt the meta parameter/function for task m. By the NTK theory, we can show that this admits closed-form solutions. In our meta-learning settings, this indicates that no explicit adaptation steps are necessary.
To see why this is the case, we first investigate the regression case, where the loss function L is the squared loss. Let x ∈ Dtestm be a test data point. As shown in Arora et al. (2019); Lee et al. (2019), with a large enough neural network we can safely assume that NTK will not change too much during the training. In this case, we can have a closed-form solution for f tm,θ as
f tm,θ(x) = fθ(x) +H(x,X tr m)H −1(Xtrm,X tr m) ( e−tH(X tr m,X tr m) − I ) ( fθ(X tr m)− Y tr ) , (5)
where e is the matrix exponential map, which can be approximated by Padé approximation (M.Arioli et al., 1996). H(Xtrm,X tr m) is an n × n kernel matrix with its (i, j) element being Θ(xm,i,xm,j), H(x,Xtrm) is a 1×n vector with its i-th element being Θ(x,xm,i), fθ(X tr m) ∈ Rn is the predictions of all training data at the initialization, and Y tr ∈ Rn is the target value of the training data. Specifically, at time t =∞, we have
f∞m,θ(x) = fθ(x) +H(x,X tr m)H −1(Xtrm,X tr m) ( Y tr − fθ(Xtrm) ) . (6)
The above results allow us to directly define an energy functional by substituting Adapt(f,Dtrm) in equation 1 with its closed-form solution f tm,θ. In other words, our new energy functional is
E(t, fθ) = ETm [ Lm(f tm,θ) ] , (7)
where f tm,θ is defined in equation 5, and Lm(f tm,θ) is the expectation of L ( f tm,θ,Dtestm ) . For classification problems, we follow the same strategy as in Arora et al. (2019) to extend regression to classification. Mores details can be found in the Appendix, including the algorithm in Appendix A.
On Potential Robustness of Meta-RKHS-II Our extensive empirical studies show that MetaRKHS-II is a more robust model than related baselines. We provide an intuitive explanation on the potential robustness of Meta-RKHS-II, as we find current theories of both robustness machine learning and NTK are insufficient for a formal explanation. Our explanation is based on some properties of both the meta-learning framework and NTK: 1) Strong initialization (meta model): For NTK to generalize well, we argue that it is necessary to start the model with a good initialization. This is automatically achieved in our meta-learning setting, where the meta model serves as the initialization for NTK predictions. Actually, this has been supported by recent research (Fort et al., 2020), which shows that there is a chaotic stage in the NTK prediction with finite neural networks, and the NTK regime can be reachable with a good initialization. 2) Low complex classification boundary: It is known that NTK is a linear model in the NTK regime. Intuitively, generating adversarial samples with a lower complex model should be relatively harder because there is less data in the vicinity of the decision boundary compared to a more complex model, making the probability of the model being attacked smaller. Thus we argue that our model can be more robust than standard meta learning models. 3) Our NTK-based model is robust enough to adapt with different time steps. And these finite time steps can be more robust to adversarial attacks than that of the infinite-time limit partly due to the complexity of back-propagating gradients. We note each of the individual factors might not be enough to ensure robustness. Instead, we argue it is the combination effect of these factors that lead to robustness of our model. Formal analysis is out of the scope of this paper and left for future work.
Connection with Meta-RKHS-I The proposed two methods choose different strategies to avoid explicit adaptation in meta-learning, which seem to be two very different algorithms. We prove below theorem, which indicates that the difference of the underlying gradient flows of the two algorithms indeed increases w.r.t. both T and the depth L of a DNN (we only consider impacts of T and L).
Theorem 5 Let fθ be a neural network with L hidden layers, with each layer being either fullyconnected or convolutional. Assume that ‖L‖∞ <∞. Then, error(T ) = |Ẽ(T, fθ)− E(T, fθ)| is a non-decreasing function of T . Furthermore, for arbitrary T > 0 we have error(T ) ≤ O ( T 2L+3 ) .
Actually, Meta-RKHS-II implicitly contains a term of functional gradient norm because E(T, fθ) = ETm [ Lm(fθ)− ∫ T 0 ∥∥∥∇θtLm(f tm,θ)∥∥∥2dt]. The difference compared to Meta-RKHS-I mainly comes from the fact that Meta-RKHS-I can be regarded as an approximation of time-discrete adaptation, while Meta-RKHS-II is based on time-continuous adaptation. In our experiments, we observe that Meta-RKHS-I is as fast as FOMAML, which means that it is more computationally efficient than the standard MAML. Meanwhile Meta-RKHS-II is the more robust model in tasks of adversarial attack and out-of-distribution adaptation.
Connection with iMAML Our proposed method is similar to the iMAML algorithm (Finn & Levine, 2019) in the sense that both methods try to solve meta-learning without executing the optimization path. Different from iMAML, which still relies on an iterative solver, our method only needs to solve a simpler optimization problem due to the closed-form adaptation.

heading: 3.5 TIME COMPLEXITY ANALYSIS
text: We compare the time complexity of our proposed methods with other first-order meta-learning methods. Without loss of generality, we analyze the complexity in the case of a L-layer MLP or L-layer convolutional neural networks. Recall that dx is the input dimension. Assume each layer has width (filter number) O(p). Let n be the data batch size, k the adaptation steps of inner-loop optimization. We summarize the time complexity in Table 1, where we simply assume the complexity of multiplying matrices with sizes a× b and b× c to be O(abc). Note in the meta-learning setting, n is typically small, indicating the efficiency of our proposed methods.

heading: 4 EXPERIMENTS
text: We conduct a set of experiments to evaluate the effectiveness of our proposed methods, including a sine wave regression toy experiment, few-shot classification, robustness to adversarial attacks, out-of-distribution generalization and ablation study. Due to space limit, more results are provided in the Appendix. We compare our models with related baselines including MAML (Finn et al., 2017), the first order MAML (FOMAML) (Finn et al., 2017), Reptile (Nichol et al., 2018) and iMAML (Rajeswaran et al., 2019). Results are reported as mean and variance over three independent runs.

heading: 4.1 REGRESSION
text: Following Finn et al. (2017); Nichol et al. (2018), we first test our proposed methods on the 1- dimensional sine wave regression problem. This problem is instructive, where a model is trained on many different sine waves with different amplitudes and phases, and tested by adapting the trained model to new sine waves with only a few data points using a fixed number of gradient-descent steps. Following Finn et al. (2017); Nichol et al. (2018), we use a fully-connected neural network with 2 hidden layers and the ReLU activation function. The results are shown in Figure 1.

heading: 4.2 FEW-SHOT IMAGE CLASSIFICATION
text: For this experiment, we choose two popular datasets adopted for meta-learning: Mini-ImageNet and FC-100 (Oreshkin et al., 2018). The cross-entropy loss is adopted for Meta-RKHS-I; while the squared loss is used for Meta-RKHS-II following Arora et al. (2019); Novak et al. (2019). Similar to Finn et al. (2017), the model architecture is set to be a four-layer convolutional neural network with ReLU activation. The filter number is set to be 32. The Adam optimizer (Kingma & Ba, 2015) is used to minimize the energy functional. Meta batch size is set to be 16 and learning rates are set to be 0.01 for Meta-RKHS-II.
The results are shown in Table 2. Note the results of Reptile is different from those in Nichol et al. (2018), because we re-evaluate it under the same setting as Finn et al. (2017), i.e., 10 steps of adaptation is applied during testing. Our results of iMAML is based on the implementation of Spigler (2019). It is observed that our proposed methods
achieve better accuracy than different baselines. Interestingly, our Meta-RKHS-I performs better than FOMAML (this is also the case in other experiments), although they share a similar objective. We conjecture the reason is because our Meta-RKHS-I restricts the function to be in an RKHS, making the functional space smaller thus easier to optimize compared to the unrestricted version of FOMAML. In terms of our two algorithms, there is not always a winner on all the tasks. We note that Meta-RKHS-I is more efficient in training. However, we show below that Meta-RKHS-II is better in terms of robustness to adversarial attacks and out-of-distribution generalization.

heading: 4.3 ROBUSTNESS TO ADVERSARIAL ATTACKS
text: We now compare the adversarial robustness of our methods with other popular baselines. We adopt both white-box and black-box attacks in this experiment. For the white-box attacks, we adopt strong attacks including the PGD Attack (Madry et al., 2017), BPDA attack (Athalye et al., 2018) and
SPSA attack (Uesato et al., 2018). For PGD attack, we use `∞ norm and compare the results on Mini-imagenet and FC-100. We compare the robust accuracy with different magnitude with 20-step attack with a step size of 2/255. For BPDA attack, we apply median smoothing, JPEGFilter and BitSqueezing as input transformation adapted from (Guo et al., 2018) as defense strategies. For SPSA attack, we follow (Uesato et al., 2018) and set the Adam learning rate 0.01, perturbation size δ = 0.01. For Black-box attack, we adopt the strong query efficient attack method (Guo et al., 2019). Follow the setting of Guo et al. (2019), we use a fixed step size of 0.2.
We consider both finite-time and infinite-time adaptation in this experiment. For finite-time adaptation, the Padé approximation with P = Q = 1 and P = Q = 2 to approximate the matrix exponential are considered (Butcher & Chipman, 1992). We use Meta-RKHS-II_t100_PQ1 and Meta-RKHSII_t100_PQ2 to denote methods using finite time t = 100, P = Q = 1 or P = Q = 2, respectively. We observe other finite time t makes similar predictions, thus we only consider t = 100. The results from the black-box attack in Figure 2 indicate the robustness of our Meta-RKHS-II. In fact, the gaps are significantly large, making it the only useful robust model in the adversarial-attack setting. Our Meta-RKHS-I is not as robust as Meta-RKHS-II, but still slightly outperforms other baselines. Regarding the white-box attack, results in Figure 3, 4 and 5 again show that our proposed Meta-RKHS-II is significantly more robust than baselines under the three strong attacks. It is also interesting to see that our Meta-RKHS-I performs slightly better than Meta-RKHS-II in some rare cases, e.g., in the Mini-ImageNet 5-way 1-shot case when the attack magnitude is not too small. More results are presented in the Appendix.
1/2 55 2/2 55 3/2 55 4/2 55 5/2 55 6/2 55 7/2 55 8/2 55 9/2 55 10 /25 5
attack magnitude
0.0
0.1
0.2
0.3
0.4
0.5
ac cu
ra cy
L infinity norm PGD Attack
1/2 55 2/2 55 3/2 55 4/2 55 5/2 55 6/2 55 7/2 55 8/2 55 9/2 55 10 /25 5
attack magnitude
0.000
0.025
0.050
0.075
0.100
0.125
0.150
L infinity norm PGD Attack
1/2 55 2/2 55 3/2 55 4/2 55 5/2 55 6/2 55 7/2 55 8/2 55 9/2 55 10 /25 5
attack magnitude
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35 L infinity norm PGD Attack
MAML FOMAML IMAML Reptile Meta-RKHS-I Meta-RKHS-II Meta-RKHS-II_t100_PQ1 Meta-RKHS-II_t100_PQ2
Figure 5: `∞ norm PGD attack on Mini-ImageNet and FC-100. Mini-ImageNet 5-way 5-shot (left), Mini-ImageNet 5-way 1-shot (middle) and FC-100 5-way 5-shot (right).

heading: 4.4 OUT-OF-DISTRIBUTION GENERALIZATION
text: We adopt similar strategy in (Lee et al., 2020) to test a model’s ability of generalizing to out-ofdistribution datasets. In this setting, the state of arts are achieved by Bayesian TAML (Lee et al., 2020). Different from their setting that considers any-shot learning with maximum number of examples for each class being as large as 50, we only focus on the standard 1 or 5 shot learning. We thus modify their code to accommodate our standard setting. The CUB (Wah et al., 2011) and VGG Flower Nilsback & Zisserman (2008) are fine-grained datasets used in this experiment, where all images are resized to 84 × 84. We follow Lee et al. (2020) to split these datasets into meta training/validation/testing sets. We first train all the methods on Mini-ImageNet or FC-100 datasets, then conduct meta-testing on CUB and VGG Flower datasets. The results are shown in Table 3. Again, our methods achieve the best results, outperforming the state-of-art method with our Meta-RKHS-II, indicating the robustness of our proposed methods. More results are presented in the Appendix.
4.5 ABLATION STUDY Table 3: Meta testing on different out-of-distribution datasets with model trained on Mini-ImageNet.
5 WAY 1 SHOT 5 WAY 5 SHOT ALGORITHM CUB VGG FLOWER CUB VGG FLOWER MAML 34.23± 1.52% 52.98± 1.76% 52.36± 0.94% 67.52± 1.30% FOMAML 35.32± 1.69% 53.86± 1.64% 52.02± 0.71% 68.83± 1.16% REPTILE 35.61± 1.38% 53.57± 1.58% 51.93± 0.89% 71.62± 1.25% IMAML 40.55± 0.61% 54.97± 0.80% 46.31± 2.03% 60.67± 1.91% BAYESIAN TAML(SOTA) 41.57± 0.60% 58.56± 0.66% 61.78± 0.56% 77.95± 0.46% META-RKHS-I 36.73± 1.26% 54.79± 1.61% 54.19± 0.73% 72.76± 1.08% META-RKHS-II 45.36± 0.87% 60.80± 1.02% 65.21± 0.64% 78.25± 0.49% We conduct several ablation studies, including: comparing Reptile with Meta-RKHS-I under different adaptation steps (results shown in the Appendix), testing the impact of choosing different time t in Meta-RKHS-II (results shown in Table 4) and the impact of network architecture with different number of CNN feature channels (results shown in the Appendix). It is interesting to see that a finite-time (around t = 10) achieves the best accuracy, although the infinite-time case guarantees a stationary point. This indicates that a stationary point achieved by limited training data in the adaptation step is not always the best choice, because the limited training data might easily overfit the model, thus achieving worse test results.

heading: 5 CONCLUSION
text: We develop meta-learning in RKHS, and propose two practical algorithms allowing efficient adaptation in the function space by avoiding some complicated adaptations as in traditional methods. We show connections between our proposed methods and existing ones. Extensive experiments suggest that our methods are more effective, achieve better generalization and are more robust against adversarial attacks and out-of-distribution generalization, compared to popular strong baselines.

heading: A ALGORITHMS
text: Our proposed algorithms for meta-learning in the RKHS are summarized in Algorithm 1.
Algorithm 1 Meta-Learning in RKHS Require: p(T ): distribution over tasks, randomly initialized neural network parameters θ.
while not done do Sample a batch of tasks {Tm}Bm=1 ∼ p(T ) for all Tm do
Sample a batch of data points Dm or Sample two batches of data points Dtrm, Dtestm . end for Evaluate the energy functional by equation 4 with {Dm}Bm=1 or Evaluate the energy functional by equation 7 with {Dtrm,Dtestm }Bm=1. Minimize the energy functional w.r.t θ.
end while

heading: B PROOF OF THEOREM 1
text: Theorem 1 If fθ is a neural network with parameter θ ∈ RP and H is the Reproducing Kernel Hilbert Space (RKHS) induced by Θ, where Θ is the Neural Tangent Kernel (NTK) of fθ, then with initialization f0 = fθ0 , the gradient flow of E(f t) coincides with the function evolution of fθt induced by the gradient flow of E(θt).
Proof Without loss of generality, we can rewrite E(f) = ETm{E(xm,ym) [C(f(xm),ym)]} with some function C(·, ·). For a neural network fθ with parameter θ ∈ RP , the gradient flow of E in RP is
dθt
dt = −∇θtE(θt).
We have
dθt
dt = −∇θt(E ◦ F )(θt)
= −ETm{E(xm,ym) [∇θtC(fθt(xm),ym)]} = −ETm { E(xm,ym) [ ∂C(f tθ(xm),ym)
∂f tθ(xm)
∂f tθ(xm)
∂θt
]} .
We know that the dynamics of fθt is
dfθt dt = dθt dt ∂fθt ∂θt
ᵀ
= −ETm { E(xm,ym) [ ∂C(fθt(xm),ym)
∂fθt(xm)
∂fθt(xm)
∂θt
]} ∂fθt
∂θt
ᵀ
= −ETm { E(xm,ym) [ ∂C(fθt(xm),ym)
∂fθt(xm)
∂fθt(x) ∂θt ∂fθt ∂θt ᵀ]} = −ETm { E(xm,ym) [ ∂C(fθt(xm),ym)
∂fθt(xm) Θt(xm, ·)
]} , (8)
where Θt is the Neural Tangent Kernel of neural network fθt (Jacot et al., 2018).
If Ht is the Reproducing Kernel Hilbert Space induced by a kernel Θt and Vxm : H → R is the evaluation functional at xm, which is defined as
Vxm(f) = f(xm),
then for an arbitrary function g and a small perturbation , we have
〈∇fVxm(f), g〉 = lim →0
Vxm(f + g)− Vxm(f)
〈∇fVxm(f), g〉 = lim →0
f(xm) + g(xm)− f(xm)
〈∇fVxm(f), g〉 = g(xm) 〈∇fVxm(f), g〉 = 〈Θt(xm, ·), g〉 ∇fVxm(f) = Θt(xm, ·) ∇ff(xm) = Θt(xm, ·).
With an initial function f0 = fθ0 ∈ H, the gradient flow of E inH is
df t
dt = −∇ftE(f t).
We have
df t
dt = −ETm{E(xm,ym)
[ ∇ftC(f t(xm),ym) ] }
= −ETm { E(xm,ym) [ ∂C(f t(xm),ym)
∂f t(xm) ∇ftf t(xm) ]} = −ETm { E(xm,ym) [ ∂C(f t(xm),ym)
∂f t(xm) Θt(xm, ·)
]} . (9)
We can complete the proof by comparing equation 8 and equation 9.

heading: C PROOF OF THEOREM 2
text: Theorem 2 If fθ is a neural network with parameter θ and H is the Reproducing Kernel Hilbert Space (RKHS) induced by Θ, where Θ is the Neural Tangent Kernel (NTK) of fθ, then

heading: M1 = Ẽ(α, fθ), and β0 = α‖∇θLm(fθ)‖2 = α‖∇fθLm(fθ)‖2H.
text: Proof Without loss of generality, we rewrite Lm(fθ) = Exm,ym [C(fθ(xm),ym)]. In regression task, we have
C(fθ(xm),ym) = 1
2 ∥∥fθ(xm)− ym ∥∥2 . In classification task, we have
C(fθ(xm),ym) = ym log(fθ(xm)) ᵀ,
where log is element-wise logarithm operation.
‖∇θLm(fθ)‖2
= ∇θLm(fθ)∇θLm(fθ)ᵀ = ∇θExm,ym [C(fθ(xm),ym)]∇θExm,ym [C(fθ(xm),ym)] ᵀ
= Exm,ym
[ ∂C(fθ(xm),ym)
∂fθ(xm)
∂fθ(xm)
∂θ
] Exm,ym [ ∂fθ(xm)
∂θ
ᵀ ∂C(fθ(xm),ym)
∂fθ(xm) ᵀ] = Exm,ym { Ex′m,y′m [ ∂C(fθ(xm),ym)
∂fθ(xm)
∂fθ(xm)
∂θ
∂fθ(x ′ m)
∂θ
ᵀ ∂C(fθ(x ′ m),y ′ m)
∂fθ(x′m) ᵀ]} = Exm,ym { Ex′m,y′m [ ∂C(fθ(xm),ym)
∂fθ(xm) Θ(xm,x
′ m) ∂C(fθ(x
′ m),y ′ m)
∂fθ(x′m)
ᵀ]}
= 〈 Exm,ym [ ∂C(fθ(xm),ym)
∂fθ(xm) Θ(xm, ·)
] ,Ex′m,y′m [ ∂C(fθ(x ′ m),y ′ m)
∂fθ(x′m) Θ(x′m, ·) ]〉 H
= 〈 Exm,ym [ ∂C(fθ(xm),ym)
∂fθ(xm) ∇fθfθ(xm)
] ,Ex′m,y′m [ ∂C(fθ(x ′ m),y ′ m)
∂fθ(x′m) ∇fθfθ(x′m) ]〉 H
= 〈 ∇fθLm(fθ),∇fθLm(fθ) 〉 H
= ‖∇fθLm(fθ)‖2H, where 〈·, ·〉H is the inner product in Reproducing Kernel Hilbert Space (RKHS) H. In the above equations, we use the definition of Neural Tangent Kernel (NTK), the property of inner product in RKHS, the definition of evaluation functional and its gradient in RKHS.
Recall that Ẽ(α, fθ) = ETm [ Lm(fθ)− α‖∇fθLm(fθ)‖2H ] and
Mk = ETm [ Lm(fθ)− k−1∑ i=0 βi ] ,
where βi = α∇θiLm(fθi)∇θLm(fθ)ᵀ and θ0 = θ,θi+1 = θi − α∇θiL(fθi ,Dtrm). The result is straightforward now.

heading: D PROOF OF THEOREM 3
text: The proof techniques we use are similar to some previous works such as (Arora et al., 2019; Allen-Zhu et al., 2019). We summaries some of the differences. Different from previous works that typically assume a neural network is Gaussian initialized, we do not have such an assumption as we are trying to learn a good meta-initialization in the meta-learning setting. Previous works try to investigate the behavior of models during training, while we focus on revealing the connection between different meta-learning algorithms. Previous work focuses on single-task regression/classification problems, while we focus on meta-learning problem.
Theorem 3 Let fθ be a fully-connected neural network with L hidden layers and ReLU activation function, s1, ..., sL+1 be the spectral norm of the weight matrices, s = maxh sh, andα be the learning rate of gradient descent. If α ≤ O(qr) with q = min(1/(LsL), L−1/(L+1)) and r = min(s−L, s), then the following holds
|Ẽ(kα, fθ)−Mk| ≤ O ( 1 L ) .
Proof We first prove the case of k = 2, i.e. applying a two-step gradient descent adaptation in MAML.
We need to prove the following theorem first.
Theorem 6 Let fθ be a fully-connected neural network with L hidden layers, and x be a data sample. Represent the neural network by fθ(x) = σ(σ(...σ(xW 1)...WL−1)WL)WL+1, where W 1, ...,WL+1 denote the weight matrices, and σ is the ReLU activation function. Let s1, ..., sL+1 be the spectral norm of weight matrices, and s = maxh sh. Let α be the learning rate of gradient descent, and f θ̃(x) be the resulting value after one step of gradient descent, and ‖ · ‖F be the Frobenius norm.
If α ≤ O(qs−L), where q = min(1/(LsL), L−1/(L+1)), then∥∥∥∂f θ̃(x) ∂θ̃ − ∂fθ(x) ∂θ ∥∥∥ F ≤ O( 1 s √ L+ 1 ).
Remark 1 Theorem 6 states that for a neural network with L hidden layers, if the learning rate of gradient descent is bounded, then the norm of derivative w.r.t all the parameters will not change
too much, although there are O(Lm2) parameters, where m denotes the maximum width of hidden layers. We use row vector instead of column vector for consistency, while it does not affect our results.
For simplicity, we will write gh(x) as gh. The bias terms in the neural network are introduced by adding an additional coordinate thus omitted in Theorem 6. Without loss of generality, we can assume ‖x ‖ ≤ 1, which can be done by data normalization in pre-processing. Let gh(x) = σ(σ(...σ(xW 1)...Wh−1)Wh) be the activation at hth hidden layer and g0(x) = x, gL+1 = fθ(x). Define diagonal matrices Dh, where Dh(i,i) = 1{g h−1Wh ≥ 0} and
bh = { Idy , if h = L+ 1 bh+1(Wh+1)ᵀDh, otherwise
where Idy is a dy × dy identity matrix. We first prove the following Lemma.
Lemma 7 Given a neural network as stated in Theorem 6, let ‖ · ‖2 denote the spectral norm, 4Wh = W̃h −Wh denote some perturbation on weight matrices, g̃h(x) denote the resulting value after perturbation, and4gh(x) = g̃h(x)− gh(x). If s ≥ 1 and ‖4Wh‖2 ≤ O(s−L/L) for all h, then
‖4gh‖ ≤ O( 1 LsL−h+1 );
If s < 1 and ‖4Wh‖2 ≤ O(q) for all h, where q = min(1/(LsL), L−1/(L+1)) and r = max(q, s), then
‖4gh‖ ≤ O(rh−1q) =
{ O(
1
LsL−h+1 ), if 1/(LsL) ≤ L−1/(L+1)
O(L−h/(L+1)), if 1/(LsL) > L−1/(L+1).
Proof Proof of Lemma 7 is based on induction.
We first prove the case of s ≥ 1. Note that g0 = x, thus4g0 = 0 ≤ O( 1 LsL−0+1 ) always holds.
For4g1, we have
‖4g1‖ = ‖σ(x W̃ 1)− σ(xW 1)‖ ≤ ‖x W̃ 1 − xW 1‖, due to the property of ReLU activation ≤ ‖x ‖‖4W 1‖2
≤ O( 1 LsL ).
Thus, the hypothesis holds for4g1. Now, assume that the hypothesis holds for4gh, then we have
‖4gh+1‖ = ‖σ(g̃hW̃h+1)− σ(ghWh+1)‖ ≤ ‖g̃hW̃h+1 − ghWh+1‖, due to the property of ReLU activation ≤ ‖g̃hWh+1 + g̃h4Wh+1 − ghWh+1‖ ≤ ‖4gh‖‖Wh+1‖2 + ‖g̃h‖‖4Wh+1‖2 ≤ O(s)‖4gh‖+ ‖gh +4gh‖‖4Wh+1‖2 ≤ O(s)‖4gh‖+O(sh)‖4Wh+1‖2 + ‖4gh‖‖4Wh+1‖2
≤ O(s)O( 1 LsL−h+1 ) +O(sh)O( 1 LsL ) +O(
1
LsL−h+1 )O(
1
LsL )
≤ O( 1 LsL−h ).
The last three inequalities come from the fact that gh = σ(σ(...σ(xW 1)...Wh−1)Wh) ≤ O(sh) and s ≥ 1. Thus, we have proved the Lemma in the case s ≥ 1. Now, we prove the first part of the case of s < 1, i.e. ‖4gh‖ ≤ O(rh−1q). Because4g0 = 0, thus the hypothesis for4g0 always holds.
For4g1, we have
‖4g1‖ = ‖σ(x W̃ 1)− σ(xW 1)‖ ≤ ‖x W̃ 1 − xW 1‖ ≤ ‖x ‖‖4W 1‖2 ≤ O(q).
Thus, the hypothesis holds for4g1. Now, we assume that the hypothesis holds for4gh. Then, we have
‖4gh+1‖ = ‖σ(g̃hW̃h+1)− σ(ghWh+1)‖ ≤ ‖g̃hW̃h+1 − ghWh+1‖ ≤ ‖g̃hWh+1 + g̃h4Wh+1 − ghWh+1‖ ≤ ‖4gh‖‖Wh+1‖2 + ‖g̃h‖‖4Wh+1‖2 ≤ O(s)‖4gh‖+ ‖gh +4gh‖‖4Wh+1‖2 ≤ O(s)O(rh−1q) +O(sh)q + qO(rh−1q) ≤ O(rhq).
The last inequality comes from the fact that r = max(q, s) and sh < s < 1.
Next we consider the second part of the case of s < 1.
If 1/(LsL) ≤ L−1/(L+1), we know that q = 1/(LsL) and
1/(LsL) ≤ L−1/(L+1)
L1/(L+1) ≤ LsL
L−L/(L+1) ≤ sL
L−1 ≤ sL+1
L−1s−L ≤ s,
which means q ≤ s, thus r = s. Then, we have
‖4gh‖ = O(rh−1q) = O(sh−1q) = O(sh−1L−1s−L) = O( 1 LsL−h+1 ).
If 1/(LsL) > L−1/(L+1), we know that q = L−1/(L+1) and q > s; then, r = q and
‖4gh‖ = O(rh−1q) = O(qh−1q) = O(qh) = O(L−h/(L+1)).
Thus, we can conclude that Lemma 7 also holds for the case of s < 1, which completes the proof.
We now prove a similar Lemma for4bh.
Lemma 8 Given a neural network as stated in Theorem 6, let ‖ · ‖2 denote the spectral norm, 4Wh = W̃h −Wh denote some perturbation on weight matrices, b̃h denote the resulting value after perturbation, and4bh = b̃h − bh.

heading: If s ≥ 1 and ‖4Wh‖2 ≤ O(s−L/L) for all h, then
text: ‖4bh‖ ≤ O( 1 Lsh );
If s < 1 and ‖4Wh‖2 ≤ O(q) for all h, where q = min(1/(LsL), L−1/(L+1)), then ‖4bh‖ ≤ { O(L−1s−h), if 1/(LsL) ≤ L−1/(L+1) O(L(h−L−1)/(L+1)), if 1/(LsL) > L−1/(L+1).
Proof Recall that bh = { Idy , if h = L+ 1 bh+1(Wh+1)ᵀDh, otherwise
where Idy is a dy × dy identity matrix and Dh(i,i) = 1{g h−1Wh ≥ 0}. It is easy to see that ‖bh‖ ≤ O(sL−h+1), because ‖Dh‖2 ≤ 1 and ‖Wh‖2 ≤ s. We first prove the case of s ≥ 1. We know that4bL+1 = 0 ≤ O(s−L−1/L) always holds. For h ≤ L, we can re-write bh as
bh = Idy (W L+1)ᵀDL(WL)ᵀDL−1...(Wh+1)ᵀDh.
Then, we have
bh(gh)ᵀ = Idy (W L+1)ᵀDL(WL)ᵀDL−1...(Wh+1)ᵀDh(gh)ᵀ. (10)
Because of the fact that
fθ = g L+1 = xW 1D1W 2D2...DLWL+1 = ghWh+1Dh+1...DLWL+1
and gh = ghDh, Dh = (Dh)ᵀ. We can re-write equation 10 as
bh(gh)ᵀ = fᵀθ .
Thus,
‖b̃h(g̃h)ᵀ − bh(gh)ᵀ‖ = ‖f θ̃ − fθ‖ = ‖4g L+1‖ ≤ O( 1
L )
by Lemma 7. Consequently, we have
‖b̃h(g̃h)ᵀ − bh(gh)ᵀ‖ = ‖4bh(gh)ᵀ +4bh4(gh)ᵀ + b̃h4(gh)ᵀ‖ ≤ O( 1 L ).
Since ‖gh‖ ≤ O(sh), we know that
‖4bh‖ ≤ O( 1 Lsh ), ‖4bh‖ ≤ O(sL−h+1)
always hold. Since L ≥ 1, s ≥ 1, we simply have ‖4bh‖ ≤ O( 1 Lsh ).
Now, we prove the case of s < 1. Similarly, we have
‖b̃h(g̃h)ᵀ − bh(gh)ᵀ‖ = ‖f θ̃ − fθ‖ = ‖4g L+1‖ ≤ O( 1
L ).
Similarly, we must have
‖4bh‖ ≤ O( 1 Lsh ), ‖4bh‖ ≤ O( 1 Lrh−1q ),
where q = min(1/(LsL), L−1/(L+1)) and r = max(q, s) by Lemma 7.
If 1/(LsL) ≤ L−1/(L+1), then sL+1 ≥ 1/L. We thus have
O( 1
Lrh−1q ) = O(
LsL−h+1
L ) = O(
sL+1
sh ) ≥ O( 1 Lsh ).
Hence, we get ‖4bh‖ ≤ O( 1 Lsh ).
If 1/(LsL) > L−1/(L+1), then sL+1 < 1/L. We have
O( 1 Lrh−1q ) = O(L−1 · Lh/(L+1)) ≤ O(L−1 · s−h) = O( 1 Lsh ).
Thus, we get ‖4bh‖ ≤ O(L(h−L−1)/(L+1)).
Lemma 9 Given a neural network as stated in Theorem 6, let ‖ · ‖F be the Frobenius norm, W 1, ...,WL+1 be the weight matrices in the neural network,4Wh = W̃h−Wh be the perturbation on weight matrices, θh be the parameter vector containing all the elements in Wh,4θh = θ̃h − θh be the perturbation on parameter vectors, and f θ̃(x) be the resulting value after perturbation.
If s ≥ 1 and ‖4Wh‖2 ≤ O(s−L/L) for all h, for any weight matrices the following holds∥∥∥∂f θ̃(x) ∂θ̃h − ∂fθ(x) ∂θh ∥∥∥ F ≤ O( 1 sL );
If s < 1 and ‖4Wh‖2 ≤ O(q) for all h, where q = min(1/(LsL), L−1/(L+1)), for any weight matrices the following holds ∥∥∥∂f θ̃(x)
∂θ̃h − ∂fθ(x) ∂θh ∥∥∥ F ≤ O( 1 sL ).
Proof We first prove the case of dy = 1, i.e. the output of neural network is 1-dimensional.
In this case, we know that∥∥∥∂f θ̃(x) ∂θ̃h − ∂fθ(x) ∂θh ∥∥∥ F = ∥∥∥∂f θ̃(x) ∂W̃h − ∂fθ(x) ∂Wh ∥∥∥ F = ∥∥∥4∂fθ(x) ∂Wh ∥∥∥ F
and the derivative to Wh is ∂fθ(x)
∂Wh = (bh)ᵀgh−1.
Then, we have ∥∥∥4∂fθ(x) ∂Wh ∥∥∥ F = ‖(b̃h)ᵀg̃h−1 − (bh)ᵀgh−1‖F
= ‖(b̃h)ᵀgh−1 − (bh)ᵀgh−1 + (b̃h)ᵀ4gh−1‖F ≤ ‖(4bh)ᵀgh−1‖F + ‖(bh +4bh)ᵀ4gh−1‖F .
Recall the fact that gh ≤ O(sh) and bh ≤ O(sL+1−h). When s ≥ 1, from Lemma 7 and Lemma 8 we know that
‖4gh‖ ≤ O( 1 LsL−h+1 ), ‖4bh‖ ≤ O( 1 Lsh ).
Then, we have∥∥∥4∂fθ(x) ∂Wh ∥∥∥ F ≤ O(sh−1)O( 1 Lsh ) +O(sL+1−h)O(
1
LsL−h+2 ) +O(
1
LsL−h+2 )O(
1
Lsh )
≤ O( 1 sL ).
When s < 1, from Lemma 7 and Lemma 8 we know that
‖4gh‖ ≤
{ O(
1
LsL−h+1 ), if 1/(LsL) ≤ L−1/(L+1) O(L−h/(L+1)), if 1/(LsL) > L−1/(L+1)
and
‖4bh‖ ≤ { O(L−1s−h), if 1/(LsL) ≤ L−1/(L+1) O(L(h−L−1)/(L+1)), if 1/(LsL) > L−1/(L+1).
If 1/(LsL) ≤ L−1/(L+1), we have∥∥∥4∂fθ(x) ∂Wh ∥∥∥ F ≤ O(sh−1)O( 1 Lsh ) +O(sL−h+1)O(
1
LsL−h+2 ) +O(
1
LsL−h+2 )O(
1
Lsh ).
Since 1/(LsL) ≤ L−1/(L+1) implies L−1 ≤ sL+1 (from proof of Lemma 7), we have 1
Lsh ≤ sL−h+1.
Then we can conclude that ∥∥∥4∂fθ(x) ∂Wh ∥∥∥ F ≤ O( 1 sL ).
If 1/(LsL) > L−1/(L+1), we have∥∥∥4∂fθ(x) ∂Wh ∥∥∥ F ≤ O(sh−1)O(L(h−L−1)/(L+1)) +O(sL+1−h)O(L−(h−1)/(L+1))
+O(L−(h−1)/(L+1))O(L(h−L−1)/(L+1)).
Since 1/(LsL) > L−1/(L+1) implies L−1 > sL+1 (from proof of Lemma 7), we have
L(h−L−1)/(L+1) > sL−h+1, 1
L(h−1)/(L+1) > sh−1.
Then we have ∥∥∥4∂fθ(x) ∂Wh ∥∥∥ F ≤ O( 1 L ) ≤ O( 1 sL ), because s < 1.
We have proved the Lemma for the case of dy = 1.
For the case of dy > 1, we know that∥∥∥∂f θ̃(x) ∂θ̃h − ∂fθ(x) ∂θh ∥∥∥2 F = dy∑ i=1 ∥∥∥∂f θ̃,i(x) ∂θ̃h − ∂fθ,i(x) ∂θh ∥∥∥2 F ≤ O( dy s2L2 ),
where fθ,i(x) is the ith dimension of fθ(x). The last inequality directly comes from the 1- dimensional case.
Since dy is a constant, we ignore it. Then, we have∥∥∥∂f θ̃(x) ∂θ̃h − ∂fθ(x) ∂θh ∥∥∥ F ≤ O( 1 sL ),
which completes the proof.
Now we can prove Theorem 6, if W̃h is obtained by one step gradient descent starting from Wh, θ̃ is obtained by one step gradient descent starting from θ, and learning rate is α. Then, for any weight matrix we have
‖4Wh‖2 = ‖α∇WhL(θ)‖2 ≤ ‖α∇WhL(θ)‖F = ‖α∇θhL(θ)‖F
= α ∥∥∥∑ni=1
n [fθ(xi)− yi]
∂fθ(xi)
∂θh ᵀ∥∥∥ F
≤ α ∑n i=1
n ci  dy∑ j ∥∥∥∂fθ,j(xi) ∂Wh ∥∥∥2 F 1/2
≤ α ∑n i=1 n ci √ dyO(s L−h+1)O(sh−1) ≤ αO(sL),
where ci = ‖fθ(xi)− yi ‖ are some constants.
If α ≤ O(s−2L/L) when s ≥ 1, then for any weight matrix we have
‖4Wh‖2 ≤ αO(sL) ≤ O(s−L/L).
If α ≤ O(qs−L) where q = min(1/(LsL), L−1/(L+1)) when s < 1, then for any weight matrix we have
‖4Wh‖2 ≤ αO(sL) ≤ O(q).
By Lemma 9, we can conclude that∥∥∥∂f θ̃(x) ∂W̃h − ∂fθ(x) ∂Wh ∥∥∥ F ≤ O( 1 sL ).
Then, we have ∥∥∥∂f θ̃(x) ∂θ̃ − ∂fθ(x) ∂θ ∥∥∥ F = [ L+1∑ h=1 ∥∥∥∂f θ̃(x) ∂θ̃h − ∂fθ(x) ∂θh ∥∥∥2 F ]1/2 ≤ O( 1
s √ L+ 1 ).
When s ≥ 1, we know that s−L ≤ 1 ≤ LL/(L+1).
Then, we have 1
LsL ≤ 1 L ≤ L−1/(L+1).
Thus, we know 1/(LsL) = min(1/(LsL), L−1/(L+1)) when s ≥ 1. For the case of s ≥ 1, we can rewrite α ≤ O(s−2L/L) = O(qs−L), where q = min(1/(LsL), L−1/(L+1)), which completes the proof of Theorem 6.
Now, we prove Theorem 3 with k = 2, i.e. two-step gradient descent adaptation. We know that
β1 = α∇θ̃Lm(fθ̃)∇θLm(fθ) ᵀ, ‖∇fθLm(fθ)‖2H = ‖∇θLm(fθ)‖2.
Thus, we have∣∣β1 − α‖∇fθLm(fθ)‖2H∣∣ = ∣∣α∇θ̃Lm(fθ̃)∇θLm(fθ)ᵀ − α∇θLm(fθ)∇θLm(fθ)ᵀ∣∣ =α‖∇θ̃Lm(fθ̃)−∇θLm(fθ)‖‖∇θLm(fθ)‖
=α ∥∥∥E(xm,ym){ [fθ̃(xm)− ym] ∂fθ̃(xm)∂θ̃ ᵀ − [fθ(xm)− ym] ∂fθ(xm) ∂θ ᵀ}∥∥∥∥∥∇θLm(fθ)∥∥ =α ∥∥∥E(xm,ym){ [fθ(xm)− ym +4fθ(xm)] [∂fθ̃(xm)∂θ̃ +4∂fθ(xm)∂θ
]ᵀ − [fθ(xm)− ym] ∂fθ(xm)
∂θ ᵀ}∥∥∥∥∥∇θLm(fθ)∥∥ =α ∥∥∥E(xm,ym){4fθ(xm) [∂fθ̃(xm)∂θ̃ +4∂fθ(xm)∂θ
]ᵀ + [fθ(xm)− ym]4 ∂fθ(xm)
∂θ ᵀ}∥∥∥∥∥∇θLm(fθ)∥∥ ≤α [ O( 1
L )O(sL
√ L) +O( 1
L )O(
1
s √ L
) +O( 1
s √ L ) ] ∥∥∇θLm(fθ)∥∥ ≤α [ O(
sL√ L ) +O( 1 s √ L ) ] ∥∥∇θLm(fθ)∥∥, because L ≥ 1 ≤ [ O( qrsL√ L ) +O( qr s √ L )
] ∥∥∇θLm(fθ)∥∥, where q = min(1/(LsL), L−1/(L+1)), r = min(s−L, s) ≤O( q√
L ) ∥∥∇θLm(fθ)∥∥.
In the case of dy = 1, we have ∥∥∥∂fθ(x) ∂Wh ∥∥∥ F = (bh)ᵀgh−1 ≤ O(sL),
which has already been shown in the proof of Lemma 9. Then, we have ∥∥∇θLm(fθ)∥∥ = O( √√√√L+1∑
h=1
∥∥∥∂fθ(x) ∂θh ∥∥∥2) = O( √√√√L+1∑
h=1
∥∥∥∂fθ(x) ∂Wh ∥∥∥2 F ) ≤ O(sL √ L+ 1).
In the case of dy ≥ 1, the bound is simply scaled by a constant of √ dy .
Thus we have∣∣β1 − α‖∇fθLm(fθ)‖2H∣∣ ≤ O( q√ L ) ∥∥∇θLm(fθ)∥∥ ≤ O(qsL) ≤ O( 1 L )
because q = min(1/(LsL), L−1/(L+1)), which completes the proof for the case of k = 2.
For the case of k > 2, we only need to make sure that the bound on learning rate always holds. Fortunately, since k is a finite constant, according to what we have already showed in the proof of previous lemmas, every step of gradient descent will not change the spectral norm of the weight matrix too much: ‖4Wh‖2 ≤ O(s−L/L) for all h if s ≥ 1, and ‖4Wh‖2 ≤ O(q) for all h if s < 1, where q = min(1/(LsL), L−1/(L+1)). Thus, we may assume that the bound on learning rate always holds during the adaptation. Using triangle inequality to generalize the results from k = 2 to k > 2, i.e. for all 1 ≤ i ≤ k − 1, we have∣∣βi − α‖∇fθLm(fθ)‖2H∣∣ ≤ O( 1L). Recall that
Ẽ(α, fθ) = ETm [ Lm(fθ)− α‖∇fθLm(fθ)‖2H ] and
Mk = ETm [ Lm(fθ)− k−1∑ i=0 βi ] ,
where βi = α∇θiLm(fθi)∇θLm(fθ)ᵀ and θ0 = θ,θi+1 = θi − α∇θiL(fθi ,Dtrm). The result is straightforward now.

heading: E PROOF OF THEOREM 4
text: Theorem 4 Let fθ be a convolutional neural network with L− l convolutional layers and l fullyconnected layers and with ReLU activation function, and dx be the input dimension. Denote by Wh the parameter vector of the convolutional layer for h ≤ L− l, and the weight matrices of the fully connected layers for L− l + 1 < h ≤ L+ 1. ‖ · ‖2 means both the spectral norm of a matrix and the Euclidean norm of a vector. Define
sh = { √ dx‖Wh‖2, if h = 1, ..., L− l ‖Wh‖2, if L− l + 1 < h ≤ L+ 1
and let s = maxh sh and α be the learning rate of gradient descent. If α ≤ O(qr) with q = min(1/(LsL), L−1/(L+1)) and r = min(s−L, s), the following holds
|Mk − Ẽ(kα, fθ)| ≤ O ( 1 L ) .
Proof We prove Theorem 4 by first transforming the convolutional neural network into an equivalent fully connected neural network and then applying Theorem 3.
First of all, we assume that there are ch channels in hth convolutional layer’s output gh(x), where h = 0, ..., L− l. For fully-connected layers, define cL−l = ... = cL+1 = 1. We may represent the dimensionality of input data by x ∈ Rdxc0 . Instead of using matrices, we represent the output of every convolutional layer by a dxch length vector gh = [ gh1 , g h 2 , ..., g h dx ] , where every
ghi = [ ghi,1, g h i,2, ..., g h i,ch ] is a ch length vector contains value of different channels at the same position.
We assume that for every element ghi,j of g h i , its value is completely determined by elements of set Qh−1i , where Q h−1 i contains kch−1 elements with fixed positions in g
h−1 for a given i. In other words, every element of the output of a convolutional layer is determined by some elements with fixed positions from output of the previous layer. This is exactly how convolutional layer works in deep learning.
If we use gh−1 Qh−1i to represent the concatenation of gh−1a,b ∈ Q h−1 i , then g h−1 Qh−1i is a kch−1 length vector, where k is the kernel size. Then we have
ghi = σ(g h−1 Qh−1i Uhi )
where Uhi,j ∈ Rkch−1×ch is a kch−1 × ch matrix.
For notation simplicity, one can define a matrix Uh ∈ Rdxch−1×dxch , where every column of Uh only has kch−1 non-zero elements, and it satisfies
gh = σ(gh−1Uh)
By the property of convolutional layer, we know the following facts: • One can represent Uh by Uh = [ V h1 , V h 2 , ..., V h dx ] where V hi ∈ Rdxch−1×ch is sub-matrix
of Uh; • Every V hi contains the same set of elements as Wh, while these elements are located at
different positions; • Every V hi can be obtained by any other V hj by swapping rows;
Let’s define UL−l = WL−l, ..., UL+1 = WL+1 for the fully-connected layer and output layer. Then we can represent the neural network just as in Theorem 3 by fθ(x) = σ(σ(...σ(xU1)...UL−1)UL)UL+1, and x ∈ Rdxc0 . Now let th be the spectral norm of Uh, and t = maxh th. By Theorem 3, we know that we want α ≤ O(qr), where q = min(1/(LsL), L−1/(L+1)), r = min(s−L, s). Because every V hi contains the same set of elements, we know that every V h i has the same Frobenius norm. Because every V hi can be obtained by any other V h j by swapping rows, we know that every V hi has the same rank.
We know that
1√ r ‖V h1 ‖F ≤ ‖V h1 ‖2 ≤ ‖Uh‖2 ≤ ‖Uh‖F =
√ dx‖V h1 ‖F = √ dx‖Wh‖2
where ‖ · ‖F denotes Frobenius norm, r denotes the rank of V h1 . The last equality holds because matrix V h1 and vector W h have the same set of elements.
Let’s define
sh = { √ dx‖Wh‖2, if h = 1, ..., L− l ‖Wh‖2, if L− l + 1 < h ≤ L+ 1
and s = maxh sh. From above we know that th = Θ(sh), because sh/ √ dxr ≤ th ≤ sh. So we also have t = Θ(s). Then the conclusion is straightforward.

heading: F REVISION OF THEOREM 3 AND THEOREM 4 IN CLASSIFICATION CASE
text: We now show how to obtain similar results of Theorem 3 and Theorem 4 in classification problem, where cross-entropy loss is used instead of squared loss. We need two more restrictions in the classification case:
1. There exist matrix A and B such that gLA ≤ softmax(gLWL+1) ≤ gLB for all data points, where softmax is the softmax operation at the last layer.
2. For any data point x whose belongs to cth class, there exists a constant > 0 such that fθ,c(x) ≥ , i.e. the output of neural network has a lower bound on the true class position.
The proof is actually similar to the proof in regression case. We briefly talk about the differences here.
Firstly, in the classification case, softmax function is used at the last layer. By the first restriction, we can get rid of softmax function by introducing new matrices, which further leads to bound of the learning rate as in regression case.
Secondly, if the loss function is the cross-entropy loss, we have:
∇θLm(fθ) = E(xm,ym) [
1
fθ,cm(xm)
∂fθ,cm(xm)
∂θ ] where cm denotes the class of xm, e.g. if xm belongs to the third class, then cm = 3. fθ,cm(xm) denotes the cthm dimensional element of fθ(xm). We want a lower bound of fθ,c(x) exists, so that the gradient ∇θLm(fθ) can be further bounded. Then we can prove similar theorems just follow the steps in regression case.

heading: G PROOF OF THEOREM 5
text: Theorem 5 Let fθ be a neural network with L hidden layers, with each layer being either fullyconnected or convolutional. Assume that ‖L‖∞ <∞. Then, error(T ) = |Ẽ(T, fθ)− E(T, fθ)| is a non-decreasing function of T . Furthermore, for arbitrary T > 0 we have:
error(T ) ≤ O ( T 2L+3 ) .
Proof Recall that E(t, fθ) is defined based on f tm,θ , which is the resulting function whose parameters evolve according to the gradient flow dθtm dt = −∇θtmL(f t m,θ,Dtrm).
We actually have the following (Santambrogio, 2016):
‖4θ‖ = ‖θ0 − θt‖ ≤ O( √ t).
For simplicity and clearness, we use4 to denote the change of any vectors and matrices. Thus, we know that
‖4Wh‖2 ≤ ‖4Wh‖F ≤ ‖4θ‖ ≤ O( √ t).
Just like the proofs of Lemma 7, Lemma 8 and Lemma 9, we show that
‖4gh‖ ≤ O(th/2), ‖4bh‖ ≤ O(t(L−h+1)/2), ∥∥∥4∂fθ(x)
∂θ ∥∥∥ F ≤ O(t(L+1)/2 √ L+ 1)
by mathematical inductions; we skip the details here. Note that different from some previous theorem, here we focus on time t, and thus hide the effect of the spectral norms by treating them as constants.
Then, we have∥∥∥4(∇θLm(fθ))∥∥∥ =‖∇θtLm(f tm,θ)−∇θLm(fθ)‖
= ∥∥∥E(xm,ym){ [f tm,θ(xm)− ym] ∂f tm,θ(xm)∂θt ᵀ − [fθ(xm)− ym] ∂fθ(xm) ∂θ ᵀ}∥∥∥ = ∥∥∥E(xm,ym){4fθ(xm) [ ∂f tm,θ(xm)(xm) ∂θt +4∂fθ(xm) ∂θ ]ᵀ + [fθ(xm)− ym]4 ∂fθ(xm) ∂θ
ᵀ}∥∥∥ ≤O(tL+1 √ L+ 1).
Recall that: E(T, fθ) = ETm [ Lm(fTm,θ) ] = ETm [ Lm(fθ) + ∫ T 0 ∇tLm(f tm,θ)dt ]
= ETm [ Lm(fθ) + ∫ T 0 dθt dt ∇θtLm(f tm,θ)dt ]
= ETm [ Lm(fθ)− ∫ T 0 ∥∥∥∇θtLm(f tm,θ)∥∥∥2dt ]
and Ẽ(T, fθ) = ETm [ Lm(fθ)− T‖∇fθLm(fθ)‖2H ] = ETm [ Lm(fθ)− T‖∇θLm(fθ)‖2 ] .
Because Ẽ(T, fθ)− E(T, fθ)
= ∫ T 0 ∥∥∥∇θtLm(f tm,θ)∥∥∥2dt− T‖∇θLm(fθ)‖2 =
∫ T 0 ∥∥∥∇θLm(fθ) +4(∇θLm(fθ))∥∥∥2dt− T‖∇θLm(fθ)‖2 =
∫ T 0 2∇θLm(fθ)4 ( ∇θLm(fθ) )ᵀ + ∥∥∥4(∇θLm(fθ))∥∥∥2dt,
we have error(T ) = |Ẽ(T, fθ)− E(T, fθ)| ≤ O ( L+ 1 2L+ 3 T 2L+3 ) = O ( T 2L+3 ) by simple calculation.
On the other hand, observe that
Ē(T, fθ) = ETm [ Lm(fθ)− ∫ T 0 ∥∥∥∇θtLm(f tm,θ)∥∥∥2Hdt ] ,
Ẽ(T, fθ) = ETm [Lm(fθ)− T ∥∥∥∇θLm(fθ)∥∥∥2
H ].
We let
G(τ) = ∫ τ 0 ∥∥∥∇θtLm(f tm,θ)∥∥∥2dt, and assume that∇θtLm(f tm,θ) is continuous at t = 0. Then, we have G′(τ) = ‖∇θtLm(f tθ)‖2.∥∥∥E(T, fθ)− Ẽ(T, fθ)∥∥∥ =∥∥∥ETm [∫ T 0 ∥∥∥∇θtLm(f tm,θ)∥∥∥2dt− T∥∥∥∇θLm(fθ)∥∥∥2H ]∥∥∥
= ∥∥∥ETm(G(T )− T ·G′(0))∥∥∥,
where TG′(0) = G(0) + TG′(0) (note that G(0) = 0) is a first order approximation to G(T ) at τ = 0. When T = 1, G(T ) − TG′(0) can be taken as a local truncation error (i.e., the error that occurs in one step of a numerical approximation). When T increases, the difference is no better than the global truncation error (in T steps):
∥∥∥G(T )− T∑ i=0 (i− (i− 1))G′(i) ∥∥∥ =∥∥∥ T∑ i=0 ∫ i+1 i ∥∥∥∇θtLm(f tm,θ)∥∥∥2 − ∥∥∥∇θiLm(f t=im,θ)∥∥∥2dt∥∥∥ ≈ ∥∥∥ T∑ i=0 ∫ i+1 i 2 · 4itLm(fm,θ) · ∇Lm(f tm,θ)dt ∥∥∥,
where 4itLm(fm,θ) = ∇θtLm(f tm,θ) − ∇θiLm(f tm,θ) as shown previously , i is the i-th time step, and G′(i) is the gradient of G at time step i. Now we can see that ∥∥∥E(T, fθ) − Ẽ(T, fθ)∥∥∥ highly relates to the difference between ∇θtLm(f tm,θ) at different time steps (i.e. 4itLm(fm,θ)), ∇θtLm(f tm,θ) and T . The first two terms relate to how flat or sharp the hyperplane of Lm(fm,θ) is near t = 0. We can wrap it as a constant C0(L, t = 0). Then, the error is at least C0(L, t = 0) ·O(T ). For the hyperplane smooth enough, we can further get a first order approximation of4itLm(fm,θ) and yieldC(L, t = 0)O(T 2), whereC(L, t = 0) can be analogized as the second order derivative of L.

heading: H SOME EXPERIMENTAL DETAILS
text: H.1 IMPLEMENTATION OF CLASSIFICATION FOR META-RKHS-II
As we mentioned earlier, our proposed energy functional with closed form adaptation can not be directly applied to classification problem. We handle this challenge following Arora et al. (2019). For a dy class classification problem, every data x is associated with a Rdy one-hot vector y as its label. For C classes classification problem, its encoding is C dimensional vector and we use −1/C and (C − 1)/C as its correct and incorrect entries encoding. In the prediction, Y tr is replaced by the encoding of training data. fθ(x) is replaced by fθ(x)ᵀ[1, ..., 1] ∈ Rn×dy for dimension consistency. During the testing time, we compute the encoding of the test data point, and choose the position with largest value as its predicted class.

heading: I EXTRA EXPERIMENTAL RESULTS
text: I.1 COMPARISON WITH RBF KERNEL
One interesting question is, without introducing extra model components or networks, what will the results of other kernel be? We provide the results of using RBF (Gaussian) kernel here: 42.1± 1.9 (5-way 1-shot) and 54.9 ± 1.1 (5-way 5-shot) on Mini-ImageNet, 32.4 ± 2.0 (5-way 1-shot) and 38.2± 0.9 (5-way 5-shot) on FC-100, which are worse than the NTK based Meta-RKHS-II, showing the superiority of using NTK.
I.2 MORE RESULTS ON OUT-OF-DISTRIBUTION GENERALIZATION
We provide some more results on out-of-distribution generalization experiments here. From the results we can find that the proposed methods is more robust and can generalize to different datasets better.
I.3 MORE RESULTS ON ADVERSARIAL ATTACK
We now show some more extra results on adversarial attack in the following figures. Consistent to the results in main text, we can find that our proposed methods are more robust to adversarial attacks.
I.4 IMPACT OF GRADIENT NORM IN META-RKHS-I
In this experiment, we compare between our proposed Meta-RKHS-I and Reptile. We evaluate the trained models with different adaptation steps in testing-time. The comparison is shown in Figure 7. As we can see, our Meta-RKHS-I always gets better results than Reptile, which supports our idea that the learned function should be close to task-specific optimal and have large functional gradient norm. These two conditions together lead to the ability of fast adaptation.
I.5 IMPACT OF NETWORK ARCHITECTURE FOR DIFFERENT META-LEARNING MODELS
In this section, we compare different meta-learning models with feature channels of 100 and 200 of the CNN network structure with 4 or 5 CNN layers respectively.

Review summaries:
In this paper, the authors view MAML from the lens of Reproducing Hilbert Kernel Hilbert Spaces (RKHS) by applying tools from the theory of Neural Tangent Kernels (NTKs). Based on these insights, they develop two meta-learning algorithms that avoid gradient-based inner-loop adaptation. Their algorithms are theoretically grounded and exhibit improved empirical performance.  

The authors propose two meta-learning algorithms in the reproducing kernel Hilbert space (RKHS) induced by the recently proposed Neural Tangent Kernels (NTK). The authors show how their algorithms obviate an explicit inner loop or task-adaptation step in the meta-learning training phase. In first algorithm, no explicit adaptation function is used, whereas in the second, a close form adaptation function which invokes the NTK is proposed - which is a simpler adaptation than that of MAML and hence, offers computational efficiency. The work is interesting and  supported by theory inspired from the NTK theory, and adds to the newly expanding literature in the use of kernels in meta-learning (unlike the authors’ claim in the introduction, theirs is not the first meta-learning paradigm in the RKHS cf (Wang et al 2020, Cerviño et al 2019)). The authors perform extensive experiments on regression and classification datasets and compare their results with other MAML-type algorithms. The experimental results do not show significant gains in terms of performance over the existing MAML approaches, except in the case of out-of-distribution datasets and adversarial attacks, where it is shown to outperform the others. The performance similarity to other methods is not surprising since the proposed approaches can be seen as an efficient approximations of the MAML.

This paper mainly deals with the computational issues of Model Agnostic Meta-Learning (MAML). Specifically, it proposes two meta-learning algorithms where the hypothesis class (i.e. the mapping function set) is defined in RKHS induced by NTK. Extensive experimental studies on many tasks (i.e. regression, few-shot image classification, robustness to adversarial attack, and out-of-distribution generalization) illustrate its superiority compared with other baselines.