Effective Meta-Regularization by Kernelized Proximal Regularization
Authors: Weisen Jiang
James Kwok
Yu Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: meta-learning

Abstract: We study the problem of meta-learning, which has proved to be advantageous to accelerate learning new tasks with a few samples. The recent approaches based on deep kernels achieve the state-of-the-art performance. However, the regularizers in their base learners are not learnable. In this paper, we propose an algorithm called MetaProx to learn a proximal regularizer for the base learner. We theoretically establish the convergence of MetaProx. Experimental results confirm the advantage of the proposed algorithm.

Effective Meta-Regularization by Kernelized Proximal Regularization
Weisen Jiangh James T. Kwok?, Yu Zhang
Department of Computer Science and Engineering, Southern University of Science and Technology *Department of Computer Science and Engineering, Hong Kong University of Science and Technology
3 Peng Cheng Lahoratory
{ujiange iangar, j james ust ust.hk, .k, yu. au.zhang.... ust@gmail com
Abstract
We study the problem of meta-learning, which has proved to be advantageous accelerate learning new tasks with a few samples. The recent approaches based on deep kernels achieve the state-of-the-art performance. However, the regularizers their base learners are not learnable. In this paper, we propose an algorithm called MetaProx to learn a proximal regularizer for the base learner. We theoretically establish the convergence of MetaProx. Experimental results confirm the advantage of the proposed algorithm.
Introduction
Humans, by leveraging prior knowledge and experience, can easily learn new tasks from handful of examples. In contrast, deep networks are data-hungry, and a large number of training samples are required to avoid overfitting. To reduce the aabor-intensive and time-consuming process of data labeling, meta-learning (or learning to learn) [3 37] aims to exact meta- meta-knowledge from seen tasks to accelerate learning on unseen tasks. Recently, meta-learning has been receiving increasing attention due to its diverse successful applications in few-shot learning 41
12
39
35). hyperparameter
optimization 14, neural architecture search 24 44
and reinforcement learning 29).
Many meta-learning algorithms operate on two levels. A base learner learns task-specific models in the inner loop, and meta-learner learns the meta-parameter in the outer loop. popular class of
25 TI 38), such as the well-known MAML (12) learns model initialization such that good model for unseen task be learned from limited samples by a few gradient updates. However, computing the meta-grudient requires pookkpropagating through the entire inner optimization path, which infeasible for large models and/or there are many gradient steps. During testing, it is common for MAML's base learner to perform many gradient steps seek a more accurate solution [12. However, for regression using a linear base learner and square loss, we will show that though the meta- meta-learner can converge to the optimal meta- meta-initialization, the base learner may overfit the training data at meta-testing.
algorithms is based on mntaainitializzaiin 12
in which the base learner learns the task-specific model by minimizing the loss with proximal regularizer (a biased regularizer from the meta-parameten Denevi et al. [7] uses a linear model with efficient closed-form solution for the base learner. However, extending nonlinear base learners requires computing the meta-gradient using matrix inversion, which can be infeasible for deep networks (28).
Another class of meta-learning algorithms is based on meta-regularizatton 28 43
To introduce nonlinearity to the base learner, recent approach is to make of the kernel trick. For example, R2D2 [4] and MetaOptNet 22 use deep kernels 42 in meta-learning for few-shot
"Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

classification. Specifically, the deep network is learned in the meta-leamer, while a base kernel is used in the base learner. Though they achieve state-of-the-arg performance, their base learners use Tikhonov regularizer rather than learnable proximal regularizer as in meta-regularizztion methods.
As learning meta-regulariizatoon has been shown to be effective in linear models for regression 7 and classification [8). in this paper we propose a kernel-based algorithm to meta-learn a proximal regularizer for a nonlinear base learner. After kernel extension, the learnable function in the proximal regularizer is a function in the reproducing kernel Hilbert space (RKHS) induced by the base kernel. critical point of the meta-loss and its global convergence is also established. Experiments on various benchmark regression and classification datasets demonstrate the superiority of the proposed algorithm over the state-of of-the-arts.
The proposed algorithm guaranteed to converge
Notations. Vectors are denoted by lowercase boldface, and matrices by uppercase boldface. For a vector x, ||ж|| = =vE, 2² and diag(x) constructs a diagonal matrix with x on the diagonal. || lI is the norm on the Hilbert space H. [O o²) is the univariate normal distribution with mean zero and variance o², (Vmm, Σ) is the multivariate normal distribution with mean and covariance matrix Σ.
2 Related Work
In meta-learning, a collection T of tasks are used to learn meta- parameter O and base learner's parameters {Wi..... W[ Each task T is sampled from a given distribution p(T), and has support set S. ((x, Wi) i I,..., n.. and a query set Q- {(x;, yi)
1...., " where Rdd the features and y the labels. Let f(; w) be model parameterized by and C(D:w) = E (x.))) {[f(x; w). be the supervised loss on data set D, where ((---) is loss function that is assumed to be convex w.r.t. the first argument. In each meta-training iteration, a batch B of tasks are randomly sampled from T. The base learner takes task t from and the meta--araaeeeer 0 to build the model ((::,)). After all tasks in the batch are processed by the base learner, the meta-learner minimizes the loss ΣTEB C(Qr: w.) w.r.t 0, and the iteration repeats. During meta-testing, given an unseen task r' ~ p(T), model f(; w is similarly learned from Sy, and O. Finally, its performance is evaluated on QT'.
Popular meta- learnig algorithms usually construct the task specific model by: (i) meta- initialization [12] 25
or (iii) metric learn- ing [35] 4 22 27]. A representative meta-learning algorithm based meta-inittillizatinn is MAML [2]. Its base learner learns w, by gradient descent from learnable initialization. Comput- ing the geta-ggadieet VoL(Qr; w,) needs to back-propagate through the inner optimization path and involves second-order derivatives. This can be expensive for large models and/or when there are many gradient steps.
II 38) (ii) regulaaauuiiization 7 8
28
43
For meta-learning algorithms based on meta-regularizzatio,, the base learner learns w, by minimizing proximal regularized loss
L(S+: w) 2 011
(1)
where À > O is the regularization parameter. The meta- meta-gradie cesta-gradient can be computed directly from the learned w,, without back- sooekppoppggting through the inner optimization trajectory (28). Though more efficient, this still takes (ng) time (28). where is the number of parameters in A.
Metric learning methods have been widely studied in few-shot learning meta-learner maps raw samples to an embedding space via a deep network, then feeds the em- beddings the base learner train simple task model. Typical models include non-parametric prototype classifier (ProtoNet (35)) ). linear models like ridge regression (R2D2 (4). SVM classifier (MetaOptNet-SVV. (22), and softmax classifier (ANIL (27)). In particular, the base learners in R2D2 41 MetaOptNet [22] and DKT [26] seek solutions in the dual space, which achieve state-of-- the-art performance. However, their base learners use Tikhonov regularizer, which is not learnable as in meta-regularizzation approaches.
39 35
22 27
The
3 Meta-Regularization by Kernelized Proximal Regularization
3.1 Meta-Initialization versus Meta-Regularizatiin
In this section, we consider a simple regression setting with linear model and square loss. Each task is linear regressor with parameter w E _ We assume that each input x is sampled

---

end for Algorithm 1 MAML. [12].
Algorithm CommonMean 77.
Require: step size and batch size 6;
Require: hyperparameier step size yjt, batch size b;
1: for forf-= 1, 2........ do
1: for = 1, 2.3.... do
2: sample batch B3 of tasks from p(T);
2: sample batch B₁ of tasks from p(r);
3: base learner:
3: basc learne::
4: for r E B, do
4: for B, do
5: w! 1b
X (X, t/ y-);
5:
W (prox) minw 211x, w y- 2 -
g+ = -x Vv.(X wlal
y)
6:
g+ Ex.,)ER Vo, (x w! y)²
7: end for
7: end for
meta-learerr: 3/141 = 1/. % Eres, gri
8: meta-learner: oc+1 = 0g 7 E Br:
9:
9:
end for
from (O, I)),, and the output is obtained as x w +. where & ~ (O0 o²) is the random noise. We compare two representative meta-leaming algorithms: (i) MAML [12] which is based on meta-initializztion and performs one gradient descent step in the inner loop of the bilevel optimization problem; and (ii) learning around a common mean (denoted CommonMean) [7 which is based on meta-regularizztion. It learns the model parameters for task r by minimizing the loss with proximal regularizer around the meta-parant pera-parameter A:
( arggmnn Σ (Xi,))ES 2 (w (WTxx x, -1) 1x,-3)² ||w -| =XI++X
1
(2)
W
where X [xX] [x]x...;xxT]] is the sample matrix from Sr, and ( Vn. is the corre- sponding label vector. Algorithms [and]] show MAML and CommonMean, respectively, for this problem.
Recently, Balcan al. study the convex online meta-learning setting and show that both approaches achieve the same average task regret. Here, we consider the offline setting. First, the following Proposition shows that both MAML and CommonMean converge to the same mneaa-aarameter. All proofs are in the appendix.
Proposition 1. Let 11tt. Assume that 1/02. Both th; in MAMI. (with one inner gradient step) and 02 in CommonMean converge to w E, w
The following Proposition shows that w in Proposition Dis also the best ub (for MAML) or (for CommonMean) with the smallest population risk for this meta-learning problem.
Proposition 2. Assume that y < 1/02 We have w == argming E,Es, Bq- Ex.iee (x w 3)² ")²==rgminn, E, L...s,B.EEx.nup (x' w((d)) -y)
(proe)
During meta-testing, we sample a task r' ~ p(T) with parameter wi.. Let X,, be the sample matrix from Sy,, and yr the corresponding label vector. We assume that X,, is full rank and n., d. To simplify notations, we drop the subscript r' in the following. Let the singular value decomposition of X be UΣVT (where Σ diag((v) (pn .,.,, D)), and V be V's orthogonal complement.
As only forward passes are needed, it is common for the base learner MAML to perform multiple gradient steps [2] With the convex loss and linear model here, the base learner obtain globally optimal solution w (gdao) directly (which is equivalent to taking infinite gradient steps). As is common few-shot learning, the number of support samples much smaller than feature dimensionality. Hence, ((dee)) not unique but depends on the learned initialization. (gdso) (a(gdoc) + V ((sle)))) For gradient descent, its update direction X XT(Xw y)
(gdeo) be written as
VEUT [Xw y) always in the span of V and so (ggdo)) remains unchanged.
Let w and 0 be written as w = Va V-b* and A = Vag + V- Moreover, let à = a² and b bo -b*.
Proposition (117). Assume that r minisjsn, 1/v3. We have Eg (( 1| 1101|111bbbl 11b (adoc)
b" b+1 + == m
o where the expectation is over the label noise vector g.
3

---

(y- For CommonMean, using the Woodbury matrix identity, w(prox) = 0 0+x -XXXXXXXXX "y (60 = V(a, + ΣU (AI "y X0)) + b = Va + V b b(Peus) where b (PrOx) = bo-
Assume that w ((ddo)) is initialized with O. Since ((ddoo) remains unchanged, w/ and w<ddoo) only differ the components lying in the column space of V.
Proposition 4. (ellw(pra)) wI |] + شو
+ + X+t?
Aā,
2 (1e7))t
de
where the
expectation is over the label noise vector &.
As can be seen, when the labels are noise-free (o 0), w ((doo)) performs better than w (prx)) However, when the labels are noisy, as n2 < d. gradient descent always converges to zero training error and overfits the noisy labels. On the other hand, the estimation error of w/rmx) equals to that of ((goo)) when X = O. For X > 0, it trades off between fitting the noisy labels (the last term in Proposition 4 and introducing an estimation bias of a" (the second term in Proposition T..
3.2 Proposed Algorithm
It is straightforward to use the dual formulation for the CommonMean algorithm. When the square loss is used as ((--). ), it is easy to see that the dual variable has the closed-form solution
a+ = A )- X,0)-
(3)
Compared with the primal formulation, we only need to invert n, n, matrix (instead of the d x d matrix AI + X.X,), In meta-learning, usually ns < d (e.g.. fy = 5). From the dual solution Oxy,
the primal solution can be recovered as W, = 0 + A x ar Given a query example (x, y) Qr. the model predicts " xx w, x 0 X- x x a,. The loss gradient is Vo v) ViE(ū, y) V Voğ. where Vi((ū, v) denotes the gradient w.r.t. the first argument, and Voj = X - X (V X, Voa, = - xI,, thus very low (O + +dd)).
The complexity of computing Vel y) is
The dual formulation also allows introduction of nonlinearity with the kernel trick. Based on deep
kernels [2). recent sfate-of-thh-an (R2D2 [4] MetaOptNet (22] and DKT (26) propose to a base kernel in the base learner and update the deep network in the meta-leamer. However, their regularizers are not learnable. We consider to learn proximal regularizer. An input x is mapped to z = NN(x; Φb) in an embedding space& E via a deep network parameterized by Φ. With the dual formulation, 0 in (2) allows extra flexibility over (22 4 7). Specifically, 0 becomes function fe E H. the reproducing kernel Hilbert space (RKHS) corresponding to a given kernel function K. on H by minimizing
Ex 8. The base learner obtains a model f,
> I1 foll
min Σ (x..)es. {(f(z;), vi) fEH
(4)
where || -. || is the norm on H. By setting fe O, this recovers the ofate-of-the-arts of MetaOpt- Net [22). R2D2 [4], and DKT [26). However, Proposition 4 suggests that a good So provides good meta-knowledge by biasing the task model.
By the representer theorem [34), the solution of is fr = fo + Ex.,,)ss a, Kr,KK,,, where K2. -( E H. and a = -[ar,ii.... ar, is obtained from the convex program
min Σ (x..)ES ((F a,, K( Z,)a..
(5)
where Z, = =ZT : 2T, and KCZ,,, Z,, is the kernel matrix. Note that the hyperparameter À in (4) is absorbed into Z as the network is learnable. With the square loss, the dual solution of (5) is Qr (I+ KC,,, Z,)) - functions, the dual problem has no closed-form solution, but this has only n. variables (which is usually small) and can be solved efficiently.
fo(Z,)) where (Z, = [fo(z]), :..; Sol (Zn,)] )} For general loss ..;
After the base learner has obtained the dual solution a,, the meta-learner updates fo and network parameter & by one gradient descent step on the validation loss Exx,)e E(ū, v). where = (z) = fo(z) + IC(Z,, 2) ax. Using the chain rule, (i v) = Vi y) (0. The first component V, y) can be computed directly and the second component
( = V (ooz)) + (V (((,,,))) Ta ar +
(K z).
(6)

---

Both V folz)) and V Via, z) can be obtained by direct differentiation. By the chain zı)i...; Zn,)] E puttn; is the input to the dual problem. V(o can be directly computed. When the -(I K(Z,, 2,
rule, V (( = vp,, oi where p =
square loss is used, Vp
10a For a general loss, ax, is obtained by solving the convex program. Hence, ar depends implicitly on p and Vpa, can be obtained by implicit differentiation. Denote the dual objective in (5) by @²
0pda @p0a (P,, g(p.a). By the implicit function theorem [32], vpa, - a,) where Vag(p, 0²
E(EENEES Vie(f, (2i), (,)(( 2,) 2,) + K(Z,, Zr). Z,) (Zn((+(Zn,) In,,)), v where & is the Kronecker product. The whole procedure, called MetaProx, is shown in Algorithm Let nig and no be the numbers of parameters in Φ and 0, respectively. Computing V(e () (ū. takes (n + (no + nz)) time, which is linear in the number of meta-parameters. This is lower than the other meta-learning algorithms (e.g., MAML [2] with single step takes (n time, iMAML 28] (n CommonMean 7 ))))).
0pda (p,
(K( 8 a +v OI I+00,, D vn): Vie(S= (Zn D,yn,
Algorithm 3 MetaProx.
Require: step size nh. batch size b;
1: for = 1,2, 2.. . do
2: sample a batch B, of tasks from T;
3: base learner:
4: for r E B, do
5: 2; NN(x;; Φ.) for each (xi, yi) Sr:
6:
(z; a)) fo, (z) + K(Z,, a denote the task model w.r.t. dual variables;
7:
argmin E Ex..nees. {(J-(z; a:), vi) + a K. Z-)ax 8: gr Ex.))E V( v). where ύ = f- (z; and z NN(x; db);:
9: end for
10: meta-leamer: (0 x = (0,, Φ1) V EreB,Bri 8y:
end for
We consider extension from regression to N-way classification. For task т, the base learner learns the model f- = 15th : )(N) for the N classes by minimizing
Classification.
\ Σ c=1 [
min ge.,.... EN (x..)es, E(ý;, vi)
Soto 11
where ÿ. = [f()) (z.)....: (2,) and fom,...... ..... fa(N) E H are functions learned by the meta- learner. By the representer theorem 34 f. [fgm: + KC(Z,.)) )) Ta. :......::: faiNs fain++ + K(Z,..) (N))
(1) where (a :.... axa1 ( is obtained from the convex program min, ()) (1)) ((...... Σ Ex.gies e(y,, vi)
De a (c] K(,, Z, (()) As latt) : { N', and both N n, are typically very small (e.g., N = 5, n, = in 5- 5-shot classification), this convex program can be solved efficiently. The meta-learner then updates the network parameter and (Saca,....,Je descent step on the validation loss E(x,yi). E(y,, Computing V
by one gradient eyy y) takes
((Nn,,) + (Nn.)?(n, + E )) time, where Nate) is the size of g(c) This is again linear in the number of meta-parameters and thus very efficient.
3.3 Theoretical Analysis
+ ErET Eixnjee, {(o(z; ar), be the empirical loss of Eysp(r) Exx,)eq {(Jo(z; v)), where z NN(x: Φ). With the linear kernel and square loss, the dual solution (3) is affine in the meta veta--arameter, and so is the primal solution w. = 0 x-X ar, Thus, the meta-loss Cmetu (0, Φ) is convex and convergence follows from convex optimization S 77 After introducing nonlinearity, the meta-loss is no longer convex. The following introduces Lipschitz-mmoothness assumptions, which have been commonly used in stochastic non-convex optimization [15 31 and meta-learning in non-convex settings |I| 43).
Let Cmeta ())
5

---

Assumption 1 (Smoothness). (i) The deep network NN Φ) is Lipschitz-smooth. i.e. V NNNxN Φ) V& 6')ll S $1 1|Φ 6'll with a Lipschitz constant 31 > (ii) the kernel K(2, z is Lipschitz-smooth w.r.l (z, ) (ii) so(z) is Lipschitz-smooth w.r.t (0. 2); (iv) (e((ū:)) y) is Lipschitz w.r.l D. i.e. [V(((.y) VPPll' y) S Balú 8' with a Lipschitz constant B2; (v) Bx-TIIV Lixapeo- {(So ), V (0 O 2)1² of where
T~ T denotes uniformly sample lask from T.
The following Lemma guarantees smoothness of the meta-loss.
Lemma 1. (0, Φ) is Lipschitz-smoott w.r.t. (0, () with Lipschitz constant Bmetu- Theorem 1. Let the step size be Tt miT(//TT, 1/23mn)). Algorithm satisfies minssesT V (o. (0, 30)1= O (c2/VT), where the expectation is taken over the random training samples.
This rate is the same as MAML M1 [9] and Meta-MinibatchProx
For MAML with J > 1 gradient steps, et al. 19] assumes that the step size in the inner loop of the order 1/J. This slows down inner loop learning when / is large. On the other hand, MetaProx does not have this restriction, as its meta meta- graddent depends only on the last iterate rather than all iterates along the trajectory.
43
Next, we study the global convergence of MetaProx. Prior work 13 focus on the case where fimeta (,, Φ)) is strongly convex in (0, <). This can be restrictive deep learning. We instead only require {V: y) to be strongly convex in ÿ. This assumption is easily met by commonly-used loss functions such as the square loss and logistic loss with compact domain. A recent work [40] studies the global convergence of MAML in wver-parameterized neural networks. pnee-parameterizztion is closely related to the assumption of uniform conditioning [88 23
Assumption 2 (Uniform conditioning (23). A multivariable function M(0, o) is p-uniformly conditioning if its tangent kernel [[8] satisfies min Amin V (o. V ( M(0 Φ)) 2
J > O, where Amin() (-) is the smallest eigenvalue of the matrix argument.
Assume that the loss e(-,-) -) is p-strongly convex w.r.t the first argument and Assumption holds. Let Xr. be the jth query example of task т, Zrg be its embedding, and Pr.)
So(2.)) + k(Zr; Toa ar be its prediction, where a the dual solution. Let M(0, Φ))
[Dm..i....; -.li..-;;ra. :'r,,, .n.i..... ........ Bry .1rr..1.---..- .1:..-iBryyj.n be an auxiliary function which maps the meta-paran meta-parameter to predictions on all query examples. The following Theorem shows that the proposed algorithm converges to a global minimum of the empirical risk Lmena (0, Φ) at the rate of O / /T). The rate improved to exponential the meta-learner adopts full gradient descent.
Theorem 2. Assume that M(O, +) is uniform conditioning. (i) Let ÿ)- minn -miiii//// (1/VT, 1/28mm)) Algo-
rithm 3 satisfies mini sisT m.L.moto (0., Φb) min (0)6) fm (f, ()) O (d2/vT). where the expecia- tion is taken over the random training samples. (ii) Let 17 < min(1/28... 4/T/pu) and B, T.
Algorithm 3 satisfies Lieea (O,, Φ.) min .) mieet (0, +) = O(1 - -///1/T))*)
Experiments
4.1 Few-shot Regression
Data sets. Experiments are performed on three data sets.
(i) Sine. This is the sinusoid regression problem in [12.. Samples w's are uniformly sampled from [-5, 5). Each task 7 learns a sine function y a, sin (ar + br) + E, where a, E [0.1,5), b, E [0. T). and .N(0,, o is the label noise. We consider both of 0 (noise-free) and of = 1. In addition to the 5-shot setting in [12). we also evaluate on the more challenging 2-shot setting. We randomly generate a meta- seeta-training set of 8000 tasks, a meta-validation set of 1000 tasks for early stopping, and meta-testing set of 2000 tasks for performance evaluation.
(ii) Sale. This is real-world dataset from (36], which contains weekly purchased quantities of 811 products over 52 weeks. For each product (task), a sample is to predict the sales quantity for the current week from sales quantities in the previous 5 weeks. Thus, each product contains 47 samples. We evaluate on the 5-shot and 1 I-shot settings. We randomly split the tasks into a meta-training set of 600 tasks, meta- meta-validation set of 100 tasks, and a meta-testing set of 111 tasks.

---

(iii) QMUL, which is multiview face dataset [16] from Queen Mary University of London. This consists of grayscale face images of 37 people (32 for meta-training and 5 for meta-testing). We follow the setting in (26) and evaluate the model on 10-shot regression. Each person has 133 facial images covering viewsphere of +900 in yaw and +330 in tilt at 10" increment. A task a trajectory taken from the discrete manifold for images from the same person. The regression goal is to predict the tilt given an image. In the in-range setting, meta-training tasks are sampled from the entire manifold. In the more challenging oul-of-range setting, meta-training tasks are sampled from the sub-manifold with yaw in [-90", 0 In both settings, meta-testing tasks are sampled from the entire manifold. We randomly sample 2400 tasks for meta-training, and 500 tasks for meta-testing. As in [26]. we do not use a meta-validation set since the dataset is small.
Network Architecture. For Sine and Sale, we use the network in 12). which is a small multilayer perceptron with two hidden layers of size 40 and ReLU activation. For QMUL, we use the three-layer convolutional neural network in [26). The embeddings are always from the last hidden layer. We use
simple linear kernel as base kernel, and fz 0 z.
Implementation Details. We use the Adam optimizer (0 with learning rate of 0.001. Each mini- batch has 16 tasks. For Sine and Sale, the model (ø and fe) is meta-trained for 40, 000 iterations. To prevent overfitting on the meta-training set, we evaluate the meta-validation performance every 500 iterations, and stop training when the loss on the meta-validation set has no significant improvement for 10 consecutive evaluations. For QMUL, we follow (26) and meta-train the model for 100 iterations. We repeat each experiment 30 times. For performance evaluation, we use the average mean squared error (MSE) on the meta-testing set.
Baselines. On Sine and Sale, we compare MetaProx with CommonMean 7 MAML 12]
MetaOptNet-RR [22], Meta-MinibatthPPro 43 and iMAMI. 28). CeamonMean is a linear model, and MetaOptNet-RR is equivalent to MetaProx when fe = O. Following [12. we set the num- ber of inner gradient steps for MAML to during meta-training, and 20 during meta-validation and meta-testing. Both Meta-MinibatchProx (43 and iMAML (28) are meta-regularizztion approaches. For QMUL, compare MetaProx with the baselines reported in 226 (namely, DKT (26.. Feature Transfer (10), and MAML). As further baselines, we also compare with Meta-MinibatchProx and MetaOptNet- to evaluate the improvement of MetaProx due to the learnable fe.
IMAML
Meta MinibatchProx
MetaOppNettRR
MetaProx
meta-loss
mete--oss
Rette-i Aoss mn
To loss
500 1000 training time
500 training
1000
training
training time
(b) 2-shot, of
(c) 2-shot, of
(d) 5-shot, of O.
(e) 5-shot, of 1.
Figure 1: Convergence curves for few-shot sinusoid regression. Best viewed in color.
Results on Sine. Figure shows the convergence curves of MetaProx and the baselines. We do not show the convergence of CommonMean, as it does not neural network backbone as the other methods. As can be seen, MetaProx converges much faster and better than the won-kernel-based methods (MAML, iMAML and Meta-MiinibatchProx). In the 2-shot settings, MetaProx converges to a loss smaller than that of MetaOptNee--RR.
Figure 2 shows the learned functions on 2 meta-testing tasks (ri with (a 4.6, b = 3.2) and 72 with (a = 3.7, = 0.5)) in the 5-shot setting and more challenging 2-shot setting. As can be seen, MetaProx always fits the target curve well. Though MAML, iMAML and Meta-MiiniatchPProx can fit the support samples, it performs worse in regions far from the support samples. This is especially noticeable in the 2-shot setting.
Table shows the MSE on the meta-testing set. Obviously, CommonMean (a linear model) fails in this nonlinear regression task. MetaProx and MetaOptNet-RR significantly outperform the other baselines. MetaProx (with the learned fo) performs better than MetaOptNet- particularly when the data is noisy.
7

---

support points
ground fruth
MAMIL
IMAME
Mata- Matta-MribatccPrrx
MataOptNect-F AR
MetaProx
50
5.0
5.0
25-
25
25
-10
-10
(a) task 71, 0g
0.
(b) task T1, 0
2
1.
(c) task 72, de
(d) task 72, 0 1.
24
2.5
2.
25
50
-10
(e) task Ti = 0.
(f) task T1, of
(g) task T2, of 0.
(h) task 72. oz
Figure 2: Sinusoid regression: Two meta-testing tasks Tι and 72 with different oe in 2-shot (b) settings. Best viewed in color.
(a)
(c) and 5-shot (e)(h)
Table 1: Average MSE (with 95% confidence intervals) of few-shot regression the Sine and Sale datasets. (The confidence intervals in Sale experiments are =0.001 for methods)
Sine (2-shot)
Sine (5-shot)
Sale
noise-free
noisy
noise-free
noisy
1-shot
5-shot
CommonMean
7
4.58 = 0.07
4.59 + 0.07
4.29 + 0.06
4.31 + 0.06
0.090
0.074
MAML
12
1.24 0.12
1.91 0.13
0.41 + 0.03
1.15 0.05
0.069
0.063
iMAML 28
1.12 0.11
1.84
0.38 + 0.02
1.02 1 0.05
0.068
0.063
Meta-MimibatchProox
43
1.15 0.08
1.87 0.09
0.37 + 0.02
1.01 0.03
0.081
0.064
MetaOptNet-RR
22
0.18 0.01
0.79 0.01
0.01 0.00
0.19 119++00
0.088
0.068
MetaProx (propused)
0.11 0.01
0.01
0.01 0.00
0.13 0.01
0.061
0.060
Table Average MSE (with 95% confidence intervals) of few-shot regression on QMUL (10-shot).
Results of the first four methods are from 126].
method
in-range
out-of-range
Feature Transfer 10
0.22 + 0.03
0.18 = 0.01
MAML [12]
0.21 = 0.01
0.18 0.02
DKT + RBF 226
0.12
0.14 = 0.03
DKT Spectral 26
0.10
0.11 0.02
Meta- Siia-MinibatccProox
43
0.171 + 0.022
0.193 + 0.025
MetaOptNet-RRR
0.021 0.007
0.039 0.009
MetaProx (proposed)
0.012 + 0.003
0.020 + 0.005
As can be seen from Table J the linear model (CommonMean) performs poorly as expected. MetaProx again outperforms the other baselines, particularly in the more challenging 1-shot setting.
Results on Sale.
Results on QMUL. Table 2 shows that MetaProx achieves the lowest MSE and the kernel methods RBKT+RBF, SpT+raa,,a,, MetaOptNett and MetaProx) perform better than non- non-kernel- nen-kereel-basd methods (Feature Transfer. MAML, and Metta-MinbaachhPoo)) MetaProx with the learnable fo reduces the errors MetaOptNNet-RR by half.
4.2 Few-shot Classification
Datasets. We use the standard 5-way K-shot setting (K or on the mini-ImageNer [39 dataset, which consists of 100 randomly chosen classes from ILSVRC-2012 [33. Each class contains 600 84 x images. We use the commonly-used split in (30 the 100 classes are randomly split into 64 for meta-training, 16 for meta-validation, and 20 for meta-testing.
use the Conv4 in [12 39 and ResNet- 12 in [22] As the cosine similarity is more effective than E distance in few-shot classifica- tion (,), we adopt the cosine kernel K(zz, z') cos(z, z') as base kernel, where z is the em- bedding of sample x extracted from the last hidden layer as in regression. For each c = 1.... 5, fore: [Kqu).... Ka g(e) is weighted prototype classifier the embedding space, where
Network Architecture. For the network backbone,

---

Table 3: Accuracies (with 95% confidence intervals) of 3-way few-shot classification on mini- ImageNer using Com4. means that the result is obtained by rerunning the code with our setup here. Other results from their original publications (Result on the 5-shot setting is not reported in iMAMI. 28
method
1-shot
5-shot
MAML
12
48.7 + 1.8
63.1 + 0.9
FOMAML
12
48.1 + 1.8
63.2 = 0.9
REPTILE 25
50.0
66.0 0.6
iMAML 28
1.8
Meta-MinibatchProx
5.9
67.4 0.9
ANIL 22
4.4
61.5 0.5
R2D2
4
65.4 4.2
ProtoNet
35
4.8
68.2
MetaOptNeetSVMMiin)
22
0.9
66.9
MctaOptNett-V
22
5.9
67.2 0.6
MetaProx (proposed)
52.4 1.0
68.8 - 0.8
(2) ( (5) are the class centroids computed from S,, and the weights (0()) ..... ((5) are meta- parameters.
Baselines. We compare MetaProx with the state-of of-the-arts: (i) meta instiaiiiiili: MAML [22 and its variants FOMAML [12), and REPTILE [25]: (ii) meta-regularrzzttinn iMAML 22 and
Meta-MinibatchProx 4 and (iii) metric learning: ANIL (27). R2D2 (4), ProtoNet
35 and
MetaOptNet 22 with SVM using the linear kernel and cosine kernel.
Table 4: Accuracies (with 95% confidence intervals) of 5-way few-shot classification mini- ImageNer using ResNet-12. means that the result is obtained by rerunning the code with our setup here.
method
1-shot
5-shot
FOMAMLÍ [1
57.41 0.71
72.12 + 0.54
ANIL 27
59.66 = 0.68
73.28 = 0.49
ProtoNet
35
59.25 - 0.64
75.60 = 0.48
MetaOptNet-SVM(iin)
62.31 0.64
78.21 + 0.42
Metaa
22
62.75 0.42
78.68 + 0.24
MetaProx (proposed)
63.82 + 0.23
79.12 + 0.18
Implementation Details. The entire model is trained end-to-end. E(ý, y) is the cross-entropy loss. The CVXPYLayers package [ used solve the dual problem. We train the model for 80, 000 iterations, and each mini-batch has 4 tasks. We the Adam optimizer (20) with initial learning rate of 0.001, which then reduced by half every 2, 500 iterations. To prevent overfitting, we evaluate the meta-validation performance every 500 iterations, and stop training when the meta-validation accuracy has no significant improvement for 10 consecutive evaluations. We report the classification accuracy averaged over 600 tasks randomly sampled from the meta-testing set.
Results. Tables 3 and Landjshow the results for Conv4 and ResNet- /2, respectively. As can be seen, MetaProx is always the best. Compared with MetaOptNNet-SVM, MetaProx is better due to the learnable regularizer.
5 Conclusion
In this paper, we proposed MetaProx, an effective meta-regularization algorithm for meta-learning. MetaProx combines deep kernel and meta-regularizztion. By reformulating the problem in the dual space, a learnable proximal regularizer is introduced to the base learner. The meta-parameters in the regularizer and network are updated by the meta-learner. We also established convergence of MetaProx. Extensive experiments on standard datasets for regression and classification verify the effectiveness of learning proximal regularizer.

---

Acknowledgements
This work is supported by NSFC grant 62076118.
References
[1] A. Agrawal, B. Amos, S. Barratt. S. Boyd, S. Diamond, and Z. Kolter. Differentiable convex optimization layers. In Neural Information Processing Systems, pages 9562-9574, 2019.
[2] M.-F. Balcan, M. Khodak, and A. Talwalkar. Provable guarantees for gradient-based meta- veta-learnnigg In International Conference Machine Learning, pages 424-433 433, 2019.
[3] Y. Bengio, S. Bengio, and J. Cloutier. Learning synaptic learning rule. In International Joint Conference on Neural Networks, pages 969-975, 1991.
[4] L. Bertinetto, J. F. Henriques, Torr, and A. Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations, 2018.
[5] S. Boyd, S. Boyd, and Vandenberghe. Convex optimization. Cambridge University Press, 2004.
[6] W.-Y. Chen, Y.C.. Liu, Kira, Y.-.. F. Wang, and J.-B. Huang. closer look at few-shot classification. In International Conference on Learning Representations, 2018.
[7] G. Denevi, C. Ciliberto, D. Stamos, and M. Pontil. Learning to learn around a common mean. In Neurul Information Processing Systems, pages 10169-10111 101991119,,20000 2018
[8] G. Denevi, C. Ciliberto, R. Grazzi, and M. Pontil. Learning-to-lees stochastic gradient descent with biased regularization. In International Conference Machine Learning, pages 1566- 1575, 2019.
[9] G. Denevi, M. Pontil, and C. Ciliberto. The advantage of conditional meta-learning for biased regularization and fine tuning. Neural Information Processing Systems, 33, 2020.
[10] J. Donahue, Y. Jia, O. Vinyals, Hoffman, N. Zhang, Tzeng, and T. Darrell. Decaf: deep convolutional activation feature for generic visual recognition. In International Conference on Machine Learning, pages 647-655 2014.
[11] A. Fallah, A. Mokhtari, and A. Ozdaglar. On the convergence theory of gradient-based model-agnostic meta-lcarning algorithms. In International Conference on Artificial Intelligence and Statistics, pages 1082-1092, 2020.
[12] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126-1135, 2017.
[13] C. Finn, A. Rajeswaran, Kakade, and S. Levine. Online meta-lcarning. In International Conference on Machine Learning, pages 1920-1930, 2019.
[14] L. Franceschi, Frasconi, S. Salzo, R. Grazzi, and M. Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning, pages 1578-1577, 2018.
[15] S. Ghadimi and G. Lan. Stochastic first- and poroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, ((44):2341-266) 2013.
[16] S. Gong, S. McKenna, and J. J. Collins. An investigation into face pose distributions. In International Conference on Automatic Face and Gesture Recognition, pages 265-270, 1996.
[17] S. Gunasckar, J. Lee, Soudry, and N. Srebro. Characterizing implicit bias in terms of optimization geometry. International Conference on Machine Learning, pages 1832-1841, 2018
[18] A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Neural Information Processing Systems, 2018.
[19] K. Ji, J. Yang, and Y. Liang. Multi-step model-agnostic meta-learning: Convergence and improved algorithms. Preprint amXiv:2002.078335 2020.
[20] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
10

---

[21] J. Lee. L. Xiao, S. Schoenholz, Y. Bahri, R. Novak. J. Sohl-Dickstenn, and J. Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Neural Information Processing Systems, 2019.
[22] K. Lee, S. Maji, A. Ravichandran, and S. Soatto. Meta-lcaming with differentiable convex optimization. In IEEE Conference on Computer Vision and Patter Recognition, pages 10657-10665, 2019.
[23] C. Liu, L. Zhu, and M. Belkin. Toward theory of optimization for per-paaameteriied systems of non-linear equations: the lessons of deep learning. Preprint arXiv:2003.0033 000007, 2020.
[24] H. Liu, K. Simonyan, and Y. Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2018.
[25] A. Acchiam, and Schulman. On first-order wetaordeemetelleeee meta-lcarning algorithms. Preprint 1 803.02999, 2018.
[26] M. Patacchiola, J. Turner, E. J. Crowley, M. O'Boyle, and A. J. Storkey. Bayesian meta-learning for the few-shot setting via deep kernels. In Neural Information Processing Systems. 2020.
[27] A. Raghu, M. Raghu, S. Bengio, and O. Vinyals. Rapid learning or feature reuse? Towards understanding the effectiveness of MAML. In International Conference on Learning Representations, 2019.
[28] A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine. Meta-lcarning with implicit gradients. In Neural Information Processing Systems, pages 113-124, 2019.
[29] K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen. Efficient off-policy meta-reinforvement learning via probabilistic context variables. In International Conference on Machine Learning, pages 5331-5340, 2019.
[30] S. Ravi and H. Larochelle. Optimization as model for few-shot learning. In International Conference on Learning Reprexentations, 2017.
[31] S. J. Reddi, A. Hefay, S. Sra, B. Poczos. and A. Smola. Stochastic variance reduction for nonconvex optimization. In International Conference on Machine Learning, pages 314-323, 2016.
[32] W. Rudin. Principles of Mathematical Analysis. McGraw-Hill, New York, 1976.
[33] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, 7. Huang, A. Karpathy, A. Khosla,
M. Bernstein, A. C. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 11(()):11-252, 2015.
[34] B. Schölkopf, R. Herbrich, and A. Smola. A generalized representer theorem. In International Conference on Computational Learning Theory, pages 416-426, 2001.
[35] J. Snell, K. Swersky, and R. 7emel. Prototypical networks for few-shot learning. In Neural Information Processing Systems, pages 4077-4087, 2017.
[36] S. C. Tan
P. San Lau. Time series clustering: superior alternative for market basket analysis. In International Conference on Advanced Data and Information Engineering, pages 241-248, 2014.
[37] S. Thrun and I., Pratt. Learning to learn: Introduction and overview. In Learning to learn, pages 3-17. Springer, 1998.
[38] F. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, U. Evci, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.-A. Manzagol, and Larochelle. Meta-dataset: dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2020.
[39] O. Vinyals. C. Blundell, T. Lillicrap. and D. Wierstra. Matching networks for one shot learning. In Neural Information Processing Systems, pages 3630-3638, 2016.
[40] H. Wang, R. Sun, and Li. Glohal convergence and induced kernels of gradient-based meta-learning with neural nets. Preprint arxiv:2006. 1 14606, 2020.
[41] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni. Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys, 53(3):1-33 34, 2020.
[42] A. G. Wilson, 7. Hu, R. Salakhutdinov, and F. P. Xing. Deep kernel learning. In International Conference on Artificial Intelligence and Statistics, 2016.
[43] P. Zhou, X. Yuan, H. Xu, S. Yan, and J. Feng. Efficient meta learning via minibatch proximal update. In Neural Information Processing Systems, pages 1534-1544, 2019.
[44] B. Zoph and Q V. Le. Neural architecture search with reinforcement learnig.. In International Conference Learning Representations, 2017.


Reviews:
[{"review_id": "qIfhxmcRNbT", "reviewer": "Reviewer_pmQA", "summary": "The authors proposed a kernelized proximal regularization method (MetaProx) for meta learning. MetaProx combines deep kernel and proximal meta regularization, allowing learnable regularization by reformulating the proximal problem in dual space. Thorough theoretical analysis are provided as well as empirical studies on regression and classification task.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is well written and easy to follow. The proofs are clear. The derived propositions supported the motivation of introducing learnable regularization. \nMy major concerns about this paper are as follow:\n1. The author mentioned that the proposed method can introduce nonlinearity to the model other than CommonMean method. By using a linear kernel as base kernel, the only difference between MetaProx and CommonMean is that MetaProx utilizes nonlinear neural networks as embedding network while CommonMean doesn’t. And that’s also the reason it outperforms CommonMean. It makes the work looks a bit incremental from this perspective. Though I think the author made a good point about why it outperforms MetaOpt-SVM work. \n\n2. In few-shot classification task, the original MetaOpt-SVM work mentioned that their method works better in large backbone network setting (e.g. ResNet12). However, in smaller backbone network setting, they do not have much advantages and hard to outperform ProtoNet. It might be fairer and more convincing to compare under different backbone setting to show consistent results.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "k570KHwYEyH", "reviewer": "Reviewer_2wb5", "summary": "This paper proposes a new meta-learning algorithm. Following the meta-regularization method, this paper proposes a new proximal regularizer instead of the traditional regularizer to reduce the computational complexity significantly. The algorithm is reasonable and interesting which also shows high performance in experiments.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " Originality:\n\nIn meta-learning community, how to extract meta-knowledge for fast adaptation to a future task is the key problem. Metric-based algorithms learn meta-knowledge using a shared feature extractor and gradient-based meta-algorithms learn meta-knowledge by learning a common initialization. The above two branches draw a lot of attention these years, while meta-regularization, learning a common mean, is natural and reasonable but less popular. This paper discovers the potential of the meta-regularization algorithms.\n\nThis paper proposes a new meta-regularizer which is interesting and novel.\n\nQuality:\npos: The idea is interesting and novel. The theoretical analysis and the experiments are solid.\n\ncons: The motivation and implications of the new proposed regularizer are not very clear. Why we need to propose a new regularizer instead of the commonly used mean square loss?  Apart from the computational cost, is there any key difference between the proposed one and the traditional one?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It will be better to clarify the key motivation and implications to apply the proposed regularizer.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hour"}, "extra_text": {}}, {"review_id": "huA8ivPteV5", "reviewer": "Reviewer_S5bU", "summary": "This paper introduces a new functional regularisation for meta learning. Often times meta learning algorithms can overfit to the training datasets and hence an introduction of a regularisation in terms of the outer loop parameters is a sensible choice. In particular, they build upon the Common mean algorithm and consider the dual form in which they can derive closed form solutions to the optimisation problem. Their main contribution is that the method remains computationally cheap, competitive in terms of performance and provide some theoretical insights into their proposed method MetaProx.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Reason to accept the paper:\n- The paper introduces a novel regularisation for meta learning in the function space and obtains closed-form solutions to the ir optimisation problem. \n-  The proposed method is computationally efficient and achieves competitive results compared to previous methods.\n-  In particular they show that setting f_{\\theta} = 0 reverts back to the popular MetaOPTNet algorithm and demonstrate that their additional regularisation improves upon the original method.\n-  They show empirically that their proposed methods performs better than most methods in particular MetaOPTNet\n-  (I have not gone through the proofs of their theoretical claims and will base my opinion on this aspect from other more confident reviewers in this area)\n\n\nReason to reject/neutral the paper and areas that need clarification:\n\n- Given that you have dropped the hyperparameter \\lambda in your objective expression (You say you are learning the kernel so this is not necessary anymore), it seems to me that initialization of your networks seems crucial and could potentially make optimization very hard. Can you comment on this?\n\n- The top of page 5 is very confusing and not very clear. I was wondering if you could give a more intuitive explanation of what is happening and why for example the implicit function theorem has to be used etc. I understand that this is to reduce complexity but a clearer explanation of this would be helpful.\n\n- What is K in your paper? Is this the kernel that is learning through the feature maps? You define a K on the Z's. Have you tried to use a gaussian kernel on top of the deep kernel features instead of the just a linear kernel? You use the word RKHS and seems quite misleading (As they are usually associated with Infinite dim feature) if in reality all you are doing is deep kernel learning. This is not a problem perse just wanted this clarified as K is never properly defined (line 138-139)\n\n-  Your Fig 1/2 is very misleading as you only show MAML papers such as [1] or [2] should be added. Else i do not see the point of this figure. Also the width is crucial in MAML and hasnt been properly studied \n\n- Table 1/2/3 is missing [1, 2] which has very well documented code and beats MetaOptNet. Hence this comparison should not be omitted.\n\n- Table 3: 5-shot, your method is not statistically significantly better than Proto-net and hence bolding your number is misleading\n\n\n\n[1] : https://arxiv.org/pdf/1912.02738.pdf CODE: https://github.com/jinxu06/metafun-tensorflow\n[2]: https://arxiv.org/pdf/2003.11539.pdf CODE: http://github.com/WangYueFt/rfs/", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Overall the paper proposes a new simple method for regularising the meta learning training. The write up of the paper could be improved in particular the dual problem setup as well as their proposed computational tricks as mentioned above. Also in terms of the experiments i believe that methods mentioned above should be included for completeness or at least justified why they weren't added. I understand that the proposed method improves upon metaoptnet but that doesn't mean that it is necessarily the best method and hence should be compared to others as well.\n\nI will be awaiting the rebuttal and am happy to raise my score if the above have been addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "acvbv1SthaJ", "reviewer": "Reviewer_Nvyf", "summary": "The authors proposed an effective meta-regularization algorithm (MetaProx) for meta-learning. A learnable proximal regularizer with deep kernel is introduced to learn the base learner. The computation complexity of meta-gradient is linear in the number of meta-parameters. Experiments on synthetic dataset and mini-ImageNet show that the proposed method outperforms the popular few-shot learning methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.\tThis paper is well-written. The research problem and proposed method are clearly introduced. \n2.\tIt is good to have theoretical analysis on the smoothness of the meta-loss and the convergence. \n3.\tThe proposed method is verified to be effective in few-shot regression and classification tasks.\n4.\tThe paper states that the new method is much faster than other meta-learning methods e.g. MAML. However, the paper lacks the comparison of training and converging time. \n5.\tIn few-shot classification experiments, the simple Conv4 network is used. However, the state-of-the-art few-shot learning methods use ResNet and Wide Residual Networks as the backbone. Does the proposed method still achieve better performance than the competitors with large models?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation is not provided. No negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper initially received mixed ratings. Most concerns from the reviewers have been well addressed by the rebuttal and after the discussion period, all reviewers are now on the accept side. In particular, the reviewers have been convinced by the experimental results provided in the rebuttal. The area chair agrees with their assessment and follows their recommendation. 

The final version of the paper should include these results and take into account the discussions that took place during the rebuttal period.
Decision: Accept (Poster)