Online Hyperparameter Meta-Learning with Hypergradient Distillation
Authors: Hae Beom Lee
Hayeon Lee
JaeWoong Shin
Eunho Yang
Timothy Hospedales
Sung Ju Hwang
Venue: ICLR 2022 Spotlight (2022)
Keywords: Hyperparameter Optimization
Meta-learning

Abstract: Many gradient-based meta-learning methods assume a set of parameters that do not participate in inner-optimization, which can be considered as hyperparameters. Although such hyperparameters can be optimized using the existing gradient-based hyperparameter optimization (HO) methods, they suffer from the following issues. Unrolled differentiation methods do not scale well to high-dimensional hyperparameters or horizon length, Implicit Function Theorem (IFT) based methods are restrictive for online optimization, and short horizon approximations suffer from short horizon bias. In this work, we propose a novel HO method that can overcome these limitations, by approximating the second-order term with knowledge distillation. Specifically, we parameterize a single Jacobian-vector product (JVP) for each HO step and minimize the distance from the true second-order term. Our method allows online optimization and also is scalable to the hyperparameter dimension and the horizon length. We demonstrate the effectiveness of our method on three different meta-learning methods and two benchmark datasets.

Published as a conference paper at ICLR 2022
ONLINE HYPERPARAMETER META-LEARNING WITH HYPERGRADIENT DISTILLATION
Hae Beom Lee', Hayeon Lee', Jaewoong Shin
Eunho Yang 12, Timothy M. Hospedales* Sung Ju Hwang!
KAIST!, AITRICS", Lunit South Korea, University of Edinburgh', Samsung AI Centre, Cambridge", United Kingdom fhaebeom. lee, hayeon926, shinjw148, eunhoy, sjlvang32)@kaist.ac. (t www.shppplllseeeeecco ac.uk
ABSTRACT
Many gradient-based meta-learning methods assume set of parameters that do not participate in inner-optimization, which can be considered as hyperparame- ters. Although such hyperparameters can be optimized using the existing gradient- based hyperparameter optimization (HO) methods, they suffer from the following issues. Unrolled differentiation methods do not scale well to high-dimensional hy- perparameters or horizon length, Implicit Function Theorem (IFT) based methods restrictive for online optimization, and short horizon approximations suffer from short horizon bias. In this work, we propose a novel HO method that can overcome these limitations, by approximating the second-order term with knowl- edge distillation. Specifically, we parameterize a single Jacobian- vector product (JVP) for each HO step and minimize the distance from the true second-order term. Our method allows online optimization and also is scalable to the hyperpa- rameter dimension and the horizon length. We demonstrate the effectiveness of our method on two different meta- meta-learning methods and three benchmark datasets.
INTRODUCTION
Meta-learning (Schmidhuber, 1987; Thrun & Pratt, 1998) aims to learn learning process itself over a task distribution. Many gradient-based meta-learning approaches assume a set of parameters that do not participate in inner-optimization (Leee & Choi, 2018; Flennerhag et al., 2019; Raghu et al., 2019) which can be seen as hyperparameters. Those hyperparameters are important in helping the inner-learner converge faster and generalize better. As they are usually very -igheiimensional such as element-wiss learning rates (Li et al., 2017), we cannot meta-learn them with simple hyperparam- eter optimization (HO) techniques such as random search (Bergstra & Bengio, 2012) or Bayesian optimization (Snoek al., 2012) due the too extensive search space.
In this case, we can use gradient- based HO methods that directly optimize the high- dimensional hyperparameters by minimizing the validation loss w.r.t. the hyperparame- ters (Bengio, 2000). Due to the expensive com- putational cost of evaluating the hypergradients (i.e. the gradient w.r.t. the hyperparameters) there has been lot of efforts improve the ef-
Table 1:
Comparison between the various gradient- based HO algorithms. 1-step denotes one-step looka- head approximation (Luketina al., 2016).
FMD
RMD
DrMAD
IFT
I-step
Ours
High-dim.
0
Online opt.
0
X
X
4
0
Coestant memory
0
Horizon
O
X
fectiveness and the efficiency of the algorithms. However, unfortunately, none of the existing algorithms satisfy the following criteria the same time that should be met for their practical use: 1) scalable to hyperparameter dimension, 2) online optimization, 3) memory-efficient, 4) avoid short-horizon bias. Please See Table for the compari- of existing gradient-based HO algorithms in the above four criteria.
Forward- Mode Differentiation (FMD) (Franceschi et al., 2017) in Table is an algorithm that forward- propapgatsss Jacobians (i.e. derivatives of the update function) from the first to the last step, which is analogous to real-time recurrent learning (RTRL) (Williams & Zipser, 1989) recurrent neural networks. FMD allows online optimization (i.e. update hyperparameters every inner-step)

---

Published as a conference paper at ICLR 2022
with the intermediate Jacobians and also computes the hypergradients over the entire horizon. How- ever, a critical limitation is that the time and space complexity linearly increases w.r.t. the hyperpa- rameter dimension. Thus, we cannot FMD for solving many practical meta-learning problems that come with millions of hyperparameters, which is the main problem we tackle in this paper.
Secondly, Reverse-Mode Differentiation (RMD) (Maclaurin et al., 2015) back-propagater the Jacobian-vector products (JVPs) from the last to the initial step. which is structurally identical to backprop through time (BPTT) (Werbos, 1990). RMD is scalable the hyperparameter dimension, but the space complexity linearly increases w.r.t. the horizon length (i.e., the number of inner- gradient steps used compute the hypergradient). It is possible to reduce the memory burden by checkpointing some of the previous weights and further interpolating between the weights to ap- proximate the trajectory (Fu et al., 2016). However, RMD and its variants are not scalable for online optimization. This is because they do not retain the intermediate Jacobians unlike FMD and thus need to recompute the whole second-order term for every online HO step.
Thirdly, algorithms based Implicit Function Theorem (IFT) are applicable to high-dimensional HO (Bengio, 2000; Pedregosa, 2016). Under the assumption that the main model parameters have arrived at convergence, the best-response Jacobian, i.e. how the converged model parameters change w.r.t. the hyperparameters, can be expressed by only the information available at the last step, such as the inverse of Hessian at convergence. Thus, we do not have to explicitly unroll the previous update steps. Due to the heavy cost of computing (2020) propose approximate it by an iterative method, which works well for high-dimensional HO problems. However, still it is not straightforward to use the method for online optimization because of the convergence assumption. That is, computing hypergradients before convergence does not guarantee the quality of the hypergradients.
Searse-Hessian-vee
product, Lorraine et al.
To our knowledge, the short horizon approximation such as one-step lookahead (1-step in Ta- ble 1) (Luketina et al., 2016) is the only existing method that fully supports online optimization, while being scalable to the hyperparameter dimension at the same time. computes hypergradients only over a single update step and ignores the past learning trajectory, which is computationally efficient as only a single JVP is computed per each online HO step. However, this approximation suffers from the short horizon bias (Wu et al., 2018) by definition.
In this paper, we propose a novel HO algorithm that can simultaneously satisfy all the aforemen- tioned criteria for practical HO. The key idea is to distill the entire second-order term into a single JVP. As a result, we only need to compute the single JVP for each online HO step, and at the same time the distilled JVP can consider longer horizons than short horizon approximations such as one-step lookahead or first-order method. We summarize the contribution of this paper as follows:
We propose HyperDistill, novel HO algorithm that satisfies the aforementioned four eri- teria for practical HO, each of which is crucial for a HO algorithm to be applied to the current meta-learning frameworks.
We show how to efficiently distill the hypergradient second-order term into single JVP.
We empirically demonstrate that our algorithm converges faster and provides better gener- alization performance convergence, with three recent meta-learning models and on two benchmark image datasets.
2 RELATED WORK
Hyperparameter optimization When the hyperparameter dimension is small (e.g. less than 100), random search (Bergstra & Bengio, 2012) or Bayesian optimization (Snoek et al., 2012) works well. However, when the hyperparameter is high-dimensional, gradient- based HO is often preferred since random or Bayesian search could become infeasible. One of the most well known methods for gradient-based HO are based on Implicit Function Theorem which compute or approximate the in- verse Hessian only at convergence. Bengio (2000) computes the exact inverse Hessian, and Luketina et al. (2016) approximate the inverse Hessian with the identity matrix, which is identical to the one- step lookahead approximation. Pedregosa (2016) approximates the inverse Hessian with conjugate gradients (CG) method. Lorraine et al. (2020) propose Neumann approximation, which is numer- ically more stable than CG approximation. On the other hand, Domke (2012) proposes unrolled differentiation for solving bi-level optimization, and Shaban et al. (2019) analyzes the truncated unrolled differentiation, which is computationally more efficient. Unrolled diffrentiation can be fur-
2

---

Published as a conference paper at ICLR 2022
ther categorized into forward (FMD) and reverse mode (RMD) (Franceschi et al., 2017). FMD is more suitable for optimizing oiw-dimesiional hyperparamters (Im et al., 2021; Micaclli & Storkey, 2020), but RMD is more scalable to the hyperparameter dimension. Maclaurin et al. (2015) proposes a more memory-efficient RMD, which reverses the SGD trajectory with momentum. Fu et al. (2016) further reduce memory burden of RMD by approximating the learning trajectory with linear inter- polation. Luketina et al. (2016) can also be understood as a short horizon approximation of RMD for online optimization. Our method also supports online optimization, but the critical difference is that our algorithm can alleviate the short horizon bias (Wu et al., 2018). RMD is basically type of backpropagation and it is available in deep learning libraries (Grefenstette et al., 2019).
Meta-learning Meta-learning (Schmidhuber, 1987; Thrun & Pratt, 1998) aims to learn a model that generalizes there exists a variety of approaches, in this paper we focus on gradient- brasee meta-learning (Finn al., 2017), especially the methods with high-dimensional hyperparameters that do not partici- pate in inner-optimization. For instance, there have been many attempts to precondition the inner- gradients for faster inner-ottimization. either by warping the parameter space with every pair of consecutive layers interleaved with a warp layer (Lee & Choi, 2018; Flennerhag et al., 2019) or di- rectly modulating the inner-gradients with diagonal (Li al., 2017) or block diagonal matrix (Park & Oliva, 2019). Perturbation function is another form of hyperparameters that help the lener-l generalize better (Lee et al., 2019; Ryu et al., 2020; Tseng et al., 2020). is also possible to let the whole feature extractor be hyperparameters and only adapt the last fully-connected layer (Raghu et al., 2019). On the other hand, some of the meta-learning literatures do not assume a task distri- bution, but tune their hyperparameters with holdouot validation set, similarly to the conventional HO setting. In this case, the one-step lookahead method (Luketina et al., 2016) is mostly used for scalable online HO, in context of domain generalization (Li et al., 2018), handling class imbal- ance (Ren et al., 2018; Shu et al., 2019), gradient-based neural architecture search (Liu et al., 2018), and coefficient of norm-based regularizer (Balaji al., 2018). Although we mainly focus on meta- learning setting in this work, whose goal is to transfer knowledge through a task distribution, it is straightforward apply our method to conventional HO problems.
distribution of tasks (Vinyals et al., 2016; Ravi & Larochelle, 2016). While
3 BACKGROUND
In this section, we first introduce RMD and its approximations for efficient computation. We then introduce our novel algorithm that supports high-dimensional online HO over the entire horizon.
3.1 HYPERPARAMETER UNROLLED DIFFERENTIATION
We first introduce notations. Throughout this paper, we will specifiy as weight and X as hy- perpurameter. The series of weights wo, WI, .......y WT evolve with the update function we ((w,-1,>; D,) over steps / = I....,T: T. The function & takes the previous weight W and the hyperparameter À as inputs and its form depends on the current mini-batch De. Note that W1, ......,, WT are functions w.r.t the hyperparameter X. The question is how to find a good hyperparameter À that yields a good response wr at the last step. In gradient-based HO, we find the optimal À by minimizing the validation loss Lval as function of X.
min C (wT(() A)
(I)
Note that we let the loss function (.-)) itself be modulated by X for generality. According to the chain rule, the hypergradient is decomposed into
d. (Wr,A)
aL (wr,
ac A) dwr
dA
OWT dA
3 First-order
A0: Second-order term
On the right hand side, the first-order (FO) term gF0 directly computes the gradient w.r.t À by fixing wr. The second-order (SO) term 50 computes the indirect effect of À through the response wr-
(wr.A) be easily computed similarly to g but the response Jacobian dwr. is more dx
at
Our
computationally challenging it is unrolled into the following form.
dwr
Σ 1=1
T
II
T
Ag
B where
A₂ =
@D(w,-10 (.))
B =
0 00 Di)
(3)
dX
Ow,-1
0X
3

---

Published as a conference paper at ICLR 2022
Algorithm Reverse-HG (RMD)
Algorithm 2 DrMAD (Fu et al., 2016)
I: Input: The last weight wr and all the previous
Input: The last weight wT and the initial weight wo.
weights wo...., WT-1.
Output: Approximated hypergradient g + g
2: Output: Hypergradient g/a + go0
o =2(w7.)) FD ac(WT.))
9 50
Owy g
3: a + 22(wr.)) gº + BL(wT.A) 00 + Dwy
4: for t T downto 1 do
02
4: for T downto do gf <- 5000 aB.
i4 + (1 (/) we + =
80 t 9
+ +aB,
6: a <- aA,
7:
< a', aaa,,
7:
end for
end for
8: return geD + g00
return gn0 + 50
Eq. (3) involves the Jacobians (A) and {B) at the intermediate steps. Evaluating them or their vector products are computationally expensive in terms of either time (FMD) or space (FMD, RMD) (Franceschi et al., 2017). Therefore, how to approximate Eq. (3) is the key to developing an efficient and effective HO algorithm.
3.2 REVERSE-MODE DIFFERENTIATION AND ITS APPROXIMATIONS
Basically, RMD is structurally analogous to backpropagation through time (BPTT) (Werbos, 1990). In RMD, we first obtain or a (wr,A) and back-propagate and B from the last to the first Bwr
step in the form of JVPs (See Algorithm 1). Whereas RMD is much faster than FMD we only need to compute one or two JVPs per each step, it usually requires to store all the previous weights Wo....; WT-1 1 to compute the previous-stey JVPs, unless consider reversible training with mo- mentum optimizer (Maclaurin et al., 2015). Therefore, when w is high-dimensional, RMD is only applicable to short-horizon problems such as few-shot learning (e.g. T in Finn et al. (2017)).
Trajectory approximation. Instead of storing all the previous weights for computing and B, we can approximate the learning trajectory by linearly interpolating between the last weight wT and the initial weight wo. Algorithm 2 illustrates the procedure called DrMAD (Fu et al., 2016), where each intermediate weight wy is approximated by Wi (1 +) Wo + fur for t = I,..., T - 1. A and respectively. However, although DrMAD dramatically lower the space complexity, it does not reduce the number of JVPs per each hypergradient step. For each online HO step t = I..... T we need to compute 2t - JVPs, thus the number of total JVPs to complete a single trajectory accumulates up to > (2t - 1) = T2, which is definitely not scalable an online optimization algorithm.
B are also approximated by Å. (0),,, 8th--1 I.A.D.) and B. = a0 I.A:D,)
0A
Short-horizon approximations. One-step lookahead approximation (Luketina et al., 2016) cur- rently one of the most popular high-dimensional online HO method that can avoid computing the excessive number of JVPs (Li et al., 2018; Ren et al., 2018; Shu et al., 2019; Liu et al., 2018; Balaji et al., 2018). The idea is very simple; for each online HO step we only care about the last previ- ous step and ignore the rest of the learning trajectory for computational efficiency. Specifically, for each step i ., T we compute the hypergradient by viewing w as constant, which yields dw: = dwe, dx
0x mm B, (See Eq. (3)). Or, may completely ignore all the second-order deriva- tives for computational efficiency, such that ZX 0 (Flennerhag al., 2019; Ryu et al., 2020). While those approximations enable online HO with low cost, they are intrinsically vulnerable to short-horizon bias (Wu et al., 2018) by definition.
4 APPROACH
We next introduce our novel online HO method based on knowledge distillation. Our method can overcome all the aforementioned limitations the same time.
4.1 HYPERGRADIENT DISTILLATION
The key idea is to distill the whole second-order term पن = a Ei= (II) A,) B, in Eq. (2) into a single JVP evaluated at a distilled weight point w and with a distilled dataset D. We denote the normalized JVP as fe(w, D) oaa ad(w,A:D) with a(x)) = प Specifically, we want to solve the following knowledge distillation problem for each online HO step / = 1,...,
4

---

Published as a conference paper at ICLR 2022
Ti. wi: Di arg IT((w. D) 91 a,w,D
(4)
so that we use ni fe([,, DF) instead of g: Online optimization is now feasible because for cach online HO step ...... T only need to compute the single JVP fe(ws. Df) rather than computing 2t JVPs for RMD or DrMAD. Also, unlike short horizon approximations, the whole
trajectory information is distilled into the JVP, alleviating the short horizon bias (Wu al., 2018).
Notice that solving Eq. (4) only w.r.t π is simply a vector projection.
(e D) = fwW D)"g." g.
(5) of (w, D)ll2 1, easily convert
Then, plugging Eq. (5) into in Eq. (4) and making the optimization problem Eq. (4) into the following equivalent problem (See Appendix A).
w DJ arg max The (w, 1,, D), गे क. (w) D?).
w..
w and Dj match the hypergradient direction and कद matches the size.
Technical challenge. However, solving Eq. (6) requires to evaluate g. for 1,..., T, which is tricky as ot is the target we aim to approximate. We next show how to roughly solve Eq, (6) even without evaluting gt (for wi: D2) or by sparsely evaluating an approximation of gt (for m7))
4.2 DISTILLING THE HYPERGRADIENT DIRECTION
Hessian approximation. In order to circumvent the technical difficulty, we start from making the optimization objective # (ww D) ft(w, D)" go in Eq. (5) simpler. We approximate oo
80 gt
04
II f=++1
A
B
a,
i=1
2=1
with 7 2 O, which we tune on meta-validation set. Note that Eq. (7) yet too expensive to use for online optimization as it consists of JVPs. We thus need further distillation, which we will explain later. Eq. (7) is simply a Hessian identity approximation. For instance, vanilla SGD with learning rate 17 Inner corresoonds to A, V Vwy-n (W,-1 Inner V Ctrain (W1-1, A)). Approximating the Hessian
V wy Crain (wj-1, A) kl, we have Aj -0(0-11 I.D:))) (a) have
17 Inmerr kl (1 (m) yd. Plugging
Eq. (7) Eq. (5) and letting fe(w;-1, D:) == σ aaa
(e D) R R
(w, w. D) Σ Sesi Selw, DJ fe(wi-1,
(8)
where dG; = = Bil2 2 Instead of maximizing म, directly, we now maximize सं w.r.t. w and D as proxy objective.
Lipschitz continuity assumption. Now we are ready to see how to distill the hypergradient di- rection W and Dj withour evaluating goo The important observation is that the maximum of a in Eq. (8) is achieved when f((w, D) is well-aligned to the other fe ... f( Dz). This intuition is directly related to the following Lipschitz continuity assumption on ft.
(9) where K > is the Lipschitz constant. Eq. (9) captures which (w, D) minimize [[,e D) fe(w:-1, D2) over 1,..., f, which is equivalent to maximizing fe(w, D)T [e D since [lf( 1. For the metric || ||x, we let 2 (ww, D)Il
(w, D) - fe(w;-1, D2) I K] (w, (w, D) - (w,-1. D.)llx, for i
K [| 1/2 + K where KI, 2 0 are additional constants that we introduce for notational convenience. Taking square of the both sides of Eq. (9) and summing over all I,...,t, we can easily derive the following lower bound of tt (See Appendix B).
Σ 8c.
K? 2
wi-1
K2 Σ Se D D & #e D)
(10)
We now maximize this lower bound instead of directly maximizing t.. Interestingly, it corresponds to the following simple minimization problems for w and D.
min
8 w -::--w;111 wi-1
min D
Σ Sr D - D.1l3-
(11)
1=1
5

---

Published as a conference paper at ICLR 2022
Algorithm 3 HyperIDistill
Algorithm 4 w..nearrstttmtt.oonn......
1: Input: > E [0, 1), initial A, and initial Φb.
1: Input: wp f- á
2: Output: Learned hyperparameter A.
2: for l to do
3: for m 1 to M de
3: Wi f- $(W--1, A; Di)
4: if vm RsttmmationPrrod then
end for
4- i.-.iiaattttmmmtiooooooooo
5: ar 4- w .2
5000 <
6: end if
6: for t = downto do
7:
wD
7:
$1-1 t 11(1
[1 T
Wp - WT
8:
for
t to 7 do
"i, 1); -... (13), Eq. (14).
8:
g 50 t g0000 + aB, (Eq. (16))
10:
ni 1- (t; in Eq. (15)
9:
a f- aA,
11:
uh + 1--((i-1.. Di)
10:
8 T ++1
12:
g 9t + n felit; DF)
11:
wT, D; + Eq. (17), Eq. (18)
13:
À A ー-1 1
12:
es at L-2
14:
end for
13: 2s f ||v, T-y
Ye t ovv))
15:
n Repale ($ WT)
14: end for
16:
end for
15: return
(a /)/(2²))
Efficient sequential update. Eq. (11) tells how to determine the distilled 3º and D; for each
HO step 1 1,..., T. ,...,,T
Since ||o, Bill2 expensive to compute, we approximate as [|a,Bollp Rs
||, By 2 R ... l|,, B.l2. yielding the following weighted average the approximated solution for wy-
7--2
w El=r 1
W00 +
gd W1
W
(12)
Ei=
Ei=
7
The following sequential update allows to efficiently evaluate Eq. (12) for each HO step. Denoting Pe (7 1 >0) 7²) [0, 1), we have
= 1: wi +
2 wi + +P:w + (t)
(13)
Note that the online update in Eq. (13) does not require to evaluate o.00 It only requires to incor- porate the past learning trajectory Wh: wI...., W1,- through the sequential updates. Therefore, the only additional cost is the memory for storing and updating the weighted running average wt.
For D, we have assumed Fuclidean distance metric as with w, but it is not straightforward to think of Euclidean distance between datasets. Instead, we simply interpret and 1 - Pt as probabilities with which we proportionally subsample each dataset.
t 1 D D1, 2 D + ss (Di-1.Pi) USS (D,, 1 pt)
(14)
where SS be a better distance metric for datasets and a corresponding solution, but we leave as a future work. See Algorithm 3 for the overall description of our algorithm, which we name HyperDistill.
(D. SS(D. denotes random subSampling of round_off( instances from D. There may
Role o Note that Eq. (12) tells us the role y as a decaying factor. The larger the y the longer the past learning trajectory we consider. In this sense, our method is generalization of the one-step lookahead approximation, i.e. 7 O, which yields w," = W and D; = Dr, ignoring the whole information about the past learning trajectory except the last step. 7 0 may be too pessimistic for most of the cases, so we need to find better performing y for each task carefully.
4.3 DISTILLING THE HYPERGRADIENT SIZE
Now we need to plug the distilled W and D; into (, J D) in Eq. (5) to obtain the scaling factor oे Se(w;, Di)" gi for online HO steps / = 1,..., T. However, whereas evaluating the sin- gle JVP fe(w; D;) is tolerable, again, evaluating .i is misleading as is the target we aim to approximate. Also, it not straightforward for गद to apply a similar trick we used in Sec. 4.2.
I Linear estimator. We thus introduce a linear function e. (1: 0) that estimates मं by periodically fitting 0 E RR. the parameter of the estimator. Then for each HO step / we could use (1; 0) instead of fully evaluating .... Based on the observation that the form of lower bound in Eq. (10) roughly proportional to Σi= ôt, = -E= 70- ||ay Bi we conveniently set e, (t: 6) to as follows:
d4(wt x; D:)
(t; 0) = 0. 110a ~n where it == a
(15)
0x

---

Published as a conference paper at ICLR 2022
Collecting samples. We next see how to collect samples {x,, vv) Vo)) === for fitting the parameter 0, where Is ||uall2 1-2 and ys T fa(w", D2) g. = a (v,)) g." For this, we need to efficiently collect:
Forward pass
50
91
1. g the second-order term computed over the horizon of size 8.
Back-prop. (DrMAD)
2. Us. the distilled JVP computed over the horizon of size
Figure 1: Collecting g²
1) 3²0 Note that DrMAD in Algorithm 2 (line 6) sequentially back-propasee propaagtes g0 for t
T. T....,1. 1. The important observation is that, step t, this incomplete second term g
E aT AT ... A Br can be seen as the valid second-order term computed over the hori- zon of size = T t 1. This is because the reparameterization = =T - + gives
80 94
Σ
as+(r-s)
(16)
for s = I....,77 T. nothing but shifting the trajectory index by - steps so that the last step is always T. Therefore, we can efficiently obtain the valid second-order term g²0h for all 1,.... T through the single backward travel along the interpolated trajectory (See Figure 1). Each ge requires to compute only one or two additional JVPs. Also, as we use DrMAD instead of RMD, we only store Wp such that the memory cost is constant w.r.t. the total horizon size T.
2) Ug: For computing the distilled JVP Us. we first compute the distilled wy and D; as below, similarly to Eq. (13) and (14). Denoting ps = (1 - 111---1111 () - -*)), we have
1: wi- + WT-1
8 2: W + pow 11 + (1-Ps)wT- pa)
(17)
1: Di + Dr. $ $22: D t ss (D; SS (Dr- ps) (18)
We then compute the unnormalized distilled JVP U. (T- (0(WT.A:D]]
Estimating 0. Now we are ready estimate 0. For x. we have I, || 1 and collect = (I1,.... IT). For y. we have y, (vv,) T g and collect = (V1...., BT). Finally, we estimate 0 = (o" u)/(<"an x)) See Algorithm and Algorithm 4 for the details. Practically, we set EstimationFee (1,51, 101,.................... J. Thus, the computational cost LinearEst imation marginal in terms of the wall-clock time (see Table 3).
od in Algorithm 3 to every 50 completions of the inner-optimizations, i.e.
5 EXPERIMENTS
Baselines. We demonstrate the efficacy of our algorithm by comparing to the following baselines. 1) First-Order Approximation (FO). Computationally the most efficient HO algorithm that com- pletely ignores the second-order term, i.e. g 0. 2) One-step Look-ahead Approximation (1-step). (Luketina et al., 2016) 'he short-horizon approximation where only single step is un- rolled to compute each hypergradient. 3) DrMAD. (Fu et al., 2016) An approximation of RMD that linearly interpolates between the initial and the last weight to save memory (see Algorithm 2). 4) Neumann IFT (N.IFT). (Lorraine et al., 2020) An IFT based method that approximates the inverse-Hessianc vector product by Neumann series. Note that this method supports online optimiza- tion around convergence. Specifically, among total T = 100 inner-steps, N.IFT(N, K) means for the last K steps perform online HO each with N inversion steps. It requires total (N + +1) K JVPs. We tune (N, K) among ((2, 25), (5, 10), (10, 5)). roughly computing 50 JVPs per inner-opt. 5) HyperDistill. Our high-dimensional online HO algorithm based the idea of knowledge dis- tillation. We tune the decaying factor within 7 E {0.9, 0.99, 0.999, 0.9999}. The linear regression done every 50 ontrmooptinization problems.
Target meta-learning models. We test on the following three meta-learning models.
1) Almost No Inner Loop (ANIL). (Raghu et al., 2019) The intuition of ANIL is that the need for task-specifi, adaptation diminishes when the task distribution homogeneous. Following this intuition, based on a typical 4-layer convolutional network with 32 channels (Finn et al., 2017), designate the three bottom layers as the sighh-imensional hyperparameter and the 4th convolutional layer and the last fully connected layer the weight, similarly to Javed & White (2019).
7

---

Published as a conference paper at ICLR 2022
ANIL / /iiyllmggeeet
WarpGrad / /CIARRO0
WarpGrad TinylmageNet
MetaWeightNet / CIFAR100
1.8
Fb
1.2
NJFT 5)
1.6
N.FT 5)
1.4
N.FT
1.7
1-4tep
1.1
1.3
1.6
DIMAD
1.5
1.2
1.5
12
HyperDetil
1.0
1.4
1.1
1.4
1.0
1.3
0.9
1.
1.2
0.0
200 400 600 800
1K
200 400 600 800
1K
200 400 600 800
1K
200
400
600
800
1K
ont Solved
onner-opt. Selved
Inner-opt. Solved
Inner-opt. Solved
Figure 2: Meta-training convergence measured in L (WT, A) with T = 100 inner-steps. We report mean and and 95% confidence intervals over 5 meta-training runs.
Online optim.
# JVPs inner-opt.
ANIL
WarpGrad
MelaWeightNet CIFARI00
tinylmageNet
CIFARIO0
tinylmageNet
FO 1-step
O
0
53.62
58. 18.16
53.54
N/A
O
50
53,90=043
58. 38.182052
49.9712246
58.452040
DrMAD
X
199
49.84=1185
55, 13+4..4
50.71+1.16
57.03++042
Neumann IFT
2
{55, 60, 75}
53,7620.31
58. 88.88
50.15
59.34.20.27
HyperDistill
2 58
56.37
60.91
55.04.19.2
60.82.003
Table 2: Meta-test performance measured in test classification accuracy (%). We report mean and and 95% confidence intervals over 5 meta-training runs.
2) WarpGrad. (Flennerhag et al., 2019) Secondly, we consider WarpGrad, whose goal is to meta- learn non-linear warp layers that facilitate fast inner-optimization and better generalization. We use 3-layer convolutional network with 32 channels. Every layer is interleaved with two warp layers that do not participate in the inner-optimization, which is the high-dimensional hyperparameter.
3) Meta Weightee,, (Shu et al., 2019) Lastly, we consider solving the label corruption problem with Meta Neighte, which meta-learns small MLP taking a ID loss reweighted loss. The parameter of the MLP is considered as a nigh-diimessooal hyperparameter. Labels are independently corrupted to random classes with probability 0.4. Note that we aim to meta-learn the MLP over a task distribution and apply to diverse unseen tasks, instead of solving single task. Also, in this meta model the direct gradient is zero, g = 0. In this case, x² in HyperDistill has a meaning of nothing but rescaling the learning rate, so we simply set π =.
input and output
Use of Reptile. Note that for all the above meta-learning models, meta-learn the weight ini- tialization with Reptile (Nichol et al., 2018) as well, representing a more practical meta-learning scenario than learning from random initialization. We use the Reptile learning rate Reptile 1. Note that @ in Algorithm 3 and Algorithm 4 denotes the Reptile initialization parameter.
Task distribution. We consider the following two datasets. To generate each task, we randomly sample 10 classes from each dataset (5000 examples) and randomly split them into 2500 training and 2500 test examples. 1) TinyImageNet. (Le & Yang, 2015) This dataset contains 200 classes of general categories. We split them into 100, 40, and 60 classes for meta-training, meta-validation, and meta-test. Each class has 500 examples of size 64 x 64. 2) CIFAR100. (Krizhevsky et al., 2009) This dataset contains 100 classes of general categories. We split them into 50, 20, and 30 classes for meta-training, veta-validation, and meta-test. Each class has 500 examples of size 32 x 32.
Experimental setup. Meta-training: For inner-optimization of the weights, we use SGD with momentum 0.9 and set the learning rate 4 Inner 0.1 for Meta NetaWWightNet and Inner = 0.01 for the others. The number of inner-steps is 7 100 and batchsize is 100. We use random cropping and horizontal flipping as data augmentations. For the hyperparameter optimization, we also use SGD with momentum 0.9 with learning rate Hyper 0.01 for MetaWeightNet and Hyper = 0.001 for the others, which we linearly decay toward 0 over total MM- 1000 inner-optimizationn. We perform parallel meta-learning with meta-batchsize set to 4. Meta-testing: We solve 500 tasks to measure average performance, with exactly the same inner-optimization setup as meta-training. We repeat this over 5 different meta-train meta-training runs and report mean and 95% confidence intervals (see Table 2). Code is publicly available at: https: /ttp::////tthtt com/ hawbeom-bbbbe..eee. lee weee/hhpeedisst.e
5.1 ANALYSIS
We perform the following analysis together with the WarpGrad model and CIFARIO0 dataset. HyperDistill provides faster convergence and better generalization. Figure 2 shows that Hy- DerDistill shows much faster meta-training convergence than the baselines for all the meta-learning
8

---

Published as a conference paper at ICLR 2022
0.99
0.98
NJIT (10.11 HoperOtettll
DIRAD
0.8
999
9
0.9
Eu.77,, y=2.9
N.IT 110.
HygerDaall
y.00.9
Sslep
0.8
0.7
0.97
0.7
0.6
0.96
0.6
0.5
0.5
0.95
200
400
600
800
1000
200
400
600
800
1000
200
400
600
800
1000
# Inner-opt. Solved
Inner-opt. Solved
Inner-opt. Solved
(a)
(b)
(c)
Figure 3: Cosine similarity to exact RMD in terms of (a) hypergradients g +g00 (b, sccond-order term
$0
The curves in (b) correspond to Eq. (7) with various
100
Meta-test/ Acc.
62
HyperDastil NIFT
13C
0.8
DYMAD
66
1-step
0.6
FO
26
58
0.4
1.
-66
Fitted
300
500
50 100
150
200
Inner-apt.
#JVP per inner-opt.
(a)
(b)
(c)
(d)
(e)
Figure 4: (a,b) Samples collected from Algorithm 4 and correspondingly fitted linear estimators (0). (c) A fitted estimator and the range actual ground-truth estimator (the shaded area one sigma). (d) The stability of 8 estimation. (e) Mcta-test performance vs. computational cost in terms of the number JVPs per inner-opt.
models and datasets considered. We that the convergence of offline method such DrMAD significantly worse than a simple first-order method, demonstrating the importance of frequent update via online optimization. HyperDistill shows significantly better convergence than FO and 1-step because is online and at the same time alleviates the short horizon bias. As a result, Ta- ble 2 shows that the meta-test performance of HyperDistill is significantly better than the baselines, although it requires comparable number of JVPs per each inner- inner-optimiztion.
HyperDistill is reasonable approximation of the true hypergradient. We see from Figure 3(a) that the hypergradient obtained from HyperDistill is more similar to the exact RMD than those ob- tained from FO and 1-step, demonstrating that HyperDistill can actually alleviate the short horizon bias. HyperDistill is even comparable to N. N10, 1) that computes 11 JVPs, whereas HyperDistill computes only single JVP. Such results indicate that the approximation DrMAD in Eq. (16) are accurate enough. Figure 3(b) shows that with careful tuning of (e.g. 0.99), the direction of the approximated second-order term in Eq. (7) can be much second-order term of I-step (y = 0). In Figure 3(c), as HyperDistill distills such a good approx- imation, it can provide a better direction of the second-order term than 1-step. Although the gap may seem marginal, even N.IFT(10, 1) performs similarly, showing that matching the direction of the second-order term without unrolling the full gradient steps is inherently challenging problem. Figure 4(a) and 4(b) show that the samples collected according to Algorithm 4 is largely linear, supporting our choice of Eq. (15). Figure 4(c) and 4(d) show that the range of fitted 0 accurate and stable, explaining why we do not have to perform the estimation frequently. Note that DrMAD approximation (Eq. (16)) is accurate (Figure 3(a) and 3(C)), helping to predict the hypergradient size. HyperDistill is compuatationally efficient. Figure 4(e) shows the superior computational effi- ciency of HyperDistill in terms of the trade-off between meta-test performance and the amount of JVP computations. Note that wall-clock time is roughly proportional to the number of JVPs per inner-optimizzation. In Appendix we can that the actual increase in memory cost and wall- cock time is very marginal compared to 1-step approximation.
used in Eq. (7) and
accurate than the
CONCLUSION
In this work, we proposed a novel HO method, HyperDistill, that can optimize high-dimensional hyperparameters in an online manner. was done by approximating the exact second-orde -oderr term with knowledge distillation. We demonstrated that HyperDistill provides faster coveerenne and better generalization performance based on realistic meta-learning methods and datasets. We also verified that it is thanks to the accurate approximations we proposed.

---

Published as a conference paper at ICLR 2022
Acknowledgements This work was supported by Google AI Focused Research Award, Cen- ter for Applied Research in Artificial Intelligence (CARAI) grant funded by DAPA and ADD (UD190031RD), the Engineering Research Center Program through the National Research Foun- dation of Korea (NRF) funded by the Korean Government MSIT (NRF-201 A5A 1059921), and Institute of Information & communications Technology Planning & Evaluation (ITP) grant funded by the Korea governmen((MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Pro- gram(KAIST)).
REFERENCES
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gen- eralization using meta-regularization. Advances in neural information processing systems, 31, 2018.
Yoshua Bengio. Gradient-based optimization of hyperparameters. Neural computation, 12(8):1889) 1900, 2000.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012.
Justin Domke. Generic methods for optimization-based modeling. In Artificial Intelligence and Statistics, pp. 318-326. PMLR, 2012.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- leatnle for fast adaptation of deep networks. In International conference machine learning, pp. 1126-1 1135. PMLR, 2017.
Sebastian Flennerhag, Pablo G Moreno, Neil D Lawrence, and Andreas Damianou. Transferring knowledge across learning processes. In International Conference on Learning Representations, 2018.
Sebastian Flennerhag, Andrei A Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. In International Conference on Learning Representations, 2019.
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In International Conference Machine Learning, pp. 1165-1173. PMLR, 2017.
Jie Fu, Hongyin Luo, Jiashi Feng, Kian Hsiang Low, and Tat-Seng Chua. Drmad: distilling reverse- mode automatic differentiation for optimizing hyperparameters of deep neural networks. Pro- ceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pp. 1469- 1475, 2016.
Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala. Generalized inner loop meta- learning. arXiv preprint 1rxiv:1910.001227,20 2019.
Daniel Jiwoong Im, Cristina Savin, and Kyunghyun Cho. Online hyperparameter optimization by real-time recurrent learning. arXiv preprint arXiv:2 2021.
Khurram Javed and Martha White. Meta-learning representations for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Hae Beom Lee, Taewook Nam, Eunho Yang, and Sung Ju Hwang. Meta dropout: Learning to perturb latent features for generalization. In International Conference Learning Representations, 2019.
10

---

Published as a conference paper at ICLR 2022
Yoonho Lee and Seungjin Choi. Gradient-basee meta-leaming with learned layerwise metric and subspace. International Conference on Machine Learning, pp. 2927-2936 PMLR, 2018.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta- learning for domain generalization. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few- shot learning. arXiv preprint arXivv/707.09883 2017.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations, 2018.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions hyperparameters by implicit differentiation. In International Conference Artificial Intelligence and Statistics, pp- 1540-1552. PMLR, 2020.
Jelena Luketina, Mathias Berglund, Klaus Greff, and Tapani Raiko. Scalable gradient-basse tuning of continuous regularization hyperparameters. In International conference machine learning, pp. 1952-2960. PMLR, 2016.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient- based hyperparameter optimiza- tion through reversible learning. In International conference machine learning, pp. 211..11111111 2113-2122. PMLR, 2015.
Paul Micaelli and Amos Storkey. Non Non-greedy gradient-baut based hyperparameter optimization over long horizons, 2020.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: wet::0200000000 02999, 2018.
Eunbyung Park and Junier B Oliva. Meta-curvature. Advances in Neural Information Processing Systems, 32, 2019.
Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In International con- ference on machine learning, pp. 737-746. PMLR, 2016.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference Learning Rep- resentations, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference machine learning, pp. 2018.
PMLR,
Jeong Un Ryu, Jaewoong Shin, Hae Beom Lee, and Sung Hwang. Metaperturb: Transferable regularizer for heterogeneous tasks and architectures. Advances in Neural Information Processing Systems, 13:11111115 2020.
Jürgen Schmidhuber. Evolutionary principles self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back- propagation for bilevel optimization. The 22nd International Conference on Artificial Intelligence and Statistics, pp.
PMLR, 2019.
Jaewoong Shin, Hae Beom Lee, Boqing Gong. and Sung Ju Hwang. Large-scale meta-learning with continual trajectory shifting. In International Conference on Machine Learning, pp. 9603-9613. PMLR, 2021.
Jun Shu, Qi Xie. Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- neight-net: Learning an explicit mapping for sample weighting. Advances in neural information processing systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2022
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
Sebastian Thrun and Lorien Pratt (eds.). Learning to Learn. Kluwer Academic Publishers, Norwell, MA, USA, 1998. ISBN 0-7923-8047-9.
Hung- Yu Tseng, Hsin- Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang. Cross-domain few-shot classification via learned feature-wise transformation. In International Conference on Learning Representations, 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Paul Werbos. Backpropagation through time: what it does and how do it. Proceedings of the IEEE, 15500): 1990.
Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989.
Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in
stochastic meta-optimization. In International Conference Learning Representations, 2018.
12

---

Published as a conference paper at ICLR 2022
A DERIVATION OF EQUATION (6)
Let f = fe(w, D) and g = gt for notational simplicity. Note that || = 1. Then,
मe (w, D) = arg min || ʃ gll
arg argmin | g|2
arg min n f² -2 f + g6g
(19)
f
Plugging this into a in Eq. (19) and with the assumption #(w, D) = -f" 2 0, we have
w., D arg min ( - g + 9'g
wDD
arg max(f (f g)²
w,D
arg max fTg
w,D
arg max ( D).
(20)
Note that Eq. (20) results from encoding the closed-form solution fz(w, D) already. Therefore, the above is a joint optimization so that we do not have to repeat alternating optimizations between (w, D) and m.
B DERIVATION OF EQUATION (10)
Let = Se(w, D) and Si f1(w=-1; D:) for notational simplicity. Note that || = =|
[| = and we are given the following inequalities.
|| fll < K][(w, D) - (wi-1, D.)llx, for i= I....,t.
Taking square of both sides and multiplying 60.i,
26,, - S < Klo,,ilww- - wi- 11? + K3o,,.11D- Dill? for i= I,....
Summing the t incqualities over all i = 1,...,tt
S
i=1
K?
de - -;; 2 K3 1=1
Σ
D0 D - D.122
1=1
Rearranging the terms,
81,4 K2
11 w - -w,-1l -K,
8 D D211
i=1
<
Σ
S ft
= =(w D)
13

---

Published as a conference paper at ICLR 2022
C META-VALIDATION PERFORMANCE
ANIL/ ANIL/TinylmageNet
WarpGrad / CIFAR100
WarpGrad TinylmageNet
MWN CIFAR100
60
68
60
66
66
58
65
64
54
56
54
1-stap
52
DIMAO
48
HyperDistil
56
50
59
48
58
200 400 600
800
1K
200
400
600
800
1K
200
400
600 800
1K
200 400 600 800 1K
Inner-opt. Solved
* inner-opt. Solved
# Inner-opt. Solved
Inner-opt. Solved
Figure 5: Meta-validation performance. report mean and confidence intervals over meta- training runs.
Figure shows the vata-validation performance as the meta- veta-training proceeds. We can see that our HyperDistill shows much faster meta-convergence and shows better generalization at convergence than the baselines, which is consistent with the meta-training convergence shown in Figure 2.
D HYPER-HYYERPARAMETES ANALYSIS
ANIL TinylmageNet
WarpGrad CIFAR100
WarpGrad / TinylmageNet
MetaWeightNeet CIFAR100
61
59
60
I-tep
50
59
DrMAD
57
30
0000000000
48
35
68
0.9
0.99
0.949
0.9999
5.9
0.09
3,999
0. 0.9899
3.8
0.99
2. 0.9
0,9999
0.8
0.99
0.999
0,6994
Gamma
Gamma
Gamma
Gamma
Figure 6: Meta-test performance by varying the value y. Red stars denote the actuall y we used for each experiment (we found them with a meta-validation set) and the corresponding performance.
Our algorithm, HyperDistill has hyper-hyperppramter y that we tune with meta-validation set the range {0.9, 0.99, 0.999, 0.9999). Figure shows that with all the values of y and for all the experimental setups consider, HyperDistill outperforms all the baselines with significant margins. This demonstrates that the performance HyperDistill is not much sensitive to the value of 7.
E MORE DETAILS OF METAWEIGHTNET EXPERIMENTS
0.5
0.5
0.5
0.5
6
10
6
10
6
10
6
Loss
Loss
Loss
Loss
(a) 1-step
(b) DrMAD
(c) N.IFT
HyperDistill
Figure 7: Learned loss weighting function with each algorithm.
We provide the additional experimental setup for the MetaWeightNet (Shu et al., 2019) experiments. We use 200(R) - loss weighting network architecture, following the original paper. Also, found that lower bounding the output of the weighting function with 0.1 can stabilize the training. Figure shows the resultant loss weighting function learned with each algorithm. We see that the learned weighting function with HyperDistill tend to output lower values than the baselines.
14

---

Published as a conference paper at ICLR 2022
F COMPUTATIONAL EFFICIENCY
ANII NinylmageNet (Mb) (s inner-opt.)
WarpGrad
MetaWeightNer CIFARIOO
CIFARIO0
tinylmageNet (Mb) inner-opt.)
(Mb) (s/ inner-opt.)
(Mb) (s inner-opt.)
FO 1-step
1430/ 6.23
1092 / 5.24
1840 7,00
N/A 1214 6.17 1262 17.57
1584 / 6.80
1650 / 6.88
3844 / 18.81
DrMAD
1442 20.88
1734 19.83
4148 / 57.09
Neumann IFT
1392 7.98
1578 7.44
3286 / 21.49
1262 / 6.93
HyperDistill
1638 / 6.92
1714/ 8.68
4098 22.15
1206 6.04
Table 3: Memory and wall-clock time required by a single process. We used RTX 2080 Ti for the measure- ments. Note that 1 processes in parallel in actual experiments (meta-batchsize is 4), which requires roughly x4 memory than values reported this table.
Table shows the computational efficiency measured in actual memory usage and average wall- clock time required complete single inner-optimization. We can see from the table that whereas our HyperDistill requires slightly more memory and wall-clock time than 1-step or Neumann IFT method, the additional cost definitely tolerable considering the superior meta-test performance shown in Table 2.
G SINUSOIDAL REGRESSION
Sinusoidal Regression
FO
MSE
Test MSE
1.2
(3, HyperDistill
DrMAD
FO 1-step DrMAD N.IFT 0.502+t.146
0.567
1.0
0.670
0.8
1.086
0.6
HyperDistill 0.327
0.4
5 10
15
20
25 30
Table 4: Meta-test performance.
Inner-opt, Solved
Figure 8: Meta-convergence
In this section, we conduct sinusoidal experiments to demonstrate the efficacy of our method on a regression task.
Task distribution. Each task is to regress a curve sampled from the following distribution of sinusoidal functions; the amplitude and phase sampled from 2/(0.1, and (0, π), respectively. The range of input is [-5,5) 5], and the input and output dimensions are both 1 (Finn et al., 2017). We consider 10-shot regression problems.
Meta-model and network architecture. We set the meta-model to ANIL (Raghu et al., 2019). Given the 4-layer fully-connected ReLU network (1-100-100-100-1 the first three layers are set to the hyperparameter, and only the last layer is adapted to given tasks.
Experimental setup. For inner-optimization of the weights, we use SGD with momentum 0.9 and set the learning rate to y Inner parameter optimization,
0.01 The number inner-steps is 30. For the hyper- use Adam optimizer (Kingma & Ba, 2015) with the learning rate 4 Hyper 0.001. The number of aaner-optimizatte
solved per each meta-convergence M = 30. We perform parallel meta-learning with the meta-batchsize set to 10. Meta-testing: We solve 1000 tasks to measure average mean squared error (MSE), with exactly the same inner-optimization setup as meta-training. We repeat this over different meta-training runs and report mean and 95% confi- dence intervals (see Table 4).
In Figure 8 and Table 4, we see that HyperDistill shows better meta mneaa-oonvrrgene and meta-test performance than all the baselines. Comparing to FO and 1-step baselines, still important to consider longer horizons even in this relatively fewer-shot learning scenario. Also, DrMAD shows poor performance, demonstrating the importance of online optimization.
Results.
that it is
15

---

Published as a conference paper at ICLR 2022
H STANDARD LEARNING SCENARIO
First-stage
Second-stage
1.8
0.75
1-step
1.6
NJFT
0.7
Lo80
1.4
HyperDistill
Test ACC
0.65
Test ACC
1-step N.IFT
70.15
70.85
1.0
0.55
1-step
HyperDistill
72.68
0.5
NJIF
HyperDistil
0.8
Table 5: Meta-test performance.
0.45
0
2K
4K
6K Steps
8K
10K
2K
4K
6K
BK
10K
#
Steps
Figure 9: Meta-train convergence
Figure 10: Meta-test convergence
In this section, instead of meta-learning setting which involves some task distribution, we consider standard learning scenario where we are given only a single classification task.
Two-stage learning. We consider the following two-stage learning scenario, which is reasonable way to cast a standard classification task into a meta-learning problem (Liu et al., 20.8). In the first- stage we split the whole CIFARIO 10 (Krizhevsky et al., 2009) training dataset into two sets with equal number of instances (each with 25,000 instances). We then use one as a training set to optimize the weight wo and the other as a validation set to optimize the hyperparameter A. In the second stage, we merge the two datasets into the original one and re-train w with it from the random initialization, while the learned hyperparameter in the first stage fixed.
Meta WeightNet (Shu et al., 2019). Again, we consider MetaWeightNet which we used in the experimental section 5. We use the same network structure for the loss weighting network and the same label corruption strategy. Note that the original paper uses I-step strategy.
Experimental setup. In the first-stage, for both weight and hyperparameter X, we use SGD with momentum 0.9 and set the learning rate to 0.01 The number of training steps is set to T = 10, 000. In the second-stage, dataset, while fixing À obtained from the first stage. We repeat this two-stage process 5 times and report mean and 95% confidence intervals (see Table 5). Jxpee--hperparameterss For Neumann IFT, we compute the hypergradients (each with 5 inversion steps) for every 5 gradient steps. For HyperDistill, we set 7 0.9.
mentioned above, we reinitialize and train with the merged
Results. In Figure 9, 10 and Table 5, we see that HyperDistill shows much better meta- convergence and meta-test performance than all the baselines. The results demonstrate the effec- tiveness of our method for solving standard HO problems. Note that we cannot consider FO because there is no direct gradient, i.e. g for this MetaWeightNe model. Also, do not consider the offline methods like DrMAD because now the horizon length became too long to backpropagate all the way through the learning process.
FURTHER ANALYSIS ON SHORT HORIZON BIAS
In this section, we demonstrate the effect of short horizon bias by showing the convergence plots with the varying decaying factor (see Figure 11). Note that in this controlled experiment we want to see the effect of only, so we fix the scaling factor as π 1.
WarpGrad / CIFAR100
0.96
y=0
0.94
0=0.
0.92
0-0.6
In the right Figure 11, 7 = 0 corresponds to I-step which com- putes the hypergradients by unrolling only single step, thus suf-
Loss
0.9
Y- 0.99
0.88
fers from short horizon bias. As we increase 7, the convergence roughly improves as well, demonstrating that the short horizon bias can be alleviated by increasing 7. Also, see Figure 3(b) which shows how well the different values of y can recover the true hypergradients. The best performing y seems 0.99 in terms of the cosine similarity to the true hypergradients.
0.86
0.84
0.82
100
200
300 400 500
# Inner-opt. Solved
Figure 1 11: short horizon bias
16

---

Published as a conference paper at ICLR 2022
J EXPERIMENTS WITH FOMAML
WarpGrad CIFARI00
WarpGrad CIFAR100 00
1.55
54
1.5
1.45
Test ACC
Test ACC
50
FO
45. 35.01 11.29
1.35
48
1 1-step
46.03
HyperDistill
51.43
1.3
1.25
1-step
HyperDistill
44
Table 6: Meta-test performance.
0
200
400
600 800
1K
200 400 600 800 1K
Solved
Inner-opt. Solved
Figure 12: Meta-train convergence Figure 13: Meta-val, convergence
In order to demonstrate that HyperDistill works well with other meta-learning algorithms than Reptile (Nichol et al., 2018), we consider first-order MAML (FOMAML) (Finn et al., 2017). Note that we need first-order approximation of MAML because the original MAML with second order derivative is too expensive with the long horizon T = 100. We can see from Figure 12 and Figure 13 that our method provides much faster and better meta-convergence than FO and 1-step. As result, in Table 6 our HyperDistill achieves significantly better meta-test performance than the baselines. Note that we not report the performance of DrMAD and Neumann IFT becuase they fail to meta-converge with FOMAML.
Note that the performance with FOMAMI much worse than with Reptile, which is well known results from the previous literature (Flennerhag al., 2018; Shin et al., 2021). This is because FOMAML ignores the whole learning process except the very last step's s gradient information. This becomes more critical with longer horizons the last step gradient becomes arbitrary uninformative to the initialization, We thus recommend using Reptile instead of FOMAML.
17


Reviews:
[{"review_id": "bQXBXxPP9sE", "reviewer": "Reviewer_G8V6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n-------------\n- This paper is very clearly presented; the background is appropriately detailed and provides sufficient information to compare HyperDistill to the relevant literature. The theory underlying the distillation process is well-explained and easy to follow; the experiments are well-described, easy to interpret, and illustrate well the various advantages of HyperDistill as it compares to other modern methods.\n\n- HyperDistill addresses the shortcomings that are pointed out in the other modern gradient-based HO methods (scalability; limited horizon; applicability to the online setting). In practice, across a variety of datasets, HyperDistill consistently achieves lower validation loss and improves test accuracy.\n\n- HyperDistill is in part motivated by having a long-term horizon (in comparison to the one-step lookahead approach). The authors compare explicitly to this baseline, and show that HyperDistill achieves a better reconstruction of the gradient, and recover one-step lookahead as a special case.\n\n- The set of experiments is Experimentally, HyperDistill improves over the many baselines that are considered by the authors are well-designed, consider a variety of relevant baselines, and presents unambiguously the advantages of HyperDistill.\n\nWeaknesses\n-----------------\nThe main weakness that I see in this work lies in the creation of the distilled dataset D (section 4.2). The authors elide over the difficulty of building this dataset, mentioning only in passing that defining a distance over datasets is difficult. This could be done more carefully, starting by defining precisely what space the minimization problem (11) operates over, and then explicitly solving (or bounding the gap of the proposed solution to) the problem.\n\nQuestions / comments\n------------------------------\n- Nit: it might be worth clarifying the dimensionality and spaces that \\pi, \\w_t, and \\D_t lie in e.g., in (4).\n\n- Similarly, I would clarify the norm being used in (4), especially as the choice of norm becomes important for the distilled dataset later on.\n\n- Some typos: \"longer horizonS\" in Section 1, \"identitcal\" in Section 2, \"completely ignoreS\" in Section 5", "summary_of_the_paper": "This paper develops a practical gradient-based hyperparameter optimization method, HyperDistill, that meets the following criteria\n  a) scalability in hyperparameter dimension and memory constraints, \n  b) accuracy (hyper-gradient update terms do not depend on only the last step of gradient updates)\n  c) applicability to the online setting.\n\nThe main difficulty lies in estimating the gradient of the weights with respect to the hyperparameters. The authors do so by approximating it as a single Jacobian-vector product, using a \"distilled\" weight and dataset pair.\n\nExperimentally, HyperDistill achieves better validation losses and higher quality true hypergradient estimates than a variety of recent, relevant baselines, and does in an efficient fashion.", "summary_of_the_review": "The paper is clearly written, the theoretical justification for the paper is well-presented and intuitive, and experiments confirm the value of the proposed method.\n\nMy only criticism is the comparative lack of care with which the distilled dataset is analyzed; this stands out particularly in contrast to how the  rest of the technical difficulties are addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YgiuFriu9jJ", "reviewer": "Reviewer_w7Q6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is well-motivated, and the paper is clearly written. My major concerns are about the experiments:\n\n1. In this paper, the authors evaluate the performance on two image classification datasets. It might be more convincing if the authors can evaluate the performance on more kinds of tasks (e.g., regression tasks). Besides, can HyperDistill boost the performance under different meta-learning algorithms? More analysis will make the findings more convincing. \n\n2. HyperDistill is a general hyperparameter optimization method. And the authors mention that the proposed method is easy to be applied on traditional hyperparameter optimization (HPO) tasks. Could you evaluate the effectiveness of the proposed method on some supervised learning datasets and compare it with HPO algorithms.\n\n3. Minor: are the tinyImagenet and CIFAR100 tasks few-shot learning tasks?\n\n---- After Rebuttal---\n\nI am happy with the authors' response and decide to keep my score.\n", "summary_of_the_paper": "The authors propose a hyperparameter optimization algorithm in meta-learning, where parameters w/o being involved in the inner loop optimization are treated as hyperparameters. The proposed algorithms approximate the second-order hypergradients via knowledge distillation. They further evaluate the effectiveness on tinyImageNet and CIFAR100.\n", "summary_of_the_review": "The motivation of the proposed method is clear, but more comprehensive experiments are needed to verify the effectiveness of Hyperdistill.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-dQu_yhkeg5", "reviewer": "Reviewer_GYz2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pos:\n1, The topic is interesting and important which can be impactful in machine learning research community.\n2, The idea is reasonable and novel. Applying knowledge distillation in second-order gradient computation is novel.\n\nCons:\n1, The written can be more clear by simplifying the notations.\n2, The experiments can be extensive by applying the methods in more datasets, for instance, meta-dataset.", "summary_of_the_paper": "This paper proposes a new online hyperparameter optimization algorithm. Applying meta-learning for hyperparameter optimization is reasonable and interesting but suffers from the second-order gradient computation. Implicit Function Theorem and Unrolled differentiation can be used to approximate the meta-gradient but also causes various problems. In this paper, the authors propose an interesting method by approximating the second-order approximation with knowledge distillation. ", "summary_of_the_review": "I think the topic is novel and impactful and the idea is novel and interesting. Experiments show the effectiveness of the method. Hence, I recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-WiLvzZPQ1", "reviewer": "Reviewer_zUNV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n* The paper is well-written and easy to follow. The introduction and related work well summarize the current state of the problem and motivate the proposed method.\n* The empirical experiments are rigorous and well supports the claims made in the paper. \n \nConcerns:\n* One question I had was on the definition of the short-horizon bias. I believe that the short-horizon bias occurs for a particular type of hyperparameter such as learning rate. For many regularization hyperparameters, my understanding was we still don’t know the greedy update performs worse in the long run. I think a clarification of the definition in the main text would be helpful. \n* In Figure 2, what would happen to the convergence if we vary \\gamma? As the \\gamma gets smaller, does the convergence look more similar to the 1-step or DrMAD? I think this experiment would strengthen the claim that HyperDistill is robust against short-horizon bias.\n* What are additional hyperparameters HyperDistill introduce? Does HyperDistill robust to this new set of hyperparameters? Since the theory part of the algorithm mostly relies on several approximations/assumptions, I believe that the ablation study on all datasets would be helpful. \n\nMinor comments/questions: \n* In the introduction, it says “computing hypergradients before convergence does not guarantee the quality of the hypergradients”. I was wondering if this statement is true. The IFT works the same way even if the training loss is a constant above 0. \n* In the experiment section, under the task distribution paragraph, TinyImageNet is not cited properly.\n* In appendix E, it would be helpful to directly cite the original paper.\n* In Figure 7, DrMAD stays at 1 constant. Does it mean that it is not learning anything?\n", "summary_of_the_paper": "The paper proposes a novel hyperparameter optimization algorithm in meta-learning to overcome previous limitations of only being able to see a longer horizon and not being scalable to high dimensional hyperparameters. In particular, the authors propose to distill the hypergradient second-order term into a one-step Jacobioan-vector product. The authors show that, with their approximated algorithm, it is possible to perform hyperparameter optimization for higher dimensional hyperparameters and longer horizon length even in an online setting. Empirically, the authors show the advantages of the proposed approach in several benchmark datasets.", "summary_of_the_review": "Overall, I vote for accepting. I believe that the paper is well-written, well-motivated and the claims made in the paper are well justified empirically. Moreover, the content is relevant to the ICLR community.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel methodology for performing meta learning for gradient-based hyperparameter optimization.  The approach overcomes limitations (scaling, e.g.) of previous methods through distilling the gradients of the hyperparameters.  The paper received 4 reviews, of which all were positive (6, 6, 8, 8).  The reviewers appreciated the technical clarity of the paper and found the proposed approach sensible, novel, technically sophisticated and effective.  The main concerns were regarding the comprehensiveness of the experiments and technical presentation of the dataset distillation.  It seems that the reviewers found the author response (lots of results were added) satisfactory regarding these points.  Thus the recommendation is to accept.
Decision: Accept (Spotlight)