On Enforcing Better Conditioned Meta-Learning for Rapid Few-Shot Adaptation
Authors: Markus Hiller
Mehrtash Harandi
Tom Drummond
Venue: NeurIPS 2022 Accept (2022)
Keywords: Few-shot learning
meta-learning
condition number
preconditioning

Abstract: Inspired by the concept of preconditioning, we propose a novel method to increase adaptation speed for gradient-based meta-learning methods without incurring extra parameters. We demonstrate that recasting the optimisation problem to a non-linear least-squares formulation provides a principled way to actively enforce a well-conditioned parameter space for meta-learning models based on the concepts of the condition number and local curvature. Our comprehensive evaluations show that the proposed method significantly outperforms its unconstrained counterpart especially during initial adaptation steps, while achieving comparable or better overall results on several few-shot classification tasks – creating the possibility of dynamically choosing the number of adaptation steps at inference time.

On Enforcing Better Conditioned Meta-Learning for Rapid Few-Shot Adaptation
Markus Hiller! Mehrtash Harandi²
Tom Drummond
'Scoool of Computing and Information Systems, The University of Melbourne
DDpartmennt of Electrical and Computer Systems Engineering, Monash University markus hillerSstudent, unimelb edu. mehrtash www.aaaaaannnnnniaa edu tom. drummond@unimell edu. au
Abstract
Inspired by the concept of preconditioning. we propose novel method to increase adaptation speed for gradient-based meta-learning methods without incurring extra parameters. We demonstrate that recasting the optimisation problem to non-linear least-squares formulation provides principled way to actively enforce a well- conditioned parameter space for meta-learning models based on the concepts the condition number and local curvature. Our comprehensive evaluations show that the proposed method significantly outperforms its unconstrained counterpart especially during initial adaptation steps, while achieving comparable or better overall results on several few-shot classification tasks creating the possibility dynamically choosing the number of adaptation steps inference time.
Introduction
Learning a task from only very limited number of datapoints in the target domain poses central challenge in machine learning. Gradient-based meta-learning as introduced in MAML (Finn) al., 2017) has proved be versatile tool to successfully address such few-shot problems, and
has inspired a significant number of follow-up works tackling a diverse set of applications like
regression (Nichol al. 2018 Finn et al. 2018), image classification (Zintgraf et al. [019 Antoniou
et al. 2019 Rajeswaran et al. 2019), reinforcement (Al-Shedivat et 2018 8) and online continual
learning (Gupta 2020).
Broadly speaking, such approaches define a family of tasks and attempt to solve a bi-level optimization
problem. The outer loop aims to learn effective meta initialization of the parameters capturing properties that generalize well across different training tasks. This learned set of parameters is then adapted to novel and previously unseen task in only a few steps during the inner loop.
Intuitively, one might expect that each step taken during the inner-loop adaptation ought to be as efficient as possible, enabling the model to quickly converge towards the desired goal (i.e., solving the new task). Evaluating popular methods like MAML (Finn et al.) 2017) however shows that this is not the case (Figures and Particularly the initial steps fall short of this expectation. The reason for this behaviour is readily explained: Gradient directions do generally not point towards the actual desired minimum (Figure T) result of the often ill-conditioned parameter space in which the optimization performed.
This behaviour is undesirable for a number of reasons. Having high contribution towards reaching the optimum from the start opens up the possibility of dynamically choosing the number steps to take at inference time, interesting property for applications with limited or changing access to computational resources, and might further lead to a better overall convergence during the final steps.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Convé
Convv
Reshetto
ResMet18
80
25
24
Dimension 2
Test Accuracy
60
80
30
-
10
-50
-10
-25
255
-30
-25
00
2.5
50
1
Dimension
Dimension
Inner loop update steps
(a) KH >> 1
(b) Ki 2
(c)
Figure 1: Importance of a well-conditioned parameter space. a and (b) show the difference in convergence if optimization using SGD is performed in an ill-conditioned parameter space with high condition number ки >> 1 or well-conditioned one (λH R 1), respectively. Depicted is an example for two dimensions using 10 update steps with learning rate 0.5. (c) Average accuracies the CUB test set for the baseline without (MAML) and with our proposed conditioning constraint. Displayed are the first two out of five inner-loop update steps for different network architectures.
With this goal of rapid adaptation from the very first step in mind, we draw inspiration from the field of optimization specifically from the idea of preconditioning. Instead of performing parameter updates directly, suitable preconditioning matrix P can be chosen and multiplied to the gradient Vo. implicitly transforming the optimization problem to a space where updates can be performed more efficiently and resulting in modulated parameter update
((k) (k-1)) P oi (
The reader might notice that this modified gradient descent update rule directly reduces to the well- known Newton's 's method the preconditioning matrix chosen to be the inverse of the Hessian matrix (P H with H p0a0 or an approximation thereof, and would thus enjoy a quadratic convergence rate via uppropriate scaling of the gradients based on the local curvature. The actual choice and computation of preconditioner is however non-trivial for most optimization problems, especially in the context of deep learning. Variants developed recent years (L.i et al. 2017) Gupta al. 2018: Park & Oliva 2019 Simon et al.] (022)) can all be seen as proposing different ways of estimating suitable preconditioning matrices, often leading to a significant increase in parameters.
8²
In contrast to previous works, we propose a new way to achieve preconditioning without incurring any additional parameters. Reconsidering the definition of preconditioned optimization stated in Equation
Our method:
II. we raise the following hypothesis:
If fe is solety parameterized by 0 and is learned, there exists a set of parameters that achieves comparable performance on unseen tasks but possesses low condition number which allows fast convergence with gradient descent when choosing the preconditioning matrix to P I
In other words, the preconditioning matrix P can be absorbed into this 'new' Icarnable function fø- enabling the problem to formulated and optimized directly well-conditioned space.
This raises the following questions: (i) Can the conditioning property of a model be actively influenced without negatively affecting its ability to converge and to solve given task? (ii) If so, does this indeed lead to faster aas-tttaptt In this paper, we set out to provide insights and answers to these and additional questions. In particular, our contributions include the following:
1. We show that the condition number of a network is clear indicator for its few-step performance on an unseen task.
2. We demonstrate a principled way to derive the condition number network by reformulat- ing the optimization problem to non-linear least- squares form over the space of available samples the support set.
3. We introduce novel cost function based around the condition number to actively encourage learning of a well-conditioned parameter space for the model during training, allowing for rapid adaptation at inference time in very few steps.

---

4. We conduct in nn-depth analyses regarding different network architectures, ability to continue adaptation beyond the training horizon, performance regarding an extended set of N -ay --shot few-shot classification scenarios, and demonstrate the efficacy of our method on all five popular few-shot classification benchmarks.
2 Preliminaries
One highly influential algorithm that tackles few-shot learning in gradient-based fashion has been Model Agnostic Meta-Learning (MAML) proposed by Finn et al. (2017). We start by introducing this algorithm to define the specific problem setting we are aiming to solve and to provide context for the following discussions. This is followed by brief overview of the condition number which constitutes one of the main concepts our method based on.
2.1 Problem setting and MAML
Let Dirain and D be the training and validation set of some given task T ~ p(T) (e.g. image [D] classification), respectively. Further, assume that D = {æ, V/ Vi)==1 1- with 2; X, yi y for some small [D) The predictor function of the model is denoted as f X x R + y and is parameterized by IR. MAML (Finn et 2017) then seeks determine a universal meta initialization 0* by solving
Σ C (Dral, (Mtraii 0*)):
arg min
(2)
Trpp()
where the inner-loop specispc parameters are given by
( - 0 aV aV gi
(Dirain o
for & E 21. K] with 0 = and step size a. In other words, given a task T MAML starts from the meta-initialisation 0* and performs K gradient update steps (so-called inner-loop updates) using the few available training samples D for this specific task to obtain the adapted specifc parameters O!*) (Equation (3. It then uses the unseen samples of D of this task together with the adapted parameters {!k) to improve the meta-initialisation 0* in the so-called outer-loop update (Equation (2) Note that this is possible since (k)) is dependent on its initialisation 0º 0* as defined in Equation and gradients can thus be computed throughout all K inner-loop updates.
2.2 Condition number a matrix
One way to quantify local curvature of parameter space and represent the expected convergence behaviour using single scalar value condition number кy of the Hessian H, function that be optimized. For a square-summable sequence space g² (i.e., Euclidean), the condition number of H can be computed via its maximum and minimum singular values o (H,), or its eigenvalues X (H,)
) in case H normal,
| (H,) Mminn HH,)
Amax (Hy) Amin (Hy)]
OR
KH
if H
The high condition number associated with the optimization problem depicted in Figure (a) repre- sents higher difference in local curvature across the parameter space, leading slow convergence caused by the initial gradients not pointing towards the minimum. In contrast, the better conditioned problem shown in (b) with significantly lower condition number Kh z approaches the desired minimum on direct way very few steps.
Learning a better-conditioned parameter space
We start this section by introducing how the objective used in few-shot learning can be reformulated as linear least-squares problem, providing the basis to approximate its Hessian. We then introduce our proposed method to enforce well conditioned parameter space and discuss variation with constraining only parameter subset that allows better scalability models with higher parameter counts.

---

3.1 Reformulated problem setting
We reconsider the minimization objective of one inner-loop step k used perform the task specific adaptation. The average loss over the support set samples in the inner loop can be reformulated to describe non-linear least squares problem
[D
[
C C(Dtrain ((x)
Σ
( Dis (
Ti (00)))
(5)
Diraann
with r: (b representing the residual of each individual sample and
| the number of
available task- speciffc samples.
As introduced in the previous section, the Hessian of function describes its local curvature and can be used to gain insights into the convergence behaviour if the function is to be minimized via some gradient-based optimisation method (Figure T Note that the dependence of gradient, Jacobian and Hessian on the dataset Dirrin and parameters ((k) is being partially dropped from the notation for improved readability during the following derivation. The dependence of the residual r; r:(O0 (0 on the
parameters (()) is additionally abbreviated via its simplified version r; (0), and Dirain is referred to
as D. Using the residual-based notation of Equation (5) together with the introduced simplifications
regarding notation, the gradient of (DDrainn e with respect to some parameter f is given by
I 9j = 2
Ôr, 0 08,
(6)
2=
and the elements of its Hessian, computed by differentiating the gradient elements gj with respect to parameter mn are given by
DD1
(r)(0) (r:(0)
+ r:(()) (r)
H jm 2
00,
D0...
30 ,0,00m.
Ignoring the second-orde derivative terms of the Hessian leads to the well-known Gauss-Newton method in which the Hessian approximated via the use of the Jacobian in the form of
D
(r)
Ør.(0)
DD
H
R 2
2
JaJan
or 2J
(8)
1=1 08;
00 m
is to be noted that we did not modify any component of the original problem formulation at any point. Thus, the Jacobian product approximating the Hessian of our objective function directly reflects the convergence behaviour of the optimisation problem. More specifically, computing the condition number Kи (Section 2.2 provides scalar measure how well- ill-posed the problem is.
3.2 Conditioning constraint and overall objective
Computing the condition number by using the maximum and minimum eigenvalues as defined in Equation 44 ignores the distribution of the remaining | eigenvalluss and would unnecessarily weaken the training signal. Instead we propose to use measure considering the distribution of all eigenvalues X(H) based the observation that well-conditioned problem with ни s 1 can only be achieved all eigenvalues are almost identical. We model this requirement by penalizing high variance of the approximated Hessian's 's logarithmic eigenvalues and define this conditioning constraint as new loss function over the K inner-loop updates
K
Lx (o(k) (g())(trraaa))) (Derain 0²))
Σ
Var (logio (> (J(4))(()T)
K
This formulation contains two design choices. The first choice to use logarithmic eigenvalues is founded in the fact that the actual magnitude is irrelevant for the convergence property since only its relative magnitude compared to all other eigenvalues is of importance. Using the logarithmic

---

Table Validating baseline re-implementation. Results of our re-implementation vs. the original work of Finn et al. (2017). Means and 95% confidence intervals of 600 randomly generated test
episodes for 5-way I-shot and 5-way 5-shot classification on the mindmmgeNNet using a Conv4 model.
5-shot
1-shot
Method
Reported
Ours
Reported
Ours
MAMI (Finn et al.
2017
63.1140.92
64.50=0.69
18.70+1.84
48.15+0.80
form to compute the variance provides scale-invariant way of penalising deviation and thus aligns with the ratio-based definition of the condition number, since returns the same loss penalty for A 0.1, A2 = 0.2 as for Àı = 10, A2 20 both of which would have an identical condition number x = 2 and thus be considered to have similar convergence properties. The second choice of equally weighting all K inner-loop update steps is based simplicity, and future research might explore whether non-equal weighting is beneficial.
Due to the number of samples being significantly smaller than the number of parameters and the resulting rank-deficiency of J J, only subset of size |Dtrain of the total parameter space is directly influenced by the condition loss Lx at each task iteration, whereas the remainder is its null space. This allows us instead compute the smaller product JJ T with equivalent non-zero eigenvalues. to be noted that the gradients used for this computation will however backpropagate to all parameters involved forming J, thus appropriately constraining all relevant parameters.
The overall objective for the outer loop of the meta-learning process with weighting factor
Σ L (DDral 0 (Dirain 0*))
7Cr (0(K) (Derain 0*))
arg min
(10)
0-
Prep(T)
encourages the network to learn a set of initialization parameters 0" that not only generalizes well across tasks (C) but also possesses well-posed problem structure with low condition number in the inner loop that allows rapid adaptation in very few steps (L,). This is possible since the condition number representing the convergence quality computed via the approximated Hessian which in turn is dependent on the network's 's parameters, allowing the computation of gradients all the way back to the initialization 0"
3.3 Constraining parameter subset
To reduce complexity, improve scalability and considering the fact that the network might face some trade-off between accuracy and well-conditioned problem formulation induced by the objective in Equation (10), we build upon the findings of Raghu et al. (2019) that mainly the last few layers undergo significant adaptation during inner-loop updates. We thus propose and investigate constraining only a specific subset the network's overall parameters through our proposed conditioning loss, namely the indicated 'last few layers'. This significantly reduces computational requirements and makes our approach more scalable deeper networks. The only modification required slight adaptation of Equation (10 by replacing ((k) in L.v with the new parameter subset (K) C (kK)
4 Experimental evaluation
In our experimental evaluation, we aim to gain empirical insights to answer the following questions: (i) Can the conditioning property of model be actively influenced without negatively affecting its ability to converge and to solve a task? (ii) If so, does this indeed lead to faster task-adaptation? (iii) How does it impact networks of various depths and trained on different datasets? (iv) Does better-conditioned parameter space provide advantages across different many-way multi-shot settings? (v) Can we expect continued adaptation beyond the training horizon?
4.1 Experimental setup
We choose MAML. (Finn et al. 2017) as our baseline for comparison due to its vast popularity and the influence regarding follow-up works it inspired since its introduction (see Section To validate
5

---

MAML ConvA Convé
Convé ours Conv6
MANL Convé
ours Conv4 ours Conv6
MAML Conu4
ours Conu
00.0
MANI Convé
Conv6
Convi
70.0
4.5
Conditon numbar [10ghal
00.0
4.0
Validaiion accureee
00.0
4.5
40
4.0
19
2.5
10
2.0
1.3
20
1.0
2 Innenloop adaptetion stops
200
400
000
500
1000
200
400
000
800
1000
Training Reeraiion
Training Reeaaions
(a)
(b)
(c)
Figure 2: Enforcing well-conditioned adaptation. Comparison of baseline method MAML FFinn et al. 2017) and the same method with our proposed conditioning constraint. Results were obtained on Imgred for two different architectures. (a) Test accuracy over inner-loop adaptation steps. (b) Validation accuracy of inner-loop adaptation step 1 during training. (c) Condition number of parameters adaptation step during training, depicted in logarithmic scale.
the implementation of our MAML baseline, we report the results achieved by the original and our re-implemented baseline in Table T showing results are on-par with the original version.
We evaluate our method on all five popular few-shot classification datasets, namely minilmageNet (Vinyals et al., 2016 Ravi & Larochelle] (017)) liered (Ren et al. 20188). CIFAR-FS (Bertinetto et al. 2019). FC100 (Oreshkin et al. 2018) and
(Wah et al.
2011), hereafter referred to as CCU' We follow previous work like Chen et al. (2019) and report the averaged test accuracies of 600 randomly sampled experiments using the A samples of the respective k-shot setting as support and 16 query samples for each class. Our baselines and methods are trained on a single NVIDIA RTX 3090 for 60,000 episodes for 1-shot and 40,000 for 5-shot tasks, with the best model picked based highest validation accuracy. We use 5 inner-loop updates and y 1.
4.2 Condition number indicates few-step performance
As very first point, we provide insights into the basis this work is built on: The condition number of a network is closely related to its adaptation capabilities using very few steps. To this end, we take a closer look at the behaviour of a network's condition number during training and the associated performance in terms of validation accuracy after the first inner-loop update step (out of five), for methods trained without or with our proposed conditioning objective (Equation (IO). The results depicted in [igureZdemmosse
a clear dependence between the progress of the validation accuracy (b) and its associated condition number (c) in two ways: 1) Low condition number seems to generally align with high validation accuracy, and 2) sudden increases in condition number indicate instability regarding the validation accuracy. We further observe that the lower condition numbers of both models trained with the proposed constraint ('ours') seem improve convergence speed early on during training, leading to slightly steeper increase in validation accuracy.
This dependence between condition number and validation accuracy exists across all inner-loop update steps, with corresponding discussion provided in the supplementary material of this paper.
4.3 Per-step adaptation with enforced conditioning
Having demonstrated the existing dependence between the condition number and performance of model, we now concentrate on our hypothesis raised in the introduction that actively constraining the network's parameter space to enforce better conditioning is possible without negatively affecting overall performance on unseen tasks. In Figure () can be clearly observed that models trained with L.c ('ours') exhibit significantly lower condition numbers than their unconstrained counterparts (cMAML'). " demonstrating that our proposed loss is indeed able to actively encourage learning of a better-condiiioned parameter space.
The performance of these models achieved on novel tasks of the test set depicted across all five inner-loop update steps in Figure 2 aa) for a 5-way 5-shot scenario. While all networks seem to learn

---

S-way wawwy
35-way
shott
20
70
Accuracy
30
30
20
20
3
3
Inner loop update steps
Inner loop update steps
(a)
(b)
Figure 3: Many-way multi-shot scenarios. Reported are the average accuracy for the baseline with- oul conditioning (MAML) and the version with our proposed conditioning loss on the minilmageNet testset. (a) depicts the results for various N-way 5-shot, (b) for 5-way K -shhot scenarios.
similarly general meta initialization (step 0), the effect of using our proposed Lx is significant especially during the first steps, increasing the test accuracy by 35.68% and 39.48% (Conv4), and 39.98% and 40.95% (Conv6) for adaptation steps and 2, respectively. While the unconstrained models mostly catch up during the following steps, their adaptation behaviour prevents any use of such trained models with fewer than the maximum number of adaptation steps used in training (here 5) in stark contrast to our models which achieve 86% (Conv4) and 89% (Conv6) of their top accuracy already after one single step. We further observed that the faster initial convergence of our method seems to consistently lead to slightly higher overall performance across most experiments we performed, especially for deeper architectures (Section 4.5 with K 2
4.4 Effect of constrained parameter subsets
We now take closer look two distinet questions: (i) is the condition number of parameter subset 'representative' of the actual condition number of the network (based on all parameters) and if so, (ii) how do different subsets influence the networks adaptation performance? To this end, we train a Conv4 architecture and enforce conditioning on various parameter subsets of the last few layers. The development of the average condition numbers during training with respect to only the parameters of the classifier ("cls") and to all parameters of the network ('all") indicate that the condition number of this subset indeed representative for the entire network (Figure D an observation that applies to all evaluated subsets except the parameters of the embedding layer's batchnorm ('eBN"). For further discussion, please see the supplementary material. Table shows that all methods demonstrate significant improvements of test accuracy in the first few steps compared to the unconstrained baseline, and all except 'eBN" achieve superior overall performance after steps. Despite these results indicating the slight advantage of selecting subset 'emb', we conduct the following experiments using the "cls' subset for the reason that constraining the parameters of the classifier is independent of the backbone and allows for easier transition between different models.
4.5 Increased network depths and adaptation beyond the training horizon
We conduct an extensive set of experiments with four different architectures (Conv4, Conv6, ResNet10, ResNet18) comparing the baseline and our method across all 5 datasets to evaluate the performance for changing network depths, both in 1-shot and 5-shot settings. The results in Table 4 demonstrate the efficacy of the proposed constraint across all models (please refer to the supplementary material for complete results). We additionally observe that the center-conditioned parameter space seems to also encourage convergence to a better overall optimium especially for deeper architectures, e.g. ResNet18 with 73. 73.28% vs. baseline 71.06%. Interestingly, smaller uncon- strained baseline models like Conv4 seem to naturally learn better conditioned parameter space in 1 1-shot scenarios for most datasets, which might indicate some relationship between the condition number and the combination task difficulty, gradient noise and network capacity. Additional results in the lower part of Table 4 show that while our methods are trained using five inner-loop update steps and achieve very fast adaptation from the very first step onwards, they are additionally able to continuously adapt further beyond their training horizon when given the possibility.
7

---

Table 2: Influence of condition parameter selection on test accu- racy (CUB dataset, Conv4). The condition loss Lx is enforced w.r.t. one multiple parameter groups. cls: classifier (last layer): emb: embedding layer; eBN: BN of embedding layer. Highlighted in gray the 'default' setting used throughout this work (simplicity).
Avg. condition number (logiel
1.8
w.r.t. cls
3.7
MH
2.6
C. w.r.t.
5-way 5-shot
cls emb eRN
step 1
step
step
step 41
step
1.3
20.04 02
24.16+0.86
64.30=0.s
74.71+0.78
77.06=0.00
1.2
61.78 18.7820.7 75
78.0810.71
75.6420.71
78.7520.cs
77.21
1.1
59.85.0 90
73.54 ro
76.61 20.00
77.73:0.m
78.1410.67
200
400
600
800
1000
Training Iterations
62.22.20.7
60.87..0.9
73.04...77
74.15110.74
74.86
V
64.27
73.4520.71
76.0220.09
77.30+0.67
77.77:0.05
Figure 4: Average condition number during training w.r.t
55.54=0.83
71.63=0.76
75.55=0.71
76.78+0.70
77.12=0.0s
63.46=2076
73.4920.74
76.31=0.59
77.67
78.1020.07
60.93
71.39
7:18330.m1
76.40=0.0e
27.28
cls subset and all parameters.
4.6 Many-way and multi-shot scenarios
We finally investigate the influence of better conditioned parameter space for N-way K K-shot applications with more than the 'default' N = classes and K >> 5 support samples. Figure demonstrates that our method sustains the previously demonstrated fast adaptation from the first step on and seems to handle settings with the increased difficulty of more classes slightly better than the baseline. Scenarios that provide more than 5 samples per class (Figure (b) show that while our method continues to consistently outperform the baseline by significant margins during the first few steps (e.g. by 53% for 20-shot at step 3), the convergence of the unconstrained baseline seems to get delayed even further with increasing sample numbers indicating that the conditioning property can not be improved simply through more available data, but is in contrast rather adversely affected.
4.7 Preconditioning methods - Numbeer of parameters and performance
We compare our approach to other recently- published preconditioning methods for a 5- way 5-shot classification task evaluated on the minilmageNeet test set. In this context, we want to draw attention the fact that while most published
ours (Corv6)
Methods
70
HAMI
Meta-SC3D
ModGrad
MC
Test Accuracy
ModGrad
Warp Warp-MAML
methods compare based on the used backbone, is worth considering their introduction of addi- tional parameters that are required at both training and inference time. We therefore outline the gains in accuracy in relation to the total number of pa- rameters in Fi While all methods are able to improve the test accuracy upon their respective baselines, this comes at a cost. Meta-SGD (Li)
Warp-MtAML
(Conv4)
MC
67
MAML
65
Mcta-SG0
63
32.9K
12L.KK
463-4 63
1.2M
10.8M
Total Parameters
et al. 2017) is able to improve by 1.5% relative to
Figure 5: Preconditioning methods, number of
its Conv4 MAML baseline (Finn et al. (017))
parameters and accuracies obtained for 5-way 5-shot evaluated the minilmageNet test set.
but doubles the number of paraneters (+100%)
ModGrad (Simon et al. (2220) is able to notably
outperform its Conv4 (64) baseline by 7.2% but requires an increase in parameters of +873%, while Warp-MAML. (Flennerhag et al.) 2019) and MC (Park & Oliva 2019) achieve their relative improve- ments of 3.5% and 3.0% at the cost of incurring additional parameter costs of +96% and +2235%, respectively (both Conv4 (128).. Our approach in contrast improves upon all baselines without adding any additional parameters property that allows us to instead choose larger backbones like Conv6 and thus significantly outperform other methods across the entire parameter-vs.-acuuee spectrum. Please refer the supplementary material for more detailed information.
4.8 Taking fewer but bigger steps at inference time
Our motivational example depicted in Figure T implies that well-conditioned parameter space should provide more 'direct' path towards the optimum and thus enable us to take fewer but bigger steps.
8

---

We therefore investigate the behaviour of our method when taking only one step but with greater step size inference time. Note that the network has been trained as in previous experiments with the default setting of using 5 inner-loop adaptation steps with a learning rate 'Ir' during training.
Our obtained results (Table 3) clearly indicate that better conditioned parameter space does indeed provide the ability to take fewer but bigger steps towards the minimum to adapt faster at inference time. While our approach evaluated on the same setting as used during training (5 steps with 'Ir') still slightly outperforms its 1-step variants and thus hints at a not (yet) ideally-conditioned parameter space, our method taking only step with an appropriately scaled '5-Ir' step size demonstrates promising task adaptation and even outperforms the respective 5-step MAML baseline model.
Table 3: Varying number of steps and step size at inference time. Test accuracy for 5-way 5-shot on minilmageNet. Model has been trained with default settings of 5 inner-loop update steps with learning rate Ir. Reported are the classification accuracies on the test set, averaged over 600 tasks.
Conv6
Osteps
step- Ir
step- 2-1r
1 step 3-Ir
1step 4-1r
1 step 5-Ir
5steps Ir
MAML
20.0020.00
20.1920.07
20.0020.00
20. 1420.0
23. 23.28:0.37
26.64 26.64
65.96 0.71
ours
20.1410::0
62.31
65.6710.1
65.8719.71
65.7210.71
66.5410.70
68.43 0.71
5
Limitations
Applying our proposed condition loss in its naive form to all parameters of very deep networks might be infeasible due to high parameter counts involved in computing the Jacobians. Our experiments however clearly demonstrate that constraining a representative subset like the classifier is sufficient to improve the overall conditioning property of the model and can significantly improve the convergence behaviour during initial steps (Section 4.4).
Our method further makes use of second order derivatives (just like the baseline MAML) and thus needs to store the computational graph throughout the inner-loop updates, property that didn') 't raise any issues in our experiments but might be a point of concern for applications with extremely limited memory- a general point worth considering with second-order methods.
Related work
Gradient-basee based meta-learning. Methods most related to approach generally aim to find a set of model parameters that can adapt to novel tasks within few steps of gradient descent. Early seminal works (Andrychowiez et al., 2016 Finn al., 2017) have spiked significant follow-up research like first order variants (Nichol et al. 2018). probabilistic reformulations (Finn et al.) 2018) and generative
approaches (Rusu et al. 20188, among many more (Ravi Larochelle 2017 Rajeswaran et al. 2019
Zintgraf et al. 2019). While some have derived theoretical guarantees (Balcan et al. [019 Grant al., 20188. others improved training stability (Antoniou al. (019)). Two main challenges have to be overcome to solve a previously unseen task with a highly limited number of data samples in this way: overfitting (Sun et al., 2019) and rapid adaptation. While several works have attempted the first (Rusu et al. 2018: Zhou et al. 2018: Zintgraf et al.) (2019), learning an efficient way of updating the model's 's parameters that enables fast adaptation remains an open problem (Flennerhag et al.] (211)) On common drawback is the unknown adaptation quality, i.e. how many updates have to be performed to achieve satisfactory results on a new task. This is commonly solved by either simply choosing the de-facto default value of five steps, or the highest number of steps that seems feasible for a given task based on empirical results. In contrast, our proposed approach of constraining methods to learn well-conditioned parameter space achieves fast adaptation from the very first step onwards and gradually increases, allowing users to dynamically choose the number at inference time.
Gradient modulation. Few works have attempted to tackle the challenge of finding more suitable way of updating the parameters. Early methods use LSTMs to update models iteratively by learning both the initialisation and update strategy ( (Andrychowicz et al.) 2016: Ravi & Larochelle 2017 but are difficult to train (Li et al., 2017). This idea has been further developed by Li et al. 2017)) towards MGD, where the initialisation is meta-leamed together with update direction and parameter-specific learning rates. While this can be seen as a form of preconditioned optimisation, more direct attempts have recently been proposed. An interleaving- based method is used to improve

---

Table 4: Increasing the network depth and adaptation steps. Evaluations are conducted all five popular FSL datasets: CUB-200 2---2000-211 2011 (Wah et al. (2011) minilmageNet (Vinyals et al. 2016).
rigredlmageNet (Ren et al. 2018 CIFAR-FS (Bertinetto et al. 2019) and FC100 (Oreshkin et al.)
2018). Reported are the classification accuracies on the unseen test set, averaged over 600 tasks.
S-shot
I-shot
Network
Method
step 17
step 21
step 31
step
step 51
step 1f
step 2
step 3
step 41
step
Cono4
MAML
20.00=0:01
20.00+0.00
52.87+0.76
61.12e0.7
61.50lcc.0
38, 76
12.33-0.74
15, 99.99
47.07+9.81
18,15-0.0
ours
a6.09aw.s
28
64.01am.77
64.89zu07
65.26
43.6820.7
47.65-
48. 48.502077
48.76
48.94-
minlangee
Conv6
MAML
20.19=wer
24 21.17am:
57,25
64.24
65. 67.96
21.40a
30.85=0.0
45.91
500300a0..
61.22-0.a
62.31
06.6611.71
67.63.0.7
68.21107
68.4316m7
50:9210ss
52.98_....
63.1810.p
53.28.0..
53.34_
ResNet10
MAMI.
20.06.0o5
20.
10.03
41.7410.74
69.40 63.2810.7
69.431o.71
20.0210.01
22.14 0.30
50.99
56.47.10.0
57.35 0.50
ours
63. 73
68 96400/2
71.40a0.70
72.1850.00
72. 72.46400.7 o7
50.446a0.2
66.86
54
a4
57.9749..5
08, 56
R8
MAML
20.12:006
20. 00000.00 oo
71. 10.00aa... n4
20.0220.00 00
20.79-.0.17
b2.81
56.4240.00
50.84=000
ours
68.60 18.60=071
72 06
72.98
13.10 73. 28.0.00
511 16600000
57.01-0.51
57.31=009
57.52
57.61-099
Dataset
Method
step 1
step 21
step 31
step 4' step 51
step 11
step 2"
step 31
step 41
step 5"
MAML
20.00=0.00
30.48so.11
43.96ian.s0
79.566074
81.564e.::
20.02eo..z
25.57=0.54
72.20=0.s
74.0640.00
74.52-084
CUB
ours
81.57=001
81.0
$1.00e0.03
81.1740.44
84163300
81
$1.6640.64
66,36 os
72,93=0.56
713380077
71.9640.07
75.22-0097
ResNetf
allmnNerr
MAMI
20.00.0e1
20.10
10
38.0010.m
68.5710as
70.901emm
20:0210.0
21.19_0a
51.301oss
56.8011.00
57.71_1100
OUTS
70.13
73.60
74.3130.74
74.55e074
74.67a0.74
54.79as..0
57.44=w.an
57.64:007
57.71
67.80=w.a
CIFAR-FS
MAML
20.01
20 00.0433.00
45.75a0..a
75.001001
79.59 no
20.00 00
23.41=0.30
133 04
67 67.06=2..9 on 67 67.44-0
ouNS
10.3
79.4916m07
79.96.10.6
60.0010
80. 18
65.73.1..0
68.339_..0
08.751101
69.04.1.00
69.00_1 1.00
MAML
20.00=0.0
20. 00e0..0
20.000=0.00
18, 16:076
48. S.
20.000e0.00
20,01=0.02
32 30
31.37=0.00
35, 89.19-0.70
FC100
ours
77.11
50.
1650.7
50.8220.74
51 088
51.2246.74 Ra
33.24....0
36.05-0.71
76.70
36.90aa.m
37.02-0.7
Network
Method
step
step
step 25
step
step 1001
step 51
step 101
step 251
step 501
step 1001
Conv4
ours
77.21=00r
77. 184.000
77.74 77 77.88800s
78 06
63 26:0.00
63.12-0.50
633 1e
63.5444000
G3. G3.17-1.00
Conv6
80.65e0es
80.90sa.e 81.15a0.32
$1.25e0022
31.335a..oz
68.87a1.
69.12-1.00
69.19s1.05
69. 16.100
69. 69.15-1.00
ResNor10
ours
83.82en.59
84 51.0940.0 81.11+0.10
81,2660037
84 2844.8
71.9940.06
71.90-0.91
75. 77
75.2140.00
75.33-9097
R8sNec18
ours
84.662u.4
84.8316.
84.9430.3 85.01zuas
85.01
75.22
75.52=waa6 75.66
75.7430.0
75.81-was
the convergence speed via meta-learning parameterized preconditioning matrix by Flennerhag et al. (201)), while Park & Oliva (2019) instead learn curvature information to transform the gradients driving the task-specific adaptation and SSmon et al. (2020) modulate the gradients using an additional module and low-rank approximation. However, all these approaches aim to estimate some form of additional preconditioning matrix transform the gradients, incurring an extensive number of additional parameters and overhead at both training and inference time. In contrast, our method imposes constraint directly onto the parameter space of our network and thus targets the same idea of well-conditioned optimisation space but within the already existing capacity of the network.
Condition number. The relevance of the condition number has been discussed in the context of deep learning by Goodfellow et al. (2016), and few works have made use of it as a 'passive metric', e.g. to monitor and assess quality of convergence in the context of Generative Adversarial Networks (Zuo) et al. (02TT. We take a different approach and show that in the bi- nature of gradient- based few-shot meta-learning, we can actively manipulate the learning process of the function learn a well optimisation problem that demonstrates superior convergence properties.
7 Conclusion
In this paper, we demonstrated how to actively encourage a deep network to learn a better-condiiioned parameter space that allows significantly faster adaptation to novel tasks in few-shot settings. To this end, we introduced a new constraint that is easily integrated into optimisation-base few-shot methods via an additive loss term. Our constraint is based around the condition number of local curvatures and does not incur any additional parameters. We showed the efficacy of our approach through comprehensive experiments on five datasets with various different architectures and few-show scenarios, significantly outperforming unconstrained methods during initial adaptation.
Acknowledgements. The authors would like to thank Finn al. (2017) and Chen et al. 20199 for sharing their insights, experimental details and code.
Part of this research was conducted while MH was with Monash University. This research was partially undertaken using the LIFF HPC-GPGPU Facility hosted at the University of Melbourne. This Facility was established with the assistance of LIEF Grant LE170100200.
10

---

References
Al-Shedivat, M.. Bansal, T. Burda, Y., Sutskever, I., Mordatch, I., and Abbeel, P. Continuous adaptation via meta-learning in nonstationary and competitive environments. International Conference on Learning Representations, 2018.
Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., Shillingford, B., and De Freitas, N. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, 2016.
Antoniou, A., Edwards, H., and Storkey, A. How to train your MAML. In International Conference Learning Representations, 2019.
Balcan, M.-F,, Khodak, M., and Talwalkar, Provable guarantees gradient-basee based meta- meta-learning. In International Conference on Machine Learning. PMLR, 2019.
Bertinetto, L., Henriques, J. F., Tom, P.. and Vedaldi, A. Metanleanning with differentiable closed-form solvers. In International Conference on Learning Representations, 2019.
Chen, W.-Y,, Liu, Y.-.., Kira, Z., Wang, Y.-.., and Huang. J.-B. A closer look at few-shot classification. In International Conference on Learning Representations, 2019.
Finn, C., Abbeel. P., and Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning PMLR, 2017.
Finn, C., Xu, K., and Levine, Probabilistic model-agnostic meta-learning. Advances Neural Information Processing Systems, 2018.
Flennerhag, S., Rusu, A. A., Pascanu, R., Visin, F,, Yin, H., and Hadsell, R. Meta-lcarning with warped gradient descent. In International Conference on Learning Representations. 2019.
Goodfellow, L, Bengio, Y., and Courville, A. Deep learning. MIT press, 2016.
Grant, E., Finn, C.,, Levine, S., Darrell, T., and Griffiths, T. Recasting gradient-based meta-learning as hierarchical Bayes, In International Conference on Learning Representations, 2018.
Gupta, G., Yadav, K., and Paull, L. Look-ahead
lcarning for continual learning. In Advances Neural Information Processing Systems, volume 33, 2020.
Gupta, V., Koren, T., and Singer, Y. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning. PMLR, 2018.
Li, Z., Zhou, F., Chen, F., and Li, H. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv: arXiv: 707.09835, 2017.
Nichol, A., Achiam, J., and Schulman, J. On first-order meta-learning algorithms. arXiv preprint vrXiv:183300299) 2018.
Oreshkin, B., Rodríguez López, P., and Lacoste, A. Tadam: Task dependent adaptive metric for improved few-shot learning, Advances in Neural Information Processing Systems, 31, 2018.
Park, E. and Oliva, J. B. Meta-curvature. Advances in Neural Information Processing Systems, 32, 2019.
Raghu, A., Raghu, M., Bengio, S., and Vinyals, O. Rapid learning or feature reuse? Towards understanding the effectiveness of MAML. In International Conference on Learning Representations, 2019.
Rajeswaran, A., Finn, C., Kakade, S. M., and Levine, S. Mcta-Icaminn with implicit gradients. In Advances in Neural Information Processing Systems, 2019.
Ravi, S. and Larochelle, H. Optimization model for few-shot learning. In International Conference Learning Representations, 2017.
Ren, M., Triantafilluu, E., Ravi, S., Snell, J., Swersky, K., Tenenbaum, B., Larochelle, H., and Zemel.
R. S. Meta-lcaming for semi-supervised few-shot classification. International Conference on Learning Representations, 2018.
Rusu, A. A., Rao. D., Sygnowski, J., Vinyals, O., Pascanu, R., Osindero, S., and Hadsell, R. Meta-learning with
latent embedding optimization. In International Conference Learning Representations, 2018.
11

---

Simon, C.. Koniusz, P., Nock, R., and Harandi, M. On modulating the gradient for meta-learning, In European Conference Computer Vision. Springer, 2020.
Sun, Q. Liu, Y., Chua, T.-.,, and Schiele, B. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2019.
Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 29, 2016.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The Caltech-UCS Birds-200-2011 Dataset. Technical Report CNS-TR-2011 California Institute of Technology. 2011.
Zhou, F., Wu, B., and Li, Z. Deep meta-learning: Learning to learn in the concept space. arXiv preprint vrXiv:102.005: 2018.
Zintgraf, L.,, Shiarli, K., Kurin, V., Hofmann, K., and Whiteson, S. Fast context adaptation via meta-learning In International Conference on Machine Learning. PMLR, 2019.
7nO, Y., Avraham, G., and Drummond, T. Improved training of generative adversarial networks using decision forests. In Proceedings the IEEE/CVF Winter Conference Applications Computer Vision, 2021.
12

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material and supplementary material; Code to be released upon acceptance
URL)? [Yes] See Section 4.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section |4. and supplementary material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Section and supplementary material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1] directly stated.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.1 and supplementary material.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "lAP9NWvj-P", "reviewer": "Reviewer_XM8W", "summary": "Improving the performance of a deep neural network on a new, unseen task with a limited number of new datapoints and adaptation epochs is one of the central problems in modern deep learning. The authors propose to improve the performance of MAML, a benchmark algorithm for few-shot adaptation problems, by pre-conditioning the parameter space using the condition number of the network. Instead of directly using the condition number to condition the bi-level optimization problem of MAML, the authors propose to consider the distribution of all eigenvalues using the approximated logarithmic eigenvalues for increased expressiveness. This simple modification to MAML is shown to be highly effective at improving the few-shot adaptation process across diverse datasets.", "questions": "- Can the proposed method also improve MAML's performance in reinforcement learning setting? Experimentally demonstrating the applicability of the proposed method across different problem set-ups or tasks would strengthen the paper's contribution.\n- The authors mention that \"computing the condition number by using the max and min eigenvalues... would unnecessarily weaken the training signal\" (section 3.2). Does using the condition number as opposed to Hessian's logarithmic eigenvalues actually lead to a meaningful difference in adaptation performance?\n- Some of the adaptation works have found that only adapting bath norm parameters, instead of the whole network parameters, to reduce the computational cost can be sufficient to improve the adaptation performance. In that regard, it is interesting that adapting cls+emb+eBN parameters performs worse than adapting cls+emb parameters (Table 2), any insight on why additionally adapting bn parameters leads to worse performance?", "limitations": "The authors adequately acknowledge the computational complexity of their method in section 5.\nPlease refer to weakness and questions for additional concerns and questions. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- The proposed method is theoretically sound, interesting, and novel. Although it is a simple modification to an existing algorithm (MAML), there is enough novelty to be recognized in the idea to better-condition a parameter space using the condition number of the Hessian matrix. In addition to the strong theoretical motivation behind the proposed approach, the authors experimentally demonstrate the relationship between the condition number of a network and its adaptation capabilities. \n- The proposed method is shown to be highly effective at allowing more rapid few-shot adaptation. According to the experimental results, the proposed method improves adaptation performance of MAML across all adaptation steps, but the degree of improvement is particularly noteworthy under a limited number of adaptation steps (1 or 2). \n- The writing is concise and clear. It was easy to follow how the modified objective function is derived by introducing the conditioning constraint to the MAML bi-level optimization problem. \n\nWeakness\n- Maybe more baseline approaches from meta-learning and/or few-shot learning, other than MAML, could be added for comparison. Can MAML + the conditioned parameter space (proposed method) outperform more recent meta-learning algorithms? Also, correct me if I'm wrong, but it appears that the parameter space conditioning can be integrated with other meta-learning algorithms as well. Does it lead to performance improvement regardless of the choice of base meta-learning algorithm?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hnnjHQZRJCB", "reviewer": "Reviewer_bs8R", "summary": "The paper proposes a regularisation term for the outer-loop of MAML to encourage a well conditioned parameter space that improves inner loop adaptation. The experiments suggest that the condition number is correlated to the performance within a few gradient updates. Training with the proposed regularisation term is shown to empirically improve upon MAML.  \n\n", "questions": "\nIt is not clear to me why MAML's training collapses in Figure 2. \n\n\nAlthough there has been previous work in preconditioning, this is the first method that I'm aware of that does so without additional parameters. On Page 9 Line 299-300, it is mentioned that existing preconditioning methods \"[incur] an often extensive number of additional parameters\". This seems to suggest that some methods may not require a lot more parameters. If so, how does the number of parameters required compare to the existing preconditioning methods and how do the performance differ? This would allow a potential user to make a decision between which method to try.\n\nI find it interesting that the proposed method can scale to many more gradient steps (Table 3). How does MAML fare when scaling to those same number of gradient steps at test time? Does MAML continue to perform similarly or better than with 5 gradient steps? I am not convinced of the benefit of choosing the number of adaptation steps at inference steps. For example, at test time, it makes sense to evaluate the model on the same number of inner loop updates as it was trained on. Although, it is beneficial if improvements can be further gained with more training (as we see in Table 3). \n\n\nOn Page 9 Line 316, it's mentioned that the proposed method \"significantly [outperforms] unconstrained methods during initial adaptation\". However, the only method compared is MAML, so it would be more precise to say \"significantly [outperforms] its unconstrained counterpart during initial adaptation\"\n\nIf the parameter space is well-conditioned, then I would expect that a single large gradient step from the meta-initialization could already train a good model. For example, instead of evaluating the model by fine-tuning with 5 inner loop updates with an inner loop learning rate of 0.1, I'm curious as to the performance of fine-tuning with 1 inner loop update with an inner loop learning rate of say 0.5.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n - Does not require additional parameters for adaptation unlike existing preconditioning methods. \n - Interesting analysis regarding the connection between condition number and few-step performance on an unseen task. \n - Promising experiment results show an empirical benefit of preconditioning. \n\nWeaknesses:\n - Requires higher-order gradients (computationally expensive) like MAML\n - Computing the conditioning term is expensive. As such, instead of applying the conditioning constraint to the entire model, only a small subset of parameters can have the conditioning constraint applied.\n - Missing comparison with existing preconditioning methods (For example, see Section 2.1 of WarpGrad) \n\nOverall, I would rate the paper:\n\nNovelty: Medium\n\nClarity: High\n\nSignificance: Medium\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZtKem4aEMSX", "reviewer": "Reviewer_FA57", "summary": "This paper presents the idea of enforcing better condition numbers for inner-loops in meta-learning frameworks. They reformulated meta-learning loss as the least-square formulation, which enables them to easily approximate condition numbers. Then, it was used as an additional loss term to minimize it, which results in better performance in few-shot classification tasks. They demonstrated that their approach converged rapidly, especially for the first few iterations.", "questions": "1. All 5 step training was used for all experiments for both MAML and the proposed method. Since MAML might overfit the initial parameters for 5 step performance, so they learned not to update too much in the first few iterations. I am curious how MAML performs, using 1 step training, and 1 step testing accuracy, which is a fairer comparison I think.\n\n2. Missing MAML results on CUB for step 5 - step 100 in Table 3.\n\n3. Missing comparisons to prior arts. Only MAML baseline is not good enough as time of now. For examples, miniImageNet results, the accuracies are way lower than current state-of-the-arts methods. I am not complaining about not achieving SOTA, but it’s better to be upfront with the readers. Especially, you should have shown the performance of methods related to ‘preconditioning’, e,g, [1-3], you can still argue that your method does not require additional parameters.\n\n4. checklist 3-(c), the answer was yes, but I can’t find any error bars w/ different random seeds. I was expecting the ‘shaded area’ in training curves, e.g., standard deviation or confidence interval, at least for the main curves. The standard deviations you reported in the tables are not w/ different random seeds. It’s average over test tasks from my understanding. So, your answers for checklist 3-(c) should be ‘no’.\n\n5. How did you compute eigenvalues? is it differentiable so that you can easily use them as a part of loss function? no explanation about this in the current texts. Also, are there other related works that are using eigenvalues as a part of loss function? \n\n6. Figure 2-(c), you plotted L_k, which is considered as condition numbers. But, was it approximated (by eq 8) and also surrogate condition numbers (by eq (9))? Since you are using L_k as an additional loss function, it is obvious that this number decreases. I am curious if the real condition number is decreasing.\n\n7. In figure 1, (a) and (b), were they a toy example? or from MAML? If MAML, how did you pick only two dimensions in the parameters? not clear given the current texts.\n\n[1] Meta-Curvature, Park et al., NeurIPS 2019\n[2] On Modulating the Gradient for Meta-Learning, Simon et al., ECCV 2020\n[3] Meta-learning with warped gradient descent, Flennerhag et al., ICLR 2020", "limitations": "The authors addressed some limitations in the main text. and I do not see any negative societal impact.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I generally like the idea of achieving rapid convergence without introducing more parameters. The paper is well written and easy to follow. The idea of using least-square reformulation to easily approximate eigenvalues is quite interesting. However, I have some concerns about experimental results below in the question section. I would revise my score depending on the authors' response.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Y6DrS_4etf_", "reviewer": "Reviewer_DdSu", "summary": "- The authors propose a regularization method for improving the conditioning of MAML.\n- Experiments show the method consistently reduces number of steps for adaption, and improves accuracy", "questions": "Is it the real condition number or the variance of logs in Figure 2?", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- method seems very effective at reducing number of required MAML steps\n- consistently outperforms MAML\n- simple additional loss during training, no cost at inference time\n- authors study effect of using parameter subset\n\nWeaknesses:\n- precondition loss only applied in top layers (as mentioned by authors), but this may not be a big problem, as the top layers are the ones that are adapted the most during meta-testing", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept