Reducing Variance in Meta-Learning via Laplace Approximation for Regression Tasks
Authors: Alfredo Reichlin
Gustaf Tegnér
Miguel Vasco
Hang Yin
Mårten Björkman
Danica Kragic
Venue: Accepted by TMLR ()

Abstract: Given a finite set of sample points, meta-learning algorithms aim to learn an optimal adaptation strategy for new, unseen tasks. Often, this data can be ambiguous as it might belong to different tasks concurrently. This is particularly the case in meta-regression tasks. In such cases, the estimated adaptation strategy is subject to high variance due to the limited amount of support data for each task, which often leads to sub-optimal generalization performance. In this work, we address the problem of variance reduction in gradient-based meta-learning and formalize the class of problems prone to this, a condition we refer to as \emph{task overlap}. Specifically, we propose a novel approach that reduces the variance of the gradient estimate by weighing each support point individually by the variance of its posterior over the parameters. To estimate the posterior, we utilize the Laplace approximation, which allows us to express the variance in terms of the curvature of the loss landscape of our meta-learner. Experimental results demonstrate the effectiveness of the proposed method and highlight the importance of variance reduction in meta-learning.

Published in Transactions on Machine Learning Research (10/2024)
Reducing Variance in Meta-Learning via Laplace Approximation for Regression Tasks
Alfredo Reichlin'
alfrei@kth.co
KTH Royal Institute of Technology
Gustaf Tegnér'
gustafte@kth.ss
KTH Royal Institute of Technology
Miguel Vasco
miguelsu@kth.ss se
KTH Royal Institute of Technology
Hang Yin University of Copenhagen
hayi@di kw. dk
Mårten Björkman
celle@kth.co se
KTH Royal Institute of Technology
Danica Kragic
dani@kth. se
KTH Royal Institute of Technology
Reviewed on OpenReview: https: // h/t:://opppee opwnreview.eeeee.e net/ forum? Uc2mgNPkEB
Abstract
Given finite set of sample points, meta-learning algorithms aim to learn optimal adaptation strategy for new, unseen tasks. Often, this data can be ambiguous it might belong to different tasks concurrently. This is particularly the case in meta-regression tasks. In such cases, the estimated adaptation strategy is subject to high variance due to the limited amount of support data for each task, which often leads to sub-optimal generalization performance. this work, we address the problem of variance reduction in gradient based meta-learning and formalize the class of problems prone to this, a condition we refer to as task overlap. Specifically, gradient estimate by weighing each support point individually by the variance of its posterior over the parameters. To estimate the posterior, we utilize the Laplace approximation, which allows us to express the variance in terms of the curvature the loss landscape of our meta-learner. Experimental results demonstrate the effectiveness of the proposed method and highlight. the importance of variance reduction meta-learning.
propose novel approach that reduces the variance of the
1 Introduction
Meta-learning, also known as learning-to-leam is concerned with the development of intelligent agents capable of adapting to changing conditions in the environment. The central idea of meta-learning is to learn prior over a distributtion of similar learning tasks, enabling fast. adaptation to novel tasks given only a limited number data points. This approach has proven successful in many domains, such as few-shot learning (Snell et al. 2017), image completion Gaannelo et al. 2018b), and imitation learning tasks (Finn et al. 2017b).
One instance of this is Cradient-Based Meta-Learning (GBML), which was introduced in Finn 2017a GBML methods employ bi-level optimization procedure, where the learner first adapts its parameters
* EEqual contribution.

---

Published in Transactions on Machine Learning Research (10/2024)
T
8
Figure 1: The Problem of Task Overlap in Regression Tasks:
On the left: Three support points of a single task marked out. These points are shared between different tasks which are marked out by the opaque curves. On the right: Each support point induces distribution in the parameter space. The true function parameters 0* lie at the intersection over the possible function values.
given a few samples denoted the support-set, which then evaluates on another, more complete, query-set. The shared prior defined the parameter initialization. A central issue ariscs in identifying accurate parameters from the support set. The limited amount of support points used for adaptation, measurement error, ambiguity between the tasks inevitably leads to uncertainty in this parameter identification. To enable more efficient adaptation, more effective priors must be learned.
When learning over a continuous space of tasks in the context of meta-regressiin ambiguity often arises. Consider the problem of regressing sinnsoidal waves with varying amplitudes and phases as depicted in fig. on the left. Each pair of (x,y) y) used to adapt to the specific wave can correspond to infinite other waves and, as such, doesn't uniquely identify the task at hand. We refer this condition as task overlap. In this case, the aggregation of the information conveyed by each data point in the adaptation dataset has to consider this uncertainty. GBML models struggle in this regard as this uncertainty is not taken into account.
In this work, propose Laplace Approximation for Variance-reduced Adaptation (LAVA), a method that introduces a novel strategy for aggregating the information of the support-set in the adaptation process. The koy idea our method is to recognize each support point as inducing a unique posterior distribution over the task parameters (fig. on the right). The complete posterior distribution can then be expressed as the joint, posterior over the samples. Thus, our model form of Bayesian model-averaging (Hoeting et al. 1999), where each model is induced by a single point from the support. We approximate the posterior distribution w.r.t each support point through Laplace's approzimation, which constructs Gaussian distribution where the variance of the estimator is function of the Hessian of the negative log-likelihood (MacKay 2003] Chapter 27). In turn, the joint posterior is again Gaussian, from which the optimal value can be expressed as its mean. In contrast to other Bayesian GBML methods, the posterior approximation is not built on the full posterior but rather on the single posteriors induced by every point in the support data. This allows us to optimally aggregate the information these points share and reduce the variance of this estimate in the case of ambiguity.
Our contributions include insight into the adaptation process of GBML methods by identifying class of continuous regression problems where GBML methods particularly subject to high variance in their adaptation procodure. We introduce method for modeling the variance that cach data point carries over the parameters and optimally aggregate these posterior distributions into the lask-adapled parameters. Finally, we demonstrate the performance of our method on regression of dynamical systems and real-world experiments and showcase state-of-the-art performance compared to standard GBML methods.
2 Preliminaries
2.1 Problem Formulation
Let p(T) be a distribution of tasks that share common generative process. Each task 5 T defines a function fr X represented subsets X' x y of inputs X' Rd and outputs C RRK. For each task,
2

---

Published in Transactions on Machine Learning Research (10/2024)
assume we are given finite dataset sampled ii.d. i.c., D, X X y.. These points can be further divided into two sets, the support set D. and the query set DJ such that D, D u DJ. We denote N = |
M = | as the size of the support and query set respectively, which we assume, for simplicity, is the same across all tasks.
From the support set, D.,, we are interested estimating the underlying function defined by each task, f-. Following previous work, (Andrychowicz et al. 2016 Ravi & Larochelle 2016), this can be described as estimating map from the support data to set of parameters that can be used to approximate the true task's function i.e,, A: X' x x - θ with A(DA) 0, such that fe, = fr. We define similarity in terms of mean-squared error and compute it using the M points of the query set with the following loss function:
(()..
1 M
Σ
fo, (a+) -vt 12 s.t. 0, = A(D.).
(1)
The performance is tightly bound to the crror in the estimator A which in part arises from uncortainty in the support data. In this work, we consider the uncertainly that is induced when data points can belong to multiple tasks, which we denote as task overlap.
For case of notation, let fx(x) = x(T,,) denote the true data-gencrating process. Definition (Task Overlap). We define task overlap as the condition for which Yo X the map f(,x): T is non-injective.
From a probabilistic perspective, this property is equivalent in stating that the conditional task-distribution P(T x., v) will have non-zero covariance for all (x, E X x y.
An example of task overlap can be seen by considering a sine wave regression problem. Let the distribution of tasks be sinusoidal waves with changing amplitude and phase i.e., y = A, sin(x + Φ,). In this case, the space of tasks is defined by the union of possible amplitudes and phases and thus has dimension 2. A single sample in the form of a tuple (z, is, however, insufficient to identify this task unambiguously. In fact, there exists an infinite number of viable amplitudes and phases for which the sine can pass through such point, fig. I on the left. In the terms above, there is no injective map between (x, y) and (A,,,,) The set of all possible solutions induced by this single point defines a distribution of solutions in the task-adapted parameter's space. All of these solutions are equally probable if conditioned on this single point. When multiple (and different) samples are considered for the adaptation step, the inference of the sine wave can be exact. There exists a unique sine wave that passes through all of these points at the same time. In terms of task parameters, this can be seen as considering the intersection of the solutions induced by each point; see fig. on the right. When using single-point estimate, the information about this distribution of possible optimal task parameters is lost.
2.2 Gradient-Based Meta-Learning
A notable family of methods to solve the problem described in eq. 1 are GBML algorithms. In these, learning the adaptation process is formulated as bi-level optimization procedure. A set of perampersmssers to is learned such that the inference process A corresponds to a single gradient descent step on the loss computed using the support data and 0y:
0, 0-=00 0, voo((0 D.) -=)0
(2)
where a is a scalar value denoting the learning rate. The resulting estimate of the task-adapted parameters is then optimized to approximate the underlying task function using eq. (1). This, effectively, defines an overall optimization procodure on the peta-parameters 6y. Gradient-based Meta-Lcarning does not require additional parameters for the adaption procedure, as it operates in the same parameter space as the learner. Moreover, it is proven to be a universal function approximator (Finn & Levine 2018), making it one of the most common models for meta-learning.
As noted in previous work Grant et al. 2018), the bi-level formulation of GBML can be interpreted from Bayesian view. We interested, for each task, in finding set of adapted parameters that maximize
3

---

Published in Transactions on Machine Learning Research (10/2024)
the likelihood of the query data i.c., m 8,). This is achieved by estimating prior in the form of On such that, when combined with the evidence from the support data, it leads to the highest probability estimate. GBML simplifies the process by defining the estimate of the adapted parameters as the solution to eq. (2). This, in fact, is equivalent to estimating the most probable set of parameters given the prior Bg and the support data D.:
0 = arg max p(0 0o; D.))
A full derivation can be found in appendix A.2 The quality of the estimate of the adapted parameters is, however, limited by the finiteness of the support data. In fact, while it is unbiased (appendix A.5 it may suffer from high variance.
2.3 Variance Reduction
In this section we discuss the variance problem related to meta-learning and techniques to reduce it.
Assuming the support set is composed of N i.i.d. samples from the task, the variance of the estimated adapted parameters can be expressed as follows:
Var
N2 Var [6.)
(4)
which in the case of task overlap is non-zero. In fig. we depict the variance of this estimate for a simple sinusoidal regression task, together with a comparison to proposed varianco-roduced estimation described in section g As can be seen, for standard GBML, the variance of the estimated parameters cannot get below the variance induced by the finite sampling of the support data. On the other hand, LAVA's 's estimation has decreasing variance as training progresses.
To address the problem high variance, we leverage upon a general method for reducing the variance of a sum of random variables (Hartung et al. 2011). Tet oI,..., ON be a set of random variables with the same mean 0 and different covariance Σ; for each i = I,...,N: N. Let Ô = Σ WiO, denote a weighted average with weights W; ddd, We wish to estimate W,'. that yield 0 with the minimum variance. We can define this variance reduction problem as:
min Var Σ W.O,
subject Σ W =1
Ww
(5)
Proposition (Variance Reduction). The solution to eq.
5 is given by:
W =
ET
(6)
We provide proof in appendix A.6 for completeness.
3 Method
In this section, we describo a novel method to model the ostimation of the task-adapted paramoters more accurately in the task overlap regime. Given a finite set of support points DS, we want to approximate the optimal posterior of eq. (3). Let p(0),;, yi) denote the posterior w.r.t one data point. Given the conditions of task overlap, (definition 1. this induces a distribution in parameter space. Another way of interpreting this posterior is that each data point provides evidence of what the possible true function could be. The full posterior p(0) DS) will thus lie at the intersection of all marginal posteriors. Given the i.i.d. assumption, this implics that:
N
(0 x II
((0 x., v.).
(7)
4

---

Published in Transactions on Machine Learning Research (10/2024)
38
Figure The space of task-parametert adapts to the Hessian. Sum of the logarithm of the loss for support points over different parameters for the sine experiment. Values increase from white to dark blue. The red cross and the red diamond indicate the prior and the posterior, orange points the single task-adapted parameters. Top row: Results for CAVIA. Bottom row: Results for LAVA, included is also the covariance for each support point.
A derivation can be found appendix A.4 We propose to model the probability of the adapted parameters given each support point as Gaussian distribution, Ai, Σ,), by means of Laplace's approximation (Kass et al., 1991p. In particular, we define the maximum posteriori (MAP) estimate to be the GBML original formulation i.e., 0, t, a C. D.). The variance of the adaptation can then be defined as the inverse of the Hessian of the loss function evaluated in the adapted parameters i.c., Σ, H This approximation allows rewrite the estimate of the overall adapted parameters in eq. (7) the product of these Gaussians.
The most probable set of parameters given the provided support data will thus be the mean of the resulting Gaussian distribution:
N
N
0
H
H.O.,
One interpretation of GBML methods such as MAML Fiin Levine 2018 is that the posterior estimate assumes equal covariance across all marginal posteriors, which in turn implies that the MAP estimate equals the average of the parameters. In fact, in the case of no task overlap, eq. 8 becomes equivalent to eq. (2). This follows from the fact that we perform single gradient slep to approximale arg max p(0 Due to linearity of the gradient operator, arg max, p(0) D corresponds to the average of {arg maxg p(0 Wi) However, in the case anisotropic uncertainty in the adapted parameters, as in the task overlap regime, this leads to a sub-optimal estimation. On the other hand, when the Laplace approximation faithfully describes the underlying distrribution, the estimate proposed in eq. (8) corresponds the minimum variance estimator. This can be seen from proposition where D; H The final assumption to make the results coincide is that the expectation over the adapted parameters the same, i.e. Ep(r)[0,] = for all i = I....,N. This 0* can be seen the optimal for the given task but is not necessary for the proof.
Compared to simple parameter averaging defined in eq. 4 we achieve lower variance for any support set. DS p(r). Thus, expectation over all tasks, we achieve variance-reduced estimate.
The overall training procedure follows from GBML methods. We optimize the set of parameters 6g through the objective of eq. (T) as a bi-level optimization procedure where 0, is now defined according to eq. (8) Differently from GBML, the Laplace approximation allows for reshaping the parameter space the learned model. Embodding this new adaptation process the optimization allows for a precise approximation. To minimize the loss, the model has shape the parameter space such that, locally, the Laplace approximation exactly estimate the posterior. This imposes certain constraints the structure of the parameter space
5

---

Published in Transactions on Machine Learning Research (10/2024)
Support Size 2
Support Size
Support Size 10
Support Size 20
-15
-18
-15
-24
-15
-25
-20
-15
-25
-30
-10
Time 10-30-
Time 10
10
Time 10
ANIL
CAVIA
LAVA
Figure 3: Evaluation of Computation Time: We evaluate the performance of the different methods (measured in terms of MSE) as a function of the computational time (in seconds) across different support sizes. The results show that, considering the same execution time (x-axis), LAVA outperforms both CAVTA and ANIL, with lower MSE. This result holds across all evaluated support sizes.
arising from task overlap. This learned parameter space can be precisely shaped to allow for an accurate Laplace Approximation due to the flexible nature of parametric neural networks. Consider once again the sine wave regression example. Each support point gives evidence for a space of possible solutions. When considering a model with parameter space of dimension 2, the space of solutions in the adapted parameters is a oinedimenla curve. The optimal posterior corresponds to an intersection of the curves induced by each support point. In contrast to other GBML methods, LAVA allows for straightening out the region of parameters that minimize the inner loss function. This in turn allows for shaping of the parameter space that enables a precise Laplace Approximation. We illustrate this in fig. 2
3.1 Computational Implementation
A limitation of the described method lies in an increased time complexity. Computing the Hessian on each single support point can severely affect the training time of methods that already require complex second-order calculations like in GBML, especially for over-parameterized neural networks. To minimize the computational burden, we consider performing adaptation in a subspace of the parameter space. In most of the experiments, opt for the implementation of ANIL (Raghu et al., 2020), which performs adaptation over the last layer of neural network as they argue that most of the adaptation takes place in the final layers. This allows to compute the Hessian in closed form, drastically reducing the computational cost and keeping it independent of the dimensionality of the input space.
Nevertheless, the Hessian computation can result in an increase in computational time and LAVA is, in fact, more expensive than the standard GBML model. However, this computational complexity increase is paired with stronger performances. TAAV provides more effective adaptation technique as one of its main features is the efficient use of the limited information given by the support. In this regard, LAVA provides better trade-off between performances and computational complexity. To better analyze this trade-off we compared LAVA's performances against two GBML baselines in the Sine regression experiment by varying the number of inner loop adaptation steps. As shown in fig. 3 LAVA has a computational complexity comparable to CAVIA (Zintgraf et al., 2019) with 2 inner loop gradient steps and ANIL with 3. In appendix A.8 we provide a description of how to compute the Hessian when the loss is in the form of eq. !I
Computing sccond-order derivatives, however, is known to be numerically unstable for neural net- works, Martens 2016). We found that simple regularization considerably stabilizes the training, Following Warton (2008), we take weighted average between the computed Hessian and an identity matrix before
aggregating the posteriors. For all of our experiments, we substitute each Hessian H; in eq. ( with the following:
1
H, = I+e
(H, +el).
(9)
where é scalar value and I is the identity matrix of the same dimensionality of H,, In our experiments, consider 0.1.
6

---

Published in Transactions on Machine Learning Research (10/2024)
4 Related Work
Gradient-Based Meta-Learning methods were first introduced with MAML Finn et al. 2017a) and then expanded into many variants. Among these, Meta-SGD (Li et al.) 2017) includes the learning rate as meta parameter to modulate the adaptation process, Reptile (Nchol & Schulman 2018 gets rid of the inner gradient and approximates the gradient descent step to the first-order. In CAVIA (Zintgraf ct al. 2019 the adaptation is performed over a set of condit ioning parameters of the base learner rather than on the entire parameter space of the network. Other works instead make use of a meta-learned preconditioning matrix in various forms to improve the expressivity of the inner optimization step. (Lee & Choi 2018 Park & Oliva
2019 F Tennerhag et al., 2020
Bayesian Meta-Learning formulates meta-learning learning prior over model parameters. Most of the work in this direction is concerned with the approximation of the intractable integral resulting from the marginalization of the task parameters. This has been attempted using second-order Laplace approximation of the distribution (Grant et al., 2018), variational methods (Nguyen et al. 2020), MCMC methods (Yoon let al.) 2018), or Diffusion models (Pavasovic et al., 2023). While these Bayesian models can provide a better trade-off between the posterior distribution of the task-adapted parameters and the likelihood of the data (Chen & Chen 2022), they require approximating the full posterior and marginalizing over it. LLAMA (Grant et al. 2018) proposes to approximate the integral around the MAP estimate through the Laplace Approximation on integrals. Given one gradient step, the posterior estimate is still performed by averaging. As we have shown, the averaging fails to account for inter- eeterddeenneenciss between the support points arising from task overlap.
Model-based Meta-Learning relies on using another model for learning the adaptation. One such approach is HyperNetwork (Ha et al. 2022], which learns a separate model to directly map the entire support data the task-adapted parameters of the base learner. In Gordon et al. (2019), this is implemented using amortized inference, while in Kirchmeyer et al. (2022), , the task-adapted parameters A Alternativey, the HyperNetwork can be used to define a distribution of candidate functions using the few-shot adaptation data (Garnelo et al. 2018b a) and additionally extend it using an attention module (Kim et al., 2019). Lastly, memory modules can be iteratively used to store information about similar seen tasks (Santoro et al. 2016) or to define a new optimization process for task-adapted parameters (Ravi & Larochelle] 2017]. All of these methods can potentially solve the aggregation of information problem implicitly as the support data are processed concurrently. However, the learned model is not model-agnostic and introduces additional parameters.
context to the base learner.
5 Experiments
To begin with, we test the validity of using the Laplace approximation to compute the task-adapted parameters for simple sine regression problem. Additionally, we show how LAVA exhibits a much lower variance in the posterior estimation in comparison to standard GBML. In appendix A.5 we demonstrate the unbiasedness of GBML and evaluate the noise robustness of GBML and LAVA against a model-basod approach.
We further evaluate our proposed model on dynamical systems tasks of varying complexity in regard the family of functions and dimensionality of the task space well as regression of two real-world datasets. We compare the results of our model against other GBML models. In particular, our baselines include context-basedl GBML. method, LLAMA (Grant) et al. 2018), VFML and VR-MAML (Wang et al. 2021 Yang & Kwok 2022) as variance-reduuced meta- learning methods, and MetaMix (Chen et al. 2021 as a meta-data augmented method. Experimental details are given in the appendix A.1
ANIL [Raghu et al.] 2020), CAVTA, (Zintgral et al. 2019h
5.1 Sine Wave Regression
To develop a further intuition of our model, we conduct experiments on the sinc-wave regression problem introduced in Finn & Levine (2018). We investigate two aspects of our model. The first is in regards the geometry of the learned parameters space (appendix A.7 and how well the Laplace approximation
7

---

Published in Transactions on Machine Learning Research (10/2024)
Log Variance
-10
-15
Cavia [Ds]=2
-25
(Dsje5
Lave [Dsjm5
Cavia [Ds]-10
Lava (Ds)-10 10
-30
200
300
402
Epochs
Figure 4: Estimator's variance. Variance of the task-adapted parameters given the same task but different support data points. Left: For CAVTA Center: For LAVA. Right: Log variance of the distribution of the adapted parameters during training.
can accurately capture distributions in parameter space. The other aspect investigate is measuring the variance reduction in our estimate. In the sine experiment, the dimensionality of the true task parameters is two, allowing us to visualize the learned parameter space. To this end, we train both LAVA and GBML, with a conditional model (akin to CAVIA ZZitggraf et al. 2019 on the sine-wave regression problem with a context vector dimension two.
Given a single (x; y) tuple, there exists one-dimensional space of sine waves that pass through that point. This makes the aggregation challenging and thus allows us to test the benefits of approximating this subspace with the Laplace Approxination.
Laplace Approximation Assumption The first ablation aims at testing the quality of the Laplace approximation in modeling the distribution of the task parameters given each single data point. After the models have converged, we visualize the loss landscape induced by different support data points when using different lask-adapted parameters. In particular, we sample a support dataset of three (z, tuples and compute the logarithm of the mean squared error between the prediction of both models and the true y sampled from grid of tasks. The idea is that cach data point's loss is minimized by continuous space of parameters. When summing up the losses of these support points, in fact, very well-defined valleys can be noticed in the loss landscape. We illustrate the distribution of this sum of losses for grid of task-adapted parameters in fig. 2 as heat map for CAVIA (top row) and LAVA (bottom row). Additionally, the prior context (red cross), the single task adapted parameters (orange dots) and the aggregated final posterior (red diamond) are also shown. For our method, we provide a visualization of the Hessian matrix for each single task-adapted parameter.
Variance Estimation For the second ablation, evaluate the variance of the posterior estimate for both CAVTA and LAVA using the sine regression framework. The variance of the estimator describes the difference between the task-adapted parameters from the sampling of different support sets from the same task. For this experiment, we fix the sine task and sample 100 different support sets each with 10 (æ, y) tuples. For each of these support points, we compute the resulting task-adapted parameters. We depict the spread of these distributions in fig. (left. and center) for CAVTA and LAVA respectively. Note that the scale of the parameter space as well as the inner lcarning rate the same for the two methods. The right of fig. 4 illustrates the log variance of the task-adapted parameters during training for different support sizes. As be seen in the figure, at the beginning of training, the variance increases suddenly the models learn to use these parameters to solve the meta-learning problem. However, while CAVIA's context paramcters variance remains high during the rest of the training, our method learns to reduce it consistently.
5.2 Differential Equations
We evaluate further on set of complex regression tasks in the form of dynamical systems prediction. We consider 5 sets of Ordinary Differential Equations (ODFs):
8

---

Published in Transactions on Machine Learning Research (10/2024)
AtzHugh-Nagumo
Pendulum
Van der Pol
Mass-Spring
Ground
CAVIA
LAVA
Figure 5: ODEs qualitative results. Rollout Trajectories for the dynamical systems' prediction with CAVIA, LAVA and ANIL. We consider the dynamics of systems with random parameters for 5 initial conditions. LAVA (blue linc) is the only model that consistently predicts the evolution the system (red dotted line).
FitzHugh-Nagumo: A model for excitable systems, e.q. modeling the spike of neuron.
Mass-Spring System: A classic mass-spring dynamics model for prediction of the effects of gravity upon mass connected to spring.
Pendulum: Describes the motion of an object of mass m connected to rodd of length L suspended from pivot.
Van Der Pol Oscillator: An oscillating system that undergoes non-linear damping, determined by the parameter μ..
Cartpole: An inverse dynamics problem of an actuated cartpole with varying mass.
For each system, we consider samples from its vector field as our target data. We train all models for 300 epochs and compare their MSE on reconstructing the vector field, we present the results in table For CAVIA and ANIL, we perform grid search over up to inner optimization steps and select the best-performing one. For single gradient step, LAVA outperforms both CAVIA and ANIL even with large number of adaptation steps. This increase in performance can be attributed to LAVA's 's use of second-order information through the curvature of the loss landscape. Further details about the dataset construction and choice of parameters are given in appendix A.1
To further evaluate the top-performing models, we visualize roll-out trajectories devised from integrating the learned vector field in fig. We can note that LAVA provides strong roll-out prediction in almost all cases.
Table MSE | by Dynamical System and Model for Support Size 10, including Cartpole results
Model
FitzHugh -Nauumo
Mass-Spring (x10-2)
Pendulum (x10-3)
Van der Pol
Cartpole
Name
CAVIA
0.19 + 0.05
1.00 + 1.36
0.34 = 0.11
3.57 + 1.18
1.14 + 0.07
ANIL
0.64 + 0.25
0.09 + 0.04
0.22 = 0.07
12.29 6.84
1.81 + 0.57
LLAMA
1.18 0.57
0.51 0.05
3.29 0.37
6.78
1.77 0.32
VFML
1.27 0.25
0.53 + 0.07
4.62 + 0.81
6.36
1.86 0.62
Metamix
3.56 1.21
0.65 0.11
3.68
46.01
4.49 0.15
VR
3.88 - 0.95
3.28 + 0.28
21.55 8.52
32.26
2.34 0.46
LAVA
0.17 0.05
0.02 0.00 0.00
0.07=
1.50
1.02 0.25
5.3 Real World Datasets
In the next experiment, we evaluate our model on two real-world datascts. The first is a regression task on the Beijing Air Quality Dataset (Zhang et. al. 2017) while in the second consider regressing real-world

---

Published in Transactions on Machine Learning Research (10/2024)
and synthetic Prequency-modulated radio signal subject to noise in the RadioML 2018.01A dataset
0 O'Shea
et al. 2018).
Air Quality: We consider the Beijing Air Quality Dataset (Zhag et al. 2017) which is a time-series dataset containing recordings of air quality across 12 monitoring sites. The dataset contains hourly measurements during the time period from 01-03-2014 to 28-2-2017. We consider each monitoring site a separate task. Each task can be viewed as an interpolation lask. A random contiguous subsequence of size nis + my is selected which is then split randomly into support- and query-set. For each measurement, a time-variable t is appended, indicating positional embedding in the time-scries. The task is then predicting the air-quality measurement from the given time t. We present the MSE in the left-most column of table 2 In this task, LAVA clearly outperform the baselines.
RadioML: Lastly, we evaluate the performances of LAVA compared with the baselines on regressing rreguency-modulated radio signals. To do so, we make use of the RadioMT. 2018.01. A dataset described in O'Shea et al. (2018). These are both synthetic and real-world radio signals recorded over-the-air and subject to noise. We devise the task as being able to regress each signal given a few data point rocordings. For testing, we withhold 25% of the signals. We present the MSE in table 2 on the right-most. column. Regression of these signals poses considerable challenge for meta-learning methods. The change in frequency between signals exacerbates the task overlap conditions of the task. The results highlight this, as LAVA is the only method able to correctly regress the signals.
Table 2: MSE | of model preec-predictions on the Air- Air-Quality dataset and RadioML dataset.
Model Name
Air-Quality (x10-3)
RadioML
ANIL
7.63 + 0.31
0.047 + 0.0139
CAVIA
8.25 + 0.16
0.5016 + 0.0
VFML
9.21 = 0.46
0.156 = 0.1351
LLAMA
15.37 7.36
0.5017 0.0
Metamix
56.76 + 13.52 NaN
0.0471 + 0.0018
VR
0.491 + 0.0081
LAVA
4.65 + 0.30
0.0038 + 0.0002
6 Discussion and Conclusion
In this paper, we characterized the problem of task overlap for under-determined inference frameworks. We have shown how this is a cause of high variance in the posterior parameters estimate for GBML models. In this regard, we have proposed LAVA, a novel method to address this issue that generalizes the original formulation of the adaptation step in GBML. In particular, the task-adapted parameters are reformulated as the average of the gradient step of each single support point weighted by the inverse of the Hessian of the negative log-likelihood. This formulation follows from the Laplace approximation of every single posterior given by each support data point, resulting in the posterior being the mean of product of Gaussian distributions. Empirically we have shown how our proposed adaptation process suffers from a much lower variance and overall increased performance for a number of experiments.
For regression tasks, the assumption of task overlap is of particular interest. On the other hand, classification tasks are inherently discrete and do not suffer from the problem of task overlap to the same extent regressionn like problems. Nonetheless, for completion, we provide in appendix AA10 the results of few-shot classification experiment on mini-Imagened In this experiment, LAVA performs similarly to the other GBML baselines, confirming the different nature of the overall problem. However, the discrete nature of classificationn problems presents an avenue for future work in the possibility of incorporating adaptation over a categorical distribution of parameters. A second limitation is the computational burden of computing the Hessian. As described in section 3.1 and further discussed appendix A.8 we overcome this limitation by restricting the adapted parameters to the last layer only (akin to ANIL, Raghu et al. 2020)). This allows to compute
10

---

Published in Transactions on Machine Learning Research (10/2024)
the Hessian in closed form. We show that the time complexity of our model is comparable to 2 inner steps of CAVTA and 3 inner steps of ANIL while showcasing superior performance.
An interesting extension to the proposed method would be to explore techniques to approximate the Hessian. This would allow us to extend the adapted parameters to the whole model. Possible approximations could include the Fisher information matrix or the Kronecker-factored approximate curvature (Martens & Grosse 2015) estimate the covariance of the Laplace approximation. Alternatively, it might be interesting to explore the direction of fully learning this covariance by following approach similar to model-based methods.
References
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
Lisha Chen and Tianyi Chen. Is bayesian model-agnostic meta learning better than model-agnostic meta learning, provably? In International Conference on Artificial Intelligence and Stalislics, pp. 7733-1774, 2022.
Yangbin Chen, Yun Ma, Tom Ko, Jianping Wang, and Qing Li. Metamix: Improved meta-learning with interpolation-baseed consistency regularization. 2020 25th International Conference on Pallern Recognition (ICPR), pp. 407 407-414, 2021.
Chelsea. Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent. approximate any learning algorithm. In International Conference on Learning Representations, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126 1135, 2017a.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. In Conference on robot learning, pp. 368, 2017b.
Sebastian Flennerhag, Andrei A Rusu, Razvan Pascamu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. In International Conference on Learning Representations, 2020.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International conference on machine learning, pp. 1704-1713, 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint 1XX::1807.016330 2018b.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta-learning probabilistic inference for prediction. In International Conference on Learning Representations, 2019.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning as hicrarchical bayes. In International Conference on Learning Representations, 2018.
David Ha, Andrew M Dai, and Quoc V Le. Hypernetworks. In International Conference Learning Representations, 2022.
Joachim Hartung, Guido Knapp, and Bimal K Sinha. Statistical meta- annta-analyis with applications. John Wiley & Sons, 2011.
Jennifer A Hocting, David Madigan, Adrian E Raftery, and Chris Volinsky. Bayesian model averaging: a tutorial (with comments by m. clyde, david draper and ei george, and rejoinder by the authors. Statistical science, 14(4):382-41 12882881772111 1999.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Robert E Kass, Luke Tierney, and Joseph B Kadane. Laplace's method in bayesian analysis. Contemporary Mathemalics, 115:89 990009990100 1991.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Fslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arxiu: 14t.1911198880 2014.
Matthieu Kirchmeyer, Yuan Yin, Jérémie Donå, Nicolas Baskiotis, Alain Rakotomamonjy, and Patrick Gallinari. Generalizing to new physical systems via context-infornet dynamics model. In International Conference on Machine Learning, pp. 11283-11300 2022.
Yoonho Lee and Seungjin Choi. Cradient-based meta-learning with learned layerwise metric and subspace. In International Conference on Machine Learning, pp. 2927-2936, 2018.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arxiv: 7077700.9999 09835, 2017.
David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.
James Martens. Second-order optimization for neural networks. University of Toronto (Canada), 2016.
James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp. 2408-2417, 2015.
Cuong Nguyen, Thanh-Tore Toan Do, and Gustavo Carnciro. Uncertainty in model-agnostic meta-learning using variational inference. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3090-3100, 2020.
Alex Nichol and John Schulman. Reptile: scalable metalcarning algorithm. arxie preprint Xiv: 1et..88880000 02999. 2(3):4, 2018.
Timothy James O'Shea, Tamoghna Roy, and T Charles Clancy. Over-the-air deep learning based radio signal classification. IFEE Journal of Selected Topics in Signal Processing, 12(1):168 179, 2018
Eunbyung Park and Junier B Oliva. Meta-curvature. Advances in Neural Information Processing Systems, 32, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Autiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Krunoslav Lehman Pavasovic, Jonas Rothfuss, and Andreas Krause. Mars: Meta-learning score matching in the function space. In The Eleventh International Conference on Leorning Representations, 2023.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature rouse? towards understanding the effectiveness of maml. In International Conference on Learning Representations, 2020.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International conference learning representations, 2016.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International conference on learning representations, 2017.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Mcta-learning with memory-angmented neural networks. In International conference on machine learning, pp. 1842-1850, 2016.
12

---

Published in Transactions on Machine Learning Research (10/2024)
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advonces in neural informalion processing systems, 30, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for one shot learning. Advances in newral information processing systems, 29, 2016.
Ling Xiao Wang, Kevin Huang, Tengyu Ma, Quanquan Gu, and Jing Huang- Variance-reduced first-order meta-learning for natural language processing tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2615, 2021.
David Warton. Penalized normal likelihood and ridge regularization of correlation and covariance matrices. Journal of the American Statistical Association, 103(481):3444 :44--3443349, 2008.
Hansi Yang and James Kwok. Elficient variance reduction for meta-learning. In International Conference Machine Learning, pp. 15070-25099 2022.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian modcl-agnostic meta-lcarning. Advances in neural information processing systems, 31, 2018.
Shuyi Zhang, Bin Guo, Anlan Dong, Jing He, Ziping Xu, and Song Xi Chen. Cautionary tales on air-quality
improvement in beijing. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, (220552)):201770)
2017.
Luisa Zintgral, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whileson. Fast context adaptation via meta-learning. International Conference on Machine Learning, pp. 1199-7702, 2019.
13

---

dr d di m kr. Published in Transactions on Machine Learning Research (10/2024)
A
Appendix
A.1
Experimental Details
For all of the experiments and all the baselines, we fix the architecture of the meta-learner fo to be a multi-layer perceptron with 3 hidden layers of 64 hidden units together with ReLU activations. We use a meta batch size of 10 tasks and train all the models with Adam {Kingma & Ra 2014) optimizer with a learning rate of 10-3 We use the inner learning rate a 0.1 for the adaptation step and MSE as the adaptation loss. All experiments
run for 5 different seeds to compute mean and standard deviations. For LLAMA use 1 == 10 for PLATIPUS scale the KL loss by 0.1, for BMAML we use 10 particles and use MSE rather than the chaser loss for fair comparison. Other experiment-specific details include:
A.1.1 Differential Equations
FitzHuughNagguoo
du dt de (u bv),
c(u u² + v),
dt
with a, 200.,,[0.122000
The initial positions are sampled as u, E 4 2.5].
Mass-Spring:
di
with mass m and spring constant k sampled from 2100.5, 1.5] with initial conditions I, i ~ 11, Pendulum: The variables modeled is the angle of the pendulum and its angular velocity. The dynamics are described as:
d0
dt ml2
db
dt mggl sin.
We sample mass m, length / and gravity g 4(0.5, 1.5] and initial conditions 0 u[- 3/ ]] and θ -~ u[111]]
Van-Der Pol Oscillator We can describe through a first-order ODE as:
dr dl y, dy
p(1 -x²)g- x.
dt
We let 4(0.1,5.0] and x, - 3].
Cartpole: The dynamics of rigid body can be described as follows:
M(q)ū + C(g, 9) g(g) Bu.
Here, q is a generalized coordinate vector, M is a matrix describing the mass, C' is a force matrix and g(q) the gravity vector. The system takes external force as input contained in u which gets mapped into general forces by the matrix B. The inverse dynamics task involves predicting the control inputs u given target trajectory {q(s)}. We generate trajectories of actuated cartpole with varying mass M and train the inverse-dynamies task.
14

---

Published in Transactions on Machine Learning Research (10/2024)
A.1.2 RadioML
The RadioMT, 2018.01A dataset (OSheaa et al. 2018 consists of measurements of over-the-air radio commu- nications signals. The original datasct consists of signals with different modulation techniques and varying signal-to-noise (SNR) ratios. For the experiment presented in the paper, restrict the dataset to signals with frequency modulation and an SNR greater or equal to 20 dB. Each signal is composed of two channels describing the In-phase and Quadrature components of the signal (I/Q). Here we consider the first 100 time-steps of each of these signals as the curve to be regressed.
A.2 Bayesian View on GBML
probabilistic inference problem from empirical Bayes perspective (Grant) et a.. 2018). The objective of GBML involves inferring set of meta parameters 0g that maximize the likelihood of the data D = U.Dr D, for all tasks. Keeping the notation above and marginalizing over the task adapted parameters, the GBML inference problem can be written as follows:
GBML be formulated
0g arg max p (D arg max II p(D?) (D) 0)d0.,
(10)
where ((D910,) corresponds to the likelihood of each task's data given the adapted parameters and DD,, 0) is the postcrior probability of the task-adapted parameters.
The integral in eq. 10 can be simplified by considering a marimum a posteriori (MAP) estimate of the posterior:
-- = arg max p(0,)D,,
6,
This simplifies the intractable integral to the following optimization problem:
8, arg max
Il p(D 0, =07).
0
A.3 Variance of parameters
Here provide short account of the variance of the GBML parameters estimator reported in eq.
Var
[0]
Var
0o
V
On
N
[(... DS)
Var
0n
a V 0. C(Oo DS)
N
Var
0.
N2
A.4 Intersection of the Posteriors
We show that the posterior over the parameters conditioned on the support set proportional the intersection of the posteriors given each single point in the support. The main idea here is applying the Bayes rule twice and making of the i.i.d. assumption the support data.
15

---

Consider the variance reduction problem defined in Published in Transactions on Machine Learning Research (10/2024)
(0) D P(D 0)p(0)
p(D))
[ p(at, 0) p(0)
p(Ds)
N
p(0 2.
p(0)
m()
p(DS)
N
p(O
Iii Dt)
p(0)N-1
N
x p(0 Iz.
II
A.5 Gradient-Based Meta-Learning is an Unbiased Estimator
Here, we show that GBML with one gradient step is an unbiased estimator. Define the loss for one task as: [(O,T) Pewp(a))) [[),
Then the gradient w.r.t 0 is unbiased estimator:
E Expp(a)r) (oL(0, = VoBeeptatt [2(0 = VoL(O,T).
Moreover, we measure empirically the bias of GBML and LAVA estimators. As comparison, include fully learned network implemented as HyperNetwork (Ha et al. 2022) that takes input the entire support dataset and outpuis the adapted parameters directly. Both the adaptation and the aggregation are learned end-to-end together with the downstream task.
We train these three models until convergence on the sine regression dataset. Then, we measure their performance on each task corrupted by Gaussian noise with standard deviation of 3 on the support labels. test how the performance changes when increasing the support size. We
The experiment is designed show the difference in the loss between adaplation with and without noise for the three models and for different support sizes in fig, 0 Thus, performances of the noiscless adaptation. Idcally, an unbiased estimator converges to the correct postcrior with enough samples as long as the noise has zero mean. As can be seen in the figure, GBML methods are much more robust to these kinds perturbations, while learned networks are not unbiased.
effectively testing the ability of these estimators to recover the
A.6 Variance reduction
Below we give an account the proof of proposition [
eq. 5
min
Var
Σ W.0+
subject to
Σ
W1 =1
(11)
1=1
We have that
n
n
Var
W;O,
Σ W.E.WF.
By introducing Lagrange multiplier X, we reach the following optimization problem:
min
F(O,W),
W
with F(O,W) = Σ W.E.WT +A
Σ W. -
16

---

Published in Transactions on Machine Learning Research (10/2024)
6-
CAVTA
Hypereework
5
LAVA
200.
3
E(0, D3
103
200
300
400
500
108
Figure 6: GBML is an unbiased estimator. Scaled MSE when adding noise to the support labels for the sine experiment with increasing support size.
By taking the derivative w.r.t W2 and using the fact that E, is symmetric we find that
dF dW;
2W,C, X.
Setting this equal to 0, we get
A 2 W₁
(12)
From the condition defined in eq. 11
have
1,
2 1=1
2
Σ
Plugging this into eq. 12 it follows that
W, =
D
x--
Given these weights, W,, the distribution of Ei W,O, follows normal distribution equivalent to the estimate in eq.
A.7 Geometry of the Parameter Space
In order to minimize the loss described in eq. 11 using LAVA's adaptation (eq- (8)), we implicitly make an assumption over the geometry of the task space. In this section, expand this assumption and provide example of the limitations of using Gaussians to represent the distribution of solutions in parameter space. In fact, the model has to shape the parameter space such that, locally, the Laplace approximation can exactly istimate the posterior. This imposes certain constraints on the structure of the parameter space arising from task overlap.
From the assumption of task overlap (definition we have that f(,x) T y non-injective, or that for each vI X V/ the pre-image Mz,y f (v) defines subspace of the task-manifold and Un. Mz.y
17

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm LAVA Pseudo-Code
Require: p(T) distribution of tasks
Require: 0, hyperparameters.
Ensure:
Output results
1:
Randomly initialize dg
while not done do Sample batch of tasks T p(T)
for all E T do Sample D.
6:
for all (2i, Vt) E D do
Evaluate 0, 9o a V VoL Iz 36)
Evaluate H, dez [0,, Zz, yi)
9:
Evaluate H. = Itc
H + cT)
10: end for
11: Evaluate H E, Hl
Evaluate 0, 1111 E; H.,
12:
13: end for
Update Og = 0g Vo ETET [(0,, D&) using each D2
14:
15: end while
defines cover of the task space T. As the Laplace approximation estimates the space with a normal distribution, we require the underlying fiber Mz.9 to be simply connected, or in other words, each element (x, y) induces a path-connected space of model parameterizations in which every path can be continuously deformed into another one.
An example of space of tasks with a non-trivial topology could be an annulus or a disk with a hole inside it. Such space arise in problems of goal-oriented navigation. Consider agent in 2-dimensional plane P where the task provides the control inputs to reach given goal position specified by coordinate (z, E P. Given trajectories provided by expert demonstrations, infer the goal-conditioned policy. In this setting, cach support point corresponds to a position of the agent, and the loss is defined as the L2 distance to the goal. If the goal can be situated anywhere on the plane P, then a single support point (xx, Us) defines a space of possible goal positions as a circle around the current position. By continuity, this implies that the space of possible parameters Mz..,, that yield the possible policies must be non-simply connected as well. This poses problems for the Laplace approximation, the Gaussian distribution implicitly holds assumption the topological properties.
A.8 A Note on Computational Complexity
Here we provide a description of how we compute The Hessian in closed form with respect to the loss defined in eq. [ and how to implement it in practice on a standard automatic differentiation framework.
Assuming the dimensions on the output to be: y C RXXI In the case of a multi-layer perception, define
fo E 1x the network up to the last layer excluded and augment it with an appropriate mue-padiing to vectorize the bias i.c.,
(x)
R(d+1)x1 Moreover, define the adaptation parameters to be the
1
weights and the bias of the last layer i.c., 0 = [W.b] E (d)((1+) We can rewrite the loss of eq. data point of the support as:
1 for one
C(0, ||0 -112
The corresponding Hossian can be written as:
H, 2Txxk (x.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm 2 Hessian Pseudo-Code in PyTorch.
Require: Mis f Wo, bp
1: a = (x;)
2: Hw = Tx @ (z )")
HA = 1exk Ø
Hw
Hb
H 2
Hb
TExk
The notation on the most standard automatic differentiation frameworks like PyTorch (aazkk et al. 2019 differs somehow from the above way computing the Hessian. In Pseudo-code algorithm necessary code for computing the Hessian.
we provide the
A.9
Additional Experiments
A.9.1 Condition Number
We measure the condition number of the Hessian for LAVA under both the conditional setting and when only updating the final layer. We calculate the condition number x(H) for H = Σi= H; and H calculated similarly with the approximate Hessians defined in eq. g We present the results across different support sizes in table 3 below.
Condition Number x(H)
Regularized Condition Number K( s(H)
Model
Epoch=1
Epoch-200
Epoch=1
Epoch=200
LAVA + CAVIA (dim=2)
4.96 x 100
1.94
10
1.04 10
1.96 10
LAVA + CAVIA (dim=4)
1.31 x 10
6.26
10a
1.04 10
41.03 109
LAVA + CAVIA (dim=8)
5.28 x 106
1.78 x 106
1.03 x 10
66.73 10"
LAVA + CAVIA (dim=16)
1.35 x 1010
5.31 x 109
1.04 x 10
36.00 x 10"
LAVA + ANIL
1.89 x 10th
1.36 x 10
74.36 10"
217.86 x 10
Table 3: Condition and Approximate Condition Numbers for Support Size
The condition number increases with the number of adaptation parameters increases. A high condition number signifies that the matrix close to being singular, meaning it is difficult to invert. In the context of matrix inversion, this implies that small errors in the input data can lead to disproportionately large errors in the output, making the inversion process highly sensitive and potentially unreliable. In the table, also present the condition number convergence. This approximate condition number is derived from the approxinale Hessian, as defined in eq. (9). Initially, the regularization promotes an even distribution of singular values, but as training progresses, the matrices become increasingly skewed. This behavior is expected, as the Hcssian for each support point H, should be elongated in specific directions, resulting in a relatively high condition number.
A.9.2 Additional Sine Results
Here we provide additional results for the sine regression experiment. Using the same experimental settings described in section 5.1 and appendix A.I present MSE and standard deviations for 5 seeds for LAVA ablation. The MSE
provide results for support size equal to value is noticcably higher as one support point is not sufficient to identify the underlying task accurately. Nevertheless, results show comparable performances between LAVA and ANIL when |D as eq. (S) becomes equivalent to the adaptation described in eq. 2 Qualitative results comparing the effect different support points and the addition of Gaussian noise on the support are shown in fig. g The figure shows how
and baselines in table 4 In particular,
the prior of the model (Green dotted line) evolves into the posterior for LAVA (orange curve) and ANIL (blue curve), this is compared with the ground truth curve (green dashed line). The effects of task overlap
19

---

Published in Transactions on Machine Learning Research (10/2024)
Models
|D$] =1
|D = 2
[ = 10
[D$| 20
ANIL
171.07 + 10.19
46.05 + 6.42
1.32 + 0.1
0.37 + 0.02 0.02
CAVIA
247.6 4.02
56.75 2.5
2
1.44 0.23
VFML
286.0 + 0.95
106.31 7.14
31.72
18.63 1.8
LLAMA
248.48 4.12
109.75
29.08 4.81
18.31200
Mctamix
= 13.53
120.12
12.04
24.85 2.22
VR
318.21 17.05
338.3 22.91
283.05 21.38
292.28 32.98
LAVA
169.38 9.18
7.01 2.02
0.11 + 0.02
0.09 + 0.02
Table 4: Model performance MSE (10-2) based different support sizes the Sine wave regression problem.
can be seen in the plots in the left column. The support size is small, GBML models like ANIL manage to fit the support points (red crosses) but fail to identify the true underlying distribution.
A.10 Mini-Imagenet
Model
5-ways 1-shot
5-ways 5-shot
ANIL
45.94 0.94
62.86 + 0.26
CAVTA
47.84 0.41
63.09 + 0.51
LLAMA
40.19 + 0.85
56.50 + 0.15
VFML
49.60 0.5
66.20 = 0.80
Metamix
50.51 = 0.86
65.73 = 0.72
VR
49.20 + 1.40
63.60 + 0.80
LAVA
46.69 + 1.45
61.51 + 0.97
Table 5: Results Mini-Imagent with support sizes 1 and 5
We further experiment with classification the Mini Imagenet dataset (Vinyals et al. 2016 We use the training-sct split as used in Ravi & Larochelle (2017) which leaves 64 classes for training, 16 for validation and 20 for test. We experiment with 5-way classification in either 1-shot or 5-shot setting. We train the models for 1000 epochs and perform model selection by choosing the one with the best performance on the validation set. We present results on the test set table 5
Standard classification benchmarks such as Mini-Imagenet test the capability of the model to incorporate high-dimensional data in the form of images. Some of the methods optimized toward image data and attempt to efficiently learn well-structured representation space of images, such that the adaptation reduces to modilying decision boundaries. In particular, few-shot image classification problems in this form are inherently discrete problems that do not suffer as extensively from the task overlap assumption as outlined in definition T In this context, the Gaussian assumption on the parameter distribution not an accurate one, as each element of the support induces mulli--mddal distribution the parameter space. Thus variance reduction becomes inaccurate, and unbiased methods such ANIL prove better. Nevertheless, LAVA manages get competitive performances on this benchmark as well.
A.11 ODEs Computational Complexity
We provide additional computational complexity analysis for the ODEs experiments. As it can be noticed in fig. 7 for most of the experiments LAVA provides comparative advantage in terms of computational complexity and general performances.
20

---

Published in Transactions on Machine Learning Research (10/2024)
Fitzuugh-Nagumo
Mass-Spring
Pendulum
0.4
-25
-10
Log MSE
Log MSE
-15
-0.2
-25
-0.6
-25
Time
10-3
Time in 10-3
Time 10-3
AVIL
CAVIA
LAVA
ANL
CAVA
LAVA
ANIL
CAVIA
LAVA
Van der Pol
Cartpole
1.25
1.00
Log MSE
0.75
0.50
0.25
Log
0.1
0.00
-01
-0.25
-0.50
Time in 10
Time in 10
LAVA
LAVA
Figure 7: Further Evaluation of Computational Time. Computational Complexity results for ODEs regression with support size 5 and different number of adaptation steps.
21

---

Published in Transactions on Machine Learning Research (10/2024)
Suppport Size 2
Suppport Size 10
Suppport Size 20
Noise 0.0
Noise std 0.1
Naise
Noise std 0.3
Noise 0.5
Suppport
Prior
ANIL
LAVA
GT
Figure 8: Qualitative Results for Sine Regression. Estimated sinusoidal curves with varying amounts of support size and noise in the support data compared with GBML.
22


Reviews:
[{"review_id": "2AMXVmqLSJ", "reviewer": "Reviewer_TjnE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Pros__:\n- The problem the paper focuses on is interesting and worth studying;\n- The paper is well organized, and the visualizations are easy to read;\n- Extensive empirical results demonstrate the effectiveness of the proposed LAVA method.\n\n__Cons__:\n- The definitions of the formulations are not well-organized, which makes the paper unreadable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper mainly focuses on the task overlap phenomenon in the context of gradient-based meta-learning. Generally, this is an interesting topic. As far as I know, few works have discussed this problem. In this paper, in order to solve the task overlap problem, Laplace Approximation for Variance-reduced Adaptation (LAVA) is proposed. Extensive empirical results reveal the effectiveness of the proposed LAVA method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Could you please provide more explanations about the Laplace approximation of the probability of the adapted parameters?\n\n2. I feel confused about the definition of $\\mu_i$ and $\\Sigma_i$ in Section 3 of Page 4. It would be better to provide more justifications for the definition here.\n\n3. The problem studied in the paper is very interesting. I also noticed that Pavasovic et al. [1] also studied this problem from the perspective of score matching. Could you please discuss LAVA and their work?\n\n[1] Pavasovic et al. MARS: Meta-learning as score matching in the function space, ICLR 2023.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vUT99HLsLm", "reviewer": "Reviewer_vixc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The method is clearly introduced.\n2. The authors have presented extensive experiments.\n\n\nWeaknesses:\n1. I wonder if the authors can validate the choice of modelling parameter distribution as Gaussian.\n2. The authors may discuss how the proposed method can improve the baseline gradient-based methods when support size is extremely limited, e.g. 1-shot.\n3. I notice that the proposed method performs worse than ANIL on miniImageNet 5-shot experiment. Is there any proper reason for this result?\n4. Fig.3 seems not so intuitive. The authors may update this figure for better understanding.\n5. It seems the proposed method can decrease the standard deviation among different trials in some settings (e.g. Pendulum and Van der Pol in Tab.1) while also having large s.d. in Tab.4. More discussion on this would be better.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper mainly targets meta-learning. The authors point out that the previous gradient based meta-learning cannot well handle the ambiguity among tasks specified as task overlap. To this end, they propose a novel method which adopts Bayesian model-averaging to aggregate posterior information induced by each support sample. The proposed method shows strong performance on several different meta-learning benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses and provide more discussion.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "I5p6aUQ4tU", "reviewer": "Reviewer_cuAw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I want to preface this section by stating that I am not an expert in this area (i.e., meta-learning) so I apologize for any inaccuracies.\n\n# Strengths\n* The notion of task overlap and addressing variance reduction by aggregation in GBML is good.\n* The paper is overall well-written and clearly structured, with a lot of figures.\n\n# Weaknesses\n* I am not convinced about the theoretical claims in paper. The theoretical analysis is too heuristic, and the authors should be more carefull in their analysis. Some quantities are defined with words and thereby the analysis becomes too superficial.\n* Despite efforts to mitigate it, the computation of the Hessian remains resource-intensive, which may limit the method's scalability to larger networks or datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Laplace Approximation for Variance-reduced Adaptation (LAVA) to address high variance in Gradient-Based Meta-learning (GBML), especially due to task overlap. LAVA reduces variance by weighing each support point based on the variance of its posterior, using the Laplace approximation and computing the Hessian of the loss function.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Corrections/comments\n* The authors implicitly assume that there exists some true function parameters $\\theta^*$. This should be putted into an assumption. In addition, I am unsure about the convexity of the objective. Thus, the authors should carefully make an assumption about that there exists some set of true parameters, $\\Theta^*$.\n* The notion of task overlap is presented as a property, i.e., Property 1. For me, this is clearly an assumption. \n* The example after Property 1 is not properly defined. Having a running examples (i.e., the sine wave regression problem) is a good, but the authors should define its component properly.\n* Sec. 2.2. change to equations 1 and 2. (Better: change to (1)-(2), e.g., see comment below about cleveref).\n* Define how $\\theta_{0}$ is learned.\n* The Hessian is very expensive to compute. Why do the authors use some quasi method instead? The authors could explore the correlations and then maybe just use a quasi Hessian method which only update the diagonal. E.g., in the same flavour of using Adagrad instead of some quasi-Newton method to make the gradient steps.\n* In the discussion about variance reduction I think the authors could explore more how bit the \"mini-batch\" should be to have reasonable variance reduction.\n* In section 3.1 the authors discuss the computation costs of the Hessian, and propose to use some quasi-Hessian in (9). Why is the Hessian ill-posed? And, what is the $\\epsilon$ used for this weighted aveaged?\n* In the experiments section the LAVA methods seem to outperform all baselines in terms of epochs. I do not think it is a fair comparison to use second-order information, and thereby I think the experiments should be done in terms of computational costs.\n* Correct the missing dots and comma in all equations.\n* Don't define equations numbers on equations that are not referred to.\n* Correct the references to equations, e.g., use the cleveref package. And then use (1) and not write Equation 1.\n* Correct the format of the citations.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All of the reviewers voted positively for this paper, citing the solid empirical analysis and sharp presentation in formulating their "overlap" problem of interest. I agree with them, and believe the paper can be accepted as-is (up to de-anonymization for the camera-ready version).
Decision: Accept (poster)