Uncertainty-Aware Meta-Learning for Multimodal Task Distributions
Authors: Cesar Almecija
Apoorva Sharma
Navid Azizan
Venue: Submitted to ICLR 2023 (2023)
Keywords: Meta-learning
Bayesian inference
neural network linearization
uncertainty estimation
Gaussian Process
NTK

Abstract: Meta-learning or learning to learn is a popular approach for learning new tasks with limited data (i.e., few-shot learning) by leveraging the commonalities among different tasks. However, meta-learned models can perform poorly when context data is limited, or when data is drawn from an out-of-distribution (OoD) task. Especially in safety-critical settings, this necessitates an uncertainty-aware approach to meta-learning. In addition, the often multimodal nature of task distributions can pose unique challenges to meta-learning methods. In this work, we present UnLiMTD (Uncertainty-aware meta-Learning for Multimodal Task Distributions), a novel method for meta-learning that (1) makes probabilistic predictions on in-distribution tasks efficiently, (2) is capable of detecting OoD context data at test time, and (3) performs on heterogeneous, multimodal task distributions. To achieve this goal, we take a probabilistic perspective and train a parametric, tuneable distribution over tasks on the meta-dataset. We construct this distribution by performing Bayesian inference on a linearized neural network, leveraging Gaussian process theory. We demonstrate that UnLiMTD’s predictions compare to, and outperform in most cases, the standard baselines, especially in the low-data regime. Furthermore, we show that UnLiMTD is effective in detecting data from OoD tasks. Finally, we confirm that both of these findings continue to hold in the multimodal task-distribution setting.

Under review as a conference paper at ICLR 2023
UNCERTAINTY-AWARE META-LEARNING FOR MULTI- MODAL TASK DISTRIBUTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Meta-learning learning to learn is popular approach for learning new tasks with limited data (i.... learning) by leveraging the commonalities among different tasks. However, meta-learned models can perform poorly when context data is limited, or when data is drawn from out-of-distribution (OoD) task. Especially in safety-critical settings, this necessitates an uncertainty-awer proach to meta-leariing In addition, the often multimodal nature of task distri- butions can pose unique challenges to meta-learning methods. this work, we present UNLIMITD (uncertainty-avert butions), novel method for meta-learning that (1) makes probabilistic predictions on in-distribution tasks efficiently, (2) is capable of detecting OoD context data at heterogeneous, multimodal task distributions. To
ap-
meta- meta-learning for multimodal task distri-
test time, and (3) performs achieve this goal, we take a probabilistic perspective and train a parametric, tune- able distribution over tasks on the performing Bayesian inference on a linearized neural network, leveraging Gaus- sian process theory. We demonstrate that UNLIMITD's predictions compare fa- vorably to, and outperform in most cases, the standard baselines, especially in the low-data regime. Furthermore, we show that UNLIMITD is effective in detecting data from OoD tasks. Finally, we confirm that both of these findings continue to hold in the multimodal task dusk-distribution setting.
We construct this distribution by
INTRODUCTION
Learning to learn is essential in human intelligence but is still a wide area of research in machine learning. Mela-learning has emerged as popular approach to enable models to perform well on new tasks using limited data. It involves first a veea-ttaiinng process, when the model learns valuable features from a set of tasks. Then, at test time, using only few datapoints from new, unseen task, the model (1) adapts to this new task (i.e., performs few-shot learning with context data), and then (2) infers by making predictions on new, unseen query inputs from the same task. A popular baseline for meta-learning, which has attracted large amount of attention, is Model-Agnoste (MAML) (Fin et al., 2017 7). in which the adaptation process consists of fine-tuning the parameters of the model via gradient descent.
Meta-Learning
However, meta-learning methods can often struggle in several ways when deployed in challenging real-world scenarios. First, when context data is too limited to fully identify the test-time task, accurate prediction can be challenging. As these predictions can be untrustworthy, this necessitates the development of meta- veta-learning methods that can express uncertainty during adaptation (Yoon) et al. 2018 Harrison et al. 2018). In addition, meta-learning models may not successfully adapt "unusual" tasks, i.c., when test-time context data is drawn from an out-of-distribution (OoD)
task not well represented in the training dataset (Jeong & Kim 2020; Twata & Kumagai 2022) Finally, special care has to be taken when learning tasks that have a large degree of heterogeneity. An important example is the case of tasks with multimodal distribution, i.e., when there are no common features shared across all the tasks, but the tasks can be broken down into subsets (modes)
in way that the ones from the same subset share common features (Vuorio et al. 2019).
Our contributions. We present UNLIMITD (uncertainty- meta-learning for multimodal lask distributions), a novel meta-learning method that leverages probabilistic tools to address the aforementioned issues. Specifically, UNLIMITD models the true distribution of tasks with leam- able distribution constructed over linearized neural network and uses analytic Bayesian inference to perform uncertainty- aware adaption. We present three variants (namely, UNLIMITD-I, UNLIM

---

Under review as a conference paper at ICLR 2023
Data sampled from p(f)
Construct ig ir)
Meta-training
Data exomple
Exomple of functions drown from Pr (
Figure I: The true task distribution p(f) can be multimodal, i.e. containing multiple clusters of tasks (e.g., lines and sines). Our approach UNLIMITD fits p(f) with a parametric, tuneable distribution Pe (f) yielded by Bayesian linear regression linearized neural network.
ITD-R, and UNLIMITD-F) that reflect a trade-off between learning a rich prior distribution over the weights and maintaining the full expressivity of the network; we show that UNLIMITD-F strikes balance between the two, making it the most appealing variant. Finally, we demonstrate that (1) our method allows for efficient probabilistic predictions on in-distribution tasks, that compare favorably to, and in most cases outperform, the existing baselines, from OoD tasks at test time, and that (3) both these findings continue to hold in the multimodal task-distribution setting.
(2) it is effective in detecting context data
The rest of the paper is organized as follows. Section 2 formalizes the problem. Section S presents background information on the linearization of neural networks and Bayesian linear regression. We We discuss related work in detail in Section
detail our approach and its three variants in Section 4 Finally, we present our experimental results concerning the performance UNLIMITD Section and conclude in Section
PROBLEM STATEMENT
A task T consists of function f. from which data is drawn. At test time, the prediction steps are broken down into (1) adaptation, that identifying Sa using context datapoints (X', x Y*) from the task, and (2) inference, that is making predictions for on the query inputs X Later the predictions can be compared with the query ground-truths Y; to estimate the quality of the prediction. for example in terms of mean squared error (MSE). The meta-training consists in learning valuable features from a cluster of tasks, which is a set of similar tasks (e.g., sines with different phases and amplitudes but same frequency), tasks from the same cluster. We take a probabilistic, functional perspective and represent a cluster by p(J). theoretical distribution over the function space that describes the probability of task belonging the cluster. Learning p(f) is appealing, it allows for performing OoD detection in addition to making predictions. Adaptation amounts to computing the conditional distribution given test context data, and one can obtain uncertainty metric by evaluating the negative log- -ikeelidd (NLL) of the context data under p(f).
that test time the predictions
be accurate on
Thus, our goal is to construct parametric, learnable functional distribution Pe(s) that approaches the theoretical distribution p(f), with structure that allows tractable conditioning and likelihood computation, even in deep learning contexts. In practice, however, we are not given p(f), but only meta-training dataset D that assume is sampled from p(f): D [X', Y where N is the
number tasks available during training. and (Xi, T the entire pool of data from which can draw subsets of context data (X', Y"). Consequently, in the meta-training phase, aim to optimize Pe (.f) to capture properties of p(f), using only the samples in D.
Once we have Pe (/), we can evaluate it both in terms of how it performs for few-shot learning (by comparing the predictions with the ground truths in terms of MSE). as well as for OoD detection (by measuring how well the NLL of context data serves to classify in in-distribution tasks against OoD tasks, measured via the AUC-ROC score).
2

---

Under review as a conference paper at ICLR 2023
3 BACKGROUND
3.1 BAYESIAN LINEAR REGRESSION AND GAUSSIAN PROCESSES
Efficient Bayesian leaaning requires a tractable inference process at test time. In general, this only possible analytically in a few cases. One of them is the Bayesian linear regression with Gaussian noise and a Gaussian prior on the weights. Viewing it from a nonparametric, functional approach, this model is equivalent to Gaussian process (GP) (Rasmussen & Williams 2005).
Let X = (Z)1...., IR) E R be a batch of K Ny-dimensional inputs, and let = (y1...., yK) R.NKK be a vectorized batch of Ng Ni-dimensional outputs. In the Bayesian linear regression model, these quantities are related according to y ((X) XX)TT + E IR, where @ RR are the weights of the model, and the inputs are mapped via RNNK RPN,K Notice how this a generalization of the usual one-dimensional linear regression (Ny 1).
If we assume a Gaussian prior on the weights 0 ~ NM,, Σ) and a Gaussian noise e ~ No0 Σg) with Σ. = I, then the model describes a multivariate Gaussian distribution on y for any X. Equivalently, this means that this model describes a GP distribution over functions, with mean and covariance function (or kernel)
Mprior (1x) (b(2)) T Jr,
(1)
cOVprior (Dt,, Ft,) = ((2)))) EE(()))))) E. l ((((tt.: TT2)) ) Σg.
This GP enables tractable computation of the likelihood of any batch of data (X, Y) given this distribution over functions. The structure of this distribution is governed by the feature map & and the prior over the weights, specified by μe and E.
This distribution can also easily be conditioned to perform inference. Given batch of data, the posterior predictive distribution is also GP, with an updated mean and covariance function
Mpost (I++) ( X) (K5 X X) + Ee) Y,
COV post
(It pos
Tta.) ( -) k X) (kx(X, Ee) (x (
Here, Moou (X.) represents our model's adapted predictions for the test data, which we can compare to Y. to evaluate the quality of our predictions, for example, via mean squared error (assuming that test data is clean, following Rasmussen & Williams (2005)). The diagonal of cOV (X., X.) can be interpreted as a per-input level of confdence that captures the ambiguity in making predictions with only a limited amount of context data.
3.2 THE LINEARIZATION OF A NEURAL NETWORK YIELDS AN EXPRESSIVE LINEAR REGRESSION MODEL
As discussed, the choice feature map o plays an important role in specifying a linear regression model. In the deep learning context, recent work has demonstrated that the linear model obtained when linearizing deep neural network with respect to its weights at initialization, wherein the
Jacobian of the network operates as the feature map, can well approximate the training behavior of wide nonlinear deep neural networks (Jacot et al. 2018: Neal 1996. Azizan et al. 2021 Lee al. 2017].
Let f be a neural network f (0, x1) + Yt, where 0 E R are the parameters of the model, x E IR., N= is an input and E Ny an output. The linearized network (w.r.t. the parameters) on Op
S(0, ..) - f(0,, mx) ( (0 Oo), where (o( RR x N= NP is the Jacobian of the network (w..t. the parameters).
In the case where the model accepts a batch of K inputs X (#I...., .K) and returns Y (vi,..., yK), generalize f into g : TR' IR N.N. Consequently, we generalize the linearization:
XK
IR NoKK with Y g(0, X)-
g X) g(O, X) J(O ( 0o).
where J(.-) : : X + I xxK is shorthand for (o(g)(.... Note that we have implicitly vectorized the outputs, and throughout the work, we will freely interchange the matrices N and the vectorized matrices N,K
3

---

Under review as a conference paper at ICLR 2023
This linearization can be viewed the N2 K -i----mimmiionnl linear regression
Φ (X) ERRK
(3)
where the feature map (-) : R x 7 R P is the transposed Jacobian J ((0. .)", The parameters of this linear regression 0 (0 0b) are the correction to the parameters chosen as the linearization point. Equivalently, this can be seen as kernel regression with the kernel ko, (X,, X2) = [On, X1 (O,, X2) which is commonly referred to the Neural Tangent Ker- nel (NTK) of the network. Note that the NTK depends on the linearization point dg. Building on these ideas, [addoxx et al. (2021) show that the NTK obtained via linearizing DNN after has been trained on task yields GP that well-suited for adaptation and fine-tuning to new, similar tasks. Furthermore, they show that networks trained on similar tasks tend to have similar Jacobians, suggesting that neural network linearization can yield an effective model for multi-task contexts such as mett-lea.ning. In this work, we leverage these insights to construct our parametric functional distribution Pe (f) via linearizing neural network model.
4 OUR APPROACH: UNLIMITD
In this section, we describe our meta-learning algorithm UNLIMITDand the construction a para- metric funetional distribution (f( that can model the true underlying distribution over tasks p(f)- First, we focus on the single cluster case, where a Gaussian process structure on Pe(f) can effee- tively model the true distribution of tasks, and detail how we can leverage meta-training data D from single cluster of tasks train the parameters & of our model. Next, we will generalize our approach to the multimodal setting, with more than one cluster of tasks. Here, we construct Pe (f) as mixture of GPs and develop training approach that can automatically identify the clusters present the training dataset without requiring the meta-training dataset to contain any additional structure such as cluster labels.
4.1
TRACTABLY STRUCTURING THE PRIOR PREDICTIVE DISTRIBUTION OVER FUNCTIONS VIA A GAUSSIAN DISTRIBUTION OVER THE WEIGHTTS
In our approach, we choose ia (f) to be the GP distribution over functions that arises from a Gaussian prior on the weights of the linearization of neural network (equation (3). Consider particular task T and a batch of K context data (X', )) The resulting prior predictive distribution, derived from equation I after evaluating on the context inputs, is Y| N(Hyix Σypx:), where
x J(u. X X*),, Σyjx: J JO.. X' X')EJ X + E.).
In this setup, the parameters of pe that wish optimize are the linearization point On. and the parameters of the prior over the weights (11, E). Given this Gaussian prior, it is straightforward to compute the joint NLL of the context labels Y'.
NLL(X', Y") (II''-AII
- [ + log dot Σvix: + Ny log 2n
(5)
The NLL (a) serves as a loss function quantifying the quality of È during training and (b) serves as an uncertainty signal at test time to evaluate whether context data (X) is OoD. Given this model, adaptation is tractable can efficiently make probabilistic predictions by evaluating the mean and covariance of the resulting posterior predictive distribution on the query inputs, using equation
condition this GP the context data analytically. In addition,
4.1.1 PARAMETERIZING THE PRIOR COVARIANCE OVER THE WEIGHTS
When working with deep neural networks, the number of weights P can surpass 105, While remains tractable to deal with 6g and μ., whose memory footprint grows linearly with P, it can quickly become intractable matrix over the weights Σ PxP, Thus, must impose some structural assumptions the prior covariance to scale to deep neural network models.
make computations with (let alone store) a dense prior covariance
Imposing a unit covariance. One simple way to tackle this issue would be to remove Σ from the learnalee parameters E, i.e. fixing it to be the identity Σ = Ip. In this case, = (0,, 11). This
4

---

Under review as a conference paper at ICLR 2023
computational benefit comes at the cost of model expressivity, as lose degree of freedom in how we can optimize our learned prior distribution Pe(.). In particular, we are unable to choose a prior over the weights of our model that captures correlations between elements of the feature map.
Learning low-dimensional representation of the covariance. An alternative is to learn a low- rank representation of Σ, allowing for a learnable weight-space prior covariance that can encode correlations. Specifically, we consider a covariance of the form Σ "" diag(s"))4 where Q is a s-dimensional subspace of R while s is learnable. In this case, the
fixed projection matrix parameters that are learned are Ç = (0o. μ4, s). We define S == diag(s"). The computation of the covariance of the prior predictive (equation 4 could then be broken down into two steps:
{
== ((o. Q
J(Ob, ((06, X')JJ (0o. X ASA
which requires a memory footprint of (P + K)), if we include the storage of the Jacobian.
Because NrK < P in typical deep learning contexts, it suffices that s < P so that it becomes tractable deal with this new representation of the covariance.
trade-off between feature-map expressiveness and learning a rich prior over the weights. Note that even if a low-dimensional representation of Σ enriches the prior distribution over the weights, also restrains the expressiveness of the feature map in the kernel by projecting the P- dimensional features J (On, X) on a subspace of size < P via Q. This presents a trade-off: we use the full feature map, but limit the weight-space prior covariance to be the identity matrix by keeping Σ = I (case UNLIMITD-I). Alternatively, we could learn a low-rank representation of Σ by randomly choosing s orthogonal directions in RP, with the risk that they could limit the expressiveness of the feature map if the directions are not relevant to the problem that is considered (case UNLIMITD-R) As compromise between these two cases, matrix more intelligently and project to the most impactful subspace of the full feature map - in this way, we can reap the benefits of tuneable prior covariance while minimizing the useful features that the projection drops. To select this subspace, we construct this projection map by choosing the top 8 eigenvectors of the Fisher information matrix (FIM) evaluated on the training dataset D (case UNLIMITD-F). Recent work has shown that the FIM for deep neural networks tends to have rapid spectral decay (Sharma et al. 202T), which suggests that keeping only a few of the top eigenvectors of the FIM is enough to encode an expressive task-tailored prior. See Appendix A. for more details.
can choose the projection
4.1.2 GENERALIZING THE STRUCTURE A MIXTURE OF GAUSSIANS
When learning on multiple clusters of tasks, p(f) can become non-unimodal, and thus cannot be accurately described by a single GP. Instead, we can capture this multimodality by structuring foe (f)) as mixture of Gaussian processes.
Building a more general structure.
We assume that at train time, a task T comes from any j=1 with equal probability. Thus, we choose to construct (f) as an weighted
cluster {C, Lima
mixture of a Gaussian processes.
For each element of the mixture, the structure is similar to the single cluster case, where the pa- rameters of the cluster's weight-space prior are given by [4y,E,) We choose to have both the projection matrix Q and the linearization point 0g (and hence, the feature map Φ(-) ((o, can compute
shared across the clusters. This yields improved computational efficiency, the projected features once, simultaneously, for all clusters. This yields the parameters Eo (M..8,)..
(0o. Q, (1),,))....
This can be viewed mixture of linear regression models, with a common feature map but sepa- rate, independent prior distributions over the weights for each cluster. These separate distributions encoded using the low-dimensional representations S, for each Σ,. Notice how this is gener- alization of the single cluster case, for when a 1, (f) becomes Gaussian and &. E
Prediction and likelihood computation. The NLL of a batch of inputs under this mixture model can be computed as
NLLmisu (X', Y log log sum CopppNNNN,LA YY)......... -LLL, (X', Y!)).
In theory, it is possible to drop and extend the identity covariance to the multi-cluster setting, however, this leads to each cluster having an identical covariance function, and thus is not effective at modeling heterogeneous behaviors among clusters.
5

---

Under review as a conference paper at ICLR 2023
Algorithm UNLIMITD-I: meta-training with identity prior covariance
1: Initialize Oo. μ..
2: for all epoch do
3: Sample n. tasks (T' (X', Y'
4: for all T (X', Y") do
NLL; < GAUSSNLL(Y'; Ju, JJ" + E.)
J(0o. x
end for
7: Update 8.. μ with Vagun E, NLL:
8:
end for
Algorithm 2 UNLIMITD-R and UNLIMITD-F: meta-training with learnt covariance
1: if using random projections then
2:
Find random projection Q
3: Initialize o. 14,
4:
else using FIM-based projections then
5: Find intermediate Oo. with UNLIMITD-I
Alg. Alg.
6: Find Q via FIMPROJ(S); initialize
end
for all epoch
9: Sample n tasks IT' (X', Y
10:
for all T, (X", Y')
11: NLL: GAUSSNLL(Y': Ju, "Q" diag(s +2))
DJ J(0u, X')
12: end for
13: Update lo ,s, with Vooumus E, NII,
end for
where ALL,(X²s Y*) NLL with respect to each individual Gaussian, as computed in equa- tion 5 and log sum exp computes the logarithm of the sum of the exponential of the arguments, taking care avoid underflow issues.
To make exact predictions, we would require conditioning this mixture model. As this is not directly tractable, we propose to first infer the cluster from which a task comes from, by identifying the Gaussian Gj,, that yields the highest likelihood for the context data (x' y*). Then, we can adaps by conditioning 9j. with the context data and finally infer by evaluating the resulting posterior distribution the queried inputs X
4.2 META-TRAINING THE PARAMETRIC TASK DISTRIBUTION
The key to our meta-learning approach to estimate the quality of fc(()) via the NLL of context data from training tasks, and use its gradients to update the parameters of the distribution &. Optimizing this loss over tasks in the dataset draws Pa (f) closer to the empirical distribution present in the dataset, and hence towards the true distribution p(f)-
We present three versions of UNLIMITD, depending on the choice of structure of the prior covari- ance over the weights (see Section 4.1.1 for more details). UNLIMITD-I (Algorithm I the meta- training with the fixed identity prior covariance. UNLIMITD-R and UNLIMITD-F (Algorithm) 2 learn oiw-dimensional representation that prior covariance, either with random projections or with MiM-dased projections.
Computing the likelihood. In the algorithms, the function GAUSSNLL(Y'; m, K) stands for NLL of Y under the Gaussian ((m, K) (see equation 5). In the mixture case, we instead use MIXTNLL, which wraps equation 6 and calls GAUSSNI. for the individual NLL computations (see discussion in Section 4.1.2 In this case, de becomes {n, J/ and becomes {s, |/ when applicable.
Finding the FIM-based projections. The FIM-based projection matrix aims identify the ele- ments of Φ = J ((,, X) that are rolevam for the problem (see Section 1.11 and Appendix(A.] However, this feature map evolves during training, because it is Og-dependent. How do we ensure that the directions we choose for Q remain relevant during training? We leverage results from Fort et al. (2020), stating that the NTK (the kernel associated with the Jacobian feature map, see Sec- tion 3.2) changes significantly at the beginning of training and that its evolution slows down

---

Under review as a conference paper at ICLR 2023
training goes This suggests that heuristic, we compute the FIM-based directions af- ter partial training, as they are unlikely to deviate much after the initial training. For this reason, UNLIMITD-F (Algorithm 2) first calls UNLIMITD-I (Algorithm IT before computing the FIM- based Q that yields intermediate parameters On and Ja. Then the usual training takes place with the learning of s in addition to 0n and μ.
5 RELATED WORK
Bayesian inference with linearized DNNs. Bayesian inference with neural networks is often intractable because the posterior predictive has rarely a closed-form expression. Whereas UNLIM- ITD linearizes the network to allow for practical Bayesian inference, existing work has used other approximations to tractably express the posterior. For example, it has been shown that in the infinite- width approximation, the posterior predictive of a Bayesian neural network behaves like GP (Neal, 1996 Lee et al., 2017). This analysis can in some cases yield a good approximation to the Bayesian posterior of a DNN (Gariia-Alonso et al. 2018). It is also common to use Laplace's 's method to ap- proximate the posterior predictive by a Gaussian distribution and allow practical use of the Bayesian framework for neural networks. This approximation relies in particular on the computation of the Hessian of the network: this is in general intractable, and most approaches Newton approximation of the Hessian instead (Ritter et al., 2018). Recently, it has been shown that the Laplace method using the Gauss-Newtor approximation is equivalent to working with a certain linearized version of the network and its resulting posterior GP (Immer et al. 202T).
the so-called Gauss-
Bayesian inference is applied in a wide range of subjects. For example, recent advances in transfer learning have been possible thanks to Bayesian inference with linearized neural networks, Maddox jet al. (20T)) have linearized pre-trained networks and performed domain adaptation by conditioning the prior predictive with data from the new task: the posterior predictive is then used to make predictions. Our approach leverages a similar adaption method and demonstrates how the prior distribution can be learned in a meta-learning setup.
Meta-lcarning. MAML is meta-learning algorithm that uses adaptation few steps of gradient descent (Finn et al., 2017). It has the benefit of being model-agnostic (it can be used on any model for which we can compute gradients w.r.t. the weights), whereas UNLIMITD requires the model to be differentiable regressor. MAMI has been further generalized probabilistic meta-learning models such as PLATIPUS or BaMAML (Yoon et al. 2018 Finn et al. 2018). where the simple gradient descent step is augmented to perform approximate Bayesian inference. These approaches, like ours, learn (during meta-training) and make use of (at test-time) prior distribution on the weights. In contrast, however, UNLIMITD uses exact Bayesian inference at test-time. MAML has also been improved for multimodal meta-learning via MMAMI (Vuorio al.) 2019 Abdollahzadeh ict al. 2021). Similarly to our method, they add a step to identily the cluster from which the task comes from Vuuorio et al. 2019). Ool detection in meta-learning has been studied by Jeong & Kim 2020). who build upon MAML to perform OoD detection in the classification setting, to identity unseen classes during training. Iwata & Kumagai (2022) also implemented OoD detection for classification, by learning a Gaussian mixture model a latent space. UNLIMITD extends these ideas to the regression task, aiming to identify when test data is drawn from an unfamiliar
function.
ALPaCA is Bayesian meta-learning algorithm for neural networks, where only the last layer is Bayesian (Harrison et al., 2018). Such framework yields an exact linear regression that uses as feature map the activations right before the last layer. Our work is a generalization of ALPaCA, in the sense that UNLIMITD restricted to the last layer matches ALPaCA's approach. More on this link between the methods is discussed in Appendix A.2
6 RESULTS AND DISCUSSION
We wish to evaluate four key aspects of UNLIMITD. (1) At test time, how do the probabilistic pre- dictions compare baselines? (2) How well does the detection of context data from OoD tasks per- form? (3) How do these results hold in the multimodal setting? (4) Which approach performs better between (a) the identity covariance (UNLIMITD-I). (b) the low-dimensional covariance with ran- dom directions (UNLIMITD-R) and the compromise (c) using FIM-based directions (UNLIMITD- F) (see trade-off in Section 4.1.II? That is, what is best between learning rich prior distribution over the weights, keeping a full feature map, and compromise between the two?
7

---

Under review as a conference paper at ICLR 2023
STEi end (0-+)
45
254
Trat
Figure 2: Example of predictions for varying number of context inputs K, after meta-training with UNLIMITD-F. Top: UNLIMITD-F, infinite task dataset. Bottom: UNLIMITD-F, finite task dataset. The standard deviation is from the posterior predictive distribution. Note how the uncer- tainty levels coherent with the actual prediction error. Also, note how uncertainty decreases when there is more context data. Notice how UNLIMITD-F recovers the shape of the sine even with low number of context inputs. Finally, note how UNLIMITD-F is able to reconstruct the sine even when trained on fewer tasks (bottom). More comprehensive plots available in Figure
We consider a cluster of sine tasks, one of linear tasks and of quadratic tasks, regression prob- lems inspired from Vuorio et al. (2019). Details on the problems can be found in Appendix
Unimodal meta-learning
The meta-learned prior accurately fits the tasks. First, we investigate the performance of UNLIMITD on a unimodal task distribution consisting of sinusoids of varying amplitude and phase, using the single GP structure for Pe (f). We compare the performance between UNLIMITD-I, UNLIMITD-R and UNLIMITD-F. We also compare the results between training with an infinite amount of available sine tasks (infinite task dataset), and with a finite amount of available tasks (finite task dataset). More training details can be found in Appendix A.5 AA.5 Examples of predictions at the test time are available in Figure 2 along with confidence levels.
In both OoD detection and quality of predictions, UNLIMITD-R and UNLIMITD-F perform better than UNLIMITD-I (Figure 3). and this is reflected in the quality of the learned prior ÿo (J) in each case (see Appendix A)..I). With respect to the trade-off mentioned in Section 4.1. we find that for small networks, rich prior over the weights matters more than the full expressiveness of the feature map, making both UNLIMITD-R and UNLIMITD-F appealing. However, after running further experiments on a deep-learning image-domain problem, this conclusion does not hold for deep networks (see Appendix(A.33 (UNLIMrTTDI and UNLIMrTTD- are appealing that case). Thus, UNL.IMITD-F is the variant that we retain, for it allows similar or better performances than the other variants in all situations.
where keeping expressive feature map is important
Note how UNLIMITD-F outperforms MAML: it achieves much better generalization when decreas- ing the number of context samples K (Figure 3). Indeed, UNLIMITD-F trained with a finite task dataset performs better than MAML with an infinite task dataset: it is able to capture better the common features of the tasks with a smaller task dataset.
Multimodal meta-learning: Comparing the mixture model against a single GP. Next, we con- sider multimodal task distribution with training data consisting of sinusoids as well as lines with varying slopes. Here, we compare the performance between choosing the mixture structure or the single GP structure (see discussion in Section 4.1.2). : in both cases, we use UNLIMITD-F. More training details can be found in Appendix A.6).
Both the OoD detection and the prediction performances are better with the mixture structure than with the single GP structure (Figure 44) indicating that the mixture model is a useful structure for (f). This is reflected in the quality the learned priors (see Appendix A.3.2 for qualitative results including samples from the learned priors). Note how the single GP structure still performs better than both MAMI and MMAMI. for prediction, especially in the low-data regime. This demonstrates
8

---

Under review as a conference paper at ICLR 2023
CTELATTR
(a) Examples context data from in-dist. and OoD tasks
(b) AUC for OoD detection
(c) MSE on predictions
Performance of UNLIMITD for OoD detection and inference, func- tion of the number of context datapoints K. The training dataset consists of sinusoids, while OoD tasks are lines and quadratic tasks. We compare different variants (UNLIMITD-I, UNLIMITD-R and UNLIMITD-F)), and against MAML for predictions. We also compare training with a finite and infinite task dataset. Note how UNL.MITTD- and UNLIMMTTD-F have efficient OoD detection and outperform MAML in predictions. Also, note how MAML trained with an infinite task dataset performs worse than UNLIMrTD-R and UNLIMrTD-F trained on finite task dataset.
Figure 3: Unimodal case:
(a) Examples of context data from in-tist. and OoD tasks
(b) AUC for OoD detection
(c) MSE on predictions
Figure 4: Multimodal case: Performance of UNLIMITD for OoD detection and inference, as function of the number context datapoints K. The training dataset includes both sines and lines, while OoD tasks quadratic functions. We compare the different variants (UNLIMiTD-F with a single GP mixture model), and against MAML/MMAN for predictions. Note how both versions of UNLIMITI-FF yield better predictions than the baselines. In particular, even with single GP, UNLIMITD-F outperforms the baselines.
the strength of our probabilistic approach for multimodal meta-learning: even the probabilistic assumptions are not optimal, the predictions are still accurate and can beat baselines.
7 CONCLUSION
We propose UNLIMITD, a novel meta-learning algorithm that models the underlying task distri- bution using parametric and tuneable distribution, leveraging Bayesian inference with linearized neural networks. We compare three variants, and show that among these, the Fisher based param- eterization, UNLIMITD-F, effectively balances scalability and expressivity, even for deep learning applications. We have demonstrated that (1) our approach makes efficient probabilistic predictions on in-distribution tasks, that compare favorably to, and often outperform, baselines, (2) it allows for effective detection of context data from OoD tasks and (3) that both these findings continue to hold the multimodal tast-disttibution setting.
There are several avenues for future work. One direction entails understanding how the performance of UNLIMITD-F is impacted if the FIM-based directions computed too early in training and the NTK changes significantly afterwards. One could also generalize our approach to non- non-Gaussian likelihoods, making UNLIMITD effective on classification tasks. Finally, further research can push the limits of multimodal meta-learning, e.g. by implementing non-pane non-parametrie Bayesian methods to automatically infer an optimal number of clusters, thereby eliminating a hyperparameter of the current approach.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Milad Abdollahzadeh, Touba Malekzadeh, and Ngai-Man Man Cheung. Revisit multimodal meta- learning through the lens of multi-task learning. Advances in Neural Information Processing Systems, 34:14632-14644, 2021.
Navid Azizan, Sahin Lale, and Babak Hassibi. Stochastic mirror descent on overparameterizee nonlinear models. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for contin- ual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762-3773.
PMLR, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference machine learning, pp. 1 1126-1135. PMLR, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning, Advances in neural information processing systems, 31, 2018.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850-5861, 2020.
Ning Gao, Hanna Ziesche, Ngo Anh Vien, Michael Volpp, and Gerhard Neumann. What matters for meta-learning vision regression tasks? In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14776-14786, 2022.
Adrià Garriga-Alonses Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional net- works as shallow gaussian processes, arXiv preprint #rXiv:8808.05887 2018.
James Harrison, Apoorva Sharma, and Marco Pavone. Meta-learning priors for efficient online bayesian regression. In International Workshop on the Algorithmic Foundations of Robotics, pp- 318-337. Springer, 2018.
Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of bayesian neural nets via local linearization. In International Conference on Artificial Intelligence and Statistics, pp. 703-711, PMLR, 2021.
Tomoharu wata and Atsutoshi Kumagai. Meta-learning for duitt--diittiuution detection via density estimation in latent space. arXiv preprint 2tti::2000009444 2022.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Taewon Jeong and Heeyoung Kim. Ood-maml: Meta-learning for few-shot out-of-distribution de- tection and classification. Advances in Neural Information Processing Systems, 33:3907-3916, 2020.
Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint 17110:771..00166 2017.
Wesley Maddox. Shuai Tang, Pablo Moreno, Andrew Gordon Wilson, and Andreas Damianou, Fast adaptation with linearized neural networks. In International Conference Artificial Intelligence and Statistics, pp. 2737-2745. PMLR, 2021.
Radford M Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks, pp. 29-53. Springer, 1996.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005. ISBN 026218253X.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In 6th International Conference on Learning Representations, ICLR 2018-Conferee Conference Track Proceedings, volume 6. International Conference Representation Learning. 2018
10

---

Under review as a conference paper at ICLR 2023
Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv: /706v 170,00440044444, 2017.
Apoorva Sharma, Navid Azizan, and Marco Pavone. Sketching curvature for efficient out-of- distribution detection for deep neural networks. In Uncertainty in Artificial Intelligence, pp.
1958-1967. PMLR, 2021.
Joel A Tropp, Alp Yurtsever, Madeleine Udell, and Volkan Cevher. Practical sketching algorithms for low-rank matrix approximation. SIAM Journal on Matrix Analysis and Applications, 38(4): 4454-1482 2017.
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim. Multimodal model-agnostic meta- learning via task-aware modulation. Advances in Neural Information Processing Systems, 32, 2019.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning, Advances in neural information processing systems, 31, 2018.
A APPENDIX
A.1 A TRACTABLE WAY OF FINDING THE PERTURBATION DIRECTIONS IN WEIGIIT SPACE THAT IMPACT THE MOST THE PREDICTIONS OF AN ENTIRE DATASET
Deep neural networks have large number of parameters, making the feature map Oo, high- dimensional. However, recent work has shown that only a small subspace of the weight space impactful. For example, to perform continual learning. Farajtabar et al. (2020) leverage the fact that it is sufficient update the parameters orthogonally to few directions only to avoid catastrophic forgetting. Sagun et (2017) have shown that the Hessian of a deep neural network can be summaa- rized in a few number of directions, due to rapid spectral decay. This encourages finding a method extract these meaningful directions of the weight space.
A.1.1 LINK WITH THE FISHER INFORMATION MATRIX
We define these main directions as the ones that have the most impact on the predictions of a whole dataset. To find them, we first find a way to quantify the influence of an infinitesimal weight pertur- bation. Using the second-order approximation that quantity, we then describe in the deep-learning context tractable way to find the directions.
Setting. We take the same setting as Section 3.2 and we describe a method to quantify the influ- ence of parameter perturbation i on the predictions of a dataset of tasks D. To do so, we leverage assume a Gaussian pdf over the observations for given inputs and parameters Po (Y X) (g X), Σz), where the covariance of the noise Σ,
probabilistic interpretation of the model:
diagonal Σg a. (just in Section 3.1
Perturbation of the prediction of a batch of inputs. Before quantifying the influence of the perturbation on the predictions of the whole task dataset, we do it for the prediction of a batch of inputs (0,, X).
We borrow the method from Sharma et al. (2021): we quantify the influence of a parameter pertur- bation by computing the Lullbeck-Liillr divergence between pe, (Yl)) and Po.+ (Y\x)) The expansion is:
8(0.. X) DKL(Pe, xT
Ø F(O. X)² + 0
(7)
where F(O,, X) = J(Op, E. ( X) = - 2 X) J(n, X) E RxP is the empir- ical Fisher Information Matrix (FIM) of the batch of inputs X, computed on the parameters O,.
define the influence of a parameter perturbation on the whole training dataset D, by generalizing the previous definition:
Generalization:
perturbation of the prediction of dataset. Now
N
5(0,, (0) = N Σ 8(0g. x (0)

---

Under review as a conference paper at ICLR 2023
Using equation / this quantity verifies:
(00. ~ OT
Σ F(O.. x¹)
0 o(11013)
which gives natural definition for the FIM of the whole dataset by analogy with equation
F D) = N 1=1
F X
No J ( J(00: X') E PxP
The expansion in equation S shows that the FIM of the dataset is second-order approximation de- scribing the influence of a parameter perturbation over the entire dataset. In particular, the eigenvec- tors of F(, D) with the highest eigenvalues the directions that impact the most the predictions.
A.1.2 COMPUTING THE TOP FIGENVECTORS OF THE FISHER INFORMATION MATRIX OF THE DATASET IN DEEP LEARNING CONTEXT
Naively computing the top eigenspace of F(O,. D) requires processing P matrix, which is intractable in the deep-learning context (where P can surpass 10%). Instead, decide to the method used by Sharma et al. (202I). which leverages (namely matrix sketching) developed by Trrpp et al. (207)).
www-raan--appoxiimatte based technique
Sketching the FIM of the dataset. The key idea behind this technique is to build two small random sketches of the FIM, (Y, W) := S (0. D)), that with high probability contain enough information to reconstruct the top eigenvectors of F(O.. D). The linearity of S simplifies the sketching process by breaking down the computation into individual sketches:
N Σ (Y' W Σ s (J J.) xx) =: No?
N
Y, W) = ((F D)) = oo?
i=1
In particular, the sketch (Y, W) can be updated in-place and does not require to store all the individ- ual sketches. Given sketch budget k + (Sharma al. (2021) recommends choosing == 2s and / : 4s and two random normal matrices Q E RXXP and V RiXP, the random individual sketches (W Y*) are defined as:
{
Y = (AL JJ)
ER
W = (VJ.) J ERIP
where J, = J(o,, X
Sketch-based computation of the top eigenspace. Once the sketches are computed, the function FIXEDRANKSYMÁPERR by Tropp al. (2017) computes the first cigenvectors and eigenvalues of the FIM the dataset. Overall, the memory footprint to find the sketches and the top eigenspace is O(P(s + M)). where is the number of queried eigenvectors and M is the size of X'. As long P. this computation is tractable, given that N,M & iss usual deep learning contexts. Algorithm summarizes the process that yields the FlM-based projections via sketching. the scaling coefficient o--² as it doesn't 'tt affect the computation, given that we only want orthogonal eigenvectors and eigenvalues.
drop
A.2 UNLIMITD AS GENERALIZATION OF ALPACA
Restraining the linearization to the last layer. Remember the linear regression of equation 3 that we obtained by linearizing the network with all its layers. Let's separate the parameters of the network 0 into two: the parameters of all the layers but the last one X, and the parameters of the last layer p: X AUp. p.
We assume that the last layer is dense with biases: we will note nd the weight matrix and pa the biases of this last layer. N₂ will stand the dimension of the activations right before the last
12

---

Under review as a conference paper at ICLR 2023
Algorithm 3 Computing the FIM-based projections
Require: s (desired size of the subspace)
1: k < 28+1
2: [ + 4s 3
3: Draw 52 1R V RIXP, two random normal matrices
4: Initialize Y 0 T Pxk W = 0 E R
5: for all training task T do
6: J: - J( X)
7: Y 1/N ((.
8: W W I/N(JJ, J
9: end for
layer: in particular, pd R-NxNN and p RRy, p will stand the size of p: in our case, p' = No x Ng Ng. We implicitly vectorize p. such that:
voe pm)
NRNON,NN,
p
We now restrain the linear regression to the last layer as follows:
(po xx(x)) Po) +
(9)
where:
xx(+) NN - R stands for the function that maps the inputs and the activations right before the last layer;
J :R x No + Xx stands for the jacobian of the last layer with respect to the parameters p. We can write the jacobian in closed-form due to the linearity of the last layer:
J' Jo (X)) = (X)(X) & IN, IN,) E KN
Note that (po, (x)) does not depend on the linearization point Po (could be expected, given the linearity of the last layer).
P Pn E PRP' is the correction to the last parameters. We will note the correction to the weights as 8D == p Pe and the correction to the bias as p = pd p
Using Kronecker's product identities, the linear regression equation 9 can be rewritten:
y = rb (X)
p +
(10)
Also, side note, another way of getting this lincar regression (equation 10) is to rewrite the initial linearization of Section 3.2 but with respect to p only:
S(AU S(A po. v.) = p" (xx) + pd vo v)())) pb)
= (p"h P (x)) + (p" -8))
(>))) + +
Doing this way has the benefit to show that the linearization exact when restricted to the last layer. For non-linear neural networks, the lincarization is always approximation.
Adapting UNLIMITD to the last layer. Just like what did in the general case, we the GP theory to make Bayesian inference on this linear regression. The new parameters of the GP Pe (f) are now { = (A, nz E), where and Σ are the parameters of the Gaussian prior over p. Note how Po have disappeared from E: contrary the general case, the linearization point optimized, as it does not impact the computation. Also note that A has replaced Po. as it parameterizes the feature map.
13

---

Under review as a conference paper at ICLR 2023
EOINTLL POINTA
SEEE THIS NARTIN
(a) Mean E.f) (UNLIMITD-I, UNLIMITD-R, UNLIMITD-F)
(b) Covariance (UNLIMITD-I)
f
(c) Covariance cov(f(0), f())) (UNLIMITD-R, UNLIMITD-F)
5b 5ch functions of P (f) after different meta- veta-trainings on the sine cluster (UNLIMITD-I, UNLIMI D-R and UNLIMITD-F, with finite infinite dataset). Note the scale, and how UNLIMITD-I has a less accurate covariance function that UNLIMITD-R and UNLIMITD-F.
Figure 5: Mean (5a) and covariance
Comparison with ALPaCA. In the ALPaCA setting, the linear regression is:
p' x +e
(1I)
Note that 6" still plays the same role in the linear regression, but is not any correction anymore.
Substracting 11 10 yields:
y yo
UNLIMITD restricted to the last layer is closely related to ALPaCA, in that they both perform linear regression with the same kernel: the only difference lies the additional bias term that not considered in ALPaCA. Thus, we can think of UNLIMITD as a generalization of ALPaCA to all the layers of the network.
A.3 ADDITIONAL RESULTS
A.3.1 ADDITIONAL RESULTS (SINGLE-CLUSTER CLUSTER CASE)
Quality of the priors. To qualitatively analyze the prior after meta-training on the sines, we plot the mean functions and the covariance functions of the resulting GP, that is pf( (Figure 5 All the trainings yield a similar mean for Pe (f)) (that is cosine with amplitude 1.5 and offset 1) 5a) which close to the theoretical mean of p(J) (a cosine with amplitude 2.5 and offset 1). The covariance function of UNLIMITD-R and UNLIMITD-F (Figure 5c) resembles what we would expect for p(/) (e.g. periodicity, negative correlation between O and etc.), but it is not the case for UNLIMITD-I (Figure (56). This empirical analysis confirms the quantitative comparison in terms of OoD detection and prediction performance carried in Section 6
Examples of predictions. Fumariissummarizss the predictions of the model meta-trained on sines, for a varying number of context inputs K, breaking down all the different cases of training.
A.3.2 ADDITIONAL RESULTS (MULTI-CLUSTER CASE)
Quality of the priors. Figure 7 and Figure show the mean and covariance functions of the GP (when training with a single GP), and the mean and covariance functions the GPs composing the mixture(When training with mixture of GPs). We note that in the case of the mixture, both of the Gaussians composing the mixture have correctly captured the common features shared by each of the clusters (respectively the linear and the sine cluster): e.g. the mean of the line cluster is the zero- function, which matches Figure 7b the correlation between rendered for linear tasks (Figure 8b). etc.
and the other inputs is correctly
When learning with a single GP, the learnt mean and covariance do not match any of the two clusters. For example, the mean has an intermediate offset between the offset of the sine cluster and the line cluster. This empirical analysis comforts the conclusions of Section 6 the mixture model yields better results than the single GP case.
14

---

Under review as a conference paper at ICLR 2023
A.3.3 UNLIMITD YIELDS EFFECTIVE PREDICTIONS LARGE- SCALE VISION PROBLEMS
We consider regression-visite meta-learning problem from Gao et al. (2022) Shapenet aiming to predict object orientations in space. this problem, each task consists of a different object of which we want to predict the orientation. For each task, the context data consists of some images of the same object, but with different orientations; the query inputs are other images of the same object. with unknown orientations. Details on the problems and the datasets can be found in appendix [A.4
We train deep learning model ShapenetlD, and we compare the performances on the test set be- tween UNLIMITD-I, UNLIMITD-R and UNLIMITD-F (Figure More training and test details can be found in the Appendix A.7
Both UNLIMITT-II and UNLIMITD-F yield better performances than MAMI., achieving low angle errors. UNLIMITD- R however gives poor results, worse than that of MAML.
In terms of the trade-off of Section 4.1.1, our conclusion from small networks does not scale to deep models. Here, the loss of valuable features perceptible (that what happens with UNLIM- ITD-R, when the randomness of the directions may drop such features), and not learning a rich prior over the weights is not burdensome (UNLIMITD-I) However, UNLIMITD-F plays role of com- promise, by learning the prior covariance while keeping the few important features of the jacobian, as it gives comparable results to UNLIMITD-I
A.4 DETAILS ON THE REGRESSION PROBLEMS
A.4.1 SIMPLE REGRESSION PROBLEMS
Our simple regression problems are inspired by Vuorio al. (2019)'s work. They consists of three clusters of different types of tasks, and varying offset. The first cluster consists of sines with constant frequency and offset, but with varying amplitude and phase:
for A sin + ()) 1/A [0.1, [0,
The second cluster consists of lines with a varying slope and no offset:
( arla [-1,
The last cluster consists of quadratic functions, with varying quadratic coefficient and phase, and with a constant offset:
+ a(x 3) + 0.5 E [-0.2 E [-
In all these clusters, we add an artificial Gaussian noise on the context observations (0, 0.05). The query datapoints remain noiseless, to remain coherent with the assumption from Section 3.1 borrowed from Rasmussen & Williams (2055).
A.4.2 VISION PROBLEM
We consider the veta-learning vision, regression problem recently created by Gao et al. (2022) (namely Shapenet1D), that consists in estimating objects orientation from images. The objects have vertical angular degree of freedom, and Figure 10 shows an example of such objects in different positions.
We use the same training and test datasets Gao et al. (202).. with no kind of augmentation (in particular, no artificial noise on the ground-truth angles). In particular, we use the same intra- category (IC) evaluation dataset (that is. objects from the same categories as the objects used for training) and cross-category (CC) evaluation dataset (that is, objects from different categories the ones used for training).
A.5 TRAINING AND TEST DETAILS FOR THE SINGLE CLUSTER CASE
A.5.1 TRAINING DETAILS OF UNLIMITD (SINGLE CLUSTER CASE)
The model is neural network with hidden layers, 40 neurons each, with ReLU non-linean non-linearity. In our single-cluster experiment, the cluster is the sine cluster (see Appendix A.4).
In the case where there are an infinite number of available sine tasks during training (ie N = +oc), the training is performed with n. 24 tasks per epoch, and at each epoch the context inputs
15

---

Under review as a conference paper at ICLR 2023
randomly drawn from [-5, 5] (which means that M +o0). In the where restrict the available tasks to finite number, we randomly choose = 10 tasks and M = 50 context datapoints
task before training (they are shared among all the "finite" trainings) and perform the trainings with n 6 tasks per epoch.
For all the trainings, the number of context inputs seen during training is K = 10.
For all trainings. we train UNLIMITD on 60, 000 epochs. When dealing with the UNLIMITI-F case, we allow half of the epochs (30, 000) to the training before finding the intermediate Og and fa (see Algorithm 2). and the other half after.
In the UNLIMITD-FF case with infinite available tasks, finite number of tasks is needed to compute the FIM: thus we build an artificial finite dataset of N 100 and M P (arbitrary, but chosen so that J(o, X JJJJ( X gets a chance to be full-rank ie contain as much information as possible), that is used only for that computation.
In the UNLIMITD-F and the UNLIMITD-R cases, the subspace size is 10.
For all trainings, the meta-optimizer is Adam with an initial learning rate of 0.001. The noise a. 0.05, equal to the noise added to the context data. We compute the NLL using Rasmussen & Williams 2005)'s implementation.
A.5.2 TRAINING DETAILS OF MAMI (SINGLE CLUSTER CASE)
We train MAML baselines to compare our results with that of MAML. A large part of these hyper- parameters directly inspired from Finn et al. 2017)'s work.
The model is neural network with 2 hidden layers, 40 neurons each, with ReLU non-linearity. In our single-cluster experiment, the cluster is the sine cluster (see Appendix (A.4).
In the case where there are an infinite number of available sine tasks during training (ie N = +o0), the training is performed with n 24 tasks per epoch, and at each epoch the context and query inputs are randomly drawn from [-5, 5]. In the case where we restrict the available tasks to a finite number, we randomly choose N = 10 tasks and M = 50 datapoints per task (used for both context and query batches) before training (they are shared among all the "finite" trainings) and perform the trainings with n = 6 tasks per epoch.
For all trainings, the number of context datapoints is K = 10, and the number of query datapoints 10. We meta-train for 70, 000 epochs.
For all trainings, the meta-optimizer is Adam with an initial learning rate of 0.001. The inner deaming-rate is kept constant, at 0.001. The number of inner updates is 5 during training, and 10 at test time.
A.5.3 TEST DETAILS (SINGLE-CLUSTER CASE)
For the OoD detection evaluation, we plot the AUC as a function of the number of the context inputs K. The AUC is computed using the NI.L of the context inputs wrt to (f) (our uncertainty metric). The mus-positives are the OoD tasks (lines and quadratic tasks) flagged as such; the false-positives are the in-distribution tasks (sines) flagged as OoD.
For the predictions, we plot the average and ci95 of the MSE on 1,000 tasks (100 queried inputs each). In the UNLIMrTD-R case, also compute the average and ci95 on different random projections trainings.
A.6 TRAINING AND TEST DETAILS FOR THE MULTI CLUSTER CASE
A.6.1
TRAINING DETAILS OF UNLIMITD (MUL.TI-CUSSTR) CASE)
The model neural network with 2 hidden layers, 40 neurons each, with ReLU non-linearity. In our multi-cluster experiment, a = 2: the clusters consist of the sine cluster and the linear cluster (see Appendix A.4).
For all the trainings, the training is performed with n 24 tasks per epoch (with an infinite number of available sine tasks and linear tasks during training ie N = +oo), and at each epoch the context inputs randomly drawn from [-5,5] (which means that M +00). In accordance with
16

---

Under review as a conference paper at ICLR 2023
equal probability assumption from Section 4.1.2 each epoch n/2 12 tasks come from the sine cluster and n/2 = 12 tasks come from the linear cluster.
For all trainings, the number of context inputs seen during training is K = 10.
For all trainings, train UNLIMITD algorithm on 60, 000 epochs: because deal with the UN- LIMITD-F case, we allow half of the epochs (30, 000) to the training before finding the intermediate 0, and {Hy 1 (see Algorithm 2 and the other half after.
In the UNLIMITD-F case, finite number of tasks is needed to compute the FIM: thus we build an artificial finite dataset of N = 100 and M = (arbitrary, but chosen so that J J(O.. X J ) gets chance to be full-rank ie contain as much information as possible), that is used only for that computation.
For all trainings, the subspace size is 8 10.
For all trainings, the meta-optimizer is Adam with initial learning rate of 0.001. The noise 0- = 0.05, equal to the noise added to the context data. We compute the NLL using Rasmussen & Williams 2005)'s 's implementation.
When training with MIXT, make the meta-learning can effectively differentiate the two clusters. Also, in the MIXT case, the mean is unique before computing the FIM. Once have found the projection directions, initialize (11 p2) with the intermediate μ. thus yielding two Gaussians.
initialize 81 and 82 randomly with ((0, 0.5I), that
A.6.2 TRAINING DETAILS OF MAML (MULTI-CLUSTER CASE)
We reimplement and train a MAML baseline to compare our results with that of MAML. A large part of these hyperparameters is directly inspired from Finn et al. 22017's work.
The model neural network with 2 hidden layers, 40 neurons each, with ReLU non-linearity. In our multi-cluster experiment, = 2: the clusters consist of the sine cluster and the linear cluster (see Appendix A.44)
The training is performed with n 24 tasks per epoch (with an infinite number of available sine and linear tasks during training ie N = +oo) and at each epoch the context and query inputs are ran- domly drawn from 5, 5). In accordance with our equal probability assumption from Section 4.2 at each epoch n/2 = 12 tasks come from the sine cluster and n1/ = 12 tasks come from the Innear cluster.
For all trainings, the number of context datapoints is K 10, and the number of query datapoints is L = 10. We meta-train for 70, 000 epochs.
For all trainings, the meta-optimizer is Adam with initial learning rate of 0.001. The inner learning-rate is kept constant, at 0.001. The number of inner updates is 5 during training, and 10 at test time.
A.6.3 TRAINING DETAILS OF MMAML (MULTI-CLUSTER CASE)
We train a MMAML by using Vuorio al. 2019)' is code, using the best setting mentionned in their paper (FiLM). We adapt their code to train it on our sine and linear clusters: we add an offset to their sine cluster (via their parameter b pi as), change the phase from sin(x - () to sin(x + pp) and specify via the arguments the slope range ((11, 1) and the y-intercept range (0, O]. because it does not vary in our case) of the line tasks. We also change the number context inputs that we set to K= 10, to remain coherent with the rest of the trainings. The rest of the hyperparameters are kept identical to the command specified the repository of MMAML. Finally, at test time we make the query ground ground-truths noiseless, to remain coherent with the rest of the test conditions.
=-10,
A.6.4 TEST DETAILS (MUI.TI-CLUSTER CASF)
For the OoD detection evaluation, we plot the AUC as function of the number of the context inputs K. The AUC is computed using the NLL of the context inputs wrt to pe (f) (our uncertainty metric). The true-positives are the OoD tasks (quadratic tasks) flagged as such; the false-positives are the in-distribution tasks (lines and sines) flagged as OoD.
17

---

Under review as a conference paper at ICLR 2023
For the predictions, plot the average and ci95 of the MSE 1,000 tasks (100 queried inputs each): half of them are sines and half them are lines.
A.7 TRAINING AND TEST DETAILS FOR THE VISION PROBLEM
The model is deep neural network, identical to the one used by Gao et al. 2022 except for the last layer: instead of doing a one-dimensional regression (where the output stands for angle prediction), we perform two-dimensional regression (where the output stands for cosine and sine prediction). This choice is motivated by the fact that the Gaussian noise assumed in Section 3.1 cannot capture the complexity an angle error (e.g. predicting 361 should yield low error when compared to the ground-truth angle 09), but better renders the MSE that can be applied on cosine and sine.
A.7.1 TRAINING DETAILS OF UNLIMITD (VISION CASE)
For all the trainings, there are n 10 tasks per epoch and K = 15 context inputs per task. When dealing with the UNLIMITD-F case, we allow half of the epochs (5, 000) the training before finding the intermediate 0, and şl (see Algorithm 2. and the other half after.
In the UNLIMITD-F and UNLIMITD-R cases, the subspace size is 100.
For all the trainings, the meta-optimizer is Adam with an initial learning rate 0.001. The noise is 0e = 0.01. We compute the NL. using Rasmussen Williams 2005)'s implementation.
A.7.2 TRAINING DETAILS OF MAML (VISION CASE)
We train a MAML baseline compare our results with that MAML. large part of these hyper- parameters directly inspired from Gao et al. (002)..
The training is performed with 10 tasks per epoch. The number of context datapoints is K 15, and the number of query datapoints is L
We meta-train for 50, 000 epochs.
For all trainings, the meta-optimizer is Adam with an initial learning rate of 0.0005. The inner learning-re rate is kept constant, at 0.002. The number of inner updates is during training, and 20 test time.
A.7.3 TEST DETAILS (VISION PROBLEM)
At test time, wrap model with the arctan function convert the predictions angle predic- tions. Then, we compare the angle predictions with the ground-truth angles. To do so, we use the following error from Gao et al. (2022) to compare two angles:
E(3 = min ( (E,t,st, E,, -e.ga.& E3.
where E3t,9- = ly = 360 - y*1 and Eg.9- = ly - y²
When plotting the performance (Figure 9. we plot the average and ci95 on 100 tasks (15 queried in- puts each). For UNLIMITD-R, the average and ci95 are computed different random projection trainings.
18

---

Under review as a conference paper at ICLR 2023
15
-12
32E--311 (CHS
24
232
21-
23-
1.2
(a) UNLIMITD-I (inf)
(b) UNLIMITD-R (inf)
(c) UNLIMITD-F (inf)
Sibae Jeafi
VOAueUNEE (eafl
SDMwercc (eal)
23-
234
214
Cancer
23-
264
1.3
-35
SEE IC
(d) UNLIMITI-II (fin)
(c) UNLIMITD-R (fin)
(f) UNLIMITI-F (fin)
Figure 6: Example of predictions for varying number of context inputs K, after different meta- trainings on the sine cluster (UNLIMrTD-I, UNLIMITD-R and UNLIMrTD-F, in the infinite and finite case). Standard deviation is from the posterior predictive distribution. Note how UNLIMITD- and U perform better than UNLIMITD-I when comes to reconstructing the sine with smaller amount of context inputs.
19

---

Under review as a conference paper at ICLR 2023
1 13
1.4
(a) Mean function BB() of the (UNLIMITD-F with single GP).
(b) Mcan function B(f) the GP of mixture (UNLIMITD-F with mixture GPs).
(c) Mcan function B(f) of the second the mixture
(UNLIMITD-F with a mixture of GPs).
Figure 7: Mean functions of the GP (when learning with GP) the GPs composing the mixture (when learning mixture of GPs), after training on both the sine and line cluster with UNLIMITD- F. Note how the mean of the single GP is intermediate between the ones of the mixture.
Covariance cov(f(1), f()) the first GP of the mixture (UNLIMITD- (UNLIMITD-F with a single GP).
(a) Covariance function cov(f f())) of the GP
(b)
function
function cov(f(0), f(x)) of the second GP of the mixture (UNLIMITD-
(c) Covariance
F with a mixture of GPs)
F with a mixture of GPs)
Figure 8: Covariance functions of the GP (when learning with GP) the GPs composing the mixture (when learning a mixture of GPs), after training on both the sine and line cluster with UNLIMITD-F. Note how the covariance of the single GP is not accurate for any of the two clusters.
Daler
Aude
UNLIMITIL
UNLIDTIDE ENLISITD
SUASIL
Figure 9: Left: Examples of vision tasks. Right: angle error function of the number of context datapoints K, after different meta-tran trainirinnngs on Shapenet (UNLIMrTD-I, UNLIMITD-R and UNLIMITD-F vs MAML). Note how UNLIMITD-L and UNLIMITD-F make better predictions than MAML. Also note that contrary to the small network case, UNLIMITD-R performs worse than MAML and than the two other variants.
20

---

Under review as a conference paper at ICLR 2023
80
80
80
80
80
88
300
100
100
120
300
120
100
100
68
100
120
(a) Angle: 4"
Angle:
(c) Angle: 202
Angle: 261
20
40
(b) Angle: 26
(d) Angle: 1300
(f) Angle: 243
(h) Angle: 285
Figure 10: Examples of images and ground-truth angles from the Shapenet D dataset (Gao al.
2022
21


Reviews:
[{"review_id": "lUnb182fxrx", "reviewer": "Reviewer_xBkp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an uncertainty-aware meta-learning method for multi-modal task distributions. \nIt utilizes a learnable parametric distribution on a linearized neural network to model the real distribution over tasks. The method uses Bayesian inference to achieve uncertainty-aware adaptivity and learns multi-modal task distributions through a mixture of Gaussian processes.\n\nWhat contributions does it make:\n1.It combines meta-learning and uncertainty from a probabilistic perspective.\n2.It makes efficient probabilistic prediction on in-distribution tasks by parametrically learning prior distributions, while efficiently detecting OoD context data at test time.\n3.It learns multi-modal task distributions through mixed Gaussian processes.", "summary_of_the_review": "The method uses Bayesian inference to achieve uncertainty-aware adaptivity and learns multi-modal task distributions through a mixture of Gaussian processes. The idea makes sense and the experimental results perform well. ", "strength_and_weaknesses": "The main strengths:\n1.It models the true distribution of tasks by performing Bayesian inference on a linearized neural network.\n2.By defining different covariances in the prior distribution, three variants are proposed:  identity covariance, low-dimensional covariance with random directions, and covariance based on Fisher information matrix, which can balance scalability and expressivity of the model.\n3.The experimental results perform well. The MSE is small which shows the accurate probability prediction on in-distribution tasks. The AUC-ROC scores also demonstrate the effective detection of context data from OoD tasks.\n\nThe main weaknesses:\n1.The review of related work on uncertainty in meta-learning is small and it is difficult to locate the main contribution of this paper.\n2.The Ood detection only extends from classification to regression, which is not very innovative.\n3.The experiments only focus on the comparison between various models proposed by authors and lack of comparison with other methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hAyHsZqH0j", "reviewer": "Reviewer_eEkt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose to tackle limited data scenario in meta-learning setting. They make use of Bayesian linear regression and Gaussian processes to model uncertainty when the data is limited. The experimental results demonstrate the efficacy of their method.", "summary_of_the_review": "In summary, the authors proposed interesting methods for meta-learning for limited data regime, but the experimental results are very limited. Therefore, I recommend reject.", "strength_and_weaknesses": "Strength\n- Paper is well written\n- Method has been developed well based on theoretically solid Bayesian regression and Gaussian processes.\n\nWeaknesses\n- The experimental results are all toy examples, such as sinusoidal regression and mixture of simple regression problems. Those experimental settings are far from real-world scenarios and also the amount itself is also very limited.\n- Insufficient baselins. There are many works that solves exactly the same problem, including all the Bayesian version of MAML and probabilistic versions of Prototypical-like networks. But they only compare agains themselves or simple vanilla MAML baselines. To name a few, see the below references. There should be tons of more methods the authors should compare against.\n\nReferences\n- Harrison et al., Meta-Learning Priors for Efficient Online Bayesian Regression\n- Kim et al., Bayesian Model-Agnostic Meta-Learning\n- Finn et al., Probabilistic Model-Agnostic Meta-Learning\n- Gordon et al., Meta-Learning Probabilistic Inference For Prediction\n- Willette et al., Meta-Learning Low Rank Covariance Factors for Energy-Based Deterministic Uncertainty\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WQ4lc2cSpz", "reviewer": "Reviewer_pWMB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method  that is able to deal with regression problems in a meta-learning framework. The method estimates a parametric and tuneable distribution, leveraging Bayesian inference with linearized neural networks. The method is flexible and able to deal with unimodal and multimodal task distributions. Experiments on sine-waves prediction are provided to showcase the effectiveness of the proposed solution.", "summary_of_the_review": "The paper provides an interesting angle on the few-shot regression problem, which is significantly different from previous work. However, the limited empirical evaluation, shortage of baselines, and possible scalability issues, arise serious concerns that need to be addressed in the rebuttal.", "strength_and_weaknesses": "\n\nStrength\n--------\n\n- The method can be used in a variety of settings, which makes it flexible.\n\n- The possibility of quantifying uncertainty is a plus, this makes the method superior to classical meta-learning approaches like MAML.\n\n- The method is able to deal with the challenging multimodal task distribution setting, which differentiates it from previous probabilistic methods.\n\n\nWeaknesses\n----------\n\n1) My main concerns regard the empirical evaluation, in particular the tasks used. The sine-wave experiments are a basic way to evaluate the performance of the proposed method. A more challenging version of the sine-wave is the out-of-range condition, where some methods have failed to generalize, for more details see Patacchiola et al. (2020). Overall, the empirical evaluation needs to be bulked up. Reporting the results on a larger set of regression problems would be beneficial, e.g. see the few-shot datasets used in Sendera et al. (2021).\n\n2) Another concern regards the baselines used in the experiments. The authors compare against MAML (e.g. see Figure 3), which is a pretty poor baseline for the regression case. The authors disregarded a line of work rooted in probabilistic and Bayesian theory that has been quite successful in dealing with unimodal task distributions in meta-learning. Those methods are neither mentioned nor discussed in the paper. The only exception is a good comparison with ALPaCA (Harrison et al. 2018) provided in the appendix. Notable examples of paper that should be discussed and compared against are R2-D2 (Bertinetto et al. 2018), ADKL (Tossou et al. 2019), Deep Kernel Transfer (Patacchiola et al. 2020), and NGGPs (Sendera et al. 2021). In particular, Patacchiola et al. (2020) provide a table that compares all these methods on the sine-wave experiments (in-range and out-of-range conditions). It is important to see how the proposed method compares against these stronger baselines.\n\n3) Working in weight space can be problematic, in particular when representing the covariance matrix. This has been addressed by the authors in Section 4.1.1. My concern here is that using a low-dimensional representation of the covariance may be appropriate in some cases but it may not be expressive enough in others. Given the limited set of experiments and backbones tested, it is hard to get a grasp on this point. For instance, it is not clear how the expressiveness of the covariance-matrix is affected by using neural networks of different size/type. I could not find a satisfying report on this, just a few lines in Section 6 (paragraph \"Unimodal meta-learning\"). It would be useful to see how the performance of the I-R-F variants change when using models of different size (while keeping fixed the dataset) and with deeper backbones (e.g. in visual regression problems).\n\n4) A corollary of the previous point is the computational complexity of the method, in terms of both time and space. In particular, it would be useful to see how the computational complexity of the I-R-F variants changes when using common backbones and how it compares to other probabilistic methods. This is quite important here, as it can be a major bottleneck that hinder scalability.\n\n5) Minor corrections. In Equation 4 there are unmatched brackets. Figure 3 is hard to read, especially when it comes to the values on the right side where the curves are very close, a tabular version may be better. A brief description of the variants of UNLIMITD called I-R-F should be added earlier in the manuscript to improve clarity (e.g. at the end of the Introduction).\n\n\nReferences\n----------\n\nBertinetto, L., Henriques, J. F., Torr, P. H., & Vedaldi, A. (2018). Meta-learning with differentiable closed-form solvers. arXiv preprint arXiv:1805.08136.\n\nHarrison, J., Sharma, A., & Pavone, M. (2018, December). Meta-learning priors for efficient online bayesian regression. In International Workshop on the Algorithmic Foundations of Robotics (pp. 318-337). Springer, Cham.\n\nPatacchiola, M., Turner, J., Crowley, E. J., O'Boyle, M., & Storkey, A. J. (2020). Bayesian meta-learning for the few-shot setting via deep kernels. Advances in Neural Information Processing Systems, 33, 16108-16118.\n\nSendera, M., Tabor, J., Nowak, A., Bedychaj, A., Patacchiola, M., Trzcinski, T., ... & Zieba, M. (2021). Non-Gaussian Gaussian Processes for Few-Shot Regression. Advances in Neural Information Processing Systems, 34, 10285-10298.\n\nTossou, P., Dura, B., Laviolette, F., Marchand, M., & Lacoste, A. (2019). Adaptive deep kernel learning. arXiv preprint arXiv:1905.12131.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0RmSX4eqDf1", "reviewer": "Reviewer_J2Ni", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a Gaussian Process (GP)-based approach to meta-learning. Using GP, the method allows probabilistic predictions for in-distribution tasks and detection for out-of-distribution (OoD) tasks. Several practical algorithm variants are introduced to tackle computational requirements of GP. Empirically, the proposed method is compared to MAML on regression problems and show improved performance.", "summary_of_the_review": "The proposed method introduces a GP-based approach to meta-learning, which grants several desirable properties including probabilistic prediction and OoD task detection. The approach appears well-motivated and sound. However, current empirical evaluation is limited: only MAML baseline is considered and only regression setting is considered. It is difficult to assess the efficacy of the proposed method compared to the broader literature.", "strength_and_weaknesses": "Strength:\n1. The paper is well-written and easy to follow. All elements of the method are well-motivated. The GP-based approach is sound.\n2. Detailed discussion is presented for the practical algorithm, highlighting the trade-offs and design constraints.\n3. The empirical comparison with MAML is detailed and shows improved performance.\n\nWeakness:\n1. The method so far only tackles regression problem, excluding a large segment of the meta-learning literature on classification. In principle, it should be straightforward to transform classification into regression (see the ridge regression approach in [1, 2]), thus allowing the proposed method to be compared to wider sets of baselines in more settings.\n2. Related to 1, the empirical comparison is very limited. Only MAML is used as the baseline. It is also unclear why OoD detection is only evaluated among the different variants of the proposed method, rather against baselines discussed in related work section.\n3. Computational requirements should be better discussed with respect to other methods. For instance, does the method require second-order derivatives during optimization? If so, does this either 1) limit the size of the network or 2) require multiple-gpus or TPU to optimize? The network used for the experiments should be mentioned in the paper.\n\n[1] The Role of Global Labels in Few-Shot Classification and How to Infer Them, Wang et. al, Neurips 2021\n\n[2] Meta-learning with differentiable closed-form solvers, Bertinetto et. al, ICLR 20119\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject