Meta-Learning Universal Priors Using Non-Injective Change of Variables
Authors: Yilang Zhang
Alireza Sadeghi
Georgios B. Giannakis
Venue: NeurIPS 2024 poster (2024)
Keywords: Meta-learning
prior expressiveness
change-of-variable formula
Primary area: deep_learning_architectures

Abstract: Meta-learning empowers data-hungry deep neural networks to rapidly learn from merely a few samples, which is especially appealing to tasks with small datasets. Critical in this context is the *prior knowledge* accumulated from related tasks. Existing meta-learning approaches typically rely on preselected priors, such as a Gaussian probability density function (pdf). The limited expressiveness of such priors however, hinders the enhanced performance of the trained model when dealing with tasks having exceedingly scarce data. Targeting improved expressiveness, this contribution introduces a *data-driven* prior that optimally fits the provided tasks using a novel non-injective change-of-variable (NCoV) model. Unlike preselected prior pdfs with fixed shapes, the advocated NCoV model can effectively approximate a considerably wide range of pdfs. Moreover, compared to conventional change-of-variable models, the introduced NCoV exhibits augmented expressiveness for pdf modeling, especially in high-dimensional spaces. Theoretical analysis underscores the appealing universal approximation capacity of the NCoV model. Numerical experiments conducted on three few-shot learning datasets validate the superiority of data-driven priors over the prespecified ones, showcasing its pronounced effectiveness when dealing with extremely limited data resources.

Meta-Learning Universal Priors Using Non-Injective Change of Variables
Yilang Zhang Department of ECE University of Minnesota Minneapolis, MN 55414 zhan74538umn edu
Alireza Sadeghi Department of ECE University of Minnesota Minneapolis, MN 55414 sadeg012@umn. edu
Georgios Giannakis
Department of ECE
University of Minnesota Minneapolis, MN 55414 georgios@unn. edu
Abstract
Meta-learning empowers data-hungry deep neural networks rapidly learn from merely a few samples, which is especially appealing to tasks with small datasets. Critical in this context is the prior knowledge accumulated from related tasks. Existing meta-learning approaches typically rely on preselected priors, such as a Gaussian probability density function (pdf). The limited expressiveness of such pri- ors however, hinders the enhanced performance of the trained model when dealing with tasks having exceedingly scarce data. Targeting improved expressiveness, this contribution introduces a data-drives prior that optimally fits the provided tasks using novel non-ing non-injeen -injinecctvve change- of variable (NCoV) model. Unlike preselected prior pdfs with fixed shapes, the advocated NCoV model can effectively approx- imate a considerably wide range of pdfs. Moreover, compared to conventional change-of-variabe models, the introduced NCoV exhibits augmented expressive- ness for pdf modeling, especially in high- nigh-dimensional spaces. Theoretical analysis underscores the appealing universal approximation capacity of the NCoV model. Numerical experiments conducted on three few-shot learning datasets validate the superiority of data-driven priors over the prespecified ones, showcasing its pronounced effectiveness when dealing with extremely limited data resources.
Introduction
Advances in deep learning (DL) have boosted the notion "learning from data" with field-changing
performance improvements reported across wide range of applications (24 15 52 Large-scale DL. models with high fitting capacity have documented ability to cope with the "curse of dimensionality" by providing compact low-dimensional representations of high-dimensional data. Nonetheless, these high-capacity models typically require protracted training using massive data records. Humans on the contrary, can perform exceptionally well on tasks such as object recognition concept comprehension with merely a few samples. How to acquire the learning ability of humans in the DL training processes thus appealing and imperative for number of application domains, especially when data are scarce or costly to annotate. Examples of such applications include machine translation [52], medical imaging [31], and robot manipulations 27
Meta-learning also referred to as "learning to learn," seeks to gather the prior knowledge shared across set of inter-related tasks, to enable quickly solving an unseen yet related learning task using minimal training samples [10]. This form of higher-level learning effectively extracts domain-generia inductive biases from prior tasks, which can be subsequently transferred to learn with limited data. This mirrors the capability that humans excel at - leveraging past experiences to rapidly acquire skills. Meta-learning holds the promise of yielding powerful priors with which
task even
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

DL models can generalize better, require fewer data for training, and adapt more effectively to new tasks in dynamically changing environments.
Conventional approaches to meta-learning have relied on hand-crafted techniques to extract prior knowledge 47 (66. With the advent of DL and growing volume of data, there has been paradigm shift from such cumbersome procedures towards more efficient data-driven strategies. In particular, the prior information encoded in hyperparameters, which are shared across tasks and can be fine- tuned using the validation data of all tasks. Utilizing these informative hyperparameters, task-specific learning can be performed even with limited data. Early attempts adopted a neural network (NN) with its weights serving as the shared hyperparameters [55] 45 33 The task-invarian NN leverages the shared hyperparameters, and training data per task, output the task- wssk-specific model. However, the selection of an appropriate NN architecture is tailored to the choice of the task-specific models. In addition, NNs inherently lack interpretability and robustness due to their "black box" nature.
Unlike NN-based meta-learning, aggeos-g meta-learning (MAML) does not rely on any pre- sumptions about task-specific models [IO]. Instead, relies on an iterative optimizer to learn the task-specific model. The task-invariant prior information is embodied in the initialization of the optimizer, which is shared across tasks. By learning an informative initialization, task-specific learning can rapidly converge to a local minimum within a few iterations. Interestingly, the ini- tialization generated by MAML can be viewed as a learnable mean of an implicit Gaussian prior probability density function (pdf) over the task-specifi model parameters [18]. Building on MAML, several optimization- based meta-learning algorithms have been advocated to learn different prior pdfs [29 37 25 1 (58. In addition, theoretical studies have been carried out to further offer insights into these approaches 13 39 9 61. Nevertheless, the prior models of most existing meta-learning methods are confined to preselected pdfs, such as the Gaussian one, and thus have limited expres- siveness, meaning fitting ability. Consequently, generalizing meta-learning to domains that deal with scarce datasets, and need sophisticated priors, remains a challenging and largely uncharted territory.
To improve the prior expressiveness in meta-learning, this contribution puts forth what term non-injective change-off of of-varialle (NCoV) model, which enables learning universal data-drivon prior from related tasks. The contribution of the resultant method named MctaNCoV threefold:
i) Our novel NCoV model is proven capable of mapping a known source pdf to an arbitrary target pdf. This markedly enhances the model expressiveness, especially in high-dimensional spaces.
ii) Theoretical analysis is provided to demonstrate that a parametric NCoV can approximate a broad spectrum of pdfs, that in turn enables versatile plug-in priors for meta-learning. Moreover, this parametric NCoV inherently provides a insk-invariant initialization, rather nicely eliminating the need for its explicit learning.
ii) Numerical tests on three benchmark few-shot learning datasets corroborate our theoretical analysis, and underscore the superior prior expressiveness of the proposed MetaNCoV method compared to meta-learning approaches with prespecified pdfs.
2 Problem setup
Meta-learning relies on
prior information from a collection of T given tasks (indexed by settings. For each t, there a dataset D. := [(x] y } N;
t ,I....,77 ,T). to deal with consisting of N₂ (data, label) pairs. The dataset is divided into a training subset Diru C DD, and validation subset Dy := D, Di In addition, a new task indexed by * is also provided, with its
training set Drr and an unannotated test set Dist := [x"] Ny
| for which the corresponding labels
{y"), J are to be inferred. The major premise of meta-learning is that the aforementioned tasks are related through their underlying data distributions or problem structures. This relationship makes it feasible to employ a unified large-scale model such as a deep NN to fit all tasks, with each task tailored by its specific model parameter &, E Rd. However, as the cardinality |D!Tm can be much smaller than d, directly optimizing Ø, over D. could readily lead to overfitting.
Niat
Meta-learning addresses this issue by capitalizing on the relationships among tasks. Specifically, since 7 considerably large in meta-learning, a lask-invariant prior can be extracted to capture knowledge across tasks, thereby facilitating the data-limited per-task training. This nested structure of prior extraction and per-task training lends itself to a bilevel optimization problem. The inner- (task-level) optimizes the per-task parameter s using Dim and the prior provided by luter-level,
2

---

while the outer-level (meta-level) evaluates the trained { using {Dral)T= prior parameterized by R."
and refines the
The bilevel optimization objective of meta-learning can be expressed as
min
2 (0); Diah)
(Ia)
s.t. 0 (0) = argmin L(o,; D( + R) t == I....
(lb)
4:
where the loss function & assesses the fit of a task-specific model to a designated dataset, and the regularizer R quantifies the impact of task-invariant prior. From the Bayesian viewpoint, C( ) = - log ply? (0,; x/r) can be interpreted as the negative log-likelihood (nll), and Ri 0) log pl 0) is the negative log-prior (nlp), where X!r" denotes the matrix collecting all the data vectors in Dim and y! is the corresponding label vector. Using Bayes' rule, it follows that Φ; = ø p( p(o.lyt trm. X!rm, O) is the maximum posteriori (MAP) estimator.
Unfortunately, the global optimum of in (Ib) generally unreachable when the postulated model is a nonlinear function of Φ.. Hence, a feasible alternative is to rely on an approximate solver d R of obtained by a tractable optimizer. Depending on how the alternative solver is acquired, meta-learning algorithms can be categorized as either NN- or optimization-baat based ones. The former harnesses an NN optimizer Z = (D 0) to model the training process that maps Dern to Φ., with the sought prior encoded in the NN's learnable weights 0 41 17]. Despite the effectiveness of NN optimizers in fitting complex mappings, it is hard to decipher the learned prior due to their black-box nature. To improve the interpretability and robustness of the approximate solver, aptimization-basee meta-learning decodes the "tractable optimizer" as a cascade of a few optimization iterations. The prior is captured by the shared hyperparameters of the optimizer. The first effort towards this direction is termed MAML [0). which relies on a K K-step gradient descent (GD) optimizer
( - o. (0) - V (0); Di k == o...,..,
(2)
where lask-invariant initialization of = Φ 0 parameterizes the prior information, and Φ Φ gives the desired approximate solver. Interestingly, despite the absence of an explicit regularization term (that is, R( 0) 0), it has been shown that, under second-order Taylor approximation,
MAML's GD solver 2 satisfies [18]
1 =110; 2211. 2
( R (0) = argmin D!T))
+
+.
where the precision matrix A, is determined by a. K, and V2L(0" : This observation indicates MAML's optimizer approximately amounts to an implicit Gaussian prior p(O;; 0) Ro N( Φ0 A,'), with the shared initialization Φd = 0 serving as its mean vector.
Building upon MAML, various methods have been investigated to learn different prior pdfs in both implicit and explicit forms. For example, recent advances further render the (per-step) precision matrix learnable by replacing it with a A that is common across tasks. Letting A denote the parameter of A, the prior parameter is thus augmented as 0 := [ 0 [. where T denotes transposition. However, a complete parametrization of A would result in 0 having prohibitively high dimensionality, that is, D~ (d²)). To ensure scalability with respect to D. A should have sufficiently simple structure such as isotropic (3), diagonal (2,, and or block diagonal [26] [7]
37) matrices. Inspired by transfer learning, one can instead split the model into an embedding "body" and a classiffer/regressor
"head," and learn their priors independently; that is, with Φb body and dthead denoting the corresponding partitions of Φh. the prior is presumed factorable as p( p( 0) = p(oshody, 0 0)p((())) head 0). On the one hand, the head typically has a nontrivial prior such as the Gaussian one K 255 On the other hand, the body
body's 's prior is intentionally restricted to a degenerate pdf body .) == b where of body a subvector 0, and 5(-) is the Dirae delta function. This eliminates the need for optimizing Φ hody in (Ib). thus markedly lowering the overall complexity for solving
[.. Although freezing
the body in (bb) allows for escalating the dimension of bb it often leads to degraded empirical performance [38] compared to the full update (2). In addition to Gaussian and degenerate pdfs, sparse priors (Laplace distributions) have been investigated in the context of network pruning [50].
3

---

3 Meta-Learning using non-injective change of variables
Existing meta-learning algorithms rely on a preselected pdf to parameterize the prior. However, the chosen pdf can have limited expressiveness; that is, it may have insufficient ability to offer an accurate fit due to its prefixed shape. Consider for instance a Gaussian prior pdf, which is inherently unimodal, symmetric, log-cone oggcconave, and infinitely differentiable by definition. Such prior may not be well-suited for tasks with multimodal or asymmetric parametric pdfs.
In this work, we propose to learn a data-driven prior pdf that optimally fits the given tasks using a novel non-injective change-of-varrable (NCoV) model. We thus term the proposed method as Meta-learning with NCoV (MetaNCoV). In contrast to preselected prior pdfs with fixed shapes, the advocated prior model can dynamically adjust its form to approximate a considerably wide range of pdfs, as will be demonstrated both theoretically and numerically, Furthermore, compared to conventional change-of-variable models such as generative adversarial networks (GANs) [6] and normalizing flows (NFs) 43), the introduced NCoV exhibits enhanced capacity for pdf estimation, especially in high-dimensional spaces. Change-of- varriaaab models and their applications in pdf estimations will be first elaborated. All the proofs are delegated to the Appendix.
3.1 Pdf estimation via change of variables
The key idea of change-of-variable model is to identify a transformation S. through which known pdf pz can be altered to approximate a target pdf q. For instance, GANs [16] and variational autoencoders (VAFs) 21 seek generator/enccoder such that high-dimensional pdfs can be acquired from a low-dimensional latent Gaussian pdf pz = (O, I). Due to the dimensional discrepancy between signal and latent spaces, these models are typically utilized to estimate signals living low-dimensional manifold; c.g., images.
To enhance the model capactiy as well as pdf tractability, NFs were introduced in 43 as a surrogate variational model for posterior inference. Recently, they have been shown also effective in estimating prior pdfs from a set of unannotated samples [6] [4] 77]. The formulation of NFs relies on the well- known change-of-variable formula. Given continuous random vector Rd with prior pdf pz. and bijection f IRd + Rdd then Z' = ((Z) is also a continuous random vector with analytical pdf
pz (z)) |dct. Jy('))
pz (z)) = ( ( (2')) |det J,- ()))
(a.e.)
where Jz(2), denotes the Jacobian of f at z' Rdd det is the determinant, and det J, * 0 almost everywhere (a.e.) for bijective f. To ensure the invertibility of f. a prudent choice is to model it as a composition of sequence of bijective functions - fi f2 o... fn.
In Bayesian inference [43, q is an intractable posterior, and f is optimized to minimize the KL- divergence between Pz' and q, equivalently, maximize the so-termed evidence lower bound (ELBO). For density estimation (6), the wanted q is an unknown prior pdf, while f is acquired via maximum likelihood training. The obtained f can be leveraged in important applications: i) probability estimation Pz (v) se q(v) for a given sample v ~ q using 3 and ii) generation of sample x' f(z), pz for which pz R q.
When d 1, the probability integral transform (PIT) suggests that, the optimal -==Q=1 Pz leads to precisely Pz Q a.e., where Q. Pz and Pz are the cumulative distribution functions (cdfs) corresponding to q. pz and pz', and g > 0 a.e. ensures Q is bijective. The resultant cdf Pz Pz fz-1 is pushforward measure, also notated as Pz: f" # Pz. In high-dimensional spaces (d > 1) however, the existence of such an f* may not hold due to the invertibility assumption of f"; see examples in e.g., [22] Section 4]. In fact, it has been shown that NFs capable of modeling pdfs with a full support; i.c., when q > 0 on Rd (36]
3.2 Improved pdf estimation via non-injective change of variable
To improve the fitting capacity of change-of-variabl models for generic g. especially those in high- dimensional spaces or without full support, the fresh idea of this work is to waive the injectivity assumption on f. In doing so, we can generalize the PIT to an arbitrary 4, as illustrated in the following theorem.

---

Theorem 3.1 (Multivariate PIT), Let Pz RR + [0.1] be the cdf of continuous random vector Z == [Z1,..., Za]" with {2:)i=1 mutually independent. For any differentiable a.e. cdf Rdd + [0. 1). there exists a non-decreasing function f : Rd +> IRd that the random vector Z' == (Z) has cdf
Pz = =Q (a.e.).
(4)
Remark 3.2 (Choice of source distribution). In the theorem, the prior distribution for the source random vector Z can be chosen arbitrarily, as it continuous and has mutually independent entries. Popular choices include standard Gaussian N (0,, Ia) and uniform 24 ([0, 11").
Remark 3.3 (Comparison with injective NFs). While conventional NFs (3) require Jj # 0 a.e. to ensure the injectivity of f. Theorem 3.1 relaxes this assumption to allows I being non-injective and thus enables Z' fZ) to match an arbitrary target distribution (even discrete one) in high- dimensional space. is worth mentioning that the mild assumption on the differentiability of Q is merely used to guarantee the existence of g. which can be easily satisfied. However, one limitation of the advocated NCoV is that it generally has no analytical solution for the resultant surrogate pdf
pz
pz(2)6(z' - [(z)]dz.
(5)
As a remedy, efficient numerical integration can be performed to estimate pz when d is small. Additional comparisons with NFs and optimal transport are deferred to Appendix
suggests the existence of the optimal f that incurs the exact match pz = the expression for such an f relies on the sought g, which is typically intractable or unknown. Therefore, feasible alternative is to resort to a tractable parametric [(; Oy), which approximates f* by learning f from the provided data. To further compare NCoVs with NFs, we will focus exclusively on Sylvester NF [51) in the following sections, but our analysis can be readily generalized to other transformations; see Remark 3.7 Sylvester NF was introduced in [5]] to improve the expressiveness of planar NF 43 by increasing its "width". In particular, Sylvester NF adopts the form
While Theorem 3.1
(Zz = Z + Ao + c); Z E Rd
(6)
where A E Rd B E Rm , RT are learnable weights with m being the number of hidden neurons (a.ka. width), a is an entry-wise nonlinear operator, and 0 == [vec(A) vec(B) c. It can be easily verified that the Sylvester NF boils down to the planar one when other NFs, one can also increase the "depth" of the flows by stacking multiple Sylvester NF layers into a chain fi f2 c ..... fn- The next theorem states that, the optimal f can be approximated to arbitrary precision using sufficiently wide one-layer Sylvester NCoV.
1. Akin to
Definition 3.4. A random vector Rd is said be tail-convergent if i) it has pdf p IRd R+ U {0), and ii) for Ve >> 0 there exists bounded E' C R2 for which
J 34A CCC.
Theorem 3.5 (Universal approximation via Sylvester NCoVs). Let Pz denote the edf of tail- convergent continuous random vector Z E Rdd with mutually independent entries, and Q Lip- schitz edf of a tail-convergent random vector. For any e >> 0, there exists cdfs P, Q for which the corresponding pdfs P.q vanishes outside compact sets Ep, Egg and
P2(v)) P(v)) < e, (Qz(v)
< VVERd.
(
Moreover, let E C Ep be any set on which the optimal f" matching Pz to Q is injective and right-continuous. There exists a Sylvester NCoV f and cero-measure set Ev, such that
ls(Z) - (Z) <e, VZ EE Eo. P - fzz(()) <E, Vz E E
(9a)
(9b)
We have shown that when f* is injective, the cdf of the optimally transformed Z' - (Z) can be written as a pushforward Q = Pz Pz fr-1, Likewise, this relationship remains valid when restricting f to set E on which f is injective. However, since the Sylvester NCoV ff may not be injective on E, one cannot directly compare Q with Pz o f-t. Fortunately, this pushforward can be equivalently written as (z(z) = (2), Vx E E; see Lemma [.l the Appendix. Utilizing this alternative relationship, Theorem 3.5 states that the Sylvester NCoV ʃ not only approximates f" a.e on Ep. but also results in pushforward approximation Pz Re Q f a.c. on E.
5

---

Algorithm MetaNCoV algorithm
Input: (D, step sizes a and 8, batch size B, and maximum iterations and R.
Initialization: randomly initialize 09.
for ...., R do Sample random T C (1,...,77 T) of cardinality B. for f do
Initialize zi argmilly, (z(z.). for k ...., K do Descend 2 z av [C[f(z];; ( Rz(z,)). end for Approximate solver 2, = =
end for Update 07+ = 8 - - LIET Vo, (7) end for Output: 0, 07.
Estimated
SCoVss f(z)
307
E30
034
30
pdt's of interest, Q()
041
CN
00
622
500
(9)
(b)
(c)
Figure 1: Transforming standard Gaussian pdf into multi multi-modal target pdfs using Sylvester NCoVs.
Remark 3.6 (Mild assumptions). The assumptions in Theorem 3.5 are mild and common. In particular, tail-convrreence only requires the probability of large deviation diminishing to the norm of the random vector goes to +oo, while imposing no constraint on the decaying rate. This assumption can be easily satisfied by wide family of distributions, even including the heavily-tailed ones. Under this benign assumption, (8) suggests Pz and Q can be approximated by alternatives P. Q with pdfs p, q having truncated tails. This is crucial to universal approximation, which typically requires f to be bounded Lebesgue integrable (5). Moreover, the Lipschitzness of Q is solely utilized to ensure the boundness of its gradient, namely the pdf q. This can be also readily met by most practical cdfs.
Remark 3.7 (Generalization to other NFs). Although Theorem 3.5 primarily focuses on one-layer Sylvester NCoVs. similar analysis for other transformation f can be acquired by employing different universal approximation models. For instance, results for multi-layer planar NCoVs and multi-layer Sylvester NCoVs can be respectively established leveraging 30 and [2]
Remark 3.8 (Influence of e). It is worth noting that the width m of the Sylvester NCoV depends as well as the optimal f". Smaller typically leads to larger m. Additionally, the nonlinearity a must be sigmoidal; see Definition B.3 in the Appendix for details.

---

3.3 Meta-learning universal priors via MetaNCoV
Next, we elucidate how universal priors can be learned in meta-learning by harnessing the proposed NCoV model. Different from existing works that rely on prespecified prior forms such as Gaussian pdfs, the novel concept of this work to learn a data-driven prior that optimally conforms with the given tasks. This is achieved by transforming the random vector Z E Rd with a known prior pz to Z' f (Z whose pdf is given by (5). This pz' acts as a surrogate model for the unknown p( f), and learning the prior parameter 0 thus boils down to optimization of the transformation parameter , Nevertheless, as discussed in Remark 3..3] pz' typically has no close-form expression when is non-injective. Therefore, instead of directly optimizing Φ,, we propose to optimize the latent vector 2, corresponding to o;- = J (Zz; 0y), which yields
min
(0)) Di
(10a)
s.t. (0,) = argmin (z)(z; Df); + Rz(z;), VI
(10b)
2,
where R2(z)) == 11--lpp pzz) is the nlp regularizer, and z is thus the MAP estimator for 2І-
Similar to 2 the global task-level minimizer z is generally infeasible to attain. Hence, tractable
alternative to rely on an approximate GD solver. Interestingly, our formulation (TO) naturally offers
convenient initialization using the muximum priori estimator
Z = argmax pz (z:) = argmin Rz(z:), Vt
(11)
2;
As an example, choosing Pz = (Oa, Ia) automatically gives z = O₂ and the corresponding Φf f(2;; 0y). This elegantly removes the need for separately learning the task iivariint initialization 8". which is exactly the maximum a priori estimator of the preselected Gaussian prior pdf p(0,: 0) = N( A,). In fact, the task-invarian initialization reflects our optimal guess of o; before accessing any task-specific data, and can be naturally derived by maximizing the prior pdf. It also worth noting that while the idea of optimizing the latent (instead of primal) variables shares similarities with 44] the latent space in [4] is designed from different perspective, which is oow-dimeniional to the end, and requires an initialization.
To this end, TO can be solved using standard alternating optimizer. The resultant MetaNCoV algorithm is listed step-by-step in Algorithm | where the inner-level (TOb) and outer-level (Ta)) are respectively optimized using K K-step GD and mini-batch stochastic GD.
While our idea has the potential to be broadened beyond meta-learning, we must emphasize that our current setup is specifically tailored to meta-learning, which does not require tractable pdf, but rather demands enhanced prior expressiveness. We should also highlight that the intractability of (5) prohibits learning NCoV via conventional approaches such as maximum likelihood training and evidence power-bound maximization this thus necessitates careful attention and extra certain designs when applying NCoV to other domains.
4 Numerical tests
In this section, we test and showcase the empirical superiority of MetaNCoV on both synthetic and real datasets. All datasets descriptions and hyperparameter setups are deferred to the Appendix [C] Codes for reproducing the results are available at httpps:: //t:::///tttttt com/ zhangyilang Maannyyylagg/Mtta/Cooo MetaNCCV
4.1 Tests with toy data
Here, we investigate an intricate yet interesting scenario to demonstrate the efficacy of NCoVs to approximate complex multi-modal pdfs in two-dimensional (2D) settings. The primary objective is to transform standard Gaussian random vector Z ~ (02xx1 I2x2) into multi-modal complex pdfs. The outcomes of this experiment are presented in Figure | The lower row displays the ground-truth pdfs of interest, while the upper row showcases the numerically estimated pdfs of the transformed random vector Z' (Z), where / a Sylvester NCoV, and the pdf of Z' is estimated via (5). As clearly evidenced in these results, the advocated NCoVs exhibit their capability to effectively convert
7

---

Table 1: Performance comparison of MetaNCoV against meta-learning methods having different priors. For fairness, only methods with a 4-block CNN backbone have been included. The highest accuracy as well as the mean accuracies within its 95% confidence interval are bolded.
5-class minilmageNet
Method
Prior model
1-shot (%)
5-shot (%)
Meta-LSTM
41
RNN-based
43 43.44_0.77
60.60 40.71
MAML 10
implicit Gaussian
48. 18.70 1.84
63. 1551114m 1..92
MetaSGD 29
diagonal Gaussian
50.47 =1 1,88
64.03 1.994
R21D2 13
degenerate body & Gaussian head
51.810.2
68.4
MC [37
block-diagonal Gaussian
54.08=0.998
67. 67.99
Warp-MAML 12
Gaussian
52.3+0.8
68, 68.440.6
MAML + L2F
implicit Gaussian
52.10
69.38
MeTAL M
implicit Gaussian
52.63
70.52
Minimax MAML
Minimax-MAMLL
58
inverted Gaussian & entropy
51.70=00422
68.41+1.25
MAML + MetaNCoV
57. 57.74 1.47
70.7220.70
NCoV-based
MetaSGD + MetaNCoV
59.10=1.52
71.48=0.66
Table 2: Performance comparison using the WRN-28 28--208-0-- 10 features 44 and validation tasks used in the training phase of meta-learning.
iidicattss that both training
5-class minilmageNeet 1-shot (%)
5-class tieredImageNet 1-shot (%)
Method
Crop
5-shot (%)
5-shot (%)
MetaSGD 29
56.5840.21
68. 68.8440.19
59.75
69.0440.22
LEO
61.76...00
77.5910.
66.33 10.05
81. 10.40.09
center
MC 37
61.221.0.10
15.12.0.17
66. 66.2010.10
82.21 10.08
MCI
61.85400.10 10
77.02+..11
+0.
82.61 +0.08
MetaSGD + MetaNCoV
59.42+1.32
70.2440.73
60.36
75.08
MC MetaNCoV
center
63.40+1.80
76. 12.6.86
12.38
86.47 =0.56
LEO
[44
multiview
40.20
20.20
79.49
MC
37
64.40.1o.10
80.21
MC MetaNCoV
multiview
66.54.1.22
36.5211..54
basic Gaussian distribution into intricate multi-modal distributions in 2D. The expressiveness of Sylvester NCoVs and numerical comparison of NEs with NCoVs are postponed to Appendix D
4.2 Performance evaluation using real data
Next, the empirical performance of MetaNCoV is assessed on three real datasets for meta-learning.
The experimental setups follow from the standard M-class N --shot few-shot classification protocol 41 10]. In particular, Derm per task f consists of M randomly drawn classes, each containing N labeled data. The default sasc-spec model is standard 4-block convolutional NN (CNN) (55]. Each block of the CNN comprises 3 3 convolution layer, a batch normalization layer, ReLU activation, and a x max pooling layer. After the convolutional blocks, linear regressor with softmax activation appended to perform classification. Following the practices of 37 12], the number of convolutional channels is set to 128 to improve its fitting capacity. Additionally, to be consistent with Theorem 3.5 Sylvester Ns Vs are adopted in all the tests.
To illustrate the benefit of learning more expressive priors, the first test compares MetaNCoV with other meta-learning algorithms having different prespecified priors using the minilmageNet dataset [55]. As a plug-in prior model, our MetaNCoV can be readily integrated with other meta- learning methods that adopt different lask- optimizers. In this test, we implement MetaNCoV with MAML 10) and MetaSGD 29). The results are listed Table where the performance metric is the average classification accuracy with 95% confidence interval on new tasks. It is seen that our MetaNCoV outperforms all the competitors. This empirically confirms the superiority of data-driven priors over the prespecified pdfs, as well as the effectiveness of MetaNCoV in learning expressive
8

---

Table 3: Performance comparison of MetaNCoV against meta-learning and metric-learning methods on the CUB-20-2011 dataset. For fairness, the backbone model is 4-block CNN.
5-class CUB-200-2011
Method
Type
1-shot (%)
5-shot (%)
MatchingNet [55
metric-learning
45.30 11.0
59.50+1.01
MAML. [T]
meta-learning
58.13
70.30
ProtoNet [8
metric-learning
37.36=1.00
45. 45.28+1.03
RelationNet 44
metrie-learning
58.99
71.20=0.10
DN4 28
metric-learning
53.15=0084
81. 00.0000060 =0.60
MattML 62
meta-learning
66.29
0.56
80.34
MAML + MetaNCoV
69. 20.24411 1.36
80.41
meta-learning
MetaSGD + MetaNCoV
69.94 11
80.54
prior. Moreover, a remarkable performance gain can be observed on the 1-shot dataset. This justifies the claim that prior can be particularly informative when the training data are extremely scarce. For an apples-to-apples comparison, methods that use pre-trained feature extractors more complicated models (e.g., residual networks) are not included in this table. The compatibility of MetaNCoV to these models will be demonstrated in the subsequent tests.
The second test evaluates MetaNCoV on minilmageNet and mieredI feature embeddings extracted using a pre-traines Wide ResNet WoRec-RRN22 28-18 backbone [4]. Compared to the 4-block CNN. this model has a greater number of parameters and thus enhanced expressiveness. The results are summarized in Table 2 where MetaNCoV is implemented with MetaSGD (9) and MC (37]. In all tests, MetaNCOV brings about notable performance improvement compared the corresponding baselines. This validates MetaNCoV's effectiveness and flexibility as a plug-in prior module.
The last test assesses the performance of MetaNCoV the CUB-200-2011 dataset [57]. In contrast to the previous two datasets that contain nature images of distinct objects, this dataset specifically focuses on birds of various species. While the classification of nature objects primarily relies on low-level features such as shapes and colors, classifying various birds requires further recognition of high-level features including textures and segmentations. To learn these complicated features. the model needs to be either trained with sufficient data, or equipped with a powerful prior. Table 3 showcases the performances different meta- and metric-learning methods on such a dataset. Again, our MetaNCoV method is markedly effective on the 1-shot dataset where data are exceptionally limited. This highlights the significance of an expressive prior. For the 5-shot dataset where data are relatively abundant, its performance is also comparable to the state-of-the-art ones.
4.3 Ablation study
Next, ablation tests are conducted to ana- lyze the performance gain of MctaNCoV, The test is carried out on the minilmageNet dataset, with results gathered in Table 4 The first ablation investigates the impact of the advocated NCoVs over the injec- tive ones. To ensure the injectivity of the Sylvester NF f. we follow the QR param- eterization recommended see the improved performance of NCoV due to its enhanced expressiveness, which numerically verifies Theorem 3.1 and Remark 3.3. The second ablation examines the influence of nonlinear function in the Sylvester NCoVs. By changing the from sigmoid to the popular ReLU activa- tion, a degradation of empirical performance can be observed. This observation corroborates with Additional experiments and visualizations be found in Appendix D
Table 4: Ablation tests for MetaNCoV.
5-class minilmageNet
Ablation setup
1-shot (%)
5-shot (%)
NCoV (baseline) Injective NE
59.10 10
71.48 20.68
56.72+1.46
69.41 69.41+0.68 20.68
ReLU o
56.54+1.46
69.84+0.68
[5I) One can
Remark 3.8
4.4 Cross-domain generalization
This subsection showcases the generalization capacity of MetaNCoV in cross-domain few-shot learning. This test is more challenging compared to previous ones due to the domain gap between

---

Table 5: Performance comparison of MetaNCoV against meta-learning algorithms in cross-domain few-shot learning setups. The prior models are trained on minilmageNet and tested on three datasets.
5-class TicredImageNet
5-class CUB
5-class Cars
Method
1-shot (%)
5-shot (%)
1-shot (%)
5-shot (%)
1-shot (%)
5-shot (%)
MAML [10]
51.61+0.20
65. 75.767
40.51+0.08
53.09+0.1e
33.57+0.11 14
44.56+0:21
ANIL
38
52.8210.29
41.12:0.11 66.52.0.28
55.82
34.7710.31
46.55.0.29
BOIL
35
53.23::0.41
69.37
44.20::0.15
60.92:0.11
36.1210.29
50.64 50.6440.22
MparseMAM++ [56
53.91
69.9240.21
43.4341104
62.02
37.14
53. 53.184u.44
GAP 19
58.56+0.93
72.82+u.77
44.74+0.75
64.78
38.4440.77
55.0440.77
MetaNCoV
61.50+1.40 19
73. 13.10+0.77
47. 47.84+1.49
65.2770.78
41.66+1.48
57.19
the meta-training and meta-testing phases. By shifting the task domain, this test aims to assess the overfitting of the learned prior specific domain. Our test setup follows from [35], where the prior
model is meta-trai meta-trained on the minilmageNNt (55] dataset, and meta-tested on magredI [42].
Cars 23 and CUB [57] datasets. Our MetaNCoV is implemented with MetaSGD (29] in this test.
As shown in Table [ our method consistently outperforms popular meta-learning approaches in such
a setup, especially in the 1-shot case. This not only confirms the cross- cross-domain generalization of
MetaNCoV, but again justifies the importance of expressive prior when data are exceedingly limited.
5 Conclusions and outlook
An informative prior plays crucial role training large-scale model with limited scall-scale data. This work introduced novel NCoV model for learning an expressive task-invarian prior. By transforming a known pdf of a continuous random vector, the NCoV model enables a large family of target pdfs. As a flexible plug-in prior model, our MetaNCoV method offers enhanced prior expressiveness compared to existing meta-learning methods that rely on preselected prior pdfs. Numerical studies validate our theoretical analysis, and highlight the superior performance of the proposed method, especially when datasets are scarce. Our future research agenda includes i) investigation of more generic universal approximation theorems; ii) bilevel convergence analysis for the MetaNCoV method; and, iii) implementation of MetaNCoV with alternative transformations, backbone models, and meta-learning methods.
Acknowledgments
This work was supported by NSF grants 2220292, 2312547, 2212318, and 2126052.
References
[1] Sungyong Baik, Janghoon Choi, Heewon Kim, Dohee Cho, Jaesik Min, and Kyoung Mu Lee. Meta-learning with task-adaptive loss function for few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9465-9477 October 2021.
[2] Sungyong Baik, Scokil Hong, and Kyoung Mu Lec. Learning to forget for meta-lcarning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2020.
[3] Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In Proceedings of International Conference on Learning Representations, 2019.
[4] Wei-Yu Chen, Yen-Cheng Liu, 7solt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look few-shot classification. In International Conference on Learning Representations, 2019.
[5] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
[6] Laurent Dinh. David Krueger, and Yoshua Bengio. NICE: non-linear independent components estimation. In Workshop Track Proceedings of International Conference on Learning Representations, 2015.
[7] Laurent Dinh, Jascha Sohl-Dicksee Dickstein, and Samy Bengio. Density estimation using real NVP. In Proceedings of International Conference Learning Representations, 2017.
10

---

[8] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradient-based model-agnostic meta-learning algorithms. In Proceedings the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 26 28 Aug 2020.
[9] Alec Farid and Anirudha Majumdar. Generalization bounds for meta-learning via pac-bayes and uniform stability. In Advances in Neural Information Processing Systems, volume 34, pages 2173-2186, 2021.
[10] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning, volume 70, pages 1126- 1135, 06- 11 Aug 2017.
[11] Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model- meta-learning. In Advances in Neural Information Processing Systems, volume 31, 2018
[12] Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-lcarning with warped gradient descent. In International Conference on Learning Representations, 2020.
[13] Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel pro- gramming for hyperparameter optimization and meta-learning. In Proceedings of International conference machine learning, pages 1568 1577. PMLR, 2018.
[14] Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. MADE: masked autoencoder for distribution estimation. In Proceedings of the 32nd International Conference on Machine Learning. volume 37, pages 881-889, 07-09 Jul 2015.
[15] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
[16] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, volume 27, 2014.
[17] Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Mela- learning probabilistic inference for prediction. In Proceedings of International Conference on Learning Representations, 2019.
[18] Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning as hierarchical Bayes. Proceedings of International Conference on Learning Representations, 2018.
[19] Suhyun Kang, Duhun Hwang. Moonjung Eo, Taesup Kim, and Wonjong Rhee. Meta-learning with geometry adaptivv preconditioner. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10080-16090, June 2023.
[20] Thomas A Keller. Jorn W.T. Peters, Priyank Jaini, Emiel Hoogeboom, Patrick Forré, and Max Welling. Self normalizing flows. Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5378-5387 18-24 Jul 2021.
[21] Diederik Kingma and Max Welling. Auto-encodie encoding variational Bayes. In Proceedings of International Conference on Learning Representations. 2014.
[22] Zhifeng Kong and Kamalika Chaudhuri. The expressive power of class of normalizing flow models. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108. pages 3599- 3609, 26 28 Aug 2020.
[23] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei, 3d object representations for fine-grained categorization. In Proceedings of the IEEE International Conference Computer Vision Workshops, June 2013.
[24] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, volume 25, 2012.
[25] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differen- tiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patiern Recognition, June 2019.
[26] Yoonho Lee and Seungjin Choi. baddient-based meta- meta-learning with learned layerwise metric and subspace. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 2927 2936. 10-15 Jul 2018.
11

---

[27] Sergey Levine, Peter Pastor, Alex Krizhevsky. Julian Ibarz, and Deirdre Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of rubotics research, 37(4-5):421 2018.
[28] Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo. Revisiting local descriptor based image-to-class measure for few-shot learning. In Procedingss of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, June 2019.
[29] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning to learn quickly for few-shot learning. arXiv preprint
2017.
[30] Hongzhou Lin and Stefanie Jegelka. Resnet with one-neuron hidden layers is universal approximator In Advances in Neural Information Processing Systems, volume 31, 2018.
[31] Geert Litjens. Thijs Kooi, Babak Ehteshami Bejuordi, Amaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. A survey on deep lcarning in medical image analysis. Medical Image Analysis, 42:60-88, 88, 2017.
[32] Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: view from the width. In Advances in Neural Information Processing Systems, volume 30, 2017.
[33] Tsendsuren Munkhdalai and Hong Yu. Meta networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Muchine Learning Research, pages 2564-2563, 06-11 Aug 2017.
[34] Didrik Nielsen, Priyank Jaini, Emiel Hoogeboom, Ole Winther, and Max Welling. Survae flows: Surjections to bridge the gap between vaes and flows. In Advances in Neural Information Processing Systems, volume 33, pages
2020.
[35] Jaehoon Oh, Hyungjun Yoo, ChangHwan Kim, and Se-Young Yun. BOIL: Towards representation change for few-shot learning. In International Conference on Learning Reprexentutions, 2021.
[36] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. 22(1), jan 2021.
[37] Funbyung Park and Junier B Oliva. Meta-curvature. In Advances in Neural Information Processing Systems, volume 32, 2019.
[38] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of MAML. In Proceedings of International Conference on Learning Representations, 2020.
[39] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances Neural Information Processing Systems, volume 32, 2019.
[40] Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning. In International Conference on Learning Representations, 2019.
[41] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In Proceedings of International Conference on Learning Representations, 2017.
[42] Mengye Ren, Sachin Ravi, Eleni Triantafillou, Jake Snell, Kevin Swersky, Josh B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learnin for semi-supervised few-shot classification. Interna- tional Conference on Learning Representations, 2018
[43] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 1530-1538, 07-09 Jul 2015.
[44] Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals. Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-Iearnin with latent embedding optimization. In International Conference on Learning Representattinss 2019.
[45] Adam Santoro, Sergey Bartunov, Mattew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory- augmente neural networks. In Proceedings of The 33rd International Conference on Machine Learning. volume 48 of Proceedings of Machine Learning Research, pages 1842 -1850, New
York, New York. USA, 20 22 Jun 2016.
12

---

[46] Juergen Schmidhuber. A general method for incremental sell-improvement and multi-agent learning, pages 81-123, 1999.
[47] Jürgen Schmidhuher. A neural network that embeds its own meta-levels, In IEEE International Conference Neural Networks, pages 407-412, 1993.
[48] Jake Snell. Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
[49] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedingy of the IEEE Conference on Computer Vision and Puttern Recognition, June 2018.
[50] Hongduan Tian, Bo Liu, Xiao-Tong Yuan, and Qingshan Liu. Meta-learning with network pruning. In Computer Vision ECCV 2020. pages 675-700, Cham, 2020.
[51] Rianne den Berg, Leonard Hasenclever, Jakub Tomezak, and Max Welling. Sylvester normalizing flows variational inference. proceedings of the Conference on Uncertainty Artificial Intelligence, 2018.
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
[53]
Cédric Villani. Topics in optimal transportation, volume 58. American Mathematical Soc., 2021.
[54] Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
[55] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, volume 29, 2016.
[56] Johannes von Oswald, Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas Zucchet, and João Sacramento. Learning where learn: Gradient sparsity in meta and continual learning. In Advances in Neural Information Processing Systems, volume 34, pages 5250-5263, 2021.
[57] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Calttch-UUSSD 2irds-200-2011 dataset. Technical Report CNS-TR-201 1-001, California Institute of Technology, 2011.
[58] Lianzhe Wang. Shiji Zhou, Shanghang Zhang, Xu Chu, Heng Chang, and Wenwu Zhu. Improving generalization of meta-learning with inverted regularization at inner-level. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7826-7835, June 2023.
[59] Jaesik Yoon, Tacsup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic mela-learning. In Advances in Neural Information Processing Systems, volume 31, 2018.
[60] Xueting Zhang, Dehin Meng, Henry Gouk, and Timothy M. Hospedales. Shallow bayesian meta learning for real-world few-shot recognition. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 651-660, October 2021.
[61] Yilang Zhang, Bingcong Li, Shijian Gao, and Georgios B. Giannakis. Scalable bayesian meta-learning through generalized implicit gradients. In Proceedings the AAAI Conference on Artificial Intelligence, volume 37(9), pages 1298-11306 Jun. 2023.
[62] Yaohui Zhu, Chenlong Liu, and Shuqiang Jiang. Multi-attention meta learning for few-shot fine- fine-grained image recognition. In Proceedings of the Twenty-Nint International Joint Conference on Artificial Intelligence, ICCA1200, 2021.
13

---

A Proof of Theorem 3.1
Theorem A.1 (Multivariate PIT. restated). Let Pz: =R [0, be the caf of continuous random vector Z == 17,..., Za] with {Zz mutually independent. For any differentiable a.e. cdf Q Rdd + [0, 11. there exists a non-decreasing function S Rdd + Rd that the random vector Z' (Z) has cdf
Pz = Q (a.e.).
(12)
Proof. We claim that the i-th entry of transformation f adopts the form
f = gi o his I....,d
(13)
where 9i : R. + IR will be specified soon, and (Z) = (2), P2,)) The proof follows from the mathematical induction on d.
First consider the base case 1. By univariate probability transformation, f" Q Pz directly verifies Theorem 3.1 and the claim (33), where Q Q='(u) := inf(u Q(v) 2 u),
1]. What f*. Since Q and Pz are cdfs, they are non-deceeasing by
remains is the proof for the monotonicity definition. Using the monotonicity of Q. it holds that {u Q(v) 2 un S {o Q((v) 2 u2). Vu 2
U2. From the definition of Q-1, we have (wn) 2 (Q1(wa)) (w2). Vun 2 U2 meaning that Q
also non-decreasing. As a result, the composition f Q- a Pz is non-decreasing.
Subsequently, assuming Theorem 3.1 and the validity of claim [3 for d I,..., do. we establish them for d = do + 1. This induction-based argument gives the proof of Theorem 3.1 For notational compactness, define random variable U, == Pz, (Z:),
.......dd. do + and likewise random
UL. = [U,....,U U.] h; (Z). The univariate PIT indicates that each U; ~ [[0 1]. Besides, since do [Z) are mutually independent, it follows that (U:)
do are also mutually independent.
Let :RR + R denote the transformation provided by the inductive hypothesis for case d do- The first do entries of the desired /* (for case = do + 1) can be defined as fīxd, (Z) f(Z).d.) Z E 1d Thus, the inductive hypothesis suggest the joint cdf for Zind, = fi (Z) Re is Pzz (E) = Q(E )oo] & E Rddo (notice that Q(-) a function defined on Rd for
d= do 1, while Q(,, +00)) on (R.d)).
Next, it will be shown that (12) can be obtained upon defining få,+1 = Ido+1 o hdo+1 with
Ido+1 1do+1(U1:dHH (Urdo++) = Q7 Indo (Udo+1 [gi(U)),...... Ido (UI:do)1T)
inf {u Qds+1f1:do (v [en (U1), Ido (U (1st)) )1 [T) 2 Udot1}
min {ur Qdo+1ll:do (v (UI).... 9do (U (UId)))) ) 2 Udot1}
(14)
where conditional edf 1d (vds+ +1 dimensional random vector V obeying edf Q, and (a) is because the conditional edf is non-decreaiigg and right continuous so the infimum can be attained. First notice that the transformed random vector
Vi:de) Pr(V/ (o(Vo.+1 S Sdb+1 V 3 V1:a)) for (do 1)-
z' = fz) 191 o hi))), ... Ido+ hdg ( (Ui,, ... Ido+1 (U 1U1d
(15)
has edf
Pz (v) | PZ
Viudo
(E) P7 1 (od,+1 +1 E)dE, Ro++
(16)
where the equality is due to Pxy = PX) for random variables X, Y.
14

---

On one hand, it holds that
Pzig Biid (vag+1
{) Pr (Z S Wdo+1 L1:00 = E)
(0) Pr(gdo+1 (UI:do+1)= S Udo+1 gz(U1) :.. (11d)) = Edo)
(b) Pr (Q (Udo+1lE) S &dutt gi(U)) = E1..... Ido (U (Udd)) = Edu
Pr (Qdot (1d)+1lE) Vdo+1
Pr (Udo+1 111+1/1:d (vdo
d Qdo+ 111:d (Vdo+1lE)
(17)
where (a) is from
15
(b) uses
14 (c) follows from the mutual independency of (2 and
(d) is because Udo+1 (0, 1].
On the other hand, it has already been shown using the inductive hypothesis that, the random vector Zi.do = di (Z) has edf Pzisen (E) = Q(E +,o]T), & E R Since Q is differentiable a.e., we have the corresponding pdf Po
(E) = SR a(L5 m) )dn == Qi (E) a.c. As a result, it follows
from 16 and (1 that
(z(v)= / Vi:do
@1:do (E) 1| (Mdy 1E)dE Q(v) (a.e.
(18)
where we use (xy = Px)) (x)E()) again. It should be noted that the only type of discontinuities for a monotone function is the jump discontinuity and there are at most countably many of them. Consequently, Pz' may fail match Q only on a set of measure zero.
Finally, we will prove the monotonicity of this constructed f by showing that Jy- ż (dx 1)- First notice from (14 that Ido+1 (d.+1(U1:dot (1::++1) is a conditional edf non-decreasing w.r.t. Ud By the definition of hdg+1, we have ([hd)+1(2]]ann 0Zag+1
P2 (Zoo+1) 2 O because Pzzg 11
8fig (2)) 0gin 1(hha (1) 11hhg111 (2))
cdf. As a result, applying the chain rule leads to
87
alhio 12/dg
87
B0dg (0dg++1Usss (1)d+++) B[Mag Ouag+1 (1
0. Additionally, the inductive hypothesis implies f is non-
0Zao+1
decreasing on Rd, that Jj = Odgx To the end, fz)
T. (Z)] has
block triangular Jacobian
(z)
852p+1
(Z)
871:dg
asāg+1 (2)
To
02
012
It has been shown that diagonal blocks (Z) > Odg xdo and which completes the proof.
2 0. Thus, it follows that
82d6+1
J
B Proof of Theorem
3,5
To aid the proof of Theorem 3.5 the following lemma offers an alternative expression for 4
Lemma B.I. Consider the notational conventions of Theorem 3.1 Let E C Rdd be the set on which f is injective and right-contimuuus. Then, it holds a.e. that
Pz = Q (z), V E.
(19)
Proof. With in effect, holds a.e. that
(z(z) Pr(Z 2)
(()) (z()))=))((()))))- P (()
(20)
where (a) is because f is injective thus increasing and right-continuous E. This proves 19
The next lemma suggests the edf of tail-convergent random vector can be approximated by truncating its pdf on sufficiently large compact set.
15

---

Lemma B.2. For any tail-convrreett random vector with cdf P: : [0, 11, and Ve > 0, there exists a cdf P. : Red - [0, 1] for which the pdf p vanishes outside compact set E C Rd and
[P(v) P(v) <,, Vv ER..
(21)
Proof. For a tail-convergent random vector, Definition 3.4 suggests that for Ve > 0, there exists bounded E' C IRd such that fgst.r g < e/2. Taking E = clEE to be the closure, it follows from the definition of closure that E is compact and
(a)
P
p <<<// 6/2.
(22)
ROVE
where (a) is due to E' - E. Now define
p/(1 FRU, on E'
p
(23)
0.
otherwise
Notice that Jaa SED-
SEmEEP) 1, which verifies p is a valid pdf. Thus, the SE 1/
induced cdf
P(v)= JISIESV) ((E))L.
(24)
It then follows for Vv E IRd that
IP(v) - P(v)l (P(v)-((v)l-
/EEEES)) P 1-E(E)dEE
S VEIIESSOAR p(E) D(E)dE + + P(E) F(E)
SUUVE P(E)DE
SRAVE P(E)DE I (EIESWINE P(E) JSSESSNE P(E)
SENE P(E)dE
1 JEMNE P(E)dE / P(E)dE +
I JRNN
PlE)dE
P(E)dE + JAN P(E)dE
(25)
RAVE
(
S c/2 + 4/2
(26)
where (a) uses 23 and (b) from 22p.
0
Next, the classic universal approximation theorem will be generalized to suit for the case of NCoVs.
Definition B.3 (5). A function
IR. R is said be sigmoidal if
{,. 0.
1, as + +00 as
a -
(27)
Definition B.4 (55, generalized), Let E be compact set with positive Borel measure. A function a IR + said to be discriminatory on E if for a finite signed regular Borel measure μ,, it holds that
/ (b z c)dp(z) = E
(28)
for all b E R and E implies f = 0. Theorem B.5 (15 Theorem 1]). Let be bounded measurable sigmoidal function. Then finite sum of the form
N
G(z) = Σ
(b) z + c,
(29)
j=1
is dense in C[(0,11*).
16

---

Corollary B.6. Let a be bounded measurable sigmoidal function, and E C. R locally compact set of finite Borel measure. Then finite sum of the form
N
G(z) = Σ (b) 2 (b)(()2+c))
(30)
==1
dense in C(F)..
Proof. When μ (E) = 0, one can take Eg E and the Corollary holds trivially. Next we consider the case p(E)= 0.
The original proof TheoremmB..]relle on Lebesgue Bounded Convergence Theorem, Hahn-Banach theorem, and Riesz Representation Theorem. All these four theorems hold for locally compact set E with finite Borel measure.
The proof of Corollary B.6 follows by i) generalizing the definition of discriminatory in to definition B.4, and ii) replacing [0, 1j the proof of Theorem B.5 with E.
Building upon Lemma B.1 Lemma follows.
B.2 and Corollary B.6 the proof of Theorem 3.5 is provided as
Theorem B.7 (Universal approximation via Sylvester NCoVs, restated). Let Pz denote the cdf of tail-convergent continuous random vector Z E Rd with mutually independent entries, and Q Lipschitz edf of tail-convergent random vector. For any > 0, there exists cdfs P. Q for which the pdfs P. q vanishes outside compact sets Ey: Egg and
Pz(v)- - P(v)) < (Qz()V) <,, VV
(31)
Moreover, let E S E, be any set on which the transform f matching Pz to Q (ef. Theorem 3.1 injective and right-continuous. There exists a Sylvester NCoV f and a zero-measure set Eo, such that
Is(Z) (z)) <6, VZ Ep
Fa.
(32a)
P2(()) - (0)(2 <,, Vz E E En.
(32b)
Proof. Since Z is tail-convergent, Lemma B.2 suggests that there exists edf Pz for which the pdf fz vanishes outside compact Ep TR. and (Z)(z) Pz(z)] c/4, Vz Rd Similarly, there also a edf Q for which ğ supported compact set Eq. and (Q(z) Q()) e/4, Vz Rd.
Moreover, let LQ be the Lipschitz constant of Q. Then, 22 Lipschitz with constant
23
and 24 indicates Q is also
J
Lo/(1
(33)
J ER
LQ - e/8
Using Rademacher theorem, we have Q differentiable a.e.. Then, let f Rd + Rd denote the optimal transform by Theorem 3.1 which matches P to Q. Lemma B. suggests that
Pz(z) = f= Vz E.EE. E.
(34)
Let Ž and Ż' = f() be random vectors obeying cdfs Pz and Q. Since pz is supported on Ep, implies that f can have arbitrary value outside Ep, which will not change pz'
We assert that f bounded on Ep. Otherwise, for any > O, there must be some zo E Ep such that [l B, and using (5) that
(z))) - (5* (z(2))(5 (z)]dz
2 / (z((2)(zz z/dz
pz(Z0) >
(35)
17

---

where the inequality is because f is non-decreasing. Since B can be arbitrarily large, this contradicts with the fact that supp(q) Eq is compact (cf. Lemma B.2).
Moreover, the monotonicity of f indicates the only possible discontinities of it must be jump discontinuities, and there are most countably many of them. Let Fo Eep be the where f is discontinuous, From the countability of Eo we have ((b))) = O. Thus, ft (z) is bounded and continuous on Ex Es, where fi and 2; are the i-th entries of f and z.
Then, applying Corollary B.6 implies that there exists a G2(z) form 30 such that
- -G: - =zill c(1-e/8) 2To
Vz E Ep Eo.
(36)
Now, define fz(z) = G.(z) + 2,, = I.....d on R.. One can easily verify from 6 that such a
definition renders f Sylvester NCoV, and (9a) holds. Moreover, it follows for Vz F.p Eg that
Pz(z)) Q f(z)) < Pz(z) - f(z)) (2((2) - f(z) - Q fz)
P2((z) f())) c/4+ c/4
(a) f"()) - Q f(x)) + e/2
( Lo
- 1 c/8
[ff()) - f(2)ll2 c/2
(6)
3 6/2 c/2=6
(37)
where (a) follows from 34 (b) is due to 33 and (c) uses (36. The proof is thus completed.
C Detailed setups of numerical tests
Our codes are run server equipped with an Intel Core 17-12700 CPU, and an NVIDIA RTX A5000 GPU. The experimental setups adopted in this paper will be next elaborated.
C.1
Toy tests
The numerical tests for 2D toy examples demonstrated in Figure aare are carried over by training Sylvester NCoV with width m 50. We used SGD optimizer with learning rate of 10-3 and momentum of 0.9. The ground truth samples used to train the model were generated by transforming 2D Gaussian random vectors Z -O200 I2x2) through non-injective ground truth transformation (Z) = (B sin(Z) + c), where sin(.) function is applied element-wise to each dimension of vector Z separately. A set of i.i.d. samples {z,, f (z, (zz)). fa=1 was randomly generated and used to train the Sylvester NCoV model. The results presented in Figure I were obtained using three different settings of ground truth (-). In each of these settings, the elements of the underlying matrices A, B, and the vector were generated from Gaussian distributions with zero mean and unit variance. The result for 1D case presented in Figure was obtained using a smaller Sylvester NCoV with width m 3, trained using SGD with learning rate of 10-2 and momentum of 0.6, and the histogram of the generated samples was normalized to represent pdf. To generate training samples, the probability integral transform (PIT). Specifically, we first draw ground truth Z' from mixture of Gaussians, denoted as Z' ~ q(2'), where q := Σ N of). Here, μ1 = -10 and μ22 = 10, with of = o3 = 1. Then, we rely on the inverse transformation f*=1(Z") to find its paired Z, where (Z) (Q (z) is the ground truth transformation obtained via PIT. Having find this mapping, we draw a set of i.i.d. samples (2,, (*x to train Sylvester NCoV model in ID.
employed
C.2 Few-shot classifications
brief description of the three benchmark datasets used in our experiments provided next. MinilmageNet (55] contains 60, 000 images sampled from the full ImageNet (II.SVRC-12) dataset, which are divided into 100 classes, each with 600 instances. All images cropped and resized into
18

---

Table 6: Hyperparameter setups.
Hyperparameter
Notation
Value
Task-level iterations
K
Task-level learning (ConvNeet-4)
a
10-2
Task-level learning rate (WRN-28-10)
a
Meta-level iterations
60, 000
Meta-level learning rate
10-3
Meta-level SGD batch size
84 x 84 pixels. In the experiments, we adopt the dataset split suggested by 41 where 64, 16 and 20 disjoint classes can accessed during the training, validation, and testing phases meta-learning.
Tageed 42 a larger subset of the ImageNet dataset, composed 779, 165 images from 608 classes. Likewise, all the images preprocessed to have size 84 84. Instead of using random split, classes are partitioned into 34 categories according to the hierarchy of ImageNet dataset. Each category contains 10 30 classes. These categories are further grouped into different 20 for training, for validation, and 8 for testing.
CUB-200-222 2011000-2011 [57] is extended version of the Caltech-UCSI Birds(CUBB)-o -00 dataset, which consists of 11, 788 line-grained images from 200 bird species. The dataset split follows from [4], dividing the species into 100 training, 50 validation, and 50 testing classes. Similar to the preceding two datasets, the images are also resized to 84 x 84.
The hyperparameters used for the few-shot classification experiments are the same as those in MAML [0), which are listed Table To enhance the statbility of the training process, we use SGD with Nesterov momentum instead of Adam as the optimizer for [10a). The width of Sylvester NCoV is determined through a grid search using the validation tasks. For minilmageNNe dataset with 4-block CNN model,
10 in the 1-shot experiment and minilmageNet and ticredImageNet with WRN-28- 10 embeddings, m fixed to be 10 under both in all the tests.
5 in the 5-shot one. For
center and multi-view crops. For the CUB dataset, we
D Additional experiments
D.1
NCoVs in ID
Here, demonstrate the efficacy of Sylvester NCoVs in approximating mixture of Gaussians in one-dimensional (1D) scenario. The primary objective is to transform ID Gaussian random variables, denoted Z ~ (O, 1), into mixture of Gaussians using trained Sylvester NCoV. As depicted in
0.35
Ground truth Mixture of Gaussians histogram generated samples
0.25
1500
counts
pdf
1250
0.15
300
0.10
252
0.05
Z
0.00
-20
-15
-10
-5
10
15
20
Sylvester NCoV Z' fZ)
Z'
Figure 2: Transforming 1D Gaussian random variable ~ NN(0 1) to a mixture of Gaussians
Z' = F(Z) using Sylvester NCoV f(-).
19

---

Estimated pdt' via Sylvester NCoVs, Z' f())
3004
308
Estimated pdfs via Injective Sylvester Flow, Z' = f()
Mieriiee Salemdte
kiw Subember Now
pdfs of interest, (())
8.08
845
340
8418
18)
(b)
(6)
Figure 3: Comparison of NFs and NCoVs in learning 2D toy pdfs.
Figure 2 (left), we illustrate the histogram of the original random variable Z and the estimated pdf of the transformed random variables Z' = f(Z) ) on the right-hand side. The dashed red curve represents the ground truth mixture of Gaussians that we seek estimate. This experiment demonstrates the ability of Sylvester NCoVs to effectively transform a basic Gaussian random variable in ID to the
desired mixture of Gaussians Z' pz) = E.m [ (2), where µ 100 and μ22 10,
with of o² 1.
D.2 Comparison of NEs and NCoVs using toy data
To demonstrate the expressiveness of NCoVs compared an injective Sylvester NE, conducted 2D toy test showcased in Figure 3 We have employed the QR factorization technique described in [5I] to enforce the injectivity constraint for Sylvester NF, which requires m S Here, m the width of the NE, and d of Sylvester NF, m thus chosen as 2. For fair comparison, our NCoV is implemented with the same setup. The three rows in the figure display respectively the learned NCoVs, injective NFs, and the target pdf q. It is seen that NCoVs are more expressive than valid NFs when learning complex distributions.
is the dimension of for this 2D example. To ensure the injectivity
To illustrate the ability of NCoVs in learning pdfs whose support full in Rd we consider checkerboard pdf. The test has been visualized in Figure where we employ an injective Sylvester NF and non-injective Sylvester NCoV learn the target distribution. our test, we stacked 40 base injective Sylvester NFs, and compared the results with those obtained using Sylvester NCoV with same number parameters and the same training setup. observed that NCoV demonstrates remarkable performance in learning the checkerboard pdf compared with that of the injective NE. This observation underscores the expressiveness of NCoVs effectively learning particular distributions, especially in scenarios involving discrete distributions those with incomplete support.
20

---

Table 7: Performance comparison using 4-block CNN backbone with different number of channels.
5-class minilmageNet 1-shot (%)
Method
Channels per block
5-shot (%)
MAML 10
32-32-32-32 32-32-32-32
48.70
63. 63.11+0.92
MetaSGD 29
50.47+1.87
64.03+0.94
Meta-LSTM 41
64-64-64-64
43.44...77
60.60
MAML [10
64-64-64-64
49.5+1.8
R2D2 3
64-64-64-64
49.5+0.2
65.4
MAML L2F
64-64-64-64
52.10=0.50
69.38
Minimax-MAN
158
64-64-64-64
51.70=0.12
68.41+1.29
MAML + MetaNCoV (ours)
64-64-64-64
55.86+1.49
68.90+0.71
MetaSGD MetaNCoV (ours)
64-64-64-64
57.44+1.48
69.15
R2D2 3
96-192-384-512
51.8+0.2
68.4
MC [37]
128-128-128-128
54.08=0.93
60.99
Warp-MAML 12
128-128-128-128
52.3
68. 68.44
MAML MetaNCoV (ours)
128-128-128-128
57.74
70.72
MetaSGD + MetaNCoV (ours)
188-128-12--123
59.10+1.52
71.48+0.68
D.3 Influence of backbones
Next, we show the performance comparison using the 4-block CNN backbone with different number of channels. All the hyperparameters are the same with Appendix C and the results are gathered in Table One can observe tht reducing the number of channels from 128 to 64 leads to slightly decreased accuracies, yet still superior than all the state-of-the--e art competitors thanks to the improved expressive power of NCoV.
D.4 Complexity and scalability
To demonstrate the scalability of MetaNCoV, this test measures numerically the time and space complexities of MetaNCoV and several popular meta-learning methods during the meta-training phase. The test is conducted on the minilmageNeet dataset, and the comparison is conducted using the 4-block CNN with 64-channel and 128-channel setups, respectively. The experimental results comparable to
are summarized the Table X It is observed that MetaNCoV has dimension MetaSGD, yet notably smaller than MC. One can also see that the increase of D only brings about a marginal growth in both time and space consumption. This is because the key factor affecting the complexity is the Hessian-vector product (HVP) computations when backpropagating the meta- gradient V LoL(d, (0); ) Since all the methods relied K-step GD to obtain (0) (where MC adopted an additional Kronecker-factorizze preconditioner (37]), the complexity for computing this meta-gran eeta-gradient is ((Kd) in both time and space.
377
pdd
Non-irjective Sylvester NCoV
Injective Sylvester NF
Figure 4: Learning a 4 x checkerboard pdf with NFs and NCoVs.
21

---

Table 8: Time and Space complexity comparison.
64-channel (d 193)
MAML
MetaSGI
MC
MAML + MetaNCoV
Dimension D of 0
121, 093
242, 186
2, 710, 896
605, 465
Time (Relative) GPU space (MB)
x1
x1.022 1, 704
x1.189
x1.199
1, 700
1,
724
128-channel (d = 365)
MAML
MetaSGD
MC
MAML + MetaNCoV
Dimension D of 0
463,363
926, 730
10,8 9,,19,952 x2.023
2,316, 825
Time (Relative)
x1.806
x1.828
x2.028
GPU space (MB)
2, 982
3, 002
3,052
3, 040
E
Additional related works and comparisons
Normalizing flows (NFs). Our NCoV is similar conventional NFs in the sense that they are both distribution models relying on the change-of-variide formula. Nevertheless, NCoV is distinct from NF in four key aspects: expressiveness, tractability, training strategy, and application fields.
i) First, NCoV has been proved capable of modeling an arbitrary pdf, whereas NF is restricted to pdfs with full support.
ii) Second, while NFs aim for closed-form pdf (3), NCoV sacrifices its tractability for augmented expressiveness.
iii Third, NFs typically trained in an unsupervised manner by maximizing the likelihood function Pς(z) over a dataset sampled from target pdf q. In contrast, NCoV requires latent optimization (10) given the intractability of likelihood.
iv) Finally, NFs can be thus utilized in applications including probability estimation and sample generation, while NCoV is tailored specifically to learn a prior over model parameters in the meta-learning context.
Non-injective transformations. Moreover, it is worth mentioning that non-injective transformations have been also considered by (34] to enhance the expressiveness. While (34) allows Z and Z' to have different dimensions, it does not provides theoretical guarantees of its expressiveness. In contrast, the design of our MetaNCoV is based on Theorem 3.1 that requires / being mapping from IR to IRd
In addition, other works such as [20] also forgo the analytical invertibility, but typically necessitate an efficient approximation of f-!
Bayesian meta-learning, Our approach is also related to Bayesian meta-learning 60
59
I1I. While these approaches aim to quantify the uncertainties in Φ, by identifying its posterior p(oly!'" X 0) p(y! Φ X )p((, 0), our MetaNCoV is deterministic by formulating the task-level optimization (10b) as a MAP problem. In addition, these methods also rely on tractable prior and (surrogate) posterior pdfs of prefixed forms such as Gaussian. For example, (60) relies on predefined conjugate prior over the extracted features (rather than parameters Φ,) to ensure the tractability of the feature posterior. In 40 Section 3.3], the surrogate (variational) posterior is prespecified as a diagonal Gaussian distribution, while the prior is fixed to be the Gaussian -ammaa form. For [59]. particle sampling with SVGD utilized parameterize and optimize the prior, where RBF kernels (i.c., Gaussian kernels) are selected to interpolate the particles. As also noted in [I] Section 4.2], both the prior and surrogate posterior distributions are predefined as diagonal Gaussian forms. In contrast, our approach forgoes the tractability for enhanced prior expressiveness.
Optimal transport. Lastly, the introduced NCoV model is related to the optimal transport problem in statistics (54), which aims to minimize the total cost of transporting one distribution to another. However, the existence of the transport maps f can impose strong assumptions on both the source and target distributions, and is closely connected to the choice of cost function. For instance, the well-known Brenier's 's theorem suggests that ff exists when Z and Z' have finite second moments (or. q has a compact support), and Z assigns no mass to any set of Hausdorff dimension d - 1 [53]. In comparison. Theorem 3.1 puts constraints only on the source random vector Z. as we can choose Pz flexibly to suit our needs. This enables Pz to match an arbitrary target Q.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims have been proved both theoretically in Section B and numeri- cally in Section which are the key contributions of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations have been discussed in Remark 3.3
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification:
All the assumptions are clearly presented in the theorem statements, and the corresponding proof' are provided in Appendices and B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our methodology is clearly stated in Algorithm 1 Codes and scripts for reproducing our results are also uploaded, with instructions incorporated.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All the datasets used in the experiments are available online, for which proper citations and descriptions have been provided in Appendix [ Our codes and scripts used in the experiments are also uploaded as the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/mipss.cc/pulli/guidiee htt///CooSSSeiiaaeeeeeee swssssssiinyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All detailed experimental setups including hyperparameters and optimizers included in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: 95% confidence interval of the averaged classification accuracy has been adopted as the uncertainty metric in all the tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The server (including CPU and GPU information) used in our experiments is specified in Appendix C while the time and space (memory) complexities have been
provided in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpg//ggg///////oooo...///////p hctpp//blcccttttcccccc..........
Answer: [Yes]
Justification: This research project conforms every aspect of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work mainly focuses on algorithm design and its theoretical guarantees.
There is no malicious societal impact direct path to any negative applications.
Guidelines:
26

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The data and model used in this paper are from safe public databases available online.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The original owners the assets used this paper have been properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
27

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: A "readme" document is included in the supplementary submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
There is no crowdsourcing nor human subjects in this research projection.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There crowdsourcing nor human subjects in this research projection, and thus no necessity for IRB approvals.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "XQKflgctKQ", "reviewer": "Reviewer_VcV4", "summary": "This paper proposes to use non-injective change of variables theorem to make the prior in meta-learning more flexible than fixed shaped ones that have previously been used. Abundent theoretical analysis and experimental results support their claim that more flexible meta-level prior pdf can significantly boost the performance of meta-learning in a few simple benchmark settings.", "questions": "I wonder about the positioning of this paper. Does the proposed method have to be specifically tailored to meta-learning only? Why don't you broaden the scope of the paper by doing experiments on more standard non meta-learning setup (like in the synthetic experiments)?", "limitations": "The authors partially addressed the limitations of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to understand\n- Theoretical analysis is done rigorously\n- Experimental results are strong (but limited to too outdated benchmarks)", "weaknesses": "Too limited benchmarks. While the idea and the theoretical analysis are nice, the benchmarks used in this paper -- mini-Imagenet, tieredImageNet, and CUB are too outdated in my opinion. They were popular in 5 years ago, and nowadays people are actually no longer interested in these benchmarks, especially in the LLM era. I think the minimum level should be meta-dataset. Could you provide the results on meta-dataset?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the interest in this work and the constructive feedback provided, which have been carefully addressed as follows. \n\n**Response to weaknesses**\n\nDue to limited time and computational resources, we are unfortunately unable to provide the results on the full MetaDataset that contains 240GB data. However, it is feasible to conduct a similar experiment on a smaller scale.\n\nBefore providing the details, we would like to emphasize that MetaDataset aims to assess the *cross-domain generalization* performance of meta-learning algorithms, which comprises 10 datasets from various domains, including ImageNet, Aircraft, and CUB. The standard setup involves meta-training a model on ImageNet (or the entire MetaDataset), and then meta-testing the trained model on all datasets. \n\nFollowing the cross-domain few-shot learning setup in [1], we meta-train our prior model on miniImageNet, and meta-test it on tieredImageNet, Cars, and CUB datasets. As miniImageNet is a subset of the full ImageNet, we believe this test to some extent reflects the promising cross-domain generalization performance of our proposed algorithm. As shown in the Table below, our method consistently outperforms popular meta-learning approaches in such a setup, especially in the 1-shot case. This not only confirms the cross-domain generalization of MetaNCoV, but again justifies the importance of expressive prior when data are exceedingly limited. \n\n|Method|5-way|TieredImageNet|5-way|CUB|5-way|Cars|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n| |1-shot|5-shot|1-shot|5-shot|1-shot|5-shot|\n|MAML [2]|$51.61_{\\pm 0.20}$|$65.76_{\\pm 0.27}$|$40.51_{\\pm 0.08}$|$53.09_{\\pm 0.16}$|$33.57_{\\pm 0.14}$|$44.56_{\\pm 0.21}$|\n|ANIL [1]|$52.82_{\\pm 0.29}$|$66.52_{\\pm 0.28}$|$41.12_{\\pm 0.15}$|$55.82_{\\pm 0.21}$|$34.77_{\\pm 0.31}$|$46.55_{\\pm 0.29}$|\n|BOIL [3]|$53.23_{\\pm 0.41}$|$69.37_{\\pm 0.23}$|$44.20_{\\pm 0.15}$|$60.92_{\\pm 0.11}$|$36.12_{\\pm 0.29}$|$50.64_{\\pm 0.22}$|\n|Sparse-MAML+ [4]|$53.91_{\\pm 0.67}$|$69.92_{\\pm 0.21}$|$43.43_{\\pm 1.04}$|$62.02_{\\pm 0.78}$|$37.14_{\\pm 0.77}$|$53.18_{\\pm 0.44}$|\n|GAP [5]|$58.56_{\\pm 0.93}$|$72.82_{\\pm 0.77}$|$44.74_{\\pm 0.75}$|$64.88_{\\pm 0.72}$|$38.44_{\\pm 0.77}$|$55.04_{\\pm 0.77}$|\nMetaNCoV (ours)|$\\mathbf{61.50}_{\\pm 1.49}$|$\\mathbf{73.10}_{\\pm 0.74}$|$\\mathbf{47.84}_{\\pm 1.49}$|$\\mathbf{65.27}_{\\pm 0.73}$|$\\mathbf{41.66}_{\\pm 1.48}$|$\\mathbf{57.19}_{\\pm 0.75}$|\n\n[1] A. Raghu, M. Raghu, S. Bengio, and O. Vinyals, \"Rapid learning or feature reuse? towards understanding the effectiveness of MAML,\" ICLR 2020.\n\n[2] C. Finn, P. Abbeel, and S. Levine, \"Model-agnostic meta-learning for fast adaptation of deep networks,\" ICML 2017.\n\n[3] J. Oh, H. Yoo, C. Kim, and S.-Y. Yun, \"Boil: Towards representation change for few-shot learning,\" ICLR 2021.\n\n[4] J. von Oswald, D. Zhao, S. Kobayashi, S. Schug, M. Caccia, N. Zucchet, and J. a. Sacramento, \"Learning where to learn: Gradient sparsity in meta and continual learning,\" NeurIPS 2021.\n\n[5] S. Kang, D. Hwang, M. Eo, T. Kim, and W. Rhee, \"Meta-learning with a geometry-adaptive preconditioner,\" CVPR, 2023.\n\n**Response to questions**\n\nWhile our idea has the potential to be broadened beyond meta-learning, we must emphasize that our current setup is specifically tailored to meta-learning, which does not require a tractable pdf, but rather demands enhanced prior expressiveness. We should also highlight to the reviewer that the *intractability* of pdf prohibits learning NCoV via conventional approaches such as maximum likelihood training and evidence lower-bound maximization -- this thus necessitates careful attention and extra certain designs when applying our idea to other domains.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aZ0YyulxDn", "reviewer": "Reviewer_4d6S", "summary": "One of representative optimization-based meta-learning method is Model-agnostic meta-learning (MAML), where the inner loop can be interpreted as solving MAP. Here, the prior distribution over model parameters is defined as Gaussian pdf and the shared initialization is the mean parameter of the Gaussian pdf. The choice of Gaussian distribution for prior distribution may lack expressiveness; therefore, this paper proposes a new meta-learning method by introducing non-injective change-of-variable (NCoV) model for prior distribution over task parameters. Furthermore, the proposed method does not need to meta-learn shared initialization as done in MAML by feed-forwarding the zero vector into the normalizing flow (choosing the base distribution of normalizing flow as standard Gaussian). The empirical results shows the efficacy of proposed method in few-shot learning settings.", "questions": "Please see weakness section.", "limitations": "This paper discuss limitation in the conclusion section.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper propose a principled meta-learning method to improve expressiveness of prior distribution over model parameters, by theoritically derived method for non-injective change-of-variable models.\n\n- The proposed method is more efficient when the dimensionality of parameter increases, while the pre-defined prior distribution baselines is not scalable to the dimensionality of parameters. \n\n- The empiricial results are very strong.", "weaknesses": "- This paper does not provide the number of \"meta-\"parameters for each method in the experiment section. I wonder whether the improvement comes from the expressiveness of proposed method or the sufficient number of parameters $\\theta_f$ for modeling normalizing flow.\n\n- This paper does not discuss relevant literature in main paper (does not include related work section). Especially, I am curious about the novelty of NCoV models derived in this paper in comparison to the literature of normalizing flows. If NCoV models have already been studied in the literature, this paper seems to lack novelty. If not, I would like to recommend the authors to include related work or discussion to highlight their contribution with respect to normalizing flows.\n\n- This paper follows experimental setups for the convention of meta-learning, but the empircial validation is conducted in small-scale settings. This paper argue that the proposed method is more scalable to dimensionality of model parameters, therefore, it would be better to highlight the argument by conducting experiments on more larger models (e.g., VIT).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable feedback provided. The questions raised are addressed one-by-one below. \n\n**Regarding weaknesses 1 and 2**\n\nWe would like to kindly bring to the reviewer's attention that the model complexity with meta-parameter dimension $D$ has been already studied extensively in Appendix D.4; additionally, related works have been provided in Appendix E of the original submission. Moreover, we have referred to them in lines 327 and 193 of the main paper to guide the readers to relevant Appendices. As evident from Table 6 in Appendix D.4, parameters of our MetaNCoV is notably less than MC, and its time- and space-complexity is comparable to popular optimization-based approaches. Furthermore, while NFs are thoroughly studied, the NCoV models have gained little attention in machine learning and statistical learning due to their intractability; see lines 701-706 in Appendix E for related works and corresponding comparisons. \n\n\n**Regarding weakness 3**\n\nOur main claim in this work is that MetaNCoV empowers more *expressive* (rather than *scalable*) prior in high-dimensional spaces; cf. lines 12, 64, 176, and 188. This argument has been also empirically confirmed in our numerical tests in Section 4 and Appendix D. In terms of scalability, MetaNCoV is comparable to popular optimization-based approaches such as MAML and MC; see Table 6. However, it is still challenging to directly apply these algorithms (even vanilla MAML) to extremely large-scale models such as vision transformers. Potential solutions include restricting the task-level update (1b) to only a *subset* of parameters, or resorting to corresponding first-order variants such as FO-MAML.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XzVxq3ocvu", "reviewer": "Reviewer_BS6h", "summary": "This paper proposes a novel approach to meta-learning using normalizing flows for modelling the prior distribution of parameters, with the task solver in the inner loop as the maximum a posterior estimator. Compared with the previous methods, such as MAML, which assume a Gaussian prior, suffering from limited expressiveness, non-injective Sylvester normalizing flows (NFs) are utilised for more flexibility. The method involves propagating samples for each task through the flow and updating the flow based on an average overall task. Experimental results show that this approach consistently outperforms some existing meta-learning techniques.", "questions": "See the weaknesses section", "limitations": "See the weaknesses section", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper is well-written and easy to follow. The motivation for modelling the meta-prior with a very flexible distribution is valid. \n2. The background of meta-learning in the few-shot learning is well introduced.", "weaknesses": "1. I find the non-injective Sylvester Flows part confusing. In traditional Sylvester Flow models, a specific structure is used to ensure invertibility by imposing conditions on the diagonals of triangular matrices. But in this work, I do not see it. \n\n2. In line 256, the authors claim that while optimization Eq.10b, the latent variables share similarities with LEO, which is confusing. The latent variable in LEO is introduced by its specific architecture and in this submission it refers very different thing. Can the authors explain more about this? \n\n3. [1] handles the prior learning problem for meta-learning from a similar perspective and outperforms MetaNCoV in some settings. \n\n[1] Zhang X, Meng D, Gouk H, Hospedales TM. Shallow bayesian meta learning for real-world few-shot recognition. In Proceedings of the IEEE/CVF international conference on computer vision 2021 (pp. 651-660).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing the insightful suggestions. The concerns are addressed one-by-one as follows. \n\n**Regarding Weakness 1**\n\nOur Sylvester NCoV *intentionally* forgoes the injectivity constraint to enhance prior expressiveness, as illustrated in Theorem 3.1. Upon waiving the injectivity, the function $f$ is no longer bijective and thus can be non-invertible. Consequently, the Jacobian $J_f$ needs not to be a positive definite triangular matrix. \n\n**Regarding Weakness 2**\n\nThe claim in line 256 aims to illustrate that both MetaNCoV and LEO optimize a latent variable instead of the primal $\\boldsymbol{\\phi}_t$. We agree with the reviewer that our latent variable is designed from a *different* perspective. We will rewrite this sentence to emphasize this difference and avoid potential confusion. \n\n**Regarding Weakness 3**\n\nOur method distinguishes from [1] in three key aspects: \n- Prior to learn. The proposed MetaNCoV aims to learn the prior $p(\\boldsymbol{\\phi}_t)$ over *model parameters* $\\boldsymbol{\\phi}_t$, while [1] propose to learn the prior $p(g(\\mathbf{x}_t^n))$ over the extracted *features* $g(\\mathbf{x}_t^n)$. Additionally, [1] relies on *preselected* prior form (i.e., conjugate prior) to ensure tractability, whereas our goal is to meta-learn a *data-driven* prior of *universal* forms. \n- Task-level optimization. This work belongs to the optimization-based meta-learning, which finetunes the entire nonlinear model during task-level optimization. In contrast, [1] is a metric-based approach that freezes the feature extractor and adapts merely the last linear layer of the model. \n- Randomness. Our approach is deterministic (cf. maximum a posteriori (10b)), focusing on enhanced prior expressiveness without necessitating a tractable pdf. In comparison, [1]'s task-level optimization seeks the probabilistic posterior distribution, which concentrates on uncertainty quantification and pdf tractability. \n\nTherefore, [1] is parallel to our work and not directly comparable. Thanks for pointing out this related work; these differences will be highlighted in Appendix E of the revised paper.\n\n\n[1] X. Zhang, D. Meng, H. Gouk, and T. Hospedales, \"Shallow Bayesian meta learning for real-world few-shot recognition,\" ICCV 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q0mwCd1n55", "reviewer": "Reviewer_XQHx", "summary": "The paper proposes to learn a more expressive parameter prior compared to predefined ones for a meta-learner. The problem is formulated in a Bayesian model-agnostic meta-learning context and its solution leverages a class of non-invective normalising flows to learn the parameter prior (aka Sylvester flows). Experiments are conducted on mini-ImageNet and CUB datasets, by comparing the proposed approach applied to existing meta-learning algorithms against their base counterparts, showcasing the benefits of an adaptive prior in terms of few-shot classification performance.", "questions": "In addition to addressing the two above-mentioned weaknesses, can you elaborate more on how the model handles uncertainty, given its probabilistic nature?", "limitations": "An important limitation of the proposed solution is its computational complexity. A thorough evaluation and discussion about the trade-off between computation and performance should be also provided.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is overall well written and clear **Clarity**\n2. The considered problem of learning a prior is relevant in the context of meta-learning **Relevance**\n3. While Sylvester normalising flows are not part of the contribution of the paper, its theoretical analysis gives a nice motivation to their use. Moreover, the theoretical part of the paper seems to be correct and sound **Soundness**\n4. Experiments show convincingly that the proposed solution outperforms base methods **Significance**\n5. Code is available during submission, demonstrating the openness to share it publicly **Openness**", "weaknesses": "1. One of the major concern with the proposed solution is in its novelty and therefore its contribution **Novelty**. There are several variants of probabilistic formulations for model-agnostic meta-learning with corresponding learnt prior. How does the proposed solution differ from them and what is therefore the novel contribution? See for instance [1-3].\n2. Experiments miss important baselines **Quality**. See for instance [1-3].\n\n**Reference**\n\n[1] Amortised Bayesian Meta-Learning. ICLR 2019\n\n[2] Bayesian Model-Agnostic Meta-Learning. NeurIPS 2018\n\n[3] Probabilistic Model-Agnostic Meta-Learning. NeurIPS 2018", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the questions. The issues raised are addressed one-by-one next. \n\n**Response to weakness 1**\n\nThis contribution differs remarkably from probabilistic meta-learning in three key aspects. \n- Deterministic algorithm. Our MetaNCoV is a *deterministic* meta-learning approach where task-level optimization (10b) is formulated as a maximum a posteriori (MAP) problem. This MAP estimator can be efficiently approximated by gradient descent. In comparison, probabilistic meta-learning methods aim to approximate the intractable posterior $p(\\boldsymbol{\\phi}_t | \\mathbf{y}_t^\\mathrm{trn}; \\mathbf{X}_t^\\mathrm{trn}, \\boldsymbol{\\theta}) \\propto p(\\mathbf{y}_t^\\mathrm{trn} | \\boldsymbol{\\phi}_t; \\mathbf{X}_t^\\mathrm{trn}) p(\\boldsymbol{\\phi}_t; \\boldsymbol{\\theta})$ using variational inference [1,3] or particle sampling [2]. \n- Prior Expressiveness. While probabilistic meta-learning relies on tractable prior and (surrogate) posterior pdfs of prefixed forms, such as Gaussian [1,3], our approach forgoes the tractability for enhanced prior expressiveness. This allows for a data-driven prior of universal forms beyond the Gaussian family. \n- Theoretical Analysis. The theoretical analysis also distinguishes the current paper with probabilistic meta-learning methods [1-3], which are based on empirical designs. \n\n**Response to weakness 2**\n\nWe will add these three additional baselines to Table 1 for a more comprehensive comparison.\n\n**Response to questions and limitations**\n\nOur method is *deterministic* rather than probabilistic, which cannot directly handle uncertainty quantification due to the intractability of the prior pdf, as mentioned in Remark 3.3. \n\nThe complexity analysis and performance trade-off have been already investigated in our submission, with results respectively reported in Appendices D.4 and D.3. It can be seen that our MetaNCoV has comparable time and space complexity relative to popular optimization-based meta-learning approaches. \n\n[1] S. Ravi, and A. Beatson, \"Amortised Bayesian meta-learning,\" ICLR 2019. \n\n[2] J. Yoon, T. Kim, O. Dia, S. Kim, Y. Bengio, and S. Ahn, \"Bayesian model-agnostic meta-learning,\" NeurIPS 2018. \n\n[3] C. Finn, K. Xu, and S. Levine, \"Probabilistic model-agnostic meta-learning,\" NeurIPS 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary of the paper:**
The paper advocates for learning a flexible (non-prescribed) prior in meta learning and achieves this by the change of variable trick from a fixed prior; differently to NFs, however, without requiring the invertibility. Particularly, it works on optimization-based meta learning and uses Sylvester NF without enforcing the Jacobian structure. It evaluates the method on several few-shot learning benchmarks of meta learning. 

**Summary of the reviews:**
The reviewers found the paper clearly written and well-motivated, the theoretical and empirical results rigorously achieved, and the empirical results to be significantly improving over the baselines with fixed form of priors. 

On the other hand, they were concerned about limited benchmarks and novelty over works within Bayesian (or generally probabilistic) meta learning.

**Summary of the rebuttal and discussions:**
In the rebuttal period, the authors provided additional results on a lower-scale version of MetaDataset for cross-domain generalization where the trends are seen to be similar. They also argue for the MAP aspect of their work when optimizing with the learned prior on test tasks which trades-off the uncertainty quantification of Bayesian meta learning for a tractable expressivity of their learnt priors.

**Consolidation report:**
The authors successfully addressed the main criticisms of the reviewers that led to their eventual leaning towards acceptance.

**Recommendation:**
The AC agrees with the unanimous rating and suggests acceptance.
Decision: Accept (poster)