Meta-Learning Universal Priors Using Non-Injective Change of Variables

Authors: yilang  zhang
alireza  sadeghi
georgios b giannakis
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 22:21:28

Abstract: Meta-learning empowers data-hungry deep neural networks to rapidly learn from merely a few samples, which is especially appealing to tasks with small datasets. Critical in this context is the *prior knowledge* accumulated from related tasks. Existing meta-learning approaches typically rely on preselected priors, such as a Gaussian probability density function (pdf). The limited expressiveness of such priors however, hinders the enhanced performance of the trained model when dealing with tasks having exceedingly scarce data. Targeting improved expressiveness, this contribution introduces a *data-driven* prior that optimally fits the provided tasks using a novel non-injective change-of-variable (NCoV) model. Unlike preselected prior pdfs with fixed shapes, the advocated NCoV model can effectively approximate a considerably wide range of pdfs. Moreover, compared to conventional change-of-variable models, the introduced NCoV exhibits augmented expressiveness for pdf modeling, especially in high-dimensional spaces. Theoretical analysis underscores the appealing universal approximation capacity of the NCoV model. Numerical experiments conducted on three few-shot learning datasets validate the superiority of data-driven priors over the prespecified ones, showcasing its pronounced effectiveness when dealing with extremely limited data resources.

Main paper: 
 introduction: 
 Advances in deep learning (DL) have boosted the notion of "learning from data" with field-changing performance improvements reported across a wide range of applications [24, 15, 52] . Large-scale DL models with high fitting capacity have documented ability to cope with the "curse of dimensionality" by providing compact low-dimensional representations of high-dimensional data. Nonetheless, these high-capacity models typically require protracted training using massive data records. Humans on the contrary, can perform exceptionally well on tasks such as object recognition or concept comprehension with merely a few samples. How to acquire the learning ability of humans in the DL training processes is thus appealing and imperative for a number of application domains, especially when data are scarce or costly to annotate. Examples of such applications include machine translation [52] , medical imaging [31] , and robot manipulations [27] .Meta-learning, also referred to as "learning to learn," seeks to gather the prior knowledge shared across a set of inter-related tasks, to enable quickly solving an unseen yet related learning task using minimal training samples [10] . This form of higher-level learning effectively extracts domain-generic inductive biases from prior tasks, which can be subsequently transferred to learn a new task even with limited data. This mirrors the capability that humans excel at -leveraging past experiences to rapidly acquire new skills. Meta-learning holds the promise of yielding powerful priors with which DL models can generalize better, require fewer data for training, and adapt more effectively to new tasks in dynamically changing environments.Conventional approaches to meta-learning have relied on hand-crafted techniques to extract prior knowledge [47, 46] . With the advent of DL and growing volume of data, there has been a paradigm shift from such cumbersome procedures towards more efficient data-driven strategies. In particular, the prior information is encoded in hyperparameters, which are shared across tasks and can be finetuned using the validation data of all tasks. Utilizing these informative hyperparameters, task-specific learning can be performed even with limited data. Early attempts adopted a neural network (NN) with its weights serving as the shared hyperparameters [55, 45, 33] . The task-invariant NN leverages the shared hyperparameters, and training data per task, to output the task-specific model. However, the selection of an appropriate NN architecture is tailored to the choice of the task-specific models. In addition, NNs inherently lack interpretability and robustness due to their "black box" nature.Unlike NN-based meta-learning, model-agnostic meta-learning (MAML) does not rely on any presumptions about task-specific models [10] . Instead, it relies on an iterative optimizer to learn the task-specific model. The task-invariant prior information is embodied in the initialization of the optimizer, which is shared across tasks. By learning an informative initialization, task-specific learning can rapidly converge to a local minimum within a few iterations. Interestingly, the initialization generated by MAML can be viewed as a learnable mean of an implicit Gaussian prior probability density function (pdf) over the task-specific model parameters [18] . Building on MAML, several optimization-based meta-learning algorithms have been advocated to learn different prior pdfs [29, 37, 25, 1, 58] . In addition, theoretical studies have been carried out to further offer insights into these approaches [13, 39, 8, 9, 61] . Nevertheless, the prior models of most existing meta-learning methods are confined to preselected pdfs, such as the Gaussian one, and thus have limited expressiveness, meaning fitting ability. Consequently, generalizing meta-learning to domains that deal with scarce datasets, and need sophisticated priors, remains a challenging and largely uncharted territory.To improve the prior expressiveness in meta-learning, this contribution puts forth what we term non-injective change-of-variable (NCoV) model, which enables learning a universal data-driven prior from related tasks. The contribution of the resultant method named MetaNCoV is threefold: i) Our novel NCoV model is proven capable of mapping a known source pdf to an arbitrary target pdf. This markedly enhances the model expressiveness, especially in high-dimensional spaces. ii) Theoretical analysis is provided to demonstrate that a parametric NCoV can approximate a broad spectrum of pdfs, that in turn enables versatile plug-in priors for meta-learning. Moreover, this parametric NCoV inherently provides a task-invariant initialization, rather nicely eliminating the need for its explicit learning. iii) Numerical tests on three benchmark few-shot learning datasets corroborate our theoretical analysis, and underscore the superior prior expressiveness of the proposed MetaNCoV method compared to meta-learning approaches with prespecified pdfs.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main claims have been proved both theoretically in Section 3 and numerically in Section 4, which are the key contributions of this paper. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
problem setup: 
 Meta-learning relies on task-invariant prior information from a collection of T given tasks (indexed by t = 1, . . . , T ), to deal with data-limited settings. For each t, there is a dataset D t := {(x n t , y n t )} are to be inferred. The major premise of meta-learning is that the aforementioned tasks are related through their underlying data distributions or problem structures. This relationship makes it feasible to employ a unified large-scale model such as a deep NN to fit all tasks, with each task tailored by its specific model parameter ϕ t ∈ R d . However, as the cardinality |D trn t | can be much smaller than d, directly optimizing ϕ t over D trn t could readily lead to overfitting.Meta-learning addresses this issue by capitalizing on the relationships among tasks. Specifically, since T is considerably large in meta-learning, a task-invariant prior can be extracted to capture knowledge across tasks, thereby facilitating the data-limited per-task training. This nested structure of prior extraction and per-task training lends itself to a bilevel optimization problem. The inner-level (task-level) optimizes the per-task parameter ϕ t using D trn t , and the prior provided by outer-level, while the outer-level (meta-level) evaluates the trained {ϕ t } T t=1 using {D val t } T t=1 , and refines the prior parameterized by θ ∈ R D .The bilevel optimization objective of meta-learning can be expressed asEQUATIONwhere the loss function L assesses the fit of a task-specific model to a designated dataset, and the regularizer R quantifies the impact of task-invariant prior. [41, 17] . Despite the effectiveness of NN optimizers in fitting complex mappings, it is hard to decipher the learned prior due to their black-box nature. To improve the interpretability and robustness of the approximate solver, optimization-based meta-learning decodes the "tractable optimizer" as a cascade of a few optimization iterations. The prior is captured by the shared hyperparameters of the optimizer. The first effort towards this direction is termed MAML [10] , which relies on a K-step gradient descent (GD) optimizerEQUATIONwhere task-invariant initialization ϕ 0 t = ϕ 0 = θ parameterizes the prior information, and φt = ϕ K t gives the desired approximate solver. Interestingly, despite the absence of an explicit regularization term (that is, R(ϕ t ; θ) = 0), it has been shown that, under second-order Taylor approximation, MAML's GD solver (2) satisfies [18] φt (θ) ≈ ϕ * t (θ) = argminϕ t L(ϕ t ; D trn t ) + 1 2 ∥ϕ t -ϕ 0 ∥ 2where the precision matrix Λ t is determined by α, K, and ∇ 2 L(ϕ 0 ; D trn t ). This observation indicates MAML's optimizer approximately amounts to an implicit Gaussian prior p(ϕ t ; θ) ≈ N (ϕ t ; ϕ 0 , Λ -1 t ), with the shared initialization ϕ 0 = θ serving as its mean vector. Building upon MAML, various methods have been investigated to learn different prior pdfs in both implicit and explicit forms. For example, recent advances further render the (per-step) precision matrix learnable by replacing it with a Λ that is common across tasks. Letting θ Λ denote the parameter of Λ, the prior parameter is thus augmented asθ := [ϕ 0⊤ , θ ⊤ Λ ],where ⊤ denotes transposition. However, a complete parametrization of Λ would result in θ having prohibitively high dimensionality, that is, D = O(d 2 ). To ensure scalability with respect to D, Λ should have a sufficiently simple structure such as isotropic [39] , diagonal [29] , and or block diagonal [26, 37] matrices. Inspired by transfer learning, one can instead split the model into an embedding "body" and a classifier/regressor "head," and learn their priors independently; that is, with ϕ body t and ϕ head t denoting the corresponding partitions of ϕ t , the prior is presumed factorable as p(ϕ t ; θ) = p(ϕ body t ; θ)p(ϕ head t ; θ). On the one hand, the head typically has a nontrivial prior such as the Gaussian one [3, 25] . On the other hand, the body's prior is intentionally restricted to a degenerate pdf p(ϕ body t ; θ) := δ(ϕ body t -ϕ body ), where ϕ body is a subvector of θ, and δ(•) is the Dirac delta function. This eliminates the need for optimizing ϕ body t in (1b), thus markedly lowering the overall complexity for solving (1) . Although freezing the body in (1b) allows for escalating the dimension of ϕ body t , it often leads to degraded empirical performance [38] compared to the full update (2) . In addition to Gaussian and degenerate pdfs, sparse priors (Laplace distributions) have been investigated in the context of network pruning [50] .Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations have been discussed in Remark 3.3. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
meta-learning using non-injective change of variables: 
 Existing meta-learning algorithms rely on a preselected pdf to parameterize the prior. However, the chosen pdf can have limited expressiveness; that is, it may have insufficient ability to offer an accurate fit due to its prefixed shape. Consider for instance a Gaussian prior pdf, which is inherently unimodal, symmetric, log-concave, and infinitely differentiable by definition. Such a prior may not be well-suited for tasks with multimodal or asymmetric parametric pdfs.In this work, we propose to learn a data-driven prior pdf that optimally fits the given tasks using a novel non-injective change-of-variable (NCoV) model. We thus term the proposed method as Meta-learning with NCoV (MetaNCoV). In contrast to preselected prior pdfs with fixed shapes, the advocated prior model can dynamically adjust its form to approximate a considerably wide range of pdfs, as will be demonstrated both theoretically and numerically. Furthermore, compared to conventional change-of-variable models such as generative adversarial networks (GANs) [16] and normalizing flows (NFs) [43] , the introduced NCoV exhibits enhanced capacity for pdf estimation, especially in high-dimensional spaces. Change-of-variable models and their applications in pdf estimations will be first elaborated. All the proofs are delegated to the Appendix.The key idea of change-of-variable model is to identify a transformation f , through which a known pdf p Z can be altered to approximate a target pdf q. For instance, GANs [16] and variational autoencoders (VAEs) [21] seek a generator/encoder such that high-dimensional pdfs can be acquired from a low-dimensional latent Gaussian pdf p Z = N (0, I). Due to the dimensional discrepancy between signal and latent spaces, these models are typically utilized to estimate signals living on a low-dimensional manifold; e.g., images.To enhance the model capactiy as well as pdf tractability, NFs were introduced in [43] as a surrogate variational model for posterior inference. Recently, they have been shown also effective in estimating prior pdfs from a set of unannotated samples [6, 14, 7] . The formulation of NFs relies on the wellknown change-of-variable formula. Given a continuous random vector Z ∈ R d with prior pdf p Z , and a bijection f : R d → R d , then Z ′ := f (Z) is also a continuous random vector with analytical pdfEQUATIONwhere J f (z ′ ) denotes the Jacobian of f at z ′ ∈ R d , det is the determinant, and det J f ̸ = 0 almost everywhere (a.e.) for bijective f . To ensure the invertibility of f , a prudent choice is to model it as a composition of a sequence of bijective functionsf = f 1 • f 2 • . . . • f n .In Bayesian inference [43] , q is an intractable posterior, and f is optimized to minimize the KLdivergence between p Z ′ and q, or equivalently, maximize the so-termed evidence lower bound (ELBO). For density estimation [6] , the wanted q is an unknown prior pdf, while f is acquired via maximum likelihood training. The obtained f can be leveraged in two important applications: i) probability estimation p Z ′ (v) ≈ q(v) for a given sample v ∼ q using (3), and ii) generation of a sample z ′ = f (z), z ∼ p Z for which p Z ′ ≈ q.When d = 1, the probability integral transform (PIT) suggests that, the optimal f * = Q -1 • P Z leads to precisely P Z ′ = Q a.e., where Q, P Z and P Z ′ are the cumulative distribution functions (cdfs) corresponding to q, p Z and p Z ′ , and q > 0 a.e. ensures Q is bijective. The resultant cdf P Z ′ = P Z • f * -1 is a pushforward measure, also notated as P Z ′ = f * # P Z . In high-dimensional spaces (d > 1) however, the existence of such an f * may not hold due to the invertibility assumption of f * ; see examples in e.g., [22, Section 4] . In fact, it has been shown that NFs are capable of modeling pdfs with a full support; i.e., when q > 0 on R d [36] .To improve the fitting capacity of change-of-variable models for generic q, especially those in highdimensional spaces or without full support, the fresh idea of this work is to waive the injectivity assumption on f . In doing so, we can generalize the PIT to an arbitrary q, as illustrated in the following theorem. Theorem 3.1 (Multivariate PIT). Let P Z : R d → [0, 1] be the cdf of continuous random vector Z := [Z 1 , . . . , Z d ] ⊤ with {Z i } d i=1 mutually independent. For any differentiable a.e. cdf Q : R d → [0, 1], there exists a non-decreasing function f * : R d → R d that the random vector Z ′ := f * (Z) has cdfEQUATIONRemark 3.2 (Choice of source distribution). In the theorem, the prior distribution for the source random vector Z can be chosen arbitrarily, as if it is continuous and has mutually independent entries. Popular choices include standard Gaussian N (0 d , I d ) and uniform U([0, 1] d ). Remark 3.3 (Comparison with injective NFs). While conventional NFs (3) require J f ̸ = 0 a.e. to ensure the injectivity of f , Theorem 3.1 relaxes this assumption to allows f being non-injective and thus enables Z ′ = f (Z) to match an arbitrary target distribution (even discrete one) in a highdimensional space. It is worth mentioning that the mild assumption on the differentiability of Q is merely used to guarantee the existence of q, which can be easily satisfied. However, one limitation of the advocated NCoV is that it generally has no analytical solution for the resultant surrogate pdfEQUATIONAs a remedy, efficient numerical integration can be performed to estimate p Z ′ when d is small. Additional comparisons with NFs and optimal transport are deferred to Appendix E.While Theorem 3.1 suggests the existence of the optimal f * that incurs the exact match p Z ′ = q, the expression for such an f * relies on the sought q, which is typically intractable or unknown. Therefore, a feasible alternative is to resort to a tractable parametric f (•; θ f ), which approximates f * by learning θ f from the provided data. To further compare NCoVs with NFs, we will focus exclusively on Sylvester NF [51] in the following sections, but our analysis can be readily generalized to other transformations; see Remark 3.7. Sylvester NF was introduced in [51] to improve the expressiveness of planar NF [43] by increasing its "width". In particular, Sylvester NF adopts the formEQUATIONwhere A ∈ R d×m , B ∈ R m×d , c ∈ R m are learnable weights with m being the number of hidden neurons (a.k.a. width), σ is an entry-wise nonlinear operator, andθ f := [vec(A) ⊤ , vec(B) ⊤ , c ⊤ ] ⊤ .It can be easily verified that the Sylvester NF boils down to the planar one when m = 1. Akin to other NFs, one can also increase the "depth" of the flows by stacking multiple Sylvester NF layers into a chainf 1 • f 2 • . . . • f n .The next theorem states that, the optimal f * can be approximated to arbitrary precision using a sufficiently wide one-layer Sylvester NCoV. Definition 3.4. A random vector on R d is said to be tail-convergent if i) it has a pdf p : R d → R + ∪ {0}, and ii) for ∀ϵ > 0 there exists a bounded E ⊂ R d for whichEQUATIONTheorem 3.5 (Universal approximation via Sylvester NCoVs). Let P Z denote the cdf of tailconvergent continuous random vector Z ∈ R d with mutually independent entries, and Q a Lipschitz cdf of a tail-convergent random vector. For any ϵ > 0, there exists cdfs P , Q for which the corresponding pdfs p, q vanishes outside compact sets E p , E q , andEQUATIONMoreover, let E ⊆ E p be any set on which the optimal f * matching PZ to Q is injective and right-continuous. There exists a Sylvester NCoV f and a zero-measure set E 0 , such thatEQUATION)We have shown that when f * is injective, the cdf of the optimally transformed Z ′ = f * (Z) can be written as a pushforwardQ = P Z ′ = P Z • f * -1 .Likewise, this relationship remains valid when restricting f * to a set E on which f * is injective. However, since the Sylvester NCoV f may not be injective on E, one cannot directly compare Q with P Z • f -1 . Fortunately, this pushforward can be equivalently written asP Z (z) = Q • f * (z), ∀z ∈ E; see Lemma B.1 in the Appendix. Utilizing this alternative relationship, Theorem 3.5 states that the Sylvester NCoV f not only approximates f * a.e. on E p , but also results in pushforward approximation P Z ≈ Q • f a.e. on E.Input: {D t } T t=1 , step sizes α and β, batch size B, and maximum iterations K and R. Initialization: randomly initialize θ 0 f . for r = 0, . . . , R -1 do Sample random T r ⊂ {1, . . . , T } of cardinality B.for t ∈ T r do Initialize z 0 t = argmin zt R Z (z t ). for k = 0, . . . , K -1 do Descend z k+1 t = z k t -α∇ z k t [L(f (z k t ; θ r f ); D trn t ) + R Z (z k t )]. end for Approximate solver ẑt = z K t . end for Update θ r+1 f = θ r f -β 1 B t∈T r ∇ θ r f L(f (ẑ t (θ r f ); θ r f ); D val t ). end for Output: θf = θ R f .Estimated pdfs via Sylvester NCoVs, 𝒁 ′ = 𝑓(𝒁) Remark 3.6 (Mild assumptions). The assumptions in Theorem 3.5 are mild and common. In particular, tail-convergence only requires the probability of large deviation diminishing to 0 as the norm of the random vector goes to +∞, while imposing no constraint on the decaying rate. This assumption can be easily satisfied by a wide family of distributions, even including the heavily-tailed ones. Under this benign assumption, (8) suggests P Z and Q can be approximated by alternatives P , Q with pdfs p, q having truncated tails. This is crucial to universal approximation, which typically requires f * to be bounded or Lebesgue integrable [5] . Moreover, the Lipschitzness of Q is solely utilized to ensure the boundness of its gradient, namely the pdf q. This can be also readily met by most practical cdfs.pdfs of interest, 𝑞(⋅) (a) (b) (c)Remark 3.7 (Generalization to other NFs). Although Theorem 3.5 primarily focuses on one-layer Sylvester NCoVs, similar analysis for other transformation f can be acquired by employing different universal approximation models. For instance, results for multi-layer planar NCoVs and multi-layer Sylvester NCoVs can be respectively established leveraging [30] and [32] . Remark 3.8 (Influence of ϵ). It is worth noting that the width m of the Sylvester NCoV depends on ϵ as well as the optimal f * . Smaller ϵ typically leads to larger m. Additionally, the nonlinearity σ must be sigmoidal; see Definition B.3 in the Appendix for details.Next, we elucidate how universal priors can be learned in meta-learning by harnessing the proposed NCoV model. Different from existing works that rely on prespecified prior forms such as Gaussian pdfs, the novel concept of this work is to learn a data-driven prior that optimally conforms with the given tasks. This is achieved by transforming the random vector Z ∈ R d with a known prior p Z to Z ′ = f (Z; θ f ), whose pdf is given by ( 5). This p Z ′ acts as a surrogate model for the unknown p(ϕ t ; θ), and learning the prior parameter θ thus boils down to optimization of the transformation parameter θ f . Nevertheless, as discussed in Remark 3.3, p Z ′ typically has no close-form expression when f is non-injective. Therefore, instead of directly optimizing ϕ t , we propose to optimize the latent vector z t corresponding to ϕ t = f (z t ; θ f ), which yieldsEQUATIONwhere R Z (z t ) := -log p Z (z t ) is the nlp regularizer, and z * t is thus the MAP estimator for z t . Similar to (2), the global task-level minimizer z * t is generally infeasible to attain. Hence, a tractable alternative is to rely on an approximate GD solver. Interestingly, our formulation (10) naturally offers a convenient initialization using the maximum a priori estimatorEQUATIONAs an example, choosing p Z = N (0 d , I d ) automatically gives z 0 t = 0 d and the corresponding ϕ 0 t = f (0 d ; θ f ). This elegantly removes the need for separately learning the task-invariant initialization ϕ 0 , which is exactly the maximum a priori estimator of the preselected Gaussian prior pdf p(θ t ; θ) = N (ϕ 0 , Λ t ). In fact, the task-invariant initialization reflects our optimal guess of ϕ t before accessing any task-specific data, and can be naturally derived by maximizing the prior pdf. It also worth noting that while the idea of optimizing the latent (instead of primal) variables shares similarities with [44] , the latent space in [44] is designed from a different perspective, which is low-dimensional to the end, and requires an initialization.To this end, (10) can be solved using a standard alternating optimizer. The resultant MetaNCoV algorithm is listed step-by-step in Algorithm 1, where the inner-level (10b) and outer-level (10a) are respectively optimized using K-step GD and mini-batch stochastic GD.While our idea has the potential to be broadened beyond meta-learning, we must emphasize that our current setup is specifically tailored to meta-learning, which does not require a tractable pdf, but rather demands enhanced prior expressiveness. We should also highlight that the intractability of (5) prohibits learning NCoV via conventional approaches such as maximum likelihood training and evidence lower-bound maximization -this thus necessitates careful attention and extra certain designs when applying NCoV to other domains.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: All the datasets used in the experiments are available online, for which proper citations and descriptions have been provided in Appendix C. Our codes and scripts used in the experiments are also uploaded as the supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
numerical tests: 
 In this section, we test and showcase the empirical superiority of MetaNCoV on both synthetic and real datasets. All datasets descriptions and hyperparameter setups are deferred to the Appendix C. Codes for reproducing the results are available at https://github.com/zhangyilang/MetaNCoV.Here, we investigate an intricate yet interesting scenario to demonstrate the efficacy of NCoVs to approximate complex multi-modal pdfs in two-dimensional (2D) settings. The primary objective is to transform a standard Gaussian random vector Z ∼ N (0 2×1 , I 2×2 ) into multi-modal complex pdfs. The outcomes of this experiment are presented in Figure 1 . The lower row displays the ground-truth pdfs q of interest, while the upper row showcases the numerically estimated pdfs of the transformed random vector Z ′ = f (Z), where f is a Sylvester NCoV, and the pdf of Z ′ is estimated via (5) . As clearly evidenced in these results, the advocated NCoVs exhibit their capability to effectively convert Next, the empirical performance of MetaNCoV is assessed on three real datasets for meta-learning.The experimental setups follow from the standard M -class N -shot few-shot classification protocol [41, 10] . In particular, D trn t per task t consists of M randomly drawn classes, each containing N labeled data. The default task-specific model is a standard 4-block convolutional NN (CNN) [55] . Each block of the CNN comprises a 3 × 3 convolution layer, a batch normalization layer, a ReLU activation, and a 2 × 2 max pooling layer. After the convolutional blocks, a linear regressor with softmax activation is appended to perform classification. Following the practices of [37, 12] , the number of convolutional channels is set to 128 to improve its fitting capacity. Additionally, to be consistent with Theorem 3.5, Sylvester NCoVs are adopted in all the tests.To illustrate the benefit of learning more expressive priors, the first test compares MetaNCoV with other meta-learning algorithms having different prespecified priors using the miniImageNet dataset [55] . As a plug-in prior model, our MetaNCoV can be readily integrated with other metalearning methods that adopt different task-level optimizers. In this test, we implement MetaNCoV with MAML [10] and MetaSGD [29] . The results are listed in Table 1 , where the performance metric is the average classification accuracy with 95% confidence interval on new tasks. It is seen that our MetaNCoV outperforms all the competitors. This empirically confirms the superiority of data-driven priors over the prespecified pdfs, as well as the effectiveness of MetaNCoV in learning an expressive prior. Moreover, a remarkable performance gain can be observed on the 1-shot dataset. This justifies the claim that prior can be particularly informative when the training data are extremely scarce. For an apples-to-apples comparison, methods that use pre-trained feature extractors or more complicated models (e.g., residual networks) are not included in this table. The compatibility of MetaNCoV to these models will be demonstrated in the subsequent tests.The second test evaluates MetaNCoV on miniImageNet and tieredImageNet feature embeddings extracted using a pre-trained Wide ResNet(WRN)-28-10 backbone [44] . Compared to the 4-block CNN, this model has a greater number of parameters and thus enhanced expressiveness. The results are summarized in Table 2 , where MetaNCoV is implemented with MetaSGD [29] and MC [37] . In all tests, MetaNCoV brings about notable performance improvement compared to the corresponding baselines. This validates MetaNCoV's effectiveness and flexibility as a plug-in prior module.The last test assesses the performance of MetaNCoV on the CUB-200-2011 dataset [57] . In contrast to the previous two datasets that contain nature images of distinct objects, this dataset specifically focuses on birds of various species. While the classification of nature objects primarily relies on low-level features such as shapes and colors, classifying various birds requires further recognition of high-level features including textures and segmentations. To learn these complicated features, the model needs to be either trained with sufficient data, or equipped with a powerful prior. Table 3 showcases the performances of different meta-and metric-learning methods on such a dataset. Again, our MetaNCoV method is markedly effective on the 1-shot dataset where data are exceptionally limited. This highlights the significance of an expressive prior. For the 5-shot dataset where data are relatively abundant, its performance is also comparable to the state-of-the-art ones. Next, ablation tests are conducted to analyze the performance gain of MetaNCoV.The test is carried out on the miniImageNet dataset, with results gathered in Table 4 . The first ablation investigates the impact of the advocated NCoVs over the injective ones. To ensure the injectivity of the Sylvester NF f , we follow the QR parameterization recommended in [51] . One can see the improved performance of NCoV due to its enhanced expressiveness, which numerically verifies Theorem 3.1 and Remark 3.3. The second ablation examines the influence of nonlinear function σ in the Sylvester NCoVs. By changing the σ from sigmoid to the popular ReLU activation, a degradation of empirical performance can be observed. This observation corroborates with Remark 3.8. Additional experiments and visualizations can be found in Appendix D.This subsection showcases the generalization capacity of MetaNCoV in cross-domain few-shot learning. This test is more challenging compared to previous ones due to the domain gap between the meta-training and meta-testing phases. By shifting the task domain, this test aims to assess the overfitting of the learned prior to a specific domain. Our test setup follows from [35] , where the prior model is meta-trained on the miniImageNet [55] dataset, and meta-tested on tieredImageNet [42] , Cars [23] , and CUB [57] datasets. Our MetaNCoV is implemented with MetaSGD [29] in this test.As shown in Table 5 , our method consistently outperforms popular meta-learning approaches in such a setup, especially in the 1-shot case. This not only confirms the cross-domain generalization of MetaNCoV, but again justifies the importance of expressive prior when data are exceedingly limited.A Proof of Theorem 3.1Theorem A.1 (Multivariate PIT, restated). Let P Z : R d → [0, 1] be the cdf of continuous random vector Z := [Z 1 , . . . , Z d ] ⊤ with {Z i } d i=1 mutually independent. For any differentiable a.e. cdf Q : R d → [0, 1], there exists a non-decreasing function f * : R d → R d that the random vector Z ′ := f * (Z) has cdfEQUATIONProof. We claim that the i-th entry of transformation f * adopts the formEQUATIONwhere g i : R i → R will be specified soon, and h i (Z) := [P Z1 (Z 1 ), . . . , P Zi (Z i )] ⊤ . The proof follows from the mathematical induction on d.First consider the base case d = 1. By univariate probability transformation, f * = Q -1 • P Z directly verifies Theorem 3.1 and the claim (13) , whereQ -1 (u) := inf{v | Q(v) ≥ u}, u ∈ [0, 1].What remains is the proof for the monotonicity of f * . Since Q and P Z are cdfs, they are non-decreasing by definition. Using the monotonicity ofQ, it holds that {v | Q(v) ≥ u 1 } ⊆ {v | Q(v) ≥ u 2 }, ∀u 1 ≥ u 2 . From the definition of Q -1 , we have Q -1 (u 1 ) ≥ Q -1 (u 2 ), ∀u 1 ≥ u 2 , meaning that Q -1 is also non-decreasing. As a result, the composition f * = Q -1 • P Z is non-decreasing.Subsequently, assuming Theorem 3.1 and the validity of claim ( 13) for d = 1, . . . , d 0 , we establish them for d = d 0 + 1. This induction-based argument gives the proof of Theorem 3.1. For notational compactness, define random variable U i := P Zi (Z i ), i = 1, . . . , d 0 + 1, and likewise random vectorU 1:i := [U 1 , . . . , U i ] = h i (Z). The univariate PIT indicates that each U i ∼ U[0, 1].Besides, since {Z i } d0+1 i=1 are mutually independent, it follows that {U i } d0+1 i=1 are also mutually independent. Let f : R d0 → R denote the transformation provided by the inductive hypothesis for case d = d 0 . The first d 0 entries of the desired f * (for case d = d 0 + 1) can be defined as f * 1:d0 (Z) := f (Z 1:d0 ), Z ∈ R d0+1 . Thus, the inductive hypothesis suggest the joint cdf forZ ′ 1:d0 = f * 1:d0 (Z) ∈ R d0 is P Z ′ 1:d 0 (ξ) = Q([ξ ⊤ , +∞] ⊤ ), ξ ∈ R d0 (notice that Q(•) is a function defined on R d0+1 for d = d 0 + 1, while Q([•, +∞] ⊤ ) is on R d0 ).Next, it will be shown that ( 12) can be obtained upon defining f * d0+1 = g d0+1 • h d0+1 withEQUATIONwhere conditional cdf  and (a ) is because the conditional cdf is non-decreasing and right continuous so the infimum can be attained. First notice that the transformed random vectorQ d0+1|1:d0 (v d0+1 |v 1:d0 ) := Pr(V ′ d0+1 ≤ v d0+1 | V 1:d0 ⪯ v 1:d0 ) for (d 0 + 1)- dimensional random vector V obeying cdf Q,Z ′ = f * (Z) = [g 1 • h 1 (Z), . . . , g d0+1 • h d0+1 (Z)] ⊤ = [g 1 (U 1 ), . . . , g d0+1 (U 1:d0+1 )] ⊤ (15)has cdfEQUATIONwhere the equality is due to P XY (x, y) =x -∞ p X (ξ)P Y |X (y|ξ)dξ for random variables X, Y .On one hand, it holds thatEQUATIONwhere (a) is from ( 15), (b) uses ( 14), (c) follows from the mutual independency of {U i } d0+1 i=1 , and(d) is because U d0+1 ∼ U[0, 1].On the other hand, it has already been shown using the inductive hypothesis that, the random vectorZ ′ 1:d0 = f * 1:d0 (Z) has cdf P Z ′ 1:d 0 (ξ) = Q([ξ ⊤ , +∞] ⊤ ), ξ ∈ R d0 . Since Q is differentiable a.e., we have the corresponding pdf p Z ′ 1:d 0 (ξ) = R q([ξ ⊤ , η] ⊤)dη := q 1:d0 (ξ) a.e.. As a result, it follows from ( 16) and ( 17) thatEQUATIONwhere we use P XY (x, y) =x -∞ p X (ξ)P Y |X (y|ξ)dξ again. It should be noted that the only type of discontinuities for a monotone function is the jump discontinuity and there are at most countably many of them. Consequently, P Z ′ may fail to match Q only on a set of measure zero.Finally, we will prove the monotonicity of this constructed f * by showing that J f * ⪰ 0 (d0+1)×(d0+1) . First notice from ( 14) that g d0+1 (U 1:d0+1 ) is a conditional cdf non-decreasing w.r.t. U d0+1 . By the definition of h d0+1 , we have∂[h d 0 +1 (Z)] d 0 +1 ∂Z d 0 +1 = P ′ Z d 0 +1 (Z d0+1 ) ≥ 0 because P Z d 0 +1is a cdf. As a result, applying the chain rule leads to∂f * d 0 +1 (Z) ∂Z d 0 +1 = ∂g d 0 +1 (h d 0 +1 ) ∂[h d 0 +1 ] d 0 +1 ∂[h d 0 +1 (Z)] d 0 +1 ∂Z d 0 +1 = ∂g d 0 +1 (U 1:d 0 +1 ) ∂U d 0 +1 ∂[h d 0 +1 (Z)] d 0 +1 ∂Z d 0 +1≥ 0. Additionally, the inductive hypothesis implies f is nondecreasing on R d ; that is J f ⪰ 0 d0×d0 . To the end, f * (Z) := [ f (Z 1:d0 ) ⊤ , f * d0+1 (Z)] ⊤ has a block triangular JacobianJ f * (Z) =   J f (Z) ∂f * d 0 +1 (Z) ∂Z 1:d 0 0 ⊤ d0 ∂f * d 0 +1 (Z) ∂Z d 0 +1   .It has been shown that diagonal blocks J f (Z) ⪰ 0 d0×d0 and∂f * d 0 +1 (Z)∂Z d 0 +1 ≥ 0, Thus, it follows that J f * ⪰ 0 (d0+1)×(d0+1) , which completes the proof.To aid the proof of Theorem 3.5, the following lemma offers an alternative expression for (4). Lemma B.1. Consider the notational conventions of Theorem 3.1. Let E ⊆ R d be the set on which f * is injective and right-continuous. Then, it holds a.e. thatEQUATION)Proof. With (4) in effect, it holds a.e. thatP Z (z) = Pr(Z ⪯ z) (a) = Pr(f * (Z) ⪯ f * (z)) = P Z ′ (f * (z)) = Q(f * (z)) = (Q • f * )(z) (20)where (a) is because f * is injective thus increasing and right-continuous on E. This proves (19) .The next lemma suggests the cdf of tail-convergent random vector can be approximated by truncating its pdf on a sufficiently large compact set.Lemma B.2. For any tail-convergent random vector with cdf P : R d → [0, 1], and ∀ϵ > 0, there exists a cdf P : R d → [0, 1] for which the pdf p vanishes outside a compact set E ⊂ R d , andEQUATIONProof. For a tail-convergent random vector, Definition 3.4 suggests that for ∀ϵ > 0, there exists a boundedE ′ ⊂ R d such that R d \E ′ p < ϵ/2. Taking E := cl(E ′ ) to be the closure, it follows from the definition of closure that E is compact andEQUATION)where (a) is due to E ′ ⊆ E. Now defineEQUATIONNotice thatR d p = E p = E p/(1 - R d \E p) = 1, which verifies p is a valid pdf. Thus, the induced cdf isEQUATIONIt then follows for ∀v ∈ R d thatEQUATIONEQUATIONwhere (a) uses (23) , and (b) is from (22) .Next, the classic universal approximation theorem will be generalized to suit for the case of NCoVs.). A function σ : R → R is said to be sigmoidal if EQUATIONEQUATIONis dense in C([0, 1] d ).Corollary B.6. Let σ be a bounded measurable sigmoidal function, and E ⊂ R d a locally compact set of finite Borel measure. Then finite sum of the formEQUATIONis dense in C(E).Proof. When µ(E) = 0, one can take E 0 = E and the Corollary holds trivially. Next we consider the case µ(E) > 0.The original proof of Theorem B. Theorem B.7 (Universal approximation via Sylvester NCoVs, restated). Let P Z denote the cdf of tail-convergent continuous random vector Z ∈ R d with mutually independent entries, and Q a Lipschitz cdf of a tail-convergent random vector. For any ϵ > 0, there exists cdfs P , Q for which the pdfs p, q vanishes outside compact sets E p , E q , andEQUATIONMoreover, let E ⊆ E p be any set on which the transform f * matching PZ to Q (cf. Theorem 3.1) is injective and right-continuous. There exists a Sylvester NCoV f and a zero-measure set E 0 , such thatEQUATIONProof. Since Z is tail-convergent, Lemma B.2 suggests that there exists a cdf PZ for which the pdf pZ vanishes outside a compact set E P ⊂ R d , and |P Z (z) -PZ (z)| < ϵ/4, ∀z ∈ R d . Similarly, there is also a cdf Q for which q is supported on a compact set E Q , and|Q(z) -Q(z)| < ϵ/4, ∀z ∈ R d .Moreover, let L Q be the Lipschitz constant of Q. Then, ( 22), ( 23) and (24) indicates Q is also Lipschitz with constantEQUATION)Using Rademacher theorem, we have Q differentiable a.e.. Then, let f * : R d → R d denote the optimal transform by Theorem 3.1, which matches P to Q. Lemma B.1 suggests thatEQUATION)Let Z and Z′ = f * ( Z) be random vectors obeying cdfs PZ and Q. Since pZ is supported on E P , (5) implies that f * can have arbitrary value outside E P , which will not change pZ ′ .We assert that f * is bounded on E P . Otherwise, for any B > 0, there must be some z 0 ∈ E P such that ∥f * (z 0 )∥ > B, and using (5) thatEQUATION)where the inequality is because f * is non-decreasing. Since B can be arbitrarily large, this contradicts with the fact that supp(q) = E Q is compact (cf. Lemma B.2).Moreover, the monotonicity of f * indicates the only possible discontinities of it must be jump discontinuities, and there are at most countably many of them. Let E 0 ⊆ E p be the set where f * is discontinuous. From the countability of E 0 we have µ(E 0 ) = 0. Thus, f * i (z) -z i is bounded and continuous on E p \ E 0 , where f * i and z i are the i-th entries of f * and z. Then, applying Corollary B.6 implies that there exists a G i (z) of form (30) such thatEQUATIONNow, definef i (z) = G i (z) + z i , i = 1, . . . , d on R d .One can easily verify from ( 6) that such a definition renders f a Sylvester NCoV, and (9a) holds. Moreover, it follows for ∀z ∈ E p \ E 0 thatEQUATIONwhere (a) follows from ( 34), (b) is due to (33) , and (c) uses (36) . The proof is thus completed.Our codes are run on a server equipped with an Intel Core i7-12700 CPU, and an NVIDIA RTX A5000 GPU. The experimental setups adopted in this paper will be next elaborated.The numerical tests for the 2D toy examples demonstrated in Figure 1 are carried over by training a Sylvester NCoV with a width m = 50. We used SGD optimizer with learning rate of 10 -3 and momentum of 0.9. The ground truth samples used to train the model were generated by transforming 2D Gaussian random vectors Z ∼ N (0 2×1 , I 2×2 ) through a non-injective ground truth transformation f * (Z) = A σ(B sin(Z) + c), where sin(•) function is applied element-wise to each dimension of vector Z separately. A set of i.i.d. samples {z i , f * (z i )} 10 5 i=1 was randomly generated and used to train the Sylvester NCoV model. The results presented in Figure 1 were obtained using three different settings of ground truth f * (•). In each of these settings, the elements of the underlying matrices A, B, and the vector c were generated from Gaussian distributions with zero mean and unit variance. The result for 1D case presented in Figure 2 was obtained using a smaller Sylvester NCoV with width m = 3, trained using SGD with learning rate of 10 -2 and momentum of 0.6, and the histogram of the generated samples was normalized to represent a pdf. To generate training samples, we employed the probability integral transform (PIT). Specifically, we first draw ground truth Z ′ from a mixture of Gaussians, denoted as Z ′ ∼ q(z ′ ), where q := 2 k=11 2 N (µ k , σ 2 k ).Here, µ 1 = -10 and µ 2 = 10, with σ 2 1 = σ 2 2 = 1. Then, we rely on the inverse transformation f * -1 (Z ′ ) to find its paired Z, where f * (Z) := (Q -1 • P Z )(Z) is the ground truth transformation obtained via PIT. Having find this mapping, we draw a set of i.i.d. samples {z i , f * (z i )} 10 5 i=1 to train Sylvester NCoV model in 1D.A brief description of the three benchmark datasets used in our experiments are provided next.MiniImageNet [55] contains 60, 000 images sampled from the full ImageNet (ILSVRC-12) dataset, which are divided into 100 classes, each with 600 instances. All images are cropped and resized into The hyperparameters used for the few-shot classification experiments are the same as those in MAML [10] , which are listed in Table 6 . To enhance the statbility of the training process, we use SGD with Nesterov momentum instead of Adam as the optimizer for (10a). The width m of Sylvester NCoV is determined through a grid search using the validation tasks. To demonstrate the expressiveness of NCoVs compared to an injective Sylvester NF, we conducted a 2D toy test showcased in Figure 3 . We have employed the QR factorization technique described in [51] to enforce the injectivity constraint for Sylvester NF, which requires m ≤ d. Here, m is the width of the NF, and d = 2 is the dimension of z for this 2D example. To ensure the injectivity of Sylvester NF, m is thus chosen as 2. For a fair comparison, our NCoV is implemented with the same setup. The three rows in the figure display respectively the learned NCoVs, injective NFs, and the target pdf q. It is seen that NCoVs are more expressive than valid NFs when learning complex distributions.To illustrate the ability of NCoVs in learning pdfs whose support is not full in R d , we consider a 4 × 4 checkerboard pdf. The test has been visualized in Figure 4 , where we employ an injective Sylvester NF and a non-injective Sylvester NCoV to learn the target distribution. In our test, we stacked 40 base injective Sylvester NFs, and compared the results with those obtained using a Sylvester NCoV with the same number of parameters and the same training setup. It is observed that NCoV demonstrates remarkable performance in learning the checkerboard pdf compared with that of the injective NF. This observation underscores the expressiveness of NCoVs in effectively learning particular distributions, especially in scenarios involving discrete distributions or those with incomplete support. Next, we show the performance comparison using the 4-block CNN backbone with different number of channels. All the hyperparameters are the same with Appendix C, and the results are gathered in Table 7 . One can observe tht reducing the number of channels from 128 to 64 leads to slightly decreased accuracies, yet still superior than all the state-of-the-art competitors thanks to the improved expressive power of NCoV.To demonstrate the scalability of MetaNCoV, this test measures numerically the time and space complexities of MetaNCoV and several popular meta-learning methods during the meta-training phase. The test is conducted on the miniImageNet dataset, and the comparison is conducted using the 4-block CNN with 64-channel and 128-channel setups, respectively. The experimental results are summarized in the Table 8 . It is observed that MetaNCoV has a dimension D comparable to MetaSGD, yet notably smaller than MC. One can also see that the increase of D only brings about a marginal growth in both time and space consumption. This is because the key factor affecting the complexity is the Hessian-vector product (HVP) computations when backpropagating the metagradient ∇ θ L( φt (θ); D val t ). Since all the methods relied on a K-step GD to obtain φt (θ) (where MC adopted an additional Kronecker-factorized preconditioner [37] ), the complexity for computing this meta-gradient is O(Kd) in both time and space.Injective Sylvester NF Target pdf 𝑞(⋅) Normalizing flows (NFs). Our NCoV is similar to conventional NFs in the sense that they are both distribution models relying on the change-of-variable formula. Nevertheless, NCoV is distinct from NF in four key aspects: expressiveness, tractability, training strategy, and application fields.i) First, NCoV has been proved capable of modeling an arbitrary pdf, whereas NF is restricted to pdfs with full support. ii) Second, while NFs aim for closed-form pdf (3), NCoV sacrifices its tractability for augmented expressiveness. iii) Third, NFs are typically trained in an unsupervised manner by maximizing the likelihood function p f (Z) over a dataset sampled from target pdf q. In contrast, NCoV requires latent optimization (10) given the intractability of likelihood. iv) Finally, NFs can be thus utilized in applications including probability estimation and sample generation, while NCoV is tailored specifically to learn a prior over model parameters in the meta-learning context.Non-injective transformations. Moreover, it is worth mentioning that non-injective transformations have been also considered by [34] to enhance the expressiveness. While [34] allows Z and Z ′ to have different dimensions, it does not provides theoretical guarantees of its expressiveness. In contrast, the design of our MetaNCoV is based on Theorem 3.1 that requires f being a mapping from R d to R d . In addition, other works such as [20] also forgo the analytical invertibility, but typically necessitate an efficient approximation of f -1 .Bayesian meta-learning. Our approach is also related to Bayesian meta-learning [60, 40, 59, 11] . While these approaches aim to quantify the uncertainties in ϕ t by identifying its posterior p(ϕ|y trn t ; X trn t , θ) ∝ p(y trn t |ϕ t ; X trn t )p(ϕ t ; θ), our MetaNCoV is deterministic by formulating the task-level optimization (10b) as a MAP problem. In addition, these methods also rely on tractable prior and (surrogate) posterior pdfs of prefixed forms such as Gaussian. For example, [60] relies on a predefined conjugate prior over the extracted features (rather than parameters ϕ t ) to ensure the tractability of the feature posterior. In [40, Section 3.3] , the surrogate (variational) posterior is prespecified as a diagonal Gaussian distribution, while the prior is fixed to be the Gaussian-Gamma form. For [59] , particle sampling with SVGD is utilized to parameterize and optimize the prior, where RBF kernels (i.e., Gaussian kernels) are selected to interpolate the particles. As also noted in [11, Section 4.2] , both the prior and surrogate posterior distributions are predefined as diagonal Gaussian forms. In contrast, our approach forgoes the tractability for enhanced prior expressiveness.Optimal transport. Lastly, the introduced NCoV model is related to the optimal transport problem in statistics [54] , which aims to minimize the total cost of transporting one distribution to another. However, the existence of the transport maps f can impose strong assumptions on both the source and target distributions, and is closely connected to the choice of cost function. For instance, the well-known Brenier's theorem suggests that f exists when Z and Z ′ have finite second moments (or, q has a compact support), and Z assigns no mass to any set of Hausdorff dimension d -1 [53] . In comparison, Theorem 3.1 puts constraints only on the source random vector Z, as we can choose P Z flexibly to suit our needs. This enables P Z ′ to match an arbitrary target Q. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: All detailed experimental setups including hyperparameters and optimizers are included in Appendix C.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: 95% confidence interval of the averaged classification accuracy has been adopted as the uncertainty metric in all the tables.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: This research project conforms every aspect of the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This work mainly focuses on algorithm design and its theoretical guarantees. There is no malicious societal impact or direct path to any negative applications.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The data and model used in this paper are from safe public databases available online.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The original owners of the assets used in this paper have been properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: A "readme" document is included in the supplementary submission.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Justification: There is no crowdsourcing nor human subjects in this research projection.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Justification: There is no crowdsourcing nor human subjects in this research projection, and thus no necessity for IRB approvals.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.38th Conference on Neural Information Processing Systems (NeurIPS 2024).Conclusions and outlookAn informative prior plays a crucial role in training a large-scale model with limited small-scale data. This work introduced a novel NCoV model for learning an expressive task-invariant prior. By transforming a known pdf of a continuous random vector, the NCoV model enables a large family of target pdfs. As a flexible plug-in prior model, our MetaNCoV method offers enhanced prior expressiveness compared to existing meta-learning methods that rely on preselected prior pdfs. Numerical studies validate our theoretical analysis, and highlight the superior performance of the proposed method, especially when datasets are scarce. Our future research agenda includes i) investigation of more generic universal approximation theorems; ii) bilevel convergence analysis for the MetaNCoV method; and, iii) implementation of MetaNCoV with alternative transformations, backbone models, and meta-learning methods. 
 
 Appendix:  


Decision: **Summary of the paper:**
The paper advocates for learning a flexible (non-prescribed) prior in meta learning and achieves this by the change of variable trick from a fixed prior; differently to NFs, however, without requiring the invertibility. Particularly, it works on optimization-based meta learning and uses Sylvester NF without enforcing the Jacobian structure. It evaluates the method on several few-shot learning benchmarks of meta learning. 

**Summary of the reviews:**
The reviewers found the paper clearly written and well-motivated, the theoretical and empirical results rigorously achieved, and the empirical results to be significantly improving over the baselines with fixed form of priors. 

On the other hand, they were concerned about limited benchmarks and novelty over works within Bayesian (or generally probabilistic) meta learning.

**Summary of the rebuttal and discussions:**
In the rebuttal period, the authors provided additional results on a lower-scale version of MetaDataset for cross-domain generalization where the trends are seen to be similar. They also argue for the MAP aspect of their work when optimizing with the learned prior on test tasks which trades-off the uncertainty quantification of Bayesian meta learning for a tractable expressivity of their learnt priors.

**Consolidation report:**
The authors successfully addressed the main criticisms of the reviewers that led to their eventual leaning towards acceptance.

**Recommendation:**
The AC agrees with the unanimous rating and suggests acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes to use non-injective change of variables theorem to make the prior in meta-learning more flexible than fixed shaped ones that have previously been used. Abundent theoretical analysis and experimental results support their claim that more flexible meta-level prior pdf can significantly boost the performance of meta-learning in a few simple benchmark settings.
strength_weakness: strength_weakness: Too limited benchmarks. While the idea and the theoretical analysis are nice, the benchmarks used in this paper -- mini-Imagenet, tieredImageNet, and CUB are too outdated in my opinion. They were popular in 5 years ago, and nowadays people are actually no longer interested in these benchmarks, especially in the LLM era. I think the minimum level should be meta-dataset. Could you provide the results on meta-dataset?
questions: questions: I wonder about the positioning of this paper. Does the proposed method have to be specifically tailored to meta-learning only? Why don't you broaden the scope of the paper by doing experiments on more standard non meta-learning setup (like in the synthetic experiments)?
limitations: limitations: The authors partially addressed the limitations of their work.
paper_summary: paper_summary: One of representative optimization-based meta-learning method is Model-agnostic meta-learning (MAML), where the inner loop can be interpreted as solving MAP. Here, the prior distribution over model parameters is defined as Gaussian pdf and the shared initialization is the mean parameter of the Gaussian pdf. The choice of Gaussian distribution for prior distribution may lack expressiveness; therefore, this paper proposes a new meta-learning method by introducing non-injective change-of-variable (NCoV) model for prior distribution over task parameters. Furthermore, the proposed method does not need to meta-learn shared initialization as done in MAML by feed-forwarding the zero vector into the normalizing flow (choosing the base distribution of normalizing flow as standard Gaussian). The empirical results shows the efficacy of proposed method in few-shot learning settings.
strength_weakness: strength_weakness: - This paper does not provide the number of "meta-"parameters for each method in the experiment section. I wonder whether the improvement comes from the expressiveness of proposed method or the sufficient number of parameters $\theta_f$ for modeling normalizing flow.

- This paper does not discuss relevant literature in main paper (does not include related work section). Especially, I am curious about the novelty of NCoV models derived in this paper in comparison to the literature of normalizing flows. If NCoV models have already been studied in the literature, this paper seems to lack novelty. If not, I would like to recommend the authors to include related work or discussion to highlight their contribution with respect to normalizing flows.

- This paper follows experimental setups for the convention of meta-learning, but the empircial validation is conducted in small-scale settings. This paper argue that the proposed method is more scalable to dimensionality of model parameters, therefore, it would be better to highlight the argument by conducting experiments on more larger models (e.g., VIT).
questions: questions: Please see weakness section.
limitations: limitations: This paper discuss limitation in the conclusion section.
paper_summary: paper_summary: This paper proposes a novel approach to meta-learning using normalizing flows for modelling the prior distribution of parameters, with the task solver in the inner loop as the maximum a posterior estimator. Compared with the previous methods, such as MAML, which assume a Gaussian prior, suffering from limited expressiveness, non-injective Sylvester normalizing flows (NFs) are utilised for more flexibility. The method involves propagating samples for each task through the flow and updating the flow based on an average overall task. Experimental results show that this approach consistently outperforms some existing meta-learning techniques.
strength_weakness: strength_weakness: 1. I find the non-injective Sylvester Flows part confusing. In traditional Sylvester Flow models, a specific structure is used to ensure invertibility by imposing conditions on the diagonals of triangular matrices. But in this work, I do not see it. 

2. In line 256, the authors claim that while optimization Eq.10b, the latent variables share similarities with LEO, which is confusing. The latent variable in LEO is introduced by its specific architecture and in this submission it refers very different thing. Can the authors explain more about this? 

3. [1] handles the prior learning problem for meta-learning from a similar perspective and outperforms MetaNCoV in some settings. 

[1] Zhang X, Meng D, Gouk H, Hospedales TM. Shallow bayesian meta learning for real-world few-shot recognition. In Proceedings of the IEEE/CVF international conference on computer vision 2021 (pp. 651-660).
questions: questions: See the weaknesses section
limitations: limitations: See the weaknesses section
paper_summary: paper_summary: The paper proposes to learn a more expressive parameter prior compared to predefined ones for a meta-learner. The problem is formulated in a Bayesian model-agnostic meta-learning context and its solution leverages a class of non-invective normalising flows to learn the parameter prior (aka Sylvester flows). Experiments are conducted on mini-ImageNet and CUB datasets, by comparing the proposed approach applied to existing meta-learning algorithms against their base counterparts, showcasing the benefits of an adaptive prior in terms of few-shot classification performance.
strength_weakness: strength_weakness: 1. One of the major concern with the proposed solution is in its novelty and therefore its contribution **Novelty**. There are several variants of probabilistic formulations for model-agnostic meta-learning with corresponding learnt prior. How does the proposed solution differ from them and what is therefore the novel contribution? See for instance [1-3].
2. Experiments miss important baselines **Quality**. See for instance [1-3].

**Reference**

[1] Amortised Bayesian Meta-Learning. ICLR 2019

[2] Bayesian Model-Agnostic Meta-Learning. NeurIPS 2018

[3] Probabilistic Model-Agnostic Meta-Learning. NeurIPS 2018
questions: questions: In addition to addressing the two above-mentioned weaknesses, can you elaborate more on how the model handles uncertainty, given its probabilistic nature?
limitations: limitations: An important limitation of the proposed solution is its computational complexity. A thorough evaluation and discussion about the trade-off between computation and performance should be also provided.

Comments: title: Answer
Answer
Answer
comment: Thank you for your constructive questions and suggestions. We will incorporate the points discussed above into our revised manuscript.
Thank you for your valuable time and insightful questions, as well as for updating the rating.
I thank the authors for their efforts in addressing my concerns in terms of weaknesses 1 and 2. I still hold different ideas about the comparison with [1] but I will change my score to borderline accept.
Dear authors

I thank the authors for the effort in clarifying my concerns. I will change my score to weak accept.
Thank you for your time and insightful suggestions, as well as for raising the score. We will incorporate the above discussions and comparisons into the revised version of our manuscript.
Dear authors,

thank you for the additional clarifications and experiments.
There are no additional questions from my side. If you include these experiments and the discussion provided in the answers in the paper, I think this will contribute to make the paper more solid and convincing.
I'm going to raise my score and I want also to extend my congratulations for the hard work.
Thank you for your follow-up questions, which are addressed in the following. 

**Regarding prefixed priors in probabilistic meta-learning**

In [Section 3.3, 1], the surrogate (variational) posterior is prespecified as a diagonal Gaussian distribution, while the prior is fixed to be the Gaussian-Gamma form. For [2], particle sampling with SVGD is utilized to parameterize and optimize the prior, where RBF kernels (i.e., Gaussian kernels) are selected in Section 5 to interpolate the particles. As also noted in [Section 4.2, 3] and [Algorithm 1, 3], both the prior and surrogate posterior distributions are predefined as diagonal Gaussian forms. 

**Regarding the exclusion of probabilistic methods**

The primary rationales for excluding these probabilistic methods is their high computational complexities compared to deterministic ones. In particular, [1] and [3] rely on variational inference, which requires sampling from the surrogate posterior $q(\boldsymbol{\phi}_t; \mathbf{v}_t)$ (with variational parameter $\mathbf{v}_t$) at each optimization step to calculate the expected NLL $\mathbb{E} _{q(\boldsymbol{\phi}_t; \mathbf{v}_t)} -\log p (\mathbf{y}_t^\mathrm{trn} | \boldsymbol{\phi}_t; \mathbf{X}_t^\mathrm{trn})$. In [2], a group of particles is maintained to parameterized the parameter distribution. As a result, these probabilistic methods [1-3] suffer from $M\times$ time- and space-complexity compared to deterministic ones, where $M$ is the number of samples. 

However, it is worth noting that our MetaNCoV consistently outperforms these probabilistic methods by a significant margin. Please refer to the table below for a comparison on the 5-class miniImageNet dataset. 

|Method|1-shot|5-shot|
|-|-|-|
|ABML [1]|$45.0_{\pm 0.6}$|$62.8_{\pm 0.74}$|
|BMAML [2]|$53.80_{\pm 1.46}$|not provided|
|PLATIPUS [3]|$50.13_{\pm 1.86}$|not provided|
|MAML+MetaNCoV (ours)|$\mathbf{57.74}_{\pm 1.47}$|$\mathbf{70.72}_{\pm 0.70}$|
Thank you for the clarifying answers, which have addressed some of my questions.  I have some additional questions: Could you please elaborate more on the "prefixed forms" for the above-mentioned probabilistic baselines? Specifically, is the Gaussian assumption imposed both at the prior and posterior level? Moreover, what is the rationale for not considering the above-mentioned methods as important baselines for comparison?