Year: 2021

Abstract:
Learning to infer the conditional posterior model is a key step for robust metalearning. This paper presents a new Bayesian meta-learning approach called Neural Variational Dropout Processes (NVDPs). NVDPs model the conditional posterior distribution based on a task-specific dropout; a low-rank product of Bernoulli experts meta-model is utilized for a memory-efficient mapping of dropout rates from a few observed contexts. It allows for a quick reconfiguration of a globally learned and shared neural network for new tasks in multi-task few-shot learning. In addition, NVDPs utilize a novel prior conditioned on the whole task data to optimize the conditional dropout posterior in the amortized variational inference. Surprisingly, this enables the robust approximation of task-specific dropout rates that can deal with a wide range of functional ambiguities and uncertainties. We compared the proposed method with other meta-learning approaches in the few-shot learning tasks such as 1D stochastic regression, image inpainting, and classification. The results show the excellent performance of NVDPs.

heading: 1 INTRODUCTION
text: In traditional machine learning, a large amount of labeled data is required to train deep models (LeCun et al., 2015). In practice, however, there are many cases where it is impossible to collect sufficient data for a given task. On the other hand, humans can quickly understand and solve a new task even from few examples (Schmidhuber, 1987; Andrychowicz et al., 2016). This distinguishing characteristic of humans is referred to as the meta-learning ability, which enables them to accumulate the past learning experiences into general knowledge and to utilize it for efficient learning in the future. Incorporating the meta-learning capability into artificial machines has long been an active research topic of machine learning (Naik & Mammone, 1992; Vinyals et al., 2016; Snell et al., 2017; Ravi & Larochelle, 2017; Finn et al., 2017; Lee et al., 2019c; Hospedales et al., 2021).
Recently, Bayesian meta-learning methods have been attracting considerable interest due to incorporating the uncertainty quantification of Bayesian frameworks into the efficient model adaptation of meta-learning approaches. An earlier study (Grant et al., 2018) extended the optimization-based deterministic approach model-agnostic meta-learning (MAML) (Finn et al., 2017) to a hierarchical Bayesian frameworks (Daumé III, 2009). Later, the optimization-based variational inference (Yoon et al., 2018; Finn et al., 2018; Ravi & Beatson, 2019; Lee et al., 2019a; Nguyen et al., 2020) and the model-based Bayesian meta-learning (Gordon et al., 2018; Garnelo et al., 2018a;b; Iakovleva et al., 2020) methods were presented. These methods achieved outstanding outcomes in various few-shot1 regression and classification tasks. In fact, the adaptation of deep models that can estimate uncertainties is a key study for reliable machine learning systems in real-world applications with high judgmental risks such as medical AI or autonomous driving systems.
Inspired by the recent studies, we propose a new model-based Bayesian meta-learning approach, neural variational dropout processes (NVDPs). The main contribution of this work is to design a new type of Neural Network (NN) based conditional posterior model that can bypass the under-fitting and posterior-collapsing of the existing approaches (Gordon et al., 2018; Garnelo et al., 2018a;b; Iakovleva et al., 2020). NVDPs extend the simple yet effective posterior modeling of Variational Dropout (VD) (Kingma et al., 2015; Gal & Ghahramani, 2016; Molchanov et al., 2017; Hron et al., 2018; Liu et al., 2019) in the context of meta-learning; a novel low-rank product of Bernoulli experts metamodel is utilized in the dropout posterior to learn task-specific dropout rates conditioned on a few
1The few-shot learning assumes only a few examples are available for each task, with the number of tasks being large (Lake et al., 2015).
learning examples. In this way, a full conditional posterior model over the NN’s parameters can be efficiently obtained. In addition, we also propose a new type of prior to optimize the conditional dropout posterior in variational inference, which supports the robust training of conditional posteriors. Although it is developed in conjunction with NVDPs, the formulation allows its adoption to other methods, and we show the effectiveness. We have evaluated NVDPs compared with other recent methods on various few-shot learning tasks and datasets. The experiments show that NVDPs can circumvent the posterior-collapsing and under-fitting and achieve outstanding performances in terms of log-likelihood, active learning efficiency, prediction accuracy, and generalization.

heading: 2 BAYESIAN META-LEARNING

heading: 2.1 AMORTIZED VARIATIONAL INFERENCE FRAMEWORK IN THE MULTI-TASK DATA
text: A goal in meta-learning is to construct models that can quickly solve new tasks from small amounts of observed data. To achieve this, it is important to learn general (or task-invariant) structures from multiple tasks that can be utilized for the efficient model adaptation when necessary. Bayesian metalearning (Gordon et al., 2018; Garnelo et al., 2018b; Ravi & Beatson, 2019) formulates this objective of meta-learning as an amortized variational inference (VI) of the posterior distribution in multi-task environments. Suppose a collection of T related tasks is given, and each t-th task has the training data Dt containing N i.i.d. observed tuples (xt,yt) = (xti, yti)Ni=1. Then, the evidence lower-bound (ELBO) over the log-likelihood of the multi-task dataset can be derived as:
T∑ t=1 log p(Dt) ≥ T∑ t=1 {Eq(φt|Dt;θ)[log p(yt|xt, φt)]− KL(q(φt|Dt; θ)||p(φt))}. (1)
Here, the p(yt|xt, φt) is the likelihood (or NN model) on t-th training data and φt is t-th task-specific variable (i.e. a latent representation or weights of NN) and q(φt|Dt; θ) is a tractable amortized variational posterior model utilized for the approximation of the true unknown posterior distribution of φt for each given t-th task (i.e p(φt|Dt)) (Kingma & Welling, 2013; Gordon et al., 2018; Garnelo et al., 2018b; Ravi & Beatson, 2019; Iakovleva et al., 2020). The parameter θ represents the common structures that can be efficiently learned across multiple different tasks. The pre-specified prior distribution p(φt) in the KL term provides a stochastic regularization that can help to capture the task-conditional uncertainty and prevent the collapsing of q(φt|Dt; θ). In fact, the maximization of the the evidence lower bound (ELBO), right side of equation 1, with respect to the conditional variational posterior model is equivalent to the minimization of ∑T t=1 KL(q(φ
t|Dt; θ)||p(φt|Dt)). Essentially, the goal in the amortized VI of Bayesian meta-learning is to learn the inference process of the conditional posterior distribution via the variational model q(φt|Dt; θ) and the shared general structure θ across multiple tasks since these task-invariant knowledge can later be utilized for the efficient adaption of the NN function on new unseen tasks. The approximation of the conditional posterior also enables the ensemble model approaches and the uncertainty quantification.

heading: 2.2 THE EXISTING CONDITIONAL POSTERIOR AND PRIOR MODELINGS
text: Many of the recent Bayesian meta-learning approaches can be roughly categorized into the optimization-based (Grant et al., 2018; Yoon et al., 2018; Finn et al., 2018; Ravi & Beatson, 2019; Lee et al., 2019a; Nguyen et al., 2020) or the model-based posterior approximation approaches (Gordon et al., 2018; Garnelo et al., 2018a;b; Kim et al., 2019; Iakovleva et al., 2020). In the optimizationbased approaches, the task-specific variable φt can be seen as the adapted NN weights. For example, the deterministic model-agnostic meta-learning (MAML) (Finn et al., 2017) can be considered as learning a Dirac delta variational posterior modeling (i.e., q(φt|DtC ; θ) ≈ δ(φt − SGDj(DtC , θ))) where the goal is to learn the shared global initialization NN’s parameters θ such that a few j steps of SGD updates on the small subset2 DtC of the t-th dataset Dt provides a good approximation of the task-specific weights φt. Many optimization-based Bayesian meta-learning approaches such as LLAMA (Grant et al., 2018), PLATIPIS (Finn et al., 2018), BMAML (Yoon et al., 2018), and ABML (Ravi & Beatson, 2019) have incorporated Gaussian type of posterior and prior models into
2The subset DtC(⊆ Dt) is also known as a context set for each task. A small size S of context set (e.g., 1-shot, 5-shot, or random) is often used in the few-shot learning tasks (Lake et al., 2015).
the deterministic adaptation framework of MAML in order to improve the robustness of models. The optimization-based approaches are applicable to various types of few-shot learning tasks, but the adaptation cost at test time is computationally expensive due to the inversion of Hessian or kernel matrix; this may restrict their usage in the environments with low computational resources.
On the other hand, the model-based Bayesian meta-learning approaches such as VERSA (Gordon et al., 2018) or Neural Processes (Garnelo et al., 2018a;b) allow an instant estimation of the Bayesian predictive distribution at test time via NN-based conditional posterior modeling. VERSA employs an additional NN-based meta-model gθ(·) to directly predict the Gaussian posterior of agent NN’s task-specific weight φt from the small context set (i.e., q(φt|DtC ; θ) = N (φt|(µ, σ) = gθ(DtC))). In this case, the shared structure θ represents the meta-model’s parameters. However, the direct approximation of agent NN weights in VERSA could limit their scalability due to the large dimensionality of the weights; they only consider the task-specific weights for the single softmax output layer. In addition, VERSA did not utilize any task-specific prior p(φt), so the conditional posterior could turn into a deterministic one while training with the Monte Carlo approximation (Iakovleva et al., 2020). Instead of the NN’s weights, the Neural Processes (NPs) model the conditional posterior with a latent representation (i.e. q(zt|Dt; θ) = N (zt|(µ, σ) = gθ(Dt))) (Garnelo et al., 2018b). Regarding the task-specific latent representation as a second input to the likelihood p(yt|xt, zt) (similar to Kingma & Welling (2013) or Edwards & Storkey (2017)) provides an efficient way to transfer the conditional information (Garnelo et al., 2018b). However, the following studies indicate that NPs often suffer from under-fitting (Kim et al., 2019) and posterior-collapsing (Grover et al., 2019; Le et al., 2018). We conjecture that these behaviors are partially inherent to its original variational formulation; the conditional posteriors of NPs depend on the whole task data while training, which did not empirically show good generalization performance at test. In addition, it is well known that the highly flexible NN-based likelihood can neglect the latent representation in variational inference (Hoffman & Johnson, 2016; Sønderby et al., 2016; Kingma et al., 2016; Chen et al., 2017; Yeung et al., 2017; Alemi et al., 2018; Lucas et al., 2019; Dieng et al., 2018); we also demonstrate the posterior collapsing of the NPs in our experiments.

heading: 3 NEURAL VARIATIONAL DROPOUT PROCESSES
text: This section introduces a new model-based Bayesian meta-learning approach, called Neural Variational Dropout Processes (NVDPs). Unlike the existing methods such as NPs or VERSA employing conditional latent representation or direct modeling of NN’s weights, NVDPs extend the posterior modeling of the variational dropout in the context of meta-learning. We also introduce a new type of task-specific prior for optimizing the conditional dropout posterior in variational inference.

heading: 3.1 A CONDITIONAL DROPOUT POSTERIOR
text: Variational dropout (VD) (Kingma et al., 2015; Gal & Ghahramani, 2016; Molchanov et al., 2017; Hron et al., 2018; Liu et al., 2019) is a set of approaches that models the variational posterior distribution based on the dropout regularization technique. The dropout regularization randomly turns off some of the Neural Network (NN) parameters during training by multiplying the discrete Bernoulli random noises to the parameters (Srivastava et al., 2014; Hinton et al., 2012). This technique was originally popularized as an efficient way to prevent NN model’s over-fitting. Later, the fast dropout (Wang & Manning, 2013; Wan et al., 2013) reported that multiplying the continuous noises sampled from Gaussian distributions works similarly to the conventional Bernoulli dropout (Srivastava et al., 2014) due to the central limit theorem. The Gaussian approximation of the Bernoulli dropout is convenient since it provides a fully factorized (tractable) posterior model over the NN’s parameters (Wang & Manning, 2013; Wan et al., 2013). In addition, VD approaches often utilize sparse priors for regularization (Kingma et al., 2015; Molchanov et al., 2017; Hron et al., 2018; 2017; Liu et al., 2019). This posterior and prior distribution modeling in VD enables the learning of each independent dropout rates over the NN parameters as variational parameters. This distinguishes the VD approaches from the conventional dropouts that use a single fixed rate over all parameters. However, the dropout posterior in the conventional VD becomes fixed once trained and it is unable to express the conditional posterior according to multiple tasks.
Conditional dropout posterior. We propose a new amortized variational posterior model that can be efficiently adapted for each given task. Suppose we train a fully connected NN of L layers, then a conditional dropout posterior3 based on the task-specific dropout rates Pt over the K × D dimensional deterministic parameters θ of each l-th layer of the NN can be given as follows:
q(φt|DtC ; θ) = K∏ k=1 D∏ d=1 q(φtk,d|DtC) = K∏ k=1 D∏ d=1 N (φtk,d|(1−Ptk,d)θk,d,Ptk,d(1−Ptk,d)θ2k,d). (2)
The parameter4 θk,d is shared across different tasks, representing the common task-invariant structure. Then the task-specific NN parameter φt are fully described by the mean and variance of each independent Gaussian distribution via θk,d and Pk,d. Note that the variational posterior is explicitly conditioned on the subset of the t-th training dataset DtC = {xti,yti}Si=1 (⊆ Dt) known as the t-th context set. The key idea of conditional posterior modeling in NVDPs is to employ a NNbased meta-model to predict the task-specific dropout rate Ptk,d from the small context set DtC . The meta-model learning to approximate Ptk,d for each given task is defined as:
Ptk,d = s(ak) · s(bd) · s(c), where (a,b, c) = gψ(rt). (3) The set representation rt is defined as the mean of features obtained from each data in t-th context set DtC (i.e., rt = ∑S i=1hω(x t i,y t i)/S, where hω is a feature extracting NN parameterized by ω), summarizing order invariant set information (Edwards & Storkey, 2017; Lee et al., 2019b). The gψ(·) is Meta NN model parameterized by ψ to predict a set of logit vectors (i.e., a ∈ RK ,b ∈ RD and c ∈ R). Then, the sigmoid function with a learnable temperature parameter τ (i.e., sτ (·) = 1/(1 + exp(−(·)/τ) ) is applied to them to get the low-rank components of task-specific dropout rates Ptk,d: the row-wise s(ak), column-wise s(bd), and layer-wise dropout rate s(c). In other words, the task-specific dropout rate Ptk,d is obtained by multiplying low-rank components of the conditionally approximated dropout rates from the meta-model gψ(DtC) (see Figure1 (a)). The product of n Bernoulli random variables is also a Bernoulli variable (Leemis, 2020). By exploiting this, we interpret the approximation of the task-specific dropout rates in terms of the low-rank product of Bernoulli experts. Unlike VERSA whose meta-model’s complexity isO(LKD) to model the full NN weight posterior directly, the complexity of meta-model in NVDPs is O(L(K+D+1)); the meta-model’s role is only to predict the low-rank components of the task-specific dropout rates. With the shared parameter θ, this can greatly reduce the complexity of posterior distribution approximation of the high dimensional task-specific weights from only a few learning examples. The product model tends to give sharp probability boundaries, which is often used for modeling the high dimensional data space (Hinton, 2002).

heading: 3.2 THE TASK-SPECIFIC VARIATIONAL PRIOR
text: To optimize the conditional dropout posterior in equation 2, a specification of the prior distribution p(φt) is necessary to get a tractable derivation of the KL regularization term of the ELBO in equation 1. The question is how we can define the effective task-specific prior. In fact, one important
3The original Gaussian approximation in the VD is q(φk,d) = N (φk,d|θk,d, αθ2k,d) with α = Pk,d/(1 − Pk,d). But, NVDPs extend the Bernoulli dropout model (Wang & Manning, 2013; Wan et al., 2013).
4We omit the layer index l of the parameter θl,k,d for brevity.
requirement in the choice of the prior distribution in the conventional VD framework (Kingma et al., 2015) is that the analytical derivation of the KL term (i.e., KL(q(φt|DtC ; θ)||p(φt)) in equation 1) should not depend on the deterministic NN parameter θ. This allows the constant optimization of the ELBO w.r.t all the independent variational parameters (i.e., θl,k,d and Pl,k,d for all l = 1 . . . L, k = 1 . . .K and d = 1 . . . D). One way of modeling the prior is to employ the log-uniform prior p(log(|φ|) ∝ c as in the conventional VD (Kingma et al., 2015). However, a recently known limitation is that the log-uniform prior is an improper prior (e.g., the KL divergence between the posterior and the log-uniform prior is infinite). This could yield a degeneration of the dropout posterior model to a deterministic one (Molchanov et al., 2017; Hron et al., 2018; 2017; Liu et al., 2019). Besides, the conventional prior used in VD approaches does not support the task-dependent regularization.
Variational prior. We introduce a new task-specific prior modeling approach to optimize the proposed conditional posterior; it is approximated with the variational prior defined by the same dropout posterior model in equation 2 except that the model is conditioned on the whole task data (i.e. p(φt) ≈ q(φt|Dt)). The KL divergence between the conditional dropout posterior (with the only context set) and the variational prior (with the whole task data) can be derived as: KL(q(φt|DtC)||q(φt|Dt)) = K∑ k=1 D∑ d=1 { Ptk,d(1−Ptk,d) + (P̂tk,d −Ptk,d)2 2P̂tk,d(1− P̂tk,d) + 1 2 log P̂tk,d(1− P̂tk,d) Ptk,d(1−Ptk,d) }
(4)
where both Pk,d and P̂k,d are the dropout rates predicted from the meta-model but with different conditional set information: Pk,d is obtained from the small context set DtC , while P̂k,d is from the whole task set Dt. Interestingly, the analytical derivation of the KL is independent of the shared parameter θ, thus this satisfies the necessary condition to be used as a prior in the VI optimization of the dropout posterior. Figure 1(b) depicts the variational prior is conditioned on the whole dataset.
The shared posterior model for the task-specific prior introduced here was inspired by recent Bayesian meta-learning approaches (Garnelo et al., 2018b; Kim et al., 2019; Iakovleva et al., 2020), but some critical differences are that: 1) we have developed the variational prior to regularize the task-specific dropout rates in the optimization of the conditional dropout posterior, 2) the denominator and the numerator of the KL divergence term in equation 4 are reversed compared with the existing approaches. In fact, some other recent studies of the amortized VI inference (Tomczak & Welling, 2018; Takahashi et al., 2019) analytically derived that the optimal prior maximizing VI objective is the variational posterior aggregated on the whole dataset: p∗(φ) = ∫ D q(φ|D)p(D). Thus, we hypothesized the conditional posterior model that depends on the whole dataset should be used to approximate the optimal prior p∗(φt) ≈ q(φt|Dt) since the variational model conditioned on the aggregated set representation with much larger context (e.g.,Dt ⊇ DtC) is likely to be much closer to the optimal task-specific prior than the model conditioned only on a subset. The experiments in Section 5 demonstrate that the variational prior approach provides a robust regularization for the conditional dropout posterior and the similar formulation is also applicable to the latent variable based conditional posterior models (Garnelo et al., 2018b; Kim et al., 2019).

heading: 3.3 STOCHASTIC VARIATIONAL INFERENCE
text: With the conditional dropout posterior defined in equation 2 and the KL regularization term in equation 4, we can now fully describe the ELBO objective of NVDPs for the multi-task dataset as:
T∑ t=1 log p(Dt) ≥ T∑ t=1 {Eq(φt|DtC ;θ)[log p(y t|xt, φt)]− KL(q(φt|DtC ; θ)||q(φt|Dt; θ))}. (5)
The goal is to maximize the ELBO equation 5 w.r.t. the variational parameters (i.e. θ, ψ, ω, and τ ) of the posterior q(φt|DtC ; θ) defined in equation 2. The optimization of these parameters is done by using the stochastic gradient variational Bayes (SGVB) (Kingma & Welling, 2013; Kingma et al., 2014). The basic trick in the SGVB is to parameterize the random weights φt ∼ q(φt|Dt) using a deterministic differentiable transformation φt = f( ,DtC) with a non-parametric i.i.d. noise ∼ p( ). Then, an unbiased differentiable minibatch-based Monte Carlo estimator, L̂(θ, ψ, ω, τ),
of the ELBO of NVDPs can be defined as:
T
T ′ T ′∑ t=1 {N M M∑ i=1 log p(yti |xti, f( ,DtC)− ( Pt(1−Pt) + (P̂t −Pt)2 2P̂t(1− P̂t) + 1 2 log P̂t(1− P̂t) Pt(1−Pt) )}. (6)
T ′ and M are the size of randomly sampled mini-batch of tasks and data points, respectively, per each epoch (i.e.,{xti,yti}Mi=1 = Dt ∼ {Dt}T ′
t=1 ∼ D). DtC is the subset set of Dt discussed in section 3.1 and 3.2. The transformation of the task-specific weights is given as φtk,d = f( k,d,DtC) = (1 − Ptk,d)θk,d + √ Ptk,d(1−Ptk,d)θk,d k,d with k,d ∼ N(0, 1) from the equation 2. The inter-
mediate weight φtk,d is now differentiable with respect to θk,d and P t k,d. Also, P t k,d (and P̂ t k,d) is deterministically computed by the meta-model parameterized by ψ, ω, and τ as in equation 3. Thus, the estimator L̂(θ, ψ, ω, τ) in equation 6 is differentiable with respect to all the variational parameters and can be optimized via the SGD algorithm. In practice, a local reparameterization trick5 is further utilized to reduce the variance of gradient estimators on training (Kingma et al., 2015).

heading: 4 RELATED WORKS
text: Gaussian Processeses (GPs) (Rasmussen & Williams, 2005) are classical Bayesian learning approaches that are closely related to NVDPs; GPs allow an analytical derivation of the posterior predictive distribution given observed context points based on Gaussian priors and kernels. However, classical GPs require an additional optimization procedure to identify the suitable kernel for each task and the time complexity is quadratic to the number of contexts. Recently introduced model-based Bayesian meta-learning methods (Garnelo et al., 2018b;a; Kim et al., 2019; Gordon et al., 2018; Iakovleva et al., 2020) offer NN-based conditional posterior models that can be efficiently learned from data and later provide an instant posterior predictive estimation at test time. One important aspect that these model-based methods have in common is the utilization of permutation invariant set representation as an input to the meta-models (Edwards & Storkey, 2017; Lee et al., 2019b; Bloem-Reddy & Teh, 2020; Teh & Lecture, 2019), which makes conditional posterior models invariant to the order of the observed context data. This property, known as exchangeability, is a necessary condition to be a stochastic process according to Kolmogorov’s Extension Theorem (Garnelo et al., 2018b; Kim et al., 2019; Øksendal, 2003). NVDPs also use the set representation to approximate the task-specific dropout rates in the conditional dropout posterior. Another related work to our NVDP approach is Meta Dropout (Lee et al., 2019a). They also consider a novel input-dependent dropout approach; Gaussian-based input dependent stochastic perturbation is applied to each pre-activation features of NN functions. Their approach is constructed based on optimization-based Bayesian meta-learning approaches (Finn et al., 2017; Li et al., 2017) without directly considering the set representation. On the contrary, NVDPs extend the dropout posterior model of Variational Dropout (VD) (Kingma et al., 2015; Wang & Manning, 2013; Wan et al., 2013; Gal & Ghahramani, 2016; Liu et al., 2019; Molchanov et al., 2017) to conditional models and propose a new type of variational prior in the VI framework of Bayesian meta-learning (Garnelo et al., 2018b; Kim et al., 2019). Gal & Ghahramani (2016) discussed the link between VD and GPs.

heading: 5 EXPERIMENTS
text: Metrics in regression. In the evaluation of the conditional NN models’ adaptation, the newly observed task data D∗ is split into the input-output pairs of the context set D∗C = {xi, yi}Si=1 and the target set D∗T = {xi, yi}Ni=1 (D∗C 6⊆ D∗T in evaluation (Garnelo et al., 2018b; Gordon et al., 2018)). (1) The log-likelihood (LL), 1N+S ∑ i∈D∗C∪D∗T Eq(φ∗|D∗C)[log p(yi|xi, φ ∗)], measures the performance of the NN model over the whole dataset D∗ conditioned on the context set. (2) The reconstructive log-likelihood (RLL), 1S ∑ i∈D∗C Eq(φ∗|D∗C)[log p(yi|xi, φ ∗)], measures how well the model reconstructs the data points in the context set. A low RLL is a sign of under-fitting. (3) The predictive log-likelihood (PLL), 1N ∑ i∈D∗T Eq(φ∗|D∗C)[log p(yi|xi, φ ∗)], measures the prediction on the data points not in the context set (i.e. the target set). A low PLL is a sign of over-fitting. 5We can sample pre-activations Bm,d for a mini-batch of size M directly using inputs Am,k (Bm,d ∼
N (Bm,d| ∑K k=1Am,k(1−P t k,d)θk,d, ∑K k=1A 2 m,kP t k,d(1−Ptk,d)θ2k,d)).
GP Samples. The 1D regression task is to predict unknown functions given some observed context points; each function (or data point) is generated from a Gaussian Process (GP) with random kernel, then the standard data split procedure is performed (i.e. S ∼ U(3, 97) and N ∼ U [S + 1, 100)) at train time. For the baseline models, Neural Process model (NP) and NP with additional deterministic representation (NP+CNP) described in (Garnelo et al., 2018b;a; Kim et al., 2019) is compared. We also adopted the variational prior (VP) into the representation based posterior of NP and its variants (NP+VP and NP+CNP+VP). For all models (including our NVDP), we used the same fixed variance and learned variance likelihood architecture depicted in (Kim et al., 2019): the agent NN with 4 hidden layers of 128 units with LeLU activation (Nair & Hinton, 2010) and an output layer of 1 unit for mean (or an additional 1 unit for variance). The dimensions of the set representation rt were fixed to 128. The meta NNs in the conditional dropout posterior in NVDP has 4 hidden layers of 128 units with LeakyReLU and an output layer of 257 units (i.e. K+D+1) for each layer of the agent model. All models was trained with Adam optimizer (Kingma & Ba, 2015) with learning rate 5e-4 and 16 task-batches for 0.5 million iterations. On the validation, 50000 random tasks (or functions) were sampled from the GP function generator, and the split data of S ∼ U(3, 97) and N = 400−S were used to compute the log-likelihood (LL) and other evaluation metrics.
Table 1 summarizes the validation results of 1D regression with the GP dataset. NVDP achieves the best LL scores compared with all other baselines in both the fixed and learned variance likelihood model settings. The NVDPs record high RLL on the observed data points and excellent PLL scores on the unseen function space in the new task; this indicates that the proposed conditional dropout posterior approach can simultaneously reduce the under-fitting and over-fitting of agent model compared with the other baselines. When VP is applied to NP or NP+CNP, the PLL scores tend to increase by meaningful margins in all cases. This supports that the proposed variational prior (VP) approach can also reduce the over-fitting of latent representation-based conditional posterior. Figure 2 visualizes the few-shot 1D function regression results in both model settings. In the fixed variance setting, the functions sampled from NPs show high variability but cannot fit the context data well. NP+CNP can fit the context data well but loses epistemic uncertainty a lot due to the collapsing of the conditional posterior. On the other hand, the functions from NVDPs show a different behavior; they capture the function’s variability well while also fitting the observed context points better. NVDPs also approximate the mean and variance of unknown function well in learned variance setting.
Active learning with regression. To further compare the uncertainty modeling accuracy, we performed an additional active learning experiment on the GP dataset described above. The goal in active learning is to improve the log-likelihood of models with a minimal number of context points. To this end, each model chooses additional data points sequentially; the point with maximal variance across the sampled regressors were selected at each step in our experiment. The initial data point is randomly sampled within the input domain, followed by 19 additional points that are selected according to the variance estimates. As seen in Figure 3, NVDPs outperform the others due to their accurate variance estimation especially with a small number of additional points, and show steady improvements with less under-fitting.
Image completion tasks. The image completion tasks are performed to validate the performance of the models in the more complex function spaces (Garnelo et al., 2018b;a; Kim et al., 2019). Here, we treat the image samples from MNIST (LeCun et al., 1998) and CelebA (Liu et al., 2015) as unknown functions. The task is to predict a mapping from normalized 2D pixel coordinates xi (∈ [0, 1]2) to pixel intensities yi (∈ R1 for greyscale, ∈ R3 for RGB) given some context points. We used the same learned variance baselines implemented in the GP data regression task (except the 2D input and 3D output for the agent NN and rt = 1024 were used for CelebA). At each iteration, the images in the training set are split into S context and N target points (e.g., S ∼ U(3, 197), n ∼ U [N + 1, 200) at train and S ∼ U(3, 197), N = 784− S at validation). Adam optimizer with a learning rate 4e-4 and 16 task-batches with 300 epochs were used for training. The validation is performed on the separated validation set. To see the generalization performance on a completely new dataset, we also tested the models trained on MNIST to the Omniglot validation set.
Table 2 summarizes the validation results of image completion tasks. NVDPs achieve the outperforming LLs compared with all other baselines. Interestingly, NVDPs (trained on the MNIST
dataset) also achieve the best results on the Omniglot dataset. Figure 4 shows the image reconstruction results with a varying number of random context pixels. NP generated various image samples when the number of contexts was small (e.g.,m <= 30 or half), but those samples could not approximate the true unknown images well compared with the other models. The NP+CNP achieves crisp reconstruction results compared with NP and also shows a good generalization performance on the Omniglot dataset, but the sampled images (or functions) from NP+CNP had almost no variability due to its posterior collapsing behavior. On the other hand, the samples from NVDPs exhibit comparable reconstruction results while also showing a reasonable amount of variability. In addition, NVDPs also present an outstanding generalization performance on the unseen Omniglot dataset. This implies that NVDPs not only fit better in the complex function regression but can also capture more general knowledge that can be applied to new unseen tasks.
Few-shot classification tasks. NVDPs can also be successfully applied for few-shot classification tasks; we have tested NVDPs on the standard benchmark datasets such as Omniglot (Lake et al., 2015) and MiniImagenet (Ravi & Larochelle, 2017) with other baselines: VERSA (Gordon et al., 2018), CNP (Garnelo et al., 2018a), Matching Nets (Vinyals et al., 2016), Prototypical Nets (Snell et al., 2017), MAML (Finn et al., 2017), Meta-SGD (Li et al., 2017) and Meta-dropout (Lee et al., 2019a). For the classifier, we used one-layer NNs with the hidden units of 512. For the meta-model, we used two-layer NNs with the hidden units of 256 similar to VERSA’s conditional posterior model. For an image input, the NN classifier outputs one logit value per each class; class-specific dropout rates for the NN classifier are computed with the image features rt ∈ R256 aggregated by the same class in the few-shot context examples. The same deep (CONV5) feature extractor architecture is used as in VERSA (Gordon et al., 2018). For each class, 1 or 5 few-shot context samples (i.e. labeled images) are given. Among 5 or 20 classes, only the logit value related to the true label is maximized. We use the same batch sizes, learning rate, and epoch settings depicted in VERSA (Gordon et al., 2018). Table 3 summarizes the results. NVDPs achieve higher predictive accuracy than the model-based meta-learning approaches CNP and VERSA and is comparable with the stateof-the-art optimization-based meta-learning approaches such as MAML or Meta-Dropout on the Omniglot dataset. NVDPs also record a good classification accuracy in the MiniImageNet dataset.

heading: 6 CONCLUSION
text: This study presents a new model-based Bayesian meta-learning approach, Neural Variational Dropout Processes (NVDPs). A novel conditional dropout posterior is induced from a meta-model that predicts the task-specific dropout rates of each NN parameter conditioned on the observed context. This paper also introduces a new type of variational prior for optimizing the conditional posterior in the amortized variational inference. We have evaluated the proposed method compared with the existing approaches in various few-shot learning tasks, including the 1D regression, image inpainting, and classification tasks. The experimental results demonstrate that NVDPs simultaneously improved the model’s adaptation on the context data, the functional variability, and generalization to new tasks. The proposed variational prior could also improve the variability of the representationbased posterior model (Garnelo et al., 2018b). Adopting the advanced set representations of (Lee et al., 2019b; Kim et al., 2019; Volpp et al., 2020) or investigating more complicated architectures (Chen et al., 2019; Gordon et al., 2020; Foong et al., 2020) to NVDPs are interesting future works.

heading: A DERIVATION OF THE ELBO OF NVDPS
text: This section describes a detailed derivation of the evidence lower-bound (ELBO) of NVDPs in equation 5. Given the context dataDtC = (xtC ,ytC) and target dataDt = (xt,yt), the KL divergence between the true unknown posterior distribution over parameter p(φt|xt,yt) and the (conditional) variational posterior q(φt|DtC) is given by:
T∑ t=1 DKL(q(φ t|DtC)||p(φt|Dt)) = T∑ t=1 ∫ q(φt|DtC) log q(φt|DtC) p(φt|xt,yt) dφt
= T∑ t=1 ∫ q(φt|DtC) log q(φt|DtC)p(yt|xt) p(yt|xt, φt)p(φt) dφt (7)
= T∑ t=1 ∫ q(φt|DtC) { log q(φt|DtC) p(φt) + log p(yt|xt)− log p(yt|φt,xt) } dφt
= T∑ t=1 DKL(q(φ t|DtC)||p(φt)) + log p(yt|xt)− Eq(φt|DtC)[log p(y t|φt,xt)]. (8)
The step (7) is due to Bayes rule of p(φt|xt,yt) = p(y t|φt,xt)p(φt) p(yt|xt) where φ
t is often assumed to be independent of xt. By reordering (8), we get
T∑ t=1 log p(yt|xt) ≥ T∑ t=1 Eq(φt|DtC)[log p(y t|xt, φt)]−DKL(q(φt|DtC)||p(φt)) (9)
≈ T∑ t=1 Eq(φt|DtC)[log p(y t|φt,xt,DtC)]−DKL(q(φt|DtC)||q(φt|Dt)) (10)
The lower-bound in (9) is due to the non-negativity of the omitted term∑T t=1DKL(q(φ
t|DtC)||p(φt|Dt)) In the lower-bound of (9), the choice of p(φt) is often difficult since the biased prior can lead to over-fitting or under-fitting of model. However, some recent studies of the amortized VI inference (Tomczak & Welling, 2018; Takahashi et al., 2019) analytically discussed that the optimal prior in the amortized variational inference is the aggregated conditional posterior model on whole dataset: p∗(φ) = ∫ D q(φ|D)p(D). Usually the aggregated posterior cannot be calculated in a closed form due to the expensive computation cost of the integral. However, the aggregation on whole dataset in the model-based conditional posterior is constructed based on the set representation. This motivates us to define the conditional posterior given the whole task dataset as an empirical approximation of the optimal prior (i.e., p∗(φt) ≈ q(φt|Dt)). We call this a variational prior. Thus, the approximation of pt(φ) in (9) with the variational prior yields the approximated lower bound of (10).

heading: B DERIVATION OF THE KL DIVERGENCE IN THE ELBO OF NVDPS
text: One important requirement in the choice of the prior is that the analytical derivation of the KL divergence term in equation (10) should not depend on the deterministic NN parameter θ (Kingma et al., 2015; Molchanov et al., 2017; Hron et al., 2018; 2017; Liu et al., 2019). This allows the constant optimization of the ELBO w.r.t all the independent variational parameters (i.e. θl,k,d and Pl,k,d for all l = 1 . . . L, k = 1 . . .K, and d = 1 . . . D). In fact, the conditional posterior defined in
(2) is a Gaussian distribution, thus KL(q(φt|DtC ; θ)||q(φt|Dt; θ)) is analytically defined as:
DKL(q(φ t|DtC ; θ)||q(φt|Dt; θ)))
= Pt(1−Pt)θ2 + ((1−Pt)θ − (1− P̂t)θ)2
2P̂t(1− P̂t)θ2 + log √ P̂t(1− P̂t)θ2√ Pt(1−Pt)θ2 − 1 2
(11)
= Pt(1−Pt)@θ2 + (P̂t −Pt)2@θ2
2P̂t(1− P̂t)@θ2 +
1 2 log
P̂t(1− P̂t)@θ2
Pt(1−Pt)@θ2 − 1 2 (12)
= Pt(1−Pt) + (P̂t −Pt)2
2P̂t(1− P̂t) +
1 2 log P̂t(1− P̂t) Pt(1−Pt) − 1 2
(13)
where both P and P̂ are the dropout rates predicted from the meta-model via the equation (3) but with different conditional set information: P is obtained from the small context set DtC , while P̂ is from the whole task set Dt. (11) is derived using the analytical formulation of KL divergence between two Gaussian distributions. (13) is equivalent to the KL term defined in (4) of the manuscript (except that the constant term 1/2 is omitted for brevity). Interestingly, the analytical derivation of the KL is independent of the shared parameter θ, thus this satisfies the necessary condition to be used as a prior in the VI optimization of the dropout posterior.
The KL divergence in (13) intuitively means that the dropout rates predicted from small context set should be close to the dropout rates predicted from much larger context set while training. The experiments validated that this surprisingly works well to induces a robust conditional functional uncertainty. However, one practical issue while training the dropout rates with the KL term (13) is that the dropout rates can converge to zeros during the early training periods due to the larger gradients from the KL than from the likelihood. In practice, we adopt the dropout rate clipping technique often used in other Variational Dropout approaches (Kingma et al., 2015; Hron et al., 2018; Liu et al., 2019; Molchanov et al., 2017; Gal & Ghahramani, 2016; Liu et al., 2019). We use the dropout rate range of (0.1, 0.99) for all experiments.

heading: C ADDITIONAL EXPERIMENTAL DETAILS ON THE 1D REGRESSION.
text: Setup. We explored the 1D function regression on the data generated from the synthetic GPs with varying kernels6 in the previous work (Garnelo et al., 2018b;a; Kim et al., 2019) that is suitable for measuring the uncertainty adaptation. For the baseline models, Neural Process model (NP) and NP with additional deterministic representation (NP+CNP) described in (Garnelo et al., 2018b;a; Kim et al., 2019) is compared. We also adopted the variational prior (VP) into the representation based posterior of NP and its variants (NP+VP and NP+CNP+VP). For all models (including our NVDP), we used the same fixed variance and learned variance likelihood architecture depicted in (Kim et al., 2019): the agent NN with 4 hidden layers of 128 units with LeLU activation (Nair & Hinton, 2010) and an output layer of 1 unit for mean (or an additional 1 unit for variance). The dimensions of the set representation rt were fixed to 128. The meta NNs in the conditional dropout posterior in NVDPs have 4 hidden layers of 128 units with LeakyReLU and an output layer of 257 units (i.e. K+D+1) for each layer of the agent model. All models was trained with Adam optimizer (Kingma & Ba, 2015) with learning rate 5e-4 for 0.5 million iterations. We draw 16 functions (a batch) from GPs at each iteration. Specifically, at every training step, we draws 16 functions f(·) from GP prior with the squared-exponential kernels, k(x, x′) = σ2f exp(−(x − x′)2/2l2), generated with lengthscale l ∼ U(0.1, 0.6) and function noise level σf ∼ U(0.1, 1.0). Then, x values was uniformly sampled from [−2, 2], and corresponding y value was determined by the randomly drawn function (i.e. y = f(x), f ∼ GP). And the task datapoints are split into disjoint sample of m contexts and n targets as m ∼ U(3, 97) and n ∼ U [m+ 1, 100), respectively. In test or validation, the numbers of the contexts and targets were chosen as m ∼ U(3, 97) and n = 400 −m, respectively. 50000 functions were sampled from GPs to compute the log-likelihood (LL) and other scores in validation.
The architecture of NVDP model is as follows:
6https://github.com/deepmind/neural-processes
(Deterministic) Feature Encoder. r(xts,yts) : 2× |C| lin+relu−→ 128× |C|︸ ︷︷ ︸
6 times
mean−→ 128.
Decoder (Agent). f(xi) : 1 lin+relu−→ 128︸ ︷︷ ︸
4 times
lin+relu−→ 2 split−→ (µ, σ) (where σ = 0.1 + 0.9· softplus(logstd)).
Meta Model. g(l)(r) : dr lin+leakyReLu−→ 128︸ ︷︷ ︸
4 times
lin+leakyReLu−→ (K(l), D(l), 1) split−→ (a,b, c)
where g(l)(r) is the meta NN for the l-th layer of the decoder, and K(l) × D(l) is the number of parameters in the l-th layer.

heading: D ADDITIONAL EXPERIMENTAL DETAILS ON THE IMAGE COMPLETION TASK.
text: Setup. The image completion tasks are performed to validate the performance of the models in the more complex function spaces (Garnelo et al., 2018b;a; Kim et al., 2019) Here, we treat the image samples from MNIST (LeCun et al., 1998) and CelebA (Liu et al., 2015) as unknown functions. The task is to predict a mapping from normalized 2D pixel coordinates xi (∈ [0, 1]2) to pixel intensities yi (∈ R1 for greyscale, ∈ R3 for RGB) given some context points. For 2D regression experiments, we used the same learned variance baselines implemented in the GP data regression task, except the input and output of the decoder are changed according to dataset, e.g., xi ∈ [0, 1]2, and yi ∈ R1 for MNIST (or ∈ R3 and rt = 1024 for CelebA). At each iteration, the images in the training set are split into S context and N target points (e.g., S ∼ U(3, 197), n ∼ U [N + 1, 200) at train and S ∼ U(3, 197), N = 784 − S at validation). Adam optimizer with a learning rate 4e-4 and 16 batches with 300 epochs were used for training. The validation is performed on the separated validation images set. To see the generalization performance on a completely new dataset, we also tested the models trained on MNIST to the Omniglot validation set.

heading: E ADDITIONAL EXPERIMENT ON 1D FUNCTION REGRESSION ON A TOY TRIGONOMETRY DATASET
text: Setup. To further investigate how the proposed meta-model utilizes the common structures of the NN parameters, we trained a small NVDP model (of the size of 13-12-12-2) on a mixture of scaled trigonometric functions: x is sampled in a range of [−π, π], and y is determined by y = a ∗ f(2 ∗ x − b ∗ π) where f is one of three functions sine, cosine, and tanh with the probability of one third, and a ∼ U(1.5, 2) and b ∼ U(−0.1, 0.1). We used the training procedures in the next section except the the simple model architecture and learning rate 5e− 4. For the small NVDP model on 1D function regression with Trigonometry dataset, following architecture was employed:
(Deterministic) Feature Encoder. r(xts,yts) : 2× |C| lin+relu−→ 12× |C|︸ ︷︷ ︸
6 times
mean−→ 12.
Decoder (Agent). f(xi, r) : (1 + dr) lin+relu−→ 12︸ ︷︷ ︸
2 times
lin+relu−→ 2 split−→ (µ, σ) (where σ = 0.1 + 0.9· soft-
plus(logstd)).
Meta Model. g(l)(r) : dr lin+Mish−→ 12︸ ︷︷ ︸
4 times
lin+Mish−→ (K(l), D(l), 1) split−→ (a,b, c)
where g(l(r) is the meta NN for the l-th layer of the decoder, and K(l) × D(l) is the number of parameters in the l-th layer.
Results. Figure 11 displays the trained NVDP on the trigonometry function dataset. The NVDP could capture the probability of dropout for each parameter θ of the agent, successfully predicting the task-specific trigonometric functions conditioned on the small (S = 4) context set. This shows that the task-specific dropout rates can transform a single conventional NN agent to express multiple functions. It is interesting to see that the contexts from sine and cosine functions yield similar dropout rates for the second layer. On the other hand, the contexts from tanh function result in different dropout structures for all layers.

Review summaries:
The paper proposes a novel Bayesian meta-learning approach to model conditional posterior distribution via an amortized variational inference framework based on task-specific dropout. For tasks with few learning examples, the paper discusses the a novel low-rank product of Bernoulli expert meta-models to efficiently obtain full conditional posterior model over the neural network’s parameters. Few-shot learning experiments from Gaussian Process samples show that the model achieves better performance. The proposed method also outperforms existing models on active learning task with better uncertainty calibration. The method also shows better performance on image completion and few-shot classification tasks.

I will put a few bullet points here. For a more detailed summary, see below. - Novel meta-learning approach - Utilises dropout to adapt a base network to new tasks - Amortised variational inference is used to train meta model and base model jointly - The meta model outputs task specific dropout rates - This results in a conditional dropout posterior akin to variational dropout - A *variational prior* over task-dependent dropout rates allows simple calculation of the KL-divergence - The approach is evaluated in 4 different problem setting. SOTA or competitive in most.

The authors present an interesting model-based meta-learning approach, Neural Variational Dropout Processes (NVDP), by constructing a conditional dropout posterior to predict task-specific dropout rates of model parameters, conditioned on the observations. Together with NVDP, the authors also propose a new type of prior, the task-specific variational prior, approximated by the same posterior model conditioned on the whole task data, which supports robust training. They show the NVDPs improve the model's adaptation, functional variability and generalization to new tasks empirically in the experiments.