Neural Variational Dropout Processes
Authors: Insu Jeon
Youngjin Park
Gunhee Kim
Venue: ICLR 2022 Poster (2022)
Keywords: Meta Learning
Few-shot Learning
Bayesian Neural Networks
Variatinoal Dropout

Abstract: Learning to infer the conditional posterior model is a key step for robust meta-learning. This paper presents a new Bayesian meta-learning approach called Neural Variational Dropout Processes (NVDPs). NVDPs model the conditional posterior distribution based on a task-specific dropout; a low-rank product of Bernoulli experts meta-model is utilized for a memory-efficient mapping of dropout rates from a few observed contexts. It allows for a quick reconfiguration of a globally learned and shared neural network for new tasks in multi-task few-shot learning. In addition, NVDPs utilize a novel prior conditioned on the whole task data to optimize the conditional dropout posterior in the amortized variational inference. Surprisingly, this enables the robust approximation of task-specific dropout rates that can deal with a wide range of functional ambiguities and uncertainties. We compared the proposed method with other meta-learning approaches in the few-shot learning tasks such as 1D stochastic regression, image inpainting, and classification. The results show the excellent performance of NVDPs.

Published as a conference paper at ICLR 2022
NEURAL VARIATIONAL DROPOUT PROCESSES
Insu Jeon', Youngjin Park?, Gunhee Kim'
SScoul National University; "Everdoubling LLC., Seoul, South Korea insu jusnueeeepepe Bonégmail amail. com, young]in@everdoubb ing. ai, gunhecl snu. ac kr
ABSTRACT
Learning to infer the conditional posterior model is key step for robust meta- learning. This paper presents a new Bayesian meta-learning approach called Neu- ral Variational Dropout Processes (NVDPs). NVDPs model the conditional poste- rior distribution based on a task-specific dropout; a low-rank product of Bernoulli experts meta-model is utilized for memory-efficient mapping of dropout rates from a few observed contexts. It allows for a quick reconfiguration of globally learned and shared neural network for new tasks in multi-task few-shot learn- ing. In addition, NVDPs utilize a novel prior conditioned on the whole task data to optimize the conditional dropout posterior in the amortized variational infer- ence. Surprisingly, this enables the robust approximation of task-specific dropout rates that can deal with a wide range of functional ambiguities and uncertainties. We compared the proposed method with other meta-learning approaches in the few-shot learning tasks such as ID stochastic regression, image inpainting, and classification. The results show the excellent performance of NVDPs.
INTRODUCTION
In traditional machine learning. large amount of labeled data is required to train deep models (LeCun et al., 2015). In practice, however, there are many cases where it is impossible to collect sufficient data for given task. On the other hand, humans can quickly understand and solve new task even from few examples (Schmidhuber Andrychwccc et al. (016). This distinguishing characteristic of humans is referred to as the neta-learning ability, which enables them to accumulate past learning experiences into general knowledge and to utilize it for efficient learning in the future.
Incorporating the meta-learning capability into artificial machines has long been an active research
topic of machine learning (Naik& & Mammone 1992, Vinyals et al. 2016, Snell et al. 2017 Ravi &
Larochelle 2017 Finn et al. |017] [ee al., 2019c Hospedales al. (021))
Recently, Bayesian meta-learning methods have been attracting considerable interest due to incor- porating uncertainty quantification of the Bayesian framework into the efficient model adaptation of meta-lcarning approaches. An earlier study (Grant et al.] (018)) extended the optimization-based deterministic approach, model-agnostic meta-learning (MAML) (Finn et al. 2017). to a hierarchi- cal Bayesian framework (Daumé III 2009). Later, optimization- based variational inference Yoon jet al. 2018 Finn et al.] 2018)] Ravi & Beatson 2019 Lee et al., 2019a Nguyen et al.] 2020) and
model based based Bayesian meta-leaaming methods (Gordon al. 20118] Garnelo al. [01 lakovl- eva et al., 2020) were presented. These methods achieved outstanding results in various few-shof" regression and classification tasks. In fact, the adaptation of deep models that estimate uncer- tainties is a core building block for reliable machine learning systems in real-world applications with high judgmental risks, such as medical AI or autonomous driving systems.
Inspired by recent studies, we propose a new model-based Bayesian meta-learning approach, neural variational dropout processes (NVDPs). The main contribution of this work is to design a new type of Neural Network (NN) based conditional posterior model that can bypass the under- fiittr--iinigg and
posterior collapsing of existing approaches (Gordon et al. 2018 Garnelo et al. [0188]] Iakovleva
et al., 2020). NVDPs extend the simple yet effective posterior modeling of Variational Dropout
(VD) Kingma et al. 2015. Gal Ghahramani] 2016) Molchanov al.) 2017 Hron et al. 2018
'The few-shot learning assumes only few examples are available for each task, with the number of tasks heing large (Lake al. 2015).

---

Published as a conference paper at ICLR 2022
Liu et al. 2019) in the context of meta-learning A novel low-rank product of Bemoulli experts meta-model is utilized in the dropout posterior to learn task-specific dropout rates conditioned on a few learning examples. In this way, full conditional posterior model over the NN's parameters can be efficiently obtained. In addition, we also propose a new type of prior to optimize the conditional dropout posterior in variational inference, which supports the robust training of conditional poste- riors. Although it is developed in conjunction with NVDPs, the formulation allows its adoption to other recent methods, and we show the effectiveness. We have evaluated NVDPs compared with other methods on various few-shot learning tasks and datasets. The experiments show that NVDPs can cireumvent the under-fitting and posterior collapsing and achieve outstanding performances in terms of log-likelihood, active learning efficiency, prediction accuracy, and generalization.
2
BAYESIAN META-LEARNING
2.1 AMORTIZED VARIATIONAL. INFERENCE FRAMEWORK IN THE MULTI-TASK DATA
A goal in meta-learning is to construct a model that can quickly solve new tasks from small amounts of observed data. To achieve this, it is important to learn a general (or task-invariant) structure from multiple tasks that can be utilized for the efficient model adaptation when necessary. Bayesian meta- learning methods (Gordon et al. 2018 Garnelo et al. 2018b Ravi & Beatson 2019) formulate this objective of meta-learning multi-task environments. Suppose a collection of T related tasks is given, and each t-th task has the training data D' containing N i.i.d. observed tuples (x', y²) (ar,, vt =- Then, the evidence lower-bound (ELBO) over the log-likelihood of the multi-task dataset can be derived as:
amortized variational inference (VI) of the posterior distribution in
log log:(D';:)
Σ
(ooooppp/oogp(''>>> Φ9] ('(g(o'D';0)
Here, pxy" (t) a likelihood (or NN model) on the 1-th training data and of is 1-th task taec-spec variable (i.e. latent representation or weights of NN) and q( |D'; 8) is tractable amortized vari- ational posterior model utilized to approximate the true unknown posterior distribution over of for each given 1-th task data (i.e., p (D') (Kingma & Welling 2013 Gordon et al. 2018 Garnelo jet al., (018b)) Ravi & Beatson 2019, lakovleva et al.; 2020). The parameter B represents the com- mon structure that can be efficiently learned across multiple different tasks. The prespecified prior distribution p((²) in the Kullback-Leibler (KL) divergence term provides a stochastic regularization that can help capture the task-conditional uncertainty and prevent the collapsing of q(o' 0)) In fact, the maximization of the ELBO, right side of equation I with respect to the conditional variational posterior model is equivalent to the minimization of E._ KL(g(0 (D'; () Essentially, the goal in the amortized VI of Bayesian meta-learning is to learn the inference process of the true conditional posterior distribution via the variational model "( (D;;0) and the shared general structure 0 across multiple tasks since this task iask-invariant knowledge can later be utilized for the efficient adaptation of the NN function new unseen tasks. The approximation of the conditional posterior also enables ensemble modeling and uncertainty quantification.
2.2 THE EXISTING CONDITIONAL POSTERIOR AND PRIOR MODELINGS
Many of the recent Bayesian meta-learning approaches can be roughly categorized into the
optimization-bl based (Grant al. 2018 Yoon al. 2018, Finn et al. 2018 Ravi & Beatson 2019
Lee et al. [2019a] 1 Nguyen et al. 2020) or the model - based posterior approximation approaches (Gor-)
don et al.
2018 Garnelo al. 2018a b Kim et al. 2019 Iakovleva al.) 2020). In the optimization-
based approaches, the task-specific variable of can be seen as the adapted NN weights. For example,
the deterministic model-agnostic meta-learning (MAML) (Finn al. 2017) can be considered as learning a Dirac delta variational posterior modeling (i.e., q(6" D t) ~ ò(ø" - SGD, (D' 0))) where the goal to learn the shared global initialization NN's parameters such that few j steps of SGD updates on the small subse D' of the 1-th dataset De provides a good approximation of the task-specific weights &². Many optimization-ban based Bayesian meta-learning approaches such
Thhe subset (C D³) is known as context (or support) set for each task. A small S size of context set (c.g., 1-shot, 5-shot, or random) is often used in the few-shot learning tasks (L.kee et al. 2015).
2

---

Published as a conference paper at ICLR 2022
as LLAMA (Grant et al. 2018), PLATIPIS (Finn al. 2018), BMAML (Yoon et al. 2018), and ABML (Ravi & Beatson 2019) have incorporated the Gaussian type of posterior and prior models into the deterministic adaptation framework of MAML in order to improve the robustness of mod- approaches can be applied to various types of few-shot learning tasks, but the adaptation cost at test time is computationally expensive due the inversion of the Hessian or kernel matrix; they may not be suitable for environments with limited computing resources at test.
els. Optimization-basse
On the other hand, the model-based Bayesian meta-learning approaches such VERSA (Gordon et al. al. (0188) or Neural Processes (Garnelo et al., 20188 b) allow an instant estimation of the Bayesian predictive distribution at test time via -a--dase conditional posterior modeling. VERSA employs model (-) to directly predict the Gaussian posterior of agent NN's
an additional NN-based task-specific weight o² from the small context set (i.e, (( 0) = N(o o) = go (P')))- In this case, the shared structure O represents the meta-model's parameters. However, the direct approximation of agent NN weights in VERSA could limit their scalability due to the large dimen- sionality of the NN's weights; they only consider the task-specific weights for the single softmax output layer. In addition, VERSA did not utilize any task-specific prior p(ő"), so the conditional posterior could collapse into deterministic one while training with the Monte Carlo approximation (lakovleva et al. 2020). Instead of the NN's weights, the Neural Processes (NPs) model the condi-
tional posterior with latent representation (i.e., (=='ID';0) 0) (N==' o) = go (D))) (Garnelo et al. 2018b). Regarding the task-specific latent representation as a second input to the likelihood p(y" 2 (similar to Kingma & Welling (2013) or Edwards & Storkey (2017)) provides another
efficient way to transfer the conditional information (Ganeloo et al. 2018b) However, the following studies indicate that NPs often suffer from under-fitting (Kim et al.) 2019) and posterior-collapsing behavior (Groer et al. 2019 Le et al. 2018). We conjecture that these behaviors are partially inher- ent its original variational formulation. Since the variational posterior in NPs rely on the whole task data while training, the posterior do not exhibit good generalization performance at test. In addition, well known that the highly flexible NN-based likelihood can neglect the latent repre- sentation in variational inference (Hoffman & Johnson 2016;; Sønderby et al. 2016 Kingma al. Chen et al.) 2017;: Yeung et 2017]] Alemi et al.] [0188] Lucas et al. 2019 Dieng et al,
2016
2018
could also observe the posterior collapsing behavior of the NPs in our experiments.
3 NEURAL VARIATIONAL DROPOUT PROCESSES
This section introduces new model-based Bayesian meta-learning approach called Neural Varia- tional Dropout Processes (NVDPs). Unlike the existing methods such as NPs VERSA employing conditional latent representation or direct modeling of NN's weights, NVDPs extend the posterior modeling of the Variational Dropout (VD) in the context of meta-learning We also introduce new type of task- speccficc prior to optimize the conditional dropout posterior in variational inference.
3.1 A CONDITIONAL DROPOUT POSTERIOR
Variational dropout (VD) (Kigmma et al. 2015 Gal & Ghahramani] 2016 Molchanov et al., 2017
Hron et al. 2018 Liu et al. 2019) set of approaches that models the variational posterior dis- tribution based on the dropout regularization technique. The dropout regularization randomly turns off some of the Neural Network (NN) parameters during training by multiplying discrete Bernoulli random noises to the parameters (Srivastava et al.) 2014 Hinton et al. 2012). This technique was originally popularized as efficient way prevent NN model's over-fitting Later, the fast dropout Wang & Manning 2013) Wan et al. 2013) reported that multiplying the continuous noises sampled from Gaussian distributions works similarly to the conventional Bernoulli dropout (Srivastava al., 2014) due to the central limit theorem. The Gaussian approximation of the Bernoulli dropout is convenient since it provides a fully factorized (tractable) posterior model over the NN's parameters (Wang & Manning 2013 Wan et al. 2013). In addition, VD approaches often utilize sparse pri- ors for regularization (Kingma et al. 2015 Molchanov et al.) 2017 Hron et al. 2018 2017 Liu ct al. 2019). This posterior and prior distribution modeling in VD enables the learning of each inde- pendent dropout rate over the NN parameters as variational parameters. This distinguishes the VD approaches from the conventional dropouts that use single fixed rate over all parameters. However, the dropout posterior in the conventional VD becomes fixed once trained and it is unable to express the conditional posterior according multiple tasks.
3

---

Published as a conference paper at ICLR 2022
PS
02
S
Meta NN
x
P
X
Pia
y
p
y
Sharec/NN zonmmeter
Meta NN
PLa
P L/
I+... Support
Pla
Pia
Oip (+)
Conditional posterfor
(a) The low-rank product of Bernoulli experts meta-model
(b) The probabllistic graphical model of NVDP's
Figure I: (a) The low-rank product of Bernoulli experts meta-model of conditional dropout posterior. (b) The probabilistic graphical model of Neural Variational Dropout Processes (NVDPs) with the variational prior. Where {at, y Jim1 is the N i.ddd samples from the f-th training dataset DD among T tasks. The context set D (#4,, vt 1=1 is small subset of the I-th training dataset.
Conditional dropout posterior. We propose a new amortized variational posterior model that can be efficiently adapted for each given task. Suppose we train a fully connected NN of L layers, then conditional dropout posterior based on the task-specific dropout rates P over the K D dimensional deterministic parameters of each 1-th layer of the NN can be given as follows:
K II
D II II R= d=1
9(0 (D);
II d=1
(okk
(O -kk,))k P (1 Pk Pka,)0k. (2)
The p fle,d is shared across different tasks, representing the common task insk-invariant struc- ture. In the equation 2 the task-specific NN parameter of are fully described by the mean and variance of each independent Gaussian distribution via Ou.d and P Note that the variational pos- terior model explicitly conditioned the subset of the t-th training dataset Dj {x(, yi (S DD') known as the t-th context set. The key idea of conditional posterior modeling in NVDPs employ an NN-based meta-model predict the task ssec-ipcc dropout rate Pk.d from the small context set Dic The meta-model to approximate Pi for each given task is simply defined as:
Pk = s(a)) s(ba) s(c), where (a, b. c) = go (r*).
(3)
Here, the representation yf is defined the of features obtained from each data in 1-th context set De (i.e., r' = w), summarizing order invariant set information (Edwards & Storkey 2017 Lee et al. 2019b). The (.).) is meta NN model parameterized by v to predict set of logit vectors (i.e., a R RD and C R). Then, the sigmoid function with a learnable temperature parameter T (i.e., s.)
hh h yx). SS where h. is feature extracting NN parameterized by
1/(1 + exp(((.)/7) ) is applied to them to get the low-rank components of task-specific dropout rates Pk the row-wise s(ax), column-wise s(ba), and layer-wise dropout rate s(c). In other words, the task -ask-specific dropout rate Pk. is obtained by multiplying low-rank components of the conditionally approximated dropout rates from the NN-based meta-model go (D)) (see Figure (a)).
The product of n Bernoulli random variables is also Bernoulli variable (Leemis 2020 By ex- ploiting this property, we interpret the approximation of the task-specific dropout rates in terms of the low-rank product of Bernoulli experts. Unlike VERSA whose meta-model's complexity is OLLKD) to model the full NN weight posterior directly, the complexity of meta-model in NVDPs is O(I(KK the task-specific dropout rates. With the shared parameter 0, this can greatly reduce the complex- ity of the posterior distribution approximation of the high-dimensional task-specific NN's weights using only a few observed context examples. The product model tends to give sharp probability boundaries, which is often used for modeling the high dimensional data space (Hinton 2002)
In addition, the meta-sodel's role is only predict the low-rank components of
3.2 THE TASK-SPECIFIC VARIATIONAL PRIOR
To optimize the conditional dropout posterior in equation 2 specification of the prior distribution p(o') is necessary to get a tractable derivation the KL regularization term the ELBO in equa-
Thee original Gaussian approximation in the is ()) = N a 2g 2 with a PR.a(( (1
P Ph.e). But, NVDPs extend the Bernoulli dropout model WWag & Manning 2013 Wan et al. 2013
We omit the layer index of the parameter fr..d d for brevity.
4

---

Published as a conference paper at ICLR 2022
tion The question is how can define the effective task-specific prior. In fact, important requirement in the choice of the prior distribution in the conventional VD framework (Kingma et al.
2015) is that the analytical derivation of the KL term (i.e., KL(g ( (Pa ())pp(()) in equation should not depend on the deterministic NN parameter 0. This allows the constant optimization of the ELBO w.r.t all the independent variational parameters (i.e., 01,k.d and PI, for all I 1... T., One way of modeling the prior is to employ the log-uniform prior
k = 1.... K and d = 1..... plloo((o))) x c as in the conventional VD (Kingma et al. 2015). However, recently known limita- tion is that the log-uniform prior is an improper prior (e.g., the KL divergence between the posterior and the log-unin ungguufformm prior is infinite). This could yield degeneration of the dropout posterior model to deterministic one (Molchanov et al. 2017: Hron et al.) 2018 2017 Liu et al.) 2019). Besides, the conventional prior used VD approaches does not support a task-dependent regularization.
Variational prior. We introduce a new task-specific prior modeling approach to optimize the proposed conditional posterior; it is approximated with the variational prior defined by the same dropout posterior model equation exccep that the prior is conditioned on the whole task data (i.e., p((²) R 1(0 (D'). The KI. divergence between the conditional drepout posterior (with the only context set) and the variational prior (with the whole task data) can be derived as:
K
Pt k
Pt Pk.a)) + (P)
P P)
1
P
P k,
KL(g(d' (
log
2P 2Pa P Phaa)
2
P
P (,
k=1d=1
(4)
where both Pkld and P. Pk.d are the dropout rates predicted from the meta-model but with different
conditional set information: PR.a is obtained from the small context set Dé while PA.aa is from the whole task set D'. Interestingly, the analytical derivation of the KL is independent of the shared parameter 8 thus this satisfies the necessary condition to be used prior in the VI optimization of the dropout posterior. Figure 1(b) depicts the variational prior is conditioned on the whole dataset.
The shared posterior model for the task-specific prior introduced here was inspired by recent Bayesian meta-learning approaches (Garnelo et al., 2018b Kim et al. 2019 lakovleva et al. 2020), but some critical differences are that: 1) we have developed the variational prior regularize the task-specific dropout rates in the optimization of the conditional dropout posterior, 2) the denom- inator and the numerator of the KL divergence term in equation are reversed compared with the existing approaches. In fact, some other recent studies of amortized VI inference (Tomezak & Welling (Tkaahaah et al., (2199) analytically derived that the optimal prior maximizing VI ob- jective is the variational posterior aggregated on the whole dataset: p (0())= = fr ((0) (())D))())) Thus, we hypothesized the conditional posterior model that depends on the whole dataset should be used to approximate the optimal prior p" (Ø¹) R q (D*) since the variational model conditioned on the aggregated set representation with much larger context (e.g..D' 2 D'))) is likely to be much closer to the optimal task-specific prior than the model conditioned only on subset. The experiments in Section 5 demonstrate that the proposed variational prior approach provides a reliable regulariza- tion for the conditional dropout posterior and the similar formulation is also applicable to the latent variable based conditional posterior models (Garnelo et al. 2018b Kim et al. 2019).
(D'
3.3 STOCHASTIC VARIATIONAL INFERENCE
With the conditional dropout posterior defined in equation 2 and the KL regularization term in equation 4
we can now fully describe the ELBO objective of NVDPs for the multi-task dataset as:
Σ log log'(DD)) 2 Σ E
poy Φ¹)l
(p: (5)
0))
The goal is to maximize the ELBO equation 5 w.r.t. the variational parameters (i.e. 0, v. w, and
7) of the posterior q(, Di; 0) defined in equation 2 The optimization of these parameters is done by using the stochastic gradient variational Bayes (SGVB) (Kingma & Welling 2013 Kingma al. 2014). The basic trick the SGVB to parameterize the random weights
q0 D
using a deterministic differentiable transformation ø' = f(c, D')) with a nor-parametric i.i.d. noise p(e). Then, unbiased differentiable minibatch-based Monte Carlo estimator, (0, 1h 7,,,
5

---

Published as a conference paper at ICLR 2022
of the ELBO of NVDPs can be defined as:
M
f(t -( P'(1 - -P') + (P' -P?? +
1 P (1 P() log P1(1 Pt) }} (6)
2P P())
T" and M are the size of randomly sampled mini-batch of tasks and data points, respectively, per each epoch (i.e., y = D' ~ {D') D). D is the subset set of D' discussed in sec-
tion 3.1 andnnn 3.2 The transformation of the task-specific weights is given as Hi, = f(Kkad Dz)= =
(1 P-P K, On.a + V /Pk - - de, with Ex,d (N, 1) from the equation The inter-
mediate weight Ka is now differentiable with respect to 8k.d and pk.d Also, Pk.d (and P
Pk.a) is
deterministically computed by the meta-model parameterized by th, uz, and in equation D Thus,
the estimator [(0,, 4,,,7) 7) in equation 6 is differentiable with respect all the variational param- eters and can be optimized via the SGD algorithm. In practice, a local reparameterization trick) is further utilized to reduce the variance of gradient estimators on training (Kingma et al., 2015).
RELATED WORKS
Gaussian Processes (GPe) (Rasmussen& & Williams 2005 are classical Bayesian learning approaches closely related to NVDPs; GPs allow an analytical derivation of the posterior predictive distribution given observed context points based on Gaussian priors and kernels. However, classical GPs re- quire an additional optimization procedure to identify the suitable kernel for each task, and the time complexity quadratic to the number of contexts. Recently introduced model-based Bayesian meta- learning methods (Garnelo et al. 201 8b 2020) offer NN- conditional posterior models that can be efficiently learned from data and later provide an instant posterior predictive estimation at test time. One important aspect of these model- based methods have in common is the utilization of permulation invariant set representation as an input to the modal (Edwards & Storkey, 2017| Lee et al. 2019b) Glleem-Reddy & Teh 22020] Teh & Lecture 2019). which makes a conditional posterior invariant to the order of the observed context data. This property. known as exchangeability, is necessary condition to define stochas- tic process according to Kolmogorov's 's Extension Theorem (Garnelo et al.] 2018b; Kim et al. 2019 Øksendal 2003). NVDPs also utilize the set representation to approximate the task-specific dropout rates in the conditional dropout posterior. Another related work to NVDPs is Meta Dropout (Lee) al., 2019a). They also proposed a unique input-dependent dropout approach; Gaussian-based stochastic perturbation is applied to each preactivation feature of NN functions. Their approach, however, is constructed using optimization-
Kim et al., 2019, Gordon et al.) 2018 Iakovleva et al.)
meta-leariig methods (Finn et al., 2017 Li al., 2017) without directly considering the set representation. NVDPs, employing set representation, extend the dropout posterior of Variational Dropout (VD) (Kingma et al. 2015): Wang & Manning 2013 Wan et al.) 2013, Liu et al.] 2019 Molchanov et al., (017)) in the context of model-based meta-learning and introduce a new idea of variational prior that can be universally applied to other
Bayesian learning approaches (Garnelo et al.) 2018b] Kim et al. 2019). Gal & Ghahramani (2016) also discussed the theoretical connection between VD and GPs.
EXPERIMENTS
Metries in regression. In the evaluation of the conditional NN models' adaptation, the newly observed task data D is split into the input-output pairs of the context set Di- = {zs, 1- and the target set Dj- = {z,, yı W (D& & Dj. in evaluation (Garnelo et al. 2018b Gordon et al. ). (1) The log-likelihood (LL), N+3
I Eie EDDD E [Pa) p(y: xx, Φ²)). measures the
performance of the NN model over the whole dataset D* conditioned on the context set. (2) The reconstructive log-like ligeliiddoood (RLL), $ EiED; Eyto ( p(y (x,, dd)], measures how well the model reconstructs the data points in the context set. A low RLL is a sign of under-fitting, (3) The predictive likelihooo (PLL), NV EiED, Eggor (D)) (log p(y: [ o² }}, measures the prediction on the data points in the target set (not in the context set). A low PLL is a sign of over-fitting.
'ee can sample pre-activations Bm.d for mini-batch of size M directly using inputs Am, (Bm,d ~
(B EK, Am, P Pr.a)Ok.d, E A2 Pk (1 Pt (.,))
6

---

Published as a conference paper at ICLR 2022
Learned veriance Fixee varfance
C NP-CNP
NP
Figure 2: The ID few-shot regression results of the models on GP dataset in fixed variance and learned variance settings. The black (dash-line) represent the true unknown task function. Black dots are few context points (S = 5) given to the posteriors. The blue lines (and light blue area in learned variance settings) are mean values (and variance) predicted from the sampled NNs.
GP Dataset
NP
NP+VP
NP+CNP
NP+CNP+VP
NVDP
Fixed
LL -0.9((#0.08)
-0.95(#0.05) -0.96(#0.06)
-0.94(=0.04) -0.4((00.05)
Variance RLL -0.97(#0.06)
-0.95(#0.04)
-0.93(#0.02)
-0.93(+0.02)
-0.93(=0.01)
PLL -0.98(+0.08)
-0.97(#0.06)
-0.95(#0.05)
-0.91(40.05)
-0.95
Learned LL 0.19(#1.87)
0.48(+0.77)
0.70(10.89)
0.70(=0.73)
0.83(10.61)
Variance RLLL 0.72(+0057)
0.71(+0055)
1.03(+0039)
1.00(+0.41)
1.10
PLI. 0.16(+1.90)
0.46( +.778)
0.66(+0.92)
0.68(+0.75)
0.81 (+0.6))
Table 1: The validation result of the ID regression models on the GP dataset in fixed variance and learned variance settings. The higher LLs are the better. The models of NP, NP with variational prior (NP+VP), NP with deterministic path (NP+CNP). NP+CNP+VP and NVDP (ours) are compared.
GP Samples. The ID regression task is to predict unknown functions given some observed context points; each function (or data point) is generated from a Gaussian Process (GP) with random kernel, then the standard data split procedure is performed (i.e. ~ (3, 97) and N ~ UIS + 1, 100)) at train time. For the baseline models, Neural Process model (NP) and NP with additional deterministic representation (NP+CNP) descrihed in (Garnelo et al. 2018b a Kim et all, 2019) is compared. We also adopted the variational prior (VP) into the representation based posterior of NP and its variants (NP+VP and NP+CNP+VP). For all models (including our NVDP), we used the same fixed variance and learned variance likelihood architecture depicted in (Kim et al. 2019): the agent NN with hidden layers of 128 units with LeLU activation (Nair & Hinton (2110) and an output layer of I unit for mean (or an additional 1 unit for variance). The dimensions of the set representation re were fixed to 128. The meta NNs in the conditional dropout posterior in NVDP has 4 hidden layers of 128 units with LeakyRel.U and an output layer of 257 units (i.e. K D+1) for each layer of the agent model. All models were trained with Adam optimizer (Kingma & Ba 2015) with learning rate 5e-4 and 16 task-batches for 0.5 million iterations. On validation, 50000 random tasks (or functions) were sampled from the GP function generator, and the split data of S ~ U33, 97) and N 400 - S were used to compute the log- licelioodoood (LL) and other evaluation metrics.
Table 1 summarizes the validation results of ID regression with the GP dataset. NVDP achieves the best LL scores compared with all other baselines in both the fixed and learned variance likelihood model settings. The NVDPs record high RLL on the observed data points and excellent PLL scores on the unseen function space in the new task; this indicates that the proposed conditional dropout posterior approach can simultaneously mitigate the under-fitting and over-fitting of the agent model compared with the other basclines. When VP is applied to NP or NP+CNP, the PLL scores tend to increase by meaningful margins in all cases. This demonstrates that the proposed variational prior (VP) approach can also reduce the over- fitting of latent representation-bant Figure visualizes the fhw-shot ID function regression results in both model settings. In the fixed variance setting, the functions sampled from NPs show high variability but cannot fit the context data well. NP+CNP can fit the context data well but loses epistemic uncertainty due to the collapsing of conditional posterior. On the other hand, the functions from NVDPs show a different behavior; they capture the function's 's variability well while also fitting the observed context points better. NVDPs also approximate the mean and variance of unknown function well in learned variance setting.
conditional posterior.
7

---

Published as a conference paper at ICLR 2022
10
30
100
Felf
1024
10
30
ID0
half
784
30
100
harf
7B4
10
30
100
half
784
/1
4
2
2
RR
7
4
4
4
2
J
5
6
5
7
6
6
6
6
3
4
4
4
5
J
5
5
6
2
7
8
3
NP-CNP
7
6
4
1
4
0
3
3
2
0
0
&
8
2
6
4
4
4
7
3
3
0
2
8
8
8
NVDP
3
4
4
4
4
6
2
2
3
2
a
8
8
8
9 (b) MNIST
4
4
6
2
3
2
8
2
S
8
(a) CelebA
[c] MNIST (train) to Omniglot (test)
Figure 4: The results from the 2D image completion tasks on CelebA, MNIST, and Omniglot dataset.
Given the observed context points (10, 30, 100, half, and full pixels), the mean values of two inde- pendently sampled functions from the models (i.e. NP, NP+CNP and NVDP (ours)) are presented.
Image Dataset
NP
NP+VP
NP+CNP
NP+CNP+VP
NVDP
MNIST
LL 0.51(=00.51)
0.76(+0.14)
0.83(+0.21)
0.86(10.16)
0.90(=0.16)
RLL 0.94(=0.18)
0.90(=0.10)
1.12(+0.09)
1.15(=0.05) 1.1((+0.09)
PLL (=0.5)))
0.75(=0.14)
0.80(+0.20)
0.83(+0.15)
0.88(=0.15)
MNIST (train)
L.I. 0.35(+0.29) 0.56(+0.10)
0.64(+0.17)
0.68(+0.13)
0.70(-0.13)
to
0.98(10.09) RLL 0.72(10.19) 0.73(10.12) 0.95(10.09)
0.99(10.07)
Omniglot (test)
PLL 0.32
0.54(+0.11)
0.60(+0.16)
0.64(+0.13)
0.66(+0.12)
CelebA
LL 0.51 0.60(=0.12)
0.76(#0.15)
0.77(+0.15)
0.83( 0.83 0.
RLL +00.11 0.68(=0.06)
(.005)
0.92(+0.05)
0. 0.04
PI.. (-027)))27) 0.59(+0.12) 0.75(+ 0.76(+0.15)
0.82 0
Table 2: The summary of 2D image completion tasks the MNIST, CelebA, and Omniglot dataset.
Few-Shot Active Learning Experiment
Active learning with regression. To further compare the uncertainty modeling accuracy, we performed an ad- ditional active learning experiment on the GP dataset de- scribed above. The goal in active learning is to improve the log-likelihood of models with a minimal number of context points. To this end, each model chooses addi- tional data points sequentially; the points with maximal variance across the sampled regressors were selected at each step in our experiment. The initial data point is ran- domly sampled within the input domain, followed by 19 additional points that are selected according to the vari- ance estimates. As seen in Figure 3, NVDPs outperform the others due to their accurate variance estimation, es- pecially with small number of additional points, and show steady improvement with less over wer-fitting.
log fit-illhhood
MVDP
NP CNP VP
NP
NP VP
number of context points
Figure 3: Active learning performance on regression after up to 19 selected data points. NVDPs estimation to quickly improve LLs, while other models are learning slowly.
use its uncertainty
Image completion tasks. The image completion tasks are performed validate the performance of the models in the more complex function spaces (Garnelo et al. 201 a Kim et al. 2019)
Here, we treat the image samples from MNIST (LeCun et al.] 1998) and CelebA (Liu et 2015) as unknown functions. The task is to predict a mapping from normalized 2D pixel coordinates x; (E [0, 1J2) to pixel intensities y: (E E for greyscale, E R for RGB) given some context points. We used the same learned variance baselines implemented in the GP data regression task (except the 2D input and 3D output for the agent NN and r' 1024 were used for CelebA). At each iteration, the images in the training set are split into S context and N target points (c.g.. S ~ (3, 197), n ~ U[N + 1, 200) at train and S (3, 197), N = 784 - 5 at validation). Adam optimizer with a learning rate 4 and 16 task batches with 300 epochs were used for training. The validation is performed on the separated validation set. To see the generalization performance on a completely new dataset, we also tested the models trained on MNIST to the Omniglot validation set.
Table summarizes the validation results of image completion tasks. NVDPs achieve the out- performing L.S compared with all other baselines. Interestingly, NVDPs (trained on the MNIST
8

---

Published as a conference paper at ICLR 2022
Omniglor dataset
MinilmageNer dataset
5-way accuracy (%)
20-way accuracy (%)
5-way accuracy (%)
Method
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
Matching Nets
98.1
98.9
93.8
98.5
46.6
60.0
Prototypical Nets
97.4
99.3
95.4
98.7
46.61 + 0.78
65.77 + 0.70
CNP
95.3
98.5
89.9
96.8
48.05
62.71
0.58
Meta-SGD (MSGD)
96.16
+ 0.14 98.54
+ 0.07
48.30
+
65.55 + 0.56
MSGD + Meta-dropout
97. 02
+ 0.13
99.05
= 0.05
50.87
0.66
65.55
+: 0.57
MAML
98.70
0.40 99.90
0.10
95.80
0.63
98.90
+ 0.20
48.70
63.11
= 0.92
VERSA
99.70
0.20 99.75 - 0.13
97.66
0.29
98.77
1 0.18
53.40
1.82
67.37 0.86
NVDP
99.70 + 0.12 99.86 + 0.28 97.98 0.22 98.99 0.22 54.06 18.
1.04
Table 3: Few-shot classification results on Omniglot and MinilmageNet dataset. The baselines are
Matching Nets, Prototypical Nets, MAML, Meta-SGD, Meta-dropout, CNP, VERSA, and NVDP (our). Each value corresponds to the classification accuracy (%) (and std) on validation set.
dataset) also achieve the best results on the Omniglot dataset. Figure 4 shows the image reconstruc- tion results with a varying number of random context pixels. NP generated various image samples when the number of contexts was small (e.g.. m S 30 or half), but those samples could not approx- imate the true unknown images well compared with the other models. The NP+CNP achieves crisp reconstruction results compared with NP and also shows good generalization performance on the Omniglot dataset, but the sampled images (or functions) from NP+CNP had almost no variability due to its posterior collapsing behavior. On the other hand, the samples from NVDPs exhibit com- parable reconstruction results while also showing a reasonable amount of variability. In addition, NVDPs also present an outstanding generalization performance This implies that NVDPs not only fit better in the complex function regression but can also capture more general knowledge that can be applied to new unseen tasks.
the unseen Omniglot dataset.
Few-shot classification tasks. NVDPs can also be successfully applied for few-shot classification tasks: we have tested NVDPs on standard benchmark datasets such as Omniglot (Lake et al. 2015)
and Minilmagenet (Ravi & Larochelle| 2017) with other baselines: VERSA (Gordon et al. 2018)
CNP (Garnelo et al.J 8a), Matching Nets (Vinyals et al. 2016). Prototypical Nets (Snell et al, 2017]. MAML (Finn et al. 2017). Meta-SGD (Li et al.] 2017) and Meta-dropout (Lee et al. 201 9a). For the classifier, we used one-layer NNs with hidden units of 512. For the meta-model, we used two-layer NNs with hidden units of 256 similar to VERSA's conditional posterior model. For an image imput, the NN classifier outputs one logit value per each class; specifcc dropout rates for the NN classifier are computed with the image features r I225 aggregated by the same class in the few-shot context examples. The same deep (CONV5) feature extractor architecture is used as in VERSA (Gordon et al. 2018). For each class, or 5 few-shot context samples (i.e., labeled images) are given. Among 5 or 20 classes, only the logit value related to the true label maximized. We use the same batch sizes, learning rate, and epoch settings depicted in VERSA (ordon et al. 2018). Table 3 summarizes the results. NVDPs achieve higher predictive accuracy than the model- based meta-learning approaches CNP and VERSA and is comparable with the state- of-the-au art optimization- based meta-learning approaches such as MAML or Meta-Dropout on the Omniglot dataset. NVDPs also record a good classification accuracy in the MinilmageNet dataset.
CONCLUSION
This study presents new model-based Bayesian meta-learning approach, Neural Variational Dropout Processes (NVDPs). A novel conditional dropout posterior is induced from a meta-model that predicts the task-specific dropout rates of each NN parameter conditioned on the observed con- text. This paper also introduces a new type of variational prior for optimizing the conditional posterior in the amortized variational inference. We have evaluated the proposed method compared with the existing approaches in various few-shot learning tasks, including 1D regression, image in- painting, and classification tasks. The experimental results demonstrate that NVDPs simultancously improved the model's 's adaptation to the context data, functional variability, and generalization to new tasks. The proposed variational prior could also improve the variability of the representation based posterior model (Gamelo et al.) 2018b). Adapting the advanced set representations of (Lee et al. 2019b] Kim et al. 2019:] Volpp et al. (020)) or investigating more complex architectures (Chen et al,
Mor9ono al. 2020 Foong al. 2020) for NVDPs would be interesting future work.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENT
This work was supported by Center for Applied Research Artificial Inteliggene(CARAA) grant funded by Defense Acquisition Program AAAi and Agency for Defense Develop- ment( ADD) (UDI9003IRD) and Basic Science Research Program through National Research Foun- dation of Korea (NRF) funded by the Korea government (MSIT) 10--22RRRAABBBOS0U0 Gunhee Kim is the corresponding author. Insu Jeon thanks Jaesik Yoon, Minui Hong, and Kangmo Kim for helpful comments.
REFERENCES
Alexander A Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing broken elbo. In PMLR, 2018.
Marein Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In NeurlPS, 2016.
Benjamin Bloem Bloem-Reddy and Yee Whye Teh. Probabilistic symmetry and invariant neural networks. JMLR, 2020.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In ICLR, 2019.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In ICLR, 2017.
Hal Daumé III. Bayesian multitask learning with latent hierarchies. In UAI, 2009.
Adji B Dieng, Yoon Kim, Alexander M Rush, and David M Blei. Avoiding latent variable collapse with generative skip models. In AISTAT, 2018.
Harrison Edwards and Amos Storkey. Towards A Neural Statistician. In ICLR, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks, In PMLR, pp. 1126-1135, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning In NewrlPS, 2018.
Andrew YK Foong, Wessel P Bruinsma. Jonathan Gordon, Yann Dubois. James Requeima, and Richard Turner. Meta-learning stationary stochastic process prediction with convolutional neu- ral processes. In NeurlPS, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In ICML, pp. 1050-1059, 2016.
Marta Garnelo, Dan Rosenbaum, Chris J. Maddison, Tiago Ramalho, David Saxton, Murray Shana- han, Yee Whye Teh, Danilo J. Rezende, and SM Eslami. Conditional neural processes. In JCML, 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, SM Eslami, and Yee Whye Teh. Neural processes. In ICML Workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018b.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Meta- learning probabilistic inference for prediction. In ICLR, 2018.
Jonathan Gordon, Wessel P Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard Turner. Convolutional conditional neural processes. ICLR, 2020.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning hierarchical bayes. In ICLR, 2018.
10

---

Published as a conference paper at ICLR 2022
Aditya Grover, Dustin Tran, Rui Shu, Ben Poole, and Kevin Murphy. Probing uncertainty estimates of neural processes. In NeurIPS Workshop on Bayesian Deep Learning. 2019.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 1778)1177-- 2002.
Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdi- nov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the varia- tional evidence lower bound. NeurIPS Workshop Advances in Approximate Bayesian Infer- ence, volume 1, pp. 2, 2016.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. TPAMI, 2021.
Jiri Hron, Alexander G. de G. Matthews, and Zoubin Ghahramani. Variational gaussian dropout is not bayesian. In NeurIPS, 2017.
Jiri Hron, Alexander G. de G. Matthews, and Zoubin Ghahramani. Variational bayesian dropout: pitfalls and fixes. In ICML, 2018.
Ekaterina lakovleva, Jakob Verbeek, and Karteek Alahari. Meta- Meeta-lerning with shared amortized variational inference. In JCML, pp.
PMLR, 2020.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gareel,, Ali Eslami. Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In ICML, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2013.
Diederik P Kingma, Danilo J Rezende, Shakir Mohamed, and Max Welling. Semi-supervised learn- ing with deep generative models. In NeurlPS, 2014.
Durk P. Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. In NeurIPS, pp. 2575-2583, 2015.
Durk P Kingma, Tim Salimans, Rafal Jozefowiez, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In NewrlPS, pp. 4743-4751, 2016.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 133--133), 2015.
Tuan Anh Le, Hyunjik Kim, Marta Gamelo,, Dan Rosenbaum, Jonathan Schwarz, and Yee Whye Teh. Empirical evaluation neural process objectives. In NeurIPS Workshop Bayesian Deep Learning, 2018
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, 1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):336444, 2015.
Hae Beom Lec, Taewook Nam, Eunho Yang, and Sung Ju Hwang. Meta dropout: Learning to perturb latent features for generalization. In ICLR, 2019a.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set eermutationinvvaiiantt transformer: A framework for attention-baser permutation invariant neural networks, In ICML, 2019b.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In CVPR, pp. 10657-10665. 2019c.

---

Published as a conference paper at ICLR 2022
Larry Leemis. The product of mutually independent bernoulli random variables is bernoulli. http: wath. wm. edu/-leemis htt/sccat///a/eeeeeee /ttp:arr//////////////ooooo pfp 2020.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few- shot learning. arXiv preprint arXiv:7707.09$3. 2017.
Yuhang Liu, Wenyong Dong, Lei Zhang. Dong Gong, and Qinfeng Shi. Variational bayesian dropout with a hierarchical prior. In CVPR, pp. 7124-7133, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
James Lucas, George Tucker, Roger B Grosse, and Mohammad Norouzi. Don't 't blame the elbo! linear vae perspective on posterior collapse. NeurlPS, pp. 9403-9413, 2019.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural 2017.
networks. In ICML, pp.
Devang Naik and Richard Mammone. Meta-neural networks that learn by learning. In IJCNV, volume 1, pp. 437-442. IEEE, 1992.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, pp. 814, 2010.
Cuong Nguyen, Thanh-Toar Do, and Gustavo Carneiro. Uncertainty in model-agnostic meta- learning using variational inference. In WACV, pp. 3090-3100, 2020.
Bernt Øksendal. Stochastic differential equations. Springer, 2003.
Carl Edward Rasmussen and Christopher Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). MIT Press, 2005.
Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning In ICLR, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017.
Jürgen Schmidhuber. Evolutionary principles self-referential learning, or on learning how to learn: the mela-mela... hook. PhD thesis, Technische Universität München, 1987.
Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for few-shot learning. In NeurlPS, 2017.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. How to train deep variational autoencoders and probabilistic ladder networks. In ICML, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, pp. 1929-1958 2014.
Hiroshi Takahashi, Tomoharu Iwata, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi. Varia- tional autoencoder with implicit optimal priors. In AAAI, pp. 5066-5073, 2019.
Yee Whye Teh and IMSM Lecture. On statistical thinking in deep learning blog
post. https:// ptttps://isstee http://mttat.ttt/et www.oo/gg/g/ooooooooooeeeeeeeeeo wp- http://ppcootttt/oppo//ooocaeaaa wtwpbbulltttttnn LLNGmml--OOGGOO0 Statistical Chatistical_Thin Thi nk Bikiing_ff ig_in Bie_____eeee__eeeeee Leee___earning..d 2019.
Jakub Tomezak and Max Welling. Vae with vampprior. In AISTATS, pp. 1214- 1223, 2018.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In NeurlPS, 2016.
Michael Volpp, Fabian Flürenbrock, Lukas Grossberger, Christian Daniel, and Gerhard Neumann. Bayesian context aggregation for neural processes. ICLR, 2020.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In ICMI., pp. 1058-1066, 2013.
12

---

Published as a conference paper at ICLR 2022
Sida Wang and Christopher Manning. Fast dropout training. In ICML, pp. 118 118-126, 2013.
Serena Yeung, Anitha Kannan, Yann Dauphin, and Li Fei-Fei. Tackling over-pruning in variational autoencoders. In ICML, 2017.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In NeurIPS, pp. 7343-7353, 2018.
A DERIVATION OF THE ELBO OF NVDPS
This section describes a detailed derivation of the evidence lower-bound (ELBO) of NVDPs in equation 5 Given the context data D' (x y' and target data D (x* y²). the KL divergence between the true unknown posterior distribution over parameter p(o' ,x'")) variational posterior (D)) is given by:
yy) and the (conditional)
T E/ a(0')D') log ((6'(D() p(0² x² yd) do
DeL De)
g0 ID'c P'))p(''''' dos ply' xx 0² p(
((011D))) log
/ ((00)D')) log
106 (Dc) p(t))
log logp(y"(x') logg(y' xt) Jais
Σ (aal (D) + looloogy"y xx') (oy x²))
(8)
The step (7) is due Bayes rule of p(( ,y) ply" x((0") independent of x By reordering (8), we get
where of is often assumed to be
ply*
ply xx)=
E R logply Dxu(g(o'DDolllo(()))
Rotor Dog x², (()] (
(10)
due to the non-negativity of the omitted term In the lower-bound of (9), the choice of p(0²) is often dif- ficult since the biased prior can lead to over-fitting under-fittin of the model. However, some recent studies of the amortized VI inference (Tomzaak & Welling 2018 Takahashi et al.) 2019) analytically discussed that the optimal prior the amortized variational inference is the aggregated conditional posterior model on whole dataset: (Ø) Sp q (DD((). Usually, the aggregated posterior cannot be calculated in a closed form due to the expensive computation cost of the integral. However, the aggregation posterior is constructed based on the set representation. This motivates us to define the conditional posterior given the whole task dataset as empirical approximation of the optimal prior (i.e., p" (ő') zs )D'). We call this a variational prior. Thus, the approximation p'()) in (9) with the variational prior yields the approximated lower bound of (10).
The lower-bound Ei= (o'
the whole dataset in the model-based conditional
B DERIVATION OF THE KL DIVERGENCE IN THE ELBO OF NVDPs
An essential requirement in the choice the prior is that the analytical derivation of the KL diver- gence term in equation (10) should not depend on the deterministic NN parameter (Kingma al.) 2013] Molchanov et al. 2017) Hron et al.) 2018 2017 Liu et al., 2019). This allows the constant optimization of the FL w.n.t all the independent variational parameters (i.e. O and P Pika for
13

---

Published as a conference paper at ICLR 2022
all - 1 = I. and d = 1..... D). In fact, the conditional posterior defined in (2) is
Gaussian distribution, thus (L(g(0 | ()) is analytically defined as:
P'(1 - P'(1--')0' + ((1 P') (1 p')))2
PI(1 P:)
1
+ +lo
(11)
2P - Pr)62
V /PI(1
Pt
2
- P'(1-P')88 PP''8 + (p'- -P')288
1
P¹(1 P') BR 1
+ log
(12)
2P((1 - -PI)OR
2
P1(1
PT) RR
2
(1 - PO) + (P'-Pt)
P²( - PT) 1
+ log
(13)
2P'(1-P) P1) P!)
PL( - ('(1-Pt) Pt)
2
where both P and P the dropout rates predicted from the meta-model via the equation (3) but with different conditional set information: P is obtained from the small context set D'c while P is from the whole task set D'. (11) is derived using the analytical formulation of KL divergence be- tween two Gaussian distributions. (13) is equivalent to the KL term defined in (4) of the manuscript (except that the constant term 1/2 is omitted for brevity). Interestingly, the analytical derivation of the KL is independent of the shared parameter 9, thus this satisfies the necessary condition to be used as prior in the VI optimization of the dropoul posterior.
The KI divergence in (13) intuitively means that the dropout rates predicted from a small context set should be close to the dropout rates predicted from a much larger context set while training. The experiments validated that this surprisingly works well to induce a robust conditional functional uncertainty. However, one practical issue while training the dropout rate with the KL term (13) that the dropout rate could converge to zeros during the early training period due to the larger gradients from the KL than from the likelihood D In practice, we adopt the dropout rate clipping technique often used in other Variational Dropout approaches (Kingma et al.] (2015) Hron et al, 2018 [Liu et al.) 2019; Molchanov et al., 2017 Gal & Ghahramani 2016] Liu et al.) 2019). We use the dropout rate range of (0.01, 0.99) for all experiments.
C ADDITIONAL EXPERIMENTAL DETAILS ON THE ID REGRESSION.
Setup. We explored the 1D function regression on the data generated from the synthetic GPs with varying kernely in the previous work (Garnelo et al. 201 8b a Kim et al. 2019) that is suitable for measuring the uncertainty adaptation. For the baseline models, Neural Process model (NP) and NP with additional deterministic representation (NP+CNP) described in (Garnelo et al. 201 8b Kim cet al., 20099p is compared. We also adopted the variational prior (VP) into the representation-baate posterior of NP and its variants (NP+VP and NP+CNP+VP), For all models (including our NVDP), we used the same fixed variance and learned variance likelihood architecture depicted in (Kim et al, 2019]: the agent NN with 4 hidden layers of 128 units with LeLU activation (Nair & Hinton 2010) and an output layer of 1 unit for mean (or an additional unit for variance). The dimensions of the set representation r' were fixed to 128. The meta NNs in the conditional dropout posterior in NVDPs have 4 hidden layers of 128 units with LeakyReLU and an output layer of 257 units (i.c. K+D+1) for each layer of the agent model. All models were trained with Adam optimizer (Kingma & Ba 2013) with learning rate 5e-4 for 0.5 million iterations. We draw 16 functions (a batch) from GPs at each iteration. Specifically, at every training step, we draw 16 functions f(-) from GP prior with the squared-exponential kernels, k(z,x') = of exp(- (x - x²)² /2/2) generated with length- scale ~ (0.1.006) and function noise level o ~ (00.1, Then, values was uniformly sampled from [-2, and corresponding y value was determined by the randomly drawn function split into disjoint sample of contexts
(i.e. y fx), ~ GP). And the task data points and n targets as m ~ U(3,97 .7) and n ~ Umm + 1, 100), respectively. In the test or validation, the numbers of contexts and targets were chosen as m U(3, 97) and n 400 m, respectively. 50000 functions were sampled from GPs to compute the log-likelihood (LL) and other scores for validation.
It also turns out to be a peating-point exception prohlem. We later set the minimum noise (i.c., eps) to le 10 in the calculation of log, sqrt, and division function, and the collapsing problem did not occur again. https: thub. com coom/deepnn n -processee
14

---

Published as a conference paper at ICLR 2022
The architecture of NVDP model is follows:
(Deterministic) Feature Encoder. r(x², y')) 2 x |C|
linerelu
128 |C
mean
128.
times
Decoder (Agent), f(x))= linerelu 128 lin+reln
split (1, σ) (where a = 0.1 + 0.9. softplus(losst)).
Meta Model. (r) : dr lin+leakyRel.
128
lintkeakRReL
(())
spli (a, b.
times
where (r) is the meta NN for the 1-th layer of the decoder, and K()) x D is the number of parameters in the 1-th layer.
GP Dataset
CNP
Fixed
LL -0.94(+0.05)
Variance
RLL 0093(100.01)
PLL 00.44(10.05)
Learned
LL
0.72(10.54)
Variance RLL 1.03(++0.8)
PLL 0.69(+0.53)
Table 4: An additional summary of the 1D regression with the GP with random kernel dataset. The deterministic baseline (CNP) is presented. We could observe that the performance the CNP is close to slightly better than the NP+CNP in Tables I of the manuscript. However, the CNP model could lose the functional variability as shown in Figure 2.
D ADDITIONAL EXPERIMENTAL DETAILS ON THE IMAGE COMPLETION TASK.
Setup. The image completion tasks are performed to validate the performance of the models in more complex function spaces (Garnelo al. 20T8b a Kim et al. 2019) Here, treat the image samples from MNIST LeCunn et al. 1998) and CelebA (Liu et al. (0155) as unknown functions. The task predict mapping from normalized 2D pixel coordinates x; (E [0, 1J²) to pixel intensities (E RR for greyscale, E R for RGB) given some context points. For 2D regression experiments, we used the same learned variance baselines implemented in the GP data regression task, except the input and output of the decoder are changed according to dataset, e.g., 24 E [0, 1J², and yi E R 1024 for CelebA), At each iteration, the images in the training
for MNIST (or R and set are split into S context and N target points (e.g., S U(3. 197), and S ~ (3, 197). = 784 at validation). Adam optimizer with a learning rate 4e-4 and 16 batches with 300 epochs were used for training. The validation is performed the separated validation images set. To see the generalization performance on a completely new dataset, we also tested the models trained on MNIST the Omniglot validation set.
U[N + 1, 200) train
E ADDITIONAL EXPERIMENTS ON ID FUNCTION REGRESSION ON A TOY TRIGONOMETRY DATASET
Setup. To further investigate how the proposed meta-model utilizes the common structures of the NN parameters, trained small NVDP model (of the size of 13-12-12-2)
mixture of scaled trigonometric functions: x is sampled in a range of [-T, т]. and is determined by т) where is one of three functions sine, cosine, and tanh with the probability of one third, and a ~ ((155, 2) and b ~ ((-0.1, 0.1.) We used the training procedures in the next section except the the simple model architecture and learning rate 5c - 4.
f(2 *
For the small NVDP model on ID function regression with Trigonometry dataset, the following architecture was employed:
15

---

Published as a conference paper at ICLR 2022
2
NP
-2
-2,
2
0
2
2
NP+VP
0
-2-
-22
2
2
2
NP+CNP
0
0
-2
-2
2
2
2
2
NVDP
0
-2
-2
2
Sample 1
Sample 2
Figure 5: The additional 1D few-shot regression results of the models on GP dataset fixed variance settings. The black dotted lines represent the true unknown task functions. Black dots are few context points (S 5) given to the posteriors. The blue lines are mean values predicted from sampled NNs.
(Deterministic) Feature Encoder. r(x;,
IC
linerelu 12
IC]
meam 12.
times
16

---

Published as a conference paper at ICLR 2022
NP
-22
-2-
0
O
2
NP+VP
-2
-2_
0
2
2
NP+CNP
-2 2
-2
2
NVDP
-22
-2
0
2
0
2
Sample 1
Sample 2
Figure 6: The additional 1D few-shot regression results of the models on GP dataset in learned variance settings. The black (dash-line) represent the true unknown task function. Black dots are a few context points (S = 5) given to the posteriors. The blue lines and light blue area are mean values and variance predicted from the sampled NNs, respectively.
Decoder (Agent). ((x,r) (1 d,)
lintrelu 12
linerelu
split (v, (where 0.1 + 0.9- soft-
times
plus(logstd)).
17

---

Published as a conference paper at ICLR 2022
Image Dataset
CNP
MNIST
LL 0.87(=0.16)
RLL 1.14(_0.08)
PLL 0.83(10.15)
MNIST
LL 0.68(-0.11)
to Omniglot
RLL 0.99(=0.08)
PLL 0.64(=0.13)
CelebA
LL 0.78(=0.15
RLL 0.93(=0.05)
PLL 0.77(=
Table 5: An additional summary of the 2D image completion tasks on the MNIST, CelebA, and Omniglot dataset. The deterministic baseline (CNP) is presented. We could observe that the per- formance of the CNP is close to or slightly better than the NP+CNP in Tables 2 of the manuscript. However, the CNP model could lose the functional variability as shown in Figure 4.
10
30
100
half
784
10
30
100
half 784
Conoex
4
S
:
NP
Stmple Sample
7
4
4
4
7
C
4
4
4
0
4
NP+VP
Sample 2 Sample
4
q
4
5
9
4
9
4
5
9
NP+CNP
Sample 2 Sampk
4
4
9
4
1
4
NP+CNP+VP
Sample 2 Sample Sample Sumpa
4
4
5
4
4
5
5
NVDP
4
4
7
5
S
d
4
MNIST
Figure 7: The additional results from the 2D image completion tasks MNIST dataset. Given the observed context points (10, 30, 100, half, and full pixels). the mean values of two independently sampled functions from the models (i.e. NP, NP+CNP and NVDP (ours)) presented.
Meta Model. (r) : d, MineMish
12 lineMish (K()), D(),1) 1) split (a. b. c)
where (r( is the meta NN for the 1-th layer of the decoder, and (()) D()) the number of parameters in the 1-th layer.
18

---

Published as a conference paper at ICLR 2022
10
30
100
half
784
10
30
100
half
784
Conted
2
R
π
NP
Sample Somple
J
5
6
5
9
6
6
6
6
3
3
5
9
6
7
θ
3
0
NP+VP
Sample Sampl
3
3
6
3
8
8
6
3
3
3
3
7
6
6
3
6
NP+CNP
Sample Sample
7
3
3
6
8
8
7
3
3
6
2
8
8
8
NP+CNP+VP
Sample Somple
3
2
5
2
3
8
8
2
2
3
3
3
NVDP
Sample Sample
3
2
2
5
MNIST (train) to Omniglot (test)
Figure 8: The additional results from the 2D image completion tasks on Omniglot dataset. Given the observed context points (10, 30, 100, half, and full pixels), the mean values of two independently sampled functions from the models (i.e. NP, NP+CNP and NVDP (ours)) are presented.
Results. Figure 11 displays the trained NVDP the trigonometry function dataset. The NVDP could capture the probability of dropout for each parameter of the agent, successfully predicting the task-specific trigonometric functions conditioned that the task-specific dropout rates can transform a single conventional NN agent to express multiple functions. is interesting to see that the contexts from sine and cosine functions yield similar dropout rates for the second layer. On the other hand, the contexts from tanh function results in different dropout structures for all layers.
the small (S = context set. This shows
19

---

Published as a conference paper at ICLR 2022
10
30
100
half
1024
10
30
100
half
10
Sanple 2 Sample Sompic 2 Sample 1 Samplle 2 Sample: Contee
NP
NP+VP
NP+CNP
NP+CNP+VP
Sample 2 Sumple Sample Sample 1
NVDP
CelebA
Figure 9: The additional results from the 2D image completion tasks on CelebA dataset. Given the observed context points (10, 30, 100, half, and full pixels), the mean values of two independently sampled functions from the models (i.e. NP, NP+CNP and NVDP (ours)) are presented.
20

---

Published as a conference paper at ICLR 2022
10
30
100
half
1024
10
30
100
half
1024
Conted
NP
Sample 2 Sample 1
NP+VP
Samcle 2 Sample
NP+CNP
Sample Sample
NP+CNP+VP
Sample 2 Sample Sample Sample
NVDP
CelebA
Figure 10: The additional results from the 2D image completion tasks CelebA dataset. Given the observed context points (10, 30, 100, half, and full pixels), the mean values of two independently sampled functions from the models (i.e. NP, NP+CNP and NVDP (ours)) presented.
(a)
[b]
075
10
30
10
10
(c)*
(d)
10
10
Figure 11: (a) sine, (b) cosine, (C) tanh function (left) with the probabilities of using parameters (1- P6) (right) predicted with small NVDPs (13-12-12-2) conditioned on 4-shot context points (black dots), and (d) the trigonometry dataset (left) and the deterministic shared NN parameters 0 (right).
21


Reviews:
[{"review_id": "wZ0RvQZUbSi", "reviewer": "Reviewer_HrLQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Summary\nThe authors propose Neural Variational Dropout Processes (NVDPs), a novel approach to meta-learning. First, the task of meta-learning is interpreted as performing Bayesian inference of the model parameters under different environments (multi-task learning). For training, the authors perform approximate Bayesian inference by optimizing a variational lower bound on the multi-task likelihood (eq. (1)). Two neural networks are trained: a base neural network with deterministic parameters that is used across tasks, and a meta neural network that outputs task-specific dropout rates for the weights of the base network given a small support set of data points S (amortized posterior). The authors link their approach to variational dropout (Kingma et al., 2015), but utilize Bernoulli dropout instead of a Gaussian approximation. For regularization the authors use a 'variational prior' that is essentially the aggregated posterior over all data points in a task. This formulation of approximate posterior and variational prior allows the authors to derive a KL-term that is independent of the base network's deterministic parameters (appendix B).\n\nThe authors place their work in the context of model-based approximate posterior approaches (Gordon et al., 2018; Garnelo et al., 2018a;b; Kim et al., 2019; Iakovleva et al., 2020) which all utilize a formulation akin to (eq. 1), with a semantic differences in the outputs of the approximate posterior. The authors claim that their formulation overcomes the scalability problems of VERSA (Gordon et al., 2018), and avoids posterior collapse as is sometimes observed in Neural Processes (NPs) (Garnelo et al., 2018a;b).\n\nThe proposed method is evaluated on several regression and classifications problems: \n\n1. A 1d regression problem where tasks are sampled from a Gaussian Process\n2. Active learning on the above task\n3. Image completion on CelebA, MNIST, and Omniglot (only test)\n4. Few-shot classification on Omniglot and MiniImageNet\n\nIn all problems domains, the results suggest either state-of-the-art or compentitive performance of NVDPs as compared to other approaches.\n\n## Opinion & Recommendation\nI am in support of accepting this paper. The paper makes a nice connection between meta-learning and variational dropout, resulting in an overall elegant approach. The use of dropout for adaptation of a model to different tasks is (as far as I am aware) novel, which would already justify publication. NVDPs are closely related to Neural Processes (NPs). However NPs can suffer from the fact that the outputs of the approximate posterior are ignored by the downstream decoder model. This is not the case for NVDPs as they directly modify the decoder through the dropout process.\n\nThe presented results give a mostly positive impression of the approach. Judging from the image completion problems, however, the limitations of all compared approaches (including NVDPs) becomes apparent. Given a fully sampled support set the resulting image reconstructions are still very blurry. See Kim et al. (2019, fig. 4a) for much more crisp reconstructions. The author's claim \"On the other hand, the samples from NVDPs exhibit sharp reconstruction results while also showing a reasonable amount of variability.\" is in my option an overstatement.\n\n## Questions\n- The authors mention at the end of section 3 that the local reparametrization trick is used for variance reduction. But it looks like it is still necessary loop over different tasks *t*. Is this true? If so, could you discuss the computational limitations of this for large *T* ?\n- Eq. (3) gives a low rank approximation of the dropout rate *P*. It is a product of 3 Bernoulli probabilities. Could you make some comments about the numerical stability of this approximation?", "summary_of_the_paper": "I will put a few bullet points here. For a more detailed summary, see below.\n- Novel meta-learning approach\n- Utilises dropout to adapt a base network to new tasks\n- Amortised variational inference is used to train meta model and base model jointly\n- The meta model outputs task specific dropout rates\n- This results in a conditional dropout posterior akin to variational dropout\n- A *variational prior* over task-dependent dropout rates allows simple calculation of the KL-divergence\n- The approach is evaluated in 4 different problem setting. SOTA or competitive in most.", "summary_of_the_review": "I recommend acceptance of the presented paper. A novel idea, combined with a good practical justification, and strong results make this paper worth for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gbF8fvBDOPH", "reviewer": "Reviewer_uz6u", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Some strengths:\n1. The newly proposed NVDPs are more robust to the model-collapsing or overfitting issues, comparing with existing approaches.\n2. The proposed variational prior can regularize the task-specific dropout rates in the optimization of the conditional dropout posterior.\n3. Comparing with previous related approaches, the complexity is greatly reduced, making the algorithm more applicable to larger data set with scalability.\n\nSome weaknesses:\n1. This approach still falls into the SGVB framework when coming to the optimization of the variational parameters, using the re-parameterization trick. This limits its novelty, and restrains the potential of this framework to investigate more complicated distributions other than Bernoulli.\n2. Recently, meta learning has been popular in some application domains, e.g., NLP. The experiments are majorly focused on highly optimized data sets, weakening its wider application on more practical tasks and usage in real scenarios.\n", "summary_of_the_paper": "The authors present an interesting model-based meta-learning approach, Neural Variational Dropout Processes (NVDP), by constructing a conditional dropout posterior to predict task-specific dropout rates of model parameters, conditioned on the observations. Together with NVDP, the authors also propose a new type of prior, the task-specific variational prior, approximated by the same posterior model conditioned on the whole task data, which supports robust training. They show the NVDPs improve the model's adaptation, functional variability and generalization to new tasks empirically in the experiments.", "summary_of_the_review": "I recommend this paper to be accepted, as its novelty on the newly developed variational prior differs from existing works. This is based on the hypothesis that the conditional posterior model to approximate the optimal prior shall depend on the whole data set. This claim is supported by the assumption that the variational model conditioned on the aggregated set with larger context shall be closer to the optimal task-specific prior, than only conditioned on a subset, which is a specific task. In the experiments, the variational prior shows a robust regularization effect.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8s4rAPLNXcn", "reviewer": "Reviewer_8oez", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ The paper is clearly written and easy to follow.\n+ The evaluation of NP-based methods using Reconstructive Log-likelihood (RLL) and predictive log-likelihood (PLL) is an interesting idea.\n+ The reasoning of the novel prior is clear and understandable. \n+ The experiments are comprehensive.\n\nWeakness: \n- Predictive accuracy (Table 3) alone is not sufficient to evaluate the strength of the uncertainty-aware few shot classification model, i.e. many recent few-shot models can easily perform significantly better than the results presented here (e.g. Meta-Learning with Differentiable Convex Optimization).\n- Experiments to demonstrate the applicability of uncertainty estimates (e.g., an active learning setting similar to Bayesian MAML) will further strengthen the results.\n- The comparison with CNP models alone is missing. The evaluation is done with NP + CNP instead.\n- Comparison with state-of-the-arts NP works will help better evaluate the model’s performance, for example, \"Convolutional conditional neural processes, Gordon et al. ICLR 2020\", \"Meta-learning stationary stochastic process prediction with convolutional neural processes,\nFoong et al, NeurIPS 2020\", or \"Attentive neural processes, Kim et al, ICLR 2019\".\n- Although the authors claim to address the issues such as mode-collapsing or over-fitting behavior in the context of meta-learning, the results to support the claim are missing in the paper.\n\n", "summary_of_the_paper": "The paper proposes a novel Bayesian meta-learning approach to model conditional posterior distribution via an amortized variational inference framework based on task-specific dropout. For tasks with few learning examples, the paper discusses the a novel low-rank product of Bernoulli expert meta-models to efficiently obtain full conditional posterior model over the neural network’s parameters. Few-shot learning experiments from Gaussian Process samples show that the model achieves better performance. The proposed method also outperforms existing models on active learning task with better uncertainty calibration. The method also shows better performance on image completion and few-shot classification tasks.", "summary_of_the_review": "an interesting idea, but there is room to further strengthen the paper, particularly for the experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel model-based Bayesian meta-learning approach that combines a novel conditional dropout posterior a new variational prior for the data-efficient learning and adaptation of deep neural networks. It is applied to tasks such as 1D stochastic regression, image inpainting, and classification.

Overall, this paper received positive reviews: Reviewers thought that the "the paper makes a nice connection between meta-learning and variational dropout, resulting in an overall elegant approach" and that the "reasoning of the novel prior is clear and understandable" while the "experiments are comprehensive".

Given the agreement of the reviewers and the novel use of dropout for adaptation of a model to different tasks, I recommend accepting this paper.
Decision: Accept (Poster)