Neural Variational Dropout Processes

1 INTRODUCTION . In traditional machine learning , a large amount of labeled data is required to train deep models ( LeCun et al. , 2015 ) . In practice , however , there are many cases where it is impossible to collect sufficient data for a given task . On the other hand , humans can quickly understand and solve a new task even from few examples ( Schmidhuber , 1987 ; Andrychowicz et al. , 2016 ) . This distinguishing characteristic of humans is referred to as the meta-learning ability , which enables them to accumulate the past learning experiences into general knowledge and to utilize it for efficient learning in the future . Incorporating the meta-learning capability into artificial machines has long been an active research topic of machine learning ( Naik & Mammone , 1992 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ; Ravi & Larochelle , 2017 ; Finn et al. , 2017 ; Lee et al. , 2019c ; Hospedales et al. , 2021 ) . Recently , Bayesian meta-learning methods have been attracting considerable interest due to incorporating the uncertainty quantification of Bayesian frameworks into the efficient model adaptation of meta-learning approaches . An earlier study ( Grant et al. , 2018 ) extended the optimization-based deterministic approach model-agnostic meta-learning ( MAML ) ( Finn et al. , 2017 ) to a hierarchical Bayesian frameworks ( Daumé III , 2009 ) . Later , the optimization-based variational inference ( Yoon et al. , 2018 ; Finn et al. , 2018 ; Ravi & Beatson , 2019 ; Lee et al. , 2019a ; Nguyen et al. , 2020 ) and the model-based Bayesian meta-learning ( Gordon et al. , 2018 ; Garnelo et al. , 2018a ; b ; Iakovleva et al. , 2020 ) methods were presented . These methods achieved outstanding outcomes in various few-shot1 regression and classification tasks . In fact , the adaptation of deep models that can estimate uncertainties is a key study for reliable machine learning systems in real-world applications with high judgmental risks such as medical AI or autonomous driving systems . Inspired by the recent studies , we propose a new model-based Bayesian meta-learning approach , neural variational dropout processes ( NVDPs ) . The main contribution of this work is to design a new type of Neural Network ( NN ) based conditional posterior model that can bypass the under-fitting and posterior-collapsing of the existing approaches ( Gordon et al. , 2018 ; Garnelo et al. , 2018a ; b ; Iakovleva et al. , 2020 ) . NVDPs extend the simple yet effective posterior modeling of Variational Dropout ( VD ) ( Kingma et al. , 2015 ; Gal & Ghahramani , 2016 ; Molchanov et al. , 2017 ; Hron et al. , 2018 ; Liu et al. , 2019 ) in the context of meta-learning ; a novel low-rank product of Bernoulli experts metamodel is utilized in the dropout posterior to learn task-specific dropout rates conditioned on a few 1The few-shot learning assumes only a few examples are available for each task , with the number of tasks being large ( Lake et al. , 2015 ) . learning examples . In this way , a full conditional posterior model over the NN ’ s parameters can be efficiently obtained . In addition , we also propose a new type of prior to optimize the conditional dropout posterior in variational inference , which supports the robust training of conditional posteriors . Although it is developed in conjunction with NVDPs , the formulation allows its adoption to other methods , and we show the effectiveness . We have evaluated NVDPs compared with other recent methods on various few-shot learning tasks and datasets . The experiments show that NVDPs can circumvent the posterior-collapsing and under-fitting and achieve outstanding performances in terms of log-likelihood , active learning efficiency , prediction accuracy , and generalization . 2 BAYESIAN META-LEARNING . 2.1 AMORTIZED VARIATIONAL INFERENCE FRAMEWORK IN THE MULTI-TASK DATA . A goal in meta-learning is to construct models that can quickly solve new tasks from small amounts of observed data . To achieve this , it is important to learn general ( or task-invariant ) structures from multiple tasks that can be utilized for the efficient model adaptation when necessary . Bayesian metalearning ( Gordon et al. , 2018 ; Garnelo et al. , 2018b ; Ravi & Beatson , 2019 ) formulates this objective of meta-learning as an amortized variational inference ( VI ) of the posterior distribution in multi-task environments . Suppose a collection of T related tasks is given , and each t-th task has the training data Dt containing N i.i.d . observed tuples ( xt , yt ) = ( xti , yti ) Ni=1 . Then , the evidence lower-bound ( ELBO ) over the log-likelihood of the multi-task dataset can be derived as : T∑ t=1 log p ( Dt ) ≥ T∑ t=1 { Eq ( φt|Dt ; θ ) [ log p ( yt|xt , φt ) ] − KL ( q ( φt|Dt ; θ ) ||p ( φt ) ) } . ( 1 ) Here , the p ( yt|xt , φt ) is the likelihood ( or NN model ) on t-th training data and φt is t-th task-specific variable ( i.e . a latent representation or weights of NN ) and q ( φt|Dt ; θ ) is a tractable amortized variational posterior model utilized for the approximation of the true unknown posterior distribution of φt for each given t-th task ( i.e p ( φt|Dt ) ) ( Kingma & Welling , 2013 ; Gordon et al. , 2018 ; Garnelo et al. , 2018b ; Ravi & Beatson , 2019 ; Iakovleva et al. , 2020 ) . The parameter θ represents the common structures that can be efficiently learned across multiple different tasks . The pre-specified prior distribution p ( φt ) in the KL term provides a stochastic regularization that can help to capture the task-conditional uncertainty and prevent the collapsing of q ( φt|Dt ; θ ) . In fact , the maximization of the the evidence lower bound ( ELBO ) , right side of equation 1 , with respect to the conditional variational posterior model is equivalent to the minimization of ∑T t=1 KL ( q ( φ t|Dt ; θ ) ||p ( φt|Dt ) ) . Essentially , the goal in the amortized VI of Bayesian meta-learning is to learn the inference process of the conditional posterior distribution via the variational model q ( φt|Dt ; θ ) and the shared general structure θ across multiple tasks since these task-invariant knowledge can later be utilized for the efficient adaption of the NN function on new unseen tasks . The approximation of the conditional posterior also enables the ensemble model approaches and the uncertainty quantification . 2.2 THE EXISTING CONDITIONAL POSTERIOR AND PRIOR MODELINGS . Many of the recent Bayesian meta-learning approaches can be roughly categorized into the optimization-based ( Grant et al. , 2018 ; Yoon et al. , 2018 ; Finn et al. , 2018 ; Ravi & Beatson , 2019 ; Lee et al. , 2019a ; Nguyen et al. , 2020 ) or the model-based posterior approximation approaches ( Gordon et al. , 2018 ; Garnelo et al. , 2018a ; b ; Kim et al. , 2019 ; Iakovleva et al. , 2020 ) . In the optimizationbased approaches , the task-specific variable φt can be seen as the adapted NN weights . For example , the deterministic model-agnostic meta-learning ( MAML ) ( Finn et al. , 2017 ) can be considered as learning a Dirac delta variational posterior modeling ( i.e. , q ( φt|DtC ; θ ) ≈ δ ( φt − SGDj ( DtC , θ ) ) ) where the goal is to learn the shared global initialization NN ’ s parameters θ such that a few j steps of SGD updates on the small subset2 DtC of the t-th dataset Dt provides a good approximation of the task-specific weights φt . Many optimization-based Bayesian meta-learning approaches such as LLAMA ( Grant et al. , 2018 ) , PLATIPIS ( Finn et al. , 2018 ) , BMAML ( Yoon et al. , 2018 ) , and ABML ( Ravi & Beatson , 2019 ) have incorporated Gaussian type of posterior and prior models into 2The subset DtC ( ⊆ Dt ) is also known as a context set for each task . A small size S of context set ( e.g. , 1-shot , 5-shot , or random ) is often used in the few-shot learning tasks ( Lake et al. , 2015 ) . the deterministic adaptation framework of MAML in order to improve the robustness of models . The optimization-based approaches are applicable to various types of few-shot learning tasks , but the adaptation cost at test time is computationally expensive due to the inversion of Hessian or kernel matrix ; this may restrict their usage in the environments with low computational resources . On the other hand , the model-based Bayesian meta-learning approaches such as VERSA ( Gordon et al. , 2018 ) or Neural Processes ( Garnelo et al. , 2018a ; b ) allow an instant estimation of the Bayesian predictive distribution at test time via NN-based conditional posterior modeling . VERSA employs an additional NN-based meta-model gθ ( · ) to directly predict the Gaussian posterior of agent NN ’ s task-specific weight φt from the small context set ( i.e. , q ( φt|DtC ; θ ) = N ( φt| ( µ , σ ) = gθ ( DtC ) ) ) . In this case , the shared structure θ represents the meta-model ’ s parameters . However , the direct approximation of agent NN weights in VERSA could limit their scalability due to the large dimensionality of the weights ; they only consider the task-specific weights for the single softmax output layer . In addition , VERSA did not utilize any task-specific prior p ( φt ) , so the conditional posterior could turn into a deterministic one while training with the Monte Carlo approximation ( Iakovleva et al. , 2020 ) . Instead of the NN ’ s weights , the Neural Processes ( NPs ) model the conditional posterior with a latent representation ( i.e . q ( zt|Dt ; θ ) = N ( zt| ( µ , σ ) = gθ ( Dt ) ) ) ( Garnelo et al. , 2018b ) . Regarding the task-specific latent representation as a second input to the likelihood p ( yt|xt , zt ) ( similar to Kingma & Welling ( 2013 ) or Edwards & Storkey ( 2017 ) ) provides an efficient way to transfer the conditional information ( Garnelo et al. , 2018b ) . However , the following studies indicate that NPs often suffer from under-fitting ( Kim et al. , 2019 ) and posterior-collapsing ( Grover et al. , 2019 ; Le et al. , 2018 ) . We conjecture that these behaviors are partially inherent to its original variational formulation ; the conditional posteriors of NPs depend on the whole task data while training , which did not empirically show good generalization performance at test . In addition , it is well known that the highly flexible NN-based likelihood can neglect the latent representation in variational inference ( Hoffman & Johnson , 2016 ; Sønderby et al. , 2016 ; Kingma et al. , 2016 ; Chen et al. , 2017 ; Yeung et al. , 2017 ; Alemi et al. , 2018 ; Lucas et al. , 2019 ; Dieng et al. , 2018 ) ; we also demonstrate the posterior collapsing of the NPs in our experiments .

Summary: The authors present an interesting model-based meta-learning approach, Neural Variational Dropout Processes (NVDP), by constructing a conditional dropout posterior to predict task-specific dropout rates of model parameters, conditioned on the observations. Together with NVDP, the authors also propose a new type of prior, the task-specific variational prior, approximated by the same posterior model conditioned on the whole task data, which supports robust training. They show the NVDPs improve the model's adaptation, functional variability and generalization to new tasks empirically in the experiments.