Neural Variational Dropout Processes

1 INTRODUCTION . In traditional machine learning , a large amount of labeled data is required to train deep models ( LeCun et al. , 2015 ) . In practice , however , there are many cases where it is impossible to collect sufficient data for a given task . On the other hand , humans can quickly understand and solve a new task even from few examples ( Schmidhuber , 1987 ; Andrychowicz et al. , 2016 ) . This distinguishing characteristic of humans is referred to as the meta-learning ability , which enables them to accumulate the past learning experiences into general knowledge and to utilize it for efficient learning in the future . Incorporating the meta-learning capability into artificial machines has long been an active research topic of machine learning ( Naik & Mammone , 1992 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ; Ravi & Larochelle , 2017 ; Finn et al. , 2017 ; Lee et al. , 2019c ; Hospedales et al. , 2021 ) . Recently , Bayesian meta-learning methods have been attracting considerable interest due to incorporating the uncertainty quantification of Bayesian frameworks into the efficient model adaptation of meta-learning approaches . An earlier study ( Grant et al. , 2018 ) extended the optimization-based deterministic approach model-agnostic meta-learning ( MAML ) ( Finn et al. , 2017 ) to a hierarchical Bayesian frameworks ( Daumé III , 2009 ) . Later , the optimization-based variational inference ( Yoon et al. , 2018 ; Finn et al. , 2018 ; Ravi & Beatson , 2019 ; Lee et al. , 2019a ; Nguyen et al. , 2020 ) and the model-based Bayesian meta-learning ( Gordon et al. , 2018 ; Garnelo et al. , 2018a ; b ; Iakovleva et al. , 2020 ) methods were presented . These methods achieved outstanding outcomes in various few-shot1 regression and classification tasks . In fact , the adaptation of deep models that can estimate uncertainties is a key study for reliable machine learning systems in real-world applications with high judgmental risks such as medical AI or autonomous driving systems . Inspired by the recent studies , we propose a new model-based Bayesian meta-learning approach , neural variational dropout processes ( NVDPs ) . The main contribution of this work is to design a new type of Neural Network ( NN ) based conditional posterior model that can bypass the under-fitting and posterior-collapsing of the existing approaches ( Gordon et al. , 2018 ; Garnelo et al. , 2018a ; b ; Iakovleva et al. , 2020 ) . NVDPs extend the simple yet effective posterior modeling of Variational Dropout ( VD ) ( Kingma et al. , 2015 ; Gal & Ghahramani , 2016 ; Molchanov et al. , 2017 ; Hron et al. , 2018 ; Liu et al. , 2019 ) in the context of meta-learning ; a novel low-rank product of Bernoulli experts metamodel is utilized in the dropout posterior to learn task-specific dropout rates conditioned on a few 1The few-shot learning assumes only a few examples are available for each task , with the number of tasks being large ( Lake et al. , 2015 ) . learning examples . In this way , a full conditional posterior model over the NN ’ s parameters can be efficiently obtained . In addition , we also propose a new type of prior to optimize the conditional dropout posterior in variational inference , which supports the robust training of conditional posteriors . Although it is developed in conjunction with NVDPs , the formulation allows its adoption to other methods , and we show the effectiveness . We have evaluated NVDPs compared with other recent methods on various few-shot learning tasks and datasets . The experiments show that NVDPs can circumvent the posterior-collapsing and under-fitting and achieve outstanding performances in terms of log-likelihood , active learning efficiency , prediction accuracy , and generalization . 2 BAYESIAN META-LEARNING . 2.1 AMORTIZED VARIATIONAL INFERENCE FRAMEWORK IN THE MULTI-TASK DATA . A goal in meta-learning is to construct models that can quickly solve new tasks from small amounts of observed data . To achieve this , it is important to learn general ( or task-invariant ) structures from multiple tasks that can be utilized for the efficient model adaptation when necessary . Bayesian metalearning ( Gordon et al. , 2018 ; Garnelo et al. , 2018b ; Ravi & Beatson , 2019 ) formulates this objective of meta-learning as an amortized variational inference ( VI ) of the posterior distribution in multi-task environments . Suppose a collection of T related tasks is given , and each t-th task has the training data Dt containing N i.i.d . observed tuples ( xt , yt ) = ( xti , yti ) Ni=1 . Then , the evidence lower-bound ( ELBO ) over the log-likelihood of the multi-task dataset can be derived as : T∑ t=1 log p ( Dt ) ≥ T∑ t=1 { Eq ( φt|Dt ; θ ) [ log p ( yt|xt , φt ) ] − KL ( q ( φt|Dt ; θ ) ||p ( φt ) ) } . ( 1 ) Here , the p ( yt|xt , φt ) is the likelihood ( or NN model ) on t-th training data and φt is t-th task-specific variable ( i.e . a latent representation or weights of NN ) and q ( φt|Dt ; θ ) is a tractable amortized variational posterior model utilized for the approximation of the true unknown posterior distribution of φt for each given t-th task ( i.e p ( φt|Dt ) ) ( Kingma & Welling , 2013 ; Gordon et al. , 2018 ; Garnelo et al. , 2018b ; Ravi & Beatson , 2019 ; Iakovleva et al. , 2020 ) . The parameter θ represents the common structures that can be efficiently learned across multiple different tasks . The pre-specified prior distribution p ( φt ) in the KL term provides a stochastic regularization that can help to capture the task-conditional uncertainty and prevent the collapsing of q ( φt|Dt ; θ ) . In fact , the maximization of the the evidence lower bound ( ELBO ) , right side of equation 1 , with respect to the conditional variational posterior model is equivalent to the minimization of ∑T t=1 KL ( q ( φ t|Dt ; θ ) ||p ( φt|Dt ) ) . Essentially , the goal in the amortized VI of Bayesian meta-learning is to learn the inference process of the conditional posterior distribution via the variational model q ( φt|Dt ; θ ) and the shared general structure θ across multiple tasks since these task-invariant knowledge can later be utilized for the efficient adaption of the NN function on new unseen tasks . The approximation of the conditional posterior also enables the ensemble model approaches and the uncertainty quantification . 2.2 THE EXISTING CONDITIONAL POSTERIOR AND PRIOR MODELINGS . Many of the recent Bayesian meta-learning approaches can be roughly categorized into the optimization-based ( Grant et al. , 2018 ; Yoon et al. , 2018 ; Finn et al. , 2018 ; Ravi & Beatson , 2019 ; Lee et al. , 2019a ; Nguyen et al. , 2020 ) or the model-based posterior approximation approaches ( Gordon et al. , 2018 ; Garnelo et al. , 2018a ; b ; Kim et al. , 2019 ; Iakovleva et al. , 2020 ) . In the optimizationbased approaches , the task-specific variable φt can be seen as the adapted NN weights . For example , the deterministic model-agnostic meta-learning ( MAML ) ( Finn et al. , 2017 ) can be considered as learning a Dirac delta variational posterior modeling ( i.e. , q ( φt|DtC ; θ ) ≈ δ ( φt − SGDj ( DtC , θ ) ) ) where the goal is to learn the shared global initialization NN ’ s parameters θ such that a few j steps of SGD updates on the small subset2 DtC of the t-th dataset Dt provides a good approximation of the task-specific weights φt . Many optimization-based Bayesian meta-learning approaches such as LLAMA ( Grant et al. , 2018 ) , PLATIPIS ( Finn et al. , 2018 ) , BMAML ( Yoon et al. , 2018 ) , and ABML ( Ravi & Beatson , 2019 ) have incorporated Gaussian type of posterior and prior models into 2The subset DtC ( ⊆ Dt ) is also known as a context set for each task . A small size S of context set ( e.g. , 1-shot , 5-shot , or random ) is often used in the few-shot learning tasks ( Lake et al. , 2015 ) . the deterministic adaptation framework of MAML in order to improve the robustness of models . The optimization-based approaches are applicable to various types of few-shot learning tasks , but the adaptation cost at test time is computationally expensive due to the inversion of Hessian or kernel matrix ; this may restrict their usage in the environments with low computational resources . On the other hand , the model-based Bayesian meta-learning approaches such as VERSA ( Gordon et al. , 2018 ) or Neural Processes ( Garnelo et al. , 2018a ; b ) allow an instant estimation of the Bayesian predictive distribution at test time via NN-based conditional posterior modeling . VERSA employs an additional NN-based meta-model gθ ( · ) to directly predict the Gaussian posterior of agent NN ’ s task-specific weight φt from the small context set ( i.e. , q ( φt|DtC ; θ ) = N ( φt| ( µ , σ ) = gθ ( DtC ) ) ) . In this case , the shared structure θ represents the meta-model ’ s parameters . However , the direct approximation of agent NN weights in VERSA could limit their scalability due to the large dimensionality of the weights ; they only consider the task-specific weights for the single softmax output layer . In addition , VERSA did not utilize any task-specific prior p ( φt ) , so the conditional posterior could turn into a deterministic one while training with the Monte Carlo approximation ( Iakovleva et al. , 2020 ) . Instead of the NN ’ s weights , the Neural Processes ( NPs ) model the conditional posterior with a latent representation ( i.e . q ( zt|Dt ; θ ) = N ( zt| ( µ , σ ) = gθ ( Dt ) ) ) ( Garnelo et al. , 2018b ) . Regarding the task-specific latent representation as a second input to the likelihood p ( yt|xt , zt ) ( similar to Kingma & Welling ( 2013 ) or Edwards & Storkey ( 2017 ) ) provides an efficient way to transfer the conditional information ( Garnelo et al. , 2018b ) . However , the following studies indicate that NPs often suffer from under-fitting ( Kim et al. , 2019 ) and posterior-collapsing ( Grover et al. , 2019 ; Le et al. , 2018 ) . We conjecture that these behaviors are partially inherent to its original variational formulation ; the conditional posteriors of NPs depend on the whole task data while training , which did not empirically show good generalization performance at test . In addition , it is well known that the highly flexible NN-based likelihood can neglect the latent representation in variational inference ( Hoffman & Johnson , 2016 ; Sønderby et al. , 2016 ; Kingma et al. , 2016 ; Chen et al. , 2017 ; Yeung et al. , 2017 ; Alemi et al. , 2018 ; Lucas et al. , 2019 ; Dieng et al. , 2018 ) ; we also demonstrate the posterior collapsing of the NPs in our experiments .

Summary: I will put a few bullet points here. For a more detailed summary, see below. - Novel meta-learning approach - Utilises dropout to adapt a base network to new tasks - Amortised variational inference is used to train meta model and base model jointly - The meta model outputs task specific dropout rates - This results in a conditional dropout posterior akin to variational dropout - A *variational prior* over task-dependent dropout rates allows simple calculation of the KL-divergence - The approach is evaluated in 4 different problem setting. SOTA or competitive in most.