Non-Gaussian Gaussian Processes for Few-Shot Regression

1 Introduction . Gaussian Processes ( GPs ) [ 33 , 46 ] are one of the most important probabilistic methods , and they have been widely used to model distributions over functions in a variety of applications such as multi-modal regression [ 56 ] , time-series prediction [ 3 , 27 ] and meta-learning [ 29 , 45 ] . Recent works propose to use GPs in the few-shot learning scenario [ 4 , 29 , 39 , 49 ] , where the model is trained to solve a supervised task with only a few labeled samples available . This particular application is well-fitted to GPs since they can determine the posterior distribution in closed-form from a small set of data samples [ 29 ] . ⇤Corresponding author : marcin.sendera @ doctoral.uj.edu.pl 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , the generalization capabilities of GPs come at the price of reduced flexibility when the modeled distributions are complex , e.g. , they have high skewness or heavy tails . Furthermore , GPs assume a high similarity between subsequent tasks . This condition is rarely met in realworld applications where tasks can vary during time , as is the case in heteroscedastic regression . These limitations of GPs also extend to multi-modal learning or , more generally , to multi-label regression [ 56 ] . In this work , we address those drawbacks by modeling the GPs posterior predictive distributions with a local non-Gaussian approximation . We do so by introducing a new method that we have named Non-Gaussian Gaussian Processes ( NGGPs ) . In NGGPs , we leverage the flexibility of Continuous Normalizing Flows ( CNF ) [ 16 ] to model arbitrary probability distributions . In particular , we propose an invertible ODE-based mapping that operates on each component of the random variable vectors . This way , we can compute a set of CNFs parameters shared across all vectors , with the resulting mapping incorporating the information of the context to model different noise for periodic functions . Figure 1 shows how NGGPs are able to capture the overall structure of a problem , whereas standard GPs fail . NGGPs are able to reconstruct a multi-modal sine function while adapting to local dissimilarities thanks to the contextualization provided by the ODE-based mapping . We provide empirical evidence that NGGPs outperform competitive state-of-the-art approaches on a diversified set of benchmarks and applications in a few-shot learning scenario ; the code is released with an open-source license2 . The contributions of our work can be summarized as follows : • We introduce Non-Gaussian Gaussian Processes ( NGGPs ) , a new probabilistic method for modeling complex distributions through locally non-Gaussian posteriors . • We show how invertible ODE-based mappings can be coupled with GPs to process the marginals of multivariate random variables resulting in more flexible models . • We extensively test NGGPs on a variety of few-shot learning benchmarks , achieving stateof-the-art performances in most conditions . 2 Related Work . The related work section is divided into three parts . First , we present a general Few-Shot Learning problem . Then , we discuss GPs , focusing on models , which use flow architectures . Finally , in the third paragraph , we describe existing approaches to Few-Shot Learning , which use Gaussian Processes . Few-Shot Learning Few-Shot Learning aims at solving problems in which the number of observations is limited . Some of the early methods in this domain have applied a two-phase approach by pre-training on the base set of training tasks and then fine-tuning the parameters to the test tasks [ 4 , 28 ] . An alternative approach is given by non-parametric metric-learning algorithms , which aim at optimizing a metric , that is then used to calculate the distance between the target observations 2https : //github.com/gmum/non-gaussian-gaussian-processes and the support set items [ 48 , 38 , 42 ] . Another popular approach to few-shot learning is Model Agnostic Meta-Learning ( MAML ) [ 9 ] and its variants [ 12 , 24 , 32 , 54 , 14 , 52 , 6 ] . MAML aims at finding a set of joined task parameters that can be easily fine-tuned to new test tasks via few gradient descent updates . MAML can also be treated as a Bayesian hierarchical model [ 10 , 15 , 18 ] . Bayesian MAML [ 55 ] combines efficient gradient-based meta-learning with non-parametric variational inference in a principled probabilistic framework . A few algorithms have been focusing exclusively on regression tasks . An example is given by ALPaCA [ 17 ] , which uses a dataset of sample functions to learn a domain-specific encoding and prior over weights . Gaussian Processes GPs have been applied to numerous machine learning problems , such as spatio-temporal density estimation [ 7 ] , robotic control [ 53 ] , or dynamics modeling in transcriptional processes in the human cell [ 21 ] . The drawback of GP lies in the computational cost of the training step , which is O ( n3 ) ( where n denotes the number of observations in the training sample ) . In [ 41 ] , the authors extend the flexibility of GPs by processing the targets with a learnable monotonic mapping ( the warping function ) . This idea is further extended in [ 22 ] , which shows that it is possible to place the prior of another GP on the warping function itself . Our method is different from these approaches , since the likelihood transformation is obtained by the use of a learnable CNF mapping . In [ 26 ] , the authors present the Transformed Gaussian Processes ( TGP ) , a new flexible family of function priors that use GPs and flow models . TGPs exploit Bayesian Neural Networks ( BNNs ) as input-dependent parametric transformations . The method can match the performance of Deep GPs at a fraction of the computational cost . The methods discussed above are trained on a single dataset , that is kept unchanged . Therefore , it is not trivial to adapt such methods to the the few-shot setting . Few-Shot Learning with Gaussian Processes When the number of observations is relatively small , GPs represent an interesting alternative to other regression approaches . This makes GPs a good candidate for meta-learning and few-shot learning , as shown by recent publications that have explored this research direction . For instance , Adaptive Deep Kernel Learning ( ADKL ) [ 45 ] proposes a variant of kernel learning for GPs , which aims at finding appropriate kernels for each task during inference by using a meta-learning approach . A similar approach can be used to learn the mean function [ 11 ] . In [ 37 ] , the authors presented a theoretically principled PAC-Bayesian framework for meta-learning . It can be used with different base learners ( e.g. , GPs or BNNs ) . Topics related to kernel tricks and meta-learning have been explored in [ 47 ] . The authors propose to use nonparametric kernel regression for the inner loop update . In [ 43 ] , the authors introduce an information-theoretic framework for meta-learning by using a variational approximation to the information bottleneck . In their GP-based approach , to account for likelihoods other than Gaussians , they propose approximating the non-Gaussian terms in the posterior with Gaussian distributions ( by using amortized functions ) , while we use CNFs to increase the flexibility of the GPs . In [ 29 ] , the authors present Deep Kernel Transfer ( DKT ) : a Bayesian treatment for the meta-learning inner loop through the use of deep kernels , which has achieved state-of-the-art results . In DKT , the deep kernel and the parameters of the GP are shared across all tasks and adjusted to maximize the marginal log-likelihood , which is equivalent to Maximum-Likelihood type II ( ML-II ) learning . DKT is particularly effective in the regression case since it is able to capture prior knowledge about the data through the GP kernel . However , in many settings , prior assumptions could be detrimental if they are not met during the evaluation phase . This is the case in few-shot regression , where there can be a significant difference between the tasks seen at training time and the tasks seen at evaluation time . For instance , if we are given few-shot tasks consisting of samples from periodic functions but periodicity is violated at evaluation time , then methods like DKT may suffer in terms of predictive accuracy under this domain shift . In this work , we tackle this problem by exploiting the flexibility of CNFs . 3 Background . Gaussian Processes . The method proposed in this paper strongly relies on Gaussian Processes ( GPs ) and their applications in regression problems . GPs are a well-established framework for principled uncertainty quantification and automatic selection of hyperparameters through a marginal likelihood objective [ 35 ] . More formally , a GP is a collection of random variables such that the joint distribution of every finite subset of random variables from this collection is a multivariate Gaussian [ 31 ] . We denote Gaussian Process as f ( · ) ⇠ GP ( µ ( · ) , k ( · , · ) ) , where µ ( x ) and k ( x , x0 ) are the mean and covariance functions . When prior information is not available , a common choice for µ is the zero constant function . The covariance function must impose a valid covariance matrix . This is achieved by restricting k to be a kernel function . Examples of such kernels include the Linear kernel , Radial Basis Function ( RBF ) kernel , Spectral Mixture ( Spectral ) kernel [ 50 ] , or Cosine-Similarity kernel [ 33 ] . Kernel functions can also be directly modeled as inner products defined in the feature space imposed by a feature mapping : X ! V : k ( x , x0 ) = h ( x ) , ( x0 ) iV ( 1 ) An advantage of the formulation above is that it can be easily implemented by modeling through a neural network . Throughout this work , we call this technique the NN Linear kernel ( sometimes called Deep Kernel [ 29 ] ) . Since every kernel can be described in terms of Equation ( 1 ) , such an approach may be desired if no prior information about the structure of the kernel function is available . Gaussian Processes provide a method for modeling probability distributions over functions . Consider a regression problem : yi = f ( xi ) + ✏i , for i = 1 , . . . , m , ( 2 ) where ✏i are i.i.d . noise variables with independent N ( 0 , 2 ) distributions . Let X be the matrix composed of all samples xi and let y be the vector composed of all target values yi . Assuming that f ( · ) ⇠ GP ( 0 , k ( · , · ) ) , we obtain : y|X ⇠ N ( 0 , K+ I ) , ( 3 ) where ki , j = k ( xi , xj ) . Analogously , inference over the unknown during the training samples is obtained by conditioning over the normal distribution . Let ( y , X ) be the train data and let ( y⇤ , X⇤ ) be the test data . Then the distribution of y⇤ given y , X , X⇤ is also a Gaussian distribution [ 34 ] : y⇤|y , X , X⇤ ⇠ N ( µ⇤ , K⇤ ) , ( 4 ) where : µ⇤ = K ( X⇤ , X ) K ( X , X ) + 2I 1 y K⇤ = K ( X⇤ , X⇤ ) + 2I K ( X⇤ , X ) K ( X , X ) + 2I 1 K ( X , X⇤ ) Continuous Normalizing Flows . Normalizing Flows ( NF ) [ 36 ] are gaining popularity among generative models thanks to their flexibility and the ease of training via direct negative log-likelihood ( NLL ) optimization . Flexibility is given by the change-of-variable technique that maps a latent variable z with know prior p ( z ) to y from some observed space with unknown distribution . This mapping is performed through a series of ( parametric ) invertible functions : y = fn · · · f1 ( z ) . Assuming known prior p ( z ) for z , the log-likelihood for y is given by : log p ( y ) = log p ( z ) NX n=1 log det @ fn @ zn 1 , ( 5 ) where z = f 11 · · · f 1n ( y ) is a result of the invertible mapping . The biggest challenge in normalizing flows is the choice of the invertible functions fn , . . . , f1 . This is due to the fact that they need to be expressive while guaranteeing an efficient calculation of the Jacobian determinant , which usually has a cubic cost . An alternative approach is given by CNF models [ 16 ] . CNFs use continuous , time-dependent transformations instead of sequence of discrete functions fn , . . . , f1 . Formally , we introduce a function g ( z ( t ) , t ) that models the dynamics of z ( t ) , @ z ( t ) @ t = g ( z ( t ) , t ) , parametrized by . In the CNF setting , we aim at finding a solution y : = z ( t1 ) for the differential equation , assuming the given initial state z : = z ( t0 ) with a known prior . As a consequence , the transformation function f is defined as : y = f ( z ) = z+ Z t1 t0 g ( z ( t ) , t ) dt . ( 6 ) The inverted form of the transformation can be easily computed using the formula : f 1 ( y ) = y Z t1 t0 g ( z ( t ) , t ) dt . ( 7 ) The log-probability of y can be computed by : log p ( y ) = log p ( f 1 ( y ) ) Z t1 t0 Tr ✓ @ g @ z ( t ) ◆ dt where f 1 ( y ) = z . ( 8 )

Summary: The authors propose a novel GP-based model for few-shot regression. The model makes GPs more flexible by adding Continuous Normalizing Flow to the output, specifically an ODE-based mapping. Moreover, this mapping is contextualized by making it input-dependent, which adds more expressivity to the predictive posterior. This flexibility makes the model better suited for modelling complex distributions (e.g. multimodality cases) and, as shown experimentally, is beneficial for few-shot regression, where tasks might differ (domain shift).  The proposed model is experimentally shown to perform better than the main competitor, Deep Kernel Transfer (DKT), on a variety of few-shot regression datasets.  Overall, this is a nicely written paper with an interesting novel method and good experimental results. Some parts of the model, however, are not given enough explanation and testing.