Learning to Learn Dense Gaussian Processes for Few-Shot Learning

1 Introduction . Meta learning [ 31 , 4 , 18 ] , also referred to as learning to learn , aims at learning to acquire shared knowledge from a set of related tasks so as to fast resolve novel tasks sampled from the same underlying task distribution . Meta learning largely stimulates the rise of few-shot learning [ 9 ] , and the recent advantages are mainly driven by designing learning algorithms that learn from massive tasks and acquire prior knowledge , which combined with a small amount of labeled data , induces models that produce reliable predictions on novel tasks . In practice , the acquisition of prior knowledge can be realized in different forms . Model agnostic meta-learning ( MAML ) [ 9 ] learns to adapts to new tasks by few iterations of gradient descend , which inspires many follow-up methods [ 3 , 12 , 22 , 46 ] . The adaptation of the entire network makes it hard to be scaled to large networks , and many recent efforts focus on adapting the last classification layer only [ 14 , 5 ] , while assuming a universal feature extractor that is shared across all tasks . Despite the remarkable progress , challenges remain due to the uncertainty in making predictions with very limited data , which requires models to have high robustness for few-shot learning . Gaussian processes [ 25 ] serve as a powerful model for the inference of functions , which enjoy appealing properties including natural uncertainty quantification and robustness when the amount of data is limited . By combining the strong learning ability of deep neural networks , Gaussian processes with deep kernels further demonstrate improved performance on supervised learning [ 44 ] . To alleviate the computational cost , sparse Gaussian processes were widely studied by learning a sparse set of ⇤Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . inducing variables to effectively approximate the full dataset . In sparse Gaussian processes , the inducing variables usually in the forms of pseudo data or a subset of training data are expected to capture statistics of the whole dataset . Gaussian processes offer desirable robustness to data scarcity by capturing uncertainty , and enjoy the high expressiveness of deep kernels . This makes Gaussian processes a well-suited learning model for few-shot learning as demonstrated in recent works [ 33 , 23 , 38 ] . It is shown in deep kernel transfer ( DKT ) [ 23 ] that a Gaussian process with a meta-learned deep kernel can deliver strong performance through a simple kernel transfer . DKT relies on an important assumption that a universal deep kernel can be obtained through training on a limited amount of tasks . However , this assumption can be too restricted to generalize to novel tasks . In practice , when solving novel tasks , it is hardly guaranteed that the learned deep kernels can well explain the data due to the scarce labeled data , and failing to properly fit the Gaussian process prior can lead to inaccurate predictions . It is therefore crucial to leverage more knowledge to be shared among Gaussian processes for individual tasks , which remains an outstanding problem for Gaussian process few-shot learning . In this paper , we introduce learning to learn Gaussian processes by a dense set of inducing variables for few-shot learning . In particular , we assume a Gaussian process prior over the prediction functions of few-shot learning tasks , which is specified by a set of inducing variables learned from data . These inducing variables are dense in the sense that they have a much larger cardinality compared to the support set of each individual task . Under the meta-learning setting , the dense inducing variables are learned to collect the shared knowledge from experienced tasks to improve the learning of new tasks effectively with limited data . To achieve task-specific prediction functions , we propose to adapt the inducing variables to each task by the gradient descent update . Without adding extra model parameters , the adaptation permits online fitting to the Gaussian processes prior to each individual task , and therefore allows the model to better fit any novel tasks at test time . In addition , in contrast to sparse Gaussian processes , we learn the inducing variables in a low-dimensional deep feature space , which significantly reduces the compute cost of both learning and adapting the dense inducing variables . The resultant dense Gaussian processes inherit the strong learning ability of deep kernels and enjoy the robustness to data scarcity and innate ability of uncertainty quantification , providing an effective few-shot learner . We validate the effectiveness of the proposed dense Gaussian processes on a variety of few-shot learning tasks . We observe that dense Gaussian processes achieve substantial performance improvements over vanilla deep Gaussian processes for few-shot learning and deliver state-of-the-art performance on common benchmarks for few-shot learning . Moreover , our dense Gaussian processes demonstrate strong generalization on cross-domain few-shot learning tasks . In summary , we make three major contributions as follows : • We introduce dense Gaussian processes by learning a set of dense inducing variables for few-shot learning . The inducing variables specify a Gaussian process prior over predictive functions across tasks , which collect shared knowledge from experienced tasks and provide strong inductive bias for learning new tasks efficiently and effectively . • We propose gradient descent based adaptation to establish task-specific prediction functions based on the Gaussian process prior . The gradient descent-based adaptation is efficient without any auxiliary network components for inferring task representations . • We conduct extensive experiments on common benchmark datasets for few-shot classification . The proposed dense Gaussian processes achieve consistent improvements over previous Gaussian processes with deep kernels and deliver comparable or even better results than state-of-the-art methods . 2 Methodology . In this section , we start with a brief review of few-shot learning ; and the key ingredients that build the foundation of the proposed method . We then introduce in detail learning to learn Gaussian processes with dense inducing variables for few-shot learning . 2.1 Backgrounds . Few-shot learning . The common practice of training few-shot learners is to formulate it as a episodic training . In each episode , the training is realized by simulating a few-shot learning task , where the learner is provided with a set of few label data called the support set S , and required to make predictions on the corresponding query set Q with unlabeled test samples . Taking few-shot image classification as an example , it is defined as a N -way K-shot learning problem , where K is usually a small number , e.g. , K = 5 . Typically , in each episode indexed by t , one few-shot classification task is generated by first sampling N categories from the training data , each of which contains K samples to form the support set St = { x1 , . . . , xN⇥K } . An adaptation to the few-shot classification model is then performed on St , by , e.g. , computing prototypes [ 32 ] , or updating the network parameters [ 9 ] . After adaptation , the query set is formed by evenly sampling M samples from each category , i.e. , Qt = { x01 , . . . , x0N⇥M } , to evaluate the updated model , and the error is propagated back to update the parameters . Gaussian processes with deep kernels . Gaussian processes ( GP ) [ 25 ] place a prior distribution p ( f ) over the target functions f . Consider we have a set of n datapoints , D = ( xi , yi ) N i=1 with additive noise yi = f ( xi ) + ✏i , where ✏i ⇠ N ( 0 , 2 ) . The GP prior over f can be written as , p ( f ) = N ( 0 , Knn ) , ( 1 ) where Knn = K ( xn , xn ) is the kernel matrix of n training datapoints , and the likelihood is p ( y|f ) = nY i=1 p ( yi|fi ) = nY i=1 N ( fi , 2 ) . ( 2 ) Given a test data point x⇤ , its predictive posterior is characterized by a Gaussian distribution , f⇤ ⇠ N ( f⇤|µ⇤ , ⇤ ) , where µ⇤ = k⇤ T ( Knn + 2I ) 1y ⇤ = k ( x ⇤ , x⇤ ) k⇤ T ( Knn + 2I ) 1k⇤ ( 3 ) Given a deep neural network ✓ : X ! Z , which maps from input space to deep feature space , we obtain the deep kernel function [ 44 ] , k ( xi , xj |✓ ) = k ( ✓ ( xi ) , ✓ ( xj ) ) = k ( zi , zj ) . ( 4 ) The dimensionality of deep feature space Z is usually smaller than the original input space X , especially for image data , leading to more efficient kernel learning and inference . In this way , the GP prior in our method is parameterized by the network as p ( f ) = N ( f |0 , Knn ) , where Knn = K ( zn , zn ) = K ( xn , xn| ✓ ) is the covariance matrix calculated based on deep features , and the function values f are defined over deep features f = f ( Z ) = f ( ( X ) ) . Sparse Gaussian Processes The posterior inference in a Gaussian Process induces a cubic complexity of O ( n3 ) , which scales poorly with a huge amount of datapoints n. Sparse Gaussian processes [ 36 , 34 , 17 ] are widely studied to improve the scalability of standard GPs , and a popular direction introduces m inducing variables { Z̃i } mi=1 to approximate the full GP marginal likelihood , where typical m ⌧ n in favor of improved efficiency . In sparse Gaussian processes , the optimization for the inducing variables { Z̃i } mi=1 can be achieved through variational inference [ 36 ] . And the GP prior [ 36 ] is augmented with the function values of inducing variables fm , f ( Z̃ ) as p ( f , fm ) = p ( f |fm ) p ( fm ) , and the joint distribution can therefore be expressed as : p ( y , f , fm ) = p ( y|f ) p ( f |fm ) p ( fm ) , ( 5 ) where it is assumed that y is conditionally independent of f given fm . By minimizing the distance between the true posterior p ( f , fm|y ) and the variational approximate posterior q ( f , fm ) = p ( f |fm ) q ( fm ) , where q ( fm ) is a free variational Gaussian distribution N ( fm|µ , ⇤ ) , the maximization of log likelihood amounts to maximizing a variational lower bound : log p ( y ) Z f , fm q ( f , fm ) log p ( y , f , fm ) q ( f , fm ) dfdfm , L ( Z̃ , , q ) ( 6 ) 2.2 Learning to Learn Dense Gaussian Processes . In this section , we derive our dense Gaussian processes by inducing variables under the episodic training setting of meta-learning for few-shot learning . Adapting Gaussian Process Priors For a C-way K-shot classification task , we adapt f c ⇠ GPc ( 0 , Kc ) , and the samples from the posterior predictive are combined based on a multi-class likelihood such as the softmax function . We would like to learn a Gaussian Process prior over the prediction function , which can be adapted to a few-shot learning task by its support set with a few gradient descent updates . The Gaussian process prior is established by a set of dense inducing variables , which is much larger than the support set in each task . The inducing variables are learned from data which serve to collect shared knowledge from experienced tasks to facilitate efficient learning of new tasks . Formally , in an episode of the C-way K-shot task , the GP learner is provided with a support set St = { xck , y c k } C , K c=1 , k=1 . We adapt a task-specific GP prior in the form of specific kernels K c ( c , Z̃c ) for each class within the task , which can be achieved through an adaptation based on Eq . 6 , with both parameters initialized by the meta representation 0 , Z̃0 , c ( 0 ) , Z̃c ( Z̃0 ) = argmax , Z̃ L ( , Z̃ , q ) . ( 7 ) As shown in [ 36 ] , L ( , Z̃ , q ) can be first maximized by analytically solving the optimal qo ( fm ) , qo ( fm ) = N ( fm|µ o , ⇤o ) , µo = 2Kmm⌃Kmny , ⇤ o = Kmm⌃Kmm , ( 8 ) where now n = |S| = C ⇥ K , thus Kmn denotes the covariance matrix between m inducing variables and n support set data , and ⌃ = ( Kmm + 2KmnKTmn ) 1 . The Derivation is provided in the Appendix A . Thus , the lower bound can be written as , L ( , Z̃ ) = log N ( y|0 , 2I + K̂nn ) 1 2 2 Tr ( Knn K̂nn ) , ( 9 ) where K̂nn = KnmK 1mmKTnm . Note that ( 9 ) can be adopted for multi-class classification on the support set . To be specific , consider a binary classification task where y 2 { 0 , 1 } , it can be solved by the proposed dense GP regression by assuming a Gaussian likelihood . With the one-versus-rest scheme , the regression model can be extended to solve the C-way classification task . Specifically , a regressor is learned for each class to predict whether a sample x belongs to this class ( with y = 1 ) or not ( with y = 0 ) . Each regressor is formulated as a single dense GP with its own adapted inducing variables . By maximizing ( 9 ) , we can obtain an adapted GP prior for each class that parameterized by Algorithm 1 Learning to learn dense inducing variables . 1 : Given : A description of the tasks as N -way K-shot with M query samples at each episode . 2 : Given : Inner-loop and out-loop learning rates ↵ and , number of steps I for inner-loop optimization . 3 : Initialize and Z̃ 4 : repeat 5 : Sample task with support set S = { xck } C , K c=1 , k=1 and query set Q = { x 0 j } M j=1 in each task . 6 : Extract feature vectors for both S and Q by applying zck = ( x c k ) and z 0 j = ( x 0 j ) , respectively . 7 : for Class n = 1 : N do 8 : for Inner iterations i = 1 : I do 9 : Perform inner-loop adaptation and tuned the per-class inducing variables Z̃c , i for a class-adapted GP prior by applying ( 14 ) . 10 : end for 11 : end for 12 : With Z̃c , I obtained , calculate the predictive posterior using ( 11 ) , and perform the outer-loop adaptation by applying ( 15 ) . 13 : until Converge 14 : Return and Z̃0 . Z̃c , c. Ideally , the well-trained inducing variables can sparsely approximate the full dataset , and reduce the prohibitive computation complexity to O ( nm2 ) . Given a query sample x⇤ , the posterior predictive distribution for class n , q ( f c⇤ |xc⇤ ) = N ( µc⇤ , ( 2 ) c⇤ ) can be calculated as µc⇤ = k⇤m⌃K c mny ( 2 ) c⇤ = k c ( x⇤ , x⇤ ) K c ⇤m ( K c mm ) 1 ( kc⇤m ) > + kc⇤m⌃ ( k c ⇤m ) > , ( 10 ) and the multi-class posterior predictive can be expressed as , p ( yc⇤ = 1|x⇤ ) = exp ( f c⇤ · ⌧ ) P c0 exp ( f c0 ⇤ · ⌧ ) , with f c⇤ ⇠ q ( f c ⇤ |x⇤ ) , ( 11 ) where ⌧ is a learned temperature for MLE . Note the trainable temperature ⌧ here allows automatic scaling to the unnormalized multi-class posterior predictive , and a similar implementation can also be found in [ 5 ] . Although sharing the same formulation , the ⌧ here serves a completely different role as the temperature used in uncertainty calibration . The loss of empirical risk on the query set , acting as the outer-loop criteria , can be then be calculated as LOuter ( Z̃ 0 , 0 ) = X ( x⇤ , y⇤ ) 2Q log p ( y⇤|x⇤ ) . ( 12 ) In this way , the GP prior adapted to the task t by St is further supervised by the query set Qt . This in turn incorporates more knowledge from each task to consolidate task-shared meta knowledge into 0 , Z̃0 , which enables new tasks to be learned efficiently . Dense Inducing Variables The aforementioned few-shot episodic training deep kernel allows an inner-loop adaptation to the model , and achieves better adaptation to the GP prior on novel tasks . However , performing adaptation to the entire deep kernel involves updating the deep neural network that parameterized the kernel , which introduces dramatically high computation cost and latency . Meanwhile , explicitly leveraging the meta knowledge learned from massive tasks , e.g. , with memory that allows adaptive information retrieval [ 48 ] , or data hallucination [ 45 ] that expands the support set of current tasks have demonstrated impressive results on few-shot learning . Inspired by the idea of explicit knowledge representation in few-shot learning , and data approximation through inducing variables in sparse Gaussian processes , we introduce dense inducing variables , which are learned as a rich set of inducing variables that carry the meta-knowledge of the entire task distribution in the latent deep feature space , and permit efficient model adaptation to novel tasks . Specifically , instead of acting as a sparse approximation to the data of any particular task , the dense inducing variables here specify a shared Gaussian process prior over prediction functions of all tasks , are introduce to support the entire tasks distribution , from which a potentially infinite number of tasks can be sampled from . To achieve effective modeling of the cross task GP prior , we allow the dense inducing variables to be have a larger cardinality than the support set in a single task , i.e. , m |S| . Notably , unlike standard sparse Gaussian processes that learn the dense inducing variables as pseudo inputs in raw data space of high dimension , we propose to directly learn inducing variables in the same latent space of the feature from the deep neural network . By doing so , learning inducing variables effectively bypasses the heavy computation cost , and also enables efficient online adaptation to the inducing variables . With dense inducing variables specifying a hyperprior over predictive functions across tasks , and included as hyperparameters in the deep kernel , we now introduce the online update , which induces an adapted Gaussian process prior over the prediction function for a specific task , through a gradient descent based adaptation to the dense inducing variables . Specifically , given the support set of the task with support set St , we perform adaptation of the Gaussian process prior by using the few labeled data , and rewrite ( 7 ) as LInner ( Z̃ ) = log N ( y|0 , 2I + K̂nn ) 1 2 2 Tr ( Knn K̂nn ) , ( 13 ) and adapt the inducing variables by Z̃c , i = Z̃c , i 1 ↵rZ̃c , i 1LInner ( Z̃ c , i 1 ) , ( 14 ) where ↵ specifies an inner-loop learning rate , and each Z̃0n initialized with Z̃0 . Thanks to the firstorder approximation to the computationally heavy second-order derivative , the training with ( 14 ) can be remarkably accelerated . Given the adapted inducing variables on the i-th task , the predictive posterior of function values on the query set can then be efficiently obtained by performing meta updates to both the feature extractor and the dense inducing variables calculated based on ( 12 ) , = r LOuter ( , Z̃ 0 ) Z̃0 = Z̃0 rZ̃0LOuter ( , Z̃ 0 ) , ( 15 ) with specifying the learning rate for the outer loop . Note that we assume a universal feature extractor and update it only in the outer loop , which is commonly used in meta-learning for efficiency and generalization to more powerful backbones . Meanwhile , the specification of the task-specific Gaussian process prior through adapting dense inducing variables endows our model with high flexibility and strong generalization to novel tasks without scarifying efficiency , as we will demonstrate in our experiments . In contrast to the previous work that simply learns transferable Gaussian processes on few-shot learning [ 23 ] , our method adopts episodic learning to adapt the Gaussian process prior to each task by updating the dense inducing variables , which parameterize the deep kernel jointly with the deep feature extractor . The overall training procedure of the dense Gaussian process model is summarized in Algorithm 1 .

Summary: This paper develops a way to learn Gaussian processes with dense inducing variables by meta-learning for few-shot learning. The algorithm provides a strong inductive bias for learning new tasks by specifying a shared Gaussian process prior over prediction functions of a set of tasks. It adapts the inducing variables to each task by efficient gradient descent to achieve task specific predictions. Experiments show that this approach can achieve competitive performance in all the tasks considered, and state-of-the-art results in certain tasks.