Learning to Learn Dense Gaussian Processes for Few-Shot Learning
Year: 2022

Abstract:
Gaussian processes with deep neural networks demonstrate to be a strong learner for few-shot learning since they combine the strength of deep learning and kernels while being able to well capture uncertainty. However, it remains an open problem to leverage the shared knowledge provided by related tasks. In this paper, we propose to learn Gaussian processes with dense inducing variables by meta-learning for few-shot learning. In contrast to sparse Gaussian processes, we define a set of dense inducing variables to be of a much larger size than the support set in each task, which collects prior knowledge from experienced tasks. The dense inducing variables specify a shared Gaussian process prior over prediction functions of all tasks, which are learned in a variational inference framework and offer a strong inductive bias for learning new tasks. To achieve task-specific prediction functions, we propose to adapt the inducing variables to each task by efficient gradient descent. We conduct extensive experiments on common benchmark datasets for a variety of few-shot learning tasks. Our dense Gaussian processes present significant improvements over vanilla Gaussian processes and comparable or even better performance with state-of-the-art methods.

heading: 1 Introduction
text: Meta learning [31, 4, 18], also referred to as learning to learn, aims at learning to acquire shared knowledge from a set of related tasks so as to fast resolve novel tasks sampled from the same underlying task distribution. Meta learning largely stimulates the rise of few-shot learning [9], and the recent advantages are mainly driven by designing learning algorithms that learn from massive tasks and acquire prior knowledge, which combined with a small amount of labeled data, induces models that produce reliable predictions on novel tasks. In practice, the acquisition of prior knowledge can be realized in different forms. Model agnostic meta-learning (MAML) [9] learns to adapts to new tasks by few iterations of gradient descend, which inspires many follow-up methods [3, 12, 22, 46]. The adaptation of the entire network makes it hard to be scaled to large networks, and many recent efforts focus on adapting the last classification layer only [14, 5], while assuming a universal feature extractor that is shared across all tasks. Despite the remarkable progress, challenges remain due to the uncertainty in making predictions with very limited data, which requires models to have high robustness for few-shot learning.
Gaussian processes [25] serve as a powerful model for the inference of functions, which enjoy appealing properties including natural uncertainty quantification and robustness when the amount of data is limited. By combining the strong learning ability of deep neural networks, Gaussian processes with deep kernels further demonstrate improved performance on supervised learning [44]. To alleviate the computational cost, sparse Gaussian processes were widely studied by learning a sparse set of
⇤Equal contribution.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
inducing variables to effectively approximate the full dataset. In sparse Gaussian processes, the inducing variables usually in the forms of pseudo data or a subset of training data are expected to capture statistics of the whole dataset.
Gaussian processes offer desirable robustness to data scarcity by capturing uncertainty, and enjoy the high expressiveness of deep kernels. This makes Gaussian processes a well-suited learning model for few-shot learning as demonstrated in recent works [33, 23, 38]. It is shown in deep kernel transfer (DKT) [23] that a Gaussian process with a meta-learned deep kernel can deliver strong performance through a simple kernel transfer. DKT relies on an important assumption that a universal deep kernel can be obtained through training on a limited amount of tasks. However, this assumption can be too restricted to generalize to novel tasks. In practice, when solving novel tasks, it is hardly guaranteed that the learned deep kernels can well explain the data due to the scarce labeled data, and failing to properly fit the Gaussian process prior can lead to inaccurate predictions. It is therefore crucial to leverage more knowledge to be shared among Gaussian processes for individual tasks, which remains an outstanding problem for Gaussian process few-shot learning.
In this paper, we introduce learning to learn Gaussian processes by a dense set of inducing variables for few-shot learning. In particular, we assume a Gaussian process prior over the prediction functions of few-shot learning tasks, which is specified by a set of inducing variables learned from data. These inducing variables are dense in the sense that they have a much larger cardinality compared to the support set of each individual task. Under the meta-learning setting, the dense inducing variables are learned to collect the shared knowledge from experienced tasks to improve the learning of new tasks effectively with limited data.
To achieve task-specific prediction functions, we propose to adapt the inducing variables to each task by the gradient descent update. Without adding extra model parameters, the adaptation permits online fitting to the Gaussian processes prior to each individual task, and therefore allows the model to better fit any novel tasks at test time. In addition, in contrast to sparse Gaussian processes, we learn the inducing variables in a low-dimensional deep feature space, which significantly reduces the compute cost of both learning and adapting the dense inducing variables.
The resultant dense Gaussian processes inherit the strong learning ability of deep kernels and enjoy the robustness to data scarcity and innate ability of uncertainty quantification, providing an effective few-shot learner. We validate the effectiveness of the proposed dense Gaussian processes on a variety of few-shot learning tasks. We observe that dense Gaussian processes achieve substantial performance improvements over vanilla deep Gaussian processes for few-shot learning and deliver state-of-the-art performance on common benchmarks for few-shot learning. Moreover, our dense Gaussian processes demonstrate strong generalization on cross-domain few-shot learning tasks.
In summary, we make three major contributions as follows:
• We introduce dense Gaussian processes by learning a set of dense inducing variables for few-shot learning. The inducing variables specify a Gaussian process prior over predictive functions across tasks, which collect shared knowledge from experienced tasks and provide strong inductive bias for learning new tasks efficiently and effectively.
• We propose gradient descent based adaptation to establish task-specific prediction functions based on the Gaussian process prior. The gradient descent-based adaptation is efficient without any auxiliary network components for inferring task representations.
• We conduct extensive experiments on common benchmark datasets for few-shot classification. The proposed dense Gaussian processes achieve consistent improvements over previous Gaussian processes with deep kernels and deliver comparable or even better results than state-of-the-art methods.

heading: 2 Methodology
text: In this section, we start with a brief review of few-shot learning; and the key ingredients that build the foundation of the proposed method. We then introduce in detail learning to learn Gaussian processes with dense inducing variables for few-shot learning.

heading: 2.1 Backgrounds
text: Few-shot learning. The common practice of training few-shot learners is to formulate it as a episodic training. In each episode, the training is realized by simulating a few-shot learning task, where the learner is provided with a set of few label data called the support set S, and required to make predictions on the corresponding query set Q with unlabeled test samples. Taking few-shot image classification as an example, it is defined as a N -way K-shot learning problem, where K is usually a small number, e.g., K = 5. Typically, in each episode indexed by t, one few-shot classification task is generated by first sampling N categories from the training data, each of which contains K samples to form the support set St = {x1, . . . ,xN⇥K}. An adaptation to the few-shot classification model is then performed on St, by, e.g., computing prototypes [32], or updating the network parameters [9]. After adaptation, the query set is formed by evenly sampling M samples from each category, i.e., Qt = {x01, . . . ,x0N⇥M}, to evaluate the updated model, and the error is propagated back to update the parameters.
Gaussian processes with deep kernels. Gaussian processes (GP) [25] place a prior distribution p(f) over the target functions f . Consider we have a set of n datapoints, D = (xi,yi) N i=1 with additive noise yi = f(xi) + ✏i, where ✏i ⇠ N (0, 2). The GP prior over f can be written as, p(f) = N (0,Knn), (1)
where Knn = K(xn,xn) is the kernel matrix of n training datapoints, and the likelihood is
p(y|f) = nY
i=1
p(yi|fi) = nY
i=1
N (fi, 2). (2)
Given a test data point x⇤, its predictive posterior is characterized by a Gaussian distribution, f⇤ ⇠ N (f⇤|µ⇤, ⇤), where
µ⇤ = k⇤ T (Knn + 2I) 1y
⇤ = k(x ⇤,x⇤) k⇤ T (Knn + 2I) 1k⇤ (3)
Given a deep neural network ✓ : X ! Z, which maps from input space to deep feature space, we obtain the deep kernel function [44],
k(xi,xj |✓) = k( ✓(xi), ✓(xj)) = k(zi, zj). (4) The dimensionality of deep feature space Z is usually smaller than the original input space X, especially for image data, leading to more efficient kernel learning and inference. In this way, the GP prior in our method is parameterized by the network as p(f) = N (f |0,Knn), where Knn = K(zn, zn) = K(xn,xn| ✓) is the covariance matrix calculated based on deep features, and the function values f are defined over deep features f = f(Z) = f( (X)).
Sparse Gaussian Processes The posterior inference in a Gaussian Process induces a cubic complexity of O(n3), which scales poorly with a huge amount of datapoints n. Sparse Gaussian processes [36, 34, 17] are widely studied to improve the scalability of standard GPs, and a popular direction introduces m inducing variables {Z̃i}mi=1 to approximate the full GP marginal likelihood, where typical m ⌧ n in favor of improved efficiency.
In sparse Gaussian processes, the optimization for the inducing variables {Z̃i}mi=1 can be achieved through variational inference [36]. And the GP prior [36] is augmented with the function values of inducing variables fm , f(Z̃) as p(f , fm) = p(f |fm)p(fm), and the joint distribution can therefore be expressed as: p(y, f , fm) = p(y|f)p(f |fm)p(fm), (5) where it is assumed that y is conditionally independent of f given fm. By minimizing the distance between the true posterior p(f , fm|y) and the variational approximate posterior q(f , fm) = p(f |fm)q(fm), where q(fm) is a free variational Gaussian distribution N (fm|µ,⇤), the maximization of log likelihood amounts to maximizing a variational lower bound:
log p(y) Z
f ,fm
q(f , fm)log p(y, f , fm)
q(f , fm) dfdfm , L(Z̃, , q) (6)

heading: 2.2 Learning to Learn Dense Gaussian Processes
text: In this section, we derive our dense Gaussian processes by inducing variables under the episodic training setting of meta-learning for few-shot learning.
Adapting Gaussian Process Priors For a C-way K-shot classification task, we adapt f c ⇠ GPc(0,Kc), and the samples from the posterior predictive are combined based on a multi-class likelihood such as the softmax function. We would like to learn a Gaussian Process prior over the prediction function, which can be adapted to a few-shot learning task by its support set with a few gradient descent updates. The Gaussian process prior is established by a set of dense inducing variables, which is much larger than the support set in each task. The inducing variables are learned from data which serve to collect shared knowledge from experienced tasks to facilitate efficient learning of new tasks.
Formally, in an episode of the C-way K-shot task, the GP learner is provided with a support set St = {xck,y c k} C,K c=1,k=1. We adapt a task-specific GP prior in the form of specific kernels K
c( c, Z̃c) for each class within the task, which can be achieved through an adaptation based on Eq. 6, with both parameters initialized by the meta representation 0, Z̃0,
c( 0), Z̃c(Z̃0) = argmax ,Z̃ L( , Z̃, q). (7)
As shown in [36], L( , Z̃, q) can be first maximized by analytically solving the optimal qo(fm),
qo(fm) = N(fm|µ o,⇤o), µo = 2Kmm⌃Kmny, ⇤ o = Kmm⌃Kmm, (8)
where now n = |S| = C ⇥ K, thus Kmn denotes the covariance matrix between m inducing variables and n support set data, and ⌃ = (Kmm + 2KmnKTmn) 1. The Derivation is provided in the Appendix A. Thus, the lower bound can be written as,
L( , Z̃) = log N (y|0, 2I + K̂nn) 1
2 2 Tr(Knn K̂nn), (9)
where K̂nn = KnmK 1mmKTnm. Note that (9) can be adopted for multi-class classification on the support set. To be specific, consider a binary classification task where y 2 {0, 1}, it can be solved by the proposed dense GP regression by assuming a Gaussian likelihood. With the one-versus-rest scheme, the regression model can be extended to solve the C-way classification task. Specifically, a regressor is learned for each class to predict whether a sample x belongs to this class (with y = 1) or not (with y = 0). Each regressor is formulated as a single dense GP with its own adapted inducing variables. By maximizing (9), we can obtain an adapted GP prior for each class that parameterized by
Algorithm 1 Learning to learn dense inducing variables. 1: Given: A description of the tasks as N -way K-shot with M query samples at each episode. 2: Given: Inner-loop and out-loop learning rates ↵ and , number of steps I for inner-loop
optimization. 3: Initialize and Z̃ 4: repeat 5: Sample task with support set S = {xck} C,K c=1,k=1 and query set Q = {x 0 j} M j=1 in each task. 6: Extract feature vectors for both S and Q by applying zck = (x c k) and z 0 j = (x 0 j), respectively. 7: for Class n = 1 : N do 8: for Inner iterations i = 1 : I do 9: Perform inner-loop adaptation and tuned the per-class inducing variables Z̃c,i for a
class-adapted GP prior by applying (14). 10: end for 11: end for 12: With Z̃c,I obtained, calculate the predictive posterior using (11), and perform the outer-loop adaptation by applying (15). 13: until Converge 14: Return and Z̃0.
Z̃c, c. Ideally, the well-trained inducing variables can sparsely approximate the full dataset, and reduce the prohibitive computation complexity to O(nm2).
Given a query sample x⇤, the posterior predictive distribution for class n, q(f c⇤ |xc⇤) = N (µc⇤, ( 2)c⇤) can be calculated as
µc⇤ = k⇤m⌃K c mny
( 2)c⇤ = k c(x⇤,x⇤) K c ⇤m(K c mm) 1(kc⇤m) > + kc⇤m⌃(k c ⇤m)
>, (10)
and the multi-class posterior predictive can be expressed as,
p(yc⇤ = 1|x⇤) = exp(f c⇤ · ⌧)P c0 exp(f c0 ⇤ · ⌧) , with f c⇤ ⇠ q(f c ⇤ |x⇤), (11)
where ⌧ is a learned temperature for MLE. Note the trainable temperature ⌧ here allows automatic scaling to the unnormalized multi-class posterior predictive, and a similar implementation can also be found in [5]. Although sharing the same formulation, the ⌧ here serves a completely different role as the temperature used in uncertainty calibration. The loss of empirical risk on the query set, acting as the outer-loop criteria, can be then be calculated as
LOuter(Z̃ 0, 0) =
X
(x⇤,y⇤)2Q
log p(y⇤|x⇤). (12)
In this way, the GP prior adapted to the task t by St is further supervised by the query set Qt. This in turn incorporates more knowledge from each task to consolidate task-shared meta knowledge into 0, Z̃0, which enables new tasks to be learned efficiently.
Dense Inducing Variables The aforementioned few-shot episodic training deep kernel allows an inner-loop adaptation to the model, and achieves better adaptation to the GP prior on novel tasks. However, performing adaptation to the entire deep kernel involves updating the deep neural network that parameterized the kernel, which introduces dramatically high computation cost and latency. Meanwhile, explicitly leveraging the meta knowledge learned from massive tasks, e.g., with memory that allows adaptive information retrieval [48], or data hallucination [45] that expands the support set of current tasks have demonstrated impressive results on few-shot learning. Inspired by the idea of explicit knowledge representation in few-shot learning, and data approximation through inducing variables in sparse Gaussian processes, we introduce dense inducing variables, which are learned as a rich set of inducing variables that carry the meta-knowledge of the entire task distribution in the latent deep feature space, and permit efficient model adaptation to novel tasks.
Specifically, instead of acting as a sparse approximation to the data of any particular task, the dense inducing variables here specify a shared Gaussian process prior over prediction functions of all tasks, are introduce to support the entire tasks distribution, from which a potentially infinite number of tasks can be sampled from.
To achieve effective modeling of the cross task GP prior, we allow the dense inducing variables to be have a larger cardinality than the support set in a single task, i.e., m |S|. Notably, unlike standard sparse Gaussian processes that learn the dense inducing variables as pseudo inputs in raw data space of high dimension, we propose to directly learn inducing variables in the same latent space of the feature from the deep neural network. By doing so, learning inducing variables effectively bypasses the heavy computation cost, and also enables efficient online adaptation to the inducing variables.
With dense inducing variables specifying a hyperprior over predictive functions across tasks, and included as hyperparameters in the deep kernel , we now introduce the online update, which induces an adapted Gaussian process prior over the prediction function for a specific task, through a gradient descent based adaptation to the dense inducing variables.
Specifically, given the support set of the task with support set St, we perform adaptation of the Gaussian process prior by using the few labeled data, and rewrite (7) as
LInner(Z̃) = log N (y|0, 2I + K̂nn) 1
2 2 Tr(Knn K̂nn), (13)
and adapt the inducing variables by
Z̃c,i = Z̃c,i 1 ↵rZ̃c,i 1LInner(Z̃ c,i 1), (14)
where ↵ specifies an inner-loop learning rate, and each Z̃0n initialized with Z̃0. Thanks to the firstorder approximation to the computationally heavy second-order derivative, the training with (14) can be remarkably accelerated.
Given the adapted inducing variables on the i-th task, the predictive posterior of function values on the query set can then be efficiently obtained by performing meta updates to both the feature extractor and the dense inducing variables calculated based on (12),
= r LOuter( , Z̃ 0)
Z̃0 = Z̃0 rZ̃0LOuter( , Z̃ 0),
(15)
with specifying the learning rate for the outer loop.
Note that we assume a universal feature extractor and update it only in the outer loop, which is commonly used in meta-learning for efficiency and generalization to more powerful backbones. Meanwhile, the specification of the task-specific Gaussian process prior through adapting dense inducing variables endows our model with high flexibility and strong generalization to novel tasks without scarifying efficiency, as we will demonstrate in our experiments.
In contrast to the previous work that simply learns transferable Gaussian processes on few-shot learning [23], our method adopts episodic learning to adapt the Gaussian process prior to each task by updating the dense inducing variables, which parameterize the deep kernel jointly with the deep feature extractor . The overall training procedure of the dense Gaussian process model is summarized in Algorithm 1.

heading: 3 Experiments
text: In this section, we support the proposed methods with strong experimental results. We present extensive experiments on both in-domain and cross-domain few-shot image classification. Visualizations of the uncertainty quantification with examples of few-shot regression are provided in Appendix Figure A. We finalize the discussions with the ablation study on the effectiveness of important components of the proposed method.

heading: 3.1 Datasets and Settings
text: We perform experiments on the widely used few-shot learning benchmarks including miniImageNet, tieredImageNet, CIFAR-FS , and Caltech-UCSD [42] (CUB). In miniImageNet [41], there are 100
image classes from a subset of ImageNet [8], with 600 images for each class. We follow the standard practice [9] to split the training, validation, and testing sets with 64, 16, and 20 classes, respectively. tieredImageNet [28] is a large subset of ImageNet that contains 608 classes with 1,300 samples in each class. Specifically, in tieredImageNet, there are 351 classes from 20 categories for training, 97 classes from 6 categories for validation, and 160 classes from 8 different categories for testing. Samples for both miniImageNet and tieredImageNet are random cropped and resized to 84⇥ 84 for training, and standard center cropping is performed to the testing images. The 200 classes in the CUB dataset is divided into 100, 50, and 50 classes, for training, validation, and testing, respectively. CIFAR-FS adopts all the 100 classes in the CIFAR-100 dataset with training, validation, and testing splits of 64, 16, and 20 classes, respectively. Each class contains 600 image samples. The image resolution for CIFAR-FS is 32⇥ 32.

heading: 3.2 Few-Shot Image Classification
text: The results of few-shot classification on miniImageNet, tieredImageNet, CIFAR-FS , and CaltechUCSD (CUB) are reported in Tables 1 and 2. For comprehensive comparisons, we adopt two backbone networks with different scales. On all three datasets, we achieve comparable or often better performance than state-of-the-art methods. Our method yields the accuracy of 72.64 on the miniImageNet dataset under the 5-way 5-shot setting, which surpasses the previous best method by up to 5%. In particular, our dense Gaussian processes achieve much better performance than the recent Gaussian processes based methods [23, 33]. This demonstrates the effectiveness of our method by learning dense inducing variables. For more experimental detail please refer to Appendix Section B.

heading: 3.3 Cross-Domain Few-Shot Image Classification
text: The efficient adaptation to the GP prior through updating the dense inducing variables enables the proposed method with improved generalization to novel tasks with unseen categories. To fully demonstrate the remarkable generalization, we perform cross-domain few-shot image classification. Following common practice, we adopt two cross-domain settings, Omniglot ! EMNIST and miniImageNet! CUB. The results are reported in Table 3. The proposed dense Gaussian processes demonstrate high generalizability across domains for few-shot classification and achieve much better performance than the recent Gaussian process based methods [23, 33] in the setting of generalization from Omniglot to CUB.

heading: 3.4 Ablation Study
text: We perform extensive ablation studies to provide insights into the effectiveness of the proposed dense Gaussian processes for few-shot learning. These experiments are performed on the CUB dataset.
Effectiveness of dense inducing variables The proposed dense inducing variables enable shared knowledge to be collected from experienced tasks and applied to new unseen tasks. We show how the number of inducing variables affects the performance of our model. As shown in Figure 2, we report performance on both 5-way 1-shot and 5-way 5-shot experiments. We test a sequence of values of m starting from 8, which is just slightly larger than the number of support set samples in a 5-way 1-shot experiment. And the value of m is progressively increased to 1024. It is clearly demonstrated that, on both experiments with different shots, the performance grows monotonically w.r.t. the number of inducing points that are utilized. And the performance tends to saturate when m > 256. To maintain a balance between efficiency and performance, we choose m = 256 throughout all the experiments. The results in Figure 2 also validate the effectiveness of specifying the GP prior through ‘dense’ inducing variables, with much more elements compared to the support set in each task.
Effectiveness of inner-loop adaptation In the proposed method, the task-specific GP prior adaptation is achieved by few-step gradient descent to the dense inducing variables. For the selection of the best configuration of the inner-loop optimization, we present comparisons performed with the number of steps I in the inner-loop gradient descent, and the inner-loop learning rate ↵. As shown in
Figure 14, we plot the classification accuracy for 5-way 5-shot setting, and show the performance obtained by conducting 1 to 20 steps of inner-loop update with three different values of ↵. It is clearly shown that the performance reaches the peak within 5 steps of inner-loop update, and it does not improve with even more steps of adaption, especially when combined with larger learning rates. This indicates the efficiency of the proposed adaption of the Gaussian process prior to each task. Moreover, a large ↵ can lead to performance drop very fast when performing more than 1 step of inner-loop update. To keep the balance between accuracy and efficiency, we choose ↵ = 0.01 and I = 5 throughout all the experiments.

heading: 4 Related Work
text: Metric learning [32] has been explored for few-shot learning, which assumes that different related tasks would share the same metric to measure similarity. Vinyals et al. [41] proposed the matching network, which learns to map a small labeled support set and an unlabelled example to its label, obviating the need for fine-tuning to adapt to new class types. This work was originally developed for one-shot learning, and extended to a few-shot setting by Snell et al. [32]. They proposed the prototypical network to learn a metric space in which classification can be performed by computing distances to prototype representations of each class and the prototype of each class is the cluster of samples in that class. To enhance the expressivity of the prototypes, Allen et al. [1] proposed infinite mixture prototypes to adaptively represent both simple and complex data distributions for few-shot learning. Satorras et al. [30] solved few-shot learning with the prism of inference on a partially observed graphical model.
Meta-learning based on optimization has been extensively studied [27, 10, 2]. The core idea of optimization-based methods is to learn an optimization procedure that is shared across tasks, which can be applied to new tasks for quick adaptation, once learned in the meta-train stage. Ravi et al. [27] proposed an LSTM based meta-learner to learn the exact optimization algorithm used to train a neural network classifier in the few-shot regime; in their methods, apart from the learned optimization algorithm, a good initialization of model parameters is also obtained after training. Finn et al. [11] proposed a model agnostic meta-learning (MAML) algorithm, which was built upon the assumption that related tasks sharing initial parameters of neural networks could be adapted to specific tasks with a few steps of gradient descent updates. It was believed that the initial weights combined with a few more steps of gradient descent can approximate any learning algorithm, and thus gradient-based meta-learning had a number of practical benefits. Finn et al. [13] extended MAML in a probabilistic framework. Rajeswaran et al. [24] developed the implicit MAML [10] algorithm, which depended only on the solution to the inner level optimization. Zintgraf et al. [50] updated context parameters with one or several gradient steps on a task-specific loss that serves as an additional input to the model and were adapted on individual tasks.
It has also been investigated to explicitly design a meta-learner to learn a base-learner. Bertinetto et al. [6] explored the feasibility of incorporating fast solvers with closed-form solutions as the base learning component of a meta-learning system. Gordon et al. [15] developed meta-learning approximate probabilistic inference for prediction. The support set was used to produce the parameter distribution of the classifier, which was applied to the query set for prediction. Mishra et al. [20] proposed a generic meta-learner architecture that used a novel combination of temporal convolutions and soft attention. Zhen et al. [49] introduced learning to learn kernels with variational random features for few-shot learning, where a specific kernel is inferred for each individual tasks conditioning on global context information collected by LSTMs. Recently, memory has generated increasing attention in the machine learning community, which was used to augment deep neural networks [43, 16]. It has also been introduced to the meta-learning framework for few-shot learning [29, 21, 48].
Gaussian processes have recently been introduced for Few-shot Learning [39, 23, 33, 33]. Tossou et al. [39] learn a parameterized kernel operator that can be combined with a differentiable kernel algorithm during inference. Deep Kernel Transfer (DKT) [23] explore Gaussian processes for fewshot classification, which learns covariance functions parameterized by deep neural networks. More recently, Titsias et al. [38] applies Gaussian processes to meta-learning by maximizing the mutual information between the query set and a latent representation of the support set. Snell et al. [33] develop a Gaussian process classifier by combining Pólya-Gamma augmentation and the one-vs-each softmax approximation [37] in order to efficiently marginalize over functions rather than model parameters. In our work, we explore a dense set of inducing variables in Gaussian processes for few-shot classification. The resultant dense Gaussian processes enable more knowledge to be shared among related tasks and transferred to new unseen tasks for efficient and effective learning. Notably, the idea of inducing points in GP also stimulates progress in many other directions of research. Set transformer [19] introduced a transformer architecture with an inducing-point-inspired design for efficient computation. This serves as a piece of supportive evidence for using inducing points in meta-learning, as we show in this paper that using inducing points to collect and accumulate shared knowledge from previously seen, related tasks offers a strong inductive bias for efficiently and effectively solving new tasks with few labeled data only.

heading: 5 Conclusion
text: In this paper, we introduce learning to learn dense inducing variables for few-shot learning. By specifying a Gaussian process prior prediction functions, we introduce dense inducing variables, which are learned from data to collect shared knowledge from previous work to facilitate efficient and effective learning of new tasks. With a universal deep feature extractor learned from the training tasks, we achieve task-specific Gaussian process prior by adapting the dense inducing variables through an inner-loop few-step gradient optimization. The resultant dense Gaussian processes are endowed with strong generalization and robustness. The effectiveness of the proposed method is validated on a variety of benchmark datasets, and state-of-the-art performance across tasks is observed. Notably, the substantial performance improvements on the cross-domain few-shot learning further demonstrate the strong generalization of the proposed method.

heading: 6 Acknowledgements
text: This work was supported by the DARPA TAMI program under No. HR00112190038.

Review summaries:
This paper proposes an extension of the Gaussian process (GP) approach presented in [1] for the setup of few-shot learning. With each new task, the data is first processed by a neural network (NN) to obtain low-dimensional feature representations, and then a GP classifier acts on these features. Since the number of data points per task is limited, the authors suggest augmenting the model with inducing inputs that are shared among all tasks. Updating the parameters follows the standard meta-learning procedures (e.g., MAML). Using the lower bound presented in [1], an internal loop updates the task-specific inducing inputs. And by maximizing the predictive likelihood on the query set, an outer loop updates both the global inducing inputs and the network parameters. Experiments on standard FSL and cross-domain few-shot benchmarks show an improved accuracy over relevant baseline methods.  

This paper develops a way to learn Gaussian processes with dense inducing variables by meta-learning for few-shot learning. The algorithm provides a strong inductive bias for learning new tasks by specifying a shared Gaussian process prior over prediction functions of a set of tasks. It adapts the inducing variables to each task by efficient gradient descent to achieve task specific predictions. Experiments show that this approach can achieve competitive performance in all the tasks considered, and state-of-the-art results in certain tasks.

This paper presents a meta-learning algorithm based on Gaussian processes. The meta-learning or few-shot image classification algorithms based on Gaussian processes have been developed recently due to their principled uncertainty modeling mechanism, which can be particularly useful for few-shot learning settings where data are scarse. The core idea of the presented algorithm is its use of dense inducing variables; usually, the inducing point methods for GPs are employed when we have large training data so the direct computation of predictive or marginal distribution is costly. Hence, the inducing points methods are usually set to be sparse because they are introduced mainly for a computational reason. On the other hand, This paper interprets the inducing points as a shared knowledge that can be transferred between tasks and proposes to encode such meta-knowledge into them. To this end, the proposed method uses a dense set of inducing points so that rich meta-knowledge obtained during meta-training can be embedded into the inducing point and can be used to quickly adapt to new tasks. The resulting algorithm is a typical gradient-based meta-learning where the goal is to maximize marginal likelihood in the inner loop and the predictive likelihood in the outer loop, and the algorithm is trained via episodic training. The proposed algorithm is validated on common benchmarks on few-shot image classification and demonstrated decent performance.