Learning to Learn Dense Gaussian Processes for Few-Shot Learning
Authors: Ze Wang
Zichen Miao
Xiantong Zhen
Qiang Qiu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Few-shot learning
Gaussian processes
inducing points

Abstract: Gaussian processes with deep neural networks demonstrate to be a strong learner for few-shot learning since they combine the strength of deep learning and kernels while being able to well capture uncertainty. However, it remains an open problem to leverage the shared knowledge provided by related tasks. In this paper, we propose to learn Gaussian processes with dense inducing variables by meta-learning for few-shot learning. In contrast to sparse Gaussian processes, we define a set of dense inducing variables to be of a much larger size than the support set in each task, which collects prior knowledge from experienced tasks. The dense inducing variables specify a shared Gaussian process prior over prediction functions of all tasks, which are learned in a variational inference framework and offer a strong inductive bias for learning new tasks. To achieve task-specific prediction functions, we propose to adapt the inducing variables to each task by efficient gradient descent. We conduct extensive experiments on common benchmark datasets for a variety of few-shot learning tasks. Our dense Gaussian processes present significant improvements over vanilla Gaussian processes and comparable or even better performance with state-of-the-art methods. 

Learning to Learn Dense Gaussian Processes for Few-Shot Learning
Ze Wang Zichen Miao Xiantong Zhen Qiang Qiu
Purdue University! University of Amsterdam Inception Institute of Artificial Intelligence {zewang, miaoz, qqiu)@purdue.edd edu x z.zeenuuva.n nl
Abstract
Gaussian processes with deep neural networks demonstrate to be strong learner for few-shot learning since they combine the strength of deep learning and kernels while being able to well capture uncertainty. However, it remains an open problem to leverage the shared knowledge provided by related tasks. this paper, we propose to learn Gaussian processes with dense inducing variables by meta-learning for few-shot learning. In contrast to sparse Gaussian processes, we define a set dense inducing variables be of much larger size than the support set in each task, which collects prior knowledge from experienced tasks. The dense inducing variables specify shared Gaussian process prior over prediction functions of all tasks, which are learned in a variational inference framework and offer a strong inductive bias for learning new tasks. To achieve task ses-sspecfiic prediction functions, we propose to adapt the inducing variables to each task by efficient gradient descent. We conduct extensive experiments on common benchmark datasets for variety of few-shot learning tasks. Our dense Gaussian processes present significant improvements over vanilla Gaussian processes and comparable or even better performance with state-of-the-an methods.
Introduction
Meta learning knowledge from a set of related tasks so as to fast resolve novel tasks sampled from the same underlying task distribution. Meta learning largely stimulates the rise of few-shot learning (9. and the recent advantages are mainly driven by designing learning algorithms that learn from massive tasks and acquire prior knowledge, which combined with small amount of labeled data, induces models that produce reliable predictions on novel tasks. In practice, the acquisition of prior knowledge can be realized in different forms. Model agnostic meta-learning (MAML) ( learns adapts to tasks by few iterations of gradient descend, which inspires many follow-up methods 3 [2] 22 46
also referred to as learning to learn, aims at learning to acquire shared
The adaptation of the entire network makes hard to be scaled to large networks, and many recent efforts focus on adapting the last classification layer only [4 5 while assuming universal feature extractor that shared across all tasks. Despite the remarkable progress, challenges remain due to the uncertainty in making predictions with very limited data, which requires models to have high robustness for few-shot learning.
Gaussian processes 25 serve as powerful model for the inference of functions, which enjoy appealing properties including natural uncertainty quantification and robustness when the amount of data is limited. By combining the strong learning ability of deep neural networks, Gaussian processes with deep kernels further demonstrate improved performance on supervised learning 44. alleviate the computational cost, sparse Gaussian processes were widely studied by learning sparse set of
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

inducing variables to effectively approximate the full dataset. In sparse Gaussian processes. the inducing variables usually in the forms of pseudo data or a subset of training data are expected to capture statistics of the whole dataset.
Gaussian processes offer desirable robustness to data scarcity by capturing uncertainty, and enjoy the high expressiveness of deep kernels. This makes Gaussian processes well-suited learning model for few-shot learning as demonstrated recent works 5% 25 58 It is shown in deep kernel transfer (DKT) 2 that Gaussian process with a meta-learned deep kernel can deliver strong performance through a simple kernel transfer. DKT relies on an important assumption that a universal deep kernel be too
can be obtained through training limited amount of tasks. However, this assumption restricted to generalize to novel tasks. In practice, when solving novel tasks, it is hardly guaranteed that the learned deep kernels well explain the data due to the scarce labeled data, and failing to properly fit the Gaussian process prior can lead to inaccurate predictions. It is therefore crucial to leverage more knowledge to be shared among Gaussian processes for individual tasks, which remains an outstanding problem for Gaussian process few-shot learning.
In this paper, we introduce learning to learn Gaussian processes by dense set of inducing variables for few-shot learning. In particular, we assume a Gaussian process prior over the prediction functions of few-shot learning tasks, which specified by a of inducing variables learned from data. These inducing variables are dense in the sense that they have much larger cardinality compared to the support set of each individual task. Under the meta-learning setting, the dense inducing variables are learned to collect the shared knowledge from experienced tasks to improve the learning of new tasks effectively with limited data.
To achieve task-specific prediction functions, we propose to adapt the inducing variables to each task by the gradient descent update. Without adding extra model parameters, the adaptation permits online fitting to the Gaussian processes prior to each individual task, and therefore allows the model to better fit any novel tasks at test time. In addition, in contrast to sparse Gaussian processes, we learn the inducing variables in low-dimensional deep feature space, which significantly reduces the compute cost of both learning and adapting the dense inducing variables.
The resultant dense Gaussian processes inherit the strong learning ability of deep kernels and enjoy the robustness to data scarcity and innate ability of uncertainty quantification, providing an effective few-shot learner. We validate the effectiveness of the proposed dense Gaussian processes on variety of few-shot learning tasks. We observe that dense Gaussian processes achieve substantial performance improvements over vanilla deep Gaussian processes for few-shot learning and deliver state-of-the performance on common benchmarks for few-shot learning. Moreover, our dense Gaussian processes demonstrate strong generalization on cross-domain few-shot learning tasks.
In summary, we make three major contributions as follows:
We introduce dense Gaussian processes by learning a set of dense inducing variables for few-shot learning. The inducing variables specify Gaussian process prior over predictive functions across tasks, which collect shared knowledge from experienced tasks and provide strong inductive bias for learning new tasks efficiently and effectively.
We propose gradient descent based adaptation to establish task-specific prediction functions based on the Gaussian process prior. The gradient descent-based adaptation is efficient without any auxiliary network components for inferring task representations.
We conduct extensive experiments on common benchmark datasets for few-shot classi- fication. The proposed dense Gaussian processes achieve consistent improvements over previous Gaussian processes with deep kernels and deliver comparable or even better results than ofate-of-the-art methods.
2 Methodology
In this section, we start with brief review of few-shot learning: and the key ingredients that build the foundation of the proposed method. We then introduce in detail learning to learn Gaussian processes with dense inducing variables for few-shot learning.

---

2.1 Backgrounds
Few-shot learning. The common practice of training few-shot learners is to formulate it as a episodic training. In each episode, the training is realized by simulating a few-shot learning task, where the learner is provided with a set of few label data called the support sei S, and required to make predictions
the corresponding query set Q with unlabeled test samples. Taking few-shot image classification as an example, it is defined as a N N-way K-shot learning problem, where K is usually a small number, e.g., K 5. Typically, in each episode indexed by t, one few-shot classification task is generated by first sampling N categories from the training data, each of which contains K samples form the support set S {xi.....; XNx K J. An adaptation to the few-shot classification model is then performed on St, by, e.g., computing prototypes 32]. or updating the network parameters [9] After adaptation, the query set is formed by evenly sampling M samples
from each category, i.e., Q, (x',...,'''x XNNMM J. to evaluate the updated model, and the error is propagated back to update the parameters.
Gaussian processes with deep kernels. Gaussian processes (GP) 2 place prior distribution p(f) over the target functions f. Consider we have set of n datapoints, D (x,, 31)) with additive noise y; fxx;) + Gj where ~ NN(0 o²). The GP prior over f can be written as,
p(f) N(, Knn),
where Knn x(x,,;,,) is the kernel matrix of training datapoints, and the likelihood is
p(ylf) II p(y. II (E,
Given test data point x., its predictive posterior is characterized by Gaussian distribution, f, ~ NNf. J.), where
μ1. k. (KCn + 0 o'I)-ly
(3)
G. = k(xx, xx) - k, 7 (KC,n + o²1)-²k.
Given a deep neural network &
X + Z, which maps from input space to deep feature space, we obtain the deep kernel function 44
(xx, x ()) k (xx)))), (x,)) k(2,,2))
(4)
The dimensionality of deep feature space Z is usually smaller than the original input space X, especially for image data, leading to more efficient kernel learning and inference. In this way, the GP prior in our method is parameterized by the network & p(f) (f Kan), where K,n. = (Zn,, z,,) = K(x,, x,) is the covariance matrix calculated based on deep features, and the function values are defined over deep features f = f(Z) (X)).
Sparse Gaussian Processes
The posterior inference in a Gaussian Process induces a cubic com- plexity of On²)) which scales poorly with a huge amount of datapoints n. Sparse Gaussian processes 17 are widely studied to improve the scalability of standard GPs, and popular direction introduces inducing variables {Z.) J.m to approximate the full GP marginal likelihood, where typical m & n in favor of improved efficiency.
36 34
In sparse Gaussian processes, the optimization for the inducing variables {Z, } can be achieved through variational inference [36]. And the GP prior 36) is augmented with the function values of inducing variables f,,. - fZZ) as pff, f.)) p(f fm. )(f.)) and the joint distribution can therefore be expressed as:
(5) where is assumed that y conditionally independent of f given fm By minimizing the dis- tance between the true posterior p(f, f. y)) and the variational approximate posterior ff.. = p( (f..). where (f...) a free variational Gaussian distribution A A(fm iu, A), the maximization of log likelihood amounts to maximizing variational lower bound:
ply, f. f...) = ply (Ptm:/))((((,m)((C...
log p(y) 2 {-
qf,, (.C.f.))oo ply, fm)) dfdff A L(Z, .)))
(f,
P.C.
3

---

obtain adapted GP prior for each class that parameterized by 1 11 1 T
z
f.
Z,
fm
x
y
y,
S
ž'
Figure 1: The graphical illustration of the proposed dense Gaussian processes for few-shot learning.
2.2 Learning to Learn Dense Gaussian Processes
In this section, derive dense Gaussian processes by inducing variables under the episodic training setting of meta-learning for few-shot learning.
Adapting Gaussian Process Priors For C-way K-shot classification task, we adapt f (0, K*), and the samples from the posterior predictive are combined based on a multi-class likelihood such as the softmax function. We would like to learn Gaussian Process prior over the prediction function, which can be adapted to few-shot learning task by its support set with a few gradient descent updates. The Gaussian process prior is established by a set of dense inducing variables, which is much larger than the support set in each task. The inducing variables are learned from data which serve to collect shared knowledge from experienced tasks to facilitate efficient learning of new tasks.
Formally, in an episode of the C-way K-shot task, the GP learner is provided with a support set S, = {xf y C.K for each class within the task, which can be achieved through an adaptation based on Eq. g with both parameters initialized by the meta representation t. 2o.
/====, We adapt a task-specific GP prior in the form of specific kernels ( 2r)
$< ) (Z)) arg max L(D, Z.
4.
As shown in 36. L( Z, can be first maximized by analytically solving the optimal (f,,),
g N (f. 4 A²), μ o mm EKmny, A = Kmm EK mm.s
where now n = |S| = C x K, thus Kmn denotes the covariance matrix between m inducing variables and support set data, and Σ = (K (Kmm +o -2 KT )- The Derivation is provided in the Appendix A Thus, the lower bound can be written as,
L( Ž) log (yl00, 02 Knn) 20 Tr(K,,, Knn);
where K... K Kum K mpm nm Note that can be adopted for multi-class classification on the support set. To be specific, consider a binary classification task where y {0, 1), it can be solved by the proposed dense GP regression by assuming a Gaussian likelihood. With the one-versus-rest scheme, the regression model can be extended to solve the C-way classification task. Specifically, regressor is learned for each class to predict whether sample x belongs to this class (with y = 1) or not (with y = 0). Each regressor is formulated as a single dense GP with own adapted inducing variables. By maximizing 9)
KT

---

Algorithm Learning to learn dense inducing variables.
1: Given: A description of the tasks as N-way K-shot with M query samples at each episode.
2: Given: Inner-lopp and out-loop learning rates a and X, number of steps for inner-loop optimization.
3: Initialize & and Z
4: repeat
5: Sample task with support set S [x and query set (x]])== in each task.
6: Extract feature vectors for both S and Q by applying z = o (xx) and z = + (xx)). respec-
tively.
7:
for Class n = do
8: for Inner iterations
do Perform inner-loop adaptation and tuned the per-class inducing variables z for class-adapted GP prior by applying 14.
9:
10:
end for end for
11:
12:
With Z obtained, calculate the predictive posterior using and perform the outer-loop adaptation by applying 15
13:
until Converge
14:
Return + and Z
Z T Ideally, the well-trained inducing variables sparsely approximate the full dataset, and reduce the prohibitive computation complexity to Onnm²).
Given query sample x., the posterior predictive distribution for class n, (ff(x{)) NN (o²) can be calculated as
,4, = Kmm EK
(10)
( = (xx, X.) - K (K (kkm) ki. E(kmm)
and the multi-class posterior predictive be expressed as,
exp(ff
ply: = 1|x.) = Ee exp(fs' -7) with f. ~ ((f))x)),
(1I)
where T is learned temperature for MLE. Note the trainable temperature 7 here allows automatic also be
scaling the unnormalized multi-class posterior predictive, and similar implementation found in [5]. Although sharing the same formulation, the here serves a completely different role as the temperature used in uncertainty calibration. The loss of empirical risk the query set, acting the outer-loop criteria, can be then be calculated as
Cowerr
Σ - logg((y./x.))
(12)
(X..).EE
In this way, the GP prior adapted to the task by S, further supervised by the query set Q, This in turn incorporates more knowledge from each task to consolidate task-shared meta knowledge into T Zo, which enables new tasks to be learned efficiently.
Dense Inducing Variables The aforementioned few-shot episodic training deep kernel allows an inner-loop adaptation to the model, and achieves better adaptation to the GP prior on novel tasks. However, performing adaptation to the entire deep kernel involves updating the deep neural network that parameterized the kernel, which introduces dramatically high computation cost and latency. Meanwhile, explicitly leveraging the meta knowledge learned from massive tasks, e.g., with memory that allows adaptive information retrieval 44 data hallucination 45 that expands the support set of current tasks have demonstrated impressive results on few-shot learning. Inspired by the idea of explicit knowledge representation in few-shot learning. and data approximation through inducing variables in sparse Gaussian processes, we introduce dense inducing variables, which are leamnd as a rich set of inducing variables that carry the neta--nooledge of the entire task distribution in the latent deep feature space, and permit efficient model adaptation to novel tasks.
5

---

Specifically, instead of acting as sparse approximation to the data of any particular task, the dense inducing variables here specify shared Gaussian process prior over prediction functions of all tasks, are introduce to support the entire tasks distribution, from which a potentially infinite number of tasks can be sampled from.
To achieve effective modeling of the cross task GP prior, we allow the dense inducing variables to be have a larger cardinality than the support set in single task, i.e., m >> [S]. Notably, unlike standard sparse Gaussian processes that learn the dense inducing variables as pseudo inputs in raw data space of high dimension, we propose to directly learn inducing variables in the same latent space of the feature from the deep neural network. By doing so, learning inducing variables effectively bypasses the heavy computation cost, and also enables efficient online adaptation to the inducing variables.
With dense inducing variables specifying hyperprior over predictive functions across tasks, and included as hyperparameters in the deep kernel Φ, we now introduce the online update, which induces the prediction function for specific task, through gradient
an adapted Gaussian process prior descent based adaptation to the dense inducing variables.
Specifically, given the support set of the task with support set St, we perform adaptation of the Gaussian process prior by using the few labeled data, and rewrite as
1
Inner (Z)
log M(y00, 021 Kmn)
Tr Tr(Knn Kan),
(13)
20²
and adapt the inducing variables by
Z Z aV 2o.-1 Linner (Z (Z
(14)
where a specifies an inner-loop learning rate, and each Z% initialized with 2o. Thanks to the first-
order approximation the computationally heavy second-orde derivative, the training with (4) can be remarkably accelerated.
Given the adapted inducing variables on the i-th task, the predictive posterior of function values on the query set can then be efficiently obtained by performing meta updates to both the feature extractor and the dense inducing variables calculated based on 12
Φ- = + - XV Coteer ② 2
(15)
2 Z AV 20 Lower 4 2n),
with A specifying the learning rate for the outer loop.
Note that we assume a universal feature extractor 4 and update it only in the outer loop, which is commonly used in meta-learning for efficiency and generalization to more powerful backbones. Mcanwhile, the specification of the task-specific Gaussian process prior through adapting dense inducing variables endows our model with high flexibility and strong generalization to novel tasks without scarifying efficiency, as we will demonstrate in our experiments.
In contrast to the previous work that simply learns transferable Gaussian processes on few-shot learning 233 our method adopts episodic learning to adapt the Gaussian process prior to each task by updating the dense inducing variables, which parameterize the deep kernel jointly with the deep feature extractor 4. The overall training procedure of the dense Gaussian process model is summarized in Algorithm T
3 Experiments
In this section, we support the proposed methods with strong experimental results. We present exten- sive experiments on both in-domain and cross-domain few-shot image classification. Visualizations of the uncertainty quantification with examples of few-shot regression provided in Appendix Figure A We finalize the discussions with the ablation study on the effectiveness of important components of the proposed method.
3.1 Datasets and Settings
We perform experiments on the widely used few-shot learning benehmarks including miniImageNet, tieredImageNet, CIFAR-FS, and Caltech-UCSD [2] (CUB). In minilmageNet [1], there are 100

---

Table Performance comparisons on minilmageNet and the Conv-4 network with 95% confidence
intervals.
minilmmgeNNe
Iagred
CIFAR-FS
Methods
I-shot
5-shot
I-shot
5-shot
I-shot
5-shot
MatshingNets
43.56 0.84
55.31
207
MAML 110
48.70 1.84
63.11
=092
51.67 181
70.30
58.90
71.52
Reptile
2R
49.97 0.32
65,99
0.58
R2-D2 6
48.70 000
65.50
0.0
60.20
70.91 091
VERSA [E]
53.31 1,80
67.30
0.1
62.51
1.70
75.11
0.1
RelationNets (35
50.44 0.82
65.32
0.70
54.48 0.9
65.32 1.70
55.00
1.01
69.30
0.80
ProtoNets 101
47.40 0,60
65.41
0.5
53.31
000
72.69
0.74
55,50
0.70
72.01
0.60
VSM 48
54.73 1.00
68.01 0.90
56.88
1771
74.65
0.81
63.42
1.90
77.93
0.00
DKT 23
49.73 0.07
64.00 0.09
OVE GP Cosine (ML)
50.02 0.35
64.58
0.51
OVE PG GP Cosine (PI.)
48.00 0.24
67.14
0.23
Our
56.32 1, 0.28
72.64.1 0,26
58,431 0.38
76.17 0.34
64.17 1.3.1
78.42 0.26
Table 2: Performance comparisons on the CUB dataset with 95% confidence intervals.
CUB (Conv-4)
CUB (ResNet-10)
Methods
1-shot
5-shot
1-shot
5-shot
Feature Transfer
46. 1 0.64
68
1.779
63.64 0.1
81.27 1. 0.57
ABML
49.57 1 0.42
68.94
0.166
Baseline
61.75 + 0.95
78,51
0.5
69.55
= 0.99
85.17
0.50
MatchingNet
60.19 # 1.02
75.11
0.35
71.29
= 0.87
83.47
0.58
ProtoNet 32
52.52 # 1.90
75.93
0.46
73.22
= 0:22
85.01
0.52
RelationNet 6-
62.52 1 0.34
78.22
0.07
70.47
1. 0.99
83.70
0.55
MAML [10
56.11 1 0.69
74.84
1 0.62
70.32
=: 0.99
80,93 1 0.71
Bayesian MAML
47
55.93 + 0.71
72.87
0:26
DKT
62.96 # u.62
77.76
0.62
72.27 = 0.30
85.64 # 0.29
OVE GP + Cosine (ML)
33
63.98 + 043
77.44
0.18
OVE PG GP + Cosine (PI.)
33
60.11 # 0.26
79.07
+ 0.05
Ours
69.18 0.41
81.48
0.58
78.83
0.67
89.97
0.63
image classes from subset of ImageNet S with 600 images for each class. We follow the standard practice 9 to split the training, validation, and testing sets with 64, 16, and 20 classes, respectively. rieredlmageNet (2 is large subset of ImageNet that contains 608 classes with 1,300 samples in each class. Specifically, in tiered there are 351 classes from 20 categories for training, 97 classes from 6 categories for validation, and 160 classes from 8 different categories for testing. Samples for both minilmageNet and tieredImageNet are random cropped and resized to 84 x 84 for training, and standard center cropping is performed to the testing images. The 200 classes in the CUB dataset is divided into 100, 50, and 50 classes, for training, validation, and testing, respectively. CIFAR-FS adopts all the 100 classes in the CIFAR- dataset with training, validation, and testing splits of 64, 16, and 20 classes, respectively. Each class contains 600 image samples. The image resolution for CIFAR-FS FS is 32 x 32.
3.2 Few-Shot Image Classification
The results of few-shot classification on minilmageNet, tieredImageNett, CIFAR-FS and Caltech- UCSD (CUB) reported in Tables I and 2 For comprehensive comparisons, adopt two backbone networks with different scales. On all three datasets, we achieve comparable or often better performance than state-of-the-an methods. Our method yields the accuracy of 72.64 on the minilmageNet dataset under the 5-way 5-shot setting, which surpasses the previous best method by up to 5%. particular, our dense Gaussian processes achieve much better performance than the recent Gaussian processes based methods [23, B3 This demonstrates the effectiveness of our method by learning dense inducing variables. For more experimental detail please refer to Appendix
Section B
7

---

Table 3: Performance comparisons for cross-domain few-shot classification with 95% confidence intervals.
Omniglot 1-shot
EMNIST 5-shot
minilmageNett - CUB 1-shot
Methods
5-shot
Feature Transfer
64.22
86.10
32.77
0.35
50.34
ABML 126
76.37 A 0.29
87.96
0.28
29.35
a: 0.26
45.74 4: 0.39
Baseline ++
56.84 = 0.91
80.01
= 0.92
39.19
2012
57.31 0.11
MatchingNel 40
75.01 + 2.09
87.41
# 1.79
36.98
= 0.06
50.72 + 0.36
ProtoNet 62
72.04 0.82
87.22
1.01
33.27
1.09
52.16 : 0.17
RelationNet
35
75.62 1.00
87.84 0:27
37.13 0.20
51.76 1.48
MAML IO
72.68
83.54
34.01 1.25
48.83 0.22
Bayesian MAML 447
63.94
65.26
33.52 = 0.36
51.35 +
DKT 23
75.40 1.10
90.30 0.49
40.14 1 0.18
56.40 1: 1.34
OVE GP (MI.)
68.43 = 0.67
86.22 = 0.20
39.66 0.18
55.71
# 0.31
OVE PG GP (PL)
35
77.00 0.50
87.52 0.19
37.49 0.11
57.23
#: 0.31
Ours
78.32
90.78 0.24
43.45 0.38
60.48 0.53
3.3 Cross-Domain Few-Shot Image Classification
The efficient adaptation to the GP prior through updating the dense inducing variables enables the proposed method with improved generalization to novel tasks with unseen categories. To fully demonstrate the remarkable generalization, we perform cross-domain few-shot image classifica- tion. Following common practice, we adopt two cross-domain settings, Omniglot + EMNIST and minilmageNet->> CUB. The results are reported in Table B The proposed dense Gaussian processes demonstrate high generalizability across domains for few-shot classification and achieve much better performance than the recent Gaussian process based methods [x 33 in the setting of generalization from Omniglot to CUB.
3.4 Ablation Study
We perform extensive ablation studies to provide insights into the effectiveness of the proposed dense Gaussian processes for few-shot learning. These experiments are performed the CUB dataset.
Effectiveness of dense inducing variables The proposed dense inducing variables enable shared knowledge to be collected from experienced tasks and applied to new unseen tasks. We show how the number of inducing variables affects the performance our model. As shown in Figure g performance on both 5-way 1-shot and 5-way 5-shot experiments. We test a sequence of values of m starting from 8, which is just slightly larger than the number of support set samples a 5-way 1-shot experiment. And the value of is progressively increased to 1024. It is clearly demonstrated that, on both experiments with different shots, the performance grows monotonically w.r.t the number of inducing points that utilized. And the performance tends to saturate when m 256. To maintain a balance between efficiency and performance, we choose m = 256 throughout all the experiments. The results in Figure 2 also validate the effectiveness of specifying the GP prior through 'dense' inducing variables, with much more elements compared to the support set in each task.
we report
Effectiveness of inner-loop adaptation In the proposed method, the task-specific GP prior adap- tation is achieved by few-step gradient descent to the dense inducing variables. For the selection of the best configuration of the inner-loop optimization, we present comparisons performed with the number of steps I in the inner-loop gradient descent, and the inner-loop learning rate a. As shown in Figure 14, we plot the classification accuracy for 5-way 5-shot setting, and show the performance obtained by conducting to 20 steps of inner-loop update with three different values of a. It is clearly shown that the performance reaches the peak within 5 steps of inner-loop update, and it does not improve with even more steps of adaption, especially when combined with larger learning rates. This indicates the efficiency of the proposed adaption of the Gaussian process prior each task. Moreover, large can lead to performance drop very fast when performing more than I step of inner-loop update. To keep the balance between accuracy and efficiency, we choose a = 0.01 and I = 5 throughout all the experiments.
8

---

Performance with different values
Performance different I and
Accuracy 1-shot
Accuracy S-shot
Accuracy
0-0.1
0-0.01
16
64
124
255
512
1024
10
20
Figure 2: The performance with different sizes of inducing variables.
Figure 3: The performance with different num- bers of adaptation steps and hyper-parameters.
4 Related Work
Metric learning T has been explored for few-shot learning, which assumes that different related tasks would share the same metric to measure similarity. Vinyals et al. [1 proposed the matching network, which learns to map small labeled support set and an unlabelled example its label, obviating the need for fine-tuning adapt to new class types. This work was originally developed for one-shot learning, and extended to a feo-shot setting by Snell et al. 32]. They proposed the prototypical network to learn metric space in which classification distances to prototype representations of each class and the prototype of each class the cluster of samples in that class. To enhance the expressivity of the prototypes, Allen al. I proposed infinite mixture prototypes to adaptively represent both simple and complex data distributions for few-shot learning. Satorras al. observed graphical model.
be performed by computing
BC solved few-shot learning with the prism of inference on a partially
Meta-leamnigg based optimization has been extensively studied (27 10
The core idea of optimization- methods is to learn an optimization procedure that is shared across tasks, which can be applied to new tasks for quick adaptation, once learned in the
stage. Ravi al. 27 proposed an LSTM based meta-learner to learn the exact optimization algorithm used to train neural network classifier in the few-shot regime; in their methods, apart from the learned optimization algorithm, good initialization of model parameters is also obtained after training. Finn et al. M proposed a model agnostic meta-learning (MAML) algorithm, which was built upon the assumption that related tasks sharing initial parameters of neural networks could be adapted to specific tasks with few steps of gradient descent updates. It was believed that the initial weights combined with a few more steps of gradient descent can approximate any learning algorithm. and thus gradient-based meta-learning had a number of practical benefits. Finn et al. [13 extended MAML in a probabilistic framework. Rajeswaran al. [24] developed the implicit MAML. 100 algorithm, which depended
only on the solution to the inner level optimization. Zintgraf et al. 550] updated context parameters with one or several gradient steps on a task-specific loss that serves as an additional input to the model and were adapted on individual tasks.
has also been investigated explicitly design a meta- meta-learner learn a base-learner, Bertinetto et al. 6 explored the feasibility of incorporating fast solvers with closed-form solutions as the base learning component
meta-lcarning system. Gordon et al. [15 developed meta-learning approximate probabilistic inference for prediction. The support set was used to produce the parameter distribution of the classifier, which was applied to the query set for prediction. Mishra et al. (20) proposed a generic meta-learner architecture that used novel combination of temporal convolutions and soft attention. Zhen et al. 49 introduced learning to learn kernels with variational random features for few-shot learning, where specific kernel is inferred for each individual tasks conditioning on global context information collected by LSTMs. Recently, memory has generated increasing attention in the
machine learning community, which was used augment deep neural networks 43 16)
has also
been introduced to the meta- meta-lcarning framework for few-shot learning
20
21
48

---

Gaussian processes have recently been introduced for Few-shot Learning 39 23 33 333 Tossou al. K9 learn a parameterized kernel operator that can be combined with a differentiable kernel algorithm during inference. Deep Kernel Transfer (DKT) 23 explore Gaussian processes for few- shot classification, which learns covariance functions parameterized by deep neural networks. More recently, Titsias et al. [X applies Gaussian processes to meta-learning by maximizing the mutual information between the query set and a latent representation of the support set. Snell et al. [33] develop Gaussian process classifier by combining Pólya-Gamma augmentation and the one-vs-each softmax approximation K in order to efficiently marginalize over functions rather than model parameters. In our work, we explore a dense set of inducing variables in Gaussian processes for few-shot classification. The resultant dense Gaussian processes enable more knowledge to be shared among related tasks and transferred to new unseen tasks for efficient and effective learning. Notably, the idea of inducing points in GP also stimulates progress in many other directions of research. Set transformer [19] introduced transformer architecture with an inducing muducing-point-iigg point-inspired design for efficient computation. This serves piece of supportive evidence for using inducing points in meta-lcarning, as we show in this paper that using inducing points to collect and accumulate shared knowledge from previously seen, related tasks offers strong inductive bias for efficiently and effectively solving new tasks with few labeled data only.
5 Conclusion
In this paper, we introduce learning to learn dense inducing variables for few-shot learning. By specifying a Gaussian process prior prediction functions, we introduce dense inducing variables, which are learned from data to collect shared knowledge from previous work to facilitate efficient and effective learning new tasks. With a universal deep feature extractor learned from the training tasks, we achieve task-specific Gaussian process prior by adapting the dense inducing variables through an inner-loop few-step gradient optimization. The resultant dense Gaussian processes are endowed with strong generalization and robustness. The effectiveness of the proposed method is validated on variety of benchmark datasets, and state-of-the-art performance across tasks is observed. Notably, the substantial performance improvements the cross tross-domain few-shot learning further demonstrate the strong generalization the proposed method.
Acknowledgements
This work was supported by the DARPA TAMI program under No. HR001 12190038.
References
[1] Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. In ICML, 2019.
[2] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In NeurlPS, 2016.
[3] Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. arXiv preprint arXiv: 1810000000 2018
[4] Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Geesei. On the optimization of a synaptic learning rule. In Preprints Conf. Optimality Artificial and Biological Neural Networks, volume 2, 1992.
[5] Luca Bertinetto, Joao Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. arXiv preprint arXiv:1805.088 2018.
[6] Luca Bertinetto, Joao Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In ICLR, 2019.
[7] Wei-Yu Yu Chen, Yen-Cheng Liu, 7solt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at shot classification. arXiv preprint 1w04::1904.04232. 2019.
10

---

[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 255. IEFE, 2009.
[9] Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1126-115 135. PMLR, 2017.
[10] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In ICML, 2017.
[11] Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. In ICLR, 2018.
[12] Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. arXiv preprint 1806,08800002880
2018.
[13] Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 9537-9548, 2018.
[14] Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Meta-leamiigg probabilistic inference for prediction. arXiv preprint
2018. [15] Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Meta-learning probabilistic inference for prediction. In ICLR, 2019.
[16] Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwiñska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):411-476 2016.
[17] James Hensman, Nicolo Fusi, and Neil D Lawrence. Gaussian processes for big data. arXiv preprint 1309.6835... 2013.
[18) Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. arXiv preprint WxXi:2200.004439 2020.
[19] Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Tch. Set transformer: A framework for attention-based permutation-invarian neural networks. In ICML, pages 3744-3753. PMLR, 2019.
[20] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive leta-learner. In ICLR, 2018
[21] Tsendsuren Munkhdalai and Hong Yu. Meta networks. In ICML, 2017.
[22] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint 1803.0999,0 2018.
[23] Massimiliano Patacchiola, Jack Turner, Elliot J Crowley, Michael O'Boyle, and Amos J Storkey. Bayesian meta-learning for the few-shot setting via deep kernels. Advances in Neural Informa- fion Processing Systems, 2020.
[24] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In NeurIPS, 2019.
[25] Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pages 63-71. Springer, 2003.
[26] Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning. In ICLR, 209.
[27] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR,
2017.
[28] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenen- baum, Hugo Larochelle, and Richard S Zemel. Meta-leaming for coiissuprvised few-shot classification. International Conference on Learning Representations, 2018.
[29] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory augmented neural networks. In ICML, 2016.
11

---

[30] Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks.
In ICLR, 2018.
[31] Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis. Technische Universität München, 1987.
[32] Jake Snell, Kevin Swersky, and Richard $ Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 2017.
[33] Jake Snell and Richard Zemel. Bayesian few-shot classification with one-vs-each pólya-gamma augmented gaussian processes. International conference on learning representations, 2021.
[34] Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs.
Advances in neural information processing systems, 18: 1257-1212 2005.
[35] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1 1199-1208, 2018.
[36] Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics, pages 567-574. PMLR, 2009.
[37] Michalis K Titsias. One-vs-each approximation to softmax for scalable estimation of probabili- ties. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages
2016.
[38] Michalis K Titsias, Sotirios Nikoloutsopoulos, and Alexandre Galashov. Information theoretic meta learning with gaussian processes. arXiv preprint arXiv:2009.03222 2020.
[39] Prudencio Tossou, Basile Dura, Francois Laviolette, Mario Marchand, and Alexandre Lacoste. Adaptive deep kernel learning. arXiv preprint arXiv:1905. 12131,, 2019.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[41] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 2016.
[42] Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, Belongie, and P. Perona. Caltech-UCSD Birds 200. (CNS-TR-2010-001), 2010.
[43] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv preprint arXiv:1410.3916, 2014.
[44] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric Xing. Deep kernel learning. In Artificial intelligence and statistics, pages 370-378. PMLR, 2016.
[45] Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. In International Conference on Learning Representations, 2021.
[46] Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 7343-7353, 2018.
[47] Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In NIPS, 2018
[48] Xiantong Zhen, Yingjun Du, Huan Xiong, Qiang Qiu, Cees Snoek, and Ling Shao. Learning to learn variational semantic memory. NeurlPS, 2020.
[49] Xiantong Zhen, Haoliang Sun, Yingjun Du, Jun Xu, Yilong Yin, Ling Shao, and Cees Snoek. Learning to learn kernels with variational random features. ICML. 2020.
[50] Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learnin, In ICML, 2019.
12


Reviews:
[{"review_id": "kp0BRcxhIki", "reviewer": "Reviewer_jkmZ", "summary": "This paper proposes an extension of the Gaussian process (GP) approach presented in [1] for the setup of few-shot learning. With each new task, the data is first processed by a neural network (NN) to obtain low-dimensional feature representations, and then a GP classifier acts on these features. Since the number of data points per task is limited, the authors suggest augmenting the model with inducing inputs that are shared among all tasks. Updating the parameters follows the standard meta-learning procedures (e.g., MAML). Using the lower bound presented in [1], an internal loop updates the task-specific inducing inputs. And by maximizing the predictive likelihood on the query set, an outer loop updates both the global inducing inputs and the network parameters. Experiments on standard FSL and cross-domain few-shot benchmarks show an improved accuracy over relevant baseline methods.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think that using Gaussian processes with NNs for few-shot challenges is a promising direction. However, there are some issues I would like the authors to address before I can pass it. \n\nThis paper presents the following merits:  \n- The paper shows strong results in a variety of experiments and against many baselines.  \n- Using inducing variables to generate a richer dataset is a good idea.  \n- The paper is written clearly and is easy to follow besides several points that will be described next.\n\nSome points that I found problematic:  \n- There are two crucial points that were not entirely clear to me, and since the code wasn't provided with the submission I could not get an answer myself. I will be happy to get clarification for both issues.  \n(1) This paper builds on [1] which was designed for regression tasks with Gaussian likelihoods. To use this method, it seems that the authors referred to the discrete class labels as if they are Gaussian distributed. If indeed I understood correctly, it does not seem like a natural thing to do. Can you please clarify that point and further explain the intuition.   \n(2) Unlike training in which the inducing inputs are jointly learned across tasks, when testing the model the classes are new and there should be no information sharing between tasks. Is that correct? If so, can you please clarify if and how does your method gain from sharing the inducing inputs of these new classes?  If the inducing location is only adjusted based on the inner loop iterations of the current task, does that suffice? I think that some analysis and more explanations regarding this part are required.  \n- Many implementation details are missing. For example, did you perform a grid search over hyperparameters (besides the ones shown in the ablation study)? what temperature did you use in the softmax? how many epochs of training did you use? etc. In general, the text should provide much more details about the datasets, the experimental setups, and the baseline methods.  \n- I think this paper omits important GPC references [2, 3, 4]. [2] handled a few-shot scenario as well, and all methods used inducing inputs in the embedding space of a NN.  \n- From my experience temperature scaling often impact significantly on the classifier performance (for example, see [5] for a recent study on this topic). If indeed you adjusted the temperature for your method, did you try to tune it for relevant baseline methods as well? Specifically, I think that the performance of the main competitor OVE-GP (PL) should also be examined under the same conditions.  \n\nMinor comments:\n- There are some typing mistakes: gradient decend (L25), to learned (L152), Our (Table 1), double citation of the same reference (L234), the cross-domain tasks in Table 3 should be: Omniglot -> EMNIST & miniImageNet -> CUB.\n- It is not clear from the text what network did you use to obtain the results in Table 1.\n\n\n[1] Titsias, M. (2009, April). Variational learning of inducing variables in sparse Gaussian processes. In Artificial intelligence and statistics (pp. 567-574). PMLR.  \n[2] Achituve, I., Navon, A., Yemini, Y., Chechik, G., & Fetaya, E. (2021). GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning. arXiv preprint arXiv:2102.07868.  \n[3] Wilson, A. G., Hu, Z., Salakhutdinov, R., & Xing, E. P. (2016, December). Stochastic variational deep kernel learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems (pp. 2594-2602).  \n[4] Bradshaw, J., Matthews, A. G. D. G., & Ghahramani, Z. (2017). Adversarial examples, uncertainty, and transfer testing robustness in Gaussian process hybrid deep networks. arXiv preprint arXiv:1707.02476.  \n[5] Wenzel, F., Roth, K., Veeling, B., Swiatkowski, J., Tran, L., Mandt, S., ... & Nowozin, S. (2020, November). How Good is the Bayes Posterior in Deep Neural Networks Really?. In International Conference on Machine Learning (pp. 10248-10259). PMLR.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not address the potential limitations and negative societal impact of their work.  I think that authors should address this in the rebuttal.  \nTo the best of my judgment, there is no potential negative societal impact for this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "hv9p8YDPQq0", "reviewer": "Reviewer_JcrC", "summary": "The authors propose a solution to few shot learning by using Gaussian processes with dense inducing points from the perspective of few shot learning, which allows for learning a shared set of inducing points which are then specialized into task specific inducing points. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n\n- The results are strong and outperform all baselines methods.\n- The method build off of work in GP's and meta learning\n- The authors do a good job of citing past work in both GP and meta learning literature.\n\n## Cons\n\n- L40 (about GP's) ... \"without suffering from computational cost in regular learning tasks\" --> What does this mean? There is no computational cost? Even if you are saying there is reduced computational cost, is this even true? GP's scale as $\\mathcal{O}(n^3)$\n- Equation 7 has the order of arguments for the variational lower bound changed from equation 6. Is this a typo?\n- The paper mentions dense inducing variables many times, but the inducing variables are actually quite sparse w.r.t. the task distribution. They are only dense when considered against the typical few shot learning task sizes. I think this distinction should be made clearer. \n- L299 says that figure 3 indicates the efficiency of the proposed solution. This is a very bold claim that lacks evidence. One could easily also say that it indicates the method overfits easily to tasks, as the performance drops quickly with more inner steps. This overfitting is something that other methods such as [1] do not show even with large numbers of adaptation steps.\n- There is no analysis of the added runtime for training/inference, as well as trainable variable counts. This information should be included somewhere in order to better understand the relation to other methods. The method leads me to suspect that there would be a minimal parameter increase, but a large increase in training/inference times. \n- The authors don't describe any limitations of the work. \n- The authors do not apply the problem to the very common few shot regression tasks. Although these are usually simple, I think the GP aspect make a particularly compelling use case for regression tasks which should be discussed.\n\nMy score is reflective of mostly bullet points 5 and 7 which need to be addressed in order to make the submission stronger. \n\n ## Minor\n\n- L14 significantly --> significant\n- L28 progresses --> progress\n- L29 the few shot learning --> few shot learning\n- L35 \"widely developed\" --> What does \"widely\" developed mean?\n- L50 \"we learning to learn\" --> we learn to learn\n- L73 \"learning new task\" --> learning new tasks\n- L131 \"few gradient descent update\" --> few gradient descent updates\n- L148 \"an learned\" --> a learned\n- L154 \"achieves better prior fitting the novel dataset\" --> sentence needs to be rewritten to make sense\n\n## References\n\n[1] Nichol, A., Achiam, J., & Schulman, J. (2018). On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no mention of limitations or societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4-5 hours"}, "extra_text": {}}, {"review_id": "E_KPKVpDGIs", "reviewer": "Reviewer_yiLY", "summary": "This paper develops a way to learn Gaussian processes with dense inducing variables by meta-learning for few-shot learning.\nThe algorithm provides a strong inductive bias for learning new tasks by specifying a shared Gaussian process prior over prediction functions of a set of tasks. It adapts the inducing variables to each task by efficient gradient descent to achieve task specific predictions.\nExperiments show that this approach can achieve competitive performance in all the tasks considered, and state-of-the-art results in certain tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The paper is relatively well written. \n2. In terms of originality, I am not aware of other similar works.\n3. The empirical results are pretty strong, achieving state-of-the-art results in certain settings.\n\n**Detailed comments**:\n1. Are there any theoretical guarantees for algorithm 1?\n2. In equation 5, should it be $p(y|f_m)$ instead of $p(y|f)$?\n3. Minor: some formatting issues at Line 20, 24.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no discussion on the limitations and potential negative societal impact of the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "E5-2fTXf-0R", "reviewer": "Reviewer_PL2R", "summary": "This paper presents a meta-learning algorithm based on Gaussian processes. The meta-learning or few-shot image classification algorithms based on Gaussian processes have been developed recently due to their principled uncertainty modeling mechanism, which can be particularly useful for few-shot learning settings where data are scarse. The core idea of the presented algorithm is its use of dense inducing variables; usually, the inducing point methods for GPs are employed when we have large training data so the direct computation of predictive or marginal distribution is costly. Hence, the inducing points methods are usually set to be sparse because they are introduced mainly for a computational reason. On the other hand, This paper interprets the inducing points as a shared knowledge that can be transferred between tasks and proposes to encode such meta-knowledge into them. To this end, the proposed method uses a dense set of inducing points so that rich meta-knowledge obtained during meta-training can be embedded into the inducing point and can be used to quickly adapt to new tasks. The resulting algorithm is a typical gradient-based meta-learning where the goal is to maximize marginal likelihood in the inner loop and the predictive likelihood in the outer loop, and the algorithm is trained via episodic training. The proposed algorithm is validated on common benchmarks on few-shot image classification and demonstrated decent performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading the paper. The paper is well-written and clear, so easy to follow. The motivation is well-set, and I like the idea of transferring knowledge through inducing points. In my opinion, the main message of the paper is the use of dense inducing points whose size is much larger than typical support sets, and I think this idea can be applied to other related approaches.\n\nAlthough not explicitly related, but I'd like to point out one related work that is worth citing. Set transformer (Lee et al., 2019) introduces an inducing point-based self-attention block to process set-structured data and is applied to meta-learning problems such as amortized clustering. There, the authors showed that their method with inducing points performs better than the full self-attention blocks without inducing points, and claim that the inducing points act as transferrable knowledge that can be shared among different tasks. I guess this view of interpreting inducing points is somewhat similar to the method proposed in this paper.\n\nThe experimental results are not that compelling; I'm not saying this because the tables in the paper do not list all the recent SOTA few-shot learning results (which would probably take an entire page). As the authors emphasized at the beginning of the text, meta-learning-based Gaussian processes provide a principled way of estimating uncertainties. In order to properly assess such aspects of GP-based meta-learning, I think the competing algorithms should be evaluated with the metrics related to uncertainty estimates (e.g., expected calibration error, negative log-likelihood, out-of-distribution detection performance, active learning, ...). It would have been good to have a minimal example on toy-regression tasks with meta-learned credible intervals displayed. \n\nAnother thing that needs to be checked is the complexity of the algorithm. Figure 2 reports that the performance starts to saturate with the number of inducing points $m=256$. Then the time-complexity of computing marginal likelihood is roughly $O(nm^2)$, which is quite heavy since $m \\gg n$. How are the actual running times (training time and inference time) compared to the baselines? I guess the proposed method can be quite slower than the baselines without GPs (which just typically requires a few forward passes through feature extracting neural nets) and GP-based without inducing points ($O(n^3)$).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not address the limitations and potential negative societal impact as shown in the checklist.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agree that the idea of using a dense set of inducing points to share information across tasks in the few-shot setting is novel and interesting. The discussions yielded a number of clarifications as well as new experiments on few-shot regression and uncertainty quantification. Please be sure to add these to the final draft.
Decision: Accept (Poster)