Learning to Learn Kernels with Variational Random Features
Authors: Haoliang Sun
Yingjun Du
Jun Xu
Yilong Yin
Xiantong Zhen
Ling Shao
Venue:  (2020)
Keywords: Meta-learning
few-shot learning
Random Fourier Feature
Kernel learning

Abstract: Meta-learning for few-shot learning involves a meta-learner that acquires shared knowledge from a set of prior tasks to improve the performance of a base-learner on new tasks with a small amount of data. Kernels are commonly used in machine learning due to their strong nonlinear learning capacity, which have not yet been fully investigated in the meta-learning scenario for few-shot learning. In this work, we explore kernel approximation with random Fourier features in the meta-learning framework for few-shot learning. We propose learning adaptive kernels by meta variational random features (MetaVRF), which is formulated as a variational inference problem. To explore shared knowledge across diverse tasks, our MetaVRF deploys an LSTM inference network to generate informative features, which can establish kernels of highly representational power with low spectral sampling rates, while also being able to quickly adapt to specific tasks for improved performance. We evaluate MetaVRF on a variety of few-shot learning tasks for both regression and classification. Experimental results demonstrate that our MetaVRF can deliver much better or competitive performance than recent meta-learning algorithms.

Under review as a conference paper at ICLR 2020
LEARNING TO LEARN KERNELS WITH VARIATIONAL RANDOM FEATURES
Anonymous authors Paper under double-blind review
ABSTRACT
Meta-learning for few-shot learning involves a meta- that acquires shared knowledge from a set of prior tasks to improve the performance of base-learner on new tasks with small amount of data. Kernels are commonly used in machine learning due to their strong nonlinear learning capacity, which have not yet been fully investigated in the meta-learning scenario for few-shot learning. In this work, we explore kernel approximation with random Fourier features in the meta-learning framework for few-shot learning. We propose learning adapative kernels by meta variational random features (MetaVRF), which is formulated as a variational infer- ence problem. To explore shared knowledge across diverse tasks, our VRF deploys an LSTM inference network to generate informative features, which can establish kernels of highly representational power with low spectral sampling rates, while also being able to quickly adapt to specific tasks for improved performance. We evaluate VRF on a variety of few-shot learning tasks for both regression and classification. Experimental results demonstrate that our MetaVRR can deliver much better or competitive performance than recent meta-learning algorithms.
INTRODUCTION
Humans have the instinct to effortlessly learn new concepts from a few examples and show great generalization ability to new samples. However, existing machine learning models, e.g., deep neural networks (DNNs) (Krizhevsky et al., 2012; He et al., 2016a), rely highly on large-scale annotated training data (Deng et al., 2009) to achieve satisfactory performance. The huge gap between human intelligence and DNNs motivates us to try and progress the task of learning from few samples, a.k.a. few-shot learning (Fei-Fei et al., 2006; Lake et al., 2015; Ravi & Larochelle, 2010))..e,,017)..
Learning to learn, or meta-learning (Schmidhuber, 1992), has recently received great interests in the machine learning community and offers promising tool for few-shot learning (Andrychowicz et al., 2016; Ravi & Larochelle, 2017; Finn et al., 2017). Generally speaking, a meta-learner (Ravi & Larochelle, 2017; Bertinetto et al., 2019) is trained to improve the performance of a base-learner on individual tasks, which is also fast adapted to solve new tasks. The crux of meta- meta-learning for few-shot learning is explore the common knowledge, such good parameter initialization (Finn et al., 2017) or efficient optimization update rule (Andrychowicz et al., 2016; Ravi & Larochelle, 2017). 1)),, shared across different tasks. The knowledge is accumulated and distilled throughout the learning stage, making the model adaptable to new but related tasks (Finn et al., 2017).
Kernel approximation by random Fourier features (RFFs) (Rahimi & Recht, 2007) is an effective technique for efficient kernel learning (Gärtner et al., 2002), which has recently become increas- ingly popular (Sinha & Duchi, 2016; Carratino al., 2018). It resorts the Fourier transform of ihift-invariant kernels and constructs explicit feature maps using the Monte Carlo approximation of the Fourier representation. The desired kernel function is approximated by the inner products between these random features. Though demonstrating great potential as strong base learner, kernel approximation with random features has not yet been fully explored the meta-learning scenario for few-shot learning. It has already been shown that the classification performance of the kernel with random features does not correlate well with the accurate approximation of kernels. Learning adaptive kernels with random features, for instance, by data-driven sampling strategies (Sinha & Duchi, 2016), can improve the performance with a low sampling rate compared to using universal random features (Avron et al., 2016; Chang al., 2017). However, since only few samples

---

Under review as a conference paper at ICLR 2020
8
S
State
State
State
1:t-2
Cell
cl:t-1
Cell
c
Cell
clit+1 sos
LSTM Meta-Learner
Bases
Bases
6f
Bases
Base-Learner 1 c-1
Base-Learner 1e
Base-Learner 1
Figure I: The inference framework of our meta variational random features (MetaVREl with a long short-term memory (LSTM) network (t the task index). The meta-leamnrr employs an LSTM network to infer random Fourier features from the support set St of the current task and bases w' of previous tasks, which explores task dependency to extract shared knowledge, The base-learner is classifier with kernels obtained by variational features from the meta-learner.
available in each task, it is challenging to learn adaptive kernels with data-driven random features while maintaining high representational capacity for few-shot learning tasks. To obtain powerful kernels for few-shot learning tasks, we need to fully explore the relationship among diverse tasks and capture their shared knowledge to generate informative random features.
In this work, we propose meta variational random features (MetaVRF) to approximate kernels in a data-driven manner for few-shot learning, which integrates variational inference and kernels in the meta-learning framework. Learning kernels with random Fourier features for few-shot learning allows us to leverage the universal approximation property of kernels to capture shared knowledge in related tasks, and meanwhile it enables to learn adaptive basis functions to quickly and efficiently adapt to new tasks. Learning adaptive kernels with data-driven random features can be naturally cast into variational inference that approximates probability density through optimization, where the posterior over the random basis function is the spectral distribution of translation-invarian kernel.
The inference of the posterior is conducted in the context of tasks to exploring their dependency for capturing shared knowledge. We adopt a long short-term memory (LSTM) based inference network (Hochreiter & Schmidhuber, 1997), which establishes task context inference to capture the task dependency. Specifically, during the inference, the cell state in the L.TTM carries and accumulates the shared knowledge which is updated for each task throughout the course of learning. The remember and forget operations in the LSTM use new information to episodically refine the cell state by gaining experience from batch of tasks, which can eventually produce random features of highly representational capability for all tasks. For an individual task, the task specific information is first extracted from the support set, and then combined with the shared knowledge in the shared cell state together as the joint condition, to infer the adaptive spectral distribution of the kernels. As result, the task context inference can not only learn to extract and maintain the shared knowledge across tasks, but also leverage the task-specific knowledge achieve adaptive kernel to the current task. The inference framework of our MetaVRE is illustrated in Figure
Extensive experiments on variety of few-shot learning problems such as regression and classification demonstrate that, our MetaVRE method achieves competitive or even better performance when compared to otate-of-the-an algorithms. Due the advantages of kernels, applied to test settings with different ways and shots from those of training setting, in which the promising results again validate the effectiveness of our MetaVRE for few-shot learning.
Meta VRF can be
2

---

Under review as a conference paper at ICLR 2020
2 PROBLEM STATEMENT
In this section, we describe the setup of meta-lcarning for few-shot learning and introduce the kernel ridge regression as the base-learner, where kernels are approximated by random Fourier features.
2.1 META-LEARNING WITH KERNELS
We adopt the episodic training strategy (Ravi & Larochelle, 2017) commonly used for few-shot classification in meta-learning which usually involves the meta- veeta-training and meta-test stages. In the meta-training stage, a meta-learner is trained to enhance the performance of a base-learner on meta-training set with batch of few-shot learning tasks, where a task is usually referred as episode (Ravi & Larochelle, 2017). In the meta-test stage, the base learner is evaluated on meta-test set with different classes of data samples from the meta-tratinng set.
For the few-shot classification problem, we sample C-way k-shot classification tasks from the mela- training set, where k is the number of labelled examples for each of the C. classes. Given the I-th task with a support learn the parameters a' of the predictor for using a standard learning algorithm with kernel trick a = A()(X), Y), where S {X, J. Here, A the base-learner and & X' + R is mapping function from A' to a dot product space H. The similarity measure k(x.x') = (x()), ((x') is usually called a kernel (Hofmann al., 2008).
St = { yxx, Ex and query set Q' {(x,, )= (8* Q 2),, we
In traditional supervised learning. the base-learner for the t-th single task usually uses a universal kernel to map the input onto a dot product space for efficient learning. Once the base-learner is trained on the support set, its performance is evaluated on the query set by the following loss function
Σ L (0()))))
(K)))
where L(-) can be any differentiable function, e.g., cross-entropy loss. In the meta-learning setting for few-shot learning. we usually consider batch of tasks. Thus, the meta-learner is trained by optimizing the following objective function w.r.f. the empirical loss on T tasks
ΣΣ
L (fai (x)). with a' = A ( (X),),, (E
E
22
where J² is the feature mapping function which can be obtained by learning task-specific kernel k' for each task with data-driven ramdom Fourier features.
In this work, we employ kernel ridge regression (KRR), which has an efficient closed-form solution, as the base-lcarner A for few-shot learning. The kernel value in the Gram matrix K E Ck can be computed as k(x, x') = for a single task can be obtained by solving the following objective w.r.t. the support set of this task,
where "T" is the transpose operation. The base-learner A
A arg min Tr[(Y - aY)(Y - aK) I I T.
(3)
This produces closed-form solution a (AI+ K))+K)-a ())KKK-l- ly. The learned predictor is then applied the query set for prediction of the query set X:
ý= fa = aK,,
(4)
where バ = ((X)((X) E RC is with each element as k(x,, x) between the samples from the support and query sets. Note that we also treat À in Eq. (3) as a trainable parameter by leveraging the meta-learning setting, and all these parameters are learned by the meta-leamer.
In order to obtain task- speciic kernels, we propose to learn kernels with random Fourier features, which not only allows us to obtain task-adaptive kernels but also enables to capture shared knowledge of different tasks by exploring their dependency.
2.2 RANDOM FOURIER FEATURES
Random Fourier features (RFFs) were proposed to construct explicit random feature maps using the Monte Carlo approximation the Fourier representation (Rahimi & Recht, 2007), which is derived from Bochner's 's theorem (Rudin, 1962).
3

---

Under review as a conference paper at ICLR 2020
Theorem (Bochner's theorem) (Rudin, 1962) A continuous, real valued, symmetric and shift- invariant function k(x, x') = k(x -x)) on IR is a posirive definite kernel if and only if is the Fourier transform p(w) of positive finite measure such that
k(x,x'
ww (x=xxxxx))) dp(w) (x))(( where (x = eow" ww
(5)
It is guaranteed that Sw (w) (x)* is an unbiased estimation of k(x, with sufficient RFF bases {w} drawn from p(w) (Rahimi & Recht, 2007).
For predefined kernel, e.g., radius basis function (RBF), we sample from its spectral distribution using the Monte Carlo method, and obtain the explicit feature map:
(x)
[cos(w] x + b1), cos(w + bD)..
where fwi.-.. wD) } are the random bases sampled from p(w), and [b,,..., .pp are D biases sampled from uniform distribution with a range [0, 2x.. Finally, the kernel value k(x, x') = 2(x)x(x')) in K is computed as the dot product of their random feature maps with the same bases.
Learning adaptive kernel with data-driven random Fourier features is essential to find the posterior distribution and the specific spectral distribution of kernels. In the following section, we introduce our meta variational random features (MetaVRF), in which random Fourier bases are treated as latent variables inferred from the support set in the meta-learning setting.
3
META VARIATIONAL RANDOM FEATURES
3.1 META EVIDENCE LOWER BOUND
From the probabilistic perspective of view, the goal of few-shot learning is to maximize the conditional predictive log- likelihood of samples from the query set Q. We treat the random Fourier base w of the kernel latent variable:
max Σ
logp(y|x, max P
Σ log S,ylx,S,))lla Sydw.
(xY)ER
(x.y)EQ
In order infer the posterior p(wly, x, S) over w, which is generally intractable, resort using a variational distribution qo (w(w)S) to approximate this posterior, where the base is conditioned on the support set S by leveraging meta-learning We can obtain the variational distribution by minimizing the Kullback-Leibler (KI.) divergence
DKL (alllslw(S))lllc (lllwll,, x, S)]-
By applying the Bayes" rule to the posterior plwly, x, S), we can derive the meta ELBO as logply'x, N
Σ Bgs( log x, (,)x,S,w)) Dru.oo(ulS))l((w>> S)] .MetaELBO (xy)EQ
The first term of meta ELBO is the predictive log-likelihood conditioned the observation x, S and the inferred RFF bases w. Maximizing it enables us to make an accurate prediction for the query set by utilizing the inferred bases from the support set. The second term in our meta ELBO minimizes the discrepancy between the meta variational distribution qu (W(()) and the meta prior p S), which encourages samples from support and query sets to share the same random Fourier bases. The full derivation of the meta ELBO is provided in the Appendix B.
We now obtain the objective by maximizing the meta ELBO with respect batch of tasks:
C= Σ
Σ Ess log p(y|x, S w') DKL (k'l" (lp("""))lpp"""))) |x, [x, (Ip(w"(x, (xy)Ee
(10)
where St is the support set of the t-th task associated with its specific bases [w/]2 Directly optimizing the above objective does not take into count the task dependency. We introduce task context inference by making the posterior conditioned on both the support set of the current task and the bases from previous tasks.
4

---

Under review as a conference paper at ICLR 2020
t-th task
S'
y
X
wt-1
wt
Q
Figure 2: Illustration of Met in a directed graphical model. (x, y) is test sample in the query set Q The base w' is inferred by conditioning on both the base w from the previous task and the support set St of the current task.
3.2 TASK CONTEXT INFERENCE
To leverage the knowledge shared across tasks, we propose task context inference for random feature bases. Specifically, the bases {wg)2= of the t-th task should rely on the base in the previous - 1 task, denoted as [ Jd=1- The directed graphical model with related variables is shown in Figure 2. To compute the probability of w' conditioned on w'-1, we replace the previous variational posterior golw (o( with 4s (w|S² w. Therefore, the objective of the task context inference is:
T
C
1 T
Σ (X.Y)EQ'
(o log ply|x,S' S','') - DKLlas(w'le "'' (')
E=1
(11)
Note that the variational approximate posterior q6 diagonal covariance. Given the support set as input, the mean w/ and standard deviation wa are output from the LSTM inference network 6(-). To enable the back- procpproppgation of the LSTM inference
Row ( is a multivariate Gaussian with
network with the sampling operation during training, we leverage the reparametrization trick (Kingma & Welling, 2013) as
(1)) W + Wa O e!) with ((0,)..
(12)
We use a permutation invarrian instance pooling layer to aggregate the support set St of a group of examples, which essentially takes the average over the feature vectors of samples in the support set into a single vector, as in (Zaheer et al., 2017). In practice, the feature representation e is extracted for each image x by shared convolutional network (.-),i.e. = (x).. The aggregation of samples in the support set is denoted as é.
We propose an LSTM-type inference network inspired by the fact that the long-term memory can be carried and refined in cell states during its update (Gers & Schmidhuber, 2000). Specifically, we design simplified variant of LSTM with two gates remove the effect of the short-term memory. The common knowledge shared by tasks is stored in the cell state and updated with new information in each episode. During inference, LSTM can remove trivial information and add knowledge with highly representational ability to the cell state. Once we have updated the cell state, the shared knowledge stored within it is combined with the task-specific information from the input support set to infer the spectral distribution. The update steps in the LSTM network are
f sigmoid(W; (W, |o - - + by);
2" = sigmoid(W, [E*, - c '1+ b,): é = tanh(W, [e'] + ba); c' = f c--++ è w/ = tanh(W, le' e¹] b.).
(13)
We is computed in the same way. After training, the final state c is the shared knowledge from the meta-training set, which is directly used in the meta-testing set. In the meta-testing stage, use the LSTM inference network @ to obtain w,, and w, from S and wT. Then, a set of bases are sampled from qa (w( (w])T.TS)) to construct the random features in the kernels. Finally, the base-learner is optimized to obtain the predictor for testing.
5

---

Under review as a conference paper at ICLR 2020
4 RELATED WORK
Meta-learning, or learning to learn, endues machine learning models the ability to improve their performance with for a number of training tasks. It has received increasing research interest with breakthroughs in many directions (Finn et al., 2017; Rusu et al., 2019; Gordon et al., 2019; Aravind Ra- jeswaran, 2019). Metric-basee methods cast few-shot learning as a matching problem (Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018; Satorras & Estrach, 2018; Allen et al., 2019) by learning shared similarity function for diverse tasks. Graphical neural network (GNN) based model generalizes the matching methods by learning the message propagation from the support set and transfer it to the query set (Garcia & Bruna, 2018). Gradient-based methods (c.g.. MAML (Finn et al., 2017)) learn an appropriate initialization of model parameters and adapt it on new tasks with only a few gradient steps (Finn & Levine, 2018; Zintgraf et al., 2019; Rusu et al., 2019). Some other interesting works propose to directly learn the gradient optimization process of networks using RNNs (Ravi Larochelle, 2017; Andrychowicz et al., 2016). Memory-based methods learn to leverage an external memory module to store and leverage key knowledge for quick adaptation (Santoro et al., 2016; Ramalho & Garnelo, 2019). Bayesian meta-learning methods (Edwards & Storkey, 2017; Finn et al., 2018; Gordon ct al., 2019) usually rely on hierarchical Bayesian models to learn the shared statistical information among different tasks and reason about the uncertainty over models. Differentiable solution methods (Liu et al., 2019; Bertinetto et al., 2019) learn a universal feature embedding and obtain a task-speciff learner with closed-form solution.
While those meta- veta-learning algorithms have achieved great success in solving few-shot learning tasks, it remains an open challenge explore shared knowledge across prior tasks. In this work, we introduce kernel learning with data-driven random Fourier features to explore task dependency to extract the shared knowledge.
Kernel learning with random Fourier feattures is a versatile and powerful tool in the machine learning and statistics communities (Bishop, 2006; Hofmann et al., 2008; Shervashidze et al., 2011). 1). Pioneering works in this category (Bach et al., 2004; Gönen & Alpaydın, 2011; Duvenaud et al., learning manner. As an innovative feature
2013) learn to combine predefined kernels in a map, several studies have focused on random Fourier features (RFFs) (Rahimi & Recht, 2007), with recent works (Wilson & Adams, 2013) learning kernels the frequency domain by modeling the spectral distribution as mixture of Gaussians and computing its optimal linear combination. Instead of modeling the spectral distribution with explicit density functions, other works focus on optimizing the random base sampling strategy (Yang et al., 2015; Sinha & Duchi, 2016).
To the best our knowledge, our work is the first to extend kernel learning with random features to the meta- meta-learning framework for few-shot learning. We propose training meta-learner to infer the spectral distribution of random features from the support set. The task-specific kernel can be leveraged by the base-learner for the supervised learning. Compared with RFFs, our learned Meta VRF achieves superior performance on few-shot learning tasks with a low sampling rate of bases.
EXPERIMENTS
In this section, we evaluate our MetaVRF on several few-shot learning problems for both regression and classification. We conduct classification experiments on three commonly-used benchmark datasets, i.e., Omniglot (Lake et al., 2015), minilmageNNt (Vinyals et al., 2016) and CIFAR-FS (Krizhevsky et al., 2009). More details about the three datasets are provided in the Appendix C. We also conduct deeper analysis to validate the effectiveness of our MetaVRF.
5.1 FEW-SHOT REGRESSION
We begin with k-shoo regression problem, and compare our Meta VRF with MAML (Finn el., 2017) as the baseline method. We follow the MAML work (Finn et al., 2017) to fit a target sine function y A sin (wr + b). with only few annotated samples. A E [0.1, denote the amplitude, frequency, and phase, respectively, which follow a uniform distribution within the corresponding interval. The goal is estimate the target sine function given only n randomly sampled date points. In our experiments, we consider input in the range of x E [-3, 5j. and conduct three tests under the conditions of k 3,5, 10. For fair comparison, we compute the feature
E [0.8, 1.2), and b E [0, m]

---

Under review as a conference paper at ICLR 2020
3-shot
5-shot
1 10-shot
5
2
5
0.849
0.096
0.294
0.304
0.021
0.024
0.240
0.002
0.003
-5
22
-5
-4
-
-4
-2
-2
MAML
MetaVRF (wlo LSTM)
MetaVRF
Ground Truth
Support Samples
Figure 3: Performance comparison for few-shot regression. three methods and ground truth.
Values are in term of MSE between
embedding using small multi-layer perceptron (MLP) with two hidden layers of size 40, following the same settings as used in MAML. We have also experimented with our MetaVRF without using LSTM (MetaVRE w/o LSTM) for the inference.
The comparison results are plotted in Figure 3. The proposed MetaVRF can fit the function accurately with even three shots. With the increase of shots, our MetaVRF can perform better, almost entirely fitting the target function with 10 shots. It clearly shows that the performance can be improved by adopting the LSTM inference, which verifies the benefits of exploring task dependency, In addition, we can see that our Meta VRF performs much better than MAML which is representative meta- learning algorithm for all three settings with different numbers of shots. More few-shot regression results are provided in the Appendix E.
5.2 FEW-SHOT CLASSIFICATION
For the classification task, we compare the proposed MetaVRF with the baseline method, random Fourier features (RFFs) and other state-of-the models. In particular, RFFs are the random feature computed from the universal kernel without adaptation.
Experimental settings. Image features are extracted via a shallow convolutional neural network. We employ the same architecture used in (Gordon et al., 2019). We do not use any fully connected layer for these CNNs. The dimension of all feature vectors is 256. The inference network for RFFs is replaced with feed-forward network with couple of dense blocks. The key hyperparameter of the number bases D in Eq. (6), is set to = 512 for Meta VRF in all experiments, while we set D 2048 for RFFs. The sampling rate of MetaVRE is much lower than in previous studies on RFFs, in which D is usually set to be 5 to 10 times the dimension of the input features (Yu et al., 2016; Rahimi & Recht, 2007). All details provided in the Appendix D.
Quantitive analysis. We evaluate our Meta on three few-shot classification benchmarks using the same meta-testing protocol (Gordon al., 2019), and present the results with 95% confidence intervals. All reported results are for models trained from scratch for few-shot learning. Table and 2 report the performance of MetaVRE compared to the current state of the arts with shallow CNN architectures. Our Meta VRF method achieves the new tifte-of-th-aane results on most of the challenging datasets, (e.g., 54.3% for 5-way 1-shot on minilmageNett, which is 1% improvement over the second-best method). On the Omniglot dataset, previous approaches did not specify the splits for training, validation, and testing, which may result in unfair comparisons. Our Meta VRF method falls within the error bars of the state-of-the -te--ar models on all experiments under 5-way 1-shot, 5-way 5-shot, 20-way 1-shot and 20-way 5-shot settings.
Since the feature extraction network used in our Meta is different from that used in previous works, in Table 2, we show the results of methods that have the same training procedures and conventional CNN architectures ours. In Vinyals et al. (2016); Snell et al. (2017); Ravi & Larochelle (2017); Finn et al. (2017); Sung et al. (2018), they usually use 64 filters in each convolutional layer. Ravi & Larochelle (2017); Finn al. (2017) set the number of filters 32 for minilmageNetto avoid
7

---

Under review as a conference paper at ICLR 2020
Table 1: Classification accuracies (%) on mind and CIFAR-FS.
minilmageNet,, 5-way 5-shot
CIFAR-FS, 5-way
Method
1-shot
1-shot
5-shot
MATCHING NET (Vinyals et al., 2016)
44.2
57
MAML (Finn et al., 2017)
48.7=1.8
63.1
18.9+199
71.5
MAML (64C)
46.7+1.7
61.1=0.1
18.9
71.5+
META-LSTM (Ravi & Larochelle, 2017)
43.4+10.8
60.6+0.77
PROTO NET (Snell et al., 2017)
47.4110.6
65.4100.5
55.510.77
72.0+10.6
RELATION NET (Sung al., 2018)
50.4100.8
65.3400.7
55.0+1.0
69.3
SNAIL (32C) by (Bertinetto al., 2019)
45.1
55.2
GNN (Garcia Bruna, 2018)
50.3
66.4
61.9
75.3
PLATIPUS (Finn al., 2018)
50.1
VERSA (Gordon et al., 2019)
53.3
67.310..
6..5
75.1
R2-D2" (Bertinetto et at., 2019)
50.5+0.2
65.4+0.2
62.3+0.2
77.4+002
R2-D2 (Devos ct al., 2019)
51.7+
63.3+0.9
60.2+
70.9+0.9
CAVIA (Zintgraf et al., 2019)
51.8++077
65.6+0.6
IMAML (Aravind Rajeswaran, 2019)
49.3
RFFs (2048d)
54.0=#19
65.44099
61.3
75.1
RF (w/o LSTM, 512d)
52.9=1.8
67.3
6..3
75.9 +0.9
METAVRF (512d)
5..3
68.0.10.
62.9
76.3
'training with 20 ways, test on 5 ways.
Table 2: Classification accuracies (%) on Omniglot.
Omniglot, 5-way
Omniglot, 20-way
Method
1-shot
5-shot
1-shot
5-shot
SIAMESE NET (Koch, 2015)
96.7
98.4
88
96.5
MATCHING NET (Vinyals et al., 2016)
98.1
93.8
MAML (Finn et al., 2017)
98.7110.4
99.9.10.1
95.8110.3
98.9-10.2
PROTO NET (Snell et al., 2017)
98.5=0.2
99.5#0.1
95,3+002
98.7=0.1
SNAIL (Mishra et al., 2018)
99.1=0.2
99.8 =0.1
97.6 #0.3
99.4
GNN (Garcia & Bruna, 2018)
99.2
99.7
97.4
99.0
VERSA (Gordon et al., 2019)
99.7-0.2
99.8+
97.7+-0.3
98.8+0022
R2-D2 (Bertinetto ct al., 2019)
98.6
99.7
94.7
98.9
IMP (Allen al., 2019)
98.4-0.3
99.5+
95.0+
98.6+0011
RFFs (2048d)
99.5=0.2
99.5+0.2
97.2+00.3
98.3+-022
METAVRF (w/o LSTM, 512d)
99.6=0.2
99.6+t0.2
97.0+
98.4=0.2
METAVRF (512d)
99.8
99.8++0.1
97.5+
99.0-0.2
overfitting. For a fair comparison, we increase the number of filters in MAML from 32 to 64 but obtains the inferior performance, indicating that MAML might be heavily prone to overfitting the model size increases. In original SNAIL,, they use the deep ResNet- embedding. We present the result of SNAIL reported in Bertinetto et al. (2019), where the result is obtained using similar shallow networks as ours. The graph neural network (GNN) (Garcia & Bruna, 2018) leverages large embedding network with [64, 96, 128, 256] filters and one fully connected layer. For R2- D2 (Bertinetto et al., 2019), the numbers of filters are [96, 192, 384, 512] and we present the results of two of its variants with 64 channels. Different training and testing conditions may also affect the performance of the compared methods. We maintain consistent conditions to be consistent in our experiments but present the results of R2-D2 (Bertinetto et al., 2019) trained in 20 ways for 5-way tasks. The performance of the reproduced R2-D2 (Devos al., 2019) is heavily impaired when this strategy not employed.
In addition, recent interesting works of (Rusu et al., 2019; Gidaris & Komodakis, 2019; Li et al., 2019; Qiao et al., 2018; Gidaris Komodakis, 2018) are not included for comparison because they rely on pre- tre-trained embeddings we adopt a relatively shallow convolutional architecture for feature extraction to demonstrate the effectiveness of the proposed Meta VRF instead of relying huge, powerful convolutional networks.
sarge-scale deep architectures, e.g., ResNet (He et al., 2016b). In contrast,
8

---

Under review as a conference paper at ICLR 2020
minilmageNet
CIFAR-FS
70
80
70
Accuracy (%
Aecuracy (%)
60
50
40
Me
40
MetaVRFF
30
30
500
1000
1500
2000
500
1000
1500
2000
(a) Number of Base
(b) Number of Base
Figure 4: Performance with different numbers 1) of bases.
Accuracy (%)
Accuracy (%)
99
95
98
90
20-way, 5-shot
20-way, 5-shot
20-way, 1-shot
97
20-way, 1-shot
-way, 5-shot
-way 5-shot
-way, 1-shot
-way, 1-shot
60
100
(a) Number of Way
(b) Number of Shot
Figure 5: Performance on test tasks with varied ways and shots on Omniglot.
5.3 FURTHER ANALYSIS
Efficiency. Figure 4, we demonstrate the effectiveness on bases our MetaVRR with a low sampling rate by comparing it with conventional random Fourier features (RFFs). We present the mean testing accuracy of fully trained models w.t.. RFFs and our MetaVRF as the number of bases in Eq. (6) varies. Given the same number of sampled bases, Meta method consistently achieves much higher performance than RFFs, e.g., as shown for the 5-way, 5-shot condition in Figure 4. The results verify the effectiveness of our MetaVRF in learning adaptive kernels and exploring the tasks dependencies.
Versatility. Our MetaVRF also shows promising performance when the number of ways C and shots between training and testing are inconsistent. In Figure 5, we plot the testing accuracy of the trained models on one particular C-waay-k-hhot task, with varied C and in the testing stage. The results demonstrate that the trained model can maintain reasonable discriminability for a high number of testing ways. particular, the model trained for the 10--way5--sho task can retain high accuracy of 94% when tested under the 100-way condition, as shown in Figure 5(a). The results also indicate that our Me exhibits considerable robustness and flexibility to great variety testing conditions.
CONCLUSIONS
In this paper, we explore kernel approximation based on random Fourier features in the meta-learning framework for few-shot learning. We propose the novel meta variational random features (Meta VRF), which leverages variational inference and meta-learning to infer the spectral distribution random Fourier features in data-driven way. MetaVRF representational power and a relatively low spectral sampling rate by using an LSTM inference network to explore the shared knowledge. In practice, LSTM inference network demonstrates the great ability to quickly adapt to specific tasks for improved performance. Experiments on few-shot learning tasks for several benchmark datasets demonstrate the state-of-the-art performance over previous methods and the importance of our contribution.
generate random Fourier features of high

---

Under review as a conference paper at ICLR 2020
REFERENCES
Kelsey R Allen, Evan Shelhamer, Hanul Shin, and Joshua B Tenenbaum. Infinite mixture prototypes for few-shot learning. 2019. 6,
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In NIPS, 2016. 6
Sham Kakade Sergey Levine Aravind Rajeswaran, Chelsea Finn. Meta-learning with implicit gradients. arXiv preprini arXiv: 1909.099000 2019. 6, 8
Haim Avron, Vikas Sindhwani, Jiyan Yang. and Michael W Mahoney. Quasi-monte carlo feature maps for shift-invariant kernels. The Journal of Machine Learning Research, 17(1):4096-41) 2016. 1
Francis R Bach, Gert RG Lanckriet, and Michael I Jordan. Multiple kernel learning, conic duality, and the algorithm. In ICMI., pp. 6. ACM, 2004. 6
Matthias Bauer, Mateo Rojas-Carulla, Jakub Bartlomiej Šwiętkowski, Bernhard Schölkopf, and Richard E Turner. Discriminative k-shot learning using probabilistic models. arXiv preprint arXiv:1706.00326. 2017. 13
Luca Bertinetto, Joao Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. 2019. 1, 6, 8. 14
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006. 6
Luigi Carratino, Alessandro Rudi, and Lorenzo Rosasco. Learning with sgd and random features. In NeurlPS, pp. 10192-10203, 2018.
Wei-Cheng Chang, Chun-Liang Li, Yiming Yang, and Barnabas Poczos. Data-driven random fourier features using stein effect. arXiv preprint arXiv: 1xXiv:7755008255 2017. 1
J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, and L.. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR, 2009.
Arouut Devos, Sylvain Chatel, and Matthias Grossglauser. Reproducing meta-learning with differen-
tiable closed-form solvers. 2019. 8
David Duvenaud, James Robert Lloyd, Roger Grosse, Joshua B Tenenbaum, and Zoubin Ghahramani. Structure discovery in nonparametric regression through compositional kernel search. arXiv preprint 1302. 2013. 6
Harrison Edwards and Amos Storkey. Towards neural statistician. ICLR, 2017. 6
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594-61) 611, 2006. 1
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. In ICLR, 2018. 6
Chelsea Finn, Pieter Abbecl, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. ICML, pp. 1126-1135. JMLR. org, 2017. 1, 6, 7,
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In NeurIPS, pp. 9516-9527, 2018. 6,
Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. 2018. 6, 8
Thomas Gärtner, Peter A Flach, Adam Kowalczyk, and Alexander Smola. Multi-instance kemels. In ICML, volume 2, pp. 7, 2002. 1
Felix A Gers and Jürgen Schmidhuber. Recurrent nets that time and count. In Proceedings of the TEE--NNS-ENNN International Joint Conference on Neural Networks, volume 3, pp. 189-194 IEEE, 2000.
10

---

Under review as a conference paper at ICLR 2020
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In CVPR, pp. 4367-4375, 2018. 8, 13
Spyros Gidaris and Nikos Komodakis. Generating classification weights with gnn denoising autoen- coders for few-shot learning. In CVPR, pp. 21-30, 2019.
Mehmet Gönen and Ethem Alpaydın. Multiple kernel learning algorithms. Journal of machine learning research, 12(Jul):2211-222 2011.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Meta- learning probabilistic inference for prediction. 2019, 6,7, 7,8
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, PP. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016b. 8
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1997. 2
Thomas Hofmann, Bernhard Schölkopf, and Alexander Smola. Kernel methods in machine learning. The annals of statistics, pp. 1171-1220, 2008. 3, 6
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1:Xiv:1412.6960 2014. 14
Diederik P Kingma and Max Welling. arXiv:1312.61 14, 2013. 5
Auto-encoding variational bayes.
arXiv preprint
Gregory Koch. Siamese neural networks for one-shot image recognition. In ICML Workshop, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In NIPS, pp. 1097-1105, 2012.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. 6, 14
Brenden Lake, Ruslan Salakhutdinov, and Joshua Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 133--1338,,,, 2015. 1, 6, 14
Hongyang Li, David Figen, Samuel Dodge, Matthew Zeiler, and Xiaogang Wang. Finding task- relevant features for few-shot learning by category traversal. In CVPR, pp. 1-10, 2019. 8
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint 1707.09777. 2017. 13
Yanbin Liu, Juho Lee, Minseop Park, Sachoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. ICLR, 2019,
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. 2018. 8, 13
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. arXiv preprint arXiv: 1112::17122099000 2017.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In NeurIPS, pp. 721-731, 2018. 13
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan Yuille. Few-shot image recognition by predicting parameters from activations. In CVPR, pp. 7238, 2018. 8, 13

---

Under review as a conference paper at ICLR 2020
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In NIPS, pp. 1177-1184, 2007. 1,3,4, 7
Tiago Ramalho and Marta Garnelo. Adaptive posterior learning: few-shot learning with a surprise- based memory module. ICLR, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. 2017. 1, 3, 6, 14
Walter Rudin. Fourier analysis on groups, volume 121967. Wiley Online Library, 1962. 3,4
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211 1-252, 2015. 14
Andrei Rusu, Dushyant Rao, Jakub Sygnowski. Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. 2019. 6, 8, 13
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmented neural networks. In ICML, pp. 1842-1850, 2016.
Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks. In ICLR, 2018.
Schmidhuber. Learning control west- memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139, 1992. 1
Nino Shervashidze, Pascal Schweitzer, Erik Jan van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Reseurch, 12(Sep): 2539-2561, 2011. 6
Aman Sinha and John C Duchi. Learning kernels with random features. In NIPS, pp. 1298 1298-1306.
2016. 1,6 6
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NIPS, pp. 4077 4087, 2017. 6, 7
Flood Sung, Yongxin Yang, I. Zhang, Tao Xiang, Philip HS Torr, and Timothy Hospedales. Learning to compare: Relation network for few-shot learning. In CVPR, pp. 1199-1208, 2018. 6, 7,
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NIPS, pp. 3630-3638, 2016. 6, 7, 8, 14
Andrew Wilson and Ryan Adams. Gaussian process kernels for pattern discovery and extrapolation. In ICML, pp. 1067-1075, 2013. 6
Zichao Yang, Andrew Wilson, Alex Smola, and Le Song. A la carte-learning fast kernls.. In Artificial Intelligence and Statistics, pp. 1098-1106, 2015.
Felix Xinnan X Yu, Ananda Theertha Suresh, Krzysztof M Choromanski, Daniel N Holtmann-Rica and Sanjiv Kumar. Orthogonal random features. In NIPS, pp. 1975-1983, 2016. 7
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint S.t::6605001444 2016. 13
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander Smola. Deep sets. NIPS, pp. 3391-3401, 2017.
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning, In ICML, pp. 7693-7702, 2019. 6,
12

---

Under review as a conference paper at ICLR 2020
A
RESULTS OF DEEP ARCHITECTURES
Table 3: Classification accuracies (%) on minilmageNet with 5 way using deep architectures
Method
1-shot
5-shot
META-SGD (Li al., 2017)
54.2440.033
70.86+00.04
(GIDARIS & KOMODAKIS, 2018)
56.20=
73.00=0.64
(BAUER ET AL.,, 2017)
56.30:10.40
73.90+10.30
(MUNKHDALAT ET AT.,, 2017)
57.10+0..70
70.04+0.63
(QLAO ET AL., 2018)
59.60:10.41
73.54:0.19
LEO (Rusu al., 2019)
61.76:10.08
77.59=0.12
SNAIL (Mishra et al., 2018)
55.71++0.99
68.88+10.22
TADAM (Oreshkin et al., 2018)
58.50
76.70+10.00
METAVRF (ours)
63.40-L0.06
77.85+00.28
We evaluate method by using pre-trained embeddings and compare with latest methods with deep embedding architectures. We use the pre-trained embeddings from a 28-layer Wide Residual Network (WRN-28- 10) (Zagoruyko & Komodakis, 2016), in similar fashion to (Rusu al., 2019; Bauer et al., 2017; Qiao et al., 2018). Specifically, the WRN-28- 10 was trained to classify images which are all from the meta-training set into the corresponding categories. We select activations in the 21th layer, with average pooling over spatial dimensions, as feature embeddings. The dimension of pre-trained embeddings is 640. As shown in Table. 3, our Meta achieves the state-of-the-an preformance on both cases of 1-shot and 5-shot.
B DERIVATIONS OF META ELBO
For singe task, we begin with maximizing log-likelihood of the conditional distribution p(y x, S) derive the ELBO of Meta By leveraging Jensen's inequality, we have the following steps
log plyl,, S) = log
p(yix, S, w/w)p(//xxx.ccmmmmmoo S)
(14)
/
p(y|x, )p(w)> S) go(w|s) dw (S) go (ww))
log
(15)
/ log
"pllxx S, w) xx S)
(16)
Egg(w)s) log plyyx,S, S.S. w/)) DKL fos (w/)
(17)
Meta ELBO
We can also derive Meta ELBO from the KL. divergence between the posterior p(wly, x, S) and its variational posterior (o(w)S) following
DKL
x, S)] Eq (w]S) [log (w]))
(18)
log p(wpp(wwyp.. x,
(19)
w (w(()) log 90 (w|S) log p x,
S)p(w[xx
(20)
p(y|x,S)
-logply/x S) + Bnu()(s) (w (8) log w x, 8) logp(w|x,
(21)
Bns()))) (logp(y/es (wo, x, S)] +
(22)
0.
(23)
Therefore, the lower bound of the evidence p(y(x) is at the RHS of lgy((y|x,S) 2 Ega(w(s) log (p(y xx, S, w')] DRI. las (w] S))
(24)
which is consistent with Eq. (17).
13

---

Under review as a conference paper at ICLR 2020
Table 4: The fully connected network v)() used for regression.
Output size
Layers
Input training samples
40
fully connected, RELU
40
fully connected, RELU
Table 5: The CNN architecture (+)) for Omniglot.
Output size
Layers
28x28x1
Input images
14x14x64
conv2d (3x3, stride=l, SAME, RELU), dropout 0.9, pool (2x2, stride=2, SAME)
7x7x64
conv2d (3x3, stride=1. SAME, RELU), dropout 0.9, pool (2x2. stride=2, SAME
4x4x64
conv2d (3x3, stride=1, SAME, RELU), dropout 0.9, pool (2x2, stride=2, SAME)
2x2x64
conv2d (3x3, stride=1, SAME, RELU), dropout 0.9, pool (2x2, stride=2, SAME)
256
flatten
C FEW-SHOT CLASSIFICATION DATASETS
Omniglot (Lake et al., 2015) is a benchmark of few-shot learning that contain 1623 handwritten characters (each with 20 examples). All characters are grouped in 50 alphabets. For fair comparison against the state of the arts, we follow the same data split and pre-processing used in Vinyals et al. (2016). The training, validation, and testing are composed of random split of [1100, 423). The dataset is augmented with rotations of 90 degrees, which results in 4000 classes for training, 400 for validation, and 1292 for testing. The number examples fixed as 20. images are resized to 28. For C-way, k-shot task at training time, we randomly sample C' classess from the 4000 classes. Once we have C' classes, (k + 15) examples of each are sampled. Thus, there are C'k examples in the support set and Cx 15 examples in the query set. The same sampling strategy is also used validation and testing.
minilmageNNet (Vinyals al., 2016) is challenging dataset constructed from ImageNet (Rus- sakovsky et al., 2015), which comprises a total 100 different classes (each with 600 instances). All these images have been downsampled 84x84. We use the splits of Ravi & Larochelle (2017), where there are [64, 16, 20] classes for training, validation and testing. We use the same episodic manner as Omniglot for sampling.
CIFAR-FS (CIFARI00 few-shots) (Bertinetto et al., 2019) is adapted from the CIFAR-I00 dataset (Krizhevsky et al., 2009) for few-shot learning. Recall that in the image classification benchmark CIFAR- -10, there are 100 classes grouped into 20 superclasses (each with 600 instances). CIFAR-FS use the same split criteria (64, 16, 20) with which minilmageNet has been generated. The resolution of all images is x2x32.
D MORE EXPERIMENTAL DETAILS
We train all models using the Adam optimizer (Kingma & Ba, 2014) with learning rate of 0.0001. The other training setting and network architecture for regression and classification on three datasets different following.
D.1 FEATURE EMBEDDING NETWORKS
Regression, The fully connected architecture for regression tasks is shown in Table 4. We train all three models (3-shot, 5-shot, 10-shot) over total of 20, 000 iterations, with 6 episodes per iteration.
Classification. The CNN architectures for Omniglot, CIFAR-FS, and minilmageNee are shown in Table 5, 6, and 7.
14

---

Under review as a conference paper at ICLR 2020
Table The CNN architecture ++)) for CIFAR-FS
Output size
Layers
x3
Input images
16x16x64
con22d (3x3, stride=1, SAME, RELU), dropout 0.5, pool (2x2, stride=2, SAME
8x8x64
conv2d (3x3, stride=1, SAME,, RELU), dropout 0.5, pool (2x2, stride=2, SAME
4x4x64
con22d (3x3, stride=1, SAME, RELU), dropout 0.5, pool (2x2, stride=2, SAME)
2x2x64
conv2d (3x3, S RELU), dropout 0.5, pool (2x2, stride=2, SAME)
256
flatten
Table 7: The CNN architecture (6()) for minilmageNet
Output size
Layers
84x84x3
Input images
42x42x64
conv2d (3x3, stride=1, SAME, RELU), dropout 0.5, pool (2x2, stride=2, SAME)
21x21x64
conv2d (3x3, stride=1, SAME, RELU), dropout 0.5, pool (2x2, stride=2, SAME)
10x10x64
conv2d (3x3, stride=1, SAME, RELU), dropout 0.5, pool (2x2, stride=2, SAME)
5x5x64
conv2d (3x3, stride=1, SAME, RELU), dropout 0.5, pool (2x2, stride=2, SAME)
2x2x64
conv2d (3x3, stride=1, SAME, RELU), dropout 0.5, pool (2x2, stride=2. SAME)
256
flatten
D.2 INFERENCE NETWORKS
The architecture of the inference network for the regression task is in Table 8, For few-shot classifica- tion tasks, all models share the same architecture, as Table 9.
D.3 PRIOR NETWORKS
The architecture of the prior network for the regression task is in Table 10. For few-shot classification tasks, all models share the same architecture, as in Table 11.
D.4 OPTIMIZATION SETTINGS
The number of training iterations and the batch size (episodes per iteration) are listed in Table 12.
Table 8: The inference network o(-) used for regression.
Output size
Layers
40
Input samples feature fully connected, ELU fully connected, ELU
40
40
40
LSTM cell, Tanh to par, log 0
15

---

Under review as a conference paper at ICLR 2020
Table 9: The inference network o(-) used for Omniglot, minilmageNet, CIFAR-FS
Output size
Layers
k x 256
Input feature
256
instance pooling
256
fully connected, ELU fully connected, ELU fully connected, ELU
256
256
256
LSTM cell, tanh Pws log o
Table 10: The prior network used for regression.
Output size
Layers
80
The concatenation of query feature and aggregated support features fully connected, ELU
40
40
fully connected, ELU
40
fully connected to Mas log o
Table 11: The prior network used for Omniglot, minilmageNet, CIFAR-FS
Output size
Layers
512
The concatenation of query feature and aggregated support features instance pooling
256
256
fully connected, ELU fully connected, ELU fully connected to Man log o
256
256
Table 12: Iteration and batch size for all datasets.
Dataset
Iter.
Batch size
Regression
20, 000
25
Omniglot
100, 000
6
CIFAR-FS
200, 000
8
minilmageNet
100, 000
16

---

Under review as a conference paper at ICLR 2020
E
FEW-SHOT REGRESSION
3-shot
5-shot
10-shot
4
0.5
0.0
0-
-0.5
-2
-2
0,500
0.393
0.06
-1.0
0.033
0.01
2.5
3-shot
5-shot
10-shot
1
0-
0-
-1
0.41
0.200
0.04
-2
-2.5
-2.5
-2.5
3-shot
5-shot
10-shot
2-
-2
2
-
3.72
0.553
0.353
-4
0:516
0.52
0.303
0.092
0,025
0.005
-5.0
-2.5
0.0
2,5
5,0
-5,0
-2.5
0.0
2.5
5.0
-5.0
-2.5
0,0
2.5
5.0
3-shot
5-shot
10-shot
0.5
0.0
-2-
-1.0
0.025
2.5
-2.5
3-shot
5-shot
10-shot
1.0
1.4
0-
-055
0.005
-0.5
0,078
0.519
-1.0
0,203
-1.0
-5.0
-2.5
0.0
2.5
5.0
-2.5
2.5
-2.5
3-shot
5-shot
10-shot
1
-2
661
5.100
B805
220
5,006
5.0
2.5
0.0
2.5
5.0
5.0
2.5
0.0
2.5
5.0
5.0
2.5
0.0
2.5
5.0
MAML
MetaVRF (wlo LSTM)
MRtaVRF
Ground Truth
Support Samples
Figure 6:
More results few-shot regression.
17


Reviews:
[{"review_id": "rkx5Y4zMcr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies meta-learning problem with few-shot learning settings. The author proposes a learn each task predictive function via the form of random Fourier features, where the kernel is jointly learned from all tasks. The novel part is the parametrization of inference network using LSTM such that the random feature samples of t-th task conditional depending on all previous task 1,...,t-1, which is an interesting way of modeling kernel spectral distribution. The experiment results show improvement of the proposed methods compared to SoTA meta learning algorithms.\n\nIn general, the writing of the paper is clear, and the proposed method is interesting and novel. However, there are parts missing in the experiment setting.  I would love to increase my score if the author could address the following questions/comments:\n(1) How do you choose the meta prior distribution? It should be a basic kernel family such as RBF Gaussian or mixture of RBF?\n(2) In Table 1 and Table 2, the benefit of using LSTM only gives very marginal improvement over w/o LSTM. Are the results statistically significant? \n(3) The experiment missed the simple kernel learning baseline, such as kernel alignment [1] and its variants [2]. If using these task-independent way to do kernel learning, what’s their performance compared to you proposed method?  \n(4) When learning the RFF spectral distribution using LSTM over a sequence of tasks, does the order of task matter?  \n\n\n[1] Learning kernels with random features, NIPS 2016.\n[2] Implicit kernel learning, AISTATS 2019.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygdMZIoYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a meta-learning framework for learning adaptive kernels using a meta-learner. For representing kernels, the paper learns a variational posterior for the kernel features, by maximizing the Evidence lower Bound. Furthermore, to plug the kernel learning into the meta-learning framework, they let the variational feature posterior to condition on the current support set for adapting and to use a modified LSTM network for accumulating information. Empirically, they compare the proposed MetaVRF with multiple baselines in the standard fewshot classification benchmarks and demonstrate superior performance. They also illustrate that their adaptively-learnt Fourier feature outperforms the standard variational Fourier features.\n\nStrengths, \n1, The idea of learning kernels in meta-learning is interesting. In fact, learning a kernel is equivalent to learning a distance between objects. If a reasonable distance between objects can be learnt, using the corresponding kernel should be able to achieve superior performance even if the kernel doesn't adapt in each episode. \n2, The proposed method achieves competitive performances. In particular, Figure 5 shows how the performance changes when the test-shot and test-way are varied. It seems surprising that the MetaVRF achieves >90% accuracy for 100-way test when trained on only 5-way 5-shot.\n\nWeakness,\n1, The notations in the paper are not well presented. (a) In eq(2), the formula $alpha^t=\\Lambda(\\Phi^t(x), y)$ is not exact, cuz $\\alpha^t$ should depend on the whole support set $S^t$ while $(x,y)$ is only one instance in $S^t$.  (b) In eq(11), the variational posterior $q(w | S^t, w^{1:t-1})$ is not exact either. Because $w^{1:t-1}$ are random variables, they cannot be observed and cannot be conditioned on. Similar issues also exist in the caption of Figure 1.\n2, The paper doesn't introduce what is the likelihood $log p(y| x, S, w)$. It is unclear how the kernel regression is adopted in classification. \n3, The meta-prior $p(w| x, S)$ depends on the feature of the query point, which doesn't seem to be a common practice in variational inference. It would be beneficial if the authors could explain this and probably validate it empirically.\n4, The motivations of the modified LSTM should be clarified more. Does the paper remove h_t in LSTM for removing short-term memory ? Why $\\hat{c}_t$ only depends on $e^t$ instead of $[e^t, c^{t-1}]$ ? \n5, The paper compares with multiple competitive baselines. However, the settings for the baselines should be better presented. For example, it is strange that the numbers of SNAIL are different with the numbers in their paper. And SNAIL on Omniglot is not reported. Furthermore, another competitive method TADAM (Oreshkin et. al., 2019) should also be compared with.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}, {"review_id": "BkgAgpcVcr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on the topic of meta-learning for few-shot learning and explores kernel approximation with random fourier features for this problem. The authors propose to learn adaptive kernels by meta variational random features, and evaluate their approach on different few-shot learning tasks, comparing it against recent meta-learning algorithms. \n\nThe paper is well-motivated and well-written. On page 8, the authors mention related works that were not included for comparison because they rely on pre-trained embeddings or large-scale deep architectures. It would have been interesting to see the difference in performance.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper looks at meta learning using random Fourier features for kernel approximations. The idea is to learn adaptive kernels by inferring Fourier bases from related tasks that can be used for the new task. A key insight of the paper is to use an LSTM to share knowledge across tasks.

The paper tackles an interesting problem, and the idea to use a meta learning setting for transfer learning within a kernel setting is quite interesting. It may be worthwhile relating this work to this paper by Titsias et al. (https://arxiv.org/abs/1901.11356), which looks at a slightly different setting (continual learning with Gaussian processes, where information is shared through inducing variables).

Having read the paper, I have some comments/questions:
1. log-likelihood should be called log-marginal likelihood (wherever the ELBO shows up)
2. The derivation of the ELBO confuses me (section 3.1). First, I don't know whether this ELBO is at training time or at test time. If it was at training time, then I agree with Reviewer #1 in the sense that $p(\omega)$ should not depend on either $x$ or $\mathcal {S}$. If it is at test time, the log-likelihood term should not depend on $\mathcal{S}$ (which is the training set), because $\mathcal S$ is taken care of by $p(\omega|\mathcal S)$. However, critically, $p(\omega|\mathcal S)$ should not depend on $x$. I agree with Reviewer #1 that this part is confusing, and the authors' response has not helped me to diffuse this confusion (e.g., priors should not be conditioned on any data).
3. The tasks are indirectly represented by a set of basis functions, which are represented by $\omega^t$ for task $t$. In the paper, these tasks are then inferred using variational inference and an LSTM. It may be worthwhile relating this to the latent-variable approach by Saemundsson et al. (http://auai.org/uai2018/proceedings/papers/235.pdf) for meta learning. 
4. The expression "meta ELBO" is inappropriate. This is a simple ELBO, nothing meta about it. If we think of the tasks as latent variables (which the paper also states), this ELBO in equation (9) is a vanilla ELBO that is used in variational inference.
5. For the LSTM, does it make a difference how the tasks are ordered?
6. Experiments: Figure 3 clearly needs error bars, and MSEs need to be reported with error bars as well; 
6a) Figures 4 and 5 need error bars.
6b) Error bars should also be based on different random initializations of the learning procedure to evaluate the robustness of the methods (use at least 20 random seeds). I don't think any of the results is based on more than one random seed (at least I could not find any statement regarding this).
7. Table 1 and 2: The highlighting in bold is unclear. If it is supposed to highlight the best methods, then the highlighting is dishonest in the sense that methods, which perform similarly, are not highlighted. For example, in Table 1, VERSA or MetaVRF (w/o LSTM) could be highlighted for all tasks because the error bars are so huge (similar in Table 2).
8. One of the things I'm missing completely is a discussion about computational demand: How efficiently can we train the model, and how long does it take to make predictions? It would be great to have some discussion about this in the paper and relate this to other approaches. 
9. The paper evaluates also the effect of having an LSTM that correlates tasks in the posterior. The analysis shows that there are some marginal gains, but none of the is statistically significant. I would have liked to see much more analysis of the effect/benefit of the LSTM.

Summary: The paper addresses an interesting problem. However, I have reservations regarding some theoretical bits and regarding the quality of the evaluation. Given that this paper also exceeds the 8 pages (default) limit, we are supposed to ask for higher acceptance standards than for an 8-pages paper. Hence, putting everything together, I recommend to reject this paper.
Decision: Reject