Title: LEARNING TO LEARN KERNELS WITH VARIATIONAL RANDOM FEATURES

Abstract: Meta-learning for few-shot learning involves a meta-learner that acquires shared knowledge from a set of prior tasks to improve the performance of a base-learner on new tasks with a small amount of data. Kernels are commonly used in machine learning due to their strong nonlinear learning capacity, which have not yet been fully investigated in the meta-learning scenario for few-shot learning. In this work, we explore kernel approximation with random Fourier features in the meta-learning framework for few-shot learning. We propose learning adapative kernels by meta variational random features (MetaVRF), which is formulated as a variational inference problem. To explore shared knowledge across diverse tasks, our MetaVRF deploys an LSTM inference network to generate informative features, which can establish kernels of highly representational power with low spectral sampling rates, while also being able to quickly adapt to specific tasks for improved performance. We evaluate MetaVRF on a variety of few-shot learning tasks for both regression and classification. Experimental results demonstrate that our MetaVRF can deliver much better or competitive performance than recent meta-learning algorithms.

1. INTRODUCTION:

Humans have the instinct to effortlessly learn new concepts from a few examples and show great generalization ability to new samples. However, existing machine learning models, e.g., deep neural networks (DNNs) (Krizhevsky et al., 2012; He et al., 2016a) , rely highly on large-scale annotated training data (Deng et al., 2009) to achieve satisfactory performance. The huge gap between human intelligence and DNNs motivates us to try and progress the task of learning from a few samples, a.k.a. few-shot learning (Fei-Fei et al., 2006; Lake et al., 2015; Ravi & Larochelle, 2017) .

Learning to learn, or meta-learning (Schmidhuber, 1992) , has recently received great interests in the machine learning community and offers a promising tool for few-shot learning (Andrychowicz et al., 2016; Ravi & Larochelle, 2017; Finn et al., 2017) . Generally speaking, a meta-learner (Ravi & Larochelle, 2017; Bertinetto et al., 2019 ) is trained to improve the performance of a base-learner on individual tasks, which is also fast adapted to solve new tasks. The crux of meta-learning for few-shot learning is to explore the common knowledge, such as a good parameter initialization (Finn et al., 2017) or efficient optimization update rule (Andrychowicz et al., 2016; Ravi & Larochelle, 2017) , shared across different tasks. The knowledge is accumulated and distilled throughout the learning stage, making the model adaptable to new but related tasks (Finn et al., 2017) . Kernel approximation by random Fourier features (RFFs) (Rahimi & Recht, 2007 ) is an effective technique for efficient kernel learning (Gärtner et al., 2002) , which has recently become increasingly popular (Sinha & Duchi, 2016; Carratino et al., 2018) . It resorts to the Fourier transform of shift-invariant kernels and constructs explicit feature maps using the Monte Carlo approximation of the Fourier representation. The desired kernel function is approximated by the inner products between these random features. Though demonstrating great potential as a strong base learner, kernel approximation with random features has not yet been fully explored in the meta-learning scenario for few-shot learning. It has already been shown that the classification performance of the kernel with random features does not correlate well with the accurate approximation of kernels. Learning adaptive kernels with random features, for instance, by data-driven sampling strategies (Sinha & Duchi, 2016) , can improve the performance with a low sampling rate compared to using universal random features (Avron et al., 2016; Chang et al., 2017) . However, since only a few samples are

Cell Cell:

Base-Learner ! " The inference framework of our meta variational random features (MetaVRF) with a long short-term memory (LSTM) network (t is the task index). The meta-learner employs an LSTM network to infer random Fourier features from the support set S t of the current task and bases ω t-1 of previous tasks, which explores task dependency to extract shared knowledge. The base-learner is a classifier with kernels obtained by variational features from the meta-learner.

available in each task, it is challenging to learn adaptive kernels with data-driven random features while maintaining high representational capacity for few-shot learning tasks. To obtain powerful kernels for few-shot learning tasks, we need to fully explore the relationship among diverse tasks and capture their shared knowledge to generate informative random features.

In this work, we propose meta variational random features (MetaVRF) to approximate kernels in a data-driven manner for few-shot learning, which integrates variational inference and kernels in the meta-learning framework. Learning kernels with random Fourier features for few-shot learning allows us to leverage the universal approximation property of kernels to capture shared knowledge in related tasks, and meanwhile it enables us to learn adaptive basis functions to quickly and efficiently adapt to new tasks. Learning adaptive kernels with data-driven random features can be naturally cast into variational inference that approximates probability density through optimization, where the posterior over the random basis function is the spectral distribution of a translation-invariant kernel.

The inference of the posterior is conducted in the context of tasks to exploring their dependency for capturing shared knowledge. We adopt a long short-term memory (LSTM) based inference network (Hochreiter & Schmidhuber, 1997) , which establishes task context inference to capture the task dependency. Specifically, during the inference, the cell state in the LSTM carries and accumulates the shared knowledge which is updated for each task throughout the course of learning. The remember and forget operations in the LSTM use new information to episodically refine the cell state by gaining experience from a batch of tasks, which can eventually produce random features of highly representational capability for all tasks. For an individual task, the task specific information is first extracted from the support set, and then combined with the shared knowledge in the shared cell state together as the joint condition, to infer the adaptive spectral distribution of the kernels. As a result, the task context inference can not only learn to extract and maintain the shared knowledge across tasks, but also leverage the task-specific knowledge to achieve an adaptive kernel to the current task.

The inference framework of our MetaVRF is illustrated in Figure 1 .

Extensive experiments on a variety of few-shot learning problems such as regression and classification demonstrate that, our MetaVRF method achieves competitive or even better performance when compared to state-of-the-art algorithms. Due to the advantages of kernels, our MetaVRF can be applied to test settings with different ways and shots from those of training setting, in which the promising results again validate the effectiveness of our MetaVRF for few-shot learning.

2. PROBLEM STATEMENT:

In this section, we describe the setup of meta-learning for few-shot learning and introduce the kernel ridge regression as the base-learner, where kernels are approximated by random Fourier features. 

Q t = {(x i , ỹi )} m i=1 (S t , Q t ⊆ X )

, we learn the parameters α t of the predictor f α t using a standard learning algorithm with kernel trick α t = Λ(Φ(X), Y ), where S t = {X, Y }. Here, Λ is the base-learner and Φ : X → R X is a mapping function from X to a dot product space H. The similarity measure k(x, x ) = Φ(x), Φ(x ) is usually called a kernel (Hofmann et al., 2008) .

In traditional supervised learning, the base-learner for the t-th single task usually uses a universal kernel to map the input onto a dot product space for efficient learning. Once the base-learner is trained on the support set, its performance is evaluated on the query set by the following loss function

EQUATION

where L(•) can be any differentiable function, e.g., cross-entropy loss. In the meta-learning setting for few-shot learning, we usually consider a batch of tasks. Thus, the meta-learner is trained by optimizing the following objective function w.r.t. the empirical loss on T tasks

EQUATION

where Φ t is the feature mapping function which can be obtained by learning task-specific kernel k t for each task t with data-driven ramdom Fourier features.

In this work, we employ kernel ridge regression (KRR), which has an efficient closed-form solution, as the base-learner Λ for few-shot learning. The kernel value in the Gram matrix K ∈ R Ck×Ck can be computed as k(x, x ) = Φ(x)Φ(x ) , where " " is the transpose operation. The base-learner Λ for a single task can be obtained by solving the following objective w.r.t. the support set of this task,

EQUATION

This produces a closed-form solution α = (λI + K) -1 Y . The learned predictor is then applied to the query set for prediction of the query set

EQUATION

where K = Φ(X)Φ( X) ∈ R Ck×m is with each element as k(x, x) between the samples from the support and query sets. Note that we also treat λ in Eq. (3) as a trainable parameter by leveraging the meta-learning setting, and all these parameters are learned by the meta-learner.

In order to obtain task-specific kernels, we propose to learn kernels with random Fourier features, which not only allows us to obtain task-adaptive kernels but also enables us to capture shared knowledge of different tasks by exploring their dependency.

2.2. RANDOM FOURIER FEATURES:

Random Fourier features (RFFs) were proposed to construct explicit random feature maps using the Monte Carlo approximation of the Fourier representation (Rahimi & Recht, 2007) , which is derived from Bochner's theorem (Rudin, 1962) .

Theorem 1 (Bochner's theorem) (Rudin, 1962) A continuous, real valued, symmetric and shiftinvariant function k(x, x ) = k(x -x ) on R d is a positive definite kernel if and only if it is the Fourier transform p(ω) of a positive finite measure such that

EQUATION

It is guaranteed that ζ ω (x)ζ ω (x) * is an unbiased estimation of k(x, x ) with sufficient RFF bases {ω} drawn from p(ω) (Rahimi & Recht, 2007) .

For a predefined kernel, e.g., radius basis function (RBF), we sample from its spectral distribution using the Monte Carlo method, and obtain the explicit feature map:

EQUATION

where Learning adaptive kernel with data-driven random Fourier features is essential to find the posterior distribution and the specific spectral distribution of kernels. In the following section, we introduce our meta variational random features (MetaVRF), in which random Fourier bases are treated as latent variables inferred from the support set in the meta-learning setting.

{ω 1 , • • • , ω D }

3 META VARIATIONAL RANDOM FEATURES

3.1. META EVIDENCE LOWER BOUND:

From the probabilistic perspective of view, the goal of few-shot learning is to maximize the conditional predictive log-likelihood of samples from the query set Q. We treat the random Fourier base ω of the kernel as a latent variable: 

In order to infer the posterior p(ω|y, x, S) over ω, which is generally intractable, we resort to using a variational distribution q φ (ω|S) to approximate this posterior, where the base is conditioned on the support set S by leveraging meta-learning. We can obtain the variational distribution by minimizing the Kullback-Leibler (KL) divergence

EQUATION

By applying the Bayes' rule to the posterior p(ω|y, x, S), we can derive the meta ELBO as

log p(y|x, S) ≥ (x,y)∈Q E q φ (ω|S) log p(y|x, S, ω) -D KL [q φ (ω|S)||p(ω|x, S)] = L MetaELBO . (9)

The first term of meta ELBO is the predictive log-likelihood conditioned on the observation x, S and the inferred RFF bases ω. Maximizing it enables us to make an accurate prediction for the query set by utilizing the inferred bases from the support set. The second term in our meta ELBO minimizes the discrepancy between the meta variational distribution q φ (ω|S) and the meta prior p(ω|x, S), which encourages samples from support and query sets to share the same random Fourier bases. The full derivation of the meta ELBO is provided in the Appendix B.

We now obtain the objective by maximizing the meta ELBO with respect to a batch of tasks:

EQUATION

)

where S t is the support set of the t-th task associated with its specific bases {ω t d } D d=1 . Directly optimizing the above objective does not take into count the task dependency. We introduce task context inference by making the posterior conditioned on both the support set of the current task and the bases from previous tasks. (x, y ) is a test sample in the query set Q t . The base ω t is inferred by conditioning on both the base ω t-1 from the previous task and the support set S t of the current task.

3.2. TASK CONTEXT INFERENCE:

To leverage the knowledge shared across tasks, we propose task context inference for random feature bases. Specifically, the bases {ω t d } D d=1 of the t-th task should rely on the base in the previous t -1 task, denoted as {ω t-1 d } D d=1 . The directed graphical model with related variables is shown in Figure 2 . To compute the probability of ω t conditioned on ω t-1 , we replace the previous variational posterior q φ (ω t |S t ) with q φ (ω|S t , ω t-1 ). Therefore, the objective of the task context inference is:

L = 1 T T t=1   (x,y)∈Q t E q φ (ω t |ω t-1 ,S t ) log p(y|x, S t , ω t ) -D KL [q φ (ω t |ω t-1 , S t )||p(ω t |x, S t )]   .

(11) Note that the variational approximate posterior q φ (ω t |ω t-1 , S t ) is a multivariate Gaussian with a diagonal covariance. Given the support set as input, the mean ω µ and standard deviation ω σ are output from the LSTM inference network φ(•). To enable the back-propagation of the LSTM inference network with the sampling operation during training, we leverage the reparametrization trick (Kingma & Welling, 2013) as

EQUATION

)

We use a permutation-invariant instance pooling layer to aggregate the support set S t of a group of examples, which essentially takes the average over the feature vectors of samples in the support set into a single vector, as in (Zaheer et al., 2017) . In practice, the feature representation e is extracted for each image x by a shared convolutional network ψ(•), i.e., e = ψ(x). The aggregation of samples in the support set is denoted as ē.

We propose an LSTM-type inference network inspired by the fact that the long-term memory can be carried and refined in cell states c during its update (Gers & Schmidhuber, 2000) . Specifically, we design a simplified variant of LSTM with two gates to remove the effect of the short-term memory. The common knowledge shared by tasks is stored in the cell state and updated with new information in each episode. During inference, LSTM can remove trivial information and add knowledge with a highly representational ability to the cell state. Once we have updated the cell state, the shared knowledge stored within it is combined with the task-specific information from the input support set to infer the spectral distribution. The update steps in the LSTM network are

EQUATION

)

ω σ is computed in the same way. After training, the final state c T is the shared knowledge from the meta-training set, which is directly used in the meta-testing set. In the meta-testing stage, we use the LSTM inference network φ to obtain ω µ and ω σ from S and ω T . Then, a set of bases are sampled from q φ (ω|ω T , S) to construct the random features in the kernels. Finally, the base-learner is optimized to obtain the predictor for testing.

4. RELATED WORK:

Meta-learning, or learning to learn, endues machine learning models the ability to improve their performance with for a number of training tasks. It has received increasing research interest with breakthroughs in many directions (Finn et To the best of our knowledge, our work is the first to extend kernel learning with random features to the meta-learning framework for few-shot learning. We propose training a meta-learner to infer the spectral distribution of random features from the support set. The task-specific kernel can be leveraged by the base-learner for the supervised learning. Compared with RFFs, our learned MetaVRF achieves superior performance on few-shot learning tasks with a low sampling rate of bases.

5. EXPERIMENTS:

In this section, we evaluate our MetaVRF on several few-shot learning problems for both regression and classification. We conduct classification experiments on three commonly-used benchmark datasets, i.e., Omniglot (Lake et al., 2015), miniImageNet (Vinyals et al., 2016) and CIFAR-FS (Krizhevsky et al., 2009) . More details about the three datasets are provided in the Appendix C. We also conduct deeper analysis to validate the effectiveness of our MetaVRF.

5.1. FEW-SHOT REGRESSION:

We begin with a k-shot regression problem, and compare our MetaVRF with MAML (Finn et al., 2017) as the baseline method. We follow the MAML work (Finn et al., 2017) embedding using a small multi-layer perceptron (MLP) with two hidden layers of size 40, following the same settings as used in MAML. We have also experimented with our MetaVRF without using LSTM (MetaVRF w/o LSTM) for the inference.

The comparison results are plotted in Figure 3 . The proposed MetaVRF can fit the function accurately with even three shots. With the increase of shots, our MetaVRF can perform better, almost entirely fitting the target function with 10 shots. It clearly shows that the performance can be improved by adopting the LSTM inference, which verifies the benefits of exploring task dependency. In addition, we can see that our MetaVRF performs much better than MAML which is a representative metalearning algorithm for all three settings with different numbers of shots. More few-shot regression results are provided in the Appendix E.

5.2. FEW-SHOT CLASSIFICATION:

For the classification task, we compare the proposed MetaVRF with the baseline method, random Fourier features (RFFs) and other state-of-the-art models. In particular, RFFs are the random feature computed from the universal kernel without adaptation.

Experimental settings. Image features are extracted via a shallow convolutional neural network. We employ the same architecture used in (Gordon et al., 2019) . We do not use any fully connected layer for these CNNs. The dimension of all feature vectors is 256. The inference network for RFFs is replaced with a feed-forward network with a couple of dense blocks. The key hyperparameter of the number of bases D in Eq. ( 6), is set to D = 512 for MetaVRF in all experiments, while we set D = 2048 for RFFs. The sampling rate of MetaVRF is much lower than in previous studies on RFFs, in which D is usually set to be 5 to 10 times the dimension of the input features (Yu et al., 2016; Rahimi & Recht, 2007) . All details are provided in the Appendix D.

Quantitive analysis. We evaluate our MetaVRF on three few-shot classification benchmarks using the same meta-testing protocol as (Gordon et al., 2019) , and present the results with 95% confidence intervals. All reported results are for models trained from scratch for few-shot learning. Table 1 and 2 report the performance of MetaVRF compared to the current state of the arts with shallow CNN architectures. Our MetaVRF method achieves the new state-of-the-art results on most of the challenging datasets, (e.g., 54.3% for 5-way 1-shot on miniImageNet, which is a 1% improvement over the second-best method). On the Omniglot dataset, previous approaches did not specify the splits for training, validation, and testing, which may result in unfair comparisons. Our MetaVRF method falls within the error bars of the state-of-the-art models on all experiments under 5-way 1-shot, 5-way 5-shot, 20-way 1-shot and 20-way 5-shot settings.

Since the feature extraction network used in our MetaVRF is different from that used in previous works, in 

5.3. FURTHER ANALYSIS:

Efficiency. In Figure 4 , we demonstrate the effectiveness on bases of our MetaVRF with a low sampling rate by comparing it with conventional random Fourier features (RFFs) . We present the mean testing accuracy of fully trained models w.r.t. RFFs and our MetaVRF as the number D of bases in Eq. ( 6) varies. Given the same number of sampled bases, our MetaVRF method consistently achieves much higher performance than RFFs, e.g., as shown for the 5-way, 5-shot condition in Figure 4 . The results verify the effectiveness of our MetaVRF in learning adaptive kernels and exploring the tasks dependencies.

Versatility. Our MetaVRF also shows promising performance when the number of ways C and k shots between training and testing are inconsistent. In Figure 5 , we plot the testing accuracy of the trained models on one particular C-way-k-shot task, with varied C and k in the testing stage. The results demonstrate that the trained model can maintain reasonable discriminability for a high number of testing ways. In particular, the model trained for the 20-way-5-shot task can retain high accuracy of 94% when tested under the 100-way condition, as shown in Figure 5 (a). The results also indicate that our MetaVRF exhibits considerable robustness and flexibility to a great variety of testing conditions.

6. CONCLUSIONS:

In this paper, we explore kernel approximation based on random Fourier features in the meta-learning framework for few-shot learning. We propose the novel meta variational random features (MetaVRF), which leverages variational inference and meta-learning to infer the spectral distribution of random Fourier features in a data-driven way. MetaVRF can generate random Fourier features of high representational power and a relatively low spectral sampling rate by using an LSTM inference network to explore the shared knowledge. In practice, our LSTM inference network demonstrates the great ability to quickly adapt to specific tasks for improved performance. Experiments on few-shot learning tasks for several benchmark datasets demonstrate the state-of-the-art performance over previous methods and the importance of our contribution.

A RESULTS OF DEEP ARCHITECTURES 

= log p(y|x, S, ω)p(ω|x, S) q φ (ω|S) q φ (ω|S) dω

≥ log p(y|x, S, ω)p(ω|x, S) q φ (ω|S) q φ (ω|S)dω (16)

EQUATION

We can also derive Meta ELBO from the KL divergence between the posterior p(ω|y, x, S) and its variational posterior q φ (ω|S) as following

EQUATION

= E q φ (ω|S) log q φ (ω|S) -log p(y|ω, x, S)p(ω|x, S) p(y|x, S)

= log p(y|x, S) + E q φ (ω|S) [log q φ (ω|S) -log p(y|ω, x, S) -log p(ω|x, S)]

= log p(y|x, S) -E q φ (ω|S) [log p(y|ω, x, S)]

EQUATION

≥ 0.

Therefore, the lower bound of the evidence p(y|x) is at the RHS of

EQUATION

which is consistent with Eq. ( 17). 

D MORE EXPERIMENTAL DETAILS:

We train all models using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0.0001. The other training setting and network architecture for regression and classification on three datasets are different as following.

D.1 FEATURE EMBEDDING NETWORKS:

Regression. The fully connected architecture for regression tasks is shown in Table 4 . We train all three models (3-shot, 5-shot, 10-shot) over a total of 20, 000 iterations, with 6 episodes per iteration.

Classification. The CNN architectures for Omniglot, CIFAR-FS, and miniImageNet are shown in Table 5 , 6, and 7. The architecture of the inference network for the regression task is in Table 8 . For few-shot classification tasks, all models share the same architecture, as in Table 9 .

D.3 PRIOR NETWORKS:

The architecture of the prior network for the regression task is in Table 10 . For few-shot classification tasks, all models share the same architecture, as in Table 11 .

D.4 OPTIMIZATION SETTINGS:

The number of training iterations and the batch size (episodes per iteration) are listed in Table 12 . 

:



Figures and Tables:

Figure1: The inference framework of our meta variational random features (MetaVRF) with a long short-term memory (LSTM) network (t is the task index). The meta-learner employs an LSTM network to infer random Fourier features from the support set S t of the current task and bases ω t-1 of previous tasks, which explores task dependency to extract shared knowledge. The base-learner is a classifier with kernels obtained by variational features from the meta-learner.

are the random bases sampled from p(ω), and [b 1 , • • • , b D ] are D biases sampled from a uniform distribution with a range of [0, 2π]. Finally, the kernel value k(x, x ) = z(x)z(x ) in K is computed as the dot product of their random feature maps with the same bases.

log p(y|x, S) = max p (x,y)∈Q log p(y|x, S, ω)p(ω|x, S)dω.

Figure 2: Illustration of MetaVRF in a directed graphical model.(x, y) is a test sample in the query set Q t . The base ω t is inferred by conditioning on both the base ω t-1 from the previous task and the support set S t of the current task.

Figure 3: Performance comparison for few-shot regression. Values are in term of MSE between three methods and ground truth.

Figure 4: Performance with different numbers D of bases.

al., 2017; Rusu et al., 2019; Gordon et al., 2019; Aravind Rajeswaran, 2019). Metric-based methods cast few-shot learning as a matching problem (Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018; Satorras & Estrach, 2018; Allen et al., 2019) by learning a shared similarity function for diverse tasks. Graphical neural network (GNN) based model generalizes the matching methods by learning the message propagation from the support set and transfer it to the query set (Garcia & Bruna, 2018). Gradient-based methods (e.g., MAML (Finn et al., 2017)) learn an appropriate initialization of model parameters and adapt it on new tasks with only a few gradient steps (Finn & Levine, 2018; Zintgraf et al., 2019; Rusu et al., 2019). Some other interesting works propose to directly learn the gradient optimization process of networks using RNNs (Ravi & Larochelle, 2017; Andrychowicz et al., 2016). Memory-based methods learn to leverage an external memory module to store and leverage key knowledge for quick adaptation (Santoro et al., 2016; Ramalho & Garnelo, 2019). Bayesian meta-learning methods (Edwards & Storkey, 2017; Finn et al., 2018; Gordon et al., 2019) usually rely on hierarchical Bayesian models to learn the shared statistical information among different tasks and reason about the uncertainty over models. Differentiable solution methods (Liu et al., 2019; Bertinetto et al., 2019) learn a universal feature embedding and obtain a task-specific learner with a closed-form solution.While those meta-learning algorithms have achieved great success in solving few-shot learning tasks, it remains an open challenge to explore shared knowledge across prior tasks. In this work, we introduce kernel learning with data-driven random Fourier features to explore task dependency to extract the shared knowledge.

Table content: <table/>



Table content: <table/>

Classification accuracies (%) on miniImageNet and CIFAR-FS.

Table content: <table><tr><td>miniImageNet, 5-way</td><td>CIFAR-FS, 5-way</td></tr></table>

Classification accuracies (%) on Omniglot.overfitting. For a fair comparison, we increase the number of filters in MAML from 32 to 64 but obtains the inferior performance, indicating that MAML might be heavily prone to overfitting as the model size increases. In original SNAIL, they use the deep ResNet-12 embedding. We present the result of SNAIL reported inBertinetto et al. (2019), where the result is obtained using similar shallow networks as ours. The graph neural network (GNN)(Garcia & Bruna, 2018) leverages a large embedding network with [64, 96, 128, 256] filters and one fully connected layer. For R2-D2(Bertinetto et al., 2019), the numbers of filters are[96, 192, 384, 512] and we present the results of two of its variants with 64 channels. Different training and testing conditions may also affect the performance of the compared methods. We maintain consistent conditions to be consistent in our experiments but present the results of R2-D2(Bertinetto et al., 2019) trained in 20 ways for 5-way tasks. The performance of the reproduced R2-D2(Devos et al., 2019) is heavily impaired when this strategy is not employed.

Table content: <table><tr><td/><td colspan="2">Omniglot, 5-way</td><td colspan="2">Omniglot, 20-way</td></tr><tr><td>Method</td><td>1-shot</td><td>5-shot</td><td>1-shot</td><td>5-shot</td></tr><tr><td>SIAMESE NET (Koch, 2015)</td><td>96.7</td><td>98.4</td><td>88</td><td>96.5</td></tr><tr><td>MATCHING NET (Vinyals et al., 2016)</td><td>98.1</td><td>98.9</td><td>93.8</td><td>98.5</td></tr><tr><td>MAML (Finn et al., 2017)</td><td colspan="2">98.7±0.4 99.9±0.1</td><td>95.8±0.3</td><td>98.9±0.2</td></tr><tr><td>PROTO NET (Snell et al., 2017)</td><td colspan="2">98.5±0.2 99.5±0.1</td><td>95.3±0.2</td><td>98.7±0.1</td></tr><tr><td>SNAIL (Mishra et al., 2018)</td><td colspan="4">99.1±0.2 99.8 ±0.1 97.6 ±0.3 99.4 ±0.2</td></tr><tr><td>GNN (Garcia &amp; Bruna, 2018)</td><td>99.2</td><td>99.7</td><td>97.4</td><td>99.0</td></tr><tr><td>VERSA (Gordon et al., 2019)</td><td colspan="2">99.7±0.2 99.8±0.1</td><td>97.7±0.3</td><td>98.8±0.2</td></tr><tr><td>R2-D2 (Bertinetto et al., 2019)</td><td>98.6</td><td>99.7</td><td>94.7</td><td>98.9</td></tr><tr><td>IMP (Allen et al., 2019)</td><td colspan="2">98.4±0.3 99.5±0.1</td><td>95.0±0.1</td><td>98.6±0.1</td></tr><tr><td>RFFS (2048d)</td><td colspan="2">99.5±0.2 99.5±0.2</td><td>97.2±0.3</td><td>98.3±0.2</td></tr><tr><td>METAVRF (w/o LSTM, 512d)</td><td colspan="2">99.6±0.2 99.6±0.2</td><td>97.0±0.3</td><td>98.4±0.2</td></tr><tr><td>METAVRF (512d)</td><td colspan="2">99.8±0.2 99.8±0.1</td><td>97.5±0.3</td><td>99.0±0.2</td></tr></table>

Classification accuracies (%) on miniImageNet with 5 way using deep architecturesWe evaluate our method by using pre-trained embeddings and compare with latest methods with deep embedding architectures. We use the pre-trained embeddings from a 28-layer Wide Residual Network (WRN-28-10)(Zagoruyko & Komodakis, 2016), in a similar fashion to(Rusu et al., 2019;Bauer et al., 2017;Qiao et al., 2018). Specifically, the WRN-28-10 was trained to classify images which are all from the meta-training set into the corresponding categories. We select activations in the 21th layer, with average pooling over spatial dimensions, as feature embeddings. The dimension of pre-trained embeddings is 640. As shown in Table.3, our MetaVRF achieves the state-of-the-art preformance on both cases of 1-shot and 5-shot.

Table content: <table><tr><td>Method</td><td>1-shot</td><td>5-shot</td></tr><tr><td>META-SGD (Li et al., 2017)</td><td colspan="2">54.24±0.03 70.86±0.04</td></tr><tr><td colspan="3">(GIDARIS &amp; KOMODAKIS, 2018) 56.20±0.86 73.00±0.64</td></tr><tr><td>(BAUER ET AL., 2017)</td><td colspan="2">56.30±0.40 73.90±0.30</td></tr><tr><td>(MUNKHDALAI ET AL., 2017)</td><td colspan="2">57.10±0.70 70.04±0.63</td></tr><tr><td>(QIAO ET AL., 2018)</td><td colspan="2">59.60±0.41 73.54±0.19</td></tr><tr><td>LEO (Rusu et al., 2019)</td><td colspan="2">61.76±0.08 77.59±0.12</td></tr><tr><td>SNAIL (Mishra et al., 2018)</td><td colspan="2">55.71±0.99 68.88±0.92</td></tr><tr><td>TADAM (Oreshkin et al., 2018)</td><td colspan="2">58.50±0.30 76.70±0.30</td></tr><tr><td>METAVRF (ours)</td><td colspan="2">63.40±0.06 77.85±0.28</td></tr><tr><td>B DERIVATIONS OF META ELBO</td><td/><td/></tr><tr><td colspan="3">For a singe task, we begin with maximizing log-likelihood of the conditional distribution p(y|x, S)</td></tr><tr><td colspan="3">to derive the ELBO of MetaVRF. By leveraging Jensen's inequality, we have the following steps as</td></tr></table>

The fully connected network ψ(•) used for regression.

Table content: <table><tr><td colspan="2">Output size Layers</td></tr><tr><td>1</td><td>Input training samples</td></tr><tr><td>40</td><td>fully connected, RELU</td></tr><tr><td>40</td><td>fully connected, RELU</td></tr></table>

The CNN architecture ψ(•) for Omniglot.Omniglot (Lake et al., 2015) is a benchmark of few-shot learning that contain 1623 handwritten characters (each with 20 examples). All characters are grouped in 50 alphabets. For fair comparison against the state of the arts, we follow the same data split and pre-processing used in Vinyals et al. -FS use the same split criteria (64, 16, 20) with which miniImageNet has been generated. The resolution of all images is 32×32.

Table content: <table><tr><td colspan="2">Output size Layers</td></tr><tr><td>28×28×1</td><td>Input images</td></tr><tr><td colspan="2">14×14×64 conv2d (3×3, stride=1, SAME, RELU), dropout 0.9, pool (2×2, stride=2, SAME)</td></tr><tr><td>7×7×64</td><td>conv2d (3×3, stride=1, SAME, RELU), dropout 0.9, pool (2×2, stride=2, SAME)</td></tr><tr><td>4×4×64</td><td>conv2d (3×3, stride=1, SAME, RELU), dropout 0.9, pool (2×2, stride=2, SAME)</td></tr><tr><td>2×2×64</td><td>conv2d (3×3, stride=1, SAME, RELU), dropout 0.9, pool (2×2, stride=2, SAME)</td></tr><tr><td>256</td><td>flatten</td></tr><tr><td colspan="2">C FEW-SHOT CLASSIFICATION DATASETS</td></tr></table>

The CNN architecture ψ(•) for CIFAR-FS Output size Layers 32×32×3Input images 16×16×64 conv2d (3×3, stride=1, SAME, RELU), dropout 0.5, pool (2×2, stride=2, SAME)

Table content: <table><tr><td>8×8×64</td><td>conv2d (3×3, stride=1, SAME, RELU), dropout 0.5, pool (2×2, stride=2, SAME)</td></tr><tr><td>4×4×64</td><td>conv2d (3×3, stride=1, SAME, RELU), dropout 0.5, pool (2×2, stride=2, SAME)</td></tr><tr><td>2×2×64</td><td>conv2d (3×3, stride=1, SAME, RELU), dropout 0.5, pool (2×2, stride=2, SAME)</td></tr><tr><td>256</td><td>flatten</td></tr></table>

The CNN architecture ψ(•) for miniImageNet Output size Layers 84×84×3 Input images 42×42×64 conv2d (3×3, stride=1, SAME, RELU), dropout 0.5, pool (2×2, stride=2, SAME) 21×21×64 conv2d (3×3, stride=1, SAME, RELU), dropout 0.5, pool (2×2, stride=2, SAME) 10×10×64 conv2d (3×3, stride=1, SAME, RELU), dropout 0.5, pool (2×2, stride=2, SAME) 5×5×64 conv2d (3×3, stride=1, SAME, RELU), dropout 0.5, pool (2×2, stride=2, SAME) 2×2×64 conv2d (3×3, stride=1, SAME, RELU), dropout 0.5, pool (2×2, stride=2, SAME) 256 flatten D.2 INFERENCE NETWORKS

Table content: <table/>

The inference network φ(•) used for regression. Tanh to µ w , log σ 2 w

Table content: <table><tr><td colspan="2">Output size Layers</td></tr><tr><td>40</td><td>Input samples feature</td></tr><tr><td>40</td><td>fully connected, ELU</td></tr><tr><td>40</td><td>fully connected, ELU</td></tr><tr><td>40</td><td>LSTM cell,</td></tr></table>

The inference network φ(•) used for Omniglot, miniImageNet, CIFAR-FS Output size Layers

Table content: <table><tr><td>k × 256</td><td>Input feature</td></tr><tr><td>256</td><td>instance pooling</td></tr><tr><td>256</td><td>fully connected, ELU</td></tr><tr><td>256</td><td>fully connected, ELU</td></tr><tr><td>256</td><td>fully connected, ELU</td></tr><tr><td>256</td><td>LSTM cell, tanh to µ w , log σ 2 w</td></tr></table>

The prior network used for regression.

Table content: <table><tr><td colspan="2">Output size Layers</td></tr><tr><td>80</td><td>The concatenation of query feature and aggregated support features</td></tr><tr><td>40</td><td>fully connected, ELU</td></tr><tr><td>40</td><td>fully connected, ELU</td></tr><tr><td>40</td><td>fully connected to µ w , log σ 2 w</td></tr></table>

The prior network used for Omniglot, miniImageNet, CIFAR-FS Output size Layers

Table content: <table><tr><td>512</td><td>The concatenation of query feature and aggregated support features</td></tr><tr><td>256</td><td>instance pooling</td></tr><tr><td>256</td><td>fully connected, ELU</td></tr><tr><td>256</td><td>fully connected, ELU</td></tr><tr><td>256</td><td>fully connected to µ w , log σ 2 w</td></tr></table>

Iteration and batch size for all datasets.

Table content: <table><tr><td>Dataset</td><td colspan="2">Iter. Batch size</td></tr><tr><td>Regression</td><td>20, 000</td><td>25</td></tr><tr><td>Omniglot</td><td>100, 000</td><td>6</td></tr><tr><td>CIFAR-FS</td><td>200, 000</td><td>8</td></tr><tr><td colspan="2">miniImageNet 100, 000</td><td>6</td></tr></table>

Peer review (iclr2020):
This paper studies meta-learning problem with few-shot learning settings. The author proposes a learn each task predictive function via the form of random Fourier features, where the kernel is jointly learned from all tasks. The novel part is the parametrization of inference network using LSTM such that the random feature samples of t-th task conditional depending on all previous task 1,...,t-1, which is an interesting way of modeling kernel spectral distribution. The experiment results show improvement of the proposed methods compared to SoTA meta learning algorithms. In general, the writing of the paper is clear, and the proposed method is interesting and novel. However, there are parts missing in the experiment setting. I would love to increase my score if the author could address the following questions/comments: (1) How do you choose the meta prior distribution? It should be a basic kernel family such as RBF Gaussian or mixture of RBF? (2) In Table 1 and Table 2, the benefit of using LSTM only gives very marginal improvement over w/o LSTM. Are the results statistically significant? (3) The experiment missed the simple kernel learning baseline, such as kernel alignment [1] and its variants [2]. If using these task-independent way to do kernel learning, what’s their performance compared to you proposed method? (4) When learning the RFF spectral distribution using LSTM over a sequence of tasks, does the order of task matter? [1] Learning kernels with random features, NIPS 2016. [2] Implicit kernel learning, AISTATS 2019.