Learning to Learn Kernels with Variational Random Features

1 INTRODUCTION . Humans have the instinct to effortlessly learn new concepts from a few examples and show great generalization ability to new samples . However , existing machine learning models , e.g. , deep neural networks ( DNNs ) ( Krizhevsky et al. , 2012 ; He et al. , 2016a ) , rely highly on large-scale annotated training data ( Deng et al. , 2009 ) to achieve satisfactory performance . The huge gap between human intelligence and DNNs motivates us to try and progress the task of learning from a few samples , a.k.a . few-shot learning ( Fei-Fei et al. , 2006 ; Lake et al. , 2015 ; Ravi & Larochelle , 2017 ) . Learning to learn , or meta-learning ( Schmidhuber , 1992 ) , has recently received great interests in the machine learning community and offers a promising tool for few-shot learning ( Andrychowicz et al. , 2016 ; Ravi & Larochelle , 2017 ; Finn et al. , 2017 ) . Generally speaking , a meta-learner ( Ravi & Larochelle , 2017 ; Bertinetto et al. , 2019 ) is trained to improve the performance of a base-learner on individual tasks , which is also fast adapted to solve new tasks . The crux of meta-learning for few-shot learning is to explore the common knowledge , such as a good parameter initialization ( Finn et al. , 2017 ) or efficient optimization update rule ( Andrychowicz et al. , 2016 ; Ravi & Larochelle , 2017 ) , shared across different tasks . The knowledge is accumulated and distilled throughout the learning stage , making the model adaptable to new but related tasks ( Finn et al. , 2017 ) . Kernel approximation by random Fourier features ( RFFs ) ( Rahimi & Recht , 2007 ) is an effective technique for efficient kernel learning ( Gärtner et al. , 2002 ) , which has recently become increasingly popular ( Sinha & Duchi , 2016 ; Carratino et al. , 2018 ) . It resorts to the Fourier transform of shift-invariant kernels and constructs explicit feature maps using the Monte Carlo approximation of the Fourier representation . The desired kernel function is approximated by the inner products between these random features . Though demonstrating great potential as a strong base learner , kernel approximation with random features has not yet been fully explored in the meta-learning scenario for few-shot learning . It has already been shown that the classification performance of the kernel with random features does not correlate well with the accurate approximation of kernels . Learning adaptive kernels with random features , for instance , by data-driven sampling strategies ( Sinha & Duchi , 2016 ) , can improve the performance with a low sampling rate compared to using universal random features ( Avron et al. , 2016 ; Chang et al. , 2017 ) . However , since only a few samples are available in each task , it is challenging to learn adaptive kernels with data-driven random features while maintaining high representational capacity for few-shot learning tasks . To obtain powerful kernels for few-shot learning tasks , we need to fully explore the relationship among diverse tasks and capture their shared knowledge to generate informative random features . In this work , we propose meta variational random features ( MetaVRF ) to approximate kernels in a data-driven manner for few-shot learning , which integrates variational inference and kernels in the meta-learning framework . Learning kernels with random Fourier features for few-shot learning allows us to leverage the universal approximation property of kernels to capture shared knowledge in related tasks , and meanwhile it enables us to learn adaptive basis functions to quickly and efficiently adapt to new tasks . Learning adaptive kernels with data-driven random features can be naturally cast into variational inference that approximates probability density through optimization , where the posterior over the random basis function is the spectral distribution of a translation-invariant kernel . The inference of the posterior is conducted in the context of tasks to exploring their dependency for capturing shared knowledge . We adopt a long short-term memory ( LSTM ) based inference network ( Hochreiter & Schmidhuber , 1997 ) , which establishes task context inference to capture the task dependency . Specifically , during the inference , the cell state in the LSTM carries and accumulates the shared knowledge which is updated for each task throughout the course of learning . The remember and forget operations in the LSTM use new information to episodically refine the cell state by gaining experience from a batch of tasks , which can eventually produce random features of highly representational capability for all tasks . For an individual task , the task specific information is first extracted from the support set , and then combined with the shared knowledge in the shared cell state together as the joint condition , to infer the adaptive spectral distribution of the kernels . As a result , the task context inference can not only learn to extract and maintain the shared knowledge across tasks , but also leverage the task-specific knowledge to achieve an adaptive kernel to the current task . The inference framework of our MetaVRF is illustrated in Figure 1 . Extensive experiments on a variety of few-shot learning problems such as regression and classification demonstrate that , our MetaVRF method achieves competitive or even better performance when compared to state-of-the-art algorithms . Due to the advantages of kernels , our MetaVRF can be applied to test settings with different ways and shots from those of training setting , in which the promising results again validate the effectiveness of our MetaVRF for few-shot learning . 2 PROBLEM STATEMENT . In this section , we describe the setup of meta-learning for few-shot learning and introduce the kernel ridge regression as the base-learner , where kernels are approximated by random Fourier features . 2.1 META-LEARNING WITH KERNELS . We adopt the episodic training strategy ( Ravi & Larochelle , 2017 ) commonly used for few-shot classification in meta-learning , which usually involves the meta-training and meta-test stages . In the meta-training stage , a meta-learner is trained to enhance the performance of a base-learner on a meta-training set with a batch of few-shot learning tasks , where a task is usually referred as an episode ( Ravi & Larochelle , 2017 ) . In the meta-test stage , the base learner is evaluated on a meta-test set with different classes of data samples from the meta-training set . For the few-shot classification problem , we sample C-way k-shot classification tasks from the metatraining set , where k is the number of labelled examples for each of the C classes . Given the t-th task with a support set St = { ( xi , yi ) } C×ki=1 and query set Qt = { ( x̃i , ỹi ) } mi=1 ( St , Qt ⊆ X ) , we learn the parameters αt of the predictor fαt using a standard learning algorithm with kernel trick αt = Λ ( Φ ( X ) , Y ) , where St = { X , Y } . Here , Λ is the base-learner and Φ : X → RX is a mapping function from X to a dot product space H. The similarity measure k ( x , x′ ) = 〈Φ ( x ) , Φ ( x′ ) 〉 is usually called a kernel ( Hofmann et al. , 2008 ) . In traditional supervised learning , the base-learner for the t-th single task usually uses a universal kernel to map the input onto a dot product space for efficient learning . Once the base-learner is trained on the support set , its performance is evaluated on the query set by the following loss function∑ ( x̃ , ỹ ) ∈Qt L ( fαt ( Φ ( x̃ ) ) , ỹ ) , ( 1 ) where L ( · ) can be any differentiable function , e.g. , cross-entropy loss . In the meta-learning setting for few-shot learning , we usually consider a batch of tasks . Thus , the meta-learner is trained by optimizing the following objective function w.r.t . the empirical loss on T tasks∑ t ∑ ( x̃ , ỹ ) ∈Qt L ( fαt ( Φt ( x̃ ) ) , ỹ ) , with αt = Λ ( Φt ( X ) , Y ) , ( 2 ) where Φt is the feature mapping function which can be obtained by learning task-specific kernel kt for each task t with data-driven ramdom Fourier features . In this work , we employ kernel ridge regression ( KRR ) , which has an efficient closed-form solution , as the base-learner Λ for few-shot learning . The kernel value in the Gram matrix K ∈ RCk×Ck can be computed as k ( x , x′ ) = Φ ( x ) Φ ( x′ ) > , where “ > ” is the transpose operation . The base-learner Λ for a single task can be obtained by solving the following objective w.r.t . the support set of this task , Λ = arg min α Tr [ ( Y − αK ) ( Y − αK ) > ] + λαKα > . ( 3 ) This produces a closed-form solution α = ( λI +K ) −1Y . The learned predictor is then applied to the query set for prediction of the query set X̃ : Ŷ = fα ( X̃ ) = αK̃ , ( 4 ) where K̃ = Φ ( X ) Φ ( X̃ ) > ∈ RCk×m is with each element as k ( x , x̃ ) between the samples from the support and query sets . Note that we also treat λ in Eq . ( 3 ) as a trainable parameter by leveraging the meta-learning setting , and all these parameters are learned by the meta-learner . In order to obtain task-specific kernels , we propose to learn kernels with random Fourier features , which not only allows us to obtain task-adaptive kernels but also enables us to capture shared knowledge of different tasks by exploring their dependency . 2.2 RANDOM FOURIER FEATURES . Random Fourier features ( RFFs ) were proposed to construct explicit random feature maps using the Monte Carlo approximation of the Fourier representation ( Rahimi & Recht , 2007 ) , which is derived from Bochner ’ s theorem ( Rudin , 1962 ) . Theorem 1 ( Bochner ’ s theorem ) ( Rudin , 1962 ) A continuous , real valued , symmetric and shiftinvariant function k ( x , x′ ) = k ( x − x′ ) on Rd is a positive definite kernel if and only if it is the Fourier transform p ( ω ) of a positive finite measure such that k ( x , x′ ) = ∫ Rd eiω > ( x−x′ ) dp ( ω ) = Eω [ ζω ( x ) ζω ( x ) ∗ ] , where ζω ( x ) = eiω > x . ( 5 ) It is guaranteed that ζω ( x ) ζω ( x ) ∗ is an unbiased estimation of k ( x , x′ ) with sufficient RFF bases { ω } drawn from p ( ω ) ( Rahimi & Recht , 2007 ) . For a predefined kernel , e.g. , radius basis function ( RBF ) , we sample from its spectral distribution using the Monte Carlo method , and obtain the explicit feature map : z ( x ) = 1√ D [ cos ( ω > 1 x + b1 ) , · · · , cos ( ω > Dx + bD ) ] , ( 6 ) where { ω1 , · · · , ωD } are the random bases sampled from p ( ω ) , and [ b1 , · · · , bD ] are D biases sampled from a uniform distribution with a range of [ 0 , 2π ] . Finally , the kernel value k ( x , x′ ) = z ( x ) z ( x′ ) > in K is computed as the dot product of their random feature maps with the same bases . Learning adaptive kernel with data-driven random Fourier features is essential to find the posterior distribution and the specific spectral distribution of kernels . In the following section , we introduce our meta variational random features ( MetaVRF ) , in which random Fourier bases are treated as latent variables inferred from the support set in the meta-learning setting .

Summary: This paper studies meta-learning problem with few-shot learning settings. The author proposes a learn each task predictive function via the form of random Fourier features, where the kernel is jointly learned from all tasks. The novel part is the parametrization of inference network using LSTM such that the random feature samples of t-th task conditional depending on all previous task 1,...,t-1, which is an interesting way of modeling kernel spectral distribution. The experiment results show improvement of the proposed methods compared to SoTA meta learning algorithms.