Few-Shot Regression via Learning Sparsifying Basis Functions

Recent few-shot learning algorithms have enabled models to quickly adapt to new tasks based on only a few training samples . Previous few-shot learning works have mainly focused on classification and reinforcement learning . In this paper , we propose a method that focuses on regression tasks . Our model is based on the idea that the degree of freedom of the unknown function can be significantly reduced if it is represented as a linear combination of a set of sparsifying basis functions . This enables using a few labelled samples to learn a good approximation of the entire function . We design a Basis Function Learner network to encode the basis functions for a task distribution , and a Weights Generator to generate the weight vector for a novel task . We show that our model outperforms current state of the art meta-learning methods in various regression tasks . 1 INTRODUCTION . Regression deals with the problem of learning a model relating a set of inputs to a set of outputs . The learned model can be thought as function y = F ( x ) that gives a prediction y ∈ Rdy given input x ∈ Rdx where dy and dx are dimensions of the output and input respectively . Typically , a regression model is trained on a large number of data points to be able to provide accurate predictions for new inputs . Recently , there have been a surge in popularity on few-shot learning methods ( Vinyals et al. , 2016 ; Koch et al. , 2015 ; Gidaris & Komodakis , 2018 ) . Few-shot learning methods require only a few examples from each task to be able to quickly adapt and perform well on a new task . These few-shot learning methods in essence are learning to learn i.e . the model learns to quickly adapt itself to new tasks rather than just learning to give the correct prediction for a particular input sample . In this work , we propose a few shot learning model that targets few-shot regression tasks . Our model takes inspiration from the idea that the degree of freedom of F ( x ) can be significantly reduced when it is modeled a linear combination of sparsifying basis functions . Thus , with a few samples , we can estimate F ( x ) . The two primary components of our model are ( i ) the Basis Function Learner network which encodes the basis functions for the distribution of tasks , and ( ii ) the Weights Generator network which produces the appropriate weights given a few labelled samples . We evaluate our model on the sinusoidal regression tasks and compare the performance to several meta-learning algorithms . We also evaluate our model on other regression tasks , namely the 1D heat equation tasks modeled by partial differential equations and the 2D Gaussian distribution tasks . Furthermore , we evaluate our model on image completion as a 2D regression problem on the MNIST and CelebA data-sets , using only a small subset of known pixel values . To summarize , our contributions for this paper are : • We propose to address few shot regression by linear combination of a set of sparsifying basis functions . • We propose to learn these ( continuous ) sparsifying basis functions from data . Traditionally , basis functions are hand-crafted ( e.g . Fourier basis ) . • We perform experiments to evaluate our approach using sinusoidal , heat equation , 2D Gaussian tasks and MNIST/CelebA image completion tasks . 2 RELATED WORK . Regression problems has long been a topic of study in the machine learning and signal processing community ( Myers & Myers , 1990 ; Specht , 1991 ) . Though similar to classification , regression estimates one or multiple scalar values and is usually thought of as a single task problem . A single model is trained to only perform regression on only one task . Our model instead reformulates the regression problem as a few-shot learning problem , allowing for our model to be able to perform regressions of tasks sampled from the same task distribution . The success achieved by deep neural networks heavily relies on a large amount of data , especially labelled ones . As labelling data is time-consuming and labor-intensive , learning from limited labelled data is drawing more and more attention . A prominent approach is meta learning . Meta learning , also referred as learning to learn , aims at learning an adaptive model across different tasks . Meta learning has shown potential in style transfer ( Zhang et al. , 2019 ) , visual navigation ( Wortsman et al. , 2018 ) , etc . Meta learning has also been applied to few-shot learning problems , which concerns models that can learn from prior experiences to adapt to new tasks . Lake et al . ( 2011 ) proposed the one-shot classification problem and introduced the Omniglot data set as a few-shot classification data set , similar to MNIST ( LeCun , 1998 ) for traditional classification . Since then , there has been a surge of meta learning methods striving to solve few-shot problems . Some meta learning approaches learn a similarity metric ( Snell et al. , 2017 ; Vinyals et al. , 2016 ; Koch et al. , 2015 ) between new test examples with few-shot training samples to make the prediction . The similarity metric used here can be Euclidean distance , cosine similarity or more expressive metric learned by relation networks ( Sung et al. , 2018 ) . On the other hand , optimization-based approaches learn how to optimize the model directly . Finn et al . ( 2017 ) learned an optimal initialization of models for different tasks in the same distribution , which is able to achieve good performance by simple gradient descent . Rusu et al . ( 2019 ) learned how to perform gradient descent in the latent space to adapt the model parameters more effectively . Ravi & Larochelle ( 2016 ) employed an LSTM to learn an optimization algorithm . Generative models are also proposed to overcome the limitations resulted from few-shot setting ( Zhang et al. , 2018 ; Hariharan & Girshick , 2017 ; Wang et al. , 2018 ) . Few-shot regression tasks are used among various few-shot leaning methods ( Finn et al. , 2017 ; Rusu et al. , 2019 ; Li et al. , 2017 ) . In most existing works , these experiment usually does not extend beyond the sinusoidal and linear regression tasks . A prominent family of algorithms that tackles a similar problem as few-shot regression is Neural Processes ( Garnelo et al. , 2018b ; a ; Kim et al. , 2019 ) . Neural Processes algorithms model the distributions of the outputs of regression functions using Deep Neural Networks given pairs of input-output pairs . Similar to Variational Autoencoders ( Kingma & Welling , 2013 ) , Neural Processes employ a Bayesian approach in modelling the output distribution of regression function using an encoder-decoder architecture . Our model on the other hand employs a deterministic approach where we directly learn a set of basis functions to model the output distribution . Our model also does not produce any latent vectors but instead produces predictions via a dot product between the learned basis functions and weight vector . Our experiment results show that our model ( based on sparse linear combination of basis functions ) compares favorably to Neural Processes ( based on conditional stochastic processes ) . Our proposed sparse linear representation framework for few shot regression makes the few shot regression problem appears to be similar to another research problem called dictionary learning ( DL ) ( Tosic & Frossard , 2011 ) , which focuses on learning dictionaries of atoms that provide efficient representations of some class of signals . However the differences between DL and our problem are significant : Our problems are continuous rather than discrete as in DL , and we only observe a very small percentage of samples . Detailed comparison with DL is discussed in the appendix . 3 PROPOSED METHOD . 3.1 PROBLEM FORMULATION . We first provide problem definition for few-shot regression . We aim at developing a model that can rapidly regress to a variety of equations and functions based on only a few training samples . We assume that each equation we would like to regress is a task Ti sampled from a distribution p ( T ) . We train our model on a set of training tasks , Strain , and evaluate it on a separate set of testing tasks , Stest . Unlike few-shot classification tasks , the tasks distribution p ( T ) is continuous for regression task in general . Each regression task is comprised of training samples Dtrain and validation samples Dval , for both the training set Strain and testing set Stest , Dtrain is comprised of K training samples and labels Dtrain = { ( xkt , ykt ) |k = 1 ... K } while Dval is comprised of N samples and labels Dval = { ( xnp , ynp ) |n = 1 ... N } . The goal of few-shot regression is to regress the entire , continuous output range of the equation given only the few points as training set . 3.2 FEW-SHOT REGRESSION VIA LEARNING SPARSIFYING BASIS FUNCTIONS . Here we discuss our main idea . We would like to model the unknown function y = F ( x ) given only Dtrain = { ( xkt , ykt ) |k = 1 ... K } . With small K , e.g . K = 10 , this is an ill-posed task , as F ( x ) can take any form . As stated before , we assume that each function we would like to regress is a task Ti drawn from an unknown distribution p ( T ) . To simplify discussion , we assume scalar input and scalar output . Our idea is to learn sparse representation of the unknown function F ( x ) , so that a few samples { ( xkt , ykt ) |k = 1 ... K } can provide adequate information to approximate the entire F ( x ) . Specifically , we model the unknown function F ( x ) as a linear combination of a set of basis functions { φi ( x ) } : F ( x ) = ∑ i wiφi ( x ) ( 1 ) Many handcrafted basis functions have been developed to expand F ( x ) . For example , the Maclaurin series expansion ( Taylor series expansion at x = 0 ) uses { φi ( x ) } = { 1 , x , x2 , x3 , ... } : F ( x ) = w0 + w1x+ w2x 2 + ... ( 2 ) If F ( x ) is a polynomial , ( 2 ) can be a sparse representation , i.e . only a few non-zero , significant wi , and most wi are zero or near zero . However , if F ( x ) is a sinusoid , it would require many terms to represent F ( x ) adequately , e.g . : sin ( x ) ≈ w1x+ w3x3 + w5x5 + w7x7 + ... + wMxM ( 3 ) In ( 3 ) , M is large and M K. Given only K samples { ( xkt , ykt ) |k = 1 ... K } , it is not adequate to determine { wi } and model the unknown function . On the other hand , if we use the Fourier basis instead , i.e. , { φi ( x ) } = { 1 , sin ( x ) , sin ( 2x ) , ... , cos ( x ) , cos ( 2x ) , ... } , clearly , we can obtain a sparse representation : we can adequately approximate the sinusoid with only a few terms . Under Fourier basis , there are only a few non-zero significant weights wi , and K samples are sufficient to estimate the significant wi and approximate the function . Essentially , with a sparsifying basis { φi ( x ) } , the degree of freedom of F ( x ) can be significantly reduced when it is modeled using ( 1 ) , so that K samples can well estimate F ( x ) . Our approach is to use the set of training tasks drawn from p ( T ) to learn { φi ( x ) } that result in sparse representation for any task drawn from p ( T ) . The set of { φi ( x ) } is encoded in the Basis Function Learner Network that takes in x and outputs Φ ( x ) = [ φ1 ( x ) , φ2 ( x ) , ... , φM ( x ) ] T . In our framework , Φ ( x ) is the same for any task drawn from p ( T ) , as it encodes the set of { φi ( x ) } that can sparsely represent any task from p ( T ) . We further learn a Weights Generator Network to map the K training samples of a novel task to a constant vector w = [ w1 , w2 , ... , wM ] T . The unknown function is modeled as wTΦ ( x ) .

Summary: The authors propose using sparse adaptive basis function models for few shot regression. The basis functions and the corresponding weights are generated via respective networks whose parameters are shared across all tasks. Elastic net regularization is used to encourage task specific sparsity in the weights,  the idea being  that with only a small number of available training examples, learning a sparse basis is  easier than learning a dense basis with many more parameters. The method is validated on both synthetic data and on image completion tasks.