Few-Shot Regression via Learning Sparsifying Basis Functions
Authors: Yi Loo
Yiluan Guo
Ngai-Man Cheung
Venue:  (2020)
Keywords: meta-learning
few-shot learning
regression
learning basis functions
self-attention

Abstract: Recent few-shot learning algorithms have enabled models to quickly adapt to new tasks based on only a few training samples. Previous few-shot learning works have mainly focused on classification and reinforcement learning. In this paper, we propose a few-shot meta-learning system that focuses exclusively on regression tasks. Our model is based on the idea that the degree of freedom of the unknown function can be significantly reduced if it is represented as a linear combination of a set of sparsifying basis functions. This enables a few labeled samples to approximate the function. We design a Basis Function Learner network to encode basis functions for a task distribution, and a Weights Generator network to generate the weight vector for a novel task. We show that our model outperforms the current state of the art meta-learning methods in various regression tasks.

Under review as a conference paper at ICLR 2020
FEW-SHOT REGRESSION VIA LEARNING SPARSIFYING BASIS FUNCTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent few-shot learning algorithms have enabled models to quickly adapt new tasks based on only a few training samples. Previous few-shot learning works have mainly focused on classification and reinforcement learning. In this paper, we propose method that focuses on regression tasks. Our model is based on the idea that the degree freedom of the unknown function can be significantly reduced if it is represented as linear combination of a set of sparsifying basis functions. This enables using a few labelled samples learn a good approximation of the entire function. We design a Basis Function Learner network to encode the basis functions for a task distribution, and a Weights Generator to generate the weight vector for novel task. We show that model outperforms current state of the art meta- learning methods in various regression tasks.
INTRODUCTION
Regression deals with the problem of learning model relating set of inputs set of outputs. The learned model can be thought as function y F(x) that gives a prediction y E Rds given input dR where dy and d, are dimensions of the output and input respectively. Typically, a regression model is trained on a large number of data points to be able to provide accurate predictions for new inputs. Recently, there have been surge in popularity on few-shot learning methods (Vinyalls et al.) 2016, Koch et al. 2015 Gidaris & Komodakis 2018). Few-shot learning methods require only few examples from each task to able to quickly adapt and perform well on new task. These few-shot learning methods in essence are learning to learn i.e. the model learns to quickly adapt itself to new tasks rather than just learning to give the correct prediction for a particular input sample.
In this work, we propose few shot learning model that targets few-shot regression tasks. Our model takes inspiration from the idea that the degree of freedom of F(x) can be significantly reduced when it is modeled linear combination of sparsifying basis functions. Thus, with few samples, we can estimate F (x)) The two primary components of our model are (i) the Basis Function Learner network which encodes the basis functions for the distribution of tasks, and (ii) the Weights Generator network which produces the appropriate weights given few labelled samples. We evaluate our model on the sinusoidal regression tasks and compare the performance to several meta-learning algorithms. We also evaluate our model on other regression tasks, namely the 1D heat equation tasks modeled by partial differential equations and the 21) Gaussian distribution tasks. Furthermore, we evaluate our model on image completion as 2D regression problem on the MNIST and CelebA data-sets, using only a small subset of known pixel values. To summarize, our contributions for this paper are:
We propose to address few shot regression by linear combination of set of sparsifying basis functions.
We propose to learn these (continuous) sparsifying basis functions from data. Traditionally, basis functions are hand-crafted (e.g. Fourier basis).
We perform experiments to evaluate our approach using sinusoidal, heat equation, 2D Gaussian tasks and MNIST/CellbA image completion tasks.

---

Under review as a conference paper at ICLR 2020
Weights Generator
P. V22
x
Mean
W
x
x
x K
Basis Function
W
Dos-Product
x
X
Learner
s
x K
Prediction
Training Samples
Test/Prediction Samples
Training Basis Functions
Test/Prediction Basis Functions
Training Labels
Weight Vector
Figure I: An overview of our model in meta-training. Our system learns the basis functions + that can result sparse representation for any task drawn from a certain task distribution. The basis functions are encoded in the Basis Function Learner network. The system produces predictions for a regression task by generating a weight vector, w for a novel task, using the Weights Generator network. The prediction is obtained by taking dot-product between the weight vector and learned basis functions.
2 RELATED WORK
Regression problems has long been topic of study in the machine learning and signal processing community (Myers & Myers, 1990: Specht 1991). Though similar to classification, regression estimates one multiple scalar values and is usually thought of as single task problem. single model is trained to only perform regression on only one task. Our model instead reformulates the regression problem as a few-shot learning problem. allowing for our model to be able perform regressions of tasks sampled from the same task distribution.
The success achieved by deep neural networks heavily relies on a large amount of data, especially labelled ones. As labelling data is time-consuming and labor-intensive, learning from limited labelled data is drawing more and more attention. A prominent approach meta learning. Meta learning, also referred as learning to learn, aims at learning adaptive model across different tasks. Meta learning has shown potential in style transfer (Zhang et al. 2019), visual navigation (Wortsman al. 2018), etc. Meta learning has also been applied to few-shot learning problems. which concerns models that can learn from prior experiences to adapt to new tasks. Lake et al. (2011) proposed the one-shot classification problem and introduced the Omniglot data set few-shot classification data set, similar to MNIST (LeCun 1998) for traditional classification. Since then, there has been a surge of meta learning methods striving to solve few-shot problems. Some meta learning approaches learn a similarity metric (Snell et al. 2017 Vinyals et al.] 2016 Koch et al.) 2015) between new test examples with few-shot training samples make the prediction. The similarity metric used here can be Euclidean distance, cosine similarity or more expressive metric learned by relation networks (Sung al., 2018). On the other hand, optimization based approaches learn how to optimize the model directly. |iinn et al. (2017) learned an optimal initialization of models for different tasks in the same distribution, which able to achieve good performance by simple gradient descent. |Rusu et al.) (2019) learned how to perform gradient descent in the latent space to adapt the model parameters more effectively. [Ravi & Larochelle| (2016) employed an LSTM to learn an optimization algorithm. Generative models are also proposed to overcome the limitations resulted from few-shot setting
(Zhang et al. 2018 Hariharan & Girshick 2017) Wang et al. 2018).
Few-shot regression tasks are used among various few-shot leaning methods (Finn et al. 2017 Rusu
cet al. 2019 et al. 2017). In most existing works, these experiment usually does not extend beyond the sinusoidal and linear regression tasks.
2

---

Under review as a conference paper at ICLR 2020
A prominent family of algorithms that tackles similar problem as few-shot regression is Neural Processes (Garnelo et al." 2018b at Kim et al. 2019). Neural Processes algorithms model the distributions of the outputs of regression functions using Deep Neural Networks given pairs of input-output pairs. Similar to Variational Autoencoders (Kingma & Welling 2013), Neural Processes employ a Bayesian approach in modelling the output distribution of regression function using an encoder-decoder architecture. Our model on the other hand employs a deterministic approach where we directly learn set of basis functions to model the output distribution. Our model also does not produce any latent vectors but instead produces predictions via dot product between the learned basis functions and weight vector. Our experiment results show that our model (based on sparse linear combination of basis functions) compares favorably to Neural Processes (based on conditional stochastic processes).
Our proposed sparse linear representation framework for few shot regression makes the few shot regression problem appears to be similar to another research problem called dictionary learning (DL) (Tosic & Frossard 201I), which focuses on learning dictionaries of atoms that provide efficient representations of some class of signals. However the differences between DL and our problem are significant: Our problems are continuous rather than discrete in DL, and only observe very small percentage of samples. Detailed comparison with DL is discussed in the appendix.
3 PROPOSED METHOD
3.1 PROBLEM FORMULATION
We first provide problem definition for few-shot regression. We aim at developing model that can rapidly regress to a variety of equations and functions based on only a few training samples. We assume that each equation
would like to regress is task T; sampled from distribution P(T). We train our model on a set of training tasks, Strain, and evaluate on a separate set of testing tasks, Stest- Unlike few-shot classification tasks, the tasks distribution p(T) is continuous for regression task in general. Each regression task is comprised of training samples Weriin and validation samples Deat, for both the training set Styain and testing Stest, Dirain is comprised of K training samples and labels IDerain {(rt y, ] 1...K} while Deat is comprised of N samples and labels Dea! {2=5 In == I...N.. The goal of few-shot regression is to regress the entire, continuous output range of the equation given only the few points as training set.
3.2 FEW-SHOT REGRESSION VIA LEARNING SPARSIFYING BASIS FUNCTIONS
Here we discuss our main idea. We would like to model the unknown function y = F(x) given only Dirain = ((ar, y = 1...K), K). With small K, e.g. K 10, this is an ill-posed task, as F(xr) can take any form. As stated before, we assume that each function we would like to regress is a task T drawn from an unknown distribution p(T).
To simplify discussion, we assume scalar input and scalar output. Our idea is to learn sparse representation of the unknown function F(x). so that a few samples {(x] ) = 1...K) } can provide adequate information to approximate the entire F(x). Specifically, we model the unknown function F(z) as a linear combination of a set of basis functions (x)
F (x) = Σ w.):(x)
(1)
Many handcrafted basis functions have been developed expand F(x). For example, the Maclaurin series expansion (Taylor series expansion at = 0) uses (z)( = {1, x, x², x2,x3, .?....:
F(x) wp + wix + W22x
If F(x) is a polynomial, 2 can be a sparse representation, i.e. only a few non-zero, significant wi. and most wi zero or near zero. However, if F(x) is sinusoid, it would require many terms to represent F(z) adequately, e.g.:
sin(x) Rs wir + wax" + w5x 8 + wys +... Warx
In 3 M is large and M > K. Given only K samples { {a s/ = 1...K. it is not adequate to determine (w:) and model the unknown function. On the other hand, if we use the Fourier basis
3

---

Under review as a conference paper at ICLR 2020
cos(2 clearly, we can obtain sparse representation: we can adequately approximate the sinusoid with only a few terms. Under Fourier basis, there are only few non-zero significant weights w;, and K samples are sufficient to estimate the significant Wi and approximate the function. Essentially, with a sparsifying basis (dx(x)) J, the degree of freedom of F(x)) can be significantly reduced when it is modeled using so that K samples can well estimate F())..
instead, ie., {0)(2)) (1, sin(x),
Our approach is to use the set of training tasks drawn from p(7) to learn {d)( that result in sparse representation for any lask drawn from p(T). The set of {0:(x)} is encoded in the Basis Function Learner Network that takes in and outputs ((x) |h (z) (.))),z) (2(x), Φar (x)]") In our framework, (z)) is the same for any task drawn from p(T). as it encodes the set of {x)) that
can sparsely represent any task from p(T). We further learn Weights Generator Network to map
the K training samples of a novel task to constant vector ur [11, W2, .... WM]" The unknown function is modeled as w (xx).
3.3 MODEL ARCHITECTURE
An overview of our model is depicted in Figure T Given a regression task T with Dtrain [(=f, v. = 1.. J,, the model is tasked to approximate the function across the entire output. The training samples, at E (Rd= first passed though the Basis Function Learner which is represented as network (2)),, parameterized by trainable parameters 0. The Basis Function Learner outputs set of learned basis functions in the form of a vector, ((x)) E Rdd,, where d& is the number of the basis functions
would like the Basis Function Learner to learn. We represent the Basis Function Learner as a series of fully connected layers followed by a ReLU non-linearity activation function (Nair& & Hinton 2010).
The set of learned basis functions ((x), together with the labels y E Rd are then passed into the Weights Generator. The Weights Generator, represented as network Ge (#(x)) yf),, with trainable parameters v/, takes the input (axt), yf and outputs weights vector wy for each training sample of a regression task. The final weights vector, w for task T is then obtained by taking a mean of the K weight vectors. The Weights Generator consists a series of B self attention blocks following by final fully connected layer to transform the output into the desired dimensions. We provide architecture details of Weights Generator network in the appendix.
The model is then applied to make prediction for any input æ. During meta-training, the validation set Deat { y In n 1...N} for a task T is given. The prediction produced by taking dot product between task-specific weights vector, w and the set of learned basis functions:
y Mired (o"))))))))
(4)
To train our model, we design a loss function L that consists of three terms. The first term is a mean-squared error between the validation set labels ' Dval and the predicted yored- We also add two penalty terms on the weights vector w generated for each task. The first penalty term is on the L.1 norm of the generated weight vectors. This is to encourage the learned weight vectors to be sparse in order to approximate the unknown function with a few significant basis functions. The second penalty term is on the L2 norm of the generated weights vector. This is used reduce the variance of the estimated weights as commonly used in regression (Zou & Hastic (2005)). The full loss function L is as follows:
La.e
1
Σ ME DDtl
(1/ U)rea)? 2 + Au > TTll + >eelw>
(5)
N
where X1 and A2 represents the weightage of the LI and 1.2 terms respectively. Note that, it turns out that our loss function for meta learning is is similar to that the Elastic Net Regression (Zou & Hastie 2005) with both LI and L2 regularization terms. However, the difference is significant: Instead of focusing on a single regression task as in (Zou & Hastie, 2005). we use this loss function to learn (i) the parameter 0 for the Basis Function Learner network, which encodes the sparsifying basis functions for any task drawn from a task distribution, and (ii) the parameter y for the Weight Generator network, which produces the weights for any novel task drawn from the same task distribution.
4

---

Under review as a conference paper at ICLR 2020
Table 1: Mean-- -eennSquarred Error results for the Sinusoidal Regression task as compared against other methods. Lower is better.
Method
5-shot
10-shot
20-shot
MAML (Finn al. 2017)
1.13 + 0.18
0.77 + 0.11
0.48 = 0.08
Meta-SGD Li et al.
2017)
0.90 = 0.16
0.53 + 0.09
0.31 = 0.05
EMAML (small) Yoon et al.
2018
0.885 + 0.117
0.615
0.371 + 0.048
EMAML (large)
0.783 0.101
0.537 0.079
0.307 = 0.040
BMAML (small) Yoon et al.
2018
0.927 = 0.116
0.735
0.459 = 0.058
BMAML (large)
0.878 0.108
0,675 0.094
0.442 = 0.055
NP (Garnelo et al.
2018b)
+:40=00200
0.561 + 0.234
0.421 + 0.088
CNP
(Garnelo al. 2018a)
0.910 + 0.234
0.630 + 0.222
0.393 = 0.145
ANP Kim et al.
2019)
0.488 = 0.188
0.216 + 0.082
0.095 = 0.068
Ours (small)
0.363 + 0.018
0.169 + 0.007
0.076 = 0.004
Ours (large)
0.199 + 0.010
0.062 0.003
0.027 = 0.002
4 EXPERIMENTS AND ANALYSIS
In this section we describe the experiments we ran and introduce the types of regression task used to evaluate our method. For all of our experiments, we set the learning rate to 0.001 and use the Adam Optimizer (Kingma& & Ba 2014) as the optimization method to preform stochastic gradient decent on our model. We implement all our models using the Tensorflow (Abadi et al., 2016) library. In the following subsections, decribe each of experiments in more detail. We include the experiments on the ID Heat Equation and 2D Gaussian regression tasks in the appendix.
4.1 1D REGRESSION
For all ID Regression tasks, the Basis Function Learner consists of two fully connected layers with 40 hidden units. For the loss function we set X = 0.001 and A2 = 0.0001.
Sinusoidal Regression, We first evaluate our model on the sinusoidal regression task which is few-shot regression task that is widely used by other few-shot learning methods as a few-shot learning task to evaluate their methods on (Finn et al. 2017 Li et al. 2017 Rusu et al. 2019). The target function is defined as y(z) = Awin(wr + b). where amplitude A, phase b, frequency w are the parameters of the function. We follow the setup exactly as in (Li et al. (017). We sample the each parameters uniformly from range A E [0.1,
[0, π.] and w E [0.8, 1.2). We train our model on tasks of batch size 4 and 60000 iterations for 5.10 and, 20 shot cases, where each training task contains K E (5, 10, 20} training samples and 10 validation samples. We compare our method against recent few-shot learning methods including Meta-SGD (Li et al., (017]. MAML (Finn et al.. 2017), EMAML .BMAMML (Yoon et al. 2018) and the Neural Processes family of methods including
E
Neural Processes (Garnelo et al. (018b) Conditional Neural Processes (Garnelo et al. (018a) and
Attentive Neural Processes (Kim et al. 2019). We use the officially released code for these three methods !
We show the results in l
We provide two variants our model in this experimental setup. The two models differ only in the size of the Weights Generator. For the "small" model the Weights Gener- ator consist of B = 1 self-attention blocks followed by a fully connected layer of 40 hidden units. The self-attention block con- sists of three parallel weight projections of 40 dimensions followed by fully con- nected layers of 80 and 40 hidden units respectively. The "large" model consists
Table 2: Mean-Squared Error results for the alternative
Sinusoidal Regression Task. Lower better.
Method
Alt. Sinusoidal 1000 tasks 10 shot
shot
EMAML
1.524 + 0.034
2.238 + 0.045
BMAML
1.412 + 0.033
2.157 + 0.049
Ours
0.918 + 0.051
2.389 + 0.103
Ours(Ensemble)
0.630 + 0.035
1.857 0.081
5

---

Under review as a conference paper at ICLR 2020
of B = self-attention blocks also followed by fully connected layer of 40 hidden units. Each self-attention block has weight projections of 64 dimensions followed by fully connected layers of 128 and 64 hidden units respectively. Both MAML and Meta-SGD. uses an architecture of 2 fully connected layers with 40 hidden units which is similar to the architecture of the Basis Learner network, though both Meta-SGD and MAML both have additional optimization for individual tasks. The Neural Process family of methods uses encoder archtecture of 4 fully connected layers with 128 hidden units and decoder architecture of 2 fully connected layers of 128 hidden units respectively which is more similar in architecture our larger model.
Similarly. we also compare our methods against two variants of EMAML and BMAML. The "small" model consist of 2 fully connected layers with 40 hidden units each while the "large" model consists of 5 fully connected layers with 40 hidden units each. This is to ensure fair comparison as both BMAMI. and EMAMI. lack a separate network to generate weight vectors but are ensemble methods that aggregate results from M, number of model instances. We set the number of model instances in BMAML and EMAML to 10.
Alternative Sinusoidal Regression We also evaluate our method on another version of the sinusoidal task as introduced by Yoon et al. (2018). The range of A remain the same while the range of b is increased to [0, 2x] and the range of w is increased to [0.5, 2.0]. An extra noise term, e is also added the function y(z). For noise e, we sample it from distribution N ~ (0, (0.0LA)²). We also fix the total number of our tasks used during training to 1000 as in (Yoon et al. 2018). For this experimental setup we also include ensemble version of our model where train 10 separate instance of our model on the same 1000 tasks and aggregate their results by taking a mean of the predictions. We evaluate our model for both 10 shot and 5 shot cases and show the mean-squared error results in Table 2 For this experimental setup, we calculate the mean-squared error from 10 randomly points from 1000 advanced sinusoidal tasks.
Our results show that our method outperforms all recent few-shot regression methods in sinusoidal tasks.
4.2 2D REGRESSION ON IMAGE DATA
We also tested our method on more challenging image data, as done in (Garnelo et al. 2018a b Kim et al.) 2019). We use MNIST (LeCun et al. 1998) and CelebA datasets (Liu et al. (015) here for qualitative and quantitative comparison. Each image can be regarded as a continuous function ʃ R + Rds, where dy = 1 if the image is gray-scale or or dy = 3 if it is RGB. The input x E 2 to I is the normalized coordinates of pixels and the output y E T is the normalized pixel value. The size of the images is 28 x 28 in MNIST and rescaled to 32 x 32 in CelebA. During meta-training, we randomly sample K points from 784(1024) pixels in one image as Iitain and another K points as Dval to form a regression task. In the meta-testing stage, the MSE is evaluated on 784(1024) - K pixels. (,.0000(12,,7000 images are used for meta- veta-training and 10,000 for meta-testing for MNIST(CelebAA dataset.
We compare our methods with NP family: CNP (arnelo al. 2018a), NP al. 2018b) and ANP (Kim et al.] 2019) for K = 50 and K = 100. Deeper network structure is adopted due to the complexity of regression on image data. Namely, we use 5 fully connected layers with 128 hidden units in Basis Function Learner and 3 attention blocks in Weight Generator for our method. The encoders and decoders in NP family are all MLPs including 4 fully connected layers with 128 hidden units. Thus, the comparison is fair in terms of network capacity. All the models are trained for 500 epochs with batch size 80. The MSE on 10,000 tasks from meta-testing set is reported with 95% confidence interval, shown in Table 3 The top results are highlighted. It can be observed that our method outperforms two of three NP methods and achieves MSE very close to most recent ANP. The outputs of regression effectiveness of our method in such challenging tasks. Note that ANP significantly improves upon NP and CNP using cross-attention, which can potentially be applied to our method as well.
CelebA image data are high-dimension predictions, which demonstrates the
Figure 2 shows the qualitative results for testing images. Red pixels in the images denote points in Dtrain- The comparison with methods from NP family is shown in the Figure 2a The regression by our method is clearly better than NP and CNP in 50-shot and 100-shot, which visually validates the quantitative results above. The qualitative results CelebA can be found in Figure 7 Appendix.

---

Under review as a conference paper at ICLR 2020
Table 3: Mean Square Error (x10-2) for 2D regression on image data.
MNIST
CelebA
Method
50-shot
100-shot
50-shot
100-shot
CNP
Garnelo al.
2018a)
4.13::0.033
3.23::0022
2.74:10.233
2.37=00019
NP
Garnelo et al.
2018b)
4.14
3.26:t0.024
2.51=#0.019
2.15=0.017
A
(im et al.
2019)
3.40::0.026
2.20::00017
2.13 =0.018
1.63=000115
Ours
3.65
2.431
2.39-10.021
1.93:10.015
GT
NP
CNP
ANP
Ours
GT
5=10
S=20
5=30
5=40
S=50
K=100 K=50
K=100 K=50
6
6
6
5
5
5
5
5
5
(a) Comparison with NP methods.
(b) Prediction with the first S largest weight parameters.
Figure Qualitative results on MNIST image data.
4.3 ANALYSIS ON BASIS FUNCTIONS
In this subsection we provide some deeper analysis on the basis functions that are learned by our method. In particular, we provide some further evidence our claim that our method learns set of sparsifying basis functions that correspond to the regression tasks that we would like to model. To demonstrate the sparsity of basis functions, we take only the largest weights in terms of [w] and their corresponding basis functions and illustrate the predicted regression function with the combination of only
weights and basis functions. We conduct this experiment on both the sinusoidal regression task and the more difficult image completion task and show these S-weights predictions in Figures 3 and 2b respectively.
The figures illustrate that method is able to produce a good prediction of the regression function with only a fraction of the full set learned basis function (40 for the sinusoidal task, 128 for the MNIST image completion task). This demonstrates the sparsity of (xx)) most of the prediction is carried out by just a small number of basis functions. This also demonstrates that our method is able force most of the information of F(x) be contained in a few terms. Therefore, using K samples to estimate the weights of these few important terms could achieve a good approximation of F(x).
4.4 ABLATION STUDIES
In this subsection we detail some ablation studies on our model to test the validity of certain design choices of our model. In particular we focus the effects of the addition of self-attention operations in the Weights Generator and also the effects of using different penalty terms on our loss function.
To test out the effects of adding the self- attention operations to our model, we conduct simple experiment where we replace the self attention operations in the self-attention block with just a single fully connected layer of equal dimensions as the self-artention weight projec- tion. Essentially, this reduces the Weights Gen- erator to be just series of fully connected layers with residual connections and layer normalization. We compare the simpler model performance on the sinusoidal regression task specified in Table with our original model and show the results in Table 4 The results show that adding the self-attention operations do improve our methods performance on the ID sinusoidal regression task.
Table Comparison of Models with and without Self- Alf-AAttntion on 10-shot Sinusoidal Regression
10-shot
Without Attention
0.163 + 0.003
With Attention
0.062 0.003
We also conducted experiments to test the effects of the different penalty terms on the the generated weights vector. In this ablation study, compared models trained using different variants of the
7

---

Under review as a conference paper at ICLR 2020
5 =
5= 10
Full Prediction
-5
5
-5
S-Weights
Full
Ground
Training Points
Prediction
Prediction
Figure 3: Top S-weights predictions for sinusoidal tasks
loss function we presented in Equation Similar to the previous study, we evaluate them on their performance the sinusoidal regression task specified in Table I The variants tested out are: (i) Loss function with only the L1-norm penalty term; (ii) Loss function with only the L2-norm penalty term (iii) Loss function with both L1 and L2-norm penalty terms. To demonstrate the sparsity of the weights vectors each variant, we also show the breakdown of the magnitude of the learned weight vectors over 100 sinusoidal tasks. We group the weight vectors into three groups : |w|| less than 0.02 to indicate weights that are near zero, |w| between 0.02 and and weights with magnitude more than We show the results the different variants Table 5 We also present histograms of the magnitude of the learned weight vectors in Figure 4
The results do show that the combination of both LI and L2 penalty terms do ultimately give the best performance for the sinusoidal regression task. In terms of sparsity, the model trained with only the LI loss term do gives the highest percentage of sparse weights though we found the model with both Li and L2 terms do give better performance while still maintaining relatively high percentage of near zero weights.
CONCLUSION
We propose a few-shot meta learning system that focuses exclusively regression tasks. Our
154
3550
261
3590
model is based on the idea of linear represen- tation of basis functions. We design Basis Function Learner network to encode the basis functions for the entire task distribution. We also design Weight generator network to gen- erate the weights from the K training samples of novel task drawn from the same task distri- bution. We show that our model has competitive performance in in various few short regression tasks.
3800
use
3230
330
28
23
Figure Histogram of magnitude of learned weight over 100 Sinusoidal tasks.
Table 5: Comparison of models trained using different penalty for 10-shot Sinusoidal Regression.
Penalty
10-shot
Percentage of Weight, w by Magnitude
w| < 0.02
0.02 <= |w|
1.0
[w] >= 1.0
LI only
0.083 + 0.004
62.45
32.925
4.625
L2 only
0.073 + 0.004
41.1
43.575
15.325
L1 L2
0.062 0.003
46.1
41.75
12.15
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint 1nXii 2016.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv: 1607.06450 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning (ICML), 2017.
Marta Gareelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference Machine Learning, pp. 1690-1699, 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural processes, 2018b.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In The IEEE International Conference Computer Vision (ICCV), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2016.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint WxXiv:1901.05700 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In International Conference Machine Learning (ICML) Deep Learning
Workshop, 2015.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. Proceedings of the Annual Meeting of the Cognitive Science Society, volume 33, 2011.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmistt/ 1998.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( 86(11):228--234, 1998.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. arXiv preprint arXiv: arXiv: 1707.09835 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Raymond H Myers and Raymond H Myers. Classical and modern regression with applications, volume 2. Duxbury press Belmont, CA, 1990.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference machine learning (ICML-10), pp. 807-814, 2010.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference Learning Representations (ICLR), 2016.

---

Under review as a conference paper at ICLR 2020
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski. Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference Learning Representations (ICLR), 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Donald Specht. A general regression neural network. IEEE transactions on neural networks, 2(6): 568- 576, 1991.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
I. Tosic and P. Frossard. Dictionary learning. IEEE Signal Processing Magazine, 28(2):27-38, March 2011.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In CVPR, 2018.
Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Learning to learn how to learn: Self-adaptive visual navigation using meta-learning. arXiv preprint arXiv:1812.0097) 2018.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian agnel-agi meta-learning. In Advances in Neural Information Processing Systems, pp. 7343-73533 2018.
Chi Zhang, Yixin Zhu, and Song-Chun Zhu. Metastyle: Three-way trade-off among speed, flexibility, and quality in neural style transfer. In The AAAI Conference on Artificial Intelligence (AAAI), 2019.
Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. Metagan: adversarial approach to few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2018
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the royal statistical society: series B (statistical methodology). 67(2):301-320 2005.
10

---

Under review as a conference paper at ICLR 2020
A
VISUALIZATION OF SINUSOIDAL BASIS FUNCTIONS
1.5
1.00
2.0
0.75
1.5
1.0
0.50
1.0
0.3
0.5
0.5
1.00
1.00
0.8
0.75
0.4
0.50
0.4
0.50
0.25
0.2
0.25
0.2
0.6
0.6
0.4
1.0
0.2
0.5
0.6
0.6
0.6
1.0
0.4
0.4
02
0.2
0.0
0.4
0.4
0.2
0.2
0.1
Figure 5: Illustration of all the non-zero basis functions that are learnt by our method on the sinusoidal regression task by order magnitude.
In this section we illustrate all of the individual non-zero basis functions learned by our model for the sinusoidal task. These functions are shown Figure Note that out of 40 of the basis functions, only 22 of the learned basis functions are non-zero functions, further demonstrating that indeed our method is forcing the model to learn a set of sparse functions to represent the tasks. Furthermore, be seen that the learned basis functions all correspond the different' "components" of sinusoidal function: most of the learned functions seem to represent possible peaks, or troughs if multiplied with negative weight at various regions of the input range whereas the top four basis function seem to model the more complicated periodic nature of the sinusoidal functions.
B ADDITIONAL ANALYSIS ON LEANED BASIS FUNCTIONS
Adding on to the experiments in Section 4.3, we also illustrate what happens when do the exact opposite. We take the prediction using the full set of weight vectors/basis function and study the effect of the prediction when we remove certain basis function from the prediction. Similar to the previous experiment,
remove the basis function by order of magnitude starting with the basis function with the largest corresponding [w]. we conduct this experiment on the sinusoidal regression task and illustrate the results in Figure
11

---

Under review as a conference paper at ICLR 2020
Full Prediction
Remove Basis
Remave 2 Basis
Remove 3 Basis
Remove 4 Basis
-5
-5
-5
Reduced Prediction
Full Prediction
Ground
Training Points
Figure 6: Reduced predictions for sinusoidal tasks
Similarly, this study also demonstrates the importance of certain basis functions as removing them caused the prediction the change drastically. In particular, notice that for sinusoidal task, removing just 4 of the most important basis functions resulted in a less accurate prediction than using just 10 of the most important basis functions.
C DETAILS ON THE WEIGHTS GENERATOR NETWORK ARCHITECTURE
Here provide more details on the architecture of the Weights Generator Network. As mentioned previously in Section 3.3. The Weights Generator Network consists of a series of self attention blocks followed by a final fully connected layer. We define self attention block such: An attention block consists of self attention operation on the input of self attention block. Following the self-attention operation, the resultant embedding is further passed through two fully connected layers. A residual connection (He et al. 2016) from the output of the self-attention operation to the output of the second fully connected layer. Finally, resultant embedding of the residual connection is then passed though layer normalization operation (Ba et al., 2016). Note that the input of the first self attention block will always be the input to the Weights Generator network, (& (xx)), yt whereas the inputs to subsequent attention blocks are the outputs of the previous attention block.
For the self-attention operation, the input transformed into query, key and value vectors though their respective weight projections. These query, key and value vectors, Q, K and V then go through a scaled dot-prood dot-product self-attention operation as introduced by (Vaswani et al. 2017):
QKT Vdk.
AU K, V) = =softman
V
(6)
D ADDITIONAL REGRESSION EXPERIMENTS
ID Heat Equation. We also evaluate method another 1D) Regression task, the 1D heat Equation task, we define it as such: Consider 1-dimensional rod of length I. with both its ends connected to heat sinks, i.e. the temperature of the ends will always be fixed at OK unless heat source is applied at the end. a constant point heat source is then applied to a random point on the rod such the the heat point source will always have temperature of 1.0K. We would like the model the temperature u(x,7 f) at each point of the rod a certain time after applying the heat source until the temperature achieves equilibrium throughout the rod. The temperature at each point after time is given by the heat equation:
du
8u 0.T
01
For experiments, set L to be 5 and randomly sample K points of range [0, 5] the heat equation curve. We fix the total number of tasks used during training to 1000 and evaluate our model on both 10 shot and 5 shot cases, similar to the experimental setup for the Advanced Sinusoidal tasks. We also compare our results to both EMAML and BMAML on this regrssion task and add an ensemble version of method for comparison. The results of our evaluation is presented in Table 6
2D Gaussian. We also evaluated our method on the for the 2D Gaussian regression tasks. For this task, we train our model to predict the probability distribution function of two-dimensional Gaussian
12

---

Under review as a conference paper at ICLR 2020
Table 6: Mean-Squared Error results for the ID Heat Equation Regression Task.
Method
1D Heat Equation 1000 tasks
10 shot
5 shot
EMAML FFin et al.
2017:
Yoon et al. 2018)
(1.02 + 0.03) x 10
1149::0.000 10
BMAML Yoon et al.
2018
(1.74 + 0.04) x 10
(1.81 = 0.04)
10
Ours
(6.32 + 0.53)
x 10
(8.30 + 0.56)
10 10-3
Ours(Ensemble)
5.13 = 0.13)
x 10 3
(7.09 = 0.16)
x 10
Table 7: Mean-Seaaaa Seann-Suarre Error results for the 2D Gaussian Regression Task.
Method
2D Gaussian 1000 tasks
10 shot
20 shot
50 shot
EMAML
(2.67 2.177:0.17 10
(2.44 0.13) x 10-3
(2.16 + 0.89) 10
BMAML
(2.26 0.15 x 10-3
(2.16 = 0.09) x 10-3
x 10 10-3 (7.83 + 0.96
(1.48 + 0.07)
Ours Ours(Ensemble)
(1.70 = 0.36)
x 10-3
(1.14
x 10-3
x 10
(1.46 0.11)
x 10-3
(0.97 + 0.10)
10
(6.09 + 0.67)
x 10
distribution. We train our model from Gaussian distribution task with mean ranging from (-2,-2) to (2, 2) and standard deviation of range [0.1, We fix the standard deviation to be of the same value in both directions. Similar to the heat equation, we use the same setup as the Advanced Sinusoidal task and compare our methods to EMAML and BMAML. We evaluate our model 10, 20 and 50 shot case. The results of our evaluation is presented in Table
Qualitative results CelebA datasets. We provide the qualitative results on CelebA datasets in Figure We note that the RGB images are complex 2D functions. We choose them to evaluate so that
the results more directly, not compare with image inpainting methods, which is also mentioned in (Garnelo et al. 2018ap. The results in Figure 7a are consistent with Figure The regression results from our method are visually better than NP and CNP. The predictions using first S largest weights are shown in Figure 7b The 2D image function is usually predicted with less than 50 weights, which suggests that the information of the 2D function is kept several terms.
E COMPARISON WITH DICTIONARY LEARNING
Our proposed sparse linear representation framework for few shot regression makes the few shot regression problem appears to be similar to another research problem called dictionary learning (DL), which focuses on learning dictionaries of atoms that provide efficient representations of some class of signals (Tosic & Frossard (01T). However the differences between DL and our problem are significant: Our problems are continuous rather than discrete as in DL, and we only observe a very small percentage of samples.
Specifically, for a given y E R², the goal DL is to learn the dictionary (n by M) for some sparse w:
y fw
In typical DL, the entire is given. Also, M > for an overcomplete dictionary (Figure 8).
In few shot regression, the goal is to predict the entire continuous function y = F(x). Therefore, viewing this as the setup in (7). n is infinite. Moreover, only a few (K) samples of is given: y F(x")). The locations of the given samples (a,²) are different for different y (different task). Therefore, our problem is significantly different and more difficult than DL. Typical DL algorithms solve 7 and return Φ. which is a n by M matrix of finite dimensions (the dictionary). In our setup, the basis matrix & has infinite entries, and Φ is encoded by the proposed Basis Function Leamnr network.
13

---

Under review as a conference paper at ICLR 2020
GT
NP
CNP
ANP
Ours
K-100 K-50
(a) Comparison with NP methods.
GT
S=10
2=
S=30
5=40 S=50
(-100 K-50
(b) Prediction with only the basis functions with the largest weights.
Figure 7: Qualitative results CelebA image data.
14

---

Under review as a conference paper at ICLR 2020
Dictionary Learning
Our method
Known
Unknown value
P
=
X
W
W
S
y
y
Figure 8: Comparing our method to Dictionary Learning
15


Reviews:
[{"review_id": "r1ggNMGw5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes an approach to few-shot regression based on\nlearning a sparse basis and a weight estimator network. The authors\nintroduce a sparsity inducing term in the loss to encourage sparse\nweight generation for tasks; these sparse coefficient vectors are then\nprojected onto the learned, task-dependent basis for\nregression. Experimental results are given on two synthetic regression\nproblems, with a comparison with the recent state-of-the-art.\n\nThis paper has some interesting ideas in it. However, it does have\nsome issues:\n\n 1. Clarity. There are several points of the proposed technique that\n    are not described clearly enough. For example, the diagram in\n    Figure 1 leads me to believe that the basis and weights generators\n    are independent (and thus not trained end-to-end). However, the\n    loss in eq. (5) seems (though it is not completely clear to me) to\n    depend on both networks (which is how I would expect things to\n    work). Also, the \"self attention blocks\" mentioned at several\n    points are never completely defines. And from the ablation study\n    is seems that the improvement form self-attention is the lion's\n    share of the overall improvement. I do not feel that it would be\n    easy to reproduce the results reported in this paper without\n    significant guesswork.\n\n 2. The experimental results are somewhat limited. The sinusoidal\n    regression problem is very artificial, as is the image regression\n    task. Focusing on regression is inherently limiting, but results\n    on more realistic regression problems would help establish more\n    clearly the significance of the contribution.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eefcvhtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a regression approach that, given a few training (support) samples of a regression task (input and desired output pairs), should be able to output the values of the target function on additional (query) inputs. The proposed method is to learn a set of basis functions (MLPs) and a weight generator that for a given support set predicts weights using which the basis functions are linearly combined to form the predicted regression function, which is later tested (using the MSE metric) w.r.t. the ground truth. The method is trained on a large collection of randomly sampled task from the target family and is tested on a separate set of random tasks. The experiments include: \n* sinusoidal wave prediction from a few samples\n* MNIST and CelebA inpainting from a set of known pixel values (in 28x28 and 32x32 resolution respectively)\n* additional experiments on heat equation and 2D Gaussian distribution task in Appendix\nThe experiments show that the proposed approach outperforms the other methods on the sinusoidal wave toy problem, and yet performs less good then than Kim et al. 2019 on MNIST and CelebA.\n\nI propose to reject the paper in its current form, and consider the following negative points for further improvement:\n\n1. The posed problem is not really few-shot learning, in (now classical) few-shot learning, such as few-shot classification on benchmarks such as miniImageNet, CUB, tieredImageNet, CIFAR-FS, FC100, etc. the meta-training is done on a disjoint set of categories and testing is done on a completely new set of categories unseen during training. The gap between disjoint visual categories is very large, and it does not come close to being tested on a different from training samples sinusoidal wave or different set of hidden pixels in inpainting on the same (seen during training) set of classes (where the basis function module could learn a set of basis functions for every class). In the proposed setting, I think a better definition would be \"learning a structured regression\" from a set of sample points to a function, and not few-shot regression.\nIf the authors would like to keep the \"few-shot flavour\", I would suggest re-formulating the experiments, and meta-train on some set of classes (e.g. inpainting over digits 0 to 4) and meta-test on a different set of classes (e.g. inpainting over digits 5 to 9). This partially holds for faces as they are all mostly different categories (different people), but in 32x32 resolution and MSE metric, I don't think they are sufficiently different.\n\n2. I would expect stronger results on the more realistic MNIST and CelebA experiments (although as suggested in 1. the setting there should be different), currently it does less well then existing method.\n\n3. An emerging important class of few-shot regression problems is few-shot object detection, where the bounding box coordinates of objects location need to be regressed. There are several papers and benchmarks in this space, and it will help the current paper to test on this challenging family of problems. Please see the following papers for benchmarks and settings:\n* LSTD: A Low-Shot Transfer Detector for Object Detection, Chen et al. 2018\n* RepMet: Representative-based metric learning for classification and one-shot object detection, Karlinsky et al. 2019\n* Few-shot Object Detection via Feature Reweighting, Kang et al. 2019", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1eXuCxRYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose using sparse adaptive basis function models for few shot regression. The basis functions and the corresponding weights are generated via respective networks whose parameters are shared across all tasks. Elastic net regularization is used to encourage task specific sparsity in the weights,  the idea being  that with only a small number of available training examples, learning a sparse basis is  easier than learning a dense basis with many more parameters. The method is validated on both synthetic data and on image completion tasks. \n\nI am leaning towards rejecting the paper. 1) Although, the paper is well written and easy to follow the technical contributions of the paper are limited. Adaptive basis functions and their sparse combinations are decades old ideas.  While the application of these ideas to few shot regression does appear to be novel,  this combination don’t seem to provide an obvious improvement over existing alternatives. 2) The empirical evidence presented is rather limited, the proposed approach only seems to outperform competitors on the synthetic sinusoidal regression experiments. Lack of strong empirical performance along with the limited novelty \n\nDetailed comments and questions: \n+ The approach naturally extends to few shot classification problems once the MSE loss in Equation 5 is replaced with an appropriate cross entropy loss. Was this considered and is the approach competitive on few shot classification problems.\n\n+ The empirical section could be significantly improved. \n\n -  Diverse synthetic data: I don’t see the value in presenting two sets of synthetic sinusoid regression experiments.  It would be better to replace the alternative sinusoid task with qualitatively different tasks. This would help the  audience ascertain whether the favorable performance demonstrated in Table 1 generalizes beyond sinusoidal signals. \n\n - Comparisons:  1.  Why are comparisons to neural processes missing in the additional synthetic experiments presented in the supplement and from Table 2? This is a particularly egregious omission since on the real data (attentive) neural processes outperform the proposed method. \n2. The ensemble approach seems to improve on the individual model significantly in Table 2. Why was this not considered for the image completion experiments? The authors would also do well to more clearly describe how the ensembling was performed. \n\n+ I find it curious that the basis functions are restricted to be non-negative. The description in 3.3 suggests that the basis function network outputs are passed through a ReLU.  What was the rational behind this design choice?\n\nMinor:\n Why are both ANP and “Ours” highlighted in Table 3, when ANP clearly outperforms and does not appear to be within statistical noise of “Ours”.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that this paper is not ready for publication.
Decision: Reject