Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper describes an approach to few-shot regression based on
learning a sparse basis and a weight estimator network. The authors
introduce a sparsity inducing term in the loss to encourage sparse
weight generation for tasks; these sparse coefficient vectors are then
projected onto the learned, task-dependent basis for
regression. Experimental results are given on two synthetic regression
problems, with a comparison with the recent state-of-the-art.

This paper has some interesting ideas in it. However, it does have
some issues:

 1. Clarity. There are several points of the proposed technique that
    are not described clearly enough. For example, the diagram in
    Figure 1 leads me to believe that the basis and weights generators
    are independent (and thus not trained end-to-end). However, the
    loss in eq. (5) seems (though it is not completely clear to me) to
    depend on both networks (which is how I would expect things to
    work). Also, the "self attention blocks" mentioned at several
    points are never completely defines. And from the ablation study
    is seems that the improvement form self-attention is the lion's
    share of the overall improvement. I do not feel that it would be
    easy to reproduce the results reported in this paper without
    significant guesswork.

 2. The experimental results are somewhat limited. The sinusoidal
    regression problem is very artificial, as is the image regression
    task. Focusing on regression is inherently limiting, but results
    on more realistic regression problems would help establish more
    clearly the significance of the contribution.


review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The authors propose using sparse adaptive basis function models for few shot regression. The basis functions and the corresponding weights are generated via respective networks whose parameters are shared across all tasks. Elastic net regularization is used to encourage task specific sparsity in the weights,  the idea being  that with only a small number of available training examples, learning a sparse basis is  easier than learning a dense basis with many more parameters. The method is validated on both synthetic data and on image completion tasks. 

I am leaning towards rejecting the paper. 1) Although, the paper is well written and easy to follow the technical contributions of the paper are limited. Adaptive basis functions and their sparse combinations are decades old ideas.  While the application of these ideas to few shot regression does appear to be novel,  this combination don’t seem to provide an obvious improvement over existing alternatives. 2) The empirical evidence presented is rather limited, the proposed approach only seems to outperform competitors on the synthetic sinusoidal regression experiments. Lack of strong empirical performance along with the limited novelty 

Detailed comments and questions: 
+ The approach naturally extends to few shot classification problems once the MSE loss in Equation 5 is replaced with an appropriate cross entropy loss. Was this considered and is the approach competitive on few shot classification problems.

+ The empirical section could be significantly improved. 

 -  Diverse synthetic data: I don’t see the value in presenting two sets of synthetic sinusoid regression experiments.  It would be better to replace the alternative sinusoid task with qualitatively different tasks. This would help the  audience ascertain whether the favorable performance demonstrated in Table 1 generalizes beyond sinusoidal signals. 

 - Comparisons:  1.  Why are comparisons to neural processes missing in the additional synthetic experiments presented in the supplement and from Table 2? This is a particularly egregious omission since on the real data (attentive) neural processes outperform the proposed method. 
2. The ensemble approach seems to improve on the individual model significantly in Table 2. Why was this not considered for the image completion experiments? The authors would also do well to more clearly describe how the ensembling was performed. 

+ I find it curious that the basis functions are restricted to be non-negative. The description in 3.3 suggests that the basis function network outputs are passed through a ReLU.  What was the rational behind this design choice?

Minor:
 Why are both ANP and “Ours” highlighted in Table 3, when ANP clearly outperforms and does not appear to be within statistical noise of “Ours”.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper proposes a regression approach that, given a few training (support) samples of a regression task (input and desired output pairs), should be able to output the values of the target function on additional (query) inputs. The proposed method is to learn a set of basis functions (MLPs) and a weight generator that for a given support set predicts weights using which the basis functions are linearly combined to form the predicted regression function, which is later tested (using the MSE metric) w.r.t. the ground truth. The method is trained on a large collection of randomly sampled task from the target family and is tested on a separate set of random tasks. The experiments include: 
* sinusoidal wave prediction from a few samples
* MNIST and CelebA inpainting from a set of known pixel values (in 28x28 and 32x32 resolution respectively)
* additional experiments on heat equation and 2D Gaussian distribution task in Appendix
The experiments show that the proposed approach outperforms the other methods on the sinusoidal wave toy problem, and yet performs less good then than Kim et al. 2019 on MNIST and CelebA.

I propose to reject the paper in its current form, and consider the following negative points for further improvement:

1. The posed problem is not really few-shot learning, in (now classical) few-shot learning, such as few-shot classification on benchmarks such as miniImageNet, CUB, tieredImageNet, CIFAR-FS, FC100, etc. the meta-training is done on a disjoint set of categories and testing is done on a completely new set of categories unseen during training. The gap between disjoint visual categories is very large, and it does not come close to being tested on a different from training samples sinusoidal wave or different set of hidden pixels in inpainting on the same (seen during training) set of classes (where the basis function module could learn a set of basis functions for every class). In the proposed setting, I think a better definition would be "learning a structured regression" from a set of sample points to a function, and not few-shot regression.
If the authors would like to keep the "few-shot flavour", I would suggest re-formulating the experiments, and meta-train on some set of classes (e.g. inpainting over digits 0 to 4) and meta-test on a different set of classes (e.g. inpainting over digits 5 to 9). This partially holds for faces as they are all mostly different categories (different people), but in 32x32 resolution and MSE metric, I don't think they are sufficiently different.

2. I would expect stronger results on the more realistic MNIST and CelebA experiments (although as suggested in 1. the setting there should be different), currently it does less well then existing method.

3. An emerging important class of few-shot regression problems is few-shot object detection, where the bounding box coordinates of objects location need to be regressed. There are several papers and benchmarks in this space, and it will help the current paper to test on this challenging family of problems. Please see the following papers for benchmarks and settings:
* LSTD: A Low-Shot Transfer Detector for Object Detection, Chen et al. 2018
* RepMet: Representative-based metric learning for classification and one-shot object detection, Karlinsky et al. 2019
* Few-shot Object Detection via Feature Reweighting, Kang et al. 2019
review_title: Official Blind Review #2

Meta-review: All reviewers agree that this paper is not ready for publication.
Decision: Reject