Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper focuses on the topic of meta-learning for few-shot learning and explores kernel approximation with random fourier features for this problem. The authors propose to learn adaptive kernels by meta variational random features, and evaluate their approach on different few-shot learning tasks, comparing it against recent meta-learning algorithms. 

The paper is well-motivated and well-written. On page 8, the authors mention related works that were not included for comparison because they rely on pre-trained embeddings or large-scale deep architectures. It would have been interesting to see the difference in performance.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper studies meta-learning problem with few-shot learning settings. The author proposes a learn each task predictive function via the form of random Fourier features, where the kernel is jointly learned from all tasks. The novel part is the parametrization of inference network using LSTM such that the random feature samples of t-th task conditional depending on all previous task 1,...,t-1, which is an interesting way of modeling kernel spectral distribution. The experiment results show improvement of the proposed methods compared to SoTA meta learning algorithms.

In general, the writing of the paper is clear, and the proposed method is interesting and novel. However, there are parts missing in the experiment setting.  I would love to increase my score if the author could address the following questions/comments:
(1) How do you choose the meta prior distribution? It should be a basic kernel family such as RBF Gaussian or mixture of RBF?
(2) In Table 1 and Table 2, the benefit of using LSTM only gives very marginal improvement over w/o LSTM. Are the results statistically significant? 
(3) The experiment missed the simple kernel learning baseline, such as kernel alignment [1] and its variants [2]. If using these task-independent way to do kernel learning, what’s their performance compared to you proposed method?  
(4) When learning the RFF spectral distribution using LSTM over a sequence of tasks, does the order of task matter?  


[1] Learning kernels with random features, NIPS 2016.
[2] Implicit kernel learning, AISTATS 2019.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a meta-learning framework for learning adaptive kernels using a meta-learner. For representing kernels, the paper learns a variational posterior for the kernel features, by maximizing the Evidence lower Bound. Furthermore, to plug the kernel learning into the meta-learning framework, they let the variational feature posterior to condition on the current support set for adapting and to use a modified LSTM network for accumulating information. Empirically, they compare the proposed MetaVRF with multiple baselines in the standard fewshot classification benchmarks and demonstrate superior performance. They also illustrate that their adaptively-learnt Fourier feature outperforms the standard variational Fourier features.

Strengths, 
1, The idea of learning kernels in meta-learning is interesting. In fact, learning a kernel is equivalent to learning a distance between objects. If a reasonable distance between objects can be learnt, using the corresponding kernel should be able to achieve superior performance even if the kernel doesn't adapt in each episode. 
2, The proposed method achieves competitive performances. In particular, Figure 5 shows how the performance changes when the test-shot and test-way are varied. It seems surprising that the MetaVRF achieves >90% accuracy for 100-way test when trained on only 5-way 5-shot.

Weakness,
1, The notations in the paper are not well presented. (a) In eq(2), the formula $alpha^t=\Lambda(\Phi^t(x), y)$ is not exact, cuz $\alpha^t$ should depend on the whole support set $S^t$ while $(x,y)$ is only one instance in $S^t$.  (b) In eq(11), the variational posterior $q(w | S^t, w^{1:t-1})$ is not exact either. Because $w^{1:t-1}$ are random variables, they cannot be observed and cannot be conditioned on. Similar issues also exist in the caption of Figure 1.
2, The paper doesn't introduce what is the likelihood $log p(y| x, S, w)$. It is unclear how the kernel regression is adopted in classification. 
3, The meta-prior $p(w| x, S)$ depends on the feature of the query point, which doesn't seem to be a common practice in variational inference. It would be beneficial if the authors could explain this and probably validate it empirically.
4, The motivations of the modified LSTM should be clarified more. Does the paper remove h_t in LSTM for removing short-term memory ? Why $\hat{c}_t$ only depends on $e^t$ instead of $[e^t, c^{t-1}]$ ? 
5, The paper compares with multiple competitive baselines. However, the settings for the baselines should be better presented. For example, it is strange that the numbers of SNAIL are different with the numbers in their paper. And SNAIL on Omniglot is not reported. Furthermore, another competitive method TADAM (Oreshkin et. al., 2019) should also be compared with.
review_title: Official Blind Review #1

Meta-review: The paper looks at meta learning using random Fourier features for kernel approximations. The idea is to learn adaptive kernels by inferring Fourier bases from related tasks that can be used for the new task. A key insight of the paper is to use an LSTM to share knowledge across tasks.

The paper tackles an interesting problem, and the idea to use a meta learning setting for transfer learning within a kernel setting is quite interesting. It may be worthwhile relating this work to this paper by Titsias et al. (https://arxiv.org/abs/1901.11356), which looks at a slightly different setting (continual learning with Gaussian processes, where information is shared through inducing variables).

Having read the paper, I have some comments/questions:
1. log-likelihood should be called log-marginal likelihood (wherever the ELBO shows up)
2. The derivation of the ELBO confuses me (section 3.1). First, I don't know whether this ELBO is at training time or at test time. If it was at training time, then I agree with Reviewer #1 in the sense that $p(\omega)$ should not depend on either $x$ or $\mathcal {S}$. If it is at test time, the log-likelihood term should not depend on $\mathcal{S}$ (which is the training set), because $\mathcal S$ is taken care of by $p(\omega|\mathcal S)$. However, critically, $p(\omega|\mathcal S)$ should not depend on $x$. I agree with Reviewer #1 that this part is confusing, and the authors' response has not helped me to diffuse this confusion (e.g., priors should not be conditioned on any data).
3. The tasks are indirectly represented by a set of basis functions, which are represented by $\omega^t$ for task $t$. In the paper, these tasks are then inferred using variational inference and an LSTM. It may be worthwhile relating this to the latent-variable approach by Saemundsson et al. (http://auai.org/uai2018/proceedings/papers/235.pdf) for meta learning. 
4. The expression "meta ELBO" is inappropriate. This is a simple ELBO, nothing meta about it. If we think of the tasks as latent variables (which the paper also states), this ELBO in equation (9) is a vanilla ELBO that is used in variational inference.
5. For the LSTM, does it make a difference how the tasks are ordered?
6. Experiments: Figure 3 clearly needs error bars, and MSEs need to be reported with error bars as well; 
6a) Figures 4 and 5 need error bars.
6b) Error bars should also be based on different random initializations of the learning procedure to evaluate the robustness of the methods (use at least 20 random seeds). I don't think any of the results is based on more than one random seed (at least I could not find any statement regarding this).
7. Table 1 and 2: The highlighting in bold is unclear. If it is supposed to highlight the best methods, then the highlighting is dishonest in the sense that methods, which perform similarly, are not highlighted. For example, in Table 1, VERSA or MetaVRF (w/o LSTM) could be highlighted for all tasks because the error bars are so huge (similar in Table 2).
8. One of the things I'm missing completely is a discussion about computational demand: How efficiently can we train the model, and how long does it take to make predictions? It would be great to have some discussion about this in the paper and relate this to other approaches. 
9. The paper evaluates also the effect of having an LSTM that correlates tasks in the posterior. The analysis shows that there are some marginal gains, but none of the is statistically significant. I would have liked to see much more analysis of the effect/benefit of the LSTM.

Summary: The paper addresses an interesting problem. However, I have reservations regarding some theoretical bits and regarding the quality of the evaluation. Given that this paper also exceeds the 8 pages (default) limit, we are supposed to ask for higher acceptance standards than for an 8-pages paper. Hence, putting everything together, I recommend to reject this paper.
Decision: Reject