Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents a method for Bayesian meta-learning. This method combines a NN feature extractor with a Gaussian Process on top. The GP kernel is linear. The information bottleneck is used to motivate a choice of approximate posterior. Using MAML to adapt the NN feature extractor weights improves performance of the composite model for few-shot learning.

---
Things I liked:
- This paper was interesting and on an important topic. 
- The information bottleneck seems some to fix some issues with variational inference takes on Bayesian meta learning (e.g. Amortized Bayesian Meta-Learning, Ravi & Beatson, 2019). The VI approach in that paper doesn't distinguish train/support and validation/query sets, and doesn't motivate the weight beta which is placed on the prior, while the information bottleneck does.
- The particular information bottleneck motivated posterior used with the GP is not specific to classification, regression, or other particular structure of prediction problem.
- I liked that the paper had plots of performance vs number of data points, as opposed to only reporting error on the full set of training tasks
- The proposed method leads to a bump in accuracy over MAML for augmented omniglot

---
There are potential areas of concern which I would like the authors to respond to. If the concerns are sufficiently addressed (or turn out to be due to my misunderstanding), I could be willing to significantly increase my score.

Main concerns:
1. What is the objective of the proposed method? Is it to get better test error than MAML, or to quantify uncertainty, or something else? If the point is to get better test error, it seems unfair to compare only to vanilla MAML given the considerable amount of literature and number of improvements in the years since. To build a case for an advantage in terms of accuracy, the paper should compare to a wider range of stronger/more recent baselines (e.g. "Bayesian Few-Shot Classification with One-vs-Each
Pólya-Gamma Augmented Gaussian Processes", Snell & Zemel 2020, does a good job of this).
2. If the motivation for the current proposed method is more than just improving the test error on these benchmarks, this motivation should be clearly explained and/or measured. The method proposed in the paper adds a lot of complexity on top of MAML and variants, so I think the paper needs to build a strong case and clarify exactly which settings this method is preferable for.
3.  One paper the authors could compare to and cite is "Meta-learning with differentiable closed-form solvers", Bertinetto et al 2018, which -- as with the current paper -- combines NN features with a closed-form linear model, and which significantly outperforms MAML.
4. The authors could also compare to the other papers they cite in Section 4, e.g. Snell & Zemel 2020 significantly outperform all the baselines and the proposed method from the current paper's results.
5. Can the authors discuss the lack of data augmentation used for Augmented Omniglot for the GP methods? It strikes me that the data augmentation is likely part of what makes Augmented Omniglot hard. In this case, removing the data augmentation for the GP methods would be giving the GP methods an unfair advantage, which could explain almost all of the GP methods' superiority on this task. A fair comparison should also compare to MAML(+variants) without data aug, and/or GP methods with data aug
6. Can the authors comment on the choice of a linear kernel for the GP? This surprised me, as for regression using a GP with a linear kernel is the same thing as doing Bayesian linear regression. Bayesian linear regression can be done exactly in O(N) time instead of O(N^3) for linear-kernel GP, so it seems strange not to use the LR formulation. It also seems surprising to have to jump through hoops to use an information bottleneck-motivated posterior in a case where we could do exact Bayesian inference cheaply. Of course, in a classification setting one needs to use approximations like Polya-Gamma augmentation, variational inference, or similar.
7. Why only show accuracy vs data points (fig 2 right) for GP not GP-MAML, and why only for augmented omniglot? This is an interesting plot which I would like to see more of in few-shot learning papers.

---
Other points of feedback:
1. I think that whenever the GP method is referred to (title, abstract, text, results tables, plots...), it should be made clear that this is a composition of a neural network feature extractor and a GP. Yes, the NN feature extractor can be seen as part of a very particular choice of kernel for the GP, and this is explained in Section 3. However, someone skimming the paper might not realize the existence of this NN at all, which would make the results very surprising. I think the paper should endeavor to give readers the correct impression on a skim. 
2. The set up in Section 2 (and "Stochastic MAML") is extremely similar to Amortized Bayesian Meta-Learning, Ravi & Beatson, 2019. As mentioned, the current submission fixes some issues with that paper, but due the similarity it wouldn't hurt to cite them.
3. The paper should explain the Augmented Omniglot benchmark better in the text (or at least refer the reader to the appendix, where it is explained). Flennerhag et al who introduced the benchmark did not use the term "Augmented Omniglot", so simply citing that paper when the term is mentioned is insufficient and left me confused before I found the discussion in the appendix, although I do think this is a good name for the benchmark.
4. How was the NN+GP method trained? I imagine this was done by backpropogating through the GP fitting procedure to get the gradient of the query-set error for training tasks (or of the IB objective for training tasks) and using this to optimize the NN weights. It would be nice to have this spelled out, as well as how this gradient was computed (whether it is simple or required some non-trivial tricks).
review_title: Interesting but not sure of advantages of proposed method vs. earlier methods

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper formulates meta-learning using the information bottleneck. The method develops a meta-learning algorithm based on Gaussian Processes, which they also interpret as a memory-based algorithm. They further describe an extension that combines the GP-based method and MAML.

The paper seems to lack focus. It consists of two orthogonal components: (1) applying the information bottleneck to meta-learning and (2) a GP-based meta-learning algorithm. Each of these components is a separate meta-learning algorithm: (1) alone can extend MAML to "stochastic MAML" ($\beta > 0$) and (2) alone is "GP" with hyperparameter $\beta=0$. The paper does not make a sufficient case for why these two components should be used in tandem (e.g. GP w/ $\beta > 0$), nor does it perform ablation experiments that each positively contributes to some measure of meta-learning performance.

The interpretation of MAML as a special case of information bottleneck (section 2.2) is not particularly novel: as the paper mentions in section 4, any previous works (LLAMA, probabilistic MAML, bayesian MAML, iMAML) have built on the interpretation of the initial parameter of MAML as encoding a prior distribution over parameters, and the gradient step as posterior inference. The fact that information bottleneck with $\beta=0$ reduces to standard learning is well-known.

The experimental results are mixed, and the paper doesn't give much of an interpretation of these results. GP and GP+MAML outperform MAML by an order of magnitude on the sinusoid regression problem but is outperformed by MAML in two of the five classification tasks considered. The experiments in the appendix are similarly mixed. GP wins on Augmented Omniglot but loses on ImageNet variants. Is the GP approach vastly superior on only regression problems, or is the difference just due to tuning and the fact that sinusoid regression is a toy problem? 

Given the relation to other probabilistic extensions of MAML, perhaps comparing against at least one of them would have made experiments more informative.

The experiments about the efficiency regarding the amount of data (Figure 2 right) are promising, but this claim would be more significant if evaluated on multiple tasks.

minor

There is a link formatting issue btw pages 2 and 3
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

The paper proposed variational approximations to the information bottleneck objective functions for meta-learning.
The authors then provided three different settings using their variational loss functions, namely SMAML, GP, and GP + MAML.
The authors' motivations for these three settings were to study the effect of stochastic gradient-based method, non-parametric method, and the combination of gradient-based method with non-parametric method.

Reason for the score:

I find the paper interesting for establishing a variational information theoretic objective function for meta-learning.
I also enjoyed the comparisons among purely gradient-based method with non-parametric method and their hybrid.
Yet, I did find some details where the motivations are a bit difficult to understand. It would be great if the authors could
clarify them.

Pro:
- The authors provided clear derivations for their variational objective functions.
- The authors provided clear kernel setups for their GP models.
- The authors provided evaluations for their three settings on both regression and classification tasks.
- The paper is easy to follow and I did not find typos.

Con / Questions:
- For the objective function in Eq.1, is there a particular reason for assuming conditional independence for $q(Z| D^{t}) $?
- For Eq.5 and Eq.6, are the models of $p_\theta (D_i^v | Z_i)$ and $p_\theta (Z_i)$ related in some ways? It seems that they are sharing the parameters $\theta$. 
- I did not find Section 3.2's motivation very convincing. The authors mentioned that the hope for combining MAML with GP is to reduce inner steps for MAML and possibly achieve better results. Yet, in Figure 4, it seems that pure GP is basically on-par with GP+MAML and outperforms GP+MAML when more inner steps are introduced for MAML. Could the authors help me with some insights for this observation?
- It seems that SMAML performed worse than MAML consistently for regression tasks and occasionally better for classification tasks. I observe that SMAML differ from MAML in two aspects, being stochastic and having the extra KL regularizer controlled by $\beta$. Are the worsened performance due to being stochastic or too much regularization from the KL term?
- On the classification tasks, it seems that GP models perform better than the standard MAML mostly when $K$ is large. It is difficult to conclude that the results are better thanks to GP being non-parametric or the variational loss function being better. This is also somewhat true for the regression tasks because we know GP is smooth and interpolates better.

-----------------------------------------------------------------
Post Rebuttal:

Many thanks for the authors to update their original paper addressing some of my questions and concerns.
Unfortunately, I still think that some aspects could be better analysed,
it is not crystal clear to me if the improvements come from the GP or their proposed variational information
theoretic function. I am keeping my original score.
review_title: Interesting ways of combining information theory, variational inference, and Bayesian non-parametric for meta-learning.

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Summary

The paper derives a meta-learning framework based on the information bottleneck principle. By adapting the variational approximation proposed in [1] to the meta-learning setting, the authors come up with a tractable objective that generalises both gradient based and memory meta-learning methods. Based on this framework, the authors proposed a new memory based meta-learning algorithm by using a GP with a deep kernel and an extension that combines this memory based method with MAML. The authors show that the method outperforms MAML in several standard meta-learning  tasks, especially in regression and many shots classification problems.

## Comments

The motivation of the paper is really clear and the paper is extremely well-written. 

From the technical perspective, the paper has two main contributions:

1. The development of a general meta-learning framework based on the information bottleneck that generalizes gradient and memory based meta-Learning
2. Two new methods that resort to GPs with deep kernels to implement the encoder in the aforementioned framework

Regarding the framework, the extension of the variational information bottleneck to the meta-learning setting is not particularly challenging being a straightforward extension of [1]. However, it is an interesting observation that this extension subsumes gradient based methods (by using parametric encodings) and memory based ones (by using nonparametric encodings). One of the things that it is missing is that the authors failed to mention that the connection between meta-learning and the information bottleneck principle had been previously explored in the literature (see [2]). While the setting is slightly different, it would make sense for the authors to address how the two papers are related, especially since the GP method proposed by the authors can be used in the transductive settings as well. 

Regarding the new methods, I think that the use of a GP encoder with a deep kernel is interesting but again not particularly challenging from the technical perspective. However, I think the combination of MAML + GP it is a very interesting observation that can open new research avenues.

The weakest part of the paper is the experiments. The authors use MAML as the unique baseline ignoring more up to date meta-learning methods which undermines the message of the paper. It would be really interesting to see how the method generalises and compares against other state of the art methods. For example, there has been several methods that try to leverage uncertainty into MAML (e.g. [3]). This would be an interesting comparison since it would help to disentangle the impact of a parametric/nonparametric encoder versus probabilistic/deterministic encoders. In addition, there are several recent methods specifically developed for transductive meta-learning that would be interesting to include in the comparison (e.g. [4]). Finally, I would appreciate a more in depth analysis of why the combined approach works better or worse than the GP one in different settings (see some questions below).

## Questions/Minors

* In section 2.2 the authors cast MAML under their framework. Two do so, they assume Z_i=\psi_i leading to p_\theta(\mathcal{D}_i^v|\psi_i) and p_\theta(\psi_i). However, the second approximation is irrelevant since further on they assume \beta=0.
* To build the stochastic extension of MAML, the authors use a stochastic encoder. However, in addition they use \beta \neq 0. It would be nice to see an ablation study to see the contribution of these separate extensions (there is an ablation study in D.3 but it is not specifically addressing this and it is not clear what is the model they use for the study). Also it would be interesting a small discussion about the importance of tying the variances of the prior and the encoder
* In section 3.1 it should be mention that it is a degenerate GP.
* Toward the end of section 3.2 a bunch of simplifications are done in order to come up with the final version of the combined algorithm: \beta to \beta_f and \beta_\psi and \beta_psi=0 which leads to a deterministic encoder and remove the need of defining a prior. However, these simplifications goes in the opposite direction of the extensions made to come up with the stochastic version of MAML. It would be great if the authors could elaborate on why that is the case and the contribution to the final performance of each of this simplifications.

## References

[1] Deep variational information bottleneck
[2] Empirical Bayes Transductive Meta-Learning with Synthetic Gradients
[3] Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
[4] Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning


review_title: Review of Information Theoretic Meta Learning with Gaussian Processes:  Interesting idea. Experimental part is weak.

Meta-review: Information bottleneck is a well-known principle that is used for clustering, dimensionality reduction, and recently deep learning. It finds a compressed representation of input X while retaining most information on the response Y. This paper addresses an attempt to interpret the meta-learning using the information bottleneck. In addition, a GP-based meta-learning method is also proposed. 
The topic itself is interesting without any doubt. However, most of reviewers have serious concerns about this work, which is summarized below. First of all, two components of this paper (IB and GP-based meta-learning) do not provide a coherent message.  While the IB interpretation is emphasized in the beginning of this paper, the main point seems to that GP-based methods can be more data efficient than gradient-based meta learning. There does not much point to GP+MAML or IB interpretation of MAML.  Experiments are not strong enough, although a few ones are added during the author responses. During the discussion with reviewers, no one support this work, so I do not have choice but to suggest rejection.

Decision: Reject