Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper addresses the issue of meta-learning in a transductive learning setting. That is, it aims to learn a model from multiple tasks and make it generalise to a new task in order to solve it efficiently. In the transductive setting, the query set (i.e., containing the unlabeled test data) of the new task is taken into account when learning the model. 

This paper takes the empirical Bayes approach to meta-learning. In order to utilise the test data that do not access to groundtruth labels, it proposes to use synthetic gradient to implement the tranductive learning. A multi-layer perceptron network is used to systhesize the gradient. Theoretical analysis is conducted to demonstrate the generalization capability of the proposed model and reveal its connection to the information bottleneck principle in the literature of neural networks. 

Overall, this is a well organised and nicely presented work. The idea on how to utilise the unlabeled test data to realise tranductive learning is novel; the analysis is thorough; and experimental study is provided to show the effectiveness of the proposed method. Meanwhile, this work can address the following issues:

1. The first paragraph on page 5, which describes the key step of syntheising gradient, can be made clearer; 
2. In the experimental study, Table 1 compares various methods with the proposed one. It will be helpful to clearly indicate for each method in comparison whether/how it also utilises the query set. This will give more context in interpreting the comparison results;
3. The advantage of the proposed method seems to diminish quickly from 1-shot to 5-shot settings. Does this mean in the case of 5 (or more)-shot setting, considering unlabeled test data with the proposed method could even adversely affect the meta-learning performance? Please comment. 
4. Since the proposed method works in a tranductive manner, it is presumed that the whole model needs to be retrained/updated once a new set of query data (e.g., for the same task or another new task) is given? In other words, how does the trained model generalise to unseen unlabeled test data? Please provide some discussion on this issue. 
5. Finally, how is the computational complexity of training the proposed EB model? 


review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The authors propose a method for transductive few-shot learning. The method is derived by taking a Bayesian perspective and recasting meta-learning as amortized variational inference, showing that results in a transductive scheme, and then using maml-style approximation of the inference (i.e., based on truncated stochastic gradient). While the idea of the paper seems intuitive, I find the writing quite confusing throughout (see my comments below) and I believe it must be improved before publishing the paper. Regarding the empirical evaluation, the results on standard benchmarks (miniImageNet and CIFAR-FS) seem reasonably strong; however, I would not call it "significantly outperform previous state-of-the-art" (as the authors claim in the abstract), since really all the top methods are in the same ballpark (the provided 95% CI overlap).


Comments:

1. In Eq. 2, if the task-specific losses are arbitrary, the whole construction is no longer a log-likelihood but rather just a loss. The authors also denote the distribution over the meta-training datasets as p_\psi(d_t), where d_t includes both inputs and targets. However, the concrete instantiations of the framework use discriminative models. Adjusting and clarifying the notation would improve the paper.

2. The way KL divergence is used in Eq. 5 is misleading since the arguments are two distributions over different sets of random variables. I would recommend keeping expected log conditional probability as a separate term (which is common in the literature).

3. Relatedly, going from ELBO to amortized VI (Eqs. 4-6) is a standard widely used VAE trick, so the derivation itself is not that informative. On the other, it would be great to include the inductive inference scheme mentioned right before Eq. 7 and compare it side-by-side with the standard amortized VI (Eq. 6). The way that part is presented now leaves the reader to derive all the details on their own.

4. Sec. 3, paragraph 1: While the original neural processes tend to underfit the data as pointed by the authors, more recent versions of the model such as attentive neural processes might work well, and perhaps worth mentioning.

5. Difference between Eq. 7 and 8 -- I believe I am misunderstanding this, but the updates look identical to me up to KL between q_\theta and a prior p_\psi. How exactly does \phi(x_t) parametrize the optimization process? I don't see how it enters into the equations. Generally, I feel deriving the method through a Bayesian perspective is quite confusing (as it is presented now) and way less clear than what is illustrated in Figure 1c.

6. Re: theoretical analysis -- it seems like the more than half a page spent on defining what generalization error is in the given setup (where all the definitions are quite standard), but then the discussion of the result, discussion of specific cases, connection to the information bottleneck bounds are all compressed down to in 1-2 sentences. This makes the "analysis" section really useless. Exemplifying the result of Thm. 1 and significantly elaborating the discussion would improve the paper.


Minor:

- The paragraph before Theorem 1: "Proposition" -> "Theorem"
- [UPD] Figure 3: titles, labels, ticks are all too small to be readable.

---------

Thanks to the authors for a detailed response. Most of my points have been addressed satisfactorily. I've updated my review accordingly.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors argue for the importance of transduction in few-shot learning approaches, and augment the empirical Bayes objective established in previous work (estimating the hyperpior $\psi$ in a frequentist fashion) so as to take advantage of the unlabelled test data.
Since the label is, by definition, unknown, a synthetic gradient is instead learned to parameterize the gradient of the variational posterior and tailor it for the transductive setting. The authors provide an analysis of the generalization ability of EB and term their method _synthetic information bottleneck_ (SIB) owing to parallels between its objective of that of the information bottleneck. SIB is tested on two standard few-shot image benchmarks in CIFAR-FS and MiniImageNet, exhibiting impressive performance and outperforming, in some cases by some margin, the baseline methods, in the 1- and 5-shot settings alike, in addition to a synthetic dataset.

The paper is technically sound and, for the most part, well-written, with the authors' motivations and explanation of the method conceived quite straightforwardly. The basic premise of using an estimated gradient to fashion an inductive few-shot learning algorithm into a transductive one is a natural and evidently potent one. The paper does, however, at times feel to be disjointed and, to an extent, lacking in focus. The respective advantages of EB and the repurposing of synthetic gradients to enable the transductive approach are clear to me, yet while they might indeed be obviously complementary, what is not obvious is the necessity of the pairing: it seems there is nothing prohibiting the substitution of the gradient for a learned surrogated just as well under a deterministic meta-initialization framework. As such, despite sporting impressive results on the image datasets, I am not convinced about how truly novel the method is when viewed as a whole. 

On a similar note, while the theoretical analysis provided in section 4 was not unappreciated, and indeed it was interesting to see such a connection between EB with information theory rigorously made, it does feel a little out of place within the main text, especially since it is not specific to the transductive setting considered, nor even to the meta-learning setting more broadly. Rather, more experiments, per Appendix C, highlighting the importance of transduction and therein the synthetic gradients and its formulation would be welcome. Indeed, it is stated that an additional loss for training the synthetic gradient network to mimic the true gradient is unnecessary; while I agree with this conclusion, I likewise do not think it would hurt to explore use of the more explicit formulation.

Considering the authors argue specifically for the importance of transduction in the zero-shot learning regime, I think it would be reasonable to expect experiments substantiating this, and the strength of their method in this regard, on non-synthetic datasets. As far as the toy problem is concerned, I am slightly confused as to the choice of baseline, both in the regard to its training procedure and as to why this was deemed more suitable than one purposed for few-shot learning, so that we might go beyond simple verification to getting some initial sense for the performance of SIB. Moreover, it is not clear from the description as to how $\lambda$ is implemented here. As it stands, Section 5, for me, offers little in the way of valuable insights. The experiments section on the whole, results aside, feels somewhat rushed; the synthetic gradients being a potential limiting factor for instance feels "tacked on" and seems to warrant more than just a passing comment.

Minor errors
- Page 7: the "to" in "let $p_\psi(w)$ to be a Gaussian" is extraneous
- Page 8: "split" not "splitted".
- Further down on the same page, "scale" in "scale each feature dimension" should be singular and Drop" is misspelled as "dropp".
- Page 9: "We report results **with using** learning rate..."
- _Equation 17_ includes the indicator function $k \neq i$ but $i$ is not defined within the context.

EDIT: changed score
review_title: Official Blind Review #2

Meta-review: Three reviewers have assessed this paper and they have scored it 6/6/6 after rebuttal. Nonetheless, the reviewers have raised a number of criticisms and the authors are encouraged to resolve them for the camera-ready submission.
Decision: Accept (Poster)