Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper describes a method that builds upon the work of Wang et al. It meta-learns to hallucinate additional samples for few-shot learning for classification tasks. Their two main insights of this paper are to propose a soft-precision term which compares the classifiers' predictions for all classes other than the ground truth class for both a few-shot training set and the hallucinated set and b) to introduce the idea of applying direct early supervision in the feature space in which the hallucination is conducted in addition to in the classifier embedding space. This allows for stronger supervision and prevents the hallucinated samples from not being representative of the classes. The authors show small, but consistent improvement in performance on two benchmarks: ImageNet and miniImageNet with two different network architectures versus various state-of-the-art meta-learning algorithms with and without hallucination. The authors have adequately cited and reviewed the existing literature. They have also conducted many experiments (both in the main paper and in the supplementary material) to show the superior performance of their approach versus the existing ones. Furthermore their ablation studies both for the type of soft precision loss and for their various individual losses are quite nice and thorough. 

Overall the contribution of this paper is incremental over Wang et al and is mainly in the introduction of their new loss terms to regularize the hallucination process. This is clearly evident from Table 2 (comparing rows 1 and 3), where much of the performance gain is attained by including the l_learner^cls term versus the collaborative loss term (comparing row 1 and row 4).

Furthermore, I would like the author to answer the following two questions:

1. The authors claim that their method is general applicable to all meta-learning methods and can be combined with them. Yet, the meta learning methods that they apply it to: prototypical networks, prototypical matching networks and cosine classifiers are all metric-learning-based meta-learning techniques. I would like the authors to outline the procedure (and preferably also show experiments) for applying their proposed technique to meta-learning based techniques that do not involve learning a metric-embedding space and instead learn the learning procedure via nested optimization, e.g. MAML and its variants.

2. In Table 4, I would like to see the results of PNM w/G or in other words the results of (Wang et al, 2018)'s method in comparison to the authors' proposed method.

3. The authors make no attempt to solve the problem of hallucinating examples for regression tasks. This is fine as it is perhaps outside he scope of their current work. However, I would like the authors to fully qualify their claims everywhere in the paper and restrict the contribution of their work to classification tasks only.


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a general meta-learning with hallucination framework called PECAN. It is model-agnostic and can be combined with any meta-learning models to consistent boost their few-shot learning performance. 

There are two key points for the proposed model. On the one hand, the authors introduce a novel precision-inducing loss which encourages the hallucinator to generate examples so that a classifier trained on them makes predictions similar to the one trained on a large amount of real examples. On the other hand, the authors introduce a collaborative objective for the hallucinator as early supervision, which directly facilitates the generation process and improves the cooperation between the hallucinator and the learner.

On the whole, the paper is well-written, and the proposed idea is novel and interesting.

I have some following major concerns about the paper:
(1) In Figure 2, the authors first sample the training set S^*_{train}, which contains n^* examples for each of the m classes, and then they randomly sample n examples per class, and obtain a subset S_{train}. Why not generate the S_{train} directly and then measure your precision-inducing loss over the real set S_{train} and S^G_{train}? I hope the authors explain it in their paper.
(2) For Function 2 in the paper, why compute the cosine distance on the probability vectors that are obtained by removing the logit for ground-truth label in original probability distributions? Could we compute the distance on the probability vectors that contains the logit for ground-truth label? I hope the authors explain it in detail.
(3) As far as I know, there are some latest work on few-shot learning in 2019, especially the work “Few-shot Learning via Saliency-guided Hallucination of Samples” and “Edge-Labeling Graph Neural Network for Few-shot Learning”. I hope the authors can compare with these two methods to further demonstrate the effectiveness of the proposed model.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors address few-shot learning via a precise collaborative hallucinator. In particular, they follow the framework of (Wang et al., 2018), and introduce two kinds of training regularization. The soft precision-inducing loss follows the spirit of adversarial learning, by using knowledge distillation. Additionally, a collaborative objective is introduced as middle supervision to enhance the learning capacity of hallucinator. 

Here are some comments:
1. The novelty is relatively limited. The idea of hallucinating has been mainly introduced in (Wang et al., 2018). The soft precision-inducing loss is a straightforward extension of knowledge distillation (Hinton et al., 2015). 

2. It is not quite clear about how to use the collaborative objective on the hallucinator. Fig.2 and the text in ' Collaboration between hallucinator and learner ' (page 5) are not quite informative. Especially, how to perform the soft precision-inducing loss (l_hal^pre) for hallucinator?
review_title: Official Blind Review #2

Meta-review: This paper describes a new approach to meta-learning with generating new useful examples.

The reviewers liked the paper but overall felt that the paper is not ready for publication as it stands.

Rejection is recommended. 
Decision: Reject