Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper considers the problem of unsupervised meta-learning, where the goal is to generate tasks for meta-training without supervision. Whereas previous work generated training and test sets from the unlabeled set for meta-training via augmentations (UMTRA) or unsupervised clustering of embeddings (CACTUs), this paper considers doing this using interpolation of the latent space representations produced by generative models. Specifically, the idea is to first train a generative model on the unlabeled set and then produce training and test sets for meta-training by decoding the interpolation of latent space representations of multiple examples from the original unlabeled set. The authors discuss 3 specific ways to produce examples for the train and test sets for meta-training in this way. They first select an anchor example from the unlabeled set that will be representative of one class in the dataset. Then, the 3 methods involve:
1. Adding noise: adding noise to the latent space representation of the anchor example to produce examples that make up the training and test set for a single class.
2. Random out-of-class sample: selecting another example from the unlabeled set and finding new examples for the class by interpolating between the anchor's and this example's representations. 
3. With Other Classes' samples: same as (2) but instead of picking another random example, the example considered is another anchor example that was used to represent a different class.

The authors evaluate their method by considering 3 few-shot learning benchmarks: (1) Omniglot; (2) CelebA few-shot identity recognition; and (3) CelebA attribute prediction. On these 3 benchmarks, they show that their method performs favorably compared to UMTRA and CACTUs.

Pros
* This paper proposes a simple yet very interesting idea for performing unsupervised meta-learning that is unique compared to previous work.
* The big benefit of this method compared to previous work is that it seems to require less tweaking per dataset. Whereas previous methods required tuning per dataset (for example, in UMTRA, selecting which augmentations to use for a specific dataset), this method requires training a generative model on the unlabeled set and using the learned latent space interpolation (where the generative model can directly learn properties of the specific dataset that can be used during interpolation). However, there are still some choices to make in terms of hyperparameters for latent space interpolation).

Cons
* I have minor concerns about the Mini-ImageNet experiments. Firstly, why are Mini-ImageNet experiments not discussed in the main paper but in the supplementary material? The difficulties of using Mini-ImageNet are mentioned, in that it is difficult to train a generative model on this more complex dataset using the limited examples in Mini-ImageNet. Thus, I believe it is a good idea to use whole ImageNet dataset as the unlabeled set, as the authors did, and the results for the method seem favorable compared to previous work. So, I think it's useful to include these results to show how this method extends to more complex images? I think a note just needs to be added that the unlabeled set this method uses is much larger than the ones used in previous work for the Mini-ImageNet comparison but I don't view this as a big negative because the data required for training is still unlabeled.
* Details of the CelebA few-shot identity recognition benchmark seem to be lacking? I don't see this benchmark mentioned in previous work so I was curious how the metrics for other methods (such as CACTUs) were generated given that this benchmark wasn't discussed in those papers? I think more details about this benchmark would be useful in general. Additionally, some citations to previous results on this benchmark would also be helpful.
review_title: Interesting work for unsupervised meta-learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work proposes LASIUM (LAtent-Space Interpolation Unsupervised Meta-learning), a novel method of generating meta-tasks in an unsupervised way. It leverages recent advances in generative models. Specifically, it uses the fact that datapoints that are close in latent space are more likely to be from the same class. The method interpolates between samples in latent space, choosing interpolation ratios.

I find this work compelling: it is built upon simple and intuitive ideas and leverages a rapidly growing research area. Its ability to use any pre-trained generative model to perform unsupervised meta-learning is promising.

The paper proposes three different variants of LASIUM, each of them compelling ideas: LASIUM -N, -RO, -OC. However, the paper does not comment on, which is best or what tradeoffs exist between the three methods. RO seems to win on Omniglot, but the paper only shows N for the hardest task (CelebA).

It would have been helpful to see how the generative model affects unsupervised meta-learning performance. For example, is being good at image generation always better for meta-learning? We cannot know from the experiments presented here because it only tries one model for each setting.

Minor comments
- the underlined phrases scattered throughout the paper helped comprehension. Nice touch.
- (very minor) page 4: inconsistent indentation for paragraph "Generating in-class...". Other bolded paragraphs are not indented.
- Table 3: Missing digit (probably 0) on MLP with dropout.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: #### Summary
- This paper proposes LASIUM, a framework for constructing tasks for unsupervised meta-learning for image classification via the use of a generative model on the unlabeled training data. Similar to Hsu et al. (2019), distance in the latent space of an unsupervised learner is taken to correspond to class-level information. Uniquely, LASIUM leverages generative modeling by proposing simple interpolation-based schemes to populate classes with latents, which are decoded by the generative model to produce image samples for the meta-learning stage. Three specific variants of LASIUM are proposed. The results indicate that the approach is competitive with respect to prior methods for the Omniglot and CelebA datasets, but curiously, results on miniImageNet are not presented.

#### Strengths
- The idea of using generative modeling and interpolation for unsupervised meta-learning for image classification is significantly different from prior works, which use data augmentation (Antoniou and Storkey 2019, Khodadadeh et al. 2019) and latent-space clustering (Hsu et al., 2019).

- Reasonable variations of the core idea of latent-space interpolation and generation are explored and assessed. This includes a VAE-specific variant which additionally leverages the encoding capability of a VAE.

- The experiments involve a reasonable variety of generative models, and the protocol seems to closely follow that of prior work.

#### Weaknesses
- Critically, experiments based on the miniImageNet dataset are completely missing. This is a benchmark that is present in all three prior works in unsupervised meta-learning for image classification ((Antoniou and Storkey 2019, Hsu et al. 2019, Khodadadeh et al. 2019), as well as the vast majority of meta-learning works. The diversity present in miniImageNet makes it a benchmark that assesses significantly different model capabilities than Omniglot or CelebA, and hence makes it an indispensable part of a comprehensive empirical evaluation.

- Unfortunately, there is a fair amount of overclaiming. "The proposed family of algorithms generate pairs
of in-class and out-of-class samples from the latent space in a principled way" -- the authors do not go beyond high-level intuition in justifying the latent interpolation schemes for task generation. "Beyond the increase in performance, we must note that the competing approaches use more domain specific knowledge (in case of UMTRA augmentations, in case of CACTUs, learned clustering)." -- how does CACTUs use more domain-specific knowledge than LASIUM? Both methods rely on an unsupervised pre-training stage to "organize" the unlabeled dataset into a latent space. 

- The use of generative models introduces a significant potential weakness: the performance of the downstream meta-learning is now additionally dependent on sample fidelity. CACTUs sidesteps this issue by only using samples from the original dataset, and AAL and UMTRA rely on carefully defined image transformations that do not compromise sample quality. This is touched upon in the discussion and described as a "limitation", but this key issue is not explored further.

- There are significant clarity issues with the discussion section. "A conceptual difference between clustering and LASIUM" -- the difference is really between CACTUs and LASIUM; clustering doesn't entail encoding by itself. "The former is shared with clustering based unsupervised learning" -- what is clustering based unsupervised learning? Do you mean CACTUs? Also, do you mean "latter" instead of "former"? "Interestingly, training of these feature extractors is based on augmentations like random crop and random flips (Caron et al. (2018)) or auto encoders (Berthelot et al. (2019))" -- what precisely is interesting about this? Do you mean to make an explicit connection between CACTUs, UMTRA/AAL, and LASIUM here? 

#### Recommendation
- I currently recommend rejection (4). The weaknesses outlined above outweigh the strengths. 

#### Questions
- Why are results on miniImageNet not included?

- Under what conditions can we expect LASIUM to be a favorable choice over CACTUs or UMTRA/AAL, if any?

- "Note that meta-learning algorithms are not sensitive to a small fraction of bad training examples in the meta-training phase (e.g. see Hsu et al. (2019); Khodadadehet al. (2019))." -- How do these works demonstrate this?

#### Minor suggestions
- There are minor typos, e.g. "proGAN" instead of "ProGAN".

------------------------------------ Post-rebuttal comments ------------------------------------

With the authors' answers, the inclusion of miniImageNet results in the main text, and revisions to the writing, my concerns have largely been addressed and I have increased my rating from a 4 to a 6.

I have a few further suggestions to make: first, make it clear in Table 4 via captioning and/or markings that UMTRA and LASIUM results for miniImageNet depend on the use of full unlabeled ImageNet data to train the AutoAugment and BigBiGAN models, respectively, while the CACTUs results do not. This results in an unfair comparison, and naturally raises the question of how LASIUM would do if we used a generative model trained only on the meta-training split of miniImageNet, which would be more in line with the protocol used for Omniglot and CelebA. It seems fair to assume that it would do worse than the current LASIUM results in Table 4, and probably the CACTUs results as well. This relates back to my point in the original review about sample fidelity: miniImageNet has the most diversity among the datasets considered, and it is hard for generative models to capture this diversity given a relatively small dataset. It would be good for this to be conveyed to the reader.

The usage of the extraneous ImageNet data for UMTRA and LASIUM does not conform to the definition of unsupervised meta-learning proposed in Hsu et al. (2019) and adopted in this work. Some discussion of the problem assumption may be warranted: in what practical circumstances would extraneous, relevant, unlabeled data be available? And when it is, why would one not use all of the data (e.g. the entirety of unlabeled ImageNet) to do unsupervised meta-learning and/or unsupervised representation learning, like in Table 12 of Hsu et al. (2019)?

Overall, despite the unfair miniImageNet protocol, I still advocate for weak acceptance as the method does show competitive results for domains that are more suited to generative modeling. For the sake of clarity for readers, though, I strongly encourage the authors to implement my additional recommendations.
review_title: OK, but miniImageNet results are missing.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a novel method for generating few-shot learning tasks from unlabeled data. The basic idea is to use a GAN or VAE to generate in-class (close in latent space) and out-of-class (far away in latent space) pairs of image data by linear interpolation in the latent space of the chosen generative model. A new in-class sample is generated by interpolating between two out-of-class samples in latent space with the interpolation weight chosen such that the generated sample is close to one of the out-of-class samples. The method is evaluated on common few-shot learning datasets with good results.

**Pros:**  
- The concept of generating few-shot learning tasks from an unlabeled dataset by performing interpolation in the latent space of a GAN or VAE is innovative and promising.
- The experiments demonstrate that the proposed method is as good as or superior to competitive unsupervised methods in basic scenarios. 
- The paper is very well written, and the ideas are clearly presented.
- The diagrams are very helpful to understanding the basic concepts.

**Concerns:**  
(1) The experiments are basic and do not explore potential limitations of the proposed method:
- All the experiments were performed with a low, fixed value of way equal to 5 and a fixed number of shots on datasets with homogeneous content. It would be interesting to see the results of using LASIUM on the more challenging meta-dataset [1] which uses tasks of variable way and shot on 10 diverse datasets. In a brief examination of the source code supplied in the supplementary material, it seems that some experimentation on meta-dataset may have been done. Any reason that those results were not presented?
- In particular, some of the datasets within meta-dataset (e.g. Fungi has a large number of classes of mushrooms that differ only subtly) require very fine-grained discrimination. It would be interesting to evaluate LASIUM in a higher way (> 25), fine-grained classification setting where classes are likely to be close in the latent space of the generator.

(2) In section 2, it says: “Our proposed approach, LASIUM, takes advantage of generative models trained on the specific domain to create the in-class and out-of-class pairs of meta-training data.” However, the goal of a few-shot learning system after meta-training is that the system should be able to make predictions on unseen input data, potentially from a different domain. As pointed out in (1) above, LASIUM has only been evaluated on a single domain. How would LASIUM handle a setting with very diverse tasks? Would more than one pre-trained GAN be needed to generate the examples? Or perhaps a GAN that has be trained on a large quantity of diverse data?

(3) LASIUM uses the latent space of a GAN or VAE to linearly interpolate between examples to generate few-shot learning training data. Were any ablation studies performed to ascertain that this is the best space for generating data? As a counter example, [3] uses linear interpolation in the raw data domain to augment training data in the supervised setting (i.e. they do not evaluate the method in the unsupervised setting) with particularly good results. The authors of [3] also consider doing the interpolation in an embedded feature space. Linear interpolation in alternative data spaces would make for a potentially interesting baseline or ablation study.

(4) Some experiment details are missing:  
- The main text (including Algorithm 1) makes no mention of using labeled data. However, in Section 4.2 it says: “For instance, the 95.29% accuracy of LASIUM-RO-GAN-MAML was obtained with only 25 labels, while the supervised approaches used 25,000.” Please elaborate on the use of labels in the various methods. Are some labels used to establish “anchor vectors”? How do the results change if you use no labels (i.e. fully unsupervised as opposed to semi-supervised)? Was labeled data also used by the competitive unsupervised approaches (i.e. CACTUs, UMTRA)?
- No detail was given on how the transfer learning results were obtained (i.e. learning rates, number of iterations, regularization, augmentation, etc.). Specifically, the concern is that the transfer learning results seem to be quite low, while [2] shows that supervised transfer learning can outperform supervised few-shot learning methods. Also, the transfer learning results for miniImageNet seem to be missing.
- What values for $\alpha$ were used in the experiments? In table 6 in the supplementary material, there is a small ablation study for selecting a good value $\alpha$ for the Omniglot experiments. It would be good to see this done for a greater range of $\alpha$ and for datasets other than Omniglot. The key question is to know how difficult it is to set $\alpha$ for each setting.
- What values are used for the rejection sampling threshold $\epsilon$ for the various methods? How sensitive is this parameter?
- For the sake of reproducibility, details on how all the baseline experiments were carried out would be beneficial.
 
(5) Some details are missing from Algorithm 1 including: a) how the unlabeled dataset gets encoded to the latent space in the VAE case and what role the unlabeled dataset plays in the GAN case; and b) the required hyperparameters for the sampling strategy (i.e. $\alpha$, $\sigma$, $\epsilon$).

**References:**  
[1] Triantafillou, Eleni, et al. "Meta-dataset: A dataset of datasets for learning to learn from few examples." arXiv preprint arXiv:1903.03096 (2019).  
[2] Tian, Yonglong, et al. "Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?." arXiv preprint arXiv:2003.11539 (2020).  
[3] Zhang, Hongyi, et al. "mixup: Beyond empirical risk minimization." arXiv preprint arXiv:1710.09412 (2017).

review_title: Review of "Unsupervised Meta-Learning through Latent-Space Interpolation in Generative Models"

Meta-review: This paper addresses a method for generating meta-tasks via latent space interpolation using a generative model, trained on the unlabeled dataset, to solve the unsupervised meta-learning. The method seems to be sound, but it lacks MiniImage-Net experiments, which is the main concern raised by most of reviewers. During the author responses, new empirical results on MiniImage-Net were added. During the discussion period with reviewers, I communicated with the reviewer with most negative comments. He/she was not fully satisfied, claiming that the protocol used for obtaining new MinImage-Net results was slightly unfair. It is suspected that the authors used a generative model trained on the ImageNet training set (of which miniImageNet is a subset) for LASIUM and  did not present results from only using miniImageNet meta-training data because it was not competitive with prior work. It should be clarified in the final paper. However, we arrived at the consensus that the paper is worth being presented. 


Decision: Accept (Poster)