Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: There exist metrics for measuring the quality and consistency of standard generative models, but these are not applicable to the one-shot setting. Therefore, the authors of this paper introduce and quantify a diversity-recognizability space where single shot generated samples can be evaluated. Furthermore, these two axes make it straight-forward to compare the generated samples to samples from a human model (a brain). Extensive experiments establish the solidity of these measures for the Omniglot dataset.
strengths_and_weaknesses: This is a strikingly solid paper, which presents a clear and novel idea. The result on how GAN-based architectures and VAE-based approaches place in different parts of the diversity-recognizability scale is highly interesting. Each experiment is well-motivated (Section 3), and their results are thoroughly transparent and well discussed. The diversity-recognizability space also presents a very appealing way to think about evaluating generative models, well situated in the psychophysics and cognitive science literature, to the best of my knowledge. Relevant architectures and methods are used both for the feature extraction as part of the metrics, as well as for the one-shot generation. It is difficult to find something to remark on. The one thing would be the fact that the method is run on a single dataset, but in my view this should be left for future work, since a very clear and useful framework is introduced here for anyone to pick up – and the experiments are extensive even for this one dataset. Thus, I do not count this as a weakness. Last, the paper is highly readable and well-structured. I believe that it can be of large value to the community.
questions: 1.It is mentioned on lines 272-274 that the number of attentional steps is less imporatnt than the number of context samples; how do you see the relevance of $\beta$ relative to the number of context samples? 

2. How relevant is context for the Omniglot dataset? Visibly, it does play a role in the experiments, but I am curious how you interpret it.

3. In the last paragraph of the conclusions, you place the most importance on creativity and the diversity part of your introduced metric. Does this mean that you think that the field should move away from GAN-based models when it comes to human-like generated data?

limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the problem of few-shot generation on the Omniglot dataset, with a focus of benchmarking existing algorithms against humans. They claim that existing metrics for generative models aren’t appropriate for evaluating one-shot generalization, and propose a different framework for this purpose. Their framework is based on two aspects: diversity and recognizability. In both cases, measuring these aspects is dependent upon a trained model. In the case of diversity, a feature extractor is used to measure intra-class variability (standard deviation of produced samples in the feature space around a ‘prototype’), and a SimCLR encoder is used for this. For recognizability, they make use of a one-shot classification model, and the accuracy of that model on generated samples is the recognizability measure. They use a Prototypical Network for this. They investigate where VAE-based and GAN-based models fall in this evaluation framework, and how this is affected by various hyperparameters. Due to the setup of the Omniglot dataset, it is also possible to compare against humans (Omniglot contains human-drawn characters which can be thought of as being the result of a human generative process). They found that the neural statistician-based VAE is the closest (on average) to humans in the diversity-recognizability space. 
strengths_and_weaknesses: # Strengths:
[+] The paper is clearly written for the most part (see a couple exceptions in the detailed comments below) \
[+] Drawing connections between humans and different generative models is interesting  \
[+] The proposed framework is well-grounded, in that it draws inspiration from psychophysics work

# Weaknesses:
[-] Limited scope \
[-] Poorly motivated \
[-] Limited significance in the current form \


Detailed comments:
================

Clarity
=====
In Figure 4, it would be clearer to actually show the size of the context set, e.g. via the size of the circle, or color or something else. Currently I assume the direction of the arrow is giving a proxy for this but this is not as informative. 

In Figure 5, in the caption: ‘Bigger squares correspond to the model showing the smallest distance to humans’. This is unclear as it sounds like the bigger the square, the smaller the distance. But these squares for the different models all have the same size and instead it’s their (y-axis) position that captures the distance. Should rephrase to ‘the squares correspond to…’ (i.e. remove the word bigger), if my understanding is correct.

As a high-level comment, it would have been useful to give some more details about the models used. For example, it is hard to understand exactly what effect we expect the context size to have on VAE-NS without a good enough understanding of that model.


Motivation and significance
=====================
To motivate their evaluation framework, the authors claim that existing metrics don’t work well if the training and testing samples are too dissimilar. In Omniglot though, this isn’t really the case. There have been several studies that show that in fact, feature reuse is sufficient for good few-shot learning performance on simple datasets like Omniglot (see [1] for an example in the context of MAML, [2] for a more general example), and in fact it was also demonstrated that good performance can even be achieved on new Omniglot classes without using the support labels at test time [3], which also hints at the simplicity of this generalization challenge. So, is this claim just a hypothesis, or is it based on some evidence on the failure of existing metrics for one-shot generation evaluation on Omniglot? Especially in the weak generalization setting considered here, this might not be an issue.

Further, for measuring recognizability, it wasn’t clear to me why we need to use a one-shot model as the ‘critic’ for whether the generated samples can be accurately classified. Why can’t we use an ‘oracle’ here that was trained on more shots from the new classes? One might argue that this is okay since this model is used just for evaluation purposes (though perhaps we do want an *example-level* split where e.g. this oracle is trained on support images but not query images from the new classes). The current setup seemed unnecessarily restrictive to me and not well motivated. If we can use an oracle trained on more shots, this also sidesteps the other difficulty mentioned by the authors as a limitation that prevents us from using existing metrics / evaluation framework for this problem.

Finally, why did the authors decide to use a different model (SimCLR) to get the encoder used for measuring diversity compared to the model used for recognizability (Prototypical Networks)? Can’t we use the same model for both of these? (and ideally report results with different choices for this model)

Limited scope
===========
Despite this paper being largely empirical, there is a limited number of models experimented with. Why not also compare against the few-shot generalization model in [4], for example?

Omniglot is also a very simple dataset and few-shot learning research (at least for classification) has drifted from it these days [5]. It would be nice to report results on more modern datasets. Quickdraw, which is mentioned in the discussions, can be a good option as it still allows comparisons to humans.

Significance
===========

All in all, due to the limited scope and insufficient motivation of the proposed framework and set of experiments, I felt that the significance of the findings is low. Further, some findings like GANs lacking diversity are already known (e.g. mode collapse is a known problem for GANs). Verifying that the same is true in the one-shot scenario is interesting, but not too surprising, especially in this weak generalization case studied in this paper.

References
=========
[1] Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML. Raghu et al. ICLR 2020.
[2] A Closer Look at Few-shot Classification. Chen et al. ICLR 2020.
[3] Are Few-Shot Learning Benchmarks too Simple ? Solving them without Task Supervision at Test-Time. Huang et al.
[4] One-shot Generalization in Deep Generative Models. Rezende et al. ICML 2016.
[5] Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples. Triantafillou et al. ICLR 2020.

questions: It’s an interesting finding that VAE-STN and VAE-SN better approximate humans for different subsets of visual classes. Would be interesting to see a similar experiment including GAN models too. Also, the authors hypothesize that this is due to the attention of STN providing a better human approximation for simple visual concepts. Isn’t this counterintuitive, as attention can help to break down a complicated image by sequentially looking at different parts? It would be nice to run some additional experiments to examine this hypothesis, e.g. keeping the architecture the same as VAE-STN but removing the attention mechanism and seeing the effects of that on which types of visual contexts it has smallest distances to humans after this modification. It would also be nice to investigate this as a function of the number of attention steps. 

Isn’t it counterintuitive that increasing the context size reduces the diversity, since allowing the model to see more examples should increase its understanding of intra-class variability?

Is there intuition on why varying the number of attentional steps for VAE-STN leads to a parabola in the recognizability/diversity space? What happens at the turn-around point? What is the significance of this finding?

How come different architectures are used for GAN models but not for VAE models?


# after the rebuttal
increased my score to a weak accept based on the discussion with the authors (please refer to my response to the authors for justification)
limitations: Aside from the limitations mentioned earlier in the review, another one is the following.
What if instead of a single image, a few images were available of the new visual concept (few-shot learning instead of one-shot learning)? The proposed framework assumes that in that case they will have to be aggregated into a prototype, but this is not necessarily the best approach. It would be nice to make it more flexible to allow evaluating any method for exploiting more than one ‘shots’.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors propose a new framework for assessing the quality of one-shot generative models for images. Particularly, the authors frame it as a two-object metric where diversity (how generated examples differ from each other) and accuracy (whether the generated example resembles the given single example) of the generative model are measured. Furthermore the human-made examples can also be put into comparison. The authors conduct the experiments of applying such two-object metric to representative one-shot generative models with detailed analysis and discussion.
strengths_and_weaknesses: Strengths of this work:
- It addresses the (relatively) less studied field of one-shot image generation and presents a framework as a metric for assessing the quality. The idea of using two objects is novel.
- This work is more of an empirical work, and it does a good job in clearing presenting experiments and the analysis. Comparison of representative one-shot image generation methods (with human performance comparable) are shown visually and the discussion is substantial. Overall, the writing quality is high and the content is substantial.
- The result presented in this paper would provide interesting insights in the one-shot image generation.


Weaknesses of this work (If they are properly addressed, I’m happy to increase my rating):
- Lack of comparison with previously leveraged metrics. It’s true that the proposed method is more holistic than the scattered ones used in previous works for evaluation, such as T-SNE, clustering, however readers would be also interested in how the new framework compass with such measures, for example how diversity and/or accuracy collrates with clustering results.
- The dataset is limited to Omniglot. Other datasets used in one-shot image generation (for example, CIFAR) should also be  considered as they may show different behaviors, which is the focus of what a metric would be studying. In such a case, human output is not feasible and thus not in the comparison, but still the impact of different methods, especially with different hyper parameters where the paper paid a lot of attention, would have significance.
- Some design choices are not clear (see Questions below)


# Changes during/after rebuttal: 
Increased score from 6 to 7.


questions: - Why using a one-shot classification model (MAML/Prototypical Net) to measure the Recognizability (mentioned in paragraph starting on line 204), instead of a full-fledged classifier trained on the whole dataset (e.g. including all data points of the  class that the one-shot example belongs to)? After all, the latter should have a better quality, and there seems to be nothing preventing the use of it, given that the proposed framework is a metric rather than a one-shot method.
- In the motivation (Sec 2.1) the author claims that likelihood-based algorithms or FID-like metrics are not ideal for the test. However this seems to be only argued but not supported by any experiments. Is there any minimal case that can demonstrate its failure in the proposed setting?
- Is the proposed method applied to other one-shot settings, such as one-shot image segmentation, etc?

limitations: No issue.
review_title: null

Meta-review: metareview: This paper studies the problem of few-shot generation on the Omniglot dataset, contrasting few-shot generative models against humans. They introduce "diversity" and "recognizability" metrics and perform an empirical analysis of how various models are situated in the diversity-recognizability plane relative to humans.

Overall this is a well-written paper, with a clear story and experiments. It provides interesting insights on how various generative models architectures relate to humans in a particular task.
confidence: Certain
Decision: Accept