Diversity vs. Recognizability: Human-like generalization in one-shot generative models
Authors: Victor Boutin
Lakshya Singhal
Xavier Thomas
Thomas Serre
Venue: NeurIPS 2022 Accept (2022)
Keywords: neuroscience
cognitive science
human generalization
one-shot image generation
generalization
generative model
spatial attention
context integration
diversity vs recognizability

Abstract: Robust generalization to new concepts has long remained a distinctive feature of human intelligence. However, recent progress in deep generative models has now led to neural architectures capable of synthesizing novel instances of unknown visual concepts from a single training example. Yet, a more precise comparison between these models and humans is not possible because existing performance metrics for generative models (i.e., FID, IS, likelihood) are not appropriate for the one-shot generation scenario. Here, we propose a new framework to evaluate one-shot generative models along two axes: sample recognizability vs. diversity  (i.e., intra-class variability). Using this framework, we perform a systematic evaluation of representative one-shot generative models on the Omniglot handwritten dataset. We first show that GAN-like and VAE-like models fall on opposite ends of the diversity-recognizability space. Extensive analyses of the effect of key model parameters further revealed that spatial attention and context integration have a linear contribution to the diversity-recognizability trade-off. In contrast, disentanglement transports the model along a parabolic curve that could be used to maximize recognizability. Using the diversity-recognizability framework, we were able to identify models and parameters that closely approximate human data.

Diversity vs. Recognizability: Human-like generalization in one-shot generative models
Victor Boutin 1.2, Lakshya Singhal', Xavier Thomas and Thomas Serre
Artificial and Natural Intelligence Toulouse Institute, Université de Toulouse, France
2 Camey Institute for Brain Science, Dpt. of Cognitive Linguistic & Psychological Sciences Brown University, Providence, RI 02912 {victor_bouting thomas thomas_serre) wbborrrrrrrrrrrerrrro.rrrrror edd
Abstract
Robust generalization new concepts has long remained distinctive feature of human intelligence. However, recent progress in deep generative models now led to neural architectures capable of synthesizing novel instances of unknown visual concepts from a single training example. Yet, a more precise comparison between these models and humans is not possible because existing performance metrics for generative models (i.c., FID, IS, likelihood) are not appropriate for the one-shot generation scenario. Here, we propose one-shot generative models along two axes: sample recognizability vs. diversity (i.e., intra-class variability). Using this framework, perform systematic evalu- ation of representative one-shot generative models on the Omniglot handwritten dataset. We first show that GAN-like and VAE-like models fall opposite ends of the diversity-rcoonnizabiit space. Extensive analyses of the effect of key model parameters further revealed that spatial attention and context integration have a linear contribution to the diversity-recoon -erogiityrr trade-off. In contrast, disentanglement transports the model along a parabolic curve that could be used maximize recognizability. Using the diversity-recognizablii framework, we were able identify models and parameters that closely approximate human data.
framework to evaluate
Introduction
Our ability to learn and generalize from limited number of samples is hallmark of human cognition. language, scientists have long highlighted how little training data children need in comparison to the richness and complexity of the language they learn efficiently to master [ 40) Similarly, children and adults alike are able to learn novel object categories from as little as a single training example 177 From computational point of view, such feats remarkable because they suggest that learners must be relying on inductive biases to overcome such challenges
3R from an ability
detect suspicious coincidences 'non-accidental' features 4 52
exploiting the principle of
compositionality 22 33
While a common criticism of modern AI approaches is their reliance on large training datasets, progress in one-shot categorization has been significant. One-shot categorization involves predicting an image category based on unique training sample per class. Multiple algorithms have been proposed including meta-lcarning algorithms [8 46
14 37 or metric-learning algorithms 4h BI 48 that are starting approach human accuracy. Perhaps less studied problem is the one-shot generation problem - aimed at creating new variations of a prototypical shape seen only once. Since the seminal work of Lake al. 32 who introduced the Bayesian Program Learning algorithm, only a handful of promising one-shot generative algorithms have been proposed 42 for more exhaustive description of prior work).
(see section 2.2
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Why have so few algorithms for one-shot image generation vs. image categorization been proposed? We argue that one of the main reasons for this lack of progress the absence of an adequate evaluation metric. As of today, one-shot generative models are evaluated using methods initially developed for models producing samples that belong to the training categories and trained on large datasets. Those metrics include the likelihood, the FID (Frechet Inception Distance), or the IS (Inception Score). In the one-shot image generation scenario in which training images are scarce and the generated samples represent new visual concepts, the likelihood, the FID, and the IS are biased K 13 38 (see 2. for more details). These limitations urge us to look for new metrics tailored for one-shot image generation.
Recent psychophysics work [52] has characterized humans' ability for one-shot generation along two main axes: samples diversity (i.e., intra-class vari- ability) and samples recognizabilityy (i.e., how easy hard they are to classify). According this frame- work, ideal generalization corresponds to combina- tion of high recognizability and high diversity. As
Box 2
Good generalization area
Box
Recognizabiliyy
illustrated in Fig. D an ideal model should be able
Box 3
to generate samples that span the entire space within the decision boundary of classifier (Box 1). In com-
Poor generalization area
parison, the model of Box 2 has learned to make
identical copies of the prototype (i.e., low diversity but high accuracy). Such a model has failed to gener- alize the visual concept exemplified by the prototype. Similarly, if the model's 's samples are so diverse that they cannot be recognized accurately shown in the Box 3 of Fig. | then the generated samples won't t look like the prototype.
Diversity
Classifier decision
Generated
samples
Prototype
boundary
Here, we borrow from this work and adapt it to create the first framework to evaluate and compare humans and one-shot generative models. Using this frame- work, we systematically evaluate array of repre- sentative one-shot generative models on the Omniglot dataset 622 We show that GAN-like and VAE-like one-shot generative models fall on opposite ends of the diversity-reoognzzabile
Figure 1: The diversity vs. recognizabil- ity framework. The best possible samples for good generalization (green area) are those that match the intra-class variations (i.e., re- main within decision boundaries; Box 1). Bad samples associated with poor generalization (red area) include strategies that involve exact copies of the prototype (Box 2) and samples that exceed the intra-class variability (Box 3)-
space: GAN-like models fall on the high recognizability - low diversity end of the space while VAE-like models fall on the low recognizability - high diversity end of the space. We further study some key model parameters that modulate spatial attention, context integration, and disentanglement. Our results suggest that spatial attention and context have an (almost) linear effect on the diversity vs. recognizabilitty trade-off. In contrast, varying the disentanglement moves the models on a parabolic curve that could be used to maximize the recognizability. Last but not least, we have leveraged the diversity vs. recognizabilitty space to identify models and parameters that best approximate the human data on the Omniglot handwritten dataset.
2 Related work
2.1 Metrics to evaluate generative models and their limitations for one-shot generation tasks
Different types of generative models are typically evaluated using different metrics. On the one hand, likelihood-basse algorithms (e.g., VAE (29 PixelCNN [39] GLOW 30, etc.) are evaluated using their own objective function applied on testing set. Likelihood provides direct estimate of the KL divergence between the data points and the model's 's samples. On the other hand, implicit generative models such as Generative Adversarial Networks (GANs) 20 for which the loss function cannot be used, are typically evaluated using other scores such as the Inception Score (IS) 45 or Frechet Inception Distance (FID) (22. IS and FID are heuristic measures used to aggregate both the sample quality and diversity in single score. The IS scores sample quality according to the

---

confidence with which an Inception v3 Net 49 assigns the correct class label to it. The FID score is the Wasserstein-2 2 distance between two Gaussian distributions: one fitted on the features of the data distribution and the other on the features of the model distribution (the features are also extracted from an Inception v3 Network).
All these metrics are problematic for the one-shot generation scenario for 2 main reasons that are intrinsically related the task: the low number of samples per class, and the dissimilarity between training and testing visual concepts. IS and FID rely on statistical distances (either KL divergence for IS or Wasserstein-2 distance for FID) that require a high number of data points N to produce an unbiased estimator of the distance. Even when used in the traditional settings (i.e.. N = 50000), it has been demonstrated that both scores are biased [3. This is to be compared with the N 20 samples typically available in popular few-shot learning datasets such as Omniglot [32] Another problem caused by the limited number of samples per class in the training set the overfitting of the Inception Net used to extract the features to compute the IS and FID 77]. To illustrate this phenomenon we have conducted a small control experiment in which we have trained a standard classifier to recognize images from the Omniglot datasets (see (177. In this experiment, we have used 18 samples per classes for training and 2 samples per classes for testing. In Fig. S2TT, we observe an increase of the testing loss while the training loss is decreasing. This is a clear sign of overfitting. Note that this overfitting is not happening when the standard classifier is replaced by a one-shot (or few-shot) classifier. This control experiment show that standard classifier are not adapted to extract relevant features in the low-data regime. Consequently, IS and FID are not suitable in the low-data regime.
The second limitation of these metrics appears because the training and the testing samples are too dissimilar. Likelihood scores are known to yield higher scores for out-of-domain data compared to in-domain data G8). Therefore, the evaluation of the novel visual concepts generated by one-shot generative models will be biased toward higher scores. In addition, both FID and IS rely on distance between features extracted by an Inception Net which comes with no guarantee that will produce meaningful features for novel categories. For example, class misalignment has been reported when the Inception Net was trained on ImageNet and tested on CIFARIO B.. Because of all the aforementioned limitations, it is pretty clear that new procedures are needed to evaluate the performance of few-shot generative algorithms.
2.2 One-shot generative models
One can distinguish between two broad classes of one-shot generative models: structured models and statistical models [16). Structured models have strong inductive biases and rigid parametric assumptions based priori knowledge such for example given hierarchy of features, known grammar or program 4. A prominent prominentexample of a structured model includes the very first algorithm for one-shot image generation, the Bayesian Program Learning (BPL) model ( Statistical models
learn visual concepts by learning statistical regularities between observed patterns [22 15 19]
Here, we focus on representative architectures of one-shot generative statistical models, which we summarize below.
VAE with Spatial Transformer Network (VAE-STN) 42 The VAE-STN is a sequential and conditional Variational Auto-Encoder (VAB) constructing images iteratively. The VAE-STN algorithm uses a recurrent neural network (i.c., an LSTM) to encode the sequence of local patches extracted by an attentional module. A key ingredient of the VAE-STN is an attention module composed of a Spatial Transformer Network (STN) 20 to Iearn to shift attention to different locations of the input image. The STN is a trainable module to learn all possible affine transformations (i.e., translation, scaling, rotation, shearing) of an input image (see S8 for samples and details of the VAE-STN).
The Neural Statistician is an extension of the conditional VAE model including contextual information. Therefore, in addition to learning an approximate inference network over latent variables for every image in the set (as done in a VAE), the approximate inference is also implemented over another latent variable, called the context variable, that specific to the considered visual concept. The context inference network is fed with a small set of images representing variations of a given visual concept. The VAE-NS has been extended to include attention and hierarchical factorization of the generative process 19 (see S9 for samples and details of the VAE-NS).
Neural statistician (VAE-NS) [33]
3

---

Data- Antaanmmmmnttiion GAN (DA-GAN) M: Data- Aatt--ugmmntationn GAN is a generative adversarial network conditioned on prototype image. The DA-GAN generator is fed with a concatenation of a vector drawn from a normal distribution and a compressed representation of the prototype. The discriminator trained to differentiate images produced by the generator from images of the dataset, while the generator has to fool the discriminator. We have trained 2 different DA-GAN, one is based on the U-Net architecture (DA-GAN-UN) and the other one on the (see S10 for samples and details of the DA-GAN-UN and SIT for samples and details of the DA-GAN-RN).
ResNet architecture (DA-GAN-
All these models are generative models conditioned by an image prototype extracted from the training or the test set. The way we have selected the prototypes is detailed in Eq. To the best of our knowledge, these models offer a representative set of one-shot generative models. We have reproduced all these models (sometimes with our own implementation when it was not available online). Our code could be found at httttt:t//tttiiiii htttps///gittubb.com//eeere www.dieersitty.s vs_ wtw.rrsseeeooiiibiittt
3 The diversity vs. accuracy framework
Let {xr be dataset composed of K concepts (i.e., classes) with N samples each (i E [I, N] and j E KJ. The framework we propose aims at evaluating the performance of generative model Po. parameterized by 0, that produces new images uz based on a single sample (or prototype) of a concept given to the generator jd (i.e., vf ~ pa (-(7))). ). For each concept j, we define a prototype as the sample closest to the center of mass for the concept j:
N
s.t. i"= argmin
f(x))
E f(x))
In Eq. [ f denotes a function that projects the input image from the pixel space to feature space. We will detail the feature extractor I shortly. Note that this definition of prototype is not unique (one could also select the prototype randomly within individual classes j), nevertheless this selection mechanism is guarantee that the selected sample will be representative of the concept.
with weak generalization split [42) Omniglot is composed of binary images representing 1, 623 classes of handwritten letters and symbols (extracted from 50 different alphabets) with just 20 samples per class. We have downsampled the original dataset to be 50 x 50 pixels. The weak generalization split consists of training set composed of all available symbols minus symbols per alphabet which are left aside for the test set. is said to be weak because all the alphabets were shown during the training process (albeit not all symbols in those alphabets). As the Omniglot dataset is hand-written by humans, we consider that these samples reflect a human generative process and we refer to this later as the human model.
Dataset. In this article, we use the Omniglot dataset
Diversity. In the proposed framework, diversity refers the intra-class variability of the samples produced by a generative model pe (- (ā+). For given prototype ø², we compute the diversity as the standard deviation of the generated samples in the feature space f:
N Σ N -
N Σ E()) N
(f(z)
s.t. of ~ po(
(2)
We use the Bessel-corrected standard deviation to keep good estimate of the data dispersion despite the relatively small number of samples (e.g., N = 20 for the Omniglot dataset used here). To verify that this diversity measure is robust to the specific choice of the feature extractor f. we explored two different settings: features learned with class supervision by a Prototypical Net 47 and features learned with self-supervision by SimCLR network [IO]. In both cases, we extracted the features from the first fully-connected layer following the last convolutional layer. The Prototypical Net was optimized so that images that belong to the same category share similar latent representations as measured by the norm.. Similarly, SimCLR leverages a contrastive loss to define a latent representation such that a sample is more similar to its augmented version than to other image samples. In SimCIR, this similarity computed with cosine similarity. These two approaches represent two ends of a continuum of methods to learn suitable representational spaces without the

---

need to explicitly learn to classify images and are thus more suitable for few-shot learning tasks 13 For the sake of comparison, we have used the exact same network architecture for both feature extractors (see sections ST and for more details on Prototypical Net and SimCIR, respectively).
We computed the samples diversity for all 150 categories of the Omniglot test set (i.e., of x in this experiment) using both the supervised and unsupervised settings. We found high linear correlation (p 0.86, p-value < 10-9) and a high rank-order Spearman correlation (p = 0.85, p-value < 10-9) between the two settings (see section S3.Ip. Hence, the two feature extraction methods produce comparable diversity measures and henceforth, we will report results using the unsupervised setting.
As additional control, have also verified that the SimCLR metric is robust changes the augmentation method used (see section (3.3) and to the specific choice of the dispersion metric (see section S3.2 for more details this comparison). We have compared the feature space of the Proto- typical Net and SimCLR using I-SNE analysis (see section S3.4) We observed strong clustering of samples belonging to the same category for both networks. suggests that the augmentation methods used by the SimCLR contrastive loss are sufficient to disentangle the class information.
Fig. shows the 10 concepts from the Omniglot test set with the lowest and highest samples diversity, respectively (for more diversity-ranked concepts with unsupervised or supervised setting, sections S4 and $5 respectively). One
that the proposed diversity metric is qualitatively similar to human judgment. Concepts with low diversity are composed of very few relatively basic strokes (e.g., lines, dots, etc) with little room for any kind of "creativity" in the generation process while more diverse concepts are composed of more numerous and more complex stroke combinations with many more opportunities for creativity.
a)
b)
W
JI
Diversity
41 R
VI
Figure 2: Samples from the top 10 Omniglot concepts (test set) associated with the lowest (a) vs. highest diversity (b). The different concepts are ranked vertically from less diverse to more diverse. Prototypes for individual concepts are shown within a red box next to actual class samples.
Recognizability. We evaluate the recogmizability of the samples produced by the one-shot generative models by leveraging one-shot classification models. As demonstrated in S17 it is also possible to use a few-shot classifier to evaluate the recognizability. We prefer one-shot classifier to match the settings proposed in 32 In order make sure our classification accuracy measure is robust to the choice of the model, we test different models which belong to the two main approaches used in machine learning for one-shot classification: metric learning and meta-lcarning B35 We selected the Prototypical Net 47] as representative metric-lcarning approach and the Model-Agnostic Meta- Learning (MAML) 18 model representative meta-learning approach. Both models were trained and tested in 1-shot 20-ways setting (see section S6 for more details on the MAML architecture and training details). We report a high Pearson (negative) correlation between the logits produced by Prototypical Net and MAML (p- -0.6 -0.60, p-value < 10-5) 5) as well as a strong Spearman rank-order correlation between the classification accuracy of both networks (p 0.62, p-value < 10-3). See section S7 for more details about this control experiment. Hence, our recognizability metric is robust to the choice of the one-shot classification model (even when those models are leveraging different approaches) and henceforth, will report results using the Prototypical Net model.
5

---

Results
4.1 GAN-like VAE-like models
For all algorithmss listed in section 2.2 we have explored different hyper-parameteen (see section 4.2 for more details), leading various models represented in the diversity vs. recognizability plot in Fig. 3h. In this figure, we have reduced each model to single point by averaging the diversity and recognizability over all classes of the Omniglot testing set. The black star corresponds to the human model, and colored data points are computed based on the samples generated by the VAE-NS, VAE-STN, DA-GAN-UN and the DA-GAN-RN, The base architectures for all algorithms (highlighted with bigger points in Fig. 3 have comparable number of parameters (2-67 M, see S8 and STO for more details the base architectures).
We observe that the GAN-like models (i.e., DA-GAN-UN and DA-GAN-RN) tend to located at the upper left side of the graph while VAE-like models (i.c., VAE-NS and VAE-STN) spread on the right side the graph. Therefore, the GAN-like models produce very recognizable samples that are highly similar to each other (high recognizability and low diversity). In contrast, VAE-like models generate more diverse but less recognizable samples. The samples in Fig. 3 illustrate this observation. The difference between GAN and ViE-like samples could be explained by their loss functions [36] The GANs' adversarial loss tends to drop some the modes of the training distribution. In general, the distribution learned by GANs put excessive
on the more likely modes but discards secondary modes [2]. This phenomenon leads to sharp and recognizable generations at the cost of reduced samples diversity. On the other hand, VAES (and likelihood-basee from wver-generalization: they cover all the modes of the training distribution and put mass in spurious regions 4 We refer the reader Fig. 4 of Lucas et al. [36] for an illustration of mode dropping in GANs and over-ge over-generee seee--eneraaization in VAEs. Our diversity vs. recognizability plot in Fig. B shows that this phenomenon holding even when the testing distribution is different from the training distribution as in the case of the one-shot generation scenario.
models in general) are suffering
a)
b)
100
8
9
Human
VAE-STN
Dacognibabiity (%)
9
G
3
6
6
6
VAE-NS
9
9
9
G
S
DA-GAN-UN
* human
VAE-NS
2
8
6
8
6
6
8
6
6
6
6
8
6
6
DA-GAN-UN
DA-GAN-RN
DA-GAN-RM RN
0.0
0.5
2.0
Diversity
Figure 3: (a) Diversity vs. recognizability plot for all tested models (colored data points) and human (black star). Each data point corresponds to the mean diversity and recognizability over all classes of the Omniglot test set. Bigger circles correspond to the base architecture of each model that all have a comparable number of parameters (s= 6 7M). The human data-point is computed based on the testing samples of the Omniglot dataset. (b) Samples produced by the different models in their base architectures (corresponding to the bigger circles in Fig. (3).. Prototypes for individual concepts are shown within red box next to actual class samples.
4.2 VAE-NS vs. VAE-STN
In this section we compare some key hyper-parameters of the VAE-NS and VAE-STN. The core idea of the VAE-NS is to integrate context information to the sample generation process. During

---

training, the context is composed of several samples that all represent variations of the same visual concept. Those samples are passed in a separate encoder to extract context statisties (denoted e in S9 used to condition the generative process. During the testing phase, the VAE-NS infers the context statisties using a single image (i.e., the prototype). We evaluate the effect of the context on the position of the VAE-NS models on the reweesstty-rcognizabblea of samples used to compute the context statistics during the training phase (from to 20 samples). Importantly, varying the number of context samples do not change the number of parameters of the network. For all tested runs, we observe a monotonic decrease of the samples diversity (see Fig. (S13)) and monotonic increase of the samples recognizability (see Fig. (T33) when the number of context samples is increased. In the transporting models from the lower-right side to the upper-left side of the plot (see Fig. Fp. dark blue curve). The effect of the number of context samples is large: the diversity is almost divided by 2 (from 2.4 1.2) and the classification accuracy is increased by 80% (from 53% to 96%). This result suggests that increasing the number of context samples for given visual concept helps the generative model to identify the properties and the features that are crucial for good recognition, but hurts the diversity of the generated samples.
space by varying the number
space, the resulting curve is monotonically
a)
b)
100
100
*2
*
B
90-
80
B
Recognbabiity (%)
80-
# of context samples
Racognizabilty
# of attention steps
50
WAE-STN VAE-NS
VAE-STN VAE-NS
humen
* human
50
1.2
1.5
1.8
2.1
2.4
2.7
1.2
1.5
1.8
2.1
Diversity
Diversity
Figure 4: (a) Effect of the number of context samples of the VAE-NS and attentional steps of the VAE-STN. Each data point represents a model with a different number of context samples for the VAE-NS (ranging from 2 to 20) or a different number of attentional steps for the VAE-STN (ranging from 20 to 90). The base architectures, highlighted with a bigger eircle, correspond to 10 context samples and 60 attentional steps for the VAE-NS and VAE-STN respectively. (b) Effect of 3. The base architectures correspond to a B = 1. In all curves, solid lines represent the mean of the parametric curves over 3 different runs. Shaded areas are computing using the standard deviation over 3 different runs. Arrows show the direction in which the tested variables (context samples, attention steps or () are increased.
In contrast to the VAE-NS, the VAE-STN uses spatial attention to sequentially attend to sub-parts of the image to decompose into simpler elements. These sub-parts then easier encode and synthesize. In the VAE-STN, one can vary the number of attentional steps (i.e., the number of attended locations) modulate spatial attention. Importantly, varying the number of attentional steps does not change the number of parameters. We have varied the number of attentional steps from 20 to 90. The relationship between the number of attentional steps, the samples diversity, and the samples recognizability is non-monotonic. We have used a parametric curve fitting method (i.e., the least curve fitting method from (1 to parameterize the curve while maintaining the order of the data point (see ST3 for more details on the fitting procedure). We report a convex parabolic relationship between the number of attentional steps and the samples diversity (see Fig. (S14)). This curve is minimal at 60 steps. We observe concave parabolic relationship between the number of attentional steps and the recognizability of samples. This curve maximal at 60 attentional steps (see Fig. (514)). In Fig. 4b we have plotted the parametric fit illustrating the position of the VAE-STN models in the
7

---

diversity-recognizzee recognizability space when one increases the number of attentional steps (the light blue curve). This curve follows quasi-linear linear trend with a sharp turn uum-around (at 60 attentional steps). The effect of the number of attentional steps on the civversity is limited compared to the effect of the number of context samples.
Both VAE-NS and VAE-STN are trained to maximize the Evidence Lower Bound (ELBO), it is then possible tune the weight of the prior in the loss function. One can operate such a modulation by changing the 3 coefficient in the ELBO loss function 124]. We refer the reader to $15 for more mathematical details about the ELBO. A high I value enforces the latent variable to be closer to a normal distribution and increases the information bottleneck in the latent space. Increasing B is known to force the disentanglement of the generative factors [9] We observe a monotonic decreasing relationship between the value of / and the samples diversity for both the VAE-STN and the VAE- NS (see Fig. S15h and Fig. [166, respectively). We report a concave parabolic relationship between B and the samples recognizability. We use the least curve fitting method to find the optimal parabolic curves (21) This curve is maximal at /3 2.5 for the VAE-STN and at / = 3 for the VAF-NS (see Fig. S15p and Fig. S16b, respectively). The overall effect of / on the position of the VAE-like models on the diversity
space is relatively similar for both the VAE-STN and the VAE-NS and follows a clear parabolic trend. These curves demonstrate that one could modulate the value of 3 to maximize the recognizability. In general, we observe that the variable controlling the context-size in the VAE-NS is the one having the biggest impact the space.
We have also varied the architecture of the DA-GAN-UN, DA-GAN-RN, VAE-NS, and VA STN by changing the size of the latent space. We did not find any common trend between the size of the latent variable, the diversity, and the recognizability (see SST more details). We observe that the DA-GAN-UN tends to produce slightly more recognizable but less diverse samples than the DA-GAN-RN while both architectures have the same number of parameters. It suggests that the extra skip connections included in the U-Net architecture, in between the encoder and the decoder of the DA-GAN-UN, allow to trade diversity for recognizabilityy
4.3 Comparison with humans
We now compare the tested models with the human data in the diversity-recognizabiie space. To perform such comparison, we first normalize all the model's diversity and recognizability (including humans) using the z-score such that both axes are scaled and centered similarly. Then, for all models, we compute the fy-distance between models and humans in the recerssty-rroggizabbiltt space. We remind that the human data point is computed using the samples of the Omniglot test set. Distances to humans as well as their distributions are reported for all models in Fig. Ba. The median of VAE- NS models is closer to humans, followed by DA-GAN-UN DA-GAN-RN and VAE-STN (medians are indicated in Fig. a with horizontal bars). The VAE-NS model showing the smallest distance is almost at the human level (see dark blue square Fig. 5b). It has a context size of 20 samples (the highest possible context size), and 3 = 2.5. The VAE-STN model that best approximates human has B = 2.25 and 60 attentional steps (see light blue square in Fig. 5p)
So far, we have reduced all models single points by averaging the diversity and recognizability values over all classes. We now study distances to humans for individual classes and for the VA NS and the VAE-STN models showing the shortest distance to humans (indicated by blue squares in Fig. (a). In Fig. 5b. we report distances to human for these 2 models and for 16 visual concepts. The visual concepts I to 8 and the VAE-STN and the VAE-NS, respectively. We observe that these visual concepts are different for the VAE-NS and the VAE-STN model. Therefore, both models are well approximating human data for some visual concepts but not for others. Interestingly, we qualitatively observe that the visual concepts to 8 look simpler (i.e., made with fewer strokes) than the visual concepts 9 to 16. It suggests that the spatial attention mechanism used by VAE-STN provides a better human approximation for simple visual concepts, while the context integration method leveraged by the VAE-NS is more relevant to mimic human data more complex visual concepts.
16 are selected so that they minimize the distance to humans with
8

---

b)
5|1
ulistdice human
b
2
Dislance human
3
1-
U
0-
VAE-STN
VAE-NS
DA-GAN-UN DA-GÁN-RN
10
WNE-NS
Figure 5: (a) Distribution of average distance humans for the VAE-NS, VAE-STN, DA-GAN- UN and DA-GAN-RN, Each data point corresponds to the mean diversity and accuracy over all classes of the Omniglot test Squares correspond the model showing the smallest distance to humans. The distance to humans is calculated with a f2-norm on the diversity-rcoognizabilyy space after z-score normalization. The horizontal line denotes the median of the model's distribution. (b) Distance to humans on 16 different visual concepts for the VAE-NS and VAE-STN models that best approximate human data (i.e., indicated by square Fig. P.. The visual concepts I to 8 selected to minimize the VAE-STN distance to humans, and visual concepts 9 to 16 minimize the VAE-NS distance to humans. Images surrounding the radar plot are the prototypes of the visual concepts.
5 Discussion
In this article, we have described novel framework for comparing computational models with human participants on the one-shot generation task. The framework measures the diversity and the recognizability the produced samples using metrics compatible with the one-shot scenario. the best of our knowledge, this is the first and only framework specifically tailored to evaluate and compare one-shot image generation models.
Among all tested algorithms, the VAE-NS is the best human approximator on Omniglot (see Fig. 5)) suggests that the context integration mechanism of the VAE-NS is an important component to reach human-like generalization. Interestingly, motor learning experiments have demonstrated that human generalization performances are also strongly related contextual information (50) Interestingly. [51] have demonstrated that a bayesian observer tends to overestimate the intra-class variance when only few context samples are accessible. Our results are in-line with this finding: Fig. 4 shows an high diversity when the number of context samples is low while the diversity is decreasing when more context samples are available. suggests that the VAE-NS is acting as bayesian observer: it overestimates intra-class variance when the context is scarse.
In addition, we demonstrate that one can tune 3 so that the model becomes closer to human data (see Fig. Ap). This is consistent with prior computational neuroscience study that has shown that disentangled VAEs (with? 1) provide good model of face-tuned neurons in the inferotemporal cortex 23 Our comparison between the VAE-NS and the VAE-STN suggests that a model which uses spatial attention mechanism better fits human data for simple visual concepts. contrast, the context integration mechanism of the VAE-NS appears to be a better human approximator for more complex visual concepts. One could thus try to combine both mechanisms towards improving the similarity with human data independent of the complexity of the visual concept. We have also found that GAN-like models (DA-GAN-RN and DA-GAN-UN) better account for human recognizability but do not approximate well the diversity of the human samples. In contrast, VAE-like models

---

(VAE-NS and VAE-STN) better account for human diversity. An interesting approach would be to leverage hybrid architecture (such the VAE-GAN +
to try better match human data.
Other candidate ingredients include the ability to harness compositionality
32 or the recurrent
processes thought to be crucial for human generalization 54
Compositionality could be introduced in one-shot generative algorithms by quantizing the latent space (as in the VQVAE (53]). As result,
each coordinate of the latent variable represents an address in a codebook, and the role of the prior is then to combine simpler concepts to generate more complex samples. One promising way to include recurrent processing into generative models is through the predictive coding framework [4]] Predictive Coding suggests that each processing step is part of an inference scheme that minimizes the prediction error [6). Previous work has demonstrated that such networks are more robust and exhibit improved generalization abilities I.. All these ingredients could be tested and compared against human abilities using the diversity/recoonizaallit framework we have proposed in this paper.
In the current version of the Omniglot dataset, the intra-class variability does not reflect the human level of creativity. is mainly due to the experimental protocol in which one asks human participants to copy a given visual concept. The Omniglot dataset could be enriched with more diverse samples, by explicitly asking human participants be creative as possible. Other drawing databases with more complex symbols such as Quick Draw! 27 could also be considered to strengthen the comparison with humans.
By decomposing the performance of the one-shot generation task along the recognizability vs. diversity axes wanted shed light the relationship between generalization and creativity (quantified by the samples diversity our framework). We hope one can make use of our framework to validate key hypotheses about human generalization abilities so that the brain. We argue that the best way to reach human-like generalization abilities is to unleash the algorithms' creativity.
better understand
Acknowledgement
This work was funded by ANITI (Artificial and Natural Intelligence Toulouse Institute) and the French National Research Agency, under the grant agreement number ANR- 19-P13A-0006 Additional funding to TS was provided by ONR (N00014-19 19-14-19-11 1114-19-1-2002- 1-2029) and NSF (IIS- LUS-1912280 and EAR-192548) Computing hardware supported by NIH Office the Director grant $100D025 181 via the Center for Computation and Visualization (CCV). We thanks Roland W. Fleming and his team for the insightful feedback and discussion about the diversity vs. recognizability framework.
10

---

References
[1] Antoniou Antreas, Storkey Amos, Edwards Harrison. Data augmentation generative adversarial networks // arXiv preprint 1.t::1111044444 2017.
[2] Arjovsky Martin, Chintala Soumith, Bottou Léon. Wasserstein generative adversarial networks // International conference machine learning. 2017. 214-223.
[3] Barratt Shane, Sharma Rishi. A note on the inception score // arXiv preprint arXiv: 180.0183 2018.
[4] Bishop Christopher M, Nasrabadi Nasser M. Pattern recognition and machine learning. 4, 2006.
[5] Boutin Victor, Franciosini Angelo, Ruffier Franck, Perrinet Laurent. Effect of top-down connec- tions in Hierarchical Sparse Coding // Neural Computation. 2020. 32, 11. 2279-2309.
[6] Boutin Victor, Zerroug Aimen, Jung Minju, Serre Thomas. Iterative VAE as predictive brain model for out-of-distribution generalization // arXiv preprint urXiv:2012.00557. 2020.
[7] Brigato Lorenzo, locchi Luca. close look deep learning with small data // 2020 25th International Conference on Pattern Recognition (ICPR). 2021. 2490-2497.
[8] Broedelet Iris, Boersma Paul, Rispens Judith, others School-Aged Children Learn Novel Categories on the Basis of Distributional Information // Frontiers in Psychology. 2022. 12, 799241.
[9] Burgess Christopher P. Higgins Irina. Pal Arka, Matthey Loic, Watters Nick, Desjardins Guil- laume, Lerchner Alexander. Understanding disentangling in beta-VAE // arXiv preprint 1eXiv:1804003599 2018.
[10] Chen Ting, Kornblith Simon, Norouzi Mohammad, Hinton Geoffrey. A simple framework for contrastive learning of visual representations // International conference on machine learning. 2020. 1597-1607.
[11] Choksi Bhavin, Mozafari Milad, Biggs O'May Callum, Ador Benjamin, Alamia Andrea, Van- Rullen Rufin. Predify: Augmenting deep neural networks with brain-inspired predictive coding dynamics // Advances in Neural Information Processing Systems. 2021. 34.
[12] Chomsky Noam. Aspects of the theory of syntax Special technical report no. 11. 1965.
[13] Chong Min Jin, Forsyth David. Effectively unbiased fid and inception score and where to find them // Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020. 6070-6079.
[14] Chowdhury Arkabandhu, Chaudhari Dipak, Chaudhuri Swarat, Jermaine Chris. Meta-Meta Classification for One-Shot Learning // Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022. 177-186.
[15] Edwards Harrison, Storkey Amos. arXiv:1606.02185. 2016.
Towards a neural statistician // arXiv preprint
[16] Feinman Reuben, Lake Brenden M. Generating new concepts with hybrid ncuro-symbolic models // arXiv preprint arXiv:2003.08978. 2020.
[17] Feldman Jacob. The structure of perceptual categories // Journal of mathematical psychology. 1997. 41, 2. 145-170.
[18] Finn Chelsea, Abbeel Pieter, Levine Sergey. Model-agnostic meta-learning for fast adaptation of deep networks // International conference on machine learning, 2017. 1126-1135.
[19] Giannone Giorgio, Winther Ole. Hierarchical Few-Shot Generative Models Fifth Workshop on Meta-Learning at the Conference on Neural Information Processing Systems. 2021.

---

[20] Goodfellow lan, Pouget-Abadie Jean, Mirza Mehdi, Xu Bing, Warde-Farley David, Ozair Sherjil, Courville Aaron, Bengio Yoshua. Generative adversarial nets Advances in neural information processing systems. 2014. 27.
[21] Grossman M. Parametric curve fitting // The Computer Journal. 1971. 14, 2. 169-172.
[22] Heusel Martin, Ramsauer Hubert, Unterthiner Thomas, Nessler Bernhard, Hochreiter Sepp. Gans trained by a two time-scale update rule converge to a local nash equilibrium // Advances in neural information processing systems. 2017.30.
[23] Higgins Irina, Chang Le, Langston Victoria. Hassabis Demis, Summerfield Christopher. Tsao Doris, Botvinick Matthew. Unsupervised deep learning identifies semantic disentanglement in single inferotemporal face patch neurons // Nature communications, 2021. 12, 1. 1-14.
[24] Higgins Irina, Matthey Loic, Pal Arka, Burgess Christopher, Glorot Xavier, Botvinick Matthew, Mohamed Shakir, Lerchner Alexander. beta-vae: Learning basic visual concepts with con- strained variational framework // arXiv preprint arXiv: 800.03559. 2016.
[25] Hinton Geoffrey, Srivastava Nitish, Swersky Kevin. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent // Cited on. 2012. 14, 8. 2.
[26] Jaderberg Max, Simonyan Karen, Zisserman Andrew, others. Spatial transformer networks // Advances in neural information processing systems. 2015. 28.
[27] Jongejan Jonas, Rowley Henry, Kawashima Takashi, Kim Jongmin, Fox-Gieg Nick. The quick, draw!-ai experiment // Mount View, CA, accessed Feb. 2016. 17, 2018. 4.
[28] Kingma Diederik P, Ba Jimmy. Adam: A method for stochastic optimization // arXiv preprint arXiv:1412.6980. 2014.
[29] Kingma Diederik Welling Max. Auto-encoding variational bayes arXiv preprint arXiv:1312.6114 14. 2013.
[30] Kingma Durk P. Dhariwal Prafulla. Glow: Generative flow with invertible 1x1 convolutions // Advances in neural information processing systems. 2018. 31.
[31] Koch Gregory, Zemel Richard, Salakhutdinov Ruslan, others Siamese neural networks for one-shot image recognition // ICML deep learning workshop. 2. 2015. 0.
[32] Lake Brenden M, Salakhutdinov Ruslan, Tenenbuum Joshua B. Human-level concept learning through probabilistic program induction // Science. 2015. 350, 6266. 1332-139 1338.
[33] Lake Brenden M, Ullman Tomer D, Tenenbaum Joshua B, Gershman Samuel J. Building machines that learn and think like people // Behavioral and brain sciences. 2017. 40.
[34] Larsen Anders Boesen Lindbo, Sonderby Søren Kaae, Larochelle Hugo, Winther Ole. Autoen- coding beyond pixels using a learned similarity metric International conference on machine learning. 2016. 1566-
[35] Li Xiaoxu, Yang Xiaochen, Ma Thanyu, Xue Jing-Hao. Deep metric learning for few-shot image classification: A selective review // atXiv preprint arXiv:2105.08 08149. 2021.
[36] Lucas Thomas, Shmelkov Konstantin, Alahari Karteek, Schmid Cordelia, Verbeek Jakob. Adap- tive density estimation for generative models // Advances in Neural Information Processing Systems. 2019. 32.
[37] Mishra Nikhil, Rohaninejad Mostafa, Chen Xi, Abbeel Pieter. A simple neural attentive meta-learner // arXiv preprint arXiv:1707.03144 2017.
[38] Nalisnick Eric, Matsukawa Akihiro, Teh Yee Whye, Gorur Dilan, Lakshminaraaanan Balaji. Do deep generative models know what they don't know? // arXiv preprint arXiv:1810.091 2018.
[39] Oord Aaron Van den, Kalchbrenner Nal, Espeholt Lasse, Vinyals Oriol, Graves Alex, others Conditional image generation with pixelenn decoders // Advances neural information processing systems. 2016. 29.
12

---

[40]
Massino. Language and learning: the debate between Jean Piaget and Noam Chomsky // Harvard Univ Press, Cambridge, MA. 1980.
[41] Rao Rajesh PN, Ballard Dana H. Predictive coding in the visual cortex: a functional inter- pretation of some extra-classical receptive-field effects // Nature neuroscience. 1999. 2, 1. 79-87.
[42] Rezende Danilo, Danihelka Ivo, Gregor Karol, Wierstra Daan, others One-shot generalization in deep generative models // International conference machine learning. 2016. 1521-1529.
[43] Richards Whitman, Feldman Jacob, Jepson A. From features perceptual categories BMVC92. 1992. 99-108.
[44] Salakhutdinov Ruslan, Tenenbaum Joshua, Torralba Antonio. One-shot learning with a hierar- chical nonparametric bayesian model // Proceedings ICML. Workshop on Unsupervised and Transfer Learning. 2012. 195-206.
[45] Salimans Tim, Goodfellow lan, Zaremba Wojciech, Cheung Vicki, Radford Alec, Chen Xi. Improved techniques for training gans // Advances in neural information processing systems. 2016. 29.
[46] Santoro Adam, Bartunov Sergey, Botvinick Matthew, Wierstra Daan, Lillicrap Timothy, Meta- learning with memory-augmented neural networks // International conference on machine learning. 2016. 1842-1850.
[47] Snell Jake, Swersky Kevin, Zemel Richard. Prototypical networks for few-shot learning Advances in neural information processing systems. 2017. 30.
[48] Sung Flood, Yang Yongxin, Zhang Li, Xiang Tao, Torr Philip HS, Hospedales Timothy M. Learning to compare: Relation network for few-shot learning // Proceedings of the IEEE conference on computer vision and pattern recognition. 2018. 1199-1208.
[49] Szegedy Christian, Vanhoucke Vincent, loffe Sergey, Shlens Jon, Wojna Zbigniew. Rethinking the inception architecture for computer vision Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. 2818-2826.
[50] Taylor Jordan A, Ivry Richard B. Context-dependent generalization // Frontiers in Human Neuroscience. 2013.7 7. 171.
[51] Tenenbaum Joshua. Bayesian modeling of human concept learning // Advances in neural information processing systems. 1998. 11.
[52] Tiedemann Henning, Morgenstern Yaniv, Schmidt Filipp, Fleming Roland W. One shot general- ization in humans revealed through drawing task // bioRxiv. 2021.
[53] Van Den Oord Aaron, Vinyals Oriol, others Neural discrete representation learning // Advances in neural information processing systems. 2017. 30.
[54] Wyatte Dean, Curran Tim, O' O'Reilly Randall. The limits feedforward vision: Recurrent processing promotes robust object recognition when objects are degraded // Journal of Cognitive Neuroscience. 2012. 24, 11. 2248-2261.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Described the conclusion.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We have briefly discussed societal impact in [ST8
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] No theoretical results
(b) Did you include complete proofs of all theoretical results? [N/A] No theoretical results
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] All the codes available on github llink given in section 2.2)
(b) Did you specify all the training details (c.g.. data splits, hyper- peeem-aaaeetess how they were chosen)? [Yes] All the training details, as well as the architectural details are given the supplementary materials.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] Error bars have been reported on 3 different runs
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In the Supplementary Information section ST
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] All the github links are disclosed in the Supplementary information, and the contributing authors are cited in the main article.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either the supplemental material URL? [Yes] The github link of our code is included in the article
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] All assets we are using are publicly available, and do not require any consent
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] No personal information in Omniglot
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "eCALGwiPgR", "reviewer": "Reviewer_fqWo", "summary": "In this paper, the authors propose a new framework for assessing the quality of one-shot generative models for images. Particularly, the authors frame it as a two-object metric where diversity (how generated examples differ from each other) and accuracy (whether the generated example resembles the given single example) of the generative model are measured. Furthermore the human-made examples can also be put into comparison. The authors conduct the experiments of applying such two-object metric to representative one-shot generative models with detailed analysis and discussion.", "questions": "- Why using a one-shot classification model (MAML/Prototypical Net) to measure the Recognizability (mentioned in paragraph starting on line 204), instead of a full-fledged classifier trained on the whole dataset (e.g. including all data points of the  class that the one-shot example belongs to)? After all, the latter should have a better quality, and there seems to be nothing preventing the use of it, given that the proposed framework is a metric rather than a one-shot method.\n- In the motivation (Sec 2.1) the author claims that likelihood-based algorithms or FID-like metrics are not ideal for the test. However this seems to be only argued but not supported by any experiments. Is there any minimal case that can demonstrate its failure in the proposed setting?\n- Is the proposed method applied to other one-shot settings, such as one-shot image segmentation, etc?\n", "limitations": "No issue.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths of this work:\n- It addresses the (relatively) less studied field of one-shot image generation and presents a framework as a metric for assessing the quality. The idea of using two objects is novel.\n- This work is more of an empirical work, and it does a good job in clearing presenting experiments and the analysis. Comparison of representative one-shot image generation methods (with human performance comparable) are shown visually and the discussion is substantial. Overall, the writing quality is high and the content is substantial.\n- The result presented in this paper would provide interesting insights in the one-shot image generation.\n\n\nWeaknesses of this work (If they are properly addressed, I’m happy to increase my rating):\n- Lack of comparison with previously leveraged metrics. It’s true that the proposed method is more holistic than the scattered ones used in previous works for evaluation, such as T-SNE, clustering, however readers would be also interested in how the new framework compass with such measures, for example how diversity and/or accuracy collrates with clustering results.\n- The dataset is limited to Omniglot. Other datasets used in one-shot image generation (for example, CIFAR) should also be  considered as they may show different behaviors, which is the focus of what a metric would be studying. In such a case, human output is not feasible and thus not in the comparison, but still the impact of different methods, especially with different hyper parameters where the paper paid a lot of attention, would have significance.\n- Some design choices are not clear (see Questions below)\n\n\n# Changes during/after rebuttal: \nIncreased score from 6 to 7.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AHh-31YTlqS", "reviewer": "Reviewer_D3RB", "summary": "This paper studies the problem of few-shot generation on the Omniglot dataset, with a focus of benchmarking existing algorithms against humans. They claim that existing metrics for generative models aren’t appropriate for evaluating one-shot generalization, and propose a different framework for this purpose. Their framework is based on two aspects: diversity and recognizability. In both cases, measuring these aspects is dependent upon a trained model. In the case of diversity, a feature extractor is used to measure intra-class variability (standard deviation of produced samples in the feature space around a ‘prototype’), and a SimCLR encoder is used for this. For recognizability, they make use of a one-shot classification model, and the accuracy of that model on generated samples is the recognizability measure. They use a Prototypical Network for this. They investigate where VAE-based and GAN-based models fall in this evaluation framework, and how this is affected by various hyperparameters. Due to the setup of the Omniglot dataset, it is also possible to compare against humans (Omniglot contains human-drawn characters which can be thought of as being the result of a human generative process). They found that the neural statistician-based VAE is the closest (on average) to humans in the diversity-recognizability space. ", "questions": "It’s an interesting finding that VAE-STN and VAE-SN better approximate humans for different subsets of visual classes. Would be interesting to see a similar experiment including GAN models too. Also, the authors hypothesize that this is due to the attention of STN providing a better human approximation for simple visual concepts. Isn’t this counterintuitive, as attention can help to break down a complicated image by sequentially looking at different parts? It would be nice to run some additional experiments to examine this hypothesis, e.g. keeping the architecture the same as VAE-STN but removing the attention mechanism and seeing the effects of that on which types of visual contexts it has smallest distances to humans after this modification. It would also be nice to investigate this as a function of the number of attention steps. \n\nIsn’t it counterintuitive that increasing the context size reduces the diversity, since allowing the model to see more examples should increase its understanding of intra-class variability?\n\nIs there intuition on why varying the number of attentional steps for VAE-STN leads to a parabola in the recognizability/diversity space? What happens at the turn-around point? What is the significance of this finding?\n\nHow come different architectures are used for GAN models but not for VAE models?\n\n\n# after the rebuttal\nincreased my score to a weak accept based on the discussion with the authors (please refer to my response to the authors for justification)", "limitations": "Aside from the limitations mentioned earlier in the review, another one is the following.\nWhat if instead of a single image, a few images were available of the new visual concept (few-shot learning instead of one-shot learning)? The proposed framework assumes that in that case they will have to be aggregated into a prototype, but this is not necessarily the best approach. It would be nice to make it more flexible to allow evaluating any method for exploiting more than one ‘shots’.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n[+] The paper is clearly written for the most part (see a couple exceptions in the detailed comments below) \\\n[+] Drawing connections between humans and different generative models is interesting  \\\n[+] The proposed framework is well-grounded, in that it draws inspiration from psychophysics work\n\n# Weaknesses:\n[-] Limited scope \\\n[-] Poorly motivated \\\n[-] Limited significance in the current form \\\n\n\nDetailed comments:\n================\n\nClarity\n=====\nIn Figure 4, it would be clearer to actually show the size of the context set, e.g. via the size of the circle, or color or something else. Currently I assume the direction of the arrow is giving a proxy for this but this is not as informative. \n\nIn Figure 5, in the caption: ‘Bigger squares correspond to the model showing the smallest distance to humans’. This is unclear as it sounds like the bigger the square, the smaller the distance. But these squares for the different models all have the same size and instead it’s their (y-axis) position that captures the distance. Should rephrase to ‘the squares correspond to…’ (i.e. remove the word bigger), if my understanding is correct.\n\nAs a high-level comment, it would have been useful to give some more details about the models used. For example, it is hard to understand exactly what effect we expect the context size to have on VAE-NS without a good enough understanding of that model.\n\n\nMotivation and significance\n=====================\nTo motivate their evaluation framework, the authors claim that existing metrics don’t work well if the training and testing samples are too dissimilar. In Omniglot though, this isn’t really the case. There have been several studies that show that in fact, feature reuse is sufficient for good few-shot learning performance on simple datasets like Omniglot (see [1] for an example in the context of MAML, [2] for a more general example), and in fact it was also demonstrated that good performance can even be achieved on new Omniglot classes without using the support labels at test time [3], which also hints at the simplicity of this generalization challenge. So, is this claim just a hypothesis, or is it based on some evidence on the failure of existing metrics for one-shot generation evaluation on Omniglot? Especially in the weak generalization setting considered here, this might not be an issue.\n\nFurther, for measuring recognizability, it wasn’t clear to me why we need to use a one-shot model as the ‘critic’ for whether the generated samples can be accurately classified. Why can’t we use an ‘oracle’ here that was trained on more shots from the new classes? One might argue that this is okay since this model is used just for evaluation purposes (though perhaps we do want an *example-level* split where e.g. this oracle is trained on support images but not query images from the new classes). The current setup seemed unnecessarily restrictive to me and not well motivated. If we can use an oracle trained on more shots, this also sidesteps the other difficulty mentioned by the authors as a limitation that prevents us from using existing metrics / evaluation framework for this problem.\n\nFinally, why did the authors decide to use a different model (SimCLR) to get the encoder used for measuring diversity compared to the model used for recognizability (Prototypical Networks)? Can’t we use the same model for both of these? (and ideally report results with different choices for this model)\n\nLimited scope\n===========\nDespite this paper being largely empirical, there is a limited number of models experimented with. Why not also compare against the few-shot generalization model in [4], for example?\n\nOmniglot is also a very simple dataset and few-shot learning research (at least for classification) has drifted from it these days [5]. It would be nice to report results on more modern datasets. Quickdraw, which is mentioned in the discussions, can be a good option as it still allows comparisons to humans.\n\nSignificance\n===========\n\nAll in all, due to the limited scope and insufficient motivation of the proposed framework and set of experiments, I felt that the significance of the findings is low. Further, some findings like GANs lacking diversity are already known (e.g. mode collapse is a known problem for GANs). Verifying that the same is true in the one-shot scenario is interesting, but not too surprising, especially in this weak generalization case studied in this paper.\n\nReferences\n=========\n[1] Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML. Raghu et al. ICLR 2020.\n[2] A Closer Look at Few-shot Classification. Chen et al. ICLR 2020.\n[3] Are Few-Shot Learning Benchmarks too Simple ? Solving them without Task Supervision at Test-Time. Huang et al.\n[4] One-shot Generalization in Deep Generative Models. Rezende et al. ICML 2016.\n[5] Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples. Triantafillou et al. ICLR 2020.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6NxdWM0ozNa", "reviewer": "Reviewer_69WY", "summary": "There exist metrics for measuring the quality and consistency of standard generative models, but these are not applicable to the one-shot setting. Therefore, the authors of this paper introduce and quantify a diversity-recognizability space where single shot generated samples can be evaluated. Furthermore, these two axes make it straight-forward to compare the generated samples to samples from a human model (a brain). Extensive experiments establish the solidity of these measures for the Omniglot dataset.", "questions": "1.It is mentioned on lines 272-274 that the number of attentional steps is less imporatnt than the number of context samples; how do you see the relevance of $\\beta$ relative to the number of context samples? \n\n2. How relevant is context for the Omniglot dataset? Visibly, it does play a role in the experiments, but I am curious how you interpret it.\n\n3. In the last paragraph of the conclusions, you place the most importance on creativity and the diversity part of your introduced metric. Does this mean that you think that the field should move away from GAN-based models when it comes to human-like generated data?\n", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a strikingly solid paper, which presents a clear and novel idea. The result on how GAN-based architectures and VAE-based approaches place in different parts of the diversity-recognizability scale is highly interesting. Each experiment is well-motivated (Section 3), and their results are thoroughly transparent and well discussed. The diversity-recognizability space also presents a very appealing way to think about evaluating generative models, well situated in the psychophysics and cognitive science literature, to the best of my knowledge. Relevant architectures and methods are used both for the feature extraction as part of the metrics, as well as for the one-shot generation. It is difficult to find something to remark on. The one thing would be the fact that the method is run on a single dataset, but in my view this should be left for future work, since a very clear and useful framework is introduced here for anyone to pick up – and the experiments are extensive even for this one dataset. Thus, I do not count this as a weakness. Last, the paper is highly readable and well-structured. I believe that it can be of large value to the community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept