Latent Representation Matters: Human-like Sketches in One-shot Drawing Tasks
Authors: Victor Boutin
Rishav Mukherji
Aditya Agrawal
Sabine Muzellec
Thomas FEL
Thomas Serre
Rufin VanRullen
Venue: NeurIPS 2024 poster (2024)
Keywords: Neuroscience
Cognitive Science
One-Shot Generative Models
Latent Diffusion Models
Human Machine alignment
Human-Machine comparison
Primary area: neuroscience_and_cognitive_science
TL;DR: Here, we demonstrate that brain-inspired regularizations in latent diffusion models close the gap between humans and machines on the one-shot drawing task.

Abstract: Humans can effortlessly draw new categories from a single exemplar, a feat that has long posed a challenge for generative models. However, this gap has started to close with recent advances in diffusion models. This one-shot drawing task requires powerful inductive biases that have not been systematically investigated. Here, we study how different inductive biases shape the latent space of Latent Diffusion Models (LDMs). Along with standard LDM regularizers (KL and vector quantization), we explore supervised regularizations (including classification and prototype-based representation) and contrastive inductive biases (using  SimCLR and redundancy reduction objectives). We demonstrate that LDMs with redundancy reduction and prototype-based regularizations produce near-human-like drawings (regarding both samples' recognizability and originality) -- better mimicking human perception (as evaluated psychophysically). Overall, our results suggest that the gap between humans and machines in one-shot drawings is almost closed.

Latent Representation Matters: Human-like Sketches in One-shot Drawing Tasks
Victor Boutin 1.2.3 Rishav Mukherji', Aditya Agrawal,, Sabine Muzellec2.3, Thomas Fel
Thomas Serre Rufin VanRullen 1.2
Artificial and Natural Intelligence Toulouse Institute, Université de Toulouse, Toulouse, France. Centre de Recherche Cerveau Cognition CNRS, Universite de Toulouse, France CCannyy Institute for Brain Science, Brown University edu
victor_boutinn
Abstract
Humans can effortlessly draw new categories from a single exemplar, feat that has long posed challenge for generative models. However, this gap has started to close with recent advances diffusion models. This one-shot drawing task requires powerful inductive biases that have not been systematically investigated. Here, we study how different inductive biases shape the latent space of Latent Diffusion Models (LDMs). Along with standard LDM regularizers (KL and vector quantization), we explore supervised regularizations (including classification and prototype-basee representation) and contrastive inductive biases (using SimCLR and redundancy reduction objectives). We demonstrate that LDMs with redundancy reduction and prototype- based regularizations produce near- hear-human--ae numan-like drawings (regarding both samples' recognizability and originality) better mimicking human perception (as evaluated psychophysically) Overall, our results suggest that the gap between humans and machines in one-shot drawings is almost closed.
Introduction
For cognitive scientists, human drawings offer a window into the brain, providing tangible insights into its visual and motor internal processes [I]. For instance, drawings have been used in clinical settings to screen for perceptual impairments following brain trauma or Alzheimer's 's disease 22 3, to assess perceptual disorders in autistic individuals 4H6 or investigate perceptual changes during child development [7] (see [ for a recent review). Drawing tasks have also proven instrumental for exploring how the brain generalizes to novel visual categories I9I1.. Cognitive psychologists routinely use the one-shot drawing task to understand how human observers can reliably form new object categories from just one exemplar 12 13 From computational viewpoint, this task is ill-defined because of the infinite number of possible sets of samples that could associated with that effortlessly produce drawings that not only easily recognizable but
exemplar. Yet, humans also original (i.e., sufficiently distinct from the reference exemplar) [12). This remarkable capability suggests that the brain leverages powerful representational inductive biases yet be discovered to form novel categories.
Computer scientists have started to make progress in identifying some of the inductive biases for machine learning algorithms to learn from limited data, For one-shot classification tasks, particularly effective representational inductive bias is to design embedding space where samples of the same category, whether seen during training or not, cluster closely. This approach spans a wide range of models ranging from representations learned via contrastive objective functions [64, prototype- based representations 17 [8] or metric matching losses [19 20). Conversely, for one-shot generation tasks, researchers have preferred architectural over representational inductive biases. For instance,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

novel architectures based on Generative Adversarial Networks (GANs) or Variational Auto- auto--Ecooders (VAEs) have incorporated forms of spatial attention [2T] or contextual integration [22 24 Recent
advances in diffusion models 25 26 make them particularly promising for one-shot generation tasks.
Indeed, clever conditioning on a context vector [2 or directly using guidance from the exemplar [27]
has led to powerful one-shot diffusion models (88] Such guidance mechanism has also proven successful in Latent Diffusion Models (LDMs) (29], which use a Regularized AutoEncoder (RAE) to compress input data and a diffusion model to learn the RAE's latent distribution. These diffusion models have started to close the gap with humans in the one-shot drawing task [30] (see section for
related work one-shot learning). While better conditioning mechanisms have driven improvements in one-shot generative models, the potential of shaping their input space with representational inductive biases inspired by one-shot classification remains largely unexplored. This raises the question: "Do representational inductive biases from one-shot classification help narrow the gap with humans in one-shot drawing tasks? ?"
In this article, we use Latent Diffusion Models (LDMss (2 to address this question. LDMs combine the flexibility of the Regularized AutoEncoder (RAE), in which one can seamlessly include various representational inductive biases in the latent space via regularization, with the high expressivity of the diffusion model. Herein, we study the impact of 6 different regularizers corresponding to distinct representational inductive biases. They are categorized into 3 groups. The first group, which serves as baseline, includes the KL and the vector quantization regularization approaches typically used in LDMs [29]. The second group involves supervised regularizers: a classification loss that promotes discriminative features mapping with categorical training labels and a prototype-basee objective function that clusters samples with their respective prototypes in an embedding space. The third group features contrastive learning regularization schemes with the SimCLR and Barlow losses. The SimCLR R objective function keeps a sample and its augmented view close in the embedding space but far apart from other samples' views. In contrast, the Barlow loss ensures that features of similar samples are decorrelated from those of dissimilar ones.
We compare those regularized LDMs against humans on the one-shot drawing task. Such task offers a leveled playfield in which humans and machines can create sketches that are directly comparable using established evaluation frameworks [1 30 12) (see section 2 for related work). More specifically, our comparison focuses on two metrics to evaluate the quality of sketches produced by humans and machines based on how distinct from the exemplar and how recognizable they are [31] - and on the alignment between humans' and machines' perceptual strategies. For the latter, we describe novel method to generate importance maps highlighting -iggoor-diaggosttcc features in LDMs. These maps are then directly compared against importance maps derived from human observers obtained through psychophysies experiments. Our results show that LDMs using prototype- (with the Barlow twin objective) regularization techniques are further
based and redundancy-reducctoo closing the gap with humans. These results are supported by both the sample's similarity and the feature importance maps alignment. Overall, our contributions can be summarized as follows:
We introduce novel representational inductive biases in Latent Diffusion Models. In partic- ular, we draw inspiration from losses that have proven effective in one-shot classification tasks (with the prototype-based, Barlow and SimCLR objective functions) to regularize the latent space of LDMs.
We derive novel explainability method to generate LDMs' feature importance maps that highlight category diagnostic features.
We systematically compare the sketches and feature importance maps derived from humans and machines, and we show that LDMs with prototype-based and Barlow regularization significantly narrow the gap with humans on the one-shot drawing task.
Our work underscores the critical role of well-designed representational inductive biases in achieving human-like performance in one-shot drawing tasks. It also sets the stage for developing generative models that are better aligned with humans.
2 Related work
Representation learning for one-shot classification tasks: unseen samples (from the query set) close to the exemplars (in the support set) has proven effective in one-shot classification. The historical approach, called metric learning, aims at creating a feature
Learning representations that bring
2

---

space in which the distances between the query and support sets are preserved 120 19 32 B3 However, the limited number of samples in the support set restricts these networks' ability to recognize novel classes. This limitation becomes more pronounced in the one-shot setting as the support set contains only one sample (the exemplar). To address this, the field has shifted towards prototype-based representations. Rather than trying to preserve the distances between query and support samples, such networks learn an embedding space in which the query samples cluster near Contrastive learning, self-supervised learning approach, offers another effective solution to mitigate sample scarcity by augmenting the training set. This method learns an embedding space where positive pairs (a sample and its augmented version) are close together, and distant from negative pairs (augmented views from different instances) [14] 15 36 39
the support samples [17 34 35
Among alternative methods, the SimCLR algorithm [14] uses cosine similarity between samples
whereas the Barlow-twins network [5] leverages the correlation matrix between features to dissociate positive and negative pairs. In this article, use the prototype-based 177, the SimCLR RR [14] and the Barlow twins [5] objectives to regularize RAFs latent space. For additional mathematical details, see section 4.1 for the prototype-based loss and section A.2.3 for SimCLR and Barlow.
Generative models for one-shot image generation tasks: Some of the main techniques involve including information from the support set into the generative process, method known as condition- ing. For instance, the Neural Statistician uses a context vector containing summary statistics from the support sets, which is then concatenated with VAE latent space [22 24 40). Similarly, GANs leverage a compressed representation of the support set as a conditioning mechanism [23]. Such a mechanism has also been used successfully
either condition 44 29 guide the denoising process of diffusion models (27 28 and latent diffusion models [29 Here, we leverage L.MMs
with classifier-free guided diffusion models (77 Such diffusion process has been shown to well
approximate human drawings in one-shot drawing tasks (30)
Human-machine comparison in one-shot drawing tasks: Cognitive scientists have developed various methods to compare the generalization abilities of machines and brains on drawing tasks. Lake et al. [44] introduced the Omniglot challenge in which both humans and machines are tasked with drawing symbols from categories represented by a single exemplar (see [45] for a review on the challenge). The authors evaluated the drawings' recognizability in visual Turing test where humans (or classifiers) had to distinguish between human-drawn and machine- deaieneeeeertedd symbols ([T]. Additional metrics, including classification uncertainty and semantic similarity, were also used to compare drawings produced by humans and machines under different time constraints [46] g While these evaluation frameworks provide useful insights into sample's 's recognizability, they do not measure how the diversity of model-generated samples compares to those created by humans. The "originality vs. recognizability" framework (3]] mitigates this issue by adding the originality metric. An originality score quantifies the similarity between the original exemplar and its corresponding variations (see section [51 for details on this evaluation framework). This evaluation framework has been used to benchmark the generalization performance of mainstream generative models Diffusion models [47]. GANs [48] and VAEs [49 against humans in the one-shot drawing setting [30]. Although Diffusion models come closest to human performance, a noticeable gap remained in this study. In this article, we use the "originality vs. recognizablility" framework from Boutin et al. [1 to evaluate representational inductive biases in Latent Diffusion Models. In particular, we demonstrate that effective biases in one-shot classification tasks also prove efficient in the one-shot drawing task.
3 Datasets
As done in previous work [31 30
M. we use the Omniglot [ and the QuickDraw-FS, [30] datasets to compare humans and machines on the one-shot drawing task. These datasets, made of handwritten symbols or drawings, offer a fair environment for comparing the generation abilitics of humans and machines [[]] 46 31 30). It is important to note that natural images generation is a task beyond human capability, making it unsuitable for a fair comparison between humans and machines.
Omniglot contains 1, 623 categories of handwritten characters from 50 different alphabets, with 20 samples per class [II. This article uses a downsampled version of the dataset (size: 48 48 pixels). We train the models on training set composed of all available symbols minus 3 symbols per alphabet left aside for the test set (similar to 21)). All the results on the Omniglot dataset are in the Appendix (see A.6)
3

---

QuickDraw-FS is made from drawings of the Quick, Draw ! challenge [50]. In this challenge, human participants are asked produce drawings in less than 20 seconds when presented with an object name. The categories are, therefore, made with semantically consistent samples that do not necessarily represent the same visual concept (e.g., the "phone" object category might contain corded phones, smartphones, phones with rotary dials, etc). The Quickraw-FS dataset mitigates this issue with categories representing the same visual concepts (see A.1 for more details). This dataset is ideally suited for purely visual one-shot generation tasks [30]. It contains 665 categories with 500 samples each. The training set is made of 550 randomly selected categories, and 115 are left aside for the testing set. We downsampled the drawings to 48 x 48 pixels to keep computational resources manageable.
For each category in both datasets, we extract 'prototypccal' sample, selected in the center of the category cluster to condition the one-shot generative models (see A.1 for more details on the exemplar selection).
4 One-shot Latent Diffusion Models
Diffusion model
Zzy
ZT
The one-shot image generation task involves synthesizing variations of visual concept not seen during training. Let x E Rº denote the image variation and y E RR the exemplar. Latent Diffusion Models (LDMs) are composed of 2 distinct stages: a first stage leverages a Regularized AutoEncoder (RAE) that extracts latent representation z E Rd (d < D) for each image (see green boxes in Fig. D and second stage consisting of diffusion model that learns the latent distribution (orange boxes in Fig. [ In the one-shot setting, the diffusion model is conditioned by zy. the latent representation of y. We call c the category label of the training set (a one-hot vector).
--
Pw
21
L reg (z)
= Z0
M
RAE
9 Encoder
Po
Decoder
4.1 Regularized Auto- Encodesss (RAEs)
x'
To describe the RAE, we use a probabilistic formulation in which 96 (z|x) is the recognition model (or the encoder), and pa (x (z) is the decoder. We train the RAEs by mini- mizing CRAE [D. In this equation, the first term is reconstruction loss (computed with a l2 distance), and the second term (Greg) covers a wide range of regularization losses. Greg includes the representational inductive biases study in this article. Those inductive biases fall into groups: the standard LDM regularizers, the supervised regularizers, and the contrastive regularizers.
Latent Diffusion Models stack a diffusion model (orange) on top of an Auto-Encoder (green).
Figure 1:
min CRAF s.t. LRAF= = -Ezogo x)) [log po (x|z)) + BLreg (z) 8.6
Standard regularizers (KL and VQ): The KL divergence in Eq. forces each coordinate of
the latent vector to be distributed following pre-determined distribution (e.g Gaussian distribution, as in the VAE [49). The vector quantized loss in Eq. transforms the continuous latent code
into a discrete code Zg using the nearest entry a codebook = {es]_t with the quantization operator: 2g nz (s.t. fzz z -+ arg mine.11 ||z 1---e eill2 as in the VQ- (5I). This quantization operation being fieeeiffereet,.ble, backpropagation to provide gradient estimator. We provide an extensive mathematical description of the VQ-VAR in
achieved using a stop- gradient operation sy[-]
App. A.2.1
LKL (L(go(z)x)) (p(z)) (with p(z) = (O I))
VAE
(2)
Lvq= (1l (z] zq || |zq] z =
VQ-VAE
Supervised regularizers (Classif. and Proto.): The classification regularizer forces discriminative features by minimizing the cross-entropy between the true labels (c) and the softmax of the logits. Here the logits are learned by linear layer (1.24) stacked on the latent space (Fq. 4.. While the classification loss is supervised by the true categorical labels, the prototype- based loss supervised

---

by the exemplars themselves (as in the Prototypical Net [17].. The prototype-basee base loss learns metric space in which classification can be performed by computing distances between the variations and their corresponding exemplars (i.e., the prototypes) 5. Here, the metric space is linked to the latent space of the RAE through a linear layer (1.5 Intuitively, the prototype-based loss finds an embedding space where the variations will be close (in terms of &g distance) from their prototypes. See A.2.2 for more details.
Ler. (
Classif.
CPR = SpR=Ea,~qa (-v)) (og(ssotmax((]))) (z) h
(zy) ||2]]
Proto. (5)
Contrastive regularizers (SimCLR and Barlow): Contrastive learning algorithms learn represen- tations that are invariant under different distortions (i.e., data augmentations). Here we define two data-augmentation operators, (.) and 7 (-), that transform the variations x into x
(x( and
xB 7 (x), respectively. We denote z A (-(x(x and z (o(- the projection of x and xB into the RAE latent space, respectively. The SimCLR regularizer based on the InfoNCE loss: it maximizes the similarity between the representation of sample and its augmented view while minimizing the similarity with negative pairs (augmented views of different instances) [4 The Barlow regularizer (as in the Barlow twins [15) forces the cross-correlation matrix between 2 and 2B to be as close to the identity matrix as possible. This causes the embedding vectors of distorted versions of samples to be similar while minimizing the redundancy between the components of these vectors. Said differently, the SimCLR loss shapes the space based on the samples' similarity, while the Barlow operates the correlation between the features of the samples. For conciseness, we have included the mathematical derivations and details on the data augmentation we used in
We leverage standard convolutional architectures (from 52 to parametrize both the encoder and the decoder. The resulting autoencoder has a 1D bottleneck = 128 for QuickDraw-F3S and 64 for Omniglot). We refer the reader to App. A for complete architectural and training details of the RAE. In the rest of the article, we evaluate the impact of these regularizations by exploring the effect of 3 (see Eq. D on LDMs.
4.2 Diffusion Model
The LDM second stage is a diffusion model that learns the data distribution in the latent space of the RAE. Diffusion models progressively denoise a pure noise zT ~ (O I) into clean latent representation Zo := z through a sequence partially denoised variables {z ) The goal is then to learn a transition probability P (2 (44) that approximates a noise injection operator v.(.) so that 2 = (8)) = / /0 + VI āge (di, is an hyperparameter of the diffusion schedule, and Gaussian noise). The Denoising Diffusion Probabilistic Model (DDPM) [47] reduces the learning of Pv (Z--1 22) to the optimization of a simple autoencoder ey trained to predict the noise from degraded latent representation z, (see A.4 for mathematical justification):
arg min E poo (1x(0) - c By (g)(-ly)
s.t. (0,, and t~ ((1
(6)
In Eq. Zy denotes the latent representation of the exemplar Eq. c could be interpreted as denoising score matching objective [53]. so the optimal model eg- matches the following score function:
Vz, log Pw- (Ze (yy) R
VI- āg Eve (Ze, Zy)
The autoencoder-like model (. Zy, 1) is a ID Unet conditioned on the time variable and zy (see AA.433 for details on the architecture and the training of the Unet). Herein, we use a classifier-free guided version of the DDPM (27 with the following score function:
V Zz log (z.lzy) (1 +)) log (z, (zy) V log Pap (z.)
(8)
This formulation introduces a guidance scale (we use 7=1) to tune how much the conditioning signal influences the final score. Such a formulation has shown effective in one-shot settings 28 30) Note that each term on the RHS of Eq. X is computed with the same network eg. using Eq. 6, is simply conditioned on a non-informative signal to compute log Pa- log (z:). We remind the reader that the training of the diffusion model begins only after the RAE training is complete, and occurs exactly
5

---

identically, regardless of the type of regularization used. The quality of images generated by the diffusion model thus directly serves compare the different regularizations. The code to train all described models is available at httpp://anonymous.. 4open. https//ccc/////aaeettatttteeeeee 526B
5
Results
5.1
Originality vs. Revognizabilty
To compare humans and machines in the one-shot drawing task, we first use the originality vs. recognizability framework [31 30]. This framework leverages 2 critic networks to evaluate the samples produced during the testing phase. The recognizability is quantified using the classification accuracy of a one-shot classifier [17], while the originality is measured using the average distance between the variations and their corresponding exemplars. This distance is computed in the feature space of self-supervised model [4]. Importantly, both human-drawn and machine-generate samples are evaluated using the same 2 critic networks. This ensures that any potential biases in the critic networks are minimized, leading to more balanced comparative analysis. Note that the originality is normalized across all tested models to range between 0 and 1. Here, we use the same originality vs. recognizability framework setting as that used Boutin al. [30]. Importantly, the originality vs. recognizability plots should be interpreted based on how close the models are to the human data point (grey star in Fig. 3). rather than focusing solely on their individual originality or recognizability scores. In simple terms, model that effectively mimics human drawings should fall near the human data point. Note also that there is an inherent trade-off between originality and recognizability: while recognizability assesses how likely the data point falls within the classifier decision boundary, originality measures how 'diffuse' the sample distribution is. Therefore a very original agent (producing highly diverse samples) will tend to have low recognizability as the samples are likely to fall outside of the classifier decision boundary.
In Fig. 3 we first evaluate how increasing the regularization weights (i.e. the 3 in Eq. 1 for each regularizer (taken separately) affects the similarity LDM samples to human drawings. To do so, we report the original- ity and the recognizability values for LDM samples trained
Exemplars
with different / values (see data points in Fig. a parametric fit (least curve fitting methods [4
3 We use
S? A
to illus-
No reg.
trate how increasing 3 affects these scores (see A.5 for more details on the parametric fit computations). We observe a similar concave shape for all curves. As 3 starts increasing, the recognizability improves while the originality decreases (except for VQ regularizer). Beyond certain 3 value, the recognizability declines, and the originality increases. In par- ticular, the maximum recognizability values for KL and VQ (obtained with 8kL= = 10 and Bvq = 5) match those of a diffusion model trained in the pixel space and barely exceed those of a non-regularized LDM (see Fig. Bl). Increasing the weight of the prototype-bart reduces the distance to human compared to the classifica- tion regularizer (the minimal distance human 0.04 for 3PR 5- 102 vs. 0.15 for Ber. 5, see Fig. 3D). Among the contrastive regularizers, Barlow regularization significantly reduces the distance to human compared to the SimCLR one (the minimal distance to human is 0.08 with 3BAR = 30 vs. 0.12 with SSimLLR 10-2 see Fig, B+). A visual inspec- tion of the samples tends to corroborate these results (see Fig. 2 and A.7 for more samples). We observe similar trends for all tested regularizers on the Omniglot dataset (see A.6).
Proto.
Classif.
Barlow
TT
SimCLR
regularizer substantially
12
T
KL
P
VQ
Figure 2: Samples from LDMs w/ different regularizers. The LDMs correspond to the larger data points
in Fig. 3
Overall, our findings indicate that not all regularizers are created equal. For supervised regularizers (see Fig. 3b), the prototype-based regularizer generates more recognizable samples compared to
the classification regularizer. This is expected since the classifier focuses separating categories in the training set, which may not be ideal for unseen categories the one-shot setting [19 [7]. In contrast, the prototype-be based regularizer clusters samples near their prototypes, leading to less

---

a)
b)
95
95
Classifier
95
SimCLR
VQ
Proto.
Barlow
90
90-
Recognizabiltty (%)
85
85
85-
80
80
80
75
75
70
70
70
0.5
0.55
0.6
0.65
0.7
0.5
0.55 Originality
0.6
0.65
0.7
0.5
0.55
0.6
0.65
0.7
Originality
Originality
Humans
No regularization
pixel baseline
Figure 3: Effect of increasing the regularization weights the originality recognizabilitt framework (QuickDraw-F'S dataset). Each data point represents an LDM trained with different values regularization weights (5). The curves represent the parametric fits, oriented the direction of an increase of 3. a): For the LDMs with "standard" regularizers, the 3 is applied on the KL (CxL in Eq. 2 or on the VQ regularizers (Lvq in Eq. 3.. b): For the supervised regularizers, the / is applied on the CL (Cor. in Eq. 4 or on the prototype-based regularizers (L in Eq. 5) c): For the contrastive regularizers, the 3 is applied on the SimCLR (CsimCLn Eq. 14) or on the Barlow regularizers (arar in Eq. 15). See A.5 for more information on the range of 3 we have explored for each regularizer. Larger data points indicate models whose performance closer to that of humans for each type of regularization. For comparison, we include LDM leveraging non-regularized RAE (hexagon marker) and a diffusion model trained directly on the pixel space (cross marker). The human performance corresponds to the recognizability and originality computed on human drawings (shown with a grey star).
overfitting and better transferability, which valuable for few-shot tasks [55] Our experiments confirm that the prototype-based regularizer generalizes better for one-shot drawing. In Fig. 3 the Barlow regularization outperforms the SimCLR regularizer in recognizability, likely due to Barlow's 's effective feature disentangling [5]. These features transfer well to new datasets, making Barlow more suitable for the one-shot drawing task. Overall, our results demonstrate that effective representational inductive biases in few-shot learning also enhance performance in one-shot drawing.
We now study the effect of the regularizers when they are used in combination. In particular, we have systematically explored the following combinations of regularizers Barlow Prototype (Fig. 4h), SimCLR. + Prototype (Fig. 4p). KL + Prototype (Fig. 4.). VQ + Prototype (Fig. 41). We observe that the Barlow Prototype and the KL + Prototype combinations produced the most human-like samples. Those regularizer's combinations recognizability is significantly higher compared to using each regularizer alone. This suggests that clustering samples around their prototypes (using the Prototype regularizer) within disentangled space (achieved via the KL or Barlow regularizer) enhances generalization. In contrast, the VQ + Prototype and the SimCLR + Prototype combinations show little no improvements.
particularly as in both cases the combined
5.2 Comparing humans and LDM perceptual strategies
While the originality vs. recognizability framework allows us to compare human and machine performances in the one-shot drawing task, it does not reveal the strategies each uses to generalize to new categories. To address this, we aim to compare the visual strategies more directly via feature importance maps. These maps emphasize the most salient features to recognize drawing.
7

---

a)
95-
95
95-
95
*
Recognizabllity (96)
90-
90-
90-
90
85-
85
85
85
80
80
80
80
Barlow
SimCLR
KL
VQ
75
Proto.
75
Proto.
75
Proto.
75
Proto.
Barlow Proto.
SimCLR Proto
Proto
VQ Proto.
70
70
70
0.5
0.6
0.7
0.5
0.7
0.5
0.7
0.5
0.6
0.7
Originality
Originality
Originality
Originality
x
Humans
No reg.
pixel
Figure 4: Combined effect of the regularization weights the originality recognizability framework (QuickDraw-F') dataset). Each data point represents an LDM trained with a combi- nation of 2 different regularizers. All combinations include the prototy tettyyee-baseee regularizers. The curves represent the parametric fits, oriented in the direction of an increase of 33 a): Barlow and prototype-hased regularizers applied either separately (plain lines) in combination (dashed-line). When applied in combinations, only the weight of the prototype-based regularizer is modified (with = 30 for Barlow). b): SimCLR and prototype-basse regularizers. When applied in combina- tions, only the weight of the
regularizer is modified, the SimCLR is set to 3 = 1. c): KL and prototye--basee regularizers. When applied in combinations, only the weight of the mototype-baseed regularizer is modified, the KL is set to 3 le 3. d): VQ and prototype-base based regularizers. When applied in combinations, only the weight of the prototype-bite based regularizer is modified, the VQ is set to B = 20. See caption in Fig.
Previous research has demonstrated that by summing the absolute values of the diffusion scores (V 2; log log ky))) throughout all diffusion steps, features in a diffusion model's generation process [30). Here, we adapt this heuristic to make it compatible with LDMs. This involves projecting each intermediate noisy state (z+) back pixel space using the RAE's decoder (po(- )). To do so, we use the chain rule, and we multiply each diffusion score by the Jacobian of the RAE decoder w.r.t x₃ (denoted variation x and its corresponding exemplar y, we can therefore compute a heatmap using Eq. 9 (see A.8. I for mathematical details). Then, we average 10 of these heatmaps, obtained with the same exemplar but for different variations belonging to the same category. This process allows us to mitigate intra-class variations while focusing on sppecoe--specific features. We call this average the feature importance map (see A.8.2 to visualize feature importance maps).
create heatmaps that highlight salient
(x+)). (xx)) For each
((x y)
(1 Vz, log pa (Ztlzy) with Zy ~ go(-ly)
(9)
i=0
We derived human feature importance maps using psychophysical data from Boutin et B3 (data
shared by the original authors). The authors collected human saliency maps through an online psychophysies experiment based on similar protocol to the ClickMe experiment [56). In this experiment, participants were presented with drawings and were asked to draw regions important for categorization (see App. in (30) for more details on the experimental protocol). We averaged the heatmaps across participants and drawings within the same category to obtain the feature importance maps we compared with those machines (sec A.8.3 for visualizing feature importance maps).
In Fig 5) we compare humans and LDMs feature importance maps. For each regularizer, we select the LDMs that produce the most human-like sketches (highlighted with larger data points in Fig. 3) Note that we exclude the VQ -Qreelulaiezed LDM from this analysis because it produces irrelevant feature importance maps, possibly due to the non- differentiability of the quantization process (see Fig. A.15) In Fig. 54, we showcase examples of the obtained feature importance maps for all other LDMs' regularizations (see also A.8.2 and for humans (see also A.8.3). We qualitatively observe that the LDMs regularized with the Barlow and the prototype-baset objectives tend to focus on sparse features. This particular aspect seems to shared with the human feature importance maps. We compute the Spearman rank correlation to quantify the similarity between human and machine
8

---

a)
b)
Human
Spearman w/ human
0.65
0.75
0.85
Proto.
Supervised
Classif.
Barlow
Contrastive
SimCLR
Standard
KL
reg.
Human Figure 5: Feature importance maps comparison. a) The visualizations include feature importance maps for humans (top row) and LDMs (six bottom rows). All the maps are overlaid on exemplars. Hot vs. cold pixels show image locations that are more vs. less important. Maps for humans were computed using psychophysical data from Boutin
(3). For the LDMs, they are obtained for each category by averaging db x y) (see Eq. over 10 different image variations (x) belonging the same category. The models' maps are computed on the more human-like LDMs for each regularization (larger data points in Fig. 5 b) Spearman's rank correlation coefficient between humans and LDMs feature importance maps. The error bar is computed as the standard deviation of the Spearman coefficients all categories (25 in total). Stars indicate the p-value (*** p 10-3 and ::p < 5.10-2) of pair-wise statistical test between models (Wilcoxon signed-rank test, see see A.8.4). The black line corresponds to an LDM without any regularization. The dashed line is the human consistency (0.88), quantifies how much two populations of humans agree with each other on feature importance maps (see A.8.3 for details the human consistency computation).
feature importance maps (see Fig. To make sure that the correlation comparison between the different LDMs is significant, have computed pairwise statistical tests (Wilcoxon signed-rank test, see A.8.4). Our results show that all considered regularizations correlate significantly more with human feature importance maps than non-regularized LDMs.. In addition, the prototype-base regularizer produces the feature importance maps with the highest correlation with humans and is significantly above all other tested regularizations (p 10-3). In the human-alli -almamnt ranking, the Barlow-reee cewllowwrruuurizeedd LDM follows the prototype-based LDM, also showing significantly higher Spearman correlation coefficient than KL, classification, Sim regularizers (p 10 10-9). All other pair-wise wise statistical tests (between KI, classification, SimCLR) are not significant enough to draw meaningful ranking.
6 Conclusion
In this article, we used Latent Diffusson Models (LDMs) study the effect of representational inductive biases for one-shot drawing tasks. We explore 6 different regularizers: KL, vector quantization, classification, prototype-based, SimCLR and Barlow regularizers. We analyzed the human/LDM, alignment from two (independent) perspectives: their performance relative to humans on the one-shot drawing task (with the recognizabilityy vs. originality framework in section 5.1) and the similarity of the underlying visual strategies (with the feature importance maps in 5.2 Overall, we observe clear alignment between the 2 analyses on the following points:
All regularized I. LDMs have an optimal regularization weight (3) where they are more aligned with humans than their non-regularized counterparts.

---

The prototype-base based regularizer is showing the best matches with human performance and attentional strategy.
In the one-shot drawing tasks, the samples' likannli could be further improved by regularizer with either the KL or the Barlow regularizers.
combining the
In conclusion, we observe that all representational inductive biases "are not created equal". However, some of them (prototype-baset and Barlow regularizers) do narrow the gap with humans in the one-shot drawing task.
Limitations
In this article, we tested six representational inductive biases, small number considering the extensive range available the represetaaioon-eamiing literature. This field encompasses hundreds of inductive biases that have proven successful in one-shot classification tasks. Therefore, other representational inductive biases might align better with human performance, both in terms of sample similarity and visual strategy. Our goal wasn't test all possible biases but demonstrate that some of them can significantly narrow the gap with humans in one-shot drawing tasks.
Another limitation of this article lies in the recognizabilityy originality framework to evaluate the drawings. This framework leverages 2 critic networks to evaluate the sample's originality and recognizability. There's ss guarantee these networks align with human perceptual judgments. Thus, the recognizability and originality scores might not reflect human perception accurately. However, since both human and model outputs evaluated using the same pre-trained critic networks, the comparison remains fair.
using
Discussion
is noteworthy that the KL and VQ regularizers, commonly used train LDMs on natural images (as in StableDiffusion (29]) are not the best-performing regularizers in the one-shot drawing task. Our study indicates that the protofype-based and the Barlow regularizers, not tested yet on LDMs trained on natural images, hold a significant potential for enhancing their one-shot ability. From a single image of new vehicle prototype or of new fashion item design, a generative model trained with these regularizers could produce relevant variations ability that current commercial applications still struggle with (see Fig. A.8.5).
Interestingly, the 2 inductive biases that align most closely with humans are directly related to prominent neuroscience theories. The prototype-based objectives provide an instantiation of the prototype theory of recognition and memory (57)61), suggesting that humans use prototype similarity recognize novel objects. Similarly, the Barlow regularization is inspired by Barlow's 's redundancy reduction theory 662 63), which posits that the brain encodes statistically independent features to climinate redundancy (and minimize energy consumption).. The effectiveness of these regularizations provides hints that the brain may use similar inductive biases to generalize to new categories. In terms of brain inspiration, although we use LDMs to model humans' one-shot generation abilities, we do not claim that these neural networks constitute a realistic model of brain processes. It is indeed unlikely that humans generate samples by iteratively denoising random noise. More biologically plausible generative models might further help to obtain better models of human behavior (e.g.. see 68).
With this paper, we highlight how specific representational inductive biases, included in the input space of generative models, can help bridge the gap with human capabilities. We believe these biases effectively as humans do, leading to exciting
will allow advancee models to generalize and create as advancements technology and creativity.
10

---

Aknowledgement
This work was funded by the European Union (ERC. GLoW, 101096017), ANITI (Artificial and Natural Intelligence Toulouse Institute) and the French National Research Agency (ANR-19-PI3A 0004). Additional funding was provided by ONR (N00014-24- 1-20266 and NSF (IIS-191220,, IIS-2402875 and EAR-1925481) Computing hardware supported by NIH Office of the Director grant S10OD02518 via the Center for Computation and Visualization (CCV).

---

References
[1] Judith E. Fan, Wilma A Bainbridge, Rebecca Chamberlain, and Jeffrey Wammes. Drawing as versatile cognitive tool. Nature Reviews Psychology, 2(9):556-560 2023.
[2] Anna Cantagallo and Sergio Della Sala. Preserved insight in an artist with extrapersonal spatial neglect. Cortex, 34(2):163-18))) 1998.
[3] Berit Agrell and Ove Dehlin. The clock-drawing test. Age and ageing. 27(3):399-444 404, 1998.
[4] Laurent Mottron and Sylvie Belleville. A study of perceptual analysis in high-level autistic subject with exceptional graphic abilities. Brain and cognition, 23(2):279-309 1993.
[5] Laurent Mottron, Jacob A Burack, Johannes EA Stauder, and Philippe Robaey. Perceptual processing among high-functioning persons with autism. The Journal of Child Psychology and Psychiatry and Allied Disciplines, 40(2):203-211, 1999.
[6] Nicholas Humphrey. Cave art, autism, and the evolution of the human mind. Cambridge Archaeological Journal, 8(2):155-111, 1998.
[7] Annette Karmiloff-Smith. Constraints representational change: Evidence from children's drawing. Cognition, 34(1):57-83, 1990.
[8] Bria Long, Judith E Fan, Holly Huey, Zixian Chai, and Michael C Frank. Parallel developmental changes in children's 's production and recognition of line drawings of visual concepts. Nature Communications, 15(1): 2024.
[9] Tomer D Ullman and Joshua B Tenenbaum. Bayesian models of conceptual development: Learning as building models of the world. Annual Review of Developmental Psychology, 2: 533-558, 2020.
[10] Joshua Brett Tenenbaum. Bayesian framework for concept learning. PhD thesis, Mas- sachusetts Institute of Technology, 1999.
[11] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 15006266):13322113333 2015.
[12] Henning Tiedemann, Yaniv Morgenstern, Filipp Schmidt, and Roland W Fleming. One-shot generalization in humans revealed through a drawing task. Elife, 1::75485, 2022.
[13] Henning Tiedemann, Yaniv Morgenstern, Filipp Schmidt, and Roland W Fleming. Probing feature spaces of object categories with drawing task. Journal of Vision, 23(9):4765 2023.
[14] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[15] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International conference machine learning. pages 12310-1222 -2320. PMLR, 2021.
[16] Chen Liu, Yanwei Fu, Chengming Xu, Siqian Yang, Jilin Li, Chengjie Wang, and Li Zhang. Learning few-shot embedding model with contrastive learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 8635-8643, 2021.
[17] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[18] Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04000 2020.
[19] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
12

---

[20] Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille. 2015.
[21] Danilo Rezende, Ivo Danihelka, Karol Gregor, Daan Wierstra, al. One-shot generalization in deep generative models. In International conference on machine learning, pages 1521-1529. PMLR, 2016.
[22] Harrison Edwards and Amos Storkey. Towards neural statistician. arXiv preprint arxiv:1666.022185, 2016.
[23] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint 1r 2017.
[24] Giorgio Giannone and Ole Winther. Hierarchical few-shot generative models. arXiv preprint vrXiv:2110.122777 2021.
[25] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
[26] Jascha
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages
PMLR, 2015.
[27] Jonathan Ho and Tim Salimans. Classifier-fre diffusion guidance. arXiv:2207.12598 2022.
arXiv preprint
[28] Bin Cheng, Zuhao Liu, Yunbo Peng, and Yue Lin. General image-to-image translation with one- shot image guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22736-22748 2023.
[29] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[30] Victor Boutin, Thomas Fel, Lakshya Singhal, Rishav Mukherji, Akash Nagaraj. Julien Colin, and Thomas Serre. Diffusion models as artists: Are we closing the gap between humans and machines? Proceedings of the 40th International Conference on Machine Learning, 2023.
[31] Victor Boutin, Lakshya Singhal, Xavier Thomas, and Thomas Serre. Diversity recognizabil- ity: Human-like generalization in one-shot generative models. Advances in Neural Information Processing Systems, 55:20933-20945 2022.
[32] Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199-1208, 2018
[33] Fusheng Hao, Fengxiang He, Jun Cheng, Lei Wang, Jianzhong Cao, and Dacheng Tao. Collect and select: Semantic alignment metric learning for few-shot learning. In Proceedings of the IEEE/CVF international Conference on Computer Vision, pages 8460-8469, 2019.
[34] Rinu Boney and Alexander Ilin. Semi-supervised few-shot learning with prototypical networks. CoRR abs/1711.118556 2017.
[35] Fangyu Wu, Jeremy Smith, Wenjin Lu, Chaoyi Pang, and Bailing Zhang. Attentive prototype few-shot learning with capsule network-based embedding. In Computer Fision-ECCY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII 16, pages 237-253. Springer, 2020.
[36] Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint arXiv: 1.073333 2016.
13

---

[37] Jean-Base Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch. Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervise learning. Advances in neural information processing systems, 33:21271-21288 2020.
[38] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighbor contrastive learning of visual representa- tions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9588 9597, 2021.
[39] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[40] Luke B Hewitt, Maxwell Nye, Andreea Gane, Tommi Jaakkola, and Joshua B Tenenbaum. The variational homoencoder: Learning to learn high capacity generative models from few examples. arXiv preprint arXiv: 1807,9999, 2018.
[41] Giorgio Giannone, Didrik Nielsen, and Ole Winther. Few-shot diffusion models. arXiv preprint arXiv:2255.15463, 2022.
[42] Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Sindiffusion: Learning diffusion model from a single natural image. arXiv preprint arXiv:2211. 2022.
[43] Vladimir Kulikov, Shahar Yadin, Matan Kleiner, and Tomer Michaeli. Sinddm: A single image denoising diffusion model. In International Conference on Machine Learning, pages 17920-17930. PMLR, 2023.
[44] Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
[45] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. The omniglot challenge: a 3-year progress report. Current Opinion in Behavioral Sciences, 29:97-104, 2019.
[46] Kushin Mukherjee, Holly Huey, Xuanchen Lu, Yael Vinker, Rio Aguina- Ariel Shamir, and Judith Fan. Seva: Leveraging sketches to evaluate alignment between human and machine visual abstraction. Advances Neural Information Processing Systems, 36, 2024.
[47] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[48] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-1 144, 2020.
[49] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. 1rXiv:1312.6114, 2013.
arXiv preprint
[50] Jonas Jongejan, Henry Rowley, Takashi Kawashima, Jongmin Kim, and Nick Fox-Gieg. The quick, draw!-ai experiment. Mount View, CA, accessed Feb, 17(2018):4, 2016.
[51] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[52] Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. arXiv preprint arXiv: 12436, 2019.
[53] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13356, 2020.
[54] M Grossman. Parametric curve fitting. The Computer Journal, 14(2):169-172. 1971.
14

---

[55] Xiaoxu Li. Xiaochen Yang, Zhanyu Ma, and Jing- Jing-Hao Xue. Deep metric learning for few-shot image classification: review of recent developments. Pattern Recognition, 138: 109381, 2023.
[56] Drew Linsley, Dan Shiebler, Sven Eberhardt, and Thomas Serre. Learning what and where to attend. arXiv preprint arXiv: 180500889, 2018.
[57] Michael I Posner and Steven W Keele. On the genesis of abstract ideas. Journal of experimental psychology, 77(31))3353 1968.
[58] Stephen K Reed. Pattern recognition and categorization. Cognitive psychology. 3(3):382-407, 1972.
[59] Donald Homa, Deborah Rhoads, and Daniel Chambliss. Evolution of conceptual structure. Journal of Experimental Psychology: Human Learning and Memory, 5(1):11, 1979.
[60] J David Smith and John Paul Minda. Prototypes in the mist: The early epochs of category learning. Journal of Experimental Psychology: Learning, memory, and cognition, 24(6):1411, 1998.
[61] John Paul Minda and J David Smith. Prototypes in category learning: the effects of category size, category structure, and stimulus complexity. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27(3):775, 2001.
[62] Horace Barlow et al. Possible principles underlying the transformation of sensory messages. Sensory communication, 1(01):217-233, 1961.
[63] Horace Barlow. Redundancy reduction revisited. Network: computation in neural systems, 12 (3):241, 2001.
[64] Rajesh PN Rao and Dana Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical eeceptive-field effects. Nature neuroscience, 2(1):79-87 1999.
[65] Bhavin Choksi, Milad Mozafari, Callum Biggs O'May, Benjamin Ador, Andrea Alamia, and Rufin VanRullen. Predify: Augmenting deep neural networks with brain-inspired predictive coding dynamics. Advances in Neural Information Processing Systems, 34: 0011440614400 14083, 2021.
[66] Victor Boutin, Aimen Zerroug, Minju Jung, and Thomas Serre. Iterative vae as predictive brain model for diitrii-iitiittoonn generalization. SVRHM workshop Neural and Information Processing Systems 34, 2020.
[67] Victor Boutin, Angelo Franciosini, Frédéric Chavane, and Laurent U Perrinet. Pooling strategies vI can account for the functional and structural diversity across species. PLOS Computational Biology, 1010220, 2022.
[68] Victor Boutin, Angelo Franciosini, Frederic Chavane, Franck Ruffier, and Laurent Perrinet. Sparse deep predictive coding captures contour integration capabilities of the early visual system. PLoS computational biology. 17(1):e1008629, 2021.
[69] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[70] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17 2017.
[71] Alexander Shmakov, Kevin Greif, Michael Fenton, Aishik Ghosh, Pierre Baldi, and Daniel Whiteson. End-to-end latent variational diffusion models for inverse problems in high energy physics. Advances in Neural Information Processing Systems, 36, 2024.
15

---

A
AppendixxSupplementary Information
A.1
QuickDraw-F'S dataset
The QuickDraw-FS dataset is built from the samples of the Quick, Draw challenge [50]. In this online experiment
www.raa..wthgoooo withgoogle. com), participants have to draw an object when presented with the category name. The resulting dataset is made of 345 object categories, with approximately 150, 000 drawings per category. The experimental protocol of the Quick, Draw ! challenge forces the participants to produce drawings that are semantically related to the category name, but those drawings do not necessarily represent the same visual concepts. For example, the "alarm clock" category includes digital and analogic types of alarm clocks, which represent 2 different visual concepts (see A.I. This property makes the original Quick, Draw dataset not optimal for purely visual one-shot generation tasks.
MY
0:
alarm clock
drums
grass
moustache
C
telephone
power outlet
Figure A.1: Examples of distinct visual concepts belonging to the same object category in the Quick, Draw ! dataset.
To mitigate this issue, previous work has proposed the QuickDraw-FF dataset. In this dataset, new categories formed based on the visual similarity of the drawings (see Appendix A in (30]). The authors have used clustering techniques in the latent space of the contrastive learning algorithms to compute the infer the new categories. The resulting dataset is made of categories representing one single visual concept. Using this dataset, one can extract a "prototype" exemplar- at the center of the cluster exemplify the category visual concepts. We include examples of drawing variations and their corresponding exemplars in Fig. A.2
C
5
2
C
C
C
5
]
5
J
5 5
C
2
2
5
5
_
2
3
2
5
]
0
0
0
5
5
0
3
5
L
C
C
C
C
5
)
5
Figure A.2: Illustration of the samples and the corresponding exemplars for 4 categories of the
QuickDraw-F55 dataset. The small image located on the top represents the exemplars of the different visual concepts. The 5 x 5 grid of drawings represents the corresponding visual concepts (randomly sampled in the cluster.
16

---

A.2 Regularized AutoEncoders
A.2.1 VQ-VAE
Let us define a codebook Z |= made of K elements (also called codewords). Each codeword has dimension s: C₁ E RR. The Vector- Quantizea Variational AutoEncoder (VQ-VAE) [51] can be decomposed into 3 stages: i) an encoder 9s (z) x) mapping the input data x to a continuous latent vector 2 E R iij discretizing operator denoted (z) which transforms z into a discretized latent vector 2:g: and iii) a decoder (x 2/q) mapping 2 to reconstructed image x. The discrete latent code 2q is calculated using a nearest-neighhor look-up in the codebook (see Eq. TOP. Said differently, each element of the continuous latent vector 2; is replaced by the nearest Oj in the codebook (here the i index corresponds to the i-th coordinate of z):
Zq:
nz nz (z)) arg min||z: ey
(10)
e,e2
Zg could then be transformed into a discretized vector by mapping each codeword with its corre- sponding address the codebook (ej - j). Note that this quantization process is equivalent to defining posterior distribution following a K K-way categorical distribution [5T].
To learn the resulting networks, one naive way would be to minimize the following loss function
arg min LV with LVQVAE - En (mmz(mmz(ss))) [log (x((2]]]
(11)
0,8,2
Eq. 11 reconstruction loss in which the information first flows through the quantized encoder, (i.e.
z 9o( (.))))) to then produce a reconstructed image (i.e. (pg(2)()(z)
However, Eq. 11 cannot be directly optimized as it has no real gradient (the arg min function is not derivable). minimize this loss function, the gradient is then approximated using straight-through estimator [36). The ttraight-through estimator involves copying the gradients from the decoder input to the encoder output. We refer the reader to line 5 in Algo. [for for practical implementation of the straight-through gradient estimator. Intuitively, since z is supposed to be very close to Zq the gradient contains mcaningful information for how the encoder has to change to minimize the reconstruction loss. During inference, the nearest embedding 2g is computed using Eq. T0 and then fed to the decoder. Due to the straight-through operation, the codebook Z does not receive any gradient information from the reconstruction term. Therefore, the codebook is learned with the simplest dictionary learning algorithm that involves minimizing the Eg distance between the quantized vector 2q and the continuous one z (i.e. ||z Zq 11))) This quantity cannot be directly minimized because there is no gradient flowing from Zq to z. To mitigate this issue, is replaced with the estimator term ||sg[z]] - 211 + |لا sg [z]] The full VQ- loss is described in Eq. 12
LVQV AE -Bzgrnz (.o((x))) [l (x + (iloglgg 2112 + ||zq
sg |z] (
(12)
The following pseudo-code illustrates how the VQ-VAE is usually implemented (see Algo. I. We follow similar implementation:
Algorithm 1: VQVAE pseudo-code
Input: dataset D, model parameters π = (0, o. 2)
for x in D do
2
z 4o (z|x) # encode
zq 1z (z) # quantize
Lreg || [zq]
21/
=11ag - sg[2)]13 # Quantization loss, sg[.] stop gradient
s
Zq = z + sg 2g z]
# strought-through gradient estimator
X po [x zq)
# decode
f ||x x| Lreg
A.2.2 Prototype-based regularization
Prototypical networks focus on learning an embedding space where data points cluster around single prototype representation for each class. A prototype is originally defined as the mean vector of
17

---

the embedded support points belonging to its class [17]. In the one-shot setting, the support set is reduced one single sample. Therefore here the prototype and the exemplar are the same.
To achieve the desired embedding space for the autoencoder we regularize the reconstruction loss with protoype-based loss. The loss uses the pairwise & distance between samples and prototype derive a probability distribution:
LPR = Bazyaallyyy logosooooololo((//aa 1
(1x"
(13)
In Eq. 13 hg (zy) represents the projection of the prototype in the embedding space while (
represents the projections of the sample. See Algo. 2.o.2foo for more details on the exact implementation of the prototype-based regularized RAF.
Algorithm 2: Prototype-basse regularizer pscudo-code
Input: dataset D y), model parameters т (0, Φ))
x: variations and y: exemplars
for (x. y) in 'D do
2
z 4s (z|x)
encode variations
Zy qs By X encode exemplar
d 11g (z) h h # pair-wise distance beteen projected and Zy
Lr log(softmax(())
6
X = (x(x(z) decode
7
L = Ix x1 + LPR
π Oc
8
A.2.3 Constrastive regularizers
Contrastive learning algorithms learn representations that are invariant under different distortions (i.c. data augmentations). Here we use two data-augmentation operators, (A) and 7 (-). that transform the variations into x 7x(x) and x 7 7B (x). respectively. We denote ZA and B the latent space projection of x and xB, respectively (i.e. (z (xA) and (o (2)))))) Here, we use two different types of contrastive regularizations that are LSimCLR (see
Maths and Algorithms:
Eq. 14) and Lar (see Eq. 15
LsimLL(ZZ zB) = Eea B
(im(h (1 ))++
Σ log
(hppsiimhh6(25)) 1b(z)))))))
(14)
Wz
LBar(z 2B) = Eynpe xB
Σ (1 sim-immmh) (
sim(( (z. h (z.) (23))1)) 2
ΣΣΣ JzL
(15)
(x, | |
with sin(ssmm(,)))
(16)
In these equations, indexes the sample in a batch, indexes the vector component of the embeddings, (z) and h (z) are linear probe stacked the RAE latent space. In the Barlow regularizer, For both networks, the linear probe projects in space of size 128.
À = 5 x 10
This is important to observe that the scalar product in Eq. 14 is computed along the vector component dimension whereas this is computed along the batch dimension in Eq. 15] Said differently, in Eq. [4] sim computes square matrix of size (batch size, batch size) (this is a pair-wise similarity matrix between samples) while it is of dimension (feature space dimension, feature space dimension) in Eq. [5 (this a correlation matrix between vector's coordinate). We refer the reader to Algo. and Algo. for the pscudo-code of the SimCLR and the Barlow regularizers, respectively.
18

---

Algorithm 3: SimCLR regularizer pseudo-code
Input: dataset D [x), model parameters π (8,0) # x: variations
for (x, y) in D do
2
XA = T (x)
# augment x in XA
3
XB - (x)
# augment x in XB
ZA (e(zA XA) # encode XA
5
ZB = (Z XB) # encode XB
Lreg LsimCLn(Z) ZA. ZB) see Eq. 14
X px(x(ZA) # decode
L = xx x + Lreg
π Oc
Algorithm 4: Barlow regularizer pseudo-code
Input: dataset D (x), model parameters т (0, # x: variations
for (x, y) in D do
XA T (x)
# augment in XA
XB B (x)
# augment in XB
ZA go (ZA XA) encode XA
BB = Us (zB xB)
# encode
6
Lreg Lar (ZA, ZB) see Eq. 15
X pa ZA) # decode
C x x
Greg
π
Augmentations:
The augmentations we use are the same for both regularizers (i.e. 7 =1)) and ()), they are randomly picked among the following transformations:
Random resized crop: with scale parameter ranging from (0.1, 0.9) and a ratio parameter ranging from (0.8, 1.2). The scale parameter tunes the upper and lower bound of the cropped area, and the ratio parameter defines the lower and upper bound for the aspect of the ratio of the crop.
Random affine transformation:
with a rotation parameter varying from (-15 to 15%), translation (from -5 pixels to pixels), zoom (with ratio from 0.75 to 1.25) and
shearing (from - 10 to 109)
Random perspective transformation: apply a scale distortion with a certain probability to simulate 3D transformations. The scale distortion we have chosen is 0.5, and is applied to the image with a probability of 50%
19

---

A.3
RAEs training and architectures
A.3.1
RAEs architectures
For the encoder, qá (z) x). and decoder, Po 2),, we leverage similar architectures than those proposed
in Ghosh et al. 52 In Table we detail the exact architecture of the RAE encoder and decoder.
Network
Layer
Input Shape
Output Shape
Param #
Conv2d
[1,48, 48, 48]
[16, 24, 24]
256
BatchNorm2d
[16. 24, 24]
[16, 24, 24]
32
ReLU
[16. 24, 24]
[16, 24, 24]
Conv2d
[16, 24, 24]
[32, 12, 12]
8,192
BatchNorm2d
[32, 12, 12]
[32, 12, 12]
64
ReLU
[32, 12, 12]
[32, 12, 12]
Conv2d
[32, 12, 12]
164,7,
32,768
Encoder: (z(x)x)
BatchNorm2d
[64, 71
[64, 7,
128
ReLU
[64,
[64. 7,
Conv2d
[64, 7,
[128,
131,072
BatchNorm2d
[128, 3,
[128, 3,3]
256
ReLU
[128. 3,
[128, 3,
Linear
[128, 3]
[d]
147,584 (d = 128)
Transsossd
[d, 1,
[128, 6, 6]
1,179,648 (d 128)
BatchNorm2d
[128, 6, 6]
[128, 6, 6)
256
ReLU
[128, 6, 6]
1128, 6]
Transsossdee
[128, 6, 6]
[64, 12, 12]
131,072
BatchNorm2d
[64, 12, 12]
[64, 12]
128
ReLU
[64, 12. 121
[64, 12]
ConnTranspose2d
[64. 12]
[32, 24. 24]
32,768
BatchNorm2d
[32, 24, 24]
[32, 24. 24]
64
Decoder: (x z)
ReLU
[32. 24, 24]
[32, 24, 24]
ConnTranspose2d
[32, 24, 24]
[16, 48, 48)
8,192
BatchNorm2d
[16, 48, 48)
[16, 48, 48]
32
ReLUU
[16, 48, 48]
[16, 48, 48
ZeroPad2d
[16, 48, 48]
[16, 49, 49]
Conv2d
[16, 49, 49]
[1, 48, 48]
257
Sigmoid
[1,48, 48]
[1, 48, 48]
Table I: The base architecture for all the autoencoders.
Note that for Omniglot and QuickDraw, have chosen different latent-space sizes (denoted d). For Omniglot 64 and for QuickDraw, d 128.
A.3.2 RAEs training details
We train the model using the Mean Squared Error loss with batch size of 128 for the reconstruction, along with different regularizations to study its effects. For both datasets, we use the Adam opti- mizer (69) with weight decay of 10-5 and learning rate of 10-4 The RAEs on the QuickDraw dataset were trained for 200 epochs and 300 epochs on the Omniglot dataset. Note that when trained the Omniglot dataset, we use learning rate scheduler in which the learning rate divided by 4 every 70 epoch.
20

---

A.4 Latent Diffusion models
In this section, we describe the mathematics behind the latent diffusion models. The following mathematical derivations are mostly derived from Sohl-Dickstein et al. (26). Song and Ermon
[25]. Ho et al. [47]. Rombach et al. [29] and are adapted to match the one-shot generation task and the
notations of this paper. Those mathematical derivations are not necessary understand this article but we include them to make it self-contained.
Herein, we consider pretrained Regularized AutoEncoder, with an encoder (o(z)x) x) and decoder p=(x(z) that map the input E RD to latent representation E R (d < D) and inversely, respectively. In the following, we will call indifferently z or Zo the latent variable corresponding to the input We will also call Zy the latent variable associated with the exemplar y. The goal of diffusion model in a one-shot latent diffusion algorithm is to learn the conditional probability of zo given the latent representation of the exemplar zy, we call this probability distribution pa (za (zy).
A.4.1 Diffusion process and noising operator in latent diffusion process
Diffusion models learn the transformation of a pure noise, called žr E IRd, into fully denoised latent representation Zo Rd. This transformation is progressive, through sequence of partially denoised latent representations {z, ) E (T-1)))), In this sequence Z is therefore sligthly more noisy than z, The idea behind the diffusion model is to learn the transition probability Pe (z Zt. 2 To do so, diffusion models introduce tractable noising process r(z, 2 1) that gradually injects noise in the latent representation. An illustration of such a directed graphical model is shown in Fig. A.3
Pult- u
Palx mn)
TAT
r(a, (--))
95(6g (x)
Poly
9g(2911)
Figure A.3: The directed graphical model considered in this work. Dotted and plain arrows represent the forward (i.e. noise injection) and the reverse processes (i.e. noise removal), respectively. Zy and 20 the latent representations of the exemplar image y and the image x, respectively (exemplified with skull drawings). Zj corresponds to the sequence of partially corrupted latent representations. Zy and zo are obtained using the RAE encoder (z(x) and can be mapped the input space using the RAE decoder po (x|2) z). The 'dummy' distributions located on top of the zi variables, illustrate the noise injection process, starting from an 'informative' multimodal distribution to a fully "uninformative' Gaussian distribution.
Here describe, in mathematical terms, the noise injection process
r(z1.T (zo) II r(z./ze1) with r(2.(2+-1) = N(z / B,4-11,B11 s.t. (3. (0, 1))
(17)
Br tunes the step size of the diffusion process. Using the successive product of Gaussian, this process could be reduced tractable noising operator (-) that injects the right amount of noise at time to obtain 2; from Zo:
In Eq. 17
21

---

Vanh-1 + v1 - One with e~ (001))
Va104-121-22 - a1
Vāizo + v1 Cef = (Zo) with 1- B and 0g
04
(18)
i=1
One could then express the probablity of z, given Zo in closed form: r(z ((z;; vā,zo, (1 I
(19)
The denoising probabilistic process, recovering the latent representation zo from noise, could be parametrized as follows:
Pe (Zo:T (2y) Paj (2 My II (x 12,, Byy)
(20)
Pab Zz- (z- [2, Zy)
=N Miy (Ze, 1, oil)
with
Paj Pop (zy)
= (N(,))
A.4.2 Loss of the Denoising Diffusion Probabilistic Model in the Latent Diffusion case
As in VAEs 49 the Evidence Lower Bound of the diffusion model could be recovered using Jensen's s inequality 477:
R Ezo)r(())) (2olz)) E Ezz))(())) log /f Pio (ro )dz.:)
Pai (Zo.r Zyy) dz1.T) r(z1.T) 220
E z log
r(z))TZZ))
Pw (Zo:T) Zyy) r(21:T (O0
E Ex log
Expr (ao)
Pip (ZO:T ky)
E (r log
= -TVVBB
r ((21:1 (20)
The Variational Lower Bound could be written sum of KL terms (26]:
LVLB = E, log r(z1:T Zoo)
Pv 20.77 (zy)
2 2
E, log p(zT Zzy
using Eq. 17 and 20
Pvl 12 [ec, Zy)
r(z, (zel (2(-1)
E,
log po (ZT (2T|la))
log
Peo (2--1 (2, Zay!)
r (2zl(z(-1)
+ log r(zi|zo)
E,
- log Pw lor
log
Pe (zt-1
2z, 2yy)
vo(20|210 (21, 1%y)
r T(Z
Zz, Zo)
r(zz r(zz
r(zz (zo)
E,
lo log
+ log
12,, Zyy 1t- (2,
r(z (z0))
Pyo Pv((20 121, Zyy)
2=2
r(z,-)
2e, Zu)
r(z (z0)
r(zz (Zo)) + llog
E,
(zr
log
Pw (2--1
r(z, (20)
pyo m (20 11 Zyy
r(z,-
1zc 2o)
r(zr + -oog
r(z, (2o)
E,
log Hogpp(2rlz
log
+
Pvo 22 12, 2xy)
r(zi (zo)
pai (Zo 21, 2ny
22

---

Er log r(zr 1o)
T
r(z,
21, 2n)
+
log
log logpu(zolzn) zx)
Pi (z 2r (ZT
Pio (zc-1
iee 2yy)
T
= Er KL r (ZT 1z
2o my)] +
Σ
2y)]
(21)
1
log (2o (2, 2x)
Lo E, log pv (zo|z1, zy)
T
L
with
L = E, KL r(z 1z 12, 2o p (z--1 Ze, 2y))
(22)
LT R. KTL [r(zr] 2o 2z 2t [yy]]
In the previous equations, E,, is a shortcut notation for Bxu.r~((eer)) Note that in the optimization process, Lr could be ignored because it doesn't 't depend on the model parameter v this is a pure non-informative Gaussian distribution (see Eq. (2).. Lp is modeled by Ho et al. [47] using a separate neural network. L, is KL between 2 Gaussians distributions, so it could be calculated with closed form:
va-
B
Varl -
pel2,, 2o)
20
+
2t
r(z:-1 20) = -22 2o), B.)) with
{
1- 2
- ā.
1
1-0-11
Be
Be
- αe
(23)
With (2,, zo) and 3,1 the mean and the variance of r(2:-112 22, 20), respectively. Using Eq. can express in a convenient way:
18
1
(Ze
vI 11-60 Cec)
(24)
V@
Therefore can simplify te (Zr, 2o) Eq. 23
1 at dt.
(ze, 2o) = jūn vai
Zt
(25)
VI
Similarly, re-parameterize (w 12, Zy)
because is available as input at training time:
1 Ct VI
(ze,
Evj
(Zc: t)
(26)
Vat
One can apply the closed form formula of the KIL between 2 gaussians distributions to compute L4 in Eq. 22
1
L,
E,
(e 20)
M4 Mp (Z,, 1)11
2 |1oo? 112
IE,
1
1
- aq e)
1 Var-
Z:
1 -
Ew t)
cw
with Eqs. 25 and 26
11 11
Vas
v1 5s
V1
ās
(1 2
E,
ex (Vorzo + v1 cie, due, 1011
(27)
20r(1 a.) ||o?1
With further simplification of Eq. 27 47
L, = E, ||e (Vaazo + vI are, 1/12
(28)
= Er ||e c (z,, (z,, 1)
(29)
23

---

A.4.3 Architecture and Training
The DDPM model we leverage is ID-UNet to perform the diffusion process over the latent embeddings. The architecture of the UNet described in Table
Network
Layer
Input Shape Blocks din dout
Output Shape
Param #
Linear
din dout + dout dout
Block_MLP
GroupNorm
dout dout
SiLU
dout dout din din din
RMSNorm_Min
din din * 512 + din
Residual
MyAttention
din
SiLU Linear
de
de
d;
2 dout
2 * doue(ds + 1) dout(din + 3)
ResnetBlock
Block MLP
din
dout
Block_MIL
dout
dout
dont (dout + 3)
Identity
dout
dout
ResnetBlock
(dins dins de)
din
2 din + din + 4)
ResnetBlock
(den, din, de)
d
* din + din + 4)
ModuleList2
Residual
din
din
513 din
Linear
din
dout
dn dout + dout
Unet
SinusoidalPosEmb
[128]
[128]
Linear
[128]
[128]
16,512
Time Embedding
GELU
[128]
[128]
Linear
[128]
[128]
16,512
Linear
[512]
[2048]
1,050,624
ModuleList2
[2048, 128]
[1024]
21,011,456
Downscale
ModuleList2
[1024,128]
[512]
5,787,136
ModuleList2
[512,128]
[256]
1,713,920
ResnetBlock
[256,128]
[256]
198,656
Bottleneck
Residual
[256]
[256]
131840
ResnetBlock
[256, 128]
[256]
198656
ModuleList2
[256, 128]
[512]
1,316,608
ModuleList2
[512,128]
[1024]
4,730,368
Upscale
Modulel
[1024, 128]
[2048]
17,8493344
ResnetBlock+LLinen
[2048, 128]
[2048]
21,514,240
Linear
[2048)
[256]
524,544
Table 2: The neural architecture of the diffusion model used for all experiments unless stated otherwise (the parameter count is shown for the latent size of Quickdraw-Fis experiments, ie = 128).
The architectures the diffusion models for both the Quickdraw-FS and Omniglot datasets are kept identical. The only difference is that the diffusion model is applied on a latent space of size for QuickDraw and of size d = 64 for Omniglot. The models are trained on a batch size of 128 using the DDPM scheduler for 1000 time steps. 37 linearly spanning between 1.5 x 10-3 and 1.95 x 10-2 and trained for 1000 epochs. The model is optimized using the Adam W optimizer [70] with an initial learning rate of 14 Then epochs.
128
scheduler in which the learning is divided by 10 every 200
24

---

A.5 Impact of the regularization on the QuickDraw-FS dataset
Herein we systematically vary the parameter in Eq. for each type of regularization and we evaluate its effect using the originality vs. recognizability framework. To visualize this effect while maintaining the order of the syper-parametent we use the parametric fit method described in [54]. This technic involves 2 simultaneous parametric fit: i) polynomial fit (degree 2) between the hyperparameters and the originality values (shown in Fig. A.4p, Fig. A.5pp Fig. A. Fig. A and Fig. (.9p) and ii) another polynomial fit (degree 2) between the hyperparameters and the recognizability values (shown in Fig. A.4c Fig. p. Fig. A.6çp ( and Fig. A.9p). Those 2 fits could then be combined to create oriented parametric fit between the originality and the recognizabilityy (shown in Fig. A.4h, Fig. A.5p, Fig. A, Fig. [A.8p and Fig. A.9h). In these curves, the "chevron" indicates the direction in which the value of the 3 hyperparameter is increased. We have included the range of we have explored in the caption each type of regularized LDM. We use the notation (a cl to express that we explored from to with step of c.
A.5.1 Impact of the KL regularization
Herein we evaluate LDM leveraging a RAE trained with the following loss (with CKL) in Eq.
min CRAE s.t. CRAE (Ezzaa(()))) [log po (x|2)) ) BKLCKLCC
(30)
P.6
a)
b)
0.85
0.75
ofggratty
0.60
0.65
0.70
0.75
10-1
10-3
10"4
10-3
10
10a1
10.4
10-3
10-1
10-1
10-2
10"
originalty
Figure A.4: Impact of the 3xL hyperparameter the originality recognizability. Each data point corresponds to LDM trained with different value of 3KL in Eq. 30 Herein we have explored the following 3KL. range: 6:0-6:10 2: 1::0011 and 0.05 and (0.1:0.5::0.).
A.5.2 Impact of the VQ regularization
Herein we evaluate LDM leveraging a RAE trained with the following loss (with Lv in Eq.
min CRAE s.t. CRAE Ez (bx) [log po z)] (vQLvQ(z)
(31)
0.
25

---

a)
0.00
recopnzzabltt/ 1M4
0.82
0.82
orginaltt
0.00
0.80
0.64
0.78
0.78
0.3
0.64
0.65
0.66
0.67
0.68
109
1093
10'
1009
10a
10a
1043
10:
2015
107
viginalit
An
Figure A.5: Impact of the 3vq hyperparameter the originality recognizability, Each data point corresponds to LDM trained with a different value of BvQ in Eq. 31 Herein we have explored the following Q range: [1,2,5] 5] and [10:50::1] 10) and 100.
A.5.3 Impact of the CL regularization
Herein we evaluate LDM leveraging a RAE trained with the following loss (with COL) in Eq
min CRAE s.t. CRAE = -Exogo(11x) [o + BoLLCL(z)
(32)
0,6
a)
b)
C)
0.70
(M)
0.85
0.84
0.64
0.84
0.62
0.8
0.60
0.62
0.64
0.66
0.68
0.70
10-425
10635
104.5
10
10:75
10-425
10A3s
204.75
10
10475
originaity
Figure A.6: Impact of the SCL hyperparameter the originality vs. recognizability. Each data point corresponds to LDM trained with different value of BeL in Eq. 32 Herein we have explored the following SeL range [0.7:0.9::0.1] and [1: 10:: 1] and [10:40::00]
A.5.4 Impact of the prototype-based regularization
Herein we evaluate LDM leveraging a RAE trained with the following loss (with LPR) in Eq.
5
min CRAE CRAE -E (.1x) [log)((x(z)] -BpBCR(Z))
(33)
0.0
26

---

a)
b)
0.94
0.82
0.90
Recogitzblllyy (50)
0.90
orgineity
0.58
0.88
0.86
0.84
0.84
0.82
origiality
Amonn
Figure A.7: Impact of the 3PR hyperparameter on the originality vs. recognizability. Each data point corresponds to a LDM trained with a different value of 3PR in Eq. 33 Herein we have explored the following 3 range: [10-1:10- 10011: 10-1::1001111 and (0.25:0.75: 0.25 and [1.0: 10:: 1] and [15:30:5] and [100, 200, 500].
A.5.5 Impact of the SimCLR regularization
Herein we evaluate LDM leveraging RAE trained with the following loss (with LSimCIR)) in Eq.
min CRAE S.t. CRAE Egng [log pe (x 12] + AsimcLRGsmmca(e)) (34)
0,6
a)
b)
0.88
0.85
3.70
0.86
0.85
0.68
nucogiablily (N6
recopttabiley (8)
0.84
ofgra/b
0.84
0.64
0.82
0.62
0.82
0.80
0.80
030
0.78
0.78
0.575
0,600
0.625 0.650 0.675 0.700
10
10:3
10.2
10-1
10
10
10
10.3
1012
10
107
originaityy
Bascc
Prwin
Figure A.8: Impact of the 3SimCLR hyperparameter the originality recognizability, Each data point corresponds to a LDM trained with a different value of BSimCLR in Eq. (34] Herein we have explored the following $5mOLR
range 1:00::10-1::00 and 10:00:
A.5.6 Impact of the Barlow regularization
Herein we evaluate a LDM leveraging RAE trained with the following loss (with (BAR) Eq 15
min CRAF S.L. CRAF -Ezago [[o]))(((2)] + (BAARnAR(2))
(35)
8,6
27

---

a)
b)
0.01
0.89
0.88
0.88
po0ggocebbry 050
0.00
roogriabllyy (5)
orginelty
0.87
0.86
0.59
0.86
0.58
0.85
0.84
0.84
0.57
0.83
0.82
0.50
origmality
Raw
Pew
Figure A.9: Impact of the SRAR hyperparameter on the originality vs. recognizability. Each data point corresponds to LDM trained with a different value of 3ARR in Eq. 35| Herein we have explored the following ABAR range: [1:10:1] and [15:30::5] and (50, 100, 200].
A.6 Impact of the regularization on the Omniglot dataset dataset
a)
Classifier
Barlow SimCLR
Proto.
98
98
98
Recognizablitty (%)
96
96
96
94 0.25
94
0.30
0.35
0.25
0.30
0.35
0.40
0.25
0.30
0.35
Originality
Originality
Originality
Humans
No regularization
pixel baseline
Figure A.10: Effect of inereasing the regularization weights the originality recognizabilityy framework (Omniglot dataset). Each data point represents an LDM trained with different values of regularization weights (3). The curves represent the parametric fits, oriented in the direction of an increase 3. a): For the LDMs with "standard" regularizers, the 8 is applied on the KL (CxL in Eq. 2 b): For the supervised regularizations, the 3 is applied the CL (Cel in Eq. 4 or on the prototype-base the 33 is applied
regularizations (Crn in Eq. [5) c): For the contrastive regularizations. the SimCLR (LsimCL. in Eq. 14) on the Barlow regularizations (L in
Eq. 15). See A.5 for more information on the range of 3 we have explored for each regularization. Larger data points indicate models whose performance is closer that of humans for each type of regularization. For comparison, we include a LDM leveraging a non-regularized RAE (hexagon marker) and diffusion model trained directly on the pixel space (cross marker). The human performance corresponds to the recognizability and originality of human drawings (shown with grey star)
28

---

Here we present curve similar to Fig. but for LDMs trained on the Omniglot dataset. We were unable train VQ- VQ-VAE with reasonable performance on this dataset, we have excluded the VQ-regularized LDM from Fig. A.T0 We believe this issue is due to improper hyperparameter tuning as the same regularizer works reasonably well on the QuickDraw-FO dataset. We are actively working to resolve this problem.
Except for the VQ regularizer, we observe that other regularizers follow similar trend those trained on the QuickDraw-F'S dataset. In particular, the profotype-based and the Barlow regularizers outperform all others.
A.7 Samples generated by the one-shot LDMs
Here we showcase the images generated by one-shot LDMs. The exemplars used to condition the LDMs present in top line in the red frame. We randomly chose 10 exemplars from 115 possible options in the QuickDraw-FSS test set. All images below the red frame represent samples of the corresponding visual concept generated by the LDM. We use the same 10 exemplars for all the LDMs for easy comparison. All shown exemplar corresponds to the LDMs, for each regularizer, showing the shortest distance humans. They correspond to larger data points in Fig.
17
T
P
T
T
T
T
ST
7
Figure A.11: Samples generated by LDM without regularzation. For this LDM, 3 is set to 0.
29

---

a)
b)
PPP
1
T
6
R
T
E3
E
P
STTEE
TT
IT
OP
7
GUDDD
T
XO
D
P
1
X
T
X
T
J
Figure A.12:
Samples generated by LDMs with standard regularizer. KL regularizer (obtained 5).
with 3Kr. = 10-3), b) VQ regularizer (obtained with 3vq
a)
b)
T
T
PAPIO
T
T
TP
OONDO
P
P
T
RIR
T
I
Figure A.13: Samples generated by LDMs with supervised regularizers. a) classification
regularizer (obtained with BCL = 5). b) prototype-based regularizer (obtained with 3pg = 102).
30

---

a)
b)
GR
T
T
DOADD
T
T
T
?
?
Figure A.14:
Samples generated by LDMs with contrastive regularizer. a) SimCLR regularizer (obtained with 3SmCLR 10-3). b) Barlow regularizer (obtained with 3 = 30).
31

---

A.8 LDM feature importance maps
A.8.1 Mathematics behind the feature importance maps
We remind that px(x|z) is the decoder of the RAE, and that p&: (2,-1/z,, 2y) is the transition proba- bility learned by the diffustion model. To make the mathematical derivations more concise, we define the following function:
Po RRd
TR D
and
Pv
RRd
R2d
(36)
log p=(-)
2t-1
log (.2z, Zy)
(37)
To project each intermediate noisy state Zt into the pixel, we feed them into the decoder. The resulting projection is Xt (2,))(Zz) po Po
For each time step of the diffusion process, the importance feature map quantifies how the absolute value of pa (z.) changes when one varies zt. o(x, describes the accumulation, over all time steps, of these "local feature map":
ppo (zc)
x(x,y)
(38)
Ozt
Opo (ze)
(39)
Oze
1=
Opu (2x) Opo (2))
Σ
(40)
Zx,
t=ul
Jno (xx) Vx.)) (oee)
(41)
(42)
with Jpp (xx) the Jacobian of the function w.r.t X computed in (z.)). If we trade the functional notations for probabilistic ones we have:
T'
x(x,y)
Σ
(Xt) log pa (-1z, zy)
(43)
1=
32

---

A.8.2 Example of LDM feature importance maps
The LDMs' feature importance maps have been computed on 25 different categories, for each of the six different regularization methods discussed in the paper. The feature maps were calculated by taking the average of 10 misalignment maps ((x, y) as defined in Eg. All shown feature importance maps correspond to the LDMs, for each regularizer, showing the shortest distance to humans. They correspond to larger data points in Fig. 3
a)
b)
wmwm
Figure A.15: Feature importance maps for LDMs with standard regularizer. a) KL regularizer
(obtained with 8Kr. 10- b) VQ regularizer (obtained with BrQ 5).
a)
b)
Figure A.16: Feature importance maps for LDMs with supervised regularizer. a) classification
regularizer (obtained with SC1 = 5). b) prototype-based regularizer (obtained with 3Pg = 5 102).
33

---

a)
Feature importance maps for LDMs with contrastive regularizer, a) SimCLR regularizer (obtained with BSsmCLA 10-3). Barlow regularizer (obtained with BBAR 30).
Figure A.17:
A.8.3 Example of Human feature importance maps
For comparison, feature importance maps have also been computed for humans for the same 25 categories. For humans, the feature importance maps are heatmaps representing the likelihood of pixel being selected by participant as part of the ClickMe- Sae--uuicDDraa experiment (further details on the experiment provided App. S of Boutin al. 30 The same image used calculate the misalignment maps for the LDMs is presented to the participants during the CliCkMe- DuCkMe--uickDraw experiment.
wmwmn
Figure A.18: Feature Importance maps for humans
Human consistency: To evaluate how humans agree with each other the feature importance maps, we computed the human consistency. To do so we use a bootstrapping technique. For each category, divided the participants into 2 populations (randomly selected), obtaining approximately 25 annotations (heatmaps) coming from different participants for each category. We then average those annotations within the population (and the same category) to form population-wist feature importance maps. We finally compute the human consistency with the Spearman correlation between those population feature importance maps. We obtain spearman of 0.8845 (p 5.10-2)
34

---

A.8.4 Pair-wise statistical test for importance feature maps
To verify the statistical significance between the human/machine correlation we have obtained for all types of regularized LDMs we use a pair-wise statistical test. In particular, we compute the Wilcoxon signed-rank test between all pairs of LDMs. This test is non-parametric and does not consider the "Gaussianity" of the underlying population. The null hypothesis of this test (that could not be rejected when the y-value is over 0.05) is that the two tested populations are sampled from the same distribution. The alternative hypothesis (validated when the p-value below 0.05) is that first population ( columns of the Table A.8.4) is stochastically greater than the second population (rows of the A.8.4). All p-values, for all pairwise statistical tests are shown in Table A.8.4.
Barlow 5.4 10-4
SimCLR 5.9 x 10
Classif. 6.03 10 -
KL
VQ
No reg.
Proto.
1.2 10-6
2.3 10
4.7 x 10
Barlow
9.5 x 10
9.3 10
1.8 x 10
2.3 x 10
2.3 10
SimCLR
2.3 x 10-1
5.2 x 10
4.7 10-7
4.7 10
Classif
2.9 x 10
2.3 10
2.3 10
KL
2.3 10-7
4.5 x 10
VQ
9.9 X 10
Importantly those statistical tests have been computed on the Spearman correlation vector (one Spearman value per category) between feature importance maps the best-performing models (those indicated with bigger data points in Fig. 3 and those of humans.
A.8.5 Illustration the limited one-shot ability Dall-e
Herein we illustrate how current Latent Diffusion Models tend to fail at producing faithful variations when prompted with single image. We showcase some of the generations made by Dall-e 3 when conditioned on a single image of a self-balancing bike. The self-balancing bike is particularly interesting case it represents an 'unusual' vehicle that is unlikely to belong the Dall-e training database. You can observe that Dall-e generates images missing some of the key concepts of the self-bat taflballniing bike (i.e. one-wheel).
Exemplar
Variations
Figure A919; Examples of variations generated by Dall-e 3 when prompted with single image of seff-balancing bike
35

---

A.8.6 Potential limitations
In this article, we tested six representational inductive biases, small number considering the extensive range available the representation-leanee biases that have proven successful in one-shot classification tasks. Therefore, other representational inductive biases might align better with human performance, both in terms sample similarity and visual strategy. Our goal wasn't it to test all possible biases but to demonstrate that some of them can significantly narrow the gap with humans in one-shot drawing tasks.
literature. This field encompasses hundreds of inductive
Another limitation of this article lies in the recognizability vs. originality framework we are using evaluate the drawings. This framework leverages 2 critic networks evaluate the sample's originality and recognizability. There's s no guarantee these networks align with human perceptual judgments. Thus, the recognizability and originality scores might not reflect human perception accurately. However, since both human and model outputs are evaluated using the same pre-trained critic networks, the comparison remains fair.
Our approach leveraged two-stage generative models: the first stage compresses information and shapes the latent distribution with representational inductive biases (the RAE), and the second stage learns this latent distribution (the diffusion model). This type of architecture takes longer to train because requires two separate training procedures. However, this limitation could be overcome by using an end-to-end training procedure for Latent Diffusion Models, which could streamline the process
A.8.7 Computational Resources
All the experiments of this paper have been performed using Quadro-RTX600 GPUs with 16 GB memory. The training time for the RAE is approximately 24 hours and 72 hours for the Diffusion model (96 hours overall). Note that as we have explored large range of hyperparameters for all types regularization, our paper is relatively extensive in terms of computations (600 models have been trained overall, but just small part of them have been used in this article).
A.8.8 Broader Impact
This work does not present any foreseeable negative societal consequences. We think the societal impact of this work is positive. It might help the neuroscience community to evaluate the different mechanisms that allow human-level generalization and then better understand the brain.
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claim is that representational inductive biases in LDMs help to close the gap with humans on the one-shot drawing task. This claim is experimentally verified in Fig. and Fig.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations in the supplementary information (see section A.8.6). There
main limitations. First, the originality vs. recognizability framework might not be aligned with human perceptual judgment. Second, the long training time of 2-stages latent diffusion models prevents the wide adoption of the representational inductive biases we propose in this paper. Third, we have tested a limited number of regularizers, so other regularization techniques might be even better aligned with humans. Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
37

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA].
Justification:
We do not have theoretical results. This article is mainly experimental.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: the Appendix and the main text, we have extensively described the ex- periments we have run. In particular, in section A.2 we describe the models we use as well as their hyperparameters. We go even further by releasing the code to reproduce our experiments: http:// http:://anonmouss 4open. / 526B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
38

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The databases we use are already in open access. The code we used to train the anonymous GitHub link (http://anonymoous 4open. science/ )22BB). We cannot release the human data we have leveraged because did not collect them. We invite interested people send mail to the authors of [30 if they are interested in human data (the authors are open to sharing their data).
Justification:
LDMs is available
r/Lat en taatterst-s---eaa Eattters
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
poollccgguies/// CodeSubmissionoolooo....ooo oolliccyolyy for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /nips.cc//// wwwpc//ccwppcpcc..e.eeee www.cc//guieseeeeee wttp:/ooeuubiisssoooooooo..oo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in the supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We extensively describe the databases use well hyperparameter training details in section A.2 and section A.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
39

---

Justification:
We report error bars and pair-wise statistical tests on Fig. 5 (see A.8.4). Note that did not compute error bars for Fig. as our analysis relies on made on tens of models.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
In the Appendix (see App. A.8.7 we describe the type of hardware we use to train the models, the training time for each model, and the total number runs we spent to publish this paper.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We believe we conform with the NeurlPS code of ethics in every aspect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
40

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have included a broader impact section in App. A.8.8 but we do not foresee any notable societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't 't think our work poses significant risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
41

---

Justification: We use the Quickdraw database (under CC BY 4.0 license). We also used Omniglot, which is under the MIT license. We credit the creator of these assets by citing them when we introduced the databases.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi twwode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our only new asset is the code that allows to run all our experiments. This code is available publicly and is under the MIT license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We have not conducted any psychophysics experiments. However, we use human data collected by other researchers. The protocol to collect those data is extensively in their article (appendix of (30
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
42

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: We have not conducted any psychophysical experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
43


Reviews:
[{"review_id": "ORsYb46djE", "reviewer": "Reviewer_vmMb", "summary": "This paper investigates how different representational inductive biases in Latent Diffusion Models affect their performance on one-shot drawing tasks, aiming to close the gap with human abilities. The authors explore six regularization techniques: KL divergence, vector quantization, classification, prototype-based, SimCLR, and Barlow twins. They evaluate these models using the originality vs. recognizability framework and a novel method for generating feature importance maps. The results show that prototype-based and Barlow regularizations significantly narrow the gap between LDMs and human performance in one-shot drawing tasks. These regularizers outperform standard LDM regularizers (KL and vector quantization) as well as classification and SimCLR regularizers. The authors also demonstrate that the feature importance maps of LDMs with prototype-based and Barlow regularizations align more closely with human attentional strategies. Additionally, they find that combining these two regularizers yields even better results. The study highlights the potential of incorporating specific representational inductive biases in generative models to achieve more human-like generalization capabilities, with implications for both AI advancement and understanding human cognitive processes.", "questions": "- How well do your findings generalize to more complex datasets beyond simple sketches? Have you considered testing your approach on datasets with more detailed or realistic images?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper presents a study by systematically exploring various representational inductive biases in Latent Diffusion Models for one-shot drawing tasks. The application of regularization techniques from one-shot classification to generative models provides new insights into improving model performance.\n- The authors employ a wide range of regularization techniques and evaluate them using multiple metrics, including the originality vs. recognizability framework and a newly developed method for generating feature importance maps. The statistical analysis of the results adds credibility to their findings.\n- This paper demonstrates how specific inductive biases can substantially improve the performance of generative models in one-shot tasks, potentially leading to more versatile and human-like AI systems. It also shows the alignment between the most effective regularizers (prototype-based and Barlow) and prominent neuroscience theories provides interesting insights into human cognitive processes. The paper's findings could have practical applications in areas requiring rapid generalization from limited examples, such as design prototyping or creative tasks.", "weaknesses": "- The paper doesn't provide a detailed analysis of how different components of the regularizers contribute to the overall performance. This makes it challenging to understand which specific aspects of each regularizer are most crucial.\n- While the authors use human-derived feature importance maps, they don't include a human evaluation of the generated samples. Such an evaluation could provide additional insights into the perceived quality and human-likeness of the generated drawings.\n- Although the authors briefly mention combining prototype-based and Barlow regularizers, this aspect is not thoroughly explored. A more systematic investigation of different regularizer combinations could potentially yield even better results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer vmMb for the relevant and valuable comments. Please find below a point-by-point to answer that answer the main concerns of the reviewer :\n* **About the lack of analysis on the impact of the different regularizers on the performance**: In Fig 2, the comparison between the regularized latent diffusion models (shown in color) with the no-regularization baseline (hexagon marker) shows how the different regularizers contribute to the global performance (as measured by the originality vs. recognizability framework). However, we acknowledge the reviewer's concern that we did not discuss enough why the regularizers produce such effects. To mitigate this issue we have added an entire paragraph (at the end of the results section) that better explains these differences. Because it was also requested by another reviewer, we have included this paragraph in the general response (see bullet point 2). We thank the reviewer for raising this issue, as we think it improves the quality of the article.\n\n\n* **About the human evaluation of the samples**: Human evaluation of the generated samples has already been proposed by previous studies. For example, [1] did a Turing test where humans had to distinguish between human-drawn and machine-generated samples. While insightful, such experiments give few insights into how the samples differ from machine to human. On the contrary, we think originality vs. recognizability offers a finer comparison between the distribution of images drawn by humans and those generated by machines. \n\n* **About systematic investigations of the different regularizers combinations**: To address the reviewer’s concern, we have run additional experiments to systematically explore various combinations of regularizations. In particular, we have systematically explored the following combinations of regularization Barlow + Prototype: KL + prototype, SimCLR+Prototype, VQ + Prototype. Among all combinations of regularizers, it is the Barlow + Prototype and the KL + Prototype combinations that perform the best. Interestingly, the good performance of the combination  KL + Prototype was not expected, because the KL regularizer (when used separately) shows a low recognizability (see Fig2). On the other hand, the VQ + Prototype combination shows no improvement compared to the Prototype regularizer. Overall, these additional experiments confirm the potential of combining an unsupervised and a supervised regularizer to match human performance. These experiments lead to one additional figure (included in the one-page PDF allowed in the general response). Note that we have also explored all combinations between unsupervised and the classifier regularizer, but we did not observe any significant improvements (we have included those experiments in the supplementary information of the revised version).\n\n* **About generalization to more complex databases**: Note that the diffusion models as well as the RAEs we use in these articles are all known to scale well to larger datasets [2]. So in theory nothing prevents us from applying the proposed regularization in more complex settings. However, the point of our article is to compare humans with machines. To do so, one needs to leverage tasks that are accessible by both humans and machines, which is not the case with natural image synthesis: humans can hardly produce images that resemble natural images. We have chosen the one-shot drawing task because it offers a leveled playfield to compare humans and machines. To make this point clearer, we have included a sentence in line 133.\n\nOverall, we hope that our detailed response has addressed the reviewer's concerns, and convinced them to increase their rating. Should there be any remaining issues, we are more than willing to engage in further discussion.\n\n[1]: Lake, Brenden M., Ruslan Salakhutdinov, and Joshua B. Tenenbaum. \"Human-level concept learning through probabilistic program induction.\" Science 350.6266 (2015): 1332-1338. \\\n[2]: Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1wPZ58PHrK", "reviewer": "Reviewer_ZReU", "summary": "i)This paper uncovers the impact of representational inductive biases on Latent Diffusion Models through one-shot tasks, particularly in the realm of human-like sketching. It compares three distinct groups of regularizers: a standard baseline, supervised methods, and a third group consisting of self-supervised techniques.\n\nii)This paper aims to uncover the strategies employed by each regularization method to generalize to novel categories.\n\niii)This paper conducts a comprehensive comparative analysis of the effectiveness of different regularization methods in one-shot drawing tasks. The study explores various dimensions, examining the performance and differences of various regularization strategies in such tasks.", "questions": "Major:\n\ni)In one-shot drawing tasks, originality constitutes a key evaluation metric. However, when measuring the “human-like” characteristics of samples, the impact of originality is relatively minor, and its role in the evaluation process appears to be more limited.\n\nii)\nDoes this paper quantitatively analyze the stroke correlation between the generated sketches and those drawn by humans? Although the generalization curve has been considered in the evaluation of originality and recognizability, the paper does not explicitly reveal the interrelationship between the strokes.\n\niii)Has this study explored the reasons for or proposed hypotheses about the performance differences exhibited by various inductive bias methods in one-shot drawing tasks?\n\nMinor:\n\ni)Given that classification models may be influenced by their own biases or uneven distributions in the training data, the recognizability of the model does not necessarily equate to the “human-like” level of the samples. Does the paper take this potential issue into consideration?\n\nii)In the exploration of effective methods to improve the performance of one-shot drawing tasks, did this paper consider approaches other than simply adding the prototype-based and Barlow regularizers with weights?\n\niii)\nIn Figure 2, does the “pixel baseline” refer to the use of rasterized sketches during training or testing? On this point, this paper could analyze the differential impact of inductive biases when dealing with vector versus raster sketches.", "limitations": "No. How the evaluation criteria effectively align with human judgment or aesthetic standards is a question of considerable research value, especially in generative tasks. Particularly when dealing with sketches or handwriting, the consideration of strokes is an indispensable key dimension that cannot be overlooked.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is dedicated to analyzing the specific manifestations of various inductive biases in one-shot drawing tasks, contributing new findings and demonstrating a high level of originality. The paper is well written, with a clear and logical structure, reflecting a rigorous academic attitude. The experimental section is well-designed, with substantial research effort, and the execution process strictly adheres to scientific methods. The comparative analysis strategy used is both comprehensive and detailed, effectively ensuring the precision and credibility of the research findings. Additionally, the conclusions of this paper provide valuable insights for the field of one-shot drawing tasks and have significant guiding significance for subsequent research.", "weaknesses": "i)This paper conducted an extensive analysis of the impact of different inductive biases in one-shot drawing tasks. However, the analysis remains superficial, merely briefly revealing the experimental results of various methods without delving into the fundamental reasons behind the differences in outcomes among the methods. Furthermore, the paper does not propose specific solutions to the research questions addressed.\n\nii)The experimental method adopted in this study is not limited to specific modalities and is not restricted by data scale. This method may be applicable to the generation tasks of more modalities(such as photos or text2img). However, the experiment specifically chose handwriting and sketches as the research subjects for one-shot generation tasks.\n\niii) The core argument proposed by this paper is the generation of samples that resemble human-drawn sketches or handwriting. However, the paper does not provide sufficient elaboration on how to quantify the “human-like” characteristics of the samples, especially with an in-depth analysis from the perspective of stroke features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer ZReU for the detailed comments. Here is our point-by-point answer:\n\n* **About the lack of analysis of the different regularizer components**: We agree with the reviewer that we did not discuss enough the reasons for such differences. As this issue was shared with other reviewers, we have answered this concern in the bullet point 2), in the general rebuttal form. \n\n* **About considering multimodal and more complex datasets**: We acknowledge that the latent diffusion models we use do not face scalability issues and can handle multiple modalities. As our scientific question involves tight comparisons between humans and machines, we focus on tasks accessible for both humans and machines (which is not the case for natural image synthesis.Therefore, the focus on the one-shot drawing task is deliberate and aligned with the scientific question we explore in this article.\n\n* **About the methods used to compare machines and humans and about stroke analysis**: In this article, we used two independent methods to compare humans and machines —the recognizability vs. originality framework and feature importance maps. Both methods lead to the same conclusion: certain regularizers help narrow the gap with humans in one-shot drawing tasks. But we agree with the reviewer that stroke analysis could be another interesting approach to analyze human drawings. We plan to include such analysis in further work, so that we could compare all 3 different comparison methods. \n\n* **About the low originality value on impact when evaluating human samples**: There might be a misunderstanding on this specific point. The originality plays a major role when comparing various types of generative models (e.g. VAEs, GANs, …) to humans as shown by previous authors (see Fig 3 of [1], or in Fig 2 of [2] ). But in this article, we focus on one particular type of generative model that are the latent diffusion models. Because such models tend to fall close to humans in terms of originality, we have rescaled the originality vs. diversity framework to zoom in and better highlight the differences between humans and machines (note the originality scale starting at 0.5). Such a rescaling might be the reason why the reviewer thinks that originality plays a minor role in the evaluation process. It is also important to note that there is an inherent tradeoff between originality and recognizability: while recognizability assesses how likely the data point falls in the classifier decision boundary, originality measures how ‘diffuse’ the sample distribution is (the Fig 1 in [1] well illustrate this trade-off)... Therefore, a very ‘original’ agent (producing highly diverse samples) will tend to have low recognizability as the samples are likely to fall outside of the classifier decision boundary. In Fig 2, we observe that models need to trade their originality for more recognizability to better approximate human performance. We have included two sentences in the main text (line 245) to clear up this misunderstanding.\n\n* **About the possible biases of the classifier to evaluate the recognizability**: We fully agree with the reviewer that the classifier used to evaluate the recognizability might be biased. Hoewer, we think these biases have a low impact on our analysis for two main reasons. First, we use a one-shot classifier to evaluate the recognizability (similar to the original paper that introduces the originality vs recognizability metrics [1]). Such classifiers are less prone to overfitting by construction (as they learn a metric space, see [3] for more explanation), mitigating the impact of potential biases in the training distribution. Second and most importantly, all samples (either human-drawn or machine-generated) are evaluated with the same classifier. Therefore, any potential biases in the recognizability metric will equally affect both human and machine performance, ensuring that the comparison remains meaningful. We propose to include such an explanation in the main text (line 242) to clarify this point.\n\n* **About other approaches to improve performance on the one-shot drawing task** (i.e. more combinations of regularizer): As this specific point was also raised by reviewer vmMb we have run more experiments to systematically explore more combinations of regularizers. Those experiments lead to one more figure. More details are in the bullet point 3 of the general rebuttal form. \n\n* **About the pixel baseline and the rasterization**: Our pixel baseline is indeed a diffusion model (without any latent projection) to learn to generate the distribution of pixel value. It indeed leverages rasterized images, as the original quickdraw images are vectorized.\n\n* **About the alignment of the metrics with human judgment**: We fully agree with the reviewer that having evaluation metrics (such as originality and recognizability) that align with human judgment would make our analysis more impactful. We are currently working on finding more aligned metrics (using harmonization techniques combined with psychophysics experiments). We have added 2 sentences in the discussion section (line 381) to discuss this interesting point. We thank the reviewer for the valuable comment.\n\nWe believe that our response, along with the additional curves and paragraph included in the article, addresses most of the reviewer's concerns and should encourage them to raise their rating.\n\n[1] Boutin, Victor, et al. \"Diversity vs. Recognizability: Human-like generalization in one-shot generative models.\" Advances in Neural Information Processing Systems 35 (2022): 20933-20946. \\\n[2] Boutin, Victor, et al. \"Diffusion models as artists: are we closing the gap between humans and machines?.\" Proceedings of the 40th International Conference on Machine Learning. 2023. \\\n[3] Snell, Jake, Kevin Swersky, and Richard Zemel. \"Prototypical networks for few-shot learning.\" Advances in neural information processing systems 30 (2017).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0qHIfhx9q2", "reviewer": "Reviewer_1Pi8", "summary": "The authors propose to explore how different regularizers impact the performance of LDM on one-shot sketch generation, with a specific focus on evaluating the similarity between the generated sketches and real ones. It reveals that prototype- and Barlow-based regularizers are among the best, and claims the gap between humans and machines in the one-shot drawing task is almost closed.", "questions": "- It is unclear how to construct the sketch codebook used for VQ-VAE.", "limitations": "Please refer to the weaknesses.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This seems the first time that the latent diffusion model (LDM) is applied to one-shot sketch generation. \n- The authors discussed how different regularizers impact the generation results regarding the originality (diversity) and recognizablity, which is valuable. Interestingly, prototype- and Barlow-based approaches are the best.\n- The paper is well-written and easy to follow.", "weaknesses": "- This paper is more like an incremental work based on [30], the key idea of using the diversity vs recognizability framework, and importance maps to measure the generation quality of diffusion models is the same. It differs in extending the diffusion model into latent feature space and applying different regularizers, which seems minor.\n- It is a bit over-claimed that the gap between humans and machines is almost closed on the one-shot drawing task by using LDM plus proper regularizers. The qualitative results shown are not as good as the actual sketches, suffering from clear blur and distortion. The experiments are conducted on relatively simple sketch cases, which makes it hard to justify its scalability.\n- DDPM used in [30] for this task is not compared (with or without the same regularizers if possible), it would be helpful to understand how effectiveness of pushing the denoising into latent space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer 1Pi8 for the meaningful comments. Please find below a point-by-point response that addresses the reviewer’s concerns:\n\n* **About the incremental work compared to [30]** : We agree with the reviewer that our article builds on the comparison framework and some ideas introduced in [30], but we respectfully disagree that our article does not bring significant novelties compared to [30]. Here are the 2 main novelties:\n    * While [30] focuses on comparing various types of generative models (GANs, VAEs, and diffusion models), we focus on comparing the effect of inductive biases (through regularization) in the latent space of latent-diffusion models. This difference might seem minor to the reviewer, but this question of effective inductive biases is prevalent in cognitive psychology (and is still an open question, see [1, 2] below) and has never been systematically studied in latent-diffusion models from a machine learning perspective. Our results confirm how crucial such inductive biases are as it has a strong impact on the generalization performance of the one-shot drawing setting.\n    * The method used in [30] to generate feature importance maps leads to maps in which the background is dominant (see Fig 5 in [30]), preventing the authors from making any meaningful and quantifiable comparison with human feature importance maps. In our article, we introduce a novel method to derive importance maps. In contrast to [30] our method leads to importance maps that are directly comparable with those of humans. We think this is an important difference with [30] as our results are backed by two independent methods (the recognizability from originality methods, and the importance maps comparison), strengthening our claim. \n\n  To clarify and emphasize the differences with [30] we have added a few extra sentences to summarize those 2 major differences in the related work section (lines 129-130).\n\n\n* **About the claim that “the gap between humans and machines is almost close” that might be a bit overstated**: We agree with the reviewer that this claim may be somewhat overstated, considering that our comparison between humans and machines is based on a very specific task, which does not cover the full spectrum of abilities in both humans and machines. We have therefore downplayed this claim and we have replaced it in the narrower context of our article (in the abstract, and in the discussion section).\n\n\n* **About the comparison with the DDPM**: We have actually included the DDPM from [30] in our comparison, this is what we call the pixel baseline (see Fig 2). Note that our baseline is a guided version of the DDPM because previous articles have shown that guided DDPM performs better than their non-guided counterpart in the one-shot generation task. We call it ‘pixel-baseline’ , because the DDPM is directly applied in the pixel space in contrast to other latent diffusion models that are applied in a regularized latent space. But we agree with the reviewer that this designation is rather ambiguous. We have therefore changed it to ‘pixel-space DDPM’.\n\n* **About the sketch codebook of the VQ-VAE**: The codebook in the VQ-VAE could be viewed as a dictionary of vectors, with each vector being learned so that it minimizes the L2 distance with the latent coordinate. Note that the way we learn the codebook is similar to the standard procedure described in the original VQ-VAE article [3]. To be more concrete, let’s consider the case where the latent space (before discretization) is of size (4, 128) (here we ignore the batch dimension for the sake of concision). Let’s also consider we have a codebook with 512 elements (this is the codebook size we use in the article). In this case, all 512 vectors of the codebook will be of size 4, so that they match the number of channels of the latent space. During learning, the codebook vectors are learned so that they minimize the L2 distance between 128 vectors (of size 4) of the latent space (see section A.2.1. for a pseudo-code). During inference, the discretization process associates to each of the 128 vectors of the latent space the address (i.e. an integer) of the closest vector in the notebook. This allows the transformation of the continuous-valued latent space into a discretized one. We agree with the reviewer that a clear explanation of codebook learning is missing in the article. We have included a scheme to explain this process in section A.2.1 to clarify this point.\n\nOverall, we hope that our detailed response has addressed the reviewer's concerns, and convinced them to increase their rating. If some concerns remain, we will be pleased to engage into more discussion.\n\n\n[1] Goyal, Anirudh, and Yoshua Bengio. \"Inductive biases for deep learning of higher-level cognition.\" Proceedings of the Royal Society A 478.2266 (2022): 20210068. \\\n[2] Marjieh, Raja, et al. \"Using Contrastive Learning with Generative Similarity to Learn Spaces that Capture Human Inductive Biases.\" arXiv preprint arXiv:2405.19420 (2024). \\\n[3] Van Den Oord, Aaron, and Oriol Vinyals. \"Neural discrete representation learning.\" Advances in neural information processing systems 30 (2017). \\\n[30] Boutin, Victor, et al. \"Diffusion models as artists: are we closing the gap between humans and machines?.\" Proceedings of the 40th International Conference on Machine Learning. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9mxBIb0HQX", "reviewer": "Reviewer_YDjR", "summary": "This paper investigates how different regularization techniques, applied to the latent space of Latent Diffusion Models (LDMs), impact their performance on one-shot drawing tasks. The Authors explore many regularization methods: KL divergence, vector quantization, classification, prototype-based, SimCLR, and Barlow Twins. \n\nThey evaluate these methods against human performance using quantitative metrics (originality vs. recognizability) and qualitative analysis of feature importance maps. \n\nThe results show that LDMs with prototype-based and Barlow Twins regularizations produce sketches that are most similar to human drawings in terms of both recognizability and originality.", "questions": "How generalizable are these findings to more complex drawings?", "limitations": "Limitations are adequately discussed, but they are not part of the main text (they are discussed in the Appendix, pag. 34).", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper provides a comprehensive comparison thorough examination of six different regularization techniques, offering insights into their effectiveness for one-shot drawing tasks.\n\n- It introduces a novel method for generating feature importance maps in LDMs, allowing for direct comparison with human perceptual strategies.\n\n- From the practical side, the findings have potential applications in improving generative models for tasks requiring human-like generalization abilities.\n\n- The study has an highly interdisciplinary approach since it integrates computer science, cognitive science, and neuroscience, potentially offering insights into human visual cognition.\n\nThe paper is very clearly written, and the Authors provide detailed information about their experimental setup, hyperparameters, and code availability, enhancing reproducibility.", "weaknesses": "Overall, I think that the paper makes a nice contribution to our understanding of how different regularization techniques affect the latent representations in generative models and their ability to produce human-like sketches.\n\nThe weakness I see regards the possibility to generalize from the \"simple\" datasets analyzed to more complex creative processes.\nThis study primarily focuses on the QuickDraw-FS dataset, with limited exploration of the Omniglot dataset. It not very clear to me how sound can be the extrapolation from these very simple (although relevant) contexts to more complex ones. I know that this does not provide a concrete and actionable insight, but I would appreciate a comment on this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer YDjR for the positive feedback as well as the relevant comments. We especially appreciated that the reviewer highlighted the interdisciplinary approach, which was at the heart of the article. Unfortunately, interdisciplinarity has its own limitations, especially when it comes to comparing humans and machines. This is the main reason why we have limited ourselves to rather simple datasets. We detail our answer below:\n\n* **About the generalization to other and more complex datasets**: We want to clarify that our experiments on the Omniglot datasets are actually not 'limited'. We placed the Omniglot results in the supplementary information section to keep a concise main-text article, but the Omniglot analysis is consistent with that conducted on the QuickDraw Dataset. In this article, our focus has been primarily on the one-shot drawing task because it allows fair comparison between humans and machines; more complex tasks like natural image generation by contrast are beyond human capability. Nevertheless, the latent diffusion models and the regularizers we have used in the article are known to scale well on more complex datasets (e.g. [1]). We anticipate similar performance improvements on natural image datasets as observed with the QuickDraw data, especially since regularizers like Barlow, SimCLR, and prototypical have shown strong performance in one-shot classification tasks of natural images. However such natural image generation models won’t be comparable to human performance as humans can hardly synthesize such images.\n\n\n* **About the limitations**: We agree that the limitation should be in the main text and not in the appendix. We have therefore added a paragraph summarizing our limitation in the discussion section (line 351)\n\n\nOverall, we think our response has addressed the primary concern of the reviewer, clarifying that our deliberate choice of a relatively simple one-shot drawing setting allows us to draw a fair comparison between humans and machines to effectively answer our scientific question. We hope we have convinced the reviewer to increase their rating. Should there be any remaining issues, we are more than willing to engage in further discussion.\n\n\n[1] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their time, their valuable comments and reviews. While the reviewers have acknowledged our “rigorous academic attitude… that strictly adhere to scientific methods” (Reviewer ZReU), as well as the novelty (Reviewers vmMb, YdjR) and significant practical impact of our article (Reviewers vmMb, ZReU, YdjR), we have not yet convinced all reviewers that our article clearly merits acceptance at the NeurIPS conference. Three main concerns are shared among reviewers:  i) our study is conducted on drawings rather than on natural images dataset (Reviewers vmMb, ZReU,  YdjR), ii) we did not discuss enough the effects of each regularizer (Reviewers vmMb, ZReU), and iii) we did not conduct systematic experimental comparisons on the combined effects of the regularizers (Reviewers vmMb, ZReU, YdjR):
1. The one-shot drawing setting (involving sketches dataset) is a deliberate choice. It offers a leveled playfield to compare humans and machines, which is at the heart of our scientific question. Natural image generation is a feat that surpasses human capability, making it not optimal to draw fair comparisons between humans and machines in the one-shot generation setting. We have updated the article to make this point clearer.
2. We fully agree with the reviewers that we did not deepen enough the intuition why some regularizers are better performing than others. To mitigate this issue, we have added an entire paragraph at the end of the results section : 

   “The experimental results shown in Fig2 show that not all regularizers are created equal. For the supervised regularizers (Fig 2b) we observe that the prototype regularizer produces more recognizable samples compared to the classification regularizer. This behavior is expected as the classifier learns features separating the categories from the training distribution. However, in the one-shot setting, such features might not be optimal to separate the categories of the (unseen) categories of the testing set [1, 2]. On the other hand, the prototype-based regularizer learns an embedding that clusters samples near their prototypes rather than directly mapping features to labels. Such a strategy is less prone to overfitting and produces more transferable features, making it particularly valuable for few-shot categorization tasks (see [3] for more discussions). Our experiments demonstrate that the prototype regularizer also better generalizes in the one-shot drawing setting. In Fig 2c, we observe that the Barlow regularizer outperforms the SimCLR regularizer in terms of recognizability. We attribute this better performance to the Barlow loss function's ability to disentangle features effectively (as measured by linear probing in [4]). These features also transfer more readily to different datasets ([4]), making the Barlow regularizer a better candidate than the SimCLR regularizer for the one-shot drawing task. Overall, our results demonstrate that the effective representational inductive biases in few-shot learning are also leading to better performance in the one-shot drawing task.”

   In addition, this paragraph also answers the reviewer's ZReU about the ‘lack of clear response to the scientific question’.  Indeed, the new paragraph highlights the fact that regularizers proven effective in the one-shot classification tasks tend to be also effective in the one-shot drawing setting. It therefore gives a clear answer to our scientific question:  “Do representational inductive biases from one-shot classification help narrow the gap with humans in the one-shot drawing task” (clearly stated in line 46). 

3. We have included more experiments to systematically study the combined effect of the regularizers. In particular, we have systematically explored the following combinations of regularization Barlow + Prototype: KL + prototype, SimCLR+Prototype, VQ + Prototype. Among all combinations of regularizers, it is the Barlow + Prototype and the KL + Prototype combinations that perform the best. Interestingly, the good performance of the combination  KL + Prototype was not expected, because the KL regularizer (when used separately) shows a low recognizability (see Fig2). On the other hand, the VQ + Prototype combination shows no improvement compared to the Prototype regularizer. Overall, these additional experiments confirm the potential of combining an unsupervised and a supervised regularizer to match human performance. The findings, illustrated by a new figure (see attached pdf) are included in the revised version of the article. This additional analysis requested by the reviewers required the training of around 600 additional latent diffusion models, amounting to about 1800 days/gpu computation time. 

We have also included reviewer-specific answers that individually address the reviewer’s concerns. We hope this will convince the reviewers of our article's value and its suitability for publication at NeurIPS.

[1] Snell, Jake, Kevin Swersky, and Richard Zemel. "Prototypical networks for few-shot learning." Advances in neural information processing systems 30 (2017). \
[2] Vinyals, Oriol, et al. "Matching networks for one shot learning." Advances in neural information processing systems 29 (2016).
[3] Li, Xiaoxu, et al. "Deep metric learning for few-shot image classification: A review of recent developments." Pattern Recognition 138 (2023): 109381. \
[4] Zbontar, Jure, et al. "Barlow twins: Self-supervised learning via redundancy reduction." International conference on machine learning. PMLR, 2021.

Meta-review: While use of a combined evaluation of originality and recognizability is not novel in the field, the application of this evaluation framework to human sketches is a non-trivial contribution.  The testing of multiple regularizers, and the addition of combined regularizers, is likely to be impactful for the field of ML models of human-like sketching and character writing.  While the paper could be strengthened by adding evaluation of stroke features or a human evaluation of the human-likeness of the sketches, or a deeper analysis of the performance of the regularizers, I think the paper provides enough of a contribution to be useful to the field as it currently stands.
Decision: Accept (poster)