Drawing out of Distribution with Neuro-Symbolic Generative Models
Authors: Yichao Liang
Joshua B. Tenenbaum
Tuan Anh Le
Siddharth N
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neuro-Symbolic Models
Generalisation
Unsupervised Learning
Omniglot

Abstract: Learning general-purpose representations from perceptual inputs is a hallmark of human intelligence. For example, people can write out numbers or characters, or even draw doodles, by characterizing these tasks as different instantiations of the same generic underlying process---compositional arrangements of different forms of pen strokes. Crucially, learning to do one task, say writing, implies reasonable competence at another, say drawing, on account of this shared process. We present Drawing out of Distribution (DooD), a neuro-symbolic generative model of stroke-based drawing that can learn such general-purpose representations. In contrast to prior work, DooD operates directly on images, requires no supervision or expensive test-time inference, and performs unsupervised amortized inference with a symbolic stroke model that better enables both interpretability and generalization. We evaluate DooD on its ability to generalize across both data and tasks. We first perform zero-shot transfer from one dataset (e.g. MNIST) to another (e.g. Quickdraw), across five different datasets, and show that DooD clearly outperforms different baselines. An analysis of the learnt representations further highlights the benefits of adopting a symbolic stroke model. We then adopt a subset of the Omniglot challenge tasks, and evaluate its ability to generate new exemplars (both unconditionally and conditionally), and perform one-shot classification, showing that DooD matches the state of the art. Taken together, we demonstrate that DooD does indeed capture general-purpose representations across both data and task, and takes a further step towards building general and robust concept-learning systems.


Drawing out of Distribution with Neuro-Symbolic Generative Models
Yichao Liang!? 1,2, Joshua B. Tenenbaum", Tuan Anh Le
3 & N. Siddharth"
University of Oxford, 'MIT, 'Google, *University of Edinburgh
Abstract
Learning purpose representations from perceptual inputs is a hallmark of human intel- ligence. For example, people can write out numbers or characters, even draw doodles, by characterizing these tasks as different instantiations of the same generic underlying process- compositional arrangements different forms of pen strokes. Crucially, learning do one task, say writing, implies reasonable competence at another, say drawing, on account of this shared process. We present Drawing out of Distribution (DooD), a neuro-symbolic generative model of stroke-based drawing that can learn such general-purpose representations, In contrast to prior work, DooD operates directly on images, requires no supervision or expensive test-time inference, and performs unsupervised amortised inference with symbolic stroke model that better enables both interpretability and generalization. We evaluate DooD on its ability to generalise across both data and tasks. We first perform zero-shot transfer from one dataset (e.g. MNIST) to another (c.g. Quickdraw), across five different datasets, and show that DooD clearly outperforms different baselines. An analysis of the learnt representations further highlights the benefits of adopting a symbolic stroke model. We then adopt a subset of the Omniglot challenge tasks, and evaluate its ability to generate perform one-shot classification, showing that DooD matches the state of the art. Taken together, we demonstrate that DooD does indeed capture general-purpose representations across both data and task, and takes a further step towards building general and robust concept-learning systems.
exemplars (both unconditionally and conditionally), and
QuichDraw
MNIST
EMINSST
KMINEST
Ormuggt
1 Introduction
Humans can learn representations of data that are general- purpose and meaningful. Being general-purpose permits ef- fective reuse when characterizing novel observations, and being
6 5
W
4
8
7
meaningful facilitates tasks like generating or classifying ob- servations. Key to this is generic process for characterizing observations-iinerrii what features relevant and how they compose to generate the observations
6
57
0
0
6
4 4
4
+
6
C
4
6
[9] [3] 33 For example,
A 2
when observing handwritten numbers, we learn to characterise them as sequential compositions (how) of different pen strokes (what). This is general-purpose as allows characterizing novel observations, say doodles instead of numbers, simply as novel compositions of previously learnt pen strokes. It is also meaning- ful since pen- ren-strokes themselves are symbolic and interpretable.
Figure 1: DooD trained on MNIST generalises other data with no extra training. Each column denotes a target and its step-by-stcp reconstruction.
Current computational approaches capture important aspects of generalizability, but none of these is simultaneously efficient, reliable, interpretable, and unsupervised (25). At one end, symbolic approaches like Lake et al. 24 attribute generalization an explicit hierarchical composition process involving sub-strokes, strokes, and characters, and build concomitant models that demonstrate human- like generalization abilities across different tasks. At the other end, neural approaches like deep generative models [9] [8 29 and deep meta-learning [1I 32 34 favour scalable learning from raw
"qqual contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

STN
STN
St
8t
6R
4-1
O
A.C
?
2
2
stop.grddieus Spaiia Trandomars (70)
Eet
Figt
? Ze
Generative Model
Recognition Model
The generative model sequentially samples both an image location (4) and a corresponding stroke (st) at that location. The rendered stroke Z4 is composited onto intermediate rendering Ziet to produce Z-5.- A binary on/off variable (ox) determines when to continue drawing. Differentiable rendering () and differentiable affine transformations via Spatial Transformer Networks (STNs) [20] enables gradient-based learning. The recognition model conditions
Figure 2:
residual Ar, x Zet to sample where to draw next (le) and what to draw next (st), and whether to continue drawing (0t).
Both models are autoregressive via two (shared) RNNs with hidden states he and hy.
perceptual data, unfettered by explicit representational biases such as strokes and their compositions. Each comes with its own peartconings--yymeeen approaches typically need additional supervision or data processing along with expensive special-purpos inference, and neural approaches fail to generalise well and don't 't capture interpretable representations, Neuro-symbolic approaches [10) seek to make the best of both worlds by judiciously combining neural processing of raw perceptual inputs with symbolic processing of extracted features, but typically involve different set of trade-offs.
We present Drawing out Distribution (DooD), a reuro-symbolic generative model of brose- drawing that can learn cemerrl-purpont representations (Fig. T). Our model operates directly on images, requires no supervision, pre-processing, or expensive test-time inference, and performs efficient amortised inference with symbolic stroke model that helps with both interpretability and generalization, setting us apart from the current sfate-of-the-art in neuro-symbolic approaches [T0 [7]. We evaluate on two axes (a) generalization across data, which measures how well the learnt represen- tations can be reused to characterise out-of-distributioo data, and (b) generalization across task, where measure how useful the learnt representations are for auxiliary tasks drawn from the Omniglot challenge set (24.. We show that DooD significantly outperforms baselines on generalization across factor, and generalization across tasks, show that it outperforms neural models, while being competitive against SOTA neuro- ceuro-symbolic models without requiring additional support such as supervision or data augmentation.
datasets, highlighting the quality of the learnt representations
Method
The framework for DooD involves generative model over sequences of strokes and their layouts, a recognition model that conditions on given observation to predict where to place what strokes, and amortised irrrational-iffreente learning setup that these models to estimate an evidence lower bound (ELBO) as the objective.
2.1 Generative Model
Conceptually, the model can be seen as drawing figure over a sequence of steps, building up to the final image (as seen in Fig. T. At each step & the model identifies region of the image canvas to draw in, puts down Bézier curve control points within that region, renders the curve in a differentiable manner, and then composites this rendered stroke these as the layout, stroke, rendering, and compositing modules respectively (elaborated below).
the previously rendered canvas. We refer to
The model sits on substrate of recurrent neural networks (RNNs), one each for the layout and stroke modules, with hidden states iy; and h respectively. The complete setup depicted in Fig. alonn with example values (images) for the different variables involved. See Appennixx B for details.

---

Formally, the generative model defines joint distribution over an image a composed over T steps, with latent variables 4 and 8; that characterise stroke's location and form, and a binary latent variable that determines stopping,
p(z, LgT, 0gT+))
II Pon (on(or) Pstrok Tet, (i))
hi))
Peomp (x|Z<T) pon Pon(or+l/gTT) (I)
where terminal step 7" is the last & where Oz 1, Hy the rendered stroke given (st, 44,, I'e the canvas-so-far, and all variables at step & = 0 initialised to O. Note that the likelihood is evaluated only at the terminal step T.
Layout Module: At step t. given the canvas-so-fatt <<t and corresponding layout-RNN hidden state fig we define the layout as a distribution over afline transforms. This allows transforming the canvas into a "glimpse" Éet using Spatial Transformer Network (STN) (20, which allows focussing on particular canvas region. The affine transform is constructed from appropriately constrained scale (1:9) translation (,)), and rotation (4F) random variables, by employing a Gaussian Mixture Model (GMM) with M components over the collection as
M
Playout (1el0 Siset, M1) =
Σ
Om NN (1."""// 11) Na (lt |r (r),,))))) Nr (r hi), (2)
m=
Tet= - STN(S,,,,, x21).
Stroke Module: Given the sampled affine transform 4, selected "glimpse" ãeg, and corresponding stroke-RNN hidden state hf, this module defines distribution strokes parametrised 1) order Bézier splines, constructing a GMM with K components for each spline control point as
D
K nak M4,
Pstroke (Selle Te,, h3)
II
(3)
Note that the Bézier spline control points sampled from the stroke module are taken to in canonical centered coordinate frame. While this can help simplify learning by reducing the variation required to be captured by the stroke module, the points can't be rendered onto the canvas as is. We situate them properly within the context of the previously determined "glimpse" by simply applying the affine transform le the control points 3, le O 8t- The transformed control points now describe a stroke to be drawn over the whole canvas for step t. through a differentiable renderer oR, to produce the rendered stroke x, oR(;,) (see Appendix B. for details). Using the intermediate canvas to guide model unrolling is termed execution guidance (EG).
Rendering Module:
Compositing Module: At step this defines a distribution over whether the model should continue drawing strokes given the rendered canvas-so-far Fett
Pon (x))) = Bernouli (<)))
(4)
Once the model has decided to stop drawing, it stops permanently. At step t, the rendered stroke x; is composited with the canvas-so-farr det to generate zi: = xec X xt. When the model stops, i.e., at the last / when = 1 (denoted T), the likelihood, with global learnable parameter a,, is given as
Peomp (e (r)))))== Laplace (aigg,
2.2 Recognition Model
As with prior approaches, we construct an approximate posterior to facilitate learning with amortised variational inference. Using the same notation from the generative model section, we define
T
GllTT 85T,
(or+1 (5T,,5TT11 -
AUT+)
Maayour (1,, (i) Isrook Ar, hi (o: (6) t=
couple of things stand out. First, where the generative model made heavy use of the canvas-so- far Fet. the recognition model primarily uses the residual Axt Z Second, being given
3

---

the target observation x itself, the information available to the layout and stroke modules is quite different. While they needed to speculate where, and what stroke draw, the generative model, here their task is simply to isolate part of the drawing ("glimpse") and fit spline to that.
As consequence of these different characteristics, the distributions over layout and strokes in the recognition model do not need to be as flexible as the generative model-locating a curve in the residual and fitting with a spline does not typically involve much ambiguity. To factor this in, and have the variational objective be reasonable, we define corresponding distributions in the recognition model using just single component of the corresponding GMMs in the generative model as
Payoou (4e Aze, (b)) = (1F) Az, h!) -N Arc, h!) (N Ar h(),
Srz STN Am)),
D II Na(o5) N.). d=0
Istroke (8x) Swt, h.)
(8)
2.3 Learning
Having defined the generative and recognition models, we now bring them together in order to construct the variational objective that will enable learning both models simultancously from data.
logp(z) 2 Buglerr (erlorrrgrroetelee log ((gT,, SgT, 0gT+1
Note that except for the stopping criterion o, which is Bernoulli random variable, all other distribu- tions employed are reparametrizable. In order to construct an effective variational objective with this discrete variable, we employ a control variate method, NVIL (28), that helps reduce the variance of
the standard REINFORCE estimator, as is also done in related work [9] (see (ppeendxii C.2).
Furthermore, in order to ensure that the ELBO objective is appropriately balanced, employ additional weighting 3 for the KL-divergence over stopping criterion or within the objective 19 (see Appendix C.I for details). This weight plays crucial role mismatch could result in the model either stopping too early or too late, resulting in incomplete or incorrect figures respectively.
Experiments
We wish to understand how well DooD generalises across both datasets $ 3.1 and tasks ($ 3.2). For across-dataset generalization, we train DooD and Attend-Infer-Reeeer (AIR) an unsupervised part-based model, on each of five stroke-based image datasets (i) MNIST (handwritten digits) 26),, (ii) EMNIST (handwritten digits and letters) 77]. (iii) KMNIST (cursive Japanese characters) [6],
(iv) Quickdraw (doodles) [15], and (v) Omniglot (handwritten characters from multiple alphabets)
24 and evaluate how well the model generalises to unseen exemplars both within the same dataset and across other datasets. We find that DooD significantly outperforms AIR, which from ablation studies, is attributed to explicit stroke modelling and execution guidance. Note that we only compare against a fully-unsupervised approach since most datasets do not provide additional data in the form of stroke labels (as required elsewhere (10). For across-task generalization, we primarily focus on Omniglot and evaluate on three out of the five challenge tasks for this dataset (25), which include contextual generation and classification. We find that our model outperforms unsupervised baselines, and is competitive against SOTA neuro-symbolic models without requiring additional support in the form of supervision or data augmentation. We include exact details about datasets (Appendix A). our model and the baselines (Appendix B), the training procedure (Appendix C), and the evaluation procedure (Appendix D) in the supplementary material.
3.1 Across-Dataset Generalization
MNIST-trained transfer. To understand how model and AIR generalise new datasets, look at sequential reconstructions (Fg. 3). We train on MNIST and show sample reconstructions from all five datasets without fine tuning. Each model renders one step at time by rendering latent parses of increasing length, allowing us to evaluate and compare the performance of part decomposition and inference. Note that limit the maximum number of strokes to 6 throughout all experiments.
Sspecifically Difference-AMee AIR, which execution guidance and performs much better than vanilla AIR

---

(a)
MINSST
EMNIST
KMNIST
QuickDraw
Omniglot
7 4
3 60
E
C
5 / 7
6
7
DooD
X
(b) DooD
/ 77 7
TKKK D1
20 Sicaci
AIR
PPPX
PLATOOr atotkar
DooD-RR
(c) Difference AIR
AAAAA
DoeD-EG
XXXX
0
0 /
KXX
MIKI
/ 4
0 1
(d) Vanilla AIR
Figure 3: (a) Our model generalises better than AIR. Our model trained MNIST reconstructs characters from all other four datasets while the baseline AIR model's reconstructions are often inaccurate, blurry incomplete. Explicit stroke parametrization (őR) and execution-guide inference (EG) are responsible for this generalization which degrades when using our model without either of these components. (b-d) Both DooD and Difference AIR (AIR elsewhere) trained MNIST generalise to using more strokes unlike Vanilla AIR which doesn't have execution-guidde inference.
Our model reconstructs in-distribution images perfectly and out-of-distribution images near-perfectly while using fewer strokes for simpler datasets (e.g. MNIST) and more strokes for more complex datasets (e.g. Omniglot). While the AIR baseline also uses an appropriate number of steps for more complex datasets, the reconstructions degrade significantly for blurry (e.g. the car & motorbike in QuickDraw), strokes go missing (e.g. the second KMNIST image) or the reconstructions are inaccurate (e.g. the last Omniglot character).
images they are
Ablation studies. To better understand why our model generalises well. we evaluate two further variants of DooD that ablate key component each: an explicit spline decoder (DooD-öR) and execution-guided inference (DooD-EG) (Fig. 3R). In the model without explicit spline decoder, we replace the differentiable spline renderer by a neural network decoder similar to AIR. This model still differs from the AIR terms of the learnable sequential prior and the fact that we enforce explicit constrains over the latent variable ranges e.g. enforcing the mean of the control-poin Gaussian to not stray too far away from the image frame. In the variant without execution guidance, we do not perform intermediate rendering, removing the direct dependence of the generative model and the recognition model on the canvas-so-far Jeg and the residual Ar,-
Both the explicit spline decoder and the execution guidance prove to be important, Without the explicit spline decoder (DooD-oR), the reconstruction quality suffers- the strokes are blurry (c.g. first three QuickDraw images), strokes go missing (e.g. the last EMNIST image), or the reconstructions inaccurate (c.g. the last Omniglot character interpreted even without the explicit spline decoder, the model learns to be parsimonious, using fewer strokes to reconstruct simpler images (Fig. 3b-d). On the other hand, without execution guidance (DooD-EG), to be selective with the number of strokes, always using the maximum allowed number. And while the reconstructions are better than AIR and DooD-òR, it still shows instances of missing strokes (e.g. some Omniglot characters). Note that although we use the canvas-so-far in conditioning variable for
"9" due to overfitting). However,
the model is unable
manner that disallows gradients satoo_gradii just providing the different components (layout, stroke, RNN hidden states) has tangible effect.
5

---

7
5
I
2
4
V
86
8
9 7
5
I
2
4
86
g
न
7
4111
2013
3331
4
273 8
ST92
3668
7
5
I
44
9 7
5
7
3 7
61643
1716-3
34167
6311.2
65612
826-2
2003
7
5
T
H
9 7
5
I
272
200
1XNAST
7 3
I
44
H
9 7
5
6
245.3
174.4
2350
6
3913
1413
:48.
105.
:063
-56
1542
Oo
7 5 T
Q
1T4R-7
14862
11919-1
JASE A
D
211-
7 5 RES
pa
Brgat carne
Terpet cirsuse
Figure 4: When training on "source" dataset and testing another "target" dataset, our model, DooD, (left) has a higher log marginal likelihood (values in each cell) than AIR (right). Given targets on top of the tables, DooD's reconstructions (images in each cell) are high quality when transferring out of distribution, unlike AIR which often struggles. Training on MNIST or Omniglot as a source dataset leads to worse transfer (the corresponding rows are the darkest) due to larger distribution shift. Particularly AIR fails when trained on Omniglot using a Laplace likelihood (standard across all other model-dataset combination for good reconstructions), due which we employ a Gaussian likelihood just for Omniglot-trained AIR (highlighted in purple).
Quantifying zero-shot transfer. We look at how DooD and AIR trained on each of the five datasets transfers to each other dataset to further understand how our model generalises. Models are trained on each "source" dataset and tested on each "target" dataset, resulting in a x 5 table for each model (Fig. 4p. Each cell shows the log marginal likelihood of the target dataset using the model trained on the source dataset, estimated using the importance weighted autoencoder (IWAE) objective [3] with 200 samples (mean and standard deviation over five runs). We also show reconstructions obtained by running the model trained on the source dataset on a few examples from the target dataset.
Our model generalises significantly better than AIR across datasets (off diagonal cells), while also performing better within dataset (diagonal cells). For both models, the values on the diagonal are the highest in any given column, suggesting that not training directly the target dataset results in a worse performance, as expected. For both models, the row values for MNIST and Omniglot are lower than in other rows, indicating that transfer learning performance is the worst when the source dataset is MNIST or Omniglot potentially due to a larger distribution shift since MNIST has low diversity and Omniglot has little to no variation in stroke thickness, in contrast to the other datasets. However, we note that our reconstructions are high quality despite transferring out of distribution, unlike reconstructions from AIR which are qualitatively worse. For example, when transferring from simple datasets (MNIST), AIR makes incomplete, incorrect and blurry reconstructions, as we have seen before, while AIR trained on complex datasets like Omniglot results in blurry reconstructions for both in-distribution and Omniglot using a Laplace likelihood (used as standard across all other model-dataset combination). We thus employ a Gaussian likelihood just for Omniglot-trained AIR, and highlight it as an outlier.
datasets. Furthermore, AIR fails when trained on
Understanding learned representations. To better understand DooD's generalization ability, we investigate its learnt representations by clustering the inferred strokes using k-means clustering (k = 8), and study the clusters both qualitatively and quantitatively. For AIR, we cluster the corresponding pert-represnntttiion latents. We then visualise things, using a t-SNE plot (Fig. 5) of the clusters, with exemplar strokes overlaid. We find that DooD has cltter--lustered representations, with clusters denoting largely distinct types of strokes-e.g. clusters for a "/", "c",, and its horizontally flipped version. In contrast, the clusters from AIR are less sensible with some clusters even capturing full characters ("0"), comprising multiple strokes. There are also clusters which contain visually different strokes, and many visually similar strokes are assigned to different clusters. Quantitatively.

---

following Aksan et al. 1 we found DooD's s better cluster consistency is reflected in a higher
Silhouette Coefficient 30 than AIR (0.21 for DooD, 0.11 for AIR).
Figure 5: Clusters of inferred strokes for DooD (left) and inferred part representations for AIR (right) overlaid t-SNE plot. DooD's representation clusters to more semantically meaningful parts as indicated by better formed clusters.
3.2 Across- Task Generalization
Here, focus subset of the Omniglot challenge tasks (24]. to evaluate model's utility a range of auxiliary tasks that was not trained to do. Despite much progress in deep generative modelling, relevant models are still not fully task general and often result in unrealistic (e.g. blurry) samples [10 [25]. DooD combines handling raw perceptual inputs with the compositional structure of strokes, which evaluate three out of the five Omniglot challenge tasks: unconditional generation, conditional generation, and one-shot classification. We compare against AIR and state-of-the-art neuro-symbolic model (GNS [I0), where relevant. Note that GNS requires stroke and character-class supervision and practically, at least for now, only applies to Omniglot.
Unconditional generation. DooD generates realistic unconditional samples of all datasets Fig. 6). indicating that the model has learned the high-level characteristics of each dataset. The strokes are sharp, and the stroke structure composes into realistic images from each dataset. For example, there are clear digits in the MNIST samples, there are recognizable objects (cars, bicycles, glasses, and smileys) in the QuickDraw samples, and the samples for EMNIST, KMNIST and Omniglot can be easily recognised as possible instances coming from those datasets. generates samples of comparable fidelity to GNS without requiring any supervision, and as evaluated using the Fréchet inception distance (FID) [6 (smaller better), outperforms GNS (0.051 versus 0.133).
The key to being able to generate realistic prior samples is the learnable sequential prior and the symbolic latent representation. AIR doesn't 't have sequential prior, so although it is possible get good reconstructions, it is impossible for it to generate realistic unconditional samples.
Character-conditioneel generation. In order to generate new exemplars of the same Omniglot character, follow Feinman and Lake [10), Lake et al. [24] and extend our model to hierarchical generative model of an abstract character "type" or "template" that generates a concrete instance of a character "token", which is rendered out an image. We consider the previously used latent variables as the type latent variable and introduce a token model which conditions on the type latent variable. The token model introduces (i) drawing noise represented by adding a Gaussian noise with fixed standard deviations to spline control points and (ii) and an affine transformation on the noise perturbed points, whose parameters are also sampled from a Gaussian distribution (described in Appendix B. To sample a new exemplar of character, we first sample the type variable from
MNIST .013
EMNIST
KMNIST .223 + .200
QuickDraw .081 + .009
Omniglor (DooD) .051 + .007
Omniglot (GNS
Omniglot (True) .025 = .004
.131
.17 + .006
.133 + .007
7
4
8
~
1 7
S
H
2
4
2
$
7
1
12
7
L
2a h
M
X
2 2
1
4
+
1
UI
17
Ó
5
?
3
7
0
2
0
3
d
W 4
7
Figure 6: DooD generates high quality unconditional character samples for all datasets which are visually indistinguishable from the real characters as it successfully captures the layout of strokes and their forms. Omniglot samples are compared to GNS [10 and real samples. Numbers denote Fréchet inception distance (FID), with smaller being better (mean + std. over 5 runs).
7

---

DooD (QD)
DooD (OM)
AIR (OM) Z
GNS 10 H H
(OM)
o
2
H
X
E 2 7 7
R
K
X
4 = T
=
X
a
R
X X
4 H
X
H X 4 X
2
R
H
2
=
1
R
1 7
H
= R BARH
-
H
2
R
H
X
WX
Figure 7: Given a target image of a handwritten Omniglot character, our model produces realistic new exemplars by inferring an explicit stroke-based representation.
MNIST
EMNIST
KMNIST
QuickDraw
Omniglot
/ 1
/
-
9 1 0 7 1 0 9 7 0 2
P
1 0 T
6
*o
2 1 4 3 x H
4
7
2
1
"
7
R
4
0
2)
66
T
7 /
UP
39 9
0 4
0
R
2)
5
I 4 0
9 1 2 7
2
3:
h
4
PL
Figure 8: Given partially drawn character, our model can generate realistic distribution over its completions by sampling from the generative model conditioned on the image of the partial character.
recognition model, and produce different exemplars by sampling and rendering different token variables given this type variable. Distinctly from [00 (24), the parameters of the token model are not learned through supervision but simply set sensible value by examining the noise level in the output - any learned statistics can be straightforwardly plugged in.
exemplars of complex QuickDraw drawings and Omniglot characters FFg.. 7) thanks to the accurate inference and the ability to add noise to explicitly parametrised strokes. While we can add an equivalent token model for AIR by (i) adding Gaussian noise to the uninterpretable feature vector representing each part and (ii) applying a Gaussian affine transformation to the rendered image, the new exemplars are not realistic both because of worse inference and the hard-to-conttol variations of the part vectors. GNS generates realistic conditional samples, but notably still makes unnatural samples in multiple instances (e.g., in column the detachments of strokes) despite having hierarchical model learned with multi-levels of supervision.
DooD generates realistic
Partial completion. As with inferring an entire figure in the previous case, we can interpret conditional generation in a slightly different way as well-where the condition is an initialization of a number, character, or figure, and the model tries to extend/complete it as best it can (Fig. 8). To do this, we first employ the recognition model over the partial figure to compute the hidden states of the shared recurrent networks. Next, starting with these computed states, we set the canvas-so so-far t<t be the partial figure itself and then unroll the generative model from that point onwards. As can be seen in the figure, DooD can generate a varied range of completions for each initial stroke, demonstrating its versatility and the utility of its learnt representations.
One-shot classification. Finally, we can apply the type-token hierarchical generative model used for generating new exem- plars to perform within-alphabet. 20-way one-shot classification. The key quantity needed for performing this task is the poste- rior predictive score of the query image a: given set of sup- port images marginalizing over the token variables corresponding to (:(T) and gr(e), and the type variable of x((o)_ Following 110 (4], we approx- imate this score by sampling from the recognition model given r(()) and perform gradient-basee based optimization to marginalize out the token variable of (T( (details in Appendix D.2.3). We find that DooD outperforms the neural baseline (AIR), while attaining competitive accuracy in comparison to other baselines Table
Table 1: Accuracy in one-shot clas- sification, without data augmentation (DA), extra supervision (ES). or way classification (2W).
{-((e) Je. ė = arg max, p( (T) (x())) which requires
Model
DA
ES 2W
Accuracy
AIR
X
X
X
14.5%
DooD
X
X
X
73.5%
VHE
18
/
X
81.3%
GNS
10
X
/
V
94.3%
BPL
X
/
96.7%

---

without requiring additional forms of support such as data augmentation, supervision for strokes, or more complex ways of computing the classification score such as two-way scoring (predicting è arg max, p( p(x(T) x)()))).
plate))
4 Related Work
Our work takes inspiration from Lake et al. [24]'s symbolic generative modelling approach which hypothesises that the human ability to generalise comes from causal and compositional under- standing that characters are generated by composing substrokes into strokes, strokes into characters and rendering characters to images. As result, Lake et al. [24] demonstrate human-like general- ization on a wide range of tasks. However, it is trained using stroke sequences, and inference is performed using expensive MCMC sampling.
We combine features of neuro-symbolic generative models and deep generative models to able to generalise well across tasks while using amortised inference and being unsupervised. From neuro-symbolic models, we share key features of Feinman and Lake [O]'s model like (i) using the canvas-so-far far in the generative model and adopt similar feature in the recognition model like Ellis et al. [8] (ii) parametrizing parts as splines and using a differentiable spline renderer, (ii) extending the model to have type-tokee toket hierarchy for generating new exemplars and performing one-shot classification. Our model can be seen as an extension of [10] that learns directly from images and recognition model for amortised inference. Like Hewitt et al. [17], we learn how to infer stroke sequence directly from images using a differentiable renderer but infer strokes directly instead of learning a stroke bank and a more flexible parametrization of strokes based differentiable spline renderer, leading to more accurate model.
attention to focus on parts of the canvas we want to generate to or recognise, and more generally exploit the compositionality of objects from which allows neural networks to learn simpler and hence more generalizable mappings. To be able to train our model from unsupervised images, we adopt the NVIL control variate [28] used by Eslami et al. [9] to be able train a model with discrete stop- draw--i latent variable. This family of models, along with deep meta-learning approaches [1]] 32 34,, is easier to learn due to the lack of symbolic variables and results in a fast amortised recognition model. However, the lack of strong inductive biases leads to poor and unreliable generalization 25 We also share idea with other works combining deep learning and explicit stroke modelling D
Similar to deep generative modelling approaches like 9
14 23 29
12 15 27], but focus learning a principled generative model which allows tackling tasks like one-shot classification and generating new exemplars, in addition conditional and unconditional sampling.
5
Conclusion
We demonstrated that DooD generalises across datasets and across tasks thanks to an explicit symbolic parametrization of strokes and execution guidance. This allows us to train one dataset such as MNIST and generalise to a more complex, -ittr--distribttion dataset such as Omniglot. Given a compositional representation and associated learned sequential prior, DooD can be applied to additional tasks in the Omniglot challenge like generating new exemplars and one-shot classification by extending have a type-token hierarchy. Our model produces realistic new exemplars without blur and artefacts unlike deep generative models.
More broadly, DooD is example of system that successfully combines symbolic generative models to achieve generalization and deep learning models to handle raw perceptual data and perform fast amortised inference while being learned from unsupervised data. We believe these principles can be useful for building fast, reliable and robust learning systems going beyond stroke-based data.
Acknowledgements We thank Rob Zinkov for his contribution to the early versions of the renderer: Reuben Feinman, Luke Hewitt, Kevin Ellis for their thoughtful comments on the manuscript; and Reuben's 's prompt support in running the GNS model. NS was in part supported by the Turing 2.0 'Enabling Advanced Autonomy' project, funded by the EPSRC and the Alan Turing Institute.

---

References
[1] E. Aksan, T. Deselaers, A. Tagliasacchi, and O. Hilliges. Cose: Compositional stroke embed- dings. Advances in Neural Information Processing Systems, 33:10041-10052, 2020.
[2] S. Bowman, L. Vilnis, O. Vinyals, A. Dai, R. Jozefowicz, and S. Bengio. Generating sentences from a continuous space. Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-21, 2016.
[3] Y. Burda, R. Grosse, and R. Salakhutdinov. Importance weighted autoencoders, arXiv preprint arXiv:1509.00519 2015.
[4] C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Lerchner. Monet: Unsupervised scene decomposition and representation, 2019. URL https://arxiv
[5] Chung. K. Kastner, L. Dinh, K. Goel, A. C. Courville, and Y. Bengio. A recurrent latent variable model for sequential data. Advances in neural information processing systems, 28: 2980-2988, 2015.
[6] T. Clanuwat, M. Bober-Irizar, A. Kitamoto, A, Lamb, K. Yamamoto, and D. Ha. Deep learning for classical japanese literature. arXiv preprint 18/2.018/2.0111 2018.
[7] G. Cohen, S. Afshar, J. Tapson, and A. Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN) pages 2921-2926. IEEE, 2017.
[8] K. Ellis, M. Nye, Y. Pu, F. Sosa, J. Tenenbaum, and A. Solar-Lezama. Write, execute, assess: Program synthesis with a repl. Advances in Neural Information Processing Systems, 32, 2019.
[9] S. Eslami, N. Heess, T. Weher, Y. Tassa, D. Szepesvari, G. E. Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances Neural Information Processing Systems, 29:3225-3233. 2016.
[10] R. Feinman and B. M. Lake. Learning task-general representations with generative neuro- symbolic modeling. arXiv preprint 1.XY::2006.144443 2020.
[11] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
[12] Y. Ganin, T. Kulkarni, I. Babuschkin, S. A. Eslami, and O. Vinyals. Synthesizing programs for images using reinforced adversarial learning. In International Conference on Machine Learning, pages 1666-1675. PMLR, 2018.
[13] M. A. Goodale and A. D. Milner. Separate visual pathways for perception and action. Trends in neurosciences, 15(1):00-25, 1992.
[14] K. Gregor, I. Danihelka, A. Graves, D. Rezende, and D. Wierstra. Draw: recurrent neural network for image generation. In International Conference on Machine Learning, pages 1462-1471. PMLR, 2015.
[15] D. Ha and D. Eck. A neural representation sketch drawings. arXiv preprint 17X0v:100..03477. 2017.
[16] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances neural information processing systems, 30, 2017.
[17] I.. Hewitt, T. A. Le, and J. Tenenbaum. Learning to learn generative programs with memoised wake-sleep. In Conference on Uncertainty in Artificial Intelligence, pages 1278-1287. PMLR, 2020.
[18] I. B. Hewitt, M. I. Nye, A. Gane, T. Jaakkola, and J. B. Tenenbaum. The variational homoen- coder: Learning to learn high capacity generative models from few examples. arXiv preprint arXiv:1807.08919, 2018.
10

---

[19] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerch- ner. beta-VAF Learning basic visual concepts with a constrained variational framework. In Proceedings of the International Conference on Learning Representations, 2016.
[20] M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
[21] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[22] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6111 2013.
[23] A. Kosiorek, S. Sabour, Y. W. Teh, and G. E. Hinton. Stacked capsule autoencoders. Advances neural information processing systems, 32, 2019.
[24] B. M. Lake, Salakhutdinov, and B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, (33066666):133221153 1338, 2015.
[25] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. The omniglot challenge: 3-year progress report. Current Opinion in Behavioral Sciences, 29:97-104, 2019.
[26] Y. LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nnistl, 1998.
[27] F. Mellor, E. Park, Y. Gamin, I. Babuschkin, T. Kulkarni, D. Rosenbaum. A. Ballard, T. Weber, O. Vinyals, and S. Eslami. Unsupervised doodling and painting with improved spiral. arXiv preprint arXiv: 1910000010 2019.
[28] A. Mnih and K. Gregor. Neural variational inference and learning in belief networks. In International Conference on Machine Learning, pages 1791-1799. PMLR, 2014.
[29] D. Rezende, I. Danihelka, K. Gregor, D. Wierstra, et al. One-shot generalization in deep generative models. In International conference machine learning, pages 1521 15299. PMLR, 2016.
[30] P. J. Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987.
[31] Schulman, N. Heess, T. Weber, and P. Abbeel. Gradient estimation using stochastic computa- tion graphs. arXiv preprint 1506.
2015.
[32] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[33] Tenenbaum and W. Freeman. Separating style and content. Advances in neural information processing systems, 1996.
[34] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.


Reviews:
[{"review_id": "zw4zkAingeU", "reviewer": "Reviewer_xiD8", "summary": "the work models the drawing, especially, consisting of simple strokes, as a sequential generation problem. The proposed approach leveraged the spatial transformer network to learn and decouple the variance between samples originating from spatial transformations. The generative approach is applied in two task: generating new plots, and one-shot classification. ", "questions": "How did the authors choose the baseline models to compare? is every model in table 1 generative? Why they are not compared with general classification models?\n\n", "limitations": "The limitation was discussed outside the main submission. The approach handles a very specific task and the generalization to other tasks are limited. ", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ plenty of qualitative results and benchmark results are presented. the advantage is highlighted clearly\n\nWeakness:\n- AIR is a baseline from 2016, which is pretty outdated. The method proposed seems to be like an STN version of AIR, or in other way, a sequential version of STN. \n- The impact of the work seems pretty limited. STN and AIR both explored if they could apply to more realistic dataset like image classification, object detection, etc. while the proposed approach so far proved to only work on stroke-based drawing. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "y75xMWnYkWz", "reviewer": "Reviewer_dHqT", "summary": "The paper proposes a neuro-symbolic generative model for stroke based drawing. Unlike prior work in neuro-symbolic generative models, the method doesn’t require explicit supervision. It consists of two parts: 1) A generative model that models the distribution over strokes (sampled from Bezier splines) and their layouts. It is also responsible of rendering the strokes on the canvas in differentiable manner (referred as guided execution). 2) A recognition model that conditions on the target image and infers where to place the strokes. The efficacy of method is demonstrated on two sets of tasks: 1) Generalisation of stroke based reconstruction across datasets. 2) Generalisation across tasks which depict the usefulness of the representation learned.", "questions": "- What are the prior probabilities of the initial states of the variables corresponding to layout, stroke, on/off variable and image itself i.e. $p(l_0)$, $p(o_0)$, $p(s_0)$ and $p(x_0)$. Shouldn’t they be a part of Eq. 1?\n- (Related to above question) For unconditional generation, how do you sample from the sequential prior? E.g. is the initial state value is sampled from Gaussian distribution?\n- What does $\\textit{m}$ and $\\textit{k}$ correspond to in equation 2 and 3? Are they specific to GMM components weights?\n- Could Dth order Bézier splines limit the creation of certain characters/shapes? Or introduce a bias for certain strokes in generation. What is the criteria of selecting a distribution of these splines?\n- What does differentiable renderer refer to? Is it just a simple affine transformation or some intricate function of putting control points on the canvas?\n- It wasn’t clear what guided execution referred to in the paper. I had to read it again to figure out it’s the differentiable rendering. \n- The binary latent variable has some continuous relaxation? The behaviour of Bernoulli distribution (equation 4) can be random in the beginning. Does it affect the training?\n- Given the figure 1, shouldn’t the posterior be conditioned on $\\Delta x_t$, in L.H.S of equation 6?\n- Would a comparison with MWS (https://arxiv.org/pdf/2007.03132.pdf) and another part based generative model (https://arxiv.org/abs/1906.06818) make sense for generalization across datasets experiment? I would recommend to mention the second paper in the related work section.\n- Can the transfer of stroke-based reconstructions across datasets be truly called zero-shot as the recognition model have access to the image (x) from OOD dataset? The image from OOD dataset guides the reconstruction (via recognition part of the method) irrespective of whether training was explicitly done on this new dataset or not. This, in my opinion, is one-shot transfer. Curious to know what authors think about this.\n- Typo on line 36,  -> model or approaches", "limitations": "There doesn't seem to be any obvious negative societal impact of this work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is largely well written and the novelty is clear.\n- Shows impressive performance for out-of-distribution characters reconstruction, unconditional generation, conditional generation, and one-shot classification.\n\n**Weaknesses**\n\nAlthough there are no major concerns, I would appreciate if authors could address the following points.\n\n- A graphical representation of the generative model would make the generative process easier to understand.\n- The paper needs to connect certain portions better. E.g how does generative model complements recognition model or vice versa? or how does the shared hidden representations $h_{t}^{l}$ and $h_{t}^{s}$ help recognition and generation? \n- The model does not seem to be a fully generative one. I am probably confused here because of equation 1, as characters generation doesn’t seem to be conditioned on the initial states of layout, stroke and on/off variables. See my related question below.\n- I liked the transfer of stroke-based reconstructions across datasets experiments. However, I think it’s one-shot transfer instead of zero-shot as the model makes use of the single sample to guide reconstruction. See my related question below.\n- Moreover, some terms are introduced without explanations for e.g.  guided execution and bezier splines.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SXAUOYM6g8s", "reviewer": "Reviewer_uLsf", "summary": "- The paper proposes an unsupervised neuro-symbolic generative model for general-purpose pen-drawing/doodling frames as a sequential composition of strokes. \n- Analyze the model and explore its benefits in terms of interpretability and out-of-distribution generalization. \n", "questions": "- **AIR**: The first paragraph talks about decomposing to “what” and “how” etc. I think the AIR paper should be cited at this point to give it the credit of originating these ideas.\n- **Affine transformations**: Is it possible to decompose the doodles to general sub-curves that will be directly sampled relatively to the positions of the previous strokes, such that there won’t be a need to apply transformations on each of them? If I understand correctly, it may help simplify the model.\n- **Related work**: I recommend moving the related work section to the beginning of the paper after the intro rather than before the end. \n- **Figure**: In figure 3 right it’s unclear to me why it says MNIST often has 10 strokes while quickdraw has 2? Are these labels swapped or I misunderstood something? \n", "limitations": "Yea but only in the supplementary. I recommend making at least a brief version in the main paper.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- **Technical details**: The paper does a great job in describing the model in a formal, precise and detailed manner.\n- **Writing quality**: The writing quality is good and the model figure is clear and helpful.\n- **Experiments**: The paper extensively covers a variety of experiments measuring generalization across both different tasks and datasets, provides a lot of helpful visualizations and also performs ablations.\n- **Unsupervised**: The model achieves SOTA performance compared to other neuro-symbolic approaches even though it doesn’t rely on strong supervision or augmentations of the data.\n- **Out of distribution generalization**: The general-purpose framing of the approach allows it to generalize out of the training distribution (MNIST) to new characters (Japanese) and even doodles (car, bike) and to a greater number of strokes. It does better than AIR.\n- **Interpretability**: The model generates doodles in a sequential easy-to-visualize manner, decomposing them in mostly a natural manner – based on looking at the few shown examples, similar in some cases to how a person might approach the drawing task (although in other cases the decompositions are less likely). \n- **Generative capabilities**: The model can create new examples given a new doodle, and can complete a partial doodle into diverse results. \n\n**Weaknesses**\n- **Task**: The model explored by the paper is very specific to the particular task (decomposing doodles to strokes, formulating a stroke as a controlled curves that could be adjusted through spatial transformations), which is potentially a bit narrow in itself. It’s unclear to me what is the potential of the approach to scale to broader domains. It could be interesting to explore visual compositionality in a more general manner, but that’s not studied in the paper.\n- **Model complexity**: the model is quite complex, with 5 different models and subparts:  having generation and recognition models, with the former consisting of sampling the layout, then the strokes, then performing an affine transformation on them (rendering), then compositing them. I think it would be good to both add more intuitive explanation about this process, its structure, and why it is a good way to approach the task. And if there could be simplifications to the approach that would also be great. \n- **Idea presentation**: I think the presentation could be improved to highlight more the big picture, the motivation and the takeaways. A good amount of background context, related work and overall motivation are being discussed, as well as the limitations and ethical considerations.\n\nConsidering both strengths and weaknesses, I originally was in terms of scoring between 6 and 7, leaning to 7 due to technical contribution and formal presentation and to 6 due to the potentially bit narrow scope of the task, but then due to the great evaluation section I converged at 7.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AXzrvokXPXQ", "reviewer": "Reviewer_MGDL", "summary": "This paper proposes a generative model for images (drawings or characters) parameterized by sequences of strokes, and a recognition model for inferring the stroke sequence from an image, which are trained jointly by optimizing a variational lower bound.  The generative model makes use of a Bezier-spline stroke model with control points provided by a recurrent convolutional neural network, and a recurrent spatial transformer network used to frame the next stroke.  The method can be used to reconstruct images from character and drawing datasets, generalize across datasets, generate new characters (hence drawing out of distribution), and to perform one-shot character classification.", "questions": "* Given the claim that DooD stands out compared to related methods in terms of computational cost, is there any evidence to establish its computational efficiency in practice?\n* Does DooD achieve superior one-shot classification compared to other character recognition models based on static images?\n* Given that GNS (Feinman and Lake, 2021) can already 'draw out of distribution', what is the motivation for the naming of DooD?\n* Line 93: Why is the decision to stop made before composing the current stroke (rather than afterward)?  Doesn't this mean wasting the computational effort to draw the current stroke if it isn't even used?\n* Line 96: What is the justification for this Laplace noise model? Why is the scale of tha Laplace fixed? What is the range of the pixel intensities, and is the Laplace distribution appropriate given that range?  Why are the pixel intensities assumed to be conditionally independent given the composited image?  Does the addition of Laplace noise accumulate over time t?  Doesn't this mean that the variance of pixels from earlier strokes is inflated?  Why?\n* Equation 6: What is the range for the index t?\n* Supplement B.1.  The notation used here (e.g. equations 10-14) differ from the notation used in the paper.  How are the terms in equations 1-6 defined in terms of these quantities?\n* Line 119: How is beta tuned?\n", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Originality\nThis method highly resembles other approaches in the literature, e.g. splines used as drawing elements (Mihai and Hare 2013), use of variational objective (Tomczak and Welling 2018), recursive image generation (Feinman and Lake 2021), or use of spatial transformer network within a character recognition model (VHE, Hewitt et al. 2018).  The combination of recursive image generation with a variational objective was previously explored by AIR (Eslami et al. 2016), while the stroke model resembles GNS (Feinman and Lake 2021).  DooD resembles a fusion of AIR and GNS, utilizing a combination of variational objective with a stroke model, and trained on images rather than stroke sequences.  It may be the first recursive image sketching model that uses spatial transformer networks as part of its composition process.\n## Quality\nSome strengths of the paper are the evaluations on multiple datasets, including cross-dataset transfer, and variety of evaluation tasks (corresponding to the Omniglot challenge).  It is also interesting that the paper investigates the internal representation of the method compared to similar methods.  The contribution of the recurrent structure of the model is demonstrated by an ablation study.  The method qualitatively outperforms baselines in terms of quality of image reconstruction, but the difference in performance is not quantified.\n\nSome weaknesses of the paper are that many of the modeling choices are not well-motivated.  In particular, the choice to use Laplace noise which appears to accumulate over steps (see below for clarity issues) seems unusual and possibly a poor fit to the actual image distribution of human drawings.  The use of only one real baseline (AIR) seems weak, especially since AIR was not developed specifically for line drawings.\n## Clarity\nThe grammar and organization of the paper is good.  However, the details of the method are not described well.  The stroke and rendering components of the model are particularly hard to understand since key details are missing from the main text, and the notations used in the appendix are not defined in terms of the notation in the main text. I am sympathetic to the difficulty of explaining such a complicated method in the page limit provided, so I am personally OK with putting more details in the appendix.  If the clarity of the methods could be improved, I would consider raising my score, which is currently \"reject\" because of lack of reproducibility.  I am of the opinion that for the paper to be publishable, it needs to contain a complete description of the method details (either in the main text or the supplement), and not just a code submission.  That said, well-documented code would help the reproducibility of the paper even more in conjunction with a more complete description of the method.\n## Significance\nThis paper appears to present a method for modeling human line drawings and handwritten characters from static images which is claimed to have superior cross-domain generalization, computational efficiency and interpretability compared to previous approaches.  It appears that the motivation of the work is to obtain better models of handwritten characters and drawings for computer vision, rather than modeling human drawing.  The interpretability of the method does seem to be superior to AIR, and while methods not based on recursive drawing are not evaluated in terms of interpretability, it is likely that they would suffer from even poorer interpretability.  The method is evaluated against GNS, BPL, and VHE which require stroke-level data.  However, a more relevant comparison would be against one-shot image classification models that are trained on static images.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept