GENERATIVE NEURO-SYMBOLIC MODELING
Year: 2021

Abstract:
People can learn rich, general-purpose conceptual representations from only raw perceptual inputs. Current machine learning approaches fall well short of these human standards, although different modeling traditions often have complementary strengths. Symbolic models can capture the compositional and causal knowledge that enables flexible generalization, but they struggle to learn from raw inputs, relying on strong abstractions and simplifying assumptions. Neural network models can learn directly from raw data, but they struggle to capture compositional and causal structure and typically must retrain to tackle new tasks. We bring together these two traditions to learn generative models of concepts that capture rich compositional and causal structure, while learning from raw data. We develop a generative neuro-symbolic (GNS) model of handwritten character concepts that uses the control flow of a probabilistic program, coupled with symbolic stroke primitives and a symbolic image renderer, to represent the causal and compositional processes by which characters are formed. The distributions of parts (strokes), and correlations between parts, are modeled with neural network subroutines, allowing the model to learn directly from raw data and express nonparametric statistical relationships. We apply our model to the Omniglot challenge of human-level concept learning, using a background set of alphabets to learn an expressive prior distribution over character drawings. In a subsequent evaluation, our GNS model uses probabilistic inference to learn rich conceptual representations from a single training image that generalize to 4 unique tasks, succeeding where previous work has fallen short.

People can learn rich, general-purpose conceptual representations from only raw perceptual inputs. Current machine learning approaches fall well short of these human standards, although different modeling traditions often have complementary strengths. Symbolic models can capture the compositional and causal knowledge that enables flexible generalization, but they struggle to learn from raw inputs, relying on strong abstractions and simplifying assumptions. Neural network models can learn directly from raw data, but they struggle to capture compositional and causal structure and typically must retrain to tackle new tasks. We bring together these two traditions to learn generative models of concepts that capture rich compositional and causal structure, while learning from raw data. We develop a generative neuro-symbolic (GNS) model of handwritten character concepts that uses the control flow of a probabilistic program, coupled with symbolic stroke primitives and a symbolic image renderer, to represent the causal and compositional processes by which characters are formed. The distributions of parts (strokes), and correlations between parts, are modeled with neural network subroutines, allowing the model to learn directly from raw data and express nonparametric statistical relationships. We apply our model to the Omniglot challenge of human-level concept learning, using a background set of alphabets to learn an expressive prior distribution over character drawings. In a subsequent evaluation, our GNS model uses probabilistic inference to learn rich conceptual representations from a single training image that generalize to 4 unique tasks, succeeding where previous work has fallen short.
1 INTRODUCTION
Human conceptual knowledge supports many capabilities spanning perception, production and reasoning [37]. A signature of this knowledge is its productivity and generality: the internal models and representations that people develop can be applied flexibly to new tasks with little or no training experience [30]. Another distinctive characteristic of human conceptual knowledge is the way that it interacts with raw signals: people learn new concepts directly from raw, high-dimensional sensory data, and they identify instances of known concepts embedded in similarly complex stimuli. A central challenge is developing machines with these human-like conceptual capabilities.
Engineering efforts have embraced two distinct paradigms: symbolic models for capturing structured knowledge, and neural network models for capturing nonparametric statistical relationships. Symbolic models are well-suited for representing the causal and compositional processes behind perceptual observations, providing explanations akin to people’s intuitive theories [38]. Quintessential examples include accounts of concept learning as program induction [13, 46, 29, 15, 4, 28]. Symbolic programs provide a language for expressing causal and compositional structure, while probabilistic modeling offers a means of learning programs and expressing additional conceptual knowledge through priors. The Bayesian Program Learning (BPL) framework [29], for example, provides a dictionary of simple sub-part primitives for generating handwritten character concepts, and symbolic relations that specify how to combine sub-parts into parts (strokes) and parts into whole character concepts. These abstractions support inductive reasoning and flexible generalization to a range of different tasks, utilizing a single conceptual representation [29].
Published as a conference paper at ICLR 2021
HumansBPL model (centered) GNS model
Symbolic models offer many useful features, but they come with important limitations. Foremost, symbolic probabilistic models make simplifying and rigid parametric assumptions, and when the assumptions are wrong—as is common in complex, high-dimensional data—they create bias [11]. The BPL character model, for example, assumes that parts are largely independent a priori, an assumption that is not reflective of real human-drawn characters. As a consequence, characters generated from the raw BPL prior lack the complexity of real characters (Fig 1, left), even though the posterior samples can appear much more structured. Another limitation of symbolic probabilistic models is that the construction of structured hypothesis spaces requires significant domain knowledge [2]. Humans, meanwhile, build rich internal models directly from raw data, forming hypotheses about the conceptual features and the generative syntax of a domain. As one potential resolution, previous work has demonstrated that the selection of structured hypotheses can itself be attributed to learning in a Bayesian framework [47, 13, 14, 41, 24, 40]. Although more flexible than a priori structural decisions, models of this kind still make many assumptions, and they have not yet tackled the types of raw, high-dimensional stimuli that are distinctive of the neural network approach.
The second paradigm, neural network modeling, prioritizes powerful nonparametric statistical learning over structured representations. This modeling tradition emphasizes emergence, the idea that conceptual knowledge arises from interactions of distributed sub-symbolic processes [36, 32]. Neural networks are adept at learning from raw data and capturing complex patterns. However, they can struggle to learn the compositional and causal structure in how concepts are formed [30]; even when this structure is salient in the data, they may have no obvious means of incorporating it. These limitations have been linked to shortcomings in systematic generalization [35, 27] and creative abilities [31]. An illustrative example is the Omniglot challenge: in 4 years of active research, neural network models do not yet explain how people quickly grasp new concepts and use them in a variety of ways, even with relatively simple handwritten characters [31]. Surveying over 10 neural models applied to Omniglot, Lake et al. [31] found that only two attempted both classification and generation tasks, and they were each outperformed by the fully-symbolic, probabilistic BPL. Moreover, neural generative models tended to produce characters with anomalous characteristics, highlighting their shortcomings in modeling causal and compositional structure (see Fig. A13 and [31, Fig. 2a]).
In this paper, we introduce a new approach that leverages the strengths of both the symbolic and neural network paradigms by representing concepts as probabilistic programs with neural network subroutines. We describe an instance of this approach developed for the Omniglot challenge [29] of task-general representation learning and discuss how we see our Omniglot model fitting into a broader class of Generative Neuro-Symbolic (GNS) models that seek to capture the data-generation process. As with traditional probabilistic programs, the control flow of a GNS program is an explicit representation of the causal generative process that produces new concepts and new exemplars. Moreover, explicit re-use of parts through repeated calls to procedures such as GeneratePart (Fig. 2) ensures a representation that is compositional, providing an appropriate inductive bias for compositional generalization. Unlike fully-symbolic probabilistic programs, however, the distribution of parts and correlations between parts in GNS are modeled with neural networks. This architectural choice allows the model to learn directly from raw data, capturing nonparametric statistics while requiring only minimal prior knowledge.
Published as a conference paper at ICLR 2021
We develop a GNS model for the Omniglot challenge of learning flexible, task-general representations of handwritten characters. We report results on 4 Omniglot challenge tasks with a single model: 1) one-shot classification, 2) parsing/segmentation, 3) generating new exemplars, and 4) generating new concepts (without constraints); the 5th and final task of generating new concepts (from type) is left for future work. We also provide log-likelihood evaluations of the generative model. Notably, our goal is not to chase state-of-the-art performance on one task across many datasets (e.g., classification). Instead we build a model that learns deep, task-general knowledge within a single domain and evaluate it on a range of different tasks. This “deep expertise” is arguably just as important as “broad expertise” in characterizing human-level concept learning [37, 31]; machines that seek human-like abilities will need both. Our work here is one proposal for how neurally-grounded approaches can move beyond pattern recognition toward the more flexible model-building abilities needed for deep expertise [30]. In Appendix E, we discuss how to extend GNS to another conceptual domain, providing a proposal for a GNS model of 3D object concepts.
2 RELATED WORK
The Omniglot dataset and challenge has been widely adopted in machine learning, with models such as Matching Nets [49], MAML [7], and ARC [44] applied to just one-shot classification, and others such as DRAW [19], SPIRAL [8], and VHE [21] applied to one or more generative tasks. In their “3-year progress report,” Lake et al. [31] reviewed the current progress on Omniglot, finding that although there was considerable progress in one-shot classification, there had been little emphasis placed on developing task-general models to match the flexibility of human learners (Table 1). Moreover, neurally-grounded models that attempt more creative generation tasks were shown to produce characters that either closely mimicked the training examples or that exhibited anomalous variations, making for easy identification from humans (see Fig. A13 and [31, Fig. 2a]). Our goal is distinct in that we aim to learn a single neuro-symbolic generative model that can perform a variety of unique tasks, and that generates novel yet structured new characters.
Neuro-symbolic modeling has become an active area of research, with applications to learning input-output programs [42, 17, 3, 39, 48], question answering [50, 34] and image description [26, 4]. GNS modeling distinguishes itself from prior work through its focus on hybrid generative modeling, combining both structured program execution and neural networks directly in the probabilistic generative process. Neuro-symbolic VQA models [50, 34] are neither generative nor task-general; they are trained discriminatively to answer questions. Other neuro-symbolic systems use neural networks to help perform inference in a fully-symbolic generative model [26, 4], or to parameterize a prior over fully-symbolic hypotheses [22]. In order to capture the dual structural and statistical characteristics of human conceptual representations, we find it important to include neural nets directly in the forward generative model. As applied to Omniglot, our model bears some resemblance to SPIRAL [8]; however, SPIRAL does not provide a density function, and it has no hierarchical structure, limiting its applications to image reconstruction and unconditional generation.
Another class of models on the neuro-symbolic spectrum aims to learn “object representations” with neural networks [5, 18, 25], which add minimal object-like symbols to support systematic reasoning and generalization. Although these models have demonstrated promising results in applications such as scene segmentation and unconditional generation, they have not yet demonstrated the type of rich inductive capabilities that we are after: namely, the ability to learn “deep” conceptual representations from just one or a few examples that support a variety of discriminative and generative tasks.
Other works (e.g. [16, 20]) have used autoregressive models like ours with similar stroke primitives to model the causal generative processes of handwriting. We develop a novel architecture for generating
Published as a conference paper at ICLR 2021
type level
token level
procedure GENERATETYPE C 0 . Initialize blank image canvas while true do
[yi, xi] GENERATEPART(C) . Sample part location & parameters C frender(yi, xi, C) . Render part to image canvas vi ⇠ p(v | C) . Sample termination indicator if vi then
break . Terminate sample {, y1:, x1:} return . Return concept type
1
C Canvas
yi, xi Part
Image I
procedure GENERATETYPE C 0 . Initialize blank image canvas while true do
[yi, xi] GENERATEPART(C) . Sample part location & parameters C frender(yi, xi, C) . Render part to image canvas vi ⇠ p(v | C) . Sample termination indicator if vi then
break . Terminate sample {, y1:, x1:} return . Return concept type
1
pr oc
ed ur
e G
EN ER
AT ET
YP E
C
0
. In
itia liz
e bla nk im ag e ca nv as
wh ile
tr ue
do
[y i , x
i ]
G EN
ER AT
EP AR
T( C ) .
Sa m
ple pa
rt loc
at ion
& pa ra m et er s
C
f re nd
er (y
i , x
i , C
)
. Re
nd er
pa rt
to im
ag e ca nv as
v i ⇠
p( v | C
)
. Sa
m ple
te rm
ina tio
n ind ica to r
if v i
th en
br ea
k
. Te
rm ina
te sa
m ple
{ , y
1:  , x
1:  }
re tu
rn
. Re
tu rn
co nc
ep t t
yp e
1
procedure GENERATETYPE C 0 . Initialize blank image canvas while true do
[yi, xi] GENERATEPART(C) . Sample part location & parameters C frender(yi, xi, C) . Render part to image canvas vi ⇠ p(v | C) . Sample termination indicator if vi then
break . Terminate sample {, y1:, x1:} return . Return concept type
1
location model p(y ∣ C)
CNN MLP
stroke model p(x ∣ y, C)
CNN LSTM
y
C
attention
p(y)
p(Δ1)
p(Δ2 ∣ Δ1)
p(ΔT ∣ Δ1:T−1) …C
Figure 2: A generative neuro-symbolic (GNS) model of character concepts. The type model GenerateType (P (ψ)) produces character types one stroke at a time, using an image canvas C as memory. At each step, the current canvas C is fed to procedure GeneratePart and a stroke sample is produced. The canvas is first processed by the location model, a CNN-MLP architecture that samples starting location y, and next by the stroke model, a CNN-LSTM architecture that samples trajectory x while attending to the encoded canvas. Finally, a symbolic renderer updates the canvas according to x and y, and a termination model decides whether to terminate the type sample. Unique exemplars are produced from a character type by sampling from the token model conditioned on ψ, adding motor noise to the drawing parameters and performing a random affine transformation.
handwriting, which represents explicit compositional structure by modeling parts and relations with separate modules and applying intermediate symbolic rendering. Most importantly, these prior models have not made a connection to the image; therefore while they can generate handwriting as symbolic coordinates, they cannot explain how people use their causal knowledge to learn new characters from visual presentations, how they infer the strokes of a character seen on paper, or how they generate a new example of an observed character. By combining a powerful autoregressive model of handwriting with a symbolic image renderer and algorithms for probabilistic inference, we seek to replicate a spectrum of unique human concept learning abilities.
3 GENERATIVE MODEL
Our GNS model leverages the type-token hierarchy of BPL [29], which offers a useful scaffolding for conceptual models (Fig. 2, left).1 The type-level model P (ψ) defines a prior distribution over character concepts, capturing overarching principles and regularities that tie together characters from different alphabets and providing a procedure to generate new character concepts unconditionally in latent stroke format (Fig. 2, right). A token-level model p(θ|ψ) captures the within-class variability that arises from motor noise and drawing styles, and an image distribution P (I|θ) provides an explicit model of how causal stroke actions translate to image pixels. All parameters of our model are learned from the Omniglot background set of drawings (Appendix A). The full joint distribution over type ψ, token θ(m) and image I(m) factors as
P (ψ, θ(m), I(m)) = P (ψ)P (θ(m)|ψ)P (I(m)|θ(m)). (1)
Although sharing a common hierarchy, the implementation details of each level in our GNS model differ from BPL in critical ways. The GNS type prior P (ψ) is a highly expressive generative model that uses an external image canvas, coupled with a symbolic rendering engine and an attentive
1Aspects of our model were recently published in a non-archival conference proceedings [6]. The previous manuscript presents only a prior distribution that alone performs just one task (generating new concepts); our new developments include a full hierarchical model, a differentiable image renderer / likelihood, and a procedure for approximate probabilistic inference from image data. These ingredients together enable GNS to perform 4 unique conceptual tasks.
Published as a conference paper at ICLR 2021
recurrent neural network, to condition future parts on the previous parts and model sophisticated causal and correlational structure. This structure is essential to generating new character concepts in realistic, human-like ways (Sec. 5). Moreover, whereas the BPL model is provided symbolic relations for strokes such as “attach start” and “attach along,” GNS learns implicit relational structure from the data, identifying salient patterns in the co-occurrences of parts and locations. Importantly, the GNS generative model is designed to be differentiable at all levels, yielding log-likelihood gradients that enable powerful new inference algorithms (Sec. 4) and estimates of marginal image likelihood (Sec. 5).
Type prior. The type prior P (ψ) is captured by a neuro-symbolic generative model of character drawings. The model represents a character as a sequence of strokes (parts), with each stroke i decomposed into a starting location yi ∈ R2 and a variable-length trajectory xi ∈ Rdi×2. Rather than use raw pen trajectories as our stroke format, we use a minimal spline representation of strokes, obtained from raw trajectories by fitting cubic b-splines with a residual threshold. The starting location yi therefore conveys the first spline control point, and trajectory xi = {∆i1, ...,∆idi} conveys the offsets between subsequent points of a (di+1)-length spline. These offsets are transformed into a sequence of relative points xi = {xi1, ..., xidi+1}, with xi1 = 0, specifying locations relative to yi. The model samples a type one stroke at a time, using an image canvas C as memory to convey the sample state. At each step, a starting location for the next stroke is first sampled from the location model, followed by a trajectory from the stroke model. The stroke is then rendered to the canvas C, and a termination model decides whether to terminate or continue the sample. Each of the three model components is expressed by a neural network, using a LSTM as the stroke model to generate trajectories as in [16]. The details of these neural modules are provided in Appendix A. The type model P (ψ) specifies an auto-regressive density function that can evaluate exact likelihoods of character drawings, and its hyperparameters (the three neural networks) are learned from the Omniglot background set of 30 alphabets using a maximum likelihood objective. A full character type ψ includes the random variables ψ = {κ, y1:κ, x1:κ}, where κ ∈ Z+ is the number of strokes. The density function P (ψ) is also fully differentiable w.r.t. the continuous random variables in ψ.
Token model. A character token θ(m) = {y(m)1:κ , x (m) 1:κ , A (m)} represents a unique instance of a character concept, where y(m)1:κ are the token-level locations, x (m) 1:κ the token-level parts, and A(m) ∈ R4 the parameters of an affine warp transformation. The token distribution factorizes as
P (θ(m)|ψ) = P (A(m)) κ∏ i=1 P (y (m) i | yi)P (x (m) i | xi). (2)
Here, P (y(m)i | yi) represents a simple noise distribution for the location of each stroke, and P (x
(m) i | xi) for the stroke trajectory. The first two dimensions of affine warp A(m) control a global re-scaling of the token drawing, and the second two a global translation of its center of mass. The distributions and pseudocode of our token model are given in Appendix A.
Image model. The image model P (I(m) | θ(m)) is based on [29] and is composed of two pieces. First, a differentiable symbolic engine f receives the token θ(m) and produces an image pixel probability map pimg = f(θ(m), σ, ) by evaluating each spline and rendering the stroke trajectories. Here, σ ∈ R+ is a parameter controlling the rendering blur around stroke coordinates, and ∈ (0, 1) controlling pixel noise, each sampled uniformly at random. The result then parameterizes an image distribution P (I(m) | θ(m)) = Bernoulli(pimg), which is differentiable w.r.t. θ(m), σ, and .
4 PROBABILISTIC INFERENCE
Given an image I of a novel concept, our GNS model aims to infer the latent causal, compositional process for generating new exemplars. We follow the high-level strategy of BPL for constructing a discrete approximation Q(ψ, θ | I) to the desired posterior distribution [29],
P (ψ, θ | I) ≈ Q(ψ, θ | I) = K∑ k=1 πkδ(θ − θk)δ(ψ − ψk). (3)
A heuristic search algorithm is used to find K good parses, {ψ, θ}1:K , that explain the underlying image with high probability. These parses are weighted by their relative posterior probability,
Published as a conference paper at ICLR 2021
log P(I(T ) ∣ I(c)) = − 401.3
Correct match
train
test
log P(I(T ) ∣ I(c)) = − 664.8
Incorrect match
train
test
(a) Classification fits
the pen (Fig. 3A, ii). To construct a new character type, first themodel samples the number of parts k and the number of subparts ni, for each part i = 1, ..., k, from their empirical distributions as measured from the background set. Second, a template for a part Si is constructed by sampling subparts from a set of discrete primitive actions learned from the background set (Fig. 3A, i), such that the probability of the next action depends on the previous. Third, parts are then grounded as parameterized curves (splines) by sampling the control points and scale parameters
1334 11 DECEMBER 2015 • VOL 350 ISSUE 6266 sciencemag.org SCIENCE
Fig. 3. A generative model of handwritten characters. (A) New types are generated by choosing primitive actions (color coded) from a library (i), combining these subparts (ii) to make parts (iii), and combining parts with relations to define simple programs (iv). New tokens are generated by running these programs (v), which are then rendered as raw data (vi). (B) Pseudocode for generating new types y and new token images I(m) for m = 1, ..., M. The function f (·, ·) transforms a subpart sequence and start location into a trajectory.
Human parses Machine parsesHuman drawings
-505 -593 -655 -695 -723
-1794-646 -1276
Training item with model’s five best parses
Test items
1 2 3 4 5stroke order:
Fig. 4. Inferringmotor programs from images. Parts are distinguished by color, with a colored dot indicating the beginning of a stroke and an arrowhead indicating the end. (A) The top row shows the five best programs discovered for an image along with their log-probability scores (Eq. 1). Subpart breaks are shown as black dots. For classification, each program was refit to three new test images (left in image triplets), and the best-fitting parse (top right) is shown with its image reconstruction (bottom right) and classification score (log posterior predictive probability). The correctly matching test item receives a much higher classification score and is also more cleanly reconstructed by the best programs induced from the training item. (B) Nine human drawings of three characters (left) are shown with their ground truth parses (middle) and best model parses (right).
RESEARCH | RESEARCH ARTICLES
on June 1, 2020
http://science.sciencem ag.org/
D ow
nloaded from
the pen (Fig. 3A, ii). To construct a new character type, first themodel samples the number of parts k and the number of subparts ni, for each part i = 1, ..., k, from their empirical distributions as measured from the background set. Second, a template for a part Si is constructed by sampling subparts from a set of discrete primitive actions learned from the background set (Fig. 3A, i), such that the probability of the next action depends on the previous. Third, parts are then grounded as parameterized curves (splines) by sampling the control points and scale parameters
1334 11 DECEMBER 2015 • VOL 350 ISSUE 6266 sciencemag.org SCIENCE
Fig. 3. A generative model of handwritten characters. (A) New types are generated by choosing primitive actions (color coded) from a library (i), combining these subparts (ii) to make parts (iii), and combining parts with relations to define simple programs (iv). New tokens are generated by running these programs (v), which are then rendered as raw data (vi). (B) Pseudocode for generating new types y and new token images I(m) for m = 1, ..., M. The function f (·, ·) transforms a subpart sequence and start location into a trajectory.
Human parses Machine parsesHuman drawings
-505 -593 -655 -695 -723
-1794-646 -1276
Training item with model’s five best parses
Test items
1 2 3 4 5stroke order:
Fig. 4. Inferringmotor programs from images. Parts are distinguished by color, with a colored dot indicating the beginning of a stroke and an arrowhead indicating the end. (A) The top row shows the five best programs discovered for an image along with their log-probability scores (Eq. 1). Subpart breaks are shown as black dots. For classification, each program was refit to three new test images (left in image triplets), and the best-fitting parse (top right) is shown with its image reconstruction (bottom right) and classification score (log posterior predictive probability). The correctly matching test item receives a much higher classification score and is also more cleanly reconstructed by the best programs induced from the training item. (B) Nine human drawings of three characters (left) are shown with their ground truth parses (middle) and best model parses (right).
RESEARCH | RESEARCH ARTICLES
on June 1, 2020
http://science.sciencem ag.org/
D ow nloaded from the pen (Fig. 3A, ii). To construct a new character type, first themodel samples the number of parts k and the nu ber of subparts ni, for each part i = 1, ..., k, fro their empirical distributions as measured from the background set. Second, a template for a part Si is constructed by sampling subparts from a set of discrete primitive actions learned from the background set (Fig. 3A, i), such that the probability of the next action depends on the previous. Third, parts are then grounded as parameterized curves (splines) by sampling the control points and scale parameters
1334 11 DECEMBER 2015 • VOL 350 ISSUE 6266 sciencemag.org SCIENCE
Fig. 3. A generative model of handwritten characters. (A) New types are generated by choosing primitive actions (color coded) from a library (i), combining these subparts (ii) to make parts (iii), and combining parts with relations to define simple programs (iv). New tokens are generated by running these programs (v), which are then rendered as raw data (vi). (B) Pseudocode for generating new types y and new token images I(m) for m = 1, ..., M. The function f (·, ·) transforms a subpart sequence and start location into a trajectory.
Human parses Machine parsesHuman drawings
-505 -593 -655 -695 -723
-1794-646 -1276
Training item with model’s five best parses
Test items
1 2 3 4 5stroke order:
Fig. 4. Inferringmotor programs from images. Parts are distinguished by color, with a colored dot indicating the beginning of a stroke and an arrowhead indicating the end. (A) The top row shows the five best programs discovered for an image along with their l g-probability cores (Eq. 1). Subpart breaks are shown as black dots. For classification, each program was refit to three new test images (left in image triplets), and the best-fitting parse (top right) is shown with its image reconstruction (bottom right) and classification score (l g posterior predictiv probability). The correctly matching tes item ceives a much higher classification score and is also more cleanly reconstructed by the best program induced from the training item. (B) Nine human rawings of three characters (left) are shown with their grou d tr th parses (middle) and best model parses (right).
RESEARCH | RESEARCH ARTICLES
on June 1, 2020
http://science.sciencem ag.org/
D ow nloaded from
Human drawings Human parse GNS rses BPL parse
the pen (Fig. 3A, ii). To construct a new character type, first themodel samples the number of parts k and the number of subparts ni, for each part i = 1, ..., k, from their empirical distributions as measured from the background set. Second, a template for a part Si is constructed by sampling subparts from a set of discrete primitive actions learned from the background set (Fig. 3A, i), such that the probability of the next action depends on the previous. Third, parts are then grounded as parameterized curves (splines) by sampling the control points and scale parameters
1334 11 DECEMBER 2015 • VOL 350 ISSUE 6266 sciencemag.org SCIENCE
Fig. 3. A generative model of handwritten characters. (A) New types re generated by choosing primitive actions (color coded) from a library (i), combining these subparts (ii) to make parts (iii), an combining parts with relations t de ine simple programs (iv). New tokens are generated by running these programs (v), which are then rendered as raw dat (vi). (B) Pseudocode for genera ing new types y and new token images I(m) for m = 1, ..., M. The function f (·, ·) transforms a subpart sequence and start location into a trajectory.
Human parses Machine parsesHuman drawings
-505 -593 -655 -695 -723
-1794-646 -1276
Training item with model’s five best parses
Test items
1 2 3 4 5stroke order:
Fig. 4. Inferringmotor programs from images. Parts are distinguished by color, with a colored dot indicating the beginning of a stroke and an arrowhead indicating the end. (A) The top row shows the five best programs discovered for an im ge along with their log-probability scores (Eq. 1). Subpart breaks are hown as black dots. For classification, each program was refit to three new test images (left in image triplets), and the best-fitting parse (top right) is shown with its image reconstruction (bottom right) and classification score (log posterior predictive probability). The correctly matchi g test item receives a much higher classification score and is also more cleanly reconstructed by th best programs indu ed from the raining item. (B) Nine human drawings of three cha acters (left) are shown with their ground tr th parses (middle) an b st model parses (right).
RESEARCH | RESEARCH ARTICLES
on June 1, 2020
http://science.sciencem ag.org/
D ow
nloaded from
(b) Parsing
Figure 3: Classification fits and parsing. (a) Posterior parses from two training images were refit to the same test image. The first row of each grid shows the training image and its top-3 predicted parses (best emboldened). The second row shows the test image and its re-fitted training parses. Reconstructed test images are shown in the final row. The correct training image reports a high forward score, indicating that I(T ) is well-explained by the motor programs for this I(c). (b) 27 character images from 3 classes are shown alongside their ground truth human parses, predicted parses from the GNS model, and predicted parses from the BPL model.
πk ∝ π̃k = P (ψk, θk, I) such that ∑ k πk = 1. To find the K good parses, search uses fast bottomup methods to propose many variants of the discrete variables, filtering the most promising options, before optimizing the continuous variables with gradient descent (we use K = 5). See Appendix B for further details about inference, and Appendix C for generating new exemplars of a concept.
Inference for one-shot classification. In one-shot classification, models are given a single training image I(c) from each of c = 1, ..., C classes, and asked to classify test images according to their corresponding training classes. For each test image I(T ), we compute an approximation of the Bayesian score log P (I(T ) | I(c)) for every example I(c), using our posterior parses {ψ, θ(c)}1:K and corresponding weights π1:K from I(c) (Eq. 3). The approximation is formulated as
log P (I(T ) | I(c)) ≈ log ∫ P (I(T )|θ(T ))P (θ(T ) | ψ)Q(ψ, θ(c), | I(c))∂ψ∂θ(c)∂θ(T )
≈ log K∑ k=1 πk max θ(T ) P (I(T ) | θ(T ))P (θ(T ) | ψk), (4)
where the maximum over θ(T ) is determined by refitting token-level parameters θ(c) to image I(T ) with gradient descent. Following Lake et al. [29], we use a two-way version of the Bayesian score that also considers parses of I(T ) refit to I(c). The classification rule is therefore
c∗ = arg max c log P (I(T ) | I(c))2 = arg max c
log [P (I(c) | I(T ))
P (I(c)) P (I(T ) | I(c))
] , (5)
where P (I(c)) ≈∑k π̃k is approximated from the unnormalized weights of I(c) parses. 5 EXPERIMENTS
GNS was evaluated on four concept learning tasks from the Omniglot challenge [31]: one-shot classification, parsing, generating new exemplars, and generating new concepts. All evaluations use novel characters from completely held-out alphabets in the Omniglot evaluation set. As mentioned
Published as a conference paper at ICLR 2021
earlier, our goal is to provide a single model that captures deep knowledge of a domain and performs strongly in a wide range of tasks, rather than besting all models on every task. Our experiments include a mixture of quantitative and qualitative evaluations, depending on the nature of the task.
One-shot classification. GNS was compared with alternative models on the one-shot classification task from Lake et al. [29]. The task involves a series of 20-way within-alphabet classification episodes, with each episode proceeding as follows. First, the machine is given one training example from each of 20 novel characters. Next, the machine must classify 20 novel test images, each corresponding to one of the training classes. With 20 episodes total, the task yields 400 trials. Importantly, all character classes in an episode come from the same alphabet as originally proposed [29], requiring finer discriminations than commonly used between-alphabet tests [31].
Table 2: Test error on withinalphabet one-shot classification.
Model Error
GNS 5.7% BPL [29] 3.3% RCN [12] 7.3% VHE [21] 18.7% Proto. Net [45] 13.7% ARC [44] 1.5%∗
*used 4x training classes As illustrated in Fig. 3a, GNS classifies a test image by choosing the training class with the highest Bayesian score (Eq. 5). A summary of the results is shown in Table 2. GNS was compared with other machine learning models that have been evaluated on the withinalphabets classification task [31]. GNS achieved an overall test error rate of 5.7% across all 20 episodes (N=400). This result is very close to the original BPL model, which achieved 3.3% error with significantly more hand-design. The symbolic relations in BPL’s token model provide rigid constraints that are key to its strong classification performance [29]. GNS achieves strong classification performance while emphasizing the nonparametric statistical knowledge needed for creative generation in subsequent tasks. Beyond BPL, our GNS model outperformed all other models that received the same background training. The ARC model [44] achieved an impressive 1.5% error, although it was trained with four-fold class augmentation and many other augmentations, and it can only perform this one task. In Appendix Fig. A10, we show a larger set of classification fits from GNS, including examples of misclassified trials.
Parsing. In the Omniglot parsing task, machines must segment a novel character into an ordered set of strokes. These predicted parses can be compared with human ground-truth parses for the same images. The approximate posterior of GNS yields a series of promising parses for a new character image, and to complete the parsing task, we identify the maximum a posteriori parse k∗ = maxk πk, reporting the corresponding stroke configuration. Fig. 3b shows a visualization of the GNS predicted parses for 27 different raw images drawn from 3 unique character classes, plotted alongside groundtruth human parses (how the images were actually drawn) along with predicted parses from the BPL model. Compared to BPL, GNS parses possess a few unique desirable qualities. The first character class has an obvious segmentation to the human eye—evidenced by the consistency of human parses in all examples—and the GNS model replicates this consistency across all 9 predicted parses. In contrast, BPL predicts seemingly-unlikely parses for 2 of the examples shown. The second character is more complex, and it was drawn in different styles by different human subjects. The GNS model, which is trained on data from subjects with different styles, captures the uncertainty in this character by predicting a variety of unique parses. BPL, on the other hand, produces a single, ubiquitous segmentation across all 9 examples. In Appendix Fig. A11, we provide a larger set of parses from the GNS model for a diverse range of Omniglot characters.
Generating new exemplars. Given just one training image of a novel character concept, GNS produces new exemplars of the concept by sampling from the approximate conditional P (I(2), θ(2) | I(1)) of Eq. 8. In Fig. 4a we show new exemplars produced by GNS for a handful of target images, plotted next to human productions (more examples in Appendix Fig. A12). In the majority of cases, samples from the model demonstrate that it has successfully captured the causal structure and invariance of the target class. In contrast, deep generative models applied to the same task miss meaningful compositional and causal structure, producing new examples that are easily discriminated from human productions [43, 21] (see Appendix Fig. A13). In some cases, such as the third column of Fig. 4a, samples from GNS exhibit sloppy stroke junctions and connections. Compared to BPL, which uses engineered symbolic relations to enforce rigid constraints at stroke junctions, GNS misses some of these structural elements. Nevertheless, new examples from GNS appear strong enough to pass for human in many cases, which we would like to test in future work with visual Turing tests.
Concept learning experiments can be reproduced using our pre-trained generative model and source code: https://github.com/rfeinman/GNS-Modeling.
Published as a conference paper at ICLR 2021
Human
GNS
Target
for each subpart. Last, parts are roughly positioned to begin either independently, at the beginning, at the end, or along previous parts, as defined by relation Ri (Fig. 3A, iv). Character tokens q(m) are produced by executing the parts and the relations andmodeling how ink flows from the pen to the page. First, motor noise is added to the control points and the scale of the subparts to create token-level stroke trajectories S(m). Second, the trajectory’s precise start location L(m) is sampled from the schematic provided by its relationRi to previous strokes. Third, global transformations are sampled, including an affine warp A(m) and adaptive noise parameters that ease probabilistic inference (30). Last, a binary image I (m) is created by a stochastic rendering function, lining the stroke trajectories with grayscale ink and interpreting the pixel values as independent Bernoulli probabilities. Posterior inference requires searching the large
combinatorial space of programs that could have generated a raw image I (m). Our strategy uses fast bottom-up methods (31) to propose a range of candidate parses. The most promising candidates are refined by using continuous optimization
and local search, forming a discrete approximation to the posterior distribution P(y , q(m)|I (m)) (section S3). Figure 4A shows the set of discovered programs for a training image I (1) and how they are refit to different test images I (2) to compute a classification score log P(I (2)|I (1)) (the log posterior predictive probability), where higher scores indicate that they are more likely to belong to the same class. A high score is achieved when at least one set of parts and relations can successfully explain both the training and the test images, without violating the soft constraints of the learned within-class variability model. Figure 4B compares the model’s best-scoring parses with the ground-truth human parses for several characters. Results
People, BPL, and alternative models were compared side by side on five concept learning tasks that examine different forms of generalization from just one or a few examples (example task Fig. 5). All behavioral experiments were run through Amazon’s Mechanical Turk, and the experimental procedures are detailed in section S5.
The main results are summarized by Fig. 6, and additional lesion analyses and controls are reported in section S6. One-shot classification was evaluated through a series of within-alphabet classification tasks for 10 different alphabets. As illustrated in Fig. 1B, i, a single image of a new character was presented, and participants selected another example of that same character from a set of 20 distinct characters produced by a typical drawer of that alphabet. Performance is shown in Fig. 6A,where chance is 95% errors. As a baseline, themodifiedHausdorff distance (32) was computed between centered images, producing 38.8% errors. People were skilled one-shot learners, achieving an average error rate of 4.5% (N = 40). BPL showed a similar error rate of 3.3%, achieving better performance than adeep convolutional network (convnet; 13.5% errors) and the HDmodel (34.8%)—each adapted from deep learning methods that have performed well on a range of computer vision tasks. A deep Siamese convolutional network optimized for this one-shot learning task achieved 8.0% errors (33), still about twice as high as humans or ourmodel. BPL’s advantage points to the benefits ofmodeling theunderlying causal process in learning concepts, a strategy different from the particular deep learning approaches examined here. BPL’s other key ingredients also make positive contributions, as shown by higher error rates for BPL lesions without learning to learn (token-level only) or compositionality (11.0% errors and 14.0%, respectively). Learning to learn was studied separately at the type and token level by disrupting the learned hyperparameters of the generativemodel. Compositionality was evaluated by comparing BPL to a matched model that allowed just one spline-based stroke, resembling earlier analysisby-synthesis models for handwritten characters that were similarly limited (34, 35). The human capacity for one-shot learning is more than just classification. It can include a suite of abilities, such as generating new examples of a concept. We compared the creative outputs produced by humans and machines through “visual Turing tests,”where naive human judges tried to identify the machine, given paired examples of human and machine behavior. In our most basic task, judges compared the drawings from nine humans asked to produce a new instance of a concept given one example with nine new examples drawn by BPL (Fig. 5). We evaluated each model based on the accuracy of the judges, which we call their identification (ID) level: Idealmodel performance is 50% ID level, indicating that they cannot distinguish the model’s behavior from humans; worst-case performance is 100%. Each judge (N = 147) completed 49 trials with blocked feedback, and judges were analyzed individually and in aggregate. The results are shown in Fig. 6B (new exemplars). Judges had only a 52% ID level on average for discriminating human versus BPL behavior. As a group, this performance was barely better than chance [t(47) = 2.03, P = 0.048], and only 3 of 48 judges had an ID level reliably above chance. Three lesioned models were evaluated by different groups of judges in separate
SCIENCE sciencemag.org 11 DECEMBER 2015 • VOL 350 ISSUE 6266 1335
1 2
1 2
1 2
1 2
1 2
1 2
Human or Machine?
Fig. 5. Generating new exemplars. Humans and machines were given an image of a novel character (top) and asked to produce new exemplars.The nine-character grids in each pair that were generated by a machine are (by row) 1, 2; 2, 1; 1, 1.
RESEARCH | RESEARCH ARTICLES
on N ovem ber 21, 2020
http://science.sciencem ag.org/
D ow
nloaded from
for each subpart. Last, parts are roughly positioned to begin either independently, at the beginning, at the end, or along previous parts, as defined by relation Ri (Fig. 3A, iv). Character tokens q(m) are produced by execut-
ing the parts and the relations andmodeling how ink flows from the pen to the page. First, motor noise is added to the control points and the scale of the subparts to create token-level stroke trajectories S(m). Second, the trajectory’s precise start location L(m) is sampled from the schematic provided by its relationRi to previous strokes. Third, global transformations are sampled, including an affine warp A(m) and adaptive noise parameters that ease probabilistic inference (30). Last, a binary image I (m) is created by a stochastic rendering function, lining the stroke trajectories with grayscale ink and interpreting the pixel values as independent Bernoulli probabilities. Posterior inference requires searching the large combinatorial space of programs that could have generated a raw image I (m). Our strategy uses fast bottom-up methods (31) to propose a range of candidate parses. The most promising candidates are refined by using continuous optimization
and local search, forming a discrete approximation to the posterior distribution P(y , q(m)|I (m)) (section S3). Figure 4A shows the set of discovered programs for a training image I (1) and how they are refit to different test images I (2) to compute a classification score log P(I (2)|I (1)) (the log posterior predictive probability), where higher scores indicate that they are more likely to belong to the same class. A high score is achieved when at least one set of parts and relations can successfully explain both the training and the test images, without violating the soft constraints of the learned within-class variability model. Figure 4B compares the model’s best-scoring parses with the ground-truth human parses for several characters.
Results
People, BPL, and alternative models were compared side by side on five concept learning tasks that examine different forms of generalization from just one or a few examples (example task Fig. 5). All behavioral experiments were run through Amazon’s Mechanical Turk, and the experimental procedures are detailed in section S5.
The main results are summarized by Fig. 6, and additional lesion analyses and controls are reported in section S6. One-shot classification was evaluated through
a series of within-alphabet classification tasks for 10 different alphabets. As illustrated in Fig. 1B, i, a single image of a new character was presented, and participants selected another example of that same character from a set of 20 distinct characters produced by a typical drawer of that alphabet. Performance is shown in Fig. 6A,where chance is 95% errors. As a baseline, themodifiedHausdorff distance (32) was computed between centered images, producing 38.8% errors. People were skilled one-shot learners, achieving an average error rate of 4.5% (N = 40). BPL showed a similar error rate of 3.3%, achieving better performance than adeep convolutional network (convnet; 13.5% errors) and the HDmodel (34.8%)—each adapted from deep learning methods that have performed well on a range of computer vision tasks. A deep Siamese convolutional network optimized for this one-shot learning task achieved 8.0% errors (33), still about twice as high as humans or ourmodel. BPL’s advantage points to the benefits ofmodeling theunderlying causal process in learning concepts, a strategy different from the particular deep learning approaches examined here. BPL’s other key ingredients also make positive contributions, as shown by higher error rates for BPL lesions without learning to learn (token-level only) or compositionality (11.0% errors and 14.0%, respectively). Learning to learn was studied separately at the type and token level by disrupting the learned hyperparameters of the generativemodel. Compositionality was evaluated by comparing BPL to a matched model that allowed just one spline-based stroke, resembling earlier analysisby-synthesis models for handwritten characters that were similarly limited (34, 35). The human capacity for one-shot learning is more than just classification. It can include a suite of abilities, such as generating new examples of a concept. We compared the creative outputs produced by humans and machines through “visual Turing tests,”where naive human judges tried to identify the machine, given paired examples of human and machine behavior. In our most basic task, judges compared the drawings from nine humans asked to produce a new instance of a concept given one example with nine new examples drawn by BPL (Fig. 5). We evaluated each model based on the accuracy of the judges, which we call their identification (ID) level: Idealmodel performance is 50% ID level, indicating that they cannot distinguish the model’s behavior from humans; worst-case performance is 100%. Each judge (N = 147) completed 49 trials with blocked feedback, and judges were analyzed individually and in aggregate. The results are shown in Fig. 6B (new exemplars). Judges had only a 52% ID level on average for discriminating human versus BPL behavior. As a group, this performance was barely better than chance [t(47) = 2.03, P = 0.048], and only 3 of 48 judges had an ID level reliably above chance. Three lesioned models were evaluated by different groups of judges in separate
SCIENCE sciencemag.org 11 DECEMBER 2015 • VOL 350 ISSUE 6266 1335
1 2
1 2
1 2
1 2
1 2
1 2
Human or Machine?
Fig. 5. Generating new exemplars. Humans and machines were given an image of a novel character (top) and asked to produce new exemplars.The nine-character grids in each pair that were generated by a machine are (by row) 1, 2; 2, 1; 1, 1.
RESEARCH | RESEARCH ARTICLES
on N ovem ber 21, 2020
http://science.sciencem ag.org/
D ow nloaded from
for each subpart. Last, parts are roughly positioned to begin either independently, at the b ginni g, at the end, or along previous parts, as defined by relation Ri (Fig. 3A, iv). Character tokens q(m) are produced by executing the parts and the relations andmodeling how ink flows from the p n to the page. First, motor noise is added to the control points and the scale of the subparts to creat oken-level stroke trajectories S(m). Second, the trajectory’s precise star location L(m) is sampled from the schematic provided by its relationRi to previous strokes. Third, global transformations are sampled, including an ffine warp A(m) and a aptive noise parameters that eas probabilistic inference (30). Last, a binary image I (m) is creat d by a stochastic rendering function, lining the stroke trajectories with grayscale ink and interpreting the pixel values a independent Bernoulli probabilit es. Posterio inference requires searching the large
combinatorial space of programs that could have generated a r w image I (m). Our st a egy uses fa t bott m-up methods (31) to prop se a r nge of candidate parses. The most promising candidates are refined by using conti uo s optimization
and local search, forming a discrete approximation to he posterio distribution P(y , q(m)|I (m)) (section S3). Figure 4A shows the s t of discovered programs for a training image I (1) and how they are refit to different tes images I (2) to compute a classification score log P(I (2)|I (1)) (the log posterio p edictive probability), where hig er scores indicate hat they are more likely to belong to he same class. A hig score is achieved when at least one s t of parts and relations can successfully explain both e training and the tes images, without violating the soft constraints of the l arned with n-class vari b lity model. Figure 4B compares the model’s best-scoring parses with e ground-tru h human parses for several characters. Results
People, BPL, and alternative models were compared side by side on five concept learning tasks that examine different forms of generalization from just one or a few examples (example task Fig. 5). All behavioral experiments were un through Amazon’s Mechanical Turk, and the experimental procedures are d tailed in section S5.
The main results are summarized by Fig. 6, and additional lesion a alyses and controls are - ported in section S6. One-shot classification was evalu ted through a series of within-alphabet classification tasks for 10 different alphabets. As illustra ed in Fig. 1B, i, a single image of a new character was present d, and participants select d another example of tha same character from a set of 20 distinct characters produced by a typical drawer of tha alphabet. Performance is shown in Fig. 6A,where chance is 95% erro s. A a b seline, themodifiedHausdorff distance (32) was computed between cent red images, producing 38.8% erro s. People were skilled one-shot learners, achiev ng an verage erro rate of 4.5% (N = 40). BPL showed a similar erro rate of 3.3%, achiev ng better perfo mance than deep convolutional etwork (convnet; 13.5% erro s) and the HDmodel (34.8%)—each adapted from deep learning methods tha have p rformed well on a range of computer vision tasks. A deep Siamese convolutional network optimized for this one-shot learning task achieved 8.0% erro s (33), still about twice as hig as humans or ourmodel. BPL’s advant ge points to the b nefits ofmodeling theunderlying caus l process in learning concepts, a strategy different from the particular deep l arning approaches xamined here. BPL’s other key ingredi nts al o make positive contributions, a shown by hig er ro rates for BPL lesions without learning to learn (token-level only) or compositionality (11.0% erro s and 14.0%, resp ctively). Learning to learn was studie separately at the type and token level by disrupting the learned hyper a ameters of the generativemodel. Compositionality was evalu ted by comparing BPL to a matched model that allowed just one spline-based stroke, r sembling earlie analysisby-s nthesis models for handwritten characters tha were similarly limited (34, 35). The uman capacity for ne-shot learning is more than just classification. It can include a suite of abilities, such as generating ew xamples of a concept. We compared the creative outp ts produced by humans and machines through “visual Turing tes s,”where naive human judges tried to identify the machine, given paired xamples of human d machine b havior. In our most basic task, judges compared the drawings from nine humans a ked to produce a new instance of a concept given o e example with nine w examples drawn by BPL (Fig. 5) We evalu ted ach model based on the accuracy of the judges, whic we call their identification (ID) level: Idealmodel perfo mance is 50% ID level, indicating tha they cannot distinguish t e model’s behavior from humans; worst-case p rfo mance is 100%. Each judge (N = 147) completed 49 trials with blocked feedback, and judges were analyzed individually and in aggre ate. The results are shown i Fig. 6B (new exemplars). Judges had only a 52% ID level on average for discriminating human versus BPL behavior. As a group, this performance was barely better han chance [t(47) = 2.03, P = 0.048], and only 3 of 48 judges had n ID level r liably above chance. Three lesioned models were evaluated by different groups of judges in separate
SCIENCE sciencemag.org 11 DECEMBER 2015 • VOL 350 ISSUE 626 1335
1 2
1 2
1 2
1 2
1 2
1 2
Human or Machine?
Fig. 5 Gen rating ew exemplars. Humans and machines were given a image of a novel character (top) and asked to produce n w xemplars.The nine-character grids in each pair tha were g nerated by a machine are (by row) 1, 2; , 1; 1, 1.
RES ARCH | RES ARCH ARTICLES
on N ovem ber 21, 2020
http://science.sciencem ag.org/
D ow
nloaded from for each subpart. Last, parts are roughly positioned to begin either indep ndently, at the beginning, at the end, or along previous parts, as d fined by relation Ri (Fig. 3A, iv). Character tokens q(m) are produce by executi g the parts and th relations andmodeling how nk flows from the pen to the page. First, motor noise i added to the control points a d the scale of the subparts to create token-lev l stroke trajectori s S(m). Second, the trajectory’s precise start locat on L(m) is sampl d from the schematic provid by ts relationRi to previous strokes. Third, global transformations are sampled, including an affine warp A(m) and adaptive noise parameters that ease probabilistic inference (30). Last, a binary image I (m) is created by a stochastic rendering function, lining the stroke trajectori s with grayscale ink and interpreting the p xel values as independent Bernoulli probabilities. Posterior inference requires searching the lar e combinatorial space of programs at could have generated a raw image I (m). Our strategy uses fast bottom-up methods (31) to propose a ra ge of candidate parses. The most promising candidat s are refined by using continuous optimiza ion and local search, forming a discrete approximation to the posterior distribution P(y , q(m)|I (m)) (section S3). Figure 4A shows the set of discovered programs a training image I (1) and how they are refit to different test images I (2) to compute a classific tion score log P(I (2)|I (1)) (the log posterior predictive probabil ty), where higher scor s indica e that th y are m re likely to belong to the sam class. A high score is achieved when at least one set of parts and relations can successfully explain both the training and the tes images, without violating the soft constraints of the learned within-class variability model. Figure 4B compares the model’s best-scoring parses with the ground-truth human parses for several characters. Results People, BPL, and alternative models were compared ide by sid on five concept learning tasks that xamine different forms of generalization f om just o or a few examples (example task Fig. 5). All behavioral experiments were run through Amazon’s Mechanical Turk, and the exp rime tal p ocedures are detailed in section S5. The main results are summarized by Fig. 6, and additional lesion analyses and controls are reported in section S6. One-shot classification was evaluated through
a series of within-alphabet classification tasks for 10 different alphabets. As illustrated in Fig. 1B, i, a single image of a new character was presented, and participants selected another example of that same character from a set of 20 distinct characters produced by a typical drawer of that alphabet. Performance is shown in Fig. 6A,where chance is 95% errors. As a baseline, themodifiedHausdorff distance (32) was computed between centered images, producing 38.8% errors. People were skilled one-shot learners, achieving an average error rate of 4.5% (N = 40). BPL showed a similar error rate of 3.3%, achieving better performance than adeep convolutional network (convnet; 13.5% errors) and the HDmodel (34.8%)—each adapted from deep learning methods that have performed well on a range of computer vision tasks. A deep Siamese convolutional network optimized for this one-shot learning task achieved 8.0% errors (33), still about twice as high as humans or ourmodel. BPL’s advantage points to the benefits ofmodeling theunderlying causal process in learning concepts, a strategy different from the particular deep learning approaches examined here. BPL’s other key ingredients also make positive contributions, as shown by higher error rates for BPL lesions without learning to learn (token-level only) or compositionality (11.0% errors and 14.0%, respectively). Learning to learn was studied separately at the type and token level by disrupting the learned hyperparameters of the generativemodel. Compositionality was evaluated by comparing BPL to a matched model that allowed just one spline-based stroke, resembling earlier analysisby-synthesis models for handwritten characters that were similarly limited (34, 35). The human capacity for one-shot learning is more than just classification. It can include a suite of abilities, such as generating new examples of a concept. We compared the creative outputs produced by humans and machines through “visual Turing tests,”where naive human judges tried to identify the machine, given paired examples of human and machine behavior. In our most basic task, judges compared the drawings from nine humans asked to produce a new instance of a concept given one example with nine new examples drawn by BPL (Fig. 5). We evaluated each model based on the accuracy of the judges, which we call their identification (ID) level: Idealmodel performance is 50% ID level, indicating that they cannot distinguish the model’s behavior from humans; worst-case performance is 100%. Each judge (N = 147) completed 49 trials with blocked feedback, and judges were analyzed individually and in aggregate. The results are shown in Fig. 6B (new exemplars). Judges had only a 52% ID level on average for discriminating human versus BPL behavior. As a group, this performance was barely better than chance [t(47) = 2.03, P = 0.048], and only 3 of 48 judges had an ID level reliably above chance. Three lesioned models were evaluated by different groups of judges in separate
SCIENCE sciencemag.org 11 DECEMBER 2015 • VOL 350 ISSUE 6266 1335
1 2
1 2
1 2
1 2
1 2
1 2
Human or Machin ?
Fig. 5. Generating new exemplars. Humans and machines were given an image of a novel character (top) and asked to produce new exemplars.The nine-character grids in each pair that were generated by a machine are (by row) 1, 2; 2, 1; 1, 1.
RESEARCH | RESEARCH ARTICLES
on N ovem ber 21, 2020
http://science.sciencem ag.org/
D ow nloaded from
BPL
(a) Generating new exemplars
GNS
BPL
(b) Generating new concepts
Figure 4: Generation tasks. (a) GNS produced 9 new exemplars for each of 5 target images, plotted here next to human and BPL productions. (b) A grid of 36 new character concepts sampled unconditionally from GNS, shown next to BPL samples.
Generating new concepts (unconstrained). In addition to generating new exemplars of a particular concept, GNS can generate new character concepts altogether, unconditioned on training images. Whereas the BPL model us s a compli ated proc dure for uncon itional ge ration that involves a preliminary inference step and a supplemental nonparametric model, GNS generates new concepts by sampling directly from the type prior P (ψ). Moreover, the resulting GNS productions capture more of the structure found in real characters than either the raw BPL prior (Fig 1, 4b) or the supplemental nonparametric BPL prior [29]. In Fig. 4b we show a grid of 36 new character concepts sampled from our generative model at reduced temperature setting T = 0.5 (Appendix A.2). The model produces characters in multiple distinct styles, with some having more angular, line-based structure and others relying on complex curves. In Appendix Fig. A14, we show a larger set of characters sampled from GNS, plotted in a topologically-organized grid alongside a corresponding g id of “ne r st neig bor” training examples. In many cases, samples from the model have a distinct style and are visually dissimilar from their nearest Omniglot neighbor.
Marginal image likelihoods. As a final evaluation, we computed likelihoods of held-out character images by marginalizing over the latent type and token variables of GNS to estimate P (I) = ∫ P (ψ, θ, I)∂ψ∂θ. We hypothesized that our causal generative model of character concepts would yield better test likelihoods compared to deep generative models trained directly on image pixels. As detailed in Appendix D, under the minimal assump-
tion that our K posterior parses represent sharply peaked modes of the joint density, we can obtain an approximate lower bound on the marginal P (I) by using Laplace’s method to estimate the integral around each mode and summing the resulting integrals. In Table 3, we report average log-likelihood (LL) bounds obtained from GNS for a random subset of 1000 evaluation images, compared against test LL bounds from both the SG [43] and the VHE [21] models. Our GNS model performs stronger than each alternative, reporting the best overall log-likelihood per dimension.
6 DISCUSSION
We introduced a new generative neuro-symbolic (GNS) model for learning flexible, task-general representations of character concepts. We demonstrated GNS on the Omniglot challenge, showing that it performs a variety of inductive tasks in ways difficult to distinguish from human behavior.
Published as a conference paper at ICLR 2021
Some evaluations were still qualitative, and future work will further quantify these results using Visual Turing Tests [29].
Whereas many machine learning algorithms emphasize breadth of data domains, isolating just a single task across datasets, we have focused our efforts in this paper on a single domain, emphasizing depth of the representation learned. Human concept learning is distinguished for having both a breadth and depth of applications [37, 31], and ultimately, we would like to capture both of these unique qualities. We see our character model as belonging to a broader class of generative neuro-symbolic (GNS) models for capturing the data generation process. We have designed our model based on general principles of visual concepts—namely, that concepts are composed of reusable parts and locations—and we describe how it generalizes to 3D object concepts in Appendix E. As in the human mind, machine learning practitioners have far more prior knowledge about some domains vs. others. Handwritten characters is a domain with strong priors [1, 33, 23], implemented directly in the human mind and body. For concepts like these with more explicit causal knowledge, it is beneficial to include priors about how causal generative factors translate into observations, as endowed to our character model through its symbolic rendering engine. For other types of concepts where these processes are less clear, it may be appropriate to use more generic neural networks that generate concepts and parts directly as raw stimuli, using less symbolic machinery and prior knowledge. We anticipate that GNS can flexibly model concepts in both types of domains, although further experiments are needed to demonstrate this.
Our current token model for character concepts is much too simple, and we acknowledge a few important shortcomings. First, as shown in Appendix Fig. A10, there are a number of scenarios in which the parses from a training character cannot adequately refit to a new example of the same character without a token model that allows for changes to discrete variables. By incorporating this allowance in future work, we hope to capture more knowledge in this domain and further improve performance. Furthermore, although our vision for GNS is to represent both concepts and background knowledge with neuro-symbolic components, the current token-level model uses only simple parametric distributions. In future work, we hope to incorporate token-level models that use neural network sub-routines, as in the type-level model presented here.
REFERENCES
[1] M. K. Babcock and J. Freyd. Perception of dynamic information in static handwritten forms. American Journal of Psychology, 101(1):111–130, 1988.
[2] M. Botvinick, D. Barrett, P. Battaglia, N. de Freitas, D. Kumaran, and et al. Building machines that learn and think for themselves. Behavioral and Brain Sciences, 40:e255, 2017.
[3] J. Devlin, J. Uesato, S. Bhupatiraju, R. Singh, A.-R. Mohamed, and P. Kohli. Robustfill: Neural program learning under noisy i/o. In ICML, 2017.
[4] K. Ellis, D. Ritchie, A. Solar-lezama, and J. B. Tenenbaum. Learning to infer graphics programs from hand-drawn images. In NeurIPS, 2018.
[5] S. M. A. Eslami, N. Heess, T. Weber, Y. Tassa, D. Szepesvari, K. Kavukcuoglu, and G. E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In NeurIPS, 2016.
[6] R. Feinman and B. M. Lake. Generating new concepts with hybrid neuro-symbolic models. In CogSci, 2020.
[7] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
[8] Y. Ganin, T. Kulkarni, I. Babuschkin, S. M. A. Eslami, and O. Vinyals. Synthesizing programs for images using reinforced adversarial learning. In ICML, 2018.
[9] V. Garcia and J. Bruna. Few-shot learning with graph neural networks. In ICLR, 2018.
[10] A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, and D. B. Rubin. Bayesian Data Analysis (3rd ed.). CRC Press, Boca Raton, FL, 2014.
[11] S. Geman, E. Bienenstock, and R. Doursat. Neural networks and the bias/variance dilemma. Neural Computation, 4(1):1–58, 1992.
Published as a conference paper at ICLR 2021
[12] D. George, W. Lehrach, K. Kansky, M. Lázaro-Gredilla, C. Laan, B. Marthi, X. Lou, and et al. A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs. Science, 358(6368), 2017.
[13] N. D. Goodman, J. B. Tenenbaum, J. Feldman, and T. L. Griffiths. A rational analysis of rule-based concept learning. Cognitive Science, 32:108–154, 2008.
[14] N. D. Goodman, T. D. Ullman, and J. B. Tenenbaum. Learning a theory of causality. Psychological Review, 118(1):110–119, 2011.
[15] N. D. Goodman, J. B. Tenenbaum, and T. Gerstenberg. Concepts in a probabilistic language of thought. In E. Margolis and S. Laurence (ed.), The conceptual mind: New directions in the study of concepts, pp. 623–653. MIT Press, Cambridge, MA, 2015.
[16] A. Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013.
[17] A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.
[18] K. Greff, R. L. Kaufman, R. Kabra, N. Watters, C. Burgess, D. Zoran, L. Matthey, M. Botvinick, and A. Lerchner. Multi-object representation learning with iterative variational inference. In ICML, 2019.
[19] K. Gregor, I. Danihelka, A. Graves, D. J. Rezende, and D. Wierstra. DRAW: A recurrent neural network for image generation. In ICML, 2015.
[20] D. Ha and D. Eck. A neural representation of sketch drawings. In ICLR, 2018.
[21] L. B. Hewitt, M. I. Nye, A. Gane, T. Jaakkola, and J. B. Tenenbaum. The Variational Homoencoder: Learning to learn high capacity generative models from few examples. In UAI, 2018.
[22] L. B. Hewitt, T. A. Le, and J. B. Tenenbaum. Learning to learn generative programs with Memoised Wake-Sleep. In UAI, 2020.
[23] K. H. James and I. Gauthier. When writing impairs reading: Letter perception’s susceptibility to motor interference. Journal of Experimental Psychology: General, 138(3):416–31, 2009.
[24] C. Kemp and J. B. Tenenbaum. Structured statistical models of inductive reasoning. Psychological Review, 116:20–58, 2009.
[25] A. R. Kosiorek, S. Sabour, Y. W. Teh, and G. E. Hinton. Stacked Capsule Autoencoders. In NeurIPS, 2019.
[26] T. D. Kulkarni and et al. Picture: A probabilistic programming language for scene perception. In CVPR, 2015.
[27] B. M. Lake and M. Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In ICML, 2018.
[28] B. M. Lake and S. T. Piantadosi. People infer recursive visual concepts from just a few examples. Computational Brain & Behavior, 2019.
[29] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350:1332–1338, 2015.
[30] B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:E253, 2017.
[31] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. The Omniglot challenge: A 3-year progress report. Behavioral Sciences, 29:97–104, 2019.
[32] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436–444, 2015.
[33] M. Longcamp, J. L. Anton, M. Roth, and J. L. Velay. Visual presentation of single letters activates a premotor area involved in writing. Neuroimage, 19(4):1492–1500, 2003.
[34] J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In ICLR, 2019.
[35] G. F. Marcus. The Algebraic Mind: Integrating Connectionism and Cognitive Science. MIT Press, Cambridge, MA, 2003.
Published as a conference paper at ICLR 2021
[36] J. L. McClelland, M. M. Botvinick, D. C. Noelle, D. C. Plaut, T. T. Rogers, M. S. Seidenberg, and L. B. Smith. Letting structure emerge: Connectionist and dynamical systems approaches to cognition. Trends in Cognitive Science, 14:348–356, 2010.
[37] G. L. Murphy. The big book of concepts. MIT Press, Cambridge, MA, 2002. [38] G. L. Murphy and D. L. Medin. The role of theories in conceptual coherence. Psychological Review, 92(3):289–316, 1985. [39] M. I. Nye, A. Solar-Lezama, J. B. Tenenbaum, and B. M. Lake. Learning compositional rules via neural program synthesis. arXiv preprint arXiv:2003.05562, 2020. [40] A. Perfors, J. B. Tenenbaum, and T. Regier. The learnability of abstract syntactic principles. Cognition, 118(3):306–338, 2011. [41] S. T. Piantadosi, J. B. Tenenbaum, and N. D. Goodman. The logical primitives of thought: Empirical foundations for compositional cognitive models. Psychological Review, 2016. [42] S. Reed and N. de Freitas. Neural programmer-interpreters. In ICLR, 2016. [43] D. J. Rezende, S. Mohamed, I. Danihelka, K. Gregor, and D. Wierstra. One-Shot generalization in deep generative models. In ICML, 2016. [44] P. Shyam, S. Gupta, and A. Dukkipati. Attentive recurrent comparators. In ICML, 2017. [45] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In NeurIPS, 2017. [46] A. Stuhlmuller, J. B. Tenenbaum, and N. D. Goodman. Learning Structured Generative Concepts. In CogSci, 2010. [47] J. B. Tenenbaum, C. Kemp, T. L. Griffiths, and N. D. Goodman. How to grow a mind: Statistics, structure, and abstraction. Science, 331(6022):1279–1285, 2011. [48] L. Valkov, D. Chaudhari, A. Srivastava, C. Sutton, and S. Chaudhuri. HOUDINI: Lifelong learning as program synthesis. In NeurIPS, 2018. [49] O. Vinyals, C. Blundell, T. Lillicrap, and D. Wierstra. Matching networks for one shot learning. In NeurIPS, 2016. [50] K. Yi, J. Wu, C. Gan, A. Torralba, P. Kohli, and J. B. Tenenbaum. Neural-symbolic VQA:
Disentangling reasoning from vision and language understanding. In NeurIPS, 2018.
Published as a conference paper at ICLR 2021
A GENERATIVE MODEL
ψ θ I
M
Type Token Image
Figure A5: The GNS hierarchical generative model.
The full hierarchical generative model of GNS is depicted in Fig. A5. The joint density for type ψ, token θ(m), and image I(m) factors as
P (ψ, θ(m), I(m)) = P (ψ)P (θ(m)|ψ)P (I(m)|θ(m)). (6)
The type ψ parameterizes a motor program for generating character tokens θ(m), unique exemplars of the concept. Both ψ and θ(m) are expressed as causal drawing parameters. An image I(m) is obtained from token θ(m) by rendering the drawing parameters and sampling binary pixel values.
A.1 TRAINING ON CAUSAL DRAWING DATA
original stroke minimal spline
Figure A6: Spline representation. Raw strokes (left) are converted into minimal splines (right) using least-squares optimization. Crosses (left) indicate pen locations and red dots (right) indicate spline control points.
To learn the parameters of P (ψ) and P (θ(m) | ψ), we fit our models to the human drawing data from the Omniglot background set. In this drawing data, a character is represented as a variable-length sequence of strokes, and each stroke is a variable-length sequence of pen locations {z1, ..., zT }, with zt ∈ R2 (Fig. A6, left). Before training our model on background drawings, we convert each stroke into a minimal spline representation using least-squares optimization (Fig. A6, right), borrowing the B-spline tools from [29]. The number of spline control points depends on the stroke complexity and is determined by a residual threshold. Furthermore, we removed small strokes using a threshold on the trajectory length. These processing steps help suppress noise and emphasize signal in the drawings. Our generative models are trained to produce character drawings, where each drawing is represented as an ordered set of splines (strokes). The number of strokes, and the number of spline coordinates per stroke, are allowed to vary in the model.
A.2 TYPE PRIOR
The type prior P (ψ) represents a character as a sequence of strokes, with each stroke decomposed into a starting location yi ∈ R2, conveying the first spline control point, and a stroke trajectory xi = {∆1, ...,∆N}, conveying deltas between spline control points. It generates character types one stroke at a time, using a symbolic rendering procedure called frender as an intermediate processing step after forming each stroke. An image canvas C is used as a memory state to convey information about previous strokes. At each step i, the next stroke’s starting location and trajectory are sampled with procedure GeneratePart. In this procedure, the current image canvas C is first read by the location model (Fig. 2), a convolutional neural network (CNN) that processes the image and returns a
Published as a conference paper at ICLR 2021
probability distribution for starting location yi: yi ∼ p(yi | C).
The starting location yi is then passed along with the image canvas C to the stroke model, a Long Short-Term Memory (LSTM) architecture with a CNN-based image attention mechanism.The stroke model samples the next stroke trajectory xi sequentially one offset at a time, selectively attending to different parts of the image canvas at each sample step and combining this information with the context of yi: xi ∼ p(xi | yi, C). After GeneratePart returns, the stroke parameters yi, xi are rendered to produce an updated canvas C = frender(yi, xi, C). The new canvas is then fed to the termination model, a CNN architecture that samples a binary termination indicator vi:
vi ∼ p(vi | C).
Both our location model and stroke model follow a technique from [16], who proposed to use neural networks with mixture outputs to model handwriting data. Parameters {π1:K , µ1:K , σ1:K , ρ1:K} output by our network specify a Gaussian mixture model (GMM) with K components (Fig. 2; colored ellipsoids), where πk ∈ (0, 1) is the mixture weight of the kth component, µk ∈ R2 its means, σk ∈ R2+ its standard deviations, and ρk ∈ (−1, 1) its correlation. In our location model, a single GMM describes the distribution p(yi | C). In our stroke model, the LSTM outputs one GMM at each timestep, describing p(∆t|∆1:t−1, yi, C). The termination model CNN has no mixture outputs; it predicts a single Bernoulli probability to sample binary variable vi. When sampling from the model at test time, we use a temperature parameter proposed by Ha & Eck [20] (see [20, Eq. 8]) to control the entropy of the mixture density outputs.
A.3 TOKEN MODEL
procedure GENERATETOKEN(ψ) {κ, y1:κ, x1:κ} ← ψ . Unpack type-level variables for i = 1 ... κ do
y (m) i ∼ P (y (m) i | yi) . Sample token-level location x (m) i ∼ P (x (m) i | xi) . Sample token-level part A(m) ∼ P (A(m)) . Sample affine warp transformation θ ← {y(m)1:κ , x (m) 1:κ , A
(m)} return θ . Return concept token
Figure A7: Token model sampling procedure.
Character types ψ are used to parameterize the procedure GenerateToken(ψ), a probabilistic program representation of token model P (θ(m) | ψ). The psuedo-code of this sampling procedure is provided in Fig. A7. The location model P (y(m)i | yi) and part model P (x (m) i | xi) are each zeromean Gaussians, with standard deviations fit to the background drawings following the procedure of Lake et al. [29] (see SM 2.3.3). The location model adds noise to the start of each stroke, and the part model adds isotropic noise to the 2d cooridnates of each spline control point in a stroke. In the affine warp A(m) ∈ R4, the first two dimensions control global re-scaling of spline coordinates, and the second two control a global translation of the center of mass. The distribution is
P (A(m)) = N ([1, 1, 0, 0],ΣA), (7) with the parameter ΣA similarly fit from background drawings (see SM 2.3.4 in [29]).
B APPROXIMATE POSTERIOR
To obtain parses {ψ, θ}1:K for our approximate posterior (Eq. 3) given an image I , we follow the high-level strategy of Lake et al. [29], using fast bottom-up search followed by discrete selection and continuous optimization. The algorithm proceeds by the following steps.
13
Published as a conference paper at ICLR 2021
Step 1: Propose a range of candidate parses with fast bottom-up methods. The bottom-up algorithm extracts an undirected skeleton graph from the character image and uses random walks on the graph to propose a range of candidate parses. There are typically about 10-100 proposal parses, depending on character complexity (Fig. A8).
input
Figure A8: The initial “base” parses proposed for an image with skeleton extraction and random walks.
Step 2: Select stroke order and stroke directions for each parse using exhaustive search with the type prior P (ψ). Random search is used for complex parses with large configuration spaces.
Step 3: Score each of the proposal parses using type prior P (ψ) and select the top-K parses. We use K = 5 following previous work [29].
Step 4: Separate each parse into type and token {ψ, θ} and optimize the continuous type- and token-level parameters with gradient descent to maximize the full joint density P (ψ, θ, I) of Eq. 1.
Step 5: Compute weights π1:K for each parse by computing π̃k = P (ψk, θk, I) and normalizing πk = π̃k/ ∑K k=1 π̃k.
C INFERENCE FOR GENERATING NEW EXEMPLARS
When generating new exemplars, we are given a single image I(1) of a novel class and asked to generate new instances I(2) (overloading the parenthesis notation from classification). To perform this task with GNS, we first sample from our approximate posterior Q(ψ, θ | I(1)) to obtain parse {ψ, θ} (see Eq. 3), and then re-sample token parameters θ from our token model P (θ(2) | ψ). Due to high-dimensional images, mass in the approximate posterior often concentrates on the single best parse. To model the diversity seen in different human parses, we apply a temperature parameter to the log of unnormalized parse weights log(π̃′k) = log(π̃k)/T before normalization, selecting T = 8 for our experiments. With updated weights π′1:K our sampling distribution is written as
P (I(2), θ(2) | I(1)) ≈ K∑ k=1 π′kP (I (2) | θ(2))P (θ(2) | ψk). (8)
D MARGINAL IMAGE LIKELIHOODS
Let z = ψ ∪ θ be a stand-in for the joint set of type- and token-level random variables in our GNS generative model. The latent z includes both continuous and discrete variables: the number of strokes κ and the number of control points per stroke d1:κ are discrete, and all remaining variables are continuous. Decomposing z into its discrete variables zD ∈ ΩD and continuous variables zC ∈ ΩC , the marginal density for an image I is written as
P (I) = ∑
zD∈ΩD
∫ P (I, zD, zC)∂zC . (9)
Published as a conference paper at ICLR 2021
For any subset Ω̃D ⊂ ΩD of the discrete domain, the following inequality holds: P (I) ≥ ∑
zD∈Ω̃D
∫ P (I, zD, zC)∂zC . (10)
Our approximate posterior (Eq. 3) gives us K parses that represent promising modes {zD, zC}1:K of the joint density P (I, zD, zC) for an image I , and by setting Ω̃D = {zD}1:K to be the set of K unique discrete configurations from our parses, we can compute the lower bound of Eq. 10 by computing the integral ∫ P (I, zD, zC)∂zC at each of these zD.
At each zDk ∈ {zD}1:K , the log-density function f(zC) = log P (I, zDk, zC) has a gradient-free maximum at zCk, the continuous configuration of the corresponding posterior parse. These maxima were identified by our gradient-based continuous optimizer during parse selection (Appendix B). If we assume that these maxima are sharply peaked, then we can use Laplace’s method to estimate the integral ∫ P (I, zDk, zC)∂zC at each zDk. Laplace’s method uses Taylor expansion to approximate the integral of ef(x) for a twice-differentiable function f around a maximum x0 as∫ ef(x)∂x ≈ ef(x0) (2π) d 2
| −Hf (x0)| 12 , (11)
where x ∈ Rd andHf (x0) is the Hessian matrix of f evaluated at x0. Our log-density function f(zC) is fully differentiable w.r.t. continuous parameters zC , therefore we can compute H(zC) = ∂2f/∂z2C with ease. Our approximate lower bound on P (I) is therefore written as the sum of Laplace approximations at our K parses:
P (I) ≥ K∑ k=1 ∫ P (I, zDk, zC)∂zC ≈ K∑ k=1 P (I, zDk, zCk) (2π) d 2 | −H(zCk)| 1 2
(12)
E APPLYING GNS TO 3D OBJECT CONCEPTS
The GNS modeling framework is designed to capture inductive biases for concept learning that generalize across different types of visual concepts. We are actively working on applying GNS as a task-general generative model of 3D object concepts, such as chairs and vehicles, again with a neuro-symbolic generative process for parts and relations. Here, we briefly review the path forward for training GNS models of object concepts.
procedure GENERATETYPE C 0 . Initialize blank image canvas while true do
[yi, xi] GENERATEPART(C) . Sample part location & parameters C frender(yi, xi, C) . Render part to image canvas vi ⇠ p(v | C) . Sample termination indicator if vi then
break . Terminate sample {, y1:, x1:} return . Return concept type
1
procedure GENERATETOKEN C 0 . Initialize blank 3d canvas while true do
xi GENERATEPART(C) . Sample part from neural net C fupdate(xi, C) . Update 3d canvas vi ⇠ p(v | C) . Sample termination indicator if vi then
break . Terminate sample ✓ {, x1:} return . Return concept token
1
procedure GENERATETYPE C 0 . Initialize blank image canvas while true do
[yi, xi] GENERATEPART(C) . Sample part location & parameters C frender(yi, xi, C) . Render part to image canvas vi ⇠ p(v | C) . Sample termination indicator if vi then
break . Terminate sample {, y1:, x1:} return . Return concept type
category model p(k |C)
instance model p(x |k, C)
C p(k ∣ C)
C
q(z ∣ x, k, C)
zk
p(x |z)
x
p
k
Ck
p(z |k, C)
x
generate token
se at
leg s ba ck
re st
Figure A9: A GNS model for the concept of a chair.
Everyday 3D object concepts have more within-class variability than handwritten characters; for example, different chair tokens vary in the number of arm rests, legs, etc. much as different handwritten character types vary in their parts, although both domains are characterized by similar structural
15
Published as a conference paper at ICLR 2021
and statistical considerations. To address these differences within GNS, we shift up one level in the generative hierarchy and design a token-level model for generating new exemplars of an individual concept (chairs, cars, etc.) that mirrors our type-level model for characters. The architecture and sampling procedure of GNS for 3D object tokens is given in Fig. A9. As with characters, our object model produces samples one part at a time, using a 3D canvas C in place of the previous 2D canvas. The procedure GeneratePart consists of two neural network components: 1) a discriminitivelytrained category model p(k | C) that predicts the category label k of the next part given the current canvas (leg, arm, back, etc.), and 2) a generative instance model p(x | k,C) that is trained with a variational autoencoder objective to sample an instance of the next part x given the current canvas and predicted part category label. Objects and object parts are represented as 3D voxel grids, and all neural modules, including the category model and the encoder/decoder of the instance model, are parameterized by 3D convolutional neural networks. A function fupdate is used to update the current canvas with the most recent part by summing the voxel grids. A GNS model for a particular concept is trained on examples of the 3D voxel grids with semantic part labels.
F.4 GENERATING NEW CONCEPTS (UNCONSTRAINED)
In Fig. A14 we show a grid of 100 new character concepts produced by GNS, plotted alongside a corresponding grid of “nearest neighbor” training examples.
Published as a conference paper at ICLR 2021
I(c)
I(T)
Score: -618.6
I(c)
I(T)
Score: -1624.5
I(c)
I(T)
Score: -269.4
I(c)
I(T)
Score: -281.1
I(c)
I(T)
Score: -269.1
I(c)
I(T)
Score: -381.6
I(c)
I(T)
Score: -820.9
I(c)
I(T)
Score: -505.6
I(c)
I(T)
Score: -403.6
I(c)
I(T)
Score: -814.4
I(c)
I(T)
Score: -742.5
I(c)
I(T)
Score: -545.6
Figure A10: Classification fits. Each row corresponds to one classification trial (one test image). The first column shows parses from the correct training image re-fit to the test example, and the second column parses from an incorrect training image. The two-way score for each train-test pair is shown above the grid, and the model’s selected match is emboldened. The 4th and 6th row here are misclassified trials.
Published as a conference paper at ICLR 2021
Published as a conference paper at ICLR 2021
Figure A12: Generating new exemplars with GNS. Twelve target images are highlighted in red boxes. For each target image, the GNS model sampled 9 new exemplars, shown in a 3x3 grid under the target.
One-shot Generalization in Deep Generative Models
Figure 8. Unconditional samples for 52 × 52 omniglot (task 1).
For a video of the generation process, see https://www.youtube.com/ watch?v=HQEI2xfTgm4
Figure 9. Generating new examplars of a given character for the
weak generalization test (task 2a). The first row shows the test images and the next 10 are one-shot samples from the model.
3. Representative samples from a novel alphabet. This task corresponds to figure 7 in Lake et al. (2015), and conditions the model on anywhere between 1 to 10 samples of a novel alphabet and asks the model to generate new characters consistent with this novel alphabet. We show here the hardest form of this test, using only 1 context image. This test is highly subjective, but the model generations in figure 11 show that it is able to pick up common features and use them in the generations.
We have emphasized the usefulness of deep generative models as scalable, general-purpose tools for probabilistic reasoning that have the important property of one-shot generalization. But, these models do have limitations. We have already pointed to the need for reasonable amounts of data. Another important consideration is that, while our models can perform one-shot generalization, they do not perform one-shot learning. One-shot learning requires that a model is updated after the presentation of each new input, e.g., like the non-parametric models used by Lake et al. (2015) or Salakhutdinov et al. (2013). Parametric models such as ours require a gradient update of the parameters, which we do not do. Instead, our model performs a type of one-shot inference that during test time can perform inferential tasks on new data points, such as missing data completion, new exemplar generation, or analogical sampling, but does not learn from these points. This distinction between one-shot learning and inference is important and affects how such models can be used. We aim to extend our approach to the online and one-shot learning setting in future.
30-20 40-10 45-5
Figure 10. Generating new examplars of a given character for the
strong generalization test (task 2b,c), with models trained with
different amounts of data. Left: Samples from model trained on
30-20 train-test split; Middle: 40-10 split; Right: 45-5 split (right)
Figure 11. Generating new exemplars from a novel alphabet (task
3). The first row shows the test images, and the next 10 rows are one-shot samples generated by the model.
6. Conclusion We have developed a new class of general-purpose models that have the ability to perform one-shot generalization, emulating an important characteristic of human cognition. Sequential generative models are natural extensions of variational auto-encoders and provide state-of-the-art models for deep density estimation and image generation. The models specify a sequential process over groups of latent variables that allows it to compute the probability of data points over a number of steps, using the principles of feedback and attention. The use of spatial attention mechanisms substantially improves the ability of the model to generalize. The spatial transformer is a highly flexible attention mechanism for both reading and writing, and is now our default mechanism for attention in generative models. We highlighted the one-shot generalization ability of the model over a range of tasks that showed that the model is able to generate compelling and diverse samples, having seen new examples just once. However there are limitations of this approach, e.g., still needing a reasonable amount of data to avoid overfitting, which we hope to address in future work.
(a) SG
Figure 4: 5-shot samples generated by each model (more in Supplement). With a PixelCNN architecture, both Neural Statistician and Resample objectives lead to underutilisation of the latent space, producing unfaithful samples.
For the hierarchical PixelCNN architecture, however, significant differences arise between training objectives. In this case, a Neural Statistician learns a strong global distribution over images but makes only minimal use of latent variables c. This means that, despite the use of a
higher capacity model, classification accuracy is much
poorer (66%) than that achieved using a deconvolutional architecture. For the same reason, conditional samples display an improved sharpness but are no longer identifiable to the cue images on which they were conditioned (Figure 4). Our careful training suggests that this is not
an optimisation difficulty but is core to the objective, as
discussed in Chen et al. (2016).
By contrast, a VHE is able to gain a large benefit from the hierarchical PixelCNN architecture, with a 3-fold reduction in classification error (5-shot accuracy 98.8%) and conditional samples which are simultaneously sharp and identifiable (Figure 4). This improvement is in part achieved by increased utilisation of the latent space, due to rescaling of the KL divergence term in the objective. However, our results show that this common technique is insufficient when used alone, leading to overfitting to
cue images with an equally severe impairment of classi-
fication performance (accuracy 62.8%). Rather, we find that KL-rescaling and data resampling must be used tog ther in order for the benefit of the powerful PixelCNN architecture to be re lis d.
Table 2 lists the classification accuracy achieved by
VHEs with both |D| = 1 and |D| = 5, as compared
to existing d ep learning appr aches. We find t at both network ar not only state- f-the-art mongst deep generative models, but also competitive against the best discriminative models trained directly for few-shot classification. Unlike these discriminative models, VHE is also able to generate new images of a character in one shot, producing samples which are both realistic and faithful to the class of the cue image (Figure 5).
As our goal is to model shared structure acr ss images, we evaluate generative performance using joint log like-
Figure 5: One-shot same-class samples generated by our model. Cue images were sampled from previously unseen classes.
lihood of the entire Omniglot test set (rather than separately across images). From this perspective, a single element VAE will perform poorly as it treats all datapoints as independent, optimising a sum over log likelihoods for each element. By sharing latents across elements of the same class, a VHE can improve upon this considerably.
For likelihood evaluation, our most appropriate comparison is with Generative Matching Networks (Bartunov & Vetrov, 2016) as they also model dependencies within a class. Thus, we trained models under the same train/test split as them, with no data augmentation. We evaluate the joint log likelihood of full character classes from the test set, normalised by the number of elements, using importance weighting with k=500 samples from q(c; X). As can be seen in Tables 3 and 4, our hierarchical PixelCNN architecture is able to achieve state-of-the-art log likelihood results only when trained using the VHE objective.
Figure 4: 5-shot samples generated by each model (more in Supplement). With a PixelCNN architecture, both Neural Statistician and Resample objectives lead to underutilisation of the latent space, producing unfaithful samples.
For the hierarchical PixelCNN architecture, however, significant differences arise between training objectives. In this case, a Neural Statistician learns a strong global distribution over images but makes only minimal use of
latent variables c. This means that, despite the use of a
higher capacity model, classification accuracy s much poorer (66%) than that achieved using a deconvolutional architecture. For the same reason, conditional samples display an improved sharpness but are no longer identifiable to the cue images on which they were conditioned
(Figure 4). Our care ul training sug ests that this is not
an optimisation difficulty but is core t the objective, as discussed in Chen et al. (2016).
By contrast, a VHE is able to gain a large benefit from the hierarchical PixelCNN architecture, with a 3-fold reduction in classification error (5-shot accuracy 98.8%)
and conditional samples which are simultaneously sharp
and identifiable (Figure 4). This improvement is in part
achieved by increased utilisation of the latent space, due to rescaling of the KL divergence term in the objective. However, our results show that this common technique is insufficient when used alone, leading to overfitting to cue images with an equally severe impair ent of classification performance (accuracy 62.8%). Rather, we find that KL-rescaling and data resampling must be used together in order for the benefit of the powerful PixelCNN architecture to be realised.
Table 2 lists the classification accuracy achieved by VHEs with both |D| = 1 and |D| = 5, as compared to existing deep learning approaches. We find that both networks are not only state-of-the-art amongst deep generative models, but also competitive against the best discriminative models trained directly for few-shot classi-
fication. Unlike these discriminative models, a VHE
is also able to generate new images of a character in one shot, producing samples which are both realistic and faithful to the class of the cue image (Figure 5).
As our goal is to model shared structure across images, we evaluate generative performance usi g joint log like-
Figure 5: One-shot same-class samples generated by our
model. Cue images were sampled from previously un-
seen classes.
lihood of the entire Omniglot test set (rather than separately across images). From this perspective, a single element VAE will perform poorly as it treats all datapoints as independent, optimising a sum over log likelihoods for each element. By sharing latents across elements of the same class, a VHE can improve upon this considerably.
For likelihood evaluation, our most appropriate comparison is with Generative Matching Networks (Bartunov & Vetrov, 2016) as they also model dependencies within a class. Thus, we trained models under the same train/test
split as them, with no data augmentation. We evaluate the
joint log likelihood of full character classes from the test set, normalised by the number of elements, using importance weighting with k=500 samples from q(c; X). As can be seen in Tables 3 and 4, our hierarchical PixelCNN architecture is able to achieve state- f-the-art log likelihood results nly when trained using the VHE objective.
Figure 4: 5-shot samples generated by each model (more in Supplement). With a PixelC N architecture, both Neural S atist cian and Resample objectives lead to underutilisation of the latent space, producing unfaithful samples.
For the hie archical PixelC N architecture, however, significant diff rences arise between tra ing objectives. In this case, a Neural S atist cian learns a stron global distribution over images but makes only m nimal use of
latent variables c. This means that, despite the use of a
higher capacity model, clas ification ccuracy is much poo er (66%) than that achieved using a deconvolutional architecture. For the same reason, conditional samples display an improved sharpness but are no longer identifiable to the cue images on which they w re conditioned
(Figure 4). Our careful tra ing sugge t that this is not
an optimisation difficulty but is core to the objective, as discussed in Chen et al. (2016).
By contrast, a VHE is able to gain a large benefit from the hie archical PixelC N architecture, with a 3-fold reduction in classification error (5-shot accuracy 9 .8%)
and conditional samples which are simultaneously sharp
and identifiable (Figure 4). This improvement is in part
achieved by increased utilisation of the latent space, due to rescaling of the KL divergence term in the objective. However, ou result show tha this co mon technique is insufficient when used alone, leading t overfitting to cue images with an equally sev re impairment of classification performance (accuracy 62.8%). Rather, we find that KL-rescaling an d ta resampling must be used together in order for the benefit of the powerful PixelC N architecture to be realised.
Table 2 li ts the classification accuracy achieved by VHEs with both |D| = 1 and |D| = 5, as compared to existing deep lear ing approaches. We find that both networks are not only s ate-of-the-art amongst deep generative models, but also competitive against the best discriminative models trained directly for few-shot classi-
fication. Unlike th se discriminative models, a VHE
is also able to generate new images of a ch racter in one shot, producing samples which are both realistic and faithful to the class of the cue image (Figure 5).
As our goal is to model shared structure across images, w evaluate generative perf rmance using joint log like-
Figure 5: One-shot same-clas samples generated by our
model. Cue images w re sampled from previously un-
seen classes.
lihood of th entire Omniglo test set (rather than separately across images). From this perspective, a singl elment VAE will perform poorly as it treats all d tapoints
as independent, optim sing a sum over log likelihoods for each lement. By sharing latents across lements of the same class, a VHE can improve upon this considerably.
For likelihood evaluation, our most appropriate comparison is with Generative Matching Networks (Bartunov & Vetrov, 2016) as they also model dependencies within a class. Thus, we trained models under the same train/test
split as them, with no d t augmen ation. W evaluate the
joint log likelihood o full ch racter cla ses from the test set, normalised by the number of lements, using importance weighting with k=500 samples from q(c; X). As can be seen in Tables 3 and 4, our hie archical PixelC N architectur is able to achieve s ate-of-the art log likelihood results only whe trained using the VHE objective.
Figure 4: 5-shot samples generated by each mod l (more in Supplement). With a PixelCNN architecture, both eural Statistician and Resample objectives l ad to underutilisation of the lat nt sp ce, pr ducing u faithful samples.
For the hierarchical PixelCNN architecture, however, significant differences arise between training objectives. In this case, a Neural Stati tician learns a strong global distribution over images t makes only minimal use of latent variables c. This me ns that, despite th use of higher capacity model, classific tion accuracy i much poorer (66%) than that achieved using a deconvolutional architecture. For the same reason, c nditional samples display an improved sharpness but a e no longer identifiable to the cue images on w ich they were c ditioned
(Figure 4). Our careful traini g s ggests that this is not
an optimisation difficulty but is c re t the objective, as discussed in Chen et al. (2016).
By contrast, a VHE is able to gain a large benefit from the hierarchical PixelCNN architecture, with a 3-fold r - duction in classification error (5-shot curacy 98.8%) and conditional samples which are simultan ously sh p and identifiable (Figure 4). This improvement is in part achieved by increased utilisation of th latent space, due to rescaling of the KL diverge ce term in the obj ctiv . However, our results show that this common technique is insufficient when used alone, leading to overfitting to cue images with an equally s vere imp irment of classification performance ( curacy 62.8%). R ther, we find that KL-rescaling and d ta resampling must be us d together in order for the ben fit of th p w rful PixelCNN architecture to be re lised.
Table 2 lists the classification accuracy achieved by VHEs with both |D| = 1 and |D| = 5, as compared to existing deep learning approaches. We fi d that bot networks are not only state-of-the-art amongst deep gen erative models, but also competitive against the best discriminative models trained directly for few-shot classification. Unlike these discriminativ mod ls, a VHE is also able to generate new images of a character in one shot, producing samples which are both realistic and faithful to the class of t e c e image (Figure 5).
As our goal is to model shared structure acros images, we evaluate gener tiv performance using joint l g lik -
Figure 5: One-shot same-class samples generated by our
mod l. Cue images were sampl d from previously un-
seen classes.
lihood of the e tire Omniglot test s t ( ather than separately acro s images). From this persp ctive, a single elment VAE will perfor poorly as it treats all datapoint
as independent, optimising a sum over log likelihoods for
each le e t. By sh ring lat nts across elements of the
same class, a VHE can i prove upon this considerably.
For likeli ood evaluation, our m st appropriate co paris i with G erative Matching N tworks (Bartu ov & Vetrov, 2016) a they also model depend ncies within a clas . Thus, we trained models under the sa e train/test
split as them, with no ata augmentatio . We eval at the
joint log likel hood f full charact r class s from the t st set, normal s d by th numb of elem nts, sing importanc weighting with k=500 samples from q(c; X). As can be seen in Tables 3 d 4, our hier rchical PixelCNN architecture is ble to a i ve state-of-the-art log likelihood results only wh train d usi g the VHE obj ctiv .
Figure 4: 5-s ot samples g nerated by each del (mor in Supplem n ). With a PixelCNN architecture, both Neural Statistician and Resample objectives le d to und rutilisatio of the latent spac , producing nfaithful samples.
For the hiera chical PixelCNN architecture, however, significant differences ari e betwee training objectives. In this case, a Neural Statistician learns a strong global distribution over images but makes only minimal use of
latent variabl s c. This means that, despite the use of a
higher capacity model, lassification accuracy is much poorer (66%) than that achieved using a deconvolutional archite ture. For the same reason, conditional samples display an improv d sharpness but are no longer identifiable to the cue images on which they were conditioned
(Figure 4). O r careful training suggests that this is not
an optimisation difficulty but is core to the objective, as discussed in Chen et al. (2016).
By contrast, a VHE is able to gain a large benefit from t e hierarchical PixelCNN architecture, with a 3-fold reduction in classification error (5-shot accuracy 98.8%)
and conditional samples which are sim ltaneously harp
and dentifiabl (Figure 4). This i provement is in part
achieved by increased utilisation of the latent space, due to r scaling of the KL divergence term in the objective. However, our results show that this common technique is insufficient when used alone, leading to overfitting to cu images with an equally severe impairment of classificati n performance (accuracy 62.8%). Rather, we find that KL-rescaling and data resa pling must be used together in order for the benefit f the powerful PixelCNN architecture to be realised.
Table 2 lists the classification accuracy achieved by VHEs with both |D| = 1 and |D| = 5, as compared t existing deep learning approaches. We find that both netwo ks are not only state-of-the-art amongst deep genera models, but al o competitive against the best discri inative models trained directly for few-shot classification. Unlike these discri inative models, a VHE
is also able to ge erate ew images of a character in one shot, pr ducing samples which are both realistic and faithful to the class of the cue image (Figure 5).
As our goal is t model shared structure across images, w eval at ge erative p rformance using joint log like-
Figure 5: One-shot same-class samples generated by our odel. Cue images were sampled from previously un-
seen classes.
lihood of the ntire Omniglot test set (rather than separately across im ges). From this perspective, a single element VAE will perf rm poorly as it treats all datapoints as independent, optimi ing a sum over log likelihoods for each element. By sharing latents across elements of the same class, a VHE can im rove upon this considerably.
For likelihood evaluation, our most appropriate comparison is with Genera ive Matching Networks (Bartunov & Vetrov, 2016) as they also model dependencies within a class. Thus, we trained mo els under the same train/test
split as them, with no data augmentation. We evaluate the
join log likeliho d of full character classes from the test set, nor alised by the number of eleme ts, using importance weighting with k=500 sa ples from q(c; X). As ca be seen in Tables 3 and 4, our hierarchical PixelCNN architecture is ab e to achieve state-of-the-art log likelihood r sults only when trai ed usi g the VHE objective.
Figure 4: 5-shot samples g nerated by each model ( or in Supplement). With PixelCNN architect e, both Neural Statistician and R sample objectiv lead to underutilis tion of th latent space, produci g unfaithful samples.
For the hierarchical PixelCNN archit cture, however, signifi ant d fferences arise between train ng objectives. In this case, a Neural Stati tician learns strong g obal dist ibution over images but akes only mini l u e of
atent var ables c. This me ns that, d spite the use of a
higher capa ity mod l, l ssificati n accuracy is much poorer (66%) than that achieved using a deconvol tional architecture. For the same reas n, conditional amples display an improved sh rp ess but are no longer identifiable to the cue imag s on which they were conditioned
(Figure 4). Our careful training suggests that this is not
an optimisation d fficulty bu is core to the objective, as dis ussed in Chen et al. (2016).
By contrast, a VHE s ab e to gai a la ge benefit from the h erarchical PixelCNN architecture, with a 3-fold - duction in classifica error (5-sho accuracy 98.8%)
and conditional s mples which are imultaneously sharp
and identifiable (F gure 4). This improve ent is in part
achi ved by increased utilisat o of th latent spac , due
to rescaling of th KL divergence term in the obj ctive. However, our results show that this common technique is insuffici nt wh n used alo e, l ading to overfitting to cue images with an equ lly sev re impairment of classification performan e (accuracy 62.8%). Rather, we find that KL-rescaling nd data resampli g ust be used together in order for t benefit of the pow rful PixelCNN architectur to be ealised.
Tabl 2 li ts the cl ssification ac uracy chieved by VHEs with both |D| = 1 and |D| = 5, as compared to existi g dee learn ng approaches. We find that both netw rks are not only state-of-the-art amo gst de p generative models, but also competitive gain t the best discriminative o els trained directly for few-shot classification. Unlike these discrimina ive odels, a VHE is also abl to generat new images of a character in one shot, producing samples which are both realistic nd faithful to the class of the cue im ge (Figure 5).
As ur goa is to odel sh red structure across images, w valuate generative performance using joint log like-
Figure 5: One-shot same-class samples generated by our
model. Cue images were sampled fro previously un-
seen lasses.
lihood of the entire Omniglot tes set (rather han separa ly across i ages). From this per pective, a single element VAE will perform poorly as it treats all dat points as independent, optimising a sum over log likelihoods for each eleme t. By sharing latents across el m t of the same class, a VHE can improve upon this considerably.
F r likelih od evaluation, our most appropriate c mpariso is with Generative Matching Netw rks (Bartunov & Vetrov, 2016) as th y also model dependencies within a class. Thus, we traine odels under the same train/test
spl s them, with no data augment tion. W evaluate the
joint log like ihood of ful character classe from the test s t, nor ali ed by h number of eleme ts, using imporance weighting with k=500 samples fro q(c; X). As
can be seen in Tables 3 and 4, our h rarchical PixelCNN archit cture is able to a hiev state-of-the-art log likelihood results only when t ained using the VHE objective.
(b) VHE
Figure A13: New exemplars produced by the Sequential Gen rative (SG) model [43] and the Vari tional Homoencoder (VHE) [21]. (a) The SG model shows far too much variability, drawing what is clearly the wrong character in many cases (e.g. right-most column). (b) The VHE character samples are often incomplete, missing important strok s of the target class.
19
Published as a conference paper at ICLR 2021

Review summaries:
In this paper, the authors seek to combine the advantages of symbolic, compositional models and neural approaches, in particular by using "probabilistic programs with neural network subroutines" which should reflect a causal generative process. This is certainly an interesting area to explore. But I'm not convinced that the approach they propose (GNS) lives up to their stated goals.

Motivated by few-shot learning challenges such as Omniglot, the authors propose a human-like model for learning how to draw visual concepts that consists of three components: (1) a location model for picking the starting point of the next stroke given the current "canvas", (2) a stroke model that continues an existing stroke, (3) a termination model that decides when to stop drawing.  The three components of this model are trained on example glyphs that are heuristically parsed into strokes.  The authors evaluate performance of this method, called GNS (Generative Neuro-Symbolic) versus more classical program learning approaches based on pre-existing libraries of subroutines and more generic deep neural network architectures.  This approach (1) represents an advance over methods based on pre-defined subroutines in the sense that primitives such as "draw a stroke intersecting an existing stroke" are learned rather than supplied by the programmer and (2) performs much better at few-shot learning of glyphs than generic approaches.