Learning Task-General Representations with Generative Neuro-Symbolic Modeling

People can learn rich , general-purpose conceptual representations from only raw perceptual inputs . Current machine learning approaches fall well short of these human standards , although different modeling traditions often have complementary strengths . Symbolic models can capture the compositional and causal knowledge that enables flexible generalization , but they struggle to learn from raw inputs , relying on strong abstractions and simplifying assumptions . Neural network models can learn directly from raw data , but they struggle to capture compositional and causal structure and typically must retrain to tackle new tasks . We bring together these two traditions to learn generative models of concepts that capture rich compositional and causal structure , while learning from raw data . We develop a generative neuro-symbolic ( GNS ) model of handwritten character concepts that uses the control flow of a probabilistic program , coupled with symbolic stroke primitives and a symbolic image renderer , to represent the causal and compositional processes by which characters are formed . The distributions of parts ( strokes ) , and correlations between parts , are modeled with neural network subroutines , allowing the model to learn directly from raw data and express nonparametric statistical relationships . We apply our model to the Omniglot challenge of human-level concept learning , using a background set of alphabets to learn an expressive prior distribution over character drawings . In a subsequent evaluation , our GNS model uses probabilistic inference to learn rich conceptual representations from a single training image that generalize to 4 unique tasks , succeeding where previous work has fallen short . 1 INTRODUCTION Human conceptual knowledge supports many capabilities spanning perception , production and reasoning [ 37 ] . A signature of this knowledge is its productivity and generality : the internal models and representations that people develop can be applied flexibly to new tasks with little or no training experience [ 30 ] . Another distinctive characteristic of human conceptual knowledge is the way that it interacts with raw signals : people learn new concepts directly from raw , high-dimensional sensory data , and they identify instances of known concepts embedded in similarly complex stimuli . A central challenge is developing machines with these human-like conceptual capabilities . Engineering efforts have embraced two distinct paradigms : symbolic models for capturing structured knowledge , and neural network models for capturing nonparametric statistical relationships . Symbolic models are well-suited for representing the causal and compositional processes behind perceptual observations , providing explanations akin to people ’ s intuitive theories [ 38 ] . Quintessential examples include accounts of concept learning as program induction [ 13 , 46 , 29 , 15 , 4 , 28 ] . Symbolic programs provide a language for expressing causal and compositional structure , while probabilistic modeling offers a means of learning programs and expressing additional conceptual knowledge through priors . The Bayesian Program Learning ( BPL ) framework [ 29 ] , for example , provides a dictionary of simple sub-part primitives for generating handwritten character concepts , and symbolic relations that specify how to combine sub-parts into parts ( strokes ) and parts into whole character concepts . These abstractions support inductive reasoning and flexible generalization to a range of different tasks , utilizing a single conceptual representation [ 29 ] . Published as a conference paper at ICLR 2021 HumansBPL model ( centered ) GNS model Symbolic models offer many useful features , but they come with important limitations . Foremost , symbolic probabilistic models make simplifying and rigid parametric assumptions , and when the assumptions are wrong—as is common in complex , high-dimensional data—they create bias [ 11 ] . The BPL character model , for example , assumes that parts are largely independent a priori , an assumption that is not reflective of real human-drawn characters . As a consequence , characters generated from the raw BPL prior lack the complexity of real characters ( Fig 1 , left ) , even though the posterior samples can appear much more structured . Another limitation of symbolic probabilistic models is that the construction of structured hypothesis spaces requires significant domain knowledge [ 2 ] . Humans , meanwhile , build rich internal models directly from raw data , forming hypotheses about the conceptual features and the generative syntax of a domain . As one potential resolution , previous work has demonstrated that the selection of structured hypotheses can itself be attributed to learning in a Bayesian framework [ 47 , 13 , 14 , 41 , 24 , 40 ] . Although more flexible than a priori structural decisions , models of this kind still make many assumptions , and they have not yet tackled the types of raw , high-dimensional stimuli that are distinctive of the neural network approach . The second paradigm , neural network modeling , prioritizes powerful nonparametric statistical learning over structured representations . This modeling tradition emphasizes emergence , the idea that conceptual knowledge arises from interactions of distributed sub-symbolic processes [ 36 , 32 ] . Neural networks are adept at learning from raw data and capturing complex patterns . However , they can struggle to learn the compositional and causal structure in how concepts are formed [ 30 ] ; even when this structure is salient in the data , they may have no obvious means of incorporating it . These limitations have been linked to shortcomings in systematic generalization [ 35 , 27 ] and creative abilities [ 31 ] . An illustrative example is the Omniglot challenge : in 4 years of active research , neural network models do not yet explain how people quickly grasp new concepts and use them in a variety of ways , even with relatively simple handwritten characters [ 31 ] . Surveying over 10 neural models applied to Omniglot , Lake et al . [ 31 ] found that only two attempted both classification and generation tasks , and they were each outperformed by the fully-symbolic , probabilistic BPL . Moreover , neural generative models tended to produce characters with anomalous characteristics , highlighting their shortcomings in modeling causal and compositional structure ( see Fig . A13 and [ 31 , Fig . 2a ] ) . In this paper , we introduce a new approach that leverages the strengths of both the symbolic and neural network paradigms by representing concepts as probabilistic programs with neural network subroutines . We describe an instance of this approach developed for the Omniglot challenge [ 29 ] of task-general representation learning and discuss how we see our Omniglot model fitting into a broader class of Generative Neuro-Symbolic ( GNS ) models that seek to capture the data-generation process . As with traditional probabilistic programs , the control flow of a GNS program is an explicit representation of the causal generative process that produces new concepts and new exemplars . Moreover , explicit re-use of parts through repeated calls to procedures such as GeneratePart ( Fig . 2 ) ensures a representation that is compositional , providing an appropriate inductive bias for compositional generalization . Unlike fully-symbolic probabilistic programs , however , the distribution of parts and correlations between parts in GNS are modeled with neural networks . This architectural choice allows the model to learn directly from raw data , capturing nonparametric statistics while requiring only minimal prior knowledge . Published as a conference paper at ICLR 2021 We develop a GNS model for the Omniglot challenge of learning flexible , task-general representations of handwritten characters . We report results on 4 Omniglot challenge tasks with a single model : 1 ) one-shot classification , 2 ) parsing/segmentation , 3 ) generating new exemplars , and 4 ) generating new concepts ( without constraints ) ; the 5th and final task of generating new concepts ( from type ) is left for future work . We also provide log-likelihood evaluations of the generative model . Notably , our goal is not to chase state-of-the-art performance on one task across many datasets ( e.g. , classification ) . Instead we build a model that learns deep , task-general knowledge within a single domain and evaluate it on a range of different tasks . This “ deep expertise ” is arguably just as important as “ broad expertise ” in characterizing human-level concept learning [ 37 , 31 ] ; machines that seek human-like abilities will need both . Our work here is one proposal for how neurally-grounded approaches can move beyond pattern recognition toward the more flexible model-building abilities needed for deep expertise [ 30 ] . In Appendix E , we discuss how to extend GNS to another conceptual domain , providing a proposal for a GNS model of 3D object concepts . 2 RELATED WORK The Omniglot dataset and challenge has been widely adopted in machine learning , with models such as Matching Nets [ 49 ] , MAML [ 7 ] , and ARC [ 44 ] applied to just one-shot classification , and others such as DRAW [ 19 ] , SPIRAL [ 8 ] , and VHE [ 21 ] applied to one or more generative tasks . In their “ 3-year progress report , ” Lake et al . [ 31 ] reviewed the current progress on Omniglot , finding that although there was considerable progress in one-shot classification , there had been little emphasis placed on developing task-general models to match the flexibility of human learners ( Table 1 ) . Moreover , neurally-grounded models that attempt more creative generation tasks were shown to produce characters that either closely mimicked the training examples or that exhibited anomalous variations , making for easy identification from humans ( see Fig . A13 and [ 31 , Fig . 2a ] ) . Our goal is distinct in that we aim to learn a single neuro-symbolic generative model that can perform a variety of unique tasks , and that generates novel yet structured new characters . Neuro-symbolic modeling has become an active area of research , with applications to learning input-output programs [ 42 , 17 , 3 , 39 , 48 ] , question answering [ 50 , 34 ] and image description [ 26 , 4 ] . GNS modeling distinguishes itself from prior work through its focus on hybrid generative modeling , combining both structured program execution and neural networks directly in the probabilistic generative process . Neuro-symbolic VQA models [ 50 , 34 ] are neither generative nor task-general ; they are trained discriminatively to answer questions . Other neuro-symbolic systems use neural networks to help perform inference in a fully-symbolic generative model [ 26 , 4 ] , or to parameterize a prior over fully-symbolic hypotheses [ 22 ] . In order to capture the dual structural and statistical characteristics of human conceptual representations , we find it important to include neural nets directly in the forward generative model . As applied to Omniglot , our model bears some resemblance to SPIRAL [ 8 ] ; however , SPIRAL does not provide a density function , and it has no hierarchical structure , limiting its applications to image reconstruction and unconditional generation . Another class of models on the neuro-symbolic spectrum aims to learn “ object representations ” with neural networks [ 5 , 18 , 25 ] , which add minimal object-like symbols to support systematic reasoning and generalization . Although these models have demonstrated promising results in applications such as scene segmentation and unconditional generation , they have not yet demonstrated the type of rich inductive capabilities that we are after : namely , the ability to learn “ deep ” conceptual representations from just one or a few examples that support a variety of discriminative and generative tasks . Other works ( e.g . [ 16 , 20 ] ) have used autoregressive models like ours with similar stroke primitives to model the causal generative processes of handwriting . We develop a novel architecture for generating Published as a conference paper at ICLR 2021 type level token level procedure GENERATETYPE C 0 . Initialize blank image canvas while true do [ yi , xi ] GENERATEPART ( C ) . Sample part location & parameters C frender ( yi , xi , C ) . Render part to image canvas vi ⇠ p ( v | C ) . Sample termination indicator if vi then break . Terminate sample { , y1 : , x1 : } return . Return concept type 1 C Canvas yi , xi Part Image I procedure GENERATETYPE C 0 . Initialize blank image canvas while true do [ yi , xi ] GENERATEPART ( C ) . Sample part location & parameters C frender ( yi , xi , C ) . Render part to image canvas vi ⇠ p ( v | C ) . Sample termination indicator if vi then break . Terminate sample { , y1 : , x1 : } return . Return concept type 1 pr oc ed ur e G EN ER AT ET YP E C 0 . In itia liz e bla nk im ag e ca nv as wh ile tr ue do [ y i , x i ] G EN ER AT EP AR T ( C ) . Sa m ple pa rt loc at ion & pa ra m et er s C f re nd er ( y i , x i , C ) . Re nd er pa rt to im ag e ca nv as v i ⇠ p ( v | C ) . Sa m ple te rm ina tio n ind ica to r if v i th en br ea k . Te rm ina te sa m ple { , y 1 : , x 1 : } re tu rn . Re tu rn co nc ep t t yp e 1 procedure GENERATETYPE C 0 . Initialize blank image canvas while true do [ yi , xi ] GENERATEPART ( C ) . Sample part location & parameters C frender ( yi , xi , C ) . Render part to image canvas vi ⇠ p ( v | C ) . Sample termination indicator if vi then break . Terminate sample { , y1 : , x1 : } return . Return concept type 1 location model p ( y ∣ C ) CNN MLP stroke model p ( x ∣ y , C ) CNN LSTM y C attention p ( y ) p ( Δ1 ) p ( Δ2 ∣ Δ1 ) p ( ΔT ∣ Δ1 : T−1 ) …C Figure 2 : A generative neuro-symbolic ( GNS ) model of character concepts . The type model GenerateType ( P ( ψ ) ) produces character types one stroke at a time , using an image canvas C as memory . At each step , the current canvas C is fed to procedure GeneratePart and a stroke sample is produced . The canvas is first processed by the location model , a CNN-MLP architecture that samples starting location y , and next by the stroke model , a CNN-LSTM architecture that samples trajectory x while attending to the encoded canvas . Finally , a symbolic renderer updates the canvas according to x and y , and a termination model decides whether to terminate the type sample . Unique exemplars are produced from a character type by sampling from the token model conditioned on ψ , adding motor noise to the drawing parameters and performing a random affine transformation . handwriting , which represents explicit compositional structure by modeling parts and relations with separate modules and applying intermediate symbolic rendering . Most importantly , these prior models have not made a connection to the image ; therefore while they can generate handwriting as symbolic coordinates , they can not explain how people use their causal knowledge to learn new characters from visual presentations , how they infer the strokes of a character seen on paper , or how they generate a new example of an observed character . By combining a powerful autoregressive model of handwriting with a symbolic image renderer and algorithms for probabilistic inference , we seek to replicate a spectrum of unique human concept learning abilities . 3 GENERATIVE MODEL Our GNS model leverages the type-token hierarchy of BPL [ 29 ] , which offers a useful scaffolding for conceptual models ( Fig . 2 , left ) .1 The type-level model P ( ψ ) defines a prior distribution over character concepts , capturing overarching principles and regularities that tie together characters from different alphabets and providing a procedure to generate new character concepts unconditionally in latent stroke format ( Fig . 2 , right ) . A token-level model p ( θ|ψ ) captures the within-class variability that arises from motor noise and drawing styles , and an image distribution P ( I|θ ) provides an explicit model of how causal stroke actions translate to image pixels . All parameters of our model are learned from the Omniglot background set of drawings ( Appendix A ) . The full joint distribution over type ψ , token θ ( m ) and image I ( m ) factors as P ( ψ , θ ( m ) , I ( m ) ) = P ( ψ ) P ( θ ( m ) |ψ ) P ( I ( m ) |θ ( m ) ) . ( 1 ) Although sharing a common hierarchy , the implementation details of each level in our GNS model differ from BPL in critical ways . The GNS type prior P ( ψ ) is a highly expressive generative model that uses an external image canvas , coupled with a symbolic rendering engine and an attentive 1Aspects of our model were recently published in a non-archival conference proceedings [ 6 ] . The previous manuscript presents only a prior distribution that alone performs just one task ( generating new concepts ) ; our new developments include a full hierarchical model , a differentiable image renderer / likelihood , and a procedure for approximate probabilistic inference from image data . These ingredients together enable GNS to perform 4 unique conceptual tasks . Published as a conference paper at ICLR 2021 recurrent neural network , to condition future parts on the previous parts and model sophisticated causal and correlational structure . This structure is essential to generating new character concepts in realistic , human-like ways ( Sec . 5 ) . Moreover , whereas the BPL model is provided symbolic relations for strokes such as “ attach start ” and “ attach along , ” GNS learns implicit relational structure from the data , identifying salient patterns in the co-occurrences of parts and locations . Importantly , the GNS generative model is designed to be differentiable at all levels , yielding log-likelihood gradients that enable powerful new inference algorithms ( Sec . 4 ) and estimates of marginal image likelihood ( Sec . 5 ) . Type prior . The type prior P ( ψ ) is captured by a neuro-symbolic generative model of character drawings . The model represents a character as a sequence of strokes ( parts ) , with each stroke i decomposed into a starting location yi ∈ R2 and a variable-length trajectory xi ∈ Rdi×2 . Rather than use raw pen trajectories as our stroke format , we use a minimal spline representation of strokes , obtained from raw trajectories by fitting cubic b-splines with a residual threshold . The starting location yi therefore conveys the first spline control point , and trajectory xi = { ∆i1 , ... , ∆idi } conveys the offsets between subsequent points of a ( di+1 ) -length spline . These offsets are transformed into a sequence of relative points xi = { xi1 , ... , xidi+1 } , with xi1 = 0 , specifying locations relative to yi . The model samples a type one stroke at a time , using an image canvas C as memory to convey the sample state . At each step , a starting location for the next stroke is first sampled from the location model , followed by a trajectory from the stroke model . The stroke is then rendered to the canvas C , and a termination model decides whether to terminate or continue the sample . Each of the three model components is expressed by a neural network , using a LSTM as the stroke model to generate trajectories as in [ 16 ] . The details of these neural modules are provided in Appendix A . The type model P ( ψ ) specifies an auto-regressive density function that can evaluate exact likelihoods of character drawings , and its hyperparameters ( the three neural networks ) are learned from the Omniglot background set of 30 alphabets using a maximum likelihood objective . A full character type ψ includes the random variables ψ = { κ , y1 : κ , x1 : κ } , where κ ∈ Z+ is the number of strokes . The density function P ( ψ ) is also fully differentiable w.r.t . the continuous random variables in ψ . Token model . A character token θ ( m ) = { y ( m ) 1 : κ , x ( m ) 1 : κ , A ( m ) } represents a unique instance of a character concept , where y ( m ) 1 : κ are the token-level locations , x ( m ) 1 : κ the token-level parts , and A ( m ) ∈ R4 the parameters of an affine warp transformation . The token distribution factorizes as P ( θ ( m ) |ψ ) = P ( A ( m ) ) κ∏ i=1 P ( y ( m ) i | yi ) P ( x ( m ) i | xi ) . ( 2 ) Here , P ( y ( m ) i | yi ) represents a simple noise distribution for the location of each stroke , and P ( x ( m ) i | xi ) for the stroke trajectory . The first two dimensions of affine warp A ( m ) control a global re-scaling of the token drawing , and the second two a global translation of its center of mass . The distributions and pseudocode of our token model are given in Appendix A . Image model . The image model P ( I ( m ) | θ ( m ) ) is based on [ 29 ] and is composed of two pieces . First , a differentiable symbolic engine f receives the token θ ( m ) and produces an image pixel probability map pimg = f ( θ ( m ) , σ , ) by evaluating each spline and rendering the stroke trajectories . Here , σ ∈ R+ is a parameter controlling the rendering blur around stroke coordinates , and ∈ ( 0 , 1 ) controlling pixel noise , each sampled uniformly at random . The result then parameterizes an image distribution P ( I ( m ) | θ ( m ) ) = Bernoulli ( pimg ) , which is differentiable w.r.t . θ ( m ) , σ , and . 4 PROBABILISTIC INFERENCE Given an image I of a novel concept , our GNS model aims to infer the latent causal , compositional process for generating new exemplars . We follow the high-level strategy of BPL for constructing a discrete approximation Q ( ψ , θ | I ) to the desired posterior distribution [ 29 ] , P ( ψ , θ | I ) ≈ Q ( ψ , θ | I ) = K∑ k=1 πkδ ( θ − θk ) δ ( ψ − ψk ) . ( 3 ) A heuristic search algorithm is used to find K good parses , { ψ , θ } 1 : K , that explain the underlying image with high probability . These parses are weighted by their relative posterior probability , Published as a conference paper at ICLR 2021 log P ( I ( T ) ∣ I ( c ) ) = − 401.3 Correct match train test log P ( I ( T ) ∣ I ( c ) ) = − 664.8 Incorrect match train test ( a ) Classification fits the pen ( Fig . 3A , ii ) . To construct a new character type , first themodel samples the number of parts k and the number of subparts ni , for each part i = 1 , ... , k , from their empirical distributions as measured from the background set . Second , a template for a part Si is constructed by sampling subparts from a set of discrete primitive actions learned from the background set ( Fig . 3A , i ) , such that the probability of the next action depends on the previous . Third , parts are then grounded as parameterized curves ( splines ) by sampling the control points and scale parameters 1334 11 DECEMBER 2015 • VOL 350 ISSUE 6266 sciencemag.org SCIENCE Fig . 3 . A generative model of handwritten characters . ( A ) New types are generated by choosing primitive actions ( color coded ) from a library ( i ) , combining these subparts ( ii ) to make parts ( iii ) , and combining parts with relations to define simple programs ( iv ) . New tokens are generated by running these programs ( v ) , which are then rendered as raw data ( vi ) . ( B ) Pseudocode for generating new types y and new token images I ( m ) for m = 1 , ... , M. The function f ( · , · ) transforms a subpart sequence and start location into a trajectory . Human parses Machine parsesHuman drawings -505 -593 -655 -695 -723 -1794-646 -1276 Training item with model ’ s five best parses Test items 1 2 3 4 5stroke order : Fig . 4 . Inferringmotor programs from images . Parts are distinguished by color , with a colored dot indicating the beginning of a stroke and an arrowhead indicating the end . ( A ) The top row shows the five best programs discovered for an image along with their log-probability scores ( Eq . 1 ) . Subpart breaks are shown as black dots . For classification , each program was refit to three new test images ( left in image triplets ) , and the best-fitting parse ( top right ) is shown with its image reconstruction ( bottom right ) and classification score ( log posterior predictive probability ) . The correctly matching test item receives a much higher classification score and is also more cleanly reconstructed by the best programs induced from the training item . ( B ) Nine human drawings of three characters ( left ) are shown with their ground truth parses ( middle ) and best model parses ( right ) . RESEARCH | RESEARCH ARTICLES on June 1 , 2020 http : //science.sciencem ag.org/ D ow nloaded from the pen ( Fig . 3A , ii ) . To construct a new character type , first themodel samples the number of parts k and the number of subparts ni , for each part i = 1 , ... , k , from their empirical distributions as measured from the background set . Second , a template for a part Si is constructed by sampling subparts from a set of discrete primitive actions learned from the background set ( Fig . 3A , i ) , such that the probability of the next action depends on the previous . Third , parts are then grounded as parameterized curves ( splines ) by sampling the control points and scale parameters 1334 11 DECEMBER 2015 • VOL 350 ISSUE 6266 sciencemag.org SCIENCE Fig . 3 . A generative model of handwritten characters . ( A ) New types are generated by choosing primitive actions ( color coded ) from a library ( i ) , combining these subparts ( ii ) to make parts ( iii ) , and combining parts with relations to define simple programs ( iv ) . New tokens are generated by running these programs ( v ) , which are then rendered as raw data ( vi ) . ( B ) Pseudocode for generating new types y and new token images I ( m ) for m = 1 , ... , M. The function f ( · , · ) transforms a subpart sequence and start location into a trajectory . Human parses Machine parsesHuman drawings -505 -593 -655 -695 -723 -1794-646 -1276 Training item with model ’ s five best parses Test items 1 2 3 4 5stroke order : Fig . 4 . Inferringmotor programs from images . Parts are distinguished by color , with a colored dot indicating the beginning of a stroke and an arrowhead indicating the end . ( A ) The top row shows the five best programs discovered for an image along with their log-probability scores ( Eq . 1 ) . Subpart breaks are shown as black dots . For classification , each program was refit to three new test images ( left in image triplets ) , and the best-fitting parse ( top right ) is shown with its image reconstruction ( bottom right ) and classification score ( log posterior predictive probability ) . The correctly matching test item receives a much higher classification score and is also more cleanly reconstructed by the best programs induced from the training item . ( B ) Nine human drawings of three characters ( left ) are shown with their ground truth parses ( middle ) and best model parses ( right ) . RESEARCH | RESEARCH ARTICLES on June 1 , 2020 http : //science.sciencem ag.org/ D ow nloaded from the pen ( Fig . 3A , ii ) . To construct a new character type , first themodel samples the number of parts k and the nu ber of subparts ni , for each part i = 1 , ... , k , fro their empirical distributions as measured from the background set . Second , a template for a part Si is constructed by sampling subparts from a set of discrete primitive actions learned from the background set ( Fig . 3A , i ) , such that the probability of the next action depends on the previous . Third , parts are then grounded as parameterized curves ( splines ) by sampling the control points and scale parameters 1334 11 DECEMBER 2015 • VOL 350 ISSUE 6266 sciencemag.org SCIENCE Fig . 3 . A generative model of handwritten characters . ( A ) New types are generated by choosing primitive actions ( color coded ) from a library ( i ) , combining these subparts ( ii ) to make parts ( iii ) , and combining parts with relations to define simple programs ( iv ) . New tokens are generated by running these programs ( v ) , which are then rendered as raw data ( vi ) . ( B ) Pseudocode for generating new types y and new token images I ( m ) for m = 1 , ... , M. The function f ( · , · ) transforms a subpart sequence and start location into a trajectory . Human parses Machine parsesHuman drawings -505 -593 -655 -695 -723 -1794-646 -1276 Training item with model ’ s five best parses Test items 1 2 3 4 5stroke order : Fig . 4 . Inferringmotor programs from images . Parts are distinguished by color , with a colored dot indicating the beginning of a stroke and an arrowhead indicating the end . ( A ) The top row shows the five best programs discovered for an image along with their l g-probability cores ( Eq . 1 ) . Subpart breaks are shown as black dots . For classification , each program was refit to three new test images ( left in image triplets ) , and the best-fitting parse ( top right ) is shown with its image reconstruction ( bottom right ) and classification score ( l g posterior predictiv probability ) . The correctly matching tes item ceives a much higher classification score and is also more cleanly reconstructed by the best program induced from the training item . ( B ) Nine human rawings of three characters ( left ) are shown with their grou d tr th parses ( middle ) and best model parses ( right ) . RESEARCH | RESEARCH ARTICLES on June 1 , 2020 http : //science.sciencem ag.org/ D ow nloaded from Human drawings Human parse GNS rses BPL parse the pen ( Fig . 3A , ii ) . To construct a new character type , first themodel samples the number of parts k and the number of subparts ni , for each part i = 1 , ... , k , from their empirical distributions as measured from the background set . Second , a template for a part Si is constructed by sampling subparts from a set of discrete primitive actions learned from the background set ( Fig . 3A , i ) , such that the probability of the next action depends on the previous . Third , parts are then grounded as parameterized curves ( splines ) by sampling the control points and scale parameters 1334 11 DECEMBER 2015 • VOL 350 ISSUE 6266 sciencemag.org SCIENCE Fig . 3 . A generative model of handwritten characters . ( A ) New types re generated by choosing primitive actions ( color coded ) from a library ( i ) , combining these subparts ( ii ) to make parts ( iii ) , an combining parts with relations t de ine simple programs ( iv ) . New tokens are generated by running these programs ( v ) , which are then rendered as raw dat ( vi ) . ( B ) Pseudocode for genera ing new types y and new token images I ( m ) for m = 1 , ... , M. The function f ( · , · ) transforms a subpart sequence and start location into a trajectory . Human parses Machine parsesHuman drawings -505 -593 -655 -695 -723 -1794-646 -1276 Training item with model ’ s five best parses Test items 1 2 3 4 5stroke order : Fig . 4 . Inferringmotor programs from images . Parts are distinguished by color , with a colored dot indicating the beginning of a stroke and an arrowhead indicating the end . ( A ) The top row shows the five best programs discovered for an im ge along with their log-probability scores ( Eq . 1 ) . Subpart breaks are hown as black dots . For classification , each program was refit to three new test images ( left in image triplets ) , and the best-fitting parse ( top right ) is shown with its image reconstruction ( bottom right ) and classification score ( log posterior predictive probability ) . The correctly matchi g test item receives a much higher classification score and is also more cleanly reconstructed by th best programs indu ed from the raining item . ( B ) Nine human drawings of three cha acters ( left ) are shown with their ground tr th parses ( middle ) an b st model parses ( right ) . RESEARCH | RESEARCH ARTICLES on June 1 , 2020 http : //science.sciencem ag.org/ D ow nloaded from ( b ) Parsing Figure 3 : Classification fits and parsing . ( a ) Posterior parses from two training images were refit to the same test image . The first row of each grid shows the training image and its top-3 predicted parses ( best emboldened ) . The second row shows the test image and its re-fitted training parses . Reconstructed test images are shown in the final row . The correct training image reports a high forward score , indicating that I ( T ) is well-explained by the motor programs for this I ( c ) . ( b ) 27 character images from 3 classes are shown alongside their ground truth human parses , predicted parses from the GNS model , and predicted parses from the BPL model . πk ∝ π̃k = P ( ψk , θk , I ) such that ∑ k πk = 1 . To find the K good parses , search uses fast bottomup methods to propose many variants of the discrete variables , filtering the most promising options , before optimizing the continuous variables with gradient descent ( we use K = 5 ) . See Appendix B for further details about inference , and Appendix C for generating new exemplars of a concept . Inference for one-shot classification . In one-shot classification , models are given a single training image I ( c ) from each of c = 1 , ... , C classes , and asked to classify test images according to their corresponding training classes . For each test image I ( T ) , we compute an approximation of the Bayesian score log P ( I ( T ) | I ( c ) ) for every example I ( c ) , using our posterior parses { ψ , θ ( c ) } 1 : K and corresponding weights π1 : K from I ( c ) ( Eq . 3 ) . The approximation is formulated as log P ( I ( T ) | I ( c ) ) ≈ log ∫ P ( I ( T ) |θ ( T ) ) P ( θ ( T ) | ψ ) Q ( ψ , θ ( c ) , | I ( c ) ) ∂ψ∂θ ( c ) ∂θ ( T ) ≈ log K∑ k=1 πk max θ ( T ) P ( I ( T ) | θ ( T ) ) P ( θ ( T ) | ψk ) , ( 4 ) where the maximum over θ ( T ) is determined by refitting token-level parameters θ ( c ) to image I ( T ) with gradient descent . Following Lake et al . [ 29 ] , we use a two-way version of the Bayesian score that also considers parses of I ( T ) refit to I ( c ) . The classification rule is therefore c∗ = arg max c log P ( I ( T ) | I ( c ) ) 2 = arg max c log [ P ( I ( c ) | I ( T ) ) P ( I ( c ) ) P ( I ( T ) | I ( c ) ) ] , ( 5 ) where P ( I ( c ) ) ≈∑k π̃k is approximated from the unnormalized weights of I ( c ) parses . 5 EXPERIMENTS GNS was evaluated on four concept learning tasks from the Omniglot challenge [ 31 ] : one-shot classification , parsing , generating new exemplars , and generating new concepts . All evaluations use novel characters from completely held-out alphabets in the Omniglot evaluation set . As mentioned Published as a conference paper at ICLR 2021 earlier , our goal is to provide a single model that captures deep knowledge of a domain and performs strongly in a wide range of tasks , rather than besting all models on every task . Our experiments include a mixture of quantitative and qualitative evaluations , depending on the nature of the task . One-shot classification . GNS was compared with alternative models on the one-shot classification task from Lake et al . [ 29 ] . The task involves a series of 20-way within-alphabet classification episodes , with each episode proceeding as follows . First , the machine is given one training example from each of 20 novel characters . Next , the machine must classify 20 novel test images , each corresponding to one of the training classes . With 20 episodes total , the task yields 400 trials . Importantly , all character classes in an episode come from the same alphabet as originally proposed [ 29 ] , requiring finer discriminations than commonly used between-alphabet tests [ 31 ] . Table 2 : Test error on withinalphabet one-shot classification . Model Error GNS 5.7 % BPL [ 29 ] 3.3 % RCN [ 12 ] 7.3 % VHE [ 21 ] 18.7 % Proto . Net [ 45 ] 13.7 % ARC [ 44 ] 1.5 % ∗ * used 4x training classes As illustrated in Fig . 3a , GNS classifies a test image by choosing the training class with the highest Bayesian score ( Eq . 5 ) . A summary of the results is shown in Table 2 . GNS was compared with other machine learning models that have been evaluated on the withinalphabets classification task [ 31 ] . GNS achieved an overall test error rate of 5.7 % across all 20 episodes ( N=400 ) . This result is very close to the original BPL model , which achieved 3.3 % error with significantly more hand-design . The symbolic relations in BPL ’ s token model provide rigid constraints that are key to its strong classification performance [ 29 ] . GNS achieves strong classification performance while emphasizing the nonparametric statistical knowledge needed for creative generation in subsequent tasks . Beyond BPL , our GNS model outperformed all other models that received the same background training . The ARC model [ 44 ] achieved an impressive 1.5 % error , although it was trained with four-fold class augmentation and many other augmentations , and it can only perform this one task . In Appendix Fig . A10 , we show a larger set of classification fits from GNS , including examples of misclassified trials . Parsing . In the Omniglot parsing task , machines must segment a novel character into an ordered set of strokes . These predicted parses can be compared with human ground-truth parses for the same images . The approximate posterior of GNS yields a series of promising parses for a new character image , and to complete the parsing task , we identify the maximum a posteriori parse k∗ = maxk πk , reporting the corresponding stroke configuration . Fig . 3b shows a visualization of the GNS predicted parses for 27 different raw images drawn from 3 unique character classes , plotted alongside groundtruth human parses ( how the images were actually drawn ) along with predicted parses from the BPL model . Compared to BPL , GNS parses possess a few unique desirable qualities . The first character class has an obvious segmentation to the human eye—evidenced by the consistency of human parses in all examples—and the GNS model replicates this consistency across all 9 predicted parses . In contrast , BPL predicts seemingly-unlikely parses for 2 of the examples shown . The second character is more complex , and it was drawn in different styles by different human subjects . The GNS model , which is trained on data from subjects with different styles , captures the uncertainty in this character by predicting a variety of unique parses . BPL , on the other hand , produces a single , ubiquitous segmentation across all 9 examples . In Appendix Fig . A11 , we provide a larger set of parses from the GNS model for a diverse range of Omniglot characters . Generating new exemplars . Given just one training image of a novel character concept , GNS produces new exemplars of the concept by sampling from the approximate conditional P ( I ( 2 ) , θ ( 2 ) | I ( 1 ) ) of Eq . 8 . In Fig . 4a we show new exemplars produced by GNS for a handful of target images , plotted next to human productions ( more examples in Appendix Fig . A12 ) . In the majority of cases , samples from the model demonstrate that it has successfully captured the causal structure and invariance of the target class . In contrast , deep generative models applied to the same task miss meaningful compositional and causal structure , producing new examples that are easily discriminated from human productions [ 43 , 21 ] ( see Appendix Fig . A13 ) . In some cases , such as the third column of Fig . 4a , samples from GNS exhibit sloppy stroke junctions and connections . Compared to BPL , which uses engineered symbolic relations to enforce rigid constraints at stroke junctions , GNS misses some of these structural elements . Nevertheless , new examples from GNS appear strong enough to pass for human in many cases , which we would like to test in future work with visual Turing tests . Concept learning experiments can be reproduced using our pre-trained generative model and source code : https : //github.com/rfeinman/GNS-Modeling . Published as a conference paper at ICLR 2021 Human GNS Target for each subpart . Last , parts are roughly positioned to begin either independently , at the beginning , at the end , or along previous parts , as defined by relation Ri ( Fig . 3A , iv ) . Character tokens q ( m ) are produced by executing the parts and the relations andmodeling how ink flows from the pen to the page . First , motor noise is added to the control points and the scale of the subparts to create token-level stroke trajectories S ( m ) . Second , the trajectory ’ s precise start location L ( m ) is sampled from the schematic provided by its relationRi to previous strokes . Third , global transformations are sampled , including an affine warp A ( m ) and adaptive noise parameters that ease probabilistic inference ( 30 ) . Last , a binary image I ( m ) is created by a stochastic rendering function , lining the stroke trajectories with grayscale ink and interpreting the pixel values as independent Bernoulli probabilities . Posterior inference requires searching the large combinatorial space of programs that could have generated a raw image I ( m ) . Our strategy uses fast bottom-up methods ( 31 ) to propose a range of candidate parses . The most promising candidates are refined by using continuous optimization and local search , forming a discrete approximation to the posterior distribution P ( y , q ( m ) |I ( m ) ) ( section S3 ) . Figure 4A shows the set of discovered programs for a training image I ( 1 ) and how they are refit to different test images I ( 2 ) to compute a classification score log P ( I ( 2 ) |I ( 1 ) ) ( the log posterior predictive probability ) , where higher scores indicate that they are more likely to belong to the same class . A high score is achieved when at least one set of parts and relations can successfully explain both the training and the test images , without violating the soft constraints of the learned within-class variability model . Figure 4B compares the model ’ s best-scoring parses with the ground-truth human parses for several characters . Results People , BPL , and alternative models were compared side by side on five concept learning tasks that examine different forms of generalization from just one or a few examples ( example task Fig . 5 ) . All behavioral experiments were run through Amazon ’ s Mechanical Turk , and the experimental procedures are detailed in section S5 . The main results are summarized by Fig . 6 , and additional lesion analyses and controls are reported in section S6 . One-shot classification was evaluated through a series of within-alphabet classification tasks for 10 different alphabets . As illustrated in Fig . 1B , i , a single image of a new character was presented , and participants selected another example of that same character from a set of 20 distinct characters produced by a typical drawer of that alphabet . Performance is shown in Fig . 6A , where chance is 95 % errors . As a baseline , themodifiedHausdorff distance ( 32 ) was computed between centered images , producing 38.8 % errors . People were skilled one-shot learners , achieving an average error rate of 4.5 % ( N = 40 ) . BPL showed a similar error rate of 3.3 % , achieving better performance than adeep convolutional network ( convnet ; 13.5 % errors ) and the HDmodel ( 34.8 % ) —each adapted from deep learning methods that have performed well on a range of computer vision tasks . A deep Siamese convolutional network optimized for this one-shot learning task achieved 8.0 % errors ( 33 ) , still about twice as high as humans or ourmodel . BPL ’ s advantage points to the benefits ofmodeling theunderlying causal process in learning concepts , a strategy different from the particular deep learning approaches examined here . BPL ’ s other key ingredients also make positive contributions , as shown by higher error rates for BPL lesions without learning to learn ( token-level only ) or compositionality ( 11.0 % errors and 14.0 % , respectively ) . Learning to learn was studied separately at the type and token level by disrupting the learned hyperparameters of the generativemodel . Compositionality was evaluated by comparing BPL to a matched model that allowed just one spline-based stroke , resembling earlier analysisby-synthesis models for handwritten characters that were similarly limited ( 34 , 35 ) . The human capacity for one-shot learning is more than just classification . It can include a suite of abilities , such as generating new examples of a concept . We compared the creative outputs produced by humans and machines through “ visual Turing tests , ” where naive human judges tried to identify the machine , given paired examples of human and machine behavior . In our most basic task , judges compared the drawings from nine humans asked to produce a new instance of a concept given one example with nine new examples drawn by BPL ( Fig . 5 ) . We evaluated each model based on the accuracy of the judges , which we call their identification ( ID ) level : Idealmodel performance is 50 % ID level , indicating that they can not distinguish the model ’ s behavior from humans ; worst-case performance is 100 % . Each judge ( N = 147 ) completed 49 trials with blocked feedback , and judges were analyzed individually and in aggregate . The results are shown in Fig . 6B ( new exemplars ) . Judges had only a 52 % ID level on average for discriminating human versus BPL behavior . As a group , this performance was barely better than chance [ t ( 47 ) = 2.03 , P = 0.048 ] , and only 3 of 48 judges had an ID level reliably above chance . Three lesioned models were evaluated by different groups of judges in separate SCIENCE sciencemag.org 11 DECEMBER 2015 • VOL 350 ISSUE 6266 1335 1 2 1 2 1 2 1 2 1 2 1 2 Human or Machine ? Fig . 5 . Generating new exemplars . Humans and machines were given an image of a novel character ( top ) and asked to produce new exemplars.The nine-character grids in each pair that were generated by a machine are ( by row ) 1 , 2 ; 2 , 1 ; 1 , 1 . RESEARCH | RESEARCH ARTICLES on N ovem ber 21 , 2020 http : //science.sciencem ag.org/ D ow nloaded from for each subpart . Last , parts are roughly positioned to begin either independently , at the beginning , at the end , or along previous parts , as defined by relation Ri ( Fig . 3A , iv ) . Character tokens q ( m ) are produced by execut- ing the parts and the relations andmodeling how ink flows from the pen to the page . First , motor noise is added to the control points and the scale of the subparts to create token-level stroke trajectories S ( m ) . Second , the trajectory ’ s precise start location L ( m ) is sampled from the schematic provided by its relationRi to previous strokes . Third , global transformations are sampled , including an affine warp A ( m ) and adaptive noise parameters that ease probabilistic inference ( 30 ) . Last , a binary image I ( m ) is created by a stochastic rendering function , lining the stroke trajectories with grayscale ink and interpreting the pixel values as independent Bernoulli probabilities . Posterior inference requires searching the large combinatorial space of programs that could have generated a raw image I ( m ) . Our strategy uses fast bottom-up methods ( 31 ) to propose a range of candidate parses . The most promising candidates are refined by using continuous optimization and local search , forming a discrete approximation to the posterior distribution P ( y , q ( m ) |I ( m ) ) ( section S3 ) . Figure 4A shows the set of discovered programs for a training image I ( 1 ) and how they are refit to different test images I ( 2 ) to compute a classification score log P ( I ( 2 ) |I ( 1 ) ) ( the log posterior predictive probability ) , where higher scores indicate that they are more likely to belong to the same class . A high score is achieved when at least one set of parts and relations can successfully explain both the training and the test images , without violating the soft constraints of the learned within-class variability model . Figure 4B compares the model ’ s best-scoring parses with the ground-truth human parses for several characters . Results People , BPL , and alternative models were compared side by side on five concept learning tasks that examine different forms of generalization from just one or a few examples ( example task Fig . 5 ) . All behavioral experiments were run through Amazon ’ s Mechanical Turk , and the experimental procedures are detailed in section S5 . The main results are summarized by Fig . 6 , and additional lesion analyses and controls are reported in section S6 . One-shot classification was evaluated through a series of within-alphabet classification tasks for 10 different alphabets . As illustrated in Fig . 1B , i , a single image of a new character was presented , and participants selected another example of that same character from a set of 20 distinct characters produced by a typical drawer of that alphabet . Performance is shown in Fig . 6A , where chance is 95 % errors . As a baseline , themodifiedHausdorff distance ( 32 ) was computed between centered images , producing 38.8 % errors . People were skilled one-shot learners , achieving an average error rate of 4.5 % ( N = 40 ) . BPL showed a similar error rate of 3.3 % , achieving better performance than adeep convolutional network ( convnet ; 13.5 % errors ) and the HDmodel ( 34.8 % ) —each adapted from deep learning methods that have performed well on a range of computer vision tasks . A deep Siamese convolutional network optimized for this one-shot learning task achieved 8.0 % errors ( 33 ) , still about twice as high as humans or ourmodel . BPL ’ s advantage points to the benefits ofmodeling theunderlying causal process in learning concepts , a strategy different from the particular deep learning approaches examined here . BPL ’ s other key ingredients also make positive contributions , as shown by higher error rates for BPL lesions without learning to learn ( token-level only ) or compositionality ( 11.0 % errors and 14.0 % , respectively ) . Learning to learn was studied separately at the type and token level by disrupting the learned hyperparameters of the generativemodel . Compositionality was evaluated by comparing BPL to a matched model that allowed just one spline-based stroke , resembling earlier analysisby-synthesis models for handwritten characters that were similarly limited ( 34 , 35 ) . The human capacity for one-shot learning is more than just classification . It can include a suite of abilities , such as generating new examples of a concept . We compared the creative outputs produced by humans and machines through “ visual Turing tests , ” where naive human judges tried to identify the machine , given paired examples of human and machine behavior . In our most basic task , judges compared the drawings from nine humans asked to produce a new instance of a concept given one example with nine new examples drawn by BPL ( Fig . 5 ) . We evaluated each model based on the accuracy of the judges , which we call their identification ( ID ) level : Idealmodel performance is 50 % ID level , indicating that they can not distinguish the model ’ s behavior from humans ; worst-case performance is 100 % . Each judge ( N = 147 ) completed 49 trials with blocked feedback , and judges were analyzed individually and in aggregate . The results are shown in Fig . 6B ( new exemplars ) . Judges had only a 52 % ID level on average for discriminating human versus BPL behavior . As a group , this performance was barely better than chance [ t ( 47 ) = 2.03 , P = 0.048 ] , and only 3 of 48 judges had an ID level reliably above chance . Three lesioned models were evaluated by different groups of judges in separate SCIENCE sciencemag.org 11 DECEMBER 2015 • VOL 350 ISSUE 6266 1335 1 2 1 2 1 2 1 2 1 2 1 2 Human or Machine ? Fig . 5 . Generating new exemplars . Humans and machines were given an image of a novel character ( top ) and asked to produce new exemplars.The nine-character grids in each pair that were generated by a machine are ( by row ) 1 , 2 ; 2 , 1 ; 1 , 1 . RESEARCH | RESEARCH ARTICLES on N ovem ber 21 , 2020 http : //science.sciencem ag.org/ D ow nloaded from for each subpart . Last , parts are roughly positioned to begin either independently , at the b ginni g , at the end , or along previous parts , as defined by relation Ri ( Fig . 3A , iv ) . Character tokens q ( m ) are produced by executing the parts and the relations andmodeling how ink flows from the p n to the page . First , motor noise is added to the control points and the scale of the subparts to creat oken-level stroke trajectories S ( m ) . Second , the trajectory ’ s precise star location L ( m ) is sampled from the schematic provided by its relationRi to previous strokes . Third , global transformations are sampled , including an ffine warp A ( m ) and a aptive noise parameters that eas probabilistic inference ( 30 ) . Last , a binary image I ( m ) is creat d by a stochastic rendering function , lining the stroke trajectories with grayscale ink and interpreting the pixel values a independent Bernoulli probabilit es . Posterio inference requires searching the large combinatorial space of programs that could have generated a r w image I ( m ) . Our st a egy uses fa t bott m-up methods ( 31 ) to prop se a r nge of candidate parses . The most promising candidates are refined by using conti uo s optimization and local search , forming a discrete approximation to he posterio distribution P ( y , q ( m ) |I ( m ) ) ( section S3 ) . Figure 4A shows the s t of discovered programs for a training image I ( 1 ) and how they are refit to different tes images I ( 2 ) to compute a classification score log P ( I ( 2 ) |I ( 1 ) ) ( the log posterio p edictive probability ) , where hig er scores indicate hat they are more likely to belong to he same class . A hig score is achieved when at least one s t of parts and relations can successfully explain both e training and the tes images , without violating the soft constraints of the l arned with n-class vari b lity model . Figure 4B compares the model ’ s best-scoring parses with e ground-tru h human parses for several characters . Results People , BPL , and alternative models were compared side by side on five concept learning tasks that examine different forms of generalization from just one or a few examples ( example task Fig . 5 ) . All behavioral experiments were un through Amazon ’ s Mechanical Turk , and the experimental procedures are d tailed in section S5 . The main results are summarized by Fig . 6 , and additional lesion a alyses and controls are - ported in section S6 . One-shot classification was evalu ted through a series of within-alphabet classification tasks for 10 different alphabets . As illustra ed in Fig . 1B , i , a single image of a new character was present d , and participants select d another example of tha same character from a set of 20 distinct characters produced by a typical drawer of tha alphabet . Performance is shown in Fig . 6A , where chance is 95 % erro s. A a b seline , themodifiedHausdorff distance ( 32 ) was computed between cent red images , producing 38.8 % erro s. People were skilled one-shot learners , achiev ng an verage erro rate of 4.5 % ( N = 40 ) . BPL showed a similar erro rate of 3.3 % , achiev ng better perfo mance than deep convolutional etwork ( convnet ; 13.5 % erro s ) and the HDmodel ( 34.8 % ) —each adapted from deep learning methods tha have p rformed well on a range of computer vision tasks . A deep Siamese convolutional network optimized for this one-shot learning task achieved 8.0 % erro s ( 33 ) , still about twice as hig as humans or ourmodel . BPL ’ s advant ge points to the b nefits ofmodeling theunderlying caus l process in learning concepts , a strategy different from the particular deep l arning approaches xamined here . BPL ’ s other key ingredi nts al o make positive contributions , a shown by hig er ro rates for BPL lesions without learning to learn ( token-level only ) or compositionality ( 11.0 % erro s and 14.0 % , resp ctively ) . Learning to learn was studie separately at the type and token level by disrupting the learned hyper a ameters of the generativemodel . Compositionality was evalu ted by comparing BPL to a matched model that allowed just one spline-based stroke , r sembling earlie analysisby-s nthesis models for handwritten characters tha were similarly limited ( 34 , 35 ) . The uman capacity for ne-shot learning is more than just classification . It can include a suite of abilities , such as generating ew xamples of a concept . We compared the creative outp ts produced by humans and machines through “ visual Turing tes s , ” where naive human judges tried to identify the machine , given paired xamples of human d machine b havior . In our most basic task , judges compared the drawings from nine humans a ked to produce a new instance of a concept given o e example with nine w examples drawn by BPL ( Fig . 5 ) We evalu ted ach model based on the accuracy of the judges , whic we call their identification ( ID ) level : Idealmodel perfo mance is 50 % ID level , indicating tha they can not distinguish t e model ’ s behavior from humans ; worst-case p rfo mance is 100 % . Each judge ( N = 147 ) completed 49 trials with blocked feedback , and judges were analyzed individually and in aggre ate . The results are shown i Fig . 6B ( new exemplars ) . Judges had only a 52 % ID level on average for discriminating human versus BPL behavior . As a group , this performance was barely better han chance [ t ( 47 ) = 2.03 , P = 0.048 ] , and only 3 of 48 judges had n ID level r liably above chance . Three lesioned models were evaluated by different groups of judges in separate SCIENCE sciencemag.org 11 DECEMBER 2015 • VOL 350 ISSUE 626 1335 1 2 1 2 1 2 1 2 1 2 1 2 Human or Machine ? Fig . 5 Gen rating ew exemplars . Humans and machines were given a image of a novel character ( top ) and asked to produce n w xemplars.The nine-character grids in each pair tha were g nerated by a machine are ( by row ) 1 , 2 ; , 1 ; 1 , 1 . RES ARCH | RES ARCH ARTICLES on N ovem ber 21 , 2020 http : //science.sciencem ag.org/ D ow nloaded from for each subpart . Last , parts are roughly positioned to begin either indep ndently , at the beginning , at the end , or along previous parts , as d fined by relation Ri ( Fig . 3A , iv ) . Character tokens q ( m ) are produce by executi g the parts and th relations andmodeling how nk flows from the pen to the page . First , motor noise i added to the control points a d the scale of the subparts to create token-lev l stroke trajectori s S ( m ) . Second , the trajectory ’ s precise start locat on L ( m ) is sampl d from the schematic provid by ts relationRi to previous strokes . Third , global transformations are sampled , including an affine warp A ( m ) and adaptive noise parameters that ease probabilistic inference ( 30 ) . Last , a binary image I ( m ) is created by a stochastic rendering function , lining the stroke trajectori s with grayscale ink and interpreting the p xel values as independent Bernoulli probabilities . Posterior inference requires searching the lar e combinatorial space of programs at could have generated a raw image I ( m ) . Our strategy uses fast bottom-up methods ( 31 ) to propose a ra ge of candidate parses . The most promising candidat s are refined by using continuous optimiza ion and local search , forming a discrete approximation to the posterior distribution P ( y , q ( m ) |I ( m ) ) ( section S3 ) . Figure 4A shows the set of discovered programs a training image I ( 1 ) and how they are refit to different test images I ( 2 ) to compute a classific tion score log P ( I ( 2 ) |I ( 1 ) ) ( the log posterior predictive probabil ty ) , where higher scor s indica e that th y are m re likely to belong to the sam class . A high score is achieved when at least one set of parts and relations can successfully explain both the training and the tes images , without violating the soft constraints of the learned within-class variability model . Figure 4B compares the model ’ s best-scoring parses with the ground-truth human parses for several characters . Results People , BPL , and alternative models were compared ide by sid on five concept learning tasks that xamine different forms of generalization f om just o or a few examples ( example task Fig . 5 ) . All behavioral experiments were run through Amazon ’ s Mechanical Turk , and the exp rime tal p ocedures are detailed in section S5 . The main results are summarized by Fig . 6 , and additional lesion analyses and controls are reported in section S6 . One-shot classification was evaluated through a series of within-alphabet classification tasks for 10 different alphabets . As illustrated in Fig . 1B , i , a single image of a new character was presented , and participants selected another example of that same character from a set of 20 distinct characters produced by a typical drawer of that alphabet . Performance is shown in Fig . 6A , where chance is 95 % errors . As a baseline , themodifiedHausdorff distance ( 32 ) was computed between centered images , producing 38.8 % errors . People were skilled one-shot learners , achieving an average error rate of 4.5 % ( N = 40 ) . BPL showed a similar error rate of 3.3 % , achieving better performance than adeep convolutional network ( convnet ; 13.5 % errors ) and the HDmodel ( 34.8 % ) —each adapted from deep learning methods that have performed well on a range of computer vision tasks . A deep Siamese convolutional network optimized for this one-shot learning task achieved 8.0 % errors ( 33 ) , still about twice as high as humans or ourmodel . BPL ’ s advantage points to the benefits ofmodeling theunderlying causal process in learning concepts , a strategy different from the particular deep learning approaches examined here . BPL ’ s other key ingredients also make positive contributions , as shown by higher error rates for BPL lesions without learning to learn ( token-level only ) or compositionality ( 11.0 % errors and 14.0 % , respectively ) . Learning to learn was studied separately at the type and token level by disrupting the learned hyperparameters of the generativemodel . Compositionality was evaluated by comparing BPL to a matched model that allowed just one spline-based stroke , resembling earlier analysisby-synthesis models for handwritten characters that were similarly limited ( 34 , 35 ) . The human capacity for one-shot learning is more than just classification . It can include a suite of abilities , such as generating new examples of a concept . We compared the creative outputs produced by humans and machines through “ visual Turing tests , ” where naive human judges tried to identify the machine , given paired examples of human and machine behavior . In our most basic task , judges compared the drawings from nine humans asked to produce a new instance of a concept given one example with nine new examples drawn by BPL ( Fig . 5 ) . We evaluated each model based on the accuracy of the judges , which we call their identification ( ID ) level : Idealmodel performance is 50 % ID level , indicating that they can not distinguish the model ’ s behavior from humans ; worst-case performance is 100 % . Each judge ( N = 147 ) completed 49 trials with blocked feedback , and judges were analyzed individually and in aggregate . The results are shown in Fig . 6B ( new exemplars ) . Judges had only a 52 % ID level on average for discriminating human versus BPL behavior . As a group , this performance was barely better than chance [ t ( 47 ) = 2.03 , P = 0.048 ] , and only 3 of 48 judges had an ID level reliably above chance . Three lesioned models were evaluated by different groups of judges in separate SCIENCE sciencemag.org 11 DECEMBER 2015 • VOL 350 ISSUE 6266 1335 1 2 1 2 1 2 1 2 1 2 1 2 Human or Machin ? Fig . 5 . Generating new exemplars . Humans and machines were given an image of a novel character ( top ) and asked to produce new exemplars.The nine-character grids in each pair that were generated by a machine are ( by row ) 1 , 2 ; 2 , 1 ; 1 , 1 . RESEARCH | RESEARCH ARTICLES on N ovem ber 21 , 2020 http : //science.sciencem ag.org/ D ow nloaded from BPL ( a ) Generating new exemplars GNS BPL ( b ) Generating new concepts Figure 4 : Generation tasks . ( a ) GNS produced 9 new exemplars for each of 5 target images , plotted here next to human and BPL productions . ( b ) A grid of 36 new character concepts sampled unconditionally from GNS , shown next to BPL samples . Generating new concepts ( unconstrained ) . In addition to generating new exemplars of a particular concept , GNS can generate new character concepts altogether , unconditioned on training images . Whereas the BPL model us s a compli ated proc dure for uncon itional ge ration that involves a preliminary inference step and a supplemental nonparametric model , GNS generates new concepts by sampling directly from the type prior P ( ψ ) . Moreover , the resulting GNS productions capture more of the structure found in real characters than either the raw BPL prior ( Fig 1 , 4b ) or the supplemental nonparametric BPL prior [ 29 ] . In Fig . 4b we show a grid of 36 new character concepts sampled from our generative model at reduced temperature setting T = 0.5 ( Appendix A.2 ) . The model produces characters in multiple distinct styles , with some having more angular , line-based structure and others relying on complex curves . In Appendix Fig . A14 , we show a larger set of characters sampled from GNS , plotted in a topologically-organized grid alongside a corresponding g id of “ ne r st neig bor ” training examples . In many cases , samples from the model have a distinct style and are visually dissimilar from their nearest Omniglot neighbor . Marginal image likelihoods . As a final evaluation , we computed likelihoods of held-out character images by marginalizing over the latent type and token variables of GNS to estimate P ( I ) = ∫ P ( ψ , θ , I ) ∂ψ∂θ . We hypothesized that our causal generative model of character concepts would yield better test likelihoods compared to deep generative models trained directly on image pixels . As detailed in Appendix D , under the minimal assump- tion that our K posterior parses represent sharply peaked modes of the joint density , we can obtain an approximate lower bound on the marginal P ( I ) by using Laplace ’ s method to estimate the integral around each mode and summing the resulting integrals . In Table 3 , we report average log-likelihood ( LL ) bounds obtained from GNS for a random subset of 1000 evaluation images , compared against test LL bounds from both the SG [ 43 ] and the VHE [ 21 ] models . Our GNS model performs stronger than each alternative , reporting the best overall log-likelihood per dimension . 6 DISCUSSION We introduced a new generative neuro-symbolic ( GNS ) model for learning flexible , task-general representations of character concepts . We demonstrated GNS on the Omniglot challenge , showing that it performs a variety of inductive tasks in ways difficult to distinguish from human behavior . Published as a conference paper at ICLR 2021 Some evaluations were still qualitative , and future work will further quantify these results using Visual Turing Tests [ 29 ] . Whereas many machine learning algorithms emphasize breadth of data domains , isolating just a single task across datasets , we have focused our efforts in this paper on a single domain , emphasizing depth of the representation learned . Human concept learning is distinguished for having both a breadth and depth of applications [ 37 , 31 ] , and ultimately , we would like to capture both of these unique qualities . We see our character model as belonging to a broader class of generative neuro-symbolic ( GNS ) models for capturing the data generation process . We have designed our model based on general principles of visual concepts—namely , that concepts are composed of reusable parts and locations—and we describe how it generalizes to 3D object concepts in Appendix E. As in the human mind , machine learning practitioners have far more prior knowledge about some domains vs. others . Handwritten characters is a domain with strong priors [ 1 , 33 , 23 ] , implemented directly in the human mind and body . For concepts like these with more explicit causal knowledge , it is beneficial to include priors about how causal generative factors translate into observations , as endowed to our character model through its symbolic rendering engine . For other types of concepts where these processes are less clear , it may be appropriate to use more generic neural networks that generate concepts and parts directly as raw stimuli , using less symbolic machinery and prior knowledge . We anticipate that GNS can flexibly model concepts in both types of domains , although further experiments are needed to demonstrate this . Our current token model for character concepts is much too simple , and we acknowledge a few important shortcomings . First , as shown in Appendix Fig . A10 , there are a number of scenarios in which the parses from a training character can not adequately refit to a new example of the same character without a token model that allows for changes to discrete variables . By incorporating this allowance in future work , we hope to capture more knowledge in this domain and further improve performance . Furthermore , although our vision for GNS is to represent both concepts and background knowledge with neuro-symbolic components , the current token-level model uses only simple parametric distributions . In future work , we hope to incorporate token-level models that use neural network sub-routines , as in the type-level model presented here . REFERENCES [ 1 ] M. K. Babcock and J. Freyd . Perception of dynamic information in static handwritten forms . American Journal of Psychology , 101 ( 1 ) :111–130 , 1988 . [ 2 ] M. Botvinick , D. Barrett , P. Battaglia , N. de Freitas , D. Kumaran , and et al . Building machines that learn and think for themselves . Behavioral and Brain Sciences , 40 : e255 , 2017 . [ 3 ] J. Devlin , J. Uesato , S. Bhupatiraju , R. Singh , A.-R. Mohamed , and P. Kohli . Robustfill : Neural program learning under noisy i/o . In ICML , 2017 . [ 4 ] K. Ellis , D. Ritchie , A. Solar-lezama , and J . B. Tenenbaum . Learning to infer graphics programs from hand-drawn images . In NeurIPS , 2018 . [ 5 ] S. M. A. Eslami , N. Heess , T. Weber , Y. Tassa , D. Szepesvari , K. Kavukcuoglu , and G. E. Hinton . Attend , infer , repeat : Fast scene understanding with generative models . In NeurIPS , 2016 . [ 6 ] R. Feinman and B. M. Lake . Generating new concepts with hybrid neuro-symbolic models . In CogSci , 2020 . [ 7 ] C. Finn , P. Abbeel , and S. Levine . Model-agnostic meta-learning for fast adaptation of deep networks . In ICML , 2017 . [ 8 ] Y. Ganin , T. Kulkarni , I. Babuschkin , S. M. A. Eslami , and O. Vinyals . Synthesizing programs for images using reinforced adversarial learning . In ICML , 2018 . [ 9 ] V. Garcia and J. Bruna . Few-shot learning with graph neural networks . In ICLR , 2018 . [ 10 ] A. Gelman , J . B. Carlin , H. S. Stern , D. B. Dunson , A. Vehtari , and D. B. Rubin . Bayesian Data Analysis ( 3rd ed. ) . CRC Press , Boca Raton , FL , 2014 . [ 11 ] S. Geman , E. Bienenstock , and R. Doursat . Neural networks and the bias/variance dilemma . Neural Computation , 4 ( 1 ) :1–58 , 1992 . Published as a conference paper at ICLR 2021 [ 12 ] D. George , W. Lehrach , K. Kansky , M. Lázaro-Gredilla , C. Laan , B. Marthi , X. Lou , and et al . A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs . Science , 358 ( 6368 ) , 2017 . [ 13 ] N. D. Goodman , J . B. Tenenbaum , J. Feldman , and T. L. Griffiths . A rational analysis of rule-based concept learning . Cognitive Science , 32:108–154 , 2008 . [ 14 ] N. D. Goodman , T. D. Ullman , and J . B. Tenenbaum . Learning a theory of causality . Psychological Review , 118 ( 1 ) :110–119 , 2011 . [ 15 ] N. D. Goodman , J . B. Tenenbaum , and T. Gerstenberg . Concepts in a probabilistic language of thought . In E. Margolis and S. Laurence ( ed . ) , The conceptual mind : New directions in the study of concepts , pp . 623–653 . MIT Press , Cambridge , MA , 2015 . [ 16 ] A. Graves . Generating sequences with recurrent neural networks . arXiv preprint arXiv:1308.0850 , 2013 . [ 17 ] A. Graves , G. Wayne , and I. Danihelka . Neural turing machines . arXiv preprint arXiv:1410.5401 , 2014 . [ 18 ] K. Greff , R. L. Kaufman , R. Kabra , N. Watters , C. Burgess , D. Zoran , L. Matthey , M. Botvinick , and A. Lerchner . Multi-object representation learning with iterative variational inference . In ICML , 2019 . [ 19 ] K. Gregor , I. Danihelka , A. Graves , D. J. Rezende , and D. Wierstra . DRAW : A recurrent neural network for image generation . In ICML , 2015 . [ 20 ] D. Ha and D. Eck . A neural representation of sketch drawings . In ICLR , 2018 . [ 21 ] L. B. Hewitt , M. I. Nye , A. Gane , T. Jaakkola , and J . B. Tenenbaum . The Variational Homoencoder : Learning to learn high capacity generative models from few examples . In UAI , 2018 . [ 22 ] L. B. Hewitt , T. A . Le , and J . B. Tenenbaum . Learning to learn generative programs with Memoised Wake-Sleep . In UAI , 2020 . [ 23 ] K. H. James and I. Gauthier . When writing impairs reading : Letter perception ’ s susceptibility to motor interference . Journal of Experimental Psychology : General , 138 ( 3 ) :416–31 , 2009 . [ 24 ] C. Kemp and J . B. Tenenbaum . Structured statistical models of inductive reasoning . Psychological Review , 116:20–58 , 2009 . [ 25 ] A. R. Kosiorek , S. Sabour , Y. W. Teh , and G. E. Hinton . Stacked Capsule Autoencoders . In NeurIPS , 2019 . [ 26 ] T. D. Kulkarni and et al . Picture : A probabilistic programming language for scene perception . In CVPR , 2015 . [ 27 ] B. M. Lake and M. Baroni . Generalization without systematicity : On the compositional skills of sequence-to-sequence recurrent networks . In ICML , 2018 . [ 28 ] B. M. Lake and S. T. Piantadosi . People infer recursive visual concepts from just a few examples . Computational Brain & Behavior , 2019 . [ 29 ] B. M. Lake , R. Salakhutdinov , and J . B. Tenenbaum . Human-level concept learning through probabilistic program induction . Science , 350:1332–1338 , 2015 . [ 30 ] B. M. Lake , T. D. Ullman , J . B. Tenenbaum , and S. J. Gershman . Building machines that learn and think like people . Behavioral and Brain Sciences , 40 : E253 , 2017 . [ 31 ] B. M. Lake , R. Salakhutdinov , and J . B. Tenenbaum . The Omniglot challenge : A 3-year progress report . Behavioral Sciences , 29:97–104 , 2019 . [ 32 ] Y. LeCun , Y. Bengio , and G. Hinton . Deep learning . Nature , 521 ( 7553 ) :436–444 , 2015 . [ 33 ] M. Longcamp , J. L. Anton , M. Roth , and J. L. Velay . Visual presentation of single letters activates a premotor area involved in writing . Neuroimage , 19 ( 4 ) :1492–1500 , 2003 . [ 34 ] J. Mao , C. Gan , P. Kohli , J . B. Tenenbaum , and J. Wu . The neuro-symbolic concept learner : Interpreting scenes , words , and sentences from natural supervision . In ICLR , 2019 . [ 35 ] G. F. Marcus . The Algebraic Mind : Integrating Connectionism and Cognitive Science . MIT Press , Cambridge , MA , 2003 . Published as a conference paper at ICLR 2021 [ 36 ] J. L. McClelland , M. M. Botvinick , D. C. Noelle , D. C. Plaut , T. T. Rogers , M. S. Seidenberg , and L. B. Smith . Letting structure emerge : Connectionist and dynamical systems approaches to cognition . Trends in Cognitive Science , 14:348–356 , 2010 . [ 37 ] G. L. Murphy . The big book of concepts . MIT Press , Cambridge , MA , 2002 . [ 38 ] G. L. Murphy and D. L. Medin . The role of theories in conceptual coherence . Psychological Review , 92 ( 3 ) :289–316 , 1985 . [ 39 ] M. I. Nye , A. Solar-Lezama , J . B. Tenenbaum , and B. M. Lake . Learning compositional rules via neural program synthesis . arXiv preprint arXiv:2003.05562 , 2020 . [ 40 ] A. Perfors , J . B. Tenenbaum , and T. Regier . The learnability of abstract syntactic principles . Cognition , 118 ( 3 ) :306–338 , 2011 . [ 41 ] S. T. Piantadosi , J . B. Tenenbaum , and N. D. Goodman . The logical primitives of thought : Empirical foundations for compositional cognitive models . Psychological Review , 2016 . [ 42 ] S. Reed and N. de Freitas . Neural programmer-interpreters . In ICLR , 2016 . [ 43 ] D. J. Rezende , S. Mohamed , I. Danihelka , K. Gregor , and D. Wierstra . One-Shot generalization in deep generative models . In ICML , 2016 . [ 44 ] P. Shyam , S. Gupta , and A. Dukkipati . Attentive recurrent comparators . In ICML , 2017 . [ 45 ] J. Snell , K. Swersky , and R. Zemel . Prototypical networks for few-shot learning . In NeurIPS , 2017 . [ 46 ] A. Stuhlmuller , J . B. Tenenbaum , and N. D. Goodman . Learning Structured Generative Concepts . In CogSci , 2010 . [ 47 ] J . B. Tenenbaum , C. Kemp , T. L. Griffiths , and N. D. Goodman . How to grow a mind : Statistics , structure , and abstraction . Science , 331 ( 6022 ) :1279–1285 , 2011 . [ 48 ] L. Valkov , D. Chaudhari , A. Srivastava , C. Sutton , and S. Chaudhuri . HOUDINI : Lifelong learning as program synthesis . In NeurIPS , 2018 . [ 49 ] O. Vinyals , C. Blundell , T. Lillicrap , and D. Wierstra . Matching networks for one shot learning . In NeurIPS , 2016 . [ 50 ] K. Yi , J. Wu , C. Gan , A. Torralba , P. Kohli , and J . B. Tenenbaum . Neural-symbolic VQA : Disentangling reasoning from vision and language understanding . In NeurIPS , 2018 . Published as a conference paper at ICLR 2021 A GENERATIVE MODEL ψ θ I M Type Token Image Figure A5 : The GNS hierarchical generative model . The full hierarchical generative model of GNS is depicted in Fig . A5 . The joint density for type ψ , token θ ( m ) , and image I ( m ) factors as P ( ψ , θ ( m ) , I ( m ) ) = P ( ψ ) P ( θ ( m ) |ψ ) P ( I ( m ) |θ ( m ) ) . ( 6 ) The type ψ parameterizes a motor program for generating character tokens θ ( m ) , unique exemplars of the concept . Both ψ and θ ( m ) are expressed as causal drawing parameters . An image I ( m ) is obtained from token θ ( m ) by rendering the drawing parameters and sampling binary pixel values . A.1 TRAINING ON CAUSAL DRAWING DATA original stroke minimal spline Figure A6 : Spline representation . Raw strokes ( left ) are converted into minimal splines ( right ) using least-squares optimization . Crosses ( left ) indicate pen locations and red dots ( right ) indicate spline control points . To learn the parameters of P ( ψ ) and P ( θ ( m ) | ψ ) , we fit our models to the human drawing data from the Omniglot background set . In this drawing data , a character is represented as a variable-length sequence of strokes , and each stroke is a variable-length sequence of pen locations { z1 , ... , zT } , with zt ∈ R2 ( Fig . A6 , left ) . Before training our model on background drawings , we convert each stroke into a minimal spline representation using least-squares optimization ( Fig . A6 , right ) , borrowing the B-spline tools from [ 29 ] . The number of spline control points depends on the stroke complexity and is determined by a residual threshold . Furthermore , we removed small strokes using a threshold on the trajectory length . These processing steps help suppress noise and emphasize signal in the drawings . Our generative models are trained to produce character drawings , where each drawing is represented as an ordered set of splines ( strokes ) . The number of strokes , and the number of spline coordinates per stroke , are allowed to vary in the model . A.2 TYPE PRIOR The type prior P ( ψ ) represents a character as a sequence of strokes , with each stroke decomposed into a starting location yi ∈ R2 , conveying the first spline control point , and a stroke trajectory xi = { ∆1 , ... , ∆N } , conveying deltas between spline control points . It generates character types one stroke at a time , using a symbolic rendering procedure called frender as an intermediate processing step after forming each stroke . An image canvas C is used as a memory state to convey information about previous strokes . At each step i , the next stroke ’ s starting location and trajectory are sampled with procedure GeneratePart . In this procedure , the current image canvas C is first read by the location model ( Fig . 2 ) , a convolutional neural network ( CNN ) that processes the image and returns a Published as a conference paper at ICLR 2021 probability distribution for starting location yi : yi ∼ p ( yi | C ) . The starting location yi is then passed along with the image canvas C to the stroke model , a Long Short-Term Memory ( LSTM ) architecture with a CNN-based image attention mechanism.The stroke model samples the next stroke trajectory xi sequentially one offset at a time , selectively attending to different parts of the image canvas at each sample step and combining this information with the context of yi : xi ∼ p ( xi | yi , C ) . After GeneratePart returns , the stroke parameters yi , xi are rendered to produce an updated canvas C = frender ( yi , xi , C ) . The new canvas is then fed to the termination model , a CNN architecture that samples a binary termination indicator vi : vi ∼ p ( vi | C ) . Both our location model and stroke model follow a technique from [ 16 ] , who proposed to use neural networks with mixture outputs to model handwriting data . Parameters { π1 : K , µ1 : K , σ1 : K , ρ1 : K } output by our network specify a Gaussian mixture model ( GMM ) with K components ( Fig . 2 ; colored ellipsoids ) , where πk ∈ ( 0 , 1 ) is the mixture weight of the kth component , µk ∈ R2 its means , σk ∈ R2+ its standard deviations , and ρk ∈ ( −1 , 1 ) its correlation . In our location model , a single GMM describes the distribution p ( yi | C ) . In our stroke model , the LSTM outputs one GMM at each timestep , describing p ( ∆t|∆1 : t−1 , yi , C ) . The termination model CNN has no mixture outputs ; it predicts a single Bernoulli probability to sample binary variable vi . When sampling from the model at test time , we use a temperature parameter proposed by Ha & Eck [ 20 ] ( see [ 20 , Eq . 8 ] ) to control the entropy of the mixture density outputs . A.3 TOKEN MODEL procedure GENERATETOKEN ( ψ ) { κ , y1 : κ , x1 : κ } ← ψ . Unpack type-level variables for i = 1 ... κ do y ( m ) i ∼ P ( y ( m ) i | yi ) . Sample token-level location x ( m ) i ∼ P ( x ( m ) i | xi ) . Sample token-level part A ( m ) ∼ P ( A ( m ) ) . Sample affine warp transformation θ ← { y ( m ) 1 : κ , x ( m ) 1 : κ , A ( m ) } return θ . Return concept token Figure A7 : Token model sampling procedure . Character types ψ are used to parameterize the procedure GenerateToken ( ψ ) , a probabilistic program representation of token model P ( θ ( m ) | ψ ) . The psuedo-code of this sampling procedure is provided in Fig . A7 . The location model P ( y ( m ) i | yi ) and part model P ( x ( m ) i | xi ) are each zeromean Gaussians , with standard deviations fit to the background drawings following the procedure of Lake et al . [ 29 ] ( see SM 2.3.3 ) . The location model adds noise to the start of each stroke , and the part model adds isotropic noise to the 2d cooridnates of each spline control point in a stroke . In the affine warp A ( m ) ∈ R4 , the first two dimensions control global re-scaling of spline coordinates , and the second two control a global translation of the center of mass . The distribution is P ( A ( m ) ) = N ( [ 1 , 1 , 0 , 0 ] , ΣA ) , ( 7 ) with the parameter ΣA similarly fit from background drawings ( see SM 2.3.4 in [ 29 ] ) . B APPROXIMATE POSTERIOR To obtain parses { ψ , θ } 1 : K for our approximate posterior ( Eq . 3 ) given an image I , we follow the high-level strategy of Lake et al . [ 29 ] , using fast bottom-up search followed by discrete selection and continuous optimization . The algorithm proceeds by the following steps . 13 Published as a conference paper at ICLR 2021 Step 1 : Propose a range of candidate parses with fast bottom-up methods . The bottom-up algorithm extracts an undirected skeleton graph from the character image and uses random walks on the graph to propose a range of candidate parses . There are typically about 10-100 proposal parses , depending on character complexity ( Fig . A8 ) . input Figure A8 : The initial “ base ” parses proposed for an image with skeleton extraction and random walks . Step 2 : Select stroke order and stroke directions for each parse using exhaustive search with the type prior P ( ψ ) . Random search is used for complex parses with large configuration spaces . Step 3 : Score each of the proposal parses using type prior P ( ψ ) and select the top-K parses . We use K = 5 following previous work [ 29 ] . Step 4 : Separate each parse into type and token { ψ , θ } and optimize the continuous type- and token-level parameters with gradient descent to maximize the full joint density P ( ψ , θ , I ) of Eq . 1 . Step 5 : Compute weights π1 : K for each parse by computing π̃k = P ( ψk , θk , I ) and normalizing πk = π̃k/ ∑K k=1 π̃k . C INFERENCE FOR GENERATING NEW EXEMPLARS When generating new exemplars , we are given a single image I ( 1 ) of a novel class and asked to generate new instances I ( 2 ) ( overloading the parenthesis notation from classification ) . To perform this task with GNS , we first sample from our approximate posterior Q ( ψ , θ | I ( 1 ) ) to obtain parse { ψ , θ } ( see Eq . 3 ) , and then re-sample token parameters θ from our token model P ( θ ( 2 ) | ψ ) . Due to high-dimensional images , mass in the approximate posterior often concentrates on the single best parse . To model the diversity seen in different human parses , we apply a temperature parameter to the log of unnormalized parse weights log ( π̃′k ) = log ( π̃k ) /T before normalization , selecting T = 8 for our experiments . With updated weights π′1 : K our sampling distribution is written as P ( I ( 2 ) , θ ( 2 ) | I ( 1 ) ) ≈ K∑ k=1 π′kP ( I ( 2 ) | θ ( 2 ) ) P ( θ ( 2 ) | ψk ) . ( 8 ) D MARGINAL IMAGE LIKELIHOODS Let z = ψ ∪ θ be a stand-in for the joint set of type- and token-level random variables in our GNS generative model . The latent z includes both continuous and discrete variables : the number of strokes κ and the number of control points per stroke d1 : κ are discrete , and all remaining variables are continuous . Decomposing z into its discrete variables zD ∈ ΩD and continuous variables zC ∈ ΩC , the marginal density for an image I is written as P ( I ) = ∑ zD∈ΩD ∫ P ( I , zD , zC ) ∂zC . ( 9 ) Published as a conference paper at ICLR 2021 For any subset Ω̃D ⊂ ΩD of the discrete domain , the following inequality holds : P ( I ) ≥ ∑ zD∈Ω̃D ∫ P ( I , zD , zC ) ∂zC . ( 10 ) Our approximate posterior ( Eq . 3 ) gives us K parses that represent promising modes { zD , zC } 1 : K of the joint density P ( I , zD , zC ) for an image I , and by setting Ω̃D = { zD } 1 : K to be the set of K unique discrete configurations from our parses , we can compute the lower bound of Eq . 10 by computing the integral ∫ P ( I , zD , zC ) ∂zC at each of these zD . At each zDk ∈ { zD } 1 : K , the log-density function f ( zC ) = log P ( I , zDk , zC ) has a gradient-free maximum at zCk , the continuous configuration of the corresponding posterior parse . These maxima were identified by our gradient-based continuous optimizer during parse selection ( Appendix B ) . If we assume that these maxima are sharply peaked , then we can use Laplace ’ s method to estimate the integral ∫ P ( I , zDk , zC ) ∂zC at each zDk . Laplace ’ s method uses Taylor expansion to approximate the integral of ef ( x ) for a twice-differentiable function f around a maximum x0 as∫ ef ( x ) ∂x ≈ ef ( x0 ) ( 2π ) d 2 | −Hf ( x0 ) | 12 , ( 11 ) where x ∈ Rd andHf ( x0 ) is the Hessian matrix of f evaluated at x0 . Our log-density function f ( zC ) is fully differentiable w.r.t . continuous parameters zC , therefore we can compute H ( zC ) = ∂2f/∂z2C with ease . Our approximate lower bound on P ( I ) is therefore written as the sum of Laplace approximations at our K parses : P ( I ) ≥ K∑ k=1 ∫ P ( I , zDk , zC ) ∂zC ≈ K∑ k=1 P ( I , zDk , zCk ) ( 2π ) d 2 | −H ( zCk ) | 1 2 ( 12 ) E APPLYING GNS TO 3D OBJECT CONCEPTS The GNS modeling framework is designed to capture inductive biases for concept learning that generalize across different types of visual concepts . We are actively working on applying GNS as a task-general generative model of 3D object concepts , such as chairs and vehicles , again with a neuro-symbolic generative process for parts and relations . Here , we briefly review the path forward for training GNS models of object concepts . procedure GENERATETYPE C 0 . Initialize blank image canvas while true do [ yi , xi ] GENERATEPART ( C ) . Sample part location & parameters C frender ( yi , xi , C ) . Render part to image canvas vi ⇠ p ( v | C ) . Sample termination indicator if vi then break . Terminate sample { , y1 : , x1 : } return . Return concept type 1 procedure GENERATETOKEN C 0 . Initialize blank 3d canvas while true do xi GENERATEPART ( C ) . Sample part from neural net C fupdate ( xi , C ) . Update 3d canvas vi ⇠ p ( v | C ) . Sample termination indicator if vi then break . Terminate sample ✓ { , x1 : } return . Return concept token 1 procedure GENERATETYPE C 0 . Initialize blank image canvas while true do [ yi , xi ] GENERATEPART ( C ) . Sample part location & parameters C frender ( yi , xi , C ) . Render part to image canvas vi ⇠ p ( v | C ) . Sample termination indicator if vi then break . Terminate sample { , y1 : , x1 : } return . Return concept type category model p ( k |C ) instance model p ( x |k , C ) C p ( k ∣ C ) C q ( z ∣ x , k , C ) zk p ( x |z ) x p k Ck p ( z |k , C ) x generate token se at leg s ba ck re st Figure A9 : A GNS model for the concept of a chair . Everyday 3D object concepts have more within-class variability than handwritten characters ; for example , different chair tokens vary in the number of arm rests , legs , etc . much as different handwritten character types vary in their parts , although both domains are characterized by similar structural 15 Published as a conference paper at ICLR 2021 and statistical considerations . To address these differences within GNS , we shift up one level in the generative hierarchy and design a token-level model for generating new exemplars of an individual concept ( chairs , cars , etc . ) that mirrors our type-level model for characters . The architecture and sampling procedure of GNS for 3D object tokens is given in Fig . A9 . As with characters , our object model produces samples one part at a time , using a 3D canvas C in place of the previous 2D canvas . The procedure GeneratePart consists of two neural network components : 1 ) a discriminitivelytrained category model p ( k | C ) that predicts the category label k of the next part given the current canvas ( leg , arm , back , etc . ) , and 2 ) a generative instance model p ( x | k , C ) that is trained with a variational autoencoder objective to sample an instance of the next part x given the current canvas and predicted part category label . Objects and object parts are represented as 3D voxel grids , and all neural modules , including the category model and the encoder/decoder of the instance model , are parameterized by 3D convolutional neural networks . A function fupdate is used to update the current canvas with the most recent part by summing the voxel grids . A GNS model for a particular concept is trained on examples of the 3D voxel grids with semantic part labels . F.4 GENERATING NEW CONCEPTS ( UNCONSTRAINED ) In Fig . A14 we show a grid of 100 new character concepts produced by GNS , plotted alongside a corresponding grid of “ nearest neighbor ” training examples . Published as a conference paper at ICLR 2021 I ( c ) I ( T ) Score : -618.6 I ( c ) I ( T ) Score : -1624.5 I ( c ) I ( T ) Score : -269.4 I ( c ) I ( T ) Score : -281.1 I ( c ) I ( T ) Score : -269.1 I ( c ) I ( T ) Score : -381.6 I ( c ) I ( T ) Score : -820.9 I ( c ) I ( T ) Score : -505.6 I ( c ) I ( T ) Score : -403.6 I ( c ) I ( T ) Score : -814.4 I ( c ) I ( T ) Score : -742.5 I ( c ) I ( T ) Score : -545.6 Figure A10 : Classification fits . Each row corresponds to one classification trial ( one test image ) . The first column shows parses from the correct training image re-fit to the test example , and the second column parses from an incorrect training image . The two-way score for each train-test pair is shown above the grid , and the model ’ s selected match is emboldened . The 4th and 6th row here are misclassified trials . Published as a conference paper at ICLR 2021 Published as a conference paper at ICLR 2021 Figure A12 : Generating new exemplars with GNS . Twelve target images are highlighted in red boxes . For each target image , the GNS model sampled 9 new exemplars , shown in a 3x3 grid under the target . One-shot Generalization in Deep Generative Models Figure 8 . Unconditional samples for 52 × 52 omniglot ( task 1 ) . For a video of the generation process , see https : //www.youtube.com/ watch ? v=HQEI2xfTgm4 Figure 9 . Generating new examplars of a given character for the weak generalization test ( task 2a ) . The first row shows the test images and the next 10 are one-shot samples from the model . 3 . Representative samples from a novel alphabet . This task corresponds to figure 7 in Lake et al . ( 2015 ) , and conditions the model on anywhere between 1 to 10 samples of a novel alphabet and asks the model to generate new characters consistent with this novel alphabet . We show here the hardest form of this test , using only 1 context image . This test is highly subjective , but the model generations in figure 11 show that it is able to pick up common features and use them in the generations . We have emphasized the usefulness of deep generative models as scalable , general-purpose tools for probabilistic reasoning that have the important property of one-shot generalization . But , these models do have limitations . We have already pointed to the need for reasonable amounts of data . Another important consideration is that , while our models can perform one-shot generalization , they do not perform one-shot learning . One-shot learning requires that a model is updated after the presentation of each new input , e.g. , like the non-parametric models used by Lake et al . ( 2015 ) or Salakhutdinov et al . ( 2013 ) . Parametric models such as ours require a gradient update of the parameters , which we do not do . Instead , our model performs a type of one-shot inference that during test time can perform inferential tasks on new data points , such as missing data completion , new exemplar generation , or analogical sampling , but does not learn from these points . This distinction between one-shot learning and inference is important and affects how such models can be used . We aim to extend our approach to the online and one-shot learning setting in future . 30-20 40-10 45-5 Figure 10 . Generating new examplars of a given character for the strong generalization test ( task 2b , c ) , with models trained with different amounts of data . Left : Samples from model trained on 30-20 train-test split ; Middle : 40-10 split ; Right : 45-5 split ( right ) Figure 11 . Generating new exemplars from a novel alphabet ( task 3 ) . The first row shows the test images , and the next 10 rows are one-shot samples generated by the model . 6 . Conclusion We have developed a new class of general-purpose models that have the ability to perform one-shot generalization , emulating an important characteristic of human cognition . Sequential generative models are natural extensions of variational auto-encoders and provide state-of-the-art models for deep density estimation and image generation . The models specify a sequential process over groups of latent variables that allows it to compute the probability of data points over a number of steps , using the principles of feedback and attention . The use of spatial attention mechanisms substantially improves the ability of the model to generalize . The spatial transformer is a highly flexible attention mechanism for both reading and writing , and is now our default mechanism for attention in generative models . We highlighted the one-shot generalization ability of the model over a range of tasks that showed that the model is able to generate compelling and diverse samples , having seen new examples just once . However there are limitations of this approach , e.g. , still needing a reasonable amount of data to avoid overfitting , which we hope to address in future work . ( a ) SG Figure 4 : 5-shot samples generated by each model ( more in Supplement ) . With a PixelCNN architecture , both Neural Statistician and Resample objectives lead to underutilisation of the latent space , producing unfaithful samples . For the hierarchical PixelCNN architecture , however , significant differences arise between training objectives . In this case , a Neural Statistician learns a strong global distribution over images but makes only minimal use of latent variables c. This means that , despite the use of a higher capacity model , classification accuracy is much poorer ( 66 % ) than that achieved using a deconvolutional architecture . For the same reason , conditional samples display an improved sharpness but are no longer identifiable to the cue images on which they were conditioned ( Figure 4 ) . Our careful training suggests that this is not an optimisation difficulty but is core to the objective , as discussed in Chen et al . ( 2016 ) . By contrast , a VHE is able to gain a large benefit from the hierarchical PixelCNN architecture , with a 3-fold reduction in classification error ( 5-shot accuracy 98.8 % ) and conditional samples which are simultaneously sharp and identifiable ( Figure 4 ) . This improvement is in part achieved by increased utilisation of the latent space , due to rescaling of the KL divergence term in the objective . However , our results show that this common technique is insufficient when used alone , leading to overfitting to cue images with an equally severe impairment of classi- fication performance ( accuracy 62.8 % ) . Rather , we find that KL-rescaling and data resampling must be used tog ther in order for the benefit of the powerful PixelCNN architecture to be re lis d. Table 2 lists the classification accuracy achieved by VHEs with both |D| = 1 and |D| = 5 , as compared to existing d ep learning appr aches . We find t at both network ar not only state- f-the-art mongst deep generative models , but also competitive against the best discriminative models trained directly for few-shot classification . Unlike these discriminative models , VHE is also able to generate new images of a character in one shot , producing samples which are both realistic and faithful to the class of the cue image ( Figure 5 ) . As our goal is to model shared structure acr ss images , we evaluate generative performance using joint log like- Figure 5 : One-shot same-class samples generated by our model . Cue images were sampled from previously unseen classes . lihood of the entire Omniglot test set ( rather than separately across images ) . From this perspective , a single element VAE will perform poorly as it treats all datapoints as independent , optimising a sum over log likelihoods for each element . By sharing latents across elements of the same class , a VHE can improve upon this considerably . For likelihood evaluation , our most appropriate comparison is with Generative Matching Networks ( Bartunov & Vetrov , 2016 ) as they also model dependencies within a class . Thus , we trained models under the same train/test split as them , with no data augmentation . We evaluate the joint log likelihood of full character classes from the test set , normalised by the number of elements , using importance weighting with k=500 samples from q ( c ; X ) . As can be seen in Tables 3 and 4 , our hierarchical PixelCNN architecture is able to achieve state-of-the-art log likelihood results only when trained using the VHE objective . Figure 4 : 5-shot samples generated by each model ( more in Supplement ) . With a PixelCNN architecture , both Neural Statistician and Resample objectives lead to underutilisation of the latent space , producing unfaithful samples . For the hierarchical PixelCNN architecture , however , significant differences arise between training objectives . In this case , a Neural Statistician learns a strong global distribution over images but makes only minimal use of latent variables c. This means that , despite the use of a higher capacity model , classification accuracy s much poorer ( 66 % ) than that achieved using a deconvolutional architecture . For the same reason , conditional samples display an improved sharpness but are no longer identifiable to the cue images on which they were conditioned ( Figure 4 ) . Our care ul training sug ests that this is not an optimisation difficulty but is core t the objective , as discussed in Chen et al . ( 2016 ) . By contrast , a VHE is able to gain a large benefit from the hierarchical PixelCNN architecture , with a 3-fold reduction in classification error ( 5-shot accuracy 98.8 % ) and conditional samples which are simultaneously sharp and identifiable ( Figure 4 ) . This improvement is in part achieved by increased utilisation of the latent space , due to rescaling of the KL divergence term in the objective . However , our results show that this common technique is insufficient when used alone , leading to overfitting to cue images with an equally severe impair ent of classification performance ( accuracy 62.8 % ) . Rather , we find that KL-rescaling and data resampling must be used together in order for the benefit of the powerful PixelCNN architecture to be realised . Table 2 lists the classification accuracy achieved by VHEs with both |D| = 1 and |D| = 5 , as compared to existing deep learning approaches . We find that both networks are not only state-of-the-art amongst deep generative models , but also competitive against the best discriminative models trained directly for few-shot classi- fication . Unlike these discriminative models , a VHE is also able to generate new images of a character in one shot , producing samples which are both realistic and faithful to the class of the cue image ( Figure 5 ) . As our goal is to model shared structure across images , we evaluate generative performance usi g joint log like- Figure 5 : One-shot same-class samples generated by our model . Cue images were sampled from previously un- seen classes . lihood of the entire Omniglot test set ( rather than separately across images ) . From this perspective , a single element VAE will perform poorly as it treats all datapoints as independent , optimising a sum over log likelihoods for each element . By sharing latents across elements of the same class , a VHE can improve upon this considerably . For likelihood evaluation , our most appropriate comparison is with Generative Matching Networks ( Bartunov & Vetrov , 2016 ) as they also model dependencies within a class . Thus , we trained models under the same train/test split as them , with no data augmentation . We evaluate the joint log likelihood of full character classes from the test set , normalised by the number of elements , using importance weighting with k=500 samples from q ( c ; X ) . As can be seen in Tables 3 and 4 , our hierarchical PixelCNN architecture is able to achieve state- f-the-art log likelihood results nly when trained using the VHE objective . Figure 4 : 5-shot samples generated by each model ( more in Supplement ) . With a PixelC N architecture , both Neural S atist cian and Resample objectives lead to underutilisation of the latent space , producing unfaithful samples . For the hie archical PixelC N architecture , however , significant diff rences arise between tra ing objectives . In this case , a Neural S atist cian learns a stron global distribution over images but makes only m nimal use of latent variables c. This means that , despite the use of a higher capacity model , clas ification ccuracy is much poo er ( 66 % ) than that achieved using a deconvolutional architecture . For the same reason , conditional samples display an improved sharpness but are no longer identifiable to the cue images on which they w re conditioned ( Figure 4 ) . Our careful tra ing sugge t that this is not an optimisation difficulty but is core to the objective , as discussed in Chen et al . ( 2016 ) . By contrast , a VHE is able to gain a large benefit from the hie archical PixelC N architecture , with a 3-fold reduction in classification error ( 5-shot accuracy 9 .8 % ) and conditional samples which are simultaneously sharp and identifiable ( Figure 4 ) . This improvement is in part achieved by increased utilisation of the latent space , due to rescaling of the KL divergence term in the objective . However , ou result show tha this co mon technique is insufficient when used alone , leading t overfitting to cue images with an equally sev re impairment of classification performance ( accuracy 62.8 % ) . Rather , we find that KL-rescaling an d ta resampling must be used together in order for the benefit of the powerful PixelC N architecture to be realised . Table 2 li ts the classification accuracy achieved by VHEs with both |D| = 1 and |D| = 5 , as compared to existing deep lear ing approaches . We find that both networks are not only s ate-of-the-art amongst deep generative models , but also competitive against the best discriminative models trained directly for few-shot classi- fication . Unlike th se discriminative models , a VHE is also able to generate new images of a ch racter in one shot , producing samples which are both realistic and faithful to the class of the cue image ( Figure 5 ) . As our goal is to model shared structure across images , w evaluate generative perf rmance using joint log like- Figure 5 : One-shot same-clas samples generated by our model . Cue images w re sampled from previously un- seen classes . lihood of th entire Omniglo test set ( rather than separately across images ) . From this perspective , a singl elment VAE will perform poorly as it treats all d tapoints as independent , optim sing a sum over log likelihoods for each lement . By sharing latents across lements of the same class , a VHE can improve upon this considerably . For likelihood evaluation , our most appropriate comparison is with Generative Matching Networks ( Bartunov & Vetrov , 2016 ) as they also model dependencies within a class . Thus , we trained models under the same train/test split as them , with no d t augmen ation . W evaluate the joint log likelihood o full ch racter cla ses from the test set , normalised by the number of lements , using importance weighting with k=500 samples from q ( c ; X ) . As can be seen in Tables 3 and 4 , our hie archical PixelC N architectur is able to achieve s ate-of-the art log likelihood results only whe trained using the VHE objective . Figure 4 : 5-shot samples generated by each mod l ( more in Supplement ) . With a PixelCNN architecture , both eural Statistician and Resample objectives l ad to underutilisation of the lat nt sp ce , pr ducing u faithful samples . For the hierarchical PixelCNN architecture , however , significant differences arise between training objectives . In this case , a Neural Stati tician learns a strong global distribution over images t makes only minimal use of latent variables c. This me ns that , despite th use of higher capacity model , classific tion accuracy i much poorer ( 66 % ) than that achieved using a deconvolutional architecture . For the same reason , c nditional samples display an improved sharpness but a e no longer identifiable to the cue images on w ich they were c ditioned ( Figure 4 ) . Our careful traini g s ggests that this is not an optimisation difficulty but is c re t the objective , as discussed in Chen et al . ( 2016 ) . By contrast , a VHE is able to gain a large benefit from the hierarchical PixelCNN architecture , with a 3-fold r - duction in classification error ( 5-shot curacy 98.8 % ) and conditional samples which are simultan ously sh p and identifiable ( Figure 4 ) . This improvement is in part achieved by increased utilisation of th latent space , due to rescaling of the KL diverge ce term in the obj ctiv . However , our results show that this common technique is insufficient when used alone , leading to overfitting to cue images with an equally s vere imp irment of classification performance ( curacy 62.8 % ) . R ther , we find that KL-rescaling and d ta resampling must be us d together in order for the ben fit of th p w rful PixelCNN architecture to be re lised . Table 2 lists the classification accuracy achieved by VHEs with both |D| = 1 and |D| = 5 , as compared to existing deep learning approaches . We fi d that bot networks are not only state-of-the-art amongst deep gen erative models , but also competitive against the best discriminative models trained directly for few-shot classification . Unlike these discriminativ mod ls , a VHE is also able to generate new images of a character in one shot , producing samples which are both realistic and faithful to the class of t e c e image ( Figure 5 ) . As our goal is to model shared structure acros images , we evaluate gener tiv performance using joint l g lik - Figure 5 : One-shot same-class samples generated by our mod l. Cue images were sampl d from previously un- seen classes . lihood of the e tire Omniglot test s t ( ather than separately acro s images ) . From this persp ctive , a single elment VAE will perfor poorly as it treats all datapoint as independent , optimising a sum over log likelihoods for each le e t. By sh ring lat nts across elements of the same class , a VHE can i prove upon this considerably . For likeli ood evaluation , our m st appropriate co paris i with G erative Matching N tworks ( Bartu ov & Vetrov , 2016 ) a they also model depend ncies within a clas . Thus , we trained models under the sa e train/test split as them , with no ata augmentatio . We eval at the joint log likel hood f full charact r class s from the t st set , normal s d by th numb of elem nts , sing importanc weighting with k=500 samples from q ( c ; X ) . As can be seen in Tables 3 d 4 , our hier rchical PixelCNN architecture is ble to a i ve state-of-the-art log likelihood results only wh train d usi g the VHE obj ctiv . Figure 4 : 5-s ot samples g nerated by each del ( mor in Supplem n ) . With a PixelCNN architecture , both Neural Statistician and Resample objectives le d to und rutilisatio of the latent spac , producing nfaithful samples . For the hiera chical PixelCNN architecture , however , significant differences ari e betwee training objectives . In this case , a Neural Statistician learns a strong global distribution over images but makes only minimal use of latent variabl s c. This means that , despite the use of a higher capacity model , lassification accuracy is much poorer ( 66 % ) than that achieved using a deconvolutional archite ture . For the same reason , conditional samples display an improv d sharpness but are no longer identifiable to the cue images on which they were conditioned ( Figure 4 ) . O r careful training suggests that this is not an optimisation difficulty but is core to the objective , as discussed in Chen et al . ( 2016 ) . By contrast , a VHE is able to gain a large benefit from t e hierarchical PixelCNN architecture , with a 3-fold reduction in classification error ( 5-shot accuracy 98.8 % ) and conditional samples which are sim ltaneously harp and dentifiabl ( Figure 4 ) . This i provement is in part achieved by increased utilisation of the latent space , due to r scaling of the KL divergence term in the objective . However , our results show that this common technique is insufficient when used alone , leading to overfitting to cu images with an equally severe impairment of classificati n performance ( accuracy 62.8 % ) . Rather , we find that KL-rescaling and data resa pling must be used together in order for the benefit f the powerful PixelCNN architecture to be realised . Table 2 lists the classification accuracy achieved by VHEs with both |D| = 1 and |D| = 5 , as compared t existing deep learning approaches . We find that both netwo ks are not only state-of-the-art amongst deep genera models , but al o competitive against the best discri inative models trained directly for few-shot classification . Unlike these discri inative models , a VHE is also able to ge erate ew images of a character in one shot , pr ducing samples which are both realistic and faithful to the class of the cue image ( Figure 5 ) . As our goal is t model shared structure across images , w eval at ge erative p rformance using joint log like- Figure 5 : One-shot same-class samples generated by our odel . Cue images were sampled from previously un- seen classes . lihood of the ntire Omniglot test set ( rather than separately across im ges ) . From this perspective , a single element VAE will perf rm poorly as it treats all datapoints as independent , optimi ing a sum over log likelihoods for each element . By sharing latents across elements of the same class , a VHE can im rove upon this considerably . For likelihood evaluation , our most appropriate comparison is with Genera ive Matching Networks ( Bartunov & Vetrov , 2016 ) as they also model dependencies within a class . Thus , we trained mo els under the same train/test split as them , with no data augmentation . We evaluate the join log likeliho d of full character classes from the test set , nor alised by the number of eleme ts , using importance weighting with k=500 sa ples from q ( c ; X ) . As ca be seen in Tables 3 and 4 , our hierarchical PixelCNN architecture is ab e to achieve state-of-the-art log likelihood r sults only when trai ed usi g the VHE objective . Figure 4 : 5-shot samples g nerated by each model ( or in Supplement ) . With PixelCNN architect e , both Neural Statistician and R sample objectiv lead to underutilis tion of th latent space , produci g unfaithful samples . For the hierarchical PixelCNN archit cture , however , signifi ant d fferences arise between train ng objectives . In this case , a Neural Stati tician learns strong g obal dist ibution over images but akes only mini l u e of atent var ables c. This me ns that , d spite the use of a higher capa ity mod l , l ssificati n accuracy is much poorer ( 66 % ) than that achieved using a deconvol tional architecture . For the same reas n , conditional amples display an improved sh rp ess but are no longer identifiable to the cue imag s on which they were conditioned ( Figure 4 ) . Our careful training suggests that this is not an optimisation d fficulty bu is core to the objective , as dis ussed in Chen et al . ( 2016 ) . By contrast , a VHE s ab e to gai a la ge benefit from the h erarchical PixelCNN architecture , with a 3-fold - duction in classifica error ( 5-sho accuracy 98.8 % ) and conditional s mples which are imultaneously sharp and identifiable ( F gure 4 ) . This improve ent is in part achi ved by increased utilisat o of th latent spac , due to rescaling of th KL divergence term in the obj ctive . However , our results show that this common technique is insuffici nt wh n used alo e , l ading to overfitting to cue images with an equ lly sev re impairment of classification performan e ( accuracy 62.8 % ) . Rather , we find that KL-rescaling nd data resampli g ust be used together in order for t benefit of the pow rful PixelCNN architectur to be ealised . Tabl 2 li ts the cl ssification ac uracy chieved by VHEs with both |D| = 1 and |D| = 5 , as compared to existi g dee learn ng approaches . We find that both netw rks are not only state-of-the-art amo gst de p generative models , but also competitive gain t the best discriminative o els trained directly for few-shot classification . Unlike these discrimina ive odels , a VHE is also abl to generat new images of a character in one shot , producing samples which are both realistic nd faithful to the class of the cue im ge ( Figure 5 ) . As ur goa is to odel sh red structure across images , w valuate generative performance using joint log like- Figure 5 : One-shot same-class samples generated by our model . Cue images were sampled fro previously un- seen lasses . lihood of the entire Omniglot tes set ( rather han separa ly across i ages ) . From this per pective , a single element VAE will perform poorly as it treats all dat points as independent , optimising a sum over log likelihoods for each eleme t. By sharing latents across el m t of the same class , a VHE can improve upon this considerably . F r likelih od evaluation , our most appropriate c mpariso is with Generative Matching Netw rks ( Bartunov & Vetrov , 2016 ) as th y also model dependencies within a class . Thus , we traine odels under the same train/test spl s them , with no data augment tion . W evaluate the joint log like ihood of ful character classe from the test s t , nor ali ed by h number of eleme ts , using imporance weighting with k=500 samples fro q ( c ; X ) . As can be seen in Tables 3 and 4 , our h rarchical PixelCNN archit cture is able to a hiev state-of-the-art log likelihood results only when t ained using the VHE objective . ( b ) VHE Figure A13 : New exemplars produced by the Sequential Gen rative ( SG ) model [ 43 ] and the Vari tional Homoencoder ( VHE ) [ 21 ] . ( a ) The SG model shows far too much variability , drawing what is clearly the wrong character in many cases ( e.g . right-most column ) . ( b ) The VHE character samples are often incomplete , missing important strok s of the target class . 19 Published as a conference paper at ICLR 2021

Summary: In this paper, the authors seek to combine the advantages of symbolic, compositional models and neural approaches, in particular by using "probabilistic programs with neural network subroutines" which should reflect a causal generative process. This is certainly an interesting area to explore. But I'm not convinced that the approach they propose (GNS) lives up to their stated goals.