Learning to Infer and Execute 3D Shape Programs

1 INTRODUCTION . Given the table in Figure 1 , humans are able to instantly recognize its parts and regularities : there exist sharp edges , smooth surfaces , a table top that is a perfect circle , and two lower , square layers . Beyond these basic components , we also perceive higher-level , abstract concepts : the shape is bilateral symmetric ; the legs are all of equal length and laid out on the opposite positions of a 2D grid . Knowledge like this is crucial for visual recognition and reasoning ( Koffka , 2013 ; Dilks et al. , 2011 ) . Recent AI systems for 3D shape understanding have made impressive progress on shape classification , parsing , reconstruction , and completion ( Qi et al. , 2017 ; Tulsiani et al. , 2017 ) , many making use of large shape repositories like ShapeNet ( Chang et al. , 2015 ) . Popular shape representations include voxels ( Wu et al. , 2015 ) , point clouds ( Qi et al. , 2017 ) , and meshes ( Wang et al. , 2018 ) . While each has its own advantages , these methods fall short on capturing the strong shape priors we just described , such as sharp edges and smooth surfaces . A few recent papers have studied modeling 3D shapes as a collection of primitives ( Tulsiani et al. , 2017 ) , with simple operations such as addition and subtraction ( Sharma et al. , 2018 ) . These representations have demonstrated success in explaining complex 3D shapes . In this paper , we go beyond them to capture the high-level regularity within a 3D shape , such as symmetry and repetition . In this paper , we propose to represent 3D shapes as shape programs . We define a domain-specific language ( DSL ) for shapes , containing both basic shape primitives for parts with their geometric and semantic attributes , as well as statements such as loops to enforce higher-level structural priors . Project page : http : //shape2prog.csail.mit.edu Because 3D shape programs are a new shape representation , there exist no annotations of shape programs for 3D shapes . The lack of annotations makes it difficult to train an inference model with full supervision . To overcome this obstacle , we propose to learn a shape program executor that reconstructs a 3D shape from a shape program . After initial bootstrapping , our model can then learn in a self-supervised way , by attempting to explain and reconstruct unlabeled 3D shapes with 3D shape programs . This design minimizes the amount of supervision needed to get our model off the ground . With the learned neural program executor , our model learns to explain input shapes without ground truth program annotations . Experiments on ShapeNet show that our model infers accurate 3D shape programs for highly complex shapes from various categories . We further extend our model by integrating with an image-to-shape reconstruction module , so it directly infers a 3D shape program from a color image . This leads to 3D shape reconstructions that are both more accurate and more physically plausible . Our contributions are three-fold . First , we propose 3D shape programs : a new representation for shapes , building on classic findings in cognitive science and computer graphics . Second , we propose to infer 3D shape programs by explaining the input shape , making use of a neural shape program executor . Third , we demonstrate that the inference model , the executor , and the programs they recover all achieve good performance on ShapeNet , learning to explain and reconstruct complex shapes . We further show that an extension of the model can infer shape programs and reconstruct 3D shapes directly from images . 2 RELATED WORK . Inverse procedural graphics . The problem of inferring programs from voxels is closely related to inverse procedural graphics , where a procedural graphics program is inferred from an image or declarative specification ( Ritchie et al. , 2016 ; Št ’ ava et al. , 2010 ) . Where the systems have been most successful , however , are when they leverage a large shape–component library ( Chaudhuri et al. , 2011 ; Schulz et al. , 2017 ) or are applied to a sparse solution space ( van den Hengel et al. , 2015 ) . Kulkarni et al . ( 2015a ) approached the problem of inverse graphics as inference in a probabilistic program for generating 2D images , or image contours , from an underlying 3D model . They demonstrated results on several different applications using parametric generative models for faces , bodies , and simple multi-part objects based on generalized cylinders . In this work , we extend the idea of inverse procedural graphics to 3-D voxel representations , and show how this idea can apply to large data sets like ShapeNet . We furthermore do not have to match components to a library of possible shapes , instead using a neural network to directly infer shapes and their parameters . A few recent papers have explored the use of simple geometric primitives to describe shapes ( Tulsiani et al. , 2017 ; Zou et al. , 2017 ; Liu et al. , 2018 ) , putting the classic idea of generalized cylinders ( Roberts , 1963 ; Binford , 1971 ) or geons ( Biederman , 1987 ) in the modern context of deep learning . In particular , Sharma et al . ( 2018 ) extended these papers and addressed the problem of inferring 3-D CAD programs from perceptual input . We find this work inspiring , but also feel that a key goal of 3-D program inference is to reconstruct a program in terms of semantically meaningful parts and their spatial regularity , which we address here . Some other graphics papers also explore regularity , but without using programs ( Mitra et al. , 2013 ; Zhu et al. , 2018 ; Nishida et al. , 2018 ; Li et al. , 2017 ) . Work in the HCI community has also addressed the problem of inferring parametric graphics primitives from perceptual input . For example , Nishida et al . ( 2016 ) proposed to learn to instantiate procedural primitives for an interactive modeling system . In our work , we instead learn to instantiate multiple procedural graphics primitives simultaneously , without assistance from a human user . Program synthesis . In the AI literature , Ellis et al . ( 2018 ) leveraged symbolic program synthesis techniques to infer 2D graphics programs from images , extending their earlier work by using neural nets for faster inference of low-level cues such as strokes ( Ellis et al. , 2015 ) . Here , we show how a purely end–to–end network can recover 3D graphics programs from voxels , conceptually relevant to RobustFill ( Devlin et al. , 2017 ) , which presents a purely end-to-end neural program synthesizer for text editing . The very recent SPIRAL system ( Ganin et al. , 2018 ) also takes as its goal to learn structured program–like models from ( 2D ) images . An important distinction from our work here is that SPIRAL explains an image in terms of paint-like “ brush strokes ” , whereas we explain 3D voxels in terms of high-level objects and semantically meaningful parts of objects , like legs or tops . Other tangential related work on program synthesis includes Balog et al . ( 2017 ) ; Devlin et al . ( 2017 ) ; Parisotto et al . ( 2017 ) ; Gaunt et al . ( 2016 ) ; Sun et al . ( 2018a ) ; Liu et al . ( 2019 ) . Learning to execute programs . Neural Program Interpreters ( NPI ) have been extensively studied for programs that abstract and execute tasks such as sorting , shape manipulation , and grade-school arithmetic ( Reed & De Freitas , 2016 ; Cai et al. , 2017 ; Bošnjak et al. , 2017 ) . In NPI ( Reed & De Freitas , 2016 ) , the key insight is that a program execution trace can be decomposed into predefined operations that are more primitive ; and at each step , an NPI learns to predict what operation to take next depending on the general environment , domain specific state , and previous actions . Cai et al . ( 2017 ) improved the generalization of NPIs by adding recursion . Johnson et al . ( 2017 ) learned to execute programs for visual question and answering . In this paper , we also learn a 3D shape program executor that renders 3D shapes from programs as a component of our model . 3 3D SHAPE PROGRAMS . In this section , we define the domain-specific language for 3D shapes , as well as the problem of shape program synthesis . Table 1 shows our DSL for 3D shape programs . Each shape program consists of a variable number of program statements . A program statement can be either Draw , which describes a shape primitive as well as its geometric and semantic attributes , or For , which contains a sub-program and parameters specifying how the subprogram should be repeatedly executed . The number of arguments for each program statement varies . We tokenize programs for the purpose of neural network prediction . Each shape primitive models a semantically-meaningful part of an object . Its geometric attributes ( Table 1 : Geometry Params , Position Params ) specify the position and orientation of the part . Its semantic attributes ( Table 1 : Semantics ) specify its relative role within the whole shape ( e.g. , top , back , leg ) . They do not affect the geometry of the primitive ; instead , they associate geometric parts with their semantic meanings conveying how parts can be shared across object categories semantically and functionally ( e.g. , a chair and a table may have similar legs ) . Our For statement captures high-level regularity across parts . For example , the legs of a table can be symmetric with respect to particular rotation angles . The horizontal bars of a chair may lay out regularly with a fixed vertical gap . Each For statement can contain sub-programs , allowing recursive generation of shape programs . The problem of inferring a 3D shape program is defined as follows : predicting a 3D shape program that reconstructs the input shape when the program is executed . In this paper , we use voxelized shapes as input with a resolution of 32× 32× 32 . 4 INFERRING AND EXECUTING 3D SHAPE PROGRAMS . Our model , called Shape Programs , consists of a program generator and a neural program executor . The program generator takes a 3D shape as input and outputs a sequence of primitive programs that describe this 3D shape . The neural program executor takes these programs as input and generates the corresponding 3D shapes . This allows our model to learn in a self-supervised way by generating programs from input shapes , executing these programs , and back-propagating the difference between the generated shapes and the raw input . 4.1 PROGRAM GENERATOR . We model program generation as a sequential prediction problem . We partition full programs into two types of subprograms , which we call blocks : ( 1 ) a single drawing statement describing a semantic part , e.g . circle top ; and ( 2 ) compound statements , which are a loop structure that interprets a set of translated or rotated parts , e.g . four symmetric legs . This part-based , symmetry-aware decomposition is inspired by human perception ( Fleuret et al. , 2011 ) . Our program generator is shown in Figure 2 . The core of the program generator consists of two orthogonal LSTMs . The first one , the Block LSTM , connects sequential blocks . The second one , the Step LSTM , generates programs for each block . At each block , we first render the shape described by previous program blocks with a graphics engine . Then , the rendered shape and the raw shape are combined along the channel dimension and fed into a 3D ConvNet . The Block LSTM takes the features output by the 3D ConvNet and outputs features of the current block , which are further fed into the step LSTM to predict the block programs . The reason why we need the step LSTM is that each block might have a different length ( e.g. , loop bodies of different sizes ) . Given block feature hblk , the Step LSTM predicts a sequence of program tokens , each consisting of a program id and an argument matrix . The i-th row of the argument matrix serves for the i-th primitive program . From the LSTM hidden state ht , two decoders generate the output . The softmax classification probability over program sets is obtained by fprog : RM → RN . The argument matrix is computed by fparam : RM → RN×K , where N is the total number of program primitives and K is the maximum possible number of arguments . The feed-forward steps of the Step LSTM are summarized as ht = flstm ( xt , ht−1 ) , ( 1 ) pt = fprog ( ht ) , at = fparam ( ht ) , ( 2 ) where the pt and at corresponds to the program probability distribution and argument matrix at time t. After getting the program ID , we obtain its arguments by retrieving the corresponding row in the argument matrix . At each time step , the input of the Step LSTM xt is the embedding of the output in the previous step . For the first step , the block feature hblk is used instead . We pretrain our program generator on a synthetic dataset with a few pre-defined simple program templates . The set of all templates for tables are shown in Section A1 . These templates are much simpler than the actual shapes . The generator is trained to predict the program token and regress the corresponding arguments via the following loss lgen = ∑ b , i wplcls ( pb , i , p̂b , i ) ) +walreg ( ab , i , âb , i ) , where lcls ( pb , i , p̂b , i ) ) and lreg ( ab , i , âb , i ) are the cross-entropy loss of program ID classification and the L-2 loss of argument regression , in step i of block b , respectively . The weights wp and wa balance the losses between classification and regression .

Summary: This paper presents a methodology to infer shape programs that can describe 3D objects. The key intuition of the shape programs is to integrate bottom-up low-level feature recognition with symbolic high-level program structure, which allows the shape programs to capture both high-level structure and the low-level geometry of the shapes. The paper proposes a domain-specific language for 3D shapes that consists of “For” loops for capturing high-level regularity, and associates objects with both their geometric and semantic attributes. It then proposes an end-to-end differentiable architecture to learn such 3D programs from shapes using an interesting self-supervised mechanism. The neural program generator proposes a program in the DSL that is executed by a neural program execution module to render the corresponding output shape, which is then compared with the original shape and the difference loss is back-propagated to improve the program distribution. The technique is evaluated on both synthetic and ShapeNet tasks, and leads to significant improvements compared to Tulsiani et al. that embed a prior structure on learning shape representations as a composition of primitive abstractions. In addition, the technique is also paired with MarrNet to allow for a better 3D reconstruction from 2D images.