Learning to Infer and Execute 3D Shape Programs

1 INTRODUCTION . Given the table in Figure 1 , humans are able to instantly recognize its parts and regularities : there exist sharp edges , smooth surfaces , a table top that is a perfect circle , and two lower , square layers . Beyond these basic components , we also perceive higher-level , abstract concepts : the shape is bilateral symmetric ; the legs are all of equal length and laid out on the opposite positions of a 2D grid . Knowledge like this is crucial for visual recognition and reasoning ( Koffka , 2013 ; Dilks et al. , 2011 ) . Recent AI systems for 3D shape understanding have made impressive progress on shape classification , parsing , reconstruction , and completion ( Qi et al. , 2017 ; Tulsiani et al. , 2017 ) , many making use of large shape repositories like ShapeNet ( Chang et al. , 2015 ) . Popular shape representations include voxels ( Wu et al. , 2015 ) , point clouds ( Qi et al. , 2017 ) , and meshes ( Wang et al. , 2018 ) . While each has its own advantages , these methods fall short on capturing the strong shape priors we just described , such as sharp edges and smooth surfaces . A few recent papers have studied modeling 3D shapes as a collection of primitives ( Tulsiani et al. , 2017 ) , with simple operations such as addition and subtraction ( Sharma et al. , 2018 ) . These representations have demonstrated success in explaining complex 3D shapes . In this paper , we go beyond them to capture the high-level regularity within a 3D shape , such as symmetry and repetition . In this paper , we propose to represent 3D shapes as shape programs . We define a domain-specific language ( DSL ) for shapes , containing both basic shape primitives for parts with their geometric and semantic attributes , as well as statements such as loops to enforce higher-level structural priors . Project page : http : //shape2prog.csail.mit.edu Because 3D shape programs are a new shape representation , there exist no annotations of shape programs for 3D shapes . The lack of annotations makes it difficult to train an inference model with full supervision . To overcome this obstacle , we propose to learn a shape program executor that reconstructs a 3D shape from a shape program . After initial bootstrapping , our model can then learn in a self-supervised way , by attempting to explain and reconstruct unlabeled 3D shapes with 3D shape programs . This design minimizes the amount of supervision needed to get our model off the ground . With the learned neural program executor , our model learns to explain input shapes without ground truth program annotations . Experiments on ShapeNet show that our model infers accurate 3D shape programs for highly complex shapes from various categories . We further extend our model by integrating with an image-to-shape reconstruction module , so it directly infers a 3D shape program from a color image . This leads to 3D shape reconstructions that are both more accurate and more physically plausible . Our contributions are three-fold . First , we propose 3D shape programs : a new representation for shapes , building on classic findings in cognitive science and computer graphics . Second , we propose to infer 3D shape programs by explaining the input shape , making use of a neural shape program executor . Third , we demonstrate that the inference model , the executor , and the programs they recover all achieve good performance on ShapeNet , learning to explain and reconstruct complex shapes . We further show that an extension of the model can infer shape programs and reconstruct 3D shapes directly from images . 2 RELATED WORK . Inverse procedural graphics . The problem of inferring programs from voxels is closely related to inverse procedural graphics , where a procedural graphics program is inferred from an image or declarative specification ( Ritchie et al. , 2016 ; Št ’ ava et al. , 2010 ) . Where the systems have been most successful , however , are when they leverage a large shape–component library ( Chaudhuri et al. , 2011 ; Schulz et al. , 2017 ) or are applied to a sparse solution space ( van den Hengel et al. , 2015 ) . Kulkarni et al . ( 2015a ) approached the problem of inverse graphics as inference in a probabilistic program for generating 2D images , or image contours , from an underlying 3D model . They demonstrated results on several different applications using parametric generative models for faces , bodies , and simple multi-part objects based on generalized cylinders . In this work , we extend the idea of inverse procedural graphics to 3-D voxel representations , and show how this idea can apply to large data sets like ShapeNet . We furthermore do not have to match components to a library of possible shapes , instead using a neural network to directly infer shapes and their parameters . A few recent papers have explored the use of simple geometric primitives to describe shapes ( Tulsiani et al. , 2017 ; Zou et al. , 2017 ; Liu et al. , 2018 ) , putting the classic idea of generalized cylinders ( Roberts , 1963 ; Binford , 1971 ) or geons ( Biederman , 1987 ) in the modern context of deep learning . In particular , Sharma et al . ( 2018 ) extended these papers and addressed the problem of inferring 3-D CAD programs from perceptual input . We find this work inspiring , but also feel that a key goal of 3-D program inference is to reconstruct a program in terms of semantically meaningful parts and their spatial regularity , which we address here . Some other graphics papers also explore regularity , but without using programs ( Mitra et al. , 2013 ; Zhu et al. , 2018 ; Nishida et al. , 2018 ; Li et al. , 2017 ) . Work in the HCI community has also addressed the problem of inferring parametric graphics primitives from perceptual input . For example , Nishida et al . ( 2016 ) proposed to learn to instantiate procedural primitives for an interactive modeling system . In our work , we instead learn to instantiate multiple procedural graphics primitives simultaneously , without assistance from a human user . Program synthesis . In the AI literature , Ellis et al . ( 2018 ) leveraged symbolic program synthesis techniques to infer 2D graphics programs from images , extending their earlier work by using neural nets for faster inference of low-level cues such as strokes ( Ellis et al. , 2015 ) . Here , we show how a purely end–to–end network can recover 3D graphics programs from voxels , conceptually relevant to RobustFill ( Devlin et al. , 2017 ) , which presents a purely end-to-end neural program synthesizer for text editing . The very recent SPIRAL system ( Ganin et al. , 2018 ) also takes as its goal to learn structured program–like models from ( 2D ) images . An important distinction from our work here is that SPIRAL explains an image in terms of paint-like “ brush strokes ” , whereas we explain 3D voxels in terms of high-level objects and semantically meaningful parts of objects , like legs or tops . Other tangential related work on program synthesis includes Balog et al . ( 2017 ) ; Devlin et al . ( 2017 ) ; Parisotto et al . ( 2017 ) ; Gaunt et al . ( 2016 ) ; Sun et al . ( 2018a ) ; Liu et al . ( 2019 ) . Learning to execute programs . Neural Program Interpreters ( NPI ) have been extensively studied for programs that abstract and execute tasks such as sorting , shape manipulation , and grade-school arithmetic ( Reed & De Freitas , 2016 ; Cai et al. , 2017 ; Bošnjak et al. , 2017 ) . In NPI ( Reed & De Freitas , 2016 ) , the key insight is that a program execution trace can be decomposed into predefined operations that are more primitive ; and at each step , an NPI learns to predict what operation to take next depending on the general environment , domain specific state , and previous actions . Cai et al . ( 2017 ) improved the generalization of NPIs by adding recursion . Johnson et al . ( 2017 ) learned to execute programs for visual question and answering . In this paper , we also learn a 3D shape program executor that renders 3D shapes from programs as a component of our model . 3 3D SHAPE PROGRAMS . In this section , we define the domain-specific language for 3D shapes , as well as the problem of shape program synthesis . Table 1 shows our DSL for 3D shape programs . Each shape program consists of a variable number of program statements . A program statement can be either Draw , which describes a shape primitive as well as its geometric and semantic attributes , or For , which contains a sub-program and parameters specifying how the subprogram should be repeatedly executed . The number of arguments for each program statement varies . We tokenize programs for the purpose of neural network prediction . Each shape primitive models a semantically-meaningful part of an object . Its geometric attributes ( Table 1 : Geometry Params , Position Params ) specify the position and orientation of the part . Its semantic attributes ( Table 1 : Semantics ) specify its relative role within the whole shape ( e.g. , top , back , leg ) . They do not affect the geometry of the primitive ; instead , they associate geometric parts with their semantic meanings conveying how parts can be shared across object categories semantically and functionally ( e.g. , a chair and a table may have similar legs ) . Our For statement captures high-level regularity across parts . For example , the legs of a table can be symmetric with respect to particular rotation angles . The horizontal bars of a chair may lay out regularly with a fixed vertical gap . Each For statement can contain sub-programs , allowing recursive generation of shape programs . The problem of inferring a 3D shape program is defined as follows : predicting a 3D shape program that reconstructs the input shape when the program is executed . In this paper , we use voxelized shapes as input with a resolution of 32× 32× 32 . 4 INFERRING AND EXECUTING 3D SHAPE PROGRAMS . Our model , called Shape Programs , consists of a program generator and a neural program executor . The program generator takes a 3D shape as input and outputs a sequence of primitive programs that describe this 3D shape . The neural program executor takes these programs as input and generates the corresponding 3D shapes . This allows our model to learn in a self-supervised way by generating programs from input shapes , executing these programs , and back-propagating the difference between the generated shapes and the raw input . 4.1 PROGRAM GENERATOR . We model program generation as a sequential prediction problem . We partition full programs into two types of subprograms , which we call blocks : ( 1 ) a single drawing statement describing a semantic part , e.g . circle top ; and ( 2 ) compound statements , which are a loop structure that interprets a set of translated or rotated parts , e.g . four symmetric legs . This part-based , symmetry-aware decomposition is inspired by human perception ( Fleuret et al. , 2011 ) . Our program generator is shown in Figure 2 . The core of the program generator consists of two orthogonal LSTMs . The first one , the Block LSTM , connects sequential blocks . The second one , the Step LSTM , generates programs for each block . At each block , we first render the shape described by previous program blocks with a graphics engine . Then , the rendered shape and the raw shape are combined along the channel dimension and fed into a 3D ConvNet . The Block LSTM takes the features output by the 3D ConvNet and outputs features of the current block , which are further fed into the step LSTM to predict the block programs . The reason why we need the step LSTM is that each block might have a different length ( e.g. , loop bodies of different sizes ) . Given block feature hblk , the Step LSTM predicts a sequence of program tokens , each consisting of a program id and an argument matrix . The i-th row of the argument matrix serves for the i-th primitive program . From the LSTM hidden state ht , two decoders generate the output . The softmax classification probability over program sets is obtained by fprog : RM → RN . The argument matrix is computed by fparam : RM → RN×K , where N is the total number of program primitives and K is the maximum possible number of arguments . The feed-forward steps of the Step LSTM are summarized as ht = flstm ( xt , ht−1 ) , ( 1 ) pt = fprog ( ht ) , at = fparam ( ht ) , ( 2 ) where the pt and at corresponds to the program probability distribution and argument matrix at time t. After getting the program ID , we obtain its arguments by retrieving the corresponding row in the argument matrix . At each time step , the input of the Step LSTM xt is the embedding of the output in the previous step . For the first step , the block feature hblk is used instead . We pretrain our program generator on a synthetic dataset with a few pre-defined simple program templates . The set of all templates for tables are shown in Section A1 . These templates are much simpler than the actual shapes . The generator is trained to predict the program token and regress the corresponding arguments via the following loss lgen = ∑ b , i wplcls ( pb , i , p̂b , i ) ) +walreg ( ab , i , âb , i ) , where lcls ( pb , i , p̂b , i ) ) and lreg ( ab , i , âb , i ) are the cross-entropy loss of program ID classification and the L-2 loss of argument regression , in step i of block b , respectively . The weights wp and wa balance the losses between classification and regression .

Summary: This paper introduces a high-level semantic description for 3D shapes. The description is given by the so-called ShapeProgram,  Each shape program consists of several program statements. A program statement can be either Draw, which describes a shape primitive as well as its geometric and semantic attributes, or For, which contains a sub-program and parameters specifying how the sub-program should be repeatedly executed. The ShapeProgram is connected with an input through two networks, the program generator (encoder) and a neural program executor (decoder). Both encoder/decoder are implemented using LSTM. The key ML contribution is on the decoder, which leverages a parametrization to make the decoder differentiable. The major advantage of the proposed technique is that it does not need to specify the ShapeProgram in advance. In the same spriit of training an auto-encoder. It can be learned in a semi-supervised manner. However, in practice, one has to start with a reasonably good initial program. In the paper, this initial program was learned from synthetic data.