Learning to Describe Scenes with Programs

Human scene perception goes beyond recognizing a collection of objects and their pairwise relations . We understand higher-level , abstract regularities within the scene such as symmetry and repetition . Current vision recognition modules and scene representations fall short in this dimension . In this paper , we present scene programs , representing a scene via a symbolic program for its objects , attributes , and their relations . We also propose a model that infers such scene programs by exploiting a hierarchical , object-based scene representation . Experiments demonstrate that our model works well on synthetic data and transfers to real images with such compositional structure . The use of scene programs has enabled a number of applications , such as complex visual analogy-making and scene extrapolation . 1 INTRODUCTION . When examining the image in Figure 1a , we instantly recognize the shape , color , and material of the objects it depicts . We can also effortlessly imagine how we may extrapolate the set of objects in the scene while preserving object patterns ( Figure 1b ) . Our ability to imagine unseen objects arises from holistic scene perception : we not only recognize individual objects from an image , but naturally perceive how they should be organized into higher-level structure ( Rock & Palmer , 1990 ) . Recent AI systems for scene understanding have made impressive progress on detecting , segmenting , and recognizing individual objects ( He et al. , 2017 ) . In contrast , the problem of understanding high-level , abstract relations among objects is less studied . While a few recent papers have attempted to produce a holistic scene representation for scenes with a variable number of objects ( Ba et al. , 2015 ; Huang & Murphy , 2015 ; Eslami et al. , 2016 ; Wu et al. , 2017 ) , the relationships among these objects are not captured in these models . The idea of jointly discovering objects and their relations has been explored only very recently , where the learned relations are often in the form of interaction graphs ( van Steenkiste et al. , 2018 ; Kipf et al. , 2018 ) or semantic scene graphs ( Johnson et al. , 2015 ) , both restricted to pairwise , local relations . However , our ability to imagine extrapolated images as in Figure 1 relies on our knowledge of long-range , hierarchical relationships among objects , such as how objects are grouped and what patterns characterize those groups . In this paper , we aim to tackle the problem of understanding higher-level , abstract regularities such as repetition and symmetry . We propose to represent scenes as scene programs . We define a domainspecific language for scenes , capturing both objects with their geometric and semantic attributes , as well as program commands such as loops to enforce higher-level structural relationships . Given an image of a complex scene , we propose to infer its scene program via a hierarchical bottom-up approach . First , we parse the image into individual objects and infer their attributes , resulting in the object representation . Then , we organize these objects into different groups , i.e . the group representation , where objects in each group fall into the same program block . Finally , we describe each group with a program , and combine these programs to get the program representation for the entire scene . ∗This work was done when Yunchao Liu was a visiting student at MIT CSAIL . Our model applies deep neural networks for each stage of this process and is able to generate programs describing the input image with high accuracy . When testing on scenes that are more complex than those used for training , our hierarchical inference process achieves better generalization performance than baseline methods that attempt to infer a program directly from the image . Our model is also able to handle ambiguity , generating multiple possible programs when there is more than one way to describe the scene . Furthermore , our method generalizes to real-world images without any additional supervised training programs ; only the low-level object detection module must be re-trained . Finally , we demonstrate how our model facilitates high-level image editing , as users can change parameters in the inferred program to achieve the editing effects they want more efficiently . We show examples of such image edits , including extrapolations such as the one in ( Figure 1b ) , on both synthetic and photographic images . Our contributions are therefore three-fold : 1 . We propose scene programs : a new representation for scenes , drawing insights from classic findings in cognitive science and computer graphics . 2 . We present a method for inferring scene programs from images using a hierarchical approach ( from objects to groups to programs ) . 3 . We demonstrate that our model can achieve high accuracy on describing both synthetic and constrained real scenes with programs . Combined with modern image-to-image translation methods , our model generates realistic images of extrapolated scenes , capturing both highlevel scene structure and low-level object appearance . 2 RELATED WORK . Describing Images with Programs Ellis et al . ( 2018 ) performs a similar task as ours where handdrawn images of 2D geometry primitives are converted to high-level programs . This work uses a constraint-based SAT solver to perform program search and is much slower than neural network models . IM2LATEX ( Deng et al. , 2017 ) de-renders images into low-level LATEX markup using a neural network , while our work discovers high-level programs from an image of objects . SPIRAL ( Ganin et al. , 2018 ) uses reinforcement learning to infer a sequence of low-level drawing commands that can reproduce an image , meanwhile learning a distribution from which images can be sampled . Beltramelli ( 2018 ) learns to convert GUI images to markup-like code . Unlike these papers , our model performs program induction in 3D and infers high-level structural patterns both in object layout and color . Describing the Structure of 3D Shapes and Scenes Beyond 2D images , prior work in vision and graphics has attempted to infer high-level structure from 3D objects and 3D scenes . The most relevant to our approach are those that extract a so-called symmetry hierarchy , in which 3D geometry is hierarchically grouped by either attachment or symmetric relationships ( Wang et al. , 2011 ) . This representation has been used to train generative models of 3D shapes ( Li et al. , 2017 ) and indoor 3D scenes ( Li et al. , 2019 ) , as well as to infer a hierarchical bounding box structure from a single image of a 3D shape ( Niu et al. , 2018 ) . Our program representation bears some resemblance to the symmetry hierarchy , but it generalizes to repetitive patterns beyond symmetries and also models patterns in object visual attributes ( e.g. , color ) . CSGNet ( Sharma et al. , 2018 ) learns to parse shapes with a set of primitive and arithmetic commands ; Tulsiani et al . ( 2017 ) parses shapes into an assembly of geometric primitives . In this paper , we focus on learning the high-level scene regularities described by loop structures . Neural Program Synthesis In general , a program synthesis model outputs an explicit program by learning from examples . Recent works on neural program synthesis include R3NN ( Parisotto et al. , 2017 ) and RobustFill ( Devlin et al. , 2017 ) , which perform end-to-end program synthesis from input/output examples . Bunel et al . ( 2018 ) goes beyond the pure supervised learning setting and improves performance on diversity and syntax by leveraging grammar and reinforcement learning . These models synthesize programs based on input/output pairs , which is different from our setting , where a program is generated to describe an input image . In the vision domain , Sun et al . ( 2018 ) learns decision strategies represented as programs from demonstration videos , while we focus on describing the complex correlations among objects in static scenes . 3 METHOD . Our model combines vision and sequence models via structured representations . An object parser predicts the segmentation mask and attributes for each object in the image . A group recognizer predicts the group that each object belongs to . Finally , a program synthesizer generates a program block for each object group . Figure 2 shows an example of synthesizing programs from an input image , where a sphere is selected at random ( highlighted ) and the group that this object belongs to is predicted , which consists of six spheres . Then the program for this group ( highlighted ) is synthesized . 3.1 A DOMAIN-SPECIFIC LANGUAGE ( DSL ) FOR SCENES . In order to constrain the program space to make it tractable for our models , we introduce human prior on scene regularities that can be described as programs . More specifically , we introduce a Domain Specific Language ( DSL ) which explicitly defines the space of our scene programs . We present the grammar of our DSL in Table 1 , which contains 3 primitive commands ( cube , sphere , cylinder ) and 2 loop structures ( for , rotate ) . The positions for each object are defined as affine transformations of loop indices , while the colors are more complicated functions of the loop indices , displaying alternating ( modular ) and repeating ( division ) patterns . Furthermore , since the DSL allows unbounded program depth , we define program blocks to further reduce complexity . Each type of program block is an production instance of the Statement token , and objects that belong to the same block form a group . For example , in this work the program blocks include single objects , layered for loops of depth ≤ 3 , and single-layer rotations of ≤ 4 objects . 3.2 OBJECT PARSING . Following the spirit of The Trace Hypothesis ( Ellis et al. , 2018 ) , we use object attributes as an intermediate representation between image space and structured program space . Parsing individual objects from the input image consists of two steps : mask prediction and attribute prediction . For each object , its instance segmentation mask is predicted by a Mask R-CNN ( He et al. , 2017 ) . Next , the mask is concatenated with the original image , and sent to a ResNet-34 ( He et al. , 2015 ) to predict object attributes . In our work , object attributes include shape , size , material , color and 3D coordinates . Each attribute is encoded as a one-hot vector , except for coordinates . The overall representation of an object is a vector of length 18 . The networks are trained with ground truth masks and attributes , respectively . For the attribute network , we minimize the mean-squared error between output and ground truth attributes . 3.3 GROUP DETECTION . When we identify a distinct visual pattern , we first know which objects in the image form the pattern before we can tell what the pattern is . Motivated by this idea , we develop a group recognizer that tells us which objects form a group that can be described by a single program block . The group recognizer works after mask prediction is performed , and answers the following specific question : given an input object , which objects are in the same group with this object ? The input to the model consists of three parts : the original image , the mask of the input object , and the mask of all objects . These three parts are concatenated and sent to a ResNet-152 followed by fully connected layers . The output contains two parts : a binary vector g where g [ i ] = 1 denotes object i in the same group with the input object , and the category c of the group , representing the type of program block that this group belongs to . The network is trained to minimize the binary cross entropy loss for group recognition , and the cross entropy loss for category classification .

Summary: This paper presents a system that infers programs describing 3D scenes composed of simple primitives. The system consists of three stages each of which is trained separately. First, the perceptual module extracts object masks and their attributes. The objects are then are split into several groups. Finally, each group is mapped to a corresponding DSL program using a sequence-to-sequence network similar to the ones typically employed in neural machine translation.