The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

1 INTRODUCTION . Humans are capable of learning visual concepts by jointly understanding vision and language ( Fazly et al. , 2010 ; Chrupała et al. , 2015 ; Gauthier et al. , 2018 ) . Consider the example shown in Figure 1-I . Imagine that someone with no prior knowledge of colors is presented with the images of the red and green cubes , paired with the questions and answers . They can easily identify the difference in objects ’ visual appearance ( in this case , color ) , and align it to the corresponding words in the questions and answers ( Red and Green ) . Other object attributes ( e.g. , shape ) can be learned in a similar fashion . Starting from there , humans are able to inductively learn the correspondence between visual concepts and word semantics ( e.g. , spatial relations and referential expressions , Figure 1-II ) , and unravel compositional logic from complex questions assisted by the learned visual concepts ( Figure 1-III , also see Abend et al . ( 2017 ) ) . Motivated by this , we propose the neuro-symbolic concept learner ( NS-CL ) , which jointly learns visual perception , words , and semantic language parsing from images and question-answer pairs . NS-CL has three modules : a neural-based perception module that extracts object-level representations from the scene , a visually-grounded semantic parser for translating questions into executable programs , and a symbolic program executor that reads out the perceptual representation of objects , classifies their attributes/relations , and executes the program to obtain an answer . Project page : http : //nscl.csail.mit.edu I . Learning basic , object-based concepts . II . Learning relational concepts based on referential expressions .. NS-CL learns from natural supervision ( i.e. , images and QA pairs ) , requiring no annotations on images or semantic programs for sentences . Instead , analogical to human concept learning , it learns via curriculum learning . NS-CL starts by learning representations/concepts of individual objects from short questions ( e.g. , What ’ s the color of the cylinder ? ) on simple scenes ( ≤3 objects ) . By doing so , it learns object-based concepts such as colors and shapes . NS-CL then learns relational concepts by leveraging these object-based concepts to interpret object referrals ( e.g. , Is there a box right of a cylinder ? ) . The model iteratively adapts to more complex scenes and highly compositional questions . NS-CL ’ s modularized design enables interpretable , robust , and accurate visual reasoning : it achieves state-of-the-art performance on the CLEVR dataset ( Johnson et al. , 2017a ) . More importantly , it naturally learns disentangled visual and language concepts , enabling combinatorial generalization w.r.t . both visual scenes and semantic programs . In particular , we demonstrate four forms of generalization . First , NS-CL generalizes to scenes with more objects and longer semantic programs than those in the training set . Second , it generalizes to new visual attribute compositions , as demonstrated on the CLEVR-CoGenT ( Johnson et al. , 2017a ) dataset . Third , it enables fast adaptation to novel visual concepts , such as learning a new color . Finally , the learned visual concepts transfer to new tasks , such as image-caption retrieval , without any extra fine-tuning . 2 RELATED WORK . Our model is related to research on joint learning of vision and natural language . In particular , there are many papers that learn visual concepts from descriptive languages , such as image-captioning or visually-grounded question-answer pairs ( Kiros et al. , 2014 ; Shi et al. , 2018 ; Mao et al. , 2016 ; Vendrov et al. , 2016 ; Ganju et al. , 2017 ) , dense language descriptions for scenes ( Johnson et al. , 2016 ) , video-captioning ( Donahue et al. , 2015 ) and video-text alignment ( Zhu et al. , 2015 ) . Visual question answering ( VQA ) stands out as it requires understanding both visual content and language . The state-of-the-art approaches usually use neural attentions ( Malinowski & Fritz , 2014 ; Chen et al. , 2015 ; Yang et al. , 2016 ; Xu & Saenko , 2016 ) . Beyond question answering , Johnson et al . ( 2017a ) proposed the CLEVR ( VQA ) dataset to diagnose reasoning models . CLEVR contains synthetic visual scenes and questions generated from latent programs . Table 1 compares our model with state-of-the-art visual reasoning models ( Andreas et al. , 2016 ; Suarez et al. , 2018 ; Santoro et al. , 2017 ) along four directions : visual features , semantics , inference , and the requirement of extra labels . For visual representations , Johnson et al . ( 2017b ) encoded visual scenes into a convolutional feature map for program operators . Mascharka et al . ( 2018 ) ; Hudson & Manning ( 2018 ) used attention as intermediate representations for transparent program execution . Recently , Yi et al . ( 2018 ) explored an interpretable , object-based visual representation for visual reasoning . It performs well , but requires fully-annotated scenes during training . Our model also adopts an object-based visual representation , but the representation is learned only based on natural supervision ( questions and answers ) . Anderson et al . ( 2018 ) also proposed to represent the image as a collection of convolutional object features and gained substantial improvements on VQA . Their model encodes questions with neural networks and answers the questions by question-conditioned attention over the object features . In contrast , NS-CL parses question inputs into programs and executes them on object features to get the answer . This makes the reasoning process interpretable and supports combinatorial generalization over quantities ( e.g. , counting objects ) . Our model also learns general visual concepts and their association with symbolic representations of language . These learned concepts can then be explicitly interpreted and deployed in other vision-language applications such as image caption retrieval . There are two types of approaches in semantic sentence parsing for visual reasoning : implicit programs as conditioned neural operations ( e.g. , conditioned convolution and dual attention ) ( Perez et al. , 2018 ; Hudson & Manning , 2018 ) and explicit programs as sequences of symbolic tokens ( Andreas et al. , 2016 ; Johnson et al. , 2017b ; Mascharka et al. , 2018 ) . As a representative , Andreas et al . ( 2016 ) build modular and structured neural architectures based on programs for answering questions . Explicit programs gain better interpretability , but usually require extra supervision such as groundtruth program annotations for training . This restricts their application . We propose to use visual grounding as distant supervision to parse questions in natural languages into explicit programs , with zero program annotations . Given the semantic parsing of questions into programs , Yi et al . ( 2018 ) proposed a purely symbolic executor for the inference of the answer in the logic space . Compared with theirs , we propose a quasi-symbolic executor for VQA . Our work is also related to learning interpretable and disentangled representations for visual scenes using neural networks . Kulkarni et al . ( 2015 ) proposed convolutional inverse graphics networks for learning and inferring pose of faces , while Yang et al . ( 2015 ) learned disentangled representation of pose of chairs from images . Wu et al . ( 2017 ) proposed the neural scene de-rendering framework as an inverse process of any rendering process . Siddharth et al . ( 2017 ) ; Higgins et al . ( 2018 ) learned disentangled representations using deep generative models . In contrast , we propose an alternative representation learning approach through joint reasoning with language . 3 NEURO-SYMBOLIC CONCEPT LEARNER . We present our neuro-symbolic concept learner , which uses a symbolic reasoning process to bridge the learning of visual concepts , words , and semantic parsing of sentences without explicit annotations for any of them . We first use a visual perception module to construct an object-based representation for a scene , and run a semantic parsing module to translate a question into an executable program . We then apply a quasi-symbolic program executor to infer the answer based on the scene representation . We use paired images , questions , and answers to jointly train the visual and language modules . Shown in Figure 2 , given an input image , the visual perception module detects objects in the scene and extracts a deep , latent representation for each of them . The semantic parsing module translates an input question in natural language into an executable program given a domain specific language ( DSL ) . The generated programs have a hierarchical structure of symbolic , functional modules , each fulfilling a specific operation over the scene representation . The explicit program semantics enjoys compositionality , interpretability , and generalizability . The program executor executes the program upon the derived scene representation and answers the question . Our program executor works in a symbolic and deterministic manner . This feature ensures a transparent execution trace of the program . Our program executor has a fully differentiable design w.r.t . the visual representations and the concept representations , which supports gradient-based optimization during training . 3.1 MODEL DETAILS . Visual perception . Shown in Figure 2 , given the input image , we use a pretrained Mask R-CNN ( He et al. , 2017 ) to generate object proposals for all objects . The bounding box for each single object paired with the original image is then sent to a ResNet-34 ( He et al. , 2015 ) to extract the region-based ( by RoI Align ) and image-based features respectively . We concatenate them to represent each object . Here , the inclusion of the representation of the full scene adds the contextual information , which is essential for the inference of relative attributes such as size or spatial position . Concept quantization . Visual reasoning requires determining an object ’ s attributes ( e.g. , its color or shape ) . We assume each visual attribute ( e.g. , shape ) contains a set of visual concept ( e.g. , Cube ) . In NS-CL , visual attributes are implemented as neural operators , mapping the object representation into an attribute-specific embedding space . Figure 3 shows an inference an object ’ s shape . Visual concepts that belong to the shape attribute , including Cube , Sphere and Cylinder , are represented as vectors in the shape embedding space . These concept vectors are also learned along the process . We measure the cosine distances 〈· , ·〉 between these vectors to determine the shape of the object . Specifically , we compute the probability that an object oi is a cube by σ ( 〈ShapeOf ( oi ) , vCube〉 − γ ) / τ , where ShapeOf ( · ) denotes the neural operator , vCube the concept embedding of Cube and σ the Sigmoid function . γ and τ are scalar constants for scaling and shifting the values of similarities . We classify relational concepts ( e.g. , Left ) between a pair of objects similarly , except that we concatenate the visual representations for both objects to form the representation of their relation . DSL and semantic parsing . The semantic parsing module translates a natural language question into an executable program with a hierarchy of primitive operations , represented in a domain-specific language ( DSL ) designed for VQA . The DSL covers a set of fundamental operations for visual reasoning , such as filtering out objects with certain concepts or query the attribute of an object . The operations share the same input and output interface , and thus can be compositionally combined to form programs of any complexity . We include a complete specification of the DSL used by our framework in the Appendix A . A. Curriculum concept learning B. Illustrative execution of NS-CL Our semantic parser generates the hierarchies of latent programs in a sequence to tree manner ( Dong & Lapata , 2016 ) . We use a bidirectional GRU ( Cho et al. , 2014 ) to encode an input question , which outputs a fixed-length embedding of the question . A decoder based on GRU cells is applied to the embedding , and recovers the hierarchy of operations as the latent program . Some operations takes concepts their parameters , such as Filter ( Red ) and Query ( Shape ) . These concepts are chosen from all concepts appeared in the input question . Figure 4 ( B ) shows an example , while more details can be found in Appendix B. Quasi-symbolic program execution . Given the latent program recovered from the question in natural language , a symbolic program executor executes the program and derives the answer based on the object-based visual representation . Our program executor is a collection of deterministic functional modules designed to realize all logic operations specified in the DSL . Figure 4 ( B ) shows an illustrative execution trace of a program . To make the execution differentiable w.r.t . visual representations , we represent the intermediate results in a probabilistic manner : a set of objects is represented by a vector , as the attention mask over all objects in the scene . Each element , Maski ∈ [ 0 , 1 ] denotes the probability that the i-th object of the scene belongs to the set . For example , shown in Figure 4 ( B ) , the first Filter operation outputs a mask of length 4 ( there are in total 4 objects in the scene ) , with each element representing the probability that the corresponding object is selected out ( i.e. , the probability that each object is a green cube ) . The output “ mask ” on the objects will be fed into the next module ( Relate in this case ) as input and the execution of programs continues . The last module outputs the final answer to the question . We refer interested readers to Appendix C for the implementation of all operators .

Summary: The paper proposes a joint learning of visual representation and word and semantic parsing of the sentences given paired images and paired Q/A with a model called neuro-symbolic concept learner using curriculum learning. The paper reads well and is easy to follow. The idea of jointly learning visual concepts and language is an important task. Human reasoning involves learning and recall from multiple moralities. The authors use the CLEVR dataset for evaluation.