FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations

We present a meta-learning framework for learning new visual concepts quickly , from just one or a few examples , guided by multiple naturally occurring data streams : simultaneously looking at images , reading sentences that describe the objects in the scene , and interpreting supplemental sentences that relate the novel concept with other concepts . The learned concepts support downstream applications , such as answering questions by reasoning about unseen images . Our model , namely FALCON , represents individual visual concepts , such as colors and shapes , as axis-aligned boxes in a high-dimensional space ( the “ box embedding space ” ) . Given an input image and its paired sentence , our model first resolves the referential expression in the sentence and associate the novel concept with particular objects in the scene . Next , our model interprets supplemental sentences to relate the novel concept with other known concepts , such as “ X has property Y ” or “ X is a kind of Y ” . Finally , it infers an optimal box embedding for the novel concept that jointly 1 ) maximizes the likelihood of the observed instances in the image , and 2 ) satisfies the relationships between the novel concepts and the known ones . We demonstrate the effectiveness of our model on both synthetic and real-world datasets . 1 INTRODUCTION . Humans build a cumulative knowledge repository of visual concepts throughout their lives from a diverse set of inputs : by looking at images , reading sentences that describe the properties of the concept , etc . Importantly , adding a novel concept to the knowledge repository requires only a small amount of data , such as a few images about the concept and a short textual description ( Bloom , 2000 ; Swingley , Daniel , 2010 ; Carey & Bartlett , 1978 ) . Take Fig . 1 as an example : from just a single image that contains many objects ( Fig . 1a ) , as well as a short descriptive sentence that describes a new concept red : “ the object left of the yellow cube is red ” , humans can effortlessly ground the novel word “ red ” with the visual appearance of the object being referred to ( Bloom , 2000 ) . Supplemental sentences such as “ red is a kind of color ” ( Fig . 1b ) may provide additional information : “ red ” objects should be classified based on their hue . This further supports us to generalize the learned concept “ red ” to objects of various shapes , sizes , and materials . Finally , the acquired concept can be used flexibly in other tasks such as question answering ( Fig . 1c ) . Our goal is to build machines that can learn concepts that are associated with the physical world in an incremental manner and flexibly use them to answer queries . To learn a new concept , for example , the word red in Fig . 1a , the system should 1 ) interpret the semantics of the descriptive sentence composed of other concepts , such as left , yellow , and cube , 2 ) instantiate a representation for the novel concept that is associated with the visual appearance of the referred object , 3 ) mediate the concept representation based on the supplemental sentences that describe the property of the concept or relate it with other concepts , and 4 ) use the learned concept flexibly in different concepts . A framework that can solve these challenges will allow us to build machines that can better learn from human and communicate with human . To address these challenges , in this paper , we present a unified framework , FALCON ( FAst Learning of novel visual CONcepts ) . FALCON maintains a collection of embedding vectors for individual visual concepts , which naturally grows in an incremental way as it learns more concepts . A neuro- symbolic concept learning and reasoning framework learns new concepts by looking at images and reading paired sentences , and use them to answer incoming queries . Concretely , FALCON represents individual visual concepts , such as colors and shapes , as axis-aligned boxes in a high-dimensional space ( the “ box embedding space ” ( Vilnis et al. , 2018 ) ) , while objects in different images will be embedded into the same latent space as points . We say object X has property Y if the embedding vector X is inside the embedding box of Y . Given an input image and its paired sentence , our model first resolves the referential expression in the sentence using the previously seen concepts ( e.g. , left , yellow , and cube ) and associate the novel concept with particular objects in the scene . Next , our model interprets supplemental sentences to relate the novel concept with other known concepts ( e.g. , yellow ) . To infer the box embedding of the novel concept , we train a neural network to predict the optimal box embedding for the novel concept that jointly 1 ) maximizes the data likelihood of the observed examples , and 2 ) satisfies the relationships between the novel concepts and the known ones . This module is trained with a meta-learning procedure . Our paper makes the following contributions . First , we present a unified neuro-symbolic framework for fast visual concept learning from diverse data streams . Second , we introduce a new concept embedding prediction module that learns to integrate visual examples and conceptual relations to infer a novel concept embedding . Finally , we build a protocol for generating meta-learning test cases for evaluating fast visual concept learning , by augmenting existing visual reasoning datasets and knowledge graphs . By evaluation on both synthetic and natural image datasets , we show that our model learns more accurate representations for novel concepts compared with existing baselines for fast concept learning . Systematical studies also show that our model can efficiently use the supplemental concept descriptions to resolve ambiguities in the visual examples . We also provide discussions about the design of different modules in our system . 2 RELATED WORKS . Visual concept learning and visual reasoning . Visual reasoning aims to reason about object properties and their relationships in given images , usually evaluated as the question-answering accuracy ( Johnson et al. , 2017a ; Hudson & Manning , 2018 ; Mascharka et al. , 2018 ; Hu et al. , 2018 ) . Recently , there has been an increasing amount of work has been focusing on using neuro-symbolic frameworks to bridge visual concept learning and visual reasoning ( Yi et al. , 2018 ; Mao et al. , 2019 ; Li et al. , 2020 ) . The high-level idea is to disentangle concept learning : association of linguistic units with visual representations , and reasoning : the ability to count objects or make queries . Han et al . ( 2019 ) recently shown how jointly learning concepts and metaconcepts can help each other . Our work is an novel approach towards making use of the metaconcepts in a meta-learning setting aiming at boost the learning of novel concepts based on known ones . Few-shot visual learning . Recent work has studied learning to classify visual scene with very limited labeled examples ( Vinyals et al. , 2016 ; Sung et al. , 2018 ; Snell et al. , 2017 ) or even without any example ( Wang et al. , 2018 ; Kampffmeyer et al. , 2019 ; Tian et al. , 2020 ) . For few-shot learning , existing work proposes to compare the similarity , such as cosine similarity ( Vinyals et al. , 2016 ) and Euclidean distance ( Snell et al. , 2017 ) , between examples , while ( Sung et al. , 2018 ) introduces a learnable module to predict such similarities . In addition , ( Gidaris & Komodakis , 2018 ) learns a weight generator to predict the classifier for classes with very few examples . ( Ravi & Larochelle , 2017 ; Finn et al. , 2017 ; Nichol et al. , 2018 ) address this problem by learning the initialization for gradient-based optimization .. ( Santoro et al. , 2016 ) used external memory to facilitate learning process , while ( Munkhdalai & Yu , 2017 ) uses meta-knowledge among task for rapid adaptation . Our module design is inspired by these work , but we use a language interface : novel concepts are learnt from paired images and texts and evaluated on visual reasoning tasks . Geometric embeddings . In contrast to representing concepts in vector spaces ( Kiros et al. , 2014 ) , a geometric embedding framework associates each concept with a geometric entity such as a Gaussian distribution ( Vilnis & McCallum , 2015 ) , the intersection of hyperplanes ( Vendrov et al. , 2016 ; Vilnis et al. , 2018 ) , and a hyperbolic cone ( Ganea et al. , 2018 ) . Among them , box embeddings ( Vilnis et al. , 2018 ) which map each concept to a hyper-box in the high-dimensional space , have been popular for concept representation : ( Li , Xiang and Vilnis , Luke and Zhang , Dongxu and Boratko , Michael and McCallum , Andrew , 2019 ) proposed a smooth training objective , and ( Ren et al. , 2020 ) uses box embeddings for reasoning over knowledge graphs . In this paper , we extend the box embedding from knowledge graphs to visual domains , and compare it with other concept embeddings . 3 FALCON . The proposed model , FALCON , learns visual concepts by simultaneously looking at images , reading sentences that describe the objects in the scene , and interpreting supplemental sentences that describe the properties of the novel concepts . FALCON learns to learn novel concepts quickly and in a continual manner . We start with a formal definition of our fast and continual concept learning task . Problem formulation . Each concept learning task is a 4-tuple ( c , Xc , Dc , Tc ) . Denote c as the novel concept to be learned ( e.g. , red ) . Models learn to recognize red objects by looking at paired images xi and sentences yi : Xc = { ( xi , yi ) } . Optionally , supplementary sentences Dc = { di } describe the concept c by relating it to other known concepts . After learning , the model will be tested on downstream tasks . In this paper , we specifically focus on visual reasoning : the ability to answer questions about objects in the testing set Tc , which is represented as pairs of images and questions . There are two possible options to approach this problem . One is manually specifying rules to compute the representation for the new concept . In this paper , we focus on a meta-learning approach : to build a system that can learn to learn new concept . Our training data is data tuples for a set of training concepts ( base concepts , Cbase ) . After training , the system is evaluated on a collection of novel concepts ( Ctest ) . That is , we will provide our system with Xc and Dc for a novel concept c , and test it on visual reasoning data Tc . Thus , the system works in a continual learning fashion : the description of a new concept depends on a previously learned concept . Overview . Fig . 2 gives an overview of our proposed model , FALCON . Our key idea is to represent each concept as an axis-aligned box in a high-dimensional space ( the “ box embedding space ” , Section 3.1 ) . Given example images xi and descriptions yi ( Fig . 2a ) , FALCON interprets the referential expression in yi as a symbolic program ( Fig . 2a2 ) . An neuro-symbolic reasoning module executes the inferred program to locate the object being referred to ( Fig . 2a3 ) , see Section 3.2 . Meanwhile , supplementary descriptions Dc ( Fig . 2b ) will be translated into relational representations of concepts ( Fig . 2b1 ) , i.e. , how the new concept c relates to other known concepts . Based on the examples of the novel concept and its relationships with other concepts , we formulate the task of novel concept learning as learning to infer the best concept embedding for c in the box embedding space , evaluated on downstream tasks ( Fig . 2c ) . Once the model learned , we will be using the same neuro-symbolic module for answering questions in the testing set Tc .

Summary: The paper describes an approach to learn visual concepts with a few examples. The paper considers a problem consisting of paired images and sentences, optional concept descriptions, and the target question answering task for a new concept. The proposed approach utilizes the object detector (He 2017) to map visual objects or relationship into the box embedding space (Vilnis 2018), and also utilizes a neuro-symbolic program to identify the referring object as well as relationship among concepts. The new concept representation is inferred after the two graph neural networks, and the networks are trained to solve for the downstream task, which in this paper is question answering. The proposed approach is evaluated in three benchmarks (CUB, CLEVR, GQA) and shown to outperform the recent baselines (Hudson &Manning 2018, Mao 2019).