FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations

We present a meta-learning framework for learning new visual concepts quickly , from just one or a few examples , guided by multiple naturally occurring data streams : simultaneously looking at images , reading sentences that describe the objects in the scene , and interpreting supplemental sentences that relate the novel concept with other concepts . The learned concepts support downstream applications , such as answering questions by reasoning about unseen images . Our model , namely FALCON , represents individual visual concepts , such as colors and shapes , as axis-aligned boxes in a high-dimensional space ( the “ box embedding space ” ) . Given an input image and its paired sentence , our model first resolves the referential expression in the sentence and associate the novel concept with particular objects in the scene . Next , our model interprets supplemental sentences to relate the novel concept with other known concepts , such as “ X has property Y ” or “ X is a kind of Y ” . Finally , it infers an optimal box embedding for the novel concept that jointly 1 ) maximizes the likelihood of the observed instances in the image , and 2 ) satisfies the relationships between the novel concepts and the known ones . We demonstrate the effectiveness of our model on both synthetic and real-world datasets . 1 INTRODUCTION . Humans build a cumulative knowledge repository of visual concepts throughout their lives from a diverse set of inputs : by looking at images , reading sentences that describe the properties of the concept , etc . Importantly , adding a novel concept to the knowledge repository requires only a small amount of data , such as a few images about the concept and a short textual description ( Bloom , 2000 ; Swingley , Daniel , 2010 ; Carey & Bartlett , 1978 ) . Take Fig . 1 as an example : from just a single image that contains many objects ( Fig . 1a ) , as well as a short descriptive sentence that describes a new concept red : “ the object left of the yellow cube is red ” , humans can effortlessly ground the novel word “ red ” with the visual appearance of the object being referred to ( Bloom , 2000 ) . Supplemental sentences such as “ red is a kind of color ” ( Fig . 1b ) may provide additional information : “ red ” objects should be classified based on their hue . This further supports us to generalize the learned concept “ red ” to objects of various shapes , sizes , and materials . Finally , the acquired concept can be used flexibly in other tasks such as question answering ( Fig . 1c ) . Our goal is to build machines that can learn concepts that are associated with the physical world in an incremental manner and flexibly use them to answer queries . To learn a new concept , for example , the word red in Fig . 1a , the system should 1 ) interpret the semantics of the descriptive sentence composed of other concepts , such as left , yellow , and cube , 2 ) instantiate a representation for the novel concept that is associated with the visual appearance of the referred object , 3 ) mediate the concept representation based on the supplemental sentences that describe the property of the concept or relate it with other concepts , and 4 ) use the learned concept flexibly in different concepts . A framework that can solve these challenges will allow us to build machines that can better learn from human and communicate with human . To address these challenges , in this paper , we present a unified framework , FALCON ( FAst Learning of novel visual CONcepts ) . FALCON maintains a collection of embedding vectors for individual visual concepts , which naturally grows in an incremental way as it learns more concepts . A neuro- symbolic concept learning and reasoning framework learns new concepts by looking at images and reading paired sentences , and use them to answer incoming queries . Concretely , FALCON represents individual visual concepts , such as colors and shapes , as axis-aligned boxes in a high-dimensional space ( the “ box embedding space ” ( Vilnis et al. , 2018 ) ) , while objects in different images will be embedded into the same latent space as points . We say object X has property Y if the embedding vector X is inside the embedding box of Y . Given an input image and its paired sentence , our model first resolves the referential expression in the sentence using the previously seen concepts ( e.g. , left , yellow , and cube ) and associate the novel concept with particular objects in the scene . Next , our model interprets supplemental sentences to relate the novel concept with other known concepts ( e.g. , yellow ) . To infer the box embedding of the novel concept , we train a neural network to predict the optimal box embedding for the novel concept that jointly 1 ) maximizes the data likelihood of the observed examples , and 2 ) satisfies the relationships between the novel concepts and the known ones . This module is trained with a meta-learning procedure . Our paper makes the following contributions . First , we present a unified neuro-symbolic framework for fast visual concept learning from diverse data streams . Second , we introduce a new concept embedding prediction module that learns to integrate visual examples and conceptual relations to infer a novel concept embedding . Finally , we build a protocol for generating meta-learning test cases for evaluating fast visual concept learning , by augmenting existing visual reasoning datasets and knowledge graphs . By evaluation on both synthetic and natural image datasets , we show that our model learns more accurate representations for novel concepts compared with existing baselines for fast concept learning . Systematical studies also show that our model can efficiently use the supplemental concept descriptions to resolve ambiguities in the visual examples . We also provide discussions about the design of different modules in our system . 2 RELATED WORKS . Visual concept learning and visual reasoning . Visual reasoning aims to reason about object properties and their relationships in given images , usually evaluated as the question-answering accuracy ( Johnson et al. , 2017a ; Hudson & Manning , 2018 ; Mascharka et al. , 2018 ; Hu et al. , 2018 ) . Recently , there has been an increasing amount of work has been focusing on using neuro-symbolic frameworks to bridge visual concept learning and visual reasoning ( Yi et al. , 2018 ; Mao et al. , 2019 ; Li et al. , 2020 ) . The high-level idea is to disentangle concept learning : association of linguistic units with visual representations , and reasoning : the ability to count objects or make queries . Han et al . ( 2019 ) recently shown how jointly learning concepts and metaconcepts can help each other . Our work is an novel approach towards making use of the metaconcepts in a meta-learning setting aiming at boost the learning of novel concepts based on known ones . Few-shot visual learning . Recent work has studied learning to classify visual scene with very limited labeled examples ( Vinyals et al. , 2016 ; Sung et al. , 2018 ; Snell et al. , 2017 ) or even without any example ( Wang et al. , 2018 ; Kampffmeyer et al. , 2019 ; Tian et al. , 2020 ) . For few-shot learning , existing work proposes to compare the similarity , such as cosine similarity ( Vinyals et al. , 2016 ) and Euclidean distance ( Snell et al. , 2017 ) , between examples , while ( Sung et al. , 2018 ) introduces a learnable module to predict such similarities . In addition , ( Gidaris & Komodakis , 2018 ) learns a weight generator to predict the classifier for classes with very few examples . ( Ravi & Larochelle , 2017 ; Finn et al. , 2017 ; Nichol et al. , 2018 ) address this problem by learning the initialization for gradient-based optimization .. ( Santoro et al. , 2016 ) used external memory to facilitate learning process , while ( Munkhdalai & Yu , 2017 ) uses meta-knowledge among task for rapid adaptation . Our module design is inspired by these work , but we use a language interface : novel concepts are learnt from paired images and texts and evaluated on visual reasoning tasks . Geometric embeddings . In contrast to representing concepts in vector spaces ( Kiros et al. , 2014 ) , a geometric embedding framework associates each concept with a geometric entity such as a Gaussian distribution ( Vilnis & McCallum , 2015 ) , the intersection of hyperplanes ( Vendrov et al. , 2016 ; Vilnis et al. , 2018 ) , and a hyperbolic cone ( Ganea et al. , 2018 ) . Among them , box embeddings ( Vilnis et al. , 2018 ) which map each concept to a hyper-box in the high-dimensional space , have been popular for concept representation : ( Li , Xiang and Vilnis , Luke and Zhang , Dongxu and Boratko , Michael and McCallum , Andrew , 2019 ) proposed a smooth training objective , and ( Ren et al. , 2020 ) uses box embeddings for reasoning over knowledge graphs . In this paper , we extend the box embedding from knowledge graphs to visual domains , and compare it with other concept embeddings . 3 FALCON . The proposed model , FALCON , learns visual concepts by simultaneously looking at images , reading sentences that describe the objects in the scene , and interpreting supplemental sentences that describe the properties of the novel concepts . FALCON learns to learn novel concepts quickly and in a continual manner . We start with a formal definition of our fast and continual concept learning task . Problem formulation . Each concept learning task is a 4-tuple ( c , Xc , Dc , Tc ) . Denote c as the novel concept to be learned ( e.g. , red ) . Models learn to recognize red objects by looking at paired images xi and sentences yi : Xc = { ( xi , yi ) } . Optionally , supplementary sentences Dc = { di } describe the concept c by relating it to other known concepts . After learning , the model will be tested on downstream tasks . In this paper , we specifically focus on visual reasoning : the ability to answer questions about objects in the testing set Tc , which is represented as pairs of images and questions . There are two possible options to approach this problem . One is manually specifying rules to compute the representation for the new concept . In this paper , we focus on a meta-learning approach : to build a system that can learn to learn new concept . Our training data is data tuples for a set of training concepts ( base concepts , Cbase ) . After training , the system is evaluated on a collection of novel concepts ( Ctest ) . That is , we will provide our system with Xc and Dc for a novel concept c , and test it on visual reasoning data Tc . Thus , the system works in a continual learning fashion : the description of a new concept depends on a previously learned concept . Overview . Fig . 2 gives an overview of our proposed model , FALCON . Our key idea is to represent each concept as an axis-aligned box in a high-dimensional space ( the “ box embedding space ” , Section 3.1 ) . Given example images xi and descriptions yi ( Fig . 2a ) , FALCON interprets the referential expression in yi as a symbolic program ( Fig . 2a2 ) . An neuro-symbolic reasoning module executes the inferred program to locate the object being referred to ( Fig . 2a3 ) , see Section 3.2 . Meanwhile , supplementary descriptions Dc ( Fig . 2b ) will be translated into relational representations of concepts ( Fig . 2b1 ) , i.e. , how the new concept c relates to other known concepts . Based on the examples of the novel concept and its relationships with other concepts , we formulate the task of novel concept learning as learning to infer the best concept embedding for c in the box embedding space , evaluated on downstream tasks ( Fig . 2c ) . Once the model learned , we will be using the same neuro-symbolic module for answering questions in the testing set Tc .

Summary: This paper presents a unified meta-learning neuro-symbolic framework for fast visual concept learning from diverse data streams. It introduces a new embedding prediction module to integrate visual examples and relations to infer novel concept embeddings. It’s meta-learning continuous learning approach also uses supplementary sentences to relate concepts to one another. They show improvements for the end task of question answering with novel concepts by first meta-learning those concepts during a meta-testing phase.