FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations
Authors: Lingjie Mei
Jiayuan Mao
Ziqi Wang
Chuang Gan
Joshua B. Tenenbaum
Venue: ICLR 2022 Poster (2022)
Keywords: Neuro-Symbolic Reasoning
Concept Learning
Meta-Learning

Abstract: We present a meta-learning framework for learning new visual concepts quickly, from just one or a few examples, guided by multiple naturally occurring data streams: simultaneously looking at images, reading sentences that describe the objects in the scene, and interpreting supplemental sentences that relate the novel concept with other concepts. The learned concepts support downstream applications, such as answering questions by reasoning about unseen images. Our model, namely FALCON, represents individual visual concepts, such as colors and shapes, as axis-aligned boxes in a high-dimensional space (the ``box embedding space''). Given an input image and its paired sentence, our model first resolves the referential expression in the sentence and associates the novel concept with particular objects in the scene. Next, our model interprets supplemental sentences to relate the novel concept with other known concepts, such as ``X has property Y'' or ``X is a kind of Y''. Finally, it infers an optimal box embedding for the novel concept that jointly 1) maximizes the likelihood of the observed instances in the image, and 2) satisfies the relationships between the novel concepts and the known ones. We demonstrate the effectiveness of our model on both synthetic and real-world datasets.

Published as a conference paper at ICLR 2022
FALCON: FAST VISUAL CONCEPT LEARNING BY IN- TEGRATING IMAGES, LINGUISTIC DESCRIPTIONS, AND CONCEPTUAL RELATIONS
Lingjie Mei MIT CSAIL
Jiayuan Mao" MIT CSAIL
Ziqi Wang UIUC
Chuang Gan MIT-IBM Watson AI Lab
Joshua B. Tenenbaum MIT BCS, CBMM, CSAIL
ABSTRACT
We present a meta-learning framework for learning new visual concepts quickly, from just one or a few examples, guided by multiple naturally occurring data streams: simultancously looking at images, reading sentences that describe the objects in the scene, and interpreting supplemental sentences that relate the novel concept with other concepts. The learned concepts support downstream applica- tions, such as answering questions by reasoning about unseen images. Our model, namely FALCON, represents individual visual concepts, such as colors and shapes, axis-aligned boxes in a high-dimensional space (the "box embedding space") Given an input image and its paired sentence, our model first resolves the referential expression in the sentence and associate the novel concept with particular objects in the scene. Next, our model interprets supplemental sentences to relate the novel concept with other known concepts, such as "X has property Y" or "X a kind of Y". Finally, it infers an optimal box embedding for the novel concept that jointly 1) maximizes the likelihood of the observed instances in the image, and 2) satisfies the relationships between the novel concepts and the known ones. We demonstrate the effectiveness of our model on both synthetic and real-world datasets.
INTRODUCTION
Humans build a cumulative knowledge repository of visual concepts throughout their lives from diverse set of inputs: by looking images, reading sentences that describe the properties of the concept, etc. Importantly, adding a novel concept to the knowledge repository requires only a small amount of data, such as few images about the concept and short textual description (Bloom) 2000 Swingley. Daniel, 20100] Carey & Bartlett 1978). Take Fig. T an example: from just a single image that contains many objects [p).. as well as a short descriptive sentence that describes new concept red: "the object left of the yellow cube is red", humans can effortlessly ground the novel word "red" with the visual appearance of the object being referred to (Bloom) (000). Supplemental sentences such as "red is a kind of color" (Fig. [T)) may provide additional information: "red" objects should be classified based on their hue. This further supports us to generalize the learned concept "red" to objects of various shapes, sizes, and materials. Finally, the acquired concept can be used flexibly in other tasks such as question answering (Fig. [T].
Our goal is to build machines that can learn concepts that are associated with the physical world in an incremental manner and flexibly use them to answer queries. To learn a new concept, for example, the word red in Fig. Dh. the system should 1) interpret the semantics of the descriptive sentence composed of other concepts, such as left, yellow, and cube, 2) instantiate a representation for the novel concept with the visual appearance of the referred object, 3) mediate the concept representation based on the supplemental sentences that describe the property of the concept or relate it with other concepts, and use the learned concept flexibly in different scenarios. framework that solve these challenges will allow us to build machines that can better learn from human and communicate with human.
To address these challenges, in this paper, we present unified framework, FALCON (FAst Learning of novel visual CONcepts). FALCON maintains a collection of embedding vectors for individual
"quull contribution.

---

Published as a conference paper at ICLR 2022
Desenin CLEVR (Conoep Rol)
Doneis CUB (Cooept, Wlite-yed Nieo)
Domals RQ4 (Conoeyr Class
Palred Image-Test oabe
(a) Painel Teag--Test.
(a) Pairdd Emage Teet.
Cunrept Learning
the yellow
Tieiz whits-ya
The loee windon glas
cljact (to Sapp. Sentence
Sentence
(b Sentence
Yellow, sreee, nol colors
(Fitt-corl vinco Hadk-capped
Metul alas desetiies propenty shjoa.
difloes vireas
Test Cascs
tdi Text Clies
Step Ressoning
H:Howw traty obsi latze Wite's color of the objects haue same dugpechhj to let oft instal objoat?
sellow splare!
Is sAiieogosfstte
willbugeed sheett
cbjoatt
Vt la Jacie as opce glase
Qu Bux blck dever?
Rot Figure 1: Three illustrative examples of our fast concept learning task. Our model learns from three naturally occurring data streams: (a) looking images, reading sentences that describe the objects in the scene, and (b) interpreting supplemental sentences that relate the novel concept with other concepts. (c) The acquired novel concepts (e.g., red and white-eyed vireo transfer to downstream tasks, such visual reasoning. We present meta-learning framework to solve this task.
A.
A. Yos
AoNo.
No
visual concepts, which naturally grows in an incremental way as it learns more concepts. neuro- symbolic concept learning and reasoning framework learns new concepts by looking at images and reading paired sentences, and use them to answer incoming queries.
Concretely, FALCON represents individual visual concepts, such as colors and shapes, as axis-aligned boxes in a high-dimensional space (the "box embedddigg spuce" (Vilnis et al., 2018)). while objects in different images will be embedded into the same latent space as points. We say object X has property Y if the embedding vector X is inside the embedding box of Y. Given an input image and its paired sentence, our model first resolves the referential expression in the sentence using the previously seen concepts (c.g., left, yellow, and cube) and associate the novel concept with particular objects in the scene. Next, our model interprets supplemental sentences relate the novel concept with other known concepts (e.g.. yellow). To infer the box embedding of the novel concept, we train a neural network predict the optimal box embedding for the novel concept that jointly 1) maximizes the data likelihood of the observed examples, and 2) satisfies the relationships between the novel concepts and the known ones. This module is trained with a meta-learning procedure.
Our paper makes the following contributions. First, we present unified neuro-symbolic framework for fast visual concept learning from diverse data streams. Second, we introduce a new concept embedding prediction module that learns to integrate visual examples and conceptual relations to infer a novel concept embedding. Finally, we build protocol for generating meta-learning test cases for evaluating fast visual concept learning, by augmenting existing visual reasoning datasets and knowledge graphs. By evaluation on both synthetic and natural image datasets, we show that our model learns more accurate representations for novel concepts compared with existing baselines for fast concept learning. Systematical studies also show that our model can efficiently use the supplemental concept descriptions to resolve ambiguities in the visual examples. We also provide discussions about the design of different modules in our system.
2 RELATED WORKS
Visual concept learning and visual reasoning. Visual reasoning aims to reason about object properties and their relationships in given images, usually evaluated as the question-answering accuracy (Johnson et al.) 2017a] Hudson & Manning] 2018 Mascharka et al.) 2018 Hu et al., 2018). Recently, there has been an increasing amount of work has been focusing on using neuro-symbolic frameworks to bridge visual concept learning and visual reasoning (Yi et al. 2018 Mao et al. 2019 Li Liet al. 2020). The high-level idea is to disentangle concept learning: association of linguistic units with visual representations, and reasoning: the ability to count objects or make queries. Han et al. (2019) recently shown how jointly learning concepts and metaconcepts can help each other. Our work is an novel approach towards making use of the metaconcepts in meta-learning setting aiming at boost the learning of novel concepts based on known ones.
Few-shot visual learning. Recent work has studied learning to classify visual scene with very limited labeled examples /Vinyals et al. 2016 Sung et al. 2018 Snell et al. 2017) or even without any example (Wang et al. 2018 Kampffmeyer et al. 2019 Tian al. 2020). For few-shot learning, existing work proposes to compare the similarity, such as cosine similarity VVnyyals et al. 2016) and Euclidean distance (Snell et al. 2017), between examples, while (Sung et al. 2018) introduces learnable module to predict such similarities. In addition, (Gidaris & Komodakis] 2018) learns a weight generator to predict the classifier for classes with very few examples. (Ravi & Larochelle
2

---

Published as a conference paper at ICLR 2022
(a) Paried Image-Text
(a1) Obj-Centrie Repr.
(43) Positive Examples
Nearo-Symoolic
Obj. Det.
Program Exec.
Reusoning
Is there any white-eyed vireo?
Semaalic
(a2) Symbolic Program
white-eyad viren.
Parsing
Unique(Filter) SarppueeeFtrrWWhiieeee WWiiteeyyed Vireo))
White-eyed Virco
(b) Supplemental Sentence
Black capped virco, philadelphia vireo, white-eyed vireo are different vireos.
Semantic Parsing
Black-ciped Vinco
Philadelphia
Virco
GNN;
GNN
Vimo
White eyed Vireo
White-eyed Vireo
Wlite-eyyod
(b1) ConceptCrragh
(b2) Example Graph
Figure 2: Overview FALCON-G. FALCON-G starts from extracting object-centric representations and parse input sentences into semantic programs. It executes the program locate the objects being referred to. The example objects, together with the reconstructed concept graphs are fed into two GNNs sequentially predict the concept embedding for the "white-eyed vireo". The derived concept embedding can be used in downstream tasks such as visual reasoning.
2017 Finn et al. 2017: Nichol et al. 2018) address this problem by learning the initialization for gradient-basee based optimization.. (Santoro al., 2016) used external memory to facilitate learning process, while (Munkhdalai & Yu, 2017) uses meta- kaawlldgo among task for rapid adaptation. Our module design is inspired by these work, but we use a language interface: novel concepts are learnt from paired images and texts and evaluated on visual reasoning tasks.
Geometric embeddings. In contrast to representing concepts in vector spaces (Kiros et al. 2014), geometric embedding framework associates each concept with geometric entity such as Gaussian distribution (Vilnis McCallum 2015), the intersection of hyperplanes (Vendrov et al.) 2016 Vilni al. 2018), and hyperbolic cone (Ganea et al. 2018). Among them, box embeddings Vilnis et al. 2018) which map each concept to a hyper- box in the high- high-dimensional space, have been popular for concept representation: (Li, Xiang and Vilnis, Luke and Zhang, Dongxu and Boratko, Michael and McCallum, Andrew 2019) proposed a smooth training objective, and (Ren et al. (0020) uses box embeddings for reasoning over knowledge graphs. In this paper, we extend the box embedding from knowledge graphs to visual domains, and compare it with other concept embeddings.
3 FALCON
The proposed model, FALCON, learns visual concepts by simultaneously looking at images, reading sentences that describe the objects in the scene, and interpreting supplemental sentences that describe the properties of the novel concepts. FALCON learns to learn novel concepts quickly and in continual manner. We start with a formal definition of our fast and continual concept learning task.
Problem formulation. Each concept learning task is a 4-tuple (c, X.r No D., To). Denote e as the novel concept be learned (e.g., red). Models learn recognize red objects by looking paired images I; and sentences y;: X. = [2,, }.)}. Optionally, supplementary sentences D. = {d;} describe the concept by relating it to other known concepts. After learning, the model will be tested on downstream tasks. In this paper, questions about objects the testing set T., which represented as pairs of images and questions.
specifically focus on visual reasoning: the ability to answer
There are two possible options to approach this problem. One is manually specifying rules to compute the representation for the new concept. In this paper, we focus on a meta-learning approach: to build system that can learn to learn new concept. Our training data is data tuples for set of training concepts (base concepts, Cbase). After training, the system is evaluated on a collection of novel concepts (Ctest). That is, will provide system with X, and D. for novel concept and test it on visual reasoning data T,, Thus, the system works in continual learning fashion: the description of new concept depends on previously learned concept.
3

---

Published as a conference paper at ICLR 2022
Overview. Fig. gives overview of our proposed model, FALCON. Our key idea is represent each concept as an axis-aligned box in a high-dimensional space (the "box embedding space", Section B.I. Given example images I; and descriptions y: (Fig. 2b) FALCON interprets the referential expression in yi as a symbolic program (Fig. 2b)). An neuro-symbolic reasoning module executes the inferred program to locate the object being referred to (Fig. 3)3), see Section 3.2 Meanwhile, supplementary descriptions De (Fig. 2 will be translated into relational representations of concepts (Fig. 1D),, i.c., how the new concept relates other known concepts. Based on the examples of the novel concept and its relationships with other concepts, we formulate the task of novel concept learning as learning to infer the best concept embedding for in the box embedding space, evaluated on downstream tasks (Fig. 2 Once the model learned, we will be using the same neuro-symbolic module for answering questions in the testing set Te.
3.1 VISUAL REPRESENTATIONS AND EMBEDDING SPACES
Given the input image, our model extracts an object-based representation for the scene (Fig. 11). This is done by first using a pretrained Mask-RCNN (He et al. 2017) to detect objects the image. The mask for each object, paired with the original image is sent to ResNet-34 (He et al. 2016) model to extract its visual representation, as a feature vector. For each pair of objects, we will concatenate their feature vectors to form the relational representation between this pair of objects.
The feature vector of each object will be mapped into a geometric embedding space with a fully- connected layer. Each object-based concepts (such as object categories, colors) will be mapped into the same embedding space. Similarly, the relational representation of object pairs, as well as relational concepts (such as spatial relationships) will be mapped into another space. Here, we focus on the box embedding space (Vilnis et al.] 2018) since it naturally models the entailment relationships between concepts. It is also worth noting that other embedding spaces are also compatible with our framework, and we will empirically compare them in Section 4
Box embedding. We slightly extend the box embedding model for knowledge graphs (Vilnis et al. 2018) into visual domains. Throughout this section, we will be using object-based concepts as an example. The idea can be easily applied to relational concepts as well.
Each object- based concept is a tuple of two vectors; = (Cen(e.), (ff(e))), denoting center and offset of the box embedding, both in Rd, where d the embedding dimension. Intuitively, each concept corresponds to an axis-aligned box (hypercube) in the embedding space, centered at Cen(ez) and have edge length 2- (ff(e.). Each object is also box: (Cen(e.), 8), centered at Cen(e, (c.) with fixed edge length of 8 = 10 We constrain that all boxes should reside within 2
For each box embedding (either an embedding of an object, or a concept), we define helper functions Min(e) Cen(e) - Off(ee and Max(e) = Cen(e) (ff denote the lower- and upper-bound of the a box e. The denotational probability is the volume of the box: Pr[e] = II. (Max(c)), -Mn(()) where subscript denotes the i-th dimension of vectors. Obvsiouly, Pr[e] [0, 1]. Moreover, define the joint distribution of two boxes é1 and 62 as the volume of their intersection: Pr[e. neg] =
II: max ((M(e1,) (2); m()),)));))0) O)), where M(e,, 62) Max(e1) v Max(e2), m(),), 62) Min(e:)
e ) A Min(e2)). A and V are element-wise max and min operators. Finally, we define the conditional probability Pr[e lea] = Pr[ei n e2]/ Pr[e2]. This notation is useful in classifying objects: we will use | as the classification confidence of obj being red. Similarly, it also handles entailment of concepts: Pr[eer[cc] readily denotes the probability that concept e entails concept c'. This well suits our need for modeling entailment relations between concepts: in Fig. 2 the supplemental sentence implies "white-eyed vireo entails vireo". box embedding, this means: any instance inside the box "white-eyed vireo" should also be inside the box "vireo." We leave detailed discussions and comparisons with other embedding spaces in Appendix Section C.4
The formulation above has non-continuous gradient at the edges of the box embeddings, following (Li, Xiang and Vilnis, Luke and Zhang, Dongxu and Boratko, Michael and McCallum, Andrew 2019), we "smooth" the computation by replacing all max(-, 0) operations with the softplus operator: softplus(a) x) = T log(1 + exp(#/7) where F is a scalar hyperparameter.
3.2 NEURO-SYMBOLE REASONING
The neuro-symbolic reasoning module processes all linguistic inputs by translating them into sequence of queries in the concept embedding space, organized by a program layout. Specifically, all natural language inputs, including the sentences y; paired with the image examples, the supplementary
4

---

Published as a conference paper at ICLR 2022
sentences di and questions in Te will be parsed into symbolic programs, using the same grammar as (Han et al., 2019). Visually grounded questions including yi and questions in T. will be parsed into programs with hierarchical layouts. They contain primitive operations such Filter for locating objects with a specific color or shape, and Query for query the attribute of a specified object. Appendix Section CaationnCoonttr contains formal definition of grammar. Meanwhile, supplementary sentences de will be translated as conjunctions of fact tuples: e.g. (red, is, color), which denotes the extra information that the new concept red a kind of color. We use pre-trained semantic parsers to translate the natural language inputs into such symbolic representations.
Next, we locate the object being referred to in each (.z y/) pair. This is done by executing the program on the extracted object representations and concept embeddings. Specifically, we apply the neuro-symbolie program executor proposed in (Mao et al. 2019). contains deterministic functional modules for each primitive operations. The execution result is a distribution Pj over all objects j in image F4, interpreted as the probability that the j-th object being referred to.
The high-level idea of the neuro-symbolic program executor is to smooth the boolean value in deterministic program execution into scores ranging from 0 to 1. For example, in Fig. 2 the execution result of the first Filter [ [white-eyed vireo] operation yields to a vector with each entry v, denoting the probability that the i-th object is red. The scores are computed by computing the probability eyCCwhite-eyed- lea] for all object embeddings e; in the box embedding space. The output is fully differentiable w.r.t. the concept embeddings and the object embeddings. Similarly, the concept embeddings can be used to answer downstream visual reasoning questions.
3.3 LEARNING TO LEARN CONCEPTS
The embedding prediction module takes the located objects {o(") of the novel concept e in the image, as well as the relationship between e and other concepts as input. outputs a box embedding for the new concept e.. Here, the relationship between the concepts is represented as a collection of 3-tuples (c, rel), where é' is an known concept and rel denotes the relationship between e and d'
At high level, the inference process has two steps. First, we infer N candidate box embeddings based on their relationship with other concepts, as well as the example objects. Next, we select the concept embedding candidate with the highest data likelihood: IT. Pr(eclo,"") indexes over M example objects. The data likelihood is computed by the denotational probabilities. Following we discuss two candidate models for generating candidate concept embeddings.
(()), where i
Graph neural networks. natural way to model relational structures between concepts and example objects is to use graph neural networks (Hamilton et al. (0177). Specifically, we represent each concept as a node in the graph, whose node embeddings are initialized as the concept embedding 6,, For each relation tuple (c, c', rel), we connect and c'. The edge embedding is initialized based on their relationships rel. We denote the derived graph as Geoncepe (Fig. 2 (b1)). We also handle
example objects in similar way: for each example object % we initialize node with its visual representation, and connect with c. We denote the derived graph as Genample (Fig. 2 (b2)).
To generate candidate concept embeddings, we draw N initial guesses ei." ;i= = I,... N i.idd. from the prior distribution pa. For each candidate ei.0 we feed it to two graph neural networks sequentially: 2.0 p(0).
1.1 GNN, (et Geoncept),
M
GNN E.1. Gezampte),
2.2. arg max
II 10.0
C. c
j=0
Note that GNN;, and GNN, are different GNNs. We use the output e as the concept embedding for the novel concept c. We denote this model as FALCON-G.
Recurrent neural networks. A variant of our model, FALCON-R, replaces the graph neural network in FALCON-G with recurrent neural networks. Specifically, we process each edge in the graph in a sequential order. We first sample et.0 from p(0) and use it to initialize the hidden state of an RNN cell RNN, J. Next, for each edge (c, rel) in G Gonceett we concatenate the concept embedding of c' and the edge embedding for rel as the input. They are fed into RNN, sequentially. We use the last hidden state of the RNN as e We process all edges in G Gesample in the same manner, but with random orders for
different RNN cell RNN2, Since all edges in graph unordered, all edges in both graphs. The implementation details for both the GNN and the RNN model can be found in our supplemental material.
5

---

Published as a conference paper at ICLR 2022
Training objective. We evaluate the inferred e on downstream tasks. Specifically, given input question and its paired image, we execute the latent program recovered from the question based on the image, and compare with the groundtruth answer to compute the loss. Note that since the output of the neuro-symholic program executor is fully differentiable w.r.t. the visual representation gradient descent to optimize
and concept embeddings, objective is comprised of two terms: one for question answering LQA and the other for the prior Aprior log po (ee) the final loss function.
model. The overall training
distribution matching: Lmete LQA
3.4 TRAINING AND TESTING PARADIGM
The training and testing pipeline for FALCON consists of three stages, namely the pre-training stage, the veta-training stage, and the meta-testing stage. We process a dataset by splitting all concepts into three groups: Cbase. Cvat, and Cteat, corresponding to the base concepts, validation concepts, and test concepts. Recall that each concept in the dataset is associated with 4-tuple (c, Xe, Dr, To). In pre-training stage, model is trained emheddings for e & Chose- This step is the same as the concept learning stage of (Mao et al. 2019). except that we are using geometric embedding space. the meta- meta-training stage, we fix the concept embedding for all с E Chase. and train the GNN modules (in FALCON-G) or the RNN modules (in FALCON-R) with our meta-learning objective Lmeta- this step, we randomly sample task tuples (c, Xe, De, Te) from concepts in Cbase. and use concepts in Ceat for validation and model selection. In the meta-testing stage, we evaluate models by its downstream task performance on concepts drawn from Crest. For more details including hyperparameters, please refer to Appendix Section B
all concepts in Cbase using T.. to obtain all concept
4 EXPERIMENTS
4.1 DATASET
We procedurally generate meta-training and testing examples. This allows us have fine-grained control over the dataset distribution. At a high-level, our idea is to generate synthetic descriptive sentences and questions based on the object-level annotations of images and external knowledge bases such as the bird taxonomy. We compare FALCON with end-to-end and other concept- based method datasets: CLEVR (Johnson al. 2017a) and CUB (Wah al. 2011).
CUB Dataset. The CUB-200-2011 dataset (CUB; Wah et al. 2011 contains 12K images of 200 categories of birds. We split these 200 classes into 100 base, 50 validation and 50 test classes following Hilliard et al. (2018). We expand the concept set by including their hypernyms in the bird taxonomy (Sullivan al. 2009), resulting in 211 base, 83 validation, and 72 test concepts. We consider the two concept re relations in CUB: hypernym and cohypernym (concepts that share the same hypernym). Examples could be found in Fig. 4
CLEVR dataset. We also evaluate our method on the CLEVR dataset (Johnson et al. 2017a). Each image contains 3D shapes placed on tabletop with various properties shapes, colors, types of material and sizes. These attribute concepts are grouped into 10 base, 2 validation and 3 test concepts. We create four splits of the CLEVR dataset with different concept split. We generate the referential expressions following (Liu et al. 2019) and question-anse answer pairs following (Johnson al. 20 a0) Examples could be found in Fig. We consider only one relation between concepts: hypernym. For example, red is kind of color.
GQA dataset. Our last dataset is thee GQA dataset (Hudson & Manning, 2019). which contains image associated with scene graph annotating the visual concepts in the scene. We select subset of 105 most frequent concepts. These concepts are grouped into 89 base concepts, validation and 10 test concepts. Examples generated in this dataset be found in Fig. 4 We consider only one relation between concepts: hypernym, e.g. standing is a kind of pose.
4.2 BASELINE
We consider two kinds of approaches: end-to-end and concept-centric.l In end-to-end approaches, example images and other descriptive sentences (Xe and D.) are treated an additional input the system when responding to each test query in Ta. In this sense, the system is not "learning" the novel concept e internal representation. Instead, it reasons about images and texts that is related to and has been seen before, in order to process the queries. Their implementation details can be found in Appendix Section

---

Published as a conference paper at ICLR 2022
NSCL+LSTM
NSCLAGNN
FALCON-G
4) FALCON-R
D:
LSTMa
M
AVG
GNN
ONN
GNNo cc 0/00
RNN
RNN
90
LSTMa
Gomeeep
Gconcope
Figure 3: Different models for predicting concept embedding ce- They differ in how they process De: supplemental sentences, Geoncepti concept graphs, and o o. objects being referred to.
CNN+LSTM is the first and simplest baseline, where ResNet-34 (He et al. 2016) and LSTM (Hochre) iter & Schmidhuber 1997) are used extract image and text representations, respectively. They are fed into a lulti-layer perceptron to predict the answer to test questions.
MAC (Hudson& & Manning 2018). is a state-of thetr-rr end-to-end method for visual reasoning. It predicts the answer based on dual attention over images and texts.
We also consider a group of concept-centric baselines, clarified in 3 Like FALCON, they all maintain a set of concept embeddings in their memory. The same neuro-symbolic program execution model is used answer questions Tç. This isolates the inference of novel concept embeddings.
NSCL+LSTM, the first baseline in this group uses the same semantic parsing and program execution
module to derive the example object embeddings o. as FALCON. Then, uses an LSTM encode all o; into single vector, and uses a different LSTM to encode the natural language sentences in De into another vector. These two vectors are used to compute the concept embedding ec-
NSCL+GNN is similar to NSCL+LSTM except that supplemental sentences De are first parsed as conceptual graphs. Next, we initialize the concept embedding in the graph by taking the "average" of visual examples. We then use the same GNN module FALCON on the conceptual graph G Geoneept to derive eg- Note that NSCL+GNN only use a single GNN corresponding to Geoncept> while FALCON-G uses two GNNs-one for concept graphs and the other for example graphs.
We test concept-centric models on two more embedding spaces besides the box embedding.
Hyperplane: object and concept correspond to vectors in Rd. denoted as e,, and ec, respectively. We define Pr[eo]ec] = (eefe ee/T /T - 7), where a is the sigmoid function, T 0.125, and ; = 2d.
object and concept correspond to vectors in IRd denoted as e, and ecr respectively. We define Pr[s,/ee] = o(((e.. ee) - 7)/r), where (..) is the cosine similarity function, and 0.1, 0.2 are scalar hyperparameters. This embedding space has been proved useful in concept learning literature (Gidaris & Komodakis 2018 Mao et al. 2019).
Hypercone:
4.3 RESULTS ON CUB
We evaluate model performance on downstream cvestion-answeringg pairs in To for all test concepts E Crest- There is only one example image for each concept ((X.l = 1). Visual reasoning questions on CUB are simple queries about the bird category in this image. Thus, it has minimal visual reasoning complexity and focuses more on the fast learning of novel concepts.
The results are summarized in Table with qualitative examples in Fig. In general. our model out- performs all baselines, suggesting the effectiveness of our concept embedding prediction. FALCON-R with box embeddings achieves the highest performance. We conjecture that this is because box embeddings can better capture hierarchies and partial order relationships between concepts, such as hypernyms. NSCL+LSTM and GNN models have inferior performance with box embeddings, proba- bly because their neural modules cannot handle complex geometric structures like box containment. Unlike them, we use sample-update eedeeee-aaateeeeluuteeee evaluate procedure to predict box embeddings.
By comparing concept-centrie models based on the same embedding space (e.g.. box embeddings), we see that translating the descriptive sentences D, into concept graphs significantly improves the performance. Thus, NSCL+LSTM has an inferior performance than other methods. NSCL-GNN and FALCON-G are very similar to each other, except that in NSCL-GNN, the initial embedding of the concept e, is initialized based on the object examples, while in FALCON-G, we use multiple random samples to initialize the embedding and select the best embedding based on the likelihood function. Moreover, FALCON-G and FALCON-R R has similar performance, and it is worth noting that FALCON-R has the advantage that it can perform online learning. That is, edges in Gooncept and G Grrampte of concept e can be gradually introduced to the system to improve the accuracy of the corresponding concept embedding cc.
7

---

Published as a conference paper at ICLR 2022
Model(Type)
QA Accuracy
Model(Type)
QA Accuracy
CNN+LSTM (E2E)
51.37
CNN+LSTM (E2E)
51.50
MAC (E2E)
73.88
MAC (E2E)
73.55
Box
Hyperplane Hypercone
Box
Hyperplane Hypercone
NSCL+LSTM (C.CC)
79.53
71.71
72.48
NSCL+LSTM (C.CC)
72.23
63.18
68.48
NSCL+GNN (C.C.)
78,50
67.60
72.82
NSCL+GNN (C.C.)
73.38
62,00
72.91
FALCON-G (C.C.)
81.33
74.11
68 68.44
FALCON-G (C.C.)
76.37
64.60
70.42
FALCON-R (C.CC)
79.83
72.82
69.74
FALCON-R (C.CC)
75.84
63.32
69.69
Table I: Fast concept learning performance the CUB dataset. We use "E2E" for end-to-end methods and "C.C." for concept-centric meth- ods. Our model FALCON-G and FALCON-R outperforms all baselines.
Table 2: Fast concept learning with only visual
examples, evaluated on the CUB dataset. The task is also referred to one-shot learning. In this setting, NSCL+GNN will fallback to a Pro- totypical Network (Snell et al. 2017).
Thete i atommolo the image.
yellow stject tie oyeu degact.
Thev snall utjedd un beriiatedd
vetal ddjedt
quaaakka adutta modkllasa sternile are affieees Liads Uneriaare.
grves, pullooe
Glav mxtá dostiba
saker
cbjes
slope
lo mana bette a viets stjeet?
post
okjast?
You (8,8
Cylkde (D.6))
You 75
Figure 4: Qualitative results of FALCON-G on the CUB and CLEVR dataset. The bottom left corner of each images shows the model prediction and the confidence score. * marks a wrong prediction.
We summarize our key findings as follows. First, compared with end-to-end neural baselines (CNN+LSTM, MAC), explicit concept representation (NSCL, FALCON) is helpful (Table 3 Second, comparing modules for predicting concept embeddings from supplemental sentences De LSTMs over texts D. (NSCL+LSTM) v.s. GNNs over concept graphs Geaneept (NSCL+GNN, FALCON-G), we see that, explicitly representing supplemental sentences graphs and using GNNs to encode the information yields the larger improvement. Third, comparing modules for predicting concept
embeddings from visual examples (o), heuristically taking the average (NSCL+GNN) v.s. GNNs on example graphs (FALCON-G), we see that using example graphs is better. These findings will be further supported by results on the other two datasets.
Case study: few-shot learning. Few-shot learning (i.e., learning new concepts from just a few examples) is special case of our fast concept learning task, where D. Ø, i.e., there is no supplemental sentences to relate the novel concept e with other concepts.
Results for this few-shot learning setting are summarized in Table 2 Our model FALCON outperforms other baselines with a significant margin, showing the effectiveness of our sample-update-selles procedure for inferring concept embeddings. In this setting, NSCL+GNN computes the novel concept embedding solely based on the "average" of visual examples, and thus being a Prototypical Network (Snell et al. 2017). Note that it cannot work with the box embedding space since the offset of the box cannot be inferred from just one input example.
Extension: continual learning. Another advantage of concept-centric models is that it supports the cumulative growth of the learned concept set. That is, newly learned concepts during the "meta- testing" stage serve supporting concepts in future concept learning. This not feasible for end-to-end approaches as they require all concepts referred to in the supplemental sentence in the base concept set Chose- We extend concept-centric approaches to this continual learning setting, where test concepts Ctest are Iearned incrementally. and new concepts might relate to another concept in Ctest that has been learned before. In this setup, FALCON-G achieves the highest accuracy (79.67%) outperforming the strongest baseline NSCL+LSTM by 4.46%. Details are in Appendix Section C.I Ablation: the number base concepts. We examine how the number of base concepts contributes to the model performance. We design new split of the CUB dataset containing 50 training species (130 base concepts), compared to 100 training species (211 base concepts) of the original split. Our FALCON-G model's accuracy drops from 81.33% 3% to 76.32% when given fewer base concepts. This demonstrates that transfering knowledge from concepts already learned is helpful. See Appendix[D.. Ablation: the number of related concepts in supplemental sentences. We provide an ablation
8

---

Published as a conference paper at ICLR 2022
Table 3:
Fast concept learning perfor- mance on the CLEVR dataset. We compare both Example only and the Example +Supp. settings. The results are evaluated and averaged on the four splits of the dataset. See the main text for analysis.
Model
Example Only
Example+Supp.
A
CNN+LSTM
41.92
42.69
0.77
MAC
61.81
62.05
0.24
NSCLALSTM
56.82
61.35
4.53
NSCLAGNN
65.01
84.11
19.11
FALCON-G
68.47
88.40
19.93
FALCON-R 68.57
87.36
18.78
on the effects of the number of related concepts in supplemental sentences. We use supplemental sentences containing (no supp. sentence), 25%, 50%, 75%, and (th original setting) of all related concepts. Our model FALCON-G yields an accuracy of 76.37%, 80.20%, 80.78%, 81.20% 81 81.33%, respectively. This result shows that more related concepts included in supplemental sentences lead to higher accuracy. Details are in Appendix D.2
4.4 RESULTS ON CLEVR
We use a similar data generation protocol for the CLEVR datasct as CUB. While CLEVR contains concepts with simpler appearance variants, it has more complicated linguistic interface. More importantly, all concepts in CLEVR are disentangled and thus allow us to produce compositional object appearances. We will exploit this feature conduct systematical study concept learning from biased data. Our framework FALCON provides a principled solution to learning from biased data, which has been previously studied in (Geirhos et 2019) and (Han et al. 2019).
Our main results are summarized in Table 3 Fig. provides qualitative examples. All concept- centric models use the box embedding space. We also compare model performance on the "few-shot" learning setting, where no supplemental sentences are provided. Due to the compositional nature of CLEVR objects, learning novel concepts from just a single example is hard, and almost always ambiguous: the model can choose to classify the examples based on either its shape, or color, or material. As result, all models have a relatively poor performance when there are no supplementary sentences specifying the kind of novel concept (color/shape/etc...
Case study: learning from biased data. We provide a systematical analysis on concept learning from biased data. In this setup, each test concept e is associated with another test concept ef (both unseen during training). We impose the constraint that example objects of concept e in X, also has concept c'. Thus, and c' is theoretically ambiguous given only image examples. The ambiguity can be resolved by relating the concept e with other known concepts. Overall, our model, FALCON-G achieves the highest accuracy (88.29%) in this setting (MAC: 61.93%, NSCL+GNN 84.83%). Moreover, our model shows the most significant performance improvement after reading supplemental sentences. Details can be found in Appendix Section C.2
4.5 RESULTS ON GQA
Following similar data generation protocol as the previous datasets, we generate questions on the GQA dataset that ask whether a linguistically grounded object in the picture is associated with a novel concept. Since each GQA picture contains a large number of concepts in different image regions, it provides a more complex linguistic interface than the CUB dataset and more diverse visual appearances than the CLEVR dataset. We exploit this advantage to test our model's performance on real-world images and questions in the GQA dataset.
Our main results are shown Table Iaain the Appendix Section C.3 Fig. provides qualitative ex- amples. Overall, our model achieves the highest accuracy (55.96%) in the setting with supplementary sentences (MAC: 54.99% NSCL+IISTM: 55.41%). 1%). Due to the visual complexity in the GQA dataset, the exact decision boundary of a concept is hard to determine. Details are in Appendix Section C.3
5 CONCLUSION
We have presented FALCON, a meta-learning framework for fast learning of visual concepts. Our model learns from multiple naturally occurring data streams: simultaneously looking at images, reading sentences that describe the image, and interpreting supplemental sentences about relationships between concepts. Our high-level idea is to incorporate neuro-symbolic learning frameworks to interpret natural language and associate visual representations with latent concept embeddings. We improve the learning efficiency by meta-learming concept embedding inference modules that integrates visual and textual information. The acquired concepts directly transfer to other downstream tasks, evaluated by the visual question answering accuracy in this paper.

---

Published as a conference paper at ICLR 2022
Acknowledgements. This work is in part supported by ONR MURI N00014-16-1-2007, the Center for Brain, Minds, and Machines (CBMM, funded by NSF STC award CCF-1231216, the MIT Quest for Intelligence, MIT-IBM AI Lab. Any opinions, findings, and conclusions recommendations expressed this material are those of the authors and do not necessarily reflect the views of our sponsors.
REFERENCES
Paul Bloom. How children learn the meanings of words. The MIT Press, 2000.
Susan Carey and E. Bartlett Acquiring single word. Proceedings the Stanford Child Language Conference, 15:17-29, 01 1978.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Octavian-Fugen Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic Entailment Cones for Learning Hierarchical Embeddings. In JCML, 2018
Robert Geirhos, Patricia Rubisch. Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. -trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019.
Spyros Gidaris and Nikos Komodakis. Dynamic Few-Shot Visual Leamning Without Forgetting. In CVPR, 2018.
William L. Hamilton, Rex Ying, and Jure Leskovec. Representation Learning on Graphs: Methods and Applications. 1rXiv:/709.055 2017.
Chi Han, Jiayuan Mao, Chuang Gan, Joshua B. Tenenbaum, and Jiajun Wu. Visual Concept Metaconcept Learning. In NeurIPS, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
Kaiming Hc, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017.
Nathan Hilliard, Lawrence Phillips, Scott Howland, Artëm Yankov, Courtney D. Corley, and Nathan Hodas. Few-Shot Learning with Metric-Agnostic Conditional Embeddings. arXiv: 000.003776 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Comput., 173:1733 1997.
Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable Neural Computation via Stack Neural Module Networks. In ECCV, 2018.
Drew A Hudson and Christopher D Manning. Compositional Attention Networks for Machine Reasoning, In ICLR, 2018.
Drew A Hudson and Christopher D Manning. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. In CVPR, 2019.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, In CVPR, 2017a.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In ICCV, 2017b.
Michael Kampffmeeyo, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric Xing. Rethinking knowledge graph propagation for zero-shot learning. In CVPR, 2019.
Diederik Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization. In ICLR, 2014.
Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. In NeurlPS, 2014.
Qing Li, Siyuan Huang, Yining Hong, and Song-Chun Chun Zhu. A Competenco-aware Curriculum for Visual Concepts Learning via Question Answering. ECCV, 2020.
Li, Xiang and Vilnis, Luke and Zhang, Dongxu and Boratko, Michael and McCallum, Andrew, Smoothing the geometry of probabilistic box emheddings. In ICLR, 2019.
10

---

Published as a conference paper at ICLR 2022
Runtao Liu, Chenxi Liu, Yutong Bai, and Alan L. Yuille. CLEVR-Ref+: Diagnosing Visual Reasoning With Referring Expressions. CVPR, 2019.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua Tenenbaum, and Jiajun Wu. The Neuro- Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. In ICLR, 2019.
David Mascharka, Philip Tran, Ryan Soklaski, and Arjun Majumdar. Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning. CVPR, 2018.
Tsendsuren Munkhdalai and Hong Yu. Meta Networks. In ICML, 2017.
Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Learning Algorithms, arXiv: 1803.02999. 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, 2019.
Deep Learning Library. In NeurIPS,
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017.
Hongyu Ren, Weihua Hu, and Jure Leskovec. Query2box: Reasoning over Knowledge Graphs Vector Space using Box Embeddings. In ICLR, 2020.
Shaoqing Ren, Kaiming Ross B. Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In NewrlPS, 2015.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-Learning with Memory-Augmented Neural Networks. In ICML, 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical Networks Few-shot Learning. In NeurlPS, 2017.
Brian I.. Sullivan, Christopher Wood, Marshall J. Iliff, Rick E. Bonney, Daniel Fink, and Steve Kelling, eBird: citizen-based bird observation network in the biological sciences. Biological Conservation, 142(10):2282 2292, 2009.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning to Compare: Relation Network for Few-Shot Learning. In CVPR, 2018.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to Sequence Learning with Neural Networks. In NewrlPS, 2014.
Swingley, Daniel. Fast Mapping and Slow Mapping in Children's Word Learning. Language Learning and Development, 6:179-183, 07 2010.
Yonglong Tian, Yue Wang. Dilip Krishnan, Tenenbaum Joshua B., and Phillip Isola. Rethinking Few-Shot Image Classification: Good Embedding Is All You Need? In ECCV, 2020.
Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language. In ICLR, 2016.
Luke Vilnis and Andrew McCallum. Word Representations via Gaussian Embedding. In ICLR, 2015.
Luke Vilnis, Xiang Li, Shikhar Murty, and Andrew McCallum. Probabilistic Embedding of Knowledge Graphs with Box Lattice Mcasures. In ACL, 2018.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching Networks for One Shot Learning. In NeurlPS, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The Caltech-UCr 200-2-200-2011 Dataset. Technical report, Califoria Institute of Technology, 2011.
Xiaolong Wang, Yufci Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In CVPR, 2018.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Toralba, Pushmeet Kohli, and Josh Tenenbaum. Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. In NeurIPS, 2018.
11

---

Published as a conference paper at ICLR 2022
The appendix is organized the following. First, in Section present model details, especially the computation performed by our GNN- and RNN-based model for concept graphs and example graphs. Next, in Section we formally defines the training objective for different stages in FALCON. Finally, in Section C we supplement details and examples of our datasets and experimental setups.
A MODEL DETAILS
A.1
REMARKS ON THE OVERALL MODEL STRUCTURE
FALCON follows a modularized, concept-centric design pattern: the system maintains a set of concept embeddings, which naturally grows as learns new concepts. We use separate modules for computing novel concept embeddings (Section 3.3 and processing queries (Section These modules directly compute the novel concept embeddings, essentially classifiers on objects and their relationships, and use them answer complex linguistic queries.
Here, all images and texts processed by the system will be "compressed" as these concept embeddings in latent space. There other possible designs, such as storing all training images and texts directly, and re-processing all training data for every single query in Te- Despite heing inefficient, as will show in experiments, such design also shows inferior performance compared with FALCON.
A.2 CONCEPT EMBEDDING
Throughout the paper, We use hidden dimension of d = 100 for the box embedding space (note that the vector representation for this box embedding space is 200-d since we have both centers and offsets), and d = 512 for the hyperplane and the hypercone embedding space. For the box embedding space, we set the smoothing hyperparameter Zhang, Dongxu and Boratko, Michael and McCallum, Andrew (2019).
0.2, following Li, Xiang and Vilnís, Luke and
A.3 PRIOR DISTRIBUTION
In both FALCON-G and FALCON-R, assume prior pe on concept embeddings, from which we sample the initial guess e For box embedding space, we model the prior distribution of e. so that its center and offset along all d dimensions can be generated by the same Dirichlet distribution parameterized by 0. In particular, we sample d independent samples (Zop, Tip; pE (0.1,..d-1)
from Dirichlet distribution parameterized by 0 (0u. 02. 02). Then an initial guess for concept 1 (21p Top). and Off(c),
embedding of is given as Cen(c)p hypercone embedding space, we model the prior as the same Gaussian distribution on all d dimensions, namely c, (0, foLa), where I is the identity matrix and 0n a scalar parameter. In both embedding space choices, 0 can be learnt from back-propagation from Lmota LQA Aprior log pe (ec), and we set the hyperparameter Aprior =
22 For hyperplane and
A.4 FALCON-G
In FALCON-G, there two graphical neural networks, GNN, and GNN2, which operates on the concept graph Geoncept and example graph Gezample, respectively. Both of them are composed by stacking two graphical neural network (GNN) layers.
For convenience, we denote the input embedding of node at layer é é [0, 1) h At layer é, we first generate the message from node j to its neighbour by:
mjs MP (h))(h (6) (((())
where & denote the relation between j and i, where MP is a message passing function, which we will formally define later.
Next, for each node i, we aggregate the message passed to i by computing an element-wise maximum (i.c., pooling) of all neighbouring j's (2)
(a) where Na(i) is the set of all neighboring nodes.
12

---

Published as a conference paper at ICLR 2022
Finally, update node embeddings for based on its original embedding 1, and aggregated messages h (6) for node i. Mathematically, Nat(t)
N ((+1) UPDATE(N (0) (0) , (No())
Concretely, we implement the message passing model MPs based on specific concept embedding spaces. For box embedding spaces, we first split h as two vectors, denoted as Cen(h) ( and Off(h))) respectively. Similarly, we split n into Cen(h (5) and Off(nn)). For each hidden dimen- sion {0, 2, dd 1}, we concatenate Cenp (h (0) Off, (hh (.) Ccn,(h,)) ( and Off (h; as a 4-d vector, and applies a two-layer perceptron (MLP) on it. This MLP is shared across all hidden dimensions p. For different binary relations k, we use different MLPs. In our experiment, the aforementioned MLP has an output dimension
and a hidden dimension 20.
For the hyperplane and the hypercone embeddings, we simply concatenate h and h and send it into a MI.P associated with relation kt 'he MI.P has 1024 hidden units and 512 output units.
The update model UPDATE is implemented in similar manner. For the box embedding space,
concatenate the hidden vector associated with each hidden dimension of h (e) and (e)
(G)(() as a 12-d
vector. We send it to a feed-forward neural network with 20 hidden units and output units, which is shared across dimensions. If we denote the four output units of ((f) ((E) gr(t) to calculate the new box embedding h." ((+1)
0ig 0p we use them
ip Cen(hip ((+1))
Cen(h (9) + (0 1
Of (1
Off(h! (9) + σ ( (())
)0
where a is the sigmoid function. For hyperplane and hypercone embeddings spaces, we implement UPDATE by concatenate h( and 1: and send feed-forward neural network with 1024
hidden units and 1024 output units. If we denote its output as o ,,(() each having a length of 512, then the update model can be written
h (++1)
=
N, + +o()) 06,
where 3 denotes element- wise multiplication.
A.5 FALCON-R
In FALCON-R, concepts graph and example graphs are handled jointly in a unified framework. For each each object example o( associated with c, we treat their relationship relation. Thus, we can combine two graphs into single graph. We denote the resulting graph as G*.
"concept-example"
We flatten the graph G by randomly order all relational tuples (c,c', rel) G* We then feed them sequentially into an RNN cell. We implement the RNN cells RNN, and RNN, GNN used in FALCON-G, but is applied 2-node graph.
For j. 1, k in G.edges:
split torch. stack (I
*h[i] , chunk (2, -1),
+h[i] chunk (2, -1)
}, dim=-1)
m[ (i, 1) mlp_1 [k] (split)
for in nodes:
h_Nii]
forch.stack.
ml (1), iill for (d1, 11) in m 1f i == 11
1) max (dim-0) va lues
for
in nodes:
13

---

Published as a conference paper at ICLR 2022
Operation
Semantics
Scene
Return all objects in the scene.
Filter the set of objects from input objects that have the specific
Filter
object-level concept. Filter the set of objects that have the specific relational concept
Relate
with as input object.
AERelate
Filter out the set of objects that have the same attribute value as the input object.
Intersection
Return the intersection of two object sets.
Union
Return the union of two object sets.
Query
Query the attribute of the input object.
AEQuery
Query if two objects have the same attribute value (e.g., color).
Exist
Query an object set is empty.
Count
Return the size of object set.
Query the size of the first object set is smaller than the size of the second object set. Query the size of the first object set is greater than the size of the second object set.
CountLessThan
CountGreaterThan
Query if the size of the first set is the same as the size of the second object set.
CountEqual
Table 4: All program operations supported in CLEVR.
12
o, delta, O_prime,
2torch stack ([
*h[i].oo chunkk (2, dim=-1), *h_N[i] unbin
15
16
hlil = torch.cat(1
delta, offset torch. (c_prime) delta_prime
center worch.sigmeidiie...
19
1, dim=-1)
Snippet I: A snippet of one-step concept graph propagation based on the box embedding space. The code is written in PyTorch Paszke et al. (2019). The code snippet updates the node embedding [i] by first computing the messages passing from node j to node i in the first for-loop. aggregating the messages to node by taking the element- wise max, and finally updating the node embedding i.
A.6 SEMANTIC PARSER
In semantic parsing module, the sentence is translated into programs of primitive operations that have hierarchical layouts. For CLEVR dataset, the complete list of operation we provided is given in Table For CUB dataset, since there only one bird in the image, we only provide operations Scene, Filter and Exist listed in Examples of the output of semantic parsing module are provided in Table 5
Our semantic parser follows the general framework of Seq2seq (Sutskever al. 2014) and our implementation follows NS-VQA (Yi et al. 2018) which has been developed for V YQA tasks. We first tokenize the input sentence into sequence of word ids. Then, feed the sequence into Seq2Seq model and generate sequence of (op;, concept,), where = 1, 2, ..., Lowr where L the output sequence length. op:'s 's operations defined the DSL, and they form "flattened"
program based on which we will recover hierarchical program. Some operations such as filter has concept input, in which case we use the corresponding concept,. The semantic parser is trained on 10% of all generated data.
A.7
PROGRM EXECUTION
We mostly follow the original design of NSCL as in Mao et al. Mao et al. (2019). The neuro- symbolyymbolic executor executes the program based both the scene representation and the concept embeddings.
14

---

Published as a conference paper at ICLR 2022
Dataset
Input
Output
Filter (Intersection
Relate (Filter (Scene cyan), front), (Filterr (Scene (), cube), behind) red)
The red objects that are both front of the cyan object and behind the cube is sphere.
CLEVR
(Description Sentences)
Query (Filter( (Filte (Scene
There rubber object to the left of the metal object; What is its color?
CLEVR
(Questions)
metai), left), rubber), color)
CUB
There is frigatebird.
Scene
(Description Sentences)
CUB
Exist (Filter (Scene frigatebira))
Is there frigatebird?
(Questions)
GQA
The clouds are a white object.
Filter(Scene () clouds)
(Description Sentences)
Exist (Filter (Filter(
GQA
Is the standing man a red object?
Filter (Scene (), standing), man), red))
(Questions)
Table 5: Example questions and the programs recovered by the semantic parser. For description sentences, the semantic parser parse the referential expression in the sentence into a program. On CUB, since there is only one object in the image, the operation Scene returns the only object in the image.
Lying in the core of execution is the representation of object sets, which are immediate results during the execution. Each set of objects (such as the output of a Filter operation, a vector of length N. where N is the number of objects in the scene. Each entry is a score ranging from 0 to 1. We use the same implementation as Mao et al. (0019), with two modifications:
First, the probability of classifying object as concept Pr ea] (objClassi fy in the NSCL paper), implemented based on different embedding spaces: box, hypercone and hyperplane. The original NSCL paper uses the hypercone representation.
Second, the probability that two objects having the same attribute (named paper) are treated special kinds of relational concepts: has_same_co cor, saape,,e_shape, ctc. Thus, the AERelate operation is implemented in the same way as Relate.
fy in the NSCL.
A.8 BASELINE IMPLEMENTATION
CNN+LSTM In CNN+LSTM, we use ResNet-34 network with global average pooling layer to extract the image representation. Each image feature is feature vector of dimension 512. We concatenate the description sentences, supplemental sentences (if any). and test questions with separator to form the text input. We use a bi-directional L.TTM to encode the text into a 512- dimensional vector (we use the output state of the token). We use a two-layer LSTM, each with hidden state of dimension 512. The image representation of both images (the example image in X., and the test image in T.), together with the text encoding, are sent into a multi-layer perceptron (MLP) to predict the final answer. The MLP has two layers, with hidden dimension of 512.
MAC
The MAC network takes an image representation (encoded by CNNs) and text represen- tation (encoded by RNNs) as its input, and directly outputs distribution over possible answers. Following the original paper Hudson & Manning (2018), we use the first four residual blocks of ResNet-101 101 He et al. (2016) to extract image representations, which is 2D feature map with hidden dimension 512. The image representations the example image and the test image an stacked channel-wise into 2D feature map of hidden dimension 1024. The Similar to CNN+LSTM,
15

---

Published as a conference paper at ICLR 2022
concatenate the description sentences, supplemental sentences (if any), and test questions with sepa- rator to form the text imput. For all other parts, we use the same setting as the original paper Hudson & Manniig (2018), such as the hidden dimension for all units in MAC.
NSCL+LSTM In NSCL+LSTM, we use the same semantic parsing module and program execution
module to derive the example object embeddings of") These object embeddings are encoded by bi-directional LSTM into a 51 512-dimensional vector. We use separate bi-directional LSTM to encode the Gooncept into a 512-dimensional vector. Embeddings of related concept c', concatenated with an embedding for the relation rel, are fed in to the LSTM one at a time. Two LSTM both use as output the output state associated with the last input, and have two layers and 512 hidden dimension. These output of two LSTMs are concatenated and sent to linear transform to predict the concept embedding eg. The output of the linear transform e, will be used by the neuro-symbolic program execution module to predict the final answer queries. NSCL+LISTM is ablative model of FALCON-R. It keeps similar designs as FALCON-R except it places visual examples and linguistic guidance on different streams, whereas FALCON-R uses a single recurrent mechanism capture both the visual and linguistic information.
GNC In NNN the supplemental sentence D. is first parsed into conceptual graph Gonceept- All nodes corresponding related concepts c' are initialized by the concept embeddings of c'. The node corresponding to concept e is initialized from the "average" of all visual examples. For both hyperplane and hypercone embedding spaces, the average computed by taking the mean of (L2-normalized) example object embeddings. For box embedding spaces, it is computed as the specific boundary (i.e., the minimal bounding box that covers all examples) of visual example embeddings. For meta-learning setting without supplmentary sentences, we use the average box size of all concepts in the training set to initialize the box size for novel concepts, so the NSCL+GNNN model would become special example of the Prototypical Network. The GNN has two layers, and has the same design as our mode FALCON-G. NSCL+GNN is an ablative model of FALCON-G. keeps all designs of FALCON-G except that it directly computes the "average" of visual examples. In contrast, in FALCON-G, the example object information are gathered with a example graph neural network encoder.
B TRAINING DETAILS
Our training paradigm consists of three stages: pre-training, meta-training and meta-testing, Fig. gives illustrative overview of these three stages. Recall that we have processed the dataset by splitting all concepts into three groups: Chuses Cyai, and Ctest, and each concept in the dataset is associated with 4ttple (c, Xe, De, Tc).
In pre- vreitrannng stage, our model is trained on all concepts c in base concept set Chase- The concept embedding of and the visual representations are trained using irrestion-answer pairs from Te. This step is the same as the concept learning stage Mao et al. (2019), except that we are using geometric embedding space. In particular, based on the classification confidence scores of object properties and their relations, the model outputs a distribution over possible answers for each recovered program. On the CLEVR dataset, we use the same curriculum learning setting as Mao et al. (0199. based on the number of objects in the image and the complexity of the questions. In the meta-training stage, we fix the concept embedding for all concepts E Coase: and train the GNN modules (in FALCON-G) or the RNN modules (in FALCON-R) with our meta-learning objective Lmeta- In this step, we randomly sample task tuples (c, Xc, D., Te) of concepts Chases and use concepts in Ceat for validation and model selection. In the meta-testing state, we evaluate each model by its downstream task performance on concepts drawn from Ctest-
One issue with end-to-end methods (CNN+LSTM and MAC) is that they assume that the concept vocabulary is known. Upon seeing a new concept, they do not have the mechanism for inducing a new word embedding for this concept c. To address this issue, in the meta-training and meta-testing stages, the appearance of the novel concept c in all texts will be replaced by a special token <UNK>.
Our model is trained with the following hyperparameters. During the pre-training stage, the model trained for 50000 iterations with a batch size of 10, using an Adam optimizer Kingma & Ba (2014) with learning rate 0.0001. During the meta-training stage, the model is trained for 10000
16

---

Published as a conference paper at ICLR 2022
Stage 1. Pre-tmining on hase concepts.
Clase igray, metal, large...
many objects are gray objects?
A: 2.
Q: gray large object?
A: No.
Stage Meta-training on base concepts.
Chaor Egray, metal, large -)
Is there large object behind the large metal object?? Yes.
Descriptive sentence:
There is object in front of the sphere; large object. Supplemental sentence: Small and large are sizes.
Q: Is there large object cube? No.
Stage 3. Meta-testing on novel concepts.
Cavei (eyan, purple, +1
Descriptive sentence:
There rubber object left the metal object: Is eyon object? Yes.
There small object to the right yellow object; eyan object.
Supplemental sentence: Gray, red, blue, green, brown, yellow and cyue are colors.
Q: Is there cube to the right of the eyan object?
A: No.
Fast Concept Learning From Vision and Language
Downstream Task: Visual Reasoning
Figure 5: An overview of the overall training paradigm.
iterations with batch size of I (one novel concept per iteration). We train the model with the same optimizer setup (Adam with a learning rate 0.001). In both stages, we evaluate the model on their corresponding validation every 1000 iterations. We decrease the learning rate by factor of 0.1 when the validation accuracy fails to improve for 5 consecutive validation evaluations, i.e. the iteration with the best validation accuracy is more than validations ago.
C EXPERIMENT DETAILS
C.I THE CUB DATASET
In the CUB dataset, descriptive sentences are generated from the referential expressions produced by the templates in Th, containing only concepts from Chase- We provided an approximate of 1.8k descriptive sentences, 1055 of which for Chases 415 for Coat and 360 for Creat- Supplemental sentences are generated based on the templates Table In supplemental sentences for concept Cruse. we randomly remove related concept c' from the sentence with probability of 0.2. We generate 54k testing questions based on the templates in Table [6. 31.5k of them paired with concepts in Ctraen. 12.5k of them paired with concepts in Coal and 10k of them paired with concepts in CLest- Descriptive sentences, supplemental sentences and testing questions are zipped together to form the input for fast concept learning task. The same template generates Pre-training questions testing questions. We generated around 1.8k question-image pairs for pre-training. During training, apply standard data augmentation techniques the images, including horizontal flip, random resizing and cropping.
Extension: continual learning. In the continual learning setting, concepts that has been learned in the meta-testing instances can be used as supporting concepts for other concepts. We implement this idea the following. Since CUB is constructed based a bird taxonomy, can define the distance between two concepts by the length of their shortest path on the bird taxonomy. We define the hop number of each concept as their distance with the closest concepts é' Chasc- In the meta-testing
17

---

Published as a conference paper at ICLR 2022
(a) The descriptive sentence template CUB.
Template
There a/an <CONCEPTS.
Example
There white-eyed vireo.
Description
<CONCEPT> is the word representing concept c.
(b) The supplemental sentence template in CUB.
Template
<CONCEPT-1>, <CONCEPT- 2>>... <CONCEPT -N are kind of
<HYPERNYM>,
Example
Black-capped vireo, philadelphia vireo, warbling vireo, white-eyed vireo are a kind vireos.
Description
<CONCEPT-i> words representing and c' that share a cohypernym relation with c. <HYPERNYM> denotes c's hypernym.
(c) The test question template in CUB.
Template Example
Is there a/an <CONCEPTI T>.
there a white-eyed virco.
Description
<CONCEPT> is the word representing concept c.
Table 6: Templates for generating descriptive sentences, supplemental sentences and test questions in
CUB.
Model
Performance
Model
Performance
Box
Hyperplane
Hypercone
Box
Hyperplane
Hypercone
NSCL+LSTM
75.21
67.78
71.06
NSCL+LSTM
71.31
62.45
67.
NSCL+GNN
71.04
64.74
72.37
NSCL+GNN
72.53
61.56
71.25
FALCON-G
79.67
73.14
64.37
FALCON-G
75.28
64.16
68.73
FALCON-R
78.76
65.38
68.11
FALCON-R
75.01
62.78
67.96
(a) The continual concept learning performance on the CUB dataset.
(b) The continual concept learning performance with only visual examples on the CUB dataset. This task is also referred as few-shot learning, where NSCL+GNN will fall back to Prototypical Network Snell et al. 2017)
Table 7: The continual concept learning performance on the CUB dataset. Only concept-centric methods can be applied to this setting. FALCON-G and FALCON-R outperform all baselines. The table b serves as comparison with table a. FALCON-G and FALCON-R make the best use supplemental sentences to help concept learning.
stage, we test models with concepts of increasing hop numbers. For example, we first teach the model with 1-hop concepts, which directly relates concepts they have seen in Chase- Next, we add 2-hop concepts. During the learning of 2-hop concepts, the concept embedding of those 1-hop concepts are fixed. The detailed result for concept-centric methods for continual concept learning can be found in Table 7a The accuracy is averaged over all 72 test concepts in Crest- For reference, we also show the results of applying these models the fast concept learning tasks without supplemental sentences in Table 7 (the few-shot learning setting). This results are computed over all 72 concepts in Crest- although we do not use any supplemental sentences. Our model makes the best use of supplemental sentences in inducing novel concept embeddings.
C.2 THE CLEVR DATASET.
CLEVR dataset contains 15 individual concepts, them associates with colors, associates with shapes, 2 associated with the materials of objects and 2 associated with the sizes of objects. Each split is generated by selecting 2 colors as validation concepts Coats and another 2 colors and shape testing concepts Crest-
18

---

Published as a conference paper at ICLR 2022
(a) The descriptive sentence template CLEVR.
Template
There a <REFEXP>; is <CONCEPT>.
Example
There small object the right of the yellow object; it is eyan object.
Description
EREFEXP> is a referential expression generated by CLEVR-Ref+
Liu al. 2019). <CONCEPT> is the word for concept
(b) The supplemental sentence template CLEVR.
Template
<CONCEPT CCCCCCEE 2>... <CONCEPT N are <ATTRIBUTE>, Cube, sphere and cylinder are shapes.
Example
Description
<CONCEPT-i> are words representing с and its related concepts d' <ATTRIBUTE> is the word for attribute of object, one of shape, color, material and size.
(c) The test question generation template in CLEVR.
Template Example
<QUEST>
Does the small blue thing have the same material as the gray object?
Description
<UEEST> question generated from the original CLEVR paper
Johnson et al. (2017a).
Table 8: Templates for generating descriptive sentences, supplemental sentences and test questions in CLEVR.
Model
Example Only
Example+Supp.
CNN+LSTM
41.61
42.38
0.77
MAC
62.38
62.40
0.02
NSCL+LSTM
57.76
61.79
4.03
NSCL+GNN
64.75
84.83
20.08
FALCON-G
69.71
87.29
17.57
FALCON-R
69.70
86.21
16.50
Table 9: Ablated study of FALCON tasks w/. and w/o. supplemental sentences (+Supp.) on the biased CLEVR dataset. All concept-centric methods use the box embedding space.
On CLEVR, descriptive sentences are generated based on the CLEVR- Liu et al. 2019) dataset. Shown Table Rh, we first generate a referential expression for the target object, and associate with the novel concept The referential expression contains only concepts from Cinse. In total have generated 7.5k pairs of descriptive sentences and images, 5k of which are for Chases Ik for Coat, and another 1.5k for Cleat- supplemental sentences sentences are generated based on template in 8p by combining the concept c and its related concepts c'. We generate 225k pairs testing questions and images based on the templates in Table 150k of them are associated with concepts in Cirains 30k for Crat, and another 45k for in Crest- Testing question contains only one occurrence of concept and possibly many occurrences of base concepts. An example of the generated texts be found in Fig. 5 We generated 402k questions for pre-training.
Learning from biased data. We supplement a concrete example to illustrate how our model can learn novel visual concepts from biased data. Shown in Fig. 6 in the first task, the learner is trained on a novel concept cylinder. However, all training examples of cylinders are also purple. Thus, without any supplemental information about whether the novel concept task itself is ambiguous. Our model is capable of resolve such kind of ambiguity by interpreting supplemental sentences. We have also included the full experimental result of this setup Table 9
color or a shape, the learning
Handling non-prototypical concepts. On CLEVR, we have studied a specific type of concept- concept relationship: hypernym. It is worth noting that there is difference between the hypernym relations CUB dataset and hypernym relations CLEVR dataset. Specifically, the hypemny of a bird category is also bird category. However, the hypernym of concept in CLEVR (e.g., color,
19

---

Published as a conference paper at ICLR 2022
Task (novel concept: concept-eylindery
Learning Input
Test Question
Deseriptive sentenee:
How many cylinders are there?
There is object behind the metal
A: 3.
brown object: cylinder. Supplemental sentenee:
There object green object; is it cylinder? A: No.
Cube, sphere and cylineer shapes.
Task (novel concept: purple) Learning Examples
Test Question
Descriptive sentence:
many purple objects
There metal object to the left of the green object: it is parple object.
A: 3.
Supplemental sentence:
Q: object right green object; is it a purple object? A: No.
Gray, red, blue, green, brown, yellow and purple are colors.
Figure 6: A visualization of fast concept learning from biased data with supplemental sentences CLEVR. Notice that both tasks, the visual examples are purple cylinders. The association between the new concept and the visual appearance (shape or color) ambiguous, without supplemental sentences.
Supplemental sentemee: (novel concept: purpie)
Gray, red. blue, green, brown, yellow and purple are colors.
Type Concept Graph
Type II Concept Graph
yellow
yellow
blue
green
blue
green
purple
brown
purple
color
gray
red
gray
red
Figure 7: Two implementations of graph G Gooneept.
shape, material, etc.) is no longer concepts that can be visually grounded. They are non non-prototypical concepts because they can not be associated with a prototypical visual instance.
We have studied
different ways handle such kind of hypernym relation. They are different in terms of their way of constructing the concept graph Geoncept. namely Type and Type II. Illustrated in Fig. 7 in Type I graph, the novel concept = purple is connected with other concepts that share the same hypernym with In type II Geoncept. the non-prototypical concept color is treated node the concept graph. All concepts that are "colors", including e connected this new node color. Our experiment shows that, in the fast concept learning test with continual learning setting, our best model (FALCON-G Box Embedding) achieves a test accuracy of 88.40% using the type I concept graph. With the type II concept graph, it can only reach the accuracy of 71.34%. Thus, throughout the paper, we will be using concept graphs of type I.
C.3 THE GQA DATASET
In the GQA dataset, descriptive sentences are generated from the referential expressions produced by the templates in Table 10h, containing only concepts from Crase. We provided approximate
20

---

Published as a conference paper at ICLR 2022
(a) The descriptive sentence template in GQA.
Template
The <CONCEPT-1>> (<CONCEPT SEICEPT--2> 2>) (object) is w/n <CONCEPT> (object).
Example
The small man is sitting object.
Description
<CONCEPT> is the word for concept c. <CONCEPT-1> are other concepts.
(b) The supplemental sentence template in GQA.
Template
<CONCEPT- 1trr-11,cCCCCPPTT--- 2>>..- <CONCEPT XCCNCEEE N> describes the same property of an object.
Example
Sitting, standing describes the same property of object.
Description
<CONCEPT-1> are words representing c and its related concepts
(c) The test question generation template in GQA.
Template Example
Is the <CONCEPT-1> (<CONCEPT-13) (object) a/an <CONCEPT> (object)) Is the white woman sitting object?
Description
<CONCEPT> is the word for concept <CONCEPT- -11 i>> other concepts.
Table 10: Templates for generating descriptive sentences, supplemental sentences and test questions in GQA.
Model
QA Accuracy
Model
QA Accuracy
CNN+LSTM
54.47
CNN+LSTM
48.73
MAC
54.99
MAC
54.45
Box
Hyperplanc
Hypercone
Box Hyperplane Hypercone
NSCL+LSTM
55.41
54.19
54.81
NSCLALSTM
53.54
53.16
52.83
NSCLAGNN
55.32
50.07
54.75
NSCLAGNN 50.15
50.74
52.03
FALCON-G
55.96
54,22
54.56
FALCON-G 53.89
53.24
52.22
FALCON-R 55.42
55.73
55.79
FALCON-R 54.60
52.86
52.37
(a) Fast concept lcarning performance the GQA dataset. Our model FALCON-G and FALCON-R outperforms all baselines.
(b) Fast concept learning with only visual exam-
ples, evaluated on the GQA dataset. In this setting, NSCL+GNN will fallback to a Prototypical Net- work (Snell et al. 2017).
of 3.9k descriptive sentences, 2.3k of which for Coase, 600 for Cral and for Crear- Supplemental sentences are generated based on the templates in Table In supplemental sentences for concept Coase. we randomly remove a related concept c' from the sentence with a probability of 0.2. We generate 117k testing questions based on the templates in Table o 13.8k of them paired with concepts Crrain. 18k of them paired with concepts in Cual and 30k of them paired with concepts in Crest. We generated around 13.4k question-image pairs for pre-training. We use the object- object-based features produced by pretrained RCNN model Ren al. (2015), which provided in the dataset.
The main result is listed in Table Ia and Tlb We also compare the model performance on the "few-shot" learning setting, where supplemental sentences are provided. Out experiments shows that our model (FALCON-G or FALCON-R with box embedding spaces) outperforms other baselines in both settings.
C.4 THE CONNECTIONS BETWEEN THE BOX EMBEDDING AND OUR FORMULATION
We justify the choice of the box embedding space from two aspects. First, intuitively some supple- mental sentences in our setting can be modeled as concept entailment relationships, which can be well modeled by the box embedding space. Second, empirically, the box embedding space, among three embedding space choices studied the paper, is the best choice for representing entailment.
Modeling concept entailments (intuition). order to learn a new concept c, the model receives supplemental sentences relating with known concepts c'. A specific type of relationship is the hypernym relation. The fact that e a hypernym implies that "all objects that belong to concept è' also belong concept c". In the box embedding space, such relationship can be represented
21

---

Published as a conference paper at ICLR 2022
constraint that "the box of is inside the box c"". Thus, the geometric structures of the box embedding space naturally fits the concept entailment relations in our data.
There are two things to be noted. First, there other non-entailment relations that not be explicitly written down as a constraint. For example, the "same kind" relationship between concept red and concept b ue: they both colors. Second, instead of explicitly imposing these constraints while inferring the concept embedding for c, we use learned models to directly predict the embedding of c- but we hope neural models leverage such geometric structures.
Comparison with other box embedding spaces (empirical support). We have already compared the "box embedding" space with other, commonly-used approaches, such as the "hyperplane space" (i.e., the score is computed by dot-product), and the "hypercone space" (i.e., the score is computed by sosiia--imilarrtt) using downstream task performance measures. The box embedding space outperforms the other two (Table and of the main paper).
Other than downstream task performances, to further validate our intuition, we conduct further analysis on whether the predicted box embeddings of concepts actually satisfy the entailment relations. Specifically, using the CUB dataset, we use trained FALCON-G models based on different embedding spaces to predict the concept embeddings the validation and the test concept set. For concepts in the training set, we use the learned concept embeddings in the pre-training stage. Next, we use all concept embeddings and the underlying conditional probability measure for each space to compute Pr[c'|c] for all pairs of concepts (c, e').
We denote Pr[c'|c] as the prediction score for "c entails c" (i.e., the hypernym relation: is a hypernym of c). In particular, we use family of classifiers entails, (c, e') [[Pr[''[c] > 4.. where | is the indicator function and t threshold parameter. The conditional probability Pr[c'|d] is defined as the following. Denote e, and ee as the concept embeddings for and c'.
For the box embedding space, Pr[c/[c] neel/ necl/Preee where Prfe Nee] and Pr[ee] have been defined in Section 3.1
For the hyperplane embedding space, Pr[c']c] = o [ee"eer/ 7), where a is the sigmoid function, т 0.125, and 7 = 2d are the same set of scalar hyperparameters used for classifies objects. d is the embedding dimension.
For the hypercone embedding space, Pr[[/[c] of((eer ea) - ->//),, where (.)) is the cosine similarity function, and 0.1, = 0.2 are the same set of sealar hyperparameters used for classifying objects.
By varying the threshold f, we compute the AUC-ROC score for FALCON-G based on three different embedding spaces, shown in Table 12] This results suggest that the predicted concept embeddings from FALCON-G (Box) indeed better capture the hypernym relations between concepts. Note that the choice of specific hyperparameters: want to highlight that the hyperplane and hypercone embedding spaces do not naturally support a conditional probability measure for two concepts, and thus are not optimized for capturing concept entailments, yielding a lower AUC-ROC score.
Y, etc., will not affect the AUC-ROC score. We also
Methods
FALCON (Box) FALCON (Hyperplane) FALCON (Hypercone)
AUC-ROC Score
0.855
0.638
0.585
Table 12: AUC-ROC score of classifying entailment relationship based on concept embeddings from different embedding spaces.
D ABLATION DETAILS
This section discusses two ablation studies on how the number of base concepts in pretraining and the effect of the number of related concepts in supplemental sentences.
22

---

Published as a conference paper at ICLR 2022
of base concepts
FALCON-G
NSCL+GNN
MAC
130
76.32
75.21
65.16
211
81.33
78.50
73.88
Table 13:
Fast concept learning (with supplemental sentence) performance under different number of
base concepts.
% of all related concepts
FALCON-G
NSCL+GNN
MAC
76.37
73.38
73.55
25
80.20
77.16
73.94
50
80.78
76.93
74.13
81,20
77.77
74.23
100
81.33
78.50
73.88
Table 14: Fast concept learning performance under different percentage of related concepts in the supplemental sentence.
D.1 THE EFFECT OF THE NUMBER OF BASE CONCEPTS
In this section, we will conduct an ablation study on how the number of base concepts can contribute the model performance. We design new split of the CUB dataset derived from 50 training species (130 base concepts), 50 validation species (81 concepts), and 100 test species (155 concepts). In our original experimental setup, we have used 100 training concepts (211 base concepts).
We perform the same fast concept learning experiments on this new split, using our model FALCON- G, a concept-centric baseline NSCL+GNN, and an end-to-end baseline MAC. All concept-centric models use box embeddings spaces.
Our results are summarized in Table 13 Since we leverage the relationship between the novel concept and known concepts during the learning of the novel concept, all methods have accuracy drop when there are fewer base concepts. This demonstrates that transfering knowledge from concepts already learned is helpful. Moreover, our model FALCON-G still has the highest accuracy when the number base concepts is reduced.
D.2 THE EFFECT OF THE NUMBER OF RELATED CONCEPTS IN SUPPLEMENTAL SENTENCES
In this section, we design an ablation study on how the number of supplemental sentences would affect model performance. Since all information captured in the supplemental sentences the names of related concepts and their relations with the novel concept, we provide an ablation on the effect of the number of related concepts in supplemental sentences.
In the following experiments, we evaluate several models on the fast concept learning tasks on the CUB dataset. For each model, we use supplemental sentences containing 0% (no supp. sentence), 25%, 50%. 75%, and 100% (the setting described in the main paper) of all related concepts. Again, we compare our model FAI.CONGG. concept-centric baseline NSCL+GNN, and an end-to-end baseline MAC. All concept-centric models use box embeddings spaces.
The results are summarized in Table 14 and Fig. g We have the following observations.
1. More related concepts included in the supplemental sentence generally lead to higher accuracy, across all models.
2. In both ceoncept-centric models (FALCON-G and NSCL+GNN), the most significant improvement in test accuracy occurs between 0% and 25%. This suggests that these models can benefit from even just an incomplete set of related concept information.
just
3. Our model, FALCON-G, performs consistently the best across all percentages of related concepts.
23

---

Published as a conference paper at ICLR 2022
method
81
MAC
NSCL+GNN
80
FALCON-G
79
78
77
76
75
74
73
20
40
60
80
100
% of all related concepts in the supplementary sentence
Figure 8: Fast concept learning performance under different percentage of related concepts in the supplemental sentence.
E FAILURE MODE DETAILS
This section discusses how failure may occur in the detection module, the semantic parser, and the concept learning module, which may may not result in wrong answer.
E. FAILURE MODE IN DETECTION MODULE
There are few circumstances where the detection module from the feature extractor produces an eтror. Here we list two cases where the detection module may fail, illustrated in Fig. 15 In one scenario, the model may fail to detect one object from the scene, as in Fig. [5a Such errors usually occur among small and partially occluded objects. In the latter scenario, the model may recognize one object mask that does not correspond to any object, creating false positive, as Fig. [5bb However, the example object may be referenced correctly, as the error does not affect the evaluation of the program in the neural symbolic execution phase. In all scenarios, the detection error leads to an incorrect object Such error may propagate to the meta-leamiig stage, change the novel concept's 's embedding, and eventually result in correct wrong answer.
semantically ambiguous object being referenced during program evaluation.
E.2 FAILURE MODE IN SEMANTIC PARSER
There are few circumstances where the semantic parser produces error. Here list three cases where the semantic parser may fail, illustrated Table First, the model may produce synthetically wrong program. For example, there are invalid concept arguments in the output program. Following Johnson et al. (20175). the parser applies heuristics to the output, including removing the invalid subprograms where concept arguments are illegal, and produces feasible program, as in Table Toa In other circumstances, the model produces synthetically correct but semantically wrong program. We will evaluate the program as it is produced by the Seq2seq. Such evaluation will either lead to the right answer lead to the wrong answer. It should be no surprise that semantically incorrect program may lead to a wrong answer since the concept arguments are wrong, in Table 16b) A semantically wrong program may also lead to the correct answer if the question has redundancy in concept quantifiers that refers to the correct object, as Table [6c] Currently, there is mechanism implemented trying to recover from the semantic errors. In the future,
24

---

Published as a conference paper at ICLR 2022
Descriptive sentence: The shiny block that are behind small rubber thing is red object. Expected example object: The metallic red cube.
Question: How many red cylinders are to the right of the blue sphere?
Expected answer: 1. (The red cylinder)
Predicted answer:
Predicted example object: The metallic grey cube.
Explanation: The meta-learned concept cylinder not accurate enough; it is too ex- clusive that even the red cylinder is not con- sidered cylinder.
Explanation: The relational concept behind not accurate enough; it is so extensive that the grey cube is behind the yellow cube.
(b) Error case when imperfect visual grounding leads to a wrong answer in downstream question- answering. This is part of the task that learns the novel concept cylinder.
(a) Error case when the imperfect visual grounding leads wrong object example in the training novel
image. This part of the task that concept red.
Figure 9: visualization of failure cases of the concept learning module.
hope to incorporate methods such REINFORCE (as in Yi et al. (2088) and Mao et al. (2019)) to leverage visual information to refine the semantic parser.
E.3 FAILURE MODE IN CONCEPT LEARNING MODULE
Besides errors in the semantic parser and the detection module, the concept learning module may also lead to potential errors due to imperfect visual groundings. Here we list two cases where the concept learning module may fail, illustrated Fig. In the first scenario, the model encounters an error when filtering for a relational concept. The concept in the program is too extensive and filters out more candidates than desired. Such error results in wrong object example, as Fig. 9a In another scenario, the model encounters an eтror when filtering for novel concept recently meta-learned. The concept in the program too exclusive and filters out fewer candidates than desired. Such етror results in wrong answer in the final result, as in Fig. 9b In general, erгors may take place in the concept learner when filtering for a base concept, a relational concept, or the novel concept just meta-leerred when the visual grounding imperfect. Such errors may result in different filter results with more or fewer objects being included. Further operation on the wrong filter output may eventually end up with correct wrong
25

---

Published as a conference paper at ICLR 2022
Descriptive sentence:The blue matte thing to the right of the big cyan object is a cube.
Expected example object: The blue cube around the upper- right corner of the cyan cube.
Predicted example object: The purple metal cylinder, Detection error: The blue cube is not detected by the detec- tion module.
Descriptive sentence: The white small object is a plate. Expected example object: The white plate. Predicted example object: The lower side of the plate. X Detection error: The entirety of the white plate is not de- tected as one object
(a) Error when the detection module misses object. The blue boxes indicates the expected object that the detection module misses; the red boxes indicates the wrongly predicted objects.
Descriptive sentence: The metallic thing that is right of the tiny sphere is a cube.
Expected example object: The green cube.
Predicted example object: The green cube. Detection error: The union of the blue cube and the gray cube is detected as one object by the detection module.
Descriptive sentence:The large long object is silver object.
Expected example object: The long knife. Predicted example object: The long knife. Detection error: Shadows of the knife and the apple are detected as objects by the detection module.
(b) Error case when the detection module recognize false positive as an object. The green boxes indicates the expected and predicted object; the red boxes indicates false positives.
Table 15: A visualization of three failure cases the detection module. The first two descriptive sentences evaluated to find the wrong example object, and the last two evaluated find the right example object.
26

---

Published as a conference paper at ICLR 2022
Are there an equal number of small balls in front of the small ball and big cyan things?
Text
Expected Program
Count #qual (Fi wirter (Mil Ier (Relate(
F Filter (Filter (Scene () small), ball),
front), small), ball),
Filter (Filter (Scene (), big), cyan))
Predicted Program
Count (Filter (fite (Relate
Filter (Filter (Scene small), ball), front), small), Filter),
Filter (Filter (Scene (), big), cyan))
Fixed Program
Count (Filter (Relate(
Filter (Filter (Scene amall), ball),
front), , small),
Filter (Filter (Seene 0,, big), cyan))
(a) syntactically infcasible program. The program fixed with heuristies.
Text
Is there a metal cyan object the left of the big brown object?
Expected Program
Exists (Filter (Filter (Relate ( lilter (Fi (erreerri (Scene (), big), brown), left), metal), cyan))
Predicted Program
Exists (Filter (Relate Filter (Filter (Scene (), big), purple), left), metal), cyan))
Expected answer
Yes. Predicted answer No.
(b) syntactically feasible semantically wrong program, which yields a wrong answer.
Text
Is there big cyan object to the right of the big brown object?
Expected Program
Exists (Filter (eltteerriiisii (Relate
Filter (F (Filter (ecenneeeeenSeeeeee right), big), cyan))
(Scene (), big), brown),
Predicted Program
Exists (Fiests(Fi (Relate( Filter (Filter (Scene (), big), , purple), right), big), cyan))
Yes. (c) syntactically feasible but semantically wrong program, but the answer is correct.
Expected answer
Yes.
Predicted answer
Table 16: A visualization of three failure cases of the semantic parser. All examples are from the question answering part tasks on learning the novel concept cyan based on the image above.
27


Reviews:
[{"review_id": "XrgGV02CjR4", "reviewer": "Reviewer_98FU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very clear to read. It is a relatively novel task to learn the new visual concepts from a few images and sentences. In the experiments, the authors validate their methods in both real-world and synthetic datasets.\n\nThe model that the author provides contains a few separate components, such as some semantic parsers, that cannot be jointly optimized. The proposed method cannot recover from the errors made by those components.\n\nIt would be great to explain in a bit more details about the failure patterns of the proposed method.\n", "summary_of_the_paper": "The paper provides a method for learning new concepts from a few examples with both images and natural languages. For the evaluation, the proposed model is able to answer questions about some images related to the visual concepts.\n", "summary_of_the_review": "I find the paper interesting to read, and believe there are enough novelties in the proposed paper. I recommend a rating of “weakly accept”.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OSo7qDeaWxq", "reviewer": "Reviewer_2P5Z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I felt like this paper has potential but it just is not there yet. The writing and paper structure needs work. It took a really long to get to the true contribution of the paper, which if appears to be a new training and evaluation setup to evaluate how well a model can answer questions about novel concepts. The problem setup involves a meta-learning step where the model is presented with the novel concept in the form of a sentence and an image along with optional supplementary sentences. Aside from the problem formulation, the paper proposes a concept learning module on top of the Han et al. 2019’s Visual concept meta-concept learning model. The concept learning module is introduces as either a graph neural network or an RNN architecture that learns to relate the concepts extracted from the images, sentences and supplementary sentences.\n\nAll older concepts are learned during a pretraining stage (the one introduced in Han et al.). A new addition is the box embedding space that improves performance of the embedding space.\n\n\nThe problem I have with the paper is that it introduces (1) a new problem formulation, (2) a new meta-learning model, and (3) uses a geometric embedding space. But none of these are really justified. \n\nI would have expected to see a paper that answers: (1) Why is this an ecologically valid problem formulation? How does it relate to existing problem formulations? Why do existing baselines fail? (2) why a meta-learning approach is the best formulation to tackle the problem? And (3): Why the box embedding space? There should be more justification for by the box embedding space is the right one to use for the given data / task. The paper has the following sentence: “Here, we focus on the box embedding space (Vilnis et al., 2018) since it naturally models the entailment relationships between concepts.” but doesn’t dive into what these entailments are.\n\nAside from this, I didn’t find any ablation that studies how the model performs as the number of supplementary sentences changes. How does it scale as the number of base concepts changes or the number of object categories? Is there a cold start problem of requiring a certain number of base concepts during pre-training?\n\n\n\nMinor:\n- There are a few typos and incomplete sentences. Ex, “Our goal is to use meta-learning a system that can learn new concept quickly.” and “4) use the learned concept flexibly in different concepts”.\n\nThere are too many terms that all sound the same in the paper: “relational concepts”, “relational representations of objects”, “object-based concepts”, etc. It took me a while to wrap my head around which ones mean what. I suggest using consistent terminology throughout the paper when referring to a specific concept.\n", "summary_of_the_paper": "This paper presents a unified meta-learning neuro-symbolic framework for fast visual concept learning from diverse data streams. It introduces a new embedding prediction module to integrate visual examples and relations to infer novel concept embeddings. It’s meta-learning continuous learning approach also uses supplementary sentences to relate concepts to one another. They show improvements for the end task of question answering with novel concepts by first meta-learning those concepts during a meta-testing phase.\n", "summary_of_the_review": "Overall, I think the contribution would have been stronger with a proper grounding of why the problem is ecologically valid, why the solution proposed is an appropriate one, and why specific design choices (box embedding) were made. There are also missing ablations and rigorous evaluation of the model.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "K6gka_QPdTE", "reviewer": "Reviewer_W3YC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper's idea is very interesting and novel. I really enjoy reading this paper. I have some questions for the specific details and the datasets.\n\nFor the dataset (sect. 4.1), if I understand the approach correctly, the approach first needs a paired image and sentence to learn the base concepts. In this setting, the sentence should be a descriptive sentence that describes the object/concept in the image. If my understanding is correct, the CUB dataset setting is very natural. However, I wonder how would CLEVR and GQA dataset follow this setting? IIUC, the text in CLEVR and GQA are all questions, which are not descriptive sentences.\n\nFor the approach (sect. 3.3), IIUC, the proposed approach needs to extract concept relations from a supplemental sentence. First, the sentence is in the natural language form. How to do the semantic parsing of the sentence? The parsing might contain error. How to resolve that? As the sentence is in the natural language form, the name of the concept might in different form. How to resolve that? Second, it is pretty rare to see a paper that makes neural symbolic machine works for the real language. The neural symbolic machine works very well for the template language. Then I wonder how to design the operators in the neural symbolic machine? \n\nFor the setting, I wonder what is the setting for CUB? IIUC, CUB is a dataset for bird classification. From Fig. 4, it seems this paper treat the CUB as a retrieval task? The images of CUB dataset only contains one single bird. Therefore, I think the object centric approach might works pretty well. (Optional, this might be a stretch) I wonder would this approach works for some other datasets like Flickr30k or COCO? ", "summary_of_the_paper": "This paper tackles concept learning problem. This paper designed a neural symbolic machine based approach that is able to inference the concept embedding from a given image and sentence pair. It can also mediate the concept embedding using additional text explanation. To verify this approach, this paper follows a meta-learning setting. Specifically, the concepts are split into training, validation, and testing. The model first learn the base concepts from the training set and then quickly infer the concept embedding for the testing split. The paper achieved a superior performance over three datasets against baselines.", "summary_of_the_review": "I think this paper is a very interesting paper for concept learning from image and natural language. The motivation of this paper is very natural and the architecture is chosen appropriately. As long as the author could address my questions for the setting, datasets, and details of the approach, I would recommend accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "J3chA0szjVE", "reviewer": "Reviewer_6cyF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\n\nThe paper describes a novel technical approach to fast concept learning from a few examples. The proposed framework integrates the neuro-symbolic program executor, box embeddings, and graph neural networks to achieve the competitive performance in the considered benchmarks.\n\nThe main technical idea seems to lie in the use of the box embedding in the concept-centric (neuro-symbolic) approach. Section 2 concisely summarizes the context of this work.\n\nThe evaluation protocol looks comprehensive and convincing, including both detailed ablation and SoTA comparisons in multiple datasets.\n\n## Weakness\n\nI do not have a major concern on this paper. Below are a few minor questions.\n\nThe paper uses a pretrained Mask R-CNN for extracting objects in a given image. I wonder what is the effect of detection errors and the influence of on what dataset these detectors are trained. Presumably the neuro-symbolic reasoning module should be able to handle errors, but it seems possible to completely miss a correct object in the scene.\n\nAnother similar question is the error in language parsing. I assume the generator in this work does not incur an error in semantic parsing thanks to the use of templates, but was there any error originating from parsing / program execution?", "summary_of_the_paper": "The paper describes an approach to learn visual concepts with a few examples. The paper considers a problem consisting of paired images and sentences, optional concept descriptions, and the target question answering task for a new concept. The proposed approach utilizes the object detector (He 2017) to map visual objects or relationship into the box embedding space (Vilnis 2018), and also utilizes a neuro-symbolic program to identify the referring object as well as relationship among concepts. The new concept representation is inferred after the two graph neural networks, and the networks are trained to solve for the downstream task, which in this paper is question answering. The proposed approach is evaluated in three benchmarks (CUB, CLEVR, GQA) and shown to outperform the recent baselines (Hudson &Manning 2018, Mao 2019).", "summary_of_the_review": "The paper proposes a novel framework for fast visual concept learning that builds on a neuro-symbolic program executor, box embeddings, and graph neural networks. Even if the individual components might be from the existing work, the overall approach presents a novel technical approach. The evaluation based on downstream question answering shows the competitive performance of the proposed model over baselines in multiple benchmarks. Given the novelty and the effectiveness, I believe the paper makes a solid contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a meta learning framework to learn novel visual concepts with few examples. The proposed FALCON model uses an embedding prediction module to infer novel concept embeddings. This is done via paired image and text data as well as supplementary sentences. The resulting systems shows improvements on a series of datasets with synthetic and real images. The reviewers were supportive of this submission and praised the novelty, central ideas and experimental setups.

Concerns included:\
(a) [2P5Z] Justifying the formulations in this paper and situating it with past work -- "Why is this an ecologically valid problem formulation?", "why a meta-learning approach is the best formulation to tackle the problem?", "Why the box embedding space?"\
(b) [W3YC] More details required about the dataset and approach.\
(c) [98FU] Failure patterns

The authors provided detailed responses to these concerns. Concern (a), (b) and (c) were well addressed in the rebuttal and paper, and led to in increase in the reviewers rating.

Given the above, I recommend acceptance. But I do urge the authors to add the details provided in the rebuttal into the main paper. In particular, the concerns/suggestions by reviewer 2P5Z can hugely help in improving the paper and informing the reader.
Decision: Accept (Poster)