ZeroC: A Neuro-Symbolic Model for Zero-shot Concept Recognition and Acquisition at Inference Time
Authors: Tailin Wu
Megan Tjandrasuwita
Zhengxuan Wu
Xuelin Yang
Kevin Liu
Rok Sosic
Jure Leskovec
Venue: NeurIPS 2022 Accept (2022)
Keywords: zero-shot concept recognition
zero-shot concept acquisition
neuro-symbolic
inference time

Abstract: Humans have the remarkable ability to recognize and acquire novel visual concepts in a zero-shot manner. Given a high-level, symbolic description of a novel concept in terms of previously learned visual concepts and their relations, humans can recognize novel concepts without seeing any examples. Moreover, they can acquire new concepts by parsing and communicating symbolic structures using learned visual concepts and relations. Endowing these capabilities in machines is pivotal in improving their generalization capability at inference time. In this work, we introduce Zero-shot Concept Recognition and Acquisition (ZeroC), a neuro-symbolic architecture that can recognize and acquire novel concepts in a zero-shot way.  ZeroC represents concepts as graphs of constituent concept models (as nodes) and their relations (as edges). To allow inference time composition, we employ energy-based models (EBMs) to model concepts and relations. We design ZeroC architecture so that it allows a one-to-one mapping between a symbolic graph structure of a concept and its corresponding EBM, which for the first time, allows acquiring new concepts, communicating its graph structure, and applying it to classification and detection tasks (even across domains) at inference time. We introduce algorithms for learning and inference with ZeroC. We evaluate ZeroC on a challenging grid-world dataset which is designed to probe zero-shot concept recognition and acquisition, and demonstrate its capability.

ZeroC: A Neuro-Symbolic Model for Zero-shot Concept Recognition and Acquisition at Inference Time
Tailin Wu Department of Computer Science Stanford University tailinêcs. stanford. edu
Megan Tjandrasuwita
Department of Computer Science California Institute of Technology megantj @mit. edu
Zhengxuan Wu Department of Computer Science Stanford University wuzhengx@cs. stanford. edu
Xuelin Yang
Department of Computer Science
Stanford University xyang23@cs. stanf edu
Kevin Liu
Rok Sosič
Department of Computer Science Stanford University
Department of Computer Science
Stanford University rokecs. stanford. edu
liukêcs stanford... edu
Jure Leskovec
Department of Computer Science Stanford University jure@cs stanford edu
Abstract
Humans have the remarkable ability to recognize and acquire novel visual con- cepts in zero-shot manner. Given high-level, symbolic description of a novel concept in terms of previously learned visual concepts and their relations, humans recognize novel concepts without seeing any examples. Moreover, they acquire new concepts by parsing and communicating symbolic structures using learned visual concepts and relations. Endowing these capabilities in machines is pivotal in improving their generalization capability at inference time. In this work, introduce Zero-shot Concept Recognition and Acquisition (ZeroC), neuro-symbolic architecture that can recognize and acquire novel concepts in a zero-shot way. ZeroC represents concepts graphs of constituent concept models (as nodes) and their relations (as edges). To allow inference time composition, we employ energy-based models (EBMs) model concepts and relations. We design ZeroC architecture so that it allows a one-to-one mapping between a symbolic graph structure of concept and its corresponding EBM, which for the first time, allows acquiring new concepts, communicating its graph structure, and applying it classification and detection tasks (even across domains) at inference time. We introduce algorithms for learning and inference with ZeroC. We evaluate ZeroC on a challenging grid-world dataset which designed to probe zero-shot concept recognition and acquisition, and demonstrate its capability.
Project wehsite and code can he found at
www.:efrrdddddeeeeeeeeeeee edw/zeroc/
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

[a] training models
(b) inference: given a hierarchical concept graph, derive its model from trained models
20,
m
FiX, Gyea)
F
Spar
Cltne
Tpurp
111
m2
Tpop
Sperp
20
R(X, Mn,
F(x.my, ma (perp) F(x..m, F(x. (x..mm.. Edssps m)
(c) detection at inference:
cumbined musk me
L
L
LL
W
2010 Lat
Figure I: Composition of Concept Models. We demonstrate the model composition for novel, hierarchical concepts on the recognition of letter E. (a) During training, we learn the models for constituent concepts, concept "line" in this case, and relations, which are "parallel" and "perpendicular". (b) During inference, we take the concept graph of F and use to derive the model for F from the models of its constituents. Note that no traiiigg is performed the hierarchical concept F. (c) Example of detecting hierarchical concept pixel level with ZeroC during inference
using Eq.
Introduction
Humans learn in diverse ways. Besides learning from demonstrations of novel concept, humans can also learn concepts on a high-level. Consider learning the "rectangle" concept, for example. Suppose already mastered the concept "line" and relations of "parallel" and "perpendicular"; s/he can easily master the "rectangle" concept when told that "rectangle" consists of two pairs "lines", the lines within the pairs are "parallel," and the lines between the pairs are "perpendicular" Then s/he can directly this newly mastered concept to recognize "rectangles" in novel images.
that one has never seen such concept,
Such zero-shot concept recognition capability is still beyond the reach of deep learning models, which require many examples (as in typical supervised learning), or many tasks drawn from the same distribution (as in few-shot learning) to learn novel concept. Moreover, along with the above symbolic to instance direction, humans can easily the reverse. Suppose again that we haven't seen the "rectangle" concept, but have already mastered concepts of "line" and relations of "parallel" and "perpendicular". When seeing novel image which contains an instance easily decompose it into its constituent concepts of "lines" and their relational graph structure (from instance to symbolic). Moreover, humans can then communicate with each other about this new concept, allowing the transfer of knowledge in high-level and in succinct way. Such zero-shot concept acquisition capability at inference time is beyond the reach of today's machine learning and Al systems. Above all, endowing the above two capabilities of zero-shot concept recognition and acquisition capability to machines at inference time, will allow them to tackle more diverse tasks.
rectangle, humans can
Past efforts have attempted to address aspects of the above problem of recognizing and acquiring novel concepts inference time in zero-shot manner. An essential component is compositionality, i.e. the ability to compose new concepts from elementary ones. and 2 introduce techniques for compositional generation in the context of energy- based models (EBMs) and variational autoencoders (VAEs), respectively. However, these works only consider composing factors of variation (e.g., color, position, smiling, young). order to recognize concepts that consist of constituent concepts as parts, the structure of objects and their relationships are equally important. For instance, given 4 "line" concepts, without specifying their relational graph structure, they can easily form a trapezoid or a generic parallelogram. [3] designed transformer-bas based relational architecture for learning explicit relations, and showed that it can generalize to new objects for the relation. However, this architecture cannot generalize concepts with more complex relational graph structure.
remains an open problem how to design an architecture that has the capability to compose novel concepts at inference time, based on its internal relational graph structure of prior learned concepts

---

and relations. A separate line of works 15 8 tackle zero-shot classification, but are limited to novel composition of label features for concepts, and ignore relation features as well as the graph structure formed by concepts and relations.
Here we introduce our ZeroC architecture, which is model designed towards zero-shot concept recognition and acquisition. It models a visual concept or relation as an Energy-Based Model (EBM) with 3 inputs: an image, mask (or two masks for relation) denoting a set of pixels within the image, and a concept/relation label string. It returns a low scalar energy value if the mask(s) correctly indicates the concept or relation within the image, and high energy value otherwise. After training. the concept and relation EBMs are able to perform classification of concept/relation labels given an image, and detection of concept/relation mask(s) given an image and concept/relation label. The core contribution of this work is ZeroC's 's architecture and inference algorithms that allow a one-to-on correspondence between a specification of relational graph structure of novel concept and its corresponding composed concept model that can also perform similar downstream classification and detection, which enables zero-shot concept recognition and acquisition at inference time.
To perform zero-shot concept recognition at inference time, ZeroC can simply compose a new concept EBM, by summing over the constituent concept and relation EBMs according the specified graph structure. This new concept EBM has the same input-output format as its constituent concept EBMs, allowing the same capability of classification and detection. Note that through the above process, ZeroC have mastered concept without seeing an image, only symbolic instruction, thus is called "zero-shot". To perform zero-shot concept acquisition, we introduce an algorithm that given an instance (image) of a novel concept, parse it into a relational graph of prior learned concepts and relations. Such relational graph can then be transferred, even cross-domain, allowing models independently trained in a different domain to be able to zero-shot classify and detect this concept in its own domain.
There are currently no specialized benchmarks to test such zero-shot concept recognition and acqui- sition capabilities. Inspired by the Abstraction and Reasoning Corpus (ARC) D. have created a synthetic, grid-world based dataset with tasks that capture the essence of the above capabilities that are deceptively simple for humans, but very hard for neural models. We demonstrate that our model, trained to classify and detect elementary visual concepts and relations. is able to classify and detect novel concepts at inference time, being given a zero-shot symbolic graph structure. We also compare with a state-of-the-arr zero-shot learning model CADA-VAE 3 Due to that it is not designed for such graph-basee based composition, it significantly underperforms our ZeroC. Furthermore, we show that two independently trained models, one on 2D images and the other on 3D images, able to acquire hierarchical concepts from each other by communicating the graph structure and then perform classification and detection of these hierarchical concepts in their respective domains.
2 Method
In this section, we describe the ZeroC architecture for recognizing and acquiring novel visual concepts at inference time We give an overview of our approach, describe how it performs classification and detection, how it supports zero-shot concept recognition, how independently-tranl -traindd models can acquire hierarchical concepts from each other and, finally, introduce its learning method.
2.1
An Overview of ZeroC Architecture
The key components of ZeroC are concepts and relations. Each concept consists of a graph and an energy-based model. The concept graph describes the concept as a composition of its constituent concepts and relations between them. The base concepts that do not have any constituent concepts are called elementary concepts. Their graph is singleton. The concept masey model is used to recognize the concept in the input data. Each relation represented in ZeroC also with graph and an energy-based model. The relation graph is simply an edge that connects the two related concepts. hierarchical concept is a concept composed of constituent concepts as nodes and relations as edges according to a graph structure.
'In this paper, we focus on visual concepts. However, our framework is fully general and may be applied to other input modalities such as graphs, natural language, etc.
3

---

During training, energy-based models are learned from images and concept or relation labels. During inference, the learned models are used to recognize concepts seen during training or more complex hierarchical concepts which were not seen before. For concepts seen during training, their learned models are applied. For new hierarchical concepts not seen during training (i.e. without learned model), their models are derived from the graph of the new hierarchical concept and energy-based models of their constituent concepts and relations. Concepts and relations in ZeroC can be viewed as templates for objects and their connections, which then get grounded during inference with specific image, where those objects and relations are assigned actual values. ZeroC also handles object masks, which indicate object locations in the image.
2.2 Zero-shot Concept Recognition and Acquisition
Formally, we model a concept with Ex,M,C m, cf which maps an image x E IR,, mask a scalar energy. Similarly, we model relations with
m [0, 1]2 and concept labe Ex,M, .M..R (z, mi, m2, r), where and m2 are a pair of masks indicating two objects in the image æ, and r is relation label between the objects. The concept and relation models have probabilistic interpretation. For example, the energy function Ex,M, m, c) corresponds to the joint probability of
mx,
exp (f m,),)
Zc
where Zc E fee-Ex...cemma dx dm is a normalizing constant. Therefore, if the mask m is actually masking an object that belongs to concept e in image æ, then (x, m, m,c) will be high and Ex,M Ex (x, m, c) will be low, and vice versa. Essentially, the models define energy landscape for their respective multi-modal inputs that give low energy if the mask correctly identifies its corresponding concept.
Next, we show how ZeroC performs detection, classification, and models hierarchically composed concepts. We use the term HC-EBM to denote concept models and R-EBM for relation models.
Detection. We want infer the location mask of the concept c, given image æ. Probabilistically, we are computing Parxxc (m/(,, Dynamies (SGLD) to sample masks on the energy landscape IT
je, c). To perform detection, we employ Stochastic Gradient Langevin
X A Ex,,M i k-1 .)) + wh w ~ A()
n = m k-
(1)
2
where n is the inferred mask at the kth iteration, k = 1,2,.. Applying ||| as
K + +oo and + O, we generate samples from the distribution of Paixx 1 Zz.c
c)), where Zz,a- = f exp( hpp(-hoi Fx.M,c(r. m, c)) dm is a normalizing con- a finite K to generate sample n n 1,2, given x.
stant. In practice,
Classification. We want to determine whether the concept e appears in a given image x, i.e. compute (clx))).). We need to marginalize over the mask m:
(e|x(c|=) Px.o(E,)) Px(x) Evec S Px,M,o(a,m, dm
(x, m,c) dm
f exp (- -E, m, m,d) o)) dm
Eeec exp exp(-Ex,,,c(6 m, c)) dm
We again use SGLD 10 in Eq. to generate N samples m n [N] = (1, 2....N given c, and approximate the above integral using maximum a posteriori (MAP) estimation:
max exp Ex.M,C ii
Pe|x(c|=)= ~ Ecec [ax exp (-Ex,M,o mi c))
(2)
We will use capital letters (e.g. X, MM) to represent random variables, and small ones (e.g. x, m) to represent their instances.
"The concept label e is a categorical variable. that we use to refer to concept. like "line". "rectangle". or "cat". In the following, without confusion, we will refer to a concept using its label c.
In this paper, to index different concepts, j index relations, and index example images.

---

In practice, we only need to find the concept with the highest value in the numerator.
Zero-shot recognition of novel concepts. To master a novel hierarchical concept and directly use it for classification and detection only given its relational graph structure, we need a way to compose the previous concept and relation energy based models. Here we introduce the hierarchical composition rule, using an English letter "F" as an illustrative example
The concept F has one constituent concept, "line", and two relations, "parallel" and "perpendicular". The models for constituents are learned during training. During inference, these models are combined, using the concept graph, into a combined energy model for the letter F. Essentially, the models of all the recognized constituent concepts and relations are added together. Note that although F contains three lines, only one "line" model is needed. The concept graph acts as template and recognized line instances (objects) are matched with nodes in the template to obtain actual models. The "line" model is instantiated three times with specific values for the three identified línes and the three models plus their corresponding relations models are used to derive the hierarchical model. In addition to the models, we also need to combine the masks of all the recognized constituent objects.
Formally, we define the following composition rule for HC-EBMs plus the masks to be combined.
Definition 2.1. Hierarchical Composition Rule: Let hierarchical concept c have graph G (c, R). where C. are constituent concept nodes and R are relation edges. During inference, these nodes and edges are matched recognized concept objects and their relationships, which provides their models and masks. The combined model Ex,M Cx,,, m, c) is then sum of the models for all the nodes and edges in the graph and the mask M is the maximum of all the masks:
Ex,M,,( m, c) = =- Ex,M,.Ca,, mi, (.)
Σ TGR
Ex,M, MR, mn1 my2,
(3)
GEC
:
= max ma (MM)) [M²²}}
2.3 Zero-shot Concept Acquisition at Inference Time
Here, we introduce techniques for acquiring novel concept graph, which is useful for sharing high-level knowledge transfer between independently trained models. Such capability may have implications in future scenarios where a hypothetical self-driving car communicating the structure of a novel road sign to other cars on the road, or a 2D vision model learning a novel object and transferring it to a more accurate 3D depth model for inference. To achieve this, we need establish an equivalence relation between concepts in different domains (e.g. g. in 2D and 3D images).
Definition 2.2. Structural Equivalence for HC-EBM: Two HC-EBMs are structurally equivalent, if their graphs are isomorphic, and their constituent HC-EBMs recursively structurally equivalent
For example, HC-EBMs for a "rectangle" concept in 3D and 2D images are structurally equivalent if they have the same decomposition into two "parallel-line" HC-EBMs connected by "connect" R-EBM, and the "parallel-lines" have the same decomposition into two "line" HC-EBMs connected by a "parallel" R-EBM. Even though the HC-EBMs are grounded in different domains, they have the same abstract structure, so they represent the same hicrarchical concept.
With structural equivalence, mudependently-trainn hierarchical concepts via the graph structure. The algorithm works as follows (see Appendix A.3p: the image parsed to decompose it into a graph of concepts and relations previously learned by HC-EBM and R-BBM, then the graph is used in a different domain to compose its HC-EBM for the new concept. The key step of this process is the first, parsing step.
HC-EBMs from different domains can acquire
The parsing step can be seen as an inverse of the Hicrarchical Composition Rule and is critical in allowing the model to recognize novel hierarchical concepts in its domain using prior-lcarned concepts and relations from other domains, thus facilitating downstream knowledge transfer between domains. Alg. [ provides further details. Steps 1-2 infer the concept instances in image using SGLD (Eq. []. Here the energy E(p) Mc, c) is the summation of HC-EBMs on independent masks, each mask mi, representing a potential concept instance belonging to concept G. Etoserlap)
max (0. Em.,, EMC m,, - 1) penalizes overlapping masks. Since the image probably contains fewer concept instances than given, some masks are empty with all-zero values; Step 3 removes these.
FFor clarity, the concept graph shown has been simplified from the real model used in the experiments.
5

---

Algorithm Parsing Hierarchical Concept From Image
Require: HC-EBM E(C) with prior-learned concepts {c1, 02. ..)). R-EBM model E( with prior-learned relations (r1,72:...1)
Require: Image æ; containing unseen hierarchical concept c.
Require: Maximum number of instances N; for each concept G, i = 1, .....
1: E (x, := : Ei= E ( Ci)
+ E (Mc), where Mc := {m,, J,
I,...N; for == I...I. is the set all masks.
2: .M M. (E (x, Mc.c)) // Using Eq.
3: MM t tammove-mptt--maskk//eee
4: + (Tin Classify (E()) mi m2x), V(mj,, m32) E Mcc /l Using Eq. to classify
5: 9 <- Build-Crraph(MM R)
6: return G
Step 4 classifies relations (using Eq. 2 between pairs of detected concept instances, Step 5 combines the detected concept instances and their relations build the graph for this new hicrarchical concept
2.4 Learning
Since standard EBM-trainiig method yields poor performance as shown later in our experiments, we describe here our approach to train the models. Further details of our training algorithm are given We are optimizing the following objective:
in Alg. 2 in Appendix A.I
L
L (Iproved)
+ Oppos-st
(pus
(pu)
a'em ((mi) + aneg
(neg)
(4)
1=1
The expression of each term is given in Appendix A.2 The first loss T, (mmprovd)) is the objective proposed [2] a state-of-the-art EBM training technique. However, to enable the challenging zero-shot concept recognition and acquisition, we need more suitable inductive biases. The next three terms inject the right inductive bias for the task. ((pusstd) makes sure that the positive energy have similar level, so the composed concept EBM can identify all constituent concepts and relations, without one constituent EBM having too low energy and only recognizing it. The empty-mask regularization (em)) makes that the when the mask is empty, its energy is between the positive and negative energy, which we prove in Appendix A.3 that it is the necessary condition for correctly discover the underlying graph in Sec. 2.3 The Liee)) additionally provide additional negative examples, encouraging discovering full concept instead of a part of it (see Appendix A.2 for details).
Experiments
In this section, we set out to answer the following two questions: (a) given a specification of graph structure for novel hierarchical concept, can the composed concept model successfully perform classification and detection tasks? (b) Can HC-EBM acquire a novel concept from HC-EBM trained from another domain?
We also evaluated our model in challenging setting by comparing with other existing approaches in controlled and systematic way. Since there are no suitable dataset, we designed Hierarchical- Concept corpus, dataset based on grid wrid-world images. The dataset comes with testing images for the classification and detection tasks which contain more complex, hierarchical concepts, composed from simpler concepts the training set (See Appendix A.9 for details). The concept instances on our images have varying locations, size, and relative positions, making them more difficult than fixed-location fixed-size inference tasks such as in 3 We also provide 2D and 3D versions images for demonstrating concept acquisition across domains.

---

Table 1:
Performance of models classification accuracy for hierarchical dataset I and 2 (%) and detection for hierarchical concepts with distractors. For the latter task, we use the pixel-wise intersection-overruiiio (IoU) (%) as our metrics. The bold fonts in the tables indicate the best model comparing with baselines. The "Stattstics"" in classification predicts the class that has the most global label fraction.
Classification (acc.)
Detection (loU)
Model
HD-Letter HD-Concepp
HD-Letterrdistractor ID-Cooncptrriittracto
Statisties
46.5
53.5
5.69
12.6
Heuristies
(-)
(-)
42.3
29.2
CADA-VAE
18.0
66.0
(-)
(-)
ZeroC (ours)
84,5
70.5
72.5
84.7
ZeroC composition without R-EBM
62.5
32.5
45.3
84.3
ZeroC composition without HC-EBM
67.0
55.0
67.7
78.4
ZeroC without (ooo--ad))
43.6
65.5
76.1
81.5
ZeroC without (
64.5
59.0
60.0
84.2
ZeroC without L/em)
81.5
61.0
68.0
86.0
ZeroC with only I. Imppowedd
27.5
55.5
49.1
81.7
3.1 Zero-shot Classification and Detection of Novel Concepts
To test recogmition of novel concepts, we designed two datasets consisting of different concept and relation types. The HD-Letter hierarchical dataset consists of concept instances of "line" and relation instances of "parallel", "perp-edge" (perpendicular and touching edge). "perp-mid" (perpendicular and touching middle), together with distractor objects. Examples are provided in the form of 3- 3-tuples (image, mask, concept) for concepts, and (image, maski, mask2, relation) for relations. ZeroC models are trained via objective Eq. g (See Appendix A for examples of training datasets). At inference time, the models need to perform classification and detection on novel images with more complicated English characters of "B-shape", "T-shape" and "A-shape", given their structure graphs with up to nodes and 6 edges. For detection, the images also contain a few distractors (concepts unrelated to the ones the model is predicting). The model is asked to return a mask indicating which pixels belong to an instance of the specified novel concept. Although looking simple, this is actually very challenging task. because in order to solve the problem of classifying/detecting hierarchical concepts, the model needs to solve subgraph isomorphism problem, which is Narhard. For example, take the problem of detecting "E" shape in image with distractor of "T" and "Rectangle" (Fig. 2 (a) first subfigure), a graph with 10 nodes (lines), 45 possible edges. Note that during training time, model has only learned "line" concept, and relations "parallel", "perp-edge" (perpendicular, touching edge), and "perp-mid", but not the overall "E" shape. At inference time and given the "E" concept graph with 4 nodes (lines), 6 edges (relations), the model needs to find the "E" concept subgraph within the large image graph. This involves Cib x 4! = 5040 possible mask assignments. Additionally, model may not perfectly detect the masks for low-level concepts.
For more complex concepts and relations, we designed the HD-Concept hicrarchical dataset, which consists of training concepts of "E-shape" and "rectangle", and training relations of "inside", "enclose" and "non-overlap". The hierarchical concepts to be classified and detected are three characters which term Conceptl, Concept2 and Concept3, the ground-truth masks in Fig. 2 (a) indicate. The
three hierarchical concepts have the same multiset of concepts, but a different relation structures.
As the closest setting to our classification task is zero-shot classification, we compare ZeroC to state-of-the-ard zero-shot learning algorithm CADA-VAE R which we adapted to our setting, using the set of concepts and relations as feature embeddings to represent the concept graph (see Appendix A.6 for details). Additionally, we evaluated a "Statistics" baseline which samples random pixels on the image based on the global statisties of pixel occurrence, and a "Heuristics" baseline that randomly chooses a same-color connected object. We use a CNN-based architecture for ZeroC, as given in Appendix A.4 We add comparisons with ablation of different aspects of ZeroC, including without HC-EBM terms or R-EBM terms in the composition rule of Def. 2.1 (Eq. 3). and without different regularization terms of the objective in Eq.
Fig. shows demonstration of this experiment. From performance results in Table I we see that ZeroC achieves classification accuracy of 84.5% and 70.5% on HD-Letter and HDnCone, respectively, both higher than CADA-VAE. The gap is more significant in HD-LLtter, where there is a larger distribution shift for hierarchical concepts (as can be seen in Appendix A.9. which the
7

---

altred
grasditutt
labired
1.000
prckitee
IoU: 0.900
0.889
lalk 0.911
Casocaa
lol: 0.951
(b) Classification
cankiied work
ladecouu maha
Tota tnerg
Indiatall warks
Tixal boarg
E=-1013
3
D-0 228
pret
orsedt
E 0.62
Colfe Dasrce
E-0.095
E-0310
E
E=-0.014
comeed
Figure 2: Sample of ZeroC's Inference Results. Tasks are (a) pixel-wise detection, and (b) classification, for the datasets of HD-Letter (consisting of concept E-shape, F-shape and A-shape) and HDDConcepp Dataset (consisting of Conceptl, Concept2, Concept3). We that ZeroC has high pixel-wise detection IoU for the specified concepts.
classification, ZeroC uses each candidate concept's graph performs SGLD on composed HC-EBM to infer the mask and compute its energy. We see that ZeroC gives lower energy to the correct concept, even when sometimes candidate concepts have similar inferred mask ((b) right).
Table Performance models on acquiring concepts between models and domains at inference time (%).
Domain (2D image) Parsing
Domain image) Classification (acc.) 1 Detection (IoU)
Model
Isomorphism (acc.) 1 Edit distance
Statistics
2.33
3.14
33.3
Mask R-CNN B
Hrelation classification
35.5
1.01
--)
(-)
ZeroC, + ZeroC, (ours)
72.7
0.50
60.7
94.4
ZeroC without L ((pos-t2) ZeroCa
55.2
1.57
$4.7
90.5
ZemoC without L.mg + ZeroC;
53.5
0.99
52.5
92.1
ZeroC, without Ziom ZeroC,
50.7
1.58
51.3
95.4
ZeroC, with only (lmpmwoo)) ZeroC,
2,00
50.5
94.7
ZemCa with ground truth graph (upper bound)
61.8
94.2
joint embedding of the image and features learned by CADA VAE is insufficient to handle. The reason for the low accuracy is that CADA-VAE is not able address these out-of-domain distribution shifts (See Appendix A.7 for details). During inference, its embedding (multi-hot vector) for the graph structure can contain up 10 hots (4 lines, 6 edges), while during training, it is only up to 1hht.. This example also shows the intrinsic difficulty the task. In addition, Table [ and Fig. 2 (a) show that ZeroC is able to detect the hierarchical concepts in the presence of distractors, and that its performance significantly better than "Statistics" and "Heuristics" basclines.
The results of our ablation studies in Table 1 show that both the HC-EBM and R-EBM terms in Eq. are key to successful classification and detection. This likely because without relations and using only concept terms, ZeroC may lose its ability to distinguish between different hierarchical concepts; for example, both "E-shape" and "A-shape" have lines but different relation structures. Moreover, we see that both (ppos-sid) and (meg) improve classification and detection performance.
3.2 Acquiring Novel Hierarchical Concepts Across Domains
We show that ZeroC can acquire novel hierarchical concepts across models and domains (Fig. We train ZeroC model in domain ZeroC,, where the images are 2D and one-hot color-coded, containing the the same training concepts and relation types as in HD-Letter in Sec. 3.1 Independently, we train another model in different domain ZeroCz, where the images contain the same set of concepts and relations, but are viewed in 3D from certain camera angle, have larger size, and use RGB colors. At inference time, each test task consists of a tuple of three images showing the hierarchical concepts in the first domain to ZeroC,. ZeroC, is only allowed to send symbolic information, ZeroC2. Then ZeroC, performs classification and detection tasks on three example images in second domain. In addition, we also evaluate whether ZeroC; can parse the concept graph of the hierarchical concept correctly, using the metric of graph edit distance dedi and graph-isomorphism accuracy w.r.t. the ground-truth. Note that the graph-isomorphism accuracy is stringent metric,
to few bits, to

---

Iine
3D image
perp-adge
perp-mid
2D concept demonstration
parallel
discovered graph
inferred:
E-1:276
E-1.134
E-U.820
E
parse
Hy
channel
E=0-007
D=0.130
R=0-665
dodu=1
I
EeO 700
R50.957
Tel.111
ZeroC,
ZeroCz
Figure 3: Acquiring Hierarchical Concept between Domains. The figure shows actual example tasks and results. At inference, ZeroC; sees the 2D demonstration of three images showing three unseen concepts Cı, Oz. Ca. It first parses each image into respective concept graphs. We that except for that has edit distance dedi of I, the others have perfect parsing. ZeroC; then sends ZeroCa parsed concept graphs, which ZeroCa uses to perform classification that selects which 3D image corresponds to each concept.
which is only if the parsed graph is isomorphic to the ground-truth, and 0 otherwisd We compare with strong baseline Mask R-CNN [33 for object detection together with relation classification. The relation classification network is CNN which takes input pair of embeddings produced by the Mask R-CNN, and predicts the label of the relation (see Appendix
[A.8] for details). We did extensive tuning of the Mask R-CNN approach to obtain its best performance. We also evaluate the performance ZeroC₂ given the ground-truth structure graph, as an ideal scenario of perfect parsing, providing an upper bound for the performance.
Table 2 shows that ZeroCi achieves 72.7% graph isomorphism accuracy for parsing the hierarchical concepts in the first domain, significantly higher than Mask R-CNN relation classification. After sending the concept graphs in the second domain, ZeroC₂ achieves a classification accuracy of 60.7% and detection loU of 94.4%. Without this information and relying global statistics, the classification accuracy of ZeroC₂ is 33.3%. This demonstrates that ZeroC2 able to acquire novel hierarchical concepts at inference time, from independently trained model from different domain. The ablation study shows that I [ I (neg) and [(pes) all contribute to more accurate parsing. The standard technique of EBM training 12 insufficient achieve good parsing, leading to less accurate transfer of hierarchical concepts. We also see in the ablations that even without perfect parsing from ZeroCi, the reduction of classification accuracy for ZeroCa is small, showing that it is able to classify under noisy specification of the concept graph. We discuss further on the generality (Appendix A..33), scalability (Appendix A. 14) and computationa complexity (Appendix A. in respective Appendix sections.
4 Related Work
Our work relates to visual compositionality, concept and relation learning, and zero-shot learning.
Visual compositionality. Compositionality is a key for addressing diverse tasks given finite ba- sic knowledge. Some approaches use composition EBMs for generation [1, and VAE-basa architecture for bi-directional symbol-image generation that can also learn logical recombination of concepts 2 These two works focus composing factors of variation, e.g. color, position, smiling, young. In contrast, our work focuses on concepts that abstract objects, where the internal hierarchical relational structure is key. Moreover, while the above works focus generation, our work focuses the tasks classification and detection. A novel Bayesian-bar based method for few-shot learning on the
This is a stringent metric since, e.g. for "Eshape" that contains 4 concept nodes and 6 relation edges, an individual accuracy of 0.9 would result in ~ 0 = 0.349 isomorphism accuracy, and an individual accuracy of 0.8 would result in 0,84+6 0.107 isomorphism accuracy.

---

Omniglot dataset shows that compositionality is pivotal for improved performance [4]. They achieve compositionality via hierarchical MCMC sampling on hand-coded priors of elementary concepts and relations. In comparison, our work is neural network-based and only requires demonstrations of elementary concepts and relations, reducing hand-coded priors. Another approach introduced modular neural network, which uses composition of neural modules for visual question answer- ing [5.. Their method can be seen as composing transformations on a representation. In comparison, our composition is achieved via composition of energy landscapes. While they focus question answering on scene graphs, we focus on classification and detection of hierarchical concepts.
Concept and relation learning. There has been exciting progress in concept learning and relation learning. Works in concept learning generally represent concepts in latent space via prototypes [6. or via latent embedding such as SCAN [2]. InfoGAN [8 and
Concept 200 introduced EBMs to represent concepts with a demonstration in simple state space. 21 further introduced unsupervised learning of local and global concepts with EBMs. Regarding relation learning, non-local neural networks 22 Relation Networks 23 Neural Relational Inference
Learner 19
! and C-SWM [5 use latent complete graphs as inductive biases to represent potential relations. PrediNet explicitly represents propositions, relations, and objects with a transformer-bas architecture, and demonstrates that it can learn relations that generalize to novel shapes of objects and column patterns. In comparison to the above works, ZeroC explicitly learns both concepts and relations, which has the unique capability to recognize and acquire hierarchical concepts inference.
typically learn a joint embedding space between image and feature labels, and perform classification at inference time on images with novel classes, based on how the novel classes correspond to a set of features. Our generalization task also requires generalizing to new concepts without seeing the image (zero-shot), but has the important distinctions that at inference time, our concepts to be inferred lic at a higher hierarchy than that in training, and furthermore use the structural concept graph. In comparison, in standard zero-shot learning, concepts in training and inference lie at the same hierarchy level, and only generalize to new combinations of features (constituent concepts) while neglecting relation structures.
Zero-shot learning. Zero-shot learning methods 4
5 Conclusion
In this paper, we introduce ZeroC, a new framework for zero-shot concept recognition and acquisition at inference time. Our experiments show that in a challenging grid-world domain, ZeroC is able to recognize complex, hierarchical concepts composed of English characters in a grid- world in a zero-shot manner, being given a high-level, symbolic specification of their structures, and after being trained with simpler concepts. In addition, we demonstrate that an independently trained ZeroC is able to transfer hierarchical concepts across different domains at inference. Although this work is evaluated only in wrid-world visual domain, we are the first to address this difficult challenge. We are also excited to see its potential application in broader domains, e.g. in Al for scientific discovery. where it may infer novel patterns and concepts from data in a zero-shot manner. We hope that this work will make useful step in the development of composable neural systems, capable of zero-shot concept recognition and acquisition and hence suitable for more diverse tasks.
Acknowledgments and Disclosure of Funding
We thank Rui Yan and Blaž Škrlj for discussions and for providing feedback manuscript. We also gratefully acknowledge the support of DARPA under Nos. HR001 12190039 (TAMI), 1660011924033 (MCS); ARO under Nos. W91 INF--6-1-00034 (MURI). W911NF-16-1-017 (DURIP): NSF under Nos. OAC-1835599 (CINES), DACC-193577 (HDR), CCCF-191894 (Expedi- tions), NIH under No. Neurosciences Institute, Amazon, Docomo, GSK, Hitachi, Intel, JPMorgan Chase, Juniper Networks, KDDI, NEC, and Toshiba.
(HuBMAP). Stanford Data Science Initiative, Wu Tsai
The content solely the responsibility of the authors and does not necessarily represent the official views of the funding entities.
10

---

References
[]] Y. Du, S. Li, and I. Mordatch, "Compositional visual generation and inference with energy based models," 34th Conference on Neural Information Processing Systems, 2020.
[2] I. Higgins, N. Sonnerat, L. Matthey, A. Pal, C. P. Burgess, M. Bosnjak, M. Shanahan, M. Botvinick, D. Hassabis, and A. Lerchner, "Scan: Learning hierarchical compositional visual concepts," International Conference on Learning Representations, 2018.
[3] M. Shanahan, K. Nikiforou, A. Creswell, C. Kaplanis, D. Barrett, and M. Garnelo, "An explic-
itly relational neural network architecture," in International Conference Machine Learning. PMLR, 2020.
[4] B. Comera-Paredess and P. Torr, "An embarrassingly simple approach to zero-shot learning," in International conference on machine learning. PMLR, 2015.
[5] H. Jiang, R. Wang, S. Shan, Y. Yang, and X. Chen, "Learning discriminative latent attributes for zero-shot classification," in Proceedings of the IEFE International Conference on Computer Vision, 2017.
[6] M. Bucher, S. Herbin, and F. Jurie, "Generating visual representations for zero-shot classification." in Proceedings of the IEEE International Conference on Computer Vision Workshops, 2017.
[7] T. Zhao, G. Liu, C. Ma, E. Chen et al., "Zero-shot learning: An energy based approach." in 2018 IEEE International Conference on Data Mining. IEEE, 2018.
[8] F. Schonfeld, S. Ebrahimi, S. Sinha, T. Darrell, and Z. Akata, "Generalized zero-and few-shot learning via aligned variational autoencoders," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[9] F. Chollet, "On the measure of intelligence." arXiv preprint arXiv:1911.01547
[10] Y. Du and I. Mordatch, "Implicit generation and generalization energy- based models," arXiv preprint arXiv:1903.0866
2019.
[11] M. Welling and Y. W. Teh, "Bayesian learning via stochastic gradient langevin dynamics," in Proceedings of the 28th International Conference on Machine Learning. Citeseer, 2011.
[12] Y. Du, S. Li, Tenenbaum, and Mordatch, "Improved contrastive divergence training of energy-based models," in Proceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139, PMLR, 18-24 Jul 2021. [Online]. Available: https://proceedings www. 13t.////.////lbbmmmmm
[13] K. He, G. Gkioxari, P. Dollár, and R. Girshick, "Mask R-CNN," in Proceedings of the IEEE
International Conference on Computer Vision, 2017.
[14] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum, "Human-level concept learning through probabilistic program induction," Science, vol. 350, no. 6266, 2015.
[15] Andreas, M. Rohrbach, T. Darrell, and D. Klein, "Neural module networks," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2016.
[16] J. Snell, K. Swersky, and R. S. Zemel, "Prototypical networks for few-shot learning," Advances in Neural Information Processing Systems, 2017.
[17] K. Cao, M. Brbić, and J. Leskovec, "Concept learners for few-shot learning," in International Conference on Learning Representations, 2021.
[18] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, "Infogan: Inter- pretable representation learning by information maximizing generative adversarial nets." in Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016.
[19] J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu, "The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision," arXiv preprint arXiv: /904. /2584, 2019.
[20] I. Mordatch, "Concept learning with energy-based basee models." arXiv preprint arX1v:1811.02286.
2018.
[21] Y. Du, S. Li, Y. Sharma, Tenenhaum, and Mordatch, "Unsupervised learning of composi- tional energy concepts," Advances in Neural Information Processing Systems, vol. 34, 2021.
11

---

[22] X. Wang, R. Girshick, A. Gupta, and K. He, "Non-local neural networks," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[23] A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap,
"A simple neural network module for relational reasoning," Advances in Neural Information
Processing Systems, vol. 30, 2017.
[24] T. Kipf, E. Fetaya, K.-C. Wang, M. Welling, and R. Zemel, "Neural relational inference for interacting systems," International Conference on Machine Learning. PMLR, 2018.
[25] T. Kipf, E. van der Pol, and M. Welling, "Contrastive learning of structured world models." in International Conference on Learning Representations, 2020. [Online]. Available:
[26] J. Rajasegaran, S. Khan, M. Hayat, F. S. Khan, and M. Shah, "Self-supervised Knowledge Distillation for Few-shot Learning." Xi9:20006009785 [cs], Aug. 2020, arXiv: 2006.09785.
[Online]. Available:
[27] T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature Pyramid Net- works for Object Detection," 2017, pp. 2117-2125. [Online]. Available: https://openaccess..n theeee
com/content_cve ceppnttttet__t____eeeeee wwwp_p00w/pppaepeeee 2017//ttll/ Feature_ Pyramid Networks_westveee S01R_2017____ 2007__pper.html
[28] T.-Y. Lin. M. Maire, Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Ziticck "Microsoft COCO: Common Objects in Context," in Computer Vision ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds. Cham: Springer International Publishing, 2014.
[29] R. S. Sutton, "Sutton & barto book: Reinforcement learning: An introduction." in A Bradford Book. MIT press Cambridge, MA, 1998.
[30] E. C. Shin, I. Polosukhin, and D. Song. "Improving neural program synthesis with inferred execution traces," Advances in Neural Information Processing Systems, vol. 31, 2018.
[31] M. Chevelier-Boivvert,, D. Bahdanau, S. Lahlou, L. Willems, C. Saharia, T. H. Nguyen, and
Y. Bengio, "Babyai: A platform to study the sample efficiency of grounded language learning," arXiv preprint wwx.iii811000222.. 2018.
[32] N. Rabinowitz, Perbet, Song, C. Zhang, A. Eslami, and M. Botvinick, "Machine theory of mind," in International Conference on Machine Learning. PMLR, 2018.
[33] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, "The caltech-uess birds birds-200-2011 dataset," 2011.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In conclusion and Appendix
A00
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In
Appendix A.II
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A.3
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] The Appendix includes full details on model architecture, training and evaluation to reproduce the experimental results. Code and data will be released upon publication of the paper. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Full details to reproduce the experiments are included in the Appendix.
12

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix of respective experiment section.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A] We use our own created benchmark.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [Yes]
In Appendix A.9 we describe the engine and training/testing tasks. Full dataset will be released upon publication.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "gVO1MIoibrv", "reviewer": "Reviewer_maWL", "summary": "The main contribution of this paper is a system (ZeroC) for pattern detection and classification that, once trained on elementary concepts, is able to zero-shot adapt to complex concepts as long as the complex concept can be described as a composition of elementary concepts & relations via a graph. Individual energy-based models (EBMs) can be trained to match each elementary concept or relation, each assigning low energy to patterns that fit its ‘signature’ (i.e. template). To recognize a complex pattern, individual EBMs can be composed according to the concept-composition graph, which allows for zero-shot recognition for arbitrary combinations of learned concepts and relations.\n\nTo evaluate zero-shot concept recognition under the scenario that the proposed method is designed for, this paper also introduces two datasets since no existing dataset are sufficiently relevant. One dataset is easier, containing letters and simple relations, while the other is harder, containing more complex patterns and relations. \n\nExperiments show that ZeroC outperforms existing zero-shot concept recognition approaches on the proposed datasets, in both in-domain and cross-domain (2D images $\\rightarrow$ 3D images) settings. Additional experiments on CLEVR give clues about potentially broader usage.", "questions": "Here are a handful of questions pertaining to experimentation details that could potentially make the paper more insightful.\n- Line126: “*We want to determine if the concept c appears in a given image x. We need to marginalize over the mask m.*” How sensitive is your approach to the **tightness** of a mask? For example, does the concept c have to be fully occupying the entire masked space to be recognized?\n- Line 149: “*the mask M is the maximum of all the masks*”. I suppose each constituent is being recognized independently, right? Then is it possible that the predicted masks don’t form a contiguous region, which would indicate a detection error? How would you plan to address this case?\n- Line 230: “*The hierarchical concepts to be classified and detected are three characters which we term Concept1, Concept2, Concept3*”. These three concepts look like random patterns. How did you choose them in the first place?\n- Does your approach exhibit any flexibility in recognizing concepts that have been slightly stretched or squeezed with the height-width ratio being altered? If such flexibility is allowed, it’d be great to visualize the performance vs. the degree of distortion.\n- Line 249: “*During inference, its embedding for the graph structure can contain up to 10 hots, while during training, it is only up to 1-hot*”. Does the performance exhibit any degradation across inference examples from 1-hot up to 10-hot? If yes, how does the degradation look like versus #hots?\n- Line 263: “*… viewed in 3D from a certain camera angle …*”. How is a 3D image input represented? Is it still a 2D matrix with 3 channels or is it represented as a 3D scene with depth?\n- Line 263 again: I suppose the camera angle may distort a perpendicular relation so that it looks like an acute angle. And this distortion varies across camera angles. How did you address this and how did you choose camera angles when you constructed this dataset?\n- Line 264: “*each test task consists of a tuple of three images*”. Why three images at a time? \n- This doesn’t have to answered thoroughly but I’d love to see your thoughts: How do you expect your approach to generalize to non-90-degree angles, like the top angle in “A”, or relations about arcs? \n", "limitations": "Broader social impact and limitations are discussed in the supplementary. \n\nHere I briefly point out a few limitations and addressing any of these would make the paper stronger.\n- The datasets used for evaluation cover limited number of concepts.\n- The search process for a mask indicating “objectness” is inefficient.\n- The universality of the set of primitive concepts and relations isn’t justified.\n- The current algorithm is unable to evolve with dynamic expansion and compression as the number of concepts to model increases.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This is a good move towards systematic generalization. The paper presents a way to train a recognition model on elementary patterns and perform inference on hierarchically composed patterns, given that the complicated pattern can be symbolically described (i.e. via a graph). \n- The experiments are solid and suggest possible applicability to realistic use cases: 2D-to-3D domain adaptation, and generalization to real-world images.\n\nWeaknesses\n- Each grid-world dataset used for evaluation contains only 3 pre-defined concepts. There is still a long way to go in order to more rigorously test the model’s compositional generalizability.\n- Object localization is achieved through a somewhat clueless search with location candidates sampled from a distribution. This might lead to an efficiency issue once the task difficulty, image resolution or object crowdedness is scaled up. \n- At present, all primitive concepts and relations are hand-crafted, but the paper didn’t explain the universality of those primitives in a general object recognition task. For example, the current task setup only touches upon parallel vs. perpendicular, but it is unclear how to define a template for an “acute angle” since it is defined as a range. In other words, the question is: how much space of the visual territory do you expect that can be approximated by your set of primitives? In order to move towards recognizing real-world objects, what other primitives should be included or what bottlenecks might need to be conquered?\n- It would be great to demonstrate how the pool of primitive recognition models could be dynamically expanded and compressed to allow for progressively increasing demands. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "d2Wj0zftKrc", "reviewer": "Reviewer_Yv6P", "summary": "This work proposes a framework for zero-shot concept learning, i.e., learning and recognizing new concepts at inference time. The proposed frameworks extracts concepts and relations and composes them to learn new concepts hierarchically. The proposed method is evaluated on some synthetic datasets. \n", "questions": "How would the basic relations and concepts be defined or discovered for real world images?", "limitations": "It seems non-trivial to apply the current approach successfully to real world images, which is the main limitation and the authors might want to have more discussion about it and future works to address remaining challenges. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n(1) Zero-shot concept learning at inference time is an interesting problem and requires more attention.\n\n(2) This work proposed a neat framework for learning concepts in a compositional way using energy based models and hierarchical composition. \n\nWeakness \n\n(1) There should be more discussion about how the proposed approach can scale to real world images that can include many edges and shapes with different relations. It seems this might become prohibitively costly. \n\n(2) The evaluation would be more convincing if it can also include results on some few-shot learning datasets such omniglot or, even better, some real world images.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JSFjZTQPI17", "reviewer": "Reviewer_P8en", "summary": "This paper addresses the problem of zero-shot concept recognition. By representing concepts with a graph, and by exploiting an energy-based model, the proposed approach enables detection and classification of novel entities at inference time. The approach can be considered \"neuro-symbolic\" as it combines symbolic graph-representation with (neural) energy-based models with a one-to-one translation of one into the other.\n", "questions": "* There is no description of the computational cost of the proposed approach. For example, Equation 2, although it is computed via MAP estimation, and although it needs only to find the concept with the highest value in the numerator, appears to be costly. Could the authors comment on this point?\n", "limitations": "The experimental evaluation is conducted on a single artificial dataset. While this is comprehensible for a novel task, it is not clear to what extent the proposed approach is tailored on the domain of images (e.g., see also Algorithm 1) or whether it could be easily genaralized to other domains.\n\nAlso, a discussion of the computational cost of the proposed approach is missing.\n\nTypos:\n- Line 29, \"which requires\" -> \"which require\"\n- Line 71, \"that can given\" -> \"that given\"\n- Line 126, \"to determine if\" -> \"to determine whether\"\n- Line 134, \"a English\" -> \"an English\"", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The addressed problem is challenging and important.\n+ The proposed solution is sound and elegant.\n+ The experimental evaluation, though on a single dataset, is well conducted.\n\n- Generalization to multiple datasets and scenarios is not straightforward.\n- No discussion on computational complexity.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-SE8DRdE5kR", "reviewer": "Reviewer_AN9q", "summary": "An EBM hierarchically describing concepts and relationships is used for concept discovery in a crowded scene with irrelevant concepts. This graph can be transferred across different domains.", "questions": "1. Even though it is claimed that the algorithm can learn new concepts, isn't the structure of the energy model used in this paper already tuned to the discovery of spatial relationships between horizontal and vertical lines? Or if I run the same algorithm on data with arbitrary concepts (for instance, triangle, circle, object A inside objet B, object is a closed contour, etc.) will it be able to discover those as well?\n\n2. How does inference scale if I increase the resolution to 256 x 256? Would it work? For the size and structure of your dataset, isn't the partition function of your energy computable (albeit expensive)?\n\n3. Same question about learning\n\n4. Isn't the particular way in which you encode spatial relationships (relative) mean that this model cannot distinguish the concept of W and M, or 6 and 9, since they are both rotations of each other, and therefore satisfy the same relative relationships?\n\n5. This model resembles heavily the ones in [1] and [2], which also describe letters as graphs of lateral relationships that entangle nodes containing edges. What are the main differences? Can this model be used to solve CAPTCHAs? Experiments showing this would definitely be much more convincing as to its capabilities.\n\n6. I wasn't able to understand precisely which information is conveyed from ZeroC1 to ZeroC2. Could you clarify this section in the paper?\n\n7. When using the loss from [12], you mention that you neglect the entropy term. What's the problem with keeping it? Would the results from [12] improve had they neglected it?\n\nMinor: Figure 4 in the appendix, training relation, first column, doesn't look like a perp-edge.\n\n\n[1] A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs. Dileep George et al. Science 2017\n\n[2] Generative Shape Models: Joint Text Recognition and Segmentation with Very Little Training Data. Xinghua Lou. NIPS 2016.", "limitations": "Described above", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nTechnically correct\n\n- Addressing the relevant problem of hierarchical concept representation\n- Clarifying examples\n- Introduces a new dataset to test these ideas\n\nWeaknesses:\n\n- The size of the problem makes it too simple to brute force\n- Unclear whether inference would scale\n- Unclear how to learn arbitrary concepts from data\n- Unclear how learning would scale\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept