Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The main contribution of this paper is a system (ZeroC) for pattern detection and classification that, once trained on elementary concepts, is able to zero-shot adapt to complex concepts as long as the complex concept can be described as a composition of elementary concepts & relations via a graph. Individual energy-based models (EBMs) can be trained to match each elementary concept or relation, each assigning low energy to patterns that fit its ‘signature’ (i.e. template). To recognize a complex pattern, individual EBMs can be composed according to the concept-composition graph, which allows for zero-shot recognition for arbitrary combinations of learned concepts and relations.

To evaluate zero-shot concept recognition under the scenario that the proposed method is designed for, this paper also introduces two datasets since no existing dataset are sufficiently relevant. One dataset is easier, containing letters and simple relations, while the other is harder, containing more complex patterns and relations. 

Experiments show that ZeroC outperforms existing zero-shot concept recognition approaches on the proposed datasets, in both in-domain and cross-domain (2D images $\rightarrow$ 3D images) settings. Additional experiments on CLEVR give clues about potentially broader usage.
strengths_and_weaknesses: Strengths:
- This is a good move towards systematic generalization. The paper presents a way to train a recognition model on elementary patterns and perform inference on hierarchically composed patterns, given that the complicated pattern can be symbolically described (i.e. via a graph). 
- The experiments are solid and suggest possible applicability to realistic use cases: 2D-to-3D domain adaptation, and generalization to real-world images.

Weaknesses
- Each grid-world dataset used for evaluation contains only 3 pre-defined concepts. There is still a long way to go in order to more rigorously test the model’s compositional generalizability.
- Object localization is achieved through a somewhat clueless search with location candidates sampled from a distribution. This might lead to an efficiency issue once the task difficulty, image resolution or object crowdedness is scaled up. 
- At present, all primitive concepts and relations are hand-crafted, but the paper didn’t explain the universality of those primitives in a general object recognition task. For example, the current task setup only touches upon parallel vs. perpendicular, but it is unclear how to define a template for an “acute angle” since it is defined as a range. In other words, the question is: how much space of the visual territory do you expect that can be approximated by your set of primitives? In order to move towards recognizing real-world objects, what other primitives should be included or what bottlenecks might need to be conquered?
- It would be great to demonstrate how the pool of primitive recognition models could be dynamically expanded and compressed to allow for progressively increasing demands. 

questions: Here are a handful of questions pertaining to experimentation details that could potentially make the paper more insightful.
- Line126: “*We want to determine if the concept c appears in a given image x. We need to marginalize over the mask m.*” How sensitive is your approach to the **tightness** of a mask? For example, does the concept c have to be fully occupying the entire masked space to be recognized?
- Line 149: “*the mask M is the maximum of all the masks*”. I suppose each constituent is being recognized independently, right? Then is it possible that the predicted masks don’t form a contiguous region, which would indicate a detection error? How would you plan to address this case?
- Line 230: “*The hierarchical concepts to be classified and detected are three characters which we term Concept1, Concept2, Concept3*”. These three concepts look like random patterns. How did you choose them in the first place?
- Does your approach exhibit any flexibility in recognizing concepts that have been slightly stretched or squeezed with the height-width ratio being altered? If such flexibility is allowed, it’d be great to visualize the performance vs. the degree of distortion.
- Line 249: “*During inference, its embedding for the graph structure can contain up to 10 hots, while during training, it is only up to 1-hot*”. Does the performance exhibit any degradation across inference examples from 1-hot up to 10-hot? If yes, how does the degradation look like versus #hots?
- Line 263: “*… viewed in 3D from a certain camera angle …*”. How is a 3D image input represented? Is it still a 2D matrix with 3 channels or is it represented as a 3D scene with depth?
- Line 263 again: I suppose the camera angle may distort a perpendicular relation so that it looks like an acute angle. And this distortion varies across camera angles. How did you address this and how did you choose camera angles when you constructed this dataset?
- Line 264: “*each test task consists of a tuple of three images*”. Why three images at a time? 
- This doesn’t have to answered thoroughly but I’d love to see your thoughts: How do you expect your approach to generalize to non-90-degree angles, like the top angle in “A”, or relations about arcs? 

limitations: Broader social impact and limitations are discussed in the supplementary. 

Here I briefly point out a few limitations and addressing any of these would make the paper stronger.
- The datasets used for evaluation cover limited number of concepts.
- The search process for a mask indicating “objectness” is inefficient.
- The universality of the set of primitive concepts and relations isn’t justified.
- The current algorithm is unable to evolve with dynamic expansion and compression as the number of concepts to model increases.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes a framework for zero-shot concept learning, i.e., learning and recognizing new concepts at inference time. The proposed frameworks extracts concepts and relations and composes them to learn new concepts hierarchically. The proposed method is evaluated on some synthetic datasets. 

strengths_and_weaknesses: Strength

(1) Zero-shot concept learning at inference time is an interesting problem and requires more attention.

(2) This work proposed a neat framework for learning concepts in a compositional way using energy based models and hierarchical composition. 

Weakness 

(1) There should be more discussion about how the proposed approach can scale to real world images that can include many edges and shapes with different relations. It seems this might become prohibitively costly. 

(2) The evaluation would be more convincing if it can also include results on some few-shot learning datasets such omniglot or, even better, some real world images.

questions: How would the basic relations and concepts be defined or discovered for real world images?
limitations: It seems non-trivial to apply the current approach successfully to real world images, which is the main limitation and the authors might want to have more discussion about it and future works to address remaining challenges.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: An EBM hierarchically describing concepts and relationships is used for concept discovery in a crowded scene with irrelevant concepts. This graph can be transferred across different domains.
strengths_and_weaknesses: Strengths:

Technically correct

- Addressing the relevant problem of hierarchical concept representation
- Clarifying examples
- Introduces a new dataset to test these ideas

Weaknesses:

- The size of the problem makes it too simple to brute force
- Unclear whether inference would scale
- Unclear how to learn arbitrary concepts from data
- Unclear how learning would scale

questions: 1. Even though it is claimed that the algorithm can learn new concepts, isn't the structure of the energy model used in this paper already tuned to the discovery of spatial relationships between horizontal and vertical lines? Or if I run the same algorithm on data with arbitrary concepts (for instance, triangle, circle, object A inside objet B, object is a closed contour, etc.) will it be able to discover those as well?

2. How does inference scale if I increase the resolution to 256 x 256? Would it work? For the size and structure of your dataset, isn't the partition function of your energy computable (albeit expensive)?

3. Same question about learning

4. Isn't the particular way in which you encode spatial relationships (relative) mean that this model cannot distinguish the concept of W and M, or 6 and 9, since they are both rotations of each other, and therefore satisfy the same relative relationships?

5. This model resembles heavily the ones in [1] and [2], which also describe letters as graphs of lateral relationships that entangle nodes containing edges. What are the main differences? Can this model be used to solve CAPTCHAs? Experiments showing this would definitely be much more convincing as to its capabilities.

6. I wasn't able to understand precisely which information is conveyed from ZeroC1 to ZeroC2. Could you clarify this section in the paper?

7. When using the loss from [12], you mention that you neglect the entropy term. What's the problem with keeping it? Would the results from [12] improve had they neglected it?

Minor: Figure 4 in the appendix, training relation, first column, doesn't look like a perp-edge.


[1] A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs. Dileep George et al. Science 2017

[2] Generative Shape Models: Joint Text Recognition and Segmentation with Very Little Training Data. Xinghua Lou. NIPS 2016.
limitations: Described above
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper addresses the problem of zero-shot concept recognition. By representing concepts with a graph, and by exploiting an energy-based model, the proposed approach enables detection and classification of novel entities at inference time. The approach can be considered "neuro-symbolic" as it combines symbolic graph-representation with (neural) energy-based models with a one-to-one translation of one into the other.

strengths_and_weaknesses: + The addressed problem is challenging and important.
+ The proposed solution is sound and elegant.
+ The experimental evaluation, though on a single dataset, is well conducted.

- Generalization to multiple datasets and scenarios is not straightforward.
- No discussion on computational complexity.

questions: * There is no description of the computational cost of the proposed approach. For example, Equation 2, although it is computed via MAP estimation, and although it needs only to find the concept with the highest value in the numerator, appears to be costly. Could the authors comment on this point?

limitations: The experimental evaluation is conducted on a single artificial dataset. While this is comprehensible for a novel task, it is not clear to what extent the proposed approach is tailored on the domain of images (e.g., see also Algorithm 1) or whether it could be easily genaralized to other domains.

Also, a discussion of the computational cost of the proposed approach is missing.

Typos:
- Line 29, "which requires" -> "which require"
- Line 71, "that can given" -> "that given"
- Line 126, "to determine if" -> "to determine whether"
- Line 134, "a English" -> "an English"
review_title: null

Meta-review: metareview: The focus of this work is on the introduction of a compositional reasoning model that enables zero-shot generalization. While there are a number of limitations (e.g. the small domain, limited concepts) but reviewers were content that the demonstrated results on low-resolution image domains proved the approach can scale to more realistic task complexity. The primary open challenge for richer tasks is the identification and training of elementary concepts -- a classification that may not hold.
confidence: Certain
Decision: Accept