PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning
Authors: Yining Hong
Li Yi
Joshua B. Tenenbaum
Antonio Torralba
Chuang Gan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Part
visual reasoning
relational reasoning
concept learning
model diagnostics

Abstract: A critical aspect of human visual perception is the ability to parse visual scenes into individual objects and further into object parts, forming part-whole hierarchies. Such composite structures could induce a rich set of semantic concepts and relations, thus playing an important role in the interpretation and organization of visual signals as well as for the generalization of visual perception and reasoning. However, existing visual reasoning benchmarks mostly focus on objects rather than parts. Visual reasoning based on the full part-whole hierarchy is much more challenging than object-centric reasoning due to finer-grained concepts, richer geometry relations, and more complex physics. Therefore, to better serve for part-based conceptual, relational and physical reasoning, we introduce a new large-scale diagnostic visual reasoning dataset named PTR. PTR contains around 80k RGBD synthetic images with ground truth object and part level annotations regarding semantic instance segmentation, color attributes, spatial and geometric relationships, and certain physical properties such as stability. These images are paired with 800k machine-generated questions covering various types of reasoning types, making them a good testbed for visual reasoning models. We examine several state-of-the-art visual reasoning models on this dataset and observe that they still make many surprising mistakes in situations where humans can easily infer the correct answer. We believe this dataset will open up new opportunities for part-based reasoning. PTR dataset and baseline models are publicly available. 

PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning
Yining Hong UCLA
Li Yi Stanford University
Joshua B. Tenenbaum MIT BCS, CBMM, CSAIL
Antonio Torralba MIT CSAIL
Chuang Gan MIT-IBM Watson AI Lab
Abstract
A critical aspect of human visual perception is the ability to parse visual scenes into individual objects and further into object parts, forming part-whole hierar- chies. Such composite structures could induce rich set of semantic concepts and relations, thus playing an important role in the interpretation and organiza- tion of visual signals well as for the generalization of visual perception and reasoning. However, existing visual reasoning benchmarks mostly focus on ob- jects rather than parts. Visual reasoning based the full part-whole hierarchy is much more challenging than object-centric reasoning due to finer meer-grained concepts, richer geometry relations, and more complex physics. Therefore, better serve for part-based conceptual, relational and physical reasoning, we introduce a new large-scale diagnostic visual reasoning dataset named PTR. PTR contains around 70k RGBD synthetic images with ground truth object and part level annotations regarding semantic instance segmentation, color attributes, spatial and geometric relationships, and certain physical properties such as stability. These images are paired with 700k pachine--gneratee questions covering various types reasoning types, making them good testbed for visual reasoning models. We examine several state-of-the-art visual reasoning models on this dataset and observe that they still make many surprising mistakes in situations where humans can easily infer the correct answer. We believe this dataset will open up new opportunities for part-based reasoning. PTR dataset and baseline models are publicly available 2.
Introduction
long-standing challenge in the field of artificial intelligence is enable machines answer questions about visual scenes. Several datasets [64, 16, 29, 49, 57] have been proposed to tackle this challenge. They mostly focus on object-level features without emphasizing much on detailed part-level understanding. However, there is strong psychological evidence that human beings parse visual scenes into part- whrle hierarchies (e.g., from a scene the objects, from an object its parts), which are currently missing from existing datasets.
and
Inspired by Aristotle's 's quote "the whole is greater than the of its parts", there has been a long history of research in psychology concerning how parts and wholes are related, beginning with the Gestalt psychologists [47]. There have also been recent efforts on how to represent such part-whole hierarchies using neural networks [19]. Introducing part-whole hierarchies into visual reasoning
'The work was done while Yining Hong was a research intern at MIT-IBM Watson Al Lab.
Project page: http://ptr... http:/.ttr..a..l.ppeeee..e..
35th Conference on Neural Information Processing Systems (NcurIPS 2021), virtual.

---

budy
cart
witee
QI (Concept); How many objects with purple logs are thene? Al:
(htut: bedckchair)
Teg
Q2 (Arithmetic)) What the zum of the mumber of logs in chuir. (hint: 5+-277)
top
and number wheels the car?? A2:
seens
Chair
support
Q3 (Physies) mune objects are stable?
table
top
pedestal
(bint: only the chair unstable) Q4 Towwrrs Towards whick direction shall the cort move became stable? A4; fromt
Ginwer
bed
(hint: make the contect between the and the tubie larger)
dleap
QI (Same Attribute): y: Is the of chair and shell the tuble the same color? Al: Yes
bodyy
(hint: Both are yellowe)
II
nefrigeratin
door
Q2 (Geometryy: What's the color of the part in the chairr that be considered line, and is perpendicular the purple part of the refiigerator? A2: bloe
Acone
chair
loy
(bint: the blue leg burs of the chair perpendicular to the dours)
Q3 QB(Analogy-): The blue part of the chair has certain geometric
table
relation to purple part the refrigerator. By amalogy.
part
of the sublee has the same relation par of what color in the
choir?A3: bruwn
(hint: the blue leg bur und the purple dours have relation; lime plan-pllnc
chulii
perpendicutar)
Funetional Program of Question "How many objects with purple legy shere?
Object Categocy
Part Category
Expand
Filter Part
Filter Part
Filter Part
Count Object
Objects
Parts
Category
Color
Exist
Purt Colur
Physics
Parts
log
purple
Objects
Geometry
Figure 1: Exemplar scene graphs, questions, and programs our Part Reasoning (PTR) dataset. PTR contains five question types: concept, relation, analogy, arithmetic and physics. Top left shows scenes from our dataset, paired with hierarchical scene graphs in the middle. Top right presents questions, answers and hints to the answers. A functional program of question is shown in the bottom.
brings two unique challenges. The first is how discriminate objects using part-level attributes. Unlike previous works which refer to objects based on holistic attributes such as object category, the specific ontology nature of objects and their parts requires detailed study. The second is how to leverage part-level attributes to interconnect objects. Objects of various categories are interrelated via some common parts (e.g., leg. central support, pedestal). The reasoning performed on one category should thus be easily generalized to reasoning beings are endowed with such modular but interconnected perceptual systems. They can effortlessly discriminate between the two tables in Figure [I] based on the differences in the tops and drawers, as well as observe a connection between the bed and chair due to the similarity of legs. remains to be explored whether machines have the same hierarchical perceptual capabilities.
unseen categories with shared parts. Human
Based on the highly composite part-whole hicrarchies, visual reasoning tasks can be richer, more complex, well as more challenging. First, the introduction of parts enriches the diversity of both visual perception and question understanding. Apart from basic object-level properties such as color and category, the visual scenes and natural language questions extend upon the properties of object parts, the composition of which makes the -qage--uuettin pairs more distinctive. Second, the relations between parts go beyond simple spatial relations. Parts can often be approximated as oriented geometric primitives (e.g., line, plane), and there naturally exist abundant geometric relations between these primitives such "perpendicular" and "parallel" (e.g., Q2 in Figure I[I]). Analogical reasoning can also be established given such relationships. Third, reasoning on parts enables the understanding of implicit properties of objects, such as physics. For example, the arrangements of the parts and their geometric relations affect the stability of the objects (e.g.. Q3, Q4 in Figure [[]).
In this paper, we present large-scale ParT Reasoning dataset, or PTR for short, benchmark for part- based conceptual, relational and physical reasoning. includes 70k scenes paired with 700k questions containing whote structures. We include over 10k objects from the PartNet [45] dataset across five object categories (chair, table, bed, refrigerator, cart) with rich geometric and structural variations to construct our scenes. Our dataset consists of five types of questions: concept, geometry, analogy, arithmetic, and physics. We provide scene graph annotations including the ground-truth locations, segmentations, and properties for all objects and parts, well all the object-object

---

Dataset
3D
Objects
Parts
Relation
Diagnostic
Geometry
Analogy
Physics
VQA [3]
X
X
VCR [61]
X
X
GQA [28]
X
X
CLEVR [29]
X
X
RAVEN [62]
X
/
/
X
PTR (ours)
/
/
Table 1: Comparison between PTR and other visual reasoning benchmarks.
and part-part relationships for model diagnostic. We also provide functional programs paired with questions.
We analyze a suite state-of-the- visual reasoning models on the PTR dataset and find that they all struggle with it, especially in relational, analogical, and physical reasoning. One oracle neural symbolic model performs better than other models but highly relies on additional supervision such as object and part masks and visual attributes from simulations. Also, the performances of all the models are inferior to human performance by a large margin, suggesting that there's 's still a long way go to equip machines with human-like hierarchical perceptual and reasoning abilities.
2 Related Work
2.1 Visual Reasoning
To assess machines' ability understanding and reasoning over vision and language, a wide variety of benchmarks have been created over the recent years [29, 3, 28, 16, 61, 12, 64, 36]. In of data input, our dataset mostly resembles the CLEVR dataset [29]. which is also a diagnostic dataset of synthetic images, template-based questions and compositional programs. However, the objects in CLEVR's scenes are of simple shapes and do not contain parts to form complex logical chains of reasoning. The VQA dataset [3] contains large-scale crowd-sourced real images and numan-generater questions. Since it's not fully-controlled synthetic dataset, it shows great inherent bias. Visual Commonsense Reasoning (VCR) [61] focuses on reasoning on commonsense knowledge. The GQA dataset [28] pairs real images with synthetic compositional questions. However, none of these datasets provide part-based dense annotations and questions. RAVEN [62] associates vision with structural, relational, and analogical reasoning in hierarchical representation. The dataset asks models to choose an image according to analogy. However, their images are rather simple, mainly consisting of 2D shapes like circles and triangles, and they do not pair images with natural language questions.
Although numerous visual reasoning datasets have been proposed, the reasoning chain typically terminates at the object level for these datasets, due to the mere reference to objects based on holistic features or spatial relationships. Also, while previous datasets emphasize reasoning based on spatial relationships, humans possess diverse reasoning abilities, including analogy [62], geometry[45 40), arithmetic(63, 20, 21], and physics [13]. These aspects can effortlessly incorporated into visual reasoning to constitute a benchmark that further approaches humans' intellectual level. Thus, we create a new reauonnng dataset that focuses on multiple aspects of reasoning on phrt- relations. We compare our dataset with previous visual reasoning datasets in Table 1.
large number of visual reasoning models have been proposed. Specifically, MAC [27] combined multi-modal attention for compositional reasoning. LCGN [23] built contextualized representations for objects support relational reasoning. These methods, though embedded in neural networks, model the reasoning process implicitly. symbolic reasoning on the objects representations and language representations. Recently, methods based on transformers [33] have achieved remarkable performance on visual reasoning tasks by utilizing end-to-end object detector. In this paper, we conduct experiments and assess strengths and weaknesses of these bascline models on bast- visual reasoning.
methods [56, 42, 7, 38] explicitly perform
3

---

2.2 Part-based 3D Understanding
Understanding 3D parts has been a long-standing problem in computer vision and graphics. is important due to several reasons. Firstly, parts and their arrangements that describe the detailed object geometry and structure are helpful for object discrimination purposes [9. 1]. Secondly, parts usually action handles and their forms could reveal the object functionality [34, 43, 24], therefore crucial for interacting with objects. Thirdly, functionally related object parts usually share similar forms, making parts suitable for connecting different objects, facilitating knowledge transfer and promoting generalizable learning [17, 41]. Finally, parts are often commonly used for 3D generation and content editing purposes [11, 44, 46].
To facilitate part-based 3D understanding, various part-centric 3D datasets have been proposed the literature. ShapeNet part dataset [58] annotates 31,963 shapes from ShapeNet [4] with semantic part segmentation covering 16 categories. And each shape is annotated with 2 to 5 parts. PartNet [45] further increases the granularity of parts, contributing 573,585 finegrained part instance annotations for 26,671 shapes across 24 object categories. Going beyond semantic understanding, several datasets also focus on the functionality and articulation aspects of parts [54, 43, 24, 25, 52]. However, most of these datasets target at part identification rather than part-based conceptual, relational and physical reasoning. Chang et al. [5] collected 27,477 part instances from 2,278 models covering 90 object categories, with a focus of studying conceptual part-based reasoning. ShapeGlot [1] presents an object reference dataset with part-level conceptual reasoning implicitly considered. Want et al. [51] proposes PartNer-Chairs, which contain questions about parts of chairs. These datasets are restricted to conceptual reasoning while our PTR has broader scope covering ampler reasoning types.
Restricted by the annotations in existing large-scale datasets, previous part-based 3D understanding works mainly focus on identifying parts [15, 30, 26, 55, 32, 58] and part relations [6, 10, 31, 35, 60, 50] for individual 3D objects. Several methods also consider a more complex hierarchical structure of 3D parts [59, 44, 37, 53, 14, 22, 8]. Moreover, these works usually assume strong categorical prior and tend to have poor generalizability across categories. However, we human beings can easily do more than just part identification or single-object part relationship understanding for objects from known categories. We can reason the relationship between parts from different object instances, make analogies, conduct mental arithmetic, and even infer the physical realism stability. These tasks are not well supported by current datasets. Therefore we present PTR to close this gap and to better support the study of these human-like common sense.
3 The PTR Dataset
The PTR dataset aims to assess machines' ability to perform part-based visual reasoning. We carefully design this dataset in fully-controlled synthetic environment and enable model diagnosties on this complex reasoning task. Each image has ground-trun -tuut annotations regarding the locations, orientations and attributes of objects and parts, well as the object masks and part masks. We also provide depth images, camera locations and orientations to facilitate the possible use of 3D models on our dataset in the future. For reasoning on geometric relations of parts, we include the geometric information of a part indicating whether it can be considered as a geometric line or plane, and the line equation or plane equation. For physical reasoning, object. The images are paired with five types of questions: concept, relation, analogy, arithmetic and physics. The questions have associated functional programs.
include the stability information of each
3.1 Image Generation
PTR includes approximately 52k images for training, 9k for validation and 10k for testing. The images are rendered via Blender". To get the physical information, we apply Bullet to simulate the future motion traces of each object.
Scene, Objects and Parts The PTR universe contains five categories of objects (chair, table, bed, refrigerator and cart) from the PartNet dataset. The reasons why choose these categories are as follows: 1) These are commonly seen objects in real life scenes; 2hThe objects share a lot of generic
hhtps://www.bendderrog// "bapss//pybbulletoggwoooorpsssss

---

70k different scenes
Objects: chair, table, bod, refrigerator, cart
Parts:
chuir: ann, leg, back, seat, centril support, pedestal, leg har, wheel, arm vertical har, amn horizontal bar
table: top, drawer, shelf, log, pedestal, log bar, door, contral support, wheel
bed: sleep
back
reffigerator: body, door
cart: body, wheel
Part Colors: gray, red, blue. green, brown, eyun, purple, and yellow
Sparial Relationship: left, right, front, behind Physicx: stable, unstable. possible positional change
Geometric Relutionship: fncelinee perpendicular, line-tine parallel. plane- perpendicular, plame- plane purallel, line-plane tlnepplmme perpendiculaar line plane purallel
Figure 2:
An overview of the scene structure and concepts the PTR dataset. The PTR universe contains five object categories, each consisting of a set of part categories. Objects have spatial relationships and parts have geometric relationships. Physical properties are also defined on objects.
parts (e.g., leg, leg bar, central support, pedestal, door, wheel, etc.); 3he objects have rich physical configurations depending on the constitution of the parts. Each object can have an arbitrary number of parts. Each part takes one of eight colors (gray, red, blue, green, brown, cyan, purple, and yellow). The specific concepts used in the PTR dataset are listed in Figure 2. The distribution of the number of parts in the scenes shown Figure 3c.
When constructing the scene, we first place a floor and three walls of random texture to enrich geometric and physical information. We add jitters to the locations and orientations of the camera and the lamps. Every scene has three to six objects. To place an object, we ensure that no objects overlap (except for physical scenes which have objects stacking). We also ensure that the objects are of proper orientations so that no much burden would be placed on object detection. The training dataset and test dataset has no shared 3D object models (i.e,, the PartNet 3D models are not shared across splits, but the semantics are shared).
The physical scenes contain objects that tilted, leaned towards walls, or stacked onto other objects. We use the Bullet physics engine to simulate physical effects. We calculate the change of object locations and orientations along time steps determine the stability of object. If the change of location and orientation is within a threshold value, an object can be considered stable, For unstable objects, we move the objects in four directions (front, behind, left, right) around their original location, to account for possible changes for the objects to become stable. To avoid missing such possible changes, for each direction we define four moving distances to place the objects.
Relationships PTR has three types of relationships: spatial relationships of objects, geometric relationships of parts, and same-attribute relationships of both objects and parts.
Objects are spatially related via four relationships (left, right, in front of, behind). The spatial relationships are dependent upon the camera viewpoint.
Parts have geometric relations if they, with negligible thickness, can be considered as ID lines or 2D planes (e.g., the legs and leg bars of Figure [II] can be treated as lines, and the doors can be treated as planes). To detect such geometric primitives. for each part we randomly sample two points for lines, or three points for planes from the raw 3D data, for 1000 times. Each time we derive a line equation or plane equation from the sampled points. We ensure that distances between the points are sufficient enough with regard to the size of the part to reduce noise. The final line plane equation then approximated by averaging the equations. We then further sample 2500 points from the 3D data to see if the majority of points conform to the equation. If not, the part cannot be considered as line or plane. The came-attribute relationships involve same-category relationship between objects, as well as same-category and same-color relationships between parts.
5

---

3.2 Question Generation
We pair each image with paachine-generate questions coupled with executable functional programs. PTR contains approximately 520k questions for training, 90k for validation and 100k for testing. All questions are open-ended and can be answered with a single word. Sample questions and detailed distributions can be found in Figure 1 and Figure
Concept Conceptual questions evaluate a model's capability to understand and reason about basic part-whole relations. The reasoning tasks grounded within the compositional space of both object-level and part-level properties. Unlike previous visual reasoning datasets which refer to an object based on the object's holistic attributes, PTR distinguishes objects by the existence of parts and the attributes of parts. To shed light on the interconnected nature of part-whole hierarchies, there are also questions that do not specify any object category but only the common part categories (e.g., QI in Figure [I] makes a connection between bed and chair by asking about the objects with legs). Conceptual questions contain multiple sub-types including: query object category, query part category, query part color, count object, exist object and count part.
Relation Relational questions query about the spatial relationships of objects, geometric relationships of parts, as well as same attribute relationships of objects and parts. Upon querying about the spatial relationships, the questions take the six sub-types of the conceptual questions, except that objects are also filtered using spatial information.
As for geometric relationships, we first detect part that can be considered as a line plane, refer to the part using color, and then query the existence/count/ollo of parts in another object that has certain geometric relation to this part. There are in total six types of geometric relationships, which are listed in Figure 2. Q2 in Figure [II] is an example of geometric questions. By considering the blue leg bars lines and the purple doors as planes, we can easily examine that the two parts perpendicular.
The same attribute relationships take three forms. The first one asks the model to compare two attributes of two objects/parts and return a "yes/no" answer (e.g... Q1 of Figure 1 [II]. The second refers to an object with the "same-category" relation to another object. The third detects whether an object has the same part attributes (e.g., same color of legs) as another object.
Analogy Analogical questions query analogy on spatial relationships and geometric relationships. Specifically, given objects or parts A, B. C where B has a certain relation A, the model is required to select an object or part D that has the same relation to C. The question may query about the sristence/coun/aattibuue of D. An exemplar question is shown in Figure [II] Q3. In this question, A is the leg bars of the chair, B is the door of the refrigerator. We find that the two parts have the "line-plane perpendicular" relationship. C is the legs of the table, and the question queries about the color of D, such that C and D have the same relation as A and B.
Arithmetic Arithmetic questions take the quantities of two types of parts in two objects, and apply two types of operations to the quantities. The first type is "compare integer". The model is asked to predict whether the two quantities follow "equal/greater than/less than" relationships and return a "yes/no" answer. The other type is "sum/minus"" ".. where the sum or the difference of two numbers is queried (e.g., Figure [1] Q2). We ensure that the answer is no greater than 10 or less than 0 to facilitate the training of neural models.
Physies Physical questions query about the stability of objects. For example, Q3 in Figure [I] asks how many objects are stable. If an object is unstable, further questions can be asked concerning the possible changes to the location of the object make it stable, as in Figure [T] Q4.
Program In PTR, each question can be parsed into a hicrarchical trec-structured functional program, as shown in the functional programs in Figure 1. Different from previous datasets, PTR has three key functions: expand parts, filter part exist and filter part count, which enable the flexible transition between object-level reasoning and part-level reasoning. When the reasoning procedure switches from object level to part level, expand parts takes all the objects, and returns the parts of the objects. filter part exist filters objects by examining whether certain parts exist in the objects, and filter part count filters objects by the number of certain parts in objects. A list of program modules can be found in the Supplementary Material.
Bias control Questions are generated from pre-defined templates, and transformed into natural language questions with associated object-level and levt- attributes from the scene. We manually

---

3303
16.4%
Concept
object
Number of Scane
2302
query object part
Relation
20.8%
1922
Analogy
23.8%
12.195
count
1002
Arithmetic
query part
803
47.4%
Physies
14,8%
sumy mminus
20
40
60
100
Number Parts Scene
(a) Question Semantics
(b) Question Reasoning Types
(c) Number of Parts
Figure Analysis on question types and distributions.
define 58 templates for question generation. We sort the question types and templates each time generate a question to ensure balanced question distribution. To avoid bias, force flat answer distribution for each question type by rejection sampling. Specifically, when sampling a question question-answer answer pair, if the count of the answer exceeds the median count of all answers in that type by a margin, we discard the question-ansee annweer pair.
4 Baseline Evaluations
4.1 Baseline Models
We mainly evaluate three kinds of state-of-the-art visual reasoning methods, including heuristic models, end-to-end neural networks, and neural-symbolie models.
Q-type (Rand.), Q-type (Freq.): These are language-only baselines that examine the bias in the dataset. Q-type (Rand.) uniformly samples an answer from the answer set given specific question type. Q-type (Freq.)) selects the most frequent training set answer for each question type.
UP-DOWN: We extract part-centric features in an unsupervised manner using Slot Attention [39], and apply bottom-up and top-down attention [2] on the part-centric features.
CNN+LSTM The question is encoded by the final hidden states from Long Short Term Memory network (LSTM) and the image is encoded using features from convolutional network (CNN). The features are concatenated and fed to an MLP to predict the final answer. This is a simple baseline that examines how vanilla neural networks perform on PTR.
MAC, MAC(P) ([27)) MAC utilizes a novel Memory, Attention and Composition cell to perform iterative reasoning process. The image features and question features are incorporated via joint attention mechanism. This a strong baseline achieving state-of-the-art performance on CLEVR without any supervision on vision and language. To boost part-level reasoning, we construct part- aware features by adding the segmentation masks of all parts and their attributes to the original features, denoted MAC(P).
LCGN ([33)) The objects are represented as nodes in the graph network, and are described by context-aware representation from related objects through iterative message passing conditioned on the textual input.
MDETR ([33)) This model fuses the two modalities at an early stage, leveraging an end-to-end modular detector boost downstream tasks like visual question answering. It uses ground-truth bounding boxes of parts to train the detector.
NS-VQA ([56)) The visual recognition and language understanding performed by neural networks, which are passed to a symbolic execution module to get the final results. We treat this baseline as an oracle model where use the annotations of both vision (ground-truth masks) and language (ground-trutt) programs).
Implementation Details We use an preetraied ResNet- 101 to extract 14 x 14 X 1024 feature maps MAC, MAC(P) and LCGN. For CNN-LSTM, we use the 2048-dimensional feature from the last pooling layer. The setup of MDETR is the same as the original paper with ResNet- 101 as backbone. We first train only the task of part detection for 30 epochs, and then train the full PTR with question answering loss.

---

For NS-VQA, we use Mask R-CNN [18] to generate segmentation proposals of objects and parts, respectively. The Mask R-CNN trained on 20% of the training data annotated with ground-truth masks for 30,000 iterations. We do not include labels of categories and attributes when training segmentation. We extract the categories and attributes of objects and parts using attribute networks (ResNet-34). The extraction works in a top-down, bottom-up fashion. First, the object proposals are sent to attribute networks to propose object categories. They are then paired with contextual information (i.e., the original images), to produce pose and physics information. Second, the part proposals are augmented with top-down object proposals and categories and sent to the attribute networks, to predict the part categories and colors separately. We also output the probability of a part being a line or plane, and the orientation (parameters of the line/plane equation) of the geometric parts. Finally, in bottom-up fashion, if parts cover region that is not covered by object segmentations, we reconstruct objects by taking the summed area of the parts, and label the objects by taking into account the labels of parts. The program generator of NS- VS-VQA is trained 1,000 crestion-progrant pairs. The part segmentations and attributes are also used in MAC(P).
Model
UP-
C-
NS-
Question Type
Rand.
Freq.
DOWN
LSTM
LCGN
MAC
MAC(P)
MDETR
VQA
Human
query-ohject
20.0
26.5
71.8
73.3
76.4
76.8
80.5
79.8
96.7
95.5
exist-object
50.1
52.4
72.6
73.9
77.1
74.8
76.5
78.2
91.5
97.8
Concept
count-object
9.8
19.8
50.0
58.9
68.5
63.3
64.5
67.9
80.8
95.4
query-part
8.9
10.9
34.4
35.7
31.5
48.2
49.4
52.3
71.8
89.1
count-part
10.0
20.1
45.5
46.8
50.4
49.6
50.9
51.2
82.9
spatial
11.5
18.2
40.6
47.2
31.2
54.9
59.3
37.6
67.1
85.7
Relation
geometric
40.1
41.7
48.6
57.9
58.6
64.8
65.6
67.1
55.3
76.4
same-attribute
36.1
39.5
47.3
52.9
54.8
57.6
58.0
61.4
62.5
94.2
spatial
26.1
32.9
19.2
51.0
54.3
53.4
53.3
58.5
68.7
84.2
Analogy
geometric
12.1
26.7
22.2
26.7
23.3
23.3
30.0
34.3
32.5
73.7
compare-integer
49.7
51.7
69.7
74.0
74.5
75.0
75.3
72.0
67.1
83.3
Arithmetic
sum-minus
10.0
19.8
26.8
31.1
35.0
34.2
34.8
36.2
43.4
75.6
stabilily
37.9
43.5
50.5
56.3
60.7
59.2
61.5
38.0
74.0
Physics
possible-change
40.6
48.1
53.5
54.1
57.2
57.8
56.8
53.8
52.0
79.8
All
28.4
33.5
45.4
52.9
58.4
57.5
58.9
59.8
62.1
84.8
Supervision
A
A
A.M.S
A.M.T
M.S.P
Table 2:
Test accuracies on the PTR dataset. For the supervision types, "A" denotes the answers, "M" denotes the segmentation masks, "S" denotes the semantics of objects and parts, "T" denotes the spans of tokens associated with masks, and "P" denotes the programs of the questions. Though humans achieve an 84.8% accuracy, SOTA visual reasoning models struggle with it.
4.2 Result Analysis
We summarize the performances for each question type of baseline models in Table 2. All models trained on the training set until convergence, tuned on the validation set, and evaluated on the test set.
Overall Result From the table, neural models by a large margin. This is credited to the extra supervision both vision (ground-tre -truth part masks and attributes) and language (ground-truth truth programs). Still, it cannot achieve nearly perfect performances in basic conceptual problems due to the difficulty in perception module (i.e., mask-renn and attribute net). MDETR stands out from all the neural models. However, two additional ingredients are required to train MDETR: the bounding boxes of all the parts, and the alignment between vision and language. We further observe that augmenting MAC with part segmentations and attributes boost the accuracy. The results suggest that currently, extra supervision is crucial for both neural models and neural-symbolic models to perform compositional reasoning. On the contrary, if the part segmentations and representations are extracted in an unsupervised manner (e.g.. UP-DOWN), the results are much inferior the results of other baseline models. We also include the unsupervised segmentation results by Slot Attention [39] in the Supplementary Material. The results suggest that our benchmark can also be served as a nature testbed for unsupervised part detection.
see that the neural-symbolic model, NS-VQA outperforms
All baseline models have unsatisfactory performances compared to human evaluation, especially in question types that require specific reasoning skills, such as analogy, physics and arithmetic.
Question Types We observe variance of performances across the wide spectrum of reasoning tasks. Starting with physics problems, we find that although NS-VQA has great performances for conceptual
8

---

questions, its performance drops drastically when it comes to physical problems, worse than all neural baselines by 15% 20%. The same phenomenon can be observed in the geometry problems, where NS-VQA performs worse than most of the neural baselines. This indicates that the attribute nets current neural-symbolic models can only learn explicit attributes while struggling with implicit high-level attributes such as physics and geometry. MAC(P) is superior to MAC in "stability" problems, indicating that the physical properties objects are linked to the composition of parts.
As for tasks that focus on part-level reasoning, models augmented with part-level representations (MAC(P), MDETR, NS-VQA) have better performances than others. Specifically, MDETR excels in geometric problems, and NS VQA manages achieve good performances for geometric analogy problems while other models fail. MAC(P) outperforms MAC in query-part, count- part and geometric questions, showing that part-based representations are essential for reasoning on part-level attributes and relationships.
4.3
Data Efficiency
CNN+LSTM
LCGN
MAC
MAC(P)
MDETR
NS-VQA
Ovemil
Conceptual
Relational
Physics
80
70
Acc (15)
Apc (%5)
Acc (1))
Ace
60
50
50
50
40
40
40
40
25 Proportion Dataset
100
25
100
25
100
25
100
Propotion Dataset
Propection Dataset
Proportion Dataset
Figure 4: Compared results on data efficiency of different baseline models.
We compare different baselines systematic study on data efficiency, which is presented in Figure Specifically, we train the model with 10%, 25%, 50% and 100% randomly chosen training images paired with questions, and evaluate the model on the original testing dataset. For NS- VQA, we use the corresponding percentage annotated data for part detection and question parsing.
From Figure 4, we find that the neuro-symbolic model remains good performances across all splits. This might be the consequence of the disentanglement of perception module and reasoning module. LCGN is data-efficient achieving ~ 49% accuracy with only 10% data, but does not show large improvement with more data. This might be due the context-aware representations which provide more information with fewer data. All the models improve slightly on the physics category, which suggests that the underperformance on physics problems is rarely caused by insufficient data. Increasing the amount of data boosts the performances in the conceptual questions most, indicating that data is important for learning part-whole concepts.
4.4 Cross-Category Generalization
To examine machines' ability of interconnecting objects of different categories via common parts (i.e., generalizing to unseen categories based on seen categories), we curate a subset from the original dataset. The curated training set has only three object categories: chair, refrig- erator, and cart. The testing set contains objects of all five categories. and questions asking about the common parts of the categories (e.g... both the table and the refrigerator have doors). The cross-category training set has about 15% data of the original training set. For fair comparison, we extract approximately the same amount of data in the training set that have all five categories in the scenes, denoted as "original" in Figure 5.
LOON
MAC
MOETR @Oigieal
NS-VQA
Figure Results on cross- category generalization.

---

Figure summarizes the croesocategory generalization results. From the figure, we can see that NS- VQA shows great generalization ability, mainly because the part detection and part attribute extraction trained on one category can be easily transferred to another with similar part appearances. Of the neural models, MDETR has the best performance, which is largely due to modular detection. LCGN also achieves satisfying performance. However, the cross-category accuracy for LCGN is much lower than the original accuracy, suggesting that the good performance is due to data efficiency. MAC(P) has the same problem. Although augment image features with part segmentations, does not show good generalization ability given by parts. One explanation is that choosing the right form of part features is crucial. Explicit mechanisms in MDETR and NS-VQA can lead to better performances and generalization ability.
4.5 Ablative Studies of NS-VQA
We further eliminate perception difficulty, and investigate the challenges of our benchmark if ground- truth part-segmentations are given. We experiment with two settings: 1) Replacing masks output by Mask-RCNN with ground-truth masks (instance segmentations given); 2) Based on 1), replacing attributes output by attribute networks with ground-truth attributes (both instance and semantic segmentations given). The results are shown Figure 6.
50
Concept
Relation
Analogy
Arithmetic
Plysics
NS. NS-VQA /wo segmentation (wo semantics NS-VQA segmenttttion wol sermantces
NS. VQA segmnentation wisemantics
Figure 6: Ablative Studies of NS- VQA model
We can see that given all the ground-truth segmentations and semantics, NS-VQL can achieve perfect results in conceptual and arithmetic problems. However, still performs poorly geometric and physical problems. Given only segmentations, NS-VQA has much lower accuracy than NS-VQA with semantics, showing that existing attribute network architectures also fall short in predicting semantics on our dataset.
5 Conclusion and Future Work
In this paper, we propose PTR, novel large-scale benchmark that emphasizes visual reasoning on part-whole hierarchies. We propose several challenging question types for PTR, including: concept, relation, analogy, arithmetic and physics. We describe the detailed dataset creation procedure. We further conduct experiments on state-of-the-ar visual reasoning models on PTR. Experimental results show that current visual reasoning models underperform humans in part-based reasoning tasks by large margin. We believe this benchmark will open up opportunities for building and testing a new family of visual reasoning models.
For future work, we can leverage the depth image in PTR and experiment with RGBD-based visual perception models. Also, results show that current models fail to represent whote hierarchies delicately, encouraging the design of novel neural networks that specializes this aspect (e.g.. Hinton [19] proposes some feasible ideas.) Last but not least, all models have bad performances in physical reasoning. One promising direction integrate physics engines with visual reasoning models to boost performances.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by MIT-IBM Watson Al Lab and its member company Nexplore, ONR MURI 1000014-13-1-33333 DARPA Machine Common Sense program, DSTA, ONR (N00014-18- 1-2847) and Mitsubishi Electric.
References
[1] Panos Achlioptas, Judy Fan, Robert Hawkins, Noah Goodman, and Leonidas Guibas. Shape- glot: Learning language for shape differentiation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 8938-8947, 2019.
[2] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6077-6086, 2018.
[3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitmick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.
[4] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint / 2015.
[5] Angel X Chang, Rishi Mago, Pranav Krishna, Manolis Savva, and Christiane Fellbaum. Linking wordnet to 3d shapes. In Global WordNet Conference, 2018. 4
[6] Siddhartha Chaudhuri, Evangelos Kalogerakis, Leonidas Guibas, and Vladlen Koltun. Proba- bilistic reasoning for assembly-based 3d modeling. In ACM SIGGRAPH 2011 papers, pages
1-10. 2011.
[7] Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan- Wea Kenneth Wong, Joshua B Tenenbaum, and Chuang Gan. Grounding physical concepts of objects and events through dynamic visual reasoning. ICLR, 2021. 3
[8] Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn. Generative scene graph networks. In International Conference on Learning Representations, 2021. URL /tttps://openrviiiie net/forunt dd-Rncmmmmttt...... 4
[9] Pedro Felzenszwalb, Ross Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part-based models. IEEE transactions on pattern analysis and machine intelligence, 320)::enneee332))113223:): 3209): 2009.
[10] Noa Fish, Melinos Averkiou, Oliver Van Kaick, Olga Sorkine-Hornung, Daniel Cohen-Or, and Niloy J Mitra. Meta-represennation of shape families. ACM Transactions Graphics (TOG),
33(4):1-11, 2014.4
[11] Thomas Funkhouser, Michael Kazhdan. Philip Shilane, Patrick Min, William Kiefer, Ayellet Tal, Szymon Rusinkiewicz, and David Dobkin. Modeling by example. ACM transactions on graphics (TOG), 23(3):652-66) 2004.
[12] Chuang Gan, Yandong Li, Haoxiang Li. Chen Sun, and Boqing Gong, VQS: Linking segmenta- tions to questions and answers for supervised attention in vqa and question-focused semantic segmentation. In ICCV, pages 181 18100, 2017.
[13] Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, et al. Threedworld: A platform for interactive multi-modal physical simulation. NeurlPS, 2021. 3
[14] Lin Gao, Jie Yang, Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, and Hao Zhang. Sdm-net: Deep generative network for structured deformable mesh. ACM Transactions on Graphics (TOG), 38(6):1-15, 2019.

---

[15] Aleksey Golovinskiy and Thomas Funkhouser. Consistent segmentation of 3d models. Comput- ers Graphics, 33(3):262 2009. 4
[16] Yash Goyal, Tejas Khot, Douglas Summers- Stane Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6325-6334 6334,, 2017. 3,3
[17] Songlang Han, Jiayuan Gu, Kaichun Mo, Li Yi, Siyu Hu, Xuejin Chen, and Hao Su. Com- positionally generalizable 3d structure prediction. arXiv preprint arXiv:2012.02493; 2020.
4
[18] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask r-cnn. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2980-2988, 2017.
[19] Geoffrey Hinton. How to represent part-whole hierarchies in neural network. arXiv preprint arXiv:2102. /2627, 2021. 1, 10
[20] Yining Hong. Qing Li, Daniel Ciao, Siyuan Huang, and Song-Chun. Zhu. Learning by fixing: Solving math word problems with weak supervision. In Thirty- AAAI Conference Artificial Intelligence, 2021.
[21] Yining Hong. Qing Li, Ran Gong, Daniel Ciao, Siyuan Huang, and Song- Chun. Zhu Smart: A situation model for algebra story problems via attributed grammar. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI-21, 2021." 3
[22] Yining Hong, Qing Li, Song-Chun Zhu, and Siyuan Huang. Vigrammar: Grounded grammar induction of vision and language. ICCV, 2021. 4
[23] Ronghang Hu, Anna Rohrbach, Trevor Darrell, and Kate Saenko. Language-con coniuugge-ondiionneeddd graph networks for relational reasoning. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 10293-10302, 2019. 3,7
[24] Ruizhen Hu, Oliver van Kaick, Bojian Wu, Hui Huang, Ariel Shamir, and Hao Zhang. Learning how objects function via co-analysis of interactions. ACM Transactions on Graphics (TOG), 35 (4):1-13, 2016. 4
[25] Ruizhen Hu, Wenchao Li, Oliver Van Kaick, Ariel Shamir, Hao Zhang, and Hui Huang. Learning predict part mobility from a single static snapshot. ACM Transactions on Graphics (TOG), 36(6):1-33, 2017.4 4
[26] Qixing Huang, Vladlen Koltun, and Leonidas Guibas. Joint shape segmentation with linear programming. In Proceedings of the 2011 SIGGRAPH Asia Conference, pages 1-12, 2011. 4
[27] D. A. Hudson and Christopher D. Manning. Compositional attention networks for machine reasoning. ICLR, 2018. 3.7
[28] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
[29] J. Johnson, Bharath Haribaran, L. V. D. Maaten, Li Fei-Fei, C. L. Zitmick, and Ross B. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 2017. 1,3
[30] Evangelos Kalogerakis, Aaron Hertzmann, and Karan Singh. Learning 3d mesh segmentation and labeling. In ACM SIGGRAPH 2010 papers, pages 1-12. 2010. 4
[31] Evangelos Kalogerakis, Siddhartha Chaudhuri, Daphne Koller, and Vladlen Koltun. A proba- bilistic model for component- shape synthesis. ACM Transactions on Graphics (TOG), 31 (4):1-11, 2012. 4
[32] Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri. 3d shape segmentation with projective convolutional networks. proceedings of the IEEE conference computer vision and pattern recognition, pages 3779-3788, 2017. 4
12

---

[33] Aishwarya Kamath, Mannat Singh. Y. LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. Mdetr modulated detection for end-to-end multi-mode understanding. ArXiv, abs/2104.12763. 2021. 3,7
[34] Dov Katz and Oliver Brock. Manipulating articulated objects with interactive perception. In
2008 IEEE International Conference on Robotics and Automation, pages 272-277. IEBE, 2008.
4
[35] Vladimir G Kim, Wilmot Li, Niloy J Mitra, Siddhartha Chaudhuri, Stephen DiVerdi, and Thomas Funkhouser. Learning part-based templates from large collections of 3d shapes. ACM Transactions on Graphics (TOG), 32(4): 2013.
[36] R. Krishna, Yuke Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, Stephanie Chen, Yannis Kalantidis, L. Li, D. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123:32-73, 2016. 3
[37] Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. Grass: Generative recursive autoencoders for shape structures. ACM Transactions Graphics (TOG), 36(4):1-14, 2017.
[38] Qing Li, Siyuan Huang. Yining Hong, Yixin Chen, Ying Nian Wu, and Song-Chun. Zhu. Closed loop neural-symbolic learning via integrating neural perception, grammar parsing, and symbolic reasoning. In International Conference on Machine Learning (ICML), 2020. 3
[39] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention, 2020. 7,8, 7,8,1 19
[40] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Chun Zhu. inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In The 59th Annual Meeting of the Association for Computational Linguistics (ACL),
2021. 3
[41] Tiange Luo, Kaichun Mo, Zhiao Huang, Jiarui Xu, Siyu Hu, Liwei Wang, and Hao Su. Learning to group: a bottom-up framework for 3d part discovery in unseen categories. arXiv preprint arXiv:2002.06478, 2020. 4
[42] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. In International Conference on Learning Representations, 2019. URL https: //openreview. mett/orrmmidde J.MihrrJghhhhttt 3
[43] Roberto Martnn-Martn Clemens Eppner, and Oliver Brock. The rbo dataset of articulated objects and interactions. The International Journal of Robotics Research, 1013 2019. 4
[44] Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas J Guibas. Structurenet: Hierarchical graph networks for 3d shape generation. Siggraph Asia, 2019. 4
[45] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 909-918, 88, 2019. 2,
[46] Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas J Guibas. Structedit: Learning structural shape variations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8859-8868, 2020. 4
[47] O. Reiser and K. Koffka. Principles of gestalt psychology. The Philosophical Review, 45:412, 1936.
[48] Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, and Oren Etzioni. Diagram understanding in geometry questions. In AAAI, 2014.
13

---

[49] Alane Suhr, Stephanie Zhou, Iris Zhang, Huajun Bai. and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. ACL, 2019.
[50] Minhyuk Sung, Hao Su, Vladimir G Kim, Siddhartha Chaudhuri, and Leonidas Guibas. Com- plementme: Weakly-supervised component suggestions for 3d modeling. ACM Transactions on Graphics (TOG), 36(6):1-12 2017. 4
[51] Ruocheng Wang, Jiayuan Mao, Samuel J. Gershman, and Jiajun Wu. Language-mediated, object-centric representation learning. ACL (Findings), 2021. 4
[52] Xiaogang Wang. Bin Zhou, Yahao Shi. Xiaowu Chen, Qinping Zhao, and Kai Xu. Shape2motion: Joint analysis of motion parts and attributes from 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8876-8884, 2019.
[53] Zhijie Wu, Xiang Wang, Di Lin, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Sagnet: Structure-aware generative network for 3d-shape modeling, ACM Transactions on Graphics (TOG), 38(4):1-14, 2019.
[54] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang. Yifu Yuan, He Wang. al. Sapien: A simulated part-based interactive vironment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11097-11107, 2020. 4
[55] Zhige Xie, Kai Xu, Ligang Liu, and Yueshan Xiong. 3d shape segmentation and labeling via extreme learning machine. In Computer graphics forum, volume 33, pages 85-95. Wiley Online Library, 2014.
[56] Kexin Yi, Jiajun Wu, Chuang Gan, A. Torralba, Kohli, and J. Tenenbaum. Neural-symbolee vqa: Disentangling reasoning from vision and language understanding. In NeurlPS, 2018. 3, 7, 16
[57] Kexin Yi, Chuang Gan, Yunzhu Li, Kohli, Jiajun Wu, A. Torralba, and J. Tenenbaum. Clevrer: Collision events for video representation and reasoning. ICLR, 2020.
[58] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG), 35(6):1-12, 2016. 4
[59] Li Yi, Leonidas Guibas, Aaron Hertzmann, Vladimir G Kim. Hao Su, and Ersin Yumer. Learning hierarchical shape segmentation and labeling from online repositories. SIGGRAPH, 2017. 4
[60] Mehmet Ersin Yumer, Siddhartha Chaudhuri, Jessica K Hodgins, and Levent Burak Kara. Semantic shape editing using deformation handles. ACM Transactions on Graphics (TOG), 34 (4):1-12, 2015. 4
[61] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6720-6731, 2019. 3
[62] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu, Raven: dataset for relational and analogical visual reasoning. In CVPR, pages 5317-5327, 2019.
[63] D. Zhang, Lei Wang, Nuo Xu, B. Dai, and H. Shen. The gap of semantic parsing: A survey on automatic math word problem solvers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42:2287-2342 2020. 3
[64] Yuke Zhu, O. Groth, Michael S. Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4995-5004, 2016. 1,3
14


Reviews:
[{"review_id": "xTc1p4_OdBx", "reviewer": "Reviewer_jSyT", "summary": "This paper mainly proposed a new dataset named ParT Reasoning (PTR) dataset for reasoning the part of the objects in an image, which data is synthetic from the simulator rather than the real world. The dataset is a form of visual question answering that containing five types of questions, including concept, relation, analogy, arithmetic, and physics. The authors conducted several popular methods on their PTR dataset to validate the challenge and characteristics of the dataset.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. It is an interesting research topic for mining 3D objects by hierarchical part-based reasoning. \n2. The definition of question types is well-defined to cover six cognition-related aspects.\n2. The paper is well organized.\n3. Sufficient experiments to compare with other SOTA methods.\n\nWeakness:\nI wonder whether the hierarchical scene graphs to be used as shown in Figure 1(center) in the experiments. What do the effect of these scene graphs bring in the part-based conceptual, relational, and physical reasoning?\n\n2. In Figure 4 (Physics part), I observed the NS-VQA achieved the worst performance compared with others, which doesn't match the description in Line 321.\n\n3. In Line 348, the author claimed the MDETR has the best performance. In my opinion, the NS-VQA should be the best performer in Figure 5. Please correct me if I'm wrong.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The benchmark is based on the simulation, which exists a gap between the synthetic data and real-world data. \n\nThere is only one word for each answer.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "pQ2KuXKijK-", "reviewer": "Reviewer_XMc6", "summary": "This paper proposes a benchmark for part-based commonsense reasoning. The dataset consists of realistic-looking scenes with various objects in the scene. Several queries are designed to test a model's reasoning capability which includes concepts, relations, physics, analogy, and arithmetic. The author(s) run several baselines spanning from a random picking to an oracle model (NS-VQA) which utilizes ground truths for vision model and language programs.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) A strong aspect of the dataset is the \"analogical reasoning\" queries which is an essence of commonsense reasoning. This is an underexplored area in the literature and there hardly exists any realistic dataset that captures this aspect (RAVEN only covers this aspect to simple 2D shapes as the authors have pointed out).\n2) In general, the presentation of the paper is neat and easy to follow. The proposed parts-based dataset is significantly novel covering major aspects of commonsense reasoning. \n\n\nWeaknesses/questions:\n3) Related works:  GSGN [Ref1] seems to be closely related work. GSGN can extract the part-whole hierarchy in the form of a tree. The authors also propose a Compositional CLEVR dataset which although is less realistic as compared to PTR, but does showcase its performance on downstream tasks.\n\n4) All the baselines either use the complete image or the ground truth object representations. A baseline with self-supervised object-centric representations like [Ref2] can be a measure of where those works lie in the spectrum of fully supervised (oracle) to distributed representations (CNN+LSTM).\n\n5) The general trend of baseline models (including oracle NS-VQA) performing poorly in \"geometric type questions\" as compared to the human baseline is observed. Is this because of NOT considering 'depth' as input? (since the geometric relationships as shown in the example images (in supp. and main paper) mostly deal with figuring out the relation between lines and planes where depth plays a crucial role). I do not think an explicit experiment is required for this, but intuitive reasoning would suffice.\n\nSuggestions:\n\n6) For easy readability, it would be better to include the types of data each model in Table 2 takes for training (example: MDETR uses bounding boxes, LCGN is self-supervised, etc.)\n\nReferences:\n\n[Ref1] Generative Scene Graph Networks, Fei Deng et al, ICLR 2021.\n[Ref2] Language-Mediated, Object-Centric Representation Learning, Ruocheng Wang and Jiayuan Mao et al, ACL 2021.\n\n\nJustification for rating: I'm currently giving the paper a weak accept. If the author(s) are able to address the comments in the weaknesses section convincingly, I'm more than happy to increase my rating to an accept.\n\n\n------\nEdit (post-rebuttal): After carefully going over the author(s) rebuttal for *all* the reviewers, I have increased my score to 7. The author(s) have successfully addressed all the concerns I've had.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "gWyOFxMs0hH", "reviewer": "Reviewer_tbaK", "summary": "This paper proposed a new benchmark for visual reasoning on part-whole hierarchies, where the dataset contains 80k RGBD synthetic images with ground truth object and part level annotations. It contains five types of questions including, concept, relation, analogy, arithmetic, and physic. The dataset generation procedure is described in a detailed way. Several vanilla baselines, CNN-based baselines, and neuron symbolic baseline are included and compared.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper takes a step further on part-level and probably part-whole hierarchies understanding, where the reviewer believes it is a very important direction. It would help us to study the part-hierarchies in the context of 3D parts reasoning.\n\nThe dataset is generated via knowing 3D parts and pre-defined templates. If the templates are designed well, the image is supposed to be perfect aligned with corresponding questions as well as answers. Also, the 3D shapes and parts would do not have overlaps or incomplete, and thus provides full 3D information for tested algorithms and let them focus on the part reasoning aspects.\n\nThe most intriguing thing is studying physics and found that the neural symbolic baseline failed to beat the NNs which just memorized the data. Overall the paper is easy to follow.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The major concern of the reviewer is how much further effort do we need to answer the questions if we already know the part segmentations (instance & semantic)? For all types of questions except the physical type, it seems we can easily answer them based on part segmentations. For example, performing arithmetic is summing up semantically related parts, and performing analogy is comparing two specific parts. Will the proposed benchmark and dataset is highly dependent on the 3D segmentation backbone? If you replace the Mask RCNN with ground-truth annotations, what performance will NS-VQA achieve?\n\nThe reviewer is not sure about the quality of the design data. SCENE A: The topology of a scene is easy to tell if the shapes are separated from each other. SCENE B: But, it is hard to distinguish the topology of scenes if the shapes are superimposed with each other via stange postures. Also, such topology may be very rare and can not help predict other topologies. If the dataset contains a lot of easy SCENE A or SCENE B, it may be a little bit meaningless to perform part reasoning in a part-hierarchy way.\n\nFor the cross-domain section, the reviewer believes the generalisability lies in the shared parts for different shape categories as the author said in #346 - #347.  Therefore, a more salient way is probably training on a category and testing on a similar category and a dissimilar category? The reviewer is not sure about the similarity between refrig. and bed or cart and table, where currently your training categories contain refrig. and cart.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "W3CS6GonlC2", "reviewer": "Reviewer_Qvcs", "summary": "The paper introduces a novel benchmark for part-based reasoning, featuring 80k indoor scenes contraining pieces of furniture from the PartNet dataset. Scenes are annotated with object and part labels, as well as geometric relations. Based on these features, a dataset of questions are generated as a VQA benchmark. Several existing models are evaluated on this benchmark, and are found to underperform  human subjects, sometimes substantially.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Dataset\nThe general setup of the dataset is plausible and follows previous benchmarks such as CLEVR. The main additions are the physics simulator, the geomatric features, and the new question types. These all seem like useful additions.\n\nThe most questionable choice to me is to color all parts in bright primary colors: This makes any part segmentation task very easy, but can sometimes make it difficult even for humans to determine which part belongs to which object, as there is little visual continuity between parts belonging to the same object. Exploring a version of the dataset featuring natural textures might be a good idea.\n\n### Clarity\nThe text is generally clear and easy to follow. Some paragraphs, e.g. those on the conceptual importance of part based reasoning, appear longer than necessary. The discussion of the possible reasons for the performance of the evaluated methods remains rather speculative, as the methods are not modified or ablated to confirm the suspected reasons for their performance. The text mostly makes this clear, but e.g. line 322 appears very confident without clear evidence.\n\nMinor comments:\n - Hinton's \"Mapping part-whole hierarchies into connectionist networks\" [15] is cited in two odd places: Line 26 uses it as psychological evidence that humans parse visual scenes into part-whole hierarchies, even though the paper contains no original psychological research. Line 31 cites it as a recent effort to represent part-whole hierarchies using neural networks, even though it was published in 1990.\n - Line 75: It is not clear why the non-synthetic nature of VQA implies great bias.\n - Line 332 is broken.\n - No \"et al.\" should be used in line 365, as [16] is a single author paper.\n\n### Significance\nGiven that the paper only introduces a new dataset, and does not propose any novel algorithm or model, determining\nwhether this contribution is of sufficient significance will be key to the acceptance decision. It should be noted that the\nproposed dataset is composed entirely of existing freely available assets, and (aside from the human performance estimates) no\ncompletely novel data has been collected.\n\nSimilarly, the existing methods have been applied as-is, and no modifications to their internal structure have been proposed or attempted. As a result, I am not sure if much insight regarding their inner workings can be derived from the results.\n\n### Summary\nWhile the proposed dataset fills a reasonable niche, it features few entirely new ideas, and the analysis of existing methods remains rather shallow. Given this, my impression is that this work does not meet the significance bar for the main track of NeurIPS. It may be better suited to the datasets & benchmarks track.\n\n### Update\nAfter reading the other reviews and the authors' responses, I am increasing my score from a 5 to a 6. It has become clearer to me that both the creation of the dataset and adapting the existing methods to the part-based setting involved a considerable amount of original work. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
Thanks for submitting your work to NeurIPS. Proposing a new benchmark for visual reasoning on part-whole hierarchies is really an impressiv and important contribution. The dataset contains 80k RGBD synthetic images with ground truth object and part level annotations as well as informations about the concept, relation, analogy, arithmetic, and physic. Several vanilla baselines, CNN-based baselines, and neuro-symbolic baseline are included and compared. All reviewers agree that this is a very solid and strong contribution. I fully agree with this sentiment. Thanks for th enice work 
Decision: Accept (Poster)