Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a novel benchmark for part-based reasoning, featuring 80k indoor scenes contraining pieces of furniture from the PartNet dataset. Scenes are annotated with object and part labels, as well as geometric relations. Based on these features, a dataset of questions are generated as a VQA benchmark. Several existing models are evaluated on this benchmark, and are found to underperform  human subjects, sometimes substantially.
main_review: ### Dataset
The general setup of the dataset is plausible and follows previous benchmarks such as CLEVR. The main additions are the physics simulator, the geomatric features, and the new question types. These all seem like useful additions.

The most questionable choice to me is to color all parts in bright primary colors: This makes any part segmentation task very easy, but can sometimes make it difficult even for humans to determine which part belongs to which object, as there is little visual continuity between parts belonging to the same object. Exploring a version of the dataset featuring natural textures might be a good idea.

### Clarity
The text is generally clear and easy to follow. Some paragraphs, e.g. those on the conceptual importance of part based reasoning, appear longer than necessary. The discussion of the possible reasons for the performance of the evaluated methods remains rather speculative, as the methods are not modified or ablated to confirm the suspected reasons for their performance. The text mostly makes this clear, but e.g. line 322 appears very confident without clear evidence.

Minor comments:
 - Hinton's "Mapping part-whole hierarchies into connectionist networks" [15] is cited in two odd places: Line 26 uses it as psychological evidence that humans parse visual scenes into part-whole hierarchies, even though the paper contains no original psychological research. Line 31 cites it as a recent effort to represent part-whole hierarchies using neural networks, even though it was published in 1990.
 - Line 75: It is not clear why the non-synthetic nature of VQA implies great bias.
 - Line 332 is broken.
 - No "et al." should be used in line 365, as [16] is a single author paper.

### Significance
Given that the paper only introduces a new dataset, and does not propose any novel algorithm or model, determining
whether this contribution is of sufficient significance will be key to the acceptance decision. It should be noted that the
proposed dataset is composed entirely of existing freely available assets, and (aside from the human performance estimates) no
completely novel data has been collected.

Similarly, the existing methods have been applied as-is, and no modifications to their internal structure have been proposed or attempted. As a result, I am not sure if much insight regarding their inner workings can be derived from the results.

### Summary
While the proposed dataset fills a reasonable niche, it features few entirely new ideas, and the analysis of existing methods remains rather shallow. Given this, my impression is that this work does not meet the significance bar for the main track of NeurIPS. It may be better suited to the datasets & benchmarks track.

### Update
After reading the other reviews and the authors' responses, I am increasing my score from a 5 to a 6. It has become clearer to me that both the creation of the dataset and adapting the existing methods to the part-based setting involved a considerable amount of original work. 
limitations_and_societal_impact: Limitations and societal impact are adequately addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a new benchmark for visual reasoning on part-whole hierarchies, where the dataset contains 80k RGBD synthetic images with ground truth object and part level annotations. It contains five types of questions including, concept, relation, analogy, arithmetic, and physic. The dataset generation procedure is described in a detailed way. Several vanilla baselines, CNN-based baselines, and neuron symbolic baseline are included and compared.
main_review: This paper takes a step further on part-level and probably part-whole hierarchies understanding, where the reviewer believes it is a very important direction. It would help us to study the part-hierarchies in the context of 3D parts reasoning.

The dataset is generated via knowing 3D parts and pre-defined templates. If the templates are designed well, the image is supposed to be perfect aligned with corresponding questions as well as answers. Also, the 3D shapes and parts would do not have overlaps or incomplete, and thus provides full 3D information for tested algorithms and let them focus on the part reasoning aspects.

The most intriguing thing is studying physics and found that the neural symbolic baseline failed to beat the NNs which just memorized the data. Overall the paper is easy to follow.
limitations_and_societal_impact: The major concern of the reviewer is how much further effort do we need to answer the questions if we already know the part segmentations (instance & semantic)? For all types of questions except the physical type, it seems we can easily answer them based on part segmentations. For example, performing arithmetic is summing up semantically related parts, and performing analogy is comparing two specific parts. Will the proposed benchmark and dataset is highly dependent on the 3D segmentation backbone? If you replace the Mask RCNN with ground-truth annotations, what performance will NS-VQA achieve?

The reviewer is not sure about the quality of the design data. SCENE A: The topology of a scene is easy to tell if the shapes are separated from each other. SCENE B: But, it is hard to distinguish the topology of scenes if the shapes are superimposed with each other via stange postures. Also, such topology may be very rare and can not help predict other topologies. If the dataset contains a lot of easy SCENE A or SCENE B, it may be a little bit meaningless to perform part reasoning in a part-hierarchy way.

For the cross-domain section, the reviewer believes the generalisability lies in the shared parts for different shape categories as the author said in #346 - #347.  Therefore, a more salient way is probably training on a category and testing on a similar category and a dissimilar category? The reviewer is not sure about the similarity between refrig. and bed or cart and table, where currently your training categories contain refrig. and cart.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper mainly proposed a new dataset named ParT Reasoning (PTR) dataset for reasoning the part of the objects in an image, which data is synthetic from the simulator rather than the real world. The dataset is a form of visual question answering that containing five types of questions, including concept, relation, analogy, arithmetic, and physics. The authors conducted several popular methods on their PTR dataset to validate the challenge and characteristics of the dataset.
main_review: Strengths:
1. It is an interesting research topic for mining 3D objects by hierarchical part-based reasoning. 
2. The definition of question types is well-defined to cover six cognition-related aspects.
2. The paper is well organized.
3. Sufficient experiments to compare with other SOTA methods.

Weakness:
I wonder whether the hierarchical scene graphs to be used as shown in Figure 1(center) in the experiments. What do the effect of these scene graphs bring in the part-based conceptual, relational, and physical reasoning?

2. In Figure 4 (Physics part), I observed the NS-VQA achieved the worst performance compared with others, which doesn't match the description in Line 321.

3. In Line 348, the author claimed the MDETR has the best performance. In my opinion, the NS-VQA should be the best performer in Figure 5. Please correct me if I'm wrong.

limitations_and_societal_impact: The benchmark is based on the simulation, which exists a gap between the synthetic data and real-world data. 

There is only one word for each answer.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a benchmark for part-based commonsense reasoning. The dataset consists of realistic-looking scenes with various objects in the scene. Several queries are designed to test a model's reasoning capability which includes concepts, relations, physics, analogy, and arithmetic. The author(s) run several baselines spanning from a random picking to an oracle model (NS-VQA) which utilizes ground truths for vision model and language programs.
main_review: Strengths:
1) A strong aspect of the dataset is the "analogical reasoning" queries which is an essence of commonsense reasoning. This is an underexplored area in the literature and there hardly exists any realistic dataset that captures this aspect (RAVEN only covers this aspect to simple 2D shapes as the authors have pointed out).
2) In general, the presentation of the paper is neat and easy to follow. The proposed parts-based dataset is significantly novel covering major aspects of commonsense reasoning. 


Weaknesses/questions:
3) Related works:  GSGN [Ref1] seems to be closely related work. GSGN can extract the part-whole hierarchy in the form of a tree. The authors also propose a Compositional CLEVR dataset which although is less realistic as compared to PTR, but does showcase its performance on downstream tasks.

4) All the baselines either use the complete image or the ground truth object representations. A baseline with self-supervised object-centric representations like [Ref2] can be a measure of where those works lie in the spectrum of fully supervised (oracle) to distributed representations (CNN+LSTM).

5) The general trend of baseline models (including oracle NS-VQA) performing poorly in "geometric type questions" as compared to the human baseline is observed. Is this because of NOT considering 'depth' as input? (since the geometric relationships as shown in the example images (in supp. and main paper) mostly deal with figuring out the relation between lines and planes where depth plays a crucial role). I do not think an explicit experiment is required for this, but intuitive reasoning would suffice.

Suggestions:

6) For easy readability, it would be better to include the types of data each model in Table 2 takes for training (example: MDETR uses bounding boxes, LCGN is self-supervised, etc.)

References:

[Ref1] Generative Scene Graph Networks, Fei Deng et al, ICLR 2021.
[Ref2] Language-Mediated, Object-Centric Representation Learning, Ruocheng Wang and Jiayuan Mao et al, ACL 2021.


Justification for rating: I'm currently giving the paper a weak accept. If the author(s) are able to address the comments in the weaknesses section convincingly, I'm more than happy to increase my rating to an accept.


------
Edit (post-rebuttal): After carefully going over the author(s) rebuttal for *all* the reviewers, I have increased my score to 7. The author(s) have successfully addressed all the concerns I've had.
limitations_and_societal_impact: Not applicable
review_title: null

Meta-review: 
Thanks for submitting your work to NeurIPS. Proposing a new benchmark for visual reasoning on part-whole hierarchies is really an impressiv and important contribution. The dataset contains 80k RGBD synthetic images with ground truth object and part level annotations as well as informations about the concept, relation, analogy, arithmetic, and physic. Several vanilla baselines, CNN-based baselines, and neuro-symbolic baseline are included and compared. All reviewers agree that this is a very solid and strong contribution. I fully agree with this sentiment. Thanks for th enice work 
Decision: Accept (Poster)