PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning

1 Introduction . A long-standing challenge in the field of artificial intelligence is to enable machines to reason and answer questions about visual scenes . Several datasets [ 64 , 16 , 29 , 49 , 57 ] have been proposed to tackle this challenge . They mostly focus on object-level features without emphasizing much on detailed part-level understanding . However , there is strong psychological evidence that human beings parse visual scenes into part-whole hierarchies ( e.g. , from a scene to the objects , from an object to its parts ) , which are currently missing from existing datasets . Inspired by Aristotle ’ s quote “ the whole is greater than the sum of its parts ” , there has been a long history of research in psychology concerning how parts and wholes are related , beginning with the Gestalt psychologists [ 47 ] . There have also been recent efforts on how to represent such part-whole hierarchies using neural networks [ 19 ] . Introducing part-whole hierarchies into visual reasoning ∗The work was done while Yining Hong was a research intern at MIT-IBM Watson AI Lab . 2Project page : http : //ptr.csail.mit.edu/ 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . brings two unique challenges . The first is how to discriminate objects using part-level attributes . Unlike previous works which refer to objects based on holistic attributes such as object category , the specific ontology nature of objects and their parts requires a detailed study . The second is how to leverage part-level attributes to interconnect objects . Objects of various categories are interrelated via some common parts ( e.g. , leg , central support , pedestal ) . The reasoning performed on one category should thus be easily generalized to reasoning on unseen categories with shared parts . Human beings are endowed with such modular but interconnected perceptual systems . They can effortlessly discriminate between the two tables in Figure 1 [ I ] based on the differences in the tops and drawers , as well as observe a connection between the bed and chair due to the similarity of legs . It remains to be explored whether machines have the same hierarchical perceptual capabilities . Based on the highly composite part-whole hierarchies , visual reasoning tasks can be richer , more complex , as well as more challenging . First , the introduction of parts enriches the diversity of both visual perception and question understanding . Apart from basic object-level properties such as color and category , the visual scenes and natural language questions extend upon the properties of object parts , the composition of which makes the image-question pairs more distinctive . Second , the relations between parts go beyond simple spatial relations . Parts can often be approximated as oriented geometric primitives ( e.g. , line , plane ) , and there naturally exist abundant geometric relations between these primitives such as “ perpendicular '' and “ parallel '' ( e.g. , Q2 in Figure 1 [ II ] ) . Analogical reasoning can also be established given such relationships . Third , reasoning on parts enables the understanding of implicit properties of objects , such as physics . For example , the arrangements of the parts and their geometric relations affect the stability of the objects ( e.g. , Q3 , Q4 in Figure 1 [ I ] ) . In this paper , we present a large-scale ParT Reasoning dataset , or PTR for short , a benchmark for part-based conceptual , relational and physical reasoning . It includes ∼ 70k scenes paired with 700k questions containing part-whole structures . We include over 10k objects from the PartNet [ 45 ] dataset across five object categories ( chair , table , bed , refrigerator , cart ) with rich geometric and structural variations to construct our scenes . Our dataset consists of five types of questions : concept , geometry , analogy , arithmetic , and physics . We provide scene graph annotations including the ground-truth locations , segmentations , and properties for all objects and parts , as well as all the object-object and part-part relationships for model diagnostic . We also provide functional programs paired with questions . We analyze a suite of state-of-the-art visual reasoning models on the PTR dataset and find that they all struggle with it , especially in relational , analogical , and physical reasoning . One oracle neural symbolic model performs better than other models but highly relies on additional supervision such as object and part masks and visual attributes from simulations . Also , the performances of all the models are inferior to human performance by a large margin , suggesting that there ’ s still a long way to go to equip machines with human-like hierarchical perceptual and reasoning abilities . 2 Related Work . 2.1 Visual Reasoning . To assess machines ’ ability on understanding and reasoning over vision and language , a wide variety of benchmarks have been created over the recent years [ 29 , 3 , 28 , 16 , 61 , 12 , 64 , 36 ] . In terms of data input , our dataset mostly resembles the CLEVR dataset [ 29 ] , which is also a diagnostic dataset of synthetic images , template-based questions and compositional programs . However , the objects in CLEVR ’ s scenes are of simple shapes and do not contain parts to form complex logical chains of reasoning . The VQA dataset [ 3 ] contains large-scale crowd-sourced real images and human-generated questions . Since it ’ s not a fully-controlled synthetic dataset , it shows great inherent bias . Visual Commonsense Reasoning ( VCR ) [ 61 ] focuses on reasoning on commonsense knowledge . The GQA dataset [ 28 ] pairs real images with synthetic compositional questions . However , none of these datasets provide part-based dense annotations and questions . RAVEN [ 62 ] associates vision with structural , relational , and analogical reasoning in a hierarchical representation . The dataset asks models to choose an image according to analogy . However , their images are rather simple , mainly consisting of 2D shapes like circles and triangles , and they do not pair images with natural language questions . Although numerous visual reasoning datasets have been proposed , the reasoning chain typically terminates at the object level for these datasets , due to the mere reference to objects based on holistic features or spatial relationships . Also , while previous datasets emphasize reasoning based on spatial relationships , humans possess diverse reasoning abilities , including analogy [ 62 ] , geometry [ 48 , 40 ] , arithmetic [ 63 , 20 , 21 ] , and physics [ 13 ] . These aspects can be effortlessly incorporated into visual reasoning to constitute a benchmark that further approaches humans ’ intellectual level . Thus , we create a new visual-reasoning dataset that focuses on multiple aspects of reasoning on part-whole relations . We compare our dataset with previous visual reasoning datasets in Table 1 . A large number of visual reasoning models have been proposed . Specifically , MAC [ 27 ] combined multi-modal attention for compositional reasoning . LCGN [ 23 ] built contextualized representations for objects to support relational reasoning . These methods , though embedded in neural networks , model the reasoning process implicitly . Neural-symbolic methods [ 56 , 42 , 7 , 38 ] explicitly perform symbolic reasoning on the objects representations and language representations . Recently , methods based on transformers [ 33 ] have achieved remarkable performance on visual reasoning tasks by utilizing end-to-end object detector . In this paper , we conduct experiments and assess strengths and weaknesses of these baseline models on part-based visual reasoning . 2.2 Part-based 3D Understanding . Understanding 3D parts has been a long-standing problem in computer vision and graphics . It is important due to several reasons . Firstly , parts and their arrangements that describe the detailed object geometry and structure are helpful for object discrimination purposes [ 9 , 1 ] . Secondly , parts are usually action handles and their forms could reveal the object functionality [ 34 , 43 , 24 ] , therefore crucial for interacting with objects . Thirdly , functionally related object parts usually share similar forms , making parts suitable for connecting different objects , facilitating knowledge transfer and promoting generalizable learning [ 17 , 41 ] . Finally , parts are often commonly used for 3D generation and content editing purposes [ 11 , 44 , 46 ] . To facilitate part-based 3D understanding , various part-centric 3D datasets have been proposed in the literature . ShapeNet part dataset [ 58 ] annotates 31,963 shapes from ShapeNet [ 4 ] with semantic part segmentation covering 16 categories . And each shape is annotated with 2 to 5 parts . PartNet [ 45 ] further increases the granularity of parts , contributing 573,585 finegrained part instance annotations for 26,671 shapes across 24 object categories . Going beyond semantic understanding , several datasets also focus on the functionality and articulation aspects of parts [ 54 , 43 , 24 , 25 , 52 ] . However , most of these datasets target at part identification rather than part-based conceptual , relational and physical reasoning . Chang et al . [ 5 ] collected 27,477 part instances from 2,278 models covering 90 object categories , with a focus of studying conceptual part-based reasoning . ShapeGlot [ 1 ] presents an object reference dataset with part-level conceptual reasoning implicitly considered . Want et al . [ 51 ] proposes PartNet-Chairs , which contain questions about parts of chairs . These datasets are restricted to conceptual reasoning while our PTR has a broader scope covering ampler reasoning types . Restricted by the annotations in existing large-scale datasets , previous part-based 3D understanding works mainly focus on identifying parts [ 15 , 30 , 26 , 55 , 32 , 58 ] and part relations [ 6 , 10 , 31 , 35 , 60 , 50 ] for individual 3D objects . Several methods also consider a more complex hierarchical structure of 3D parts [ 59 , 44 , 37 , 53 , 14 , 22 , 8 ] . Moreover , these works usually assume a strong categorical prior and tend to have poor generalizability across categories . However , we human beings can easily do more than just part identification or single-object part relationship understanding for objects from known categories . We can reason the relationship between parts from different object instances , make analogies , conduct mental arithmetic , and even infer the physical realism or stability . These tasks are not well supported by current datasets . Therefore we present PTR to close this gap and to better support the study of these human-like common sense . 3 The PTR Dataset . The PTR dataset aims to assess machines ’ ability to perform part-based visual reasoning . We carefully design this dataset in a fully-controlled synthetic environment and enable model diagnostics on this complex reasoning task . Each image has ground-truth annotations regarding the locations , orientations and attributes of objects and parts , as well as the object masks and part masks . We also provide depth images , camera locations and orientations to facilitate the possible use of 3D models on our dataset in the future . For reasoning on geometric relations of parts , we include the geometric information of a part indicating whether it can be considered as a geometric line or plane , and the line equation or plane equation . For physical reasoning , we include the stability information of each object . The images are paired with five types of questions : concept , relation , analogy , arithmetic and physics . The questions have associated functional programs .

Summary: The paper introduces a novel benchmark for part-based reasoning, featuring 80k indoor scenes contraining pieces of furniture from the PartNet dataset. Scenes are annotated with object and part labels, as well as geometric relations. Based on these features, a dataset of questions are generated as a VQA benchmark. Several existing models are evaluated on this benchmark, and are found to underperform  human subjects, sometimes substantially.