PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning

1 Introduction . A long-standing challenge in the field of artificial intelligence is to enable machines to reason and answer questions about visual scenes . Several datasets [ 64 , 16 , 29 , 49 , 57 ] have been proposed to tackle this challenge . They mostly focus on object-level features without emphasizing much on detailed part-level understanding . However , there is strong psychological evidence that human beings parse visual scenes into part-whole hierarchies ( e.g. , from a scene to the objects , from an object to its parts ) , which are currently missing from existing datasets . Inspired by Aristotle ’ s quote “ the whole is greater than the sum of its parts ” , there has been a long history of research in psychology concerning how parts and wholes are related , beginning with the Gestalt psychologists [ 47 ] . There have also been recent efforts on how to represent such part-whole hierarchies using neural networks [ 19 ] . Introducing part-whole hierarchies into visual reasoning ∗The work was done while Yining Hong was a research intern at MIT-IBM Watson AI Lab . 2Project page : http : //ptr.csail.mit.edu/ 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . brings two unique challenges . The first is how to discriminate objects using part-level attributes . Unlike previous works which refer to objects based on holistic attributes such as object category , the specific ontology nature of objects and their parts requires a detailed study . The second is how to leverage part-level attributes to interconnect objects . Objects of various categories are interrelated via some common parts ( e.g. , leg , central support , pedestal ) . The reasoning performed on one category should thus be easily generalized to reasoning on unseen categories with shared parts . Human beings are endowed with such modular but interconnected perceptual systems . They can effortlessly discriminate between the two tables in Figure 1 [ I ] based on the differences in the tops and drawers , as well as observe a connection between the bed and chair due to the similarity of legs . It remains to be explored whether machines have the same hierarchical perceptual capabilities . Based on the highly composite part-whole hierarchies , visual reasoning tasks can be richer , more complex , as well as more challenging . First , the introduction of parts enriches the diversity of both visual perception and question understanding . Apart from basic object-level properties such as color and category , the visual scenes and natural language questions extend upon the properties of object parts , the composition of which makes the image-question pairs more distinctive . Second , the relations between parts go beyond simple spatial relations . Parts can often be approximated as oriented geometric primitives ( e.g. , line , plane ) , and there naturally exist abundant geometric relations between these primitives such as “ perpendicular '' and “ parallel '' ( e.g. , Q2 in Figure 1 [ II ] ) . Analogical reasoning can also be established given such relationships . Third , reasoning on parts enables the understanding of implicit properties of objects , such as physics . For example , the arrangements of the parts and their geometric relations affect the stability of the objects ( e.g. , Q3 , Q4 in Figure 1 [ I ] ) . In this paper , we present a large-scale ParT Reasoning dataset , or PTR for short , a benchmark for part-based conceptual , relational and physical reasoning . It includes ∼ 70k scenes paired with 700k questions containing part-whole structures . We include over 10k objects from the PartNet [ 45 ] dataset across five object categories ( chair , table , bed , refrigerator , cart ) with rich geometric and structural variations to construct our scenes . Our dataset consists of five types of questions : concept , geometry , analogy , arithmetic , and physics . We provide scene graph annotations including the ground-truth locations , segmentations , and properties for all objects and parts , as well as all the object-object and part-part relationships for model diagnostic . We also provide functional programs paired with questions . We analyze a suite of state-of-the-art visual reasoning models on the PTR dataset and find that they all struggle with it , especially in relational , analogical , and physical reasoning . One oracle neural symbolic model performs better than other models but highly relies on additional supervision such as object and part masks and visual attributes from simulations . Also , the performances of all the models are inferior to human performance by a large margin , suggesting that there ’ s still a long way to go to equip machines with human-like hierarchical perceptual and reasoning abilities . 2 Related Work . 2.1 Visual Reasoning . To assess machines ’ ability on understanding and reasoning over vision and language , a wide variety of benchmarks have been created over the recent years [ 29 , 3 , 28 , 16 , 61 , 12 , 64 , 36 ] . In terms of data input , our dataset mostly resembles the CLEVR dataset [ 29 ] , which is also a diagnostic dataset of synthetic images , template-based questions and compositional programs . However , the objects in CLEVR ’ s scenes are of simple shapes and do not contain parts to form complex logical chains of reasoning . The VQA dataset [ 3 ] contains large-scale crowd-sourced real images and human-generated questions . Since it ’ s not a fully-controlled synthetic dataset , it shows great inherent bias . Visual Commonsense Reasoning ( VCR ) [ 61 ] focuses on reasoning on commonsense knowledge . The GQA dataset [ 28 ] pairs real images with synthetic compositional questions . However , none of these datasets provide part-based dense annotations and questions . RAVEN [ 62 ] associates vision with structural , relational , and analogical reasoning in a hierarchical representation . The dataset asks models to choose an image according to analogy . However , their images are rather simple , mainly consisting of 2D shapes like circles and triangles , and they do not pair images with natural language questions . Although numerous visual reasoning datasets have been proposed , the reasoning chain typically terminates at the object level for these datasets , due to the mere reference to objects based on holistic features or spatial relationships . Also , while previous datasets emphasize reasoning based on spatial relationships , humans possess diverse reasoning abilities , including analogy [ 62 ] , geometry [ 48 , 40 ] , arithmetic [ 63 , 20 , 21 ] , and physics [ 13 ] . These aspects can be effortlessly incorporated into visual reasoning to constitute a benchmark that further approaches humans ’ intellectual level . Thus , we create a new visual-reasoning dataset that focuses on multiple aspects of reasoning on part-whole relations . We compare our dataset with previous visual reasoning datasets in Table 1 . A large number of visual reasoning models have been proposed . Specifically , MAC [ 27 ] combined multi-modal attention for compositional reasoning . LCGN [ 23 ] built contextualized representations for objects to support relational reasoning . These methods , though embedded in neural networks , model the reasoning process implicitly . Neural-symbolic methods [ 56 , 42 , 7 , 38 ] explicitly perform symbolic reasoning on the objects representations and language representations . Recently , methods based on transformers [ 33 ] have achieved remarkable performance on visual reasoning tasks by utilizing end-to-end object detector . In this paper , we conduct experiments and assess strengths and weaknesses of these baseline models on part-based visual reasoning . 2.2 Part-based 3D Understanding . Understanding 3D parts has been a long-standing problem in computer vision and graphics . It is important due to several reasons . Firstly , parts and their arrangements that describe the detailed object geometry and structure are helpful for object discrimination purposes [ 9 , 1 ] . Secondly , parts are usually action handles and their forms could reveal the object functionality [ 34 , 43 , 24 ] , therefore crucial for interacting with objects . Thirdly , functionally related object parts usually share similar forms , making parts suitable for connecting different objects , facilitating knowledge transfer and promoting generalizable learning [ 17 , 41 ] . Finally , parts are often commonly used for 3D generation and content editing purposes [ 11 , 44 , 46 ] . To facilitate part-based 3D understanding , various part-centric 3D datasets have been proposed in the literature . ShapeNet part dataset [ 58 ] annotates 31,963 shapes from ShapeNet [ 4 ] with semantic part segmentation covering 16 categories . And each shape is annotated with 2 to 5 parts . PartNet [ 45 ] further increases the granularity of parts , contributing 573,585 finegrained part instance annotations for 26,671 shapes across 24 object categories . Going beyond semantic understanding , several datasets also focus on the functionality and articulation aspects of parts [ 54 , 43 , 24 , 25 , 52 ] . However , most of these datasets target at part identification rather than part-based conceptual , relational and physical reasoning . Chang et al . [ 5 ] collected 27,477 part instances from 2,278 models covering 90 object categories , with a focus of studying conceptual part-based reasoning . ShapeGlot [ 1 ] presents an object reference dataset with part-level conceptual reasoning implicitly considered . Want et al . [ 51 ] proposes PartNet-Chairs , which contain questions about parts of chairs . These datasets are restricted to conceptual reasoning while our PTR has a broader scope covering ampler reasoning types . Restricted by the annotations in existing large-scale datasets , previous part-based 3D understanding works mainly focus on identifying parts [ 15 , 30 , 26 , 55 , 32 , 58 ] and part relations [ 6 , 10 , 31 , 35 , 60 , 50 ] for individual 3D objects . Several methods also consider a more complex hierarchical structure of 3D parts [ 59 , 44 , 37 , 53 , 14 , 22 , 8 ] . Moreover , these works usually assume a strong categorical prior and tend to have poor generalizability across categories . However , we human beings can easily do more than just part identification or single-object part relationship understanding for objects from known categories . We can reason the relationship between parts from different object instances , make analogies , conduct mental arithmetic , and even infer the physical realism or stability . These tasks are not well supported by current datasets . Therefore we present PTR to close this gap and to better support the study of these human-like common sense . 3 The PTR Dataset . The PTR dataset aims to assess machines ’ ability to perform part-based visual reasoning . We carefully design this dataset in a fully-controlled synthetic environment and enable model diagnostics on this complex reasoning task . Each image has ground-truth annotations regarding the locations , orientations and attributes of objects and parts , as well as the object masks and part masks . We also provide depth images , camera locations and orientations to facilitate the possible use of 3D models on our dataset in the future . For reasoning on geometric relations of parts , we include the geometric information of a part indicating whether it can be considered as a geometric line or plane , and the line equation or plane equation . For physical reasoning , we include the stability information of each object . The images are paired with five types of questions : concept , relation , analogy , arithmetic and physics . The questions have associated functional programs .

Summary: This paper proposes a benchmark for part-based commonsense reasoning. The dataset consists of realistic-looking scenes with various objects in the scene. Several queries are designed to test a model's reasoning capability which includes concepts, relations, physics, analogy, and arithmetic. The author(s) run several baselines spanning from a random picking to an oracle model (NS-VQA) which utilizes ground truths for vision model and language programs.