Beyond Object Recognition: A New Benchmark towards Object Concept Learning

1 INTRODUCTION . Object understanding is essential for intelligent machines . Recently , benefited by deep learning and large-scale datasets ( Deng et al. , 2009 ; Lin et al. , 2014 ) , category recognition ( Krizhevsky et al. , 2012 ; Ren et al. , 2015 ) has made tremendous leaps of progress . But to close the gap between human and machine perception , machines need to pursue deeper object understanding , e.g. , recognizing higher-level attributes ( Isola et al. , 2015 ) and affordances ( Gibson , 2014 ) , which may help them establish the concept of objects ( Martin , 2007 ) when interacting with complex contexts . Category apple is a symbol indicating its referent ( real apple ) . In line with symbol grounding ( Harnad , 1990 ) , machines should learn knowledge beyond category to approach concept understanding . According to cognition studies ( Ross , 2008 ; Martin , 2007 ) , attributes depicting objects from the physical/visual side play an important role in object understanding . Thus , many works ( Lampert et al. , 2009 ; Xiao et al. , 2010 ; Farhadi et al. , 2009 ) studied to ground objects with attributes , e.g. , a hammer consists of a long handle and a heavy head . Moreover , attributes can depict object state ( Isola et al. , 2015 ) . An elegant characteristic of attributes is cross-category : objects of the same category can have various states ( big or fresh apple ) , whilst various objects can have the same state ( sliced orange or carrot ) . If the category is the first level of object concept , the attribute can be seen as the second level closer to the physical fact . However , recognizing attributes is still far away from concept understanding . Given a hammer , we need to know it can be held to hit nails . That is , requiring machines to infer affordance ( Gibson , 2014 ) indicating what actions humans can perform upon objects . Here , we refer to affordance as the third level , which is closely related to common sense and causal inference ( Gibson , 2014 ) . Although affordance learning has been studied in robotics ( Do et al. , 2018 ; Hermans et al. , 2011 ) and vision ( Chuang et al. , 2018 ; Zhu et al. , 2014 ) communities in decades , it is still challenging . First , previous works ( Nguyen et al. , 2017 ; Fouhey et al. , 2015 ) often focus on recognizing affordance solely . But we usually infer affordance based on attribute observation . When we need to knock in a nail without a hammer at hand , we may find other hard or heavy objects instead , e.g. , a thick book . This profoundly reveals the causality between attribute and affordance . Second , previous works are designed for category/scale/scene-limited tasks , e.g. , in Zhu et al . ( 2014 ) , 40 objects and 14 affordances are included ; Hermans et al . ( 2011 ) collect 375 indoor images of 6 objects , 21 attributes , and 7 affordances ; a recent dataset ( Nguyen et al. , 2017 ) contains 10 indoor objects and 9 affordances . Hence , they can not support general affordance reasoning for large-scale applications . To reshape object learning , we believe it is essential to look at the above three levels in a unified and causal way based on an extensive knowledge base . Hence , we move a step forward to propose the object concept learning ( OCL ) task : given an object , machines need to infer its category , attributes , and further answer “ what can we do upon it and why ” . In a nutshell , machines need to reason affordance based on object appearance , category , and attributes . To this end , we build a largescale and dense dataset consisting of 381 object categories , 114 attributes , and 170 affordances . It contains 80,463 images of diverse scenes , 185,941 object instances in different states . Different from previous works ( Chao et al. , 2015 ; Hermans et al. , 2011 ; Zhu et al. , 2014 ) , OCL offers a more nuanced angle . It includes : ( 1 ) category-level attribute ( A ) and affordance ( B ) labels ; ( 2 ) instancelevel attribute ( α ) and affordance ( β ) labels . Besides , we consider and annotate the causal relations between three levels to evaluate the reasoning ability of models and keep the follow-up methods from fitting data only . Accordingly , based on the causal structure of OCL , we also propose a neurocausal method , Object Concept Reasoning Network ( OCRN ) , as the future baseline . It operates the instantiation from category-level to instance-level , and leverages causal intervention ( Pearl et al. , 2016 ) to alleviate the bias from categories and infer attributes and affordances . OCRN outperforms a host of baselines and shows impressive performance while follows the causal relations well . In summary , our contributions are threefold : ( 1 ) Introducing the object concept learning ( OCL ) task poses challenges and opportunities for deeper object understanding and knowledge-based reasoning . ( 2 ) Building a dataset consists of diverse objects , elaborate attributes and affordances , together with their causal relations to benchmark OCL . ( 3 ) An object concept reasoning network ( OCRN ) is introduced to reason three levels with concept instantiation and debiasing , which performs well . 2 RELATED WORK . Object Attribute . Attribute depicts the visual/physical properties like color , size , shape , etc . It usually plays the role of intermedia between pixels and higher-level concepts , e.g. , prompting object recognition ( Farhadi et al. , 2009 ) , affordance learning ( Hermans et al. , 2011 ) , zero-shot learning ( Lampert et al. , 2009 ) , and object detection ( Kumar Singh et al. , 2018 ) . Recently , several largescale datasets ( Farhadi et al. , 2009 ; Xiao et al. , 2010 ; Liu et al. , 2017 ; Patterson & Hays , 2016 ; Isola et al. , 2015 ; Krishna et al. , 2016 ; Hudson & Manning , 2019 ) are released . For attribute recognition , recent works have made progress . Beside direct multi-label classification ( Lampert et al. , 2009 ; Parikh & Grauman , 2011 ; Xiao et al. , 2010 ; Patterson & Hays , 2016 ) or leveraging the correlation between attribute-attribute and attribute-object ( Hwang et al. , 2011 ; Chen & Grauman , 2014 ; Mahajan et al. , 2011 ) , intrinsic properties ( compositionality , contextuality ( Misra et al. , 2017 ; Nagarajan & Grauman , 2018 ) , symmetry ( Li et al. , 2020b ) ) of attribute-object are also proven useful . Object Affordance . The concept of affordance is introduced by Gibson ( 2014 ) . Affordance learning has two canonical paradigms : direct mapping ( Fouhey et al. , 2015 ) or indirect method ( Zhu et al. , 2014 ; Zhao & Zhu , 2013 ; Wang et al. , 2017 ; Roy & Todorovic , 2016 ) with intermedia like object category , attribute , and 3D contents . Some studies also tried to learn affordance from human-object interactions , i.e. , encoding the relationship between object and action ( Gupta & Davis , 2007 ; Yao et al. , 2013 ; Kato et al. , 2018 ) . Visual Genome ( Krishna et al. , 2016 ) provides visual relationships between objects in the scene graph , including some human actions instead of affordances . However , these relationships cover limited and sparse affordances . Differently , we use easily accessible object images as the knowledge source , and densely annotate all attributes/affordances for all object instances . Besides the vision community , the robot community also pays much attention to affordance ( Do et al. , 2018 ; Pinto & Gupta , 2016 ; Thermos et al. , 2017 ; Pinto et al. , 2016 ) , especially for object grasping and manipulation . Recently , several datasets ( Nguyen et al. , 2017 ; Zhu et al. , 2014 ; Chao et al. , 2015 ; Chuang et al. , 2018 ) have been proposed . Zhu et al . ( 2014 ) built a knowledge base containing attribute , affordance , human pose , and human-object spatial configuration . But labeling pose and human-object relative location are costly . Chao et al . ( 2015 ) proposed a semantic category-level affordance dataset including 91 COCO objects ( Lin et al. , 2014 ) and 957 affordances . Causal Inference . There is increasing literature on exploiting causal inference ( Pearl et al. , 2016 ) in machine learning , especially with causal graphical models ( Spirtes et al. , 2000 ; Pearl et al. , 2016 ) , including feature selection ( Guyon et al. , 2007 ) and learning ( Chalupka et al. , 2014 ) , video analysis ( Pickup et al. , 2014 ; Lebeda et al. , 2015 ) , reinforcement learning ( Nair et al. , 2019 ; Dasgupta et al. , 2019 ) , etc . Recently , Wang et al . ( 2020a ) studied the causal relation between objects in images and used intervention ( Pearl et al. , 2016 ) to alleviate the observation bias in detection . Atzmon et al . ( 2020 ) analyze the causal generative model of compositional zero-shot learning and disentangle the representations of attributes and objects . In this work , we explore the causal relations between three object levels and apply backdoor adjustment ( Pearl et al. , 2016 ) to alleviate the existing bias . 3 KNOWLEDGE BASE CONSTRUCTION . Our goal is to construct a dataset that can characterize abundant knowledge in object category , attribute , affordance , and causal relations . The construction process is given as follows ( Fig . 2 ) . Data Collection . ( 1 ) Affordance : We collect 170 affordances out of 1,006 candidates from widelyused action/affordance datasets ( Chao et al. , 2015 ; Gu et al. , 2018 ; Chao et al. , 2018 ; Gupta & Malik , 2015 ; Zhu et al. , 2014 ; Nguyen et al. , 2017 ) in view of generality and commonness . ( 2 ) Category : Considering the taxonomy ( WordNet ( Fellbaum , 2012 ) ) and diversity , we collect 381 objects out of 1,742 candidates from object datasets ( Farhadi et al. , 2009 ; Xiao et al. , 2010 ; Patterson & Hays , 2016 ; Liu et al. , 2017 ; Lin et al. , 2014 ) . ( 3 ) Attribute : When viewed through the lens of the causal relation between attribute-affordance , we manually filter 500 most frequent attributes from large-scale attribute datasets ( Farhadi et al. , 2009 ; Xiao et al. , 2010 ; Patterson & Hays , 2016 ; Liu et al. , 2017 ; Krishna et al. , 2016 ) and choose 114 attributes , covering colors , deformations , supercategories , various surface , geometrical , and physical properties . ( 4 ) Image : We extract 75,578 images from object datasets ( Farhadi et al. , 2009 ; Xiao et al. , 2010 ; Patterson & Hays , 2016 ; Liu et al. , 2017 ; Krishna et al. , 2016 ) , together with their Ground Truth ( GT ) boxes . To ensure diversity , we further manually collected 4,885 Internet images of selected categories . Then , we annotate the missing box and category labels for all instances via crowdsourcing . Finally , 185,941 instances of 381 categories from 80,463 images are collected , i.e. , average 488 instances per category and 2.31 boxes per image . More details and class lists are in Appendix Sec . A.11 . OCL is long-tail distributed , where the head categories have over 5,000 instances each , but the rarest categories have only 9 instances , which challenges the robustness of vision systems . Attribute Annotation . We offer annotations in two levels : ( 1 ) Category-level attribute ( A ) contains common sense . For each category , we annotate its most common attributes . In concept learning , the usage of category-level annotation as common knowledge can date back to Osherson et al . ( 1991 ) . Following Osherson et al . ( 1991 ) , to avoid bias , annotators are given category-attribute pairs ( category names , not images ) and multi-annotator vote to build the binary category-level attribute matrixMA . ( [ 381 , 114 ] ) . ( 2 ) Instance-level attribute ( α ) is the individual attributes of each instance . The annotation unit is an attribute-instance pair . Each pair is labeled by multi-annotator . Affordance Annotation . There are also two levels of granularity : ( 1 ) Category-level affordance B , similar to A , is annotated in category-affordance pairs , indicating the common affordances of each category . Following Chao et al . ( 2015 ) , the annotators label category-level affordance matrix MB ( [ 381 , 170 ] ) similar to A annotation . ( 2 ) Instance-level affordance β is annotated for every instance with the help of object states ( Isola et al. , 2015 ) . As B is determined by common states , objects in specific states may have different affordances from B , e.g. , we can not board a flying plane . Moreover , if a service robot finds a broken cup , it may infer that the cup can still hold water as it is trained withB labels . Thus , we need detailed β beyondB . As the instances in the same state should have similar β ( all rotten apples can not be eaten ) , six experts first conclude the states . In total , 1,376 states are defined , and each category has 3.6 states on average . Next , β is annotated for each state and the instances are assigned with beta according to their states . More details please refer to Appendix Sec . A.2 It is worth noting that , as states seem to be a beneficial latent variable for OCL , in practice , we do not use it in models because the states we use are a subset of whole state space , and there can be unseen states in real-world data . Causal Relation Annotation . We annotate instance-level ( considering the context of each instance ) causality to answer which attribute ( s ) are the critical and direct causes of a certain affordance ? ( 1 ) Filtering to avoid the combinatorial explosion . Initially , we need to make binary decisions on all instance-α-β triplets , which is far beyond handleable . Fortunately , we find that most α-β classes ( e.g. , shiny and kick ) are meaningless and always of no causality on any instances . Thus , we can firstly exclude the most impossible pairs , meanwhile , guarantee the completeness of causality . Finally , we obtain about 10 % α-β classes as candidates . ( 2 ) Instance-level causality : we also adopt object states as a reference . Multiple annotators have been involved for each state-α-β triplet and are asked whether the specific attribute is the direct cause of this affordance in this state . The answers are combined and checked for all instances of a state . Finally , we obtain about 2 M instance-α-β triplets of causal relations . As we have labelled all α and β for all instances , the causal relations would be in four situations : [ 0,0 ] , [ 1,1 ] ; [ 0,1 ] , [ 1,0 ] . The former two are “ positive ” , e.g. , fresh ( 1/0 ) →eat ( 1/0 ) for an apple . While the last two are “ negative ” , e.g. , broken ( 1/0 ) →drive ( 0/1 ) for a car . Discussion . Fig . 3 shows some examples of OCL . These clear causalities are not thoroughly studied in previous datasets ( Zhu et al. , 2014 ; Nguyen et al. , 2017 ; Fouhey et al. , 2015 ; Hermans et al. , 2011 ) . We compare OCL with previous dense datasets in Tab . 1 . More analysis figures and tables are in Appendix Sec . A.3 .

Summary: This paper proposes a new large-scale benchmark dataset for object concept learning, which consists of recognizing attributes, affordances, and their causal raltions about objects in input images. Detailed annotations of object categories, attributes and affordances on both category and instance levels, and their causal relations (on the instance level) are provided. This new dataset will be helpful for the community to advance the research of causal learning from visual data.   A strong baseline method is also proposed that explicitly considers the causal structure and concept instantiation of object categories, attributes, and affordances. Although it shows better results than other baselines, there is still great room for future improvement.