Beyond Object Recognition: A New Benchmark towards Object Concept Learning

1 INTRODUCTION . Object understanding is essential for intelligent machines . Recently , benefited by deep learning and large-scale datasets ( Deng et al. , 2009 ; Lin et al. , 2014 ) , category recognition ( Krizhevsky et al. , 2012 ; Ren et al. , 2015 ) has made tremendous leaps of progress . But to close the gap between human and machine perception , machines need to pursue deeper object understanding , e.g. , recognizing higher-level attributes ( Isola et al. , 2015 ) and affordances ( Gibson , 2014 ) , which may help them establish the concept of objects ( Martin , 2007 ) when interacting with complex contexts . Category apple is a symbol indicating its referent ( real apple ) . In line with symbol grounding ( Harnad , 1990 ) , machines should learn knowledge beyond category to approach concept understanding . According to cognition studies ( Ross , 2008 ; Martin , 2007 ) , attributes depicting objects from the physical/visual side play an important role in object understanding . Thus , many works ( Lampert et al. , 2009 ; Xiao et al. , 2010 ; Farhadi et al. , 2009 ) studied to ground objects with attributes , e.g. , a hammer consists of a long handle and a heavy head . Moreover , attributes can depict object state ( Isola et al. , 2015 ) . An elegant characteristic of attributes is cross-category : objects of the same category can have various states ( big or fresh apple ) , whilst various objects can have the same state ( sliced orange or carrot ) . If the category is the first level of object concept , the attribute can be seen as the second level closer to the physical fact . However , recognizing attributes is still far away from concept understanding . Given a hammer , we need to know it can be held to hit nails . That is , requiring machines to infer affordance ( Gibson , 2014 ) indicating what actions humans can perform upon objects . Here , we refer to affordance as the third level , which is closely related to common sense and causal inference ( Gibson , 2014 ) . Although affordance learning has been studied in robotics ( Do et al. , 2018 ; Hermans et al. , 2011 ) and vision ( Chuang et al. , 2018 ; Zhu et al. , 2014 ) communities in decades , it is still challenging . First , previous works ( Nguyen et al. , 2017 ; Fouhey et al. , 2015 ) often focus on recognizing affordance solely . But we usually infer affordance based on attribute observation . When we need to knock in a nail without a hammer at hand , we may find other hard or heavy objects instead , e.g. , a thick book . This profoundly reveals the causality between attribute and affordance . Second , previous works are designed for category/scale/scene-limited tasks , e.g. , in Zhu et al . ( 2014 ) , 40 objects and 14 affordances are included ; Hermans et al . ( 2011 ) collect 375 indoor images of 6 objects , 21 attributes , and 7 affordances ; a recent dataset ( Nguyen et al. , 2017 ) contains 10 indoor objects and 9 affordances . Hence , they can not support general affordance reasoning for large-scale applications . To reshape object learning , we believe it is essential to look at the above three levels in a unified and causal way based on an extensive knowledge base . Hence , we move a step forward to propose the object concept learning ( OCL ) task : given an object , machines need to infer its category , attributes , and further answer “ what can we do upon it and why ” . In a nutshell , machines need to reason affordance based on object appearance , category , and attributes . To this end , we build a largescale and dense dataset consisting of 381 object categories , 114 attributes , and 170 affordances . It contains 80,463 images of diverse scenes , 185,941 object instances in different states . Different from previous works ( Chao et al. , 2015 ; Hermans et al. , 2011 ; Zhu et al. , 2014 ) , OCL offers a more nuanced angle . It includes : ( 1 ) category-level attribute ( A ) and affordance ( B ) labels ; ( 2 ) instancelevel attribute ( α ) and affordance ( β ) labels . Besides , we consider and annotate the causal relations between three levels to evaluate the reasoning ability of models and keep the follow-up methods from fitting data only . Accordingly , based on the causal structure of OCL , we also propose a neurocausal method , Object Concept Reasoning Network ( OCRN ) , as the future baseline . It operates the instantiation from category-level to instance-level , and leverages causal intervention ( Pearl et al. , 2016 ) to alleviate the bias from categories and infer attributes and affordances . OCRN outperforms a host of baselines and shows impressive performance while follows the causal relations well . In summary , our contributions are threefold : ( 1 ) Introducing the object concept learning ( OCL ) task poses challenges and opportunities for deeper object understanding and knowledge-based reasoning . ( 2 ) Building a dataset consists of diverse objects , elaborate attributes and affordances , together with their causal relations to benchmark OCL . ( 3 ) An object concept reasoning network ( OCRN ) is introduced to reason three levels with concept instantiation and debiasing , which performs well . 2 RELATED WORK . Object Attribute . Attribute depicts the visual/physical properties like color , size , shape , etc . It usually plays the role of intermedia between pixels and higher-level concepts , e.g. , prompting object recognition ( Farhadi et al. , 2009 ) , affordance learning ( Hermans et al. , 2011 ) , zero-shot learning ( Lampert et al. , 2009 ) , and object detection ( Kumar Singh et al. , 2018 ) . Recently , several largescale datasets ( Farhadi et al. , 2009 ; Xiao et al. , 2010 ; Liu et al. , 2017 ; Patterson & Hays , 2016 ; Isola et al. , 2015 ; Krishna et al. , 2016 ; Hudson & Manning , 2019 ) are released . For attribute recognition , recent works have made progress . Beside direct multi-label classification ( Lampert et al. , 2009 ; Parikh & Grauman , 2011 ; Xiao et al. , 2010 ; Patterson & Hays , 2016 ) or leveraging the correlation between attribute-attribute and attribute-object ( Hwang et al. , 2011 ; Chen & Grauman , 2014 ; Mahajan et al. , 2011 ) , intrinsic properties ( compositionality , contextuality ( Misra et al. , 2017 ; Nagarajan & Grauman , 2018 ) , symmetry ( Li et al. , 2020b ) ) of attribute-object are also proven useful . Object Affordance . The concept of affordance is introduced by Gibson ( 2014 ) . Affordance learning has two canonical paradigms : direct mapping ( Fouhey et al. , 2015 ) or indirect method ( Zhu et al. , 2014 ; Zhao & Zhu , 2013 ; Wang et al. , 2017 ; Roy & Todorovic , 2016 ) with intermedia like object category , attribute , and 3D contents . Some studies also tried to learn affordance from human-object interactions , i.e. , encoding the relationship between object and action ( Gupta & Davis , 2007 ; Yao et al. , 2013 ; Kato et al. , 2018 ) . Visual Genome ( Krishna et al. , 2016 ) provides visual relationships between objects in the scene graph , including some human actions instead of affordances . However , these relationships cover limited and sparse affordances . Differently , we use easily accessible object images as the knowledge source , and densely annotate all attributes/affordances for all object instances . Besides the vision community , the robot community also pays much attention to affordance ( Do et al. , 2018 ; Pinto & Gupta , 2016 ; Thermos et al. , 2017 ; Pinto et al. , 2016 ) , especially for object grasping and manipulation . Recently , several datasets ( Nguyen et al. , 2017 ; Zhu et al. , 2014 ; Chao et al. , 2015 ; Chuang et al. , 2018 ) have been proposed . Zhu et al . ( 2014 ) built a knowledge base containing attribute , affordance , human pose , and human-object spatial configuration . But labeling pose and human-object relative location are costly . Chao et al . ( 2015 ) proposed a semantic category-level affordance dataset including 91 COCO objects ( Lin et al. , 2014 ) and 957 affordances . Causal Inference . There is increasing literature on exploiting causal inference ( Pearl et al. , 2016 ) in machine learning , especially with causal graphical models ( Spirtes et al. , 2000 ; Pearl et al. , 2016 ) , including feature selection ( Guyon et al. , 2007 ) and learning ( Chalupka et al. , 2014 ) , video analysis ( Pickup et al. , 2014 ; Lebeda et al. , 2015 ) , reinforcement learning ( Nair et al. , 2019 ; Dasgupta et al. , 2019 ) , etc . Recently , Wang et al . ( 2020a ) studied the causal relation between objects in images and used intervention ( Pearl et al. , 2016 ) to alleviate the observation bias in detection . Atzmon et al . ( 2020 ) analyze the causal generative model of compositional zero-shot learning and disentangle the representations of attributes and objects . In this work , we explore the causal relations between three object levels and apply backdoor adjustment ( Pearl et al. , 2016 ) to alleviate the existing bias . 3 KNOWLEDGE BASE CONSTRUCTION . Our goal is to construct a dataset that can characterize abundant knowledge in object category , attribute , affordance , and causal relations . The construction process is given as follows ( Fig . 2 ) . Data Collection . ( 1 ) Affordance : We collect 170 affordances out of 1,006 candidates from widelyused action/affordance datasets ( Chao et al. , 2015 ; Gu et al. , 2018 ; Chao et al. , 2018 ; Gupta & Malik , 2015 ; Zhu et al. , 2014 ; Nguyen et al. , 2017 ) in view of generality and commonness . ( 2 ) Category : Considering the taxonomy ( WordNet ( Fellbaum , 2012 ) ) and diversity , we collect 381 objects out of 1,742 candidates from object datasets ( Farhadi et al. , 2009 ; Xiao et al. , 2010 ; Patterson & Hays , 2016 ; Liu et al. , 2017 ; Lin et al. , 2014 ) . ( 3 ) Attribute : When viewed through the lens of the causal relation between attribute-affordance , we manually filter 500 most frequent attributes from large-scale attribute datasets ( Farhadi et al. , 2009 ; Xiao et al. , 2010 ; Patterson & Hays , 2016 ; Liu et al. , 2017 ; Krishna et al. , 2016 ) and choose 114 attributes , covering colors , deformations , supercategories , various surface , geometrical , and physical properties . ( 4 ) Image : We extract 75,578 images from object datasets ( Farhadi et al. , 2009 ; Xiao et al. , 2010 ; Patterson & Hays , 2016 ; Liu et al. , 2017 ; Krishna et al. , 2016 ) , together with their Ground Truth ( GT ) boxes . To ensure diversity , we further manually collected 4,885 Internet images of selected categories . Then , we annotate the missing box and category labels for all instances via crowdsourcing . Finally , 185,941 instances of 381 categories from 80,463 images are collected , i.e. , average 488 instances per category and 2.31 boxes per image . More details and class lists are in Appendix Sec . A.11 . OCL is long-tail distributed , where the head categories have over 5,000 instances each , but the rarest categories have only 9 instances , which challenges the robustness of vision systems . Attribute Annotation . We offer annotations in two levels : ( 1 ) Category-level attribute ( A ) contains common sense . For each category , we annotate its most common attributes . In concept learning , the usage of category-level annotation as common knowledge can date back to Osherson et al . ( 1991 ) . Following Osherson et al . ( 1991 ) , to avoid bias , annotators are given category-attribute pairs ( category names , not images ) and multi-annotator vote to build the binary category-level attribute matrixMA . ( [ 381 , 114 ] ) . ( 2 ) Instance-level attribute ( α ) is the individual attributes of each instance . The annotation unit is an attribute-instance pair . Each pair is labeled by multi-annotator . Affordance Annotation . There are also two levels of granularity : ( 1 ) Category-level affordance B , similar to A , is annotated in category-affordance pairs , indicating the common affordances of each category . Following Chao et al . ( 2015 ) , the annotators label category-level affordance matrix MB ( [ 381 , 170 ] ) similar to A annotation . ( 2 ) Instance-level affordance β is annotated for every instance with the help of object states ( Isola et al. , 2015 ) . As B is determined by common states , objects in specific states may have different affordances from B , e.g. , we can not board a flying plane . Moreover , if a service robot finds a broken cup , it may infer that the cup can still hold water as it is trained withB labels . Thus , we need detailed β beyondB . As the instances in the same state should have similar β ( all rotten apples can not be eaten ) , six experts first conclude the states . In total , 1,376 states are defined , and each category has 3.6 states on average . Next , β is annotated for each state and the instances are assigned with beta according to their states . More details please refer to Appendix Sec . A.2 It is worth noting that , as states seem to be a beneficial latent variable for OCL , in practice , we do not use it in models because the states we use are a subset of whole state space , and there can be unseen states in real-world data . Causal Relation Annotation . We annotate instance-level ( considering the context of each instance ) causality to answer which attribute ( s ) are the critical and direct causes of a certain affordance ? ( 1 ) Filtering to avoid the combinatorial explosion . Initially , we need to make binary decisions on all instance-α-β triplets , which is far beyond handleable . Fortunately , we find that most α-β classes ( e.g. , shiny and kick ) are meaningless and always of no causality on any instances . Thus , we can firstly exclude the most impossible pairs , meanwhile , guarantee the completeness of causality . Finally , we obtain about 10 % α-β classes as candidates . ( 2 ) Instance-level causality : we also adopt object states as a reference . Multiple annotators have been involved for each state-α-β triplet and are asked whether the specific attribute is the direct cause of this affordance in this state . The answers are combined and checked for all instances of a state . Finally , we obtain about 2 M instance-α-β triplets of causal relations . As we have labelled all α and β for all instances , the causal relations would be in four situations : [ 0,0 ] , [ 1,1 ] ; [ 0,1 ] , [ 1,0 ] . The former two are “ positive ” , e.g. , fresh ( 1/0 ) →eat ( 1/0 ) for an apple . While the last two are “ negative ” , e.g. , broken ( 1/0 ) →drive ( 0/1 ) for a car . Discussion . Fig . 3 shows some examples of OCL . These clear causalities are not thoroughly studied in previous datasets ( Zhu et al. , 2014 ; Nguyen et al. , 2017 ; Fouhey et al. , 2015 ; Hermans et al. , 2011 ) . We compare OCL with previous dense datasets in Tab . 1 . More analysis figures and tables are in Appendix Sec . A.3 .

Summary: In short, the paper proposes a new task, that of object concept learning. Object concept learning is a basically a combination of object classification, attribute classification of said object, and affordance classification of said object, both at a category (affordance of any cup) and an instance (affordance of this cup) level. To this end, the paper proposes a dataset, which starts from existing ones and extends them accordingly. Further, it proposes a baseline method that is somewhat inspired by do-calculus (Pearl). Experiments show positive trends.