Beyond Object Recognition: A New Benchmark towards Object Concept Learning
Authors: Yong-Lu Li
Yue Xu
Xinyu Xu
Xiaohan Mao
Yuan Yao
Siqi Liu
Cewu Lu
Venue: ICLR 2022 Submitted (2022)
Keywords: Object Concept Learning
Attributes
Affordance
Causal Inference

Abstract: Understanding objects is a central building block of artificial intelligence, especially for embodied AI. Even though object recognition excels with deep learning, current machines still struggle to learn higher-level knowledge, e.g., what attributes does an object have, what can we do with an object. In this work, we propose a challenging Object Concept Learning (OCL) task to push the envelope of object understanding. It requires machines to reason out object affordances and simultaneously give the reason: what attributes make an object possesses these affordances. To support OCL, we build a densely annotated knowledge base including extensive labels for three levels of object concept: categories, attributes, and affordances, together with their causal relations. By analyzing the causal structure of OCL, we present a strong baseline, Object Concept Reasoning Network (OCRN). It leverages causal intervention and concept instantiation to infer the three levels following their causal relations. In extensive experiments, OCRN effectively infers the object knowledge while follows the causalities well. Our data and code will be publicly available.

Under review as a conference paper at ICLR 2022
BEYOND OBJECT RECOGNITION: A NEW BENCH- MARK TOWARDS OBJECT CONCEPT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Understanding objects is a central building block of artificial intelligence, espe- cially for embodied AL. Even though object recognition excels with deep learn- ing, current machines still struggle to learn higher-level knowledge, e.g., what attributes does an object have, what can we do with an object. this work, we propose challenging Object Concept Learning (OCL) task to push the envelope of object understanding. It requires machines to reason out object affordances and simultaneously give the reason: what attributes make an object possesses these af- fordances. To support OCL, we build a densely annotated knowledge base includ- ing extensive labels for three levels of object concept: categories, attributes, and affordances, and their causal relations. By analyzing the causal structure of OCL, we present baseline, Object Concept Reasoning Network (OCRN). It leverages causal intervention and concept instantiation to infer the three levels following their causal relations. In experiments, OCRN effectively infers the object knowl- edge while follows the causalities well, Data and code will be publicly available.
INTRODUCTION
Object understanding is essential for intelligent machines. Recently, benefited by deep learning and large-scale datasets (Deng et al. 2009 Lin al.. 2014), category recognition (Krizhevsky et al., 2012 Ren et al.] 2015) has made tremendous leaps of progress. But to close the gap between human and machine perception, machines need pursue deeper object understanding, e.g., recognizing higher-level attributes (Isola et al., 2015) and affordances (Gibson) 2014), which may help them establish the concept objects (Martin) 2007) when interacting with complex contexts.
Category apple e is symbol indicating its referent (real apple). In line with symbol grounding (Har- nad] 1990). machines should learn knowledge beyond category to approach concept understanding. According to cognition studies (Ross) 2008 Martin 2007), attributes depicting objects from the physical/visual side play an important role in object understanding. Thus, many works (Lampert) et al. 2009: Xiao al. 2010 Farhadi et al. 2009) studied to ground objects with attributes, e.g., a harmer consists of a long handle and a heavy head. Moreover, attributes can depict object state (Isola al. 2015). An elegant characteristic of attributes is cross-category: objects of the same category can have various states (big or fresh apple), whilst various objects can have the same state (s (eliced orange or carrot). If the category is the first level of object concept, the attribute can be seen as the second level closer to the physical fact.
However, recognizing attributes is still far away from concept understanding. Given a hammer. I, need to know it can be he le to hit nails. That is, requiring machines to infer affordance (Gibson 2014] indicating what actions humans can perform upon objects. Here, we refer to affordance as the third level, which is closely related to common sense and causal inference (Gibson) 2014). Although affordance learning has been studied in roboties (Do et al.] 2018 20088/Hermm Hermans et al.) 201 and vision (Chuang et al. 2018 Zhu et al. 2014) communities in decades, it is still challenging. First, previous works (Nguyen et al.) 2017 Fouhey et al.) 2015) often focus on recognizing affordance solely. But we usually infer affordance based on attribute observation. When we need to knock in nail without a hammer at hand, we may find other hard or heavy objects instead, e.g., a thick book. This profoundly reveals the causality between attribute and affordance. Second, previous works are designed for category/scall/sccnee limited tasks, e.g., in Zhu et al.] (2014), 40 objects and 14 affordances are included; Hermans al. (2011) collect 375 indoor images of objects, 21

---

Under review as a conference paper at ICLR 2022
Calegary?
Cat
Mug v
Category
Green White
Digiul
Causal
Altribate?
Relations
Cute
Painted
Attribute
Aftordance
Affordance ?
Bold
Eat
Inspect
Pot
Drink wrnkkwttty
Feed
For intelligent and embodied agents, understanding the daily objects requires the ability to perceive not only object category but also attributes and affordance. Thus, in OCL, we try to reveal object concept learning both three levels and explore their profound causal relations.
Figure I:
attributes, and 7 affordances; a recent dataset (Nguyen et al., (017) contains 10 indoor objects and 9 affordances. Hence, they cannot support general affordance reasoning for large-scale applications.
To reshape object learning, we believe it is essential to look at the above three levels in a unified and causal way based on an extensive knowledge base. Hence, we move step forward to propose the object concept learning (OCL) task: given an object, machines need to infer its category, attributes, and further answer "what can we do upon it and why". In nutshell, machines need to reason affordance based on object appearance, category. and attributes. To this end, we build a large- scale and dense dataset consisting of 381 object categories, 114 attributes, and 170 affordances. contains 80,463 images of diverse scenes, 185,941 object instances in different states. Different from previous works (Chao et al.) 2015 Hermans et al. 2011 Zhu et al. 2014), OCL offers a more nuanced angle. It includes: (I) rategory-leve attribute (A) and affordance (B) labels; (2) instance- level attribute (a) and affordance (3) labels. Besides, we consider and annotate the causal relations between three levels to evaluate the reasoning ability of models and keep the follow-up methods from fitting data only. Accordingly, based on the causal structure of OCL, we also propose neuro- causal method, Object Concept Reasoning Network (OCRN), as the future baseline. It operates the instantiation from category-level to instance-level. and leverages causal intervention (Pearl et al.) 2016) to alleviate the bias from categories and infer attributes and affordances. OCRN outperforms a host of baselines and shows impressive performance while follows the causal relations well.
In summary, our contributions are threefold: (1) Introducing the object concept learning (OCL) task poses challenges and opportunities for deeper object understanding and knowledge-based reason- ing. (2) Building a dataset consists of diverse objects, elaborate attributes and affordances, together with their causal relations to benchmark OCL. (3) An object concept reasoning network (OCRN) is introduced to reason three levels with concept instantiation and debiasing, which performs well.
2 RELATED WORK
Object Attribute. Attribute depicts the visual/physical properties like color, size, shape, etc. It usually plays the role of intermedia between pixels and higher-level concepts, e-g., prompting ob-
ject recognition (Farhadi et al. 2009), affordance learning (Hermans et al. 2011), zero-shot learn- ing (Lampert et al.] 2009), , and object detection (Kumar Singh et al. 2018). Recently, several large-
scale datasets (Farhadi et al. 2009 Xiao et al. 2010 Liu et al.] 2017 Patterson & Hays 2016 Isola cet al. 2015) Krishna et al. 2016 Hudson & Manning 2019) are released. For attribute recognition,
recent works have made progress. Beside direct multi-label classification (Lampert al., 2009 Parikh & Grauman, 2011] T. Xiao et al., 2010 Patterson & Hays 2016) or leveraging the correlation
between attribute-atribunt and attribute-object (Hwang et al. 1 Chen Grauman, 2014 Maha-
jan et al.] 2011), intrinsic properties (compositionality, contextuality (Misra et al. 2017] Nagarajan & Grauman 2018), symmetry (Li al. 2020bb) of attribute-object are also proven useful.
Object Affordance. The concept of affordance is introduced by Gibson (0144). Affordance learn-
ing has two canonical paradigms: direct mapping (Fouhey et al. (2015) or indirect method (Zhu al. 2014 Zhao & Zhu| 20133. Wang et al.] [017]] Roy & Todorovic 2016) with intermedia like object category, attribute, and 3D contents. Some studies also tried to learn affordance from human-object interactions. i.e., encoding the relationship between object and action (Gupta & Davis 2007, Yao ctt al., 2013 Kato et al., 2018). Visual Genome (Krishna et al. 2016) provides visual relationships between objects in the scene graph, including some human actions instead of affordances. However, these relationships cover limited and sparse affordances. Differently, we use easily accessible ob- ject images the knowledge source, and densely annotate all atrriuteeslfforddanes for all object instances. Besides the vision community, the robot community also pays much attention to affor- dance (Do et al. 2018 Pinto & Gupta 2016 Thermos et al. 2017 Pinto al., 2016), especially for
2

---

Under review as a conference paper at ICLR 2022
J. Data collection
Category-level annotation
Instance-level annotation
esperto
attribute
affordance
Categoey candidates
halls nound?"
Alteinut
0-3-core by votinng
SUEI!
openn
Besters car
Affordance classos
Objjcct cabogorins
0/1
propose object states
add Maliio
classify state
0/1 and rechack
AAz ppwxpplls ediblle?
Affoncaam
Objectt cotegories
images busss
Alliibule dlssses
by voting
crow cowddouriing
exporta
exclting
dacssets and Internet
d. Causality annotation
pains
cooplemror edeegory Beses
nound scroll small kick
lecal Gorch
scan them volu
light lift
Madian
OCL
black eat
cotten
annccatorss
ichonse r6sociated
-t-B pairs
Figure 2: OCL construction. a) Data collection. b) Annotating category-level attributes and affor- dances. c) Annotating instance- attributes and affordances. d) Finding direct causal relations.
object grasping and manipulation. Recently, several datasets (Nguyen al. 2017 Zhu al. 2014 Chao et al. 20155] Chuang et al. (2018) have been proposed. Zhu et al. (2014) built knowledge base containing attribute, affordance, human pose, and human-object spatial configuration. But la- beling pose and human-object relative location are costly. Chao et al.
(2155) proposed semantic
category-level affordance dataset including 91 COCO objects (Lin et al. 2014 and 957 affordances.
Causal Inference. There is increasing literature on exploiting causal inference Pearl et al. 2016)
in machine learning, especially with causal graphical models (Spirtes et al., 2000, Pearl et al. 2016). including feature selection (Guyon et al. 2007) and learning (Chalupka et al. [014)). video analy-
sis (Pickup et af., 2014, Lebeda et al. 2015), reinforcement learning (Nair et al.) 2019) Dasgupta cet al. 2019), etc. Recently, Wang al. (2020a) studied the causal relation between objects in images and used intervention (Pearl et al. (0166) to alleviate the observation bias in detection. Atzzmn et al. (2020) analyze the causal generative model of compositional zero-shot learning and disentangle the representations of attributes and objects. In this work, we explore the causal relations between three object levels and apply backdoor adjustment (Pearl et al. 2016) alleviate the existing bias.
KNOWLEDGE BASE CONSTRUCTION
Our goal is to construct dataset that can characterize abundant knowledge in object category,
attribute, affordance, and causal relations. The construction process is given as follows (Fig. 2
Data Collection. (1) Affordance: We collect 170 affordances out of 1,006 candidates from widely-
used action/affordance datasets (Chao et al.]
2015 Gu et al.] 2018 Chao et al.) [0018] Gupta & 2017 in view of generality and commonness. (2)
Malik 2015) Zhu et al.) 2014 Nguyen et al.
Category: Considering the taxonomy (WordNet (Fellbaum) (012)) and diversity, we collect 381
objects out of 1.742 candidates from object datasets (Farhadi al. 2009 Xiao et al.] 20100] Patterson
& Hays] 2016 Liu et al.) 2017 Lin et al.) (2014). (3) Attribute: When viewed through the lens of
the causal relation between attribute-affordance,
manually filter 500 most frequent attributes from large-scale attribute datasets (Farhadi et al., 2009) Xiao et al.) 2010 Patterson & Hays 2016)
Liu al.) 2017]] Krishna et al., (2016) and choose 114 attributes, covering colors, deformations, supercategories, various surface, geometrical, and physical properties. (4) Image: We extract 75,578
images from object datasets (Farbadi et al., 2009 Xiao et al., 2010 Patterson & Hays
2016 Liu
et al. K et al. al.] 2016)), together with their Ground Truth (GT) boxes. To ensure diversity,
we further manually collected 4,885 Internet images of selected categories. Then, we annotate the missing box and category labels for all instances via crowdsourcing. Finally, 185,941 instances of 381 categories from 80,463 images are collected, i.e., average 488 instances per category and 2.31 boxes per image. More details and class lists are in Appendix Sec. A.77 OCL is long-tail distributed, where the head categories have over 5,000 instances each, but the rarest categorics have only 9 instances, which challenges the robustness of vision systems.
Attribute Annotation. We offer annotations in two levels: (1) Category-level attribute (AA con- tains common sense. For each category, we annotate its most common attributes. In concept learning, the usage of category- annotation as common knowledge can date back to (osheroonn et al. (199T). Following Osherson al. (1991), avoid bias, annotators are given category-atrribute
3

---

Under review as a conference paper at ICLR 2022
bute
Apple
X
State: Locau
Stuee
Static
li
Toy
Closad v
Open V
Fresh v
Eol, V
Parked V
V
box
Kanel
Woodens
apple
Padacc
LockeoX
State:
State
Sone:
Closed v/
Open v
Fresh
X
V
Parked V
Start V
Figure 3: OCL examples. Causal relations between a (red), 3 (blue) in various contexts are listed.
Dataset
Image
Instance
# Object
# Atribute
# Allordance
APY (Farhadi et al.
2009
15,339
13,339
32
64
SUN (Xiao et al 2010
14,340
14,340
717
102
COCO-a [Pafferson & Hays 2016
84,044
188,426
29
196
ImageNet]500t 50K (Liu et 2017
150,000
150,000
1,000
25
Chao et al. 2013
91
/
957 (B)
Hermans al. 2011
375
6
21
7
Zhu ct al. 2014
4,000
4,000
40
57
14
OCL
80,463
185,941
381
114
170
Table 1: Comparison of dense annotated datasets. OCL provides the category- and instance- level
attributes (A, o),, affordances (B, (3).
pairs (category names, not images) and multi annotaton vote to build the binary category-level at- tribute matrix MA. (3881, 114)). (2) Instance-level attribute (a) is the individual attributes of each instance. The annotation unit is an
pair. Each pair is labeled by multi- aulti-annotator.
Affordance Annotation. There are also two levels of granularity: (I) Category-level affordance B, similar to A, is annotated in category-aforrlace pairs, indicating the common affordances of each category. Following Chao al. (2015), the annotators label category-level affordance matrix MB (381, 170) similar to A annotation. (2) Instance-level affordance is annotated for every instance with the help of object states (Isola et al.) 2015). As B is determined by common states, objects in specific states may have different affordances from B, e.g., we cannot board f flying plane. Moreover, if service robot finds a broken cup. it may infer that the cup can still hold water it is trained with B labels. Thus, we need detailed ,3 beyond B. As the instances in the same state should have similar 3 (all rotten apples cannot be eaten), six experts first conclude the states. In total, 1,376 states are defined, and each category has 3.6 states on average. Next, / is annotated for each state and the instances are assigned with beta according to their states. More details please refer to Appendix Sec. A.2 It is worth noting that, states seem to be a beneficial latent variable for OCL, in practice, we do not use it in models because the states we use are a subset of whole state space, and there can be unseen states in real-worke data.
Causal Relation Annotation. We annotate instance-level (considering the context of each instance) causality to answer which attribute(s) are the critical and direct causes of certain affordance? (1) Filtering to avoid the combinatorial explosion. Initially, we need to make binary decisions on all instance-a-3 triplets, which is far beyond handleable. Fortunately, we find that most a-3 classes (e.g., shiny and kick) are meaningless and always of no causality on any instances. Thus, we can firstly exclude the most impossible pairs, meanwhile, guarantee the completeness of causality. Finally, we obtain about 10% a-3 classes candidates. (2) Instance-level causality: we also adopt object states as a reference. Multiple annotators have been involved for each state-a-B triplet and asked whether the specific attribute is the direct cause of this affordance in this state. The answers are combined and checked for all instances of a state. Finally, we obtain about 2 M instance-a-f triplets of causal relations. As have labelled all and 3 for all instances, the causal relations would be in (ea)s((///)a)((10))))
four situations: [0,0], [1: [0,1], [1,0]. The former two are "positive", e.g., "negative", e.g., broken(1/0) -dr ve(0/1) for car.
for apple. While the last
Discussion. Fig. shows some examples of OCL. These clear causalities are not thoroughly studied in previous datasets (Zhu et al., 2014, Nguyen al.) 2017; Fouhey al. 2015 Hermans et al.) 2011). We compare OCL with previous dense datasets in Tab. More analysis figures and tables in Appendix Sec. A.3

---

Under review as a conference paper at ICLR 2022
4 OBJECT CONCEPT LEARNING
4.1 TASK OVERVIEW
Given instance 7 (image region in box bo), OCL aims infer attribute and affordance while following the causalities. Formally, OCL can be described as:
Pa, PPa.P3>=FLPPOO)) (TDEIII
where Pa, P3 the probabilities of a, B, given an object instance and its predicted category probability P(O[T). A means the Total Direct Effect (TDE (Pearl et al. 2016) of affordance prediction change after operate upon a model F(.). A is expected follow the GT causal relation between attribue-afforddanee. We will detail the reasoning evaluation in Sec. 5
We spilt images into train, validation (val) and test with 56,916: 11:6:::,,46:9.111 images. The val and test sets cover 221 the 381 categories, and the train set covers all categories. OCL is a long-taile recognition task and requires generalization to cover the whole object category-atrriutte affordance space with imbalanced information. Thus, it is challenging for current vision systems without the reasoning ability to understand the causalities.
4.2 OBJECT CONCEPT REASONING NETWORK
Here, we first analyze the causal structure of OCL and then propose a basic system, Object Concept Reasoning Network (OCRN), as the future baseline for the OCL task.
OCL Causal Graph. We use the causal graph to shed light on the subtle causalities OCL in
Fig. 4 Causal graph (Pearl et al.) 2016) indi- cates the underlying causalities between vari- ables O, B; 1, 3. Directed edges repre- sent the causal directions. According to the prior knowledge about the causalities between three levels, a hierarchical structure is depicted: (a) the inner triangle with dotted lines is the category-level: object category O decides the category-level attributes A and affordances B; (b) the outer triangle is the instance-level: instance visual appearance I, instance-level attributes a and affordances B. As indicates the physical noumenon, it is the source of semantic and functional properties and decides 3. As attributes can affect affordances, in both two levels, there are edges from attributes to affordances. From another perspective, 1, at, 3 are the instantiations of O, A, B respectively. We can use whole node O' to represent O, A, B (Fig. 5).
Mag
Attribute?
Affordange?
1542
toradded
Taak vill
Tegasy
Fead
(a) OCL: Object Concept Learning Figure 4: OCL causal graph. I is object appear- ance. J, 3 are the instantiations of O, A, B.
(b) Causal Graph
Object Category Bias. Formally, OCL depicted as P(a(T) and P(3II, Following the causal graph, we represent nodes {I, A, B, 3) as JJI.JA. Sa: }g} respectively in latent space. fi is the Rol pooling feature of b, extracted by a COCO pre-trained ResNet-50 (He et al. 2016). As usually imbalanced, conventional methods may suffer
the samples of different object categories from severe category bias (Wang et al. 2020a). For example, in OCL, animal accounts for 22% instances and home appliance only accounts for 3%. In P(a)T), category bias is imported following
P(a)1) P(al,,)))(0011
where P(O,)I) is the predicted category probability. That is, O confounder (Pearl et 2016) and pollutes attribute inference, especially for the rare categories (analyzed in 7
Causal Intervention. To tackle this problem, we propose OCRN utilizing intervention (Pearl al. 2016) to deconfound the confounder O for (Fig. 5). That said, in a estimation, we use do(.)
operation (Pearl et al. 2016) to eliminate the edge from O I, I,i..., P(aldo(I))
m
m
O(a(1.)))P(0))- P(O.) Σ
P(al,, P(PAL1)))(A100)) = Σ P(all, (3)
5

---

Under review as a conference paper at ICLR 2022
Dxpectation br c.
Aggregationn
Intervenckn
fo I Faltittne
P(aldo(1)
P(a
P(81La)
Fo and F't
self arlaatantoo
Expactation Po
Fp
fo - Pa
P(A(do(I.a))
The overview of OCRN. The edge from O to I is deconfounded. Thus can eliminate the bias from the imbalance. Equations below the graphs are the estimations of a, 8 w/ or w/o do(.). Attribute and affordance modules are the instantiations of category-level features. Mean object category feature fo, is converted to fA, or fB. by an FC, and then instantiated via (fr: fA,) or Fa(fr; f..). Causal paths marked in green are the conditions of F.) or Fa(-) implemented via attention policy, fo and fa after intervention are the expectations of instantiated Fa and Fa(Si, far (n.)) w.r. prior Po.. Linear-Sigmoid classifiers give the final predictions.
Figure 5:
where m 381. Aj is the category- attribute vector of jth category. As A is decided by O, (O,) 1 iif the category-attribute of jth category. P(O;) is the prior probability of the i-th category (frequency in the OCL train set).
and O.) 0 if * j. where O is the 7th category and A₁ is
4.2.1 ATTRIBUTE ESTIMATION
To implement Eq. 3 we use mean object category feature to represent category-level attribute A. We take the mean feature (fo;) of all instances in category the representation of the category (O+). It is then mapped to attribute feature space: fA, = Fa(fo.) ) where F-(-) is fully-connected layer (FC) (Fig. 5). The representation set of A fA {fA. J=1 For each category, we use fr of its all instances (train set) to calculate fo, in training and inference.
Attribute Instantiation. Next, we obtain a representation according Eq.
Ja = Σ Fa(r.dAa))P.
(4)
where Po; is the prior category probability (P(O.) Eq. 3. Eq, indicaaess the attribute instantia- tion from A to with I as the condition. Hence, we can equally translate the estimation problem into a conditioned instantiation problem. (.) is implemented with multi-heat head attention mecha- nism (Vaswani et al. (017). shown in Fig. S Both fA. and fi are fed to three independent linear transformations to obtain key, query, and value vectors K, J. and K', Q',, V'. The attention output so foftmar[[ Q17[K, K'//Vd(VV VJT where is the feature dimension. The output com- pressed by linear layer the instantiated representation F. fA,). The debiased representation fo is the expectation of F., (Jr, JA,,) w.r. Po, according to back-door adjustment (Eq. 3).
4.2.2 AFFORDANCE ESTIMATION
Similar to in B estimation, category bias also exists:
m72
(S)
With Eq. 3 a can be seen as "enforced" and deconfounded, as it is beforehand estimated, For I, we again use intervention (Pearl et al. 2016):
P( = Σ (
(6)

---

Under review as a conference paper at ICLR 2022
Affordance Instantiation. Similar to Eq. 3
P(B 1 lif j, P(B,IO.) = if #
omit the process for simplicity. An FC is used to obtain fB and Eq. 6 is implemented as:
m Folfi Solti
fB. =F((fo.
As shown in Fig. for causal inference operation (Sec.5). fa is first separated to attribute category- wise features fap by multiple independent FCs, where fag is the feature of pah attribute. The features are aggregated via
by an FC to fa. In this way, we can manipulate specific attributes of Sa by masking some certain Say. in order to evaluate the causal learning of our model. We compare different aggregation methods in the ablations. And fo = {fB. h Given the conditions {fr, fa fe, }, FF(() operate the instantiation. FB()), Fp(+) are implemented the same as Fa()), Fa except the first input condition of Fg is the concatenation of fa.
peneenatinn-compesssiiii
Learning Objectives. To drive the learning, we devise several objectives.
(I) Category-level loss Le. After obtaining the category-level representations LA. SB. we in- put them to two linear-Sigmoid classifiers 9A: 9B to classify A, B, given GT 0 in training: PA. = (fA,), ), Pn. = gn The binary cross-entropy losses of the i-th object category represented as TA and L.B.- So the total category-level loss is Lc = - + L..).
(2) Instance-level loss L1- As for the instance-level, we send fa. fa. together with the features before expectation Fa(fi, fa.), fa. to linear classifiers Ja and gg. The binary cross- entropy losses of these output logits are represented as La, Lg. The separated attribute features fap are also sent to independent binary classifiers fa, and computed losses with ground truth of pth attribute, which included T.a The instance-level loss then L,, I.a + L.g.
Finally, the total loss is L = AcLe + Lt. We adopt a two-stage policy: first inferring attributes, then reasoning affordances.
5 EXPERIMENT
Metric. For a, 33 learning, as an object can have multiple attributes and affordances (multi-label classification), we use mean Average Precision (mAP). For causal reasoning, we adopt TDE (Pearl et 20116) for evaluation. The general idea of TDE is to assess the effect of a variable: constructing the counterfactual scenario via eliminating the effect of the controlled variable, then subtracting the counterfactual from the original. Give an intuitive example, the effect of cargo can be measured by the speed difference between loaded car and an empty car. In OCL, we formulate TDE as the affordance probability change AP3. Here, we redefine AP3 as the difference between P P(2y) and p (T))(3,): (p E [1, 114], q E [1, 170]): comparing the original with its counterfactual without the effect of attribute ap. For an object, let P(5.) denote its original probability of B, estimated by trained model. As we use learned attribute feature to infer Sg. we assign zero-mask (Tang et al. 2020bb to the feature of Op and keep the other attribute features, to get PTDg(3.) with the same model parameters. After concurrently operating TDE for all [ap. Byl, we obtain huge cube of aestunce-attribut--fffie each instance. For each (Ap, By), We score instances following (Appendix Sec. A.4):
where each grid indicates the assessed effect of certain [app Ag) for
(1) TDE: If [app 3g] has annotated causality, when eliminating the effect of ap. P(5,)- (3,) should be large. If [Ap. 3.y] does not have causality, the difference should be small. Beside the change St
scale, the right change direction is also expected. We set TDE score TDE B, as:
TDE max((((,)) - ( GTg, 1,
(8)
max((PrD - 0), GTg, 0,
where GT', is the label of Ba-
(2) a-3-TDE: app. 3g should alse be infered accurately. We multiply TDE with P(ap GTa,)) )+ P(3, GT p.,) unified score. We compute AP for each (a,, 8yy] and average them to mAP.
Baselines. We list all baselines here briefly (detailed in Appendix Sec. A.5): (1) Direct Mapping
from S1 to Pa, P3 (DM-V); (2) DM from Attribute (DM-At); (3) DM from Attribute and Object
7

---

Under review as a conference paper at ICLR 2022
Method
a
3
IDE
a-B a-B-TDE IDE
DM-V
29.9
51.8
06AR
DM-L
21.2
47.5
MM
23.8
48.9
LingCorr
7.9
25.9
KPMF
25.4
49.1
Al---Lookup
18.9
30.9
28.6
51.7
DM-At
27.9
52.3
7.6
6.7
DM-AIO
28.0
52.2
8.1
7.0
Ngram
22.6
50.8
8.3
7.6
MIN-GT
33.4
9.5
9.1
(a) Proportion of correct TDE when GTa,
Attention
24.1
48.7
8.1
7.1
OCRN
31.6
53.3
9.5
9.2
DM-AI w/ LTDE
28.4
52.2
13.3
14,0
1.0
DM-AtO w/ Lypp
28.0
52.4
15.4
13.6
00.
Ngram w/ LTDE
22.2
49.9
14.1
12,9
MLN-GT w/ LYDE
33.7
12.3
11.8
05-
Attention w/ LTdE
23.9
49.0
17.8
15.5
OCRN w/ LIDE
31.5
53.6
20.3
16,9
Table 2: OCL results.
w/ LTDE indicates that
training with TDE loss. The baselines in the up- per block cannot operate TDE due to the model structure (Appendix Sec. A.5).
(b) Proportion of correct TDE when GTs, 1.
Figure 6: TDE performance different [app 3,1-
(DM-AtO); (4) DM from Bert (Devlin al., 2018) Linguistic Representation (DM-L); (5) Visual- Linguistic Representation, i.c., Multi-Modaltyy (MM); (6) Markov Logic Network (Rihhardson &
Domingos, 2006), using GT a infer (MLN-GT); (7) Linguistic Correlation of O-a, 0-3 (Ling-
Corr); (8) Kernliized Probabilistic Matrix Factorization (KPMF) (Zhou et al.) 2012); (9) A&B
Lookup, directly getting PA, PB from M'A, M'3 as Pa. P3: (10) Cause as Attention (Attention);
(11) Hierarchical Mapping from I to A/B then to a/B (HMa). (12) Ngram (Lin et al. 2012).
TDE loss. Though the machine expected to learn the causalities given a, labels only. We also wonder how it would perform given causal supervision. We adopt extra Hinge loss maximize the TDE score of all [ap, l. In detail, without the effect of one certain ap, we intend the probability of causal-related Bg to keep farther from GT&, by a margin 7 (= 0.1 in experiments), formulated
max(0, (P(3,) - PTbe((B)) GTg, = 1,
TDE 3
max max(0, (0, - (p P(3,)), GTa =.
We enumerate all annotated [atp, 3.1 of an instance to obtain LTDE. Different from the default, the total loss here is L Le + L, >TDELTDE.
Implementation Details. For a fair comparison, all methods adopt a shared COCO (Lin et al, 2014) pre-trained ResNet-50 (He al. (016)) (frozen) extract and use the same object boxes in training and inference. In OCRN, the dimension of // and all representations /A, [n, for Ja is 1024. The individual features of each attribute category is 512d and aggregated to 1024d by FC. We train the attribute module with learning rate of 0.3 and batch size of 1024 for 470 epochs. Then the attribute module is frozen, and the affordance module is trained with learning rate of 3.0e-3 and batch size of 768 for 20 epochs. In training, Ac = 0.03, ATDE = 3.
Results. Tab. presents the results. OCRN outperforms the baselines and achieves decent improve- ments on all tracks. In terms of a OCRN largely outperforms DM-V with 1.7 mAP (default) and DM-At with 3.1 mAP (w/LTDg). As for OCRN also achieves 1.0 mAP (default) and 1.4 mAP (w/LTDE) improvements compared to DM-At and DM-AtO w/LTDE. Comparatively, HMa utilizes the supervision of A, B, but performs much worse. A& Lookup directly uses GT A, B to infer a, B, but its poor performance verifies the significant difference between A, B and a, 8. Moreover, we find that all methods perform better on / than a. This may because a are more diverse than 3, e.g.,
eat lots of foods, but foods usually have various attributes (fruit Another reason is that the positive samples in 3 labels (23.2%) are much more than the positives in labels (9.4%). The different pos-neg ratio affect the learning lot and result in the above gap.
pizza).
In TDE evaluation, without the guidance of LTDES all methods achieve unsatisfied performances. However, OCRN still has an advantage. Only MLN adopting the first-order logic and GT a labels is
8

---

Under review as a conference paper at ICLR 2022
Mickl
Method
TDB
<-T-TDE
2013
wio dravenianneingg
12.1
18.3
16.1
wib La. L.s.
12.1
51.8
19.8
16.7
2007
with Ley Lis
10.0
27.0
16.6
16.4
12% Dims
11.7
51.5
150
16.0
1005
512. Dims
52.3
52.1
199
16.7
20:8 Dims
32.2
51.5
19.1
16.3
Meun eggregatian
32.2
31.3
189
16.7
300
Max pouling appregation
32.1
49.1
1900
16.8
Affondenne Class
Ransim foceteefaateal
30.4
51.8
51
5.1
Table 3: Ablation study results the Figure 7: Attribute bias (w/ Figure 8: Affordance bias (w/ and w/o deconfounding) for and w/o deconfounding) for category frying pan. category g ra ffe
val set.
comparable with OCRN. Iftaiiedd with LTDE and direct causality labels, all methods perform much better to learn complex causalities. Particularly, the typical deep learning model Attention performs best in baselines, but MI.N no longer holds the previous advantage. Relatively, OCRN shows greater improvements and outperforms Attention with 2.5 and 1.4 mAP on two TDE tracks. Due to the page limit, we provide more analyses in Appendix Sec. AA6 In Appendix Sec. [A.7 we further apply OCRN to Hbjent-bject Interaction Detection (Chao al. 2018), where OCRN boosts the performance
and verifies the efficacy and generalization of OCL.
Visualization. (1) In Fig. 6 we show the correct instance proportions (%) of OCRN and Attention
after TDE. (a) depicts the randomly chosen causal [ap. Sy] pairs with GT2, = 1 expecting P(3,) >
(Tbr(3,) (3,). (b) depicts the randomly chosen causal [ap; S,] pairs with GT 8, O expecting P(3,) <
PrDe (B,)). The higher proportions indicate that OCRN performs better than Attention TDE. (2)
To verify the deconfounding, we compare the model bias of OCRN w/ or w/o deconfounding. The bias of category O upon attribute a is measured following /hhao et al. (2017), by b(O,a) a) Ene e(O,,'' a'). When measuring data bias, c(O, a) is the number of co-occurrence of O and a in OCL, and when it comes to model bias, c(O, a) is the sum of probabilities that O are predicted positive with a. The bias of B is measured in the same manner. Fig. 7 and 8 show some examples of the biases of training data and models, indicating that OCRN deconfounding effectively prevents the model from bias toward the train set.
Ablation Study. We conduct ablation studies to verify the components of OCRN w/LTDE (Tab. 3 on the val set. (1) Deconfounding, OCRN w/o deconfounding is implemented following Eq. 2 and where PTO and a) are the category predictions of pre-trained detectors (Liu et al,
2021]. Both and 3 performances significantly decline, explained by Fig. and g (2) Losses. The performances slightly drop after removing category-level LA,, LB, but sigmificantly drop with- out instance- L,, Lg. (3) Feature dimension. For fA., fB. fai fg. smaller and larger feature sizes all have degrading effects. (4) TDE-telated implementations. We probe some different meth- ods: (a) Mean aggregation: S. = E; fay; (b) Max-pooling aggregation: S' is the max value of fap as each component; (c) Random counterfactual feature: assigned random vector as the counterfac- tual attribute feature (instead of zero vector) during TDE. These methods perform worse than the chosen setting on TDE performance but are comparable on a and 33 performance.
Overall, OCL poses extreme challenges current Al systems. expects representative learning to accurately recognize attribute and affordance from raw data meanwhile causal inference to capture the causalities within diverse instances and contexts, i.e., both the intuitive System I and logical System 2 (Bengio) 2019). From the experiments, we find that models struggle to achieve satisfying results on all tracks simultaneously. Notably, difficult to achieve a satisfying TDE score via data fitting, and there is much room for improvement. For future studies, harmonious performance on a, B, and causality learning are encouraged to capture object knowledge better. Potential directions may include causal representation learning (Schölkopf et al.] 2021). casual reinforcement learning (Bareinhoim, 2020), neural-symbolic reasoning (Besold et al. 2017). etc.
Discussion.
6 CONCLUSION
In this work, we introduce object concept learning (OCL), which expects machines to infer affor- dances and explain what attributes enable an object to possess these affordances. Accordingly, we build extensive real-world dataset and present a baseline OCRN based casual intervention and instantiation. OCRN achieves decent performance and follows the causalities well. However, OCL remains challenging and would inspire line of studies upon reasoning- object understanding.

---

Under review as a conference paper at ICLR 2022
7 ETHICS STATEMENT
This work primarily proposes a daily object dataset to advance machine understanding of objects for the future study of embodied AI, intelligent robots, etc. Our work may help the home service robots better understand the human living scenes and then help to improve our lives, especially for the health care of patients, elders, and children. All the data used in OCL are publicly available. We have cited the creators and followed the official licenses. For all images, we only provide the URL links in our future web for research uses only. Moreover, modem deep learning models are usually computationally expensive. Our proposed model OCRN also needs multiple GPUs to train. In the future, OCL encourages new line of studies to understand objects' common sense and combine causal reasoning and deep learning. We believe the future works focusing on learning the causality instead of heavily fitting data would more computationally lite, less data-hungry, and low-carbon.
REPRODUCIBILITY STATEMENT
We provide our source code, OCL dataset samples, and a video demo in the supplementary material (the zip file). It includes all the manuscripts to preprocess and load data, construct the OCRN model with the myper-parameters train and the model, evaluate the performance on the OCL benchmark, etc. Please refer to the code file for more details. The details of related datasets, OCL data processing and annotation are in the main text Sec. Appendix Sec. A.3 A.11 Our code and data will be publicly available.
REFERENCES
Yuval Atzmon, Felix Kreuk, Uri Shalit, and Gal Chechik. A causal view of compositional zero-shot recognition. arXiv preprint arXiv:2006. 14610, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
Elias Bareinboim. Towards causal reinforcement learning (crl). Tutorial at ICML, 2020.
Yoshua Bengio. From system I deep learning system deep learning. In Posner lecture at NeurlPS'2019 2019.
Tarek R Besold, Artur d' Avila Garcez, Sebastian Bader, Howard Bowman, Pedro Domingos, Pas- Hitzler, Kai-Uwe Kühnberger, Luis C Lamb, Daniel Lowd, Priscila Machado Vieira Lima, et al. Neural-symbolic learning and reasoning: A survey and interpretation. arXiv preprint arXiv:1711.03902, 2017.
Krzysztof Chalupka, Pictro Perona, and Frederick Eberhardt. Visual causal feature learning. arXiv preprint 1rXiv:1412.2309, 2014.
Yu-Wei Chao, Zhan Wang, Rada Mihalcea, and Jia Deng. Mining semantic affordances of visual object categories. In CVPR, 2015.
Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning detect human-object interactions. In WACV, 2018.
Chao-Yeh Chen and Kristen Grauman. Inferring analogous attributes. In CVPR, 2014.
Ching- Yao Chuang, Jiaman Li, Antonio Torralba, and Sanja Fidler. Learning to act properly: Pre- dicting and explaining affordances from images. In CVPR, 2018.
Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, and Zeb Kurth-Nelson. Causal reasoning from meta-reinforreement learning. arXiv preprini
2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint a 1810.04805 2018.
10

---

Under review as a conference paper at ICLR 2022
Thanh-Toam Do, Anh Nguyen, and Ian Reid. Affordancenet: An end-to-end deep learning approach for object affordance detection. ICRA, 2018.
Ali Farhadi, lan Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In CVPR, 2009.
Christiane Fellbaum. Wordnet. The encyclopedia of applied linguistics, 2012.
David F Fouhey, Xiaolong Wang, and Abhinav Gupta. defense of the direct perception of affor- dances. arXiv preprint arXiv: 01085, 2015.
Chen Gao, Yuliang Zou, and Jia-Bin Huang. ican: Instance-centric attention network for human- object interaction detection. In BMVC, 2018.
James Gibson. The ecological approach to visual perception: classic edition. Psychology Press, 2014.
Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. Ava: A video dataset of spatio-temporally localized atomic visual actions. In CVPR, 2018.
Abhinav Gupta and Larry S Davis. Objects in action: An approach for combining action under- standing and object perception. In CVPR, 2007.
Saurabh Gupta and Jitendra Malik. Visual semantic role labeling, arXiv preprint arXiv:1505.04444, 2015.
Isabelle Guyon, Constantin Aliferis, and André Elisseeff. Causal feature selection. Computational methods of feature selection, pp. 63-82, 2007.
Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335- 346, 1990.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Tucker Hermans, James M Rehg, and Aaron Bobick. Affordance prediction via learned object attributes. In ICRA Workshop. 2011.
Drew Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Sung Ju Hwang, Fei Sha, and Kristen Grauman. Sharing features between objects and their at- tributes. CVPR, 2011.
Phillip Isola, Joseph Lim, and Edward H Adelson. Discovering states and transformations image collections. In CVPR, 2015.
Keizo Kato, Yin Li, and Abhinav Gupta. Compositional learning for human object interaction. In ECCV, 2018.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJC,, 2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NIPS, 2012.
Krishna Kumar Singh, Santosh Divvala, Ali Farhadi, and Yong Jac Lec. Dock: Detecting objects by transferring common-sern knowledge. In ECCV, 2018.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In CVPR, 2009.
11

---

Under review as a conference paper at ICLR 2022
Karel Lebeda, Simon Hadfield, and Richard Bowden. Exploring causal relationships in visual object tracking. ICCV, 2015.
Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma. Hao-Shu Fang, Yanfeng Wang, and Cewu Lu. Transferable interactiveness knowledge for human-object interaction detection. In CVPR, 2019.
Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu. Hoi analysis: Integrating and decomposing human-object interaction. In NeurIPS, 2020a.
Yong-Lu Li, Yue Xu, Xiaohan Mao, and Cewu Lu. Symmetry and group in attribute-object compo- sitions. CVPR, 2020b.
Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Y. Lin, J.-B. Michel, E. Aiden, J. Orwant, W. Brockman, and S. Petrov. Syntactic annotations for the google books ngram corpus. In ACL, 2012.
H. Liu, R. Wang, S. Shan, and X. Chen. Learning multifunctional binary codes for both category and attribute oriented retrieval tasks. In CVPR, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030, 2021.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using (-sne. JMLR, 2008
Dhruv Mahajan, Sundararajan Sellamanickam, and Vinod Nair. A joint learning framework for attribute models and object descriptions. In ICCV, 2011.
Alex Martin. The representation of object concepts in the brain. Annu. Rev. Psychol., 58:25-45, 2007.
Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composition with context. In CVPR, 2017.
Tushar Nagarajan and Kristen Grauman. Attributes operators: factorizing unseen attribute-object compositions. In ECCV, 2018.
Suraj Nair, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Causal induction from visual observations for goal directed tasks. arXiv preprint arXiv:1910001751, 2019.
Anh Nguyen, Dimitrios Kanoulas, Darwin G Caldwell, and Nikos G Tsagarakis. Object-based affordances detection with convolutional neural networks and dense conditional random fields. In IROS, 2017.
Daniel Osherson, Joshua Stern, Ormond Wilkie, Michael Stob, and Edward E Smith. Default probability. Cognitive Science, 15(2):251-269, 1991.
Devi Parikh and Kristen Grauman. Relative attributes. In ICCV, 2011.
Genevieve Patterson and James Hays. Coco attributes: Attributes for people, animals, and objects. In ECCV, 2016.
Judea Pearl, Madelyn Glymour, and Nicholas Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016.
Lyndsey C Pickup, Zheng Pan, Donglai Wei, YiChang Shih, Changshui Zhang, Andrew Zisserman, Bernhard Scholkopf, and William T Freeman. Seeing the arrow of time. In CVPR, 2014.
Lerrel Pinto and Abhinav Gupta. Supersizing self-supervisiin: Learning to grasp from 50k tries and 700 robot hours. In ICRA, 2016.
12

---

Under review as a conference paper at ICLR 2022
Lerrel Pinto, Dhiraj Gandhi, Yuanfeng Han, Yong-Lae Park, and Abhinav Gupta. The curious robot: Learning visual representations via physical interactions. In ECCV, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.
Matthew Richardson and Pedro Domingos. Markov logic networks. Machine learning, 62(1-2): 107-136, 2006.
B Ross. Category learning: Learning to access and use relevant knowledge. Memory and mind: A Festschrift for Gordon H. Bower, pp. 229-246, 2008.
Anirban Roy and Sinisa Todorovic. A multi-scale cnn for affordance segmentation in rgb images. In ECCV, 2016.
Bernhard Schölkopf. Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 10965)611--33, 2021.
Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search. MIT press, 2000.
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. arXiv preprint arXiv:2009.1299 129991, 2020a.
Kaihua Tang. Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. CVPR, 2020b.
Spyridon Thermos, Georgios Th Papadopoulos, Petros Daras, and Gerasimos Potamianos. Deep fefordance-groundde sensorimotor object recognition. In CVPR, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Tan Wang, Jianqiang Huang, Hanwang Zhang, and Qianru Sun. Visual commonsense r-cnn. In CVPR, 2020a.
Xiaolong Wang, Rohit Girdhar, and Abhinav Gupta. Binge watching: Scaling affordance learning from sitcoms. In CVPR, 2017.
Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In CVPR, 2020b.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. CVPR, 2010.
Bangpeng Yao, Jiayuan Ma, and Fei-Fei. Discovering object functionality. ICCV, 2013.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv: 2017.
Yibiao Zhao and Song-Chun Zhu. Scene parsing by integrating function, geometry and appearance models. In CVPR, 2013.
Tinghui Zhou, Hanhuai Shan, Arindam Banerjee, and Guillermo Sapiro. Kernelized probabilistic matrix factorization: Exploiting graphs and side information. In SDM, 2012.
Yuke Zhu, Alireza Fathi, and Li Fei-Fei. Reasoning about object affordances in a knowledge base representation. In ECCV, 2014.
13

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
We report more details and analyses about OCL in the appendix:
Sec.
A.1
Category/AtribueeAAfffooeee Selection
Sec.
Annotation Details
Sec.
OCL Characteristics
Sec. 4
TDE Metric Details
Sec. A.5
Baseline Details
Sec. A.6
Detailed Result Analysis
Sec. A.7
Application on Human-Object Interaction (HOI) Detection
Sec. A
Comparison Debiasing
Sec.
About States
Sec. A.10
Discussion about Causality and Causal Graph
Sec.
Detailed Lists
A.1
CATEGOR/AAATRRBUTEFFFHHDDAA SELECTION
others
buildding
7%
food drink
22%
plant
5%
musical instrument
38
vehicle
furniture
too
17k
home eppliance
clothing 9%
equipment
15K
Figure 9: Super-categories of objects in OCL.
We first choose affordances, categories, and attributes, considering their causal relations. We present their word cloud in Fig- 10 The complete lists can be found in Appendix Sec. A.11
(1) Affordance: To build a general and applicable knowledge base, we collect 1,006 affordance
candidates from several widely-used action or affordance datasets (Chao et al., 2015 Gu et al.)
2018 Chao et al. 2018 Gupta Malik 2015 Zhu et al. 20144] Nguyen et al.
2017) (957 from
Chao et al. (2015). 160 from Gu et al. (2018) 8), 146 from Chao al. (2018), 97 from Gupta & Malik (0015), from Zhu et al. (2014)
from Nguyen et al. (0117) with overlaps). We find that and some of them difficult for visual recognition, e.g., accept (consider right and proper). So each candidate is scored by 5 human experts from 0.0 to 5.0 according to generality and commonness. We keep 170 top-scored affordances in our base (134
not all affordances
in common
14

---

Under review as a conference paper at ICLR 2022
Not peespranate varesint
poptterlly ly sculptuuu loture pen
vate inspect 1886 push pho Lograph Sattat S Maatebuu bru.hbruyoooe reaobic
toee goatyehicl weeecille guitas caatee
d brown, gray fluryy
couch
mas
Mfooee moa 20n parachis
snooth painte
black Bott
drag oat dy aeso hal I hi gan Ch MP
ric.
truppe sooel LTd dnd box-oox
parked
de de Carry glean
Toak bo Wedhing og rachine 24e aur Leaffi light
B ob buy Saptare
uisd] 1 aard stee ard miree eistbu deetiiin zebra
tent ruck au shay
p2a00 FUx
ghboat
frosh
who Lewh
Turry
uped Solid
ni Fa se Louch
pull
Stice
(a) Category
(b) Attribute
(c) Affordance
Figure 10: Word clouds of categories, attributes, and affordance (by positive frequencies in OCL).
from Chao al. (2033), 78 from Gu al. (2018), 127 from Chao et al. (2018), 53 from Gupta & with overlaps).
Malik (215), 13 from Zhu et al. 2014). 11 from Nguyen et al.
(2017 1)17)
(2) Category:
Considering the taxonomy (WordNet (Fellbaum 2012
we collect a pool with over
1,742 object categories from previous datasets (Farhadi et al., 2009 Xiao et al., 2010 Patterson
& Hays 2016, Liu et al. 2017) 7 (32 from Farhadi et al. (009). 28 from Patterson & Hays (2016),
717 from Xiao et al. (010)) 1000 from Liu et al. (2017). with overlups). Then we merge the similar categories according to WordNet (Fellbaum 2012). and filter out the categories which are not common daily objects (e.g. "man". "planct"), ). unrelated to the above 170 affordances (c.g.
"skyscraper") or too uncommon (e.g. "malleefowl"). Finally our database has 381 common object categories. These object categories are divided into 12 super categories, showed in Fig. 2
(3) Attribute: We extract the attribute classes from several large-scale attribute datasets (Farhadi
al. 2009 Xiao al.] 2010 Patterson & Hays| 2016 Liu et al.] [017]] Krishna al.] 2016) (64
from Farhadi et al. 2009)), 203 from Patterson & Hays (016)), 66 from Xiao et al. (2010). 25 from
Liu et al. (217)). top 500 from Krishna et al. (2016)) and manually filter the 500 most frequent attributes. Five experts give 0.0 to 5.0 scores based on their relevance to human actions and the
selected 170 affordances to better explore the causal relations between attributes and affordances. Some attributes (e.g. cloudy, compet i tive that are not helpful for affordance reasoning are discarded. Finally, 114 attributes are kept, covering colors, deformations, supercategories, various surface, geometrical, and physical properties.
A.2 ANNOTATION DETAILS
Attribute Annotation.
(1) Category-level attribute (A). Following Osherson et al. (199I), to avoid bias, annotators are given category-attribute pairs (category names, not images). They propose a 0-3 score according to the category concept in their minds (0: No, 1: Normally No, 2: Normally Yes, 3: Yes). Each pair annotated by three annotators and takes the plurality as A label. If the range of 3 proposals exceeds 1, another three annotators will re-annotate this pair until achieving consensus. We binarize the annotations (0: No, I: Yes) with threshold of and get category- attribute matrix MA (381, 114).
(2) Instance-level attribute (a). Two annotators label each pair with (No) and 1 (Yes). If they give different labels, this pair will be handed over to another two annotators until meeting consensus.
Affordance Annotation.
(1) Category-level affordance B. Following Chao et al. (015). the annotators are given category- affordance pairs. The pairs are annotated in four bins (0-3) and normalized (same as A) describe the possibility of an affordance in category. Each pair is annotated by three annotators and makes consensus the same as A. The 0-3 scores are binarized (1: Yes, No) with threshold of 2. The final category-level affordance matrix Mn is 1381,170|.
(2) Instance-level affordance 3 is annotated for every instance with the help of object states [Isola et al. 2015). As B is determined by common states, objects in specific states may have different affordances from B, e.g., we cannot board a flying plane. Moreover, if service robot finds a broken cup, it may infer that the cup can still hold water as it is trained with B labels. Thus, we need detailed 3 beyond B. As the instances in the same state should have similar 3 (all rotten apples cannot be eaten), six experts first conclude the states. The experts scan all instances of each category and use their knowledge of affordance to define all the existing states. Then all 186 K instances are dispatched to the concluded states via crowdsourcing. If some instances do not belong
15

---

Under review as a conference paper at ICLR 2022
any predefined states, they will be returned to the experts to add more states. In total, 1,376 states are defined, and each category has 3.6 states on average. Next, 3 is annotated for each state. Given state state-affordacce pair and example images, two annotators mark it with 0 (No) and 1 (Yes). The results are combined in the same way as a. Thus, each instance would have a state and the corresponding B3. An annotator would recheck each instance together with its state and labels to ensure the quality, If its state is inaccurate or the state 3 labels are unsuitable, this annotator would correct them.
Causal Relation Annotation.
(1) Filtering. Starting from the [114,170] matrix of a-8 classes, ask three experts vote the causal relation of each class. They scan all instances to answer whether the relation exists in any cases, Finally, obtain about 10% a-3 classes candidates.
(2) Instance-level causality: we also adopt object states as a reference. For each state-a-3 triplet, two annotators are asked whether the specific attribute is the direct cause of this affordance this state and gives their binary answer. We use the same method in annotating 3 to combine results and state, an expert decides whether the
assign state-level labels to instances. Next, for all instances state-level relations are reasonable for each instance in specific contexts and correct the inaccurate ones. Finally, we obtain about M instance-a-$ 3 triplets of causal relations.
A Rumning Example of Dataset Construction.
running example is shown Fig. to show the process of annotations clearly.
Collect images (banana)
Detect objects
Propose states
banana model/toy
Annotate direct causal relations
Annotate attributes and affordances
fake V furry v tasty
touch v
fake v furry tasty
touch v
eat x
eat x hold v
hold v
banana
fake V-
eat
banana model/toy
model/toy
Figure 11: A running example of dataset construction.
A.3
OCL CHARACTERISTICS
A.3.1 OBJECT BOX SIZE
We visualize the distribution of normalized object box size in Fig. 12] where the box width and height are normalized by the width and height of the whole image. shows that most objects in our knowledge base are small objects, providing abundant regional information.
A.3.2 ANNOTATOR INFORMATION
Annotators' age, major, and education degree are presented in Fig.
14 and 15
16

---

Under review as a conference paper at ICLR 2022
3900
3000
2300
2000
2900
1500
1500
1000
500
500
Figure 12: Distribution of normalized object box width (left) and height (right).
4g.35
Minde
cos
 O
50%
Aps21 345
Figure 13: Age information of Figure 14: Major information annotators.
Figure 15: Degree information of annotators.
of annotators.
A.3.3 MATRIX SAMPLES
The category-level attribute and affordance (A, B) matrices are detailed in Fig. T6 as heatmaps, the cells with dark color indicate positive samples. For example, ice cream is cold while clock is not natural, cake can be eaten while eraser can not be cooked. These are in line with our common sense.
A.3.4 STATE DISTRIBUTION
Before annotating the affordances, we first define the object states for all object categories and annotate the state affordances. In total, we define 1,376 states for 381 object categories. The state list can be found Appendix Sec. A.11. And Fig. 18 shows the state distribution per object category.
A.3.5 ATTRIBUTE AFFORDACE RELATION
We analyze the instance-level attribute-affordance relations in our knowledge base under three cri- teria. (1) Attribute Conditioned Affordance Probability. It's computed P(3]a) estimate affordance probability given an attribute. The range is [0,1]. (2) Attribute-Affordance Correla- tion. For all instances our dataset, we evaluate the label correlation of each -ffouddae-ffoo pair, whose scale is in [- (3) Attribute-A Affordancff Causality. Start with the annotated cause- effect (a - 3) labels, we count for how many times each attribute-affordance pair appear in our dataset and normalize the value by the maximum occurrences, leading to value in the range [0, I]. should be mentioned that we only annotate whether an attribute-affordamn pair has explicit and key causality, but the detailed effect (positive negative) should be referred to instance labels.
We visualize the samples of attribute-affordame some interesting properties of them. They reveal some common relations, such what between tasty and eat. However, some criteria suffer from data bias. For the condition matrix in Fig. [9 only cares about cases with positive attribute labels, which is not good in highlighting the negative
relation matrices in Fig. 19 20
21 and observe
17

---

Under review as a conference paper at ICLR 2022
truck
cake
clock
guitar
ice cream
lemon
sheep
rabbit
wheel
aircraft
panda
coffee
drill
eraser
apple
broccoli
sarong
ski mask
wallet
lasty
dry
cloth
fresh
metal
cold portable
natural
digital
round
chase
eat
collect
herd
carry
wear
break
drink
clean
cook
Figure 16: Category-Ievel attribute (A) matrix. Figure 17: Category-level affordance (B) matrix.
truck
13
12
11
Number of states
10
bagel
torch
bed
3-
2-
Object Category
Figure 18: State distributions of different object categories.
relations, e.g. the relation between natural and produce. For the former two matrices Fig. 20 they all point out the relation between tasty and pick, since most tasty objects are pickable food. This finding is simply misled by the data bias but violate the causal graph (inferece from attribute to object category, then affordance). Last, the matrix obtained from causal annotation in Fig. 21 is more sparse, clear of causality.
A.3.6 UNIFIED OBJECT REPRESENTATION
To compare attribute-only and attribute-affordaacee representation abilities, we cluster the object instances of two similar animals (zebra and horse) with their attribute labels and attribute-afffrdaace labels, respectively. The results are shown in Fig. 22 via t-SNE (Maaten & Hinton 2008). With both attribute and affordance labels, zebra and horse can be better separated than attribute only. And attribute and affordance together can differentiate specific states well, such as r riding, pull pulling etc.
18

---

Under review as a conference paper at ICLR 2022
tasty
tasty
Mul
050
moving
moving
031
moving
walking
walking
walking
natural
natural
natural
eat
stop
laad
pick
lyiite
prodtce
eat
stop
load
pick
kgnita
eroduce
eat
sto
lawn
pick
ggite
producee
Figure 19: Attribute condi- tioned affordance matrix.
Figure 20: Atribute-affordance correlation matrix.
Figure 21: Attribute-affordan causality matrix.
-40
20033
(a) Attribute Labels.
(b) Labels.
Figure 22: Clustcring using attribute and attribute-afooddan labels.
A.3.7 DIFFERENCE BETWEEN CATEGORY- AND INSTANCE-LEVV LABELS
We analyze the differences between category-level A, B labels and instance-level a, 3 labels. For each object category, we compute the average ratio of changed attribute/affordancee classes during each instantiation from A to differences between A and a as well as and B are reported respectively in Fig. 23 We find that affordance labels change more dramatically than attribute labels during instantiations. This is because each attribute change may affect several affordancees, e.g., when a common book becomes wet, we neither open nor read it.
from B B. The top-50 categories with the most significant
0.
344
3.46
3.44
035
3.6
(a) Difference between A and a labels.
(b) Difference between B and / labels.
Figure 23: Top-50 object categories with largest ratio of difference between category- and instance- level labels.
19

---

Under review as a conference paper at ICLR 2022
A.3.8 ATTRIBUTE-
CAUSAL RELATIONS
We annotate all object instances' causal relations of filtered [0p, 8,] pairs. In total, 1,085 [[pp 9g] pairs are chosen for the causality annotation, and over 2 M instance-a-8 triplets are annotated. In TDE evaluation (main text Sec. (). report the mean AP of top-300 [ap, Sgg pairs to avoid the biased influence of very rare [Op, ply] pairs that include less than 35 object instances.
A.3.9 DATA PARTITIONING
For the OCL task, our knowledge base is split into train, val, and test sets. The statistical details of the split are listed in Tab. 4
The image number ratio of the three sets is nearly 4:1:0.6, and the
instance ratio is around 5:1:1.
Set
Image
Object Instance
Object category
Train
56,916
135,148
381
Val
14,446
25,176
221
Test
9,101
25,617
221
Val+Test
23,547
50,793
221
All
80,463
185,941
381
Table 4: Detailed data split of our knowledge base.
A.3.10 IMAGES AND INSTANCES
Some additional data samples of our knowledge base are shown in Fig. 24 25a 25b 26 and 27 including samples of diverse object categories with various bounding box distributions, different attributes and affordances, and human-labeled object states and obvious causal relations.
We also detail the image and object instance counts in Fig. 28
29
and 30
A.3.11 MORE STATISTICS OF ANNOTATION
We divide A, B, a, B, causality annotation into multiple finer-grained small sets in our pipeline. Generally, we have 13, 19, 124, 140, 85 annotators-sets (381 totally) for A, B, o, 3, causality anno- tation respectively. We assign each small set to annotators. However, considering the controversial situations introduced, part of the annotation are confused cases based on their results. In the whole process, 9.6% of A, 7.7% of B, 5.2% of a, 7.9% of and 13.7% of causality are confusing and re-assigned to additional annotators. These indeterminable ones will be sent to two extra two anno- tators until making an agreement. The quality of the dataset is guaranteed by a low confusion ratio and multiple refining stages.
12 POTENTIAL BIAS
We have considered the bias issue in the construction of our dataset. 1. In our dataset, the existing datasets (ImageNet[Deng et al. (2009), COCO et al. (014)) Farhadi al. (2009), SUN Xiao et al.] (0100) are open-sourced datasets and the images collected from Internet are publicly accessi- ble too. The dataset is constructed for only non-commercial purposes. We will only provide URLs of these images to avoid copyright infringement. 2. During image collection, with general objects and are particularly careful with the images selection to avoid unsuitable con- tent, private images, implicit biases. 3. During annotation, the annotators cover different genders, ages, and fields of expertise to avoid potential annotation biases. And they are all informed how we will the annotations in our research.
choose images
20

---

Under review as a conference paper at ICLR 2022
apple
bicycle
bird
boat
cake
car
cat
horse
sandwich
Figure 24: More OCL samples of objects.
A.4 TDE METRIC DETAILS
TDE to measure whether model infers affordance with proper attention to the causality-related attribute. That said, when removing the attribute, the model is expected to have large prediction difference further away from the ground truth. We detail some settings in our TDE metric here. For the TDE score:
max mmx((P(B)) (3,), GTg, 1, P(B,),
TDE O B.
(10)
max (PT P 0), GT = =0
we want the affordance probablity change direction is right according to the GT affordance labels. Concretely, for an instance with the labeled causal relation between [ap. if the label GT3, 1, we expect the probability change P(8,) - p to be larger after eliminating the effect of ap- Because without the effect of Op, the probability of B, should be contrary to the fact (GT', 1), i.e. PrDE(3 should be much smaller than P(8g). Similarly, if its GTg, O. expect PfD (3,)
21

---

Under review as a conference paper at ICLR 2022
BE WIHH
broken
cookable
bent
drinkable
empty
feedable
framed
functionable
furry
holdable
hanging
ignitable
round
paintable
tasty
rideable
vertical
sittable
wooden
wearable
(a) attributes
(b) affordances Figure 25: More OCL samples of attributes and affordances
Object States
Images
Attributes
Affordances
parked v
touch
caged/hold
walking x
pull x
chase
fake v
drive
bus toy
rectangular
lift /
moving x
load x
tasty
baked goods raw
broken cook
wet v
wash x
potato mashed
whole cook v
solid kick
metal v
drive x
motorcycle repairing
parked check v
moving drag x
Figure 26: More OCL samples. We present objects different states, together with their key tributes and affordances.
should be much larger than P(A,). That said, P (3,) - P(B,) should be large enough to follow the causal relation. Furthermore, TDE loss, the setting follows this thought too.
22

---

Under review as a conference paper at ICLR 2022
apple
fresh V
eat v
cattle
parked v
stop x
bird
dead x
feed v
pizza
heavy x
flip v
sheep
furry V
shear V
Figure 27: More OCL samples of direct causal relations
Mrd
dlghant
monbe
Snverictrate
flas
1061
funsture
2008
4000
8000
8030
30000 12300
14000
20023 2300
Figure 28: Counts of object categories.
Moreover, in a-3-TDE, P(a, GTa,) means that the probabilities of predicted P(a.)) accords with the label GT, a. In detail, if GT = 1. then P(a, = GTa,) = P(a,): if GT, = 0, then P(a, GTa,) 11-1-PP P(a,), where P(ap) the model prediction. P(S, GT,, is similar. That said, if GT,, 1, then P(a, GT P(8g); if GT2, = 0, then P(βg GT2,) = P(Bg)- Finally, TDE P(a, GTa,) * P(g, GT,) indicates the unified performance of attribute, affordance recognition and causal inference.
23

---

Under review as a conference paper at ICLR 2022
PHrZE
Mi
spodted
2330
403K0
60930
32306
32000
Figure 29: Counts of attribute classes.
2011
DN
1835
2300 4300
1030
B33K0
3000C9
20000 32066
30X0C
30300
Figure 30: Counts of affordance classes.
For attribute/affordaace recognition only, all methods adopt the labels to learn knowledge from the data. Moreover, in the evaluation of causal relation, only the "w/ LTDE" adopts the causal relation labels. We hope the models can self-learn to mine and follow the intrinsic causalities. Thus, design the TDE operation to evaluate this ability. Some works Wang al. (2020a); Tang ett al. (2)00a]b) also try to marry supervised deep learning and causal inference which are similar OCRN method. Besides, current techniques especially the deep learning methods perform unsatisfying in our OCL task, and all show poor ability to capture the causality. We think our main aim and contribution here is reveal this important problem via OCL dataset instead of directly addressing the task we propose. A lot of works should be done to prompt this field which we hope our OCL can support.
A.5 BASELINE DETAILS
We introduce the details of all baselines here:
(1) Direct Mapping from Visual Feature (DM-V): feeding fı into MLP-Sigmoids to predict P3. Each a and 3 class owns customized MLP followed by LayerNorm Ba et al. (2016) to generate class-specific feature and shares same MLP-Sigmoid in classification.
24

---

Under review as a conference paper at ICLR 2022
(2) DM from Attribute (DM-At): training classifier with fo same with DM-V, but use the concatenated representation of attributes as fa to train the 3 classifier.
(3) DM from Attribute and Object (DM-A10): training
classifier with fr same with DM-V, but use the concatenated representation of attributes fa and objects fo to train the / classifier.
(4) DM from Linguistic Representation (DM-L): replace the input representation f1 of DM-V with linguistic feature fL, which is the expectation of Bert (Devlin et al. 2018) of category names w.r. (T):
(5) Multi-Modality (MM): mapping fr to the semantic space via minimizing the distance to its fa. The multi-modal aligned fi is fed to MI.P-Sigmoids to predict Pa, Pg-
(6) Markoy Logic Network (MLN-GT) (Richardson & Domingos 2006): adopt MLN to model the 3 relations following Zhu al. (2014). After training on OCL, infer 3 with GT to estimate its performance upper bound.
(7) Linguistic Correlation (LingCorr): measure the correlation between object and a/3 classes via their Bert (Devlin et al., 2018) cosine similarity. Pa. P3 are given by multiplying P(O to correlation matrices.
(8) Kernelized Probabilistic Matrix Factorization (KPMF) Zhoou et al. 2012): calculate the Softmax normalized cosine similarity between each testing instance and all training samples as weights. Then Pa or P3 is generated as the weighted sum of GT or 3 of training samples.
(9) A&B Lookup: return the expectation of category-level attribute or affordance vectors A,, B, w.r.t (T)
(10) Attention: map the A and fr into a shared latent space then use Sigmoid convert fi into attention to generate instantialized fa followed by MLP Mig-Sigmoids to predict a. As for B, we use the concatenated representation fr and fa to generate attenion on B.
(11) Hierarchical Mapping (HMa)) first mapping fi to category- attribute or affordance space by an MLP supervised by GT A or B. Then the mapped features are fed to an MI.P-Sigmoid to predict Pa or Pa.
(12) Ngram (Lin al., 2012): Ngram is adopted retrieve the relevance between a and 3 and generate association matrix Ma-3. Then we multiply DM predicted Pa with Ma estimate Ps.
Besides, the TDE calculation needs feature zero-masking to eliminate the effect of specific at- tributes (Tang al., 2020b). Thus, some baselines (DM-V, DM-L,, MM, LingCorr, KPMF, A& B- Lookup. HMa) cannot be evaluated with TDE. The other methods (DM-At, DM-AIO, Attention, OCRN) follow the same TDE calculation (feature masking). Two unique cases are Ngram and MLN- GT. Ngram uses attribute probabilities infer affordance. Thus, randomize the specific attribute probabilities for Ngram to operate the TDE calculation. And MLN-GT must use GT attribute labels distinguish the "positive" and "negative" causes and then reason out the effect affordance. Thus, in TDE, we directly eliminate its corresponding attribute input.
A.6 DETAILED RESULT ANALYSIS
A.6.1 DETAILED ATTRIBUTE AND AFFORDANCE PERFORMANCES
We compute and analysis the performance (AP) of OCRN on each attribute or affordance class in Fig. 31 and Fig. 32 which suggest that visually abstract concepts like fake are more difficult to model than concrete ones like metal, breakable. The performance of attribute classes is lower than affordance classes. This is mainly because the attributes have more diversities. Thus the positive instances of each attribute class are less than the affordance class.
A.6.2 ATTRIBUTE AND AFFORDANCE RECOGNITION GIVEN DETECTED BOXES
Though OCL is a high-level concept learning task with object boxes as inputs, also consider object detection evaluation for practical applications. We adopt Swin transformer (Swin) (Liu et al. 2021) as the detector. It is pretrained on COCO (Lin et al. 2014) and finetuned on OCL train
25

---

Under review as a conference paper at ICLR 2022
1047
Figure 31: AP of attribute classes.
Figure 32: AP of affordance classes.
set with GT boxes of 381 categories. On OCL test set, it achieves 22.9 APoo on object detection. Subsequently, will provide detected box b, for all models inference. We can consider the detection effect in the attribute and affordance recognition metric to build more strict criterion. Namely, all false positive detections (loU<00.3 with refer to GT boxes) as the false positives of a and /8 recognition too. Morcover, TDE calculation needs to construct the counterfactual of an object instance. If the inaccurately detected object box shifts according to the GT box, difficult to know whether the counterfactual comes from the attribute masking visual content change, using the corresponding strriute-affordanc causal relation labels of this GT box. Thus, considering the unique property of causal inference different from common recognition, here do not report the TDE score. Tab. [] shows the results given detected boxes. Due to the more strict criterion and detection quality, the performances of all methods degrade greatly. But OCRN still holds the superiority on two tracks.
Method
DM-V
7.4
11.0
DM-L
MM
5.4
9.9
LingCorr
1.7
5.6
KPMF
6.4
10.5
B-oouuo
4.1
5.8
HMa
6,5
10.9
DM-At
6.8
10.5
DM-AIO
Ngram
5.1
10.2
MLN-GT GT
Attention
5.5
10.1
OCRN
7.9
11.3
Table 5: Attribute and affordance recognition results given detected boxes from Swin Trans-
former (Luu et al. 2021
A.6.3 OCL-BASED IMAGE RETRIEVAL
We visualize the OCL reasoning performance by retrieval the top-score instances with OCRN. Some results shown in Fig. 34 The model can correctly retrieve the related images, especially on some common concepts e.g. co l Lumarr, sit.
A.7 APPLICATION ON HUMAN-OBJER INTERACTION (HOI) DETECTION
To further verify the generalization ability, we apply OCL to Human-Object Interaction (HOI) de- tection and help HOI methods boost their performances.
HOI depicts the actions performed upon objects by humans. Usually, an object has multi-affordance, i.e., person can perform different actions upon it. But in an image, just one several ac- tions/affordances are usually happening/acttvated. Without object knowledge, previous methods (Li et al. 2019 Gao al. 2018 Li al. 2020a) find the activated affordances from hundreds of actions (Chao et al. 2018). For example, for each human-object pair in HICO-DET (Chao et al. 2018), model has to select
several actions from the defined 116 actions. With OCI, things
26

---

Under review as a conference paper at ICLR 2022
Top-5 Attribute Retrieval with OCRN
Top-S Afforclance Retrievvl with OCRN
Columnar
Capture
Horizuntal
Natural
Ignibe
Rectangle
Sit
Figure 33: Top-5 attribute retrievals on the OCL Figure 34: Top-5 affordance retrievals on the test set.
OCL test set.
are different. OCI, covers many actions, so we can use OCRN to infer P3 of an object to narrow the solution space. Thus, we propose two ways:
(1) OCL Filtering: We use P3 to narrow the action space with threshold and generate Pğ- Affordances with probabilities higher than y are kept and others are set to zero (7 = 0.5). Then, HOI model only needs predict narrowed action space. practice, we multiply the prediction Phot from HOI model with P3 element-wisely to obtain the final prediction PHO Phor * P3-
(2) Hlmman-as-Probe Another more straightforward way is to predict HOI via OCL directly. We treat the human paired with the object as probe. Assuming the human feature is fh and human- object spatial configuration feature is fep (from Li et al. (2019); Gao et al. (2118). ). As P3 indicates all possible affordances, the ongoing actions can seen as the instantiation of Ps. i.e. they are activated by the "probe" In and Sap- So we use Sn and fsp to generate an attention A via MI.-Sigmoid. Then we operate P3 Antss and late fusion to get the final prediction PHoi (P3 An+ap + PHo1)/2.
Concretely, we use OCRN to enhance HOI detection models (iCAN (Gao et al. 2018). TIN et al. 2019), IDN (Li et al. (020aa) on HICO-DET (Chao et al. 2018). As OCL merely contains 15 object categories in HICO-DET (Chao et al. 2018), the rest 65 object categories are unseen. We embed OCRN into three HOI models according to OCL filtering and
Probe (Appendix Sec. A.7 and the public model checkpoints of Gao et al. (2018) Li et al. 2019 2020a are used.
The results shown in Tab. 6 With OCL filtering, iCAN (Gao al. 2018). TIN et. 2019) and IDN (Li et al., 2020a) achieve a gain of mAP by 0.65%, 0.90%, and 0.77% respectively. The Human-as-Prokh Probe is suitable for HOI detection and contributes performance boost of 1.50%, 1.46%, and 0.98% to three models. These strongly verify the efficacy and generalization ability of OCL.
Method
Full
Rare
Non-Rare
ICAN
14.84
10.45
16.15
iCAN+Filterig
15.49
8.76
17.50
iCAN+Probe
16.34
11.66
17.74
TIN
17.03
13,42
18.11
TIN+Filterng
17.93
13.79
19.17
TIN+Probe
18.49
15.02
19.58
IDN
23.36
22.47
23.63
IDN+Filterng
24.13
23.74
24.24
IDN+Probe
24.34
24.03
24.43
Table 6: Results of HOI detection (using detected object boxes).
27

---

Under review as a conference paper at ICLR 2022
A.8 COMPARISON ON DEBIASING
The motivation of the OCRN is to follow the prior knowledge of the three levels of objects with a deep learning-basee causal graph model, to pursue the object understanding beyond the common direct mapping from pixels to labels. To avoid the bias estimation such as in the Simpson's para- dox Pearl et al. (016). Thus, we use intervention to deconfound the confounder category and ex- clusive the possible spurious bias and correlation imported bias from imbalanced object categories. Overall, we propose our OCRN in a causal inference perspective instead of the pure classification viewpoint, which also suits our causal graphical model well. Similar cases are also proposed in cent works like Wang et al. (200aa) Tang et al. (2020a] bb. Moreover, to better compare our method with the common debiasing methods, we further conduct the experiments as follows.
We regard a, 3 recognition as multiple independent binary classification tasks and implement some methods introduced in Wang al. (2020b) on our strong baseline DM-V to reduce bias from object categories. We use mean bias amplification (Amp) in Zhao et al.] (2017) as bias evaluation metric: small Amp means model suffers less from data category bias. The test results are shown in Tab. 7 The proposed OCRN has comparable or smaller bias amplification than the variants of DM-V since our model follows the causal graph and exploits the tools of causal inference, while most methods for category bias are from the view of classification.
Model OCRN
Test Inference
Amp.
B Amp.
argmax, , P(yler)
0.127
0.112
DM-V Joint ND-way Soltmax
argmax, y maxg Piely, d(z)
0.151
0.158
DM-V + Joint ND-way Softmax
argmax, y Σn Pr.ly. d(z)
0.148
0.154
DM-V N-way classifier per domain
argmax, y Pr. (rl
0.135
0,112
DM-V N way classifier per domain
argmax, , sy,
0.147
0.145
Table 7: Comparison with debiasing models.
A.9 ABOUT STATES
We did not use object states in our model because there is a compositional zero-shot problem for object-state pairs, i.c., there can be unseen states in real-world data. Differently, affordances are more general. The models explicitly incorporating object states will fail to generalize these zero-shot states and it adds to the object category bias. In experiments, the state supervision during training would indeed slightly improve the affordance recognition performance, since instances in the same state lies in tight cluster in affordance label space. But this will hurt the TDE performance greatly.
A.10 DISCUSSION ABOUT CAUSALITY AND CAUSAL GRAPH
Annotating causality in the real-world is extremely difficult. In data annotation, we have met numer- ambiguities and difficulties to confirm the "right" causal relations. To address these challenges, we follow the following principles: (1) Firstly, we only emphasize on clear and strong causal rela- tions via crowdsoureing, but omit the vague ones. (2) Second, we take an object affordince-centric viewpoint to look at the possible causal relations. (3) We would rather discard than condone the controversial situation in annotation. (4) We only focus on the simple relations between one at- tribute and one affordance, instead of the very complex compositions of multiple attributes and affordances which are almost impossible to annotate. Therefore, we finally find that we can label very small percentage of all edges with the whole causal graph consisting of so many nodes (category, attributes, affordances, contexts, etc.) while keeping the quality.
Our causal graph follows the human priors from our experts and crowdsourcing annotators. Some previous works also follow this before designing the method, such as Zhu et al. (2014). From the viewpoint of causal discovery Pearl et al. )): Tang et al. (2020a b ;; Wang et al. (2020a). the above arcs (e.g., the inverted are from attribute to category in the causal graph directed acyclic graph, DAG) are indeed possible. However, here, mainly study the object concept learning problem, especially attribute and affordance learning. Thus, from the perspective of affordance learning, we think the ares from category to attribute and affordance is more vital and meaningful to us.
28

---

Under review as a conference paper at ICLR 2022
More specifically: (1) The causal graph follows the nature of human-like recognition. Given new object, we refer to the category level commonsense first, then a specific instantiation. Considering simple case, when a human wants to find apple to cat, he will pick up a fresh one instead of bad one based on the recognition of object category and attribute. (2) Definitely, the causal graph reflects the data collection pipeline. We first select object categories before collecting images and complementing instances. After that, category-level A, B and instance-level a, B are annotated. (3) Different ares are mainly attributed to task context, c.g., we have "image to label" in image recognition task, but "label to image" in image generation task. In OCL, the taxonomy viewpoint (how we define an object due to its attribute or affordance, e.g., a container is container because can be used to hold or transport something) lies out of the area of our concern. Instead, we consider more on the physical side of learning the property and functionality of an object for intelligent agent/robot. It will have a larger impact on boosting downstream tasks like robot manipulation.
A.11 DETAILED LISTS
The detailed object categories, states, attributes, and affordances are listed as follows.
A.11.1 OBJECT CATEGORY LIST
abaya, academic gown, accordion, acorn, aircraft, alpaca, ambulance, apple, armadillo, artichoke, axe, backpack, bagel, baked goods, balance beam, ball, balloon, banana, banjo. barbell, barrel, bath- tub, beach wagon, beaker, bear, bed, bee, bell pepper, bench, bib, bicycle, billiard table, binoculars, bird, bison, boat, bookcase, boot, bottle, bottle opener, box, brace, brassiere, bread, broccoli, broom, bubble, buckeye, bull, burrito, bus, butterfly, cabbage, cake, camel, camera, can opener, candle, can- non, car, cardigan, carnivore, carrot, cart, cat, cattle, cavy, cello, chain, chainsaw, chair, cheetah, chest of drawers, chicken, chime, clock, clog, closet, coat, cocktail shaker, coffee, computer key- board, computer mouse, conch, container, corn, couch, cowboy hat, crab, crocodile, crossword puz- zle, crutch, curtain, custard apple, desk, dessert, dhole, diaper, digital clock, dinosaur, dishwasher, dog, door, dough, dragonfly. drill, drum, drumstick, duck, dugong, dumbbell, eagle, eggnog, ele- phant, envelope, eraser, face powder, filing cabinet, fish, flower, flowerpot, flute, foothall helmet, footwear, fountain, fox, french horn, frog, frying pan, furniture, gazelle, giraffe, glove, goat, goblet, golf ball, golf cart, goose, gown, guacamole, guitar, hair dryer, hair slide, hair spray, hamburger, hammer, hamster, handgun, handkerchief, harmonica, harp, hartebeest, hat, hay, heater, hedgehog, helmet, hen, hippopotamus, home appliance, honeycomb, horizontal bar, horse, hot dog. hyena, ibex, ice cream, impala, insect, invertebrate, ipod, isopod, jaguar. jeans, jellyfish, jinrikisha, joy- stick, jug, kimono, kitchen utensil, knee pad, knife, knot, koala, lab coat, ladle, lamp, lampshade, lantern, laptop, lemon, letter opener, lighthouse, limousine, lion, lipstick, lizard, loudspeaker, loupe, madagascar cat, mailbox, maracas, marimba, mask, matchstick, measuring cup, meat loaf, megalith, microphone, microwave oven, military uniform, minibus, miniskirt, mirror, missile, mixing bowl, mobile home, mobile phone, modem, mongoose, monkey, mortarboard, motorcycle, mug, mush- room, nail, necklace, nipple, oar, oboe, ocarina, oil filter, orange, otter, oven, owl, oxygen mask, packet, paintbrush, panda, panpipe, paper towel, parachute, parking meter, pen, pencil case, pen- cil sharpener, penguin, petri dish, piano, picket fence, pig, pillow, pinwheel, pizza, plastic bag, plate, polar bear, pomegranate, porch, potato, pretzel, printer, rabbit, radio telescope, rain barrel, red panda, reel, refrigerator, remote control, rifle, rocking chair, rugby ball, safety pin, sandal, sandwich, sarong, saxophone, scarf, scoreboard, scorpion, screw, screwdriver, sculpture, sea lion, seat belt, sewing machine, shark, sheep, shield, shirt, shoji, shower, shower cap, ski, ski mask, sleeping bag, slide rule, snail, snake, snorkel, snowmobile, snowplow, soap dispenser, sock, solar dish, sombrero, spatula, spider, squash, squirrel, starfish, steel drum, stethoscope, strawberry, stretcher, submarine, suit, sunglasses, sunscreen, swan, swimwear, swing, syringe, tank, taxi, teapot, teddy bear, tele- phone, tennis ball, tent, tick, tie, timber wolf, toaster, toilet paper, tool, torch, towel, tower, traffic light, train, tripod, trombone, truck, trumpet, turnstile, turtle, umbrella, unicycle, vase, vegetable, vehicle, vending machine, vestment, violin, volleyball, vulture, waffle iron, wall clock, wallaby, wallet, wardrobe, warthog, washing machine, watch, weapon, whale, wheel, whistle, wild boar windmill, window shade, wine, wok, wooden spoon, wool, worm, yawl, yurt, zebra.
29

---

Under review as a conference paper at ICLR 2022
A.11.2 OBJECT ATTRIBUTE LIST
crumpled, ceramic, cold, curled, furry. black, wet, orange, brown, yellow, striped, cool, gray, leather, large. wooden, small, soft, round, old, portable, fluffy, hard, hom, messy, heavy, blue, purple, closed, new, red, thin, full, vertical, strong, dry, spotted, quadruped. whole, sharp, long, fake, open, toy, plastic, white, columnar, empty, flat, cloth, warm, leashed, solid, smooth, worn, rectangular, bipedal, tasty, curved, pink, hot, digital, electric, fresh, horizontal, short, natural, metal, cooked, green, folded, broken, bent, sliced, thick, wide, narrow, arched, puffy, cream, stone, cement, mar- ble, floral, glass, water, rubber, brick, sandy, plaid, paper, checkered, parked, moving, melted, lit, wearing, framed, stacked, tiled, standing, hanging, sitting, walking, sleeping, flying, dead, ripe, in the picture, reflective, grassy, leafy, painted, rusty.
A. OBJECT STATE LIST
(1) abaya:
common, damaged, special material. (2) academic gown: common, damaged, special material. (3) accordion:
common, in picture. (4) acorn: common, rotten. (5) aireraft: broken, flying, in factory, model, on ground, on show, part, toy. (6) alpaca: caged, dead, fighting, hugging, lying, milking, picture, playing, running, shearing, standing, statue, toy, walking. (7) ambulance: broken, building. food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped. (8) apple:
eating, fake, holding, model, picture, piece, rotten, sliced, toy, whole. (9) ar- madillo: dead, model, moving, resting, with shell. (10) artichoke: common. (11) axe: common, model, toy. (12) backpack: normal canvas. (13) bagel: eating, fake, holding, picture, raw, sliced, whole. (14) baked goods: eating, fake, holding, picture, raw, sliced, whole. (15) balance beam: common. (16) ball: flying large ball, flying small ball, holding large ball, holding small ball, kicking. playing large ball, playing small ball. (17) balloon:
flying, normal landed. (18) banana: cooked,
immature, model, on tree, peeled. picture, piece, rotten, sliced, toy, unripe. (19) banjo: common, playing. (20) barbell: common. (21) barrel: common, full. (22) bathtub: common. (23) beach wagon: broken, building, food, full, load on vehicle, model, moving, parked, part, picture, repair- ing, stained, wrapped. (24) beaker: ceramic glass, wood plastic. (25) bear: caged, dead, hugging, model, picture, running, standing, swimming, tied, title page, trapped. (26) bed: common. (27) flying, on some places. (28) bell pepper: common. (29) bench: common. (30) bib: common. (31) bicycle: broken, fake, holding, load on vehicle, model, moving, parked. picture, riding, stained. (32) billiard table: common. (33) binoculars: fixed, portable. (34) bird: caged, dead, flying, high place, hold, meat, model, perched, picture, swimming. (35) bison: caged, dead, eating, lying, model, moving, picture, toy. (36) boat:
bee:
model, moving, parked, stacked, stained. (37) bookcase: common, empty. (38) boot: common, fake, wearing. (39) bottle:
glass, glass broken, metal, plastic. (40) bottle opener: with drill. (41) box: metal closed, metal locked, metal open, paper closed, paper open, wood closed, wood open. (42) brace:
eating, fake, picture, raw, sliced, whole. (45) broccoli: cooked, fake, raw, rotten. (46) broom: common, fractured. (47) bubble: plastic, soap. (48) buckeye: with shell. (49) bull: caged, dead, eating, lying, model, moving, picture, toy. (50) burrito:
common. (43) brassiere: common. (44) bread:
eating, fake, holding, picture, raw, sliced, whole. broken, building, food, full, load on vehicle, model, moving, parked, picture, repairing, stained, toy, wrapped. (52) butterfly: flying. on hand, on some places. (53) cabbage: common. (54) cake; cating, fake, picture, raw, sliced, whole. (55) camel: sitting, standing, walking. (56) camera: common. (57) can opener: hidden blade, with blade, with drill. (58) candle: common, lit. (59) cannon:
(51) bus:
common, model. (60) car: broken, building, food, full, load on vehicle, model, moving, parked. part, picture, repairing, stained, wrapped. (61) cardigan:
common, damaged, spe- cooked, dirty, fake, rotten, metal, plastic, wood. (65) cat: caged, fake, holding, lying, moving, picture, caged, dead, fighting, hugging, lying, milking, picture, playing, running, shearing, standing, statue, toy, walking. (67) cavy: cating, holding, playing, resting. (68) cello: common. (69) chain: thick, thin. (70) chainsaw: common. (71) chair: broken, burnable, fragile, fragile and burnable, solid. (72) cheetah: caged, moving, sitting, toy. (73) chest of drawers: wood. (74) chicken: caged, dead, flying. high place, hold, meat, model, perched, picture, swimming. (75) chime:
cial material. (62) carnivore: caged, moving, sitting, toy. (63) carrot:
sliced, whole. (64) cart:
standing, toy. (66) cattle:
large, small, wood plastic. (76) clock: broken, common. (77) clog: boat, common, wearing. common. (79) coat: common, damaged, special material. (80) cocktail shaker: com- mon. (81) coffee: common. (82) computer keyboard: common. (83) computer mouse: broken, common. (84) conch: alive, shell. (85) container: closed, full. locked, open, with holes. (86) corn: common. (88) cowboy hat: common. (89) crab:
(78) closet:
kernels, rotten, whole, with husk. (87) couch:
30

---

Under review as a conference paper at ICLR 2022
alive, cooked. (90) crocodile: common. (91) crossword puzzle: common. (92) crutch: common. (93) curtain:
common. (94) custard apple: common. (95) desk: common. (96) dessert: com- mon, half eaten, hold by human. (97) dhole:
caged, moving, sitting, toy. (98) diaper: common, hanged, wearing. (99) digital clock: broken, common, hold by human, projected in screen. (100) dinosaur: fake, fossil, sculpture. (101) dishwasher: broken, closed, fixing, opened, playing with caged, in boat, in car, jumping, lying, model, moving, picture, playing, stop, tied, toy. (103) door: close, open, open with human. (104) dough: common. (105) dragonfly: on some places. (106) drill:
human pet, uninstalled, using. (102) dog:
occupied, placed. caged, dead, flying. high place, hold, meat, model, perched, picture, swimming. (110) dugong: eating, swimming. (111) dumbbell: occupied, placed. flying, in high place, on ground. (113) eggnog: common, half eaten, hold by human. (114) elephant: caged, in water, lying, on show, picture, playing, ridden, sitting, standing, statue, toy, trap, walking. (115) envelope: common, hold by human. (116) eraser:
occupied, placed. (107) drum:
(108) drumstick:
occupied, placed. (109) duck:
(112) eagle:
common, run out. (117) face powder: common, on brush. (118) filing cabinet: close, open. (119) fish: food, caught by human, normal(swimming). (120) flower:
common. (122) flute: common, wearing. (124) footwear: common, underwear close, operative. (126) fox: caged, moving, sitting, toy. (127)
common. (121) flowerpot:
common, playing. (123) foothall helmet:
swimwear, wearing. (125) fountain:
common. (128) frog: in hand, on ground, swimming. (129) frying pan: common, common, using. (131) gazelle: caged, dead, fighting, hugging, lying, milking, picture, playing, running, shearing, standing, statue, toy, walking. (132)
french horn:
hold by human, using. (130) furniture:
giraffe: caged, dead, fighting, hugging. lying, milking, picture, playing, running, shearing, standing, statue, toy, walking. (133) glove: not wearing, wearing, weaving. (134) goat: caged, dead, fighting, hugging, lying, milking. picture, playing, running, shearing, standing, statue, toy, walking. (135) goblet: empty, full. (136) golf ball: building, common. (137) golf cart: broken, building, food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped. (138) goose: caged, dead, flying, high place, hold, meat, model, perched, picture, swimming. (139) gown: com- mon, damaged, special material. (140) guacamole: pure, with food. (141) guitar: common, play- ing. (142) hair dryer: model, place, use. (143) hair slide: hold by human, place. (144) hair spray: building, hold, place. (145) hamburger: eating, fake, picture, raw, sliced, whole. (146) hammer: common. (147) hamster: holding, playing, resting eating. (148) handgun: common, hold by hu- man. (149) handkerchief: hold by human, placed. (150) harmonica: place, playing. (151) harp: caged, dead, fighting, hugging, lying, milking, picture, playing, running. shearing, standing, statue, toy, walking. (153) hat: on hand, place, toy, wearing. (154) hay: closed, running. (156) hedgehog:
place, playing, (152) hartebeest:
common. (155) heater:
common, eating. (157) helmet: place, wear. (158) hen: caged, dead, flying, high place, hold, meat, model, perched, picture, swimming. (159) hippopotamus: caged, in water, on ground. (160) home appliance: broken, placed, using. (161) honeycomb:
common, processed food. (162) horizontal bar: common, grab or used in lying, model, picture, pulling. riding, running, standing, statue, toy. (164) hot dog: cating, fake, picture, raw, sliced, whole. (165) hyena: caged, moving, sitting, toy. (166) ibex: caged, dead, fighting, hugging, lying, milking, picture, playing, running, shearing, standing, statue, toy, walking. (167) ice cream: hold, placed. (168) impala: caged, dead, fighting, hugging, lying, milking, picture, playing, running, shearing, standing, statue, toy, walking. (169) insect: caught in box,bag,cup common, hold by human, on plant. (170) invertebrate: building, caught, common, food, fossil, in water, model toy. (171) ipod: on hand, place. (172) isopod:
sports. (163) horse:
common, hold by caged, moving, sitting. toy. (174) jeans: common, wearing. (175) jellyfish: common. (176) jinrikisha: metal, wood plastic. (177) joystick: hold by human, place. (178) jug: common, damaged, special material. (180) kitchen utensil: broken, com- mon, contain something, heating. (181) knee pad: common, wearing. (182) knife: common, hold common, noddles knot. (184) koala:
human. (173) jaguar:
common. (179) kimono:
common. (185) lab coat: common, damaged, special material. (186) ladle: contain something, hold by human, placed plastic or metal, placed wood. (187) lamp: common, uninstalled, using. (188) lampshade: common. (189) lantern: lighting, making, no light. (190) laptop:
by human. (183) knot:
sliced, whole. (192) letter opener: hold by human, place. (193) lighthouse: common. (194) limousine: broken, building. food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped. (195) close, open, using. (197) lizard: dead, model, common, hold by human. (199) loupe: hold by human, place. (200) madagascar cat: caged, fake, holding, lying, moving, picture, standing, toy. (201) mailbox: steal, wooden. (202) maracas: common, using. (203) marimba: common, fake, using. (204) mask: common, wearing. (205) matchstick: common, in fire, in hand, used. (206)
placed, using. (191) lemon:
lion: caged, moving, sitting, toy. (196) lipstick:
moving, resting, with shell. (198) loudspeaker:
31

---

Under review as a conference paper at ICLR 2022
ceramic glass, wood plastic. (207) meat loaf: eating, on table. (208) megalith: common. (209) microphone: common, using. (210) microwave oven: common, using. (211) mil- itary uniform:
measuring cup:
common, damaged, special material. (212) minibus: broken, building. food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped. (213) miniskirt: wearing. (214) mirror: rear view in car, rear view out car, street mirror. (215) missile: launched, unlaunched. (216) mixing bowl: common. (217) mobile home: close, moving, open, stop. (218) mobile phone: common, using. (219) modem: common, hold by human, stripped. (220) mon- goose: common. (221) monkey: eating, fake, in cage, in house, lying, on tree rope shelf high place, pet by human. playing, resting, sitting, standing, swimming, walking running. (222) mortarboard: common, hold, thrown, wearing. (223) motorcycle: load on vehicle, model, moving, on show, pic- ture, repairing, riding, stopped, toy. (224) mug: balloon, ceramic, glass, metal. (225) mushroom: common, cooked, fake, hold by human, possible poisonous. (226) nail: bent, common, half nailed in. (227) necklace:
common. (228) nipple: with bottle, without bottle. (229) oar: fixed, portable. (230) oboe: common. (231) ocarina: common, fake, hold by human, playing. (232) oil filter: com- mon. (233) orange:
common, feed by human, hold by human, in water. (235) oven: open, pot like. (236) owl: caged, dead, flying. high place, hold, meat, model, perched, picture, swimming. (237) oxygen mask: common. (238)
immature, model, peel. picture, rotten, sliced, whole. (234) otter:
packet: closed, open. (239) paintbrush: common. (240) panda: eating resting, playing. (241)
panpipe: common, hold by human, playing. (242) paper towel: common, torn. (243) parachute:
common. (244) parking meter: box like. with pole. (245) pen: feather, metal, plastic. (246)
pencil case: metal, soft plastic canvas. (247) pencil sharpener: embedded blade. rotating. (248)
penguin: normal resting. (249) petri dish: empty, in use. (250) piano: broken, common, locked, model. (251) picket fence: normal wood. (252) pig: caged, resting. (253) pillow: common. (254)
pinwheel: normal paper. (255) pizza:
eating, fake, picture, raw, sliced, whole. (256) plastic bag: empty, filled. (257) plate: dirty, empty, with dish. (258) polar bear: caged, dead, hugging, model, picture, running, standing, swimming, tied, title page, trapped. (259) pomegranate: opened, rotten, seeds, whole. (260) porch:
empty, with chair. (261) potato: mashed, whole raw. (262) pretzel: eating, fake, picture, raw, sliced, whole. (263) printer: large machine, mini portable, small desk- top. (264) rabbit: caged, common, holding. (265) radio telescope: common. (266) rain barrel: robber, wood. (267) red panda: common, feed or hold by human. (268) reel: only reel itself, with rod, with rod and string. (269) refrigerator: closed, open full, open not full. (270) remote control: broken, common. (271) rifle: common. (272) rocking chair: wood, (273) rugby ball: alone, holding, model. (274) safety pin: architecture, broken, close, hang, hurt human, open. (275) sandal: common, wearing. (276) sandwich: eating. fake, picture, raw, sliced, whole. (277) sarong: placed, wearing. (278) saxophone: common. (279) scarf: on cat, placed, wearing. (280) score- common, hold by human, scorpion like bag. (282) serew: common, plug into something. (283) screwdriver:
board: installed, manual write, portable. (281) scorpion:
common, hold by human, in package, plug in something. (284) sculpture: common, in high place, in picture. (285) sea lion: in water, on ground. (286) seat belt: common. (287) sewing machine off, on. (288) shark: caught by wire, dead on ground, normal swim. (289) sheep: caged, dead, fighting. hugging, lying, milking, picture, playing, running, shearing, standing, statue, toy, walking. (290) shield: common. (291) shirt: common, wearing. (292) shoji: close, open. (293) shower: off, on. (294) shower cap: on head, place. (295) ski: placed, using by human. (296) ski mask: common, wearing, (297) sleeping bag: empty flatten, human inside, rolled. (298) slide rule: common, hold by human. (299) snail: common, hold by human. (300) snake: common, hold by human, in water, toy. (301) snorkel: placed, wearing. (302) snowmobile: broken, building, food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped. (303) snowplow: broken, building, food, full, load on vehicle, model, moving, parked. part, picture, repairing, stained, wrapped. (304) soap dis- penser: installed, portable. (305) sock: wearing. (306) solar dish:
common, half installed. (307) light, on back, place, wearing. (308) spatula: common, hold by human, toy. (309) spi- der: caught in box. common, hold by human, on plant. (310) squash: cooked meal, sliced, whole ripe, whole unripe. (311) squirrel: common, hold by human, toy. (312) starfish: hold, in water, on ground, on sale. (313) steel drum: occupied, placed. (314) stethoscope: common. (315) strawberry: common. (316) stretcher: empty, human on it, lift by human. (317) submarine: model, move, on ground, park. (318) suit:
sombrero:
hand, placed, wearing. (320) sunscreen: hold by human, placed. (321) swan: caged, dead, flying, high place, hold, meat, model, perched, picture, swimming. (322) swimwear: placed, wearing. (323) swing: empty, human on it. (324) syringe: contain something, empty, hold by human, stick into something. (325) tank: firing, moving, stopped, toy. (326) taxi: broken, building, food, full, load on vehicle,
common, wearing. (319) sunglasses:
32

---

Under review as a conference paper at ICLR 2022
model, moving, parked, part, picture, repairing, stained, wrapped. (327) teapot: common. (328)
teddy bear: common. (329) telephone: domestic, public. (330) tennis ball: common, damaged, unopened. (331) tent: common, using. (332) tick: common, hold by human, on plant. (333) tie:
common, wearing. (334) timber wolf: caged, moving, sitting, toy. (335) toaster: common, using. (336) toilet paper: common, exhausted. (337) tool: common, damaged old, fake, using. (338) torch: electronic not work, firing, firing hold by human. (339) towel:
common, using. (340) tower: common, portable. (341) traffic light: common, fixing. (342) train: food, model, moving, parked, picture, stained, toy. (343) tripod: common. (344) trombone: common, separated. (345) truck: broken, building. food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped. (346) trumpet:
common. (348) turtle: in hand, in water, on common, hold by human. (350) unicycle: common, riding. (351) vase: common, in hand, rotten, sliced cooked, unpicked. (353) vehicle: bro- ken, building, food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped. (354) vending machine: off, on. (355) vestment: common, damaged, special material. (356) violin: common, playing. (357) volleyball: flying, holding, static. (358) vulture: flying, in broken, fake, part, picture, placed, using. (360) wall common. (361) wallaby: eating, lying, running walking, standing. (362) wallet: common. (363) wardrobe: close, open. (364) warthog: eating, standing lying, walking running. (365) wash- ing machine: close, in wild, open, using. (366) watch: common, wearing. (367) weapon: common, fake, firing, hold by human, using. (368) whale:
common. (347) turnstile:
ground. (349) umbrella:
common. (352) vegetable:
high place, on ground. (359) waffle iron:
clock:
common, dead. (369) wheel: damaged. holding, in water, sculpture, stop, uninstalled, using, using in water, working. (370) whistle: common. (371) wild boar: eating, lying, standing, walking running. (372) windmill: common. (373) window shade: put down, taken up. (374) wine:
common, hold by human, using. (376) wooden spoon: contain something. hold by human, placed wood. (377) wool: cloth, skein, toy. (378) worm: common, in water, microorganism. (379) yawl: model, moving, parked, stacked, stained. (380) yurt: close, fake, open, using. (381) zebra: caged, lying, model, picture, rumning, standing, statue, tied, toy, wild.
common, fake, pouring. (375) wok:
A.11.4 OBJECT AFFORDANCE LIST
aim (point cause to go (blows, weapons, objects such photographic equipment) towards), assemble (create by putting components or members together), bend (form a curve), bend (change direction), blast (fire a shot), blow (exhale hard), board (get on board of (trains, buses, ships, aircraft, etc.)), break (become separated into pieces or fragments), brush (rub with a brush, or as if with a brush), buy (obtain by purchase; acquire by means of financial transaction), capture (capture as if by hunting, snaring, or trapping), carry (have with oneself; have on one's person), catch (attract and fix), catch (take hold of so to seize or restrain or stop the motion of), chase (go after with the intent to catch), check (examine so as to determine accuracy. quality, or condition). clean (make clean by removing dirt, filth, or unwanted substances from), climb (go upward with gradual or continuous progress), close (become closed), collect (gather or collect), cook (transform and make suitable for consumption by heating), cut (separate with or as if with an instrument), decant (pour out), dilute (lessen the strength or flavor of a solution or mixture), drag (pull, as against resistance). draw (bring, take, or pull out of a container or from under a cover), drink (take in liquids), drive (operate or control a vehicle), eat (cat a meal; take a meal), eat (take in solid food), edit (cut and assemble the components of), embrace (squeeze (someone) tightly in your arms, usually with fondness), exit (move out of or depart from), feed (give food to). fly (operate airplane), function (perform as expected when applied), get (succeed in catching or seizing, especially after a chase), hit (hit against; come into sudden contact with), hit (cause move by striking). hitch (to hook entangle). hold (be the physical support of; carry the weight of), hold (have or hold in one's hands or grip), hop on (get up on the back of). ignite (cause to start burning; subject to fire or great heat), inject (give an injection to). inspect (look over carefully), install (set up for use), jab (poke or thrust abruptly), jump (bypass), kick (drive or propel with the foot), kick (thrash about or strike out with the feet), let go of (release, as from one's grip), lie (be located or situated somewhere; occupy certain position), lie (be lying, be prostrate; be in a horizontal position). lie down (assume a reclining position), lift (take hold of something and move it to a different location), load (fill or place load on), melt (reduce or cause to be reduced from a solid to a liquid state, usually by heating), move (cause to move or shift into new position place, both in concrete and in an abstract sense), obstruct (block passage through), occupy (occupy the whole of), open (cause to open or to become open), open (start to operate function or cause to start operating or functioning), operate (handle and cause
33

---

Under review as a conference paper at ICLR 2022
to function), pack (arrange in container), paint (upply paint to; coat with paint), pare (remove the edges from and cut down to the desired size), photograph (record on photographic film), pick (look for and gather), plow (to break and turn over earth especially with plow), produce (create or manufacture a man-made product), pull (apply force so as to cause motion towards the source of the motion), push (move with force, "He pushed the table into corner). push (press against forcefully without moving), put (put into a certain place or abstract location), race (compete in race), raise (raise from a lower to higher position), reach (reach a destination, either real or abstract), read (interpret something that is written or printed), rend (tear or be torn violently), repair (restore by replacing a part or putting together what torn or broken), ride (sit and travel on the back of animal, usually while controlling its motions), ride (be carried or travel on in vehicle), score (gain points game), shoot (hit with a missile from a weapon), shoot (send forth suddenly, intensely, swiftly), sign (mark with one's 's signature; write one's name (on)), sit (be around, often idly or without specific purpose), sit (be seated), slide (move smoothly along a surface), smoke (inhale and exhale smoke from cigarettes, cigars. pipes), spin (revolve quickly and repeatedly around one's own axis), squeeze (press firmly), stand (hold one's ground; maintain a position; be steadfast or upright), stand (be standing; be upright), stand (remain inactive or immobile), steer (direct the course; determine the direction of travelling), sweep (move with sweeping, effortless, gliding motions), swerve (turn sharply; change direction abruptly), swing (make a big sweeping gesture or movement), switch off (cause to stop operating by disengaging switch), throw (propel through the air), tie (fasten or secure with a rope, string, or cord), toast (propose a toast to), touch (perceive via the tactile sense), transport (move while supporting, either in a vehicle or in one's hands or on onc's body), turn (let (something) fall or spill from a container), turn (cause to move around or rotate), turn (change orientation or direction, also in the abstract sense), unfold (spread out or open from closed or folded state), walk (use one's 's feet to advance; advance by steps), wash (cleanse (one's 's body) with soap and water), wash (clean with some chemical process), work (shape, form, improve a material), write (record data on a computer), write (mark or trace on a surface), adjust (make correspondent or conformable), flip (toss with sharp movement clean, or empty with a liquid), grind (make a grating or grinding sound by rubbing together), groom (give neat appearance to), herd (to gather and move group of animals), hose (water with hose), jump (cause to jump or leap). lasso (catch with a lasso), launch (to put (a boat or ship) on the water), lick (pass the tongue over), milk (take milk from female mammals), park (place temporarily), pay (give money, usually in exchange for goods or services), pet (stroke or caress gently), point (direct into a position for use), row (propel with oars), run (cause an animal to move fast), serve (put the ball into play), shear (shear the wool from), sip (drink in sips), smell (inhale the odor of; perceive by the olfactory sense), stick (pierce or penetrate or puncture with something pointed), stir (mix or add by stirring), straddle (sit or stand astride of), text on (send message). type on (write by means of a keyboard with types), zip (close with a zipper), swim, talk on phone, work on computer, ski, surf, skateboard, dry (remove the moisture from and make dry), fly (travel through the air; be airborne), greet (express grectings upon meeting somcone), hunt (pursue for food or sport (as of wild animals)), launch (propel with force), lose (fail keep or to maintain; cease to have, either physically or in an abstract sense), prepare (educate for a future role or function), rub (scrape or rub as if to relieve itching), set up (get ready for a particular purpose or event), snog (touch with the lips or press the lips (against someone's mouth or other body part) as an expression of love, greeting, etc.), stop (stop from happening or developing), talk (exchange thoughts; talk with), teach (impart skills or knowledge to), walk (accompany or escort), watch (see or watch), watch (look attentively), wear (have on one's person). wield (handle effectively), tag (touch a player while he is holding the ball), wave (signal with the hands or nod).
to cause turn over in the air), flush (rinse,
34


Reviews:
[{"review_id": "Ut5tgR1sgLk", "reviewer": "Reviewer_mgiJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I found that the writing of this paper can still be improved. For example, Table 1 is too small; there are too many bold texts in Section 3; Section 4.2 contains many paragraph subtitles, it will be better if they can be grouped according to whether they are applied to attribute prediction or affordance prediction, etc.\n\nMy primary concern about this paper, however, lies in its clarity in formulating the causal story. First, I would suggest the authors state clearly where do these causal graphs come from. Why should we predict attributes based on the category (figure 1)? Why should the image be generated based on the Category (figure 5)? Specifically,\n\n- is the causal graph representing the nature? E.g., physics.\n- is the causal graph representing the data generation/collection pipeline? E.g., we first sample a category, and then search for the images.\n- is the causal graph representing human's reasoning of the properties? This is quite tricky. For example, the arrow could be attribute -> category (e.g., a yellow, oval citrus fruit with thick skin and fragrant, acidic juice => lemon), or affordance -> category (an object that can be used to hold or transport something. => container).\n\nI guess the primary reason for introducing a specific Bayes net is for debiasing purposes. The key idea is to apply \"interference\" during inference to compensate the biases during training (e.g., category frequency, etc.)\n\nHowever, I feel that this part of the paper is not well compared with other approaches towards \"debiasing\". For example, \nhttps://openaccess.thecvf.com/content_CVPR_2020/papers/Wang_Towards_Fairness_in_Visual_Recognition_Effective_Strategies_for_Bias_Mitigation_CVPR_2020_paper.pdf\nhttps://openaccess.thecvf.com/content/CVPR2021/papers/Ramaswamy_Fair_Attribute_Classification_Through_Latent_Space_De-Biasing_CVPR_2021_paper.pdf\nThese two papers discussed several methods for \"remove biases.\" There are, of course, many other papers in this field, if the authors follow the pointers in these papers. I would like to see how the proposed model relate to these methods as well as how they compare empirically.\n\nA minor thing about the metric: I am not sure how TDE/alpha-beta-TDE should be interpreted. why is this consistency score between network \"weights\" and \"human labels\" matter at all? This is especially tricky when all prediction networks take the image feature as their inputs.\n\n", "summary_of_the_paper": "The primary contribution of this paper is two folds. First, the paper presents a new crowd-sourcing dataset that contains annotations for object categories, attributes, and affordances. Second, the paper presents a CNN training and inference pipeline that leverages \"causal interference\" to address potential biases.", "summary_of_the_review": "I think the dataset part is a solid contribution of the paper. However, for the proposed model, I think the authors are missing important baselines and discussions with related work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Se5Q8ObVrgC", "reviewer": "Reviewer_XzUp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ves:\n+ Present-day ML models lay a great emphasis on categorizing objects based solely on appearance. In case of distribution shift in a real-world dataset, such models find it difficult to generalize. A deeper understanding of objects based on their physical properties and affordances can avoid such problems, and the dataset introduced in this paper is a notable step in the right direction.\n+ The constructed dataset is large-scale which will encourage the generalization of models towards real-world data and large-scale applications.\n\nConcerns:\n- The paper has adopted a causal intervention method to solve for category biasing. However, there are simpler methods to tackle such problems (undersampling/oversampling majority/minority classes, generating synthetic samples etc.). How is the proposed method to these approaches? Why is a causal approach necessary? For a dataset that may be of wide use, it may also be good to not make its description more complex than it should be.\n- Surprisingly, for an image dataset paper, there were very few examples of the images in the paper. It may have been nice to see a few sample images in the supplementary material along with a clear description of their annotations.\n- The paper presentation is very dense. While I appreciate the detailed description of every aspect, it at times seems one level of detail beyond what is required. Besides, for what seems like a dataset created with careful thought, it would have been good to see more examples from the dataset, or a running example to explain the decisions made for the dataset design.\n- Similarly, in Sections 4 and 5, the many notations and inline equations make it difficult to follow. It may be a good idea to write problem formulation (or include them in the task overview), which lays down the assumptions, notations and objectives clearly and concisely, and then discuss the rest of the work.\n- The notion of causal relations in real-world scenes can be tricky and complex (for e.g, the notion of context -- water can be causal for a swimming fish making context causal). In the few examples presented in the paper, it was not very clear whether the relationships between the attributes and the affordances in the image are really causal. The paper mentions that two annotators were used to label such causal relationships -- I am not sure if this may be sufficient.\n\nMinor comments:\n* Section 4.1 the sentence, “The ɑ, 𝛃 probabilities ...” seems incomplete.\n* Section 4.2 “Attribute Estimation Debiasing” there is inconsistency in the notation of f_Oi.", "summary_of_the_paper": "This paper introduces a large annotated dataset for object concept learning. The proposed dataset contains annotation at two levels of granularity (category and instance level). The dataset also provides causal relations between object attributes and their affordances. The paper also provides a thorough insight into their annotation process, and introduces a baseline method called the Object Concept Reasoning Network (OCRN) based on causal intervention and instantiation. The proposed dataset is large-scale for similar ones in this space, and can be of great help to researchers working in the area of concept learning and compositionality.", "summary_of_the_review": "In general, the paper presents a dataset, which is a positive step towards concept learning tasks and will hopefully encourage future works to move beyond object categorization based only on appearance. The paper presentation seems too dense, and it’d be good to see some concrete directions of presenting the paper better to make it more accepted by the community.\n\nI will wait to hear back from the authors and other reviewers, and accordingly adjust my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NVEaOsRMWzr", "reviewer": "Reviewer_xQXf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the paper are as follows:\n\n- (Very) Important problem.\n- A new dataset is always a good contribution. Specifically, the work appears to have done the extra mile in combining lots of different and different data sources.\n\nThe overall weaknesses of the paper are as follows:\n- The technical contribution is low. Basically, this is a dataset/task proposition paper. This is also the authors' summary.\n- In that sense, it feels that the paper is not a direct fit to ICLR, which focuses more on technical contribution. \n- While it is claimed that the work is about causality, the causality in this context is rather learned with standard correlation-based approaches, which minimize standard learning objectives.\n- In eq 3, the notation is not entirely clear to me, or I am misunderstanding something. If A stands for category-level attributes, their number is different than the number of object categories. How can then be that $P(A_j|O_i)=1$ $iff i=j$, since these two indices run over different sets?\n- It is not entirely clear what is the baseline supposed to do? Basically, learn 'causal mechanisms' such that to attain 'perfect' recognition of the category-level, instance level attributes and affordances, and category label? And to do so, the do-operation intervention runs over all attributes/affordances in (3) and (6)? What exactly is the motivation of this intervention, as in selecting one attribute/affordance over another?\n- The writing is not always very clear. For instance, I am not sure what is the x axis of figure 8 and 9, I could not find it in the caption? Is it the number of training data used to train the model, and if yes, why is nearly zero for a long period in the figure?", "summary_of_the_paper": "In short, the paper proposes a new task, that of object concept learning. Object concept learning is a basically a combination of object classification, attribute classification of said object, and affordance classification of said object, both at a category (affordance of any cup) and an instance (affordance of this cup) level. To this end, the paper proposes a dataset, which starts from existing ones and extends them accordingly. Further, it proposes a baseline method that is somewhat inspired by do-calculus (Pearl). Experiments show positive trends.", "summary_of_the_review": "In summary, I think the paper brings little technical innovation, and as such it should not be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8zvSgALbjG", "reviewer": "Reviewer_mY2o", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Having such a large-scale dataset with detailed annotations and causal relationships of visual concepts (attributes and affordances in this paper) is valuable for the entire community. It will advance future research on causal learning from visual data.\n\n2. Data and code will be publicly available (they are available for review). \n\nWeaknesses:\n\nFirst of all, I have to admit I have limited knowledge about causal learning. I found some technical details are missing, particularly in Section 4 and 5, which make the technical part not easy to follow. Moreover, it would be great if more information could be provided in the paper (or in the supplementary material). Detailed comments can be found below.\n\n1. For the data annotation, more annotators were hired if the first set of annotators could not reach an agreement. What is the percentage of such confusing cases?\n\n2. What is the total cost of building such a dataset? This is useful as a reference if people would like to build a similar dataset in the future.\n\n3. Although experts were involved in designing the data annotation process, for example, defining the object states, it is not clear whether there is any way to formally monitor the quality of the annotations.\n\n4. Object states are used in the data annotation process for the instance-level affordance and causal relation parts. I would love to see the annotations of object states for such a benchmark dataset to be released as well. But they are intentionally excluded in the benchmark dataset and being used in the baseline model. Here are two questions:\n    a) Since object states were used in the annotation process, is it possible that without using the states, some annotations of afforances and causal relations may be ambiguous or do not hold at all? For example, I may argue that both the apples in the middle column of Fig 4 can not be *directly* eaten as they need to be washed first. \n    b) According to the experimental results, the performance of the proposed baseline is still far from satisfactory. What if we explicity consider object states as part of the causal structure? Will it bring any improvement?\n\n\n5. What are the context in Fig. 4? Are they attributes or object states?\n\n6. For the task definition in Eq.(1), it is not clear about what the concrete tasks are. It would be helpful to explicitly explain/define them.\n\n7. In Eq.(7) and Fig. 6, $f_{\\alpha}'$ is introduced but without formal explanations. Particularly in Fig. 6, what are $f_{\\alpha_1}$, $f_{\\alpha_2}$, and $f_{\\alpha_p}$? How are they different from $f_{\\alpha}$? Where is the binary classifier $g_{\\alpha_p}$? Is its loss part of $L_{\\alpha}$?\n\n8. Clarifications are needed for the definition of TDE. From my understanding, when $GT_{\\beta_q}=1$, it means the affordance $\\beta_q$ exists in the image. As a result, a high score is desired from the model's predicted score. Ideally, by explicitly considering the causal relationship of $\\alpha_p$ and $\\beta_q$, $P_{TDE}^{\\alpha_p}(\\beta_q)$ should be higher than $P(\\beta_q)$. So TDE should be defined as $max(P_{TDE}^{\\alpha_p}(\\beta_q) - P(\\beta_q), 0)$. Similarly, for the case where $GT_{\\beta_q}=0$, it should be $max(P(\\beta_q) - P_{TDE}^{\\alpha_p}(\\beta_q), 0)$.", "summary_of_the_paper": "This paper proposes a new large-scale benchmark dataset for object concept learning, which consists of recognizing attributes, affordances, and their causal raltions about objects in input images. Detailed annotations of object categories, attributes and affordances on both category and instance levels, and their causal relations (on the instance level) are provided. This new dataset will be helpful for the community to advance the research of causal learning from visual data. \n\nA strong baseline method is also proposed that explicitly considers the causal structure and concept instantiation of object categories, attributes, and affordances. Although it shows better results than other baselines, there is still great room for future improvement.", "summary_of_the_review": "The issues mentioned in the weaknesses part above are minor. They do not affect the value of the proposed benchmark dataset and baseline algorithm. But it would be great if they could be addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents work on a dataset for object concept learning.  The main contributions include causal relations in the dataset and a method (OCRN).  The initial reviews pointed to concerns over the nature of the causal relations, the presentation of the paper, and the OCRN method and its motivations / use of the do-operator.  The reviewers engaged in significant discussions after considering the authors' responses and the others' reviews.  After this delibration, the concerns over the dataset, its annotations, and the presentation of the methods were deemed to be better served by a full revision and reconsideration of the paper.  As such, the paper is not recommended for acceptance at this time.
Decision: Reject