IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenes
Authors: QI LI
Kaichun Mo
Yanchao Yang
Hang Zhao
Leonidas Guibas
Venue: ICLR 2022 Poster (2022)
Keywords: Inter-object Functional Relationship
Learning Interactive Policy for Exploration
Interactive Perception
3D Scene Understanding

Abstract: Building embodied intelligent agents that can interact with 3D indoor environments has received increasing research attention in recent years. While most works focus on single-object or agent-object visual functionality and affordances, our work proposes to study a novel, underexplored, kind of visual relations that is also important to perceive and model -- inter-object functional relationships (e.g., a switch on the wall turns on or off the light, a remote control operates the TV). Humans often spend no effort or only a little to infer these relationships, even when entering a new room, by using our strong prior knowledge (e.g., we know that buttons control electrical devices) or using only a few exploratory interactions in cases of uncertainty (e.g., multiple switches and lights in the same room). In this paper, we take the first step in building AI system learning inter-object functional relationships in 3D indoor environments with key technical contributions of modeling prior knowledge by training over large-scale scenes and designing interactive policies for effectively exploring the training scenes and quickly adapting to novel test scenes. We create a new dataset based on the AI2Thor and PartNet datasets and perform extensive experiments that prove the effectiveness of our proposed method.

Published as a conference paper at ICLR 2022
IFR-EXPLORE: LEARNING INTER-OBIECT FUNC- TIONAL RELATIONSHIPS IN 3D INDOOR SCENES
Li Kaichun Mo" Yanchao Yang', Hang Zhao', Leonidas Guibas?
TTinggua University
Stanford University
(liqil zhaohang01244 com
(kaichun, yanchaoy, guibas 8cs. st anfodddd
ABSTRACT
Building embodied intelligent agents that can interact with 3D indoor environments has received increasing research attention recent years. While most works focus single-object agent-object visual functionality and affordances, our work proposes to study a new kind of visual relationship that is also important to perceive and model inter-object functional relationships (e.g., switch on the wall turns on or off the light, a remote control operates the TV). Humans often room,
spend little or effort to infer these relationships, by using our strong prior knowledge (e.g., we know that buttons control electrical devices) or using only a few exploratory interactions in cases of uncertainty (e.g., multiple switches and lights in the same room). In this paper, we take the first step in building AI system learning inter-object functional relationships in 3D indoor environments with key technical contributions of modeling prior knowledge by training over large-scale scenes and designing interactive policies for effectively exploring the training scenes and quickly adapting to novel test scenes. We create a new benchmark based on the A12Thor and PartNet datasets and perform extensive experiments that prove the effectiveness of proposed method. Results show that our model successfully learns priors and fast-interactive-aaaaaa for exploring inter-object functional relationships in complex 3D scenes. Several ablation studies further validate the usefulness of each proposed module.
when entering
strategies
INTRODUCTION
When first-time entering a hotel room in a foreign country, we may need to search among buttons on the wall to find the one that turns specific light, figure out how to turn the faucets in the bathroom and adjust water temperature, or perform several operations on the TV and the remotes effort if we have used the same
to switch to favorite channel. Many of these may require models of lamp or faucet before, while the others that we are less familiar with will take more time for explore and figure how things work. We humans can quickly adapt to new indoor environment thanks to our strong prior knowledge about how objects could be functionally related in general human environments (e.g.. the room light can turned on by pressing a button toggling switch) and our amazing capability of using only few interactions while exploring the environment to learn how things work in cases of uncertainty (e.g., multiple lights and buttons in the room). In this work, we investigate how to equip AI systems with the same capabilities of learning inter-object functional relationships in 3D indoor environments.
With the recent research popularity of embodied AI in perception and interaction, many previous works have studied how to perceive, model, and understand object functionality (Kim al., 2014; Hu et al., 2016; 2018; 2020) and agent-object interaction (Montesano & Lopes, 2009; Do et al., 2018; Nagarajan Grauman, 2020; Mo et al., 2021a), just to name a few. However, these works mostly investigate single-objecusscene agent-object scenarios. A few past works (Jiang et al., 2012; Sun et al., 2014; Zhu et al., 2015; Mo et al., 2021b; Cheng et al., 2021) have studied inter-object relationships given pair of objects for specific tasks (e.g., placement, stacking). We make first step bringing up research attentions to a new type of visual quantity inter-object functional relationship, and an interesting yet underexplored challenging problem of how to learn such generic functional relationships in novel 3D indoor environments.
* Eqqull contribution.

---

Published as a conference paper at ICLR 2022
Prior knowledge
Interaction & GT obs
Posterior
i
1
Figure I: We formulate a novel task of learning inter-object functional relationships (IFRs) in novel 3D indoor scenes. Given an input 3D scene with multiple objects (left), we predict a functional scene graph (middle) with IFRs (e.g., knife cuts fruits, the microwave button triggers its door open). Our proposed method learns prior knowledge over object geometry (green dashed lines) and posteriors (yellow dashed lines) after performing several interactions (yellow hand signs). In cases of relationship uncertainty such as the stove example (right), learning from the interaction is necessary.
Given a novel indoor scene with multiple objects as the input (Fig. 1, left), we predict a complete directional graph of inter-object functional relationships (Fig. 1, middle) on how the state change of one object causes functional effects on other objects. While the semantics of individual objects already suggest some functional priors (e.g., buttons can be pressed, electrical devices can be turned on), our system learns to build up the correspondence among them (e-g., which button controls which electrical device). Also, since one may encounter unknown object categories exploring novel scenes, we do not assume semantic labels of objects and only use shape geometry as input. Our system, therefore, learns inter-object functional geometry-mappinn to what to trigger (e.g. one button among others on the wall) and what other shapes the interaction effects (e.g.. one light in the room).
Though seemingly simple one may interact with all objects in the scene to query the functional scene graph, it is actually very challenging how to efficiently and accurately perceive, reason, and explore the scene with few interactions possible. First of all, we do not want to waste time interacting with books and bags as they will not functionally trigger other objects. Secondly, the semantics of two objects may directly indicate functional relationships. For example, that buttons control electrical devices in general. Besides, in cases of having two buttons and two lights in the scene, one may guess the pairings according to spatial proximity. All of these examples indicate that learning some geometric and semantic priors among the objects may help predict many inter-object functional relationships even without any interaction. However, there are always scenarios where uncertainties exist. For example, for the stove highlighted in Fig. 1 (right), interactions have to be taken place figure out which knob controls which burner, though only two interactions are needed for the four pairs if our agent can smartly infer the other two automatically.
know
We design a two-staged approach, under a sulf-supe interactive learning-free wrotinn---ro-mee www.-frooppppppee eopo-exalooation framework, to tackle the problem. The first stage perceives and reasons over the objects in novel test scene to predict possible inter-object functional relationships using learned functional priors about the object geometry and scene layout. Then, in the second stage, our system performs a few interactions in the scene to reduce uncertainties and fastly adapts to posterior functional scene graph prediction. We jointly train reinforcement learning policy explore large-scale training scenes and use the collected interaction observations to supervise the prior/posterior networks in the system.
Since the first to explore this task our best knowledge, create new hybrid dataset based on AI2THOR (Kolve et al., 2017) and PartNet (Mo et al., 2019b) as the benchmark, on which we conduct extensive experiments comparing to various baselines and illustrating our interesting findings. Experiments validate that both stages contribute to the final predictions and prove the training synergy of the exploration policy and the prior/posterior networks. We also find reasonable generalization capabilities when testing over novel scene types with unknown object categories.
In summary, this work makes the following main contributions:
we first formulate an important, interesting, yet underexplored challenging task - Icarning inter-object functional relationships in novel 3D indoor environments;
we propose a novel self-supervised learning- Beaming--rom-exxy from-exploration framework combining a prior-inference stage and iist-interactive-daaatttto stage tackle the problem;
2

---

Published as a conference paper at ICLR 2022
we create a new dataset for benchmarking the proposed task and demonstrating the effec- tiveness of the proposed approach.
2 RELATED WORKS
Visual Functionality and Affordance. We humans perceive and interact with the surrounding 3D world to accomplish everyday tasks. To equip Al systems with the same capabilities, researchers have done many works investigating shape functionality (Kim et al., 2014; Hu et al., 2016; 2018; 2020; Lai et al., 2021; Guan et al., 2020), grasp affordance (Lenz et al., 2015; Redmon & Angelova, 2015;
Montesano & Lopes, 2009; Qin al., 2020; Kokic al., 2020; Mandikal & Grauman, 2021; Yang et al., 2020; Corona et al.. 2020; Kjellström et al., 2011; Fang et al., 2018; Nagarajan et al., 2019;
Brahmbhatt et al., 2019; Jiang al., 2021), manipulation affordance (Do al., 2018; Nagarajan et al., 2019; 2020; Nagarajan & Grauman, 2020; Mo et al., 2021a), scene affordance (Piyathilaka & Kodagoda, 2015; Rhinehart Kitani, 2016; Li al., 2019b), etc. While these works mostly study single-object/seene or agent-object interaction scenarios, our work explores inter-object relationships. Besides, we investigate how to explore functional relationships by interacting in 3D scenes, different from previous works that study interactions given specific object pairs or tasks (Jiang et al., 2012; Sun et al., 2014; Zhu et al., 2015; Mo et al., 2021b; Cheng et al., 2021).
Inter-object Relationships. Besides perceiving individual objects in the environment, understand- ing the rich relationships among them is also crucial for many downstream applications. Previous works have attempted to model object supporting and contact relationships (Fisher et al., 201 Silberman et al., 2012). spatial relationships (Galleguillos et al., 2008; Gould et al., 2008; Kulkarni et al., 2019), semantic relationships (Sadeghi & Farhadi, 2011; Divvala et al., 2014), physical relation- ships (Mitash et al., 2019), etc. More recently, researchers have introduced scene graphs (Liu et al., 2014; Johnson et al., 2015; Krishna et al., 2017; Xu et al., 2017; Li et al., 201 7b; Zhu et al., 2020; Wald et al., 2020) and hierarchies (Armeni et al., 2019; Li et al., 2019a; 2017a; Mo et al., 2019b;a) to parse the objects, object parts, and their relationships. Unlike the spatial, geometric, semantic, or physical relationships investigated in these works, we pay more attention to inter-object functional relationships in which the state change of one object causes functional effects on other objects.
Learning from Exploration and Interaction While deep learning has demonstrated great success in various application domains (Russakovsky et al., 2015; Silver et al., 2016), large-scale annotated data for supervision inevitably becomes the bottleneck. Many works thus explore self-supervised learning via active perception (Wilkes & Tsotsos, 1992), interactive perception (Bohg et al., 2017), or interactive exploration (Wyatt et al., 2011) to learn visual representations (Jayaraman & Grauman, 2018; Yang et al., 2019; Weihs et al., 2019; Fang et al., 2020; Zakka et al., 2020), objects and poses (Caicedo Lazebnik, 2015; Han et al., 2019; Deng et al., 2020; Chaplot et al., 2020b; Choi et al., 2021), segmentation and parts (Katz& & Brock, 2008; Kenney et al., 2009; Van Hoof et al., 2014; Pathak et al., 2018; Eitel et al., 2019; Lohmann et al., 2020; Gadre et al., 2021), physics and dynamics (Wu et al., 2015; Mottaghi et al., 2016; Agrawal et al., 2016; Li et al., 2016; Janner et al., 2018; Xu et al., 2019; Lohmann et al., 2020; Ehsani et al., 2020), manipulation skills (Pinto & Gupta,
2016: Levine al., 2018; Agrawal et al., 2016; Pinto et al., 2016; Pinto & Gupta, 2017: Zeng et al.,
2018; Batra et al., 2020), navigation policies (Anderson et al., 2018; Chen et al., 2019; Chaplot et al.,
2020a; Ramakrishnan al., 2021), etc. In this work, we design interactive policies to explore novel
3D indoor rooms and learn our newly proposed inter-object functional relationships.
3 PROBLEM FORMULATION
We formulate a novel problem of inferring inter-object functional relationships (IFRs) among objects in novel 3D indoor environments (Fig. 1). For the task input, each scene S comprises multiple 3D objects (O,, O2 O,). each of which is presented as 3D point cloud with 2,048 points depicting the complete shape geometry. Every object O may be subject to state changes (e.g., water is running or not from a faucet, button is pressed up/down). The set of 3D objects describe the complete 3D scene 5 - (O,, O2. O,} including both the objects with and without possible state changes. The spatial layout of the objects in the scene is explicitly modeled that each object O, = (O,,ci,s) can be decomposed into a zero-centered unit-sized shape point cloud Ô E 2g2004833 the object center position in the scene c; E RR3, and its isotropic scale E R. For the task output, we are interested to figure out the (IFRs) Ls. In a case that interacting with an object O (a trigger) will cause a functional state change of another object O, (a responder), we formally define observed IFR (O,, O)) Bs. In the task output Ls. there could be one-to-one or many-to-mane relationships between two objects, as trigger object may cause functional state changes of many responder objects and vice versa.
3

---

Published as a conference paper at ICLR 2022
Interaction & Posterior-Net
Exploration
Prior-Net
Large-scale scenes
Prior knowledge
New scene
Posterior
Figure 2: System Overview. During training (left), we jointly learn an RL policy to explore large- scale scenes and use the collected interaction observations to supervise the prior/posterior networks. In the test time (right), given novel scene, first propose possible inter-object functional relationships by applying the learned prior knowledge and then perform interactions to resolve uncertain cases.
We define a task scenario that an agent actively explores the environment and interacts with the objects to output the inter-object functional relationships following the popular embodied and real-world settings (Nagarajan & Grauman, 2020; Ramakrishnan et al., 2021). For each interaction, an agent picks an object, triggers its state change, and directly receives the state changes of the other objects that have been affected in the scene. We assume that standardized visual quantities (e.g., object segmentation, state changes) themselves (Geiger et al., 2012; Li et al., 2020), and thus that the functional effect of a relationship is directly observable. An agent is provided with large-scale scenes to explore for learning in the training stage and is asked to predict the functional scene graph (S,85) ) for a novel scene at the test time. We also abstract away complexities on robotic navigation (Anderson et al., 2018; Ramakrishnan et al., 2021) and manipulation (Nagarajan & Grauman, 2020; Mo et al., 2021a), which are orthogonal to our contribution estimating inter-object functional relationships.
provided the agent as they well-established research topics by
4 TECHNICAL APPROACH
Fig. 2 presents the system overview. Given a novel indoor scene, the prior-inference stage first produces a functional scene graph prior (5, by perceiving and reasoning over the objects in the scene S = {O), O2, "0,) Then, in the inst-adaptttion stage, the agent performs as few interactions as possible to reduce uncertainties and updates prior into posterior for the final functional scene graph prediction (S. We train an exploration policy that learns to effectively and efficiently explore large-scale training scenes and collects interaction data to supervise the prior/posterior networks.
4.1 THE PRIOR-INFERENCE STAGE
We design two prior networks - BR-Prior-Net and SR-Prior-Net, for the functional scene graph prior learning, where BR-Prior-Ner focus on modeling binary relationship priors 2. E [0, I] given only two input shape geometry (0;: O,) (Vi,)) while -R-Prior-Net reasons scene-level functional relationship priors 23 = (ri,, E [0, Oj E S} considering all the objects in the scene S = (0) Oz, On).
4.1.1 BINARY- RELATIONSHIP PRIOR NETWORK (BR-PRIOR-NET)
Given two objects the scene (e.g., a button and light), humans often have binary priors the two objects are functionally correlated or not (e.g., the button may control the light) according to geometry and semantics of the two shapes, regardless of their locations in the scene. Having this observation, we thus design binary-relationship prior network (BR- (BR-Prior-Net) to model the possibility of functional relationship for every pair of trigger object O and responder object Oj.
As shown in Fig. (left), our BR-Prior-Net takes two object point clouds O,,0, E 2048 as input and outputs the belief of IFR P,, 10. By allowing O, = =,, we also consider self-relationshhpp (e.g., triggering the button on the microwave opens its door). We train a shared PointNet++ network Qi et al. (2017), with four set abstraction layers with N [51 [512, 128, 32, 1] point resolutions, process the two point clouds and output global features f f 644 that summarize the two input shapes. We then concatenate the two obtained features and employ Multilayer Perceptron (MLP) network to estimate r E [0, 1] for the final output. We obtain the binary functional scene relationship prior S (ri, E [0, 1]|0,, 0z S} by enumerating all object pairs in the scene.
4.1.2 SCENE-RELATIONSIIE PRIOR NETWORK (SR-PRIOR-NET
While the binary-relationsie prior network empirically performs well in predicting pairwise func- tional relationships effectively scopes down the problem scale to input objects only, much
4

---

Published as a conference paper at ICLR 2022
BR-Prior-Net
Binary prior graph
SR-Prior-Net & SR-Posterior-Net
Scene prior graph
Scene posterior graph
Sharod PN++
MLP
GCN
Figure 3: Network Architecture. The BR-Prior-Ne takes two object point clouds input and estimates their functional relationship likelihood. By passing all object pairs through BR-Prior-Net, we obtain a binary functional graph prior 25. Then, employ Graph Convolutional Network (GCN) to model the scene context and predict a scene functional graph prior L' We repurpose the same GCN as SR-Posterior-Ne which produces a scene posterior graph S after each interaction stepr. The agent outputs final functional scene graph prediction Hy after addressing all uncertainties.
important scene-level information is ignored, such as relative object poses and scene object layout. Such scene context may provide clues on relationship pairings in cases of uncertainties. For example, the BR-Prior-Nes may connect all buttons to all lights in the room, but given the object layout, one may assign each button its closest light with higher probabilities. Therefore, we further introduce scene-relationship prior network (SR-Prior-Ned to take the scene context into consideration.
The SR-Prior-Net is implemented as three-layer Graph Convolutional Network (GCN) (Kipf Welling, 2016) that connects and passes messages among all objects in the scene. Fig. 3 (right) shows an overview of the design. Each object O; is represented as node in the graph, with a node
feature ni; (0) that combines the +ointNet++ geometry feature and the scene-contextual information
(e.g., object position and size). The edge between two nodes is initialized with wif = = predicted by the BR-Prior-Net. To allow better graph message passing throughout the graph. we additionally connect two nodes if the two objects are close enough (< 0.5m, which covers about 53.1 1% IFRs in our dataset) because closer objects are more likely to have functional relationships, e.g., a stove knob is close to the corresponding stove burner. We use an edge weight of w y= -=0.6, 0.6, which experimentally tuned out (see Table E.5). After three iterations of graph convolutions
(k-1)
(R)
w
vk 1,2,3
where d -1 E, with 1k-1) and C denotes the learnable parameters, the network outputs a scene- contextual embedding 1 n E R2 for each object O,. Next, for each object pair, we concatenate their feature embeddings, feed it through an MLP. and predict a likelihood score for their functional relationship r E [0, 1]. The final scene-level functional scene relationship prior R3 (ri,+ e [0, 10]1,,, Oj E S} is obtained by enumerating all object pairs in the scene.
4.2 THE FAST-INTERACTIIV-ADAAPPPA
STAGE
While the prior networks do their best to propose functionally related objects by perceiving and reasoning over objects in the scene, there are always uncertain cases that priors do not suffice (e.g., the stove example in Fig. 1 (right)), where interactions are needed to reduce such uncertainties. A smart agent only needs to perform very few interactions as needed and is capable of inferring other pairings. We therefore design JR-Posterior-Net to learn such posterior reasoning after interactions and propose test-time fast-adaptation strategy that sequentially picks few objects to interact.
4.2.1 SCENE-RELAAIOONSS
POSTERIOR NETWORK (SR- POSTTERIOOR--NTT
The scene-relationship posterior network SSR-Posterior-Ner shares the same network architecture and weights to SR-Prior-Ner introduced in Sec. 4.1.2. We only repurpose the network to model functional scene graph posteriors by simply changing the network input to the current functional scene relationship belief modulated by the observed interaction outcomes. In our experiments, we perform interactions over objects one-by-one sequentially and thus feed through SR-Posterior-Net multiple times to evolve the functional scene relationship posteriors 2.th (t = 1,2, ,TT) after each interaction, where T's denotes the total number of interactions performed in this scene.
5

---

Published as a conference paper at ICLR 2022
Concretely, when O, we use S = Gz. which is the functional scene relationship prior estimated by SR-Prior-Net without performing any interaction. For each timestep r. given the interaction observations (ei. E (0. 10, E $ after interacting with object Oi, where ei = if and only if O; triggers Oj. update the posterior sg!) with all past observations
(() f., et.j (i i1,42,"... .4,j
1,2, ,n), feed the modified posterior through SR-Posterior-Net, and obtain a functional scene relationship posterior ( for the next timestep I + 1. We compute the final functional scene { E T s t, Vi with a threshold : == 0.9.
s
relationship prediction B3 {(,,,)))) (Ts)
4.2.2 TEST-TIME FAST-INTERACTIVE- ADAPPTTIOON STRATEGY
Now, the only questions left are how pick the object interact with each timestep and when to stop. We find that a simple heuristic to explore the object that is predicted most uncertainly in our posterior works very well in our experiments. Specifically, at each step r, the agent picks O that
i = arg
max
min
v 0, ,
We stop the test-time
procedure when all predictions are certain enough,
min (2.1-1)) < vi.
with a threshold 0.05, the agent explores about 19% of objects in the room on average. can also be enlarged to reduce number of interactions. Another strategy is to fix the test-time interaction budget, e.g., 10% or 20% of the total number of objects in each room.
4.3 SELF-SUPERVISED LEARNING FROM EXPLORATION
We adopt the embodied paradigm of learning from exploration and interaction (Nagarajan & Grauman, 2020; Ramakrishnan et al., 2021) that an agent actively explores the large-scale training scenes, interacts with the environments, and collects data for supervision. It is challenging task how to effectively prohe the environments for self-supervised learning and perform as few interactions as possible for efficiency. We formulate the task as a reinforcement learning (RL) problem and learn an exploration policy that collects data for supervising the prior and posterior networks.
4.3.1 EXPLORATION POLICY
The RL policy, implemented under the Proximal Policy Optimization (PPO) (Schulman et al., 2017) framework, takes the current functional scene graph representation as the state input state (5.2)") (
at each / / and picks one object as the action output to interact with action (t) E. When O, we use R R5. which is predicted by the SR-Prior-Net (Sec. 4.1.2). After each interaction, the agent observes the interaction outcome that some or no objects have functional state changes triggered by O,, namely "bservation!) = {ee, E {0, 1) E S} where et, if and only if O, triggers O,. which will then be passed to the SR- SR-Posterior-Ne (Sec. 4.2.1) to update the functional
relationship belief (((1)) for the next timestep 1. The agent may decide stop exploring the current scene and move on to the next one under a limited interaction budget.
We design the RL reward function follows.
reward ( a. 1/
e1,l + B -y
(4)
The first term encourages interacting with O to correct the wrong or uncertain functional relationship
beliefs (t) f. j.. There are three cases for this term: 1) the agent's belief is already correct (!)) ezj+
thus 1r2 e.. = 0); 2) the belief is wrong (1 = ei.j. thus Ir:] ei,l = 1); 3) the agent is (p) uncertain (e.g.. Fi. 0.5, thus In! ei,l 0.5). We use max instead of avg the aggregation function because the IFRs are sparse and an average value may conceal the mistake. The second term rewards exploring trigger objects of interest, e.g., it encourages the agent to interact with buttons or switches, instead of boxes cabinets. The third term simulates the interaction cost to avoid redundant interactions for efficient exploration. We use a = 2, ß = and y = 1 in our experiments.
The RL network shares the same three-layer GCN backbone as the scene-relationship networks (Sec. 4.2.2 and 4.2.1) for reasoning over the functional scene graph (S. However, instead of predicting IFRs in SR-Prior-Net (Sec. 4.1 4.1.2) and SR-Posterior-Net (Sec. 4.2. 1), the GCN RL policy

---

Published as a conference paper at ICLR 2022
outputs action probabilities di 's over object nodes 0,'s, by firstly obtaining an embedding n/" R for each object O,, then concatenating with global scene feature nis all object embeddings, and finally feeding through an MLP to get an action score a; (t) In addition, we duplicate the scene feature ns and input it to the MLP to query the termination score dstop. All the action scores are finally passed through SoftMax layer for the action probability scores.
(1) computed by averaging across
4.3.2 TRAINING STRATEGY AND LOSSES
To supervise the proposed interactive
framework, we alternate the training of the exploration RL policy and the prior/posterior networks. We leverage the training synergy that the prior/posterior networks predict helpful information for better exploration of the scenes, while the exploration RL policy collects useful data to better train the two networks. In our implementation, we train the exploration policy and the prior/posterior networks to make they jointly converge.
In each loop, conditioned the current prior/posterior predictions, the exploration policy strategically explores the large-scale training scenes with a total interaction budget of 1000 (e.g.. roughly 32% of the total 3125 objects from 210 scenes). The exploration policy learns wisely allocate budgets across different scenes and pick the objects worth interacting with. Specifically, the agent can interact with m objects in one scene and explore the rest scenes with the 1000 m budget left. For a scene S = {01, O2.... ...., where the agent sequentially performs m interactions over objects (O,, 0 O,.,... i,,)). we obtain m sets of observations (fei,) [001) S}. (et,) E {0. 10, e S}.".. feim. E {0, HH00 E S}).
We then use the collected interaction observations to train the prior/posterior networks. For BR-Prior- Net, we simply train for the observed link predictions r², - et, (i = 11.12..... Jm. j = 1,2, .I). We learn one shared network for -R-Prior-Net and SR- UR-Posterior-Net but train it with two supervision sources. For learning the SR-Priors, we input the BR- BR-Prior-Net prediction 2 output the scene-level prior S and train for the observed link predictions as well ri,j -ie (i -i= 12, jm. 1,2, .!)
For training the SR-P PR-Pooters,rs, we the posterior estimate s at each timestep 1,2. m 1, /
update it with the past ground-truti observations (!) = (i i1, i2. j. = 1.2, ... .n), input to the network to obtain the posterior estimate sg(t+ / and supervise the link predictions r
(++1) + ei,)
4441,42, j = 1,2....,.. ,n). We use the standard binary cross entropy for all loss terms.
EXPERIMENTS
We evaluate our method on new hybrid dataset and compare it to several baselines that prove the effectiveness of the proposed method and its components. We also experiment with transferring to novel scene types that may contain novel shape categories. See appendix for more results.
5.1 DATASET
We create new hybrid dataset based on AI2THOR (Kolve et al., 2017) and PartNet- Mobiility (Mo et al., 2019b; Xiang et al., 2020) to support our study. AI2THOR provides 120 interactive scenes of 4 types (i.e., bathroom, bedroom, living room, and kitchen) containing more than 1500 interactive objects covering 102 categories. The PartNet-Mobility dataset contains 2,346 3D articulated object CAD models from 46 categories. We replace 83 AI2THOR objects with 888 PartNet- MorbiityMoollity objects for 31 categories (12 exist in AI2THOR while 19 are newly added) to further enrich the diversity of the shape geometry. We use 27 kinds of functional relationships originally built in AI2THOR (e.g., the microwave can be toggled on) and also enrich 21 more types of inter-object functional relationships (e.g., the pen can be used to write in the book).
In total, our dataset contains 1200 scenes covering 23360 objects from 121 categories. We enrich the origin 120 AI2THOR scenes by 1) randomly spawn objects into different locations and 2) randomly replace the object geometries of the same type. Fig. A.6 shows one example room from each of the four types. Among these, 27 are trigger objects, 29 are responders, where 11 are both trigger and responders (e.g., desk lamp), and 79 are non non-interactive background objects. We split the dataset into non-overlapping 800 training scenes and 400 test scenes. Table A.3 presents more detailed statistics.
5.2 BASELINES AND METRICS
Since we are the first to study the problem to our best knowledge, there is no external baseline for directly fair comparisons. However, to validate the effectiveness for each of our proposed
7

---

Published as a conference paper at ICLR 2022
Table Quantitative Evaluations. We compare several baselines or ablated versions, and report the precision (P), recall (R), Fl-score (F1), and Matthews correlation coefficient (MCC) given the test-time interaction budget as 10% or 20% of the per-room object counts.
Methed
F
Sulhrae
F1
MCCC
P
R
Indreae
F1
MCC
P
R
Eiiing Roses
F1
MCC
F
MCC
Kandom (UBS))
1,006
0.100
00.55
0.317
1,000
0.100
0.157
0.258
1,601
0.102
10135
0.257
1,000
0.125
0.224
0.352
Random (209g)
1,000
0.232
0.377
0.438
1,000
0.201
0.335
0.427
1,000
0.216
0.335
0.427
1,000
0.20
0.346
0.400
ANa NadiraryPror (196)
0.837
0.50
0.750
0.77
0.416
0393
0.028
0/00
0.297
03290
01315
01388
0.53
0363
0.01
1,000
0.274
0.315
0.70
0.178
0.383
0.578
0.688
0.681
0.734
Abia- Rape-n (10%)
0.543
0.934
0.708
0338
0.908
0.556
0.155
0.845
0.656
0.799
0.657
ANo Randondddage (10%)
0,891
0.13
0.766
0.366
0.776
0.10
0.299
0.701
0.624
Abo NoBttaryPror (2046)
0.755
Abla NoScenePrior
0.17
01733
0.854
0.653
0.026
0.855
Roll-aaaonnhlle (209))
0.915
0.581
0.772
0.668
0,353
0.335
Inserction Explention
1,000
0.528
0.720
0.695
1,000
0.354
0.556
Ous Fazl T195)
0.50
0311
0.556
0.31
0.30
0.785
0.731
DanwFiaal (20%)
0.937
0.994
1.957
0.958
0,677
0.967
0.924
0.927
0.892
0.924
I.983
0.905
0.548
0.890
0.964
0.665
58/59/
5/23/
50
37
47
60/61
27733
Figure 4: Qualitative Results. We show top-down views of one example scene. In the bottom row, from left to right, we show the functional scene- graph predictions given by binary priors, scene priors, posteriors with 10% interactions, posteriors with 20% interactions. In these figures, we mark the ground-truth relationships in blue lines, the predicted ones in red lines, the interaction observations in green lines, and the interacted objects with green cross symbols. We assign each object with yellow ID linking to one of the top-row subfigures for some example object zoom-ins. The green IDs in the top row mean the object has been interacted during exploration.
modules (e.g., , BR-Prior-Net, SR-Prior-Net, the fast-interactive-adaaee exploration policy), compare Ours-Final several basclines and ablated versions:
strategy, the self-supervised
Random:
the agent performs random actions in the scene the test time;
Abla-NoBinaryPriane the agent does not use the binary priors and instead initializes the input edges to SR-Prior-Net with w 1;
A Ablo-NoScenePrior: the agent uses the predicted binary-relationship prior S instead of the estimated scene-relationship prior 23 the prior knowledge;
Mola-RandomExploon the agent does use the proposed exploration policy and instead randomly explores training scenes during training;
the agent randomly picks objects to interact with during adaptation;
Curs-PriorOnly the agent directly uses the scene-relationship prior predictions S3 and threshold 0.5 to predict the final outputs without any interaction;
Interaction- Interaction-Exploratonn
try best adapt Nagarajan & Grauman (2020)
ting, though it originally tackles very different task on learning affordance over individual shapes in a scene and does not propose IFRs. We use a PointNet++ encode object features and an MLP to additionally predict set of objects that each interaction at object causes.
We use the standard precision, recall, and Fl-score for quantitative evaluations and comparisons. Additionally, we set a per-room interaction budget for fairly comparing all the methods and report their performance when the budgets are 10% and 20% of the object count in each room.
5.3 RESULTS AND ANALYSIS
Table 1 presents the quantitative evaluations and comparisons to baselines, where Final achieves the best scores in Fi-score and MCC in all comparisons under different interaction budgets. The Random baseline observes the direct interaction outcomes and thus by design gives
that Ours-
8

---

Published as a conference paper at ICLR 2022
Table 2: We present quantitative evaluations when transferring our network trained on one type of rooms to other types at the test time that may also contain unseen object types. We additionally show results transferring to RoboTHOR, whose scenes are visually different from AI2THOR.
Bathowan
Bedrwas
Living Room
Kitchen R
Babo
Method
F1
R
F1
P
R
F1
F1
P
F1
Bathroom
0.823
0.764
0.789
0.799
0.750
0.768
8.800
0.554
0.652
03800
0.630
0.688
Bodroom
0,700
0.530
0.583
0.871
0.916
0.885
0.222
0,402
0.318
0.561
0.844
0.663
Living Room
0.612
0.444
0.514
0.753
0,818
0.779
0.817
0.458
0.576
0.817
0.458
0.576
Kitchen
0.365
0.569
0.444
0.943
0.614
perfect precision but random interaction renders very low recalls and F1 scores. Compared to the ablated versions, we validate the effectiveness of our learned binary-prior, scene-prior, the proposed policy. also
has-iiterratiie--aaaattiinn strategy, and the self-supervised meets our expectation that the performance improves with more interactions (i.e., from 10% to 20%), which is further validated by Fig. D.12. We show qualitative results in Fig. 4 and analyze them below.
Is the Learned Prior Knowledge Useful? In Table 1, we see that Durs-PriorOniy already gives without performing any interaction, outperforming Random by large
decent performance margin, which proves the usefulness of the learned prior knowledge. As shown in Fig. 4 (the left-most figure in the bottom row), the prior knowledge correctly predicts many relationships (red lines) to the ground-trute ones (blue lines). In addition, both BR- AR-Prior-Ne. and SR-Prior-Nee are helpful, as observed in the ablated versions Abla-NoBinary Priory and A Abla-NoScenePPror We find that the BR- Re-PPrir--et makes a lot of false-posiae positive predictions and the them given the scene context. The performance drop of Abla-NoBinaryPrioo further illustrates the need for binary priors, which initialize SR- (R-Prior-Net with a sparse graph, better than dense one.
successfully removes
Does Rnst-Interacttie--ee tittterraciiv-AAaaaattiii Adaptation Help? The performance of Durs-PriorOnle is further im- proved given test-time interactions and posterior adaptions as Ours-Final generates better numbers. In Fig. 4, we see that some redundant red lines removed observing the interaction outcomes and applying the SR-Posterior-NG predictions. Compared to Abla-RandomAdap that performs random interactions, we find that the fast-iterraciiv-adaaattiin strategy performs better. Qualitatively, besides the stove case mentioned in Fig. 1 (right), another interesting case in Fig. 4 is that from prior knowledge the agent thinks the two knives (ID: 57/67) may cut egg (ID: 50), maybe because it looks like tomato (ID: 62) in geometry. The agent only interacts one knife (ID: 57) to correct this wrong belief and then the false-positive edge from the other knife (ID: 67) is automatically removed.
Is Exploration Policy Effective? We learn an exploration policy for efficient exploration of the training scenes. Abla-RandomExplor gives clear performance drop when using randomly col- lected data for training, proving that the exploration policy can wisely pick objects to explore.
5.4 TRANSFER TO NOVEL ROOM AND OBJECT TYPES We further test model's generalization capability to objects and rooms of novel types. In Table 2, we report quantitative evaluations where we train our systems on one type of AI2THOR rooms and test over the other types, as well as additional results of testing over totally out-of- distribution RoboTHOR (Deitke et al., 2020) test dataset (see Fig. A.7). We find that 1) it works in most cases showing that our learned skills can transfer to novel room types; 2) our system can also generalize to novel object categories. For example, as shown in Fig. 5, although the agent never sees the television nor remote in the training scenes of bedrooms, it thinks the TV might be controlled by the remote. 3) the prior knowledge is not working well because of the data distribution shift but the posterior interaction helps. For example, in Fig. the interaction helps make accurate predictions despite the prior is not very accurate.
24
Figure 5: Bedrooms to living rooms.
6 CONCLUSION
We formulate novel task of learning inter-object functional relationships in novel 3D indoor environments and propose a two-staged interactive problem. Using a newly created hybrid dataset, we provide extensive analysis of our method and report quantitative comparisons to baselines. Experiments comparing to several ablated versions validate the usefulness of each proposed module. See appendix for limitations and future works.
approach to tackle the

---

Published as a conference paper at ICLR 2022
Ethics Statement. We explore Al algorithms towards building future home-assistani robots which can aid people with disabilities, senior people, etc. While we envision that our proposed approach may suffer from the bias in training data, the privacy of using human-designed data, and domain gaps between training on synthetic data and testing in real-world environments, these are the common problems that most if not all learning-based based systems have. We do not other partieular harm issue of major concerns that our work may raise.
Reproducibility Statement. We support open-soureed research and will make sure that our results are reproducible. Thus, we promise release the code, data, and pre-trained models publicly the community upon paper acceptance.
Acknowledgements IIS-1763268.
This paper is supported by a Vannevar Bush Faculty fellowship and NSF grant
REFERENCES
Pulkit Agrawal, Ashvin Nair, Pieter Abheel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. arXiv preprint arXiv: 1ot.:v::60000719999 2016.
Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, al. On evaluation of embodied navigation agents, arXiv preprint arXiv: 1xX7::807700677 2018.
Iro Armeni, Zhi- Yang He, Jun Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5664-5673, 2019.
Dhruv Batra, Angel X Chang, Sonia Chernova, Andrew Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, et al. Rearrangement: A challenge for embodied ai. arXiv preprint arXiv:2011.011 2020.
Jeannette Bohg, Karol Hausman, Bharath Sankaran, Oliver Brock, Danica Kragic, Stefan Schaal, and Gaurav S Sukhatme. Interactive perception: Leveraging action in perception and perception in action. JEEE Transactions on Robotics, 33(6):1273 1291, 2017.
Samarth Brahmbhatt, Cusuh Ham, Charles C Kemp, and James Hays. Contactdb: Analyzing and predicting grasp contact via thermal imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770998719, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint (606.0v600005544 2016.
Juan C Caicedo and Svetlana Lazebnik. Active object localization with deep reinforcement learning. In Proceedings of the IEEE international conference computer vision, pp. 2488-2498 2496, 2015.
Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to explore using active neural mapping. In International Conference Learning Representations, 2020a. URL https: htttpp::///peeeeviieee httprriiwwwwee.ddffffeeeeee
net http://ddddddttk11XX0000
Devendra Singh Chaplot, Helen Jiang, Saurabh Gupta, and Abhinav Gupta. Semantic curiosity for active visual learning. In European Conference on Computer Vision, pp. 309-326. Springer, 2020b.
Tao Chen, Saurabh Gupta, and Abhinav Gupta. Learning exploration policies for navigation. In International Conference on Learning Representations, 2019. URL httpps://penrevieer net/ffrumt forum? d=SyMMn05F7)
Shuo Cheng, Kaichun Mo, and Lin Shao. Learning to regrasp by learning to place. In Conference on Robot Learning ( (CoRI.), 2021.
Jiwoong Choi, Ismail Elezi, Hyuk-Jae Lee, Clement Farabet, and Jose M Alvarez. Active learning for deep object detection via probabilistic modeling. arXiv preprint 16: 2021.
10

---

Published as a conference paper at ICLR 2022
Enric Corona, Albert Pumarola, Guillem Alenya, Francese Moreno-Noguee and Grégory Rogez. Ganhand: Predicting human grasp affordances multi-object scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 5031-5041, 2020.
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, Luca Weihs, Mark Yatskar, and Ali Farhadi. RoboTHOR: An Open Simulation-to-Real Embodied Al Platform. In CVPR, 2020.
Xinke Deng, Yu Xiang, Arsalan Mousavian, Clemens Eppner, Timothy Bretl, and Dieter Fox. Self-supervised 6d object pose estimation for robot manipulation. In 2020 IEEE International Conference Roboties and Automation (ICRA), pp. 3665-3671. IEEE, 2020.
Santosh K Divvala, Ali Farhadi, and Carlos Guestrin. Learning everything about anything: Webly- supervised visual concept learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3270-322 2014.
Thanh- Toan Do, Anh Nguyen, and Ian Reid. Affordancenet: An end-to-end deep learning approach for object affordance detection. In International Conference on Robotics and Automation (ICRA), 2018.
Kiana Ehsani, Shubham Tulsiani. Saurabh Gupta, Ali Farhadi, and Abhinav Gupta. Use the force, luke! learning to predict physical forces by simulating effects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 224-233, 2020.
Andreas Eitel, Nico Hauff, and Wolfram Burgard. Self-supervised transfer learning for instance seg- mentation through physical interaction. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4020 4026. IEEE, 2019.
Kuan Fang, Ti-Lin Wu, Daniel Yang. Silvio Savarese, and Joseph J Lim. Demo2vec: Reasoning object affordances from online videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2139-2147, 2018.
Zhaoyuan Fang, Ayush Jain, Gabriel Sarch, Adam Harley, and Katerina Fragkiadaki. Move to see better: Towards self-supervised amodal object detection. arXiv preprint arXiv:2012.00005 2020.
Matthew Fisher, Manolis Savva, and Pat Hanrahan. Characterizing structural relationships in scenes using graph kernels. In ACM SIGGRAPH 2011 papers, pp. 1-12. 2011.
Samir Yitzhak Gadre, Kiana Ehsani, and Shuran Song. Act the part: Learning interaction strategies for articulated object part discovery. International Conference on Computer Vision (ICCV), 2021.
Carolina Galleguillos, Andrew Rabinovich, and Serge Belongie. Object categorization using co- occurrence, location and appearance. In 2008 IEEE Conference Computer Vision and Pattern Recognition, 1-8. IEEE, 2008
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
Stephen Gould, Jim Rodgers, David Cohen, Gal Elidan, and Daphne Koller. Multi-class segmentation with relative location prior. International journal of computer vision, 80(3):300-316, 2008
Yanran Guan, Han Liu, Kun Liu, Kangxue Yin, Ruizhen Hu, Oliver van Kaick, Yan Zhang, Ersin Yumer, Nathan Carr, Radomir Mech, et al. Fame: 3d shape generation via functionality- model evolution. IEEE Transuctions on Visualization and Computer Gruphics, 2020.
Xiaoning Han, Huaping Liu, Fuchun Sun, and Xinyu Zhang. Active object detection with multistep action prediction using deep qenetwork. IEEË Transactions on Industrial Informatics, 15(6): 3723-3731, 2019.
11

---

Published as a conference paper at ICLR 2022
Ruizhen Hu, Oliver van Kaick, Bojian Wu, Hui Huang, Ariel Shamir, and Hao Zhang. Learning how objects function via co-analysis of interactions. ACM Transactions on Graphics (TOG), 35(4): 1-13, 2016.
Ruizhen Hu, Manolis Savva, and Oliver van Kaick. Functionality representations and applications for shape analysis. In Computer Graphics Forum, volume 37, pp. 603-624. Wiley Online Library, 2018.
Ruizhen Hu, Zihao Yan, Jingwen Zhang, Oliver Van Kaick, Ariel Shamir, Hao Zhang, and Hui Huang. Predictive and generative neural networks for object functionality. arXiv preprint arXiv:2006. 15520, 2020.
Michael Janner, Sergey Levine, William Freeman, Joshua B Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning about physical interactions with object-oriented prediction and planning. arXiv preprint UrXiv:1812.10972, 2018.
Dinesh Jayaraman and Kristen Grauman. End-to-end policy learning for active visual categorization. IEEE transactions on pattern analysis and machine intelligence, 41(7):1601-1614 2018.
Hanwen Jiang. Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. International Conference on Computer Vision (ICCV), 2021.
Yun Jiang, Marcus Lim, Changxi Zheng, and Ashutosh Saxena. Learning to place new objects in scene. The International Journal of Robotics Research, 1100):122111043, 2012.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2015.
Dov Katz and Oliver Brock. Manipulating articulated objects with interactive perception. 2008 IEEE International Conference on Robotics and Automation, pp. 272-277. IEEE, 2008.
Jacqueline Kenney, Thomas Buckley, and Oliver Brock. Interactive segmentation for manipulation in unstructured environments. In 2009 IEEE International Conference on Robotics and Automation, pp. 1377-1382. IEEE, 2009.
Vladimir G Kim, Siddhartha Chaudhuri, Leonidas Guibas, and Thomas Funkhouser. Shape2pose:
Human-centric shape analysis. ACM Transactions on Graphics (TOG), 33(4):1-12, 2014.
Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: 1609, (29090 2016.
Hedvig Kjellström, Javier Romero, and Danica Kragić. Visual object-action recognition: Inferring object affordances from human demonstration. Computer Vision and Image Understanding, 115 (1):81-90, 2011.
Mia Kokic, Danica Kragic, and Jeannette Bohg. Learning task-oriented grasping from human activity datasets. IEEE Robotics and Automation Letters, 5(2):3352-3359, 2020.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123 2017.
Nilesh Kulkarni, Ishan Misra, Shubham Tulsiani, and Abhinav Gupta. 3d-relnet: Joint object and relational network for 3d prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2212-2221, 2019.
12

---

Published as a conference paper at ICLR 2022
Zihang Lai, Senthil Purushwalkam, and Abhinav Gupta. The functional correspondence problem.
arXiv preprint arXiv:2109.0109) 2021.
lan Lenz. Honglak Lee, and Ashutosh Saxena. Deep learning for detecting robotic grasps. The International Journal of Robotics Research, 34(4-5):705-722 -224, 2015.
Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand- eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, 37(4-5):211-33, 2018.
Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. Grass: Generative recursive autoencoders for shape structures. ACM Transactions on Graphics (TOG), 36 (4):1-14, 2017a.
Manyi Li, Akshay Gadi Patil, Kai Xu, Siddhartha Chaudhuri, Owais Khan, Ariel Shamir, Changhe Tu, Baoquan Chen, Daniel Cohen-Or, and Hao Zhang. Grains: Generative recursive autoencoders for indoor scenes. ACM Transactions on Graphies (TOG), 38(2):1-16, 2019a.
Wenbin Li, Seyedmajid Azimi, Aleš Leonardis, and Mario Fritz. To fall or not to fall: A visual approach physical stability prediction. arXiv preprint 160. 2016.
Xiaolong Li, He Wang. Yi, Leonidas Guibas, A Lynn Abbott, and Shuran Song- Category-level articulated object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3706-3715, 2020.
Xueting Li, Sifei Liu, Kihwan Kim, Xiaolong Wang, Ming-Hsuan Yang, and Jan Kautz. Putting
humans in scene: Learning affordance in 3d indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12368-12376, 2019b.
Yikang Li. Wanli Ouyang. Bolei Zhou, Kun Wang, and Xiaogang Wang. Scene graph generation from objects, phrases and region captions. In Proceedings of the IEEE international conference on computer vision, pp. 1261-1270. 2017b.
Tianqiang Liu, Siddhartha Chaudhuri, Vladimir G Kim, Qixing Huang, Niloy Mitra, and Thomas Funkhouser. Creating consistent scene graphs using probabilistic grammar. ACM Transactions on Graphics (TOG), 33(6):1-12, 2014.
Martin Lohmann, Jordi Salvador, Aniruddha Kembhavi, and Roozbeh Mottaghi. Learning about objects by learning to interact with them. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
Chaitanya Mitash, Abdeslam Boularias, and Kostas Bekris. Physics-based scene-level reason- ing for object pose estimation in clutter. The International Journal of Robotics Research, pp. (078836919988) 191365511946551,, 2019.
Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas Guibas. Structurenet: Hierarchical graph networks for 3d shape generation. ACM Transactions on Graphics (TOG), Siggraph Asia 20/9, Articce 242, 2019a.
Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 909-918,100 2019b.
Kaichun Mo, Leonidas Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2act: From pixels to actions for articulated 3d objects. In International Conference on Computer Vision (ICCV), 2021a.
Kaichun Mo, Yuzhe Qin, Fanbo Xiang, Hao Su, and Leonidas Guibas. O20-Afford: Annotation free large-scale object-object affordance learning. In Conference Robot Learning (CoRL), 2021b.
13

---

Published as a conference paper at ICLR 2022
Luis Montesano and Manuel Lopes. Learning grasping affordances from local visual descriptors. In 2009 IEEE 8th international conference on development and learning. pp. 1-6. IEEE, 2009.
Roozbeh Mottaghi, Mohammad Rastegari, Abhinav Gupta, and Ali Farhadi. "what happens if..." learning to predict the effect of forces in images. In European conference on computer vision, pp. 269-285. Springer, 2016.
Tushar Nagarajan and Kristen Grauman. Learning affordance landscapes for interaction exploration in 3d environments. In NeurlPS, 2020.
Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman. Grounded human-object interac- tion hotspots from video. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8688-8697 2019.
Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman. Ego-topo: Envi- ronment affordances from egocentric video. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 163-172, 2020.
Deepak Pathak, Yide Shentu, Dian Chen, Pulkit Agrawal, Trevor Darrell, Sergey Levine, and Jitendra Malik. Learning instance segmentation by interaction. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pp. 2045,,045, 2018.
Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. 2016 IEEE international conference on rubolics and automation (ICRA), pp. 3406-3413. IEEE, 2016.
Lerrel Pinto and Abhinav Gupta. Learning push by grasping: Using multiple tasks for effective learning. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2161- 2168. IEEE, 2017.
Lerrel Pinto, Dhiraj Gandhi, Yuanfeng Han, Yong-Lae Park, and Abhinav Gupta. The curious robot: Learning visual representations via physical interactions. In European Conference on Computer Vision, pp. 3-18. Springer, 2016.
Lasitha Piyathilaka and Sarath Kodagoda. Affordance-map: Mapping human context in 3d scenes using cost-sensitive svm and virtual human models. In 2015 IEEE International Conference Robotics and Biomimetics (ROBIO), pp. 2035-2040. IEEE, 2015.
Charles R Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning point sets in metric space. arXiv preprint arXiv: 02413, 2017.
Yuzhe Qin, Rui Chen, Hao Zhu, Meng Song, Jing Xu, and Hao Su. S4g: Amodal single-view single-shot se (3) grasp detection in cluttered scenes. In Conference on robot learning, pp. 53-65. PMLR, 2020.
Santhosh Ramakrishnan, Dinesh Jayaraman, and Kristen Grauman. An exploration embodied visual exploration. International Journal of Computer Vision, pp. 1-34, 2021.
Joseph Redmon and Anelia Angelova. Real-time grasp detection using convolutional neural networks. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 1316-1322. IEEE, 2015.
Nicholas Rhinehart and Kris M Kitani. Learning action maps of large environments via first-person vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
580-588, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252 2015.
Mohammad Amin Sadeghi and Ali Farhadi. Recognition using visual phrases. IEEE, 2011.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (T0.:::77000347777 2017.
14

---

Published as a conference paper at ICLR 2022
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European conference on computer vision, pp. 746-760. Springer, 2012.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529((587):484 489, 2016.
Yu Sun, Shaogang Ren, and Yun Lin. Object-object interaction affordance learning. Robotics and Autonomous Systems, 6284):487 2014.
Herke Van Hoof, Oliver Kroemer, and Jan Peters. Probabilistic segmentation and targeted exploration of objects in cluttered environments. IEEE Transactions Robotics, 30(5):1198-1200 2014.
Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3961-3970, 2020.
Luca Weihs, Aniruddha Kembhavi, Kiana Ehsani, Sarah M Pratt, Winson Han, Alvaro Herrasti, Eric Kolve, Dustin Schwenk, Roozbeh Mottaghi, and Ali Farhadi. Learning generalizable visual representations via interactive gameplay. arXiv preprint arXiv: 1et::01120099 2019.
D Wilkes and JK Tsotsos. Active object recognition. In Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 136-141. IEEE, 1992.
Jiajun Wu, IIker Yildirim, Joseph Lim, Bill Freeman, and Josh Tenenhaum. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. Advances in neural information processing systems, 28: 127-135, 2015.
Jeremy L Wyatt, Peter Dayan, Ales Leonardis, and Jan Peters. Exploration and curiosity in robot learning and inference (dagstuhl seminar 11131). In Dagstuhl Reports, volume 1. Schloss Dagstuhl- Leibniz-Zentrum fuer Informatik, 2011.
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangehen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 11097-11107, 2020.
Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative mes- sage passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5410-5419, 2017.
Zhenjia Xu, Jiajun Wu, Andy Zeng, Joshua Tenenbaum, and Shuran Song. Densephysnet: Learning dense physical object representations via multi-step dynamic interactions. In Robotics: Science and Systems (RSS), 2019.
Jianwei Yang, Zhile Ren, Mingze Xu, Xinlei Chen, David Crandall, Devi Parikh, and Dhruv Batra. Embodied visual recognition. arXiv preprint arXiv:/904 204.04404, 2019.
Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Cpf: Learning a contact potential field model the hand-object interaction. arXiv preprint arXi::20/2.00924 2020.
Kevin Zakka, Andy Zeng, Johnny Lee, and Shuran Song. Form2fit: Learning shape priors for generalizable assembly from disassembly. In 2020 IEEF International Conference on Robotics and Automation (ICRA), pp. 4404-9410. IEEE, 2020.
Andy Zeng, Shuran Song, Stefan Welker, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. 2018.
Yixin Zhu, Yibiao Zhao, and Song Chun Zhu. Understanding tools: Task-oriented object modeling, learning and recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2855-2864 2015.
15

---

Published as a conference paper at ICLR 2022
Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Edmonds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, et al. Dark, beyond deep: paradigm shift to cognitive ai with humanlike common sense. Engineering, 6(3):310-345, 2020.
APPENDIX
A
MORE DATASET DETAILS
Fig. A.6 shows one example room from each of the four types in AI2THOR (Kolve et al., 2017). Each AI2THOR scene is designed manually by 3D artists from reference photos. There is randomizer that can be used to change the location of the objects, so as to enrich the scene. RoboTHOR (Deitke et al., 2020) contains 25 test scenes that have different styles from the AI2THOR scenes (see Fig. A.6). Fig. A.7 shows some Robo' Thor scenes we are using. We can see that the scenes are visually different and thus we consider them be two different distributions of scenes.
Figure A.6: Four room types of A12Thor scenes. From left to right: kitchen, bedroom, bathroom, living room. The yellow numbers object index, the blue lines IFR.
Figure A.7: RoboThor data gallery.
Table A.3 presents the detailed data statistics. We collect all A12Thor models and scenes offline and build our simulation environment with standard OpenAl Gym API (Brockman al., 2016). We construct the edges of a scene by grammar based on object classes. For many-to-many relationships
Figure A.8: Two FloorPlan 11 variances in dataset.
16

---

Published as a conference paper at ICLR 2022
18
23/33
37
35
42/43
36
31/32
Figure B.9: A qualitative result in the bathroom. There many holders used for different purposes. For example, the bath towel holder (ID: 20) is used for bath towel (ID: 19); the hand towel holder is (ID: 29/30/31/32) used for the hand towel (ID: 18); the toilet paper holder is (ID: 41) used for toilet paper (ID: 23/33). Those holders look similar and the agent fails to give accurate predictions based on prior knowledge. However, the agent knows those holders are hard to distinguish and use the must-interactiv--adapati policy to reduce the uncertainty.
(e.g., knives cut potatoes), all triggers connected to responders. For one-to-one relationships (e.g., the buttons open lamps), we follow the rules of A12Thor if possible. For cases that A12Thor suffices (e.g., there is ceil lamp instance in the A12Thor scenes), manually bind the relationships with heuristic rules based on distances and human knowledge. Among all the 48 edge types, 19 are one-to-one and 29 are many-to-many. If the object should have detailed parts (e.g., Microwave), substitute the A12Thor mesh with the PartNet model.
Table A.3: data statistics
#scenes
#shape-cats
#objects
#int-objects
#rolations
Kitchen
300
63
20880
4060
6790
AI2THOR + PartNet
Bathroom
300
40
10850
3540
3620
Bedroom
300
59
12280
2240
2520
Living Room
300
50
12710
1590
1830
RoboThor PartNet
60
49
3104
488
579
B MORE QUALITATIVE RESULTS
We show more qualitative results in this section. In Fig. B.9 we show an example inference in the bathroom. There many holders used for different purposes. For example, the bath towel holder (ID: 20) is used for bath towel (ID: 19); the hand towel holder is (ID: 29/30/31/32 used for hand towel (ID: 18); the toilet paper holder is (ID: 41) used for toilet paper (ID: 23/33). Those holders look similar and the agent fails to give accurate predictions based on prior knowledge. However, the agent knows those holders are hard distinguish and uses the iust-interactive-ane adaattttonoo policy to reduce the uncertainty. In Fig. 1. we show an example inference in the bedroom. We can see that the prior knowledge is already good without any interaction. And with interaction, the performance can be further improved. For example, the agent learns to turn on the ceiling lamp (ID: 3) by toggling a novel button (ID: 9) that doesn't appear the training scenes.
C DIFFERENT RELATIONSHIP ABLATION.
In this section discuss the agent's performance over difference types of relationships. As shown in Table C.4. Overall the agent performs better on the Many-to-Many relationship. which is reasonable because One-to- Pnee-to-One relationships would suffer from ambiguities. Another clear trend is that the agent performs much better in the Self relationship compared to the Binary relationship, which indicates the difficulties to build IFRs across objects.
17

---

Published as a conference paper at ICLR 2022
3
11
12
13
41/42
42/44
Figure B.10: qualitative result in the bedroom. We can see that the prior knowledge is already good without any interaction. And with interaction, the performance can be further improved. For example, the agent learns turn on the ceiling lamp (ID: 3) by toggling novel button (ID: 9) that doesn't I appear in the training scenes. We also show the overall performance for reference.
Table C.4: We present performance over different types relationships. One-to-One and Many-to- Many classified by number of links of objects of GT graph. Self means trigger object leads to state change of itself. Binary means to interact one object leads to state change of another object.
Method
MCC
One uo One
0335
0.983
0.33
0.951
0.95
0.741
0.900
0331
0.841
0.37
0.12
0.819
0.36
Many-to- Many
0.68
0.992
0.977
8.78
0.758
1,000
0.847
0.862
1.000
0.500
0.93
0.941
0.867
0.934
0.895
018%
Self
0.995
0.992
0.992
0.911
0.927
1,000
0.956
0.877
0.832
0.585
0.879
0.899
0.98
0.830
0.895
0.900
Binary
0.98
0.788
0.935
0.999
0.867
0.92
0.915
0.919
0.736
0.366
0.826
0.838
0.23
0.965
0.856
0859
D FAST-INTERACTIVE-AAAAE.E
BEHAVIOR
Here show the behavior of fast-interactive- adapttatooo policy. In Fig. D.1I, visualize the distribution objects that the fastiiteractiiv-aaaaaaii policy interacts with. We can see that the adaptation policy tends to interact with interesting objects with IFR (e.g., switches and knobs). In Fig. D.12, we can also see the performance steadily improve with the number interactions.
E INITIALIZATION OF EDGES IN GCN.
Here discussed the edge initialization of GCN of SR-Prior-Net and Exploration Policy. Table E.5 shows that: 1) the performance drops with only prior knowledge or distance information as edges; 2) scaling factor boost the performance. Our hypothesis is that the scaling of initial edges helps the model to distinguish them from GT edges, thus making it easier to do casual inference.
F FAILURE CASES AND ANALYSIS
There
where the object shapes very similar, but the IFR is different. For example, Fig. F.13 shows one failure case bathroom scene, where the agent confuses the hand towel with the bath towel and predicts extra edges. (the hand towel holder is used to hold the bath towel and the bath towel holder used to hold hand towel.)
G LIMITATIONS AND FUTURE WORKS.
First, work uses 3D point clouds inputs, ignoring the object color, texture, other non- geometric information. Besides, our system assumes all the objects the scene are pre-segmented and many visual quantities direction observable (e.g., the state change of objects). Future
18

---

Published as a conference paper at ICLR 2022
Tower
Object Type
6/C
Ouiect
Octh
648 Percent Interactiona
ala
Percent Interacions
Lvnng room
Kitchen
Catret
ThenSn
Homeis
Object Typ.c
Clqeect
10.00
Truder
Coueerics
Labace
6:38
Ain
Figure D.11: The distribution of objects that the tastinterractive-ddaaioon policy interacts with on the test scenes. We discard objects with less than 10 interactions in the whole test scene.
3.00
0.75
0.7
58
2.5
50
10.0
125
150
375
Figure D.12: The average Fl score in the test scenes with respect to the number of interactions.
Figure F.13: A bathroom scene where hand towel holder holds a hand towel while a bath towel holder holds a bath towel. The agent confuses the two towels and predicts two extra edges.
19

---

Published as a conference paper at ICLR 2022
Table E.S: The edge initialization of GCN of SR-Prior-Net and Exploration Policy, The experiments is conducted on the bathroom with 20% budget.
Precision
Recall
F1
Prior only
0.872
0.971
0.913
Distance only
0,835
0.986
0.902
No Scaling (y = 1.)
0.819
0.991
0.894
Scaling factor y 0.4
1.0
0.836
0.908
Scaling factor = 0.8
0.873
0.986
0.922
Ours-Final
0.873
0.989
0.923
works study how imperfect visual observations may affect network predictions and how address such challenges if any. Lastly, we also abstract away the robotic navigation and detailed manipulation. Future works may investigate building an end-to-end pipeline considering navigation and manipulation.
20


Reviews:
[{"review_id": "hC3ATcW5GdD", "reviewer": "Reviewer_RFiX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-The task is interesting. Although I do not think exploring inter-object functional relationships in 3D indoor scenes is a new task (some previous works about causality modeling proposed similar tasks, such as \"Learning Perceptual Causality from Video\"), I appreciate the collected dataset and the designed pipeline of learning the relationships for inter-object. \n\n-The dataset and the proposed pipeline may inspire some applications such as home-assistant robots, intention prediction, and so on.\n\nWeaknesses:\n-Some details are missed which may affect the reproducibility o the work. For example. the reward design and shaping are not discussed. The FAST-INTERACTIVE-ADAPTATION is a little confusing. I am not sure what \"fast\" refers to or what makes the framework fast. It is nice if the authors could clarify this point.\n\n-The motivation of the design of some components could be discussed more clearly. For example, it is not intuitive for me why the input of binary-relationship prior network is a point cloud. It is not clear for me either why two objects nodes are connected because they are near(<0.5m)\n\n-The experiment does not evaluate the proposed framework over different types of relations. For example, I am wondering what is the performance of the framework on the many-to-many relationship prediction.\n\n\n", "summary_of_the_paper": "This paper proposes to study inter-object functional relationships such as a switch on the wall turning on or off the light. To tackle this task, this work designs a two-stage approach under a self-supervised interactive learning-from-exploration framework, which can predict functional relationships between objects in new 3D indoor environments. Furthermore, this work creates a dataset based on AI2THOR and AI2THOR to demonstrate the task. ", "summary_of_the_review": "In summary, the task of this work is interesting and I think it gives some insight into exploring the general functionality of objects. The paper is well organized. I would be inclined to suggest accepting it if the motivation and some technical details could be clarified in the revision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Op1IqSYmyl", "reviewer": "Reviewer_fQs8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem is interesting and well motivated. However, it is fairly simple. It requires point cloud inputs of the objects and then only needs to predict a binary classification if any two objects interact or not. This seems like a simplified version of \"Learning Affordance Landscapes for\nInteraction Exploration in 3D Environments\", Nagarajan et al. 2020 which predicts actions performable on an object (e.g., open, close, toggle, etc.) from images direction (e.g., without pre-segmented objects/point clouds). Such an approach could be applied to this setting, by additionally predicting the object that is changed, by making the output also capture the relationships/changes when an action is performed. Related, this work does not compare to any previous works. While the ablations are interesting and well done, the lack of other comparisons is a weakness. Especially given that this introduces a new dataset, comparing previous works to establish baselines is helpful.\n\nThere also aren't comparisons in the accuracy of different types of relationships (toggleable, many-to-many, one-to-one, functional, inter-object, etc). It would be interesting to see which of these the model does well on and which it does not do well on.\n\nThe experiments on transferring between environments are interesting and is an important ability for such an approach.\n\nThe paper would benefit from some proofreading and editing, there are some sentences with grammar mistakes, especially in the introduction. Otherwise, the paper is well written and easy to follow and understand.\n\nThe RL and self-supervised learning approach seem effective for this problem.\n", "summary_of_the_paper": "This paper proposes a method to learn relationships between two objects, e.g., flipping a switch turns a light on. This is formulated as a binary classification task taking two objects as input and outputing a single a value $\\in [0,1]$ if the objects are related or not. The paper proposes a method to train this using RL to select the training (object_1, object_2) pairs. The paper also introduces a dataset for evaluation and provides a lot of experiments to study this.", "summary_of_the_review": "Overall the paper is well motivated and the problem is interesting. However, the approach is not compared to any previous works and is fairly simple (e.g., predicting a binary value if two objects are related or not). With some revisions and additional experimental comparisons, I think the paper would be much stronger.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IHO0PRprGzQ", "reviewer": "Reviewer_sxhc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "===Quality of Results===:\n\nS: Extensive experiments on the design choices have been performed, which informs the reader of the need and functioning of incorporating such modules. The proposed method is a modular approach with simple networks, which is quite easy to follow.\n\nW: Well, it is difficult to objectively judge how good the results are. Although P, R and F1 scores have been used to measure the quality of results, they do not necessarily capture the \"goodness\" score, i.e,  P and R scores are biased, or rather, hide important flaws in the data stats that have a significant bearing on the result interpretation (and therefore the Model's learned ability). How about Matthew's Correlation Coefficient (MCC)? This metric correctly captures the quality of results and sidesteps issues prevalent with P and R scores.\n\n\n\n===Comparisons===:\n\nS & W: There is no comparison to prior works since the claim is that this is the first paper proposing this problem.\nHowever, experiments have been performed to validate the system design choices, such as the need for individual prior nets (binary, as well as, scene).\n\n\n===Novelty===:\n\nW: Problem statement -- The claim is that the problem is new. Application-wise, it may be, but it is nothing more than the task of link prediction. As such, I am not sold on the novelty of the problem.\n\nNeutral: Approach -- Good modular approach, but expected.\n\nNeutral: Results -- Interpretation could be better. \n\n\n\n===Limitations===:\n\nProblem novelty and Evaluations, as described above.\n\n\n\n\n\n===Relevant Literature Coverage===:\n\nS: \"Related work\" section is comprehensive and covers a lot of relevant papers in this field.\n\n\n\n===Writing===:\n\nS: The writing is good.\n\n", "summary_of_the_paper": "The paper presents an interesting research problem of \"Learning Inter-Object Functional Relationships\" in virtual 3D indoor environments, where the goal is to make an agent understand the functional effect on an object by interacting with other objects in the scene. Two important (and related) questions arise: (a) How to understand which object(s) affects the functionality of which other object(s) in the room?, and (b) how to learn such inter-object functional priors to build a good knowledge base for reliable inference in unseen environments?\n\nThe paper addresses these two questions by presenting a reinforcement-based learning algorithm. \n\nThe technical summary of the paper is given below.\n\n===Goal===: \n\nLearning inter-object functional relationships, represented by scene graphs. That is, an agent is provided with large-scale scenes to explore for learning in the training stage and is asked to predict a functional scene graph, (S, R_s) for a new scene at test time. Here S is the scene and R_s is the relationship scene graph for the scene, S.\n\n\n\n===Assumptions===:\n\nObject segmentation and state changes (by \"state\", the authors refer to the functional outcome of an object) are provided to the agent apriori. Therefore, the functional effect of a relationship is directly observable.\n\n\n\n===Input===: \n\nA 3D scene as a set of objects {O_1, O_2, ..., O_n}, where each object O_i is explicitly modeled as (O_i^hat, c_i, s_i) where O_i^hat represents the point cloud of the object, c_i represents the 3D centroid, and s_i represents the object's isotropic scale.\n\n\n\n===Output===:\n\nFunctional scene graph, R_s, i.e., whether there exists a link between two nodes (objects) of the scene graph (scene).\n\n\n\n\n===Learning Style===: \n\nSelf-supervised learning framework\n\n\n\n\n===Neural Network===:\n\n1) Pretrained PointNet++ and an MLP (these two make up what is called a \"BR-Prior-Net\", which is short for Binary Relationship Prior Net) -- supervised training using observed GT data\n\n2) GCN, 3-layer (this is a part of what is called an \"SR-Prior-Net\", which is short for Scene Relationship Prior Net) -- Partly supervised using data from the above step and some rule-based heuristics\n\nThe goal of the above two networks is to propose functionally related objects.\n\n3) SR-Posterior-Net, which has the same architecture and weights as SR-Prior-Net. The only difference is that the input to SR-Posterior-Net, in the beginning, is the refined prediction output from SR-Prior-Net.\n\n4) A GCN trained using Reinforcement Learning (RL) algorithm to output action probability scores for each object that indicates the probability that this object triggers a state change on another object. Essentially, the RL-algorithm-based optimization on GCN is used to obtain supervision ground truth.\n\n\n\n===Dataset Used===: \n\nAI2Thor, PartNet-Mobility\n\n1) A total of 1200 scenes covering 23360 objects from 121 categories are used from the above datasets.\n\n2) Among these, 27 are trigger objects, 29 are responders, where 11 are both trigger and responders (e.g., desk lamp), and 79 are non-interactive background objects\n\n\n\n===Relevance of datasets===: \n\nThe above two datasets employed fit well for working on the proposed problem.\n\n\n===Loss===:\n\n1) For training the RL algorithm, a Policy Optimization loss is formulated.\n2) SR-Priori, BR-Priori, and SR-Posteriori are all simple link prediction tasks. So, standard binary cross entropy loss.\n\n\n\n===Objective Evaluation Metrics===:\n\nPrecision, Recall and F1-scores", "summary_of_the_review": "Essentially, the problem of inferring inter-object functionality boils down to link prediction tasks, but the meat lies in the fact that supervisory GT signals are mined from the outputs of a developed system within. \n\nI give credit to the self-supervision part, but then, the problem itself pivots towards the theme of extracting priors from data for learning a different task. Plus, the evaluation is not reliably interpretable with the results shown. As such, I am sitting on a borderline score for this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7Vuqw_R5Cq0", "reviewer": "Reviewer_TQTu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "These are some solid points in the paper:\n1. The paper introduces the extraction of the inter-object functional relationship as a novel problem to explore.\n\n2. There is a new hybrid dataset composed of 1200 scenes with 23,360 objects from 121 categories to study this problem.\n\n3. The paper introduces a strategy based on PointNet++ to infer the binary inter-object functional relationship, a graph convolutional network to model the scene and predict the functional graph for the scene.\n\n4. A significant effort has been spent to showcase different aspects of the solution.\n\n5. The authors promise to make their code and dataset available upon acceptance.\n   \nThese are suggestions I make to improve the paper:\n\n1. The authors should add results to their abstract.\n\n2. The statement  “Performs as few interactions as possible to reduce uncertainties” should be quantified. \n\n3. Please define PPO (proximal policy optimization?) in the text.\n\n4. In the following statement: “In our implementation, we train the exploration policy and the prior/posterior networks for five loops until they jointly converge.” Please clarify: it is five loops or until convergency?\n\n5. I did not understand this sentence: “on average 32% of the total object count 3125 in all 210 training scenes.” Please clarify.\n\n6. I have several comments on the sentence: “We replace 83 AI2THOR models with 888 PartNet-Mobility models for 31 categories (12 exist in AI2THOR while 19 are newly added) to further enrich the diversity of the shape geometry.”\n\na. I suggest replacing “models” with “objects” for clarity.\n\nb. Would you please elaborate on the need to modify the AI2THOR dataset? Would you please elaborate on the diversity of shape geometry you are aiming for?  \nc. I am intrigued by how do you go from 120 interactive scenes to 1200? What are the criteria to create a new scene? How does one scene differ from the rest? Would you please quantify the difference between scenes?\n\n7. I suggest you bold the best results in Table 1. ", "summary_of_the_paper": "The paper proposes a framework to study inter-object functional relationships, i.e., where the state change in one of the objects causes functional effects on other 3D objects.  The authors propose a two-stage strategy guided by a self-supervised strategy. The first stage predicts possible functional relationships between the objects. In the second stage, the system performs interactions between the objects in the scene. The authors present the inter-functional relationship with a directed graph.\n \nThe binary relationships are represented with a prior network. Processing all the objects results in a scene representation of a functional relationship. Their procedure for training an exploratory policy allows them to generate data to construct a posterior network of the functional relationship in the scene. The authors create a dataset using scenes and objects from AI2Thor, enriched with articulated objects from PartNet.\n\nThe authors present an ablation study where baselines are extracted for the binary priors, scene priors, the exploration policy, the selection of objects, and their level of interaction. ", "summary_of_the_review": "The paper introduces a novel and relevant problem. In the process, the authors construct a dataset for experimentation based largely on previous ones. The paper presents a strategy to solve the problem while the experimentation showcases different aspects of the solution. Finally, the authors promise to make their code and dataset available upon acceptance. Still, some issues need to be addressed to clarify the message in the paper further.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an approach for learning inter-object relations. The relationships are represented in terms of scene graphs, and are processed with graph convolutional networks. All of the reviewers find the problem interesting and meaningful, which is the main strength of the paper. The approach assumes good object segmentations and state changes are provided to the agent, which the AC believes is a very dangerous assumption. Object segmentation or the change detection from raw data is still an active research area and lack of end-to-end training capability of the proposed approach is a limiting aspect. Still, the authors were able to convince all the reviewers that the problem formulation and the proposed approach is valid. Experimental results were provided to support the argument.

Respecting the opinions from the reviewers, the AC recommends accepting the paper, although the AC himself/herself is very reluctant to give an accept rating.
Decision: Accept (Poster)