Year: 2021

Abstract:
Building embodied intelligent agents that can interact with 3D indoor environments has received increasing research attention in recent years. While most works focus on single-object or agent-object visual functionality and affordances, our work proposes to study a new kind of visual relationship that is also important to perceive and model – inter-object functional relationships (e.g., a switch on the wall turns on or off the light, a remote control operates the TV). Humans often spend little or no effort to infer these relationships, even when entering a new room, by using our strong prior knowledge (e.g., we know that buttons control electrical devices) or using only a few exploratory interactions in cases of uncertainty (e.g., multiple switches and lights in the same room). In this paper, we take the first step in building AI system learning inter-object functional relationships in 3D indoor environments with key technical contributions of modeling prior knowledge by training over large-scale scenes and designing interactive policies for effectively exploring the training scenes and quickly adapting to novel test scenes. We create a new benchmark based on the AI2Thor and PartNet datasets and perform extensive experiments that prove the effectiveness of our proposed method. Results show that our model successfully learns priors and fast-interactive-adaptation strategies for exploring inter-object functional relationships in complex 3D scenes. Several ablation studies further validate the usefulness of each proposed module.

heading: 1 INTRODUCTION
text: When first-time entering a hotel room in a foreign country, we may need to search among buttons on the wall to find the one that turns on a specific light, figure out how to turn on the faucets in the bathroom and adjust water temperature, or perform several operations on the TV and the remotes to switch to our favorite channel. Many of these may require no effort if we have used the same models of lamp or faucet before, while the others that we are less familiar with will take more time for us to explore and figure out how things work. We humans can quickly adapt to a new indoor environment thanks to our strong prior knowledge about how objects could be functionally related in general human environments (e.g., the room light can be turned on by pressing a button or toggling a switch) and our amazing capability of using only a few interactions while exploring the environment to learn how things work in cases of uncertainty (e.g., multiple lights and buttons in the room). In this work, we investigate how to equip AI systems with the same capabilities of learning inter-object functional relationships in 3D indoor environments.
With the recent research popularity of embodied AI in perception and interaction, many previous works have studied how to perceive, model, and understand object functionality (Kim et al., 2014; Hu et al., 2016; 2018; 2020) and agent-object interaction (Montesano & Lopes, 2009; Do et al., 2018; Nagarajan & Grauman, 2020; Mo et al., 2021a), just to name a few. However, these works mostly investigate single-object/scene or agent-object scenarios. A few past works (Jiang et al., 2012; Sun et al., 2014; Zhu et al., 2015; Mo et al., 2021b; Cheng et al., 2021) have studied inter-object relationships given a pair of objects or for specific tasks (e.g., placement, stacking). We make a first step bringing up research attentions to a new type of visual quantity – inter-object functional relationship, and an interesting yet underexplored challenging problem of how to learn such generic functional relationships in novel 3D indoor environments.
Given a novel indoor scene with multiple objects as the input (Fig. 1, left), we predict a complete directional graph of inter-object functional relationships (Fig. 1, middle) on how the state change
Under review as a conference paper at ICLR 2022 4
Teaser
of one object causes functional effects on other objects. While the semantics of individual objects already suggest some functional priors (e.g., buttons can be pressed, electrical devices can be turned on), our system learns to build up the correspondence among them (e.g., which button controls which electrical device). Also, since one may encounter unknown object categories exploring novel scenes, we do not assume semantic labels of objects and only use shape geometry as input. Our system, therefore, learns inter-object functional geometry-mappings to what to trigger (e.g., one button among others on the wall) and what other shapes the interaction effects (e.g., one light in the room).
Though seemingly simple as one may interact with all objects in the scene to query the functional scene graph, it is actually very challenging how to efficiently and accurately perceive, reason, and explore the scene with as few interactions as possible. First of all, we do not want to waste time interacting with books and bags as they will not functionally trigger other objects. Secondly, the semantics of two objects may directly indicate functional relationships. For example, we know that buttons control electrical devices in general. Besides, in cases of having two buttons and two lights in the scene, one may guess the pairings according to spatial proximity. All of these examples indicate that learning some geometric and semantic priors among the objects may help predict many inter-object functional relationships even without any interaction. However, there are always scenarios where uncertainties exist. For example, for the stove highlighted in Fig. 1 (right), interactions have to be taken place to figure out which knob controls which burner, though only two interactions are needed for the four pairs if our agent can smartly infer the other two automatically.
We design a two-staged approach, under a self-supervised interactive learning-from-exploration framework, to tackle the problem. The first stage perceives and reasons over the objects in a novel test scene to predict possible inter-object functional relationships using learned functional priors about the object geometry and scene layout. Then, in the second stage, our system performs a few interactions in the scene to reduce uncertainties and fastly adapts to a posterior functional scene graph prediction. We jointly train a reinforcement learning policy to explore large-scale training scenes and use the collected interaction observations to supervise the prior/posterior networks in the system.
Since we are the first to explore this task to our best knowledge, we create a new hybrid dataset based on AI2THOR (Kolve et al., 2017) and PartNet (Mo et al., 2019b) as the benchmark, on which we conduct extensive experiments comparing to various baselines and illustrating our interesting findings. Experiments validate that both stages contribute to the final predictions and prove the training synergy of the exploration policy and the prior/posterior networks. We also find reasonable generalization capabilities when testing over novel scene types with unknown object categories.
In summary, this work makes the following main contributions: • we first formulate an important, interesting, yet underexplored challenging task – learning
inter-object functional relationships in novel 3D indoor environments; • we propose a novel self-supervised learning-from-exploration framework combining a
prior-inference stage and a fast-interactive-adaptation stage to tackle the problem; • we create a new dataset for benchmarking the proposed task and demonstrating the effec-
tiveness of the proposed approach.
Under review as a conference paper at ICLR 2022

heading: 2 RELATED WORKS
text: Visual Functionality and Affordance. We humans perceive and interact with the surrounding 3D world to accomplish everyday tasks. To equip AI systems with the same capabilities, researchers have done many works investigating shape functionality (Kim et al., 2014; Hu et al., 2016; 2018; 2020; Lai et al., 2021; Guan et al., 2020), grasp affordance (Lenz et al., 2015; Redmon & Angelova, 2015; Montesano & Lopes, 2009; Qin et al., 2020; Kokic et al., 2020; Mandikal & Grauman, 2021; Yang et al., 2020; Corona et al., 2020; Kjellström et al., 2011; Fang et al., 2018; Nagarajan et al., 2019; Brahmbhatt et al., 2019; Jiang et al., 2021), manipulation affordance (Do et al., 2018; Nagarajan et al., 2019; 2020; Nagarajan & Grauman, 2020; Mo et al., 2021a), scene affordance (Piyathilaka & Kodagoda, 2015; Rhinehart & Kitani, 2016; Li et al., 2019b), etc. While these works mostly study single-object/scene or agent-object interaction scenarios, our work explores inter-object relationships. Besides, we investigate how to explore functional relationships by interacting in 3D scenes, different from previous works that study interactions given specific object pairs or tasks (Jiang et al., 2012; Sun et al., 2014; Zhu et al., 2015; Mo et al., 2021b; Cheng et al., 2021).
Inter-object Relationships. Besides perceiving individual objects in the environment, understanding the rich relationships among them is also crucial for many downstream applications. Previous works have attempted to model object supporting and contact relationships (Fisher et al., 2011; Silberman et al., 2012), spatial relationships (Galleguillos et al., 2008; Gould et al., 2008; Kulkarni et al., 2019), semantic relationships (Sadeghi & Farhadi, 2011; Divvala et al., 2014), physical relationships (Mitash et al., 2019), etc. More recently, researchers have introduced scene graphs (Liu et al., 2014; Johnson et al., 2015; Krishna et al., 2017; Xu et al., 2017; Li et al., 2017b; Zhu et al., 2020; Wald et al., 2020) and hierarchies (Armeni et al., 2019; Li et al., 2019a; 2017a; Mo et al., 2019b;a) to parse the objects, object parts, and their relationships. Unlike the spatial, geometric, semantic, or physical relationships investigated in these works, we pay more attention to inter-object functional relationships in which the state change of one object causes functional effects on other objects.
Learning from Exploration and Interaction. While deep learning has demonstrated great success in various application domains (Russakovsky et al., 2015; Silver et al., 2016), large-scale annotated data for supervision inevitably becomes the bottleneck. Many works thus explore self-supervised learning via active perception (Wilkes & Tsotsos, 1992), interactive perception (Bohg et al., 2017), or interactive exploration (Wyatt et al., 2011) to learn visual representations (Jayaraman & Grauman, 2018; Yang et al., 2019; Weihs et al., 2019; Fang et al., 2020; Zakka et al., 2020), objects and poses (Caicedo & Lazebnik, 2015; Han et al., 2019; Deng et al., 2020; Chaplot et al., 2020b; Choi et al., 2021), segmentation and parts (Katz & Brock, 2008; Kenney et al., 2009; Van Hoof et al., 2014; Pathak et al., 2018; Eitel et al., 2019; Lohmann et al., 2020; Gadre et al., 2021), physics and dynamics (Wu et al., 2015; Mottaghi et al., 2016; Agrawal et al., 2016; Li et al., 2016; Janner et al., 2018; Xu et al., 2019; Lohmann et al., 2020; Ehsani et al., 2020), manipulation skills (Pinto & Gupta, 2016; Levine et al., 2018; Agrawal et al., 2016; Pinto et al., 2016; Pinto & Gupta, 2017; Zeng et al., 2018; Batra et al., 2020), navigation policies (Anderson et al., 2018; Chen et al., 2019; Chaplot et al., 2020a; Ramakrishnan et al., 2021), etc. In this work, we design interactive policies to explore novel 3D indoor rooms and learn our newly proposed inter-object functional relationships.

heading: 3 PROBLEM FORMULATION
text: We formulate a novel problem of inferring inter-object functional relationships (IFRs) among objects in novel 3D indoor environments (Fig. 1). For the task input, each scene S comprises multiple 3D objects {O1,O2, · · · ,On}, each of which is presented as a 3D point cloud with 2,048 points depicting the complete shape geometry. Every object Oi may be subject to state changes (e.g., water is running or not from a faucet, a button is pressed up/down). The set of 3D objects describe the complete 3D scene S = {O1,O2, · · · ,On} including both the objects with and without possible state changes. The spatial layout of the objects in the scene is explicitly modeled that each object Oi = (Ôi,ci,si) can be decomposed into a zero-centered unit-sized shape point cloud Ôi ∈ R2,048×3, the object center position in the scene ci ∈ R3, and its isotropic scale si ∈ R. For the task output, we are interested to figure out the (IFRs) RS. In a case that interacting with an object Oi (a trigger) will cause a functional state change of another object O j (a responder), we formally define an observed IFR (Oi,O j) ∈RS. In the task output RS, there could be one-to-one or many-to-many relationships between two objects, as a trigger object may cause functional state changes of many responder objects and vice versa.
We define a task scenario that an agent actively explores the environment and interacts with the objects to output the inter-object functional relationships following the popular embodied and real-world
Under review as a conference paper at ICLR 2022
5
settings (Nagarajan & Grauman, 2020; Ramakrishnan et al., 2021). For each interaction, an agent picks an object, triggers its state change, and directly receives the state changes of the other objects that have been affected in the scene. We assume that standardized visual quantities (e.g., object segmentation, state changes) are provided to the agent as they are well-established research topics by themselves (Geiger et al., 2012; Li et al., 2020), and thus that the functional effect of a relationship is directly observable. An agent is provided with large-scale scenes to explore for learning in the training stage and is asked to predict the functional scene graph (S,RS) for a novel scene at the test time. We also abstract away complexities on robotic navigation (Anderson et al., 2018; Ramakrishnan et al., 2021) and manipulation (Nagarajan & Grauman, 2020; Mo et al., 2021a), which are orthogonal to our contribution to estimating inter-object functional relationships.

heading: 4 TECHNICAL APPROACH
text: Fig. 2 presents the system overview. Given a novel indoor scene, the prior-inference stage first produces a functional scene graph prior (S,RsS) by perceiving and reasoning over the objects in the scene S = {O1,O2, · · · ,On}. Then, in the fast-adaptation stage, the agent performs as few interactions as possible to reduce uncertainties and updates prior into posterior for the final functional scene graph prediction (S,RS). We train an exploration policy that learns to effectively and efficiently explore large-scale training scenes and collects interaction data to supervise the prior/posterior networks.

heading: 4.1 THE PRIOR-INFERENCE STAGE
text: We design two prior networks – BR-Prior-Net and SR-Prior-Net, for the functional scene graph prior learning, where BR-Prior-Net focus on modeling binary relationship priors rbi, j ∈ [0,1] given only two input shape geometry (Oi,O j) (∀i, j) while SR-Prior-Net reasons scene-level functional relationship priors RsS = {rsi, j ∈ [0,1]|Oi,O j ∈ S} considering all the objects in the scene S = {O1,O2, · · · ,On}.

heading: 4.1.1 BINARY-RELATIONSHIP PRIOR NETWORK (BR-PRIOR-NET)
text: Given two objects in the scene (e.g., a button and a light), humans often have binary priors if the two objects are functionally correlated or not (e.g., the button may control the light) according to geometry and semantics of the two shapes, regardless of their locations in the scene. Having this observation, we thus design a binary-relationship prior network (BR-Prior-Net) to model the possibility of functional relationship for every pair of trigger object Oi and responder object O j.
As shown in Fig. 3 (left), our BR-Prior-Net takes two object point clouds Oi,O j ∈ R2048 as input and outputs the belief of IFR rbi, j ∈ [0,1]. By allowing Oi = O j, we also consider self-relationship (e.g., triggering the button on the microwave opens its door). We train a shared PointNet++ network Qi et al. (2017), with four set abstraction layers with N = [512,128,32,1] point resolutions, to process the two point clouds and output global features f bi , f b j ∈ R64 that summarize the two input shapes. We then concatenate the two obtained features and employ a Multilayer Perceptron (MLP) network to estimate rbi, j ∈ [0,1] for the final output. We obtain the binary functional scene relationship prior RbS = {rbi, j ∈ [0,1]|Oi,O j ∈ S} by enumerating all object pairs in the scene.

heading: 4.1.2 SCENE-RELATIONSHIP PRIOR NETWORK (SR-PRIOR-NET)
text: While the binary-relationship prior network empirically performs well in predicting pairwise functional relationships as it effectively scopes down the problem scale to two input objects only, much important scene-level information is ignored, such as relative object poses and scene object layout.
Under review as a conference paper at ICLR 2022 6
Such scene context may provide clues on relationship pairings in cases of uncertainties. For example, the BR-Prior-Net may connect all buttons to all lights in the room, but given the object layout, one may assign each button to its closest light with higher probabilities. Therefore, we further introduce scene-relationship prior network (SR-Prior-Net) to take the scene context into consideration.
The SR-Prior-Net is implemented as a three-layer Graph Convolutional Network (GCN) (Kipf & Welling, 2016) that connects and passes messages among all objects in the scene. Fig. 3 (right) shows an overview of the design. Each object Oi is represented as a node i in the graph, with a node feature n(0)i that combines the PointNet++ geometry feature and the scene-contextual information (e.g., object position and size). The edge between two nodes is initialized with w(0)i, j = r b i, j predicted by the BR-Prior-Net. To allow better graph message passing throughout the graph, we additionally connect two nodes if the two objects are close enough (< 0.5m, which covers about 53.1% IFRs in our dataset) because closer objects are more likely to have functional relationships, e.g., a stove knob is close to the corresponding stove burner. We use an edge weight of w(0)i, j = γ = 0.6, which is experimentally tuned out (see Table E.5). After three iterations of graph convolutions
n(k)i = Θ∑ j w(k−1)j,i√ d(k−1)j d (k−1) i n(k−1)j ,∀k = 1,2,3 (1)
where d(k−1)i = 1+∑ j w (k−1) j,i and Θ denotes the learnable parameters, the network outputs a scenecontextual embedding f Si = n (3) i ∈ R32 for each object Oi. Next, for each object pair, we concatenate their feature embeddings, feed it through an MLP, and predict a likelihood score for their functional relationship rsi, j ∈ [0,1]. The final scene-level functional scene relationship prior RsS = {rsi, j ∈ [0,1]|Oi,O j ∈ S} is obtained by enumerating all object pairs in the scene.

heading: 4.2 THE FAST-INTERACTIVE-ADAPTATION STAGE
text: While the prior networks do their best to propose functionally related objects by perceiving and reasoning over objects in the scene, there are always uncertain cases that priors do not suffice (e.g., the stove example in Fig. 1 (right)), where interactions are needed to reduce such uncertainties. A smart agent only needs to perform very few interactions as needed and is capable of inferring other pairings. We therefore design SR-Posterior-Net to learn such posterior reasoning after interactions and propose a test-time fast-adaptation strategy that sequentially picks a few objects to interact.

heading: 4.2.1 SCENE-RELATIONSHIP POSTERIOR NETWORK (SR-POSTERIOR-NET)
text: The scene-relationship posterior network (SR-Posterior-Net) shares the same network architecture and weights to SR-Prior-Net introduced in Sec. 4.1.2. We only repurpose the network to model functional scene graph posteriors by simply changing the network input to the current functional scene relationship belief modulated by the observed interaction outcomes. In our experiments, we perform interactions over objects one-by-one sequentially and thus feed through SR-Posterior-Net multiple times to evolve the functional scene relationship posteriors R(t)S (t = 1,2, · · · ,TS) after each interaction, where TS denotes the total number of interactions performed in this scene.
Under review as a conference paper at ICLR 2022
Concretely, when t = 0, we use R(0)S = R s S, which is the functional scene relationship prior estimated by SR-Prior-Net without performing any interaction. For each timestep t, given the interaction observations {eit , j ∈ {0,1}|O j ∈ S} after interacting with object Oit , where eit , j = 1 if and only if Oit triggers O j, we update the posterior R (t) S with all past observations r (t) i, j = ei, j (i = i1, i2, · · · , it , j = 1,2, · · · ,n), feed the modified posterior through SR-Posterior-Net, and obtain a functional scene relationship posterior R(t+1)S for the next timestep t + 1. We compute the final functional scene relationship prediction RS = {(Oi,O j)|r(TS)i, j ∈R (TS) S > τ,∀i, j} with a threshold τ = 0.9.

heading: 4.2.2 TEST-TIME FAST-INTERACTIVE-ADAPTATION STRATEGY
text: Now, the only questions left are how to pick the object to interact with at each timestep and when to stop. We find that a simple heuristic to explore the object that is predicted most uncertainly in our posterior works very well in our experiments. Specifically, at each step t, the agent picks Oi that
i = argmax i
( max
j
( min ( r(t)i, j ,1− r (t) i, j ))) ,∀t = 0,1, · · · ,TS−1 (2)
We stop the test-time fast-interactive-adaptation procedure when all predictions are certain enough, min (
r(t)i, j ,1− r (t) i, j
) < γ,∀i, j (3)
with a threshold γ = 0.05, the agent explores about 19% of objects in the room on average. γ can also be enlarged to reduce number of interactions. Another strategy is to fix the test-time interaction budget, e.g., 10% or 20% of the total number of objects in each room.

heading: 4.3 SELF-SUPERVISED LEARNING FROM EXPLORATION
text: We adopt the embodied paradigm of learning from exploration and interaction (Nagarajan & Grauman, 2020; Ramakrishnan et al., 2021) that an agent actively explores the large-scale training scenes, interacts with the environments, and collects data for supervision. It is a challenging task how to effectively probe the environments for self-supervised learning and perform as few interactions as possible for efficiency. We formulate the task as a reinforcement learning (RL) problem and learn an exploration policy that collects data for supervising the prior and posterior networks.

heading: 4.3.1 EXPLORATION POLICY
text: The RL policy, implemented under the Proximal Policy Optimization (PPO) (Schulman et al., 2017) framework, takes the current functional scene graph representation as the state input state(t) = (S,R(t)S ) at each timestep t and picks one object as the action output to interact with action
(t) =Oi ∈ S. When t = 0, we use R(0)S = R s S, which is predicted by the SR-Prior-Net (Sec. 4.1.2). After each interaction, the agent observes the interaction outcome that some or no objects have functional state changes triggered by Oi, namely observation(t) = {ei, j ∈ {0,1}|O j ∈ S} where ei, j = 1 if and only if Oi triggers O j, which will then be passed to the SR-Posterior-Net (Sec. 4.2.1) to update the functional relationship belief R(t+1)S for the next timestep t +1. The agent may decide to stop exploring the current scene and move on to the next one under a limited interaction budget.
We design the RL reward function as follows. reward(t) = α max
j |r(t)i, j − ei, j|+β1[ei, j]− γ (4)
The first term encourages interacting with Oi to correct the wrong or uncertain functional relationship beliefs r(t)i, j ,∀ j. There are three cases for this term: 1) the agent’s belief is already correct (r (t) i, j = ei, j, thus |r(t)i, j − ei, j| = 0); 2) the belief is wrong (r (t) i, j = 1− ei, j, thus |r (t) i, j − ei, j| = 1); 3) the agent is uncertain (e.g., r(t)i, j = 0.5, thus |r (t) i, j − ei, j| = 0.5). We use max instead of avg as the aggregation function because the IFRs are sparse and an average value may conceal the mistake. The second term rewards exploring trigger objects of interest, e.g., it encourages the agent to interact with buttons or switches, instead of boxes or cabinets. The third term simulates the interaction cost to avoid redundant interactions for efficient exploration. We use α = 2, β = 1 and γ = 1 in our experiments.
The RL network shares the same three-layer GCN backbone as the scene-relationship networks (Sec. 4.1.2 and 4.2.1) for reasoning over the functional scene graph (S,R(t)S ). However, instead of predicting IFRs in SR-Prior-Net (Sec. 4.1.2) and SR-Posterior-Net (Sec. 4.2.1), the GCN RL policy
Under review as a conference paper at ICLR 2022
outputs action probabilities a(t)i ’s over object nodes Oi’s, by firstly obtaining an embedding n (t) i ∈R32 for each object Oi, then concatenating with a global scene feature n (t) S computed by averaging across all object embeddings, and finally feeding through an MLP to get an action score a(t)i . In addition, we duplicate the scene feature n(t)S and input it to the MLP to query the termination score astop. All the action scores are finally passed through a SoftMax layer for the action probability scores.

heading: 4.3.2 TRAINING STRATEGY AND LOSSES
text: To supervise the proposed interactive learning-from-exploration framework, we alternate the training of the exploration RL policy and the prior/posterior networks. We leverage the training synergy that the prior/posterior networks predict helpful information for better exploration of the scenes, while the exploration RL policy collects useful data to better train the two networks. In our implementation, we train the exploration policy and the prior/posterior networks to make they jointly converge.
In each loop, conditioned on the current prior/posterior predictions, the exploration policy strategically explores the large-scale training scenes with a total interaction budget of 1000 (e.g., roughly 32% of the total 3125 objects from 210 scenes). The exploration policy learns to wisely allocate budgets across different scenes and pick the objects worth interacting with. Specifically, the agent can interact with m objects in one scene and explore the rest scenes with the 1000−m budget left. For a scene S = {O1,O2, · · · ,On} where the agent sequentially performs m interactions over objects (Oi1 ,Oi2 , · · · ,Oim), we obtain m sets of observations ({ei1, j ∈ {0,1}|O j ∈ S},{ei2, j ∈ {0,1}|O j ∈ S}, · · · ,{eim, j ∈ {0,1}|O j ∈ S}). We then use the collected interaction observations to train the prior/posterior networks. For BR-PriorNet, we simply train for the observed link predictions rbi, j → ei, j (i = i1, i2, · · · , im, j = 1,2, · · · ,n). We learn one shared network for SR-Prior-Net and SR-Posterior-Net but train it with two supervision sources. For learning the SR-Priors, we input the BR-Prior-Net prediction RbS , output the scene-level prior RsS and train for the observed link predictions as well r s i, j→ ei, j (i= i1, i2, · · · , im, j = 1,2, · · · ,n). For training the SR-Posteriors, we use the posterior estimate R(t)S at each timestep t = 1,2, · · · ,m−1, update it with the past ground-truth observations r(t)i, j = ei, j (i = i1, i2, · · · , it , j = 1,2, · · · ,n), input to the network to obtain the posterior estimate R(t+1)S , and supervise the link predictions r (t+1) i, j → ei, j (i = it+1, it+2, · · · , im, j = 1,2, · · · ,n). We use the standard binary cross entropy for all loss terms.

heading: 5 EXPERIMENTS
text: We evaluate our method on a new hybrid dataset and compare it to several baselines that prove the effectiveness of the proposed method and its components. We also experiment with transferring to novel scene types that may contain novel shape categories. See appendix for more results.

heading: 5.1 DATASET
text: We create a new hybrid dataset based on AI2THOR (Kolve et al., 2017) and PartNet-Mobility (Mo et al., 2019b; Xiang et al., 2020) to support our study. AI2THOR provides 120 interactive scenes of 4 types (i.e., bathroom, bedroom, living room, and kitchen) containing more than 1500 interactive objects covering 102 categories. The PartNet-Mobility dataset contains 2,346 3D articulated object CAD models from 46 categories. We replace 83 AI2THOR objects with 888 PartNet-Mobility objects for 31 categories (12 exist in AI2THOR while 19 are newly added) to further enrich the diversity of the shape geometry. We use 27 kinds of functional relationships originally built in AI2THOR (e.g., the microwave can be toggled on) and also enrich 21 more types of inter-object functional relationships (e.g., the pen can be used to write in the book).
In total, our dataset contains 1200 scenes covering 23360 objects from 121 categories. We enrich the origin 120 AI2THOR scenes by 1) randomly spawn objects into different locations and 2) randomly replace the object geometries of the same type. Fig. A.6 shows one example room from each of the four types. Among these, 27 are trigger objects, 29 are responders, where 11 are both trigger and responders (e.g., desk lamp), and 79 are non-interactive background objects. We split the dataset into non-overlapping 800 training scenes and 400 test scenes. Table A.3 presents more detailed statistics.

heading: 5.2 BASELINES AND METRICS
text: Since we are the first to study the problem to our best knowledge, there is no external baseline for directly fair comparisons. However, to validate the effectiveness for each of our proposed
Under review as a conference paper at ICLR 2022
modules (e.g., BR-Prior-Net, SR-Prior-Net, the fast-interactive-adaptation strategy, the self-supervised exploration policy), we compare Ours-Final to several baselines and ablated versions:
• Random: the agent performs random actions in the scene at the test time; • Abla-NoBinaryPrior: the agent does not use the binary priors and instead initializes the
input edges to SR-Prior-Net with w(0)i, j = 1;
• Abla-NoScenePrior: the agent uses the predicted binary-relationship prior RbS instead of the estimated scene-relationship prior RsS as the prior knowledge; • Abla-RandomExplore: the agent does not use the proposed exploration policy and instead randomly explores training scenes during training;
• Abla-RandomAdapt: the agent randomly picks objects to interact with during adaptation; • Ours-PriorOnly: the agent directly uses the scene-relationship prior predictions RsS and
cuts a threshold 0.5 to predict the final outputs without any interaction; • Interaction-Exploration: we try our best to adapt Nagarajan & Grauman (2020) to our set-
ting, though it originally tackles a very different task on learning affordance over individual shapes in a scene and does not propose IFRs. We use a PointNet++ to encode object features and an MLP to additionally predict a set of objects that each interaction at a object causes.
We use the standard precision, recall, and F1-score for quantitative evaluations and comparisons. Additionally, we set a per-room interaction budget for fairly comparing all the methods and report their performance when the budgets are 10% and 20% of the object count in each room.

heading: 5.3 RESULTS AND ANALYSIS
text: Table 1 presents the quantitative evaluations and comparisons to baselines, where we see that OursFinal achieves the best scores in F1-score and MCC in all comparisons under different interaction budgets. The Random baseline observes the direct interaction outcomes and thus by design gives a perfect precision but random interaction renders very low recalls and F1 scores. Compared to the
Under review as a conference paper at ICLR 2022
ablated versions, we validate the effectiveness of our learned binary-prior, scene-prior, the proposed fast-interactive-adaptation strategy, and the self-supervised learning-from-exploration policy. It also meets our expectation that the performance improves with more interactions (i.e., from 10% to 20%), which is further validated by Fig. D.12. We show qualitative results in Fig. 4 and analyze them below.
Is the Learned Prior Knowledge Useful? In Table 1, we see that Ours-PriorOnly already gives decent performance even without performing any interaction, outperforming Random by a large margin, which proves the usefulness of the learned prior knowledge. As shown in Fig. 4 (the left-most figure in the bottom row), the prior knowledge correctly predicts many relationships (red lines) to the ground-truth ones (blue lines). In addition, both BR-Prior-Net and SR-Prior-Net are helpful, as observed in the ablated versions Abla-NoBinaryPrior and Abla-NoScenePrior. We find that the BR-Prior-Net makes a lot of false-positive predictions and the SR-Prior-Net successfully removes them given the scene context. The performance drop of Abla-NoBinaryPrior further illustrates the need for binary priors, which initialize SR-Prior-Net with a sparse graph, better than a dense one.
Does Fast-Interactive-Adaptation Help? The performance of Ours-PriorOnly is further improved given test-time interactions and posterior adaptions as Ours-Final generates better numbers. In Fig. 4, we see that some redundant red lines are removed observing the interaction outcomes and applying the SR-Posterior-Net predictions. Compared to Abla-RandomAdapt that performs random interactions, we find that the fast-interactive-adaptation strategy performs better. Qualitatively, besides the stove case mentioned in Fig. 1 (right), another interesting case in Fig. 4 is that from prior knowledge the agent thinks the two knives (ID: 57/67) may cut egg (ID: 50), maybe because it looks like a tomato (ID: 62) in geometry. The agent only interacts one knife (ID: 57) to correct this wrong belief and then the false-positive edge from the other knife (ID: 67) is automatically removed.
Is Exploration Policy Effective? We learn an exploration policy for efficient exploration of the training scenes. Abla-RandomExplore gives a clear performance drop when using randomly collected data for training, proving that the exploration policy can wisely pick objects to explore.

heading: 5.4 TRANSFER TO NOVEL ROOM AND OBJECT TYPES
text: We further test our model’s generalization capability to objects and rooms of novel types. In Table 2, we report quantitative evaluations where we train our systems on one type of AI2THOR rooms and test over the other types, as well as additional results of testing over a totally out-ofdistribution RoboTHOR (Deitke et al., 2020) test dataset (see Fig. A.7). We find that 1) it works in most cases showing that our learned skills can transfer to novel room types; 2) our system can also generalize to novel object categories. For example, as shown in Fig. 5, although the agent never sees the television nor remote in the training
scenes of bedrooms, it thinks the TV might be controlled by the remote. 3) the prior knowledge is not working well because of the data distribution shift but the posterior interaction helps. For example, in Fig. 5, the interaction helps to make accurate predictions despite the prior is not very accurate.

heading: 6 CONCLUSION
text: We formulate a novel task of learning inter-object functional relationships in novel 3D indoor environments and propose a two-staged interactive learning-from-exploration approach to tackle the problem. Using a newly created hybrid dataset, we provide extensive analysis of our method and report quantitative comparisons to baselines. Experiments comparing to several ablated versions validate the usefulness of each proposed module. See appendix for limitations and future works.
Under review as a conference paper at ICLR 2022
Ethics Statement. We explore AI algorithms towards building future home-assistant robots which can aid people with disabilities, senior people, etc. While we envision that our proposed approach may suffer from the bias in training data, the privacy of using human-designed data, and domain gaps between training on synthetic data and testing in real-world environments, these are the common problems that most if not all learning-based systems have. We do not see other particular harm or issue of major concerns that our work may raise.
Reproducibility Statement. We support open-sourced research and will make sure that our results are reproducible. Thus, we promise to release the code, data, and pre-trained models publicly to the community upon paper acceptance.

heading: A MORE DATASET DETAILS
text: Fig. A.6 shows one example room from each of the four types in AI2THOR (Kolve et al., 2017). Each AI2THOR scene is designed manually by 3D artists from reference photos. There is a randomizer that can be used to change the location of the objects, so as to enrich the scene. RoboTHOR (Deitke et al., 2020) contains 25 test scenes that have different styles from the AI2THOR scenes (see Fig. A.6). Fig. A.7 shows some RoboThor scenes we are using. We can see that the scenes are visually different and thus we consider them to be two different distributions of scenes.
Table A.3 presents the detailed data statistics. We collect all AI2Thor models and scenes offline and build our own simulation environment with standard OpenAI Gym API (Brockman et al., 2016). We construct the edges of a scene by grammar based on object classes. For many-to-many relationships (e.g., knives can cut potatoes), all triggers are connected to responders. For one-to-one relationships (e.g., the buttons open lamps), we follow the rules of AI2Thor if possible. For cases that AI2Thor suffices (e.g., there is no ceil lamp instance in the AI2Thor scenes), we manually bind the relationships with heuristic rules based on distances and human knowledge. Among all the 48 edge types, 19 are one-to-one and 29 are many-to-many. If the object should have detailed parts (e.g., Microwave), we substitute the AI2Thor mesh with the PartNet model.
Under review as a conference paper at ICLR 2022

heading: B MORE QUALITATIVE RESULTS
text: We show more qualitative results in this section. In Fig. B.9 we show an example inference in the bathroom. There are many holders used for different purposes. For example, the bath towel holder (ID: 20) is used for bath towel (ID: 19); the hand towel holder is (ID: 29/30/31/32) used for a hand towel (ID: 18); the toilet paper holder is (ID: 41) used for toilet paper (ID: 23/33). Those holders look similar and the agent fails to give accurate predictions based on prior knowledge. However, the agent knows those holders are hard to distinguish and uses the fast-interactive-adaptation policy to reduce the uncertainty. In Fig. B.10, we show an example inference in the bedroom. We can see that the prior knowledge is already good without any interaction. And with interaction, the performance can be further improved. For example, the agent learns to turn on the ceiling lamp (ID: 3) by toggling a novel button (ID: 9) that doesn’t appear in the training scenes.

heading: C DIFFERENT RELATIONSHIP ABLATION.
text: In this section we discuss the agent’s performance over difference types of relationships. As shown in Table C.4. Overall the agent performs better on the Many-to-Many relationship, which is reasonable because One-to-One relationships would suffer from ambiguities. Another clear trend is that the agent performs much better in the Self relationship compared to the Binary relationship, which indicates the difficulties to build IFRs across objects.

heading: D FAST-INTERACTIVE-ADAPTATION BEHAVIOR
text: Here we show the behavior of fast-interactive-adaptation policy. In Fig. D.11, we visualize the distribution of objects that the fast-interactive-adaptation policy interacts with. We can see that the adaptation policy tends to interact with interesting objects with IFR (e.g., switches and knobs). In Fig. D.12, we can also see the performance steadily improve with the number of interactions.
Under review as a conference paper at ICLR 2022
21
E INITIALIZATION OF EDGES IN GCN.
Here we discussed the edge initialization of GCN of SR-Prior-Net and Exploration Policy. Table E.5 shows that: 1) the performance drops with only prior knowledge or distance information as edges; 2) a scaling factor can boost the performance. Our hypothesis is that the scaling of initial edges helps the model to distinguish them from GT edges, thus making it easier to do casual inference.

heading: F FAILURE CASES AND ANALYSIS
text: There are cases where the object shapes are very similar, but the IFR is different. For example, Fig. F.13 shows one failure case in a bathroom scene, where the agent confuses the hand towel with the bath towel and predicts two extra edges. (the hand towel holder is used to hold the bath towel and the bath towel holder is used to hold hand towel.)

heading: G LIMITATIONS AND FUTURE WORKS.
text: First, our work uses 3D point clouds as inputs, ignoring the object color, texture, or other nongeometric information. Besides, our system assumes all the objects in the scene are pre-segmented and many visual quantities are direction observable (e.g., the state change of objects). Future works can study how imperfect visual observations may affect our network predictions and how to address such challenges if any. Lastly, we also abstract away the robotic navigation and detailed manipulation. Future works may investigate building an end-to-end pipeline considering navigation and manipulation.
Under review as a conference paper at ICLR 2022
0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 # of Interactions
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
F1
kitchen bedroom living_room bathroom
Figure F.13: A bathroom scene where a hand towel holder holds a hand towel while a bath towel holder holds a bath towel. The agent confuses the two towels and predicts two extra edges.
Under review as a conference paper at ICLR 2022
Table E.5: The edge initialization of GCN of SR-Prior-Net and Exploration Policy. The experiments is conducted on the bathroom with 20% budget.
Precision Recall F1 Prior only 0.872 0.971 0.913 Distance only 0.835 0.986 0.902 No Scaling (γ = 1.) 0.819 0.991 0.894 Scaling factor γ = 0.4 1.0 0.836 0.908 Scaling factor γ = 0.8 0.873 0.986 0.922
Ours-Final 0.873 0.989 0.923

Review summaries:
This paper proposes a method to learn relationships between two objects, e.g., flipping a switch turns a light on. This is formulated as a binary classification task taking two objects as input and outputing a single a value $\in [0,1]$ if the objects are related or not. The paper proposes a method to train this using RL to select the training (object_1, object_2) pairs. The paper also introduces a dataset for evaluation and provides a lot of experiments to study this.

This paper proposes to study inter-object functional relationships such as a switch on the wall turning on or off the light. To tackle this task, this work designs a two-stage approach under a self-supervised interactive learning-from-exploration framework, which can predict functional relationships between objects in new 3D indoor environments. Furthermore, this work creates a dataset based on AI2THOR and AI2THOR to demonstrate the task. 

The paper presents an interesting research problem of "Learning Inter-Object Functional Relationships" in virtual 3D indoor environments, where the goal is to make an agent understand the functional effect on an object by interacting with other objects in the scene. Two important (and related) questions arise: (a) How to understand which object(s) affects the functionality of which other object(s) in the room?, and (b) how to learn such inter-object functional priors to build a good knowledge base for reliable inference in unseen environments?  The paper addresses these two questions by presenting a reinforcement-based learning algorithm.   The technical summary of the paper is given below.  ===Goal===:   Learning inter-object functional relationships, represented by scene graphs. That is, an agent is provided with large-scale scenes to explore for learning in the training stage and is asked to predict a functional scene graph, (S, R_s) for a new scene at test time. Here S is the scene and R_s is the relationship scene graph for the scene, S.    ===Assumptions===:  Object segmentation and state changes (by "state", the authors refer to the functional outcome of an object) are provided to the agent apriori. Therefore, the functional effect of a relationship is directly observable.    ===Input===:   A 3D scene as a set of objects {O_1, O_2, ..., O_n}, where each object O_i is explicitly modeled as (O_i^hat, c_i, s_i) where O_i^hat represents the point cloud of the object, c_i represents the 3D centroid, and s_i represents the object's isotropic scale.    ===Output===:  Functional scene graph, R_s, i.e., whether there exists a link between two nodes (objects) of the scene graph (scene).     ===Learning Style===:   Self-supervised learning framework     ===Neural Network===:  1) Pretrained PointNet++ and an MLP (these two make up what is called a "BR-Prior-Net", which is short for Binary Relationship Prior Net) -- supervised training using observed GT data  2) GCN, 3-layer (this is a part of what is called an "SR-Prior-Net", which is short for Scene Relationship Prior Net) -- Partly supervised using data from the above step and some rule-based heuristics  The goal of the above two networks is to propose functionally related objects.  3) SR-Posterior-Net, which has the same architecture and weights as SR-Prior-Net. The only difference is that the input to SR-Posterior-Net, in the beginning, is the refined prediction output from SR-Prior-Net.  4) A GCN trained using Reinforcement Learning (RL) algorithm to output action probability scores for each object that indicates the probability that this object triggers a state change on another object. Essentially, the RL-algorithm-based optimization on GCN is used to obtain supervision ground truth.    ===Dataset Used===:   AI2Thor, PartNet-Mobility  1) A total of 1200 scenes covering 23360 objects from 121 categories are used from the above datasets.  2) Among these, 27 are trigger objects, 29 are responders, where 11 are both trigger and responders (e.g., desk lamp), and 79 are non-interactive background objects    ===Relevance of datasets===:   The above two datasets employed fit well for working on the proposed problem.   ===Loss===:  1) For training the RL algorithm, a Policy Optimization loss is formulated. 2) SR-Priori, BR-Priori, and SR-Posteriori are all simple link prediction tasks. So, standard binary cross entropy loss.    ===Objective Evaluation Metrics===:  Precision, Recall and F1-scores

The paper proposes a framework to study inter-object functional relationships, i.e., where the state change in one of the objects causes functional effects on other 3D objects.  The authors propose a two-stage strategy guided by a self-supervised strategy. The first stage predicts possible functional relationships between the objects. In the second stage, the system performs interactions between the objects in the scene. The authors present the inter-functional relationship with a directed graph.   The binary relationships are represented with a prior network. Processing all the objects results in a scene representation of a functional relationship. Their procedure for training an exploratory policy allows them to generate data to construct a posterior network of the functional relationship in the scene. The authors create a dataset using scenes and objects from AI2Thor, enriched with articulated objects from PartNet.  The authors present an ablation study where baselines are extracted for the binary priors, scene priors, the exploration policy, the selection of objects, and their level of interaction.