IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenes

1 INTRODUCTION . When first-time entering a hotel room in a foreign country , we may need to search among buttons on the wall to find the one that turns on a specific light , figure out how to turn on the faucets in the bathroom and adjust water temperature , or perform several operations on the TV and the remotes to switch to our favorite channel . Many of these may require no effort if we have used the same models of lamp or faucet before , while the others that we are less familiar with will take more time for us to explore and figure out how things work . We humans can quickly adapt to a new indoor environment thanks to our strong prior knowledge about how objects could be functionally related in general human environments ( e.g. , the room light can be turned on by pressing a button or toggling a switch ) and our amazing capability of using only a few interactions while exploring the environment to learn how things work in cases of uncertainty ( e.g. , multiple lights and buttons in the room ) . In this work , we investigate how to equip AI systems with the same capabilities of learning inter-object functional relationships in 3D indoor environments . With the recent research popularity of embodied AI in perception and interaction , many previous works have studied how to perceive , model , and understand object functionality ( Kim et al. , 2014 ; Hu et al. , 2016 ; 2018 ; 2020 ) and agent-object interaction ( Montesano & Lopes , 2009 ; Do et al. , 2018 ; Nagarajan & Grauman , 2020 ; Mo et al. , 2021a ) , just to name a few . However , these works mostly investigate single-object/scene or agent-object scenarios . A few past works ( Jiang et al. , 2012 ; Sun et al. , 2014 ; Zhu et al. , 2015 ; Mo et al. , 2021b ; Cheng et al. , 2021 ) have studied inter-object relationships given a pair of objects or for specific tasks ( e.g. , placement , stacking ) . We make a first step bringing up research attentions to a new type of visual quantity – inter-object functional relationship , and an interesting yet underexplored challenging problem of how to learn such generic functional relationships in novel 3D indoor environments . Given a novel indoor scene with multiple objects as the input ( Fig . 1 , left ) , we predict a complete directional graph of inter-object functional relationships ( Fig . 1 , middle ) on how the state change Under review as a conference paper at ICLR 2022 4 Teaser of one object causes functional effects on other objects . While the semantics of individual objects already suggest some functional priors ( e.g. , buttons can be pressed , electrical devices can be turned on ) , our system learns to build up the correspondence among them ( e.g. , which button controls which electrical device ) . Also , since one may encounter unknown object categories exploring novel scenes , we do not assume semantic labels of objects and only use shape geometry as input . Our system , therefore , learns inter-object functional geometry-mappings to what to trigger ( e.g. , one button among others on the wall ) and what other shapes the interaction effects ( e.g. , one light in the room ) . Though seemingly simple as one may interact with all objects in the scene to query the functional scene graph , it is actually very challenging how to efficiently and accurately perceive , reason , and explore the scene with as few interactions as possible . First of all , we do not want to waste time interacting with books and bags as they will not functionally trigger other objects . Secondly , the semantics of two objects may directly indicate functional relationships . For example , we know that buttons control electrical devices in general . Besides , in cases of having two buttons and two lights in the scene , one may guess the pairings according to spatial proximity . All of these examples indicate that learning some geometric and semantic priors among the objects may help predict many inter-object functional relationships even without any interaction . However , there are always scenarios where uncertainties exist . For example , for the stove highlighted in Fig . 1 ( right ) , interactions have to be taken place to figure out which knob controls which burner , though only two interactions are needed for the four pairs if our agent can smartly infer the other two automatically . We design a two-staged approach , under a self-supervised interactive learning-from-exploration framework , to tackle the problem . The first stage perceives and reasons over the objects in a novel test scene to predict possible inter-object functional relationships using learned functional priors about the object geometry and scene layout . Then , in the second stage , our system performs a few interactions in the scene to reduce uncertainties and fastly adapts to a posterior functional scene graph prediction . We jointly train a reinforcement learning policy to explore large-scale training scenes and use the collected interaction observations to supervise the prior/posterior networks in the system . Since we are the first to explore this task to our best knowledge , we create a new hybrid dataset based on AI2THOR ( Kolve et al. , 2017 ) and PartNet ( Mo et al. , 2019b ) as the benchmark , on which we conduct extensive experiments comparing to various baselines and illustrating our interesting findings . Experiments validate that both stages contribute to the final predictions and prove the training synergy of the exploration policy and the prior/posterior networks . We also find reasonable generalization capabilities when testing over novel scene types with unknown object categories . In summary , this work makes the following main contributions : • we first formulate an important , interesting , yet underexplored challenging task – learning inter-object functional relationships in novel 3D indoor environments ; • we propose a novel self-supervised learning-from-exploration framework combining a prior-inference stage and a fast-interactive-adaptation stage to tackle the problem ; • we create a new dataset for benchmarking the proposed task and demonstrating the effec- tiveness of the proposed approach . Under review as a conference paper at ICLR 2022 2 RELATED WORKS . Visual Functionality and Affordance . We humans perceive and interact with the surrounding 3D world to accomplish everyday tasks . To equip AI systems with the same capabilities , researchers have done many works investigating shape functionality ( Kim et al. , 2014 ; Hu et al. , 2016 ; 2018 ; 2020 ; Lai et al. , 2021 ; Guan et al. , 2020 ) , grasp affordance ( Lenz et al. , 2015 ; Redmon & Angelova , 2015 ; Montesano & Lopes , 2009 ; Qin et al. , 2020 ; Kokic et al. , 2020 ; Mandikal & Grauman , 2021 ; Yang et al. , 2020 ; Corona et al. , 2020 ; Kjellström et al. , 2011 ; Fang et al. , 2018 ; Nagarajan et al. , 2019 ; Brahmbhatt et al. , 2019 ; Jiang et al. , 2021 ) , manipulation affordance ( Do et al. , 2018 ; Nagarajan et al. , 2019 ; 2020 ; Nagarajan & Grauman , 2020 ; Mo et al. , 2021a ) , scene affordance ( Piyathilaka & Kodagoda , 2015 ; Rhinehart & Kitani , 2016 ; Li et al. , 2019b ) , etc . While these works mostly study single-object/scene or agent-object interaction scenarios , our work explores inter-object relationships . Besides , we investigate how to explore functional relationships by interacting in 3D scenes , different from previous works that study interactions given specific object pairs or tasks ( Jiang et al. , 2012 ; Sun et al. , 2014 ; Zhu et al. , 2015 ; Mo et al. , 2021b ; Cheng et al. , 2021 ) . Inter-object Relationships . Besides perceiving individual objects in the environment , understanding the rich relationships among them is also crucial for many downstream applications . Previous works have attempted to model object supporting and contact relationships ( Fisher et al. , 2011 ; Silberman et al. , 2012 ) , spatial relationships ( Galleguillos et al. , 2008 ; Gould et al. , 2008 ; Kulkarni et al. , 2019 ) , semantic relationships ( Sadeghi & Farhadi , 2011 ; Divvala et al. , 2014 ) , physical relationships ( Mitash et al. , 2019 ) , etc . More recently , researchers have introduced scene graphs ( Liu et al. , 2014 ; Johnson et al. , 2015 ; Krishna et al. , 2017 ; Xu et al. , 2017 ; Li et al. , 2017b ; Zhu et al. , 2020 ; Wald et al. , 2020 ) and hierarchies ( Armeni et al. , 2019 ; Li et al. , 2019a ; 2017a ; Mo et al. , 2019b ; a ) to parse the objects , object parts , and their relationships . Unlike the spatial , geometric , semantic , or physical relationships investigated in these works , we pay more attention to inter-object functional relationships in which the state change of one object causes functional effects on other objects . Learning from Exploration and Interaction . While deep learning has demonstrated great success in various application domains ( Russakovsky et al. , 2015 ; Silver et al. , 2016 ) , large-scale annotated data for supervision inevitably becomes the bottleneck . Many works thus explore self-supervised learning via active perception ( Wilkes & Tsotsos , 1992 ) , interactive perception ( Bohg et al. , 2017 ) , or interactive exploration ( Wyatt et al. , 2011 ) to learn visual representations ( Jayaraman & Grauman , 2018 ; Yang et al. , 2019 ; Weihs et al. , 2019 ; Fang et al. , 2020 ; Zakka et al. , 2020 ) , objects and poses ( Caicedo & Lazebnik , 2015 ; Han et al. , 2019 ; Deng et al. , 2020 ; Chaplot et al. , 2020b ; Choi et al. , 2021 ) , segmentation and parts ( Katz & Brock , 2008 ; Kenney et al. , 2009 ; Van Hoof et al. , 2014 ; Pathak et al. , 2018 ; Eitel et al. , 2019 ; Lohmann et al. , 2020 ; Gadre et al. , 2021 ) , physics and dynamics ( Wu et al. , 2015 ; Mottaghi et al. , 2016 ; Agrawal et al. , 2016 ; Li et al. , 2016 ; Janner et al. , 2018 ; Xu et al. , 2019 ; Lohmann et al. , 2020 ; Ehsani et al. , 2020 ) , manipulation skills ( Pinto & Gupta , 2016 ; Levine et al. , 2018 ; Agrawal et al. , 2016 ; Pinto et al. , 2016 ; Pinto & Gupta , 2017 ; Zeng et al. , 2018 ; Batra et al. , 2020 ) , navigation policies ( Anderson et al. , 2018 ; Chen et al. , 2019 ; Chaplot et al. , 2020a ; Ramakrishnan et al. , 2021 ) , etc . In this work , we design interactive policies to explore novel 3D indoor rooms and learn our newly proposed inter-object functional relationships . 3 PROBLEM FORMULATION . We formulate a novel problem of inferring inter-object functional relationships ( IFRs ) among objects in novel 3D indoor environments ( Fig . 1 ) . For the task input , each scene S comprises multiple 3D objects { O1 , O2 , · · · , On } , each of which is presented as a 3D point cloud with 2,048 points depicting the complete shape geometry . Every object Oi may be subject to state changes ( e.g. , water is running or not from a faucet , a button is pressed up/down ) . The set of 3D objects describe the complete 3D scene S = { O1 , O2 , · · · , On } including both the objects with and without possible state changes . The spatial layout of the objects in the scene is explicitly modeled that each object Oi = ( Ôi , ci , si ) can be decomposed into a zero-centered unit-sized shape point cloud Ôi ∈ R2,048×3 , the object center position in the scene ci ∈ R3 , and its isotropic scale si ∈ R. For the task output , we are interested to figure out the ( IFRs ) RS . In a case that interacting with an object Oi ( a trigger ) will cause a functional state change of another object O j ( a responder ) , we formally define an observed IFR ( Oi , O j ) ∈RS . In the task output RS , there could be one-to-one or many-to-many relationships between two objects , as a trigger object may cause functional state changes of many responder objects and vice versa . We define a task scenario that an agent actively explores the environment and interacts with the objects to output the inter-object functional relationships following the popular embodied and real-world Under review as a conference paper at ICLR 2022 5 settings ( Nagarajan & Grauman , 2020 ; Ramakrishnan et al. , 2021 ) . For each interaction , an agent picks an object , triggers its state change , and directly receives the state changes of the other objects that have been affected in the scene . We assume that standardized visual quantities ( e.g. , object segmentation , state changes ) are provided to the agent as they are well-established research topics by themselves ( Geiger et al. , 2012 ; Li et al. , 2020 ) , and thus that the functional effect of a relationship is directly observable . An agent is provided with large-scale scenes to explore for learning in the training stage and is asked to predict the functional scene graph ( S , RS ) for a novel scene at the test time . We also abstract away complexities on robotic navigation ( Anderson et al. , 2018 ; Ramakrishnan et al. , 2021 ) and manipulation ( Nagarajan & Grauman , 2020 ; Mo et al. , 2021a ) , which are orthogonal to our contribution to estimating inter-object functional relationships .

Summary: The paper presents an interesting research problem of "Learning Inter-Object Functional Relationships" in virtual 3D indoor environments, where the goal is to make an agent understand the functional effect on an object by interacting with other objects in the scene. Two important (and related) questions arise: (a) How to understand which object(s) affects the functionality of which other object(s) in the room?, and (b) how to learn such inter-object functional priors to build a good knowledge base for reliable inference in unseen environments?  The paper addresses these two questions by presenting a reinforcement-based learning algorithm.   The technical summary of the paper is given below.  ===Goal===:   Learning inter-object functional relationships, represented by scene graphs. That is, an agent is provided with large-scale scenes to explore for learning in the training stage and is asked to predict a functional scene graph, (S, R_s) for a new scene at test time. Here S is the scene and R_s is the relationship scene graph for the scene, S.    ===Assumptions===:  Object segmentation and state changes (by "state", the authors refer to the functional outcome of an object) are provided to the agent apriori. Therefore, the functional effect of a relationship is directly observable.    ===Input===:   A 3D scene as a set of objects {O_1, O_2, ..., O_n}, where each object O_i is explicitly modeled as (O_i^hat, c_i, s_i) where O_i^hat represents the point cloud of the object, c_i represents the 3D centroid, and s_i represents the object's isotropic scale.    ===Output===:  Functional scene graph, R_s, i.e., whether there exists a link between two nodes (objects) of the scene graph (scene).     ===Learning Style===:   Self-supervised learning framework     ===Neural Network===:  1) Pretrained PointNet++ and an MLP (these two make up what is called a "BR-Prior-Net", which is short for Binary Relationship Prior Net) -- supervised training using observed GT data  2) GCN, 3-layer (this is a part of what is called an "SR-Prior-Net", which is short for Scene Relationship Prior Net) -- Partly supervised using data from the above step and some rule-based heuristics  The goal of the above two networks is to propose functionally related objects.  3) SR-Posterior-Net, which has the same architecture and weights as SR-Prior-Net. The only difference is that the input to SR-Posterior-Net, in the beginning, is the refined prediction output from SR-Prior-Net.  4) A GCN trained using Reinforcement Learning (RL) algorithm to output action probability scores for each object that indicates the probability that this object triggers a state change on another object. Essentially, the RL-algorithm-based optimization on GCN is used to obtain supervision ground truth.    ===Dataset Used===:   AI2Thor, PartNet-Mobility  1) A total of 1200 scenes covering 23360 objects from 121 categories are used from the above datasets.  2) Among these, 27 are trigger objects, 29 are responders, where 11 are both trigger and responders (e.g., desk lamp), and 79 are non-interactive background objects    ===Relevance of datasets===:   The above two datasets employed fit well for working on the proposed problem.   ===Loss===:  1) For training the RL algorithm, a Policy Optimization loss is formulated. 2) SR-Priori, BR-Priori, and SR-Posteriori are all simple link prediction tasks. So, standard binary cross entropy loss.    ===Objective Evaluation Metrics===:  Precision, Recall and F1-scores