IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenes

1 INTRODUCTION . When first-time entering a hotel room in a foreign country , we may need to search among buttons on the wall to find the one that turns on a specific light , figure out how to turn on the faucets in the bathroom and adjust water temperature , or perform several operations on the TV and the remotes to switch to our favorite channel . Many of these may require no effort if we have used the same models of lamp or faucet before , while the others that we are less familiar with will take more time for us to explore and figure out how things work . We humans can quickly adapt to a new indoor environment thanks to our strong prior knowledge about how objects could be functionally related in general human environments ( e.g. , the room light can be turned on by pressing a button or toggling a switch ) and our amazing capability of using only a few interactions while exploring the environment to learn how things work in cases of uncertainty ( e.g. , multiple lights and buttons in the room ) . In this work , we investigate how to equip AI systems with the same capabilities of learning inter-object functional relationships in 3D indoor environments . With the recent research popularity of embodied AI in perception and interaction , many previous works have studied how to perceive , model , and understand object functionality ( Kim et al. , 2014 ; Hu et al. , 2016 ; 2018 ; 2020 ) and agent-object interaction ( Montesano & Lopes , 2009 ; Do et al. , 2018 ; Nagarajan & Grauman , 2020 ; Mo et al. , 2021a ) , just to name a few . However , these works mostly investigate single-object/scene or agent-object scenarios . A few past works ( Jiang et al. , 2012 ; Sun et al. , 2014 ; Zhu et al. , 2015 ; Mo et al. , 2021b ; Cheng et al. , 2021 ) have studied inter-object relationships given a pair of objects or for specific tasks ( e.g. , placement , stacking ) . We make a first step bringing up research attentions to a new type of visual quantity – inter-object functional relationship , and an interesting yet underexplored challenging problem of how to learn such generic functional relationships in novel 3D indoor environments . Given a novel indoor scene with multiple objects as the input ( Fig . 1 , left ) , we predict a complete directional graph of inter-object functional relationships ( Fig . 1 , middle ) on how the state change Under review as a conference paper at ICLR 2022 4 Teaser of one object causes functional effects on other objects . While the semantics of individual objects already suggest some functional priors ( e.g. , buttons can be pressed , electrical devices can be turned on ) , our system learns to build up the correspondence among them ( e.g. , which button controls which electrical device ) . Also , since one may encounter unknown object categories exploring novel scenes , we do not assume semantic labels of objects and only use shape geometry as input . Our system , therefore , learns inter-object functional geometry-mappings to what to trigger ( e.g. , one button among others on the wall ) and what other shapes the interaction effects ( e.g. , one light in the room ) . Though seemingly simple as one may interact with all objects in the scene to query the functional scene graph , it is actually very challenging how to efficiently and accurately perceive , reason , and explore the scene with as few interactions as possible . First of all , we do not want to waste time interacting with books and bags as they will not functionally trigger other objects . Secondly , the semantics of two objects may directly indicate functional relationships . For example , we know that buttons control electrical devices in general . Besides , in cases of having two buttons and two lights in the scene , one may guess the pairings according to spatial proximity . All of these examples indicate that learning some geometric and semantic priors among the objects may help predict many inter-object functional relationships even without any interaction . However , there are always scenarios where uncertainties exist . For example , for the stove highlighted in Fig . 1 ( right ) , interactions have to be taken place to figure out which knob controls which burner , though only two interactions are needed for the four pairs if our agent can smartly infer the other two automatically . We design a two-staged approach , under a self-supervised interactive learning-from-exploration framework , to tackle the problem . The first stage perceives and reasons over the objects in a novel test scene to predict possible inter-object functional relationships using learned functional priors about the object geometry and scene layout . Then , in the second stage , our system performs a few interactions in the scene to reduce uncertainties and fastly adapts to a posterior functional scene graph prediction . We jointly train a reinforcement learning policy to explore large-scale training scenes and use the collected interaction observations to supervise the prior/posterior networks in the system . Since we are the first to explore this task to our best knowledge , we create a new hybrid dataset based on AI2THOR ( Kolve et al. , 2017 ) and PartNet ( Mo et al. , 2019b ) as the benchmark , on which we conduct extensive experiments comparing to various baselines and illustrating our interesting findings . Experiments validate that both stages contribute to the final predictions and prove the training synergy of the exploration policy and the prior/posterior networks . We also find reasonable generalization capabilities when testing over novel scene types with unknown object categories . In summary , this work makes the following main contributions : • we first formulate an important , interesting , yet underexplored challenging task – learning inter-object functional relationships in novel 3D indoor environments ; • we propose a novel self-supervised learning-from-exploration framework combining a prior-inference stage and a fast-interactive-adaptation stage to tackle the problem ; • we create a new dataset for benchmarking the proposed task and demonstrating the effec- tiveness of the proposed approach . Under review as a conference paper at ICLR 2022 2 RELATED WORKS . Visual Functionality and Affordance . We humans perceive and interact with the surrounding 3D world to accomplish everyday tasks . To equip AI systems with the same capabilities , researchers have done many works investigating shape functionality ( Kim et al. , 2014 ; Hu et al. , 2016 ; 2018 ; 2020 ; Lai et al. , 2021 ; Guan et al. , 2020 ) , grasp affordance ( Lenz et al. , 2015 ; Redmon & Angelova , 2015 ; Montesano & Lopes , 2009 ; Qin et al. , 2020 ; Kokic et al. , 2020 ; Mandikal & Grauman , 2021 ; Yang et al. , 2020 ; Corona et al. , 2020 ; Kjellström et al. , 2011 ; Fang et al. , 2018 ; Nagarajan et al. , 2019 ; Brahmbhatt et al. , 2019 ; Jiang et al. , 2021 ) , manipulation affordance ( Do et al. , 2018 ; Nagarajan et al. , 2019 ; 2020 ; Nagarajan & Grauman , 2020 ; Mo et al. , 2021a ) , scene affordance ( Piyathilaka & Kodagoda , 2015 ; Rhinehart & Kitani , 2016 ; Li et al. , 2019b ) , etc . While these works mostly study single-object/scene or agent-object interaction scenarios , our work explores inter-object relationships . Besides , we investigate how to explore functional relationships by interacting in 3D scenes , different from previous works that study interactions given specific object pairs or tasks ( Jiang et al. , 2012 ; Sun et al. , 2014 ; Zhu et al. , 2015 ; Mo et al. , 2021b ; Cheng et al. , 2021 ) . Inter-object Relationships . Besides perceiving individual objects in the environment , understanding the rich relationships among them is also crucial for many downstream applications . Previous works have attempted to model object supporting and contact relationships ( Fisher et al. , 2011 ; Silberman et al. , 2012 ) , spatial relationships ( Galleguillos et al. , 2008 ; Gould et al. , 2008 ; Kulkarni et al. , 2019 ) , semantic relationships ( Sadeghi & Farhadi , 2011 ; Divvala et al. , 2014 ) , physical relationships ( Mitash et al. , 2019 ) , etc . More recently , researchers have introduced scene graphs ( Liu et al. , 2014 ; Johnson et al. , 2015 ; Krishna et al. , 2017 ; Xu et al. , 2017 ; Li et al. , 2017b ; Zhu et al. , 2020 ; Wald et al. , 2020 ) and hierarchies ( Armeni et al. , 2019 ; Li et al. , 2019a ; 2017a ; Mo et al. , 2019b ; a ) to parse the objects , object parts , and their relationships . Unlike the spatial , geometric , semantic , or physical relationships investigated in these works , we pay more attention to inter-object functional relationships in which the state change of one object causes functional effects on other objects . Learning from Exploration and Interaction . While deep learning has demonstrated great success in various application domains ( Russakovsky et al. , 2015 ; Silver et al. , 2016 ) , large-scale annotated data for supervision inevitably becomes the bottleneck . Many works thus explore self-supervised learning via active perception ( Wilkes & Tsotsos , 1992 ) , interactive perception ( Bohg et al. , 2017 ) , or interactive exploration ( Wyatt et al. , 2011 ) to learn visual representations ( Jayaraman & Grauman , 2018 ; Yang et al. , 2019 ; Weihs et al. , 2019 ; Fang et al. , 2020 ; Zakka et al. , 2020 ) , objects and poses ( Caicedo & Lazebnik , 2015 ; Han et al. , 2019 ; Deng et al. , 2020 ; Chaplot et al. , 2020b ; Choi et al. , 2021 ) , segmentation and parts ( Katz & Brock , 2008 ; Kenney et al. , 2009 ; Van Hoof et al. , 2014 ; Pathak et al. , 2018 ; Eitel et al. , 2019 ; Lohmann et al. , 2020 ; Gadre et al. , 2021 ) , physics and dynamics ( Wu et al. , 2015 ; Mottaghi et al. , 2016 ; Agrawal et al. , 2016 ; Li et al. , 2016 ; Janner et al. , 2018 ; Xu et al. , 2019 ; Lohmann et al. , 2020 ; Ehsani et al. , 2020 ) , manipulation skills ( Pinto & Gupta , 2016 ; Levine et al. , 2018 ; Agrawal et al. , 2016 ; Pinto et al. , 2016 ; Pinto & Gupta , 2017 ; Zeng et al. , 2018 ; Batra et al. , 2020 ) , navigation policies ( Anderson et al. , 2018 ; Chen et al. , 2019 ; Chaplot et al. , 2020a ; Ramakrishnan et al. , 2021 ) , etc . In this work , we design interactive policies to explore novel 3D indoor rooms and learn our newly proposed inter-object functional relationships . 3 PROBLEM FORMULATION . We formulate a novel problem of inferring inter-object functional relationships ( IFRs ) among objects in novel 3D indoor environments ( Fig . 1 ) . For the task input , each scene S comprises multiple 3D objects { O1 , O2 , · · · , On } , each of which is presented as a 3D point cloud with 2,048 points depicting the complete shape geometry . Every object Oi may be subject to state changes ( e.g. , water is running or not from a faucet , a button is pressed up/down ) . The set of 3D objects describe the complete 3D scene S = { O1 , O2 , · · · , On } including both the objects with and without possible state changes . The spatial layout of the objects in the scene is explicitly modeled that each object Oi = ( Ôi , ci , si ) can be decomposed into a zero-centered unit-sized shape point cloud Ôi ∈ R2,048×3 , the object center position in the scene ci ∈ R3 , and its isotropic scale si ∈ R. For the task output , we are interested to figure out the ( IFRs ) RS . In a case that interacting with an object Oi ( a trigger ) will cause a functional state change of another object O j ( a responder ) , we formally define an observed IFR ( Oi , O j ) ∈RS . In the task output RS , there could be one-to-one or many-to-many relationships between two objects , as a trigger object may cause functional state changes of many responder objects and vice versa . We define a task scenario that an agent actively explores the environment and interacts with the objects to output the inter-object functional relationships following the popular embodied and real-world Under review as a conference paper at ICLR 2022 5 settings ( Nagarajan & Grauman , 2020 ; Ramakrishnan et al. , 2021 ) . For each interaction , an agent picks an object , triggers its state change , and directly receives the state changes of the other objects that have been affected in the scene . We assume that standardized visual quantities ( e.g. , object segmentation , state changes ) are provided to the agent as they are well-established research topics by themselves ( Geiger et al. , 2012 ; Li et al. , 2020 ) , and thus that the functional effect of a relationship is directly observable . An agent is provided with large-scale scenes to explore for learning in the training stage and is asked to predict the functional scene graph ( S , RS ) for a novel scene at the test time . We also abstract away complexities on robotic navigation ( Anderson et al. , 2018 ; Ramakrishnan et al. , 2021 ) and manipulation ( Nagarajan & Grauman , 2020 ; Mo et al. , 2021a ) , which are orthogonal to our contribution to estimating inter-object functional relationships .

Summary: This paper proposes to study inter-object functional relationships such as a switch on the wall turning on or off the light. To tackle this task, this work designs a two-stage approach under a self-supervised interactive learning-from-exploration framework, which can predict functional relationships between objects in new 3D indoor environments. Furthermore, this work creates a dataset based on AI2THOR and AI2THOR to demonstrate the task.