Learning to Act with Affordance-Aware Multimodal Neural SLAM

1 INTRODUCTION . There is significant recent progress in learning simulated embodied agents Pashevich et al . ( 2021 ) ; Zhang & Chai ( 2021 ) ; Blukis et al . ( 2021 ) ; Nagarajan & Grauman ( 2020 ) ; Singh et al . ( 2020 ) ; Suglia et al . ( 2021 ) that follow human language instructions , process multi-sensory inputs and act to complete complex tasks Anderson et al . ( 2018 ) ; Das et al . ( 2018 ) ; Chen et al . ( 2019 ) ; Shridhar et al . ( 2020 ) . Despite this , challenges remain before agent performance approaches satisfactory levels , including long-horizon planning and reasoning Blukis et al . ( 2021 ) , effective language grounding in visually rich environments , efficient exploration Chen et al . ( 2018 ) , and importantly , generalization to unseen environments . Most prior work Singh et al . ( 2020 ) ; Pashevich et al . ( 2021 ) ; Nguyen et al . ( 2021 ) ; Suglia et al . ( 2021 ) adopted end-to-end deep learning models that map visual and language inputs into action sequences . Besides being difficult to interpret , these models show limited generalization , suffering from significant performance drop when tested on new tasks and scenes . In contrast , hierarchical approach Zhang & Chai ( 2021 ) ; Blukis et al . ( 2021 ) achieve better generalization performance and interpretability . Although hierarchical structure is helpful for long-horizon planning , its key impact is an expressive semantic representation of the environment acquired via Neural SLAM-based approaches Chaplot et al . ( 2020a ; c ) ; Blukis et al . ( 2021 ) . However , a missing component in these methods is fine-grained affordance Kim & Sukhatme ( 2015 ) ; Qi et al . ( 2019 ) . To build a robotic assistant that can follow human instructions to complete a task ( e.g. , Open the fridge and grab me a soda ) , it is essential that the agent can perform affordance-aware navigation : it must navigate to a reasonable position and pose near the fridge that enables follow-on actions open and pick-up . Operationally , the agent has to move to a location where the fridge is within reach yet without arresting the fridge door from being opened . Ideally , it should also position itself so that the soda is in its first person viewing field to allow the follow-on pick-up action . This is challenging compared to pure navigation ( where navigating to any location close to the fridge is acceptable ) . To achieve this , we propose a sophisticated affordance-aware semantic representation that leads to accurate planning for navigation setting up subsequent object interactions for success . Efficient exploration of the environment Ramakrishnan et al . ( 2021 ) ; Chen et al . ( 2018 ) needs to be addressed to establish this semantic representation - it is unacceptable for a robot to wander around for an extended period of time to complete a single task in a real-world setting . To resolve this issue , we propose the first multimodal exploration module that takes language instruction as guidance and keeps track of visited regions to explore the area of interest effectively and efficiently . This lays a foundation for map construction , which is critical to long-horizon planning . Here , we introduce Affordance-aware Multimodal Neural SLAM ( AMSLAM ) , which implements two key insights to address the challenges of robust long-horizon planning , namely , efficient exploration and generalization : 1 . Affordance-aware semantic representation that estimates object information in terms of where the agent can interact with them to support sophisticated affordanceaware navigation , and 2 . Task-driven multimodal exploration that takes guidance from language instruction , visual input , and previously explored regions to improve the effectiveness and efficiency of exploration . AMSLAM is the first Neural SLAM-based approach for Embodied AI tasks to utilize several modalities for effective exploration and an affordance-aware semantic representation for robust long-horizon planning . We conduct comprehensive empirical studies on the ALFRED benchmark Shridhar et al . ( 2020 ) to demonstrate the key components of AMSLAM , setting a new state-of-the-art generalization performance at 23.48 % , a > 40 % improvement over prior published state-of-the-art approaches . 2 RELATED WORK . Recent progress in Embodied Artificial Intelligence , spans both simulation environments Kolve et al . ( 2017 ) ; Li et al . ( 2021 ) ; Savva et al . ( 2019 ) ; Gan et al . ( 2020 ) ; Puig et al . ( 2018 ) and sophisticated tasks Das et al . ( 2018 ) ; Anderson et al . ( 2018 ) ; Shridhar et al . ( 2020 ) . Our work is most closely related to research in language-guided task completion , Neural SLAM , and exploration . Language-Guided Task Completion . ALFRED Shridhar et al . ( 2020 ) is a benchmark that enables a learning agent to follow natural language descriptions to complete complex household tasks . The agent ’ s goal is to learn a mapping from natural language instructions to a sequence of actions for task completion in a simulated 3D environment . Various modeling approaches have been proposed falling into roughly two families of methods . The first focuses on learning large end-to-end models that directly translate instructions to low-level agent actions Singh et al . ( 2020 ) ; Suglia et al . ( 2021 ) ; Pashevich et al . ( 2021 ) . However , these agents typically suffer from poor generalization performance , and are difficult to interpret . Recently , hierarchical approaches Zhang & Chai ( 2021 ) ; Blukis et al . ( 2021 ) have attracted attention due to their better generalization and interpretability . We also adopt a hierarchical structure , focusing on affordance-aware navigation thereby achieving significantly better generalization than all existing approaches . Neural SLAM and Affordance-aware Semantic Representation . Neural SLAM Chaplot et al . ( 2020a ; b ; c ) , constructs an environment semantic representation enabling map-based long-horizon planning Chaplot et al . ( 2021 ) . However , these are tested in pure navigation tasks instead of complex household tasks , and does not consider affordance Qi et al . ( 2019 ) ; Nagarajan & Grauman ( 2020 ) ; Xu et al . ( 2020 ) , which is required for tasks involving both navigation and manipulations . In Blukis et al . ( 2021 ) , the authors utilize SLAM for 3D environment reconstruction in language-guided task completion . Their approach relies heavily on accurate depth prediction ( less robust in unseen environments ) . Instead , we propose a waypoint-oriented representation which associates each object with the locations on the floor from where the agent can interact with the object . Furthermore , different from the 2D affordance map in Blukis et al . ( 2021 ) that directly predicts affordance type , our semantic representation supports more fine-grained control of the robot ’ s position and pose , which facilitates significantly better generalization . The approach in Qi et al . ( 2019 ) assumes direct access to the ground truth depth information ( not available in our setup ) and the method in Nagarajan & Grauman ( 2020 ) only focuses on pure navigation problems . Learning to Explore for Navigation . An essential step in Neural SLAM-based approaches is learning to explore the environment for map building Ramakrishnan et al . ( 2021 ) ; Chen et al . ( 2018 ) ; Jayaraman & Grauman ( 2018 ) ; Chaplot et al . ( 2020a ) . Multiple approaches have been proposed to tackle aspects of exploration in the reinforcement learning Schmidhuber ( 1991 ) ; Pathak et al . ( 2017 ) ; Burda et al . ( 2018 ) ; Chen et al . ( 2018 ) ; Jayaraman & Grauman ( 2018 ) , computer vision Ramakrishnan et al . ( 2021 ) ; Nagarajan & Grauman ( 2020 ) , and robotics Blukis et al . ( 2021 ) ; Harrison et al . ( 2018 ) communities . The central principle of prior methods is learning to reduce environment uncertainty ; different definitions of uncertainty lead to the following types of methods Ramakrishnan et al . ( 2021 ) . Curiosity-driven Schmidhuber ( 1991 ) ; Pathak et al . ( 2017 ) ; Burda et al . ( 2018 ) approaches learn forward dynamics and reward visiting areas that are poorly predicted by the model . Count-based exploration Tang et al . ( 2017 ) ; Bellemare et al . ( 2016 ) ; Ostrovski et al . ( 2017 ) ; Rashid et al . ( 2020 ) encourages visiting states that are less frequently visited . Coverage-based Chen et al . ( 2018 ) ; Jayaraman & Grauman ( 2018 ) approaches reward visiting all navigable areas by searching in a task-agnostic manner . In contrast , we propose a multimodal exploration approach utilizing egocentric visual input , language instructions , and memory of explored areas to reduce task-specific uncertainty of points of interest ( areas important to complete the task ) . We show this to be more efficient , leading to more effective map prediction and robust planning . 3 PROBLEM FORMULATION . We focus on the ALFRED challenge Shridhar et al . ( 2020 ) , where an agent is asked to follow human instructions to complete long-horizon household tasks in indoor scenes ( simulated in AI2Thor Kolve et al . ( 2017 ) ) . Each task in ALFRED consists of several subgoals for either navigation ( moving in the environment ) or object interactions ( interacting with at least one object ) . Language inputs contain a high-level task description and a sequence of low-level step-by-step instructions ( each corresponding to a subgoal ) . The agent is a simulated robot with access to the states of the environment only through a front-view RGB camera with a relatively small field of view . The agent ’ s own state is a 5-tuple ( x , y , r , h , o ) , where x , y are its 2D position , r the horizontal rotation angle , h the vertical camera angles ( also called “ horizon ” ) and o the type of object held in its hand . The state space of the agent is discrete , with navigation actions : MoveAhead ( moving forward by 0.25m ) , RotateLeft & RotateRight ( rotating in the horizontal plane by 90◦ ) and LookUp & LookDown ( adjusting the horizon by 15◦ ) . Formally , r ∈ { 0◦ , 90◦ , 180◦ , 270◦ } and h ∈ { 60◦ , 45◦ , ... , −15◦ , −30◦ , } where positive h indicates facing downward . With these discrete actions , the agent has full knowledge of the relative changes ∆x , ∆y , ∆r and ∆h . Each of the 7 object interaction actions ( PickUp , Open , Slice , etc . ) is parametrized by an binary mask for the target object , which is usually predicted with a pre-trained instance segmentation module . Featuring long-horizon tasks with a range of interactions , the ALFRED challenge evaluates an agent ’ s ability to perform tasks over unseen test scenes , while only allowing ≤1000 steps and ≤10 action failures for each task at inference time . 4 AFFORDANCE-AWARE MULTIMODAL NEURAL SLAM . Affordance-aware navigation is a major challenge in solving complex and long-horizon indoor tasks such as ALFRED with both navigation and object interactions . Specifically , given each object of interest in the scene , the agent is required to not only find and approach it but also end up at a pose ( x , y , r , h ) , that is feasible for subsequent interactions with the object . For instance , to open a fridge , the robot should approach the fridge closely enough ( so the door is within reach ) , look at it ( so that the fridge is in the field of view ) , and leave enough room to open the door . To solve a long-horizon task involving multiple navigation and object interaction subgoals , it is natural to use an explicit semantic map , either 2D or 3D , of the environment ( similar to Neural Active SLAM Chaplot et al . ( 2020a ) ) , together with model-based planning ( e.g . as in HLSM Blukis et al . ( 2021 ) ) . This line of work tends to generalize better than models that directly learn mappings from human instructions to navigation & interaction actions ( e.g. , E.T . Pashevich et al . ( 2021 ) ) . With perfect knowledge of the environment , it is possible to achieve ( nearly ) perfect performance . In practice , however , the semantic map acquired at inference time is usually far from ideal , primarily due to Incompleteness ( missing information due to insufficient exploration of the scene ) and Inaccuracy ( erroneous object location prediction on the map , especially for small objects ) . To improve exploration performance , we propose a multimodal module that , at each step , predicts an exploration action a ∈ { MoveAhead , RotateLeft , RotateRight } by taking visual observations & actions in the past , step-by-step language instructions , and the explored area map which indicates where the agent has visited . We show that , compared to existing model-based approaches on ALFRED ( e.g. , HLSM Blukis et al . ( 2021 ) which applies random exploration ) , our use of lowlevel language instructions leads to more efficient exploration . The proposed exploration module operates at the subgoal level and only predicts exploration actions ( in contrast to E.T . which directly predicts actions for the entire task ) . The extra modality ( the explored area ) facilitates exploration by providing the agent with explicit spatial information . We illustrate the exploration module in Figure 3 , elaborate its details in Section 4.3 , and empirically demonstrate its advantages in Section 5 . To deal with the inaccuracy in map prediction , we carefully design an affordance-aware semantic representation for the environments . On one hand , knowing the precise spatial coordinates of objects requires precise depth information , which is difficult to acquire due to 3D sensor noise and/or inaccuracy in predicting depth from 2D images . On the other hand , affordance-aware navigation essentially asks for poses ( x , y , r , h ) of the agent suitable for interactions with the target objects , thus requiring only coarse-grained spatial information . Given an object type o , we define such corresponding poses as waypoints Wo and then treat navigation as a path planning problem among different waypoints . To generate such waypoints , we handle large objects ( fridges , cabinets , etc . ) and small objects ( apples , mug , etc . ) differently . The waypoints for large objects are computed using 2D grid maps predicted and aggregated from front-view camera images by a CNN-based network ; for small objects , we directly search over all observations acquired during the exploration phase with the help of a pre-trained Mask RCNN He et al . ( 2017 ) ( detailed below in Section 4.2 ) .

Summary: This paper proposes a new framework that improves the state-of-the-art performance of the ALFRED benchmark (accomplishing navigation and interaction tasks given language instructions in AI2THOR environments) by 40% relatively. Two claimed main technical contributions are: 1) leveraging multimodal inputs, specifically the newly incorporated language instructions and visited area maps, during exploration, 2) proposing an affordance-aware semantic representation that marks the object locations, heights, and possible agent interaction spots. Results that are 40% relatively better than the state-of-the-art are observed and several ablation studies over key network modules are also presented.