Learning to Act with Affordance-Aware Multimodal Neural SLAM

1 INTRODUCTION . There is significant recent progress in learning simulated embodied agents Pashevich et al . ( 2021 ) ; Zhang & Chai ( 2021 ) ; Blukis et al . ( 2021 ) ; Nagarajan & Grauman ( 2020 ) ; Singh et al . ( 2020 ) ; Suglia et al . ( 2021 ) that follow human language instructions , process multi-sensory inputs and act to complete complex tasks Anderson et al . ( 2018 ) ; Das et al . ( 2018 ) ; Chen et al . ( 2019 ) ; Shridhar et al . ( 2020 ) . Despite this , challenges remain before agent performance approaches satisfactory levels , including long-horizon planning and reasoning Blukis et al . ( 2021 ) , effective language grounding in visually rich environments , efficient exploration Chen et al . ( 2018 ) , and importantly , generalization to unseen environments . Most prior work Singh et al . ( 2020 ) ; Pashevich et al . ( 2021 ) ; Nguyen et al . ( 2021 ) ; Suglia et al . ( 2021 ) adopted end-to-end deep learning models that map visual and language inputs into action sequences . Besides being difficult to interpret , these models show limited generalization , suffering from significant performance drop when tested on new tasks and scenes . In contrast , hierarchical approach Zhang & Chai ( 2021 ) ; Blukis et al . ( 2021 ) achieve better generalization performance and interpretability . Although hierarchical structure is helpful for long-horizon planning , its key impact is an expressive semantic representation of the environment acquired via Neural SLAM-based approaches Chaplot et al . ( 2020a ; c ) ; Blukis et al . ( 2021 ) . However , a missing component in these methods is fine-grained affordance Kim & Sukhatme ( 2015 ) ; Qi et al . ( 2019 ) . To build a robotic assistant that can follow human instructions to complete a task ( e.g. , Open the fridge and grab me a soda ) , it is essential that the agent can perform affordance-aware navigation : it must navigate to a reasonable position and pose near the fridge that enables follow-on actions open and pick-up . Operationally , the agent has to move to a location where the fridge is within reach yet without arresting the fridge door from being opened . Ideally , it should also position itself so that the soda is in its first person viewing field to allow the follow-on pick-up action . This is challenging compared to pure navigation ( where navigating to any location close to the fridge is acceptable ) . To achieve this , we propose a sophisticated affordance-aware semantic representation that leads to accurate planning for navigation setting up subsequent object interactions for success . Efficient exploration of the environment Ramakrishnan et al . ( 2021 ) ; Chen et al . ( 2018 ) needs to be addressed to establish this semantic representation - it is unacceptable for a robot to wander around for an extended period of time to complete a single task in a real-world setting . To resolve this issue , we propose the first multimodal exploration module that takes language instruction as guidance and keeps track of visited regions to explore the area of interest effectively and efficiently . This lays a foundation for map construction , which is critical to long-horizon planning . Here , we introduce Affordance-aware Multimodal Neural SLAM ( AMSLAM ) , which implements two key insights to address the challenges of robust long-horizon planning , namely , efficient exploration and generalization : 1 . Affordance-aware semantic representation that estimates object information in terms of where the agent can interact with them to support sophisticated affordanceaware navigation , and 2 . Task-driven multimodal exploration that takes guidance from language instruction , visual input , and previously explored regions to improve the effectiveness and efficiency of exploration . AMSLAM is the first Neural SLAM-based approach for Embodied AI tasks to utilize several modalities for effective exploration and an affordance-aware semantic representation for robust long-horizon planning . We conduct comprehensive empirical studies on the ALFRED benchmark Shridhar et al . ( 2020 ) to demonstrate the key components of AMSLAM , setting a new state-of-the-art generalization performance at 23.48 % , a > 40 % improvement over prior published state-of-the-art approaches . 2 RELATED WORK . Recent progress in Embodied Artificial Intelligence , spans both simulation environments Kolve et al . ( 2017 ) ; Li et al . ( 2021 ) ; Savva et al . ( 2019 ) ; Gan et al . ( 2020 ) ; Puig et al . ( 2018 ) and sophisticated tasks Das et al . ( 2018 ) ; Anderson et al . ( 2018 ) ; Shridhar et al . ( 2020 ) . Our work is most closely related to research in language-guided task completion , Neural SLAM , and exploration . Language-Guided Task Completion . ALFRED Shridhar et al . ( 2020 ) is a benchmark that enables a learning agent to follow natural language descriptions to complete complex household tasks . The agent ’ s goal is to learn a mapping from natural language instructions to a sequence of actions for task completion in a simulated 3D environment . Various modeling approaches have been proposed falling into roughly two families of methods . The first focuses on learning large end-to-end models that directly translate instructions to low-level agent actions Singh et al . ( 2020 ) ; Suglia et al . ( 2021 ) ; Pashevich et al . ( 2021 ) . However , these agents typically suffer from poor generalization performance , and are difficult to interpret . Recently , hierarchical approaches Zhang & Chai ( 2021 ) ; Blukis et al . ( 2021 ) have attracted attention due to their better generalization and interpretability . We also adopt a hierarchical structure , focusing on affordance-aware navigation thereby achieving significantly better generalization than all existing approaches . Neural SLAM and Affordance-aware Semantic Representation . Neural SLAM Chaplot et al . ( 2020a ; b ; c ) , constructs an environment semantic representation enabling map-based long-horizon planning Chaplot et al . ( 2021 ) . However , these are tested in pure navigation tasks instead of complex household tasks , and does not consider affordance Qi et al . ( 2019 ) ; Nagarajan & Grauman ( 2020 ) ; Xu et al . ( 2020 ) , which is required for tasks involving both navigation and manipulations . In Blukis et al . ( 2021 ) , the authors utilize SLAM for 3D environment reconstruction in language-guided task completion . Their approach relies heavily on accurate depth prediction ( less robust in unseen environments ) . Instead , we propose a waypoint-oriented representation which associates each object with the locations on the floor from where the agent can interact with the object . Furthermore , different from the 2D affordance map in Blukis et al . ( 2021 ) that directly predicts affordance type , our semantic representation supports more fine-grained control of the robot ’ s position and pose , which facilitates significantly better generalization . The approach in Qi et al . ( 2019 ) assumes direct access to the ground truth depth information ( not available in our setup ) and the method in Nagarajan & Grauman ( 2020 ) only focuses on pure navigation problems . Learning to Explore for Navigation . An essential step in Neural SLAM-based approaches is learning to explore the environment for map building Ramakrishnan et al . ( 2021 ) ; Chen et al . ( 2018 ) ; Jayaraman & Grauman ( 2018 ) ; Chaplot et al . ( 2020a ) . Multiple approaches have been proposed to tackle aspects of exploration in the reinforcement learning Schmidhuber ( 1991 ) ; Pathak et al . ( 2017 ) ; Burda et al . ( 2018 ) ; Chen et al . ( 2018 ) ; Jayaraman & Grauman ( 2018 ) , computer vision Ramakrishnan et al . ( 2021 ) ; Nagarajan & Grauman ( 2020 ) , and robotics Blukis et al . ( 2021 ) ; Harrison et al . ( 2018 ) communities . The central principle of prior methods is learning to reduce environment uncertainty ; different definitions of uncertainty lead to the following types of methods Ramakrishnan et al . ( 2021 ) . Curiosity-driven Schmidhuber ( 1991 ) ; Pathak et al . ( 2017 ) ; Burda et al . ( 2018 ) approaches learn forward dynamics and reward visiting areas that are poorly predicted by the model . Count-based exploration Tang et al . ( 2017 ) ; Bellemare et al . ( 2016 ) ; Ostrovski et al . ( 2017 ) ; Rashid et al . ( 2020 ) encourages visiting states that are less frequently visited . Coverage-based Chen et al . ( 2018 ) ; Jayaraman & Grauman ( 2018 ) approaches reward visiting all navigable areas by searching in a task-agnostic manner . In contrast , we propose a multimodal exploration approach utilizing egocentric visual input , language instructions , and memory of explored areas to reduce task-specific uncertainty of points of interest ( areas important to complete the task ) . We show this to be more efficient , leading to more effective map prediction and robust planning . 3 PROBLEM FORMULATION . We focus on the ALFRED challenge Shridhar et al . ( 2020 ) , where an agent is asked to follow human instructions to complete long-horizon household tasks in indoor scenes ( simulated in AI2Thor Kolve et al . ( 2017 ) ) . Each task in ALFRED consists of several subgoals for either navigation ( moving in the environment ) or object interactions ( interacting with at least one object ) . Language inputs contain a high-level task description and a sequence of low-level step-by-step instructions ( each corresponding to a subgoal ) . The agent is a simulated robot with access to the states of the environment only through a front-view RGB camera with a relatively small field of view . The agent ’ s own state is a 5-tuple ( x , y , r , h , o ) , where x , y are its 2D position , r the horizontal rotation angle , h the vertical camera angles ( also called “ horizon ” ) and o the type of object held in its hand . The state space of the agent is discrete , with navigation actions : MoveAhead ( moving forward by 0.25m ) , RotateLeft & RotateRight ( rotating in the horizontal plane by 90◦ ) and LookUp & LookDown ( adjusting the horizon by 15◦ ) . Formally , r ∈ { 0◦ , 90◦ , 180◦ , 270◦ } and h ∈ { 60◦ , 45◦ , ... , −15◦ , −30◦ , } where positive h indicates facing downward . With these discrete actions , the agent has full knowledge of the relative changes ∆x , ∆y , ∆r and ∆h . Each of the 7 object interaction actions ( PickUp , Open , Slice , etc . ) is parametrized by an binary mask for the target object , which is usually predicted with a pre-trained instance segmentation module . Featuring long-horizon tasks with a range of interactions , the ALFRED challenge evaluates an agent ’ s ability to perform tasks over unseen test scenes , while only allowing ≤1000 steps and ≤10 action failures for each task at inference time . 4 AFFORDANCE-AWARE MULTIMODAL NEURAL SLAM . Affordance-aware navigation is a major challenge in solving complex and long-horizon indoor tasks such as ALFRED with both navigation and object interactions . Specifically , given each object of interest in the scene , the agent is required to not only find and approach it but also end up at a pose ( x , y , r , h ) , that is feasible for subsequent interactions with the object . For instance , to open a fridge , the robot should approach the fridge closely enough ( so the door is within reach ) , look at it ( so that the fridge is in the field of view ) , and leave enough room to open the door . To solve a long-horizon task involving multiple navigation and object interaction subgoals , it is natural to use an explicit semantic map , either 2D or 3D , of the environment ( similar to Neural Active SLAM Chaplot et al . ( 2020a ) ) , together with model-based planning ( e.g . as in HLSM Blukis et al . ( 2021 ) ) . This line of work tends to generalize better than models that directly learn mappings from human instructions to navigation & interaction actions ( e.g. , E.T . Pashevich et al . ( 2021 ) ) . With perfect knowledge of the environment , it is possible to achieve ( nearly ) perfect performance . In practice , however , the semantic map acquired at inference time is usually far from ideal , primarily due to Incompleteness ( missing information due to insufficient exploration of the scene ) and Inaccuracy ( erroneous object location prediction on the map , especially for small objects ) . To improve exploration performance , we propose a multimodal module that , at each step , predicts an exploration action a ∈ { MoveAhead , RotateLeft , RotateRight } by taking visual observations & actions in the past , step-by-step language instructions , and the explored area map which indicates where the agent has visited . We show that , compared to existing model-based approaches on ALFRED ( e.g. , HLSM Blukis et al . ( 2021 ) which applies random exploration ) , our use of lowlevel language instructions leads to more efficient exploration . The proposed exploration module operates at the subgoal level and only predicts exploration actions ( in contrast to E.T . which directly predicts actions for the entire task ) . The extra modality ( the explored area ) facilitates exploration by providing the agent with explicit spatial information . We illustrate the exploration module in Figure 3 , elaborate its details in Section 4.3 , and empirically demonstrate its advantages in Section 5 . To deal with the inaccuracy in map prediction , we carefully design an affordance-aware semantic representation for the environments . On one hand , knowing the precise spatial coordinates of objects requires precise depth information , which is difficult to acquire due to 3D sensor noise and/or inaccuracy in predicting depth from 2D images . On the other hand , affordance-aware navigation essentially asks for poses ( x , y , r , h ) of the agent suitable for interactions with the target objects , thus requiring only coarse-grained spatial information . Given an object type o , we define such corresponding poses as waypoints Wo and then treat navigation as a path planning problem among different waypoints . To generate such waypoints , we handle large objects ( fridges , cabinets , etc . ) and small objects ( apples , mug , etc . ) differently . The waypoints for large objects are computed using 2D grid maps predicted and aggregated from front-view camera images by a CNN-based network ; for small objects , we directly search over all observations acquired during the exploration phase with the help of a pre-trained Mask RCNN He et al . ( 2017 ) ( detailed below in Section 4.2 ) .

Summary: The paper proposes a method for solving the ALFRED task (following language instructions to perform a set of household tasks). The model has two main components: (1) Affordance-aware Semantic Representation (2) Multi-modal exploration. The former component estimates the location of the objects in the scene. The latter provides an exploration strategy by using instructions, images, previous actions and previously explored areas as input.