Shaping embodied agent behavior with activity-context priors from egocentric video

1 Introduction . Embodied AI agents that are capable of moving around and interacting with objects in human spaces promise important practical applications for home service robots , ranging from agents that can search for misplaced items , to agents that can cook entire meals . The pursuit of such agents has driven exciting new research in visual semantic planning [ 69 ] , instruction following [ 59 ] , and object rearrangement [ 3 , 30 ] , typically supported by advanced simulators [ 46 , 33 , 58 , 22 ] where policies may be learned quickly and safely before potentially transferring to real robots . In such tasks , an agent aims to perform a sequence of actions that will transform the visual environment from an initial state to a goal state . This in turn requires jointly learning behaviors for both navigation , to move from one place to another , and object interaction , to manipulate objects and modify the environment ( e.g. , pick-up objects , use tools and objects together , turn-on lights ) . A key challenge is that changing the state of the environment involves context-sensitive actions that depend on both the agent and environment ’ s current state—what the agent is holding , what other objects are present nearby , and what their properties are . For example , to wash a plate , a plate must be in the sink before the agent toggles-on the faucet ; to slice an apple the agent must first be holding a knife , and the apple must not already be sliced . Understanding these conditions is critical for efficient learning and planning : the agent must first bring the environment into the proper precondition state before attempting to perform a given activity with objects . We refer to these states as activity-contexts . Despite its importance , current approaches do not explicitly model the activity-context . Pure reinforcement learning ( RL ) approaches directly search for goal states without considering preconditions . This requires a large number of trials for the agent to chance upon suitable configurations , and leads to poor sample efficiency or learning failures [ 69 , 59 ] . Instead , most methods resort to collecting expert demonstrations to train imitation learning ( IL ) agents and optionally finetune with 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . RL [ 69 , 59 , 31 , 15 ] . While demonstrations may implicitly reveal activity-contexts , they are cumbersome to collect ; they require expert teleoperation using specialized hardware ( VR , MoCap ) , often in artificial lab settings [ 68 , 49 , 28 ] , and need to be collected independently for each new task . Rather than solicit demonstrations , we propose to learn activity-contexts from real-world , egocentric video of people performing daily life activities . Humans understand activities from years of experience , and can effortlessly bring even a novel environment to new , appropriate configurations for interaction-heavy tasks . Egocentric or “ first-person '' video recorded with a wearable camera puts actions and objects manipulated by a person at the forefront , offering an immediate window of this expertise in action in its natural habitat . We present a reinforcement learning approach that infers activity-context conditions directly from people ’ s first-hand experience and transfers them to embodied agents to improve interaction policy learning . Specifically , we 1 ) train visual models to detect how humans prepare their environment for activities from egocentric video , and 2 ) develop a novel auxiliary reward function that encourages agents to seek out similar activity-context states . For example , by observing that people frequently carry pans to sinks ( to clean them ) or stoves ( to cook their contents ) , our model rewards agents for prioritizing interactions with faucets or stove-knobs when pots or pans are nearby . As a result , this incentivizes agents to transport relevant objects to compatible locations before attempting interactions , which accelerates learning . See Fig . 1 . Importantly , our goal is not direct imitation . Our insight is that while humans and embodied agents have very different action spaces and bodies , they operate in similar environments where the underlying conditions about what state the environment must be in before trying to modify it are strongly aligned . Our goal is thus to guide an agent ’ s exploratory interactions towards these potential progress-enabling states as it learns a new task . Moreover , because our training videos are passively collected by human camera-wearers and capture a wide array of daily actions , they help build a general visual prior for human activity-contexts , while side-stepping the heavy requirements of collecting IL demonstrations for each individual task of interest . Our experiments demonstrate the value of learning from egocentric videos from humans ( in EPICKitchens [ 13 ] ) to train visual semantic planning agents ( in AI2-iTHOR [ 33 ] ) . Our video model relates objects based on goal-oriented actions ( e.g. , knives used to cut potatoes ) rather than spatial co-occurrences ( e.g. , knives are found near spoons ) [ 66 , 11 , 48 , 65 ] or semantic similarity ( e.g. , potatoes are like tomatoes ) [ 66 ] . Our agents outperform strong exploration methods and state-of-theart embodied policies on multi-step interaction tasks ( e.g. , storing cutlery , cleaning dishes ) , improving absolute success rates by up to 12 % on the most complex tasks . Our approach learns policies faster , generalizes to unseen environments , and greatly improves success rates on difficult instances . 2 Related Work . Interaction in 3D environments Recent embodied AI work leverages simulated environments [ 46 , 52 , 58 , 33 , 22 , 58 ] to build agents for interaction-heavy tasks like visual semantic planning [ 69 ] , interactive question answering [ 25 ] , instruction following [ 59 ] , and object rearrangement [ 3 , 31 ] . Prior approaches use imitation learning ( IL ) based pre-training to improve sample efficiency , but at the expense of collecting expert in-domain demonstrations individually for each task [ 69 , 15 , 31 , 59 ] . Instead , we leverage readily available human egocentric video in place of costly IL demonstrations , and we propose an RL approach that benefits from the discovered human activity-priors . Exploration for navigation and interaction Exploration strategies for visual navigation encourage efficient policy learning by rewarding agents for covering area [ 12 , 10 , 17 ] , visiting new states [ 51 , 4 ] , expanding the frontier [ 47 ] , anticipating maps [ 50 ] , or via intrinsic motivation [ 42 , 6 ] . For grasping , prior work studies curiosity [ 27 ] and disagreement [ 43 ] as intrinsic motivation . Beyond navigation and grasping , interaction exploration [ 40 ] rewards agents for successfully attempting new object interactions ( take potato , open fridge , etc. ) . In contrast , our agents are rewarded for achieving compatible activity-contexts learned from human egocentric video . Our model incentivizes action sequences that are aligned with activities ( e.g. , putting a plate in the sink before turning-on the faucet ) , as opposed to arbitrary actions ( e.g. , turning the faucet on and then immediately off ) . Learning from passive video for embodied agents Thus far , video plays a limited role in learning for embodied AI , primarily as a source of demonstrations for imitation learning . Prior work learns dynamics models for behavior cloning [ 57 , 67 , 44 , 54 , 53 ] or crafts reward functions that encourage visiting expert states [ 1 , 16 , 55 , 36 ] . Beyond imitation , recent navigation work “ re-labels '' video frames with pseudo-action labels to learn navigation subroutines [ 9 ] or action-conditioned value functions [ 34 ] ; however , they use videos generated from simulation or from intentionally recorded real-estate tours . In contrast , our work is the first to use free-form human-generated video captured in the real world to learn priors for object interactions . Our priors are not tied to specific goals ( as in behavior cloning ) and are cast as general purpose auxiliary rewards to encourage efficient RL . Learning about human actions from video Substantial work in computer vision explores models for human action recognition in video [ 70 , 62 , 8 , 19 ] , including analyzing hand-object interactions [ 56 , 2 , 61 , 7 , 24 ] and egocentric video understanding [ 35 , 21 , 38 , 32 , 13 , 26 ] . More closely related to our work , visual affordance models derived from video can detect likely places for actions to occur [ 39 , 18 , 41 ] , such as where to grasp a frying pan , and object saliency models can identify human-useable objects in egocentric video [ 14 , 5 , 20 ] . These methods learn important concepts from human video , but do not consider their use for embodied agent action , as we propose . Semantic priors for embodied agents Human-centric environments contain useful semantic and structural regularities . Prior work exploits spatial relationships between objects [ 66 , 11 , 48 ] and room layouts [ 65 ] to improve navigation ( e.g. , beds tend to be in bedrooms ; kitchens tend to be near dining rooms ) . Recent work learns visual priors from agent experience ( not human video ) to understand environment traversability [ 47 ] , affordances [ 40 ] , or object properties like mass [ 37 ] . These priors encode static properties of objects or geometric relationships between them by learning from co-occurrences in static images . In contrast , our formulation encodes information about objects in action from video of humans using objects to answer what objects should be brought together to enable interactions , rather than what objects are typically co-located . In addition , unlike previous models that require substantial online agent experience to learn priors , we use readily available egocentric video of human activity . 3 Approach . Our goal is to train agents to efficiently solve interaction-heavy tasks . Training reinforcement learning ( RL ) agents is difficult due to sparse task rewards , large search spaces , and context-sensitive actions that depend on what the agent is holding and where other objects are . We propose an auxiliary reward derived from egocentric videos of daily human activity that encourages agents to configure the environment in ways that facilitate successful activities . For example , watching humans wash spoons in the sink suggests that the sink is a good location to bring utensils to before interacting with the nearby faucet or dish soap . In the following , we begin by defining the visual semantic planning task ( Sec . 3.1 ) . Then , we show how to infer activity-context priors from egocentric video ( Sec . 3.2 ) and how to translate those priors to an embodied agent in simulation ( Sec . 3.3 ) . We then describe our approach to encode the priors as a dense reward ( Sec . 3.4 ) . Finally , we present our policy learning architecture ( Sec . 3.5 ) . 3.1 Visual semantic planning . We aim to train an agent to bring the environment into a particular goal state specified by the task τ . Agents can perform navigation actions AN ( e.g. , move forward , rotate left/right ) and object interactions AI ( e.g. , take/put , open , toggle ) involving an object from set O . Each task is set up as a partially observable Markov decision process . The agent is spawned at an initial state s0 . At each time step t , the agent in state st receives an observation ( xt , θt , ht ) containing the RGB egocentric view , the agent ’ s current pose , and the currently held object ( if any ) . The agent executes an action on an object ot , ( at , ot ) ∼ { AN ⋃ AI } , and receives a reward rt ∼ Rτ ( st , at , ot , st+1 ) . For navigation actions , the interaction target ot is null . A recurrent network encodes the agent ’ s observation history over time to arrive at the state representation ( detailed below ) . A task reward is provided if the agent brings its environment to a goal state gτ . For example , in the “ Clean object '' task in our experiments , a washable object must be inside the sink , and the faucet must be toggled-on for success . A positive reward is given for goal completion , while a small negative reward is given at each time-step to incentivize quick task completion : Rτ ( st , at , ot , st+1 ) = { 10 if gτ satisfied , −0.01 otherwise . ( 1 ) The goal is to learn a policy πτ that maximizes this reward over an episode of length T .

Summary: The paper presented a novel method that extracts object priors from egocentric videos to guide the learning of a robotic agent to interact with objects. The key idea is to model the presence of objects and their co-occurrence from naturalistic activities in first person videos (so called activity context), which can be further used as an auxiliary reward in RL for learning to perform complex object manipulation tasks. The proposed method was demonstrated in a simulated robotic learning environment, with promising results across several tasks.