Shaping embodied agent behavior with activity-context priors from egocentric video

1 Introduction . Embodied AI agents that are capable of moving around and interacting with objects in human spaces promise important practical applications for home service robots , ranging from agents that can search for misplaced items , to agents that can cook entire meals . The pursuit of such agents has driven exciting new research in visual semantic planning [ 69 ] , instruction following [ 59 ] , and object rearrangement [ 3 , 30 ] , typically supported by advanced simulators [ 46 , 33 , 58 , 22 ] where policies may be learned quickly and safely before potentially transferring to real robots . In such tasks , an agent aims to perform a sequence of actions that will transform the visual environment from an initial state to a goal state . This in turn requires jointly learning behaviors for both navigation , to move from one place to another , and object interaction , to manipulate objects and modify the environment ( e.g. , pick-up objects , use tools and objects together , turn-on lights ) . A key challenge is that changing the state of the environment involves context-sensitive actions that depend on both the agent and environment ’ s current state—what the agent is holding , what other objects are present nearby , and what their properties are . For example , to wash a plate , a plate must be in the sink before the agent toggles-on the faucet ; to slice an apple the agent must first be holding a knife , and the apple must not already be sliced . Understanding these conditions is critical for efficient learning and planning : the agent must first bring the environment into the proper precondition state before attempting to perform a given activity with objects . We refer to these states as activity-contexts . Despite its importance , current approaches do not explicitly model the activity-context . Pure reinforcement learning ( RL ) approaches directly search for goal states without considering preconditions . This requires a large number of trials for the agent to chance upon suitable configurations , and leads to poor sample efficiency or learning failures [ 69 , 59 ] . Instead , most methods resort to collecting expert demonstrations to train imitation learning ( IL ) agents and optionally finetune with 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . RL [ 69 , 59 , 31 , 15 ] . While demonstrations may implicitly reveal activity-contexts , they are cumbersome to collect ; they require expert teleoperation using specialized hardware ( VR , MoCap ) , often in artificial lab settings [ 68 , 49 , 28 ] , and need to be collected independently for each new task . Rather than solicit demonstrations , we propose to learn activity-contexts from real-world , egocentric video of people performing daily life activities . Humans understand activities from years of experience , and can effortlessly bring even a novel environment to new , appropriate configurations for interaction-heavy tasks . Egocentric or “ first-person '' video recorded with a wearable camera puts actions and objects manipulated by a person at the forefront , offering an immediate window of this expertise in action in its natural habitat . We present a reinforcement learning approach that infers activity-context conditions directly from people ’ s first-hand experience and transfers them to embodied agents to improve interaction policy learning . Specifically , we 1 ) train visual models to detect how humans prepare their environment for activities from egocentric video , and 2 ) develop a novel auxiliary reward function that encourages agents to seek out similar activity-context states . For example , by observing that people frequently carry pans to sinks ( to clean them ) or stoves ( to cook their contents ) , our model rewards agents for prioritizing interactions with faucets or stove-knobs when pots or pans are nearby . As a result , this incentivizes agents to transport relevant objects to compatible locations before attempting interactions , which accelerates learning . See Fig . 1 . Importantly , our goal is not direct imitation . Our insight is that while humans and embodied agents have very different action spaces and bodies , they operate in similar environments where the underlying conditions about what state the environment must be in before trying to modify it are strongly aligned . Our goal is thus to guide an agent ’ s exploratory interactions towards these potential progress-enabling states as it learns a new task . Moreover , because our training videos are passively collected by human camera-wearers and capture a wide array of daily actions , they help build a general visual prior for human activity-contexts , while side-stepping the heavy requirements of collecting IL demonstrations for each individual task of interest . Our experiments demonstrate the value of learning from egocentric videos from humans ( in EPICKitchens [ 13 ] ) to train visual semantic planning agents ( in AI2-iTHOR [ 33 ] ) . Our video model relates objects based on goal-oriented actions ( e.g. , knives used to cut potatoes ) rather than spatial co-occurrences ( e.g. , knives are found near spoons ) [ 66 , 11 , 48 , 65 ] or semantic similarity ( e.g. , potatoes are like tomatoes ) [ 66 ] . Our agents outperform strong exploration methods and state-of-theart embodied policies on multi-step interaction tasks ( e.g. , storing cutlery , cleaning dishes ) , improving absolute success rates by up to 12 % on the most complex tasks . Our approach learns policies faster , generalizes to unseen environments , and greatly improves success rates on difficult instances . 2 Related Work . Interaction in 3D environments Recent embodied AI work leverages simulated environments [ 46 , 52 , 58 , 33 , 22 , 58 ] to build agents for interaction-heavy tasks like visual semantic planning [ 69 ] , interactive question answering [ 25 ] , instruction following [ 59 ] , and object rearrangement [ 3 , 31 ] . Prior approaches use imitation learning ( IL ) based pre-training to improve sample efficiency , but at the expense of collecting expert in-domain demonstrations individually for each task [ 69 , 15 , 31 , 59 ] . Instead , we leverage readily available human egocentric video in place of costly IL demonstrations , and we propose an RL approach that benefits from the discovered human activity-priors . Exploration for navigation and interaction Exploration strategies for visual navigation encourage efficient policy learning by rewarding agents for covering area [ 12 , 10 , 17 ] , visiting new states [ 51 , 4 ] , expanding the frontier [ 47 ] , anticipating maps [ 50 ] , or via intrinsic motivation [ 42 , 6 ] . For grasping , prior work studies curiosity [ 27 ] and disagreement [ 43 ] as intrinsic motivation . Beyond navigation and grasping , interaction exploration [ 40 ] rewards agents for successfully attempting new object interactions ( take potato , open fridge , etc. ) . In contrast , our agents are rewarded for achieving compatible activity-contexts learned from human egocentric video . Our model incentivizes action sequences that are aligned with activities ( e.g. , putting a plate in the sink before turning-on the faucet ) , as opposed to arbitrary actions ( e.g. , turning the faucet on and then immediately off ) . Learning from passive video for embodied agents Thus far , video plays a limited role in learning for embodied AI , primarily as a source of demonstrations for imitation learning . Prior work learns dynamics models for behavior cloning [ 57 , 67 , 44 , 54 , 53 ] or crafts reward functions that encourage visiting expert states [ 1 , 16 , 55 , 36 ] . Beyond imitation , recent navigation work “ re-labels '' video frames with pseudo-action labels to learn navigation subroutines [ 9 ] or action-conditioned value functions [ 34 ] ; however , they use videos generated from simulation or from intentionally recorded real-estate tours . In contrast , our work is the first to use free-form human-generated video captured in the real world to learn priors for object interactions . Our priors are not tied to specific goals ( as in behavior cloning ) and are cast as general purpose auxiliary rewards to encourage efficient RL . Learning about human actions from video Substantial work in computer vision explores models for human action recognition in video [ 70 , 62 , 8 , 19 ] , including analyzing hand-object interactions [ 56 , 2 , 61 , 7 , 24 ] and egocentric video understanding [ 35 , 21 , 38 , 32 , 13 , 26 ] . More closely related to our work , visual affordance models derived from video can detect likely places for actions to occur [ 39 , 18 , 41 ] , such as where to grasp a frying pan , and object saliency models can identify human-useable objects in egocentric video [ 14 , 5 , 20 ] . These methods learn important concepts from human video , but do not consider their use for embodied agent action , as we propose . Semantic priors for embodied agents Human-centric environments contain useful semantic and structural regularities . Prior work exploits spatial relationships between objects [ 66 , 11 , 48 ] and room layouts [ 65 ] to improve navigation ( e.g. , beds tend to be in bedrooms ; kitchens tend to be near dining rooms ) . Recent work learns visual priors from agent experience ( not human video ) to understand environment traversability [ 47 ] , affordances [ 40 ] , or object properties like mass [ 37 ] . These priors encode static properties of objects or geometric relationships between them by learning from co-occurrences in static images . In contrast , our formulation encodes information about objects in action from video of humans using objects to answer what objects should be brought together to enable interactions , rather than what objects are typically co-located . In addition , unlike previous models that require substantial online agent experience to learn priors , we use readily available egocentric video of human activity . 3 Approach . Our goal is to train agents to efficiently solve interaction-heavy tasks . Training reinforcement learning ( RL ) agents is difficult due to sparse task rewards , large search spaces , and context-sensitive actions that depend on what the agent is holding and where other objects are . We propose an auxiliary reward derived from egocentric videos of daily human activity that encourages agents to configure the environment in ways that facilitate successful activities . For example , watching humans wash spoons in the sink suggests that the sink is a good location to bring utensils to before interacting with the nearby faucet or dish soap . In the following , we begin by defining the visual semantic planning task ( Sec . 3.1 ) . Then , we show how to infer activity-context priors from egocentric video ( Sec . 3.2 ) and how to translate those priors to an embodied agent in simulation ( Sec . 3.3 ) . We then describe our approach to encode the priors as a dense reward ( Sec . 3.4 ) . Finally , we present our policy learning architecture ( Sec . 3.5 ) . 3.1 Visual semantic planning . We aim to train an agent to bring the environment into a particular goal state specified by the task τ . Agents can perform navigation actions AN ( e.g. , move forward , rotate left/right ) and object interactions AI ( e.g. , take/put , open , toggle ) involving an object from set O . Each task is set up as a partially observable Markov decision process . The agent is spawned at an initial state s0 . At each time step t , the agent in state st receives an observation ( xt , θt , ht ) containing the RGB egocentric view , the agent ’ s current pose , and the currently held object ( if any ) . The agent executes an action on an object ot , ( at , ot ) ∼ { AN ⋃ AI } , and receives a reward rt ∼ Rτ ( st , at , ot , st+1 ) . For navigation actions , the interaction target ot is null . A recurrent network encodes the agent ’ s observation history over time to arrive at the state representation ( detailed below ) . A task reward is provided if the agent brings its environment to a goal state gτ . For example , in the “ Clean object '' task in our experiments , a washable object must be inside the sink , and the faucet must be toggled-on for success . A positive reward is given for goal completion , while a small negative reward is given at each time-step to incentivize quick task completion : Rτ ( st , at , ot , st+1 ) = { 10 if gτ satisfied , −0.01 otherwise . ( 1 ) The goal is to learn a policy πτ that maximizes this reward over an episode of length T .

Summary: This paper introduces an approach to discover activity-context priors, that is, for a given object, the environment states that are preconditions for attempting given activities with objects. Such priors are acquired from in-the-wild egocentric videos that are captured from a camera that is worn by people performing daily activities. A visual model is trained to detect how humans prepare their environment for their activities from egocentric videos. Video-based priors are then used as auxiliary rewards to encourage agents to seek out similar activity-context states. The proposed approach accelerates learning and generalises to unseen environments.