Shaping embodied agent behavior with activity-context priors from egocentric video
Authors: Tushar Nagarajan
Kristen Grauman
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: embodied AI
learning from passive videos
visual semantic planning
interaction exploration
object interaction
egocentric video
human activity understanding
computer vision

Abstract: Complex physical tasks entail a sequence of object interactions, each with its own preconditions -- which can be difficult for robotic agents to learn efficiently solely through their own experience. We introduce an approach to discover activity-context priors from in-the-wild egocentric video captured with human worn cameras. For a given object, an activity-context prior represents the set of other compatible objects that are required for activities to succeed (e.g., a knife and cutting board brought together with a tomato are conducive to cutting). We encode our video-based prior as an auxiliary reward function that encourages an agent to bring compatible objects together before attempting an interaction. In this way, our model translates everyday human experience into embodied agent skills. We demonstrate our idea using egocentric EPIC-Kitchens video of people performing unscripted kitchen activities to benefit virtual household robotic agents performing various complex tasks in AI2-iTHOR, significantly accelerating agent learning. 

Shaping embodied agent behavior with activity-context priors from egocentric video
Tushar Nagarajan UT Austin and Facebook Al Research tushar. annutexan edu
Kristen Grauman UT Austin and Facebook Al Research graumanOcs utexas. edu
Abstract
Complex physical tasks entail a sequence of object interactions, each with its own rrecondtionsswwiee can be difficult for robotic agents to learn efficiently solely through their own experience. We introduce an approach to discover activity- context priors from in-the-wild egocentric video captured with human worn cam- eras. For a given object, an activity-context prior represents the set of other compatible objects that required for activities succeed (e.g., knife and cutting board brought together with tomato are conducive to cutling). We encode our video-based prior as an auxiliary reward function that encourages an agent bring compatible objects together before attempting an interaction. In this way, our model translates everyday human experience into embodied agent skills. We demonstrate our idea using egocentric EPIC-Kitchens video of people performing unscripted kitchen activities benefit virtual household robotic agents performing various complex tasks in AI2-TTHOR, significantly accelerating agent learning. Project page: http://vists cs. utexas. edu/proj ects/ /ggo- rewards/
Introduction
Embodied Al agents that are capable moving around and interacting with objects human spaces promise important practical applications for home service robots, ranging from agents that can search for misplaced items, to agents that can cook entire meals. The pursuit of such agents has driven exciting new research in visual semantic planning [69], instruction following [59], and object rearrangement [3, 30], typically supported by advanced simulators [46, 33, 58, 22] where policies may be learned quickly and safely before potentially transferring to real robots. In such tasks, an agent aims to perform a sequence of actions that will transform the visual environment from an initial state to goal state. This in turn requires jointly learning behaviors for both navigation, move from one place to another, and object interaction, to manipulate objects and modify the environment (e.g.. pick-up objects, use tools and objects together, turn-on lights).
key challenge is that changing the state of the environment involves context-sensitive actions that depend on both the agent and environment's current state what the agent is holding, what other objects are present nearby, and what their propertics are. For example, to wash a plate, plate must be in the sink before the agent toggles-on the faucet; to slice an apple the agent must first be holding knife, and the apple must not already be sliced. Understanding these conditions is critical for efficient learning and planning: the agent must first bring the environment into the proper precondition state before attempting to perform a given activity with objects. We refer to these states as activity-contextt
Despite its importance, current approaches do not explicitly model the activity-context Pure rein- forcement learning (RL. approaches directly search for goal states without considering preconditions. This requires large number of trials for the agent to chance upon suitable configurations, and leads to poor sample efficiency or learning failures [69, 59]. Instead, most methods resort col- lecting expert demonstrations to train imitation learning (IL) agents and optionally finetune with
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Reward agents for bringing objects together enable activity
Watch human egocentric videos
Discover activity-context priors
Bottle
Sink
Stove-knob: +0.8
Faucet: +0.6
Lettuce: +0.1
Pan
Book
Slove
Fridge
Book: +0,01
Bottle: +0.04
We discover activity-contexts for objects directly from egocentric video of human activity. A given object's activity-contex goes beyond "what objects are found together" to capture the likelihood that each other object in the environment participates in activities involving it (i.e., "what objects logether enable action"). Right panel: Our approach guides agents bring compatible objects-objects with high likelihood together to enable activities. For example, bringing pan to the sink increases the value of faucet interactions, but bringing it to the table has little effect on interactions with book.
Figure 1: Main idea. Left and middle panel:
RL [69, 59, 31, 15]. While demonstrations may implicitly reveal activity-connextss, they are cumber- some to collect; they require expert teleoperation using specialized hardware (VR, MoCap), often in artificial lab settings [68, 49, 28], and need to be collected independently for each new task.
Rather than solicit demonstrations, we propose learn activity-context from real-world, egocentric video of people performing daily life activities. Humans understand activities from years expe- rience, and can effortlessly bring even a novel environment new, appropriate configurations for interaction-heavy tasks. Egocentric or "first-person" video recorded with a wearable camera puts actions and objects manipulated by person the forefront, offering an immediate window of this expertise in action in its natural habitat.
We present reinforcement learning approach that infers people's first-hand experience and transfers them to embodied agents to improve interaction policy learning. Specifically, we 1) train visual models detect how humans prepare their environment for activities from egocentric video, and 2) develop novel auxiliary reward function that encourages agents to seek out similar activity-context states. For example, by observing that people frequently carry pans to sinks (to clean them) or stoves (to cook their contents), our model rewards agents for prioritizing interactions with faucets or stove-knobs when pots or pans are nearby. As result, this incentivizes agents to transport relevant objects to compatible locations before attempting interactions, which accelerates learning. See Fig. 1.
conditions directly from
Importantly, our goal not direct imitation. Our insight is that while humans and embodied agents have very different action spaces and bodies, they operate in similar environments where the underlying conditions about what state the environment must be in before trying to modify strongly aligned. Our goal is thus to guide agent's exploratory interactions towards these potential progress-enabling states as it learns a new task. Moreover, because our training videos are passively collected by human
weererss and capture wide array of daily actions, they help build general visual prior for human activity-contexte while side-stepping the heavy requirements of collecting IL demonstrations for each individual task of interest.
Our experiments demonstrate the value of learning from egocentric videos from humans (in EPIC- Kitchens [13]) to train visual semantic planning agents (in A12-ITHOR [33]). Our video model relates objects based on goal-oriented actions (c.g., knives used to cut potatoes) rather than spatial co-occurrences (c.g., knives are found near spoons) [66, 11, 48, 65] semantic similarity (c.g.. potatoes are like tomatoes) [66]. Our agents outperform strong exploration methods and state-of-the. art embodied policies multi-step interaction tasks (e.g., storing cutlery, cleaning dishes). improving absolute success rates by up to 12% on the most complex tasks. Our approach learns policies faster, generalizes to unseen environments, and greatly improves success rates on difficult instances.
2 Related Work
Interaction in 3D environments Recent embodied AI work leverages simulated environments [46, 52, 58, 33, 22, 58] to build agents for interaction-heavy tasks like visual semantic planning [69]. interactive question answering [25], instruction following [59], and object rearrangement [3. 31].

---

Prior approaches use imitation learning (IL) based pre- tra-ningn to improve sample efficiency, but at the expense of collecting expert in-domain demonstrations individually for each task [69, 15, 59]. Instead, we leverage readily available human egocentric video in place of costly II. demonstrations, and we propose an RI. approach that benefits from the discovered human activity-priors.
Exploration for navigation and interaction Exploration strategies for visual navigation encour- age efficient policy learning by rewarding agents for covering area [12, 10, 17], visiting new states [51,4] 4], expanding the frontier [47], anticipating maps [50], or via intrinsic motivation [42, 6]. For grasping, prior work studies curiosity [27] and disagreement [43] as intrinsic motivation. Beyond navigation and grasping, interaction exploration [40] rewards agents for successfully attempting new object interactions (take potato, open fridge, etc.). In contrast, our agents are rewarded for achieving compatible activity-contexts learned from human egocentric video. Our model incentivizes action sequences that are aligned with activities (e.g.. putting a plate in the sink before turning-on the faucet), as opposed to arbitrary actions (e.g., turning the faucet on and then immediately off).
Learning from passive video for embodied agents Thus far, video plays a limited role in learning for embodied AI, primarily as a source of demonstrations for imitation learning. Prior work learns dynamics models for behavior cloning [57, 67, 44, 54, 53] or crafts reward functions that encourage visiting expert states [1. 16, 55, 36]. Beyond imitation, recent navigation work "re-labels" video frames with pseudo-action labels learn navigation subroutines [9] or oction-conditioned value functions [34]; however, they use videos generated from simulation or from intentionally recorded real-estate tours. contrast, our work the first to use free-form human- menaangeeeeraed video captured in the real world to learn priors for object interactions. Our priors are not tied to specific goals (as in behavior cloning) and are cast as general purpose auxiliary rewards to encourage efficient RL.
Learning about human actions from video Substantial work in computer vision explores models for human action recognition in video [70, 62, 8, 19J, including analyzing hand-object interactions [56, 2, 61, 7, 24] and egocentric video understanding [35, 21, 38, 32, 13, 26]. More closely related to our work, visual affordance models derived from video can detect likely places for actions to occur [39, 18, 41], such as where to grasp frying pan, and object saliency models can identify human-useable objects in egocentric video [14, 5, 20]. These methods learn important concepts from human video, but do not consider their use for embodied agent action, as we propose.
Semantic priors for embodied agents Human-centric environments contain useful semantic and structural regularities. Prior work exploits spatial relationships between objects [66, 11, 48] and room layouts [65] improve navigation (e.g., beds tend to be in bedrooms; kitchens tend to be near dining rooms). Recent work learns visual priors from agent experience (not human video) to understand environment traversability [47], affordances [40], or object properties like mass [37]. These priors encode static properties of objects or geometric relationships between them by learning from co-occurrences in static images. In contrast, our formulation encodes information about objects in action from video of humans using objects to answer what objects should be brought together to enable interactions, rather than what objects are rypically co-located. In addition, unlike previous models that require substantial online agent experience to learn priors, we use readily available egocentric video of human activity.
3 Approach
Our goal is to train agents to efficiently solve interaction-heave tasks. Training reinforcement learning (RL) agents is difficult due to sparse task rewards, large search spaces, and context-sensitive actions that depend reward derived from egocentric videos of daily human activity that encourages agents to configure the environment in ways that facilitate successful activities. For example, watching humans wash spoons in the sink suggests that the sink is good location to bring utensils to before interacting with the nearby faucet or dish soap.
what the agent is holding and where other objects are. We propose auxiliary
In the following, we begin by defining the visual semantic planning task (Sec. 3. Then, we show how to infer activity-context priors from egocentric video (Sec. 3.2) and how to translate those priors to an embodied agent in simulation (Sec. 3.3). We then describe our approach to encode the priors as dense reward (Sec. 3.4). Finally, we present our policy learning architecture (Sec. 3.5).
3

---

3.1 Visual semantic planning
We aim train an agent to bring the environment into a particular goal state specified by the task T, Agents can perform navigation actions Av (e.g., move forward, rotate left/right) and object interactions A (e.g., take/put, open, toggle) involving an object from set O.
Each task is set up as a partially observable Markov decision process. The agent is spawned at an initial state 8p- At each time step I, the agent in state 8, receives observation (2x,, 0r ha) containing the RGB egocentric view, the agent's current pose, and the currently held object (if any). The agent executes an action on an object or, (+) ~ {Av Arr and receives a reward re ~ (st, at, Ot, St41). For navigation actions, the interaction target is null. A recurrent network encodes the agent's observation history over time arrive at the state representation (detailed below).
A task reward is provided the agent brings its environment to a goal state gr. For example, in the "Clean object" task in experiments, a washable object must be inside the sink, and the faucet must be toggled-on for success. A positive reward is given for goal completion, while a small negative reward is given at each time-step to incentivize quick task completion:
10
if gr satisfied,
R (8t, Ge, Ot, 86-1)
-0001
otherwise.
The goal is to learn a policy n, that maximizes this reward over an episode of length T.
3.2 Human activity-context from egocentric video
Learning interaction policies requires understanding of how the environment needs to be configured- -where the agent needs to be, and what objects need to be present there-before goals completed. We infer this directly from passively collected dataset of human egocentric videos.
The dataset consists of a set of clips v E V where each clip captures a person performing an interaction with some object from video object vocabulary Oy (e.g., "slice tomato", "pour kettle"). Our model uses the clip's 's frames only, not the interaction label. Our approach first extracts activity-context from each training video frame, and then aggregates their statistics over all training clips to produce an inter-object uctivity-context compatibility function, as follows.
For each frame fe in an egocentric video clip v, we use an off-the-shelf hand-object interaction model [56] that detects active manipulleed objects in contact with hands hands-resulting of class-agnostic bounding boxes. We associate object labels to these boxes using pre-trained object instance detection model. Specifically, we transfer labels from high-confidence instance detections to active object boxes with large overlap, namely, where intersection over union (IoU) > 0.5, resulting
set
in set of active object boxes and corresponding class labels D(f:)= = {bb, 0g) (.)..(b, oN) oN)). where b; denotes box coordinates and o, E Ov. These instances represent objects that are directly involved in the activity, ignoring background objects that are incidentally visible but not interacted with by hands. Detection model details are in Supp.
We infer frame fi's activity-context AC(J)) from the Cartesian product of active objects.
AC(S:) = {(0;, 0i,0j E D(f+)) D(f(),,
(2)
where o * Oj. and each o; is object that can be held and moved to different locations, opposed to fixed object like a refrigerator or sink. We include a null object token to consider cases when the agent visits locations empty-handed Fig. 2 (left) shows examples.
Each (o,; 0y) pair represents a particular object o and corresponding activity-context object (ACO) on object that is used with it in activity. These include movable objects (e.g., tools like knives and cutting-boarde in "slice tomato"), receptacles (c.g., kettles and cups in "pour water"), and fixed environment locations (e.g., sinks, faucets in "wash spoon"). This is in contrast to an object's s affordance, which defines object properties in isolation (e.g. tomatoes are sliceable whether a knife is held or not, spoons are washable even when they are inside drawers).
Finally, we define the inter-object activity-context compatibility score sla,, 03) as follows:
Σ S,(o,, 0z)
0(0,,0;)= Σ Σ Sy(o,, 0x) VEV Au
BEV
(3)

---

Unitum
WardEmbed
SconsProot
Figure 2: Left: Detected ACOs from EPIC videos. White boxes: active object detections. Red/blue boxes: sampled (ubject, ACO) tuple (Equation 2). Last column shows failure cases cases-spong not held (top), liq- washwashing false positive (bottom). Right: ACO compatibiliity scores THOR. Red: active object. blue: top corresponding ACOs. Edge thickness indicates compatibility (Equation 3). Our approach (bottom right) prioritizes object relationships for "Pan" that are relevant for activity (e.g.. StoveBurner, StoveKnob) over semantically similar or spatially co-occurring objects (c.g., Pot, Shelf in top-right, bottom-left respectively).
where S, (o, 02) measures the fraction of frames of video v where (0,, o,)) across clips, we capture dominant object relationships and reduce the effect of object detection errors.
AC(f)). By aggregating
Recall that these relationships are derived from video frames of objects in action rather than arbitrary video frames or static images, and are thus strongly tied to human activity, not static scene properties. In other words, ((0; 0j) measures how likely is brought together with Oj during an activity (e.g.. knives and cutting boards are used to cut fruits) rather than how likely they generally co-occur in space (e.g., spoons and knives are stored together, but neither enables the action of the other).
3.3 Translating to activity-context for embodied agents
Next, we go from compatible object detections from human video, to interactible objects in embodied agents' environments. The compatibility score in Equation 3 is defined over the vocabulary of detectable objects Oy, which may not be the same as the interactible objects O in the agent's enviromment. This mismatch is to be expected since we rely on in-the-wil video that is not carefully curated for the particular environments or tasks faced by the agents we train. To account for this, we first map objects in video and those in the agent's 's domain to a common Glove [45] word embedding space. Then, we re-estimate p(0m 0₂) for each object pair O by considering nearest neighbors to Oy in this space. Specifically, in Equation 3, we set
So on))
Σ Σ
Om.DnnJ((o..o))
o.EN(om) (o.)
where ((o) the nearest neighbors of environment object o among video objects Oy within fixed distance in embedding space, and ot,y measures the dot product similarity the corresponding word embeddings of objects o, and oj. The resulting compatibility scores represent relationships between objects in the agent's environment, as inferred from similar objects in egocentric video. See Fig. (right) for an illustration of top ACOs compared to other heuristics (like word embedding similarity) and Supp. for further details.
3.4 Learning interact with activity-context reward
Next, cast the compatibility scores learned from video into auxiliary reward for embodied agents. In short, the agent rewarded for interactions with any objects at any time; however, this reward is enhanced when there are compatible ACOs nearby. For example, it is more valuable to turn-on the stove when there is a pan on it than if there was knife (or no other objects) nearby. To maximize this reward, the agent must intelligently move objects in the environment locations with compatible objects before performing interactions.
To achieve this, the agent maintains persistent memory to keep track of where objects were moved to, and what other objects near it are potential ACOs (and thus, how affects the value of nearby object interactions). The memory M starts empty for all objects. At each time-step.
5

---

occup, faucet]
c cM
PPO
π
olpot, store- wwwubbrrerr
*bolll,, lanoss]
GRU
Rr + Rs.
oeup,, bottle
Palicy network
Mion
M(a
# bottle, cup
ó(pot, stove knob)
faucett
RGB observation
ACO memory (.M)
P2 decup, feweet;
@/bottle, faweet"
(Eq. 5)
Figure 3: Policy training framework. Left panel: Our policy network generates an action (at, 0a) given the current state observation. If objects are moved by the agent, the ACO memory M is updated to add objects (red) their ACO's memory (blue) following Sec. 3.4. Right panel: An auxiliary reward Rş provided for object interactions based on nearby ACOs. For example, the interaction target oe the faucet, the agent is rewarded for having the cup and bottle nearby.
if an agent holding an object o puts-down the object at location p in 3D space', identify o's neighhoring objects. For each neighhoring object o', o is potential ACO and is added to the memory: M(o') U {(o,p) p) d(o, o') < c). We set 0.5m. Conversely, if an object
picked-up from location it is removed from the list of potential ACOs of all objects it originally neighbored: M(o') {(o.p) p)} Vo' M, and its own ACO memory cleared: (M
Φ. Fig. 3 (right) shows 1. Each object (in red) was placed at its respective locations in previous time-steps, and added to the ACO memories of nearby objects (in blue).
snapshot of this memory built until time-step
At time-step the agent is rewarded if successfully interacts with an object based on this memory. The total aetivity-conten reward RS is the sum of compatibility scores for objects in memory for which 0e is a candidate ACO:
ooo, o1) 0g E Ar A(a,, = d'EM(a))
Σ
Ro(st, at O1, M)
(5)
otherwise,
where c(ar, 0:) counts the number of times particular interaction occurs. Note that the currently held object (or null, if nothing is held) included in M for all interactions. See Supp. for pseudo-code of the memory update and reward allocation step.
Unlike the sparse task reward that offered only at the goal state (See. 3.1), the activity-context reward is provided at every time-step. These dense rewards are not task-specific, meaning we do not give any privileged information about which video priors are most beneficial to a given task т. However, they nonetheless encourage the agent to take meaningful actions towards activities, helping to reach states that are strongly aligned with common task subgoals human videos.
3.5 Interaction policy training
Putting it together, our training process is as follows. We adopt an actor-critic policy to train our agent [60]. At each time-step the current egocentric frame Hi encoded using ResNet-18 [29] encoder and then average pooled and fed to an LSTM based recurrent neural network (along with encodings of the held object class and previous action) aggregate observations over time, and finally to an actor-critic network (MLP) to generate the next action distribution and value. See Fig. 3 (left). In parallel, the agent maintains and updates its activity-context memory and generates an auxiliary reward based on nearby context objects at every time-step following Sec. 3.4.
The final reward is combination of the task reward (Sec. 3.1) and the activity-conte reward:
R( De Ot, S1+11 .M- Ry (se, dt, Ot, s14 + Ag ( (St, Ot, Or, M),
where A controls the contribution of the auxiliary reward term. We train our agents using DD- PPO [64] for 5M steps, with rollouts of T = 256 time steps. Our model and all baselines use visual
We calculate from the agent's pose and depth observations using inverse projection transformation on the interaction target point in the agent's current view.

---

encoders from agents that are pre-trained for interaction exploration [40] for 5M steps, which we find benefits all approaches. See Fig. and Supp. for architecture, hyperparameter and training details.
4
Experiments
We evaluate how well agents learn complex interaction tasks using our human video based reward.
Simulator and video datasets. To train policies, we use the AI2-THHRR [33] simulator where agents can navigate: Ay= = [move forward, turn left/right 90", look up/down 30"), J. and interact with objects: A (take, put, open, close, toggle-on, toggle-off, slice). Our action space of size ||A== 110 is the union of all navigation actions and valid object interactions following [63]. We use all 30 kitchen scenes from A12-TTHOR, split into training (25) and testing (5) sets. To learn activity-conter priors, we use all 55 hours of video from EPIC-Kitcher daily, unscripted kitchen actívities in variety of homes. It consists of -40k video clips annotated for interactions spanning 352 objects (Oy) and 125 actions. Note that we use clip boundaries to segment actions, but we do not use the action labels in our method.
[13], which contains egocentric videos of
Since kitchen scenes present diverse set of object interactions in multi-step cooking activities, they are of great interest in this research domain [13, 23, 40]. Further, the alignment of domain with A12-iTHOR's ' kitchen environments provides a path to transfer knowledge from videos to agents.
Visual semantic planning tasks. We consider seven tasks in our experiments where the agent must STORE: put an object that is outside into drawer, HEAT: turn on the stove with cooking receptacle on it, COOL: store food item or container inside the fridge, CLEAN: put an object the sink and turn on the faucet to wash it, SLICE: slice food item with knife, PREP: place food item inside a pot/pan, TRASH: throw an object into the trash bin,
Each task is associated with a goal state that the environment must be in. For example in the COOL task, an object that was originally outside, must be inside the fridge, and the fridge door must be closed. See Supp. for goal states for all tasks. These tasks represent realistic scenarios in home robot assistant settings, consistent with tasks studied in recent work [59, 40].
We generate 64 episodes per task and per environment with randomized object and agent positions evaluate our agents. We report task success rates (%) on unseen test environments. This means that our model must both generalize what it observes in the real- real-world human video to the agent's s egocentric views, as well as generalize from training environments novel test environments.
Baselines. We compare several methods:
VANILLA trains a policy using only the task reward (Sec. 3.1). This is the standard approach to train reinforcement learning agents.
SCenePRiorS [66] also uses only the task reward, but encodes spatial co-occurrences between objects using graph convolutional network (GCN) model enhance its state representation. We use the authors' code.
NAVEXP [50] adds an auxiliary navigation exploration reward to the vanilla model. We use object coverage [17, 50], which rewards the agent for visiting (but not interacting with) new objects. INTExp [40] a faate-of-the-art exploration method for object interaction that adds reward for successful object interactions regardless of nearby context objects. We use the authors' code.
SCENEPRIORS represents the conventional approach of introducing visual priors into policy learning- through GCN encoders, not auxiliary rewards-and and considers spatial co-occurrence instead of activity based priors. NAVEXP and INTEXP exploration methods from recent work that incentivize all locations or objects equally. In contrast, our reward is function of the state of the environment, and how well it aligns with observed states in video of human activity.
4.1 Visual semantic planning results
Table I shows success rates across all tasks. The numbers in brackets represent the probability that an agent executing random actions will reach the goal state assuming all objects are within reach. Interaction-heavv tasks requiring long sequences of steps like STORE (pick up object, open drawer, put object inside, close drawer) are significantly harder than interaction-ligee tasks like PREP (pick up food item, put in pan)-in this particular example, by four orders of magnitude.
Our approach outperforms prior work on all interaction-heavy tasks, and performs competitively on interaction- light tasks. Compared to the VANILLA baseline, SCENE PRIORS's stronger observation
7

---

7)
STORE(Ie 7)
H 6)
CLEANG2:
SLICE(ee3) 3)
PREP/Z-3)) 3)
TRASH 3)
VANILLA
0.12.2007
0.0020.00 00
2.01
0.3520.12
0.3022.00
0.22.0.05
0.1420.00
SCENEPRIORS [66]
0.14
0.00 .0
0.04
0.35
0.36++0.05
0. 0.26+0.08
0.20#0.06
NAVEXP [50]
0.05 20.04
0.01 10.01
4.01
0.43g0.02
4.294.0.0
1.332
0.25
INTEXP 44)
0.006
0.03+00.0
0.0620.08
0.1920.05
0.26
0.1920.08
0.0220.001
OURS
0.26=#0.06
0.12+0.07
0.13+0.05
0.53+0.08
0.36+0.06
0.26=0.07
0.13+0.02
Table 1: Task success rates (%) on test environments. Numbers in brackets indicate the probability that a
random agent performs the correct sequence of interactions to complcte task (e.g., COOL is much harder than
PREP). Our video-based activity-context priors result in the best performance across all interaction hhavy tasks. Navigation-based exploration agents perform well for tasks that involve minimal object interactions (e.g. PREP, TRASH). Values are averaged over training runs.
IntExp
Ours
128
New interaction
T-0
128
interaction
Supporting ACO
Figure 4: Policy rollouts for the COOL task during early/late stage training. Red dots represent auxiliary rewards (size represents value). Our agents learn to move objects to suitable locations (near supporting ACOs, blue crosses) to increase their reward, rather than simply visit more objects. See text.
encoding results better performance with the same task reward; however, while its spatial co- occurrence prior encoding what objects are found together (rather than used logether) is helpful for object search [66]. it falls short for object interactions.
INTEXP provides a task-agnostic objective for pre-training interaction agents; it results in strong visual encoder features, but yields poor performance when used as an auxiliary reward. fails to discriminate between useful interaction sequences that are aligned with task goals (e.g., put pan on stove, turn-on stove) and arbitrary interaction sequences that maximize its reward (e.g., pick up, then put down objects sequentially). NAvEXP performs poorly on interaction-hent tasks, often achieving close to zero success rates. However, performs well on tasks like TRASH and PREP where a single object must be moved to target location for success.
Fig. 4 shows qualitative results of 100 policy rollouts. Each panel shows side-by-side comparison of agent behavior during early (IM steps) and late (5M steps) training for the same task. Each red dot represents an auxiliary reward given to the agent, sized by reward value; the yellow star denotes where the task reward is issued. INTEXP (left panel) is rewarded uniformly for each interaction (equally sized red dots), and thus has single strategy to maximize reward - perform more interactions. In contrast, our approach (right panel) can maximize reward along two axes: perform more interactions as before, or selectively move objects to favorable positions near ACOs (blue crosses). This guides agent exploration to relevant states for activities, translating to higher task success.
4.2 Training speed and task success versus difficulty
Fig. 5 shows consolidated results across all tasks, treating each episode of each task an individual instance that can be successful or not. These results give a sense of overall performance of each baseline, despite variation across tasks. See Supp. for a task-specific breakdown of results.
Fig. 5 (left) shows convergence speed. Our approach sees an increased success rate at early training epochs as dense rewards allow agents to make progress towards goals without explicit task rewards. NAVEXP performs poorly in early iterations as it prioritizes navigation actions to visit as many objects possible. Overall this result shows a key ad advantage of our idea: learning from human video not only boosts overall policy success rates, but it also accelerates the agent's 's learning process, giving "head start" from having observed how people perform general daily interactions. Importantly, our ablations (below) show that access to the video alone is not sufficient; our idea to capture functional activity-contexts is key.
Fig. (right) analyzes task success as a function of navigation difficulty. We measure this as the ideal geodesic distance to each of the objects required to reach the goal state. This value is low when
8

---

0.25
Vanilla
SceneFriors (66)
Improvement over ba maseline (%)
150
ScenePriors [66]
NavExp [50]
NavExp [50]
IntExp
IntExp [40]
100
Ours
Ours
Success
0.15
50-
0.10
0.05
0-
0.00
-50
Difficulty
Iterations (x10f)
Figure 5: Consolidated performance across all seven tasks. Left: Success rate vs. training iteration. Our dense activity-contexl rewards accelerate performance early training iterations. Right: Success rate improvement navigation difficulty. Episodes sorted and grouped ideal navigation distance task objects (low high). Our method shows largest improvements over VANILLA on difficult instances.
Cool.
STORE
HEAT
CLEAN
SLICE
PREP
TRASH
UNIFORM
0.07=0.00
0.02+0.01
0.12+0.00
0.37
0.26.0.03
0.22=0.05
0.02
WORDEMBED
0.19=0.03
0.02
0.03
0.40
0.34
0.22=0.01
0.04
SPATIALCOOC
0.33=0.05
0.02
0.06+40.0
0.4740.09
0.33
0.25=0008
0.13
OURS (INTSEQ)
0.33=0.04
+0003
0.07 20.03
0.43440.0
0.30
0.20-0.02
0.09
OURS (ACO)
0.26=2.06
0.12+0.07
0.13+0.05
0.53+20.03
0.36
0.26=n.07
0.13
Table Compatibility score ablations. Simple similarity measures (WORDEMBED) or naively using interac- tion labels in video (INTSEQ) produces sub-optimal policies. Our activity detection based priors perform the best across the majority of tasks. Results are averaged over runs.
objects are close by in small environments, and large when the agent has to move around to find specific objects in large environments. We sort episodes by this criterion into groups of increasing difficulty and compare each method against the VANILLA baseline policy. Our method has the highest success rates across difficulties and offers the largest improvements on more difficult instances.
4.3 Compatibility function ablations
Finally we investigate how different sources of priors impact our agents when incorporated into our reward scheme. Specifically, vary how we compute compatibility score 6(0;, 0j) in Equation 3. UNIFORM assigns equal compatibility to all object pairs; WORDEMBED uses Glove [45] embedding similarity; SPATIAL.COOe
spatial co-occurrence statisties derived from static images?; OURS (INTSEQ) uses transition probabilities between object interactions from ground truth, labeled sequences the same egocentric videos.
Table 2 shows the results, Uniform or semantic similarity heuristics are insufficient to capture the role of objects participating in activities. SPATIALCooC' object co-occurrences aligned for some tasks (e.g., vegetables are co-located with fridges in static images, and activities involving storing groceries), but also capture unhelpful static relationships (see Fig. 2, right). INTSEQ benefits from egocentric video; however, uses ground truth video object labels, and falsely assumes that the next object interacted with in time is necessarily relevant to the previous object. Our activity-context score is tightly linked to each object's involvement in interactions, and proves to be the most useful prior.
Conclusion
We proposed an approach to discover activity-context priors from real-world human egocentric video and use them accelerate training of interaction agents. Our work provides avenue for embodied agents to directly benefit from human experience even when the tasks and environments differ. Future work could explore policies that update human-activity prior beliefs during task-specific policy training, and policy architectures that encode such priors jointly in the state representation.
Acknowledgments: Thanks to Santhosh Ramakrishnan for helpful feedback and the UT Systems Admin team for their continued support. UT Austin is supported in part by DARPA L2M, Facebook Reality Labs, and the UT Austin NSF AT Institute. K.G. G. is paid Research Scientist by Facebook AT.
This different from SCENEPRIORS (Sec. 4.1) which encodes the priors in the state representation.

---

References
[1] Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In NewrlPS, 2018. 3
[2] Sven Bambach, Stefan Lee, David Crandall, and Chen Yu. Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions. In /CCV, 2015.
[3] Dhruv Batra, Angel X Chang. Sonia Chernova, Andrew J Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, et al. Rearrangement: A challenge for embodied ai. arXiv preprint 0Xiv:2011.01973. 2020. 1.2
[4] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In NeurlPS, 2016. 3
[5] Gedas Bertasius, Hyun Soo Park, Stella X. Yu, and Jianbo Shi. First person action object detection with egonet. In RSS, 2017. 3
[6] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. ICIR, 2019. 3
Zhe Cao. Iliaa Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-ohject interactions in the wild, arXiv preprint arXiv:20/2.09856, 2020. 3
[8] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset.
In CVPR, 2017. 3
[9] Matthew Chang, Arjun Gupta, and Saurabh Gupta. Semantic visual navigation by watching youtube videos.
NeurlPS, 2020.
[10] Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov.
Learning to explore using active neural slam. In ICLR, 2020. 3
[11] Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov. Object
goal navigation using goal- goal-oriented semantic exploration. NeurlPS, 2020. 2, 3
[12] Tao Chen, Saurabh Gupta, and Abhinav Gupta. Learning exploration policies for navigation. In ICLR,
2019. 3
[13] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti. Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018. License available at: https: //g/tttttt gith........ www/ /pic- kitchens/ spic-kitchens- 55- www--aanttaiiinn wwwllonn/llloo /ttp///ooomaatttttt/ee/ LLINEEE txt. 3, 3,7
[14] Dima Damen, Teesid Leelasawassuk, Osian Haines, Andrew Calway, and Walterio W Mayol-Cuevas. Cuevas. You-do, i-learn: Discovering task relevant objects and their modes of interaction from multi-user egocentric
video. In BMVC, 2014. 3
[15] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra, Embodied question answering. In ICCV-W, 2018. 2,3
[16] Dehidatta Dwibedi, Jonathan Tompson, Corey Lynch, and Pierre Sermanet. Learning actionable represen- tations from visual observations. In IROS, 2018. 3
[17] Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents in long-horizon horizon tasks. CVPR, 2019. 7.4
[18] Kuan Fang. Te-Lin Wu, Daniel Yang, Silvio Savarese, and Joseph Lim. Demo2vec: Reasoning object
affordances from online videos. In CVPR, 2018. 3
[19] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video
recognition. In ICCV, 2019. 3
[20] A. Furnari, S. Battiato, K. Grauman, and G. Maria Farinella. Next-active-objecct prediction from egocentric
videos. JVCIR, 2017. 3
[21] Antonino Furnari and Giovanni Maria Farinella. What would you expect? anticipating egocentric actions with rolling-umi unrooliig Istms and modality attention. /CCV, 2019. 3
[22] Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf. James Tracr, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, et al. Threedworld: A platform for interactive multi- modal physical simulation. arXiv preprint orXiv:2007.00 .04954, 2020. 1,2
[23] Z. Gao, R. Gong, T. Shu, X. Xie, S. Wang, and S. C. Zhu. Vrkitchen: an interactive 3d virtual environment for task-oriented learning. arXiv: 0tti://93..000 050500557, 2019. 7
[24] Guillermo
Shanxin Yuan, Seungryul Back, and Tae-Kyun Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In CVPR, 2018. 3
[25] Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi.
Iqa: Visual question answering in interactive environments. In CVPR, 2018. 2
[26] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhong- cong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane. Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer. Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang. Yifei Huang. Wengi Jia, Weslie
10

---

Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li. Zhengiang Li. Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland. Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Yunyi Zhu, Pablo Arbelacz, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg. Yoichi Sato, Jianho Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik, Ego4D: Around the World 3,000 Hours of Egocentric Video. CoRR, abs/21 10.07058, 2021. 3
[27] Nick Haher, Damian Mrowca, Stephanie Wang. Li F Fei-Fei, and Daniel L Yamins. Learning to play with intrnsically-motivatedd, motivated, aware agents. In NewrIPS, 2018. [28] Ankur Handa, Karl Van Wyk, Wei Yang. Jacky Liang. Wei Chao, Qian Wan, Stan Birchfield. Nathan Ratliff, and Dieter Fox. Dexpilot: Vision based teleoperation of dexterous robotic hand-arm system. ICRA,
2020. 2
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 6
[30] Unnat Jain, Luca Weihs, Eric Kolve, Ali Farhadi, Svetlana Lazebnik, Aniruddha Kembhavi, and Alexander Schwing, A cordial syne: Going heyond marginal policies for multi-agent emhodied tasks. In ECCV, 2020.
[31] Unnat Jain, Luca Weihs, Eric Kolve, Mohammad Rastegari, Svetlana Lazebnik, Ali Farhadi, Alexander Schwing. and Aniruddha Kembhavi. Two body problem: Collaborative visual task completion. In CVPR, 2019. 2,
[32] Hao Jiang and Kristen Grauman. Seeing invisible poses: Estimating 3d body pose from egocentric video.
In CVPR, 2017. 3
[33] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Farhadi. Ai2-thor: An interactive 3d environment for visual arXiv preprint arXiv: 1.XXiv:711..0 1712.05444 2017. License available at: httpe://githii com/ aal/aaalaaaaaaa///aaaaaaaaaaaaaa aaimaa///
blob/main/LICENT
1, 2, 7,4
[34] Ashish Kumar, Saurahh Gupta, and Jitendra Malik. Learning navigation subroutines from egocentric
videos. CoRI., 2020. 3
[35] Yin Li, Miao Liu, and James M Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. In ECCV, 2018. 3
[36] YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In ICRA, 2018. 3
[37] Martin Lohmann, Jordi Salvador, Aniruddha Kembhavi, and Roozbeh Mottaghi, Learning about objects by learning to interact with them. NeurIPS, 2020. 3
[38] Zheng Lu and Kristen Grauman. Story- summarization for egocentric video. In CVPR, 2013. 3
[39]
Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman. Grounded human-object interaction
hotspots from video. In ICCV, 2019. 3
[40] Tushar Nagarajan and Kristen Grauman. Learning affordance landscapes for interaction exploration in 3d
environments. NeurlPS, 2020. 3, 8,
[41] Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman. Ego-topo: Environment affordances from egocentric video. In CVPR, 2020.
[42] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell, Curiosity-driven exploration by self-supervised prediction. In ICML, 2017.
[43] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In
ICML, 2019. 3
[44] Deepak Pathak. Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. CVPR-W W.
2018. 3
[45] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014. 5, 3,
[46] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang. Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In CVPR, 2018. 1,2
[47] William Qi, Ravi Teja Mullapudi, Saurabh Gupta, and Deva Ramanan. Learning to move with affordance
maps. ICLRR 2020. 3
[48] Y. Qiu, A. Pal, and H. I. Christensen. Learning hierarchical relationships for object-goal navigation. In CoRL, 2020. 2.3
[49] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. RSS, 2018.
[50] Santhosh K. Ramakrishnan, Dinesh Jayaraman, and Kristen Grauman, An exploration of embodied visual exploration. IJCV, 2021. 3, 3,7,8 4,

---

[51]
Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent. Mare Pollefeys. Timothy Lillicrap.
and Sylvain Gelly. Episodic curiosity through reachability. ICLR, 2019.
[52] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In ICCV, 2019. 2,4
[53] Karl Schmeckpeper, Oleh Rybkin, Kostas Danillidis, Sergey Levine, and Chelsea Finn. Reinforcement
learning with videos: Combining offline observations with interaction. CoRL, 2020. 3
[54] Karl Schmeckpeper. Annie Xie, Oleh Rybkin, Stephen Tian, Kostas Danilidiis. Sergey Levine, and Chelsea Finn. Learning predictive models from observation and interaction. ECCV, 2020. 3
[55] Pierre Sermanet. Corey Lynch, Jasmine Hsu, and Sergey Levine. Time-contrastive networks: Self- supervised learning from multi-view observation. CVPR-W. W, 2017. [56] Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Foubey. Understanding human hands in contact at internet scale. In CVPR, 2020. 3,4
[57] Pratyusha Sharma, Lekha Mohan, Lerrel Pinto, and Abhinav Gupta. Multiple interactions made easy
(mime): Large scale demonstrations data for imitation. In CoRL, 2018. 3
[58] Bokui Shen, Fei Xia, Chengshu Li, Roberto Martín,,artin, Linxi Fan, Guanzhi Wang, Shyamal Buch, Claudia D' Arpino,, Sanjana Srivastava, Lyne P Tehapmi. et al. igibson, a simulation environment for interactive tasks in large realistiescenes. arXiv preprini arXiv:2012.0295
2020. 1,2
[59] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In CVPR, 2020. 1, 3.7
[60] Richard Sutton and Andrew Barto. Reinforcement learning: An introduction. MIT press, 2018.
[61] Bugra Tekin, Federica Bogo, and Marc Pollefeys. H+ o: Unified egocentric recognition of 3d hand-object poses and interactions. In CVPR, 2019. 3
[62]
Limin Wang, Yuanjun Xiong, 7he Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool, Temporal
segment networks: Towards good practices for deep action recognition. In ECCV, 2016. 3
[63] Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrangement. In
CVPR, 2021. 7
[64] Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint
arXiv:191.000355 2019. 6
[65] Yi Wu, Yuxin Wu, Aviv Tamar, Stuart Russell, Georgia Gkioxari, and Yuandong Tian. Learning and planning with a semantic model. arXiv preprint arXiv:1809. 10842, 2018. 2.
[66] Wei Yang. Xiaolong Wang. Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation
using scene priors. ICIRR, 2019. 3, 7, 8,4
[67]
Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine.
One-shot imitation from observing humans via domain-adaptive meta-learning. RSS, 2018. 3
[68] Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep
imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA, 2018.
[69] Yuke Zhu, Daniel Gordon, Eric Kolve, Dieter Fox, Li Fei-Fei, Abhinav Gupta, Roozbeh Mottaghi, and Ali Farhadi. Visual semantic planning using deep successor representations. In ICCV, 2017. 1, 2,
[70] Yi Zhu, Xinyu Li, Chunhui Liu, Mohammadreza Zolfaghari, Yuanjun Xiong, Chongruo Wu, Zhi Zhang, Joseph Tighe, R Manmatha, and Mu Li. A comprehensive study of deep video action recognition. arXiv preprint X0v::0020.0666666 2020. 3
12


Reviews:
[{"review_id": "iBhfKGm6k91", "reviewer": "Reviewer_htnr", "summary": "The paper presented a novel method that extracts object priors from egocentric videos to guide the learning of a robotic agent to interact with objects. The key idea is to model the presence of objects and their co-occurrence from naturalistic activities in first person videos (so called activity context), which can be further used as an auxiliary reward in RL for learning to perform complex object manipulation tasks. The proposed method was demonstrated in a simulated robotic learning environment, with promising results across several tasks. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nTeaching a robotic agent by learning from first person visual experience is a quite exciting idea. It is great to see the first successful demonstration of such an idea. \n\nThe design of activity context using statistics of object presence and co-occurrence is smart, avoiding the potentially complicated modeling of human body motion. The idea of using object prior as auxiliary reward in RL is interesting, though the RL part is quite standard. \n\nOverall, this is a well executed paper with a great idea. \n\n**Quality**\n\nThe proposed method is technically sound. The experiments are solid and the results are promising. I appreciate the video demo included in the supplement.\n\nMy only concern is that all results are reported under simulation. It will be quite interesting to see some demonstrations in a real world setting. This is less of a concern but more of a hope. I do believe that the ability to demonstrate this idea in the real world could be quite impactful. \n\n**Clarity**\n\nThe paper is well written and easy to follow. The key idea is clearly illustrated with nice visuals across the paper. \n\nThe authors claimed that “the model did not use the interaction / action label” in L148. Yet L242 suggests that the experiment considered 40K clips cropped from the video, where the starting and ending time of these clips are given by the action annotations. I think this part should be clarified.\n\n**Significance**\n\nTeaching robotic agents using real world videos is a critical problem in computer vision and robotics. The success of this paradigm can potentially have a major impact in robotic learning. \n\n**Post-rebuttal Update**\n\nThanks for the authors' response. After reading other reviews and responses, I remain quite positive about this paper. Thus, I stand by my previous rating and recommend to accept this paper. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "N4fDtJFVIxJ", "reviewer": "Reviewer_HGQh", "summary": "This paper introduces an approach to discover activity-context priors, that is, for a given object, the environment states that are preconditions for attempting given activities with objects. Such priors are acquired from in-the-wild egocentric videos that are captured from a camera that is worn by people performing daily activities. A visual model is trained to detect how humans prepare their environment for their activities from egocentric videos. Video-based priors are then used as auxiliary rewards to encourage agents to seek out similar activity-context states. The proposed approach accelerates learning and generalises to unseen environments.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper presents a method that is based on the insight that while humans and embodied agents have very different action spaces and bodies, they operate in similar environments. Hence, instead of tackling the gap and transfer between embodiments, the goal of the authors here is to guide agents' interactions towards states that are \"demonstrated\" by humans and that enables meaningful object interactions. The tasks and methods introduced have limited novelty, and the work is a novel and interesting combination of well-known techniques and ideas. It is clear how this work differs from previous contributions, and related work is adequately cited.\n\nQuality: The submission is technically sound and claims are supported by theoretical and experimental analysis. The proposed method and the alternative methods used as comparison are used in an appropriate way. Comparisons with other existing methods are analysed across different experiments and tests, which completes the evaluation of the proposed method. Limitations of the proposed method are also discussed, and failure cases are presented.\n\nClarity: The paper is clear and well organised. Details about the implementation and algorithms used are provided, so that enough information is available to reproduce the results.\n\nSignificance: The results presented are significant and interesting, the ideas at the basis of this work are valuable and can be used by other researchers in the field to improve performance on complex tasks that entail sequences of object interactions. The idea of guiding agents' interactions towards states that enable meaningful object interactions from human demonstration is not completely novel, but its realisation in this work is neat. \n\nOther comments:\nWould the learned visual model work on real robot/real environments as well? Can you elaborate on the simulation vs. reality gap?\nVideos are a great addition to the submission as they make it easier to appreciate the behavior of the proposed method w.r.t. others.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed limitations and edge cases. The work does not have societal impact is such.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "LpSqScWLFQZ", "reviewer": "Reviewer_dhvp", "summary": "In this paper, the authors train a vision model to learn an activity context from unlabeled video, use this activity context to derive a reward augmentation strategy, and train a model-free algorithm (PPO) to bring compatible objects (matching the distribution of objects scene during training) together.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Originality\nThe paper's novel contribution is threefold. The authors propose a novel method for learning an embedding of activity priors for video, the authors use these priors to create a reward augmentation strategy, and the authors show that when trained with this strategy, pre-existing model-free algorithms are able to outperform SOTA on interaction heavy tasks.  Learning an embedding of activity priors is not, in itself new [1]. As noted by the author's memory-based methods for reinforcement learning, are also, not in themselves, novel [2]. The authors' primary contribution is reformulating the representation of the memory in terms of activity context, and deriving the reward augmentation from this representation.\n\n* Quality\nThe paper shows strong results against four competing algorithms (NavExp, IntExp, ScenePriors). The results use both real-world image data and a realistic simulation environment. The figures are clear, and support the author’s central claim about the utility of activity context priors in driving the object aggregation behavior.\n\n* Clarity\nThe paper is clear, the authors describe the formulation of the activity context and the adaption of the activity context to the reward. The description of how to generate and store the activity context, and how this is then used by the agent policy is also clear.\n\nFigure 4. was a bit hard to parse, the contrast of the auxiliary reward against the background made it difficult to interpret, and the blue lines (which may represent the link of the supporting activity context to a given auxiliary reward) were not explained in the caption.\n\n* Significance\nVideo datasets are abundant in comparison to supervised datasets of agents performing home activities. Developing methods that can learn useful priors from unlabeled videos would be an extreme boon to the embodied learning community. This work presents an early step towards that goal, as it does a dataset with a comparatively narrow distribution of objects and features (kitchen) Furthermore, this work presents a novel reward augmentation strategy derived from the learned object-centric priors.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are a few areas that if addressed may increase the score of my review.\n\nThe paper could benefit from internal ablations, as it is unclear the contributions of each component of the model to overall performance. I would be interested to see how replacing individual components of the model (ResNet18 backbone, LSTM aggregator) contribute to the overall model performance.\n\nOne limitation is that although the activity-context model derives from unlabeled video data, the model itself relies heavily on priors learned from a dataset that includes a large (100K) subset of labeled data [2]. It would be beneficial to see if an action recognition model trained explicitly on the Epic Kitchens dataset only showed strong generalization.\n\nOn the topic of generalization, the authors make the claim that the method generalizes to unseen environments. I think it’s important to note the distinction that these activity context priors all share a common underlying distribution (drawn from common household kitchen objects). One relevant ablation would be to alter the distribution of objects to preserve object categories, but such that the distribution no longer matched the distribution of a kitchen.\n\nThe authors do not directly address societal impact in the paper. As noted, the Epic Kitchens Dataset is anonymous and publicly available. It may be the case that as data-driven approaches to learning contexts on unsupervised data evolve, ethical concerns will arise about the nature of entrained biases.\n", "review_text": "", "extra_scores": {"ethical_concerns": "Possibly with respect to biases in the video data, not sure.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "H7EsL3dxSqH", "reviewer": "Reviewer_zAh2", "summary": "The paper introduces activity-context priors learned from real-world egocentric video frames, which are used as an auxiliary reward to encourage RL agents to solve longer-term object interaction tasks. Instead of directly imitating human videos, the prior is computed automatically using statistics from an off-the-shelf object recognition model run on the EPIC-Kitchens dataset. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Although there are other methods which try to compute exploration rewards based on object interaction and novelty, the authors clearly explain how their method is novel. Compared to prior methods, this method specifically tries to encode not just which objects occur near each other, but which objects are actually used by the agent to interact together, which can reduce the impact of spurious correspondences. The experimental comparison is very thorough and the methods which the authors choose to compare to are very relevant. \n\nQuality: The evaluation tasks are very challenging for RL and the method makes significant improvements on these difficult multi-step tasks.\n\nThe compatibility score ablations are thorough and insightful. The method is able to accelerate early training performance, when the algorithm still needs to rely on the dense provided rewards for exploration.\n\nClarity: The paper is very well written. Code will be released at a later date, but the appendix provides sufficient details for reproducibility at the moment. Addressing the following points may make things even better:\n\nIt took me a bit to figure out whether $\\phi(o_i, o_j) == \\phi(o_j, o_i)$, but I see that it is not. It could be helpful to mention this explicitly.\n\nHow sensitive is the method to the auxiliary reward scaling hyperparameter $\\lambda_\\phi$? \n\nThe intro states “Specifically, we 1) train visual models to detect how humans prepare their environment for activities from egocentric video”, however, maybe this is not quite precise -- the scores seem to model which objects occur in activities together, which is not solely preparation? \n\nThe discussion of how the memory $\\mathcal{M}$ is maintained is rather complex and difficult to understand as written. It could be helpful to give a more general description of what the memory is keeping track of and when rewards are given before introducing details.\n\nSignificance: The proposed method can use existing egocentric real-world video to learn the priors, and does not need any additional demonstration data. This is impactful in that it may allow methods to better use existing large-scale data to improve the learning of RL algorithms.\n\nHowever, the performance on tasks with minimal object interactions, while competitive, is not quite as strong as prior work. This is rather surprising as it seems like the proposed method should prioritize object interactions which solve the task and still perform better than navigation-based methods which weigh all object interactions equally. Is there an explanation for why performance is not as strong on those tasks?\n\nOther comments:\n\nFigure 4 is challenging to parse -- it is difficult to see what the correct sequence of object movements to solve the task should be (and what we would like the corresponding red dots to look like).\n\nFigure 2 is great!\n\nRecommendation:\n\nThe idea of learning priors specifically to encode activities rather than encouraging any object interactions seems like a promising direction. Comparisons are made to state of the art methods and the method performs well on the most difficult tasks. The paper is well written, the work is positioned and explained well compared to relevant prior work, and the idea is presented clearly. The main negative I see is the performance on the simpler (less object interaction) tasks. So, I think this paper meets the threshold and recommend acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do well in discussing limitations of their method, including the ACO detection errors and cases where simpler methods seem to perform better. One significant limitation is that the method relies on a hand-object interaction detection model, which restricts the possible sources of data which it can use for learning (other datasets than epic kitchens). ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers an interesting problem of aiming to use egocentric human videos as a prior for guiding the behavior of an embodied agent completing tasks in AI2-iTHOR virtual environment. All of the reviews are positive about the paper, and the author response also helped address several points of feedback from the reviewers. The authors are strongly encouraged to incorporate the feedback and the changes mentioned in the author response into the revised paper.
Decision: Accept (Spotlight)