EgoEnv: Human-centric environment representations from egocentric video
Authors: Tushar Nagarajan
Santhosh Kumar Ramakrishnan
Ruta Desai
James Hillis
Kristen Grauman
Venue: NeurIPS 2023 oral (2023)
Keywords: egocentric video
3D environment
sim2real
sim-to-real
episodic memory
TL;DR: We learn "environment-aware" ego-video representations that encode not just a short 1-2s clip, but also the local surroundings of the camera-wearer (e.g., what objects are nearby, how far are they).

Abstract: First-person video highlights a camera-wearer's activities in the context of their persistent environment. However, current video understanding approaches reason over visual features from short video clips that are detached from the underlying physical space and  capture only what is immediately visible. To facilitate human-centric environment understanding, we present an approach that links egocentric video and the environment by learning representations that are predictive of the camera-wearer's (potentially unseen) local surroundings. We train such models using videos from agents in simulated 3D environments where the environment is fully observable, and test them on human-captured real-world videos from unseen environments. On two human-centric video tasks, we show that models equipped with our environment-aware features consistently outperform their counterparts with traditional clip features. Moreover, despite being trained exclusively on simulated videos, our approach successfully handles real-world videos from HouseTours and Ego4D, and achieves state-of-the-art results on the Ego4D NLQ challenge.

EgoEnv: Human-centric environment representations from egocentric video
Tushar Nagarajan?, Santhosh Kumar Ramakrishnan', Ruta Desai James Hillis Kristen Grauman University of Texas at Austin, "FAIR, Meta
Abstract
First-person video highlights a camera- wearer's activities in the context of their persistent environment. However, current video understanding approaches reason over visual features from short video clips that are detached from the underlying physical space and capture only what is immediately visible. To facilitate human- centric environment understanding, we present an approach that links egocentric video and the environment by learning representations that predictive of the camera-wess cumera-wearer's 's (potentially unseen) local surroundings. We train such models using videos from agents in simulated 3D environments where the environment is fully observable, and test them on human-captured real-world videos from unseen environments. On human-centric video tasks, we show that models equipped with our environment-awarr features consistently outperform their counterparts with traditional clip features. Moreover, despite being trained exclusively simu- lated videos, our approach successfully handles real-world videos from House and Ego4D, and achieves state-of-the-art results on the Ego4D NLQ challenge. Project page: https: //vvvviiinn//////// cs. utexas. edu/ edu/projects pttp:/pteoeteoeee/e//e///
Introduction
Egocentric video offers a unique view into human activities through the eyes of a camera-wearer:
Understanding this type of video is core building augmented reality (AR) applications that provide context-relevant assistance to humans based on their activity. Ego-video is thus the subject of several recent datasets and benchmarks that driving new research [47, 75, 13, 26].
key feature of the egocentric setting is the tight coupling of a camera- wearer and their persistent physical environment, i.c., person's 's mental model of their surroundings informs their actions. This mental model is important, for example, to reach for a cabinct door out of view, to re- visit the couch to search for misplaced phone to visit spaces configured to support certain activitics. This raises an important need for numan-centrin environment understanding to learn representations from video that capture the camera-wearer's activities in the context of their enviroument. Such representations would encode the human-enve cnmaanenvirrommnt link, and allow models to jointly reason about both (e.g.. to answer "what did the person cut near the sink?"). See Fig.
Despite its importance, there been only limited work on learning human-centric environment representations. Current video models segment video into short clips (1-2s long) and then aggregate clip features over time (c.g., with recurrent, graph or transformer- based networks) for tasks like action forecasting [24, 26, 57, 25], temporal action localization [95, 50, 51, 96], episodic memory [26, 15), and movie understanding [90]. Critically, the clip features encode what is immediately visible in short time window, and their aggregation over time does not equate linking them in physical space. Other approaches use explicit camera pose information (e.g., from SLAM) to localize the camera- wearer, but not its relation surrounding objects (e.g., forecasting [58, 68, 27]) to group activities by location, but do not learn representations for agent video (e.g., affordance prediction [67, 57]).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Ego-video
EgoEnv feature
Environment-wware pretraining
Figure 1: Main idea. Video models trained on short egocentric clips capture narrow, instantaneous view of human activity (e.g., cutting an onion at the counter) without considering the broader context to which the activity is tied (e.g.. the pan to the left to cook the onions, the fridge further behind to store leftovers). We propose ground video in its underlying 3D environment by learning representations that predictive of their surroundings, thereby enhancing standard clip-based models with complementary environment information.
To address these shortcomings, we propose to learn environment- video representations that encode the surrounding physical space. Specifically, we define the local environment state at each time-step of an egocentric video the set of objects (and their rough distance) in front, to the left, right, and behind the camera- wearer. See Fig. 2. We use this state as supervision to train a transformer-based video encoder model that aggregates visual information across video to build an enviroument memory, which can be queried to predict the local state at any point in the video.
The local state captures the rough layout of objects relative to the camera-weareh It is important for understanding physical space - provides a semantic signal to localize people (e.g.. in the as well as human behavior,
living room, from the arrangement of couches, lamps and tables) since people move towards layouts that support activities (c.g., stove-top areas, dressers). Predicting the local state thus involves capturing the natural statistics of object layouts across different homes and then translating contexts across environments to reason about new ones. Once trained, given observation from a video, model produces drop-in EgoEnv feature which encodes enviroument information to complement the action information in existing video clip features.
An important practical question is how to supervise such a representation. Sourcing local state labels requires agent and object positions and omnidirectional visibility at each time step. This is challenging as egocentric videos only offer sparse coverage of the environment. Furthermore, they are prone to object detection, tracking, and SLAM failures. Therefore, for training we turn to videos generated by agents in simulation. This allows to sample diverse, large-scale trajectories to cover the environment, while also providing ground-trutt trutth local state. Once trained with simulated video, apply our models to real-world videos from new, unseen environments.
We demonstrate our EgoEnv approach on two video tasks where joint reasoning of both human action and the underlying physical space is required: (1) inferring the room category that the camera wearer physically in as they move through their environment, and (2) localizing the answer to a natural language query in an egocentric video. These tasks support many potential applications, including AR systems that can offer context-relevant assistance.
We are the first to demonstrate the value of 3D simulation data for real-world ego-video understanding. Our experiments show that by capitalizing on both geometric and semantic cues in our proposed "local enviromment state" task, we can leverage video walkthroughs from simulated agents in HM3D scenes [64] ultimately enable downstream human-centric environment models on real-world videos from House Tours [7] and Ego4D [26]. Furthermore, models equipped with our EgoEnv features outperform both popular scene classification [97] and natural language video localization models [95], achieves state-of of-the-art results on the Ego4D NLQ challenge leaderboard.
2
Related work
Video understanding in 3D environments. Prior work encodes short video clips [4, 85, 60, 33] or temporally aggregates them for additional context [89, 24, 95, 50, 51, 96, 90]. However, these methods treat the video a temporal sequence and fail to capture the spatial context from the underlying persistent environment. For egocentric video, prior work has used structure from motion (SIM) to map people and objects for trajectory [58] and activity forecasting [27] and action grounding

---

in 3D [67, 14]. These approaches localize the camera-wearge but do not learn representations for the camera- weerer'ss surroundings. The model of [52] associates features to voxel maps to localize actions; however, they require pre-computed 3D scan of the environment at both training and inference. Prior work groups clips by rough spatial location as topological graphs [57] or activity threads [61], but they stop short of learning representations using these groups. In contrast, we explicitly learn features that relate clips based on their spatial layout, for each step of an ego-video.
Video representation learning. Traditional video understanding methods learn representations by training models on large, manually curated video action recognition datasets [4, 54, 26]. Recent self-supervised learning (SSL) approaches eliminate the supervision requirement by leveraging implicit temporal signals [88, 55, 81, 40, 31, 84, 86, 87, 66, 23]. In contrast, we learn features that encode the local spatial-state of the environment (as opposed to temporal signals). Further, we show how to leverage state information that is readily accessible in simulation, but not in video datasets (i.e., locations and semantic classes of objects surrounding the agent) for training.
Environment features for embodied AI. In embodied AI, pose-estimates are used to build maps [11, 8, 62], edge features in graphs [9, 7]. as spatial embeddings for episodic memo- ries [20], to project features to a grid map [28, 36, 5] or to learn environment features for visual navigation [65]. However, these approaches are explored solely in simulation and typically require accurate pose-estimates or smooth action spaces, and thus are not directly applicable to egocentric videos. Research on world-models [18, 29, 30] and unseen panorama reconstruction [41, 76, 44] hallucinate the effect of agent actions to aid decision decision-making in simulation. In contrast, we aim to leamn environment features for egocentric camera-wearer aid real-worle video understanding.
Learning from simulated data. Prior work has proposed cost-effective ways to generate large- scale synthetic image datasets for various vision tasks [77, 79, 16, 37, 19, 70J. In robotics, simulation environments have been developed to quickly and safely train policies, with the eventual goal of transferring them to real world applications [43, 80, 45, 6, 91, 72, 64, 78]. The resulting sim-to-real problem, where models must adapt to changes in simulator and real- domains, is an active area of research for robot navigation [79, 42, 71, 3, 1]. However, simulated data for video understanding is much less explored. Prior work has synthesized data for human body pose estimation [12, 82, 93, 17], trajectory forecasting [48], and action recognition [69]. Rather than model human behavior, our approach is the first to directly capture the environment surrounding the camera-wearer for real-world video understanding tasks.
3 Approach
Our goal is to learn EgoEnv representations that encode the local surroundings of the camera- wearer. Such representation would implicitly maintain a semantic memory of surrounding objects beyond what is immediately visible, and coupled with standard video feature, would allow models to jointly reason about activities and the underlying physical space. Directly appending the camera pose with each video frame may capture the local state; however noise in pose estimated from ego-video with quick head motions and characteristic blur limits its utility (see Supp. for experiments).
Instead, we introduce an approach that leverages simulated environments where perfect state infor- mation is available to train models that can link visual information to the physical surroundings. To this end, we first define the local state prediction task in simulation (Sec. 3). Next, we introduce our transformer-batt based architecture that predicts the local state videos (Sec. 3.2). Finally, we show how our model trained in simulation generates environment features for real-world egocentric video frames (Sec. 3.3).
3.1 Local environment state
We require a model that is aware of not just what is immediately visible in a single frame, but also of the camera-wearee camera-west wearer as the set of objects in each relative direction - i.e., what objects are to the front, left, right or behind the camera-wearer, together with their rough distance from the camera-wearer and train model to predict this state. Our definition of local state takes inspiration from cognitive science [32, 46], and offers supervision signals that are both geometric (relative object locations) and semantic (semantic object labels), which we observe leads to strong representations.
surroundings. We therefore define the local environment state of the
3

---

Local state labels
Front (couch very near)
Right (cabinet - far)
Back (couch near, plant far)
Left (book near, couch near,
piclure - far)
Video trajectory
Camera-wearer
Obstacles
Far away hidden objects
Seen vs. unseen locations
Figure 2: Local environment state task. Left panel: Top-down environment view showing the camera-wcarer path (blue gradient linc) and nearby objects (colored boxes). Middle panel: Egocentric view in each direction. Only the forward view (top left) is observed. Remaining views shown for clarity. Task: Given egocentric video of person their environment, front, to the left, right, and behind the camera-wearer at each time-step (right panel). The model only sees the forward ego-view (middle panel, top left) and does not have access to the top-down map. Note that not all parts of the environment observations based on their shared space, as well anticipate unseen surroundings based on statistics of training environments. Best viewed in color.
model must predict set of objects (and their rough distance)
during walkthrough (white vs. grey regions map) models must link
More formally, let O be of object classes. For frame f from video trajectory in an environment, the local state is a tuple (Yo, Yr). yo a x |O] dimensional matrix which represents instances of each object class in the four cardinal directions relative the namera-weares y, is a matrix of the same size containing the distance of the objects in from the camera-wearer, discretized into 5 distance ranges between 0.25 5.0m!. For direction and object class j, the labels
{
if ((Pa.P,) 5 ( = otherwise,
(1)
yr d(pa, P.) if yo 1,
where Pa- Pj are the poses of the camera- wearer and object o, respectively, d(pa.P)) Py) is the euclidean distance between them, 8 distance threshold for nearby objects (we set 8 = 5.0m, beyond which visible objects are small), P(p..P,) E [0...3] is the discretized angle of the object relative to the agent's heading (forward, right, behind, left), and d(pa, E [0...4] is discretized distance. See Fig. 2 and Supp. for empirical analysis of related alternatives, e.g., predicting just object presence or image features.
We then train model to predict the local state of the target video frame, conditioned on the video trajectory. Once trained, the model can relate what is visible frame to the camera-weare''s possibly hidden surroundings produce environment-avwe
features.
Since supervision for camera-wearer pose and every object's location non-trivial for egocentric videos, where camera localization and tracking is error prone, we leverage videos in simulated environments for training, as presented next.
3.2 Environment-awarr pretraining in simulation
To source local state labels, we generate a dataset of video walkthroughs of agents in simulated 3D environments where agent and object poses are accessible all times (see Sec. 4). We train our model in two stages described below.
3.2.1 Pose embedding learning
While ground truth camera pose is available from the simulator training time, model trained to rely on will fail on real-world egocentric video at test time, where pose estimates are noisy. With the goal of handling arbitrary indoor egocentric video, we instead explore representations that implicitly encode coarse pose information. See Supp. for pose-related experiments.
For object classes with multiple instances, select the nearest

---

walkthrough (simulation)
Input video dip
Cip fecture di
Env feature h.
Environment momory
Local state prediction
Cip encodar
P
di
P
(At- (PK--)
9L
Front: table ownn, shell
Right: bench
Reletive pose
predicton
Query frome
-Lefft: potled plant t 'e
Behind: (rwe).
Video (real world)
Pre-trained environment encoder decoder
Figure Model framework. Left: Our model first learns pose embeddings by predicting discretized relative pose between observations from simulated video walkthroughs (Sec. 3.2.1). Center: Next, it encodes observa- tions and their pose embeddings into an environment memory that is trained to predict the local environment state for query frame (Sec. 3.2.2). Right: Once trained, model builds and queries the environment memory for any time-point of interest in a real-world video, to generate an environment feature downstream video tasks in disjoint and novel scenes (Sec. 3.3). @ concatenation.
Specifically, for a sequence of RGB frames {1+)=1 and camera poses {0,) we generate pose embeddings {pe) P(5,,.... (r) using a transformer encoder network. These embeddings used to predict the relative pose between each observation pair using bilinear layer
O.g p Vop + WT (w) pp)) + +pp
where d.g is the predicted relative pose and Vq W bp trainable parameters. We discretize the relative pose into 12 angles and 4 distance ranges to provide an approximate yet robust pose estimate. The network is trained to minimize cross-entropy between the predicted and the target relative pose labels for all observation pairs Σx, encode information to help relate video observations based their location and orientation in the environment. Note that once trained, pose embeddings are inferred directly from video frame sequences- ground truth pose is only required for training.
Bt,)). The trained pose embeddings pt
3.2.2 Local state pretraining
Next, we train a model to embed visual information from a video walkthrough into an environment memory, which can then be queried infer the local state corresponding given video frame from the same video. We implement this model as a transformer encoder-decoder model.
Specifically, for a video walkthrough V with RGB frames {fe)T= and query frame fg. we predict the local state M4 (yo, yr) follows. First, pose embeddings {p, ) are generated for the video and query frames following Sec. 3.2.1. Then, each frame encoded jointly with the pose embedding using a linear transform Mp.
xt = Mp((ee))))
Next, we uniformly sample K video frames to construct an environment memory using a transformer encoder ε, which updates frame representations using self-attention:
{EI,..., cK] = [(,,,..,,K))
(5)
The resulting memory represents features for each time-step that contain propagated information from all other time-steps. Compared to prior work [24, 95, 50, 51, 96], our encoder has the ability to relate observations based on not just visual characteristics and their temporal ordering, but also their relative spatial layout in the environment. A transformer decoder 'D then attends over the memory using query Zy to produce the output EgoEnv representation hn:
hy D(. ;zg);
which is finally used to predict the local state using two linear classifiers M. and M, for object class and distance predictions respectively. The network is trained to minimize the combination losses over the predicted and the target state labels for each direction:
L(hg. No. vr) Loce (Mo (hy), vo) + ALce (M, (ha), vr),
5

---

Dining room
Eathroom
Bedrcom
Hallway
Staircase
When did I Visit the piano then overlosked the kilchen?
Bodroom
Uliliyy room
Living foom
Kitchen
Did the dishwasher open?
Places 365
Room prediction
Episodic memory (NLQ)
Figure 4: Scene understanding in third-person photos vs. human-centric environment understanding. Left: Well-framed, canonical images from Places365 are considerably different from scene content observed in egocentric video. Center and right: Real egocentric video streams from HouseTTours (top) and Ego4D (bottom) illustrating the value in modeling the underlying environment, rather than just what is visible in short clips. For example, the person does not explicitly look at the staircase while walking down it (center, top row), the spatial relation between the person, piano, and kitchen is important answer the question (right, top row).
where Lice and Lee refer to binary cross-entropy and cross-entropy losses respectively. We set = 0.1 which balances the contributions of each loss function based validation experiments. The distance loss is computed only for objects that are in the local state (yo- 1). See Fig. 3 (left) and Supp. for more architecture details.
Learning to predict the local state involves aggregating information about observed objects across time, as well anticipating unseen objects based on learned priors from the layout of objects in other scenes (e.g.. TVs are usually in front of couches; kitchens have particular arrangements of sinks, refrigerators, and stove-tops).
Once trained, given a video in a new environment and a time- tine-point of interest, our model constructs an environment memory, predicting the local state based on information aggregated throughout the test video. Importantly, h, - the EgoEnv feature - contains valuable information about the camera- wearer's surroundings, offering environment features to complement traditional video features (e.g.. for a person watching TV, also encode the couch they are sitting on, the lamp nearby).
3.3
Environment-memory for video understanding with real videos
Next, we leverage our environment-memon model for real-world video tasks. A video understanding task defines mapping from sequence of video clips {ci, CN } from longer video V to task label. We consider two tasks: (1) ROOMPRED: where the model must classify which room r, the camera-wearer is in (e.g., living room, kitchen) at time / in the video, and (2) NLQ: natural language queries, an episodic memory task popularized recently in Ego4D [26] where the model must identify the temporal window (f,, fe) in the video that answers an environment-centie query q specified in natural language. See Fig. 4. Both tasks entail human-centric spatial reasoning from video.
Current models produce clip features that encode only what is immediately visible. This is sufficient for short-horizon tasks (e.g., action recognition). but as we will show, falls short on the tasks above that require extra reasoning about the agent's ' surroundings. Our environment memoooy model addresses this by enhancing standard clip features with context from the camera- wearer's 's surroundings.
To do this, for each input clip in E V, we select the center frame Sa of the clip the query frame. Following Sec. 3.2, we uniformly sample K frames from the video around the query frame, encode them along with their pose embeddings (Fqn. 4), and build an environment memory using our environment encoder (Egn. 5). Finally, we use our decoder D to produce the output feature. This results in set of output EgoEnv features, one per input clip.
h, D(fei, ek I.).
Each environment feature then enhances the original clip feature follows
= =W [g hj] bg,
(8)
where 9: is the original clip feature for clip a (e.g., ResNet, SlowFast, EgoVLP) and We. be are linear transform parameters. See Fig. 3 (right).
The new clip features (gí..., Jyy consolidate features from what is directly visible in short video clip and features of the (potentially unseen) space surrounding the camera-wearer, Put simply, our approach implicitly widens the field of view for tasks that reason about short video clips by providing way to access features of their surroundings in a persistent, geometrically consistent manner.

---

.00
0:33
000
0.80
0.57
0.28
0.23
0.52
0.49
Video trajectory Query frame pose A Top attended views
Figure 5: Visualized attention weights. Our model learns to attend to views that help solve the local state prediction task. The query frame Q and top-3 attended views (colored boxes), their positions along the trajectory (colored circles), and their associated attention scores are shown. Sce Supp. for more examples.
Critically, we use the exact same EgoEnv representations to tackle both video understanding tasks. This is a departure from traditional sim-to-real approaches where a task-specific dataset needs to be carefully designed for every downstream task, which may be impractical. That said, given that today's s 3D assets (Matterport3D. HM3D, etc.) focus on indoor spaces, our model is best suited to videos in indoor environments. We discuss the sim-to-real gap in detail in Supp.
4 Experiments
We evaluate how our EgoEnv features learned in simulation benefit real-world video understanding.
Simulator environments For training, we use the Habitat simulator [72] with photo-realistic HM3D [64] scenes to generate simulated video walkthroughs. We generate 15k walkthroughs from 900 HM3D scenes, each 512 steps long, taken by shortest-path agent that navigates to randomly sampled goal locations (move forward, turn right/left 30%). For each time-step, we obtain the ground- truth local state from the simulator required in Sec. 3.1 (i.c., object labels and relative pose). For object labels, we map instance predictions across O 23 categories to the 3D scenes using a pretrained instance segmentation model [21] trained on the subset of scenes with semantic labels. Though the walkthroughs involve discrete actions, they share characteristics with real-world video (cameras at head-level; views covering the environment) making them suitable for transfer. See Supp. for examples and details.
Video datasets We evaluate our models on three egocentric video sources. (1) House [7] contains 119 hours of real-world video footage of house tours from YouTube. We use -32 hours of video from 886 houses where the camera can be localized and create data splits based on houses. (2) Ego4D [26] contains 3k hours of real-world video of people performing daily activities. We use all videos annotated for the NLQ benchmark and apply the provided data splits, which yields 1,259 unseen scenes. (3) Matterport3D (MP3D) [6] contains simulated video walkthroughs from 90 photo-realistic 3D scenes. We use 146 long video walkthroughs and standard data splits [5].
These datasets provide ideal test-bed for our approach. On the one hand, both HouseTouurs and Ego4D are real-world video datasets allowing us to test generalization to both real-world visuals as well as natural human activity across diverse, cluttered environments in unseen houses. On the other hand, MP3D offers novel scenes with distinct visual characteristics and object distributions compared to HM3D, allowing to test our model's robustness to domain shift in a controlled simulated video setting. Note that none of the datasets have a 1-1 alignment in object taxonomy with HM3D, meaning our downstream tasks require generalization both unseen environments and unseen objects.
We collect crowd-sourced labels for each task, which we will publicly share. For ROOMPRED these are room category labels from 21 classes (e.g., living room, kitchen) for each time-step on HouseTTure and Ego4D. For NLQ these are natural language queries and corresponding response tracks in the video. On House the kitchen", "when did I first visit the bathtub") ) and on Ego4D, we use the official NLQ benchmark annotations, which require reasoning over actions, objects, and locations (e.g.. "what tool did pick up from the table", "where did hang the pink cloth").. On MP3D, we source all labels directly from the simulator (9 room categories, and automatically generated NLQ queries from simulator object labels and locations). See Supp. for data collection details and Fig. 4 for examples.
crowd-source queries (e.g., "where did last see my phone in
Experiment setup For pre-training, we use 2048-D ImageNet- ResNet50 [35] features for each video frame. Our encoder-decoder models P, E, D are 2-layer transformers [83] with hidden dimension 128. K 64 frames are sampled from each video to populate the memory. We train
7

---

models for 2.5k epochs and select the model with the lowest validation loss. For ROOMPRED, we generate a single EgoEnv feature aligned with the query clip. For NLQ we generate one feature per input clip. Full architecture and training details are in Supp.
Baselines For ROOMPRED we use a popular scene recognition model PLACESCNN [97] as the baseline model. For NIQ we use the state-of-the-art moment localization model VSINET [94, 49, 53]. Within these two frameworks, we compare the following approaches to enhance clip representations: TRAMEFEAT uses pretrained ResNet50 [35] model to generate frame feature corresponding to each clip. OBJFEAT trains an object detector on all available simulated HM3D data and generates backbone features for each clip. We use the QueryInst Inst model [21]. MAE [34] trains a state-of-the-art self- -elf-sueeeiieed learning approach to reconstruct masked patches of walkthrough video frames. EGO [57] trains graph convolutional network (GCN) over the video graph built following [57]. EPC [65] trains an environment memory model to predict masked zone features conditioned pose. TRF (SCRATCH) trains a scene- meee-m transformer model [20] that shares our model architecture but is randomly initialized and fine-tuned for the task.
These baselines represent various strategies to incorporate environment information into clip repre- sentations ranging from frame features (MAE, FRAMEFEAT, OBJFFAT), to topologiaal graph-based features (EGOTOPO), pose-based features (EPC). Note that OBJFEAT, MAE and EPC all pre-train on the same walkthrough videos as our approach. OBJFEAT further benefits from ground-truth object labels from the simulator. Features from these approaches augment the input clip representations following Eqn. 8 - baseline architectures remain unchanged. Note that EPC requires privileged information coommation-gruun-ttuute camera poses at inference wwheres model does not.
4.1 Pose embedding and local state pretraining
We begin by evaluating the pose embedding network trained to predict relative pose discretized into 12 angles and 4 distance ranges. On the validation set, the model achieves accuracies of 48.4% on relative distance prediction and 34.4% on relative orientation prediction. Note that this task is challenging models must predict relative pose for all possible pairs of observations in trajectory using their visual features alone- - however the goal is to generate pose encodings, not to output perfect pose. Next, we evaluate how well our model can infer the local state, reporting average precision (AP) in each direction. Given a forward view, objects can reliably recognized (37.8 AP) compared to naively outputting the distribution of objects seen at training (5.4 AP). Moreover. our approach can link views in the video trajectory to also infer and anticipate objects in other directions, i... to the right, left and behind (21.5, 24.9 and 20.2 AP respectively) given the forward view. We visualize the attention weights learned by our model to link relevant observations to the query in Fig. 5. Our model learns select informative views for the task beyond just temporally adjacent frames or views with high visual overlap. For example, in the first image, the view with highest attention score (1.0) looks at the bed directly to the left (yellow box), allowing our model to benefit from information beyond its field of view.
4.2 EgoEny features for room prediction
Next, we evaluate our method on predicting what room the camera-wearer visits in the video. All models have access to the full video, but inference is at at each time-step. The PLACESCNN model is 2-layer MLP classifier trained on features from a Places365 [97] scene classification model. Features are max-pooled across a clip of N = 8 frames around the time-step of interest for additional context, as a single frame may be uninformative (e.g., facing wall). We generate environment feature aligned with the center of the clip.
Fig. 6 shows the results. We report top-1 accuracy as a function of dataset difficulty, measured by the prediction entropy of the Places365 model trained on canonical scene images. Instances are "hard" (high entropy) where frame-level information is insufficient for predicting the room type. Accuracy on the full dataset is at the far-right of the plots. All models perform better on HouseTours compared to MP3D and Ego4D since the house tours were captured explicitly to provide informative views of each room. Despite having access to all additional pre-training videos and labels as supervision, frame- level features from OBJFEAT and MAE prove to be insufficient for environment-levee reasoning. All methods except ours perform worse with the introduction of hard instances where the surrounding environment-conter is important (left to right). This is key result: despite training our models entirely in simulation, and with videos from a set of disjoint environments, our EgoEny features are useful for downstream tasks on real-world videos.
8

---

MP3D
HouseTours
Egp4D
72
70
54
acturaty (60
60
68
accuracy D90
55-
86
50
64
50-
40
12
easy - hard IX10P instances)
easy
hard (x10 instainces)
easy - hard (K10P Instances)
PlacesCNN [97)
From efeat
ObjFeat
MAE [34]
Egolopo [57]
EPC (65]
Ir (scratch)
Egatnn
Figure 6: ROOMPRED accuracy by instance difficulty. Our method outperforms baselines, especially on hard instances (smaller performance drop along curve). EPC requires pose at inference, which is unavailable in Ego4D. Results aggregated across three runs. See Supp. for averaged results with error bars.
MP3D [6]
House Tours [7]
Ego4D*
[26]
RANKI @M
@0.3
@0.5
AVG
@0.3
@0.5
AVG
@0.3
@0.5
AVG
VSLNET [94]
33.69
22.83
28.26
42.94
27.68
35.31
5.45
3.12
4.29
FRAMEFEAT
35.23
24.57
29.90
48.45
32.06
40.25
5.58
3.28
4.43
OBJFEAT
37.20
26.33
31.76
47.74
32.49
40.11
5.76
3.43
4.59
MAE [34]
35.11
24.13
29.62
44.49
27.82
36.16
5.65
3.02
4.34
EGOTOPO [57]
36.10
25.06
30.58
43.36
27.97
35.66
5.45
3.19
4.32
EPC [65]
36.85
27.64
32.24
43.22
27.82
35.52
TRF (SCRATCH)
34.18
24.65
29.42
40.54
22.32
31.43
5.25
3.12
4.19
EGOENV
38.18
26.85
32.52
51.98
34.18
43.08
6.04
3.51
4.77
Table I: NLQ results. All baselines build on VSLNet [95] with alternate features. 'Privileged access to pose at
inference, unavailable to our model, and absent in Ego4D. "Validation split. See Table 2 for test set results.
4.3 EgoEny features for episodic memory
Next we evaluate on localizing the responses to natural language queries in egocentric video. We VSLNET [94] and provide it with N 128 clips sampled uniformly from the full video to generate predictions. We use SlowFast [22] clip features and generate environment features aligned with each input clip. We use the benchmark-providee metric of Rank m, which measures temporal localization accuracy [26].
Table shows the NLQ results. Similar to ROOMPRED, instances are harder in MP3D than in HouseTours as MP3D's s shortest-path agents produce moments that are quick transitions between objects and locations. and contain only short glimpses of them. The global, video-level information from EGOTOPO improves performance slightly on all datasets. Strong image-level supervision (object labels in OBJFEAT and FRAMEFEAT) results in the largest improvements; MAE, which has access to the same data but trains self-supervised representations, does not show strong improvements. EPC performs well MP3D, where accurate pose is available from the simulator, but not House Tours with only noisy pose estimates (see Supp.). Our EgoEnv approach performs the best overall, outperforming even EPC, which (unlike EgoEnv) has access ground-truth pose inference.
Finally, Table shows official eval server results for the Ego4D NLQ benchmark. We augment the baseline [63] with our EgoEnv features. We achieve state-of ofte--of-the-ar results, ranking 1st on the public leaderboard at the time of submission, and currently ranked 3rd. Note that Ego4D contains in-the-wild videos of natural human activity in diverse scenes (e.g., workshops, gardens) compared to the simulated walkthrough videos in pretraining. Due to this sim-to-real gap, our approach performs even better instances aligned with training videos (navigation in indoor homes) as Supp. experiments show. Our leading results on the full challenge set for this major benchmark demonstrates the value of environment-centric feature learning approach, despite this gap.
4.4 Analysis of sim-to-real gap
Next, we discuss our approach in the context of the sim-to-real gap. Ego4D videos are in-the-wild. capture natural human actions and object- interactins,,,sss, and take place in diverse scenes. These scenes may significantly different from the simulated environments used for pre-training (navigation in indoor houses). In Table 3, we show results the subset of videos that are aligned with the training environments on the Ego4D NI.Q validation set. We select these using the scenario labels provided in Ego4D including indoor home scenarios (c.g.. listening to music, houschold management)

---

RANK @
@0.3
@0.5
AVG
CONE [39]
15.26
9.24
12.25
BADGERS UW-MAD. [56]
15.71
9.57
12.64
INTERN [10]
16.46
10.06
13.26
NAQ [63]
21.70
13.64
17.67
EGOENV
23.28
14.36
18.82
Table 2: Ego4D N challenge results. Our model obtains the best results against published approaches, and ranks 3rd among concurrent, unpublished work [38, 74].
and navigation-heary scenarios (walking indoors and outdoors) while excluding outdoor activities (e.g., golfing, outdoor cooking). See Supp. for the full list of scenarios. Our approach shows healthier improvements across both sets of scenarios, highlighting the effect of the sim-to-real gap. However, despite this gap, our approach is still able to outperform other approaches over all scenarios, demonstrating the value of our environment-centrré feature learning approach.
RANK K@@ M
RI @033
@I
AVG
NAQ (ALL) [63]
24.12
15.04
19.58
EGOENV (ALI.)
25.37
15.33
20.35
NAQ (INDOOR) [63]
28.91
17.97
23.44
EGOENV (INDOOR)
31.22
19.09
25.16
NAQ (NAV) [63]
23.58
15.49
19.53
EGOENV (NAV)
26.16
16.01
21.08
Table 3: Ego4D N validation set results on aligned scenes. Our approach performs better on the subset of vidcos that are aligned with our approach's 's simulated training environments (navigation, indoor houses),
4.5 Ablation experiments
Next, we discuss some important ablations of our model design. We present full details of these experiments in Supp E and F, but we discuss the main conclusions here.
Importance of pose information: We measure the effect of pose embeddings on our local state prediction task. Our models show small improvements in predicting objects in the forward view (+0.7 mAP) where scene information is directly visible, but large for other views that need to be inferred: mAP improvements of +2.2 (right), +0.9 (behind) and +1.0 (left). Further, we directly embed ground-truth pose as part of the input and see benefits on both tasks on MP3D, but not on HouseTours, due to noise in extracted pose (compared simulator -aroviddd pose MP3D).
Alternate pretraining task formulations: Next, we investigate alternate pretraining objectives instead of local state prediction including variants that only predict the object categories in each cardinal direction, but not the distances or that directly predicts the image features in each cardinal direction, among others. We find that approach that requires predicting both object labels, orientations as well as rough distances offers a balance of both cues during pretraining, translating to strong downstream performance.
Hyperparameter ablations:
We vary window size, memory size and the loss weighting term. We find that small window sizes are sufficient for localizing the room category for ROOMPRED, while larger windows are required for NLQ; and that the model is not very sensitive to the other parameters.
Conclusion
We proposed framework learn environment-aware representations in simulation and transfer them to video understanding tasks on challenging real-world datasets. Our approach outperforms state-of-the-ant representations for predicting visited rooms and retrieving important moments from natural language queries, despite a significant sim-to-real gap. Despite its strengths, there are several opportunities for future work to improve our model further. These include incorporating 3D) information into the local state task (currently defined in 2D, top-down map), generating more human-like simulated videos and integrating more explicit approaches to tackle the sim-to-real gap.
Acknowledgements Thanks to Fu-Jen Chu and Jiabo Hu for help collecting the House annotations. UT Austin is supported in part by IFML NSF AI institute. KG is paid as a research scientist at Meta.
10

---

References
[1] Peter Anderson, Ayush Shrivastava, Joanne Truong, Arjun Majumdar, Devi Parikh, Dhruv Batra, and Stefan Lee. Simato-real transfer for vision-and languagg navigation. In CoRL. 2021. 3
[2] Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi. Manolis Savva, Alexander Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects. In arXiv preprint arXiv:2006. 13171, 2020. 4
[3] Roberto Bigazzi, Federico Landi, Marcella Comia,, Silvia Cascianelli, Lorenzo Baraldi, and Rita Cucchiara. Out of the box: embodied navigation in the real world. In CAIP, 2021.
[4] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset.
In CVPR, 2017. 2,3
[5] Vincent Cartillier, Zhile Ren, Neha Jain, Stefan Lee, Irfan Essa, and Dhruv Batra. Semantic mapnet: Building allocentric semantic maps and representations from egocentric views. In AAAI, 2021. 3,
[6] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgh-d data in indoor environments. In
3DV, 2017. MatterPort3D dataset license available at: http: /tpp:///kkdrrrr vc. in. tum /wt.eeeetttttt/////////
MP_ UP_TOS.p .df.. 3, 9, 11
[7] Matthew Chang, Arjun Gupta, and Saurabh Gupta. Semantic visual navigation by watching youtube videos.
In NewrlPS, 2020. 2, 7.9,
[8] Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to explore using active neural slam. In ICLR, 2020. 3
[9] Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural topological
slam for visual navigation. In CVPR, 2020. 3
[10] Guo Chen, Sen Xing, Zhe Chen, Yi Wang. Kunchang Li, Yizhuo Li, Yi Liu, Jiahao Wang, Yin-Dong Zheng. Bingkun Huang, et al. Intervvideo-ego4:: A pack of champion solutions to ego4d challenges. arXiv preprint arXiv:2211.09522 2022. 10
[11] Tao Chen, Saurabh Gupta, and Abhinav Gupta. Learning exploration policies for navigation. In ICLR,
2019. 3
[12] Wenzheng Chen, Huan Wang. Yangyan Li, Hao Su, Zhenhua Wang, Changhe Tu, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. Synthesizing training images for hoosting human 3d pose estimation. In 3DV, 2016. 3
[13] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Tohy Perrett, Will Price, et al. Rescaling egocentric vision: collection,
pipeline and challenges for spic-kitchens. -000 In IJCV, 2022. 1
[14] Dima Damen, Teesid I. Ieelasawassuk, and Walterio Mayol-Cuevas. You-do, i-leam: Egocentric unsuper- vised discovery of objects and their modes of interaction towards video-based guidance. In CVIU, 2016.
[15] Samyak Datta, Sameer Dharur, Vincent Cartillier, Ruta Desai, Mukul Khanna, Dhruv Batra, and Devi Parikh. Episodic memory question answering. In CVPR, 2022. 1
[16] Maximilian Denninger. Martin Sundermeyer. Dominik Winkelbauer, Youssef Zidan, Dmiury Olefir, Mo- hamad Elbadrawy, Absan Lodhi, and Harinandan Katam. Blenderproe. In arXiv preprint arXiv:1911.0190
2019. 3
[17] Carl Doersch and Andrew Zisserman. Sim2real transfer learning for 3d human pose estimation: motion to the rescue. In NeurlPS, 2019. 3
[18] Alexey Dosovitskiy and Vladlen Koltun. Learning to act by predicting the future. ICT.R, 2017. 3
[19] Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In ICCV, 2021. 3
[20] Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents in long-horizon tasks. In CVPR, 2019. 3,
[21] Yuxin Fang, Shusheng Yang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances queries. In ICCV, 2021. 7, 7,8, 9
[22] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019. 9, 12
[23] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. CVPR, 2021.
[24] Antonino Furnari and Giovanni Maria Farinella. What would you expect? anticipating egocentric actions with rolling- unrolling Istms and modality attention. In ICCV, 2019. 2,
[25] Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In ICCV, 2021. 1
[26] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhong- cong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do. Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen.

---

Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang. Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li. Zhemqiang Li. Karttikeya Mangalam, Raghava Modhugu. Jonathan Munro, Tullie Murrell, Takumi Nishiyasu. Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo. Yuchen Wang, Xindi Wu, Takuma Yagi, Yunyi Zhu, Pablo Arbelacz, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianho Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the World in 3,000 Hours of Egocentric Video. In CVPR, 2022. 1, 2, 3, 6, 7. 9. 4, 5, 10, 12
[27] Jiaqi Guan, Ye Yuan, Kris M Kitani, and Nicholas Rhinchart. Generative hybrid representations for activity forecasting with no-regret learning. In CVPR, 2020. 1,
[28] Saurabh Gupta, James Davidson. Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping
and planning for visual navigation. In CVPR, 2017.
[29] David Ha and Jürgen Schmidhuher. World models. NeurlPS, 2018. 3
[30] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream control: Learning
behaviors by latent imagination. ICI., 2019.
[31] Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding.
In CVPR Workshops, 2019.
[32] Xue Han, Patrick Byrne, Michael Kahana, and Suzanna Becker. When do objects become landmarks? a vr study of the effect of task relevance on spatial memory. In PloS one, 2012. 3
[33] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d enns retrace the history of 2d cnns and imagenet? In CVPR, 2018. 2
[34] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 8,9 9
[35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR, 2016. 7,8
[36] Joao F Henriques and Andrea Vedaldi. Mapnet: An allocentric spatial memory for mapping environments.
In CVPR, 2018. 3
[37] Stefan Hinterstoisser, Vincent Lepetit, Paul Wohlhart, and Kurt Konolige. On pre-trained image features and synthetic images for deep learning. ECCV Workshops, 2018. 3
[38] Zhijian Hou, Lei Ji. Difci Gao. Wanjun Zhong, Kun Yan, Chao Li, Wing-Kwong Chan, Chong- Wah Ngo, Nan Duan, and Mike Zheng Shou. Groundnlq@ ego4d natural language queries challenge 2023. arXiv
preprint arXiv:2306. 15255, 2023. 10. 2
[39] Zhijian Hou, Wanjun Zhong, Lei Ji, Difei Gao, Kun Yan, Wing-Kwong Chan, Chong- Wahg Ngo, Zheng Shou, and Nan Duan. An efficient coarse-to-fine alignment framework @ ego4d natural language queries challenge 2022. arXiv preprint arXiv:221 1.08776, 2022. 10
[40] Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In ICCV,
2015. 3
[41] Dinesh Jayaraman and Kristen Grauman. Learning to look around: Intelligently exploring unseen environments for unknown tasks. In CVPR, 2018. 3, 12
[42] Abhishek Kadian, Joanne Truong, Aaron Gokaslan, Alexander Clegg. Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, and Dhruv Batra. Sim2real predictivity: Does evaluation in simulation predict real-world performance? In RA-I,, 2020.
[43] Nathan Koenig and Andrew Howard. Design and use paradigms for gazebo, an open-source multi-robot simulator. In IROS, 2004.
[44] Jing Yu Koh, Honglak Lee, Yinfei Yang. Jason Baldridge, and Peter Anderson. Pathdreamer: A world
model for indoor navigation. In ICCV, 2021. 3, 12
[45] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai. In
arXiv preprint arXiv:1712.05474. 2017. 3
[46] Eric Krokos, Catherine Plaisant, and Amitabh Varshney, Virtual memory palaces: immersion aids recall.
In Virtual reality, 2019. 3
[47] Yin Li, Zhefan Ye, and James M Rehg. Delving into egocentric actions. In CVPR, 2015. 1
[48] Junwei Liang, Lu Jiang, and Alexander Hauptmann. Simaug: Learning robust representations from simulation for trajectory prediction. In ECCV, 2020.
[49] Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhongeong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining@ @ ego4d challenge 2022. In arXiv preprint arXiv:2207 .0662, 2022. 8, 10, 12
[50] Tianwei Lin. Xiao Liu, Xin Li. Errui Ding, and Shilei Wen. Bmn: Boundary
network for
temporal action proposal generation. In /CCV, 2019. 1.2.5
[51] Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Yu Cheng, Wei Wei, Zichuan Xu, and Yulai Xie.
Context-aware biaffine localizing network for temporal sentence grounding. In CVPR, 2021. 2,
12

---

[52]
Miao Liu, Lingni Ma, Kiran Somasundaram, Yin Li. Kristen Grauman, James M Rehg, and Chao Li. Egocentric activity recognition and localization on 3d map. In ECCV, 2022. 3
[53] Naiyuan Liu, Xiaohan Wang, Xiaobo Li, Yi Yang, and Yueting Zhuang. Reler@ zju-alihaba submission to the ego4d natural language queries challenge 2022. In arXiv preprint S2tv:2227.003882 2022. 8, 10, 12 Antoine Miech, Dimitri Zhukov, Jean- Cann-Bppiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol 00m: Learning text-video embedding by watching hundred million narrated vidco clips. In ICCV, 2019. 3
[54]
[55] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV, 2016. 3
[56] Sicheng Mo, Fangzhou Mu, and Yin Li. A simple transformer- model for ego4d natural language queries challenge. arXiv preprint 3xxiv:2211.07044 2022. 10
[57] Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman. Ego-topo: Environment
affordances from egocentric video. CVPR, 2020. 1, 8, 9
[58] Hyun Soo Park, Jyh-Jing Hwang, Yedong Niu, and Jianho Shi. Egocentric future localization. In CVPR, 2016. 1,
[59]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perf-oerformance deep learning library. In Advances neural information processing systems, 2019. 8
[60] Fiora Pirri, Lorenzo Mauro, Edoardo Alati, Valsamis Ntouskos, Mahdieh Izadpanahkakhik, and Elham Omrani. Anticipation and next action forecasting in video: an end-to-end model with memory. In arXiv
preprint arXiv: 033288. 2019. 2
[61]
Will Price, Carl Vondrick, and Dima Damen. Unweavenet: Unweaving activity stories. In CVPR, 2022. 3 Santhosh Ramakrishnan, Ziad Al-Halah, and Kristen Grauman. Occupancy anticipation for efficient
[62]
exploration and navigation. In ECCV, 2020. 3
[63]
Santhosh Kumar Ramakrishnan, Ziad Al-Halah, and Kristen Grauman. Naq: Leveraging narrations as queries to supervise episodic memory. CVPR, 2023. 9, 10, 12
[64]
Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Drik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat Habitat-matterpe matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AL In NeurIPS Datasets and Benchmarks Track, 2021. 2, 3, 7
[65] Santhosh K Ramakrishnan, Tushar Nagarajan, Ziad Al-Halah, and Kristen Grauman. Environment
predictive coding for embodied agents. In ICLR, 2022. 3, 8, 9, 11
[66]
Adria Recasens, Pauline Luc, cannBaatiste Alayrac, Luyu Wang, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Pāträucean, Florent Altché, Michal Valko, et al. Broaden your views for self- supervised vidco learning. In ICCV, 2021. 3
[67]
Nicholas Rhinehart and Kris M Kitani. Learning action maps of large environments via first-person vision.
In CVPR, 2016. 1,3 3
[68]
Nicholas Rhinchart and Kris M Kitani. First-person activity forccasting with online inverse reinforcement learning. In ICCV, 2017.
[69]
Cesar Roberto de Souza. Adrien Gaidon, Yohann Cabon, and Antonio Manuel Lopez. Procedural generation of vidcos train deep action recognition networks. In CVPR, 2017. 3
[70]
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In ICCV, 2021.
[71] Marco Rosano, Antonino Furnari, Luigi Gulino, and Giovanni Maria Farinella. On embodied visual navigation in real environments through habitat. In ICPR, 2021. 3
[72] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai rescarch. In ICCV, 2019. 3,7
[73] Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-fron coiiiieffroo-mooiiio revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 11
[74] Jaay Shao, Xiaohan Wang. Ruijie Quan, and Yi Yang. Action sensitivity learning for the ego4d episodic memory challenge 2023. arXiv preprint Xrxiv:2306. 17,, 2023. 10, 2
[75] Gunnar A Sigurdsson, Abhinay Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego:
A large-scale dataset of paired third and first person videos. In arXiv preprint arXiv:1804, 180469966. 2018. 1
[76] Shuran Song. Andy Zeng, Angel X Chang, Manolis Savva, Silvio Savarese, and Thomas Funkhouser. 1m2pano3d: Extrapolating 360 structure and semantics heyond the field of view. In CVPR, 2018. 3, 12 [77] Hao Su, Charles R Qi. Yangyan Li, and Leonidas J Guibas. Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views. In ICCV, 2015. 3
[78] Andrew Szot, Alexander Clegg. Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot. Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants rearrange their habitat. In NewrIPS, 2021. 3
13

---

[79] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider. Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. In IROS, 2017. 3 [80] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In
IROS, 2012. 3
[81] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. In arXiv preprint T..Kv:1807.0377400 2018. 3
[82] Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmond, Michael Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In CVPR, 2017. 3
[83] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017. 7.8
[84] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. In CVPR, 2016. 3
[85] Limin Wang, Yuanjun Xiong, The Wang. Yu Qiao, Dahua Lin, Xiaoou Tang. and Luc Van Gool, Temporal
segment networks: Towards good practices for deep action recognition. In ECCV, 2016. 2 [86] Xiaolong Wang, Kaiming He, and Abhinav Gupta. Transitive invariance for self-supervised visual representation learning. ICCV, 2017.
[87] Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle-consistency of time. In CVPR, 2019. 3
[88] Donglai Wei, Joseph J Lim, Andrew Zisserman, and William T Freeman. Learning and using the arrow of time. In CVPR, 2018. 3
[89] Chao- Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick.
Long-term feature hanks for detailed video understanding. In CVPR, 2019. 2
[90] Than Wu and Philipp Krahenbuhl. Towards long-form video understanding. In CVPR, 2021. 1,
[91] Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In CVPR, 2018. Gibson dataset license agreement available http://as stanford. edu/ www.sonn/aaaeeeeeeoeeee
wwweeeaeeeeeett..... pdf. 3, 14
[92] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre- ve-training for motor
control. arXiv preprint arXiv:2203.06/ 73, 2022. 10
[93] Ye Yuan and Kris Kitani. 3d ego-pose estimation via imitation learning. In ECCV, 2018.
[94] Hao Zhang. Aixin Sun, Wei Jing, and Jocy Tianyi Zhou. Span-based localizing network for natural language video localization. ACL, 2020. 8, 9, 11,
[95] Songyang Zhang, Houwen Peng. Jianlong Fu, and Jicbo Luo. Learning 2d temporal adjacent networks for moment localization with natural language. In AAAI, 2020. 2,
[96] Chen Zhao, Ali K Thabet, and Bernard Ghanem. Video self-stitching graph network for temporal action localization. In /CCV, 2021. 2,
[97] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: 10 million image database for scene recognition. In TPAMI, 2017, 2, 8,
14


Reviews:
[{"review_id": "uDqi4FRngL", "reviewer": "Reviewer_iTsH", "summary": "The work presents an approach to learn spatial environment representations for egocentric videos. Such representations encode the camera-wearer’s (seen and unseen) local surroundings/environment. Previous approaches mostly focus on learning representations over a longer temporal space, however, understanding of the physical spatial space seem to be missing which is addressed in the work via learning of environment-aware features.\n\nThe method define a local environment state having a set of object in a relative direction to the camera wearer. The local state has both geometric information (relative object location) and semantic information (object labels). The model learns this local state by learning two matrices - a direction matrix which represents which object is in front, back, left or right and another metric which represents the distance of the object from the camera wearer in a discretized space. To define the matrices, pose information is needed. Since ground truth pose information is missing from real-world egocentric videos, a model first learns the pose information from simulated environment by minimizing a cross entropy loss between the pose of camera wearer and pose of object. \n\nNext, an environment memory is encoded from a video walkthrough of T frames and a query frame. Pose embeddings are generated for all the T frames and query frame using the learned pose model, and each frame is encoded with this pose embedding. Then, K video frames are sampled to construct an environment memory using a transformer encoder. This representation consists of both temporal and spatial information of the environment. The transformer decoder then uses this environment memory and the query frame to generate a EgoEnv representation which is combined with the original video features (eg. Resnet). These representations are finally used for the video downstream tasks. \n\nThe method is evaluated on the room prediction and NLQ challenge on the datasets - Ego4D, HouseTours, and Matterport3D.", "questions": "How much time does it take to train the pose embedding network and the rest of the model too in general?", "limitations": "No such limitations. The paper is well-written, covers all the implementation details, and discusses about all the aspects of the approach.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and has a detailed supplementary material covering the dataset statistics, model architecture details, and ablation experiments.\n2. The work shows a thorough evaluation with the method being evaluated on multiple datasets and compared with multiple baselines.", "weaknesses": "1. It would be great to discuss about the accuracy of the pose embedding learning network on the simulated network since the overall model is dependent on it.\n2. There can be more discussion as to how the sim-to-real gap gets reduced in the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1 It would be great to discuss about the accuracy of the pose embedding learning network on the simulated network since the overall model is dependent on it.\n\nThanks for the suggestion. As mentioned in L157, the pose embedding network is trained to predict relative pose discretized into 12 angles and 4 distance ranges. On the validation set, the model achieves accuracies of 48.4% on relative distance prediction and 34.4% on relative orientation prediction. Note that this task is challenging – models must predict relative pose for all possible pairs of observations in a trajectory using their visual features alone – however the goal is to generate pose encodings, not to output perfect pose. In Supp. E.2.1, we highlight the importance of these pose embeddings for local state prediction. In short, including pose embeddings leads to better object class and distance prediction, especially when objects are not immediately visible.\n\n---\n> W2 There can be more discussion as to how the sim-to-real gap gets reduced in the approach.\n\nWe make efforts to minimize the sim-to-real gap in our dataset and task design. First, we opt for photo-realistic environments from HM3D which contain high-fidelity reconstructions of diverse, real-world houses. Second, when creating the simulated trajectories in Sec. 4 (Simulator environments), we try to match the characteristics of simulated agent walkthroughs to camera-wearer movement in video datasets (L233). Specifically, we position cameras at head-level for the simulated agents, and we adjust the episode duration such that the number of rooms visited per episode by simulated agents roughly matches that in HouseTours (approximately 20 room transitions on average). Please see our Supp. video to compare the generated walkthroughs with HouseTours videos. Finally, we design our local state prediction task around capturing object and environment layout that reflects priors of real-world object distributions. We select this over other simpler alternatives like predicting image features directly, which are more susceptible to failures due to sim-to-real visual differences. We compare alternatives in Supp. E.4.\n\nWe discuss the effect of the sim-to-real gap in Supp B. To summarize, we find that our approach works best on scenes that are aligned with our simulated training data (i.e., indoor home scenarios; videos with lots of walking and less object interaction) and naturally performs worse on out-of-distribution activities (e.g., golfing, outdoor cooking).\n\n---\n> Q1. How much time does it take to train the pose embedding network and the rest of the model too in general?\n\nBoth the local state prediction and the pose embedding training is performed for 2.5k epochs (L262). Each training run takes ~24 hours when trained on two Quadro RTX 6000 GPUs. Note that this is a one time pre-training cost – once trained, the model is directly used as a feature extractor in downstream tasks and does not incur any extra training cost. \n"}, {"review_id": "8At9DQXI07", "reviewer": "Reviewer_9ZdK", "summary": "In the paper, a model is introduced to extract vector embeddings of environments using images from a first-person perspective. The model is trained in a simulated environment where a virtual agent moves around and collects images to learn about its surroundings. The model's performance was tested in two different tasks - predicting the layout of a room and remembering sequences of events. The experimental results showed that the proposed model outperforms other baselines in both downstream tasks.", "questions": "1) How was the algorithm the generate the walking pattern of the virtual agent? Were used different speeds or motions (e.g., running, walking, etc.) during the training?\n\n2) Based on the images presented in the paper, it appears that the agent did not look up or down. I am curious if this lack of movement would impact its performance in videos focused on activities such as cooking, where the camera angle is often directed downwards. Has there been any analysis on this topic?\n", "limitations": "I was unable to locate the discussion on limitations either in the paper or the lengthy supplemental material.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper is well-structured and easy to follow, with a clear motivation behind it. The tables and figures are helpful in explaining the concepts. Overall, I found the paper to be an enjoyable read.\n\n- The paper tackles the very interesting and relevant problem of scene understanding from first-person images.\n\n- The proposed model is very interesting and the experiments are enlightening and helpful in utilizing environment information of egocentric images to better understand the scene.\n\n- I believe this is a solid paper that deserves to be communicated.", "weaknesses": "Although the paper is well-written and structured, many important experiments and discussions were excluded and can only be found in the 18-page supplemental material. One example is the full ablation procedure which is solely available in the supplemental material.\n\nI couldn't find any information on the limitations of the agent, which is crucial in this work. For instance, how well the agent performs in outdoor environments? Moreover, it's important to know and if there are any differences in walking patterns between the virtual agent and real people that may affect the results. Additionally, it would be helpful to know how effective the vector embedding is in dynamic scenes where the environment is constantly changing during video acquisition.\n\nI believe that it would be beneficial to briefly discuss these questions that have been raised.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1. Although the paper is well-written and structured, many important experiments and discussions were excluded and can only be found in the 18-page supplemental material. One example is the full ablation procedure which is solely available in the supplemental material.\n\nWe tried to prioritize the most important information in the main paper, but of course, some experiments had to be moved to Supp. If our paper is accepted, we will move relevant sections to the main paper. For example, the discussion on the limitations of our approach and the ablation experiments mentioned. In the main paper, we point to specific experiments in the supplementary so that a reader knows what is available. \n\n---\n> W2. I couldn't find any information on the limitations of the agent, which is crucial in this work. \n\nPlease see the common response for a discussion on the limitations. Below are responses to specific questions.\n\n> “how well the agent performs in outdoor environments?” \n\nAs mentioned in L220-3 and Supp. B, our approach is best suited to indoor environments that are aligned with training scenes. Supp. Table 1 quantifies this for the NLQ task on Ego4D, where we see the largest improvements in indoor home scenarios (e.g., listening to music, household management) and navigation heavy scenarios (walking indoors and outdoors) and lower improvements in outdoor scenes (e.g., golfing, outdoor cooking). Note that our approach is compatible with outdoor video; however, it is limited by the availability of simulated data for activities in outdoor scenes.\n\n> Additionally, it would be helpful to know how effective the vector embedding is in dynamic scenes where the environment is constantly changing during video acquisition.\n\nInteresting point. Our walkthroughs are generated in static scenes where objects are not moved, while the real world is dynamic. We argue in Supp. C1 that a large part of real-world environments are static (e.g., counter tops, staircases; beds, couches, TV sets) which is valuable to encode even when some objects may have moved around. Our embeddings do perform better where there is primarily camera motion and low scene motion / object interaction – Supp. Table 1 shows increased performance in navigation-heavy videos (e.g., walking indoors and outdoors). Further, the larger improvements across both tasks in HouseTours, where the environment is also static, as compared to Ego4D, also hints at this effect.\n\n---\n> Q1-2. Moreover, it's important to know and if there are any differences in walking patterns between the virtual agent and real people that may affect the results. \n> How was the algorithm the generate the walking pattern of the virtual agent? Were used different speeds or motions (e.g., running, walking, etc.) during the training? Based on the images presented in the paper, it appears that the agent did not look up or down. I am curious if this lack of movement would impact its performance in videos focused on activities such as cooking, where the camera angle is often directed downwards. Has there been any analysis on this topic?\n\nDetails about the simulated agent walkthroughs are in Supp. C1. To summarize, agents follow the shortest-path between two points for a fixed number of steps. The action space is discrete (move forward by 0.25m, turn left/right). The frames were rendered into videos using a fixed FPS (i.e., equivalent to a single speed of walking). The episode length and FPS were selected to approximately match the characteristics of human walkthroughs in HouseTours (i.e., agents move between approximately 20 rooms on average in an episode). We did not experiment with agents looking up/down or in general more “human-like” head motion or navigation policies, though that is an interesting direction for future research.\n\n"}, {"review_id": "fnIvfcBvAN", "reviewer": "Reviewer_ssfq", "summary": "The paper proposes a novel framework to learn environment-aware video representations from egocentric videos. The framework can be trained on synthetic data and incorporated into various existing approaches for real-world downstream tasks including RoomPred and NLQ. Experiments demonstrate that models equipped with the framework achieve superior performances in the two downstream tasks on two real-world egocentric datasets, which demonstrates the value of synthetic data for real-world 3D understanding under egocentric videos.", "questions": "When incorporating the ground-truth camera poses into the method, I am curious about how to input the pose to EgoEnv+pose since I do not find a network parameter that transfers camera poses to pose embeddings.", "limitations": "One potential limitation is that the method has not been tested on dynamic scenes with dynamic background objects or humans such as multi-person interaction scenarios.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "(1) The method in the paper releases the need for camera poses, making it more robust to the noise in structure-from-motion algorithms compared to existing pose-based methods.\n\n(2) Experiments fully demonstrate the effectiveness of the method.\n\n(3) Comprehensive ablation studies demonstrate the value of the core designs of the method.\n\n(4) The method is trained on only automatically-generated synthetic data yet performs better than baselines on real-world scenarios, indicating that simulated data could directly benefit real-world egocentric video understanding without techniques of sim-to-real transfer.\n\n(5) The writing is clear and well-organized.", "weaknesses": "(1) The method hugely relies on the surrounding background objects in the 3D scene. When a person navigates in a scene with clean backgrounds, it might be challenging for the method to encode expressive environment features. When dealing with easy real-world instances in the RoomPred task, the method performance is slightly worse than the baselines.\n\n(2) The performance gain from the proposed method seems marginal on Ego4D.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1 The method hugely relies on the surrounding background objects in the 3D scene. When a person navigates in a scene with clean backgrounds, it might be challenging for the method to encode expressive environment features. When dealing with easy real-world instances in the RoomPred task, the method performance is slightly worse than the baselines.\n\nYes, as mentioned in L187 our method builds on learned priors of objects and their layouts and will be equivalent to standard video models in, for example, an empty room. However, empty scenes may not be aligned with the practical relevance of our models (i.e., assistive robots / AR mentioned in L58-62) where objects and cluttered scenes are key.\n\n---\n> W2 The performance gain from the proposed method seems marginal on Ego4D.\n\nWe present a detailed analysis of our approach’s performance on Ego4D in Supp. B. To summarize, Ego4D videos are in-the-wild videos of natural human activity in diverse scenes. This is in contrast to the simulated walkthroughs used for pretraining. We show that our approach performs well on samples that are aligned with the pretraining data – i.e., indoor home scenarios and navigation heavy scenarios, with lower improvements on out-of-distribution scenes like outdoor activities (e.g., golfing, outdoor cooking). Importantly, while the performance improvement is indeed lower than improvements on HouseTours, our method was the top-ranked approach on Ego4D for NLQ at the time of submission, and remains third ranked on the NLQ leaderboard, demonstrating its impact.\n\n---\n> Q1 When incorporating the ground-truth camera poses into the method, I am curious about how to input the pose to EgoEnv+pose since I do not find a network parameter that transfers camera poses to pose embeddings.\n\nThe input poses are directly transformed to the dimension of the pose embeddings ($p_t$ in Fig 3) using a linear layer. \n\n---\n> L1. One potential limitation is that the method has not been tested on dynamic scenes with dynamic background objects or humans such as multi-person interaction scenarios.\n\nFor testing, Ego4D contains videos with dynamic objects, object interactions, and social interactions. For training, yes, these are interesting future directions that will be driven by advances in simulator capabilities. \n"}, {"review_id": "bz4e1gwkH7", "reviewer": "Reviewer_eEwQ", "summary": "In this paper, the authors address the limitation of current video understanding methods that only analyze short video clips in isolation, without considering the broader context of the camera-wearer's environment. They propose an approach that establishes a connection between egocentric videos and the surrounding environment by learning predictive representations. To accomplish this, the authors train their models using videos captured by agents in simulated 3D environments, where the environment is fully observable. An interesting finding is that despite being exclusively trained on simulated videos, the proposed approach effectively handles real-world videos from HouseTours and Ego4D datasets. It also achieves state-of-the-art results in the Ego4D NLQ challenge.\n\nI have increased the recommendation after the rebuttal. The rebuttal addressed my concerns.\n\n", "questions": "1. The experiments stated in the weakness section.\n2. Per the NeurIPS requirement, I hope the authors can discuss the limitations of the paper. ", "limitations": "The authors did not state the limitations of this work in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, this paper introduces an innovative approach that bridges the gap between egocentric video and the camera-wearer's environment. By leveraging predictive representations, trained on simulated videos, the proposed method demonstrates improved performance over traditional clip-based approaches in various human-centric video tasks and real-world scenarios. The general idea of grounding egocentric video in its underlying world environment is very interesting. The proposed method to learn representations that are predictive of their surroundings and then enhance standard clip-based models is technically sound. \n\n", "weaknesses": "In my opinion, the core of this method is a pertaining process that refines a feature into a better one by implicitly seeing its surroundings.\nIn this sense, in all the experiments, EgoEnv features have knowledge about the surroundings however other features do not. Thus, one important experiment needed to be done is allowing other models to access the same amount of information but in a more straightforward manner. \n\nFor example, giving other models not only the current clip as input, but also frames that are uniformly sampled from the whole video.\n\nFrom the Tables, FrameFeat and ObjectFeat baselines are already very useful. I wonder whether a simple extension of these methods that, similarly, allows a longer temporal range as input at a time, would also perform reasonably well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1/Q1. In my opinion, the core of this method is a pertaining process that refines a feature into a better one by implicitly seeing its surroundings. In this sense, in all the experiments, EgoEnv features have knowledge about the surroundings however other features do not. Thus, one important experiment needed to be done is allowing other models to access the same amount of information but in a more straightforward manner.\n\nPlease note that we do provide such baselines.  As mentioned in L294, all models have access to the full video, while inference is performed at a particular time-step. Several baselines do already use the entire video context. TRF (scratch) and EPC both use the same inputs as our model (uniformly sampled across the video) and Ego-Topo (which builds a graph over the entire video’s frames). TRF (scratch) is in fact a combination of FrameFeat + the transformer to allow longer temporal range. \n\n---\n> Q2. Per the NeurIPS requirement, I hope the authors can discuss the limitations of the paper.\n> The authors did not state the limitations of this work in the paper.\n\nPlease see the discussion on limitations in the common response.\n"}, {"review_id": "fXCwMJvMUQ", "reviewer": "Reviewer_chrV", "summary": "This work aims to learn human centric environment representations from first person camera views. Their novel approach utilizes a transformer-based approach that encodes the local environment state at each time-step in an egocentric video is defined as a set of objects, along with their approximate distances, located in front, left, right, and behind the camera-wearer. This work claims to outperform state-of-the-art representations in predicting visited rooms and retrieving significant moments when responding to natural language queries.\n", "questions": "- The authors should elaborate on how their models would deal with motion blur and sudden movements of the camera. Is there a motion model that is implicitly learned based on the camera movements or is it assumed that the camera moves at a fixed speed? \n- Since this method is intended for AR applications, were their any evaluations done to benchmark the real-time inference performance of this method?\n", "limitations": "The authors do not explicitly state the limitations of this work, and should refer to the Weaknesses section of the review and consider stating those limitations in the paper (if applicable). ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "`Significance:` This work takes an important step towards proposing an approach that models the physical surroundings of a camera from a single egocentric perspective. This work could have multiple applications in AR, VR and robot navigation. \n\n`Originality:` Even though there has been a lot of work in the domain of video understanding in 3D environments, most approaches localize the camera-wearer but do not learn representations for the camera-wearer’s surroundings. Therefore, to the best of my knowledge, this work is novel. \n\n`Quality:` The authors provide a good theoretical background to their approach and back up their claims with rigorous experimentation on multiple datasets and tasks.\n \n`Clarity:` The paper and the supplementary materials are clearly written and easy to follow. ", "weaknesses": "- This work does not consider the influence of motion blur if the person wearing the camera makes sudden movements, making it challenging to apply this work to real environments. \n- This work also uses a semantic segmentation model to identify physical objects in the camera wearers’s environment. However, that could be limiting as their approach is restricted to objects that are classified by the segmentation model.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1. This work does not consider the influence of motion blur if the person wearing the camera makes sudden movements, making it challenging to apply this work to real environments.\n> Q1. The authors should elaborate on how their models would deal with motion blur and sudden movements of the camera. Is there a motion model that is implicitly learned based on the camera movements or is it assumed that the camera moves at a fixed speed?\n\nAs mentioned in L151, the pose embedding model was intentionally designed to account for missing or unreliable pose estimates in real-world video due to rapid camera-motion and motion blur, however, there is no explicit mechanism in place to filter them out. In terms of inputs to the model, models are trained assuming a fixed speed (i.e., a fixed FPS for the simulated agent trajectories). One of our testbeds, Ego4D, exhibits frequent motion blur; please see the Supp video for examples.\n\n---\n> W2. This work also uses a semantic segmentation model to identify physical objects in the camera wearers’s environment. However, that could be limiting as their approach is restricted to objects that are classified by the segmentation model.\n\nYes, this is an important point. As mentioned in L231, we are limited to the 23 categories present in HM3D, detected by semantic segmentation models; however, these categories capture a broad range of commonly available household objects (TV sets, couches, tools etc.) in our video datasets (Ego4D, HouseTours). Importantly, the object classes are *not* a required output of the model – they are used only as labels in the local state task to train the model to link views across a trajectory. Once trained, the prediction heads are discarded while the encoder is retained to generate the environment feature $h_q$ (L192). In short, a larger set of objects is desirable for training, but does not restrict our approach for generating downstream environment features.\n\n---\n> Q2. Since this method is intended for AR applications, were their any evaluations done to benchmark the real-time inference performance of this method?\n\nNo, real-time inference is an important feature but was not a research focus in this work. \n\n---\n> L1. The authors do not explicitly state the limitations of this work, and should refer to the Weaknesses section of the review and consider stating those limitations in the paper (if applicable).\n\nPlease see the discussion on limitations in the common response.\n\n"}]

Author rebuttal:
Thanks to all the reviewers for their effort and constructive feedback. All five reviewers recommend accepting the paper, with two recommending strong accept (5, 6, 7, 8, 8). We address common concerns shared by reviewers below.

**Limitations of the proposed approach**

We will emphasize the limitations more in the paper and summarize here.

We discuss the limitations of our approach in the context of the sim-to-real gap in L328 and Supp. B. Specifically, our model is affected by the type and diversity of pretraining data — videos of simulated agents walking around a house — limiting its generalization to unconstrained real-world video. Similarly, our approach is limited by simulator functionality – HM3D scenes support a small set of objects, which may not overlap with real-world environments, and Habitat does not support fine-grained object-interactions (e.g., chopping vegetables). As a result, we find that our approach works well on videos that are consistent with pretraining (i.e., indoor home scenarios; videos with lots of walking and less object interaction) but contributes less on out-of-distribution scenes and activities (e.g., golfing, outdoor cooking). We expect that future advancements in simulator capabilities (e.g., human motion models for agents, fine grained object interaction simulation) will help address this class of limitations.

Beyond the sim-to-real gap, our approach has other limitations. First, our model does not have specialized modules to aggregate long-term temporal (or pose) information into its representation, compared to, for example, structure from motion methods that can aggregate and re-localize observations over a long video. As a result, our approach does not see benefits from increasing the temporal window from which the memory is constructed (Supp. F3). Next, our local state prediction task is learned in a coarse 2D space – the top-down map of the environment – which does not encode fine-grained geometric relations which may be important for certain tasks (e.g., is the object placed on top of, or inside another?). Finally, our approach is computationally intensive. While pre-training is a one-time cost, generating each augmented clip feature at inference requires the computation of several frame features in the vicinity of the clip, and then aggregating that information using the transformer module. This may be limiting for real-time inference applications in the assistance setting.

Despite these limitations, our approach outperforms state-of-the-art representations for predicting visited rooms and retrieving important moments from natural language queries.


Meta-review: Reviewers are unanimously enthusiastic about this work -- noting that the approach to environmental representation learning is well-motivated and trainable in simulation. Much of the discussion during the rebuttal phase clarified limitations of the proposed approach which should be integrated with the current text.
Decision: Accept (oral)