SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tushar Nagarajan $^{2}$ , Santhosh Kumar Ramakrishnan $^{1}$ , Ruta Desai $^{2}$ , James Hillis $^{2}$ , Kristen Grauman $^{1,2}$
<sup>1</sup>University of Texas at Austin, <sup>2</sup>FAIR, Meta
First-person video highlights a camera-wearer's activities in the context of their persistent environment. However, current video understanding approaches reason over visual features from short video clips that are detached from the underlying physical space and capture only what is immediately visible. To facilitate human-centric environment understanding, we present an approach that links egocentric video and the environment by learning representations that are predictive of the camera-wearer's (potentially unseen) local surroundings. We train such models using videos from agents in simulated 3D environments where the environment is fully observable, and test them on human-captured real-world videos from unseen environments. On two human-centric video tasks, we show that models equipped with our environment-aware features consistently outperform their counterparts with traditional clip features. Moreover, despite being trained exclusively on simulated videos, our approach successfully handles real-world videos from HouseTours and Ego4D, and achieves state-of-the-art results on the Ego4D NLQ challenge. Project page: https://vision.cs.utexas.edu/projects/ego-env/
Egocentric video offers a unique view into human activities through the eyes of a camera-wearer. Understanding this type of video is core to building augmented reality (AR) applications that can provide context-relevant assistance to humans based on their activity. Ego-video is thus the subject of several recent datasets and benchmarks that are driving new research [47, 75, 13, 26].
A key feature of the egocentric setting is the tight coupling of a camera-wearer and their persistent physical environment, i.e., a person's mental model of their surroundings informs their actions. This mental model is important, for example, to reach for a cabinet door out of view, to re-visit the couch to search for a misplaced phone or to visit spaces configured to support certain activities. This raises an important need for human-centric environment understanding — to learn representations from video that capture the camera-wearer's activities in the context of their environment. Such representations would encode the human-environment link, and allow models to jointly reason about both (e.g., to answer "what did the person cut near the sink?"). See Fig. 1.
Despite its importance, there has been only limited work on learning human-centric environment representations. Current video models segment a video into short clips (1-2s long) and then aggregate clip features over time (e.g., with recurrent, graph or transformer-based networks) for tasks like action forecasting [24, 26, 57, 25], temporal action localization [95, 50, 51, 96], episodic memory [26, 15], and movie understanding [90]. Critically, the clip features encode what is immediately visible in a short time window, and their aggregation over time does not equate to linking them in physical space. Other approaches use explicit camera pose information (e.g., from SLAM) to localize the camerewearer, but not its relation to surrounding objects (e.g., forecasting [58, 68, 27]) or to group activities by location, but do not learn representations for agent video (e.g., affordance prediction [67, 57]).
Figure 1: Main idea. Video models trained on short egocentric clips capture a narrow, instantaneous view of human activity (e.g., cutting an onion at the counter) without considering the broader context to which the activity is tied (e.g., the pan to the left to cook the onions, the fridge further behind to store leftovers). We propose to ground video in its underlying 3D environment by learning representations that are predictive of their surroundings, thereby enhancing standard clip-based models with complementary environment information.
To address these shortcomings, we propose to learn environment-aware video representations that encode the surrounding physical space. Specifically, we define the local environment state at each time-step of an egocentric video as the set of objects (and their rough distance) in front, to the left, right, and behind the camera-wearer. See Fig. 2. We use this state as supervision to train a transformer-based video encoder model that aggregates visual information across a video to build an environment memory, which can be queried to predict the local state at any point in the video.
The local state captures the rough layout of objects relative to the camera-wearer. It is important for understanding physical space — it provides a semantic signal to localize people (e.g., in the living room, from the arrangement of couches, lamps and tables) — as well as human behavior, since people move towards layouts that support activities (e.g., stove-top areas, dressers). Predicting the local state thus involves capturing the natural statistics of object layouts across different homes and then translating contexts across environments to reason about new ones. Once trained, given an observation from a new video, our model produces a drop-in EgoEnv feature which encodes environment information to complement the action information in existing video clip features.
An important practical question is how to supervise such a representation. Sourcing local state labels requires agent and object positions and omnidirectional visibility at each time step. This is challenging as egocentric videos only offer sparse coverage of the environment. Furthermore, they are prone to object detection, tracking, and SLAM failures. Therefore, for training we turn to videos generated by agents in simulation. This allows us to sample diverse, large-scale trajectories to cover the environment, while also providing ground-truth local state. Once trained with simulated video, we apply our models to real-world videos from new, unseen environments.
We demonstrate our EgoEnv approach on two video tasks where joint reasoning of both human action and the underlying physical space is required: (1) inferring the room category that the camera wearer is physically in as they move through their environment, and (2) localizing the answer to a natural language query in an egocentric video. These tasks support many potential applications, including AR systems that can offer context-relevant assistance.
We are the first to demonstrate the value of 3D simulation data for real-world ego-video understanding. Our experiments show that by capitalizing on both geometric and semantic cues in our proposed "local environment state" task, we can leverage video walkthroughs from simulated agents in HM3D scenes [64] to ultimately enable downstream human-centric environment models on real-world videos from HouseTours [7] and Ego4D [26]. Furthermore, models equipped with our EgoEnv features outperform both popular scene classification [97] and natural language video localization models [95], achieves state-of-the-art results on the Ego4D NLQ challenge leaderboard.
Video understanding in 3D environments. Prior work encodes short video clips [4, 85, 60, 33] or temporally aggregates them for additional context [89, 24, 95, 50, 51, 96, 90]. However, these methods treat the video as a temporal sequence and fail to capture the spatial context from the underlying persistent environment. For egocentric video, prior work has used structure from motion (SfM) to map people and objects for trajectory [58] and activity forecasting [27] and action grounding
in 3D [67, 14]. These approaches localize the camera-wearer but do not learn representations for the camera-wearer's surroundings. The model of [52] associates features to voxel maps to localize actions; however, they require a pre-computed 3D scan of the environment at both training and inference. Prior work groups clips by rough spatial location as topological graphs [57] or activity threads [61], but they stop short of learning representations using these groups. In contrast, we explicitly learn features that relate clips based on their spatial layout, for each step of an ego-video.
Video representation learning. Traditional video understanding methods learn representations by training models on large, manually curated video action recognition datasets [4, 54, 26]. Recent self-supervised learning (SSL) approaches eliminate the supervision requirement by leveraging implicit temporal signals [88, 55, 81, 40, 31, 84, 86, 87, 66, 23]. In contrast, we learn features that encode the local spatial-state of the environment (as opposed to temporal signals). Further, we show how to leverage state information that is readily accessible in simulation, but not in video datasets (i.e., locations and semantic classes of objects surrounding the agent) for training.
Environment features for embodied AI. In embodied AI, pose-estimates are used to build maps [11, 8, 62], as edge features in graphs [9, 7], as spatial embeddings for episodic memories [20], to project features to a grid map [28, 36, 5] or to learn environment features for visual navigation [65]. However, these approaches are explored solely in simulation and typically require accurate pose-estimates or smooth action spaces, and thus are not directly applicable to egocentric videos. Research on world-models [18, 29, 30] and unseen panorama reconstruction [41, 76, 44] hallucinate the effect of agent actions to aid decision-making in simulation. In contrast, we aim to learn environment features for an egocentric camera-wearer to aid real-world video understanding.
Learning from simulated data. Prior work has proposed cost-effective ways to generate large-scale synthetic image datasets for various vision tasks [77, 79, 16, 37, 19, 70]. In robotics, simulation environments have been developed to quickly and safely train policies, with the eventual goal of transferring them to real world applications [43, 80, 45, 6, 91, 72, 64, 78]. The resulting sim-to-real problem, where models must adapt to changes in simulator and real-world domains, is an active area of research for robot navigation [79, 42, 71, 3, 1]. However, simulated data for video understanding is much less explored. Prior work has synthesized data for human body pose estimation [12, 82, 93, 17], trajectory forecasting [48], and action recognition [69]. Rather than model human behavior, our approach is the first to directly capture the environment surrounding the camera-wearer for real-world video understanding tasks.
Our goal is to learn EgoEnv representations that encode the local surroundings of the camera-wearer. Such a representation would implicitly maintain a semantic memory of surrounding objects beyond what is immediately visible, and coupled with a standard video feature, would allow models to jointly reason about activities and the underlying physical space. Directly appending the camera pose with each video frame may capture the local state; however noise in pose estimated from ego-video with quick head motions and characteristic blur limits its utility (see Supp. for experiments).
Instead, we introduce an approach that leverages simulated environments where perfect state information is available to train models that can link visual information to the physical surroundings. To this end, we first define the local state prediction task in simulation (Sec. 3.1). Next, we introduce our transformer-based architecture that predicts the local state in videos (Sec. 3.2). Finally, we show how our model trained in simulation generates environment features for real-world egocentric video frames (Sec. 3.3).
3.1 Local environment state
We require a model that is aware of not just what is immediately visible in a single frame, but also of the camera-wearer's surroundings. We therefore define the local environment state of the camera-wearer as the set of objects in each relative direction — i.e., what objects are to the front, left, right or behind the camera-wearer, together with their rough distance from the camera-wearer — and train a model to predict this state. Our definition of local state takes inspiration from cognitive science [32, 46], and offers supervision signals that are both geometric (relative object locations) and semantic (semantic object labels), which we observe leads to strong representations.
Figure 2: Local environment state task. Left panel: Top-down environment view showing the camera-wearer path (blue gradient line) and nearby objects (colored boxes). Middle panel: Egocentric view in each direction. Only the forward view (top left) is observed. Remaining views are shown for clarity. Task: Given an egocentric video of a person in their environment, the model must predict the set of objects (and their rough distance) in front, to the left, right, and behind the camera-wearer at each time-step (right panel). The model only sees the forward ego-view (middle panel, top left) and does not have access to the top-down map. Note that not all parts of the environment are seen during a walkthrough (white vs. grey regions on map) — models must link seen observations based on their shared space, as well as anticipate unseen surroundings based on statistics of training environments. Best viewed in color.
Local state labels
$\uparrow$  Front (couch - very near)
$\rightarrow$  Right (cabinet - far)
$\downarrow$  Back (couch - near, plant - far)
Left (book - near, couch - near, picture - far)
Video trajectory
Camera-wearer
Obstacles
Far away / hidden objects
Seen vs. unseen locations
More formally, let  $\mathcal{O}$  be a set of object classes. For a frame  $f$  from a video trajectory in an environment, the local state is a tuple  $(y_{o}, y_{r})$ .  $y_{o}$  is a  $4 \times |\mathcal{O}|$  dimensional matrix which represents instances of each object class in the four cardinal directions relative to the camera-wearer.  $y_{r}$  is a matrix of the same size containing the distance of the objects in  $y_{o}$  from the camera-wearer, discretized into 5 distance ranges between  $0.25 - 5.0m^{1}$ . For direction  $i$  and object class  $j$ , the labels are:
$$
y _ {o} [ i, j ] = \left\{ \begin{array}{l l} 1 & \text {i f} d \left(p _ {a}, p _ {j}\right) <   \delta \wedge \theta \left(p _ {a}, p _ {j}\right) = i \\ 0 & \text {o t h e r w i s e ,} \end{array} \right. \tag {1}
$$
$$
y _ {r} [ i, j ] = \bar {d} \left(p _ {a}, p _ {j}\right) \quad \text {i f} y _ {o} [ i, j ] = 1, \tag {2}
$$
where  $p_a, p_j$  are the poses of the camera-wearer and object  $o_j$  respectively,  $d(p_a, p_j)$  is the euclidean distance between them,  $\delta$  is a distance threshold for nearby objects (we set  $\delta = 5.0\mathrm{m}$ , beyond which visible objects are small),  $\theta(p_a, p_j) \in [0\dots 3]$  is the discretized angle of the object relative to the agent's heading (forward, right, behind, left), and  $\bar{d}(p_a, p_j) \in [0\dots 4]$  is discretized distance. See Fig. 2 and Supp. for empirical analysis of related alternatives, e.g., predicting just object presence or image features.
We then train a model to predict the local state of the target video frame, conditioned on the video trajectory. Once trained, the model can relate what is visible in a frame to the camera-wearer's possibly hidden surroundings to produce environment-aware features.
Since supervision for camera-wearer pose and every object's location is non-trivial for egocentric videos, where camera localization and tracking is error prone, we leverage videos in simulated environments for training, as presented next.
3.2 Environment-aware pretraining in simulation
To source local state labels, we generate a dataset of video walkthroughs of agents in simulated 3D environments where agent and object poses are accessible at all times (see Sec. 4). We train our model in two stages described below.
3.2.1 Pose embedding learning
While ground truth camera pose is available from the simulator at training time, a model trained to rely on it will fail on real-world egocentric video at test time, where pose estimates are noisy. With the goal of handling arbitrary indoor egocentric video, we instead explore representations that implicitly encode coarse pose information. See Supp. for pose-related experiments.
For object classes with multiple instances, we select the nearest one.
Figure 3: Model framework. Left: Our model first learns pose embeddings by predicting discretized relative pose between observations from simulated video walkthroughs (Sec. 3.2.1). Center: Next, it encodes observations and their pose embeddings into an environment memory that is trained to predict the local environment state for a query frame (Sec. 3.2.2). Right: Once trained, our model builds and queries the environment memory for any time-point of interest in a real-world video, to generate an environment feature for downstream video tasks in disjoint and novel scenes (Sec. 3.3).  $\oplus =$  concatenation.
Specifically, for a sequence of RGB frames  $\{f_t\}_{t=1}^T$  and camera poses  $\{\theta_t\}_{t=1}^T$ , we generate pose embeddings  $\{p_t\}_{t=1}^T = \mathcal{P}(f_1, \dots, f_T)$  using a transformer encoder network. These embeddings are used to predict the relative pose between each observation pair using a bilinear layer
$$
\hat {\theta} _ {i, j} = p _ {i} ^ {T} V _ {p} p _ {j} + W _ {p} ^ {T} \left(p _ {j} - p _ {i}\right) + b _ {p}, \tag {3}
$$
where  $\hat{\theta}_{i,j}$  is the predicted relative pose and  $V_{p}, W_{p}, b_{p}$  are trainable parameters. We discretize the relative pose into 12 angles and 4 distance ranges to provide an approximate yet robust pose estimate. The network is trained to minimize cross-entropy between the predicted and the target relative pose labels for all observation pairs  $\sum_{i,j} \mathcal{L}_{ce}(\hat{\theta}_{i,j}, \theta_{i,j})$ . The trained pose embeddings  $p_{t}$  encode information to help relate video observations based on their location and orientation in the environment. Note that once trained, pose embeddings are inferred directly from video frame sequences — ground truth pose is only required for training.
3.2.2 Local state pretraining
Next, we train a model to embed visual information from a video walkthrough into an environment memory, which can then be queried to infer the local state corresponding to a given video frame from the same video. We implement this model as a transformer encoder-decoder model.
Specifically, for a video walk through  $\mathcal{V}$  with RGB frames  $\{f_t\}_{t=1}^T$ , and a query frame  $f_q$ , we predict the local state  $y_q = (y_o, y_r)$  as follows. First, pose embeddings  $\{p_t\}_{t=1}^T$  are generated for the video and query frames following Sec. 3.2.1. Then, each frame is encoded jointly with the pose embedding using a linear transform  $\mathcal{M}_p$ .
$$
x _ {t} = \mathcal {M} _ {p} ([ f _ {t}; p _ {t}) ]). \tag {4}
$$
Next, we uniformly sample  $K$  video frames to construct an environment memory using a transformer encoder  $\mathcal{E}$ , which updates frame representations using self-attention:
$$
\left\{e _ {1}, \dots , e _ {K} \right\} = \mathcal {E} \left(x _ {1}, \dots , x _ {K}\right). \tag {5}
$$
The resulting memory represents features for each time-step that contain propagated information from all other time-steps. Compared to prior work [24, 95, 50, 51, 96], our encoder has the ability to relate observations based on not just visual characteristics and their temporal ordering, but also their relative spatial layout in the environment. A transformer decoder  $\mathcal{D}$  then attends over the memory using query  $x_{q}$  to produce the output EgoEnv representation  $h_q$ :
$$
h _ {q} = \mathcal {D} \left(\left\{e _ {1}, \dots , e _ {K} \right\}, x _ {q}\right), \tag {6}
$$
which is finally used to predict the local state using two linear classifiers  $\mathcal{M}_o$  and  $\mathcal{M}_r$  for object class and distance predictions respectively. The network is trained to minimize the combination of losses over the predicted and the target state labels for each direction:
$$
\mathcal {L} \left(h _ {q}, y _ {o}, y _ {r}\right) = \mathcal {L} _ {b c e} \left(\mathcal {M} _ {o} \left(h _ {q}\right), y _ {o}\right) + \lambda \mathcal {L} _ {c e} \left(\mathcal {M} _ {r} \left(h _ {q}\right), y _ {r}\right),
$$
Figure 4: Scene understanding in third-person photos vs. human-centric environment understanding. Left: Well-framed, canonical images from Places365 are considerably different from scene content observed in egocentric video. Center and right: Real egocentric video streams from HouseTours (top) and Ego4D (bottom) illustrating the value in modeling the underlying environment, rather than just what is visible in short clips. For example, the person does not explicitly look at the staircase while walking down it (center, top row); the spatial relation between the person, piano, and kitchen is important to answer the question (right, top row).
where  $\mathcal{L}_{bce}$  and  $\mathcal{L}_{ce}$  refer to binary cross-entropy and cross-entropy losses respectively. We set  $\lambda = 0.1$  which balances the contributions of each loss function based on validation experiments. The distance loss is computed only for objects that are in the local state ( $y_{o} = 1$ ). See Fig. 3 (left) and Supp. for more architecture details.
Learning to predict the local state involves aggregating information about observed objects across time, as well as anticipating unseen objects based on learned priors from the layout of objects in other scenes (e.g., TVs are usually in front of couches; kitchens have particular arrangements of sinks, refrigerators, and stove-tops).
Once trained, given a video in a new environment and a time-point of interest, our model constructs an environment memory, predicting the local state based on information aggregated throughout the test video. Importantly,  $h_{q}$  — the EgoEnv feature — contains valuable information about the camerewer's surroundings, offering environment features to complement traditional video features (e.g., for a person watching TV, also encode the couch they are sitting on, the lamp nearby).
3.3 Environment-memory for video understanding with real videos
Next, we leverage our environment-memory model for real-world video tasks. A video understanding task defines a mapping from a sequence of video clips  $\{c_1,\dots,c_N\}$  from longer video  $\mathcal{V}$  to a task label. We consider two tasks: (1) ROOMPred: where the model must classify which room  $r_t$  the camera-wearer is in (e.g., living room, kitchen) at time  $t$  in the video, and (2) NLQ: natural language queries, an episodic memory task popularized recently in Ego4D [26] where the model must identify the temporal window  $(t_s,t_e)$  in the video that answers an environment-centric query  $q$  specified in natural language. See Fig. 4. Both tasks entail human-centric spatial reasoning from video.
Current models produce clip features that encode only what is immediately visible. This is sufficient for short-horizon tasks (e.g., action recognition), but as we will show, falls short on the tasks above that require extra reasoning about the agent's surroundings. Our environment-memory model addresses this by enhancing standard clip features with context from the camera-wearer's surroundings.
To do this, for each input clip in  $c_{i} \in \mathcal{V}$ , we select the center frame  $f_{i}$  of the clip as the query frame. Following Sec. 3.2, we uniformly sample  $K$  frames from the video around the query frame, encode them along with their pose embeddings (Eqn. 4), and build an environment memory using our environment encoder  $\mathcal{E}$  (Eqn. 5). Finally, we use our decoder  $\mathcal{D}$  to produce the output feature. This results in set of output EgoEnv features, one per input clip.
$$
h _ {i} = \mathcal {D} \left(\left\{e _ {1} ^ {i}, \dots , e _ {K} ^ {i} \right\}, x _ {i}\right). \tag {7}
$$
Each environment feature then enhances the original clip feature as follows
$$
g _ {i} ^ {\prime} = W _ {\mathcal {E}} ^ {T} \left[ g _ {i}; h _ {i} \right] + b _ {\mathcal {E}}, \tag {8}
$$
where  $g_{i}$  is the original clip feature for clip  $c_{i}$  (e.g., ResNet, SlowFast, EgoVLP) and  $W_{\mathcal{E}}, b_{\mathcal{E}}$  are linear transform parameters. See Fig. 3 (right).