SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The new clip features  $\{g_1',\dots,g_N'\}$  consolidate features from what is directly visible in a short video clip and features of the (potentially unseen) space surrounding the camera-wearer. Put simply, our approach implicitly widens the field of view for tasks that reason about short video clips by providing a way to access features of their surroundings in a persistent, geometrically consistent manner.
Figure 5: Visualized attention weights. Our model learns to attend to views that help solve the local state prediction task. The query frame  $Q$  and top-3 attended views (colored boxes), their positions along the trajectory (colored circles), and their associated attention scores are shown. See Supp. for more examples.
Critically, we use the exact same EgoEnv representations to tackle both video understanding tasks. This is a departure from traditional sim-to-real approaches where a task-specific dataset needs to be carefully designed for every downstream task, which may be impractical. That said, given that today's 3D assets (Matterport3D, HM3D, etc.) focus on indoor spaces, our model is best suited to videos in indoor environments. We discuss the sim-to-real gap in detail in Supp.
We evaluate how our EgoEnv features learned in simulation benefit real-world video understanding.
Simulator environments For training, we use the Habitat simulator [72] with photo-realistic HM3D [64] scenes to generate simulated video walkthroughs. We generate  $\sim 15\mathrm{k}$  walkthroughs from 900 HM3D scenes, each 512 steps long, taken by a shortest-path agent that navigates to randomly sampled goal locations (move forward, turn right/left  $30^{\circ}$ ). For each time-step, we obtain the ground-truth local state from the simulator required in Sec. 3.1 (i.e., object labels and relative pose). For object labels, we map instance predictions across  $|\mathcal{O}| = 23$  categories to the 3D scenes using a pretrained instance segmentation model [21] trained on the subset of scenes with semantic labels. Though the walkthroughs involve discrete actions, they share characteristics with real-world video (cameras at head-level; views covering the environment) making them suitable for transfer. See Supp. for examples and details.
Video datasets We evaluate our models on three egocentric video sources. (1) HouseTours [7] contains 119 hours of real-world video footage of house tours from YouTube. We use  $\sim 32$  hours of video from 886 houses where the camera can be localized and create data splits based on houses. (2) Ego4D [26] contains 3k hours of real-world video of people performing daily activities. We use all videos annotated for the NLQ benchmark and apply the provided data splits, which yields 1,259 unseen scenes. (3) Matterport3D (MP3D) [6] contains simulated video walkthroughs from 90 photo-realistic 3D scenes. We use 146 long video walkthroughs and standard data splits [5].
These datasets provide an ideal test-bed for our approach. On the one hand, both HouseTours and Ego4D are real-world video datasets allowing us to test generalization to both real-world visuals as well as natural human activity across diverse, cluttered environments in unseen houses. On the other hand, MP3D offers novel scenes with distinct visual characteristics and object distributions compared to HM3D, allowing us to test our model's robustness to domain shift in a controlled simulated video setting. Note that none of the datasets have a 1-1 alignment in object taxonomy with HM3D, meaning our downstream tasks require generalization to both unseen environments and unseen objects.
We collect crowd-sourced labels for each task, which we will publicly share. For ROOMPRED these are room category labels from 21 classes (e.g., living room, kitchen) for each time-step on HouseTours and Ego4D. For NLQ these are natural language queries and corresponding response tracks in the video. On HouseTours, we crowd-source queries (e.g., "where did I last see my phone in the kitchen", "when did I first visit the bathtub") and on Ego4D, we use the official NLQ benchmark annotations, which require reasoning over actions, objects, and locations (e.g., "what tool did I pick up from the table", "where did I hang the pink cloth"). On MP3D, we source all labels directly from the simulator (9 room categories, and automatically generated NLQ queries from simulator object labels and locations). See Supp. for data collection details and Fig. 4 for examples.
Experiment setup For pre-training, we use 2048-D ImageNet-pretrained ResNet50 [35] features for each video frame. Our encoder-decoder models  $\mathcal{P},\mathcal{E},\mathcal{D}$  are 2-layer transformers [83] with hidden dimension 128.  $K = 64$  frames are sampled from each video to populate the memory. We train
models for 2.5k epochs and select the model with the lowest validation loss. For ROOMPRED, we generate a single EgoEnv feature aligned with the query clip. For NLQ we generate one feature per input clip. Full architecture and training details are in Supp.
Baselines For ROOMPRED we use a popular scene recognition model PLACESCNN [97] as the baseline model. For NLQ we use the state-of-the-art moment localization model VSLNET [94, 49, 53]. Within these two frameworks, we compare the following approaches to enhance clip representations: FRAMEFEAT uses a pretrained ResNet50 [35] model to generate a frame feature corresponding to each clip. OBJFEAT trains an object detector on all available simulated HM3D data and generates backbone features for each clip. We use the QueryInst model [21]. MAE [34] trains a state-of-the-art self-supervised learning approach to reconstruct masked patches of walkthrough video frames. EGOToPO [57] trains a graph convolutional network (GCN) over the video graph built following [57]. EPC [65] trains an environment memory model to predict masked zone features conditioned on pose. TRF (SCRATCH) trains a scene-memory transformer model [20] that shares our model architecture but is randomly initialized and fine-tuned for the task.
These baselines represent various strategies to incorporate environment information into clip representations ranging from frame features (MAE, FRAMEFEAT, OBJFEAT), to topological graph-based features (EGOTOPO), to pose-based features (EPC). Note that OBJFEAT, MAE and EPC all pre-train on the same walkthrough videos as our approach. OBJFEAT further benefits from ground-truth object labels from the simulator. Features from these approaches augment the input clip representations following Eqn. 8 — baseline architectures remain unchanged. Note that EPC requires privileged information—ground-truth camera poses at inference time—whereas our model does not.
4.1 Pose embedding and local state pretraining
We begin by evaluating the pose embedding network trained to predict relative pose discretized into 12 angles and 4 distance ranges. On the validation set, the model achieves accuracies of  $48.4\%$  on relative distance prediction and  $34.4\%$  on relative orientation prediction. Note that this task is challenging — models must predict relative pose for all possible pairs of observations in a trajectory using their visual features alone — however the goal is to generate pose encodings, not to output perfect pose. Next, we evaluate how well our model can infer the local state, reporting average precision (AP) in each direction. Given a forward view, objects can be reliably recognized (37.8 AP) compared to naively outputting the distribution of objects seen at training (5.4 AP). Moreover, our approach can link views in the video trajectory to also infer and anticipate objects in other directions, i.e., to the right, left and behind (21.5, 24.9 and 20.2 AP respectively) given the forward view. We visualize the attention weights learned by our model to link relevant observations to the query in Fig. 5. Our model learns to select informative views for the task beyond just temporally adjacent frames or views with high visual overlap. For example, in the first image, the view with highest attention score (1.0) looks at the bed directly to the left (yellow box), allowing our model to benefit from information beyond its field of view.
4.2 EgoEnv features for room prediction
Next, we evaluate our method on predicting what room the camera-wearer visits in the video. All models have access to the full video, but inference is at at each time-step. The PLACESCNN model is a 2-layer MLP classifier trained on features from a Places365 [97] scene classification model. Features are max-pooled across a clip of  $N = 8$  frames around the time-step of interest for additional context, as a single frame may be uninformative (e.g., facing a wall). We generate an environment feature aligned with the center of the clip.
Fig. 6 shows the results. We report top-1 accuracy as a function of dataset difficulty, measured by the prediction entropy of the Places365 model trained on canonical scene images. Instances are "hard" (high entropy) where frame-level information is insufficient for predicting the room type. Accuracy on the full dataset is at the far-right of the plots. All models perform better on HouseTours compared to MP3D and Ego4D since the house tours were captured explicitly to provide informative views of each room. Despite having access to all additional pre-training videos and labels as supervision, frame-level features from OBJFEAT and MAE prove to be insufficient for environment-level reasoning. All methods except ours perform worse with the introduction of hard instances where the surrounding environment-context is important (left to right). This is a key result: despite training our models entirely in simulation, and with videos from a set of disjoint environments, our EgoEnv features are useful for downstream tasks on real-world videos.
Figure 6: ROOMPred accuracy by instance difficulty. Our method outperforms baselines, especially on hard instances (smaller performance drop along curve). EPC requires pose at inference, which is unavailable in Ego4D. Results are aggregated across three runs. See Supp. for averaged results with error bars.
RANK1@M→	@0.3	@0.5	AVG
VSLNET [94]	33.69	22.83	28.26
FRAMEFEAT	35.23	24.57	29.90
OBJFEAT	37.20	26.33	31.76
MAE [34]	35.11	24.13	29.62
EGOTOPO [57]	36.10	25.06	30.58
EPC† [65]	36.85	27.64	32.24
TRF (SCRATCH)	34.18	24.65	29.42
EGOENV	38.18	26.85	32.52
@0.3	@0.5	AVG
42.94	27.68	35.31
48.45	32.06	40.25
47.74	32.49	40.11
44.49	27.82	36.16
43.36	27.97	35.66
43.22	27.82	35.52
40.54	22.32	31.43
51.98	34.18	43.08
Ego4D‡[26]
@0.3	@0.5	AVG
5.45	3.12	4.29
5.58	3.28	4.43
5.76	3.43	4.59
5.65	3.02	4.34
5.45	3.19	4.32
-	-	-
5.25	3.12	4.19
6.04	3.51	4.77
Table 1: NLQ results. All baselines build on VSLNet [95] with alternate features.  ${}^{ \dagger  }$  Privileged access to pose at inference, unavailable to our model, and absent in Ego4D.  ${}^{ \ddagger  }$  Validation split. See Table 2 for test set results.
4.3 EgoEnv features for episodic memory
Next we evaluate on localizing the responses to natural language queries in egocentric video. We use VSLNET [94] and provide it with  $N = 128$  clips sampled uniformly from the full video to generate predictions. We use SlowFast [22] clip features and generate environment features aligned with each input clip. We use the benchmark-provided metric of Rank  $n@m$ , which measures temporal localization accuracy [26].
Table 1 shows the NLQ results. Similar to ROOMPRED, instances are harder in MP3D than in HouseTours as MP3D's shortest-path agents produce moments that are quick transitions between objects and locations, and contain only short glimpses of them. The global, video-level information from EGOtoPO improves performance slightly on all datasets. Strong image-level supervision (object labels in OBJFEAT and FRAMEFEAT) results in the largest improvements; MAE, which has access to the same data but trains self-supervised representations, does not show strong improvements. EPC performs well on MP3D, where accurate pose is available from the simulator, but not on HouseTours with only noisy pose estimates (see Supp.). Our EgoEnv approach performs the best overall, outperforming even EPC, which (unlike EgoEnv) has access to ground-truth pose at inference.
Finally, Table 2 shows official eval server results for the Ego4D NLQ benchmark. We augment the baseline [63] with our EgoEnv features. We achieve state-of-the-art results, ranking 1st on the public leaderboard at the time of submission, and currently ranked 3rd. Note that Ego4D contains in-the-wild videos of natural human activity in diverse scenes (e.g., workshops, gardens) compared to the simulated walkthrough videos in pretraining. Due to this sim-to-real gap, our approach performs even better on instances aligned with training videos (navigation in indoor homes) as our Supp. experiments show. Our leading results on the full challenge set for this major benchmark demonstrate the value of our environment-centric feature learning approach, despite this gap.
4.4 Analysis of sim-to-real gap
Next, we discuss our approach in the context of the sim-to-real gap. Ego4D videos are in-the-wild, capture natural human actions and object-interactions, and take place in diverse scenes. These scenes may be significantly different from the simulated environments used for pre-training (navigation in indoor houses). In Table 3, we show results on the subset of videos that are aligned with the training environments on the Ego4D NLQ validation set. We select these using the scenario labels provided in Ego4D including indoor home scenarios (e.g., listening to music, household management)
RANK 1@M →	@0.3	@0.5	AVG
CONE [39]	15.26	9.24	12.25
BADGERS@UW-MAD. [56]	15.71	9.57	12.64
INTERNVIDEO [10]	16.46	10.06	13.26
NAQ [63]	21.70	13.64	17.67
EGOENV	23.28	14.36	18.82
and navigation-heavy scenarios (walking indoors and outdoors) while excluding outdoor activities (e.g., golfing, outdoor cooking). See Supp. for the full list of scenarios. Our approach shows healthier improvements across both sets of scenarios, highlighting the effect of the sim-to-real gap. However, despite this gap, our approach is still able to outperform other approaches over all scenarios, demonstrating the value of our environment-centric feature learning approach.
Table 2: Ego4D NLQ challenge results. Our model obtains the best results against published approaches, and ranks 3rd among concurrent, unpublished work [38, 74].
RANK K@M →	R1@0.3	R1@0.5	AVG
NAQ (ALL) [63]	24.12	15.04	19.58
EGOENV (ALL)	25.37	15.33	20.35
NAQ (INDOOR) [63]	28.91	17.97	23.44
EGOENV (INDOOR)	31.22	19.09	25.16
NAQ (NAV) [63]	23.58	15.49	19.53
EGOENV (NAV)	26.16	16.01	21.08
Table 3: Ego4D NLQ validation set results on aligned scenes. Our approach performs better on the subset of videos that are aligned with our approach's simulated training environments (navigation, indoor houses).
4.5 Ablation experiments
Next, we discuss some important ablations of our model design. We present full details of these experiments in Supp E and F, but we discuss the main conclusions here.
Importance of pose information: We measure the effect of pose embeddings on our local state prediction task. Our models show small improvements in predicting objects in the forward view  $(+0.7\mathrm{mAP})$  where scene information is directly visible, but large for other views that need to be inferred: mAP improvements of  $+2.2$  (right),  $+0.9$  (behind) and  $+1.0$  (left). Further, we directly embed ground-truth pose as part of the input and see benefits on both tasks on MP3D, but not on HouseTours, due to noise in extracted pose (compared to simulator-provided pose in MP3D).
Alternate pretraining task formulations: Next, we investigate alternate pretraining objectives instead of local state prediction including variants that only predict the object categories in each cardinal direction, but not the distances or that directly predicts the image features in each cardinal direction, among others. We find that our approach that requires predicting both object labels, orientations as well as rough distances offers a balance of both cues during pretraining, translating to strong downstream performance.
Hyperparameter ablations: We vary window size, memory size and the loss weighting term. We find that small window sizes are sufficient for localizing the room category for ROOMPRED, while larger windows are required for NLQ; and that the model is not very sensitive to the other parameters.
We proposed a framework to learn environment-aware representations in simulation and transfer them to video understanding tasks on challenging real-world datasets. Our approach outperforms state-of-the-art representations for predicting visited rooms and retrieving important moments from natural language queries, despite a significant sim-to-real gap. Despite its strengths, there are several opportunities for future work to improve our model further. These include incorporating 3D information into the local state task (currently defined in a 2D, top-down map), generating more human-like simulated videos and integrating more explicit approaches to tackle the sim-to-real gap.
Acknowledgements Thanks to Fu-Jen Chu and Jiabo Hu for help collecting the HouseTours annotations. UT Austin is supported in part by IFML NSF AI institute. KG is paid as a research scientist at Meta.
[79] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IROS, 2017. 3
[80] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012. 3
[81] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. In arXiv preprint arXiv:1807.03748, 2018. 3
[82] Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In CVPR, 2017. 3
[83] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 7, 8
[84] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. In CVPR, 2016. 3
[85] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016. 2
[86] Xiaolong Wang, Kaiming He, and Abhinav Gupta. Transitive invariance for self-supervised visual representation learning. In ICCV, 2017. 3
[87] Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle-consistency of time. In CVPR, 2019. 3
[88] Donglai Wei, Joseph J Lim, Andrew Zisserman, and William T Freeman. Learning and using the arrow of time. In CVPR, 2018. 3
[89] Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed video understanding. In CVPR, 2019. 2
[90] Chao-Yuan Wu and Philipp Krahenbuhl. Towards long-form video understanding. In CVPR, 2021. 1, 2
[91] Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In CVPR, 2018. Gibson dataset license agreement available at http://svl.stanford.edu/gibson2/ assets/GDS_agreement.pdf. 3, 14
[92] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173, 2022. 10
[93] Ye Yuan and Kris Kitani. 3d ego-pose estimation via imitation learning. In ECCV, 2018. 3
[94] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-based localizing network for natural language video localization. In ACL, 2020. 8, 9, 11, 12
[95] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. Learning 2d temporal adjacent networks for moment localization with natural language. In AAAI, 2020. 1, 2, 5, 9
[96] Chen Zhao, Ali K Thabet, and Bernard Ghanem. Video self-stitching graph network for temporal action localization. In ICCV, 2021. 1, 2, 5
[97] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. In TPAMI, 2017. 2, 8, 9