EPIC Fields: Marrying 3D Geometry and Video Understanding
Authors: Vadim Tschernezki
Ahmad Darkhalil
Zhifan Zhu
David Fouhey
Iro Laina
Diane Larlus
Dima Damen
Andrea Vedaldi
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Egocentric; Video; 3D

Abstract: Neural rendering is fuelling a unification of learning, 3D geometry and video understanding that has been waiting for more than two decades. Progress, however, is still hampered by a lack of suitable datasets and benchmarks. To address this gap, we introduce EPIC Fields, an augmentation of EPIC-KITCHENS with 3D camera information. Like other datasets for neural rendering, EPIC Fields removes the complex and expensive step of reconstructing cameras using photogrammetry, and allows researchers to focus on modelling problems. We illustrate the challenge of photogrammetry in egocentric videos of dynamic actions and propose innovations to address them. Compared to other neural rendering datasets, EPIC Fields is better tailored to video understanding because it is paired with labelled action segments and the recent VISOR segment annotations. To further motivate the community, we also evaluate two benchmark tasks in neural rendering and segmenting dynamic objects, with strong baselines that showcase what is not possible today. We also highlight the advantage of geometry in semi-supervised video object segmentations on the VISOR annotations. EPIC Fields reconstructs 96\% of videos in EPIC-KITCHENS, registering 19M frames in 99 hours recorded in 45 kitchens, and is available from: http://epic-kitchens.github.io/epic-fields

EPIC Fields
Marrying 3D Geometry and Video Understanding
Vadim Tschernezki **
Ahmad Darkhalil
Zhifan Zhu *+
David Fouhey
Tro Laina
Diane Larlus"
Dima Damen
Andrea Vedaldi
VGG, University of Oxford
University of Bristol
New York University
NAVER LABS Europe
Equal Contribution
Abstract
Neural rendering is fuelling unification of learning, 3D geometry and video understanding that has been waiting for more than two decades. Progress, however, is still hampered by lack of suitable datasets and benchmarks. To address this gap, we introduce EPIC Fields, an augmentation of EPIC-KITCHENN with 3D camera information. Like other datasets for neural rendering, EPIC Fields removes the complex and expensive step of reconstructing cameras using photogrammetry. and allows researchers focus on modelling problems. We illustrate the challenge of photogrammetry in egocentric videos of dynamic actions and propose innovations address them. Compared other neural rendering datasets, EPIC Fields is better tailored to video understanding because it is paired with labelled action segments and the recent VISOR segment annotations. also evaluate three benchmark tasks in neural rendering and segmenting dynamic objects, with strong baselines that showease what is not possible today. We also highlight the advantage of geometry in semi-supervised video object segmentations on the VISOR annotations. EPIC Fields reconstructs 96% of videos in EPIC- KITCHENS, registering 19M frames in 99 hours recorded in 45 kitchens, and is available from: http:/ /tap::///ppc/////eeeeeeeeeeee ki tichenes.... github. io/epic-fielde
further motivate the community, we
EPIC EPIC FIELSS calazet
Bendimaris
Figure 1: We propose EPIC Fields that extends EPIC-KITCHENS with 3D information, including full frame-rate camera pose trajectories (top). These are directly obtained from dynamic sequences of object interactions (sampled frames) without additional modalities or pre-scans. We showcase EPIC Fields through several benchmarks (bottom) that use the fusion of geometric and semantic cues.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Introduction
Recent breakthroughs in neural rendering [52, 33] have enabled a deeper integration of machine learning in geometric tasks like 3D reconstruction and rendering, creating opportunity to bring 3D geometry and video understanding closer together. By representing videos in 3D we can explain away the variability induced by the camera motion, which is dominant especially in egocentric videos. We can also integrate information extracted from each frame independently into a global, consistent interpretation of the video, demonstrated by semantic neural rendering [74, 58, 20, 68, 21, 12, 61]. However, such successes have been mostly limited to static scenarios, where only the camera moves. Indeed, 3D reconstruction still struggles with dynamic content and much work remains before we can have a 3D understanding of dynamic phenomena like actions and activities.
An obstacle to further progress in 3D video understanding is the lack of suitable development data. In this paper, we address this gap by introducing EPIC Fields, an extension of the popular EPIC- KITCHENS [5] dataset which adds reconstructed 3D cameras and new benchmark tasks assessing both 3D reconstruction and semantic video understanding.
We choose to build on EPIC-KITCHENS because it is an established benchmark for 2D video understanding with rich annotations. Furthermore, contains egocentric videos which are likely to benefit from 3D understanding, but which also challenge existing 3D reconstruction techniques due to their highly dynamic content and long duration (up to one hour). Dynamics include the motion the actor and of the objects that they manipulate, as well as object transformations (e.g. slicing carrot). Furthermore, most objects are mostly static, moving only during brief spells of active manipulation. These challenges push the limits of current dynamic 3D reconstruction methods, which are usually restricted to short videos or focus on one class of objects [44, 41, 39, 26, 22, 53].
Obtaining camera information for EPIC-KITCHENS is challenging since structure- structure-from-motii from-motion methods often fail on such complex videos. By solving this problem, we make it much easier for other researchers to start on 3D video understanding even when they are not experts in 3D vision, similar to what the setup and data introduced in works like NeRF [33] has done for static 3D reconstruction. Furthermore, while we propose specific benchmark tasks, we anticipate that researchers will be able to use our dataset to investigate many more questions than the ones we investigate here.
In summary, our first contribution is to augment PPIC-KITCHENS with camera information. To overcome the limitations [17] of traditional structure-
pipelines [48], which struggle with egocentric videos, we introduce pro-processing step that intelligently sub-samples frames from these videos, resulting in higher reconstruction reliability and speed. Our second contribution is to introduce new benchmark tasks that require or can benefit from the 3D cameras: dynamic novel view synthesis (i.e., reconstructing unseen frames given a subset of frames from a monocular video); identifying and segmenting objects that move independently from the camera, and semi-supervised video object segmentation. These benchmarks use and extend the VISOR [8] annotations to provide dense ground-truth semantic labels. We report number of baselines and conclude that, while 3D reconstruction can indeed benefit video understanding, existing approaches are challenged by the dynamic aspects of EPIC Fields.
2 Related work
Egocentric action understanding using 3D. Some egocentric datasets [38, 7. 15] contain static 3D scans of the recording locations. These typically do not contain actions, or the environments are scanned post-hoc, usually with an additional step. For instance, [38] uses stereo egocentric cameras, but no activities, and in [7, 15], reconstruction is done afterwards via hardware or additional dedicated scans. These scans are costly, which is why just 13% of Ego4D [15] data comes with a 3D scan. In contrast, we provide a pipeline for estimating camera poses from egocentric data without additional hardware or scans, which we demonstrate on an existing, challenging dataset SPIC-KITCHENS.
Inferring cameras in egocentric videos. In this work, we perform the challenging task of re- constructing 3D camera poses from egocentric videos that show dynamic activities from a single camera. Since the PITCENS [6] dataset is unscripted, the videos show natural interactions by participants in their homes, who act swiftly due to familiarity. Prior work [17, 35, 54] on these videos highlights the challenge. In [17], where ORB-SLAM was used find short clips where the camera pose was stable, the authors note that bundle adjustment failed and reconstructions lasted for

---

just 7 second intervals. Using [171, [35] found hot-spots, but commented that just 44% of the frames could be registered. Others have used additional hardware information; for instance, [54] proposed using IMU data to establish short-term trajectories. In contrast, this work shows how to reconstruct cameras for full videos in EPIC-KITCHENS, without additional assumptions, data, or hardware.
Multi-view videos. A different approach to enabling neural rendering is calibrated multiview setups. Many of these datasets, however, capture humans a "blank context", including HumanEva [51], Human3, 6M [19], AIST+ [60, 24], and ZJU-Mocap [42]. There are datasets capturing humans in complex environments, such as the Immersive Light Field dataset [1], NVIDIA Dynamic Scene Datasets [70], UCSD Dynamic Scene Dataset [29], and Plenoptic Video datasets [25]. However, these videos are short (1-2 min) and, due to the capture setup, show actions outside of their natural environment. In contrast, EPIC-KITCHENS is captured with an egocentric camera and shows long captures of indoor activities. Our contribution of reconstructing the cameras over time turns the egocentric data into the multiview data needed while retaining the naturalness of the data.
NeRF and dynamics. NeRF extensions to dynamic data can be roughly divided into approaches that add time as an additional dimension of the radiance fields [32. 59, 66, 13, 64, 25, 47, 2] and those that instead model explicitly 3D flow and reduce the reconstruction to a canonical (static) one [44, 39, 70, 40, 63, 57, 26, 9, 53, 18, 11, 27, 30]. While these methods demonstrate successes, their success depends on the dominance of camera motion over scene motion [14]. Scene motion by dynamic objects is not always common in existing datasets. Our proposed EPIC Fields contains both camera motion and fast continuous motion by the actor visible in the camera's 's field of view.
NeRF and semantics. Authors have already noted that neural rendering and 3D geometry can be helpful allies of video understanding. For instance, Semantic NeRF [74, 61] proposes to predict dense semantic labels in addition to RGB colours, while [21, 12, 50, 62] consider panoptic segmentations (things and stuff). [58, 20, 28] propose to fuse semantic features from pre-trained ViTs [3, 23, 56] into a neural reconstruction. [68, 73] represent scene as a composition of static objects given their 2D masks. Several studies employ neural rendering to separate scenes into objects and background either without or with weak supervisory signals [10, 67, 71, 59, 49, 37, 34, 65]. With a few exceptions [59, 28, 65], however, little work has been done on decomposing dynamic scenes into objects.
3 The EPIC Fields dataset
We introduce here the new EPIC Fields dataset. We first describe the content of the dataset and then the process of constructing it, including several technical innovations that made possible.
3.1 EPIC Fields in a nutshell
EPIC Fields extends EPIC-KITCHENS to include camera pose information. EPIC-KITCHENS contains videos of cooking activities collected using a head-mounted camera in 45 different kitchens. It has semantic annotations for fine-grained actions and their action-relevant objects, including 90K start-end times of actions [6]. VISOR [8] adds 272K manually annotated masks and 9.9M interpolated masks of hands and active objects. With EPIC Fields, we further contribute camera extrinsic parameters for each video frame as well as camera intrinsic parameters. Using the technique described in Section 3.2, successfully processed 671 videos spanning all 45 kitchens, resulting in 18,790,333 registered video frames with estimated camera poses.
Motivation. Our camera annotations facilitate reconstructing and interpreting videos in 3D. Figure 2 illustrates this point by mapping some 2D action annotations from EPIC-KITCHENS to the 3D space. Lifting annotations to 3D puts them in the wider context of the environment where actions occur, and enables studying the relevance of 3D egocentric trajectories to actions (for anticipation), objects (for understanding object state changes), and hand-object understanding. The figure also illustrates mapping hand meshes extracted using [46] to the 3D context of the kitchen.
Ethics, licensing, data protection. EPTC-KIS was collected with ethics approval by the University of Bristol and explicit consent from the participants. The data does not contain per- sonal identifiable information or offensive content and is provided under non-commercial license. EPIC Fields is released under the same terms.
3

---

Ploc
3
Ree
Figure 2: EPIC Fields unlocks applications that combine interactions with 3D information. We showcase examples of actions grounded in 3D (top row), and examples of integrating single-image 3D hands [46] into the kitchen reconstruction during interactions (bottom row).
3.2 Dataset construction
Because EPIC-KITCHENS videos were not collected with 3D reconstruction in mind they are difficult to reconstruct. For instance, they contain many dynamic objects: hands are visible in 95% of the frames and the focus of attention is often an object actively manipulated. Standard reconstruction pipelines operate under the assumption that the scene is static and are thus only moderately robust to dynamic objects. Other challenges include the video length (-9 mins on average) and the skewed distribution of viewpoints: videos alternate phases of small motion around hot-spots (e.g., cooking at a hob or washing the sink) and fast motion between hot-spots (e.g., moving the pot to the sink).
We address these challenges by: (1) filtering videos to reduce the number of redundant frames, computational cost, and skew; (2) using structure from motion (SIM) to reconstruct the scene from the filtered frames; (3) registering the remaining frames to the sparse reconstruction. We accept a video's reconstruction if 70% or more of its frames are registered successfully. In this manner, can reconstruct 96% of all EPIC-KITCHENS videos. We next describe each step, with details in the supplement.
Frame filtering. The goal of frame filtering is to downsample a vidco to reduce redundancy and skew while maintaining sufficient viewpoint coverage for accurate reconstruction. We filter frames by seeking temporal windows where frames have substantial visual overlap and then only keep one frame per window, similar to redundant frame mining [48, 55] and other SfM SLAM pipelines. Overlap between frames is measured by estimating homographies by matching SIFT features [31]. Given a homography H between two frames, we define their visual overlap F to be the fraction of the first frame area covered by the quadrilateral formed by warping the second frame corners by H. Windows are formed greedily, finding runs of frames (i + 1, ..... + k-) with overlap z 0.9 the first frame and discarding them. Filtering discards on average about 82% of frames in each video while also retaining sufficient number of frames in the critical transitions between hot-spots.
Sparse reconstruction. The filtered frames are fed to an off-the-shelf structure-from weo--frromtiinn pipeline. Among these, found COLMAP [48] to be more effective than MINS-MONO [45], which suffered from frequent drifts and restarts.
In Table 1 we analyse the effectiveness of the homography-based filtering algorithm by comparing to a na've filter that subsamples frames uniformly. We use 30 randomly selected videos for this experiment and report two standard SfM metrics [48]: the average reprojection error and the number of 3D points in the reconstruction. The first metric is a proxy for the accuracy of the reconstruction, and the second for its coverage. Both filtering techniques reduce the number of frames equally and thus result in similar computational complexity. However, nomography-basee
filtering also addresses the skew and results in a significantly better success rate, increased coverage, and reduced reprojection error compared to uniform subsampling. Besides considering the number of points reconstructed, Figure 3 shows qualitatively the notably improved coverage obtained by nomography-basee filtering.
Dense reconstruction, automated verification, and restart. After obtaining the sparse reconstruc- tion from the filtered subset of video frames, we use COLMAP to register the remaining frames against it, which is computationally cheap. We accept the final reconstruction if 20% of the video's frames, at full frame rate, are registered successfully. This process succeeds in 90% (631 vidcos)

---

Table Impact of frame filtering on the reconstruction quality. We compare the sparse reconstruc- tion of 30 videos using either homography- or uniform frame filtering, Naïve uniform sampling results in only 27 of the videos being reconstructed successfully (i.e, dense registration rate 70%). Furthermore, the successful reconstructions have significantly reduced coverage (-16.64%)
and increased reprojection error (+4.76%) compared to homography- based filtering.
Avg. #3D Points
Avg. Repr.
Avg. Reg. Rate
Successful
Frame sampling
Error
Reconstructions
Homography- based (ours)
27,763
0,798
98.6%
30/30
Uniformly Relative change
23,142
0,836
89.0%
27/30
-16.64%
4.76%
9.77%
-10 %
Uniform sampling
Our sampling
Figure 3: 3D reconstructions with different sampling. We compare three scenes reconstructed using either uniform frame selection homography-bast pipeline. Uniform sampling yields partial reconstructions with limited coverage. Ours demonstrates superior performance, resulting in better coverage by registering successfully more viewpoints.
of cases. When video is rejected, the reconstruction process is attempted again with higher threshold F 2 0.95; this usually doubles the number of frames that COLMAP needs to process for the reconstruction, but increases the success rate to 96%. We discuss reasons for the failure of the last 29 EPIC-KITCHENS videos the supplement.
Application to other egocentric videos. While we developed our reconstruction pipeline by con- sidering the SPIC-KITCHENS data, the approach we obtained is general and applies equally well to other egocentric video collections such as Ego4D [16]. at least for indoor locations. We give examples of these reconstructions in the supplement.
4 The EPIC Fields benchmarks, experiments and results
We define three benchmarks EPIC Fields that probe 3D video understanding. Annotations, evaluation code and baselines are released as part of EPIC Fields; further details are in the supplement.
4.1
Dynamic New-View Synthesis (D-NVS)
Given a subset of video frames input, the goal of dynamic new-view synthesis (D-NVS) is to predict other video frames given only their timestamps and camera parameters. While other D-NVS benchmarks exist, EPIC Fields is more challenging due to the first-person perspective and the large number of dynamic objects. In Table we compare EPIC Fields to commonly used datasets in D-NVS. EPIC Fields offers significant step up in complexity and scale with significantly longer videos and associated semantics. For detailed statistics, please refer to the supplement.
Video selection. Due to the computational cost of most D-NVS algorithms, we limit the D-NVS benchmark to a subset of 50 videos (14.7 hours and 2.86M registered frames) extracted from the train/val set of VISOR [8] (this selection includes 96. 1 % of the frames annotated in VISOR).
5

---

Table 2: Comparison of datasets commonly used in dynamic new-view synthesis.
Dataset
#Scenes
Seq. Length
Monocular
Semantics
Nerfies [40]
8-15 sec
X
X
D-NeRF [44]
1-3 sec
X
x
Plenoptic Video [25]
10-60
X
X
NVIDIA Dynamic Scene Dataset [70]
1-5 sec
4/
X
HyperNeRR [41]
16
8-15 sec
13 16
X
iPhone [14]
14
8- 15 sec
7/14
X
SAFF [28]
1 1-5sec
X
V
EPIC Fields [D-NVS] (ours)
50
6-37 min (Avg 22)
50/ 50
V
Iovadiin Out-u--adi Ouooaattoon
(Eoep)
H
(1
Figure 4: Definition of the three difficulty levels for the task of dynamic new-view synthesis. Validation and test frames selected to meet three reconstruction difficulty levels. In-Action frames (Hard) happen during an action and are harder to reconstruct due to the dynamics. Out-of-Action (Medium) frames happen outside action, but far from train frame. Out-of-Action (Easy) frames are near train frames. Frames in a bounding box (orange) represent either val/test frames. Frames marked with cross are discarded create larger time gap around each val/test frame (medium and hard levels). All other frames can be used for training.
Frame selection. For each video in the D-NVS benchmark, select the video frames to be used as input to the system (training) and those that remain unseen and are used for evaluation only (validation/testinn)) Specifically, we propose categorising evaluation frames into three tiers of difficulty (easy, medium, hard visualised in Figure 4), determined by the type of motion and the temporal gap between the evaluation and training frames. In-Action frames correspond to common 'put', "take", and "cut' actions annotated in SPICCKITCHENS, based on their start-stop times; they characterised by substantial object motion due hand-object interactions and are thus more difficult to reconstruct. In pursuit of a greater challenge, for the In-Action (Hard) set of frames, exclude frames from the training occurring within second of test frame. Out-of-Action frames occur outside action segments, where there is no appreciable motion except for the camera, making these frames generally easier to reconstruct. For the Out-of-Action (Medium) set, we sample 70% of the out-of-action frames with the same time gap as above. The Out-of-Action (Easy) set corresponds the remaining 30% without removing the neighbouring training frames. The reasoning is that it is generally easier to predict frame temporarily close to a training one. We assign every other evaluation frame to the validation and test sets, respectively. The average time gap between consecutive evaluation frames is 3.73 seconds. Further statistics are provided in the supplement.
Benchmark methods. To demonstrate how EPIC Fields be used and probe the limits of the state of the art in such challenging scenarios, we consider three neural rendering approaches: NeRF-W [32], NeuralDiff [59], and T-NeRF+, an extended version of T-NeRF [14].
NeuralDiff [59] is a method tailored to egocentric videos. It uses three parallel streams to separate the scene into the actor, the transient objects (that move at some point in the video), and the background that remains static. We combine the predictions of the actor and transient objects to predict our dynamic and semi-static objects, which will be relevant in Section 4.2.
NeRF-W [32] augments NeRF with the ability to 'explain' photometric and environmental (non- constant) variations by learning low-dimensional latent space that can modulate scene appearance and geometry. As a result, NeRF-W also separates static and transient components. We follow the modification from [59] to render NeRF-W applicable to vidco frames and the D-NVS task.

---

mask for each frame in Table 3: Dynamic new-view synthesis. We compare different neural rendering approaches for frames from different difficulty levels (easy, medium, hard). We report PSNR considering all pixels in each test frame. Given the mask annotations from VISOR for In-Action frames, we also report PSNR on
background (BG) and foreground (FG) pixels separately for the hard (In-Action) setting.
Hard
Method
Easy
Medium
All
BG
FG
NeRF-W [32]
21.13
19.3
17.93
18.99
13.54
T-NeRF+ [14]
21.58
19.81
18.44
19.73
13.74
NeuralDiff [59]
22.14
19.88
18.36
19.54
13.37
T-NeRF+ [14] was proposed baseline to evaluate state-of-the-art NeRFs on dynamic scenes. was shown to outperform other methods in terms of the quality of the synthesised images. We extend T-NeRF by adding another stream to the cine-conditioned NeRF architecture that models the background (static parts of the scene).
Results. To measure performance on this task, we report the Peak Signal-to-Noiss Ratio (PSNR) of the test frame reconstructions, which is a proxy for the quality of the underlying 3D reconstructions with the key advantage of not requiring. 3D ground-truth for evaluation. We report results in Table 3 for the three levels of difficulty. There a strong relationship between PSNR and difficulty: PSNR is consistently lower for all methods when rendering views during actions (hard) compared to outside actions (medium, easy). Some limitations of rendering these hard test frames are shown in Figure 5. For example, the bottom row shows that no 3D baseline renders the person's 's arm correctly, since all models struggle to interpolate the person's movement between frames. We further observe a significant gap in rendering quality if we calculate PSNR separately for foreground and background regions. We use the VISOR annotations of hands and active objects for In-Action frames to obtain this separation. These results not only highlight the existing limitations of current methods but also offer a valuable benchmark for assessing potential improvements in a targeted manner.
4.2 Unsupervised Dynamic Object Segmentation (UDOS)
The goal of Unsupervised Dynamic Object Segmentation (UDOS) is to identify which regions in each frame correspond to dynamic objects. This task can be approached in 2D only but a good proxy to assess 3D methods well, and can, in fact, be boosted by 3D modelling. Here, we extend the setting introduced in [59], using 20x more data and adopting a more nuanced evaluation protocol.
Video and frame selection. We use the same selection of videos as for the D-NVS task, but only use the In-Action frames with VISOR annotations, as they provide ground-truth dynamic object segmentations. We convert VISOR masks into a srreground-backgroung three ways, depending on objects that are currently moving, or those that have moved at a different time in the video. In the dynamic objects only setting, the foreground contains hands and other visible body parts as well as object masks only for objects that are currently being moved. We use the VISOR contact annotations to identify these objects and augment these with additional manual masks for visible body parts including torsos, legs, and feet. More details are in the supplement. In the semi-static only setting, consider only objects that moved at some point during the video, but not during the current frame. We select these objects by watching the video and identifying all objects that have moved at least once. VISOR contains annotations of these objects only on frames where they are considered active. We employ an automated method to propagate the annotations to cover all frames, resulting in set of semi-static object masks. This is the complete set of masks for all objects that have moved at any point in the video, even if they are temporarily static. More details can be found in the supplement. We combine both to report the dynamic and semi-static setting.
Benchmark methods. We use NeuralDiff and NeRF-W from the NVS task, since, by design, they decompose scenes into static and dynamic components. Additional considerations are necessary to make T-NeRF+ applicable to UDOS. In order to disentangle the modelling of both radiance fields in terms of temporal variation, we apply the uncertainty modelling from [32] model change in observed colours of pixels that occur due to dynamic effects inside the scene. This extension enables T-NeRF+ learn decomposed radiance field.
We also consider a 2D baseline, Motion Grouping (MG) [69], a state-of-the-art method for self- supervised video object segmentation. It trains segmentation model using an autoencoder-like
7

---

Ground Truth
NeRF-W
T-NeRF
NeuralDiff
Ol--O--- Action (Easy)
Our- Of Aetionn (Mediiu
In- Aetion (Hard)
Figure 5: Dynamic new-view synthesis. We compare the outputs of 3D methods NeRF-W [32], T-NeRF+ [14], and NeuralDiff [58], for novel viewpoints, across three different complexity levels. The predictions more accurate with less difficult motion shown in the first and second row. The task becomes more challenging for our hard samples.
framework. The model has two output layers; layer represents the background. and the other layer identifies one or more moving objects the foreground, including their opacity layers. These layers are then linearly composed and optimised reconstruct the input image. Since this approach unsupervised, it can be compared fairly to the 3D baselines for this task.
Results. To evaluate performance, we measure 2D segmentation accuracy on test frames using mean average precision (mAP) as in [59]. Table 4 compares unsupervised 2D baselines and 3D baselines. Depending on the type of observed motion, 3D-based methods offer advantages over 2D methods and vice versa. For example, 3D-based methods are better suited for discovering semi-static objects that are not currently in motion, i.e., they have been moved at different times within video. This is evident by the improved segmentation performance when considering this type motion (i.e., SS+D and purely SS). However, note that none of the 3D-based methods explicitly consider motion. Consequently. MG, which takes as input optical flow, performs better on purely dynamic motion, but struggles to segment objects that are temporarily not moving, This drawback of 3D-based methods, compared to 2D motion-basee methods, underscores the current challenge in capturing dynamics in neural rendering. Addressing this limitation is open question for future research.
Figure shows qualitative results. We observe that MG performs particularly well on objects that are constantly in motion, for example, the moving body parts of the person. Among the 3D methods, NeuralDiff is better at capturing dynamic objects, and, unlike MG, both NeuralDiff and T-NeRF+ are able to segment various semi-static objects well since they do not rely continuous motion.

---

Table 4: Unsupervised dynamic object segmentation. We report the mean average precision (mAP) on segmenting the semi-static (SS) and dynamic components of the scene, and also their union (SS+Dyn). All methods are trained without explicit supervision, Le., masks are used during training, only for evaluation.
Method
3D
SS+Dyn
Dynamic
MG [69]
X
55.53
12.78
64.27
NeRF-W [32]
/
45.62
20.97
28,52
T-NeRF+ [14]
64.91
24.48
44.27
NeuralDiff [59]
/
69.74
25.55
55.58
MG (2D)
NeRF-W (3D)
+-NeRF+ (3D)
NeuraDDiff (3D)
SS Dyn
SS
Dynamic
Figure 6: Unsupervised dynamic object segmentation. We compare three 3D baselines (NeRF- W [32], T-NeRF+ [14] and NeuralDiff [58]) and one 2D baseline (MG [69]) with three motion types. The 2D baseline captures the person and does well on the purely dynamic (short-range motion) setting. 3D models do this and also segment semi-static (SS) components (long-range motion, i.e., objects that were moved some time ago). DynDyn, the evaluation includes SS and dynamic components.
4.3 Semi-Supervised Video Object Segmentation (VOS)
Semi- Supi--Suee Video Object Segmentation (VOS) is standard supervisee video understand- ing task: given the mask for one or more objects in a reference frame, the goal is to propagate the segments to subsequent frames. For this task, we the train/val splits published as part of the VISOR VOS benchmark (See [8] Sec. 5. VOS is usually approached by using 2D models. Here, explore how the 3D information in EPIC Fields can be used for it instead.
Benchmark methods. We evaluate two naïve baselines for this task, one in 2D and another in 3D. For completeness, also compare these to existing, trained 2D VOS models.
Fixed 2D. We make the assumption that the pixels the first frame remain constant throughout the entire sequence. This naïve baseline is prone to failure when the camera undergoes movement.
Fixed in 3D. To better understand the potential of 3D information for VOS, compare the 2D baseline above to a 3D one. In the 3D baseline, an object mask is projected to 3D and its position in 3D is fixed throughout the sequence. The mask is then re-projected other frames using the available camera information. This works well for static objects and achieves two effects. First, objects can be reliably tracked over occlusions. Second, detecting when these objects are in or of view is by-product of estimated camera poses.
Trained 2D models. We also evaluate two state-of-the-ar models for video object segmentation, STM [36] and XMEM [4]. These are trained on the train set of VISOR.
Results. We compare the baselines on the VISOR benchmark using the evaluation metrics defined in [43] which are the region similarity J and contour accuracy F. We also distinguish the set of objects that are static, such as 'rridge. 'floor", and 'sink', and report the above metrics separately for these and all other movable objects (SS+Dyn). Table 5 shows the results where the Fixed in 3D clearly outperforms the Fixed in 2D by significant margin for the anticipated static objects

---

Table 5: Semi-Supervised VOS. We compare naive baselines in 2D and 3D, as well pretrained/fine tuned models on static and dynamic objects on the validation set of VISOR VOS. two videos from the validation set are excluded they don't 't have successful reconstructions.
VISOR VAL[8]
Static
SS
Dyn
Method
J&F
J
J&F
J
F
J&F
J
FF
Fixed in 2D Fixed in 3D
12.5
13.4
11.6
17.8
23.8
11.6
12.0
11.9
12.0
/
31.3
30.5
32.2
48.4
52.2
44.6
29.6
27.8
31.5
Pretrained STM
63.0
60.8
65.2
65.4
63.1
63.7
65.5
Fine-tuned STM
X
76.4
74.2
78.6
76.8
77.7
76.0
76.6
73.8
79.5
Pretrained XMEM
X
64.0
61.5
66.4
63.2
64.0
62.5
64.1
61.1
67.1
Fine-tuned XMEM
X
77.3
75.2
79.4
77.0
77.7
77.4
78.0
75.3
80.7
Fixed
Fixed
XMEM-pretrained
XMEM-finetuned
Reference
Figure 7: Semi-Supervised Video Object Segmentation. We compare our baselines on frames from the same sequence. The Fixed in 3D baseline can track the bin over camera motion and recognise in/out-of view. Pretrained models usually suffer from false positives in the out-of view scenes.
(430.6%) but also improves results for the remaining semi-static and dynamic 17.6%) objects. This is because such objects do remain unmoved for some duration of the videos. This highlights the additional value derived from representing objects in 3D. Figure visualises one example where the bin is successfully propagated using the Fixed in 3D baseline, including when out of view. The pretrained models struggle to propagate masks for the novel objects in the dataset or for masks that go out of the scene. These are cases that the Fixed in 3D baseline successfully handles. However, the fine-tuned models are quantitatively and qualitatively superior as they are trained on the dataset. No prior work has utilised 3D information along with learnt models for the task of semi-supervised VOS. We hope our novel benchmark can trigger new VOS approaches that tackle the combined challenge of keeping track of static objects in 3D and dynamic objects through trained propagation of objects during motion and transformations.
Conclusions
We introduced EPIC Fields, dataset study 3D video understanding. We addressed the difficult challenge of reconstructing cameras in EPIC-KITCHENS videos, introducing filtering and other tech- niques that are portable other similar reconstruction scenarios. Using these pre-computed cameras facilitates working on 3D video understanding even without significant expertise in photogrammetry.
With EPIC Fields also defined three benchmark tasks: dynamic new-view synthesis, unsupervised dynamic object segmentation, and video object segmentation. Our results show that the performance of state-of-the-art dynamic neural reconstructioon/eendriin methods strongly depends motion. particular, the gap in reconstruction quality hetween the dynamic and the static parts the videos show that there is ample margin for further improvements in the handling of dynamic objects. Similar findings apply to the segmentation of objects, where 3DD-based models can assist unsupervised video object segmentation and propagate masks of static objects over time. We hope that these results, the proposed benchmark data and code (comprising evaluation, camera reconstruction, and baselines) will assist the community in investigating further methods that combine geometry and video understanding.
the type of
Societal impact. While expect that benchmark will lead to positive impact, including applications to augmented and mixed reality including AR assistants, there are potential negative impacts well: better AR may be used for deception and many capabilities powering assistant may also aid surveillance.
10

---

Acknowledgements Project supported by EPSRC Program Grant Visual EP/T028257///. A. Dark- halil is supported by EPSRC DTP program. Z. Zhu is supported by UoB- Scholarship. I. Laina and A. Vedaldi are supported by HRC-CoG UNION 101001212. D. Damen is supported by EPSRC Fellowship UMPIRE EP/T004991/1. D. Fouhey supported by a NSF CAREER #2142529.
We thank Sanja Fidler and Amlan Kar, from the University of Toronto, for contributions to the initial idea and discussions of this project. We also thank Rhodri Guerrier for assisting in manual annotations of the test set.
References
[1] Michael Broxton, John Flynn, Ryan S. Overbeek, Daniel Erickson, Peter Hedman, Matthew DuVall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul E. Debevec. Immersive light field video with a layered mesh representation. ACM Trans. on Graphics (TOG), 2020.
[2] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
2
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[4] Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with an aakinson-shiffre memory model. In Proceedings of the European Conference on Computer Vision (ECCV), pages 640-658. Springer, 2022. 4.3
[5] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler. Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchene dataset. In Proceedings of the European Conference on Computer Vision (ECCV), 2018. 1
[6] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evan- gelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for ppic-kitchens-100 International Journal of Computer Vision (ICV), 2022. 2, 3.1
[7] Dima Damen, Teesid Leelasawassuk, Osian Haines, Andrew Calway, and Walterio Mayol- Cuevas. You-do, i-learn: Discovering task relevant objects and their modes of interaction from multi-user egocentric video. In Proceedings of the British Machine Vision Conference (BMVC),
2014. 2
[8] Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitchens visor benchmark: Video segmentations and object relations. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022. 3.1, 4.1, 4.3
[9] Yilun Du Yinan Zhang, Hlong-Xing Yu, Joshua B. Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. Proceedings of the International Conference
on Computer Vision (ICCV), 2021. 2
[10] Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. NeRF- SOS: Any-view sulf-supe object segmentation complex scenes. In Proceedings of the
International Conference on Learning Representations (ICIR), 2023. 2
[11] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In Proceed-
ings of SIGGRAPH, 2022. 2
[12] Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, An- dreas Geiger, and Yiyi Liao. Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. In Proceedings of the International Conference on 3D Vision (3DV), 2022. 1, 2 [13) Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the International Conference on Computer Vision (ICCV), 2021. 2
[14] Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dynamic view synthesis: A reality check. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022. 2, 2, 4, 6
[15] Kristen Grauman, Andrew Westbury, Fugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger. Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world
11

---

in 3,000 hours of egocentric video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[16] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagara- jan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Erie Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Frapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang. Yifei Huang, Wenqi Jia, Weslic Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano. Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi. Yunyi Zhu, Pablo Arbe- laez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3,000 hours of egocentric video, 2021. 3.2
[17] Jiaqi Guan, Ye Yuan, Kris M Kitani, and Nicholas Rhinehart. Generative hybrid representations for activity forecasting with no-regret learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 1,2
[18] Xiang Guo, Guanying Chen, Yuchao Dai, Xiaoqing Ye, Jiadai Sun, Xiao Tan, and Errui Ding. Neural deformable voxel grid for fast optimization of dynamic view synthesis. In Proceedings
of the Asian Conference on Computer Vision (ACCV), 2022. 2
[19] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6M, Large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 2014. 2
[20] Sosuke Kobayashi, Eicch Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022. 1,2
[21] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J. Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas A. Funkhouser. Panoptic neural fields: A semantic object-aware neural scene representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1,2
[22] Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. Neural human performer: Learning generalizable radiance fields for human performance rendering. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). 2021.
[23] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language- driven semantic segmentation. In Proceedings of the International Conference on Learning
Representations (ICLR), 2022. 2
[24] Ruilong Li. Shan Yang, David A. Ross, and Angjoo Kanazawa. Learn to dance with +IST++: Music conditioned 3d dance generation. In Proceedings of the International Conference on Computer Vision (ICCV), 2021. 2
[25] Tianye Li, Mira Slavcheva, Michael Zollhöfer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard A. Newcombe, and Zhaoyang Lv. Neural 3D video synthesis from multi-view video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 2,2
[26] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 1,2
[27] Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2023. 2
[28] Yiqing Liang, Eliot I Laidlaw, Alexander Meyerowitz, Srinath Sridhar, and James Tompkin. Semantic attention flow fields for monocular dynamic scene decomposition. In Proceedings of the International Conference on Computer Vision (ICCV), 2023. 2
12

---

[29] Kai-En Lin, Lei Xiao, Feng Liu, Guowei Yang, and Ravi Ramamoorthi. Deep 3d mask volume for view synthesis of dynamic scenes. In Proceedings of the International Conference on Computer Vision (ICCV), 2021. 2
[30] Jia-Wei Liu, Yan-Pei Cao, Weijia Mao, Wengiao Zhang, David Junhao Zhang, Jussi Keppo, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Devrf: Fast deformable voxel radiance fields for dynamic scenes. arXiv.cs, 01122205.17222. 2022. 2
[31] David G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal
of Computer Viston (IJCV), 60(2), 2004. 3.2
[32] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Doso- vitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2021. 2, 4.1, 4.2, 5, 6
[33] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[34] Ashkan Mirzaei, Yash Kant, Jonathan Kelly, and Igor Gillischenski. Laterf: Label and text driven object radiance fields. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
[35] Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman. Ego-topo: Environment affordances from egocentric video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2
[36] Seoung Wug Oh, Joon- Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation us- ing space-time memory networks. In Proceedings of the International Conference on Computer Vision (ICCV), pages 9226-9235, 2019, 4.3
[37] Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2021. 2
[38] Hyun Soo Park, Jyh-Jing Hwang, Yedong Niu, and Jianbo Shi. Egocentric future localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
2016. 2
[39] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, Steven M. Scitz, and Ricardo -rutinaBruuala.... Deformable neural radiance fields. CoRR, abs/2011.12945. 2020. 1.2
[40] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Bruaala Nerfies: Deformable neural radiance fields. In Proceedings of the International Conference on Computer Vision (ICCV), 2021. 2,2
[41] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, Ricardo Martin-Brualla and Steven M. Seitz. Hypernerf: a higher-dimensiooal representation for topologically varying neural radiance fields. ACM Trans. on Graphics (TOG),
40(6), 2021. 1.2
[42] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. CORR, abs/2012.155835 2020. 2
[43] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung and Luc Van Gool. The 2017 davis challenge video object segmentation. arXiv preprint arXiv:1704.0 00770.. 2017. 4.3
[44] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2021. 1, 2,2
[45] Tong Qin, Peiliang Li, and Shaojie Shen. Vins-mono:
robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics, 2018. 3.2
[46] Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmocap: A monocular 3d whole-body pose estimation system via regression and integration. In IEEE International Conference Computer Vision Workshops, 2021. 3.1,2
[47] Sara Fridovich- Keil and Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
2023. 2
13

---

[48] Johannes Lutz Schönberger and fan-Michael Frahm. Structure from--ootin revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 3.3.2,3.2 3.2
[49] Prafull Sharma, Ayush Tewari, Yilun Du, Sergey Zakharov, Rares Andrei Ambrus, Adrien Gaidon, William T. Freeman, Fredo Durand, Joshua B. Tenenbaum, and Vincent Sitzmann. Neural groundplans: Persistent neural scene representations from single image. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. 2
[50] Yawar Siddiqui. Lorenzo Porzi, Samuel Rota Bulò, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR),
2023.7 2
[51] Leonid Sigal, Alexandru O. Balan, and Michael J. Black. HumanEva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. International Journal of Computer Vision (ICV), 2010. 2
[52] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: neural scene representations. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2019.
Continuous SD-structure-awerr
[53]
Lianghhen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics, 2023. 1,2
[54] Shuhan Tan, Tushar Nagarajan, and Kristen Grauman. Egodistill: Egocentric head motion distillation for efficient video understanding. arXiv preprint arXiv:2301 1,22/7, 2023.
[55] Chengzhou Tang, Oliver Wang, and Ping Tan. Gslam: Initialization monocular visual In Proceedings of the International Conference on 3D
slam via global
Vision (3DV), 2017. 3.2
[56] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data- nata-efficient image transformers & distillation through attention. In Proceedings of the International Conference on Machine Learning (ICML), 2021. 2
[57] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video. In Proceedings of the International Conference on Computer Vision (ICCV), 2021. 2
[58] Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural Feature Fusion Fields:
3D distillation of self-supervised 2D image representation. In Proceedings of the International Conference on 3D Vision (3DV), 2022. 2,
[59] Vadim Tschernezki, Diane Larlus, and Andrea Vedaldi. NeuralDiff: Segmenting 3D objects that move in egocentric videos. In Proceedings of the International Conference on 3D Vision (3DV), 2021. 2, 2, 4.1, 4.2
[60] Shuhei Tsuchida, Satoru Fukayama, Masahiro Hamasaki, and Masataka Goto. AIST dance video database: Multi-genre, multi-dancer, and multi-camera database for dance information processing. In Proc. ISMIR, 2019. 2
[61] Suhani Vora, Noha Radwan, Klaus Greff, Hemning Meyer, Kyle Genova, Mehdi S. M. Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. NeSF: Neural semantic fields for generalizable semantic segmentation of 3D scenes. arXiv.cs, abs/2111.13266 2021. 1,2
[62] Bing Wang, Lu Chen, and Bo Yang. Dm-nerf: 3d scene geometry decomposition and manipula- tion from 2d images. arXiv preprint S.Xiv:2208.072222 2022. 2
[63] Chaoyang Wang, Ben Eckart, Simon I Lucey. and Orazio Gallo. Neural trajectory fields for dynamic novel view synthesis. arXiv.cs, abs/2105. 10555904. 2021. 2
[64] Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Fourier PlenOctrees for dynamic radiance field rendering in real-time. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2
[65] Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli. D² nerf: Self-supervised decoupling of dynamic and static objects from a monocular video. arXiv preprint arXiv:2205. 15838, 2022. 2
[66] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 2
14

---

[67] Christopher Xie, Keunhong Park, Ricardo Bartin-Brualla, and Matthew Brown. Fig-nerf: sngure-ground neural radiance fields for 3d object category modelling. In Proceedings of the International Conference on 3D Vision (3DV), pages 962-971. IEEE, 2021. 2
[68] Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning composiioooi neural radiance field for editable scene rendering. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
1.2
[69] Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. In Proceedings of the International Conference
on Computer Vision (ICCV), 2021. 4.2, 6
[70] Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from monocular camera. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2020. 2, 2,
[71] Hong-Xing Yu, Leonidas Guibas, and Jiajun Wu. Unsupervised discovery object radiance fields. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
[72] Wentao Yuan, Zhaoyang Lv, Tanner Schmidt, and Steven Lovegrove. STaR: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 2
[73] Xiaoshuai Zhang, Abhijit Kundu, Thomas Funkhouser, Leonidas Guibas, Hao Su, and Kyle Genova. Nerflets: Local radiance fields for efficient structure-aware 3d scene representation from 2d supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2023. 2
[74] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene la- belling and understanding with implicit scene representation. In Proceedings of the International Conference Computer Vision (ICCV), 2021. 1,
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We believe that our abstract and introduction accurately describe the work.
(b) Did you describe the limitations of your work? [Yes]] We have discussed the limitations in the experiments. Indeed, one of the points of our paper is to show current limitations in existing work.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We have discussed potential negative impacts in the conclusions of the paper.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] There are no theoretical results in the paper.
(b) Did you include complete proofs of all theoretical results? [N/A] There are no theoreti- cal results in the paper.
3. If you ran experiments (e.g., for benchmarks)...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No] We include all all of the data and instructions needed. We will release all code for our benchmarks no later than the publication date.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We provide information about the data splits and hyperparameters in the main paper and supplement.
(c) Did you report error bars (e-g., with respect to the random seed after running experi- ments multiple times)? [No] Our experiments are too computationally expensive to run
15

---

multiple times to produce error bars. Our models are meant as baselines that subsequent methods ought show improvement on.
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We provide a detailed documentation the compute cost of each of the steps in the supplement. Some steps, such the registration of all cameras, and providing the information to the community, we can collectively save substantial overall compute time across the community.
costly. However, believe that by doing them
4. If you are using existing assets (e.g.. code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We build exclusively on the EPIC-KITCHENS 100 dataset. Our use of the dataset is made clear throughout the paper.
(b) Did you mention the license of the assets? [Yes] When introducing EPIC-KITCHENS, we introduce its license. Our new assets will be released under the same license.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Yes, provide additional assets that will provide to the reviewers privately and then share publicly upon acceptance.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] When introducing EPPC-KITCHENS, we mention that was collected with ethics approval and with clear consent from the people who are in the data.
(e) Did you discuss whether the data you are using/curating contains personally identi- fiable information or offensive content? [Yes] When introducing EPIC-KITCHENS, we discuss that the data does not contain PII, was reviewed by participants prior to publication, and does not contain offensive content.
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] We did not do any crowdsourced annotation.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] There no participant risks for this paper. The dataset we use, however, was collected with ethics board approval.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] We did have any paid participants or workers.
16


Reviews:
[{"review_id": "MaPPiQEksz", "reviewer": "Reviewer_ZiM7", "summary": "", "questions": "", "limitations": "Please see the \"Opportunities For Improvement\". It would be great if you could explain more comparing with Ego4D", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1) This paper is very helpful for the whole research community as it provides the camera poses + some other additional information for the kitchen-visor dataset which lift large-scale 2D annotations to 3D. This would be essential to study of 3D egocentric trajectories concerning actions (for anticipation), objects (for understanding object state changes), and the interactions between hands and objects.\n2) The proposed three benchmarks (dynamic new-view synthesis, unsupervised dynamic object segmentation, video object segmentation), and the related codebase are valuable for further exploration of 3D and video understanding.\n3) The paper highlights several technical innovations that were crucial in constructing the EPIC Fields dataset. These innovations likely address challenges in processing egocentric videos, ensuring that the resulting dataset is of high quality and reliable for researchers to use.  It also successfully processes a substantial amount of data, including 671 videos spanning 45 different kitchens, resulting in over 18 million registered video frames with estimated camera poses\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ivOLqJk0Dt", "reviewer": "Reviewer_Pv5q", "summary": "", "questions": "", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Simple yet effective technique to solve for camera poses in egocentric long duration videos.\n- The outcome (the augmented EPIC Kitchen dataset) is useful for the community to study dynamic 3D scene understanding.\n- Interesting to see that in the UDOS benchmarking results, 3D methods indeed do better in semi static movable object segmentation.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "dV79IOFPUM", "reviewer": "Reviewer_cueE", "summary": "", "questions": "", "limitations": "See above \"Opportunities For Improvement\". 1 and 3 can be moved here.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1) Providing the camera poses in the second large dataset in egocentric vision is indeed beneficial. \n2) The three new benchmarks can be a test bed for the novel methods in the future.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "BAxcnRuzfm", "reviewer": "Reviewer_FW6P", "summary": "", "questions": "", "limitations": "The authors clearly addressed the societal impact of this work. better AR may be used for deception and many capabilities powering an assistant may also aid surveillance.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The authors focus on the significance of 3D information in video understanding datasets. A novel pre-processing step is applied to the classic Structure-from-Motion method in order to acquire reliable reconstruction quality.\n\nIn Section 4, the authors list three important tasks and show advantages of using 3D camera information in these tasks. These results, the proposed benchmark, and the codebase will provide valuable insights and an experimental playground to further investigate methods that combine geometry and video understanding.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: All four reviewers liked the dataset and the proposed tasks and recommended acceptance.  The AC agrees.  The authors are encouraged to revise the submission based on the reviews in the camera-ready version.
Decision: Accept (Poster)