SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Video and frame selection. We use the same selection of videos as for the D-NVS task, but only use the In-Action frames with VISOR annotations, as they provide ground-truth dynamic object segmentations. We convert VISOR masks into a foreground-background mask for each frame in three ways, depending on objects that are currently moving, or those that have moved at a different time in the video. In the dynamic objects only setting, the foreground contains hands and other visible body parts as well as object masks only for objects that are currently being moved. We use the VISOR contact annotations to identify these objects and augment these with additional manual masks for visible body parts including torsos, legs, and feet. More details are in the supplement. In the semi-static only setting, we consider only objects that moved at some point during the video, but not during the current frame. We select these objects by watching the video and identifying all objects that have moved at least once. VISOR contains annotations of these objects only on frames where they are considered active. We employ an automated method to propagate the annotations to cover all frames, resulting in a set of semi-static object masks. This is the complete set of masks for all objects that have moved at any point in the video, even if they are temporarily static. More details can be found in the supplement. We combine both to report the dynamic and semi-static setting.
Benchmark methods. We use NeuralDiff and NeRF-W from the NVS task, since, by design, they decompose scenes into static and dynamic components. Additional considerations are necessary to make  $T$ -NeRF+ applicable to UDOS. In order to disentangle the modelling of both radiance fields in terms of temporal variation, we apply the uncertainty modelling from [32] to model a change in observed colours of pixels that occur due to dynamic effects inside the scene. This extension enables  $T$ -NeRF+ to learn a decomposed radiance field.
We also consider a 2D baseline, Motion Grouping (MG) [69], a state-of-the-art method for self-supervised video object segmentation. It trains a segmentation model using an autoencoder-like
Figure 5: Dynamic new-view synthesis. We compare the outputs of 3D methods NeRF-W [32], T-NeRF+ [14], and NeuralDiff [58], for novel viewpoints, across three different complexity levels. The predictions are more accurate with less difficult motion as shown in the first and second row. The task becomes more challenging for our hard samples.
framework. The model has two output layers; one layer represents the background, and the other layer identifies one or more moving objects in the foreground, including their opacity layers. These layers are then linearly composed and optimised to reconstruct the input image. Since this approach is unsupervised, it can be compared fairly to the 3D baselines for this task.
Results. To evaluate performance, we measure 2D segmentation accuracy on test frames using mean average precision (mAP) as in [59]. Table 4 compares unsupervised 2D baselines and 3D baselines. Depending on the type of observed motion, 3D-based methods offer advantages over 2D methods and vice versa. For example, 3D-based methods are better suited for discovering semi-static objects that are not currently in motion, i.e., they have been moved at different times within a video. This is evident by the improved segmentation performance when considering this type of motion (i.e.,  $SS + D$  and purely SS). However, we note that none of the 3D-based methods explicitly consider motion. Consequently, MG, which takes as input optical flow, performs better on purely dynamic motion, but struggles to segment objects that are temporarily not moving. This drawback of 3D-based methods, compared to 2D motion-based methods, underscores the current challenge in capturing dynamics in neural rendering. Addressing this limitation is an open question for future research.
Figure 6 shows qualitative results. We observe that MG performs particularly well on objects that are constantly in motion, for example, the moving body parts of the person. Among the 3D methods, NeuralDiff is better at capturing dynamic objects, and, unlike MG, both NeuralDiff and T-NeRF+ are able to segment various semi-static objects as well since they do not rely on continuous motion.
Table 4: Unsupervised dynamic object segmentation. We report the mean average precision (mAP) on segmenting the semi-static (SS) and dynamic components of the scene, and also their union (SS+Dyn). All methods are trained without explicit supervision, i.e., no masks are used during training, only for evaluation.
Method	3D	SS+Dyn	SS	Dynamic
MG [69]	X	55.53	12.78	64.27
NeRF-W [32]	✓	45.62	20.97	28.52
T-NeRF+ [14]	✓	64.91	24.48	44.27
NeuralDiff [59]	✓	69.74	25.55	55.58
Figure 6: Unsupervised dynamic object segmentation. We compare three 3D baselines (NeRF-W [32], T-NeRF+ [14] and NeuralDiff [58]) and one 2D baseline (MG [69]) with three motion types. The 2D baseline captures the person and does well on the purely dynamic (short-range motion) setting. 3D models do this and also segment semi-static (SS) components (long-range motion, i.e., objects that were moved some time ago). In  $SS + Dyn$ , the evaluation includes SS and dynamic components.
4.3 Semi-Supervised Video Object Segmentation (VOS)
Semi-Supervised Video Object Segmentation (VOS) is a standard semi-supervised video understanding task: given the mask for one or more objects in a reference frame, the goal is to propagate the segments to subsequent frames. For this task, we use the train/val splits published as part of the VISOR VOS benchmark (See [8] Sec. 5.1). VOS is usually approached by using 2D models. Here, we explore how the 3D information in EPIC Fields can be used for it instead.
Benchmark methods. We evaluate two naive baselines for this task, one in 2D and another in 3D. For completeness, we also compare these to existing, trained 2D VOS models.
Fixed in 2D. We make the assumption that the pixels in the first frame remain constant throughout the entire sequence. This naive baseline is prone to failure when the camera undergoes movement.
Fixed in 3D. To better understand the potential of 3D information for VOS, we compare the 2D baseline above to a 3D one. In the 3D baseline, an object mask is projected to 3D and its position in 3D is fixed throughout the sequence. The mask is then re-projected to other frames using the available camera information. This works well for static objects and achieves two effects. First, objects can be reliably tracked over occlusions. Second, detecting when these objects are in or out of view is a by-product of estimated camera poses.
Trained 2D models. We also evaluate two state-of-the-art models for video object segmentation, STM [36] and XMEM [4]. These are trained on the train set of VISOR.
Results. We compare the baselines on the VISOR benchmark using the evaluation metrics defined in [43] which are the region similarity  $\mathcal{I}$  and contour accuracy  $\mathcal{F}$ . We also distinguish the set of objects that are static, such as 'fridge', 'floor', and 'sink', and report the above metrics separately for these and all other movable objects (SS+Dyn). Table 5 shows the results where the Fixed in  $3D$  clearly outperforms the Fixed in  $2D$  by a significant margin for the anticipated static objects
Table 5: Semi-Supervised VOS. We compare naive baselines in 2D and 3D, as well as pretrained/finetuned models on static and dynamic objects on the validation set of VISOR VOS. *: two videos from the validation set are excluded as they don't have successful reconstructions.
Method	3D	VISOR VAL[8]	Static	SS + Dyn
J&F	J	F	J&F	J	F	J&F	J	F
Fixed in 2D	X	12.5	13.4	11.6	17.8	23.8	11.6	12.0	11.9	12.0
Fixed in 3D *	✓	31.3	30.5	32.2	48.4	52.2	44.6	29.6	27.8	31.5
Pretrained STM	X	63.0	60.8	65.2	64.3	65.4	63.1	63.7	60.8	65.5
Fine-tuned STM	X	76.4	74.2	78.6	76.8	77.7	76.0	76.6	73.8	79.5
Pretrained XMEM	X	64.0	61.5	66.4	63.2	64.0	62.5	64.1	61.1	67.1
Fine-tuned XMEM	X	77.3	75.2	79.4	77.0	77.7	77.4	78.0	75.3	80.7
Figure 7: Semi-Supervised Video Object Segmentation. We compare our baselines on two frames from the same sequence. The Fixed in  $3D$  baseline can track the bin over camera motion and recognise in/out-of view. Pretrained models usually suffer from false positives in the out-of-view scenes.
$(+30.6\%)$  but also improves results for the remaining semi-static and dynamic  $(+17.6\%)$  objects. This is because such objects do remain unmoved for some duration of the videos. This highlights the additional value derived from representing objects in 3D. Figure 7 visualises one example where the bin is successfully propagated using the Fixed in 3D baseline, including when out of view. The pretrained models struggle to propagate masks for the novel objects in the dataset or for masks that go out of the scene. These are cases that the Fixed in 3D baseline successfully handles. However, the fine-tuned models are quantitatively and qualitatively superior as they are trained on the dataset. No prior work has utilised 3D information along with learnt models for the task of semi-supervised VOS. We hope our novel benchmark can trigger new VOS approaches that tackle the combined challenge of keeping track of static objects in 3D and dynamic objects through trained propagation of objects during motion and transformations.
We introduced EPIC Fields, a dataset to study 3D video understanding. We addressed the difficult challenge of reconstructing cameras in EPIC-KITCHENS videos, introducing filtering and other techniques that are portable to other similar reconstruction scenarios. Using these pre-computed cameras facilitates working on 3D video understanding even without significant expertise in photogrammetry.
With EPIC Fields we also defined three benchmark tasks: dynamic new-view synthesis, unsupervised dynamic object segmentation, and video object segmentation. Our results show that the performance of state-of-the-art dynamic neural reconstruction-rendering methods strongly depends on the type of motion. In particular, the gap in reconstruction quality between the dynamic and the static parts of the videos show that there is ample margin for further improvements in the handling of dynamic objects. Similar findings apply to the segmentation of objects, where 3D-based models can assist unsupervised video object segmentation and propagate masks of static objects over time. We hope that these results, the proposed benchmark data and code (comprising evaluation, camera reconstruction, and baselines) will assist the community in investigating further methods that combine geometry and video understanding.
Societal impact. While we expect that our benchmark will lead to positive impact, including applications to augmented and mixed reality including AR assistants, there are potential negative impacts as well: better AR may be used for deception and many capabilities powering an assistant may also aid surveillance.
Acknowledgements Project supported by EPSRC Program Grant Visual AI EP/T028572/1. A. Darkhalil is supported by EPSRC DTP program. Z. Zhu is supported by UoB-CSC Scholarship. I. Laina and A. Vedaldi are supported by ERC-CoG UNION 101001212. D. Damen is supported by EPSRC Fellowship UMPIRE EP/T004991/1. D. Fouhey is supported by a NSF CAREER #2142529.
We thank Sanja Fidler and Amlan Kar, from the University of Toronto, for contributions to the initial idea and discussions of this project. We also thank Rhodri Guerrier for assisting in manual annotations of the test set.
[67] Christopher Xie, Keunhong Park, Ricardo Martin-Brualla, and Matthew Brown. Fig-nerf: Figure-ground neural radiance fields for 3d object category modelling. In Proceedings of the International Conference on 3D Vision (3DV), pages 962–971. IEEE, 2021. 2
[68] Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In Proceedings of the International Conference on Computer Vision (ICCV), 2021. 1, 2
[69] Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. In Proceedings of the International Conference on Computer Vision (ICCV), 2021. 4.2, 6
[70] Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2, 2, 2
[71] Hong-Xing Yu, Leonidas Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. In Proceedings of the International Conference on Learning Representations (ICLR), 2022. 2
[72] Wentao Yuan, Zhaoyang Lv, Tanner Schmidt, and Steven Lovegrove. STaR: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 2
[73] Xiaoshuai Zhang, Abhijit Kundu, Thomas Funkhouser, Leonidas Guibas, Hao Su, and Kyle Genova. Nerflets: Local radiance fields for efficient structure-aware 3d scene representation from 2d supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2
[74] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. In Proceedings of the International Conference on Computer Vision (ICCV), 2021. 1, 2
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We believe that our abstract and introduction accurately describe the work.
(b) Did you describe the limitations of your work? [Yes] We have discussed the limitations in the experiments. Indeed, one of the points of our paper is to show current limitations in existing work.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We have discussed potential negative impacts in the conclusions of the paper.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
(a) Did you state the full set of assumptions of all theoretical results? [N/A] There are no theoretical results in the paper.
(b) Did you include complete proofs of all theoretical results? [N/A] There are no theoretical results in the paper.
3. If you ran experiments (e.g., for benchmarks)...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No] We include all all of the data and instructions needed. We will release all code for our benchmarks no later than the publication date.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We provide information about the data splits and hyperparameters in the main paper and supplement.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No] Our experiments are too computationally expensive to run
multiple times to produce error bars. Our models are meant as baselines that subsequent methods ought to show improvement on.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We provide a detailed documentation of the compute cost of each of the steps in the supplement. Some steps, such as the registration of all cameras, are costly. However, believe that by doing them and providing the information to the community, we can collectively save substantial overall compute time across the community.
(a) If your work uses existing assets, did you cite the creators? [Yes] We build exclusively on the EPIC-KITCHENS 100 dataset. Our use of the dataset is made clear throughout the paper.
(b) Did you mention the license of the assets? [Yes] When introducing EPIC-KITCHENS, we introduce its license. Our new assets will be released under the same license.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] Yes, we provide additional assets that will provide to the reviewers privately and then share publicly upon acceptance.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] When introducing EPIC-KITCHENS, we mention that it was collected with ethics approval and with clear consent from the people who are in the data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] When introducing EPIC-KITCHENS, we discuss that the data does not contain PII, was reviewed by participants prior to publication, and does not contain offensive content.
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] We did not do any crowdsourced annotation.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] There are no participant risks for this paper. The dataset we use, however, was collected with ethics board approval.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] We did not have any paid participants or workers.