EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision
Authors: Jiawei Yang
Boris Ivanovic
Or Litany
Xinshuo Weng
Seung Wook Kim
Boyi Li
Tong Che
Danfei Xu
Sanja Fidler
Marco Pavone
Yue Wang
Venue: ICLR 2024 poster (2024)
Keywords: Neural fields
spatial-temporal NeRFs
NeRF for driving
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: We present EmerNeRF, a simple yet powerful approach for learning spatial-temporal representations of dynamic driving scenes. Grounded in neural fields, EmerNeRF simultaneously captures scene geometry, appearance, motion, and semantics via self-bootstrapping. EmerNeRF hinges upon two core components: First, it stratifies scenes into static and dynamic fields. This decomposition emerges purely from self-supervision, enabling our model to learn from general, in-the-wild data sources. Second, EmerNeRF parameterizes an induced flow field from the dynamic field and uses this flow field to further aggregate multi-frame features, amplifying the rendering precision of dynamic objects. Coupling these three fields (static, dynamic, and flow) enables EmerNeRF to represent highly-dynamic scenes self-sufficiently, without relying on ground truth object annotations or pre-trained models for dynamic object segmentation or optical flow estimation. Our method achieves state-of-the-art performance in sensor simulation, significantly outperforming previous methods when reconstructing static (+2.93 PSNR) and dynamic (+3.70 PSNR) scenes. In addition, to bolster EmerNeRF's semantic generalization, we lift 2D visual foundation model features into 4D space-time and address a general positional bias in modern Transformers, significantly boosting 3D perception performance (e.g., 37.50% relative improvement in occupancy prediction accuracy on average). Finally, we construct a diverse and challenging 120-sequence dataset to benchmark neural fields under extreme and highly-dynamic settings. See the project page for code, data, and request pre-trained models: https://emernerf.github.io

Published as a conference paper at ICLR 2024
EMERNERF: EMERGENT
SCENE
DECOMPOSITION VIA SELF-SUPERVISION
Jiawei Yang",,, Boris Ivanovic", Or Litany Xinshuo Weng Seung Wook Kim', Boyi Li", Tong Che Danfei Xu Sanja Fidler Marco Pavone Yue Wang"
[yangjiaw, yue. @uscc edu, University of Southern California
8 sonffei@gatech.ee. Georgia Institute of Technology
8 f pavone@stanffrd.eddi. Stanford University
edu, University of Toronto
orlitany@gmail.com Technion
(bivarovic, xweng, seungwokk,, boy tongc) com, NVIDIA Research
ABSTRACT
We present EmerNeRF, simple yet powerful approach for learning spatial- temporal representations of dynamic driving scenes. Grounded in neural fields, EmerNeRF simultaneously captures scene geometry. appearance, motion, and semantics via self-bootstrapping. EmerNeRF hinges upon two core compo- nents: First, it stratifies scenes into static and dynamic fields. This decomposition emerges purely from supervvipn,,,,nn, enabling model to learn from general, in-the-wild data sources. Second, EmerNeRF parameterizes an induced flow field from the dynamic field and this flow field to further aggregate multi-frame features, amplifying the rendering precision of dynamic objects. Coupling these three fields (static, dynamic, and flow) enables rNeF represent highly- dynamic scenes self-sufficiently, without relying on ground truth object annota- tions pre-trained models for dynamic object segmentation or optical flow esti- mation. Our method achieves state-of-th of-the-art performance in sensor simulation, significantly outperforming previous methods when reconstructing static (+2.93 PSNR) and dynamic (+3.70 PSNR) scenes. In addition, to bolster EmerNeRF's semantic generalization, we lift 2D visual foundation model features into 4D space-time and address general positional bias in modern Transformers, signif- icantly boosting 3D perception performance (e.g., 37.50% relative improvement in occupancy prediction accuracy on average). Finally, we construct diverse and challenging 120-sequence dataset to benchmark neural fields under extreme and highly-dyname settings. See the project page for code, data, and request /t////eerrf.gthuuieieeeeeeeeeeee
pre-trained models:
INTRODUCTION
Perceiving, representing, and reconstructing dynamic scenes is critical for autonomous agents to understand and interact with their environments. Current approaches predominantly build custom pipelines with components dedicated to identifying and tracking static obstacles and dynamic ob- jects (Yang al., 2023; Guo al., 2023). However, such approaches require training each com- ponent with a large amount of labeled data and devising complex mechanisms to combine out- puts across components. To represent static scenes, approaches leveraging neural radiance fields (NeRFs) (Mildenhall et al., 2021) have witnessed Cambrian explosion in computer graphics, robotics, and autonomous driving, owing to their strong performance in estimating 3D geometry and appearance (Rematas et al., 2022; Tancik et al., 2022; Wang et al., 2023c; Guo et al., 2023). However, without explicit supervision, NeRFs struggle with dynamic environments filled with fast- moving objects, such vehicles and pedestrians in urban scenarios. In this work, we tackle this long-standing challenge and develop a self-supervised technique for building 4D sspace-time) rep- resentations of dynamic scenes.

---

Published as a conference paper at ICLR 2024
GT
(b) Rendered RGA
(c) Decomposed Stanic RGA
Decomposed Depth
Decomposed
Decomposed Dynamic Dapth
Emerged Scane Flow
Reconstructed DINCO Features
(i) Decompesed PE-Free DINOV2 Features
( Decompresed TE-Festures
Figure 1: EmerNeRF effectively reconstructs photo-realistic dynamic scenes (b), separating them into explicit static (c-d) and dynamic (e-f) elements, all via self-supervision. Notably, (g) scene flows emerge from EmerNeRF without any explicit flow supervision. Moreover, Eme RF can address detrimental positional embedding (PE) patterns observed in vision foundation models (h. j), and lift clean, PE-free features into 4D space (i). Additional visualizations can be found Appendix C.4.
We consider common setting where mobile robot equipped with multiple sensors (e.g., cam- eras, LiDAR) navigates through large dynamic environment (e.g., a neighborhood street). The fundamental underlying challenge to construct an expansive 4D representation with only sparse, transient observations, that is, to reconstruct an entire 4D space-time volume from single traversal of the volume. Unfortunately, this setting violates NeRF's multi-view consistency assumption- each point in the space-time volume will only be observed once. Recent works (Yang et al., 2023; Ost et al., 2021) seek to simplify the problem by modeling static scene components (e.g., buildings and trees) and dynamic objects separately, using combination of neural fields and mesh represen- tations. This decomposition enables exploiting multi- tallti-timetepp observations to supervise static com- ponents, but it often requires costly ground-trui -tuth annotations segment and track dynamic objects. Moreover, despite efforts shown in daily videos (Wang et al., 2021; Gao et al., 2021; Wang et al., 2023b;a; Du al., 2021), effectively modeling the temporal correspondence of dynamic objects in
2

---

Published as a conference paper at ICLR 2024
autonomous driving scenarios is still largely unexplored. Overall, learning 4D representations of dynamic scenes in the context of autonomous driving remains a formidable challenge.
Towards this end, we present Eme EmerNeRF, self-supervised approach for constructing 4D neural scene representations. As shown in Fig. 1, Eme rNeRF decouples static and dynamic scene com- ponents and estimates 3D scene flows remarkably, all from self-supervision. At high level, EmerNeRF builds a hybrid static-dynamic world representation via a density-regularized objec- tive, generating density for dynamic objects only as necessary (i.e., when points intersect dynamic objects). This representation enables our approach to capture dynamic components and exploit multi-timestep observations to self self-supervise static scene elements. To address the lack of cross- observation consistency for dynamic components, we task EmerNeRF to predict 3D scene flows and use them to aggregate remporallyddisplaced features. Intriguingly, Eme NeeRF's capability to estimate scene flow emerges naturally from this process, without any explicit flow supervision. Fi- nally, to enhance scene comprehension, we "lift" features from pre-trained 2D visual foundation models (e.g., DINOvI (Caron et al., 2021), DINOv2 (Oquab et al., 2023)) to 4D space-time. In do- ing so, we observe and rectify a challenge tied to Transformer-basee foundation models: positional embedding (PE) patterns (Fig. 1 (h)). As we will show in $4.3, effectively utilizing such general features greatly improves Emer rNeRF's semantic understanding and enables few-shot auto-labeling.
We evaluate EmerNeRF sensor sequences collected by autonomous vehicles (AVs) traversing through diverse urban environments. A critical challenge is that current autonomous driving datasets heavily imbalanced, containing many simple scenarios with few dynamic objects. To facilitate a focused empirical study and bolster future research on this topic, we present the NeRF On-The- Road (NOTR) benchmark, a balanced subsample of 120 driving sequences from the Waymo Open Dataset (Sun et al., 2020) containing diverse visual conditions (lighting, weather, and exposure) and challenging dynamic scenarios. On this benchmark, EmerNeRA significantly outperforms previous state-of-the- afr NeRF-based approaches (Park et al., 202 lb: Wu et al., 2022; Müller et al., 2022; Guo et al., 2023) on scene reconstruction by 2.93 and 3.70 PSNR on static and dynamic scenes, respec- tively, and by 2.91 PSNR on dynamic novel view synthesis. For scene flow estimation, Emer excels over Li et al. (2021a) by 42. in metrics of interest. Additionally, removing PE patterns brings an average improvement of 37.50% relative to using the original, PE pattern-laden features on semantic occupancy prediction. Contributions. Our key contributions are fourfold: (1) We in- troduce Eme rNeRF, novel 4D neural scene representation framework that excels in challenging autonomous driving scenarios. EMerNeRF performs static-dynamic decomposition and scene flow estimation, all through self-supervision. (2) A streamlined method to tackle the undesired effects of positional embedding patterns from Vision Transformers, which is immediately applicable to other tasks. (3) We introduce the NOTR dataset assess neural fields in diverse conditions and facili- tate future development in the field. (4) Eme RN achieves state-of- the- performance in scene reconstruction, novel view synthesis, and scene flow estimation.
2 RELATED WORK
Dynamic scene reconstruction with NeRFs. Recent works adopt NeRFs (Mildenhall et al., 2021; Müller et al., 2022) to accommodate dynamic scenes (Li et al., 2021b; Park et al., 2021b; Wu et al., 2022; Wang al., 2021: Gao et al., 2021; Wang et al., 2023a; Du al., 2021). Earlier methods (Bansal et al., 2020; Li et al., 2022; Wang et al., 2022; Fang et al., 2022) for dynamic view synthesis rely on multiple synchronized videos recorded from different viewpoints, restricting their use for real-world applications in autonomous driving and robotics. Recent methods, such as Nerfies (Park al., 2021a) and HyperNeRF (Park et al., 2021b), have managed to achieve dynamic view synthesis using a single camera. However, they rely on a strict assumption that all observations can be mapped via deformation back to a canonical reference space, usually constructed from the first timestep.
Of particular relevance to our work are methods like NeRR (Wu et al., 2022). SUDS (Turki et al., 2023), and NeuralGroundplans (Sharma et al., 2022). These methods also partition a 4D scene into static and dynamic components. However, NeRF underperforms significantly for outdoor scenes due to its sensitivity to hyperparameters and insufficient capacity; NeuralGroundplann relies on synchronized videos from different viewpoints reason about dynamics; and SUDS, designed for multi- traversal driving logs, largely relies on accurate optical flows derived by pre-trained models and incurs high computational costs due to its expensive flow-based warping losses. In contrast, our
3

---

Published as a conference paper at ICLR 2024
approach can reconstruct accurate 4D scene representation from single-traversie log captured by sensors mounted on a self-driving vehicle. Freed from the constraints of pre-trained flow models, EmerNeRF exploits and refines its own intrinsic flow predictions, enabling a self-improving loop.
NeRFs for AV data. Creating high-fi fidelity neural simulations from collected driving logs is crucial for the autonomous driving community, as it facilitates the closed-loop training and testing of various algorithms. Beyond SUDS (Turki et al., 2023), there is a growing interest in reconstructing scenes from driving logs. In particular, recent methods excel with static scenes but face challenges with dynamic objects (Guo et al., 2023). While approaches like UniSim (Yang et al., 2023) and NSG (Ost al.. 2021) handle dynamic objects, they depend on ground truth annotations, making them less scalable due to the cost of obtaining such annotations. In contrast, our method achieves high-fidelity simulation results purely through self-supervision, offering a scalable solution.
Augmenting NeRFs. NeRF methods are commonly augmented with external model outputs to in- corporate additional information. For example, approaches that incorporate scene flow often rely on existing optical flow models for supervision (Li et al., 2021b; Turki et al., 2023; Li et al., 2023b; Wang et al., 2021; Gao et al., 2021; Wang et al., 2023a; Du et al., 2021). They usually require cycle- consistency tests to filter out inconsistent flow estimations; otherwise, the optimization process is prone to failure (Wang et al., 2023b). The Neural Scene Flow Prior (NSFP) (Li et al., 2021a), a state- of-the-art flow estimator, optimizes neural network to estimate the scene flow at each timestep (minimizing the Chamfer Loss (Fan et al., 2017)). This per-timestep optimization makes NSFP prohibitively expensive. In contrast, our Eme rNe bypasses the need for either pre-trained op- tical flow models or holistic geometry losses. Instead, our flow field is supervised only by scene reconstruction losses and the flow estimation capability emerges on its own. Most recently, 2D signals such as semantic labels or foundation model feature vectors have been distilled into 3D space (Kobayashi al., 2022; Kerr et al., 2023; Tsagkas et al., 2023; Shafiullah et al., 2022), en- abling semantic understanding tasks. In this work, we similarly lift visual foundation model features into 4D space and show their potential for few-shot perception tasks.
SELF-SUPERVISED
NEURAL FIELDS
Learning a spatial-temporal representation of dynamic environment with multi-sensor robot is challenging due to the sparsity of observations and costs of obtaining ground truth annotations. To this end, our design choices stem from the following key principles: (I) Learn a scene decompo- sition entirely through self-supervisio and avoid using any ground-truth annotations or pre-trained models for dynamic object segmentation or optical flow. (2) Model dynamic element correspon- dences across time via scene flow. (3) Obtain a mutually reinforcing representation: static-dynamic decomposition and flow estimation can benefit from each other. (4) Improve the semanties of scene representations by leveraging feature lifting and distillation, enabling a range of perception tasks.
Having established several design principles, we are now equipped to describe EmerNeRF, a self- supervised approach for efficiently representing both static and dynamic scene components. First, $3.1 details how EmerNeRF builds a hybrid world representation with a static and dynamic field. Then, $3.2 explains how Eeer leverages emergent flow field to aggregate temporal features over time, further improving its representation of dynamic components. $3.3 describes the lifting of semantic features from pre-trained 2D models to 4D ppace-time, enhancing EmerNeR? scene understanding. Finally, $3.4 discusses the loss function that is minimized during training.
3.1 SCENE REPRESENTATIONS
Scene decomposition. To enable efficient scene decomposition, we design EmerNeRF to be hybrid spatial-temporal representation. It decomposes a 4D scene into static field S and a dynamic field D. both of which are parameterized by learnable hash grids (Müller al., 2022) H. and Ha, respectively. Alternative versatile representations such as Hexplane (Cao & Johnson, 2023) can also be employed. This decoupling offers a flexible and compact 4D scene representation for time- independent features h, H. (x) and time-varying features h. = Haa(x, f). where x (x, y, =) is the 3D location of a query point and denotes its timestep. These features are further transformed into g. and ga by lightweight MLPs (g. and ga) and used to predict per-point density o, and oa
gs, 0 (x)(
gd, ds (a(HHa
4

---

Published as a conference paper at ICLR 2024
(a)
(b)
Siyy Head
Cas
Fnstuure Head
(x,
X
Static
Re-il
Cororr Haad
Dynamic Field />
Cocrdinato
Shottirs Hoae
Prediction
(06,
Bllncor
PE Head
Intorpoiono
(
Flow Field
Y
Learnable Map (f
Patterns
Figure 2: EmerNeRF Overview. (a) EmerNeRe consists of static, dynamic, and flow field (S, D, V). These fields take as input either a spatial query x or spatial-temporal query (x, t) to generate static (feature gs, density o₂) pair
dynamic (feature g.,- density oa) pair. Of note, features g/2 from
use the forward and backward flows (v, and vo) to generate temporally-aggregnn nearby temporal features g g and gd +++ (a slight abuse of notation w.r.t. Eq. (8)). These fea- tures (along with the view direction are consumed by the shared color head which independently predicts the static and dynamic colors and Ca. (b) Eme NNeRF removes undesired positional en- coding patterns in VIT' via a learnable PE map followed by a PE head.
Multi-head prediction. NNRF uses separate heads for color, sky, and shadow predictions. To maximize the use of dense self-supervision from the static branch, the static and dynamic branches share the same color head MLP color This color head takes (g., d) and (ga. d) as input, and outputs the normalized view direction.
per-point color c, and C2 for static and dynamic items, where follow Rematas et al. (2022) and use separate sky head
Since the depth of the sky is ill-defined, to predict the sky's color from the frequency-embes embedded view direction r(d) and features from the static field, where (-)) is frequency-based positional embedding function, (2021). As in Wu et al. (2022). we use a shadow head MLP shadow to depict the shadows of dynamic objects. outputs scalar o [o, 1] for dynamic objects, modulating the color intensity predicted by the static field. Collectively, we have:
in Mildenhall et al.
C2 MTP color (gs: (d))
C MLP color (ga; (d))
Csky MLP color sky (7(d))
MLP shadow (ga)
Rendering, To enable highly-efficient rendering, we use density- based weights to combine results from the static field and dynamic field:
a
(1 - p) +
ad Cd
(4)
o. ad
o. +
To render pixel, K discrete samples {xI,..., along its ray to estimate the integral of color. The final outputs are given by:
K
K (1-E
Ć Σ Tasc + 1
Tias
Caky
(5)
i=1
where T: = 0g) is the accumulated transmittance and a; is the piece-wise opacity.
Dynamic density regularization. To facilitate static-dynamic decomposition, leverage the fact that our world is predominantly static. We We regularize dynamic density by minimizing the expectation of the dynamic density ad, which prompts the dynamic field produce density values only as needed:
C., - 1
(r(r
(6)
where N2 is the number of rays per batch, K represents the number of sampling points along a ray, and ar i) denotes the predicted dynamic density of the i-th point along the r-th ray.
3.2 EMERGENT SCENE FLOW
Scene flow estimation. To capture explicit correspondences between dynamic objects and provide link by which to aggregate temporally-displaced features, introduce additional scene flow
5

---

Published as a conference paper at ICLR 2024
field consisting of hash grid V (x. and flow predictor MLP.. This flow field maps spatial-temporal query point (x, t) to a flow vector v E R which transforms the query point to its position in the next timestep, given by:
V == MLP., M...((H((X.))
=
(7)
In practice, our flow field predicts both forward flow Vy and backward flow Vb, resulting in 6-dimensional flow vector for each point.
Multi-frame feature integration.
Next, we the link provided by the predicted scene flow to integrate features from nearby timesteps, using simple weighted summation:
Sd = 0.25 (x va, 1)) 0.5 ga(H.a(x, t)) + 0.25 gu OHaaa v
If not otherwise specified, 8's is used by default when the flow field is enabled (instead of ga in Eqs. (2) and (3)). This feature aggregation module achieves three goals: 1) It connects the flow field to scene reconstruction losses (e.g.. RGB loss) for supervision, 2) it consolidates features, denoising temporal attributes for accurate predictions, and 3) each point is enriched through the shared gradient of its temporally-linie linked features, enhancing the quality of individual points via shared knowledge.
Emergent abilities. We do not use any explicit flow supervision to guide INRF's flow estima- tion process. Instead, this capability emerges from our temporal aggregation step while optimizing scene reconstruction losses ($3.4). Our hypothesis that only temporally-consisee from Multi-frame feature integration, and this integration indirectly drives the scene flow field to- ward optimal solutions - predicting correct flows for all points. Our subsequent ablation studies in A Appendix C.I confirm this: when the temporal aggregation is disabled or gradients of these nearby features are stopped, the flow field fails to learn meaningful results.
features benefit
3.3 VISION TRANSFORMER FEATURE LIFTING
While NeRFs excel at generating high-fidelity color and density fields, they lack in conveying se- mantic content, constraining their utility for semantic scene comprehension. To bridge this gap, we lift 2D foundation model features to 4D, enabling crucial autonomous driving perception tasks such semantic occupancy prediction. Although previous works might suggest a straightforward approach (Kerr et al., 2023; Kobayashi et al., 2022), directly lifting features from state-of-the-arr vi- sion transformer (ViT) models reveals additional complexities due positional embeddings (PEs) in transformer models (Fig. (h-j)). In the following sections, we detail how we enhance EmerNe Ne with feature reconstruction head, uncover detrimental PE patterns in transformer models, and sub- sequently mitigate these issues.
Feature reconstruction head. Analogous the color head, we incorporate a feature head MLP feat and feature sky head MLP jeat.sky to predict per-point features f and sky features faky, given by:
f. MLP Feat (g.), where E {s,d}
faky MLPPieatstae (7()).
Similar to the color head, we share the feature head among the static and dynamic branches. Ren- dering these features similarly follows Eq. (5), given by:
1 (1-I Tia
Σ Traf +
faky
(10)
Positional embedding patterns. We observe pronounced and undesired PE patterns when using current sfateoo--thean foundation models, notably DINOv2 (Oquab et al., 2023) (Fig. (h)). These patterns remain fixed in images, irrespective of 3D viewpoint changes, breaking 3D multi-view consistency. Our experiments ($4.3) reveal that these patterns not only impair feature synthesis results, but also cause a substantial reduction in 3D perception performance.
Shared learnable additive prior. We base our solution on the observation that ViTs extract feature maps image-by-image and these PE patterns appear consistently across all images. This suggests that a single, globally-shared PE feature map might be sufficient to capture this shared information. Accordingly, assume additive noise model for the PE patterns; that is, they can be indepen- dently subtracted from the original features to obtain PE-free features. With this assumption, we construct learnalle and globally-shared 2D feature map 24 to compensate for these patterns. This

---

Published as a conference paper at ICLR 2024
process is depicted in Fig. 2 (b). For target pixel coordinate (u, t), first volume-render PE- free feature as in Eq. (10). Then, we bilinearly interpolate U and decode the interpolated feature using single-layer MLPPr. Pr to obtain the PE pattern feature, which is then added to the PE-free feature. Formally:
F
Σ Tafh + 1 Σ Tai faky + MLPPPE (interp ((u,v) v) .2))
(1-
(11)
PB feature
Volumc-readered PE-free feature
The grouped terms render "PE-free" features (Fig. 1 (i) and "PE" patterns (Fig. 1 (j)) respectively, with their sum producing the overall "PE-containing" features (Fig. 1 (h)).
3.4
OPTIMIZATION
Loss functions. Our method decouples pixel rays and LiDAR rays to account for sensor asynchro- nization. For pixel rays, use L2 loss for colors Ligb (and optional semantic features Lieat). binary cross entropy loss for sky supervision Csky. and shadow sparsity loss Lshadow. For LiDAR rays, combine expected depth loss with line-of-sight loss Ldepth. as proposed in Rematas et al. (2022). This line-of-sight loss promotes an unimodal distribution of density weights along ray, which we find is important for clear static-dynamic decomposition. For dynamic regular- ization, we use a density-based regularization (Eq. 6) to encourage the dynamic field to produce density values only when absolutely necessary. This dynamic regularization loss is applied both pixel rays (Co a(ikee))) and LiDAR rays (La,(IaDAR) ). Lastly, we regularize the flow field with a cycle consistency loss Leyele. See Appendix A.1 for details. summary, we minimize:
C = Ligh + Laky + Lshadow + Laulpixel) + Leycle + Lrea + Ldepth + Cog(LADAR)
(12)
pixel rays
LiDAR rays
Implementation details. All model implementation details can be found Appendix A.
EXPERIMENTS
In this section, benchmark the reconstruction capabilities of EmerNeRF against prior meth- ods, focusing on static and dynamic scene reconstruction, novel view synthesis, scene flow esti- mation, and foundation model feature reconstruction. Further ablation studies and discussion of EmerNeRF's 's limitations can be found in Appendices C.I and C.3, respectively.
Dataset. While there exist many public datasets with AV sensor data (Caesar et al., 2020; Sun al., 2020; Caesar et al., 2021), they are heavily imbalanced, containing many simple scenarios with few to no dynamic objects. To remedy this, we introduce NeRF on-The-Road (NOTR), a balanced and diverse benchmark derived from the Waymo Open Dataset (Sun et al., 2020). NOTR features 120 unique, hand-picked driving sequences, split into 32 static (the same split as in StreetSurf (Guo et al., 2023)), 32 dynamic, and 56 diverse scenes across seven challenging conditions: ego-static, high-speed, exposure mismatch, dusk/dawn, gloomy, rainy, and night. We name these splits Static- 32, Dynamic-32, and Diverse-56, respectively. This dataset not only offers consistent benchmark for static and dynamic object reconstruction, it also highlights the challenges of training NeRFs on real-world AV data. Beyond simulation, our benchmark offers 2D bounding boxes for dynamic objects, ground truth 3D scene flow, and 3D semantic occupancy- crucial for driving perception tasks. Additional details can be found in Appendix B.
4.1 RENDERING
Setup. To analyze performance across various driving scenarios, we test EmerNeRF's scene recon- struction and novel view synthesis capabilities on different NOTR splits. For scene reconstruction, all samples in a log are used for training. This setup probes the upper bound of each method. For novel view synthesis, we omit every 10th timestep, resulting in 10% novel views for evaluation. Our metrics include peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). For Dynamic scenes, we further leverage ground truth bounding boxes and velocity data to identify dy- namic objects and compute "dynamic-only" metrics; and we benchmark against HyperNeRF (Park
7

---

Published as a conference paper at ICLR 2024
Table 1: Dynamic and static scene reconstruction performance.
(a) Dynamic-32 32 Split
(b) Static-32 Split
Scene Reconstraction
Novel View Synthesis
Static Scone Reconstruction
Methods
Full Image
Dynamic-nll
Full Image
Dynamic-Dll
Methods
PSNR1
SSIM1
PSNRT
SSIMI
PSNRT
SSIMT
PSNRT
SSIMT
DPSNRT
SSIMT
HyperNeRF
D"NeRF
24.35
25.17
0.645
0.688
21.78
22.93
0.504
0,569
24.17
24.71
0.642
0.682
21.44
22.43
0,494
0.554
iNGP
StreetSurf
24.46
26.66
0.694
0,784
Ours
28.87
0.814
26.19
0.736
27.62
0.792
24.18
0.670
Ours
29.08
0.803
Table 2: Scene flow estimation on the NOTR Dynamic-32 split.
Methods
EPE3D (m) 4
Accs(%) f
(%)) f (rad)
NSFP (Li et al., 2021a)
0.365
51.76
67.36
0.84
Ours
0.014
93.92
96.27
0.64
et al., 2021b) and DeNeRF (Wu et al., 2022), two state-of-the-art methods for modeling dynamic scenes. Due to their prohibitive training cost, we only compare against them in the Dynamic-32 split. On the Static-32 split, we disable our dynamic and flow branches, and compare against StreetSurf (Guo et al., 2023) and iNGP Müller et al. (2022) (as implemented by Guo et al. (2023)). We use the official codebases released by these methods, and adapt them to NOTR. To ensure a fair comparison, we augment all methods with LiDAR depth supervision and sky supervision, and disable our feature field. Further details can be found in Appendix A.2.
Dynamic scene comparisons. Table (a) shows that our approach consistently outperforms others on scene reconstruction and novel view synthesis. We refer readers Appendix C.2 for qualitative comparisons. In them, we can see that HyperNeRF (Park et al., 2021b) and DeNeRF (Wu et al., 2022) tend to produce ower-smoothed renderings and struggle with dynamic object representation. In contrast, EmerNeRE excels in reconstructing high-fidelity static background and dynamic fore- ground objects, while preserving high-frequency details (evident from its high SSIM and PSNR values). Despite DONeRF's intent to separate static and dynamic elements, it struggles in com- plex driving contexts and produces poor dynamic object segmentation (as shown in Fig. C.6). Our method outperforms them both quantitatively and qualitatively. Static scene comparisons. While static scene representation is not our main focus, Table 1 (b) shows that Eme r NeRF achieves state- of-the-art art performance in this task well, outperforming StreetSuRF (Guo et al., 2023) which was designed for static outdoor scenes. Thus, by modeling static and dynamic components, EMerNeRF accurately reconstruct general driving scenes.
4.2 FLOW ESTIMATION
Setup. We assess Eme RE all frames of the Dynamic-32 split, benchmarking against the prior state-of-the-art, NSFP (Li et al., 2021a). Using the Waymo dataset's ground truth scene flows, we compute metrics consistent with Li et al. (2021a): 3D end-point error (EPE3D), calculated as the mean L2 distance between predictions and ground truth for all points; Accs representing the frac- tion of points with EPE3D less than 5em or relative error under 5%; Acc10, indicating the fraction of points with EPE3D under 10cm or a relative error below 10%; and 0, the average angle error be- tween predictions and ground truths. When evaluating NSFP (Li et al., 2021 a), we use their official implementation and remove ground points (our approach does not require such preprocessing).
Results. As shown in Table 2, our approach outperforms NSFP across all metrics, with significant leads in EPE3D, Accs, and Acc While NSFP (Li et al., 2021a) employs the Chamfer distance loss Fan et al. (2017) to solve scene flow, EmerNeRF achieves significantly better results without any explicit flow supervision. These properties naturally emerge from temporal aggregation step. Appendix C.1 contains additional ablation studies regarding the emergence of flow estimation.
4.3 LEVERAGING FOUNDATION MODEL FEATURES
To investigate the impact of ViT PE patterns on 3D perception and feature synthesis, we instantiate versions of Eme EF with and without proposed PE decomposition module.
8

---

Published as a conference paper at ICLR 2024
Table 3: Few-shot semantic occupancy prediction evaluation. We investigate the influence of positional embedding (PE) patterns on 4D features by evaluating semantic occupancy prediction performance. We report waamle-avveaggee micro aicurracy and class-averaged nacro-accuracy.
PE removed?
VIT model
Staric-32
Dynamic-32
Diverse-56
Average of splits
Micro Ace
Macro Acc
Micro Ace
Macro Ace
Micro Acc
Macro Ace
Miero Ace
Macro Acc
No
DINOvI
43.12%
52.71%
47.51%
54.46%
43.19%
51.11%
44.60%
52.76%
Yes
DINOvI
55.02%
57.13%
57.65%
57.77%
54.56%
53.13%
55.74%
56.67%
Relative Improvement
+8.38%
+21.35%
426.32%
+7.87%
47.42%
No
DINOv2
38.73%
50.30%
51.43%
57.03%
45.22%
54,37%
45.13%
53.90%
Yes
DINOv2
63.21%
89.41%
65.08% %
60.82%
57.86%
59.00%
62.05%
59.74%
Relative Improvement
+63.22%
+18.11%
+26.539%
+5.65%
+27.95%
+8.51%
-37,50%
+10.84%
Table 4: Feature synthesis results. We report the Seature-PNSR values under different settings.
PE removed?
ViT' model
Static-32
Dynamie-32
Diverse-56
No
DINOvI
23.35
23.37
23.78
DINOvI
23.57 (+0:23)
23.52 (50.15)
23.92 (+0.14)
No
DINOv2
21.87
22.34
22.79
Yes
DINOv2
22.70 (-083)
22.80 (+0.45)
23.21 (+0.42)
Setup. We evaluate Eme EmerNeRF's few-shot perception capabilities using the Occ3D dataset (Tian et al., 2023). Occ3D provides 3D semantic occupancy annotations for the Waymo dataset (Sun et al., 2020) in voxel sizes of 0.4m and 0.1m (we use 0.1m). For each sequence, we annotate every 10th frame with ground truth information, resulting in 10% labeled data. Occupied coordinates are input to pre-trained EmerNeRF models to compute feature centroids per class. Features from the remaining 90% of frames then queried and classified based on their nearest feature centroid. We report both micro (sample-averaged) and macro (class-averaged) classification accuracies. All models are obtained from the scene reconstruction setting, i.e., all views are used for training.
Results. Table 3 compares the performance of PEEcontaining 4D features to their PE-free coun- terparts. Remarkably, Eme EmerNe rNeRF with PE-free DINOv2 (Oquab et al., 2023) features sees a max- imum relative improvement of 153% in micro-accuracy and an average increase of 78% over its PE-containing counterpart. Intriguingly, although the DINOv1 (Caron et al., 2021) model might appear visually unaffected (Fig. C.7), our results indicate that directly lifting PF-containing features to 4D space-time is indeed problematic. With our decomposition, PÊ-frre DINOv1 features witness an average relative boost of 40.36% in micro-accuracy. As another illustration of PE patterns' im- pact, DINOv2 features perform significantly worse than DINOvI features without our PE pattern mitigation. Thus, by eliminating PE patterns, the improved performance of DINOv2 over DINOv1 carries over to 3D perception.
Feature synthesis results. Table 4 compares the Paature-PSSRR of PE-containing and PE-free mod- els, showing marked improvements in feature synthesis quality when using our proposed PE decom- position method, especially for DINOy2 (Oquab al., 2023). While DINOv1 (Caron et al., 2021) appears to be less influenced by PE patterns, our method unveils their presence, further showing that even scemingly unaffected models can benefit from PE pattern decomposition.
CONCLUSION
In this work, we present EmerNeRF, simple yet powerful approach for learning 4D neural repre- sentations of dynamic scenes. EmerNeRF effectively captures scene geometry, appearance, motion, and any additional semantie features by decomposing scenes into static and dynamic fields, learning an induced flow field, and optionally lifting foundation model features to a resulting 4D hash grid representation. EmerN RN additionally removes problematic positional embedding patterns that appear when employing Transformer-bait based foundation model features. Notably, all of these tasks (save for foundation model feature lifting) are learned in self- refevsuuee fashion, without relying on ground truth object annotations or pre-trained models for dynamic object segmentation or optical flow estimation. When evaluated on NOTR, our carefully-selected subset of 120 challenging driv- ing scenes from the Waymo Open Dataset (Sun et al., 2020), EmerNeRF achieves state-of-the-and performance in sensor simulation, significantly outperforming previous methods on both static and dynamic scene reconstruction, novel view synthesis, and scene flow estimation. Exciting areas of future work include further exploring capabilities enabled or significantly improved by harnessing foundation model features: few-shot, zero-shot, and auto-labeling via open-vocabulary detection.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
This work primarily focuses on autonomous driving data representation and reconstruction, Ac- cordingly, we use open datasets captured in public spaces which strive preserve personal privacy by leveraging state-of-the-art object detection techniques to blur people's faces and vehicle license plates. However, these instance-level characteristics. What requires more effort manage (and could potentially lead to greater harm) is maintaining a diversity of neighborhoods, and not only in terms of geography, but also population distribution, architectural diversity, and data collection times (ideally repeated traversals uniformly distributed throughout the day and night, for example). We created the NOTR dataset with diversity mind, hand-picking scenarios from the Waymo Open Dataset (Sun et al., 2020) to ensure diversity of neighhorhoods and scenario types (e.g., static, dynamic). However, as in the parent Waymo Open Dataset, the NOTR dataset contains primarily urban geographies, collected from only handful of cities in the USA.
REPRODUCIBILITY STATEMENT
We present our method in $3, experiments and results in $4, implementation details and ablation studies in Appendix A. We benchmark previous approaches and our proposed method using publicly available data and include details of the derived dataset in Appendix B. Additional visualizations, code, models, and data are available either in the appendix or at https: : cmmernerf.ggthhue
REFERENCES
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep ViT features dense visual descriptors. arXiv preprint arXiv:2112.000 05814,, 2021.
Aayush Bansal, Minh Vo, Yaser Sheikh, Deva Ramanan, and Srinivasa Narasimhan. 4d visualiza- tion of dynamic events from unconstrained multi-view videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5366-5375, 2020.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Bruall and Pratul P Srinivasan. Mip-nerf: multiscale representation for anti anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5855-5864, 2021.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. arXiv preprint arXiv:2304.06706, 2023.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for au- tonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuPlan: A closed-loop ml- ml-based planning benchmark for autonomous vehicles. IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshop on Autonomous Driving: Perception, Prediction and Planning, 2021.
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 130-141, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference Computer Vision, pp. 333-33-3300 350. Springer, 2022.
Yilun Du, Yinan Zhang. Bong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. in 2021 iece. In CVF International Conference on Computer Vision (ICCV) pp. 14304-14314, 2021.
10

---

Published as a conference paper at ICLR 2024
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object recon- struction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605-613,4 2017.
Jiemin Fang, Taoran Yi, Xinggang Wang. Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, pp. 1-9, 2022.
Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1-479-12488. 2023.
Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5712-5722111 2021.
Jianfei Guo, Nianchen Deng, Xinyang Li, Yeqi Bai, Botian Shi, Chiyu Wang, Chenjing Ding, Dongliang Wang, and Yikang Li. Streetsurf: Extending multi-view implicit surface reconstruction to street views. arXiv preprint Sikiv::200.049988 2023.
Justin KerT, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Lan- guage embedded radiance fields. In International Conference Computer Vision (ICCV), 2023.
Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In Advances in Neural Information Processing Systems, pp. 23311- 23330, 2022.
Ruilong Li, Hang Gao, Matthew Tancik, and Angjoo Kanazawa. Nerfacc: Efficient sampling accel- erates nerfs. arXiv preprint arXiv:2305.04966 2023a.
Tianye Li. Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Xueqian Li, Jhony Kaesemodel Pontes, and Simon Lucey. Neural scene flow prior. Advances in Neural Information Processing Systems, 34:7838-7851, 2021a.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space- time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6498-6508, 2021b.
Zhengqi Li, Qianqian Wang. Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neu- ral dynamic image-based rendering. In Proceedings of the IEEESCVE Conference on Computer Vision and Paltern Recognition, pp. 4273-4284, 2023b.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106 2021.
nn, Thomas Müller. Liny-cuda anda-nn, April 2021. URL https:t//ttttiitie wwt.hhww...oo///oo/aao....aa
tiny-cuda-nn.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), (44: 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint axX1:22044.07133. 2023.
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2856-2865, 2021.
11

---

Published as a conference paper at ICLR 2024
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo the IEEE/CVF International Conference on Computer Vision, pp. 508555865544 2021a.
Nerfies: Deformable neural radiance fields. In Proceedings of
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, and Steven M Seitz. Hypemerr: A higheer-dimensionall representation for topologically varying neural radiance fields. arXiv preprint 11Xiv:2106.132285 2021b.
Ricardo
Christian Reiser, Rick Szeliski, Dor Verbin, Pratul Srinivasan, Ben Mildenhall, Andreas Geiger, Jon Barron, and Peter Hedman. Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes, ACM Transactions on Graphics (TOG), 42(4): 2023.
Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan T Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12932-12942, 2022.
Nur Muhammad Mahi Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, and Arthur Szlam. Clip-fields: Weakly supervised semantic fields for robotic memory. arXiv preprint arXiv: Arxiv- 2210.05663, 2022.
Prafull Sharma, Ayush Tewari, Yilun Du, Sergey Zakharov, Rares Andrei Ambrus, Adrien Gaidon, William T Freeman, Fredo Durand, Joshua B Tenenbaum, and Vincent Sitzmann. Neural ground- plans: Persistent neural scene representations from single image. In The Eleventh International Conference Learning Representations, 2022.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pp. 2446-2454, 2020.
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srini- vasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Xiaoyu Tian, Tao Jiang, Longfei Yun, Yue Wang. Yilun Wang, and Hang Zhao. Oce3d: sarge-ccale 3d occupancy prediction benchmark for autonomous driving. arXiv preprint arXiv:2304. 14365, 2023.
Nikolaos Tsagkas, Oisin Mac Aodha, and Chris Xiaoxuan Lu. VI-fields: Towards language- grounded neural implicit spatial representations. arXiv preprint 2rXiv:2305.12427, 2023.
Haithem Turki, Jason Y Zhang, Francesco Ferroni, and Deva Ramanan. Suds: Scalable urban dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23375-12385, 2023.
Chaoyang Wang, Ben Eckart, Simon Lucey, and Orazio Gallo. Neural trajectory fields for dynamic novel view synthesis. arXiv preprint arXiv:2105.05994 2021.
Chaoyang Wang. Lachlan Ewen MacDonald, Laszlo A Jeni, and Simon Lucey. Flow supervision for deformable nerf. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2212-221. 2023a.
Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Fourier plenoctrees for dynamic radiance field rendering in real-time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13524-13534, 2022.
Qianqian Wang, Yen- Yu Chang, Ruojin Cai, Zhenggi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all at once. arXiv preprint arXiv:2306. 2344002200. 2023b.
12

---

Published as a conference paper at ICLR 2024
Zian Wang, Tianchang Shen, Jun Gao, Shengyu Huang, Jacob Munkberg, Jon Hasselgren, Zan Gojcic, Wenzheng Chen, and Sanja Fidler. Neural fields meet explicit geometric representations for inverse rendering of urban scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8370-8380, 2023c.
Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztirlli, D 2nerf: self-supervised decoupling of dynamic and static objects from monocular video. Advances in Neural Information Processing Systems, 35:32653-32666, 2022.
Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1389-1399, 2023.
13

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
In this section, we discuss the implementation details of EmerNeRF. Our code is publicly available, and the pre- preitrained models will be released upon request. See emernerf github. io for more details.
A.1
EMERNERE IMPLEMENTATION DETAILS
A.1.1 DATA PROCESSING
Data source. Our sequences are sourced from the sion, which augments raw sensor data with point cloud flow annotations. For camera images, we employ three frontal cameras: FRONT_LEFT, FRONT, and FRONT_RIGET resizing them to reso- lution of 640 x 960 for both training and evaluation. Regarding LiDAR point clouds, we exclusively use the first return data (ignoring the second return data). We sidestep the rolling shutter issue in LiDAR sensors for simplicity and leave it for future exploration. Dynamic object masks are derived from 2D ground truth camera bounding boxes, with velocities determined from the given metadata. Only objects exceeding a velocity of 1 m/s are classified dynamic, filtering out potential sen- sor and annotation noise. For sky masks, we utilize the Mask2Former aer-FFormmr-arcittcturre
ver-
ViT-Adaptera model pre- pre-trained on ADE20K. Note that, the dynamic object masks and point cloud flows used for evaluation only.
Foundation model feature extraction. We employ the officially released checkpoints of DINOv2 Oquab et al. (2023) and DINOv1 (Caron et al., 2021), in conjunction with the feature extractor implementation from Amir al. (2021). For DINOv1, we utilize the ViT-B/16, resizing images to 640 x 960 and modifying the model's 's stride to 8 to further increase the resolution of extracted feature maps. For DINOv2, we use the ViT-B/14 variant, adjusting image dimensions to 644 966 and using stride of 7. Given the vast size of the resultant feature maps, we employ PCA decomposition to reduce the feature dimension from 768 64 and normalize these features to the [0, 1] range.
A.1.2 EMERNERF
Representations. To account for the efficiency of the 4d neural volume, we build all our scene representations based iNGP (Müller al., 2022). We use the iNGP implementations from tiny-cuda-nn (Müller, 2021), and use nerface toolkit (Li et al., 2023a) for acceleration. Fol- lowing Barron et al. (2023), our static hash grid adopts resolution between 2 to 2 over 10 levels, with fixed feature-length of 4 for all hash entries. Each level's feature entries are limited to a size 2 With these settings, our model comprises approximately 30M parameters 18M fewer than StreetSurf's s SDF representation (Guo et al., 2023). Our dynamic hash encoder maintains similar configuration but with a max hash feature map size of 218 We anticipate improved performance with larger hash encoders. All MLPs have hidden layer width of 64. To address camera expo- sure variations in the wild, we employ per-image 16-dimensional appearance embeddings for the scene reconstruction task and per-camera 16 16-dimensional embeddings for the novel view synthesis task. Our method builds upon efficient hybrid representations, InstantNGP (Müller et al., 2022). Parallel efforts in improving reconstruction efficiency are also made by Cao & Johnson (2023); Fridovich- Keil et al. (2023); Chen et al. (2022). We believe their representations are also suitable for Eme RE' encoders.
Positional embedding (PE) patterns. We use learnable feature map. denoted as U, with dimen- sions 80 x 120 x 32 (H W C)) to accommodate the positional embedding patterns, as discussed in the main text. To decode the PE pattern for individual pixel located at (u, v), we first sample a feature vector from using F. r..ridssampp Subsequently, linear layer decodes this feature vector produce the final PE features.
Scene range. To define the axis-aligned bounding box (AABB) of the scene, we utilize LiDAR points. In practice, we uniformly subsample the LiDAR points by factor of 4 and find the scene boundaries by computing the 2% and 98% percentiles of the world coordinates within the LiDAR
console clo.d. http:///wo....aaaaaaa/aa/a/a//ae
com/
fl
14

---

Published as a conference paper at ICLR 2024
point cloud. However, the LiDAR sensor typically covers only 75-meter radius around the vehicle. Consequently, an unrestricted contraction mechanism is useful to ensure better performance. Fol- lowing the scene contraction method detailed in Reiser al. (2023), contraction function to project the points falling outside the determined AABB.
peecewise-projectvv
Multi-level sampling. In line with findings in Mildenhall al. (2021); Barron al. (2021), we observe that leveraging extra proposal networks enhances both rendering quality and geometry esti- mation. Our framework integrates two-step proposal sampling process, using two distinct iNGP- based proposal models. In the initial step, 128 samples are drawn using the first proposal model which consists of levels, each having a I-dim feature. The resolution for this model ranges from 2 to 29, and each level has a maximum hash map capacity of 2200 For the subsequent sampling phase, 64 samples are taken using the second proposal model. This model boasts a maximum res- olution of 2!! but retains the other parameters of the first model. To counteract the "z-aliasing" issuc- partuccllllllllly prominent driving sequences with thin structures like traffic signs and light poles, we further incorporate the anti-aliasing proposal loss introduced by Barron et al. (2023) dur- ing proposal network optimization. A more thorough discussion on this is available in Barron et al. (2023). Lastly, we do not employ spatial-temporal proposal networks, i.e., we don't parameterize the proposal networks with a temporal dimension. Our current implementation already can capture temporal variations from the final scene fields, and we leave integrating a temporal dimension in proposal models for future exploration. For the final rendering, 64 points are sampled from the scene fields.
A.1.3 OPTIMIZATION
All components in EmerNeRF are trained jointly in an end-to-end manner.
Loss functions. As we discussed in $3.4, our total loss function is
L Ligh + Laky + Labadow + Las(pixet) + Layele Lieat + Ldepth + (La((
(A1)
for pixel
for LiDAR mys
With representing ray and N, its total number, the individual loss components defined as:
1. RGB loss (Crgh): Measures the difference between the predicted color (C(r)) and the ground truth color (C(r)) for each ray.
Ligb N, E1((()) C
(A2)
2. Sky loss (Lay): Measures the discrepancy between the predicted opacity of rendered rays and the actual sky masks. Specifically, sky regions should exhibit transparency. The binary cross entropy (BCE) loss is employed to evaluate this difference. In the equation, Ó(r) is the accumulated opacity of ray r as in Equation (5). M() is the ground truth mask with for the sky region and 0 otherwise.
Lak = 0.001
1 N,
Σ BCE 100))))M(r)))))
(A3)
Shadow loss (Cshadow): Penalizes the accumulated squared shadow ratio, following Wu et al. (2022).
K
Latuidow = 0.01
Σ
Toapp
(A4)
N,
Dynamic regularization (prxe)) and (TA(D))))):: Penalizes the mean dynamic density of all points across all rays. This encourages the dynamic branch to generate density only when necessary.
C. = 0.01
Σ
Σ
oa(r i)
(AS)
N,
K
15

---

Published as a conference paper at ICLR 2024
5. Cycle consistency regularization (Layete): relf-regularizes the scene flow prediction. This loss encourages the congruence between the forward scene flow at time and its corresponding back- ward scene flow at time +1.
0.01 Leyele 2
[sg(v)(xxt)) vi
( (x vx((x,),, 1)}² [sg(vo(x,)) ()) + v's (x+ + vo (x,,t),, }
(A6)
where y (x, t) denotes forward scene flow at time t, v t), 1++1) is predicted backward scene flow at the heward-warped position at time t 1, sg means stop-gradient operation, and E represents the expectation, i.e., averaging over all sample points.
6. Feature loss (Ceet): ) Measures the difference between the predicted semantic feature (F(r) and the ground truth semantic feature (F(r) for each ray.
Clea 0.5 Nr [1F(r) F()
(A7)
7. Depth Loss (Caeph): Combines the expected depth loss and the line-of-sight loss, described in Rematas et al. (2022). The expected depth loss ensures the depth predicted through the volumetric rendering process aligns with the LiDAR measurement's depth. The line-of-sight loss includes two components: a free-space regularization term that ensures zero density for points before the LiDAR termination points and near-surface loss promoting density concentration around the termination surface. With a slight notation abuse, we have:
depph = E, [| [1 - (
(A8)
Jrae 1 w(()²al
[2e (w(l) Ke ((-2)))))
Liine-of-sigh E,,
+ EE
(A9)
depph Lexp + 0.1 Lline-of-sight
(A10)
where (rr represents rendered depth values and (r) stands for the ground truth LiDAR range values. Here, the variable t indicates offset from the origin towards the ray's direction, dif- ferentiating it from the temporal variable discussed earlier. w(() specifies the blending weights of a point along the ray. Ke(x)) NO0,, (e/3)² represents kernel integrating N is truncated Gaussian. The parameter determines the strictness of the line-of-sight loss. Following the suggestions in Rematas et al. (2022), we linearly decay from 6.0 2.5 during the whole training process.
one, where
Training. We train our models for 25k iterations using batch size of 8196. In static scenarios, we deactivate the dynamic and flow branches. Training durations on a single A100 GPU are as follows: for static scenes, feature-free training requires 33 minutes, while the feature-embedded approach takes 40 minutes. Dynamic scene training, which incorporates the flow field and feature aggregation, extends the durations to 2 hours for feature-free and 2.25 hours for feature-embedded representations. To mitigate excessive regularization when the geometry prediction is not reliable, enable line-of-se loss after the initial 2k iterations and subsequently halve its coefficient every 5k iterations.
A.2 BASLINE IMPLEMENTATIONS
For HyperNeRF (Park et al., 2021b) and DeReRF (Wu et al., 2022), we adopt their officially released JAX implementations and tailor them to our dataset. We train each model for 100k iterations using a batch size of 4096. Training and evaluation for each model take approximately 4 hours on 4 A A00 GPUs for single scene. To ensure fair comparison, we enhance both models with an additional sky head and provide them with the same depth and sky supervision as in our model. Nevertheless, since neither HyperNeRF nor DeNeRF inherently supports separate sampling of pixel rays and LiDAR rays, map LiDAR point clouds to the image plane and apply L2 loss for depth supervision. We determine a scale factor from the AABBs derived from LiDAR data to ensure scenes are encapsulated within their predefined near-far range. For StreetSuRF (Guo et al., 2023), we directly quote the reported performance from their paper, as our experiments are conducted under the same setting.
16

---

Published as a conference paper at ICLR 2024
A.3 RUN TIME ANALYSIS
In Table A.I, we conduct a runtime analysis, Hybrid methods, such as our proposed EmerNeRE and StreetSurf (Guo et al., 2023), can be trained in relatively short duration of 30 minutes hours. This efficiency comes at the expense of utilizing explicit grids for encodings. In contrast, MLP-based models like DºN Wu al. (2022) and HyperNeRF Park et al. (2021b) have fewer parameters but require longer training time. It is important to note that the number of parameters in both implicit and hybrid models does not directly correlate with their performance. For instance, as illustrated in Figure of Müller et al. (2022), an MLP with 438k parameters can achieve better image fitting than hybrid model with a total of 33.6M parameters (10k from MLP and 33.6M from the grids).
Table A. Run-time analysis. We include the parameter counts for implicit MLPs and explicit encodings like HashGrids (Müller et al., 2022). Additionally, we provide metrics such as the number of GPU hours required for training, the total training iterations, and the batch sizes used, offering a comprehensive overview of the computational requirements. Specifically, StreetSurf (Guo et al., 2023) employs near-field hash encoder (32M) and a far-field encoder (16M). Our approach has a static hash encoder (30.55M parameters), with the optional inclusion of dynamic hash encoder (10.49M parameters) and a flow encoder (9.70M parameters).
Method
# parameters in MLPs
parameters in grids
GPU hours
Iters.
Seconds / 1k iters
Batch Size
Statie:
StreetSurf
0.IM 0.065M
32M 16M
1.26lus 0.58hrs
12,500
362.88s K3.52s
8192
Ours
30.55M
25,000
8192
Dynamic:
D2NeRF
3.03M
9,68hrs
100,000
348.48s
4096
HyperNeRR
1.32M
7.32hrs
100,000
263.52s
4096
Ours w/o flow
0.065M
30.55M 10.49M
0.88 hrs
25,000
126.72s
8192
Ours wet flow
0.065M
30,55M + 10.49M 9.70M
2,03 hrs
25,000
292.32s
8192
B NERF ON-THE-ROAD (NOTR) DATASET
As neural fields gain more attention in autonomous driving, there is an evident need for a compre- hensive dataset that captures diverse on-road driving scenarios for NeRF evaluations. To this end, we introduce NeRF To-The-Rool (NOTR) dataset, a benchmark derived from the Waymo Open Dataset (Sun et al., 2020). NOTR features 120 unique driving sequences, split into 32 static scenes, 32 dynamic scenes, and 56 scenes across seven challenging conditions: ego-static, high-speed, ex- posure mismatch, dusk/dawn, gloomy, rainy, and nighttime. Examples are shown in Figure B.
Beyond images and point clouds, NOTR provides additional resources pivotal for driving percep- tion tasks: bounding boxes for dynamic objects, ground-truie 3D scene flow, and 3D semantic occupancy. We hope this dataset can promote NeRF research in driving scenarios, extending the applications of NeRFs from mere view synthesis to motion understanding, e.g., 3D flows, and scene comprehension, e.g., semantics.
Regarding scene classifications, our static scenes adhere to the split presented in StreetSuRF (Guo et al., 2023), which contains clean scenes with no moving objects. The dynamic scenes, which are frequently observed in driving logs, are chosen based on lighting conditions to differentiate them from those in the "diverse" category. The Diverse-56 samples may also contain dynamic objects, but they are split primarily based on the ego vehicle's state (c.g.. ego-static, high-speed, camera exposure mismatch), weather condition (e.g., rainy, gloomy), and lighting difference (e.g., nighttime, dusk/dawn). We provide the sequence IDs of these scenes in our codebase.
C ADDITIONAL RESULTS
C. ABLATION STUDIES
Ablation on flow emergence. Table C.1 provides ablation studies to understand the impact of other components scene reconstruction, novel view synthesis, and scene flow estimation. For these
17

---

Published as a conference paper at ICLR 2024
(1) Static
(2) Dynamic
(3) Ego-static
(4) High-speed
(5) Rainy
(6) Dusk/Dawn
(7) Nighttime
(8) Mismatch expasure
(9) Gloomy
Figure B.1: Samples from the NOTR Benchmark. This comprehensive benchmark contains (1) 32 static scenes, (2) 32 dynamic scenes, and 56 additional scenes across seven categories: (3) ego-static, (4) high-speed, (5) rainy, (6) dusk/dawn, (7) nighttime, (8) mismatched exposure, and (9) gloomy conditions. We include LiDAR visualization in each second row and sky masks in each third row.
Table C.I: Ablation study.
Scene Reconstruction
Novel View Synthesis
Scene Flow estimation
Dynamic-Only PSNR1
Image
Dynamic-Only PSNR
PSNR1
PSNR1
Aec;(%)
(a) 40-Only INGP
26.55
22.30
26.02
21.03
flow
(c) no temporal uggregation
26.95
23.90
26.60
23.98
4.53%
(d) freeze lemporally displaced fentures hefore aggregation
26.93
24.02
26.78
23.81
3,87%
(e)
18

---

Published as a conference paper at ICLR 2024
(s) GT Image
(b) Wio Sky Head
(c) Wil. Sky Head
Figure C.1: Effect of sky head. The depth of the sky ill-defined. Using a sky head helps to remove the sky floaters.
Table C.2: Ablation study on different heads.
Scene Reconstruction
Novel View Synthesis
Methods
Full Image
Dynamic-Onf
Full Image
Dynamic-Onie
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
DPSNR
SSIM
(a) use separate RGB heads
28.789
0.8042
26.107
0.7146
27.444
0.7877
24.292
0.6656
w/a. sky head
28.778
0.8097
26.143
0.7255
27.533
0.7889
24.292
0.6654
w/t feat. head but w/t PE decump.
28.775
0.8045
26,259
0.7280
27.844
0.7931
24.264
0.6629
(d) w/t feat. hend and w/o PE decomp.
28.755
0,8038
26.134
0.7238
27.469
0.7829
23,945
0.6590
(c) use projected depth
28.874
0.8097
26,200
0.7275
27.532
0,7884
24.321
0.6642
Ours default: use a shared RGB head for &dynanie fields, use sky head, w/o feat. head, use LiDAR rays (f) default
28.873
0.8096
26,223
0,7276
27.547
0,7885
24.323
0.6655
ablation experiments, all models are trained for 8k iterations, shorter duration compared to the 25k iterations in the primary experiments. From our observations: (a) Using full 4D iNGP with- out the static field results in the worst results, a consequence of the lack of multi-view supervision. (b-e) Introducing hybrid representations consistently improves the results. (c) Omitting the tem- poral aggregation step or (d) freezing temporal feature gradients (stop the gradients of g and gh+1 in Fig. 2) negates the emergence flow estimation ability, as evidenced in the final column. Combining all these settings yields the best results.
Ablation on different heads. Table C.2 provides ablation studies on 5 randomly selected sequences from the Dynamic-32 Split, where we examine the effects of different heads used by our method on rendering performance. These models are trained for 25k iterations. We observe that our method performs reasonably well across all settings. Specifically, (a) using separate RGB heads for static and dynamic fields results in slightly lower rendering quality, particularly for dynamic components, which underscores the effectiveness of our strategy of sharing RGB heads between static and dy- namic fields. (b) Removing the sky head does not significantly affect rendering quality, but it does impact depth visualization in sky regions, as shown in Figure C.I. (c, d) Augmenting Eme NE with semantic feature heads can potentially enhance rendering. but this is contingent upon enabling the proposed positional embedding artifacts decomposition. Without this decomposition, the PSNR scores for dynamic objects decrease from 24.264 (c) to 23.945 (d) in the novel view synthesis task. (f) Utilizing projected depth map, as opposed to directly sampling LiDAR rays, yields results that similar to our default setting.
Ablation on the effect of supervision to flow estimation. Following the setting in the previous ablation study, we explore how different forms of supervisions and predictions affect the accuracy of flow estimation. The results are shown in Table C.3.
We begin by discussing the impact of depth losses. (b) Using an I.2 depth loss results in flow estimations with reasonable scales (i.e., achieving satisfactory Accş and Acc1o results), but these estimates come with large angle error. (c) Applying a line-of-sight loss yields better results. The line-of-sight loss encourages a concentrated density distribution along a ray towards its termina- tion point-a peaky distribution. This strategy effectively resolves the ambiguity associated with accumulating numerous noisy flows in the 3D scene, as it focuses on the core contributors of these flows: the dynamic objects. (a) Combining these two depth losses provides the optimal performance. Figure C.2 provides qualitative comparison of methods with and without the line-of-sight loss.
19

---

Published as a conference paper at ICLR 2024
(s) GT Image
(b) forward flow (combined loss)
(c) forward (Line sight loss only)
(d) forward flow Loss only)
Figure C.2: Qualitative comparative analysis of different depth losses on flow estimation
20

---

Published as a conference paper at ICLR 2024
We next discuss the ablation study on how to utilize flows. In existing NeRF-based methods (Gao et al., 2021; Li et al., 2021b; Wang et al., 2021; 2023a), flow fields are typically employed a three-step process (referred to as "Color-warping"> 1. Temporal Warping: Sampling points warped to their positions at the previous and/or next timestep to query their corresponding density and colors from nearby frames. 2. Accumulation: These densities and colors are then accumulated along corresponding rays in the current time step. Note that for each ray, we perform three individual integrations for density and colors from the past, the current, and the next timestep. This construction results in three distinct per-ray colors. 3. RGB Loss Computation: The RGB loss is calculated for each of these three colors.
However, the warped points are occluded in the previous or next timestep, making the RGB loss computation for these points invalid. As solution, NSFF (Li al., 2021a) proposed loss weight- ing strategy to alleviate this issue. Nonetheless, this approach, referred to as the "warping -based method," computationally expensive as requires querying the RGB network three times and rendering three distinct sets of rays.
Our approach differs by aggregating the warped features from both the previous and next timestep. We only query the RGB network with aggregated features and render single color per ray. This method effectively bypasses the occlusion issues associated with forward- and bockward-warpp renderings by focusing solely on the current timestep, while leveraging information from nearby frames. Additionally, by decoding the aggregated features through a color MLP, we effectively mit- igate potential artifacts introduced by color warping, leading to more robust rendering. To validate this hypothesis, we introduce two variations of our EmerNeRF:
1. Variation 1:
Renders forward- and warkwerd-warped rays separately, applying reconstruction losses to each (resulting in three colors per ray in the current timestep), as discussed above. We call this method "Color-warping".
2. Variation 2: Implements color aggregation instead of feature aggregation. This involves predict- ing forward- and backward-waid warred per-point colors, aggregating them by taking an average of per-3D location colors from the previous, the current, and the next timestep before taking inte- gration along each ray, and rendering the final output (resulting in one color per ray). We call this method "Color-ensemble". This is to study if the color MI. helps to mitigate feature noise.
Table C.3-(d,e) shows the results. We see that our default method (a) outperforms others (d.e), with the "Color-ensemble showing the least effective results. The "Color-warping" variant also demonstrated less accurate flow estimations compared to our default method while taking longer training time due to its extra network queryings and color renderings. In addition, all these methods demonstrate promising flow estimation ability. This analysis underscores the potential of utilizing neural fields for flow estimation, especially when depth signals are available, eliminating the need for heuristic methods or pre- trained models to provide optical flow labels. We believe this observation will inspire broader range of follow-up studies.
Table C.3: Ablation on the effect of supervision to flow estimation.
Settings
EPE3D (m) Acc, AC)
(rad) 4
Combined L2 and line-of-sight loss for depth; feature-nnsemble
(a) Ours default
0.0130
0.9272
0.9599
0.5175
Depth supervision:
(b) L2 loss only
0,0367
0,9077
0,9211
1.1207
(c) Line-of-sight--lt only
0.0136
0.9296
0,9574
0.5996
how to utilize flows:
(d) Color-warping
0.0190
0.9181
0,9370
0.6868
(c) Color-ensemble
0.0414
0.8929
0,9065
1.5708
C.2 QUALITATIVE RESULTS
Figures C.3 and C.4 show qualitative comparisons between our EmerNeRF and previous methods under the scene reconstruction setting, while Figure C.5 highlights the enhanced static-dynamic de- composition of our method compared to DºNeRF (Wu et al., 2022). Moreover, Figure C.6 illustrates our method's 's superiority in novel view synthesis tasks against HyperNeRF (Park et al., 2021b) and NeR (Wu et al., 2022). Our method consistently delivers more realistic and detailed renders.
21

---

Published as a conference paper at ICLR 2024
(4) D2NARF
(b) HyperNeRF
(c) NFFF (ours)
(d) Ground Truch
Figure C.3: Qualitative scene reconstruction compppar
(+) DINeRE
Oe) Hyperkwnlt
(s) EmerNoff tours!
(40 Ground Trush
Figure C.4: Qualitative scene reconstruction comparisons.
Notably, HyperNeRF does not decompose static and dynamic components; it provides only com- posite renders, while our method not only renders high-fidelity temporal views but also precisely separates static and dynamic elements. Furthermore, our method introduces the novel capability of generating dynamic scene flows.
22

---

Published as a conference paper at ICLR 2024
Sa) D2NeRF-ful
(b) D2NNRR dynamicRGi
<c) Our EmerNeRF-ful
GT RG3
Figure C.5: Scene decomposition comparisons. Note that we utilize a green background to blend dynamic objects, whereas D2NeRF's results are presented with white background.
C.3 LIMITATIONS
Inline with other methods, EmerNeRF does not optimize camera poses and is prone to rolling shutter effects of cameras. Future work to address this issue investigate joint optimization of pixel-wise camera poses and scene representations. In addition, the balance between geometry and rendering quality remains a trade-off and needs further study. Finally, leave for future work investigate more complicated motion patterns.
C.4 VISUALIZATIONS
23

---

Published as a conference paper at ICLR 2024
CT Image
Synchosis
Ours- Novel Scene Flow Synthesis
HyperNeks
in RPNARF
Dynamic Decomposiion
Figure C.6: Qualitative novel temporal view comparison.
24

---

Published as a conference paper at ICLR 2024
Images
GT DINOv2
(f) GT DINOv1
Reconstructed DINOv2
(g) Reconstructed DINOV1
(d) Decomposed PE-free DINOv2
(h) Decomposed PE-free DINOV1
(e) Decomposed DINOv2 PE patterns
(i) Decomposed DINOv1 PE patterns
Figure C.7: Different positional embedding patterns DINOv1 (Caron et al., 2021) and DINOv2 models (Oquab et al., 2023)
25

---

Published as a conference paper at ICLR 2024
GT RGB
Rendered RGB
Decomposed Dynamic Depth
(d) Frner gad Soana Flow
GT DINOV2 Features
(f) Raconstructed DINOw? Features
(g) Decomposed PE Patterns
(h) Stacking dynamic RGB Decomposed FE-Free Static DINDV2 Features
Figure C.8: Scene reconstruction visualizations EmerNeRF. We show (a) GT RGB images, (b) reconstructed RGB images, (c) decomposed dynamic depth, (d) emerged scene flows, (e) GT DINOv2 features, (f) reconstructed DINOv2 features, and (g) decomposed PE patterns. We also stack colors of dynamic objects onto decomposed PE-free static DINOv2 features.
26

---

Published as a conference paper at ICLR 2024
(e) GT
(c) becomposed Dynamic Depth
(c) Emerged Scene Flow
GT DINOv2 Features
DINOx2
(g)) Decomposed Patterns
(h) Stacking dynomic RGB on Decompased PE Free Static DINOV2 Festures
Figure C.9: Scene reconstruction visualizations of EmerNeRE under different lighting conditions. We show (a) GT RGB images, (b) reconstructed RGB images,
(c) decomposed dynamic depth, (d) emerged scene flows, (e) GT DINOv2 features, (f) reconstructed DINOv2 features, and (g) decomposed PE patterns.
also stack colors of dynamic objects onto decomposed PE-free static DINOv2 features. EmerNeRF works well under dark environments (left) and discerns challenging scene flows in complex environments (right). Colors indicate scene flows' norms and directions.
27

---

Published as a conference paper at ICLR 2024
(o) GT
(c) Decompased Dynamic Depth
(d) Emerged Scene Flow
GI DINOV2 Features
Reconstrusted
Decomposed
(h) Stacking dynamic RG8 on Decomposed PE-free Static DINOV2 Features
Figure C.10: Scene reconstruction visualizations of EmerNeRF under differet lighting and weather conditions. We show (a) GT RGB images, (b) reconstructed RGB images, (c) decomposed dynamic depth, (d) emerged scene flows, (e) GT DINOv2 features, (f) reconstructed DINOv2 features, and (g) decomposed PE patterns. We also stack colors of dynamic objects colors onto decomposed PE- free static DINOv2 features. EmerNett works well under gloomy environments (left) and discerns fine-grained speed information (right).
28


Reviews:
[{"review_id": "f9U1AShxjV", "reviewer": "Reviewer_qWy5", "summary": "The paper introduces EmerNeRF, an approach for learning spatial-temporal representations of dynamic scenes. EmerNeRF decomposes scenes into static and dynamic fields, with an additional scene flow field modeling the movement of objects across time. The dynamic feature is computed as a weighted sum of features from nearby timesteps, where the sampling operation is determined by the self-supervised scene flow field. Additionally, the paper proposes a method to generate positional encoding (PE) free features from a pretrained feature encoder by leveraging the time-consistent property of PE features. The empirical result indicates the proposed method can achieve better novel view synthesis, flow estimation, and few-shot semantic prediction results compared to the baselines.", "questions": "- What’s the novel components that EmerNeRF propses?\n- How does EmerNeRF perform in more general dynamic scenes?\n- How does EmerNeRF compare to recent newer baseline?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The proposed dataset holds potential value for research in dynamic scene reconstruction field\n- EmerNeRF demonstrates better reconstruction quality in driving scene dataset compared to the baselines\n- The obtained scene flow exhibits high accuracy compared to baseline method", "weaknesses": "- Novelty. Many design components of EMerNeRF have been proposed in previous work, including separated static and dynamic fields [2][3], sky head [1], shadow head [2], flow field [4]. The paper lacks a detailed discussion highlighting the differences compared to these previous works.\n- Generalizability. The proposed method lacks verification in existing dynamic scene datasets used in baselines such as Nerfies [5], HyperNeRF [6]. Those datasets contain more complex deformations and a significant proportion of dynamic components. It is necessary to evaluate the robustness and understand the limitations of the proposed design modules, including dynamic density regularization and self-supervised flow field.\n- Ambiguity of equation 6. The meaning and formulation of the expectation of the dynamic density remain unclear.\n- Baseline. The quantitative evaluation lacks a more recent baseline [7].\n\n[1] Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan T Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 12932–12942, 2022\n\n[2] Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli. Dˆ2nerf: Self-supervised decoupling of dynamic and static objects from a monocular video. *Advances in Neural Information Processing Systems*, 35:32653–32666, 2022.\n\n[3] Martin-Brualla, Ricardo, Noha Radwan, Mehdi SM Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. \"Nerf in the wild: Neural radiance fields for unconstrained photo collections.\" In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 7210-7219. 2021.\n\n[4] Du, Yilun, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, and Jiajun Wu. \"Neural radiance flow for 4d view synthesis and video processing.\" In *2021 IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 14304-14314. IEEE Computer Society, 2021.\n\n[5] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 5865–5874, 2021a.\n\n[6] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. *arXiv preprint arXiv:2106.13228*, 2021b.\n\n[7] Cao, Ang, and Justin Johnson. \"Hexplane: A fast representation for dynamic scenes.\" In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 130-141. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "64S20ma0B1", "reviewer": "Reviewer_BkKZ", "summary": "This paper presents a neural field approach that can perform static-dynamic decomposition and scene flow estimation in challenging autonomous driving scenarios. A hybrid 4D scene representation consisting of static, dynamic, and flow fields is adopted and jointly trained with the goal of appearance and feature reconstruction. Experimentation reveals state-of-the-art performance in novel view synthesis and scene flow estimation.", "questions": "The experiment setup omit every 10th frame, resulting in 10% novel views for evaluation. It would be better to include ablation study on the sparsity of the sampling and how the method degrade with fewer training frames.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes an effective way of jointly modeling the static and dynamic scenes in the setting of autonomous driving. The method is technically sound, from the high-level idea of using hybrid representation for static and dynamic scenes and optimize them under the goal of appearance reconstruction to the details of carefully modeling sky and shadows in the framework.\n\nEven though there is no direct flow supervision or well-adopted supervision such as flow based warping, it's quite novel to see the scene flow estimation \"emerges\" from the temporal aggregation of features for scene reconstruction.\n\nThe experimentation is thorough and the numerical improvements over baselines are obvious.\n\nThe writing and presentation of this paper are also pretty good and easy-to-follow.", "weaknesses": "It's good to see more visualization of the model output in the appendix. But it would also be good to include more qualitative comparisons against the baselines in addition to the quantitative results. Also, adding error maps would be more intuitive to highlight the difference.\n\nThere also seems to miss the runtime analysis and comparison. How long does this approach take for sensor simulation in training and test time, and how does it compare to the existing approaches? That would be an important piece of information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4udIVprYis", "reviewer": "Reviewer_hxFT", "summary": "This paper presents a NeRF based method for learning scene representation. By decomposing the loss function into two separate terms - one handling static elements of the scene and the other dynamic elements the model is able to separate the static and dynamic components with no extra supervision. The model also estimates scene flow in the process as part of the regularization. Finally the method is also applied to \"lifting\" 2D self-supervised representations (e.g. DINO) into 4D space by a combination of readout heads and a learned positional embedding which rectifies some of the issues associated with these SSL representations.", "questions": "In relation to the above:\n\n* How much dependance is there on LiDAR data? would this method work without direct depth supervision signals?\n* It's not written explicitly in the paper - does the data include ground truth camera parameters? (extrinsic and intrinsic, I imagine the answer is yes)\n* The dynamics regularization term is a bit simplistic as it is a simple minimization of total density for dynamic elements - have you tried other regularization methods? say assuming sparsity and so on?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper presents a nice combination of ideas that have been floating around for a while, leveraging the strengths of different approaches in an appealing and relatively elegant way. \nThe paper is well presented, well executed and results are impressive all in all (but see below) - this is a good paper.", "weaknesses": "I think there are a couple of weaknesses that may require addressing\n\n* Ablation analysis is lacking - table C1 addresses some of the of the modeling decisions but there are more aspects I would have loved to see analyzed.\n* Applicability - I may be wrong, but I feel the use of driving data together with a NeRF based method (that is, need to train on each scene separately) is a bit odd - usually in this use case one would want an online inference model (e.g. an encoder) which can infer elements and structure quickly as the car/robot drives around the scene. On the other hand I feel this is not widely applicable to other types of data (say free form natural scenes with lots of unstructured movement).\n* There is a lot of focus on DINO/v2 features and their related issues - I am wondering if other SSL methods suffer from similar PE issues (specifically ones that handle motion such VideoMAE / Siamese MAE etc.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GOhS7ozTb6", "reviewer": "Reviewer_fQX6", "summary": "This paper proposes EmerNeRF to decompose a scene into static and dynamic components by learning from videos using a self-supervised manner. The authors further propose to lift visual features from foundation models into the 4D space, by learning a shared PE feature map and PE-free volumetric feature fields. The authors also construct a new benchmark by subsampling video sequences from the Waymo Open Dataset. The proposed method achieves good results on several tasks, including scene reconstruction, view synthesis, and occupancy prediction.", "questions": "Please see the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is intuitive, simple yet effective, achieving promising results on several tasks\n2. I like the study of PE patterns in vision foundation models and the solution to it. The visualization in Figure 1 is also good.", "weaknesses": "**1. Method**\n- First of all, the authors claim “no prior works in this field have explicitly modeled temporal correspondence for dynamic objects”, which is wrong. Some related works are missing here [A1-3].\n- It seems to me that the formulation of Gao et al. [A2] is very similar to the proposed method. Except Gao et al. [A2] used: (1) MLP-based NeRF instead of hash grids; (2) separate RGB heads for static and dynamic branches; (3) Not using a sky branch. However, Gao et al. [A2] showed that their method would fail without the optical flow regularization. So I wonder what is the core reason that the proposed method can work while Gao et al. [A2] cannot. Is it due to the hash grid? Is it due to ground truth depth supervision (from lidar)? Is it due to the dataset being evaluated (multiview-view, vehicle motions are rigid and thus simpler)? Or is it due to the evaluation protocol (relatively easy to do view synthesis for frame interpolation)? While I prefer such a simple method, the reason that makes it work remains unclear to me.\n\n**2. Experiments**\n- For the results in Table 1 and Table 2, is visual feature distillation being used?\n- Is it possible to show to number of parameters in Table 1?\n- The comparisons with baselines in Table 1 also seems problematic. First, the authors project lidar points to image and use a L2 loss for depth regularization for HyperNeRF and D2NeRF, which is definitely worse than directly regularizing on lidar rays due to many reasons (e.g., projection error, possible occlusions). While the authors claim that neither of these two baselines support lidar ray sampling, it is actually not difficult to incorporate this functionality. An easier way is to use the same way to regularize depth in the proposed framework. Second, StreetSuRF has its own normal supervision and lidar ray preprocessing. Simply disabling them and use the same one as in the proposed method does not seem correct to me.\n- While not necessary, would be good to see a comparison on the scene flow estimation task with a SOTA method [A4].\n\n\n**Summary**\n\nMy main concern is that I don’t know what is the core reason that makes the proposed work. It would be great if the authors can conduct more extensive ablation analysis or conduct experiments on other datasets (especially those monocular ones used for dynamic view synthesis). Also, I would expect to see more reasonable comparisons with the baseline methods. The authors should also incorporate the missing literature and discussion accordingly.\n\n\n**Reference**\n- [A1] Wang et al. “Neural Trajectory Fields for Dynamic Novel View Synthesis”\n- [A2] Gao et al. “​​Dynamic View Synthesis from Dynamic Monocular Video”. ICCV 2021\n- [A3] Wang et al. “Flow supervision for Deformable NeRF”. ICCV 2023\n- [A4] Li et al. “Fast Neural Scene Flow”. ICCV 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)