D^2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video
Authors: Tianhao Walter Wu
Fangcheng Zhong
Andrea Tagliasacchi
Forrester Cole
Cengiz Oztireli
Venue: NeurIPS 2022 Accept (2022)
Keywords: Scene Decomposition
Video Segmentation
Novel View Synthesis

Abstract: Given a monocular video, segmenting and decoupling dynamic objects while recovering the static environment is a widely studied problem in machine intelligence. Existing solutions usually approach this problem in the image domain, limiting their performance and understanding of the environment. We introduce Decoupled Dynamic Neural Radiance Field (D^2NeRF), a self-supervised approach that takes a monocular video and learns a 3D scene representation which decouples moving objects, including their shadows, from the static background. Our method represents the moving objects and the static background by two separate neural radiance fields with only one allowing for temporal changes. A naive implementation of this approach leads to the dynamic component taking over the static one as the representation of the former is inherently more general and prone to overfitting. To this end, we propose a novel loss to promote correct separation of phenomena. We further propose a shadow field network to detect and decouple dynamically moving shadows. We introduce a new dataset containing various dynamic objects and shadows and demonstrate that our method can achieve better performance than state-of-the-art approaches in decoupling dynamic and static 3D objects, occlusion and shadow removal, and image segmentation for moving objects. Project page: https://d2nerf.github.io/

DeR: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video
Tianhao Wu University of Cambridge
Fangcheng Zhong University of Cambridge
Andrea Tagliasacchi
Google Research
Simon Fraser University
Forrester Cole
Cengiz Oztireli
Google Research
Google Research
University of Cambridge
Abstract
Given a monocular video, segmenting and decoupling dynamic objects while recov- ering the static environment is a widely studied problem in machine intelligence. Existing solutions usually approach this problem in the image domain, limiting their performance and understanding of the enviromment. We introduce Decoupled Dynamic Neural Radiance Field (D² NeRF), self- euffssupeeiied approach that takes monocular video and learns 3D scene representation which decouples moving objects, including their shadows, from the static background. Our method repre- sents the moving objects and the static background by two separate neural radiance fields with only one allowing for temporal changes. A naive implementation of this approach leads to the dynamic component taking over the static one as the representation of the former is inherently more general and prone to overfitting. To this end, we propose a novel loss to promote correct separation of phenomena. We further propose a shadow field network to detect and decouple dynamically moving shadows. We introduce a new dataset containing various dynamic objects and shadows and demonstrate that method achieve better performance than state-of-the-ard approaches in decoupling dynamic and static 3D objects, occlusion and shadow removal, and image segmentation for moving objects. Project page: d2nerf. gittuubbi
1 Introduction
Reasoning about motion is fundamental task in machine vision which facilitates intelligent inter- actions with the 3D environment for applications such as robotics and autonomous driving. Given monocular RGB video captured from moving casual camera, consider the problem of dis- entangling the camera from object motion, and simultaneously recovering a 3D model of the static environment.
While decomposition of scenes in the image domain has been addressed in the literature, the use of 2D priors and inpainting technique lacks 3D understanding, leading to sub-optimal results. We approach this problem in 3D, aiming to reconstruct decoupled 3D scene representation that allows for synthesizing the dynamic and static objects separately in free-view and time-vary tinevvayying fashion.
Compared to the task static scene reconstruction [32]. modeling a scene with time-dependent effects is severely ill-posed problem. Existing works seek for robust solutions by incorporating
additional supervision such as multi-view capture 25 optical flow 10, or depth 660, but they treat
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

every part of the scene as time- dime-depen,ent, leading to poor reconstruction of background details due to limited network capacity.
In this paper, we adapt Neural Radiance Fields 32 (NeRF) and its extension HyperNeRF 40 to
time-varying scenes by decoupling the dynamic and static components of the scene into separate radiance fields. Previous techniques that decouple dynamic and static scenes either rely on pre-trained
I2 23], or are limited to a single rigid object [68] or semi-static objects [54]. Our method learns dynamic and static components separately in
object detectioon/eemmenttioonn modules [99 26
self-supervised fashion, using novel skewed-entropy loss to encourage a clean separation of static and dynamic objects.
A crucial issue in creating clean separation is properly handling shadows, dynamic objects cast shadows that cause the radiance of the shadow receiver to vary with time. When the shadow receiver is part of the static component, this time-varying change in radiance cannot be directly modeled. Our solution is to relax the static component with a time-varying shadow field that modulates the radiance, allowing the shadows cast by moving objects be captured while constraining density and color to be static.
Our method enables 3D scene decoupling and reconstruction from monocular video captured from casual equipment such as a mobile phone, and can be readily extended to multi-view videos. By separately modeling the time-van varying and ineeiindeenneett targets in the video, our method can remove the dynamic occluders and their shadows, and synthesize a clean background from novel views.
We demonstrate the effectiveness of our method in two aspects: (i) the quality of novel view synthesis of the decoupled static background for monocular videos where the dynamic objects and shadows heavily occlude the scene, and (ii) the correctness of segmentation of dynamic objects and shadows on 2D images.
We introduce a new dataset with rigid and non-rigid dynamic objects, rapid camera motion and various moving shadows in both the synthetic and real-world settings to evaluate these two aspects, and show that our method achieves better performance than -fate-of-the-arr approaches.
2 Related Work
As our method learns decoupled neural 3D representation of the dynamic and static scenes, we start this section with a review of scene representations, and then focus on methods for object motion decoupling, We also review prior works for 2D segmentation of moving objects.
Scene Representations A 3D scene representation is a data structure that encodes the geome- try and appearance of a 3D scene, upon which many algorithms and applications are developed. Recently, there has been a surge of methods that combine deep learning methods with traditional
3D representations: point clouds 17 41 II I. meshes 7 33 551. voxels (6] 59] implicit surfaces [88 63 15
14
light fields 50 52 48 and volumetric fields 28 32 Among neural
representations, NeRF 32 has attracted substantial attention due to its photo-realistic performance in novel view synthesis for scenes with complex geometry, lighting, and materials. Via differentiable volume rendering and inputs of multiple views of the scene, NeRF applies an MLP learn 5D radiance field of the scene modeling the spatially and view-dependent radiance. Various extensions of NeRF have been developed improve performance and generality such as training with only allowing for input images with inconsistent lighting and object
one or few views 67 18
35 45
locations 31 69 learning large-scat scale scenes with street or satellite views 44
61 speeding up
rendering to reduce training and inference time [2 g 44 46 66 13 34
27 53 58 65), capture of
dynamic effects within the scene [25] 43
60
36 and decomposing self-occluded
shadows to recover correct albedo
9 We further extend NeRF to decouple dynamic from static
effects.
Motion Decoupling Prior works to acquire a decoupled 3D representation of dynamic and static scenes can be divided into either supervised or self-supervised approaches. Among the supervised approaches, STNeRF [19 learns individual NeRFs with deformation fields for each human in
a dynamic scene through pre-trained human segmentation networks. Similarly, NSFF [26] and DynNeRF [2] rely pre-trained semantic and motion segmentation methods to obtain masks for

---

(x d. r2)
Composite Rendering (Sec 3.1)
x
c'(1 P)
Photometric
Skowed Entropy and Ray fization
Shadow Field
Regularization (Sec 3.3]
Figure I: Overview Given the ground truth view, camera pose and the time frame, method reconstructs the underlying scene as a composite radiance field. Dynamic objects are represented by FD, while the static scene is represented by FF. The shadow-field FF models non-static shadows within the input video.
moving objects a monocular video, and explicitly guide the training of separate NeRF networks decouple the scene based on motion. Among the self-supervised approaches, SIMONe [20] incorporates transformer encoder and variational autodecoder to simultaneously recover novel views, object segmentation masks and dynamic object trajectories, but they do not allow for synthesis of dynamic or static objects alone. NeRF-W [3I employs per- per-frame embeddings to model non -on-photoeetric consistent effects in unconstrained photo collections, but their design was not intended for clean separation between moving objects and the static scene. STaR [68] reconstructs and decomposes a rigid dynamic object and the static background simultaneously by optimizing two NeRFs and a set of time-vare time-varying object poses in self-supervised way. but it is only suitable for scenes with a single rigid dynamic object and requires multi-view videos. Conversely, our approach works with more complex scenes involving multiple non-rigid and topologically varying objects, and our method can be directly applied from monocular video. NeuralDiff (54) incorporates three NeRF-based streamlines to decompose background, object and actor from an egocentric video, and is the most similar to ours within the literature. However, its use of naive time-varying NeRF architecture leads to blurry results and therefore heavily limits its performance both scene decomposition and reconstruction.
Image Segmentation of Moving Objects Orthogonal reconstruction and disentanglement in 3D, there have also been extensive researches in self-supervised and template-free segmentation at the image level (i.e. 2D). The majority relies on motion-clues segment objects with different optical Some techniques incorporate transformer style slot-based attention scheme to learn consistent object segmentation over a sequence of optical flow images [64], while others learn background and extend the segmentation target to correlated effects such as shadow or reflectance (30]. Those approaches come with obvious shortcomings, as they focus image level segmentation and incorporate no 3D understanding, they cannot handle large scale camera motion, complicated static background and cannot recover 3D geometry, perform novel view synthesis.
flow patterns 4 64 37 62]
from single video with smooth camera movement and homographic
Method
Given monocular video captured from a freely moving hand-held camera, our method reconstructs neural scene representation that decouples moving objects from the static environment, assuming

---

As illustrated in Figure our method achieves this by learning separate radiance fields for static and dynamic portions of the scene, and doing so in a fully self-supervised fashion. We describe our architecture (Section 3. detail of our supe-vsee losses (Section (.2). and describe how, while shadows are not explicitly modeled by NeRFs, simple technique for their effective removal is attainable (Section 3.3
constant illumination and known camera poses (e.g. calibrated with COLMAP 49
3.1 Composite Neural Radiance Field
The static component builds upon NeRF [3), which represents the scene continuous spatial- dependent density and spatial-view-deeeedeen radiance c using an multi-layer perceptron JF:
の. (x) c [x, d) E TR3
E
J (x, d)
(1)
where x E R is the spatial coordinate, and d E RS |||| - is the view direction. To model the dynamic component of a scene, we adapt HyperNeRF [40] which accurately captures scenes with non-rigid motion well as topological changes by introducing additional degree of freedom and network capacity. For convenience, we denote it as a neural function FD;
o (.) E
= F d,)
(2)
c
where T; E R" is the per-frame time latent code. Given a camera ray r = 0 - td originating from o and with direction d, the two models are then composited to calculate the color C of the camera ray by integrating the radiance according to volumetric rendering within pre-defined depth range [f,, tyl:
(t) + (t T:) ct, T:)) dt /'' T(t) - In
Č(r,T,)=
(3)
T(t) = exp (-f((09(0) D
(-) +,))
ds
(4)
where we simplify our notation as otl) (r(()) and c(l) c(r((), d). Note that, with such an additive decomposition, samples from either fields are capable of terminating the camera ray and occluding the other.
3.2 Supervision Losses
To find the parameters of the static [Eq. [ and dynamic (Eq 2 NeRF networks, a photometric loss is applied to ensure that the output image sequences of the composite NeRF (Eq, (3) align with the input video frames:
(r T.) = ||C(r, T.) - C(I,T,))l
(5)
where C(r, T:) indicates the true color of camera ray obtained from the i-th input video frame. However, note the dynamic component can naturally take over the static counterpart by incorrectly assigning occupancy of static objects dynamic NeRF, and the photometric loss alone also does not guarantee a correct separation. In what follows, we design a collection of regularizers that promote such decoupling in self-supervised fashion.
Dynamic vs. Static Factorization As physical objects cannot co-exist at the same spatial location, physically realistic solution should have any position in space either occupied by the static scene or by a dynamic object, but not both. To enforce this behavior we denote the spatial ratio of dynamic vs. static density
σ (x. Ti)
w(x, Ti) D Ti) + (x) E [0,
(6)
and then penalize its deviation from a categorical {0, distribution via a binary entropy loss 688
Cr T:) = '/ Ho(w(r(t), T()) dt
H₂(x) = (xx log(x) + +11 - 2) log([ x))
(8)

---

Ho(w))
Inpur
Statc
k=1
2.875
Figure 2: Skewed entropy- (left) the skewed (k >> 1) and classical
entropy losses, A
skewed entropy encourages wider range of to decrease and has larger gradient values around 0.5, but its gradient vanishes when w approaches O. (right) The decoupled alpha masks and static components when original, properly-skewed and over-skewed binary entropy losses are applied.
rnput
b
Dynamic
Static
Dynamic
Static
w/oo F
Figure 4: Shadow ambiguity - When shadows occur fre- quently in the input data, the average shadow gets integrated the static component, and the dynamic component incor- rectly learns the differential with respect to this average and appears as a brighter surface. This can be avoided by a more direct modeling of shadow effects dynamic darkening of static regions (i.e. the shadow field).
Figure 3: Static regularization By encouraging a more con- centrated density distribution along each camera ray in static com- ponent, the recovered background contains less view-dependent arti- facts.
However, due to the strong expressive power of the dynamic networks (Eq. 2). optimizing the loss (Eq. ( leads to the technique modeling parts of the scene dynamic, regardless of whether they are dynamic or static; see Figure [ (right). To overcome this issue, we propose skewed entropy loss bias our loss slightly favor static explanations of the scene with skewness syper-parameter that, as illustrated in Figure 2 (left, 1>11), attains the desired behavior:
C.(r, Ti) (.)r,,))=. /' Ho(v(((()) di
Ray Regularization Choosing a large value of skewness causes the appearance fuzzy floaters (low-density particles) in the static portion of the scene; see T k=10), As it can be intuitively understood from Figure 2 (left,, this caused by the small gradients of Hy as approaches zero. To mitigate this effect, and reduce fuzziness in the reconstruction, we penalize the maximum of w along each camera ray:
(, = max w( w(r(t), T:)
(10)
sEfo..ty)
Such loss can be intuitively interpreted as constraining the dynamic component occupy as few pixels as possible while keeping minimal impact on the overall loss for all samples. Note that L, only removes density floaters that sit along camera rays that do not intersect with any dynamic objects.
Static Regularization We empirically found that static component may abuse the camera pose as the hint for the current time frame and learn dynamic effects as sparse clouds that lead to high- frequency appearance changes; see Figure The ambiguity comes from the fact that we are using monocular casual videos where the camera almost never visits the exact same position twice during the capture. That is, there exists one-to-one mapping between camera pose and time variable. We
5

---

solve this issue by imposing prior on the distribution of density along a ray, penalizing density distributions that would cause cloud-like artifacts 22 45
s (r) = 100=-ap 1.¹h p(t) -log p(t) dt where p(t) (r(()) Jr. o (3((s)) ds
(11)
3.3 Shadow Fields
Neural radiance fields cannot faithfully model standalone shadows without significant changes to its architecture necessary to modeling materials and illumination; see NeRFactor [71]. In simple cases where shadows the dynamic objects move rapidly, they could alternatively be learned by the dynamic radiance field semi remi-transparent layers top of the static surface. However, this tends to fail for shadows that do not move much, or that are highly correlated with the camera motion. As shadows are texture-less, understanding their movement is ambiguous, and representing them as a semi- uomi-transparent layer causes difficulties in the optimization; see Figure 4 To overcome this issue, and under the assumption of direct illumination model (i.e. negligible global illumination effects), we make the observation that a cast shadow can be represented as a pointwise reduction in the radiance of the the static scene, and incorporate this within Eq.
as:
Ö(r,T))= = f"" T()((((L- p(r(t), T()((L-(((()))) - (t) c (t) (t,, T:))
(12)
p(x, T.) E [0, 1] = Fx Ti)
(13)
where p(x, Tú)) is shadow ratio that scales-down the radiance of the static scene to incorporate the shadow. To avoid the shadow-ratio from over-explaining dark regions of the scene, we penalize its average squared magnitude along a ray:
C,(r, i1-tr [.' p(r(1), T:) dt
(14)
Finally, note that shadows cast from dynamic objects onto other dynamic objects already expressed from the radiance term of the dynamic branch, and do not need explicit modeling.
Experiments
4.1 Implementation details
Our method is easily reproducible, as we intend to release code and datasets upon publication to facilitate future research. We adopt the HyperNeRF [40] architecture as the dynamic component, which has NeRF MLP network of 8 layers, each with 256 channels, and our static NeRF component has the same architecture. Similar NeRF [32], we apply hierarchical volume sampling with 64 coarse and 64 fine samples. The optimization takes 100k iterations with batch size 1024 and an exponentially decayed learning rate from 10-3 3 10-5 This training procedure spans approximately two hours on four NVIDIA A A100-SXM-80GB GPUs. The overall loss of our method is:
= (T, + (I,
+ (I, + Ao s
(I,
(15)
where Ao Ay. Aas, Ao are the weights of the regularization terms respectively. For scenes with mixture of dynamic objects and shadows, we apply shadow decay and set =0=1.1 We set A,=0.001 for scenes featuring view-correlated dynamic shadows only. We experimentally found that the optimal choice of the hyperparameters, especially λo A, and the skewness kt are strongly influenced by the level of object motion, camera motion, and video length. Therefore, we performed grid search on our synthetic and held-out real-world scenes, and some scenes from DAVIS 42), to establish a set of hyperparameters applicable to a variety of scenarios; details about hyperparameters can be found in the suppiementary. We do not apply shadow field for evaluations on our synthetic scenes, we empirically found that shadow field not needed learn correct shadows. We also disable the view direction input for synthetic scenes they do not contain strong deeendepnddent effects.

---

Pick2
Water Cookie
Cars
Hag
NeuralDDiff
.208
.159
MG 6
.333
.699
.484
.044
.424
HN [40]
.486
.233
.187
.361
.162
.290
NeuralDiff
34
806
508
080
.368
.372
Ours
.253
.214
.153
.153
.156
.186
Ours
.448
.70
.703
.551
.693
.717
NeuraiDiff
HyperNeRF
Ours
MC
NeuraiDiff
Ours
Figure 5: Novel view synthesis LPIPS.
Figure 6: Video segmentation -J1
4.2 Evaluation
We demonstrate the performance of our method both quantitatively and qualitatively on three tasks. We focus our attention to our main objective of decoupling and removing dynamic objects, including their shadows, with 3D reconstruction of the static environment. We only include a summary of our results for novel view synthesis (in Figure 5 and video segmentation (in Figure 6 and refer the reader to the Supplementary and I D for a more in-depth discussion. We strongly encourage the readers watch videos on the project page http://d2nerf.iithub.. to better appreciate our results.
4.3 Datasets
In addition to the data obtained from HyperNeRF 40 and Nerfies [39,, we acquire more complex datasets in the real-world, as well as design synthetic dataset to enable quantitative comparisons.
Synthetic dataset We generate synthetic dataset with ground-truth masks for moving objects and their shadows with Kubric [16]. This dataset consists of five scenes containing one multiple dynamic objects from ShapeNet [5] with rigid or non-rigid motion, and the corresponding Kubric worker script is provided in supplementary material. We move the virtual camera over 10 keyframes randomly sampled from azimuth [2,2 + T/4 and altitude [1, 1.2] to generate a 200-frame video sequence for training. We also rotate the virtual camera around the center of all keyframes to generate 100 validation views with only the static background being visible. We additionally generate masks for both the dynamic objects and their shadows, allowing to quantitatively study the performance of our algorithm. Note that shadows are usually absent in existing moving- obbjects segmentation benchmarks.
Real-world dataset We also capture ten video sequences real scenes to showcase our perfor- mance. Compared HyperNeRF's, our dataset contains more challenging scenarios with rapid motion and non-trivial dynamic shadows. Note however that we canmot perform quantitative analysis for these datasets due the absence of ground-truth views of static scene ground- truth masks. Five of real scenes are captured with a similar setting as Nerfies, where we use dual-hold phone rig and synchronize the capture based on audio. We use the images captured by one of the two phones validation views for novel-view synthesis, which are discussed in the Supplementary E To demonstrate the ability of fully self-supervised scene decoupling, we do not apply any masks when registering real-world images using COLMAP 49
4.4 Scene Decoupling Table Figure
We report the evaluation of our method on its ability to decouple dynamic objects and their shad- ows, while recovering the static background. We evaluate our performance against NeRF-W [II], NSFF 26 and NeuralDiff [54]. For NeRF-W, we used only transient embedding and disabled the appearance embedding that models variable lighting for evaluation on synthetic scenes, as they have constant illumination. For NSFF, we disabled the hard mining initialization via 2D masks input. For NeuralDiff, we disabled the actor component (as our input videos are not egocentric) and only use the transient component.

---

Chairs
Pillow MS-35041
Mcan
38 SSIM PSNAL
MS-35IMI
PANK
LISPS)
NS-SSIMI
PSNIKT
PSNRI
LIU1S
LIVIS)
MS-SSEMI ISNIRT
N
218
24.23
243
.873
24.51
.791
20.65
65
23.27
.935
28.24
.177
.819
24.28
NSFF 26
200
506
24.90
620
.376
10.29
.108
.892
25. 60
662
-284
12.82
.782
.343
4.55
47%
.540
15.64
NeumiDur
4
065
.52
11.89
198
9921
25.9%
.117
910
20.
112
.722
74.42
.565
.612
20.09
.11
.831
26.27
Ours
.62
975
34.27
.990
.953
26.27
J776
.979
34.14
.995
.707
24.63
.766
.979
36.58
.080
.919
31.18
Input
NeuralDiff
Table I: Scene decoupling (quantitative) We train each scene with 200 frames, decouple the dynamic objects and shadows, and render the static component from 100 novel views to compare with ground truth. Note these are computed on the synthetic dataset, for which ground truth available.
NeRF-W
NeuralDiff
Mask
Dynamic
Static
Mask
Dynamic
Static
PIC
BALLOON
WATER
CURLS
Figure 7: Scene decoupling (qualitative) We visualize results (top) real scenes, and on (bottom) scenes from HyperNeRF [40] and Nerfies [39]. To better illustrate the decoupled object and shadow, render the dynamic component with white background. Note that since method only decouples dynamic targets, it does not include parts of objects that remain still throughout the capture, such the body in "curls" and "banana" scenes.
In the evaluation, used each method to synthesize the static background from multiple validation views with the moving objects and their shadows removed; see Figure 7 for qualitative results on real data, well as Supplementary C for qualitative results synthetic data. We compare the results with the ground truth on the synthetic data and report LPIPS [7]] Multi-Scale SSIM [57], and PSNR the metries for novel view synthesis of the decoupled static background; although NSFF similarly learns separate dynamic and static NeRFs, it is not designed to maximize the performance of scene decoupling and does not incorporate regularization the dynamic NeRF. Therefore, it is extremely unstable and can fail completely by learning everything as dynamic, leaving static component empty scene.
Table Note that
Ablations Table Figure
Figure 9
quantitatively ablate our method on our synthetic dataset; see Table 2 where "skew" means skewness is applied in the binary entropy regularization, and "2," stands for the density ratio ray regularization. We also qualitatively ablate on real scene; see Figure We also qualitatively illustrate the ablation on shadow field network, which is necessary for decoupling shadows with

---

Car
Cars
Bag
Chairs
Pillow
Mean
skew C. EMPS,
SSMM
PNSRT
MS-55IMT
MS SS1M
PHSRR
[PIPS.
M5-5SIM1
PMSR
1PUSL
MS-S5IMT
PSSR1
SSMM
PMIRT
214
.834
26.26
.119
.943
26.10
.254
.666
19.96
104
.698
24.42
.385
.671
14.24
.255
.762
22.20
182
865
25.89
260
803
22.47
.189
893
28.38
107
.693
24.44
.311
.770
15.27
.210
.805
23.29
104
31.55
151
07
.062
.95
34.27
.99
.953
26.27
.076
.979
34.14
095
.707
24.63
.076
.979
36.58
.080
.919
31.18
Table 2: Ablations (quantitative) - We train on each scene with 200 frames, decouple the dynamic
objects and shadows, and render the static component from 100 novel views for metric evaluations.
Input
w/o skew
w/o Ep
w/o skew, w/o Ep
w/ skow, w/ Ey
Figure 8: Ablations (qualitative) For scenes with slow motion strong view-dependent reflectance. C, is used together with the skewed entropy to prevent the dynamic component from incorrectly decoupling the scene. In the scene above this appears slightly darkened color the table.
Inout
P*)
Figure 9: Ablations (shadows) Our method is able remove large area of shadows, even if they are strongly correlated with the view direction (e.g., shadow cast by camera or the photographer). Note that the appearance embedding from NeRF-W [1 is not sufficient to remove shadow that present throughout the capture; see the our wehsite for additional qualitative results.
large area, slow or repetitive motion, or shadows that are highly correlated with the camera view; see Figure g
Conclusions
We presented DeNeRF, method for self-supervised 3D scene decoupling and reconstruction from casual monocular videos. Our method decouples occluders and correlated shadows, recovers clean background representations, and enables high quality novel views synthesis. Our novel skewed entropy regularizer is critical to separate dynamic from static components of the scene, while our shadow-field allows for the removal of dynamic shadows without having explicitly model the interaction between light and geometry. We demonstrate superior results for multiple tasks on existing datasets, as well as on two datasets that we introduce alongside our technique.
Limitations Similar to many NeRF-based methods, our approach relies on accurate camera reg- istration achieve success decoupling and reconstruction. Our approach also suffers from high frequency view dependentt radiance change, such those caused by the presence of reflective surface within the scene. Due to the monocular moving camera setting, those effects could be misinterpreted as dynamic effects, resulting in incorrect decoupling. Removing texture-less target that repeatedly moves within a very small range also difficult, as the motion clues are extremely ambiguous in this case; see Figure TO We also note that the skewed binary entropy is not the only loss that suits the purpose of separating two NeRF components while suppressing one of them. Many losses such the beta distribution from (28 and Laplacian distribution from [45] can behave similarly by

---

Input
Mask
Dynamic
Static
Limitations Because the hand stays around the same position for the majority of the time during the video, our method is unable to fully decouple and remove the texture-less shadow.
Figure 10:
incorporating an exponential coefficient. The main difference between our skewed entropy loss and beta or Laplacian distributions is that the former has vanishing gradient when approaching O, while the gradients of the latter become infinity. Choosing appropriate form of loss could reduce the complexity of hyperparameters without affecting the decoupling accuracy.
Acknowledgements
This work supported by UKRI Future Leaders Fellowship [grant number G104084). We thank Ziwen Cui and Ningding Wang for their help with data collection.
References
[1] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning represen- tations and generative models for 3d point clouds, 2018.
[2] Benjamin Attal, Jia-Bin Huang, Michael Zollhoefer, Johannes Kopf, and Changil Kim. Learning neural light fields with ray-space embedding networks, 2021.
[3] Bagautdinov, C. Wu, J. Saragih, P. Fua, and Y. Sheikh. Modeling facial geometry using com- positional vaes. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2877-3886, 2018.
[4] Pia Bideau and Erik Learned-Miller, It's moving! probabilistic model for causal motion segmentation in moving camera videos, 2016.
[5] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li. Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Ries 3D Model Repository. Technical Report 12.Kiv1512.0000 [cs.GR], Stanford University Princeton University Toyota Technological Institute at Chicago, 2015.
[6] Christopher B. Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: unified approach for single and multi-view 3d object reconstruction, 2016.
[7] Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. Cvxnet: Learnable convex decomposition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[8] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free, 2021.
[9] Dawa Derksen and Dario Izzo. Shadow neural radiance fields for multi-view satellite pho- togrammetry, 2021.
[10] Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[11] Haoqiang Fan, Hao Su, and Leonidas Guibas. point set generation network for 3d object reconstruction from single image, 2016.
[12] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE International Conference on Computer Vision, 2021.
10

---

[13]
Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering 200fps, 2021.
[14] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape, 2019.
[15] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, Williamm T. Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions, 2019.
[16] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi. Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang. Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubrie: a scalable dataset generator: In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[17] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. papier-máché approach to learning 3d surface generation. In Proceedings of IEEE conference on computer vision and pattern recognition, pages 216-224, 2018.
[18] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf diet: Semantically consistent few-shot view synthesis, 2021.
[19] Zhang Jiakai, Liu Xinhang, Ye Xinyi, Zhao Fugiang, Zhang Yanshun, Wu Minye, Zhang Yingliang, Xu Lan, and Yu Jingyi. Editable free-viewpoint video using layered neural representation. In ACM SIGGRAPH, 2021.
[20] Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matthew Botvinick, Alexander Lerchner, and Christopher P. Burgess. Simone: View-invariant, cumporally-abstracter object representations via unsupervised video decomposition, 2021.
[21] Kacper Kania, Kwang Moo Yi, Marek Kowalski, Tomasz Trzcinski, and Andrea Tagliasacchi. Conerf: Controllable neural radiance fields. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2022.
[22] Mijeong Kim, Seonguk Seo, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. arXiv.org, 2021.
[23] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object-aware neural scene representation. In Conference on Computer Vision and Pattern Recognition (CVPR). 2022.
[24] Fuxin Li, Taeyoung Kim, Ahmad Humayun, David Tsai, and James M. Rehg. Video segmenta- tion by tracking many figure-ground segments. In ICCV, 2013.
[25] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, and Zhaoyang Lv. Neural 3d video synthesis from multi-view video, 2021.
[26] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings the Computer Vision and Pattern Recognition (CVPR), 2021.
the IEEE/CVF Conference on
[27] David B. Lindell, Julien N. P. Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering, 2020.
[28] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Trans. Graph., 38(4)))5:::1155:144 July 2019.
[29] Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. Mixture of volumetric primitives for efficient neural rendering, 2021.
[30] Erika Lu, Forrester Cole, Tali Dekel, Andrew Zisserman, William T Freeman, and Michael Rubinstein. Omnimatte: Associating objects and their effects in video. In CVPR, 2021.
[31] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Doso- vitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
11

---

[32] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis, 2020.
[33] Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of 3d meshes. In International Conference on Machine Learning, pages
7220-7220 12200-229..00 PMLR, 2020.
[34] Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H. Mueller, Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, and Markus Steinberger. DONeRF: Towards Real- Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks. Computer Graphics Forum, 40(4), 2021.
[35] Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs, 2021.
[36] Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 2856-2865, June 2021.
[37] Anestis Papazoglou and Vittorio Ferrari. Fast object segmentation in unconstrained video. In 2013 IEEE International Conference on Computer Vision, pages 1777-1784, 2013.
[38] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE Conference on Computer Vision and Pullern Recognition, pages 165-174, 2019.
[39] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. ICCV, 2021.
[40] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. Hypemerf: A higher-dimensional representation for topologically varying neural radiance fields. ACM Trans. Graph., 40(6), dec 2021.
[41] Songyou Peng, Chiyu Jiang, Yiyi Liao, Michael Niemeyer, Mare Pollefeys, and Andreas Geiger. Shape points: A differentiable poisson solver. Advances in Neural Information Processing Systems, 34, 2021.
[42] Perazzi, Pont-Tuset, McWilliams, L. Van Gool, M. Gross, and A. Sorkine Sorkine-Horunng, benchmark dataset and evaluation methodology for video object segmentation. In Computer Vision and Pattern Recognition, 2016.
[43] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. arXiv preprint WxXiv:2011.13900 2020.
[44] Daniel Rebain, Wei Jiang. Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[45] Daniel Rebain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, and Andrea Tagliasacchi. LoInerf: Learn from one look. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[46] Christian Reiser, Songyou Peng. Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps, 2021.
[47]
Konstantinos Rematas, Andrew Liu, Pratul Srinivasan, Jonathan T. Barron, Andrea Tagliasac- chi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields, 2021.
[48] Mehdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas Funkhouser, and Andrea Tagliasacchi. Scene representation transformer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[49] Johannes Lutz Schönberger and Jan-Michacl Frahm. Structure- mruuctue-from--ae from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[50] Vincent Sitzmano, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In Proc. NeurIPS, 2021.
12

---

[51] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-aves aware neural scene representations, 2020.
[52] Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Light field neural rendering, 2021.
[53] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction, 2021.
[54] Vadim Tschernezki, Diane Larlus, and Andrea Vedaldi. NeuralDiff: Segmenting 3D objects that move in egocentric videos. In Proceedings of the International Conference on 3D Vision (3DV), 2021.
[55] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. 2018.
[56] Yi Wang. Pierre-Mane Jodoin, Fatih Porikli, Janusz Konrad, Yannick Benezeth, and Prakash Ishwar. Cdnet 2014: An expanded change detection benchmark dataset. IEEE Computer Society Conference Computer Vision and Patlern Recognition Workshops, 06 2014.
[57] Z. Wang, E.P. Simoncelli, and A.C. Bovik. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems Computers, 2003, volume 2, pages 1398-1402 Vol.2, 2003.
[58] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwa- janakomn Nex: Real-time view synthesis with neural basis expansion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[59] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes, 2015.
[60] Wenqi Xian, Jia- Bin Huang, Johannes Kopf, and Changil Kim. Space-time time neural irradiance fields for free- viewpoint video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9421-9431, 2021.
[61]
Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Citynerf: Building nerf at city scale, 2021.
[62] Christopher Xie, Yu Xiang, Zaid Harchaoui, and Dieter Fox. Object discovery in videos as foreground motion clustering, 2018
[63] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and UIrich Neumann. Disn: Deep implicit surface network for hualtquality single-view 3d reconstruction. In Advances in Neural Information Processing Systems, pages 492-502. Curran Associates, Inc., 2019.
[64] Charig Yang. Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. In ICCV, 2021.
[65] Alex Yu, Sara Fridovich- Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks, 2021.
[66] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields, 2021.
[67] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one few images, 2020.
[68] Wentao Yuan, Zhaoyang Lv, Tanner Schmidt, and Steven Lovegrove. Star: Self-supervised tracking and reconstruction of rigid objects motion with neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13144-13152, 2021.
[69] Jason Y. Zhang. Gengshan Yang. Shubham Tulsiani, and Deva Ramanan. NeRS: Neural reflectance surfaces for sparse-view 3d reconstruction in the wild. In Conference on Neural Information Processing Systems, 2021.
[70] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. In CVPR, 2018.
[71] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (TOG), 40(6):1-18, 2021.
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] . You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section.
Did you include the license to the code and datasets? [No] The code and the data proprictary.
Did you include the license the code and datasets? [N/]]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as URLY? [Yes] See Supple- mentary
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] See Section 4.2
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No] Repeated evaluations not applicable due to the limitation in amount of computational resources required for training
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.2
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] See Section 4.2
(b) Did you mention the license of the assets? [Yes] See Section 4.2
(c) Did you include any new assets either the supplemental material or as URL? (es]]
See Supplementary [A]
(d) Did you discuss whether and how consent obtained from people whose data you' 're using/curating? [Yes] Consent obtained from license
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] Only personally identifiable information would be the "curls" dataset, which is obtained from Nerfies (39) and used under their license.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and sereenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "iMrpU4ZP436", "reviewer": "Reviewer_xgPY", "summary": "This paper presents D^2NeRF, a self-supervised method that takes a monocular video and learns a 3D scene representation that decouples moving objects, including their shadows, from the static background. In addition, this paper proposes a novel loss to promote the correct separation of phenomena for static and dynamic filed. The authors further propose a shadow field network to detect and decouple dynamically moving shadows. A new dataset was proposed, containing various dynamic objects and shadows. Extensive experiments demonstrate that the proposed method can achieve better performance than state-of-the-art approaches in decoupling dynamic and static 3D objects, occlusion and shadow removal, and image segmentation for moving objects.", "questions": "please refer to the weakness. Add more comparison results with exiting methods.\n* do not compare with other motion decoupling methods, such as STNeRF[18], NSFF[24] and DynNeRF[11], SIMONe[19], STaR[63], although their experiment setting may be inconsistent with the proposed methods. However, I think in the synthetic dataset, all of them can be reproduced, please add some of their results for a complete comparison.\n* It seems to lack generalizability, just train and test in the same dataset.", "limitations": "The authors have discussed the limitation, and all of them are a considerable challenge. I have no idea to solve them, and I think it's a trade-off.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n* New dataset for static and dynamic field decomposition.\n* Self-supervised method that takes a monocular video and learns a 3D scene representation that decouples moving objects, including their shadows, from the static background. \n* Novel loss to promote the correct separation of phenomena for static and dynamic filed. \n* A shadow field network to detect and decouple dynamically moving shadows\n* SoTA results in several tasks such as decoupling dynamic and static 3D objects, occlusion and shadow removal, and image segmentation for moving objects.\n\nWeakness:\n* require the accurate camera, suffer from high-frequency view-dependent radiance change, which has been discussed.\n* do not compare with other motion decoupling methods, such as STNeRF[18], NSFF[24] and DynNeRF[11], SIMONe[19], STaR[63], although their experiment setting may be inconsistent with the proposed methods. However, I think in the synthetic dataset, all of them can be reproduced, please add some of their results for a complete comparison.\n* When I refer to the supplementary video, there are still some wrong static and dynamic field decompositions, such as the keyboard. And I think this is a tradeoff. \n* It seems to lack generalizability, just train and test in the same dataset.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "ZHmwrHDIg9p", "reviewer": "Reviewer_GJS4", "summary": "This paper proposes a new self-supervised approach for segmenting moving objects from the static background. It tackles with issues encountered during training with several techniques, such as skewed entropy, ray regularization, static regularization, and integrating shadow ratio to separate shadows.", "questions": "1. I am curious about the robustness of the proposed approach to the hyparameters. See the weakness part.", "limitations": "I appreciate the authors's discussions about the limitations of the proposed approach in Sec. 5, which helps the understanding of the suitable scenarios.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. Originality: the proposed techniques are insightful and are beneficial to the community, such as skewed entropy and ray regularization.\n2. Quality: The presented qualitative results are of high quality.\n3. Clarity: the manuscript is well-written.\n4. Significance: the task of decoupling dynamic objects and static background is important and the proposed approach tackles the long-standing problem with high-quality results.\n\n## Weakness\n\n1. I think the main weakness comes from the scene-level tuning for the hyparameter of k, $\\lambda_s$, $\\lambda_r$, $\\lambda_{\\sigma^S}$, $\\lambda_\\rho$.\n2. I understand that to push the number, the author may need grid-search for each scene. However, the lack of analysis for the sensitivity of those hyparameters makes it unclear how robust the proposed approach is.\n3. Especially considering in Sec. B of supp, we have 9 sets of parameters for 19 scenes. Even for the same scene (Banana), we have two sets of hyparameters for two tasks (decoupling and novel view).\n4. Besides the per-scene tuning results, I recommend the author to report a set of quantitative results with a single set of hyparameters if possible.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4E2lwN9l0Yp", "reviewer": "Reviewer_Xgiz", "summary": "This paper proposes a method to segment and decouple dynamic objects while recovering the static environment from a monocular video. It adapts from NeRF and its extension Hyper NeRF but with improved handling of shadow regions as well as a loss to promote correct separation of dynamic and static regions. It demonstrates plausible motion segmentation and shadow removal result compared to recent NeRF based methods.\n\n", "questions": "see the questions in \"strength and weakness\"", "limitations": "The method cannot handle high frequency view-dependent radiance change due to the monocular moving camera setting.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is a solid development for using NeRF to reconstruct from monocular videos with dynamic foregrounds. The paper in general is well-written and the claims are well supported. The adaptation it made to handle shadow and foreground & background separation is elegant and seems to be effective. It definitely holds value to people working on similar problems and deserves publication.\n\nThere are a couple of things I hope the authors could comment on.\n\n(1) In Ln. 226, “To demonstrate the ability of fully self-supervised scene decoupling, we do not apply any masks when registering real-world images using COLMAP”. In my experience, without feeding masks, COLMAP tends to make wrong estimation of camera poses when the foreground is sufficiently large, which will definitely results in wrong reconstruction of the static background. It seems the proposed do not attempt to update the camera pose during optimization, so the claim above looks confusing to me. \n\n(2) It would be nice if the author also visualizes the depth map of the reconstructed foreground / background.\n\n(3) The results of hyperNeRF looks far worse compared to the proposed method even in the dynamic region. Given that the proposed method seems to have little difference to hyperNeRF at least in the dynamic regions, the current result surprises me a bit. Could the author make additional comments on the possible reasons?\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "-Hp1TOVqbsb", "reviewer": "Reviewer_PH91", "summary": "This paper introduces a series of techniques to decouple dynamic objects from monocular videos, which present impressive static background recovery with shadow removal.  How to remove the disturbance from dynamic objects/occluders is a long-standing problem in 3D vision domain. I appreciate the simple but valuable design, i.e., the skewed entropy loss with decoupled NeRF and an extra shadow field. However, many previous methods have adopted similar techniques for foregroud/background content separation. Some related literatures are not included in the paper. Besides, it reads like a CVPR-tyle paper. I'm not sure whether it's suitable for NIPS.", "questions": "1.  The skewness hyper-parameter $k$ is critical in practice because it determines the proportion of the `dynamic'. And it's sensitive according to the provided experiments. Is it possible to automatically tune the parameter?\n2. The motivation and mechanism of the skew entropy loss is similar to the beta distribution proposed by [1]. As it is regarded one of the major contribution, a thorough comparison and discussion is deserved.\n3. The topic is NVS in dynamic environments. Some related literatures are absent[1][2].\n4. The shadow NeRF concept has been introduced in [3], which should be included in related literature.\n\n[1] Neural Volumes: Learning Dynamic Renderable Volumes from Images \n[2] Neural Scene Graphs for Dynamic Scenes\n[3] Shadow Neural Radiance Fields for Multi-view Satellite Photogrammetry", "limitations": "\n\n\n\n\n\n\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts:\n1. The skewed-entropy loss is able to flexibly separate the static and dynamic parts.\n2. Separating the time-varying shadows and the static appearace via a shadow field.\n2. The decouple performance and NVS quality significantly outperforms SOTA.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept