Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a method to segment and decouple dynamic objects while recovering the static environment from a monocular video. It adapts from NeRF and its extension Hyper NeRF but with improved handling of shadow regions as well as a loss to promote correct separation of dynamic and static regions. It demonstrates plausible motion segmentation and shadow removal result compared to recent NeRF based methods.


strengths_and_weaknesses: This paper is a solid development for using NeRF to reconstruct from monocular videos with dynamic foregrounds. The paper in general is well-written and the claims are well supported. The adaptation it made to handle shadow and foreground & background separation is elegant and seems to be effective. It definitely holds value to people working on similar problems and deserves publication.

There are a couple of things I hope the authors could comment on.

(1) In Ln. 226, “To demonstrate the ability of fully self-supervised scene decoupling, we do not apply any masks when registering real-world images using COLMAP”. In my experience, without feeding masks, COLMAP tends to make wrong estimation of camera poses when the foreground is sufficiently large, which will definitely results in wrong reconstruction of the static background. It seems the proposed do not attempt to update the camera pose during optimization, so the claim above looks confusing to me. 

(2) It would be nice if the author also visualizes the depth map of the reconstructed foreground / background.

(3) The results of hyperNeRF looks far worse compared to the proposed method even in the dynamic region. Given that the proposed method seems to have little difference to hyperNeRF at least in the dynamic regions, the current result surprises me a bit. Could the author make additional comments on the possible reasons?



questions: see the questions in "strength and weakness"
limitations: The method cannot handle high frequency view-dependent radiance change due to the monocular moving camera setting.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents D^2NeRF, a self-supervised method that takes a monocular video and learns a 3D scene representation that decouples moving objects, including their shadows, from the static background. In addition, this paper proposes a novel loss to promote the correct separation of phenomena for static and dynamic filed. The authors further propose a shadow field network to detect and decouple dynamically moving shadows. A new dataset was proposed, containing various dynamic objects and shadows. Extensive experiments demonstrate that the proposed method can achieve better performance than state-of-the-art approaches in decoupling dynamic and static 3D objects, occlusion and shadow removal, and image segmentation for moving objects.
strengths_and_weaknesses: Strength:

* New dataset for static and dynamic field decomposition.
* Self-supervised method that takes a monocular video and learns a 3D scene representation that decouples moving objects, including their shadows, from the static background. 
* Novel loss to promote the correct separation of phenomena for static and dynamic filed. 
* A shadow field network to detect and decouple dynamically moving shadows
* SoTA results in several tasks such as decoupling dynamic and static 3D objects, occlusion and shadow removal, and image segmentation for moving objects.

Weakness:
* require the accurate camera, suffer from high-frequency view-dependent radiance change, which has been discussed.
* do not compare with other motion decoupling methods, such as STNeRF[18], NSFF[24] and DynNeRF[11], SIMONe[19], STaR[63], although their experiment setting may be inconsistent with the proposed methods. However, I think in the synthetic dataset, all of them can be reproduced, please add some of their results for a complete comparison.
* When I refer to the supplementary video, there are still some wrong static and dynamic field decompositions, such as the keyboard. And I think this is a tradeoff. 
* It seems to lack generalizability, just train and test in the same dataset.
questions: please refer to the weakness. Add more comparison results with exiting methods.
* do not compare with other motion decoupling methods, such as STNeRF[18], NSFF[24] and DynNeRF[11], SIMONe[19], STaR[63], although their experiment setting may be inconsistent with the proposed methods. However, I think in the synthetic dataset, all of them can be reproduced, please add some of their results for a complete comparison.
* It seems to lack generalizability, just train and test in the same dataset.
limitations: The authors have discussed the limitation, and all of them are a considerable challenge. I have no idea to solve them, and I think it's a trade-off.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new self-supervised approach for segmenting moving objects from the static background. It tackles with issues encountered during training with several techniques, such as skewed entropy, ray regularization, static regularization, and integrating shadow ratio to separate shadows.
strengths_and_weaknesses: ## Strengths

1. Originality: the proposed techniques are insightful and are beneficial to the community, such as skewed entropy and ray regularization.
2. Quality: The presented qualitative results are of high quality.
3. Clarity: the manuscript is well-written.
4. Significance: the task of decoupling dynamic objects and static background is important and the proposed approach tackles the long-standing problem with high-quality results.

## Weakness

1. I think the main weakness comes from the scene-level tuning for the hyparameter of k, $\lambda_s$, $\lambda_r$, $\lambda_{\sigma^S}$, $\lambda_\rho$.
2. I understand that to push the number, the author may need grid-search for each scene. However, the lack of analysis for the sensitivity of those hyparameters makes it unclear how robust the proposed approach is.
3. Especially considering in Sec. B of supp, we have 9 sets of parameters for 19 scenes. Even for the same scene (Banana), we have two sets of hyparameters for two tasks (decoupling and novel view).
4. Besides the per-scene tuning results, I recommend the author to report a set of quantitative results with a single set of hyparameters if possible.
questions: 1. I am curious about the robustness of the proposed approach to the hyparameters. See the weakness part.
limitations: I appreciate the authors's discussions about the limitations of the proposed approach in Sec. 5, which helps the understanding of the suitable scenarios.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a series of techniques to decouple dynamic objects from monocular videos, which present impressive static background recovery with shadow removal.  How to remove the disturbance from dynamic objects/occluders is a long-standing problem in 3D vision domain. I appreciate the simple but valuable design, i.e., the skewed entropy loss with decoupled NeRF and an extra shadow field. However, many previous methods have adopted similar techniques for foregroud/background content separation. Some related literatures are not included in the paper. Besides, it reads like a CVPR-tyle paper. I'm not sure whether it's suitable for NIPS.
strengths_and_weaknesses: Strenghts:
1. The skewed-entropy loss is able to flexibly separate the static and dynamic parts.
2. Separating the time-varying shadows and the static appearace via a shadow field.
2. The decouple performance and NVS quality significantly outperforms SOTA.

questions: 1.  The skewness hyper-parameter $k$ is critical in practice because it determines the proportion of the `dynamic'. And it's sensitive according to the provided experiments. Is it possible to automatically tune the parameter?
2. The motivation and mechanism of the skew entropy loss is similar to the beta distribution proposed by [1]. As it is regarded one of the major contribution, a thorough comparison and discussion is deserved.
3. The topic is NVS in dynamic environments. Some related literatures are absent[1][2].
4. The shadow NeRF concept has been introduced in [3], which should be included in related literature.

[1] Neural Volumes: Learning Dynamic Renderable Volumes from Images 
[2] Neural Scene Graphs for Dynamic Scenes
[3] Shadow Neural Radiance Fields for Multi-view Satellite Photogrammetry
limitations:
review_title: null

Meta-review: metareview: This paper attacks an interesting problem with NERF, decoupling moving objects, including their shadows, from the static background.  All four reviewers recommend accepting the paper, and the weaknesses identified did not detract from substantive contributions. Therefore I am accepting this paper.
confidence: Certain
Decision: Accept