Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: An approach is presented in this paper for mapping 2D image observations to persistent 3D scene representations in order to create a novel view synthesis and disentangle the movable and immovable components of the scene. A ground-aligned 2D feature grid based on conditional neural groundplans can be employed as a memory-efficient representation of scene data. This notion is inspired by the bird's-eye-view (BEV) representation commonly used in vision and robotics. Self-supervised learning is conducted using differentiable rendering from unlabeled multi-view observations, and the method learns to complete the geometry and appearance of occluded regions.

strength_and_weaknesses: Strength：
1. The conditional neural groundplans proposed are intriguing. This allows efficient processing of scene appearance and geometry directly in 3D, through the self-supervised learning of conditional neural groundplans, a hybrid discrete continuous representation of 3D neural scenes.
2. The proposed disentanglement of static background and movable foreground objects is novel. This approach uses object motion for training an encoder-based system that can reconstruct scenes from a single image instead of decomposing video frames based on scene specifics.
3. The proposed method is promising for understanding 3D from a single image. The dynamic groundplan encodes 3D geometry, allowing us to segment and animate objects in 3D using single images and 3D bounding boxes.

Weaknesses：
1. The biggest weakness is that only results on simple synthetic data are presented. Currently, there are a number of well-annotated 3D driving scenes available. An example is KITTI 360 (https://www.cvlibs.net/datasets/kitti-360/). Could the authors explain why results on the real datasets are not presented? Is it because of some specific annotation (such as BEV correspondence or the number of views) needed by the method? Otherwise, the generalizability on natural scenes and the practical value of the method are questioned.
2. I think there are some ambiguity behind the proposed Static-Dynamic Disentanglement. A definition of Static and Dynamic should be made clear in the proposed framework. A car remaining still in the sequence will be considered as Static. Considering a situation when two moving and static cars looks similar or exactly the same, how can we distinguish from a single image if a car is static or dynamic? The method should mention the above cases as an assumption, or making a clear definition of being Static and Dynamic.
3. Groundplans implicitly assume that objects are aligned on a plane. However, in real life, it is common that scenes/objects are not placed on a flat ground plane (e.g., a curved plane). The ability of persistent 3D scene representation is a little overclaimed, if we consider the implicit assumption behind groundplans.

clarity,_quality,_novelty_and_reproducibility: The paper is well-organized and clearly written. Ideas are novel. Results in the paper are easily reproducible.

summary_of_the_review: The method is good and the idea is interesting. But only synthetic data is used for evaluation, and the access of real data should not be a problem.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper describes a 3D scene representation built from 2D observations. 

The representation allows the decomposition of static and movable objects in the scene, and can support tasks including novel view synthesis, 3D instance segmentation, and object-level scene editing. The authors propose a self-supervised training method for learning such a representation. 

Evaluations are conducted on two synthetic datasets. 
strength_and_weaknesses: Strengths include: 
* The self-supervised training removes the need for GT annotation, e.g., object bounding boxes, for the supported downstream tasks. 
* The decomposition of static and movable object feature spaces looks like quite an elegant solution to utilize the inductive bias in multi-view videos while enabling capabilities such as segmentation and object-level scene editing. 
* The results overall look promising. Single-view/few-view novel view synthesis is a very challenging task, and the comparisons with PixelNeRF and uORF clearly demonstrated the advantages in synthesis quality. The analysis of adding more views (Fig.10) is strong evidence that the learned 3D representation can figure out how 2D views are related to, and hence help improve, 3D reconstruction. 

Weaknesses include:
* There is no study of the generalization capability of the model. It seems likely that the learned models can overfit the limited appearances in the dataset (a few types of shapes/objects with mostly uniform colors). 
* Related to the last point, the evaluation is completely done on synthesized datasets, and it's not clear how well the model can learn real-world data with much more diverse appearances. Since the method does not require GT annotations, some evaluation on real-world data does not seem too difficult. 
* The are some quality issues with the presented view synthesis results. For example, while unobserved regions are understandably blurry, why are observed regions also not sharp? We can see from Fig.10 that adding more views does not help the quality of observed regions any further. 
* Training requires multi-view videos. I can't find clarification on whether these need to be from static cameras. Nothing in the model appears to have such a requirement, but if so, that's a strong limitation. Moreover, why can't the model be trained on frames from a single moving camera? 
* It's not clearly described how scene editing is done. It says that's enabled by "directly editing the neural groundplan". Does that mean editing the (movable object) feature space? The feature space is not sparse (i.e. locations without objects have non-zero features), so how does cropping in such space work, and why can that make sense? 
clarity,_quality,_novelty_and_reproducibility: Quality is overall high, with thorough evaluations and a convincing and coherent story. Clarity-wise, I find the paper easy to follow and pleasant to read. There are a few of details that I think the authors should provide more clarification (last two points in weaknesses). 

I think all major components of the work have been seen in some previous work. However, they are nicely put together to achieve a 3D presentation with the combination of some unique strengths (single-image 3D scene inference, movable object editing, self-supervised training). 

There are a decent amount of details in the paper, and the authors promise code and data release, so I think reproducibility is not a concern. 
summary_of_the_review: The major highlights of the paper are the use of multi-view videos as self-supervision and, consequently, the ability to decompose static/movable objects to support segmentation and editing. The results are promising, although they appear to be somewhat preliminary. 

My primary concerns are around generalization and the lack of evidence on how the approach performs on real, non-synthetic images. There are also some details requiring clarifications that should be easy fixes.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a bird’s eye-view (BEV) representation for a 3D scene. The representation is disentangled into the static part and the dynamic part by a 2D CNN. The authors train their representation network on multiple dynamic videos with multiple views. When given a new image, the network can generate the implicit 3D representation and decompose the scene automatically. Differential rendering is used to generate novel view images. The authors conducted experiments on two datasets and showed impressive results.
strength_and_weaknesses: ====> Strength
1. The authors introduced the bird’s eye-view (BEV) representation as a scene representation, which is novel in the area of neural rendering.
2. The disentanglement of the static and moving component of the scene adds more contribution to the proposed method. Although it is not new to automatically separate these components in the neural rendering community, using a vanilla 2D CNN to separate the feature representation is new to the reviewer.
3. The authors show that using a simple heuristic method, they can generate instance segmentation and bounding boxes for the dynamic components in the scene.
4. Good qualitative results are given.

====> Weakness
1. The details of the neural groundplans is confusing: 

    i. In page 4, there are multiple $x$ such as $x'$, $x$, $x_c$, $x_i$. Their shape should be clarified. Whether they belong to 2D or 3D point, and also their coordinate system (i.e., the image frame coordinate or the world coordinate, and how to transform between these coordinates using the camera intrinsic)

    ii. For equation like $v(x ′ ) = F(π(C^{-1}(x ′ )))$, $f(x ′ ) = ...$, the parentheses here denote indexing the feature volume $v$ and feature tensor $F$. However, it is easy to confuse them with the functions like $\pi()$ and $D()$. The authors should consider using other formulation.

    iii.The $\pi(\cdot)$ is not clear. What is the meaning of it and how to learn $\pi(\cdot)$, is it learned per image I?

    iv. ``$g_x$ are the groundplan features for the point $x$ computed by projecting the coordinates onto the groundplane...'' How is the projection conducted? Is it projected by the camera extrinsic？

    v. In page 5, the authors say ``The per-frame static groundplans are pooled to obtain a single, time-independent static groundplan.'' How is the pooling process performed? The authors need to detail it.

2. The task setting is not clear. The authors should consider formulate the problem setting in a separated subsection (e.g., train one model for multiple scenes, all images are calibrated, the view density per scene, etc.)

3. It seems the compared baselines like pixelNerf and uORF are not proposed for the topic of separating dynamic and static representation. It would be fairer to compare with ``Neuraldiff: Segmenting 3d objects that move in egocentric videos.'' that used different components for the moving and static objects.
clarity,_quality,_novelty_and_reproducibility: Given the results, the reviewer feel that this work is in good quality. And based on my expertise, it presents good originality. 

However, as stated in the weakness, the clarity of the paper needs further improvement. And although the authors give some details of the network in the appendix, the reviewer is not optimistic that it can be reproduced easily without code. 
summary_of_the_review: Overall, the paper proposed a new representation for the neural rendering, and is novel to me. However, the clarity of the paper needs further improvement. The reviewer looks forward to the author feedback and will consider increase my rating if the discussion can address my questions.

=====
The authors addressed most of my concerns during the discussion and I would increase my rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a method that maps a single 2D image of a scene to a persistent 3D scene representation, enabling novel view synthesis and disentangled representation of the movable and immovable components of the scene. The enabler is a new representation called conditional neural groudplans, motivated by the Birds’-eye-view representation. Because of the ability to separate the movable and immovable components of the scene, the representation enables a variety of downstream tasks, such as extraction of object-centric 3D representations, novel view synthesis, instance-level segmentation, 3D bounding box prediction, and scene editing. The authors conduct experiments on CLEVR and CoSY. The proposed method achieves state-of-the-art novel view synthesis compared with PixelNeRF and uORF on CLEVR. In addition, the proposed method obtain favorable instance segmentation performance compared with uORF on the CLEVR dataset.
strength_and_weaknesses: Strength:

1. The authors tackle a challenging yet valuable problem, disentanglement of the movable and immovable components of the scene from a single image. They propose a neural scene representation called conditional neural groundplans and a self-supervised framework to disentangle static background and movable foreground objects given only a single image. The framework enables a variety of downstream tasks, such as extraction of object-centric 3D representations, novel view synthesis, instance-level segmentation, 3D bounding box prediction, and scene editing. The direction would interest a wide range of communities, including 3D scene understanding, 3D neural representation, disentangled representation learning, and novel view synthesis.
2. The proposed 3D persistent scene representation learning framework elegantly leverages ideas from BEV representation, neural representation learning, volume rendering, and compositing operation (Yuan et al., 2021). Moreover, the proposed method utilizes multi-view videos (in a self-supervised manner) at training time to learn to separately reconstruct static and movable components of the scene from a single image at test time. The authors present a self-supervised neural scene representation framework for general scene understanding tasks. The reviewer found the work valuable to the community.
3. The proposed 3D scene representations enable novel view synthesis and disentangle movable and immovable components of the scene via diverse qualitative evaluation on two datasets, CLEVR and CoSY. In particular, the qualitative results shown on CoSY are impressive, as shown in the manuscript and supplementary website.

Weaknesses:

1. Persistent scene representation: By sufficient qualitative analysis, the proposed 3D scene representations enable novel view synthesis and disentangle movable and immovable elements for different scene understanding tasks. One failure case shown on the top row of p.17 is that the framework cannot effectively synthesize the viewing perspective behind the box with a green door. The framework seems unable to capture the background correctly, yielding incorrect synthesis. Moreover, the box's location seems incorrect from the images on the left. The reviewer wonder if the performance can be improved with additional observations.
2. The authors propose to have two separate groundplans neural representations for static and dynamic, respectively. As we know from the literature (e.g., Yuan et al., 2021), it is possible to keep separate 3D neural representations for novel view synthesis. It seems reasonable to tackle the goal of this paper, i.e., achieve decent performance on a variety of tasks from a single image, with separate 3D neural representations based on the proposed framework. It would be great for the authors to elaborate on the technical challenge of this direction. That could motivate the community to tackle the direction collectively.
3. Differentiable rendering based on neural groundplans feature: the reviewer found it less intuitive to understand why the differentiable rendering would work using the groundplan features g_x. Specifically, g_x is obtained based on the weighted sum of the features along the y-axis. The groundplan feature g_x contains information in a particular pillar. Why is it a better feature for volume rendering? Empirically, the design choice renders better quality than STOA methods (i.e., PixelNeRF and uORF). Could the authors please elaborate more on this design choice?
4. The design choice of BEV representation: The authors utilize a particular operation to form BEV representations, i.e., the weighted sum of the features. However, there are other approaches, e.g., Roddick and Cipolla, 2020, Philion and Fidler, 2020 and Saha et al., 2022. It would be valuable for the audience to learn the difference between these operations for representation learning.
5. The reviewer is interested in the qualitative performance of novel view synthesis without using the contracted coordinate. With the analysis, readers would be able to visualize the difference.
clarity,_quality,_novelty_and_reproducibility: 1. The overall clarity and quality of the manuscript are great and easy to understand.
2. The authors propose a new persistent 3D scene representation that can enable novel view synthesis and disentangle movable and immovable elements of a scene from a single image. The reviewer found the representation learning framework a contribution, and the work would stimulate the community to explore the direction collectively.
3. The work utilizes multiple ideas from BEV representation, neural representation learning, volume rendering, and compositing operation (Yuan et al., 2021). It would be valuable to the community to open-source the code. Note that the manuscript o provide many details about the proposed learning framework. 
summary_of_the_review: The work presents a novel 3D persistent scene representation framework and demonstrates impressive novel view synthesis performance and scene understanding tasks such as instance segmentation and localization. The reviewer found the work novel and has the potential to stimulate the community to study the direction collectively. However, there are several concerns mentioned in the Weaknesses section. The reviewer would like to get feedbacks from the authors.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose neural ground plans 3D representations to represent 3D extracted from a single image. The method obtains good qualitative results showing how to render new views for scene understanding. The reviewers had initially several concerns like:
- Results only on synthetic datasets
- Difficulties to determine if a vehicle is moving or static
- Some baseline comparisons

justification_for_why_not_higher_score: The article had negative scores. The reviewers pointed out several weaknesses and the authors addressed many of them. However, there are some points that require further research. Overall the contribution is good.
justification_for_why_not_lower_score: The article is worth publishing since it has important contributions even there is need of more research.
Decision: Accept: poster