Single-Shot Scene Reconstruction
Authors: Sergey Zakharov
Rares Andrei Ambrus
Vitor Campagnolo Guizilini
Dennis Park
Wadim Kehl
Fredo Durand
Joshua B. Tenenbaum
Vincent Sitzmann
Jiajun Wu
Adrien Gaidon
Venue: CoRL2021 Poster (2021)
Keywords: scene reconstruction
differentiable rendering

Abstract: We introduce a novel scene reconstruction method to infer a fully editable and re-renderable model of a 3D road scene from a single image. We represent movable objects separately from the immovable background, and recover a full 3D model of each distinct object as well as their spatial relations in the scene. We leverage transformer-based detectors and neural implicit 3D representations and we build a Scene Decomposition Network (SDN) that reconstructs the scene in 3D. Furthermore, we show that this reconstruction can be used in an analysis-by-synthesis setting via differentiable rendering. Trained only on simulated road scenes, our method generalizes well to real data in the same class without any adaptation thanks to its strong inductive priors. Experiments on two synthetic-real dataset pairs (PD-DDAD and VKITTI-KITTI) show that our method can robustly recover scene geometry and appearance, as well as reconstruct and re-render the scene from novel viewpoints.

Single-Shot Scene Reconstruction
Sergey Zakharov', Rareş Ambrus', Vitor Guizilini', Dennis Park', Wadim Kehl', Fredo Durand Joshua B. Tenenbaum Vincent Sitzmann", Jiajun Wu', Adrien Gaidon'
'Toyota Research Institute, Wooen Planet, 3 Massachusetts Institute Technology, Sttanford University
Abstract: We introduce novel scene reconstruction method to infer a fully ed- itable and re-renderable model of a 3D road scene from a single image. We rep- resent movable objects separately from the immovable background. and recover a full 3D model of each distinct object as well as their spatial relations in the scene. detectors and neural implicit 3D representations
We leverage transformer-bait and we build Scene Decomposition Network (SDN) that reconstructs the scene in 3D. Furthermore, we show that this reconstruction can be used in an analysis- by-synthesis setting via differentiable rendering. Trained only on simulated road scenes, our method generalizes well to real data the same class without any adaptation thanks to its strong inductive priors. Experiments dataset pairs (PD-DDAD and VKITTI-KITTI) show that our method can robustly recover scene geometry and appearance, as well as reconstruct and re-render the scene from novel viewpoints.
two synthetic-real
Introduction
Decomposing images into disjoint symbolic representations is the ultimate goal of robotics and computer vision, as it allows semantic reasoning over all scene parts. Such a decomposition bene- fits different possible applications (e.g., robotics, augmented reality, autonomous driving) where decomposed scene can be reassembled in different ways, enabling interaction or reenactment. Un- fortunately, single image scene reconstruction is inherently ill-posed due to variety of reasons. Most prominently, ambiguity in projective geometry can only be resolved with prior knowledge about the observed scene. Current saute-of-the-as methods either do not provide full decomposi- tion of the scene and regress per- ver-pixel depth [1, 2], or estimate the poses of already known objects and ignore the rest of the geometry [3, 4].
In this work, we design a pipeline to obtain an interpretable, disentangled scene representation for 3D-aware manipulation of driving scenes. Our system is capable of reconstructing full geometries and appearances of detected object instances from a single RGB image. Such a decomposition is made possible by three components: (l) scene decomposition network (SDN) that detects object instances, recovers their partial geometries and poses, and predicts the full geometry of the back- ground and RGB color behind the occluders; (2) differentiable database of colored object priors that encodes their full geometry and appearance in the form of signed distance fields (SDF) and luminance fields (LF); and (3) 2D/3D optimization pipeline that uses surfel-based differentiable rendering to fit initial observations to the object database, leveraging their shape and appearance.
Our contributions are summarized follows:
A holistic single-shot scene reconstruction system that splits the scene into background and object instances and recovers their full geometry and appearance;
An optimization pipeline based on NOCS-SDF and RGB-LF fitting as well as surfel-based dif- ferentiable rendering to refine geometries, appearances, and poses of detected instances;
A scene manipulation and generation pipeline allowing to place and render objects of various geometry and appearance, and change view points.
Conference on Robot Learning (CoRL 2021), London, UK.

---

000
Lat
CNN
E
D
MLPs
Lot
Lat
RGB
Prior D8
Recovered 3D scene
Dopth
100-1891
Deph (Bokground)
20:30 30 Optimization
NOCS
NOCS
Pose Estimation
(0)
Figure 1: Scene Decomposition Network (SDN): Given RGB image, SDN recovers partial canonicalized object shapes in the form of NOCS maps and predicts background depth. NOCS maps then used to directly estimate object poses using PnP solver. We further fit learned object priors to align with predicted partial shapes in terms of geometry, appearance, and poses.
2 Related Work
Single-Shot Scene Reconstruction Holistic single image scene reconstruction is a highly ill- posed problem, tackled from different angles by variety of works. The most common approach is monocular depth estimation that predicts per-pixel depth values [5, 1, 2]. While achieving re- markable results, these methods reconstruct the scene as whole and not incorporate knowledge about the scene's objects and layout. Alternatively, 3D detection pipelines detect separate objects and recover their masks and 3D bounding boxes [3, 6, 7, 4]. or incorporate relationships between objects by using graph or physical simulation [8, 9, 10]. While having understanding about the object appearance in 2D and their 3D pose, these methods do not recover the shapes of the objects nor the rest of the scene. There are also pipelines aiming at reconstructing objects from detections by aligning 3D CAD models to fit the predicted masks and input images [11, 12, 13], albeit without generalization to unknown instances outside the model database. Lastly, there recent works that recover room layouts together with full object geometries [14, 15], but restrict themselves to the geometric components of the recovered instances. In our work we recover the geometry and appear- ance for both background and detected objects by using a differentiable database of object priors allowing smoothly interpolate between plausible object shapes.
6DoF Object Detection via Correspondence Regression Object detection and pose estimation from RGB images is well-established but ongoing research problem due among others, the in- herent ambiguity of perspective projection. The myriad of existing solutions can loosely be grouped in three categories: direct pose regression (using neural net output the pose directly by learning from data) [16, 17, 18, 19), template-matching (comparing patches with predefined set of tem- plates either in pixel space feature space) [20, 21, 22, 23, 24, 25], and correspondence-baat methods (directly utilizing correspondences between image coordinates and 3D models of inter- est) [26, 27, 3, 4] each having their own benefits and drawbacks. The current state-of-the-an methods in object pose estimation almost exclusively belong to the latter group with such repre- sentatives as PVNet [7], CDPN [28], EPOS [27], Pix2Pose [4], GDR-Net [29] and DPOD [3, 6]. They all regress per- per-pixel correspondences and use a PnP solver to estimate a 6DoF pose given set of 2D-3D correspondences and camera parameters, minimizing reprojection errors. While some works regress manually defined UV correspondences, others regress Normalized Object Coordinate Space (NOCS) maps [30] that represent visible zyz object coordinates in RGB space. The benefits of NOCS are their straightforward creation through rendering. and their simultaneous definition of both the object's pose and shape. In our work we show how to effectively leverage this representation for geometry retrieval as well.

---

Neural Implicit Representations Recently, new prominent direction for 3D shape and appear- ance representation has emerged with DeepSDF [31]. Occupancy networks [32], and IM-Net [33], using neural networks for scalar function approximation. All three methods take 3D coordinates as
input and output either binary occupancy estimate or continuous SDF value, encoding the object's surface. Subsequently, follow-up work such as Deep Local Shapes (DLS) [34], Neural Geometric Level of Detail (NGLOD) [35] or NASA [36] improved the direction further. While DLS stores SDFs voxel grid cells to allow larger scenes, NGLOD uses an efficient Octgrid representation and a feature composition scheme based on trilinear interpolation to further improve the reconstruction quality, whereas NASA extends the approach to articulated objects.
Similar representations have been extended represent object appearance. Scene Representation Networks (SRN) [37] add a ray marching routine and also regress RGB colors at surface intersec- tions to learn from multi-view images, and Differentiable Volumetric Rendering [38] which couples an implicit shape representation with differentiable rendering. Alternatively, instead of regress- ing SDF or RGB values for predefined 3D coordinates, NeRF [39] proposes to regress density and color values along rays (5D coordinates) and compute the true color via numerical integra- tion. This approach, drawing inspiration from Neural Volumes [40] and their volumetric represen- tation, boosted interest in implicit volumetric rendering and resulted in multitude of works tack- ling problems from training and rendering time performance [41, 42, 43, 44], to covering dynamic scenes [45, 46, 47, 48], scene relighting [49, 50, 51], and composition [52. 53, 54].
While providing realistic renderings, most NeRF-based methods overfit exception of PixelNeRF [55]) and require a set of very dense view point annotations. Our pipeline, on the other hand, is capable of reconstructing scenes never seen during training, and is also not limited to viewpoints seen during training.
single scene (with the
3 Methodology
Given single RGB image of typical driving scene, our pipeline holistically reconstructs the scene by not only explaining every visible pixel, but also predicting the full geometry and appearance of detected objects and the occluded areas that not visible in the image. This is made possible by combination of three components: a scene decomposition network (SDN), a differentiable database of object priors, and 2D/3D optimization pipeline.
3.1 Scene Decomposition Network
The first component of our pipeline (Fig. 1) is the scene decomposition network, SDN. Its main role to estimate the layout of the scene, detect and identify visible objects, and recover the geom- etry of both the background and the detected objects. For clarity, it is divided into three separate blocks: a detection transformer-base based decomposition block, an object reasoning block, and a 3D reasoning block.
Detection Transformer Block (Fig. la) The core of our SDN is transformer- decomposi- tion. An encoder-decoder object detector based on transformers, a popular architecture for sequence prediction and has been first applied to computer vision in DETR [56]. DETR first generates dis- tinctive image features from an input RGB image, and feeds them to transformer module to build attention maps between features. This self-attention mechanism explicitly models all pairwise inter- actions between elements in a sequence, making this architecture particularly suitable for specific constraints of set prediction (e.g. duplicate suppression). Using the notion of object queries, we are able to retrieve output features for each the detected objects, which are then used as input to our object reasoning block and 3D reasoning block.
Object Reasoning Block (Fig. 1b) Our object reasoning block takes output features of the detec- tion transformer and uses a collection of MLPs to predict important object properties. Following [56] we use 3-layer perceptron with ReLU activation and linear projection layer to regress object class IDs and 2D bounding boxes. Additionally, we also regress SDF and LF feature vectors, denoted by 2.df and y respectively. These feature vectors provide an initial state for object reconstruction and are essential effectively reason about and refine the 3D component of these objects, as described in detail in Section 3.2.
3

---

Grid
Zgt
SDF MPP
Rocovered Surfoce
RGB
RG8
DR
Ar
Iccovered Surfnce
Recovered Surfore
MOES
Randared
Prodicted
Figure 2: Differentiable database of object priors (A) and shape, appearance, and pose opti- mization (B). Our object priors are learned using coordinate-based MLPs, taking positionally en- coded features as imput and recovering geometry in form of SDF and appearance on the object's surface. We use O-isosurface projection (marked P), to retrieve object's s surface from an SDF field. Our initial recovered object is then refined fit both predicted observations in terms masks and NOCS map, well ground truth RGB values from the input image.
3D Reasoning Block (Fig. 1c) Our 3D reasoning block aims to recover 3D scene information by splitting it into two parts: a background containing road surfaces, buildings, and other objects not detected by the detection transformer, and foreground entirely consisting of detected objects. We use the output of the transformer decoder for each object to compute multi-head attention scores of this embedding over the output of the encoder, generating M attention heatmaps per object. These masks are then used to regress the geometry for both foreground and background.
The foreground is predicted as set of normalized shape maps. Since they encode 3D coordinates via RGB, visualizing each 3D color component in 3D space allows us to recover par- tial 3D shape of the object in its normalized coordinate space (Fig. 3). This representation is crucial for our pipeline to trieve the object's full geometry and pose.
The background, on the other hand, is represented map since we solely care about its geometry. For view syn- thesis applications, we also predict the depth and appearance behind the detected objects by utilizing a GAN-like encoder-decoder generator architecture. This architecture takes masked RGB and depth images and inpaints RGB and depth values for occluded regions, employing a colly-coovolutionall PatchGAN discriminator to judge the genuineness of the inpainted maps.
depth
Partial shape recovery from predicted 2D NOCS map.
Figure
Given estimated object masks and NOCS maps, we recover three things enabling pose estimation: (1) 2D coordinates of the object in the image, (2) 3D points of the partial object shape in the canon- ical frame of reference,
(3) and their 2D-3D correspondences. This unique representation frees us from the need of storing a collection of 3D primitives and identifying them to find detected model, because both 3D and 2D information is contained in the form of a 3-channel map. We multiply the recovered normalized shape by the per class scale factor to recover absolute scale. The 6DoF pose is then estimated using PnP [57] that predicts the object pose from given correspondences and camera intrinsics. Since we get a large set of correspondences for each model, we also combine PnP with RANSAC to robustify against outliers. For the results presented in the evaluation section, we run 1000 RANSAC iterations per pose with a Ipz reprojection error threshold.
3.2 Shape, Pose, and Appearance Optimization
The second crucial component of our pipeline is a 2D/3D optimization that is enabled by the dif- ferentiable database of object priors (Fig. 2a) well as surfel-based differentiable rendering pipeline (Fig 2b). Given output features corresponding to detected objects regressed by the object reasoning block (Fig. lb), our optimization procedure generates full shapes, appearances and poses of all the objects in the scene.
We leverage differentiable database of object priors PriorDB which encodes the shape and luminance of the input models as Signed Distance Fields (SDFs) and Luminance Fields (LFss with associated latent feature vectors. Given partial shape observation, we differentiate against PriorDB

---

and find the maximum likelihood latent vector that best explains the observed shape. The RGB component is optimized similarly, with the complete procedure described in detail in See 3.2.1.
While NOCS-based SDF and LF optimization allows us to reconstruct plausible objects from im- ages, it does not account for the pose component. It might very well be the case that estimated poses are incorrect and are harming the full scene reconstruction. This can be alleviated via 2D alignment step, which can further be used as a complimentary source for shape optimization when predicted NOCS maps are noisy, resulting in a lower reprojection error and better reconstructions. Thus, to further constrain and improve the optimization, we define a differentiable rendering pipeline that al- lows us to impose 2D image losses based on renderings of the recovered SDF shape and appearance, as described in detail in Sec. 3.2.2.
3.2.1 Differentiable Database of Object Priors (Fig. 1d)
Our differential database of object priors or PriorDB represents objects as signed distance fields (with positive and negative values pointing to exterior and interior areas respectively), in which each value corresponds to the distance to the closest surface. A single MLP can be used to represent multiple geometries using a single feature vector žady and query 3D locations x = {z1, NN) as Object appearances, on the other hand, are represented as luminance fields (LF), defining the prior perceived luminance of the object as seen in the training set. Similarly to SDN, the LF module is implemented as an MLP, but it takes feature vector by concatenating zzdf and zif as well as query locations x as input, and outputs resulting luminance as fiy Zedf; Zit) =
fear 2xdf)
Both SDF and LF modules are trained using synthetic data pro- vided by Parallel Domain [58], containing high-quality realis- tic road scene renderings together with ground truth annota- tions. To train the SDF module, we use a collection of canon- ical ground truth point clouds and associated surface normals representing rendered objects. In total, we encode 27 cars in our PriorDB. We first define multi-level octree spanning the volume [-1, 1]", with points located at the corners of the imagi- nary voxels of various sizes depending on the level. For each of the point clouds we define unsigned distances by computing nearest neighbors to the octree points. Next, we define the loss Lad the distance to the surface at grid points starting from the coarsest level, gradually increasing the resolution up to the highest level (we use 3 levels of granularity as shown in Fig. 4), while only considering points located pre-defined per-level distances to the sur- face. To ensure that we learn a correct signed field, we additionally minimize the distance between the ground truth surface normals and analytically-deriver surface normals from the SDF module. Moreover, to ensure that we always regress plausible shapes regardless the input feature vector, we do an additional refinement round by sampling random feature vectors and enforcing the output to be equal to the shape of the nearest plausible neighbor.
Figure 4: Different octree levels used for training SDF primitives.
Once the SDF module is trained, we store SDF features Zed/ associated with objects and use them to train the LF module. Differently, it is trained on partial canonical shapes recovered from provided RGB renderings (see Fig. 2a), associated depth maps and poses. We compute the luminance compo- nent from RGB renderings and enforce the LF of each feature to be the same. The network weights then optimized fit the average luminance per object using an L1 loss. Additionally, we define a symmetry loss Leymm enforcing colors on the left side of the car to be similar to those on the right one. For that we compute nearest neighbors between the left side of the car and an inverted right side and then minimize the LI distance between these correspondences.
Our optimizer takes partial canonical object shape and appearance observations (recovered from predicted NOCS maps as shown in Fig. 1b), as well as initial PriorDB features. Due to the robustness of NOCS regression, and despite the occasional noisy outliers, we set it as our main optimization prior. The initially predicted SDF is used to recover the full surface of the object in the form of point cloud with surface normals using a 0-isosurface projection. Then we estimate nearest neighbors between the two point clouds and minimize the distance between the points by optimizing the feature vector zzd.. The RGB component is optimized similarly by minimizing the color difference between the correspondences, but we also allow the LF module weights to vary for a finer level of detail. In the case of RGB optimization we also use the Taymmm loss between the left and right sides of the predicted shape.
Optimization 3D
5

---

Figure 5: VKITTI2 reconstructions. Our SDN recovers partial reconstructions (left), which then completed with shape priors from PriorDB using our 2D-3D optimization pipeline (right).
3.2.2 Surfel-based Differentiable Renderer
Differentiable rendering allows to optimize objects with respect to pose and shape, and is an essential component of our optimization scheme. Since we use an SDF representation, we cannot directly use differentiable renderers for triangulated meshes. Instead, we implemented a renderer that uses surfels as the primary representation (inspired by [59]), estimating surface points and normals using an O-isosurface projection, followed by surfel-based rasterizer.
Surface Tangent Discs Surface elements or surfels are a common concept in computer graphics [60] as an alternative to connected triangular primitives. In order to render watertight surfaces, the individual surface normals must sufficiently approximate the local region of the object's 's geometry.
d Th Ps
(1a)
P P= (u d,
(1b)
n1 K (u, U. 1)T
To construct surface discs we first estimate the 3D coordinates of the resulting tangent plane given the normal 0f(p;z) of projected point pt. The distance d of the plane each 2D pixel (u, v) computed by solving system of linear equations for the plane and camera projection defined in Eq. la, where K is the inverse camera matrix. Then, we get get a 3D plane coordinate by backprojection (Eq. 1b). Finally, we estimate the distance between the plane vertex and surface point and clamp if it is larger than a disc diameter to get final discs M
3p.
- ||: - Pl12, 0).
Rendering Function Similarly to [59, 61], we combine colors from different surfel primitives based on their depth values to compose a final rendering. The distance of the primitive to the camera defines its final intensity contribution the image. To ensure that all primitive contributions sum up to 1 at each pixel we use a modified softmax function. The final rendering function is given in Eq. 2a, where I is the output image, S the estimated NOCS map, and w; the weighting masks.
I =
Σ S(p)) Wi.
(2a)
expl M Exerp(-De eeppp-DDo)MM
(2b)
Eq. 2b defines weighting masks w,, where D is the normalized depth and or is transparency weight with a - oc defining completely opaque rendering as only the closest primitive is rendered.
Optimization 2D Formally, we define three losses on the rendering output Lnocsap: Lifap and
Immaskan Inocagn compares the output of the renderer with predicted NOCS, Lifan compares the output of the renderer with input RGB images, and Lmaskan, compares the rendered and the predicted masks. The exact definitions of these losses are provided in the appendix. The final optimization loss is a weighted sum of both 2D and 3D components (see Sec. 3.2. While 3D components ensure robust shape optimization, 2D losses help to better align the object in 3D space and allow better luminance matching.
4 Experiments
In this section, we first evaluate the individual components of the SDN on common benchmarks for stereo view synthesis, depth estimation and 3D object detection. Then we demonstrate the capabili- ties of our full pipeline for the tasks of scene reconstruction, object recovery and manipulation. The evaluation is performed on four datasets: PD, VKITTI2, DDAD, and KITTI.

---

3D Reconstruction
Notrl ucang cirangement
(c) Recovered objects
eoovared background
Figure 6: Scene reconstruction using our pipeline on KITTL. Our pipeline decomposes the scene into objects (c) and background (d) and recovers full geometries and appearances (e). This allows for scene manipulation, view synthesis and novel scene arrangements (b). Despite being trained on synthetic data, our pipeline generalizes well to real images without any adaptation.
VKITTI2 [62] and KITTI [63] Virtual KITTI [64] is synthetic proxy to the KITTI dataset the standard benchmark for autonomous computer vision. It contains five virtual worlds rendered under ten different conditions weather and camera configuration, in total comprising 21 21,260 images. VKITTI2 extends the original dataset with more photo-realistic renderings. In the experiments use the first 80% consecutive frames of each configuration for training and the last 20% the clone configuration for testing. To evaluate object detection and depth estimation on real data use the KITTI-3D [65] validation split consisting of 3769 images.
Parallel Domain (PD) [58] and DDAD [2] The Parallel Domain dataset a procedurally- generated collection of fully annotated photo-realistic renderings of urban driving scenes, It contains 200 180-frame sequences resulting in a total of 36000 images. Each sequence also contains addi- tional frame with extra ground truth modalities which we use to form the PD test split with total of 200 images. The Dense Depth for Automated Driving (DDAD) dataset is real-world counterpart of the PD dataset mainly used as depth evaluation benchmark. We consider only the front camera, resulting validation sequences with 3950 images.
4.1 Scene Decomposition and 3D Reasoning
Stereo View Synthesis In this experiment quantitatively evaluate SDN for the task of view synthesis by recovering stereo-pair of the input image. We compare our results with state-of-the- art generalizable NeRF-bbaeed view synthesis method PixeINeRF [55] and depth estimation method PackNer-SfM [2]. All pipelines were trained on the same VKITT12 fraining split and evaluated on the VKITTI2 test split. From the results in Table 2, we observe that SDN can reliably re-generate the scene from novel view point thanks to its integrated prior and geometric nature, and outperforms both baselines. Please refer the appendix for qualitative results.
Depth Evaluation Next evaluate the quality of our geometric output using common metrics for depth evaluation. We argue that our NOCS maps, given estimated poses, can be transformed
Lower better
Higher is better
Train
Method
Test
AbaRell
SqRel
RMSE
RMSE
5 F: 1.25
K 1.25
6 1.25*
Monodepth2 Pac
DDAD KITTI DDAD KITTI
8.425
0.655
0.211
2,716
7,701
0.319
0.546
0.704
0.780
Ours
0.175
2,375
7,384
0.560
0.642
0.756
0.803
Monodepth2
0.437
15.621
26.578
0.731
0.025
0.118
0.241
Sac
0.340
11.290
22.332
0.309
Ours
0.238
10.402
18.228
0.346
0.535
0.682
0.704
Monodepth2
0.368
7.976
14,412
0.633
0.320
01480
0.580
PackNet-SIM
0.176
7.687
0.372
0.795
0.840
Monalepth2
0.187
3.951
6.465
0.184
01429
0.506
D.536
PucKNet-SIM
0.165
2,500
6.017
0,165
0.366
0.591
0.620
Table 1: Depth estimation results for direct transfer from synthetic (VKITTI2 and PD) to real (KITTI and DDAD) datasets. KITTI results reported with the Garg crop at distances up 80m, and DDAD results are reported without cropping and at distances up to 80m.

---

Method
PSNR1
SSIM
LPIPS)
BPV
Shape (mm)
Mask (loU)
RGB (PSNR)
PixeNNRR
16.63
0.45
0.57
Method
Easy
Med
Easy
Hard
Mean
Median
Mean
Mediun
Mean Median
PackNer-SIM
17.88
0.64
0.36
SMOKE
14.30
9,51
5.94
11.47
7.40
5.1
No ope.
46.57
14.87
83.85
87:34
16.34
Ours
18.51
0.66
0.31
Ours
16.50
11,88
10.15
10.94
7.54
6.72
9.30
85.51
3163
9.20
86.44
89.09
Table 2: View synthesis.
Table 3: KITTI-3D detection.
32.71
We evaluate the performance of our SDN for the task of stereo-pair recovery on the VKITTI2 dataset.
Reported are AP| R4o metrics of Car class with 50% IoU thresh- old. Both methods are trained on VKITTI2 train set.
Table 4: Optimization ablation.
Effect of optimization object shape and luminance estimation, and 2D amodal mask alignment.
to depth maps and significantly outperform standard depth estimation pipelines due to the prior ge- ometry knowledge. To conduct this experiment, we choose current sfae-oo-thhe-n depth estimation pipelines, Monodepth2 [1] and PackNet-SIM [2], and train them on 2 synthetic datasets VKITTI2 and PD similarly to our method. Then we evaluate the depth quality for foreground objects on real datasets- - KITTI and DDAD using number of standard depth estimation metrics. This experiment evaluates three aspects of our model: 1) the reconstructed object geometry, 2) the accuracy of the estimated poses, and 3) transferrability across domains. Table shows that our model consistently outperforms the baselines for monocular depth estimation, demonstrating the superior geometric accuracy and generalization qualities achieved thanks to the encoded object prior.
3D Detection In this experiment we aim to demonstrate how well our method generalizes to other datasets in terms of 3D detection quality. We compare against a state-of-the-art method for KITTI pose estimation, SMOKE [66], which is trained on the same VKITTI2 train set as our method. To compare with our algorithm that has access to ground-truth size of 3D boxes, we substitute the size of predicted boxes with the ones of nearest- neirest-neighbor ground-truth, when evaluating SMOKE. Table 3 shows that perform competitively with SMOKE, and demonstrates that our underlying NOCS representation generalizes to diverse shapes and poses of objects in the real-world.
4.2 Scene Manipulation and Object Recovery
The section aims to demonstrate the capability of our full pipeline to reconstruct and manipulate the scene. In Fig. 6, we show the scene decomposition results on the KITTI dataset. Our SDN takes as input single reference image (Fig. 6a) and recovers full 3D objects (Fig. 6c) as well as the scene background (Fig. 6d). Recovered objects can be freely manipulated as shown in Fig, 6b. The full 3D scene reconstruction is shown in Fig. 6e. Note that our pipeline has been trained entirely on synthetic data, and generalizes to real images without any adaptation. We refer the readers to our supplementary videos for the visualization of the full capability of scene manipulation.
Optimization ablation Lastly, we investigate how different components of our optimization pipeline affect the end result in terms of full shape and luminance estimation, as well as 2D amodal mask alignment. All evaluations are performed on the PD test split. We observe a significant per- formance increase for all considered metrics when compared to an unoptimized result as shown in Table While 3D optimization enables robust shape estimation. it does not align luminance in the projective space and also ignores object's 's pose since it is performed in the object's canonical frame. The 2D optimization step on the other hand is performed in the projective space and allows for exact luminance matching and good mask alignment. However, it performs worse with respect to shape optimization due to an additional ambiguity introduced by the pose component, which also affects the amodal mask metric. Finally, using both 2D and 3D optimization combines their strengths and yields the best overall performance.
Conclusion
We present a novel holistic scene reconstruction pipeline that disentangles the scene by representing objects separately from the scene. Our Scene Decomposition Network detects and recovers partial geometries and poses of objects, and estimates depth of the background. The partial geometries are then completed using our differentiable database of object priors and a 2D-3D optimization procedure. As result, our pipeline explains every pixel on the image and more the full object geometries and appearances. Despite being trained only on synthetic data our experiments show good generalization to real data on KITTI and DDAD datasets.
8

---

References
[1] C. Godard, O. Mac Aodha, M. Firman, and G. Brostow. Digging into self-supervised monoc- ular depth prediction. In ICCV, 2019.
[2] V. Guizilini, Ambrus, S. Pillai, A. Raventos, and A. Gaidon. 3d packing for self-supervised monocular depth estimation. In CVPR, 2020.
[3] S. Zakharov, Shugurov, and S. Ilie. Dpod: Dense 6d pose object detector in rgb images. In ICCV, 2019.
[4] K. Park, T. Patten, and M. Vincze. Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation. In ICCV, 2019.
[5] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from single image using multi-scale deep network. arXiv, 2014.
[6] I. Shugurov, Zakharov, and S. Ilie. Dpodv2: Dense correspondence-basee based 6 dof pose esti- mation. TPAMI, 2021.
[7] S. Peng. Liu, Q. Huang, X. Zhou, and H. Bao. Pynet: Pixel-wise voting network for 6dof pose estimation. In CVPR, 2019.
[8] Y. Chen, S. Huang, T. Yuan, S. Qi, Y. Zhu, and S.-C. Zhu. Holistic++ scene understanding: Single-view 3d holistic scene parsing and human pose estimation with human-object interac- tion and physical commonsense. In CVPR, 2019.
[9] Y. Du, Z. Liu, Basevi, A. Leonardis, B. Freeman, Tenenbaum, and J. Wu. Learning to exploit stability for 3d scene parsing. In NeurlPS, 2018.
[10] Huang, S. Qi, Y. Xiao, Zhu, Y. Wu, and S.-C. Zhu. Cooperative holistic scene under- standing: Unifying 3d object, layout, and camera pose estimation. arXiv, 2018.
[11] Y. Li, G. Wang, X. Ji, Y. Xiang, and D. Fox. Deepim: Deep iterative matching for 6d pose estimation. In ECCV, 2018.
[12] F. Manhardr, W. Kehl, N. Navab, and F. Tombari. Deep model-based 6d pose refinement in rgb. In 2018. 2018.
[13] I. Shugurov, I. Pavlov, S. Zakharov, and S. Ilie. Multi-view object pose refinement with differ- entiable renderer. RAL, 2021.
[14] Y. Nie, X. Han, S. Guo, Y. Zheng, J. Chang, and J. J. Zhang. TotalBdunddrrtanddin: Joint layout, object pose and mesh reconstruction for indoor scenes from single image. In CVPR, 2020.
[15] C. Zhang, Z. Cui, Y. Zhang, B. Zeng, M. Pollefeys, and S. Liu. Holistic 3d scene understanding from a single image with implicit representation. In CVPR, 2021.
[16] A. Kendall, M. Grimes, and R. Cipolla. Posenet: A convolutional network for real-time 6-dof camera relocalization. In ICCV, 2015.
[17] A. Kendall and R. Cipolla. Geometric loss functions for camera pose regression with deep learning. In CVPR, 2017.
[18] Walch, C. Hazirbas, Leal-Taixe, Sattler, S. Hilsenbeck, and Cremers. Image-based localization using Istms for structured feature correlation. In ICCV, 2017.
[19] F. Engelmann, K. Rematas, B. Leibe, and V. Ferrari. From Points Multi-Object 3D Recon- struction. In CVPR, 2021.
[20] S. Hinterstoisser, S. Holzer, C. Cagniart, S. Ilie, K. Konolige, N. Navab, and V. Lepetit. Mul- timodal templates for real-time detection of texture-less objects in heavily cluttered scenes. In ICCV, 1.

---

[21] S. Hinterstoisser, V. Lepetit, S. llie, S. Holzer, G. Bradski, K. Konolige, and N. Navab. Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In ACCV, 2012.
[22] W. Kehl, Tombari, N. Navab, S. Ilie, and V. Lepetit. Hashmod: A hashing method for scalable 3d object detection. In BMVC, 2015.
[23] P. Wohlhart and V. Lepetit. Learning descriptors for object recognition and 3d pose estimation. In CVPR, 2015.
[24] S. Zakharov, W. Kehl, B. Planche, A. Hutter, and S. Ili.. 3d object instance recognition & pose estimation using triplet loss with dynamic margin. In IROS, 2017.
[25] M. Bui, S. Zakharov, S. Albarqouni, S. Ilie, and N. Navab, When regression meets manifold learning for object recognition and pose estimation. In ICRA, 2018.
[26] O. H. Jafari, S. K. Mustikovela, K. Pertsch, E. Brachmann, and C. Rother. ipose: instance- aware 6d pose estimation of partly occluded objects. In ACCV, 2018.
[27] T. Hodan, D. Barath, and J. Matas. Epos: estimating 6d pose of objects with symmetries. In CVPR, 2020.
[28] Z. Li, G. Wang, and X. Ji. Cdpn: Coordinates- based disentangled pose network for real-time time rgb-based 6-dof object pose estimation. In CVPR, 2019.
[29] G. Wang, F. Manhardt, F. Tombari, and X. Ji. Gdr-net: Geometry-guider direct regression network for monocular 6d object pose estimation. In CVPR, 2021.
[30] H. Wang. Sridhar, Huang. J. Valentin, Song, and L. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In CVPR, 2019.
[31] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning contin- uous signed distance functions for shape representation. In CVPR, 2019.
[32] I.. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
[33] W. Chen, J. Gao, H. Ling, E. J. Smith, J. Lehtinen, A. Jacobson, and S. Fidler. Learning to predict 3d objects with an interpolation-
differentiable renderer. In NeurIPS, 2019.
[34] R. Chabra, J. E. Lenssen, E. Ulg, T. Schmidt, Straub, Lovegrove, and R. Newcombe, Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In ECCV, 2020.
[35] T. Takikawa, Litalien, K. Yin, K. Kreis, C. Loop, D. Nowrouzezahraii A. Jacobson, M. McGuire, and S. Fidler. Neural geometric level of detail: Real-time rendering with im- plicit 3d shapes. In CVPR. 2021.
[36] B. Deng, J. Lewis, T. Jeruzalski, G. Pons-Moll, G. Hinton, M. Norouzi, and A. Tagliasacchi. Nasa: neural articulated shape approximation. arXiv, 2019.
[37] V. Sitzmann, M. Zollhoefer, and G. Wetzstein. Scene representation networks: Continuous ad-structure-awar neural scene representations, NeurIPS, 2019.
[38] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In CVPR, 2020.
[39] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[40] S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv, 2019.
[41] I. Liu, Gu, K. Z. Lin, T.SS. Chua, and C. Theobalt. Neural sparse voxel fields. NewrIPS, 2020.
10

---

[42] D. Rebain, W. Jiang, S. Yazdani, K. Li, K. M. Yi, and A. Tagliasacchi. Derf: Decomposed radiance fields. In CVPR, 2021.
[43] D. B. Lindell, N. Martel, and G. Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In CVPR, 2021.
[44] M. Tancik, B. Mildenhall, T. Wang, D. Schmidt, P. P. Srinivasan, J. T. Barron, and R. Ng. Learned initializations for optimizing coordinate-basse -base neural representations. CVPR, 2021.
[45] K. Park, U. Sinha, T. Barron, S. Bouaziz, D. B. Goldman, M. Seitz, and R.-M. Brualla. Deformable neural radiance fields. arXiv, 2020.
[46] Z. Li, S. Niklaus, N. Snavely, and O. Wang. Neural scene flow fields for space-time view synthesis dynamic scenes. In CVPR, 2021.
[47] W. Xian, J.-B. Huang, Kopf. and C. Kim. Space-time neural irradiance fields for free- viewpoint video. In CVPR, 2021.
[48] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguet D-NeRF: Neural Radiance Fields for Dynamic Scenes. CVPR, 2021.
[49] R. Martin-Bann Brruala,, N. Radwan, M. S. M. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duck- worth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
[50] M. Boss, R. Braun, V. Jampani, J. T. Barron, C. Liu, and H. Lensch. Nerd: Neural reflectance decomposition from image collections. In CVPR, 2021.
[51] P. Srinivasan, B. Deng, X. Zhang, M. Taneik, B. Mildenhall, and J. T. Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR, 2021.
[52] M. Niemeyer and A. Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In CVPR, 2021.
[53] W. Yuan, Z. Lv, T. Schmidt, and S. Lovegrove. Star: Self-supervised tracking and reconstruc- tion of rigid objects in motion with neural rendering. In CVPR, 2021.
[54] J. Ost, F. Mannan, N. Thuerey, J. Knodt, and F. Heide. Neural scene graphs for dynamic scenes. In CVPR, 2021.
[55] A. Yu, V. Ye, M. Tancik, and A. Kanazawa, pixelnerf: Neural radiance fields from one or few images. In CVPR, 2021.
[56] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end detection with transformers. In ICCV, 2020.
[57] Z. Zhang. flexible new technique for camera calibration. TPAMI, 2000.
[58] Parallel domain. attps://paralleeldmminn com/. May 2021.
[59] Zakharov, W. Kehl, Bhargava, and A. Gaidon. Autolabeling 3d objects with differentiable rendering of sdf shape priors. In CVPR, 2020.
[60] H. Pfister, M. Zwicker, J. van Baar, and M. Gross. Surfels: Surface elements rendering primitives. SIGGRAPH, 2000.
[61] Liu, T. Li, W. Chen, and H. Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In ICCV, 2019.
[62] Y. Cabon, N. Murray, and M. Humenberger. Virtual kitti 2. arXiv, 2020.
[63] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
[64] A. Gaidon, Q Wang, Y. Cabon, and E. Vig. Virtual worlds as proxy for multi-object tracking analysis. In CVPR, 2016.

---

[65] A. Geiger. P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
[66] Z. Liu, Z. Wu, and R. Tóth. Smoke: Single-stage monocular 3d object detection via keypoint estimation. In CVPRW, 2020.
12


Reviews:
[{"review_id": "vRUKR0rHLJ", "reviewer": "Reviewer_JgCa", "summary": "This paper proposes a method for reconstructing, and decomposing a scene from a single RGB image. The main idea is to use a decomposition network to simultaneously predict normalized object coordinates a depth map of the background. The object poses can then be easily computed using PnP and refined using a database of object models and an optimization procedure. The results show that the proposed approach generalizes well and achieves excellent performance on view synthesis, depth estimation and object detection in this domain.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths** \n- The paper is very well written from start to finish. It is clear that the authors have spent a lot of effort on writing and polishing the paper. The paper is easy to follow and describes the method in sufficient detail (there is, however, some information that is missing related to the data used, see below).\n- The setup of the problem is very interesting and is the way I think perception problems should be addressed. By this I mean instead of just doing black-box depth prediction / object detection, the scene is decomposed in a more “symbolic” manner and the combination of each of these elements helps to describe the overall scene. This allows the method to be used for multiple tasks at the same time in an interpretable manner which is not often the case with deep learning approaches.\n- The evaluation is targeted and supports the claims made in the paper. In particular cross-domain depth estimation is tested, 3D object detection accuracy is tested and view synthesis is tested.\n- The results are good both quantitatively and qualitatively. The supplementary video showing the fully reconstructed scene and cars is quite impressive (however, it would be nice to see more examples).\n- The method shows great generalization capability. It is entirely trained on synthetic images (Parallel domain and VKITTI2) and is able to generalize well to real KITTI and DDAD. \n\n**Weaknesses**\n- The approach works only for a single class of object. For each new class, a database of object models is required that have been placed in a canonical frame of reference. This is ok for objects that don’t vary much in shape or appearance (like cars) but won’t work well for general objects. Of course, in the scenario autonomous / assisted driving this is not too much of an issue. However, it does limit the application to other areas.\n- There are not many example scenes shown in the results. There are only two individual frames in the supplementary video and one in the main paper. Since the paper addresses the problem of view synthesis it would be good to see a more diverse range of reconstructions to get a better idea of the quality. \n- There is not indication of the runtime of the method and how stable it is when applied to video prediction (which is how it would probably be applied in the autonomous vehicle / ADAS case).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This is a good paper in the sense that it presents an interesting method and achieves some solid results. However, the weaknesses I’ve considered in the recommendation are the fact that it will be difficult to adapt to other setups (apart from road scenes), and the paper limited examples showing how the approach works.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "I would like the authors to address the following in the rebuttal (or final revision):\n- How accurate are the different stages of the pipeline? I.e. how does the accuracy of the PnP compare to the final optimized result?\n- How is the scale factor determined at testing time, is there a fixed scale for each of the 27 database objects?\n- It would be good to provide more qualitative examples to show the performance of the approach across different scenes. When does the method fail? And when does it work really well?\n- How would the approach work with vehicles of other shapes like buses, vans or bicycles?\n"}}, {"review_id": "pXqkRJaTBe", "reviewer": "Reviewer_ky9j", "summary": "This paper introduces a complete pipeline to represent a scene with 3D model of moving objects and immovable background by a single image. The approach utilize NOCS-SDF, RGB-LF and surfel-based differentiable rendering to optimize the full geometry of movable object instances with the help of a object prior database. The decomposition of the scene enables the approach to edit the movable objects and re-render the scene.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The idea decomposing the scene to dynamic objects and static scene is quite commonly exploited. However, the representation of the object this paper developed has a good geometry representation. It can also recover the surface of object based on a prior object database. The combination of SDF and LF MLP makes sense from the perspective of theory and shows promising result. \n- This paper introduces the surfel-based differentiable renderer into this field for the first time. Comparing with NeRF, the surfel-based differentiable renderer is more geometrical consistent. It will not collapse when you generalize this approach to unseen scene.\n\nWeakness\n- The effectivenes of transformer-based feature extractor needs to be validated. Usually transformer-based architecture requires more data to keep the same performance as CNN. Is the transformer-based feature extractor really better than CNN in this task? Also, it would be misleading to use CNN instead of Transformer in the Figure 1. \n- It is not fair to compare this method with NeRF since NeRF deals with novel view synthesis. The fundamental principle of NeRF and this approach have a large discrepancy. It would be fair to compare it with monocular depth estimation with editable moving objects, which shares the same representation of this approach. You can compare different object representation under the framework of monocular depth estimation.\n- The details of object reasoning block are missing in the paper and supplementary material.\n- The authors claim this approach can apply to real data without domain adaptation. It would be nice if a quantitative result of the performance difference between VKITTI dataset and KITTI dataset.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work solve the monocular scene reconstruction by scene decomposition of dynamic objects and static scene. By introducing NOCS-SDF and RGB-LF based optimization and surfel-based differentiable render, the novel representation and rendering of these movable instances makes this approach generalizable to unknown scene (but not for unknown objects). The theory of this approach is solid and the overall design is reasonable for robotics application. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- Will the transformer-based feature extractor harm the performance of the depth estimation module? We have not seen many literature addressing this issue, and you are actually sharing the representation with two tasks: Depth and NOCS. A ablation study which shows the advantage of transformer-based feature extractor will be nice to have.\n- Why this approach can apply to real data without domain adaptation? Do you mean by similar scene like the example you showed in the paper: VKITTI and KITTI? If the scene changes, will it still be able to have a strong sim-to-real performance without domain adaptation? \n- The missing loss functions in the Section 3.2 can make the reader without related background knowledge hard to understand.\n- Can we have a way to show the advantage of partial shape recevery by quantitative analysis? This seems a crucial step to obtain a good final object reconstruction with differential render. "}}, {"review_id": "RN1FgRWV0kv", "reviewer": "Reviewer_pv1C", "summary": "This paper proposes a holistic single-shot scene reconstruction system that separately recovers the depth of the background and the 3D geometry and appearance of the objects.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- Results are strong.\n- It's an interesting idea to use SDF features to encode the prior dataset and fit the reconstructed partial object with the learned SDF decoder.\n\nWeakness:\n- The structure of section 3.2 is very messy:\n  - You have to read 3.2.1 and 3.2.2 before you can understand the first part of 3.2. Because this part is not high-level enough and many unexplained terms appear in this part.\n  Please briefly introduce the terms like PriorDB feature, 0-isosurface projection, and Lsymm loss.\n  - 158: The distance between which point clouds? Is the ground truth full point cloud of the objects available? Or partial point clouds are compared?\n- 207: Fig 5 does not have an \"a\n- It makes sense to fit SDF of prior shapes because cars' shapes are close. But the luminance field can be very different across the shapes. For example, in Figure 5, the rendered color of the second car on the right side of the road is orange, while in partial reconstruction, it's white.\n- No ablation study is conducted. For example, w/ or w/o the 3 components in 2D alignment. \n- Few qualitative results are shown in paper or supplementary material. No qualitative results of novel view synthesis of PixelNeRF and the proposed methods are shown, which makes the discussion not convincing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The idea is pretty cool. But the lack of qualitative results and the poor writing harms the quality of the paper. So I recommend rejecting the paper. But if the author can give a concrete rebuttal, I'll definitely reconsider.\n\nUpdate: The authors' responses address my questions. So I would like to change my recommendation to 'weak accept'.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "As stated in weakness.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "EfzKr5nRp1x", "reviewer": "Reviewer_jaEn", "summary": "This paper presented a new single view scene reconstruction method that can decompose the scene into the background and multiple objects (cars in the shown demo) and recover their full geometry and appearance. It also involves an optimisation step to refine the geometry, appearance and poses of the detected instances based on NOCS-SDF and RGB-LF fitting with a surfel-based differentiable rendering method. The quantitative evaluations show SOTA performance in terms of depth evaluation and 3D detection. It furthers shows that it can achieve scene manipulation and generation to enable novel view synthesis. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper presents a novel single-view holistic scene reconstruction/composition method. The strength is that it can recover the appearance by adding an extra latent embedding for LF. It also contains an optimisation capability using a surfel-based differentiable rendering approach, which makes the method applicable to unseen environments. \n\nIn terms of the weakness, there are a few points.\n1. Comparison: In the related work, the authors perhaps missed some recent work in the single-view scene reconstruction that can also recover the full object geometry. Such as \"Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image\" (CVPR 2020), \"Holistic 3D Scene Understanding from a Single Image with Implicit Representation\" (CVPR 2021), and some others. The authors might need to add these work in the related work part and show how their work is positioned against those work. The mentioned works can also recover the room layout and generate multi-class instances. From my perspective, the strength of this presented paper is that it can also recover the full appearance of the detected instances. However, this point is not evaluated or demonstrated in the experiment part. This innovation needs a thorough evaluation. \n\n2. Explanation of method: Section 3.2 needs a more clear explanation of the losses used for optimisation, for example, z_{sdf}, RGB loss, L_{symm}, L_{{NOCS_2D}}, L_{LF_{2D}}, L_{mask}, pose optimisation, normal distances. The equations for those losses and how they are summed under different circunsmaces are needed to be able to reproduce the results. \n\n3. Evaluations: Some qualitative results are needed to demonstrate the numbers in the view synthesis. Besides, as the pose estimation is an important step in the pipeline, it is expected to see some evaluations in the camera and object pose estimation. Also, the object and scene 3D reconstruction should also be evaluated. Since the aim is to recover the full geometry, it is expected also to evaluate the unseen part, not just the depth evaluation for the seen surface. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The presented paper shows some new ideas and novel results. However, it requires extra evaluations to prove its innovation and also needs more clear explanation in its method part. This makes me on the borderline (tends to weak accept). The final recommendation will depend on if the mentioned concerns can be solved.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "In addition to some issues mentioned in the weakness part:\n* L150: Fig.5a is not the figure referred does not include object prior database.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers received this paper positively. Both presentation and results were reported as convincing, however, the ratings were borderline (1x weak reject, 2x weak accept). The main criticism relates to the lack of qualitative results (all reviewers), evaluation shortcomings (pv1C: ablation study, JgCa: runtime and stability analysis, jaEn: accuracy of camera and pose estimation) and presentation (pv1C+jaEn: section 3.2). Given the authors revise the manuscript accordingly, I’d be happy to accept the paper. 
Decision: Accept (Poster)