DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images
Authors: Bing WANG
Lu Chen
Bo Yang
Venue: ICLR 2023 poster (2023)
Keywords: 3D Scene Decomposition
Object Manipulation
Neural Rendering

Abstract: In this paper, we study the problem of 3D scene geometry decomposition and manipulation from 2D views. By leveraging the recent implicit neural representation techniques, particularly the appealing neural radiance fields, we introduce an object field component to learn unique codes for all individual objects in 3D space only from 2D supervision. The key to this component is a series of carefully designed loss functions to enable every 3D point, especially in non-occupied space, to be effectively optimized even without 3D labels. In addition, we introduce an inverse query algorithm to freely manipulate any specified 3D object shape in the learned scene representation. Notably, our manipulation algorithm can explicitly tackle key issues such as object collisions and visual occlusions. Our method, called DM-NeRF, is among the first to simultaneously reconstruct, decompose, manipulate and render complex 3D scenes in a single pipeline. Extensive experiments on three datasets clearly show that our method can accurately decompose all 3D objects from 2D views, allowing any interested object to be freely manipulated in 3D space such as translation, rotation, size adjustment, and deformation.

Published as a conference paper at ICLR 2023
DM-NERF: 3D SCENE GEOMETRY DECOMPOSITION AND MANIPULATION FROM 2D IMAGES
Bing Wang Lu Chen Bo Yang!
Shenzhen Research Institute, The Hong Kong Polytechnic University
2 vLAR Group, The Hong Kong Polytechnic University UUniversity of Oxford b beinganggeplllg edu. hk bo. www.yanggpol...c.. edw.:ppolyu.u........ .du.hk
ABSTRACT
In this paper, we study the problem of 3D scene geometry decomposition and ma- nipulation from 2D views. By leveraging the recent implicit neural representation techniques, particularly the appealing neural radiance fields, we introduce an ob- ject field component to learn unique codes for all individual objects in 3D space only from 2D supervision. The key to this component is multiple carefully de- signed loss functions enable every 3D point, especially in non-occupied space, to be effectively optimized without 3D labels. In addition, we introduce an inverse query algorithm freely manipulate any specified 3D object shape in the learned scene representation. Notably, our manipulation algorithm can explicitly tackle key issues such as object collisions and visual occlusions. Our method, called DM-NeRF, is among the first to simultaneously reconstruct, decompose, manip- single pipeline. Extensive experiments
ulate and render complex 3D scenes three datasets clearly show that our method can accurately decompose all 3D objects from 2D views, allowing any interested object to be freely manipulated 3D space such as translation, rotation, size adjustment, and deformation.
INTRODUCTION
In many cutting-edge applications such as mixed reality on mobile devices, users may desire to vir- tually manipulate objects in 3D scenes, such moving chair making flying broomstick in 3D room. This would allow users to easily edit real scenes at fingertips and view objects from new perspectives. However, this is particularly challenging as it involves 3D scene reconstruction, de- composition, manipulation, and photorealistic rendering in single framework (Savva et al.) 2019).
traditional pipeline firstly reconstructs explicit 3D structures such as point clouds or polygonal meshes using SIM/SLAM techniques (Ozyesil al. 2017 Cadena et al. (0166) and then identi- fies 3D objects followed by manual editing. However, these explicit 3D representations inherently discretize continuous surfaces, and changing the shapes often requires additional repair procedures such as remeshing (Alliez et al. 2002). Such discretized and manipulated 3D structures can hardly retain geometry and appearance details, resulting in the generated novel views to be unappealing. Given this, it is worthwhile to design new pipeline which can recover continuous 3D scene geom- etry only from 2D views and enable object decomposition and manipulation.
Recently, implicit representations, especially NeRF (Mildenhall et al., 2020), emerge as a promis- ing tool to represent continuous 3D geometries from images. A series of succeeding methods (Boss) ctt al. [22T] Chen et al., | Zhang et al. 2021c) are rapidly developed to decouple lighting factors from structures, allowing free edits of illumination and materials. However, they fail to decompose 3D scene geometries into individual objects. Therefore, it is hard to manipulate individual object shapes in complex scenes. Recent works (ttelzner et al. [222]] Zhang el al. 202 Kania et al.] 2022 Yuan et al. 2022 Tschernekii et al., 2022 Kobayashi et al. 2022 Kim et al. 2022 Benaim al. 2022] Ren et al. 2022) have started to learn disentangled shape representations for potential geometry manipulation. However, they either focus on synthetic scenes or simple model segmenta- tion, and can hardly extend real-world 3D scenes with dozens of objects.
Ren
"Corresponding Author
Equal Contribution

---

Published as a conference paper at ICLR 2023
Object Field Component
Object Code
Object Manipulation
Step 1:
Step 2:
Point
Demity
Step
RGB
2D Views
NeRP
Viewing
Appessenne
with Powes
Sectic Representation
Direction
Figure I:
The general workflow of DM-NeRF. NeRF (green block) used as the backbone. We propose the object field and manipulation components illustrated by blue and orange blocks.
In this paper, we aim to simultancously recover continuous 3D scene geometry, segment all in- dividual objects 3D space, and support flexible object shape manipulation such as translation, rotation, size adjustment and deformation. In addition, the edited 3D scenes can be also rendered from novel views. However, this task extremely challenging as it requires: 1) an object decom- position approach amenable to continuous and implicit 3D fields, without relying on any 3D labels for supervision due to the infeasibility of collecting labels in continuous 3D space; 2) an object ma- nipulation method agrecable to the learned implicit and decomposed fields, with an ability to clearly address visual occlusions inevitably caused by manipulation.
To tackle these challenges, we design simple pipeline, DM-NeRF, which is built on the successful NeRF, but able to decompose the entire 3D space into object fields and freely manipulate their ge- ometries for realistic novel view rendering. As shown in Figure it consists of 3 major components: 1) the existing radiance field to learn volume density and appearance for every 3D point in space; 2) the object field which learns unique object code for every 3D point; 3) the object manipulator that directly edits the shape of any specified object and automatically tackles visual occlusions.
The object field is the core of DM-NeRF. This component aims to predict one-hot vector, iee. object code, for every 3D point in the entire scene space. However, learning such code involves critical issues: 1) there are no ground truth 3D object codes available for full supervision; 2) the number of total objects is variable and there is no fixed order for objects; 3) the non-occupied (empty) 3D space must be taken into account, but there are no labels for supervision as well. In Section 3.1 we show that our object field together with multiple carefully designed loss functions can address them properly, under the supervision of color images with 2D object masks only.
Once the object field is well learned, our object manipulator aims to directly edit the geometry and render novel views when specifying the target objects, viewing angels, and manipulation settings. A naïve method is to obtain explicit 3D structures followed by manual editing and rendering, so that any shape occlusion and collision can be explicitly addressed. However, is extremely inefficient to evaluate dense 3D points from implicit fields. To this end, as detailed in Section 3.2 introduce lightweight inverse query algorithm to automatically edit the scene geometry.
Overall, our pipeline can simultancously recover 3D scene geometry, decompose and manipulate object instances only from 2D images. Extensive experiments on multiple datasets demonstrate that method can precisely segment all 3D objects and effectively edit 3D scene geometry, without sacrificing high fidelity of novel view rendering. Our key contributions are:
We propose an object field to directly learn a unique code for each object in 3D space only from 2D images, showing remarkable robustness and accuracy over the commonly-used individual image based segmentation methods.
We propose an inverse query algorithm to effectively edit specified object shapes, while generating realistic scene images from novel views.
We demonstrate superior performance for 3D decomposition and manipulation, and also con- tribute the first synthetic dataset for quantitative evaluation of 3D scene editing. Our code and dataset are available at htttttts///gll http://ggtttt.u////// com vLAR - DMM NeRF
We note that recent works ObjectNeRF (Yang et al. 2021]. NSG (Ost et al. 2021) and ObjectSDF (Wu et al.] 2022) address the similar task as ours. However, ObjectNeRI only decomposes fore- ground object, NSG focuses on decomposing dynamic objects, and ObjectSDF only uses semantic label as regularization. None of them directly learns to segment multiple 3D objects as ours. A few works (Norman et al., 2022; Kundu et al., 2022 Fu et al. 2022) tackle panoptic segmentation in radiance fields. However, they fundamentally segment objects in 2D images followed by learning a separate radiance field for each object. By comparison, our method learns to directly segment all objects in the 3D scene radiance space, and it demonstrates superior accuracy and robustness than
2

---

Published as a conference paper at ICLR 2023
2D object segmentation methods such as MaskRCNN (He et al. (017) and Swin Transformer (Luu
et al. 2021b). especially when 2D object labels are noisy during training as detailed in Section
RELATED WORK
Explicit 3D Representations:
To represent 3D geometry of objects and scenes, voxel grids (Choy
et al., 2016), octree (Tatarchenko et al. 2017). meshes (Kato et al. 2018 Groueix et al.] 2018),
point clouds (Fan et al. 2017) and shape primitives (Zou et al. 2017 7 are widely used. A Although impressive progress has been achieved in shape reconstruction Yang et al. 2019a Xie al. 2019).
completion (Song et al. 2017), generation (Lin et al. 2018), and scene understanding (Tulsiani et al., 2018 Gkioxari et al. 2019). the quality of these representations are inherently limited by the spatial resolution and memory footprint. Therefore, they are hard to represent complex 3D scenes.
Implicit 3D Representations: To overcome the discretization issue of explicit representations, co- ordinate based MLPs have been recently proposed to learn implicit functions to represent continu- ous 3D shapes. These implicit representations can be generally categorized as: 1) signed distance fields (Park al. 2019). 2) occupancy fields (Mescheder et al.) (0119), 3) unsigned distance fields (Chibane et al.] 2020 Wang et al. 2022), 4) radiance fields (Mildenhall et al. (020), and 5) hy- brid fields (Wang et al. 2021). Among them, both occupancy fields and signed distance fields can only recover closed 3D shapes, and are hard to represent open geometries. These representations have been extensively studied for novel view synthesis (Niemeyer et al. 2020 Trevithick & Yang) 2021] and 3D scene understanding (Zhang et al. 2021a Zhi et al. 202 b). Thanks to the power- ful representation, impressive results have been achieved, especially from the neural radiance fields and its succeeding methods. In this paper, we also leverage the success of implicit representations, particularly NeRF, to recover the geometry and appearance of 3D scenes from 2D images.
3D Object Segmentation: To identify 3D objects from complex scenes, existing methods generally include 1) image based 3D object detection (Mousavian al.) 2017). 2) 3D voxel based detection methods (Zhou & Tuzel 2018) and 3) 3D point cloud based object segmentation methods (Yang) et al. 2019bj. Given large-scale datasets with full 3D object annotations, these approaches have achieved excellent object segmentation accuracy. However, they are particularly designed to pro- cess explicit and discrete 3D geometries. Therefore, they are unable to segment continuous and fine-grained shapes, and fail to support geometry manipulation and realistic rendering. With the fast development of implicit representation, it is desirable to learn object segmentation for implicit sur- faces. To the best of our knowledge, this paper is among the first segment all 3D objects of implicit representations for complex scenes, only with color images and 2D object labels for supervision.
3D Scene Editing: Existing methods of editing 3D scenes from images can be categorized as 1) appearance editing and 2) shape editing. A majority of works (Sengupta et al., 2019 Boss et al.,
2021 Zhang et al. 2021c. Chen al. T02T focus on lighting decomposition for appearance edit-
ing. Although achieving appealing results, they cannot separately manipulate individual objects. A
number of recent works (Munkberg et al., 202T Liu al., 2021a Jang & Agapito 202T Stelzner et al. 2021 Guandao Yang et al. (021)) start to learn disentangled shape representations for potential geometry manipulation. However, they can only deal with single objects or simple scenes, without being able to learn unique object codes for precise shape manipulation and novel view rendering. In addition, there are also plethora of works (Tewari et al., 20200, Niemeyer & Geiger [02 Dhamo et al. 2021; Alaluf et al., 2022) generation based scene editing. Although they can manipulate the synthesized objects and scenes, they cannot discover and edit objects from real-world images.
3 DM-NERF
Given a set of L images for a static scene with known camera poses and intrinsics EI,, K1) (I,, EL, Kt)). NeRF uses simple MLPs to learn the continuous 3D scene ge- ometry and appearance. In particular, it takes 5D) vectors of query point coordinates p (x. y, 2) and viewing directions v- = (0, Φ) as input, and predicts the volume density a and color c = (r, g, b) for point p. In our pipeline, we leverage this vanilla NeRF the backbone to learn continuous scene representations, although other NeRF variants can also be used, Our method aims to decompose all individual 3D objects, and freely manipulate any object in the 3D scene space. To achieve this, design an object field component to parallelly learn an object code for every query point p. together with object manipulator to edit the learned radiance fields and object fields.
3

---

Published as a conference paper at ICLR 2023
Est.ompry + sad.obs
Object Field Component
Oljjrr Gode Projection
Fdd. obj
Dopihh Estimattio
play,
rgh
Ground Truch
Scene Representation.
NeRF
Innuna Mark
Figure/ 2: The architecture of our pipeline. Given 3D point p. we learn an object code through using 2D and 3D supervision signals.
three loss functions: t2 abilEad empty faa obj in Eqs 9
3.1 OBJECT FIELDS
Object Field Representation: As shown in Figure 2 given the input point p. we model the object field as function of its coordinates, because the object signature of a 3D point is irrelevant to the viewing angles. The object field is represented by one-hot vector Basically, this one-hot object code aims to accurately describe the object ownership of any point in 3D space.
However, there two issues here: 1) the total number of objects in 3D scenes is variable and it can be or many; 2) the entire 3D space has a large non-occupied volume in addition to solid objects. To tackle these issues, we define the object code as H + dimensional, where H is predefined number solid objects that the network is expected to predict in maximum. We can safely choose a relative large value for H/ in practice. The last dimension of is particularly reserved represent the non-occupied space. Notably, this dedicated design is crucial for tackling occlusion and collision during object manipulation discussed Section 3.2 Formally, the object field is defined as:
=((), where E R #++1
(1)
The function f is parameterized by series of MLPs. If the last dimension of code is 1, it represents the input point p is non-occupied or the point empty.
Object Code Projection: Considering that it is infeasible collect object code labels in continuous 3D space for full supervision while it is fairly easy and low-cost to collect object labels on 2D images, aim to project the object codes along the query light ray back to 2D pixel. Since the volume density a learned by the backbone NeRF represents the geometry distribution, we simply approximate the projected object code of a pixel ó using the sampling strategy and volume rendering formulation of NeRF Formally, it is defined as:
K
ThOROs,
where
To = -E=erp-- o.,)). 0g =
k=1
with K representing the total sample points along the light ray shooting from pixel, a representing
the learned density of the jth sample point, 6g representing the distance between the (k: + 1)th and hthh sample points. From this projection formulation, can easily obtain 2D masks of 3D object codes given the pose and camera parameters of any query viewing angles.
Object Code Supervision: As shown in the right part of Figure 2. having projected 2D object predictions at hand, we choose 2D images with object annotations for supervision. However, there are two issues: 1) The number and order of ground truth objects can be very different across different views due to visual occlusions. For example, as to the same 3D object in space, its object annotation image #1 can be quite different from its an- notation in image #2. Therefore, it is non-trivial to consistently utilize 2D annotations for supervision. 2) The 2D annotations
Hungarian
"Macching"
Hungarion coceonee MMtcching
2D Object Predictions
GT Object Labels
Figure 3: Illustration of 2D object matching and supervision for Ezg_onj- only provide labels for 3D solid objects, non-occupied 3D space is never recorded in 2D images. Therefore, it is impossible to directly supervise non-occupied space, i.e., the last dimension of ô, from 2D annotations. These issues make adaptions of existing 2D methods ineffective, e.g., Mask- RCNN (He et al., 2017) or Swin Transformer (Liu et al. 2021b). fundamentally because they do not consider the consistency between 3D and 2D, but just segment objects on individual images.
To tackle the first issue, we use the Optimal Association and Supervision strategy proposed by 3D-BoNet (Yang al. 2019b). As illustrated in Figure 3 assuming we generate L images of 2D object predictions (T1
}r), T RUXV xVH++) and have the corresponding J. images of 21D
4

---

Published as a conference paper at ICLR 2023
ground truth object labels objects and T represents the number of ground truth objects. Note that the ground truth number of objects in each image is usually different, but here use the same T to avoid an abuse of notation.
RUXVXT, in which H is the predefined number of
For each pair, we firstly take the first H dimensions (solid object predictions) to be .M& RN where N UxV, while the last dimension is never used this stage. Likewise, is reshaped to be . E R NxT_ Then, M and M are fed into Hungarian algorithm (Kuhn 1955) to associate every ground truth 2D object mask with unique predicted 2D object mask, according to Soft Intersection-over- Union (sloU) and Cross-Entropy Score (CES) (Yang et al., (019b). Formally, the Soft Intersection-over Union (sIoU) cost between the hth predicted object mask and the eth ground truth object mask in the 1 pair is defined as follows:
and reshape them
E (MR MP)
C
(3)
Σ where MT and M'" are the th values of Mr, and M,. The Cross- Eoos--Entrop Score (CES) between M₁. and M, is formally defined as:
ME Σ Mr Em=((M M.))
N Σ n=1
CES
M log MR + (1 MM") log(1 - MX)
(4)
N
After the optimal association based (Criel + CCFS), reorder the predicted object masks to align with the T ground truth masks, and then we directly minimize the cost of all ground truth objects in every pair. The final loss fddoos is defined by averaging across all L image pairs.
T
T
1 b2d.obj L EsIoU CES)), where sIoU: = T (CC,) CES:
((CES)) (5)
T 1=1
1=1
To tackle the second issue, we turn to supervise the non-occupied object code in 3D space with the aid of estimated surface distances. In particular, given specific query light ray on which we sample K 3D points to compute the projected 2D object code ó, we simultaneously compute an approximate distance between camera center and the surface hit point along that query light:
k-1
d TR where k=11 where K represents the total sample points along light ray, a; is the learned density, and 6g is the distance between (k 1)th and k sample points, as same as Equation 2
Ta = expp Σ a;
O% = exp( kk)
(6)
As shown in Figure 4 once we have the surface distance d hand, we can easily know the relative position between ev- ery tt sample point and the surface point along the light ray. Naturally,
can then identify the subset of sample points surely belonging to empty space as indicated by green points, the subset of sample points near the surface as indicated by red points, and the remaining subset of sample points behind the surface as indicated by black points. Such geometric informa- tion provides critical signals to supervise empty space, i.e., the last dimension of object code o. Note that, the sample points
Pk
d'
Figure 4: Empty points identification. behind the surface may not surely belong to empty space, and therefore cannot be used supervi- sion signals. We use the following kernel functions to obtain a surfaceness score S and an emptiness score eg for the th sample point with the indicator function represented by 1().
8k erp( (dg d)²), e (1 sx) I(d Ad dz
where de represents the distance between camera center and the k*th sample point, and Ad is hy- perparameter to compensate the inaccuracy of estimated surface distance d. Note that, the indicator function is defined as: 1() = d Ad dg > 0, and otherwise 1( = 0. used mask out the sample points behind surface point during loss calculation. Given the emptiness and surfaceness scores for the total K sample points along the ray, we use the simple log loss to supervise the last dimension of object code denoted as oF H+1
K
1
Σ km
(x log(o (++1) + st log(1 - ox #+1))
Esdemmpyy
(8)
K
5

---

Published as a conference paper at ICLR 2023
Since there should be no solid object at all in the empty space, we apply the following fsdobj loss on the first H dimensions of the object code to push them to be zeros, being complementary to the existing fad loss. The ht dimension of the 1thh sample point's ' object code is denoted by of.
H
1
log - -ok)
Ck
K
h=1
To sum up, we firstly project object codes in 3D space back to 2D pixels along light rays using volume rendering equation, and then use optimal association strategy to compute the loss value with 2D object labels only. In addition, we introduce the key emptiness and surfaceness scores for points in 3D space with the aid of estimated surface distances. These unique scores are used to supervise the non-occupied 3D space. The whole object field is jointly supervised by:
l2d + endppypyy" + ozd
(10)
3.2 OBJECT MANIPULATOR
If we want to manipulate a specific 3D object shape (e.g,, translation, rotation and size adjustment), how does the whole scene look like in a new perspective after manipulation, assuming the object code and manipulation matrix are precomputed from users' interactions (e.g., click, drag or zoom)?
Intuitively. there could be two strategies: 1) firstly project 3D scene into 2D images, and then edit objects 2D space. 2) firstly edit objects in 3D space, and then project into 2D images. Compared with the first strategy which would inevitably incur inconsistency across multiple images due to the independent edits on individual views, the latter more favourable. Remarkably, our object field component can support the latter strategy. Regarding such a manipulation task on implicit fields, the core question is: how do we edit the codes alc/o of every sample point along the query light ray, such that the generated novel view exactly shows the new appearance? This is nontrivial as: First, we need to address potential collisions between objects during manipulation. This is quite intuitive, thanks to our special design of emptiness score the last dimension of object code o. Second, due to visual occlusions, object codes behind surface points may not be accurate as they are not sufficiently optimized. By comparison, the projected object code Ô along a light ray tends to be more accurate primarily because we have ground truth 2D labels for strong supervision. At last, we need a systematic procedure to update the codes with the known manipulation infor- mation. To this end, we design inverse query approach. Inverse Query: We design a creative inverse query approach to address above issues, realizing the favourable strategy: editing in 3D space followed by 2D projection. In particular, as shown in Figure 5 for any 3D sample point ps along a spe- eific query light ray, given the target (i.e., to-be-edited) ob- ject code o, and its manipulation settings: relative translation Ap=(Ar, Ay, Az), rotation matrix 3x3, and scaling factor 1>0, we firstly compute an inverse 3D point Pas, and then eval- uate whether P& and P belong to the target object, and lastly decide to whether edit the codes or not. Formally, we introduce Inverse Query Algorithm I to con- duct single light ray editing and rendering for object shape manipulation. Naturally, we can shoot a bunch of rays from any novel viewing angles to generate images of manipulated 3D scenes.
Pk
Px
inverse point
Figure Inverse points computation.
3.3 IMPLEMENTATION
To preserve the high-que qushityulity of image rendering, our loss in Equation [0 only used to optimize the MLPs of object field branch. The backbone is only optimized by the original NeRF photo-metric loss (Mildenhall et al. 2020). The whole network is end-to-end trained from scratch. The single hyper-parameter for our object field Ad set as 0.05 meters in all experiments.
EXPERIMENTS
4.1 DATASETS
DM-SR: To the best of our knowledge, there is no 3D scene dataset suitable for quantitative evalua- tion of geometry manipulation. Therefore, we create synthetic dataset with 8 different and complex indoor rooms, called DM-SR. For each scene, we generate the following 5 groups of images: Group (w/o Manipulation): Color images and 2D object masks at 400x pixels are rendered from viewpoints on the upper hemisphere. We generate 300 views for training.

---

Published as a conference paper at ICLR 2023
Algorithm Our Inverse Query Algorithm to manipulate the learned implicit fields. (1) o; the target object code, one-hot with H + dimensions. (Ap, R 0] represent the manipulation information for the target object. (2) {p: ... Ps: ..pp) represent the K sample points along a specific query light ray r. Note that, we convert all object codes into hard one-hot vectors for casy implementation.
Input:
The target object code ot, manipulation information [Ap, Raxa 0}:
The sample points (pi... .Pp .....pa "Pk...p along a specific query light ray r;
Output:
The final pixel color è rendered from the query ray after manipulation:
The final pixel object code rendered from the query ray after manipulation:
Preliminary step:
Obtain the projected pixel object code light ray before manipulation;
NOTE: The loop below shows how to edit per sample point in 3D space.
for in (Pi pp."'."" .Px do
Compute the inverse point par for pe: Pv (1/1)R ' "px Ap):
Obtain the codes (ax, Ck. 0&) for the point ps:
Obtain the codes {dv, Ctr, Ogr } for the inverse point par
Tackle visual occlusions:
if ou O, and Oi do:
Note: the target object is behind the surface but will be manipulated;
Obtain new implicit codes {āx, Ex. ] for pa after manipulation:
if ou f o, and o/- + and O
do: collision detected, EXIT.
if оı * o; Og o, do:
{ōr, Ce. o t {ow, Cx', 0g'.0w O
if 0 O: and Og Oe do:
(Ox, C2 ō - Og.. Cx', ow)
if 0g = Oe and og o o do:
[āx Cx. 5s - (0,0,0}
o * and og o do:
(ōx, o.) + {ar: Ci. o
After the above for loop, every point ps will get new implicit codes (ōx, Ez )x
NOTE: The step below shows how to project edited 3D points 2D image. According to volume rendering equation, the final pixel color object code are:
ē Ek ThārCe. 0 >x Trāuō
where T4 = eupl E G.b;), ae -erp( oxde..
Group (Translation Only): One object is selected to be translated along x or axis with ~ 0.3m.
Group 3 (Rotation Only): One object is selected to be rotated around axis with about 90 degrees. Group (Scaling Only): One object is selected to be scaled down about 0.8x smaller.
One object is selected be simultaneously trans- lated about ~ 0.3m, rotated about 90 degrees, scaled down about 0.8 smaller.
Group 5 (Joint Translation/RootaiinnSSCcie
For each group, 100 views generated for testing the same viewpoints.
Replica (Straub et al. 2019) is a reconstruction based 3D dataset of high fidelity scenes. We request the authors of Menantic-NeRE (Zhi et al. 2021a) to generate (180 training/ / 180 testing) color images and 2D object masks with camera poses at 640x480 pixels for each of 7 scenes. Each scene has 59~99 objects with very diverse sizes.
Replica:
ScanNet: ScanNet (Dai et al. 2017) is a large-scale challenging real-world dataset. We select scenes (~10 objects in each one) for evaluation. Each scene has ~3000 raw images with 2D object masks and camera poses, among which we evenly select 300 views for training and 100 for testing.
4.2 BASELINE AND METRICS
Scene Decomposition: The most relevant work us is ObjectNeRI (Yang al. 2021). but its design is vastly different: 1) requires a point cloud as input for voxelization in training, but we do not. needs GT bounding boxes of target objects manually prune point samples during editing, but we do not need any annotations in editing. 3) It only learns to binarily segment the foreground object and background, by pre-defining an Object Library training and editing. However, our object code is completely learned from scratch. This means that ObjectNeRF is not comparable due to the fundamental differences. Note that, the recent Semantic-NeRF ZZhi et al., 202 is also not comparable because it only learns 3D semantic categories, not individual 3D objects. In fact,
notice that all recent published relevant works (Yang et al. 2021 Zhi et al. 202 la Wu et al. 2022 Kundu et al.; 2022 Norman et al.) 2022 Fu et | 2022) do not directly tackle and evaluate multiple 3D object segmentation in literature. In this regard, we use the powerful Mask-RCNN (He et al. 2017) and Swin Transformer (Liu et al. 2021b) as baselines.

---

Published as a conference paper at ICLR 2023
Since train our DM-NeRF in scene-specife fashion, for faimess, also fine-tune Mask-RCNy and Swin Transformer models (pretrained under Detectron2 Library) on every single scene. In particular, we carefully fine-tune both models using up 480 epochs until convergence with learning rate 5e and then pick up the best models on the testing split of each scene for comparison.
Object Manipulation:
Since there is method that can directly manipulate objects in continuous radiance fields, we adapt the recent Point-NeRF (Xu et al.) 2022) for comparison, because it can recover both explicit 3D scene point clouds and novel views. Adaptation details are in Appendix.
Metrics: We use the standard PSNR/SSIM/LPIPP scores to evaluate color image synthesis (Milden- hall et al. 2020), and use AP of all 2D test images evaluate 3D scene decomposition.
4.3 3D SCENE DECOMPOSITION
Training with 100% Accurate 2D Labels: We evaluate the performance of scene decomposition on datasets. For DM-SR dataset, we evaluate method and baselines images of Group only, while the images of Groups 2/3/4/5 are used for object manipulation. For every single scene in these datasets, using 100% accurate 2D object ground truth labels, train separate model for our method, and fine-tune separate models for Mask-RCNN (MR) and Swin Transformer (Swin')).
Analysis: Table T shows that method, not surprisingly, achieves excellent results for novel view rendering thanks to the original NeRF backbone. Notably, our method obtains nearly perfect ob- ject segmentation results across multiple viewing points of complex 3D scenes in all three datasets, clearly outperforming baselines. Figure 10 shows that our results have much sharper object bound- aries thanks to the explicit 3D geometry applied in our object field.
DM-SR
Replics Datrecl
StomNNi Duitact
Synthees
Dectrmpaation
Aosal Syntheen
Decorppaates SwiaT
Symthms.
Licormpontinn
PSNE-
SSIM
1PPPS
MR
SwiaT
Ours
PSNRR
SSIMF 1PMP5
Ours
PSNRT
SS1MT [
SwinT
Ours
Balleoom
Bedioon
Office
0012
59.92
Dinsing
Office
07.58
93.25
Küches
0ffic-3
27.76
Rascptitt
70.33
0058.00
99.17
Rea
Room,0
76.90
79.83
81.63
$3.59
98.12
98.67
84.75
90.24
99.40
Average
81.03
Aossage
85.25
95.00
Table 1: Quantitative results three datasets. The metric for object decomposition is AP0.75
Robustness to Noisy 2D Labels: Since our method inherently has multi-view consistency while the 2D segmentation methods do not, it is expected that method has better robustness to inaccurate and noisy 2D labels in training. To validate this advantage, we conduct the following experiments on DM-SR dataset. Particularly,
illustrated in Figure [ randomly assign incorrect object labels to different amounts of image pixels of all training images 109%/50%///7%%/88///9999 and then our method and baselines are all trained with these noisy 2D labels.
Analysis: As shown in Table 2. it can be seen that our method still achieves an excellent object segmentation score (AP0.75 = 74.08) on resting/novel views, even though 80% of 2D labels are in- correct in training. By contrast, both baselines fail catastrophically once more than 50% of labels are noisy in training. Basically, this is because our dedicated losses stsp:o///es...ppttkeeee itly take into account object geometry consistency across multi-views, thus allowing the estimated 3D object codes to be resistant against wrong labels, whereas the existing 2D object segmentation methods only independently process single images, being easily misled by wrong labels in train- ing. This clearly demonstrates the remarkable robustness of our method. More experiments and quantitative/qualtttiie results regarding robustness are in Appendix.
explic-
Extension to Panoptic Segmentation: Our method can be casily extended to tackling panoptic segmentation by adding extra semantic branch parallel to object code branch. Due to the limited space, quantitative and qualitative results are provided in Appendix.
0% Noisy Label
10% Noisy Labels
50% Naisy Labels
70% Noisy Labels
80% Noisy Labels
90% Noisy Labels
Figure 6: Examples of 2D object labels with different levels of noise.
8

---

Published as a conference paper at ICLR 2023
10%
Noisy Labels
50%
Noisy Labels
70%
Noisy Labels
80% Noisy Labels
90% Noisy Labelss
MR
SwinT
Ours
MR
SwinT
Ours
MR
SwinT
Ours
MR
SwinT
Ours
MR
SwinT
Ours
Bathromm
98.93
98.96
99.81
62.94
99.63
7.71
99.02
1.89
1.79
7.11
9.69
Bedroom
98.61
98.85
100.0
75.41
92.94
100.0
3.00
3.96
100.0
2.96
3.18
82.83
2.95
2.75
4.25
Dinning
95.57
97.88
98.41
46.52
43.50
85.48
1.40
1.05
81.91
1.51
1.09
63.50
1.36
0.96
14.44
Kitchen
100.0
84,93
93.60
100.0
8.42
100.0
4.02
4.69
1.80
Reception
80.9
91.00
100.0
32.78
42.11
100.0
1.03
6.27
100.0
0.73
1.96
100.0
0.22
1.64
37.63
Rest
93.58
94.50
99.64
52.75
$1.99
99.32
1.63
1.57
99.33
2.46
2.02
66.74
1.63
1.69
11.11
Study
93.07
97.94
98.58
49.03
60.04
97.97
1.44
6,24
98.03
1.50
72.62
1.40
5.48
40.72
Office
95.09
97.00
100.0
66.28
69.11
100.0
4.13
2.96
100.0
2.61
3.94
97.00
2.66
2.48
0.00
Average
94.33
96.87
99.56
57.78
64.53
97.80
2.32
4.77
97.29
2.21
3.88
74.08
2,00
3.35
14.96
Table 2: Quantitative object decomposition AP0.75
Scores On noisy DM-SR dataset.
Translation
Rotation
PSNR1
SSIM
LPIPS)
App.3
PSNR
SSIMF
LPIPS_
App.9
Point-NcRF
25.79
0.847
0.202
25.21
0.818
0.203
Ours
33,94
0.975
0.033
89.33
31.94
0.969
0.038
85.68
Ablation Ours (w/o Exd)
32.84
0.967
0.048
87.26
30.38
0.945
0.090
82.46
Ablation Ours (w/o VO)
33.54
0.970
0.045
86.93
30.57
0.953
0.076
82.43
Scale
Joint
PSNR1
SSIMI
LPIPS.
AP0.0
PSNR1
SSIMI
LPIPS
AP0.0 f
Point-NeRF
25.83
0.848
0.202
23.51
0.816
0.226
Ours
33,40
0,971
0.037
86.05
30.65
0.965
0.045
81.70
Ablation Ours (wlo Fas))
31.84
0,959
0.062
83.31
29.95
0.947
0.088
77.36
Ablation 2: Ours (w/o VO)
32.43
0.964
0.054
76.33
29.85
0.951
0.075
74.71
Table 3: Quantitative results of object manipulation and ablation studies on DM-SR dataset.
4.4 3D OBJECT MANIPULATION
In this section, we directly use our model trained on the images of Group to test on the remaining images of Groups 2/3/4/5 in DM-SR dataset. In particular, with the trained model, we feed the known manipulation information of Groups 2/3/4/5 into Algorithm|] | generating images and 2D object masks. These (edited) images and masks are compared with the ground truth 2D views. For comparison, we carefully train Point-NeRF models in seene-specific fashion and apply the same manipulation information its learned point clouds followed by novel view rendering.
Analysis: Table 3 shows that the quality of our novel view rendering is clearly better than Point- NeRF (Xu et al. 2022), although decreases after manipulation compared with mon-manipulation in [ primarily because the lighting factors are not decomposed and the illumination of edited objects shows discrepancies. However, the object decomposition still nearly perfect, as also shown for deformation manipulation in Figure [1 More results are in Appendix.
4.5 ABLATION STUDY
To evaluate the effectiveness of our key designs of object field and manipulator, we conduct two ab- lation studies. 1) We only remove the loss functions (Ead_empty + fad_obj) which are jointly designed to learn correct codes for empty 3D points, denoted as w/o Ezd. 2) During manipulation, we remove the step "Tackle visual occlusions" in Algorithm I denoted as w/o VO. As shown in Table 3 both of our loss functions for empty space regularization and visual occlusion handling step are crucial for accurate 3D scene decomposition and manipulation. More ablation results are Appendix.
DISCUSSION AND CONCLUSION
We have shown that it is feasible to simultaneously reconstruct, decompose, manipulate and render complex 3D scenes in single pipeline only from 2D views. By adding an object field component into the implicit representation, we successfully decompose all individual objects in 3D space. The decomposed object shapes can be further freely edited by our visual occlusion aware manipulator.
One limitation is the lack of decomposing lighting factors, which is non-trivial and left for future work. In addition, manipulation of originally occluded objects or parts may produce artifacts due to the inaccuracy of learned radiance fields, although these artifacts can be easily repaired by applying simple heuristics such as continuity of surfaces or using better neural backbones.

---

Published as a conference paper at ICLR 2023
Acknowledgements: This work was supported in part by National Natural Science Foundation of China under Grant 62271431, in part by Shenzhen Science and Technology Innovation Commission under Grant NYYY12220022122000000 Grants 25207822 & 15225522.
in part by Research Grants Council of Hong Kong under
REFERENCES
Yuval Alaluf, Ron Mokady, and Amit H Bermano. HyperStyle: StyleGAN Inversion with Hyper- Networks for Real Image Editing. CVPR, 2022.
Pierre Alliez, Mark Meyer, and Mathieu Desbrun. Interactive geometry remeshing. TOG, 2002.
Sagie Benaim, Frederik Warburg, Peter Ebert, and Christensen Serge. Volumetric Disentanglement for 3D Scene Manipulation. arXiv:2206.. 02776, 2022.
Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan T. Barron, and Hendrik. Neural-PIL:
Neural Pre-Integrated Lighting for Reflectance Decomposition. NeurIPS, 2021.
Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, Jose Neira, Ian D. Reid, and John J. Leonard. Past, Present, and Future of Simultaneous Localization and Mapping: Towards the Robust- cobust-Perception Age. IEEE Transactions on Robotics, 2016.
Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang. Clement Fuji Tsang, Sameh Khamis, Or Litany, and Sanja Fidler. DIB-R++: Learning to Predict Lighting and Material with Hy- brid Differentiable Renderer. NeurIPS, 2021.
Julian Chibane, Aymen Mir, and Gerard Pons-Moll. Neural Unsigned Distance Fields for Implicit
Function Learning. NeurIPS, 2020.
Christopher B. Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction. ECCV, 2016.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nicßner. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR, 2017.
Helisa Dhamo, Fabian Manhardt, Nassir Navab, and Federico Tombari. Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs. ICCV, 2021.
Haoqiang Fan, Hao Su, and Leonidas Guibas. A Point Set Generation Network for 3D Object
Reconstruction from Single Image. CVPR, 2017.
Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao. Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Seg- mentation. 3DV, 2022.
Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh R-CNN. ICCV, 2019.
Thibault Groueix, Matthew Fisher, V G Kim, B C Russell, and M Aubry. A Papier-Mache Approach to Learning 3D Surface Generation. CVPR, 2018
Guandao Yang, Serge Belongie, Bharath Hariharan, and Vladlen Koltun. Geometry Processing with Neural Fields. NeurlPS, 2021.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask R-CNN. IEEE International Conference on Computer Vision, pp. 2961-2969, 2017.
Wonbong Jang and Lourdes Agapito. CodeNeRIF: Disentangled Neural Radiance Fields for Object Categories. ICCV, 2021.
Kacper Kania, Kwang Moo Yi, Marek Kowalski, Tomasz Trzciński, and Andrea Tagliasacchi. CoN- eRF: Controllable Neural Radiance Fields. CVPR, 2022.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3D Mesh Renderer. CVPR, 2018.
Mira Kim, Jaehoon Ko. Kyusun Cho, Junmyeong Choi, and Daewon Choi. AE-NeRF: Auto- Encoding Neural Radiance Fields for 3D-Aware Object Manipulation. arXiv:2204. 13426, 2022.
Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing NeRF for Editing via Feature Field Distillation. NewrIPS, 2022.
Harold W. Kuhn. The Hungarian Method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83-97 1955.
10

---

Published as a conference paper at ICLR 2023
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, An- drea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation. CVPR, 2022.
Lin, Chen Kong, and Simon Lucey. Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction. AAAI, 2018.
Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun- Yan Zhu, and Bryan Russell. Editing Conditional Radiance Fields. ICCV, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. ICCV, 2021b.
Lars Mescheder, Michael Oechsle, Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR, 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and
Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV, 2020.
Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3D Bounding Box Esti- mation Using Deep Learning and Geometry. CVPR, 2017.
Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting Triangular 3D Models, Materials, and Lighting From Images. arXiv:2111.12503 2021.
Michael Niemeyer and Andreas Geiger. GIRAFFE: Representing Scenes as Compositional Gener- ative Neural Feature Fields. CVPR. pp. 11453-11464, 2021.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable Volu- metric Rendering: Learning Implicit 3D Representations without 3D Supervision. CVPR, 2020.
M Norman, Andrea Simonelli, Lorenzo Porzi, Samuel Rota, Matthias Nießner, and Peter Kontschieder. AutoRF: Learning 3D Object Radiance Fields from Single View Observations.
CVPR, 2022.
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural Scene Graphs for Dynamic Scenes. CVPR, 2021.
Onur Ozyesil, Vladislav Voroninski, Ronen Basri, and Amit Singer. A Survey of Structure from Motion. Acta Numerica, 26:305-364, 2017.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. CVPR, 2019.
Zhongzheng Ren, Aseem Agarwala, Bryan Russell, and Alexander G Schwing. Neural Volumetric Object Selection. CVPR, 2022.
Mike Roberts and Nathan Paczan. Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding. ICCV, 2021.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied Al Research. ICCV, 2019.
Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W. Jacobs, and Jan Kautz. Neural Inverse Rendering an Indoor Scene from a Single Image. ICCV, 2019.
Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva, and Thomas Funkhouser. Semantic Scene Completion from Single Depth Image. CVPR, 2017.
Karl Stelzner, Kristian Kersting, and Adam Kosiorek. Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation. arXiv:2104.01144 148, 2021.
Julian Straub, Thomas Whelan, Lingni Ma. Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, and Hauke M Strasdat. The Replica Dataset A Digital Replica of Indoor Spaces. arXiv: /90.:00990007777 2019.
11

---

Published as a conference paper at ICLR 2023
Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Octree Generating Networks: Efficient Convolutional Architectures for Heshuuusolttion 3D Outputs. ICCV, 2017.
Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj. Florian Bernard, Hans-Peter Seidel, Patrick Perez, Michael Zollhofer, and Christian Theobalt. StyleRig: Rigging StyleGAN for 3D Con- trol over Portrait Images. CVPR, 2020.
Alex Trevithick and Bo Yang. GRF: Learning General Radiance Field for 3D Representation and Rendering. ICCV, 2021.
Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations. 3DV, 2022.
S Tulsiani, S Gupta, D Fouhey, A A. Efros, and J Malik. Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene. CVPR, 2018.
Bing Wang. Zhengdi Yu, Bo Yang, Jie Qin, Toby Breckon, Ling Shao, Niki Trigoni, and Andrew Markham. RangeUDF: Semantic Surface Reconstruction from 3D Point Clouds.
arXiv:2204.09133, 2022.
Peng Wang. Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. NeurlPS, 2021.
Qianyi Wu, Xian Liu, Yuedong Chen, Kejie Li, Chuanxia Zheng, Jianfei Cai, and Jianmin Zheng. Object-Compositional Neural Implicit Surfaces. ECCV, 2022.
Haozhe Xie, Hongxun Yao, Xiaoshuai Sun, Shangchen Zhou, Shengping Zhang, and Xiaojun Tong. Pix2Vox: Context-aware 3D Reconstruction from Single and Multi-view Images. ICCV, 2019.
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neu- mann. Point-NeRF: Point-based Neural Radiance Fields. arXiv, 2022.
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning copiettCoomosiiioonall Neural Radiance Field for Editable Scene Ren- dering. ICCV, 2021.
Bo Yang, Stefano Rosa, Andrew Markham, Niki Trigoni, and Hongkai Wen. Dense 3D Object Reconstruction from a Single Depth View. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019a.
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, and Niki Trigoni. Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurlPS, 2019b.
Yu-jie Yuan, Yang-tian Sun, Yu-kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. NeRF-Editing Geometry Editing of Neural Radiance Fields. CVPR, 2022.
Cheng Zhang, Zhaopeng Cui, Yinda Zhang, Bing Zeng, Marc Pollefeys, and Shuaicheng Liu. Holis- tic 3D Scene Understanding from Single Image with Implicit Representation. CVPR, 2021a. la.
Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fugiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu. Editable rrce-viewpoin video using layered neural representation. ACM Transactions on Graphics, 40(4), 2021b. ISSN 15577368.
Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, and Jonathan T. Barron. NeRFactor: Neural Factorization of Shape and Reflectance Under an Un- known Illumination. SIGGRAPH Asia, 2021c.
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison. In-Place Scene La- belling and Understanding with Implicit Scene Representation. ICCV, 2021a.
Shuaifeng Zhi, Edgar Sucar, Andre Mouton, lain Haughton, Tristan Laidlow, and Andrew J. Davi- son. iLabel: Interactive Neural Scene Labelling. arXiv:2111.14633 2021b.
Yin Zhou and Oncel Tuzel. VoxeINet: End-to-End Learning for Point Cloud Based 3D Object Detection. CVPR, 2018.
Chuhang Zou, Ersin Yumer, Jimei Yang, Duygu Ceylan, and Derek Hoiem. 3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks. ICCV, 2017.
12


Reviews:
[{"review_id": "mGAdV37sDu", "reviewer": "Reviewer_tL8f", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends on the power NeRF, by introducing an additional vector to present the object code for each 3D point. The authors designed a few loss functions and showed that with these loss functions the object code for the 3D points can be directly optimized from 2D image annotations. The authors then used the learnt object code to translate/rotate/scale the objects and showed promising experimental results.", "summary_of_the_review": "This paper presents a good idea and executes it well. The design of the object code vector and the loss functions is neat. The authors show superior accuracy compared with MaskRCNN. They further used this to demonstrate an application of manipulating the objects. This paper is a resubmission of an ECCV 2022 submission. The authors addressed lots of comments from the previous review cycle, and have made it a good submission for ICLR.", "strength_and_weaknesses": "+ The idea is novel. The design of the object code and the loss functions is simple and elegant.\n+ The authors have done extensive experiments on a few different datasets and showed superior results comparing with MaskRCNN\n+ This paper is a resubmission of an ECCV 2022 submission. The comments in the review and rebuttal have been addressed quite well in this submission.\n\n- The authors mainly compared their approach with only one existing approach (MaskRCNN). However, I do agree with the authors' explanation in section 4.2 that most relevant work focuses on similar but different problems thus it is hard to do a fair comparison with these works.\n- In all manipulation results, the proposed method still has a ghost boundary along the model pre-manipulation. Can you discuss this limitation?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ccqR2vN_Pcm", "reviewer": "Reviewer_Tr2a", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of 3D object segmentation and manipulation from images with instance segmentations. It uses NeRF and ask it to further produce an object label per 3D point. They introduce a Hungarian matching based loss to supervise the object label fields with 2D instance segmentations (although that part wasn't clear in the paper). They show better results than the 2D mask-rcnn baseline in terms of 2d segmentation.", "summary_of_the_review": "Overall, the problem setup is novel and the solution is new. However, the method is not clearly described and the comparison to baseline is not well-designed.", "strength_and_weaknesses": "Strength\n- The problem of 3D object segmentation from images is an important problem, and this paper takes an interesting step to solve it.\n- Novelty. The Hungarian matching-based loss seems to be an interesting solution to handle the 2D instance label mismatch during loss evaluation. However, the paper did not provide much detail. For instance, what is the cost function when matching the rendered labels to the ground-truth. When it may fail?\n\nWeakness\n- Clarity. My major concern is on writing and clarity. The method is hard to keep track of, and I don't think readers can easily understand how it work. To improve the writing, the authors may want to improve the figures, use precise notation for losses, and put adequate emphasis on certain parts. See minor comments for more.\n- Experiments. I found the comparison against over-fitted mask-rcnn to be not informative. Assuming the ground-truth segmentation is perfect, mask-rcnn fine-tuned per scene could easily overfit to the ground-truth labels and produce high accuracy. The fact it did not produce high accuracy made me think it is not well-trained. I can only see this experiment to be meaningful if the input segmentation is noisy, and the paper aims to clean it up, while mask-rcnn is not able to do so. \n- Generalization to real data and other setups. The method seemed to be trained on synthetic data with good instance segmentation. If we were to apply it to real world images, we need to answer how sensitive is the method to bad 2D instance segmentations. A related question is how to handle results of existing detectors (for instance Mask-RCNN only gives instance segmentation but not the background and stuff classes). Does it work with panoptic segmentation?\n\nMinor comments\n- Fig 1: object manipulation block is left empty, is this intended?                      \n- Fig 2: losses appeared but not discussed or referred to. Ideally, by looking at the figure, readers would be able to understand the gist of the method.\n- Fig 3: does this corresponds to l-2d-obj? Please make it clear in the caption.\n- Eq 4: $\\delta_k$ is not defined. Is it the same as $d_k$? If $\\delta_k$ is the depth, Eq 4 shouldn’t be called surface distance since it’s measuring the distance to the closest surface along the ray instead of the closest surface in the scene.                                                                                            \n- Eq 5: how does the indicator function work here? If $d-\\Delta d-d_k =0$, value is one?\n- Eq 7: The name of $l_{3d-obj}$ is slightly confusing as it optimizes the emptiness of the object label, but the name indicates it has similar functionality as $l_{2d-obj}$, which optimizes the object label.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WwpxDJVVrG", "reviewer": "Reviewer_BSjr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission proposed an extension of NeRF called DM-NeRF, which can segment individual objects and manipulate them (translation, rotation, deformation, etc.). To supervise object segmentation, the authors use their algorithm to generate multiple 2D object code predictions and match them with ground truth object labels using the Hungarian algorithm. To manipulate an object, they perform the inverse operation on sample points along a ray. The empirical results show high-quality novel view synthesis and better decomposition than Mask-RCNN.", "summary_of_the_review": "The authors proposed to extend NeRF object instance segmentation, providing a way to supervise the object code prediction which produced good results (better than Mask-RCNN). They also proposed an object manipulator which supports translating/rotating/scaling an object but generates artifacts and did not consider lighting conditions, which is my major concern.", "strength_and_weaknesses": "- The task studied in this work is important. As the authors mentioned, many cutting-edge tasks may require obtaining and manipulating individual 3D objects in a 3D scene.\n- The object code supervision proposed by the authors is natural and generates good results. As the authors mentioned, supervising the object code prediction is challenging as the ordering of the objects is not determined. The multiple views of 2D object code with the Hungarian algorithm is a good solution to it (idea originated from 3D-BoNet), and it produces better results than Mask-RCNN.\n- The object manipulator produced reasonable results but has space for improvement. Empirically, the manipulator produced artifacts (Figure 7, first line). At the end of the video “DM-NeRF_2D.mp4”, when the table is being rotated, they are some very unrealistic artifacts as well as unrealistic light.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GD6LwwAgFZ", "reviewer": "Reviewer_1RkG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel formulation based on Neural Radiance Fields (NERFs) to jointly reconstruc, semantically decompose, manipulate and render complex scenes in a unified pipeline.", "summary_of_the_review": "\nThere are multiple positives to this one. However, the main missing bits hurt all three levels of contributions made by the submitted paper as mentioned above.", "strength_and_weaknesses": "** Strengths:\n\n- (1) Readability.\n\nIn its current state, the paper is very well written. The main key ideas are well explained, motivated, justified and articulated.\n\n- (2) The relative positioning is very clearly and explicitely established.\n\nThis includes multiple discussions, in particular the differential wrt other existing object-centric NERF prior work.\n\n- (3) The overall maturity of the proposed package.\n\n- (4) The Related Work discussion is strong, overall.\n\nIts structure, populated references are pertinent and up to date, while the level of insights is a very good trade-off to make an efficient use of the text real estate.\n\n- (5) The performance of the proposed contribution, despite the relatively competitive field. \n\n** Weaknesses:\n\n- (1) The experimental quantitative evaluation lacks perspective.\n\nThis applies throughout and hurts the contributions: \n(i) decomposition contribution: Mask-RCNN (He 2017) is now considered a standard baseline rather than a strong state-of-the-art reference. In turn, the near perfect performance of the proposed method on the proposed dataset raises questions regarding the pertinence of the selected scenes to evaluate altogether. Having a more compelling baseline or multiple ones could help the reader accept the proposed contribution at the method level, near perfect. In absence of the aforementioned, it is still difficult to accept as is.\n\n(ii) Object Manipulation: there should be a better justification for a complete lack of a baseline, outside the ablative study as an experimental contribution.\n\n\n(iii) The contributed dataset: the proof of concept applications to validate its usefullness and quality is strongly tied to (i) above, which makes it slightly problematic as it currently stands.\n\n\n- (2) How much does it cost? (Conceptual discussion)\n\nThere are missing information about typical runtimes, resource usage and specificities, in particular in contrast to the closest related NERF-based derivatives out there.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster