Slot-guided Volumetric Object Radiance Fields
Authors: DI QI
Tong Yang
Xiangyu Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D object-centric representation learning
NeRF
3D-aware slot
TL;DR: This research tackles the challenge of 3D object-centric representation learning.  By using object slots as a guidance to compose object NeRFs, we precisely decompose 3D scenes into individual objects with limited training resources.

Abstract: We present a novel framework for 3D object-centric representation learning. Our approach effectively decomposes complex scenes into individual objects from a single image in an unsupervised fashion. This method, called \underline{s}lot-guided \underline{V}olumetric \underline{O}bject \underline{R}adiance \underline{F}ields~(sVORF), composes volumetric object radiance fields with object slots as a guidance to implement unsupervised 3D scene decomposition. Specifically, sVORF obtains object slots from a single image via a transformer module, maps these slots to volumetric object radiance fields with a hypernetwork and composes object radiance fields with the guidance of object slots at a 3D location. Moreover, sVORF significantly reduces memory requirement due to small-sized pixel rendering during training. We demonstrate the effectiveness of our approach by showing top results in scene decomposition and generation tasks of complex synthetic datasets (e.g., Room-Diverse). Furthermore, we also confirm the potential of sVORF to segment objects in real-world scenes (e.g., the LLFF dataset).  We hope our approach can provide preliminary understanding of the physical world and help ease future research in 3D object-centric representation learning.

Slot-guided Volumetric Object Radiance Fields
Di Qi
Tong Yang MEGVII Technology Inc. yangtonggmegvii..cmm com
Xiangyu Zhang' MEGVII Technology Inc.
MEGVII Technology Inc.
qidi (mmevvi..comm
zhangziangnuumegvii..cc
Abstract
We present novel framework for 3D object-centric representation learning. Our approach effectively decomposes complex scenes into individual objects from sin- gle image in an unsupervised fashion. This method, called slot-guided Volumetric Object Radiance Fields (sVORF), composes volumetric object radiance fields with object slots guidance to implement unsupervised 3D scene decomposition. Specifically, sVORF obtains object slots from single image via a transformer module, maps these slots to volumetric object radiance fields with hypemetwork and composes object radiance fields with the guidance of object slots at a 3D location. Moreover, sVORF significantly reduces memory requirement due small-sized pixel rendering during training. We demonstrate the effectiveness approach by showing top results in scene decomposition and generation tasks of complex synthetic datasets (e.g., Room-Diverse). Furthermore, we also confirm the potential of sVORF to segment objects in real-world scenes (e.g., the LLFF dataset). We hope our approach can provide preliminary understanding of the physical world and help ease future research in 3D object-centric representation learning.
Introduction
As humans, we can understand scenes, perceive discrete objects within it, and interact with these objects in a 3D environment. This object-centric, geometric understanding of the 3D world is fundamental ability in human vision [1]. In computer vision, researchers attempt to replicate this fundamental ability machine learning models with a high interest, due to its wide application ranging from robotics [2] autonomous navigation. To this end, machine learning models should bear two characteristics: unsupervised representation learning manner and 3D-aware generative mode [3].
Recently, with the advances neural radiance fields (NeRFs) [4] and representation learning [5, 6, 7]. there are some works [8, 9, 3] to achieve these two characteristics. These works lear to decompose objects and understand 3D scene geometry from RGB supervision via novel view synthesis in an unsupervised manner. To this end, volumetric-based methods [8] utilize volume rendering mechanism to implement a 3D-aware differentiable generative process without supervision. computational cost of volumetric-basee
avoid high methods, light-field based methods [9, 3] use light field formulation [10] to scale its application to large numbers objects in a scene. However, existing works suffer from some limitations. Light-field based methods lack strict multi-viww consistency [10] and fall into mask bleeding issues [3], which are harmful for object-centric representation learning. Besides, volumetric-basee methods fail to decompose scenes caused by attention rank collapse [8] and do not perform well in complex multi-object scenes.
In this paper, we propose novel framework for 3D object-centric representation learning, allevi- ating the issues of existing works. Our method, called slot-guided Volumetric Object Radiance
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Fields (sVORF), adopts volumetric rendering to synthesis novel views and use object slots as guidance to compose volumetric object radiance fields. Concretely, we firstly use an efficient trans- former module to extract object slots from a single image and learn object-aware slot features with the help of self-attention mechanisms [11]. Then we utilize hypernetwork to map these slots to volumetric object radiance fields. Finally, at each 3D location, we compose object radiance fields with the guidance of object slots for volumetric rendering. Thus, vVORF can avoid instinctive limits of light field formulation and resolve mask bleeding issues. Meanwhile, with the benefits from the guidance of object slots, sVORF can learn 3D-aware slot features and facilitate network optimization, caaumttrrc-bssele methods. Moreover, instead of rendering whole image [8, 9], sVORF only render a small amount of image pixels during the training phase, which significantly reduces the demand for training resources.
alleviating the issues faced by existing
To validate the effectiveness of SVORF, we conduct experiments on four synthetic datasets to assess the ability of scene decomposition (e.g., segmentation in 3D) and scene generation(e.g. novel view sythesis, scene editing in 3D). Our results demonstrate that sVORF can precisely decompose 31) scenes into individual objects and produce high-quality novel view images. Specially, sVORF outperforms other state-of-the-art methods by a significant margin in complex multi-object scenes. In ablation experiments, we also analyze the effects of core components and show their strength. Besides, we show the robustness of sVORF on unseen object appearance and unfamiliar spatial arrangements. Furthermore, extend our validation process to complex real-world LLFF data, confirming that our approach can segment objects in complex scenes with high accuracy.
To sum up, our contributions are three fold. First, we introduce a novel approach for 3D-centric representation learning, named sVORF, that effectively decompose objects from single image. Second, our slot-guided scene composition method avoids the shortcomings of existing methods and significantly reduces the memory requirements during training phase. Third, we validate the effectiveness of our proposed method on synthetic datasets and confirm the extendability of our approach on real-world scenes.
Related Work
2.1 Neural Scene Representation and Rendering
Neural scene representations parameterize 3D scenes with a deep network that map xyz coordinates to signed distance functions or occupancy fields. Equipped with differentiable rendering functions, they can be optimized using only 2D images, relaxing the requirement 3D ground truth. particular, Neural Radiance Fields (NeRFs) [4] can use an MLP to compute radiance values (color and density) for a given 5D coordinate (spatial location (x, y, Z)) and viewing direction (0, Φ)) and produce novel views with remarkably fine details. Numerous subsequent works have been introduced to address some its shortcomings and expand its applications, including rendering accelcration [12, 13, 14, 15, 16, 17, 18, 19, 20], NeRF with few images [21, 22, 23, 24), 3D reconstruction [25, 26, 27, 28, 29] and 3D scene semantic understanding [30, 31, 32, 33, 34]. However, these volumetric methods need hundreds of evaluations for ray, leading an expensive cost for rendering. To address this issue, Light Field Networks (LFN) [10, 24] directly map an input ray to an output color, making only a single evaluation of the MLP per ray.
2.2 3D Object-centric Representation Learning
Driven by the effectiveness of NeRF, recent research has attempted to combine 2D self-supervised object-centric models [5, 7] with neural scene representations decompose a 3D scene to individual objects. Earlier work [35] utilizes a slot-based encoder and NeRFs as 3D representations decompose 3D scene with extra multi-view dense depth supervision. Likewise, uORF [8] explicitly model the separation of objects and background with only images in training to address complex scenes. In order to avoid expensive cost of volume rendering, OSRT [3] and COLF [9] further replace the volumetric parametrization with a light field formulation. However, existing methods face some limitations. ObSuRF [35] need dense depth as supervision and can not handle complex scenes, while uORF [8] encounters expensive computation cost during training. Although light field formulation method [3, 9] resolve computation cost issues, they lack strict multi-view consistency [10] and easily fall into mask bleeding issues [3].

---

Decomposition
Composition
Volume Rendering
Global pooling
Target View
Podliral Encodiags +
Haper
NDP
MLP
Self-A leetion
la c.)
(dip, Ci.)
Encader
Cross Atteriin
A Alentios Block
(M my)
Aggesption Block
0d
@ Maal Surr
d)
Ieput View
Tarpet
Figure sVORF overview. The image encoder Eg (I) processes the source view of a scene to generate 2D image features that serve prior. Next, these features are fed into the Scene Decomposition module to infer object and background slots. A hypernetwork then maps these slots to volumetric object radiance fields. Finally, the object slots provide guidance for the recombination of object radiance fields to render arbitrary views with 3D-consistent object decomposition.
Different from the above methods, our method adopts volume representation to parameterize 3D scenes, avoiding the limits of light field formulation. Also, approach has low computation cost by only sampling a small amount of rays and avoids using depth supervision during training. Moreover, our method can be applied to object segmentation in real-world scenes (e.g. LLFF dataset) [36].
3 Method
The aim of our method is to decompose a scene to set of object-centric 3D representations given a single input image. An overview of our approach is provided in Figure 1. We firstly extract image features from an input image and decompose object slots from image features. Then we map these slots volumetric object radiance fields and compose these object radiance fields with the guidance of object slots for novel view synthesis.
3.1 Preliminaries: NeRF
We begin by briefly reviewing the Neural Radiance Fields (NeRFs). A radiance field encodes scene as a continuous volumetric radiance field f. The input to f is a location x E R and a viewing direction d E R2 while the output is an RGB color value NcRFs parameterize f as an MLP Dg. To make MLP learns high-frequency functions, the input coordinates x and viewing directions d are mapped into higher dimensional space with sinusoids function 7(-) before being passed into the ML. [37]. This process, known as positional encoding, allows the ML.P to better capture high-frequency scene content. Therefore, the function T can be formulated as:
R and volume density
R
&s : (1(x), rd)) (a,
Given N sampled points along a ray r and its predicted color and volume density {(c; o;), N}}, the expected color C(r) of camera ray can be derived from volume rendering:
C(r) ΣT. T exp (-a,6))) c, exp
where 8, indicates the distance between adjacent samples. The reconstruction loss between the rendered and true pixel colors is used during training optimize the parameters & of MLP.
3.2 Scene Decomposition from 2D Prior
Following [35, 9], we define the scene as combination of entities, where the first represent the objects and the last one represents the background. To obtain these entities, we first extract image feature as 2D prior using an encoder E(I from a scene image Instead of using slot attention module as existing methods, adopt efficient transformer module T infer object and

---

background slots from image feature, termed as S = [= Compared to slot attention module, this transformer module simple and easy train without Gated Recurrent Unit (GRU) block. For this transformer module, we take a global image feature ū avg_pooll E(T)) as initial object
slots Z = } i-1 Combining with K learned positional encodings, these slots explicitly model all pairwise interactions berween all slots and learn object-aware features via self-attention, which avoids inherent ambiguities in estimating color and geometry at occluded views, Section 4.4 for details. Then these slots bind and explain the input image representation with cross-attention to image feature. Finally, the slots are transformed to S = {s i- via feed forward network (FFN). The whole process can be formulated as follows:
K
S = (ZZ E(T))
3.3 Scene Composition
Given the infered slots S = { ==1 we recompose these slots to novel views of the same scene, which critical for achieving 3D object-centric representation learning. To achieve it, there are two forms: Spatial Broadcast (SB) [35, 8, 9, 3] and Slot Mixers (SM) [3]. SB and SM do scene composition on RGB space and feature space, respectively. SB can utilize 3D geometric bias (3D point or 3D ray) in the scene composition, facilitating the network optimization. Conversely, due to composition on feature space, SM is hard to optimize without 3D geometric bias. But SM can learn 3D-aware slot features, which is useful for scene decomposition. Combining the advantages of two forms, we propose a new method for scene composition. We transform slots into volumetric neural radiance fields to utilize explicit geometric bias and avoid the limits of light field formulation [10]. Then, we compose all volumetric neural radiance fields with the guidance of slots, making slot features 3D-aware.
Objects as Neural Radiance Fields To transform slot to its radiance field, we utilize a hyper- network [38, 39] H to map the slot s; directly to the parameters Φ of the associated object Neural Radiance Field. The mapping process is formulated as:
6i H(:))
where = 1, 1.... K, K is the number of object slots.
With the radiance field aa we can map location E I& with a viewing direction d E IR² to tuple of color c; and density o; of corresponding object:
i (7(x), >d)) + (at, c;)
(5)
In this way, we transform feature space to RGB space, introducing 3D geometric bias for scene composition.
Composing Mechanism Given K Object NeRFs ( ) compose their outputs { and {c.) at 3D location with the guidance of object slots. Specifically, we firstly leverage a feature aggregation block Dc to gather object slots and obtain an aggregate feature for query location 7(x):
z = ())
(6)
where D is a cross-attention layer [40] network. Then we pass z to an attention block D and compute a normalized dot-product similarity between z and each slot feature in S:
m (2
(7)
where m (m,.... mx), D is a cross-attention layer without linear operation. Finally, we compute the combined density ō and color é as follows:
K
K
c=
m.C.
(8)
1=0
Note that our model can train with small amount of sample rays, leading to significant reduction for computation and memory cost during training. We speculate that this characteristic benefits from our composing mechanism. In order to decompose small objects from scene, sample much more rays to cover the regions of these small objects, resulting in heavy training cost. However, our composing mechanism can perform well on small objects with a small amount of rays, addressing this large sampling cost, see Section 4.4.
essential to

---

3.4 Loss Functions
Reconstruction Loss We train our model across multiple scenes and only take view images as the supervisory signal. The reconstruction loss is formulated as:
Crecon Σ [[(()) -C
where the set of rays each batch, and Cge is the groundtruth color.
Connectivity Regularization We observe that some object radiance fields exist semi- cmiitrrasspaeent clouds, especially when the number of slots is much larger than the total number of objects in scene. To solve this issue, we apply connectivity regularization Lronnect to each Φs. by referring to the distortion loss presented in Mip-NeRF360 [41]:
Lconnect (1, w) =
w)=
Σ m 1
4 + 4+1
4
+
1+1
Σ
w (4i-1 - 44)
(10)
2
2
where w = (T, (1 exp (-0,,,))) is the weights along ray and is the normalized ray distance.
Total Loss The overall training loss function is formulated follows:
Lrecon + Aconuect Lconnect
(11)
where Aconnect is the scale balance the connectivity regularization Leonnect, which is be 0.01 in our experiments.
4 Experiments
Datasets
Following uORF [8], we experiment on several datasets increasing order of complexity.
CLEVR-567 [8]: The CLEVR [42] dataset is a widely used benchmark for evaluating object decom- position in computer vision. CLEVR-567 is multicamera variant of this dataset with 1,000 scenes for training and 500 scenes for testing. Each scene consists of with 5-7 CLEVR objects that randomly positioned and oriented with clean background. The objects in the scenes comprised of three geometric primitives: cubes, spheres, and cylinders. We follow uORF's setup in using a "Rubber" material with largely diffuse properties.
CLEVR-3D [35]: This dataset is also variant of CLEVR dataset, in which each scene consists of 3-6 basic geometric shapes of 2 sizes and colors. In particular, each scene includes 3 fixed views: the two target views are the default CLEVR imput view rotated by 120" and 240", respectively. Following ObSuRF [35], we train 35k scenes and test on the first 320 scenes of each validation set [43, 6].
Room-Chair [8]: This dataset contains 1,000 scenes designated for training and 500 for testing. In this dataset, each scene includes 3-4 chairs of identical shape with 3 different textures background.
Room-Divers [8]: This dataset is an upgraded Room-Chair. Each scene contains 4 distinct chairs, whose shape chosen randomly from ShapeNet [44] chair shapes, and range of background that is selected from 50 unique textures. There 5,000 scenes for training and 500 for testing.
MultiShapeeNer (MSN) [35]: This dataset comprises 11,733 distinct shapes, with each scene populated by 2-4 objects with different categories sourced from the ShapeNetV2 3D model dataset.
Local Light Field Fusion (L.FF) [36]: This dataset includes real scene scenarios with complex forground and background, making it highly challenging. We specifically utilize the forward-facing scenes Flower and Fortress from the LLFFF dataset, with each scene consisting of 27 training images and 6 testing images.
Baselines We compare our model with 2D object-centric learning method Attentiton [43] and four competitive 3D methods, namely uORF [8], COLF [9], ObSuRF [35] and OSRT [3], in terms of scene decomposition and novel view synthesis. Both COLF and OSRT are based on light field, while uORF and ObSuRF employ volumetric parameterization, which is similar to our approach.
5

---

Table 1: Scene segmentation results. Bold and Underline indicate state-of-the-ee (SOTA) and the
second best.
CLEVR-567
Room-Chair
Koom-Diverse
Model
3D metric
2D
metric
3D metric
2D metric
3D metric
2D metric
NV-ARI
ARI
Fg-ARIt
NV-ARI t ARI
FggARIt
NV-ARI
ARI t Fg-ARRI'
Slot Attention [43]
N/A
3.5
93.2
N/A
38.4
40.2
N/A
17.4
43.8
uORF [8]
83.8
86.3
87.4
74.3
78.8
88.8
56,9
65.6
67.9
COLF [9]
46.6
59.5
92.6
83.5
83.9
92.4
54.5
70.7
71.7
sVORF
81.5
82.7
92.0
87.0
87.8
92.4
75.6
78.4
86.6
Table 2: Results on CLEVR-3D dataset. Modeel is re-evaluated using the unified test set for fair comparison. Bold and Underline indicate state-of-the-art (SOTA) and the second best.
(a) 3D Segmentation.
(b) Novel View Synthesis.
Model
Supervision
ARI
Fg-ARI't
Model
LPIPS І
SSIM
PSNR t
ObSuRF [35]
imagh+depth
94.6
95.7
ObSuRF [35]
N/A
N/A
33.69
OSRT [3]
image
42.7
97.0
OSRT [3]
0.0367
0.9719
36.74
sVORF
image
86.0
96.3
sVORF
0.0258
0.9759
37.52
Metries To evaluate the quality of novel view synthesis, we use Learned Perceptual Image Patch Similarity (LPIPS) [45], Structural Similarity Index (SSIM) [46], and Peak Signal-to-Noise Ratio (PSNR). For scene segmentation in 3D, we measure clustering similarity using Adjusted Rand Index (ARI). The ARI score ranges from 0 to 1, with score of O indicating random segmentation and score of indicating perfect segmentation. For a fair comparison, we evaluate two types of 2D ARI metrics: ARI and FG-ARI, as well as three types of 3D ARI metries: NV-ARI, ARI", and FG-ARI". Specifically, both ARI and FG-ARI are computed on the source image to facilitate comparison with 2D methods. The FG-ART is further calculated solely the foreground regions, using ground-truth data. Similarly, ARI and FG-ARI" are calculated on all images. Lastly, the NV-ARI is computed on synthesized novel views.
4.1 Scene Segmentation in 3D
Given the soft slot masks of 3D locations, 2D segmentation masks are inferred through volume rendering. To clarify, we begin by mapping the pixel p in the rendered view to a ray r for sampling N points along it. Then, calculate the soft mask of each sampling point at all object NeRFs and derive the composite density ā using the mask information. Finally, we render the segmentation mask along the ray based the composite density yield the segmentation of pixel p.
Setup
Results We compare our method with uORF and COLF, and present the results Table 1. The comparison reveal that method outperforms all baselines in terms of NV-ARI and ARI values, particularly the NV-ARI, in both Room- Chhai and sVORF can effectively identify 3D objects from single image with better multi-view consistency. Furthermore, in the more complex Room-Diverse scene, our approach achieves comprehensive and distinct improvements
scenes. provides evidence that
other methods, indicating the robustness of our design. Moreover, report the performance on CLEVR-3D in Table 2a to ensure equitable comparisons with ObSuRF and OSRT. Compared with OSRT, sVORF achieves significantly higher ARI and similar FG-ARI without using depth information, which further proves that our volume parameterization can alleviate mask bleeding problems, as shown in Figure However, we encounter exceptions in the ARI and NV-ARI scores on the CLEVR-567 dataset, and the ARI" on CLEVR-3D dataset. Our benchmark scores are slightly lower than those of uORF and ObSuRF. We attribute this to the penalty imposed on reconstructing shadows of foreground objects. which are not included in the ground truth object masks. This inadequacy is illustrated Figure 3. Since the light source remains fixed in the CLEVR dataset, it is straightforward to model the shadow as translucent layer belonging to the object. Nonetheless, the high FG-ARI indicates that our proposed method proficient in forming factorized representations, which segment objects in a scene effectively.

---

Table Comparison on novel view synthesis from single image.
CLEVR-567
Room-Chair
Room-Divers
Model
LPIPS +
SSIM
PSNR
LPIPS +
SSIM
PSNR
LPIPS + SSIM
PSNR
uORF [8]
0.0859
0.8971
29.28
0.0821
0.8722
29.60
0.1729
0.7094
25.96
COLF [9]
0.0608
0.9346
31.81
0.0485
0.8934
30.93
0.1274
0,7308
26.02
SVORF
0.0211
0.9701
37.20
0.0824
0.8992
33.04
0.1637
0.7825
29.41
Furthermore, to validate our method on more challenging scenarios, we conduct experiments on the MSN dataset and show the results in Table 4. Compared with ObSuRF, sVORF achieves significantly higher Fg-ARI and comparable ARI without using depth information, which demonstrates the model's ability decouple more complex scenarios. Qualitative results are shown Figure 2.
Table 4: Comparison on MSN dataset.
Model
ARI*
Fg-ARI't
PSNR
ObSuRF [35]
64.1 63.4
81.4
27.41
sVORF
84.1
30.51
Figure
Qualitative results <VORF on MSN.
4.2 Novel View Synthesis
Setup For each test scene, we reserve one view an input and use the other views to evaluate the the quality of reconstruction.
Results In Table 3, sVORF outperforms all baselines most metrics, despite not employing the coarse-to-fine training schedule. These findings suggest that our method effective in producing high-quality images. The slightly lower LPIPS performance be attributed to the lack of perceptual loss in the s sVORF training process, which does not explicitly optimize the LPIPS performance. Moreover, there exists trade-off between structural and perceptual loss within the model. The excellent SSIM performance of our model indicates that the s sVORF can generate complex object shapes. Specifically, illustrated in Figure 3, our method generated diverse chair shapes with higher accuracy than COLF and uORF, even though it did not fully recover the background texture. addition, the results presented in Table 2b indicate that sVOFF outperforms OSRT in all metrics, despite using nearly half of the parameters as OSRT (48. 733 v.s. 83. MM). Without the addition depth information, our method synthesizes multi-view images with higher quality than obsurf, as shown in Table 4.
Room-Chair and Room-Diverse
4.3 Scene Design and Editing in 3D
Input image
GT
Input view
Novel view
Setup We examine the potential of our method for basic scene editing on the Room-Chair dataset. Following uORF, our investigation involves two types of modifications, namely, moving objects and changing backgrounds. For editing the posi- tion of foreground object, we move all query point coordinates its object NeRF, based on the
Chang:: Move
targeted movement. order to relocate the slot, an
affine transformation is applied the 3D sample points before passing them to the corresponding object NeRF.
Figure 4: 3D scene manipulation for moving object and changing background.
Meanwhile, for changing the background, we substitute the original background texture by replacing the original background slot feature with that of the target image.

---

GT
uORF
COLF
Ours
GT
OSRT
Our
Impuat view
Novel view
Novel seg.
GT
DORF
COLF
OurS
GT
DORF
COLF
Ours
Iuput view
Novel vieew
Nord sags
Figure 3: Qualitative Comparison. We compare the reconstructions the input view, a novel view, well the novel view segmentation using the uORF [8], COLF [9], OSRT [3), and our method four datasets. Our method produces a finer segmentation and more precise shapes.
Results Figure 4 shows that edited images maintain harmonious quality while accurately executing object movement and background replacement, affirming the strong correlation between slots and 3D objects sVORF, as well as the accuracy of our model in 3D scene segmentation.
4.4 Ablation Studies
Table 5: Ablation studies on the CLEVR-567 dataset.
In this section, we conduct ablation studies on the CLEVR-567 dataset to gain deeper understanding of how the various parts con-
Model
NV-ARI +
FG-ARI f
sVORF (w/o NVS)
15.1
31.2
sVORF (w/o CR) sVORF (density-weighted)
81.1
89.3
tribute to the overall effectiveness of our ap-
81.4
88.6
proach. More ablation studies on architec- tures detailed Appendix
VVRF (w/o SA)
79.6
85.4
sVORF (ours)
81.5
92.0
Role of Novel View Synthesis To investigate the influence of the novel view synthesis setup on the training process, modify our reconstructed target view to equal with the input view, i.e. we turns sVORF into a 2D image auto-encoder. As shown in Figure 5, the model divides images based areas rather than objects. This division led to substantially lower ARI results, presented Table 5. Above observations confirm the crucial role of viewpoint changes in scene decomposition. Specifically, the mapping between slots and 3D representations captures changes occurring in various regions due to differences in viewing angles between the target and source views. Changes that occur within regions belonging the same object are more closely approximated, which allows the model to converge to slots based on object clustering.
Connectivity Regularization As previously mentioned, novel view synthesis setup ensures distinc- tiveness between objects, while implementing Connectivity Regularization guarantees connectivity within the same objects. These two factors work together to achieve a clear decomposition of the scene. Figure visualizes that our Connectivity Regularization ensures that points on an object belong to the same slot by preventing semi-transpan transprranp clouds from being modeled by object NeRFs.

---

Numerically, shown in Table 5, the model's FG-ARI displays a significant increase from 89.3 to 92.0 when implementing connectivity regularization compared the model without it.
Composing Mechanism As described in Section 3.3, our composing mechanism can perform well on small objects with few rays. To verify this, we implement the common density-weight mean combination mode model and present the visualization in Figure 5. Our composing mechanism outperforms the density-weighted mean combination mode decoupling effect, especially for small objects that may otherwise be segmented into attachments of other objects.
Self-A Ateetionn in Scene Decomposition If an object in the input image is occluded, it may be mistakenly segmented into the slot of another object that corresponds to the occluded area. This issue can be resolved by leveraging the self-attention layer in the decomposition module as it facilitates the interaction between slots. As illustrated in the Figure 5, removing the self-attention layer leads to incorrect division of the occluded object, highlighting the importance of inter-slot interaction in preventing the collapse of the corresponding slot of the occluded object.
Image
GT
w/o NVS
w/o CR
Dengity-weighted
w/o SA
Ours
Ingalt view
Novel view
Figure 5: Qualitative comparison of ablation studies on CLEVR-567 dataset. Specifically, we evaluate the impact of four techniques Novel View Synthesis (NVS), Comnectivity Regularization (CR), Composing Mechanism, and Self-Attention (SA).
4.5 Generalization
We conduct three sets of generalization experiments. The three subsets assess the ability of VORF to generalize to unseen object appearances, unfamiliar spatial arrangements and grayscale images respectively.
Object Appearance For unseen object appearance, we follow the dataset in uORF, which is similar to CLEVR-567. The training set of this dataset excludes red cylinders and blue spheres, while the testing set includes only these types of objects. The results presented Table 6. It is noteworthy that our approach has encountered the appearance of test objects but achieves comparable results to the models trained standard CLEVR-567 dataset.
Table 6: Object Appearance
Model
ARI 1
Fg-ARIf
Nott-Attenioon [43]
2.2
N/A
uORF [8]
85.5
N/A
SVORF
82.0
92.6
SVORF(567)
83,9
95.8
Spatial Arrangements To assess the model's ability generalize a greater number objects with unseen, challenging arrangements, we train our method using 11 slots on the CLEVR-567 dataset and evaluate on the packed-CLa Becked-CLEVR-111 dataset provided by uORF. During training, we randomly mask off four slots prevent slot collapse, but we utilize all slots during testing. The performance of our method on the unseen object arrangements test is shown in Table and found to be reasonably well.
Table 7: Spatial Arrangements
Model
ARI 1
ARIA
Slat-Attention [43]
5.7
N/A
uORF [8]
83.2
N/A
SVORF
81.0
85.5

---

RGB color To explore whether the sVORF mainly relies on RGB color for scene decomposition, we conduct an evaluation on a grayscale version of CLEVR-567 dataset. The model used in the evaluation is only trained on RGB CLEVR-567 dataset. The model achieves 87.5 FG-ARI on the grayscale test set, which on par with 92.0 FG-ARI on the default RGB images. The evaluation results demonstrate that s sVORF really learns to decompose the scene intrinsically. For qualitative results, see details in Appendix D.
4.6 Object Segmentation on Real Images
Image
Ours
GT
We demonstrate the effectiveness of our approach real-world scenarios by validating on LLFF datasets. To handle complex scenarios, we imple- ment VTT-Base [47] as the backbone instead of ResNet34 [48] and train the network from scratch. Qualitative visualizations displayed Figure show that sVRRF effectively segments the object within the scene, demonstrating the potential of approach to understand general real scenarios.
Nov
Figure 6: Object segmentation on real images.
4.7 Training Speed and Memory Consumption
Setup We compare sVORF with uORF [8] and COLF [9] in terms of memory consumption and training speed. For all three methods, we train models on the CLEVR-567 dataset using batch size of I on V100 and record their memory usage and time taken for one epoch. Since both COLF and uORF render and supervise images at 64 x 64 resolution to learn the coarse structure first before supervising them 128 128 resolution, we record their performance during both stages.
Results As shown in Table 8, sVORF offers substantially shorter training time (3m24s vs. 9m2s) and lower memory con- sumption (4.6G vs. 23.7G) than the volumetrie-decoderr based uORF method, while displaying comparable performance to the light field-base based COLF
Table 8: Memory Consumption and Performance Comparison.
Volumetric Field
Light Field
Model
uORF [8]
COLF [9]
SVORF
coarse
fine
coarse
fine
Memory
4.6G
23.6G
23.7G
2.8G
3.8G
Training Time
3m24s
8m42s
9m2s
Im56s
2m22s
In terms of total training time, for the CLEVR-567 and Room-Chair datasets, we train sVORF for approximately 7 hours using Nvidia RTX 2080 Ti GPUs with batch size 16. The uORF and COLF models are trained on an Nvidia RTX V100 GPU for approximately 7 and days, respectively, with batch size of 1. For the CLEVR3D dataset, sVORF is trained for approximately 2 days using Nvidia RTX V100 GPUs with batch size 16, while OSRT is trained for approximately day 8 A100 GPUs with batch size of 256. These results demonstrate the effectiveness VORF in overcoming the challenges of volumetric decoders that demand extensive resources, while also contributing to enhanced training efficiency.
Conclusion
We present sVORF, novel method for 3D object-centric representation learning. By adopting volumetric rendering to synthesis novel views and using object slots as a guidance to compose volumetric object radiance fields, sVORF precisely decompose 3D scenes into individual objects with limited training resources. It significantly outperforms existing SOTA methods, particularly in complex multi-object scenes. In addition. we demonstrate sVORF's ability for realistic scenario decomposition, indicating promising direction for understanding the physical world.
10

---

References
[1] Elizaheth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1):89-
96, 2007.
[2] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson. Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint mrXiv22303.08378, 2023.
[3] Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. Advances in Neural Information Processing Systems, 35:9512-9524, 2022.
[4] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[5] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint 1xXiv:190.111399 2019.
[6] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object repre- sentation learning with iterative variational inference. In International Conference on Machine Learning, pages 2424-2433. PMLR, 2019.
[7] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint urXiv:2001.02407. 2020.
[8] Hong-Xine Yu, Leonidas Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. arXiv preprint arXiv:2107.07905 2021.
[9] Cameron Smith, Hong-Xing Yu, Sergey Zakharov, Fredo Durand, Joshua Tenenbaum, Jiajun Wu, and Vincent Sitzmann. Unsupervised discovery and composition of object light fields. arXiv preprint arXiv: 2022.
[10] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 34:19313- 2021.
[11] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 213-229. Springer, 2020.
[12] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin, Fast- nerf: High-fidelity neural rendering at 200fps. Proceedings the IEEE/CVF International Conference on Computer Vision, pages 14346-14355, 2021.
[13] David B Lindell, Julien NP Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[14] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances Neural Information Processing Systems, 33:15651-15663. 2020.
[15] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1- 15, 2022.
[16] Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14153-14161, 2021.

---

[17] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 14335-14345, 2021.
[18] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul Srinivasan, Jonathan Barron, and Ren Ng. Learned initializations for optimizing coordinate-basee neural repre- sentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[19] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752-5761 2021.
[20] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXtv:2010.07492, 2020.
[21] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587 2021.
[22] Qianqian Wang. Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690-4699, 2021.
[23] Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for occlusion-aware image-based rendering. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7824-7833, 2022.
[24] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 6229 0238, 2022.
[25] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint 10..vv2206..//.
2021.
[26) Yiqun Wang, Ivan Skorokhodov, and Peter Wonka. Improved surface reconstruction using high-frequency details. arXiv preprint arXiv:2206.078500 2022.
[27] Wenqi Yang, Guanying Chen, Chaofeng Chen, Zhenfang Chen, and Kwan- Yee K Wong. Ps-nerf: Neural inverse rendering for multi-view photometric stereo. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part I, pages 266 284. Springer, 2022.
[28] Wang Yifan, Lukas Rahmann, and Olga borkine-Horuung Geomeetyy-consisteet neural shape representation with implicit displacement fields. arXiv preprint arXiv:2106.051181 2021.
[29] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002. 2020.
[30] Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any-view self-supervised object segmentation on complex scenes. arXiv preprini wwXiv:2209.08776.1 2022.
[31] Xinhang Liu, Jiaben Chen, Huai Yu, Yu-Wing Tai, and Chi- Tang. Unsupervised multi- view object segmentation using radiance field propagation. arXiv preprint 20.0:222200000:0 2022.
12

---

[32] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison. In-place scene labelling and understanding with implicit scene representation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15838-15847, 2021.
[33] Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, An- dreas Geiger, and Yiyi Liao. Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. arXiv preprint arXiv:2203.15224 2022.
[34] Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buló, Norman Müller, Matthias NieBner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. arXiv preprint arXiv:2212.098822 2022.
[35] Karl Stelzner, Kristian Kersting, and Adam Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.01148, 2021.
[36] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4):1-14, 2019.
[37] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
[38] David Ha, Andrew Dai, and Quoe V Le. Hypemetworks. arXiv preprint 100109660 2016.
[39] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-awart neural scene representations, Advances in Neural Information Processing Systems, 32, 2019.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[41] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5470-5470 2022.
[42] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2901-2910, 2017.
[43] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances Neural Information Processing Systems, 33:11525- 2020.
[44] Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information rich 3d model repository. arXiv preprint
2015.
[45] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[46] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
[47] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint 2020.
13

---

[48] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
14

---

A Additional Training and Architecture Details
A.1 Training details
We utilize the Adam optimizer with learning rate of 0.0001, .1 0.9, and 82 0.999. Additionally, we implement learning rate warm-up for the initial 1,000 iterations. The minimum number K of objects in each scene is customized separately follows: K = 8, 7,5, 5, and 5 for the CLEVR-567, CLEVR-3D, Room-Chair, Room-Diverse, LLFF, and MSN datasets, respectively.
To allow training
high resolution, such as 256 x 256, we render individual pixels instead of large-sized patches. Specifically, we randomly sample a batch of 64 rays from the set of all pixels in the dataset, and then follow the hicrarchical volume sampling [4] to query 64 samples from the coarse network and 128 samples from the fine network.
In addition, we train our model from scratch, with the exception of the Room- Diveree dataset is more complex and requires an incremental learning approach. Specifically, we initialize our models for Room-Diverse using weights from model that have previously been trained on the CLEVR-567 dataset.
dataset. The
A.2 Architecture details
Image encoder For the image encoder E, we utilize the well-known ResNet34 [48] architecture as the backbone, followed by three upsampling layers. Specifically, given the source image I E 2226x250x3 we extract features F1 E 1228x x28 F2 @64x64x12 and F3 x232x32x256 from the conv2, conv3 and conv4 layer of the ResNet34 architecture, respectively. Note that the max-pooling layer in conv2 is not used. Subsequently, these features are fed into the following three up-sampling layers within a U-net expansive path, resulting in the feature F E R 128 128x128888 128x512 The architecture of upsample layers shown in Table 9.
Table 9: The architecture of upsample layers.All convolutional kernel sizes are 3x3, All activation functions for convolutional layers are ReL.U. "+" indicates channel concatenation with a feature map of same resolution sourced from ResNet34.
Layer name
Input shape
Output shape
Stride
Conv1
32 x 32 x 256
32 x 32 x 512
1
Bilinear Upsampler Conv2
32 x 32 512
64 x 64 512
64 (512 + 128)
64 x 64 512
Bilinear Upsampler Conv3
64 x x 512
128 x 128 x 512
128 x 128 x (512 + 64)
128 x 128 x 512
Object NeRF To represent each object NeRF, employ simple 4-layer MLP that each layer has 128 channels and followed by a ReLU activation. In addition, we use skip connection mechanism that add the first layer's 's activation to the third layer's activation. Note that the last layer outputs the RGB value with a sigmoid activation function.
Transformer-based module The efficient transformer module is a standard transformer decoder. Specifically, we firstly use each slot z; as query and interact with other object slots with multi-head self-attention operation. Then we employ multi-head cross-attention operation to attend into and aggregate features from the flattend image features E(I). Finally, we pass the resulting slot features into a feed forward network (FFN) to get the final slots. This transformer module is simple and easy to train than the slot attention module, it does not contain a Gated Recurrent Unit (GRU) block.
Composition module The aggregation block performs a cross-attention operation, which aggregates object representations with the 3D location x as the query to obtain the corresponding feature 2. The attention block computes the similarity between S and 2 after mapping them into the same space through a linear layer, thus obtaining the probability distribution of belonging to each slot. In the initial stages of our experiments, we observed that utilizing both modules simultaneously yielded superior results. We speculate that this improvement may be attributed to a good feature space alignment between 3D points and slot features using the proposed aggregation block.
15

---

B Object Segmentation on Real Images
Setup In this study, we use VIT-Base [47] as the feature extractor for complex datasets of size 1008x 756 with intricate textures. The adopted backbone networks are trained from scratch, and the method strictly follows unsupervised learning. To conserve memory, we resize the source image to 256x 256 and select only 7168 rays from the target view during each training iteration. Moreover, we define the task as foreground-backgroundd segmentation. As such, we set the number of slots to and map it to two NeRFs. Each NeRF consists of simple 4-layer MLP.
Unlike the setup on other datasets, we train sVORF on both LLFFF scenes as two separate models Following the setup in NeRF-SOS, we divide the data of each scene into training and testing sets, en- suring there no overlap between these two sets.
Results Figure shows the complete segmentation results of SORF both Flower and Fortress datasets. The results demonstrate the potential of sVORF for 3D segmentation on non-object-centric real scenes with cluttered backgrounds. Additionally, the segmentation results in Figure 8. Unlike sVORF with ViT-Base, sVORF with ResNet34 produces coarse segmentation and still segments foreground object from complex scenes.
ResNet34 the backbone and provide
Input view
Novel views
Images
GT
Figure 7: Qualitative results of 3D segmentation on real images.
C More Ablation Studies
This section provides more ablation studies on architecture, assessing the efficacy of the transformer- based module (Section 3.2), hypernetwork (Section 3.3) and combination module (Section 3.3), respectively. The quantitative results are reported in Table 10.
Transformer-based module substitute the transformer with slot attention and observe that slot attention fails to achieve the decomposition task in our model, shown in the second row of Table 10.
16

---

Reconstructions
Sepmenstations
Figure 8: Qualitative results of sVORF with ResNet-34 on LLFF dataset.
Table 10: Ablation studies on the CLEVR-567 dataset.
Model
NV-ARI f FG-ART 1
sVORF (w/o Hypernetwork)
21.6
65.9
sVORF (w Slot-Attention)
14.1
76.8
VORF (w SM)
28.4
71,2
VVRRF (ours)
81.5
92.0
Based on this comparison, we can conclude that our transformer-based module has better scene decomposition than slot attention in our training setting.
Hypernetwork We replace the hypernetwork with directly using the slots for conditioning the radiance fields per object like uORF [8]. As shown in the first row of Table 10, the model's performance significantly decreases. We speculate that using hypernetwork can provide stronger 3D geometric bias than directly using the slots for conditioning the radiance fields per object.
Our composition module Slot Mixers To compare our composition method with Slot Mixers decoder, we have some modifications on the SM decoder [3]. First, we use a 3D point x instead of the target ray the query aggregate the weighted slot feature. Second, slot feature into the corresponding radiance field. Third, based on the radiance field, we can obtain find that the composition performance
transform the weighted
the density and color of the 3D point æ. In the experiment, of the SM method is lower than our proposed composition method. Specifically, the SM method exhibits 3D-inconsistent segmentation results, shown in the third of Table 10. proves that the introduction of 3D geometric bias is really important for scene decomposition.
D More Qualitative Results
Depth maps We illustrate the depth maps of sVORF on different datasets Figure 9, The results show that our method can learn high quality of 3D geometry.
RGB
Depth
Figure 9: Reconstructed depth maps of sVORF on different datasets.
Object-level radiance fields. We provide the visualization of learned object-level radiance fields on CLEVR-567 dataset Figure 10. further demonstrated that our method can achieve very clean scene decomposition.
17

---

Depth
Figure 10: Visualization of learned object-level radiance fields.
Grayscale images. We provide the multi-view results of the qualitative evaluation of the model on grayscale version of the CLEVR-567 dataset in Figure 11.
Gray
Figure 11: Multi-view qualitative results on a grayscale version the CLEVR-567 dataset.
E
Limitations
There are three limitations in sVORF. First, although sVORF can generalize to more objects at test time, shown in Section 4.7, the maximum slot number of test scenes is restricted by the training setting. In other words, we need know the maximum number of objects/slots in the scene, and ensure that the number of slots set is equal to exceeds this maximum value. Second, like other 3D representation learning methods, the model's training process requires curated multi-view images, which are difficult obtain and necessitate specialized equipment, particularly in real scenes. Finally, although our method performs well on some complex scenes, such as MSN, it is still a challenge for method to decompose and understand real-world scenes like human. Addressing the limitations related to the slot number, training dataset and the generalization capabilities on real scenes is potential area for future rescarch.
18


Reviews:
[{"review_id": "CYsN9uoktS", "reviewer": "Reviewer_uYMS", "summary": "This paper proposes sVORF to tackle the challenging problem of ‘unsupervised’ scene decomposition by learning a series of slot-guided neural radiance fields. Inferring object-level NeRF from a hyper-network instead of costly GRU modules, the overall pipeline can be efficiently trained with on a collection of multi-view images and produce promising segmentation results on several synthetic benchmark datasets. ", "questions": "Please see the weaknesses above.", "limitations": "Limitations have been discussed in the supplement.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\n- The overall ideas is well motivated and easy to follow, and the main pipeline could be well taken.\n\n- Extensive experiments on adequate synthetic datasets are conducted. The numerical results are promising  compared to SOTA baselines, with detailed ablations.", "weaknesses": "- Limited unique contribution compared to previous work like uORF.\n\n   I think this paper proposes a good extension on top of uORF while my main concern is that the unique contributions are somewhat limited. The overall pipeline is built on top of previous work uORF by replacing the expensive GRU modules to hyper-network inferred compositional object-level NeRFs, which I see as the main difference bringing efficiency advantages.\n\n- I do appreciate the promising performance on synthetic datasets, however I think more discussions and results towards the main statements (consistency, efficiency against current baselines) need to be further strengthened.\n    - The method of using radiance fields instead of light-fields stand out in terms of strict multi-view consistency. Therefore, discussions and analysis of multi-view consistency of segmentation masks (on synthetic and real-world cases) are expected to show to validate this point.\n\n\n- Are the adopted backbone networks (ResNet34 and ViT-Base of real-world image segmentation) pre-trained in any way or trained from scratch (L318)? As I am not sure if the used datasets are sufficient to train such large network. If so, is the method strictly unsupervised anymore? More clarifications are expected. Also, for real-world segmentation, I am wondering how does the backbone impact the performance if ResNet34 is used. \n\n- How does appearance affect the perform? Does the mode really learn to decompose the scene intrinsically or a very good ‘colour/apperance’ segmentor? I see some good evidence from fig2 but find the networks struggles to separate close-by objects with similar colour as well as shadows. Moreover, related to last point of consistency, is this ‘incorrect pattern’ also consistent across view?\n\n\n - Related to last point, visualization of learned object-level radiance fields are expected. Readers could know if the method indeed decompose the scenes into objects and background as we expect.\n\n- It would be more exciting to see how the methods work on real-world images with more than 1 foreground objects.\n\n- What is the total training time required to converge sVORF as only per-epoch time is provided.\n\n-L118. (GEU) --> (GRU)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and helpful suggestions!\n\n$\\bf{Q1}$: Limited unique contribution compared to previous work like uORF.\n\n$\\bf{A1}$:Unlike the previous work like uORF, our method has two unique contributions. $\\textbf{Firstly}$, we use a transformer-based module instead of the GRU module to extract object and background slots. This module is simple and easy to optimize without GRU block. When we substitute the transformer with slot attention, we observe that slot attention fails to achieve the decomposition task in our model, as shown in Fig 5 in the rebuttal PDF. Based on this comparison, we can conclude that our transformer-based module has a better scene decomposition than slot attention in our training setting. $\\textbf{Secondly}$, we propose a slot-guided scene composition module to recompose the slots to novel views. Compared to a conditional NeRF in uORF, this module uses a hypernetwork to transform a slot to its radiance field and utilizes explicit geometric bias to obtain density and color. This design performs better than using conditional NeRF, as shown in Table in our global response. We speculate that using hypernetwork can provide stronger 3D geometric bias than directly using the slots for conditioning the radiance fields per object. Besides, this module uses slots as a guidance to compose individual objects and background. This scheme can make slot features 3D-aware, which is useful for scene decomposition. As shown in section Composing Mechanism, this scheme outweighs the density-weighted mean used in uORF largely in FG-ARI metric.\n\n$\\bf{Q2}$: Limited discussions and results towards the main statements (consistency, efficiency against current baselines).\n\n$\\bf{A2}$: $\\textbf{Multi-view consistency}$: To our knowledge, the generation and segmentation quality of novel view can show the multi-view consistency of the proposed method.  As shown in Table 1, our method beats all baselines in terms of NV-ARI (ARI on synthesized novel views) in both Room-Chair and Room-Diverse scene. Besides, our method outperforms other baselines on novel view synthesis, as shown in Table 3.    \n$\\textbf{Efficency}$: We provide the details about the training speed and memory consumption in Appendix A.1.  \nIn addition, we provide some examples of depth maps for each dataset to show the effectiveness of our method in Fig 3 of the rebuttal PDF. \n\n$\\bf{Q3}$: Is the method strictly unsupervised anymore? How does the backbone impact the performance if ResNet34 is used in real-world cases?\n\n$\\bf{A3}$: The adopted backbone networks are trained from scratch, and the method strictly follows unsupervised learning. \nWe use ResNet34 as the backbone and provide the segmentation results in Fig 2 of the rebuttal PDF. Unlike sVORF with ViT-Base, sVORF with ResNet34 produces a coarse segmentation and still segments foreground object from complex scenes.\n\n$\\textbf{Q4}$: How does appearance affect the perform? Is the ‘incorrect pattern’ of shadow also consistent across view?\n\n$\\textbf{A4}$: To explore whether the sVORF mainly relies on RGB color for scene decomposition, we conduct an evaluation on a grayscale version of CLEVR-567 dataset. The model used in the evaluation is only trained on RGB CLEVR567 dataset. The model achieves 87.5 FG-ARI on the grayscale test set, which is on par with 92.0 FG-ARI on the default RGB images. The evaluation results demonstrate that sVORF really learns to decompose the scene intrinsically. For qualitative results , please refer to the rebuttal PDF.  \nThe ‘incorrect pattern’ of the shadow remains consistent across different views, as illustrated in Fig 5 in the rebuttal PDF.\n \n$\\textbf{Q5}$: Visualization of learned object-level radiance fields.\n\n$\\textbf{A5}$: Thank you for this suggestion! We provide the visualization of learned object-level radiance fields on CLEVR-567 dataset in Fig 4 in the rebuttal PDF. It is further demonstrated that our method can achieve very clean scene decomposition.\n \n$\\textbf{Q6}$: Experiments on real-world images with more than 1 foreground objects.\n\n$\\textbf{A6}$: To validate our method on more challenging scenarios, we conduct a preliminary experiment on the MSN dataset. The MSN dataset comprises $\\textbf{11,733 distinct shapes}$, with each scene populated by 2-4 objects sourced from the ShapeNetV2 3D model dataset. The results are shown in below:\n\n| Model | Supervision |  ARI $\\uparrow$ | Fg-ARI $\\uparrow$ | PSNR $\\uparrow$|\n| ----------- | ----------- | ------------ | ----------- | ----------- |\n| ObSuRF | image+depth  | $\\bf{64.1}$ | 81.4 | 27.41 |\n| sVORF | image |  63.4 | $\\bf{84.1}$ | $\\bf{30.51}$ |\n\nCompared with ObSuRF, sVORF achieves significantly higher Fg-ARI and comparable ARI without using depth information, which demonstrates the model's ability to decouple more complex scenarios. Please refer to Fig 1 in the rebuttal PDF for visual results. \n\n$\\textbf{Q7}$: The total training time.\n\n$\\textbf{A7}$: For the CLEVR-567 and Room-Chair datasets, sVORF is trained with a batch size of 16 for approximately 7 hours on 8 Nvidia RTX 2080ti GPUs. For the larger CLEVR3D datasets, sVORF spends approximately 2 days using a batch size of 16 on 8 Nvidia RTX V100 GPUs.\n\n$\\textbf{Q8}$: Typo in L118 \n\n$ \\textbf{A8}$: Thank you for catching that typo and we will fix it in the final version."}, {"review_id": "3PSMJPJuUx", "reviewer": "Reviewer_JyjT", "summary": "This paper introduces sVORF, a compositional method for representing scenes as a collection of slots that are parametrized as per-object radiance fields. The proposed model can be used to perform novel-view synthesis, as well as semantic segmentation in 3D. Moreover, the proposed compositional representation, also enables performing simple editing operations in the scene. In more detail, given an input image, sVORF first extracts image features from the image and passes them to a transformer encoder-decoder module that extracts slots for the objects in the scene and the background. Subsequently, these slots are mapped to volumetric object radiance fields using a hypernetwork. Using the per-object radiance fields, the scene is rendered by simply compositing their outputs at a 3D location with the guidance of each slot. Unlike [8] that was the first work that explored discovering slots in 3D data, sVORF maps slots to per-object radiance fields using a hypernetwork. The authors evaluate the performance of their model on the CLEVR-567, CLEVR-3D, Room-Chair, Room-Diverse and the LLFF datasets and compare with several compositional representations that represent slots as radiance fields [8, 35] or as light fields [9]. From their experimental results, it seems that the proposed model outperforms all baselines both on the novel view synthesis task as well as on the 3D segmentation task.\n\nOverall, I think the proposed idea for discovering slots in 3D is interesting and the authors show that their approach is working for simple scenes with a few objects. In terms of novelty, the proposed pipeline is quite similar to [8], with the main difference being the use of the hypernetwork for mapping slots to per-object radiance fields. Looking at the quantitative results, it seems that the proposed model is better than the baselines on most tasks, thus I am leaning towards accepting the paper but I am still a bit concerned by the fact that the proposed model and [8] are relatively similar and that the proposed model is only evaluated on extremely simple scenes.", "questions": "1. Unless I am missing out something, I think that the proposed model assumes that the number of objects/slots is known right? In other words, although we might not know what are the objects, we know how many objects/slots exist in the scene, right? This is never clearly mentioned in the text. I think the authors should clarify it.\n\n2. I think that the per-object NeRFs should be locally defined, namely each NeRF is augmented with a local affine transformation defining its pose in 3D. Otherwise, it would not be possible to change the location of a specific object/slot to a new position. Is this really the case?\n\n3. I think that from the provided information for the composing mechanism in L146-150 it is not 100% clear how are the aggregation $D^c$ and the attention bloc $D^a$ really implemented. One thing that is not very clear to me is why do we need both? Can the authors please explain?\n\n4. In L264-272, the authors present an ablation study that tries to investigate the impact of the Novel View Synthesis setup. To be honest, I am not 100% sure that I understand what the authors mean by \"we modify our reconstructed target view to equal with the input view\". Can the authors please clarify?\n\n5. I am wondering what is the quality of the 3D geometry of the proposed model and how does it compare to baselines. Since the authors represent the scene using a compositional NeRF-based representation I think it would be useful to also show some depth maps at least in the supplementary.", "limitations": "The authors discuss the limitations of their work in their supplementary material but I was not able to find any discussion about the potential negative societal impact of their work. That being said, I think that for this paper this discussion is not 100% necessary.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors conduct multiple experiments and compare their model with several baselines on multiple datasets. From the quantitative evaluation, we note that the proposed model outperforms all baselines on most tasks. I particularly liked the scene editing experiment in Sec. 4.3.\n\n2. Although some things are not 100% clear I think the paper is easy to read and relatively nicely written. I really liked Figure 1 that is a pictorial representation of the overall pipeline.", "weaknesses": "1. One of the major weaknesses of this work is that the authors only demonstrate the performance of their model on very simple datasets that consist of few objects that in most cases belong to the same object class e.g. chairs. Although I am aware that all baselines evaluate their models on similar setups I think it would have been great if the authors could show that their model works on more challenging scenarios, with different objects and more diverse backgrounds. A rather simple indoor dataset that the authors could consider is the 3D-FRONT dataset. Another less exciting alternative would be to also show results on the City-Block dataset introduced in [9]. However, since also this dataset contains scenes with cars, I think that the 3D-FRONT would be a more appropriate benchmark.\n\n2. In Sec. 3.2 and in L116-117 the authors state that the adopt an efficient transformer module to infer the object and the background slots form the image features. Can the authors describe how this efficient transformer module compares to the classical slot attention module used in [8]? Is the efficient transformer module a standard transformer encoder/decoder? I think that to make things very clear, it would have been very useful to properly define (e.g. using a math expression) how is the cross-attention defined? In addition, can the authors also clarify why/how is this model efficient?\n\n3. Although, the authors provide several ablation studies, I believe that they should have ablated the impact of using a hypernetwork as opposed to directly using the slots for conditioning the per-object radiance fields. This is more similar to [8] and it could better explain why is the proposed model outperforming [8], although their quite similar.\n\n4. I am bit concerned regarding the limited technical contribution of this work. Unless I am missing out something I think the proposed paper is quite similar to [8]. That being said, the proposed method seems to be significantly better than [8]. This could be either due to the use of the hypernetwork or the use of the attention module for inferring the slots. I believe that it is very important to clarify what are the differences and also try to justify the improvement in performance though ablations e.g. ablate the use of a hypernetwork for mapping slots to per-object radiance fields.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and helpful suggestions!\n\n$\\textbf{Q1:}$ Experiments on more challenging scenarios.\n\n$\\textbf{A1:}$ To validate our method on more challenging scenarios, we conduct a preliminary experiment on the MSN dataset. The MSN dataset comprises $\\textbf{11,733 distinct shapes}$, with each scene populated by 2-4 objects sourced from the ShapeNetV2 3D model dataset. The results are shown in below:\n\n| Model | Supervision |  ARI $\\uparrow$ | Fg-ARI $\\uparrow$ | PSNR $\\uparrow$|\n| ----------- | ----------- | ------------ | ----------- | ----------- |\n| ObSuRF | image+depth  | $\\bf{64.1}$ | 81.4 | 27.41 |\n| sVORF | image |  63.4 | $\\bf{84.1}$ | $\\bf{30.51}$ |\n\nCompared with ObSuRF, sVORF achieves significantly higher Fg-ARI and comparable ARI without using depth information, which demonstrates the model's ability to decouple more complex scenarios. Please refer to Fig 1 in the rebuttal PDF for visual results. \n\n$\\textbf{Q2:}$ Details of the efficient transformer module.\n\n$\\textbf{A2:}$ The efficient transformer module is a standard transformer decoder. Specifically, we firstly use each slot $\\textbf{z}_i$ as query and interact with other object slots with multi-headed self-attention operation. Then we employ multi-headed cross-attention operation to attend into and aggregate features from the flattend image features $E(\\textbf{I})$. Finally, we pass the resulting slot features into a feed forward network (FFN) to get the final slots.  \nAs mentioned in the main paper, this transformer module is simple and easy to train than the slot attention module, as it does not contain a Gated Recurrent Unit (GRU) block. Additionally, when we replace the transformer with slot attention, we observe that slot attention fails to achieve the decomposition task, see Fig 5 in the rebuttal. Based on this comparison, we can conclude that our transformer-based module has a better scene decomposition than slot attention in our training setting. \n  \n$\\textbf{Q3:}$ The impact of using a hypernetwork.\n\n$\\textbf{A3:}$ To validate the effectiveness of the hypernetwork, we conduct an ablation study, which replaces the hypernetwork with directly using the slots for conditioning the radiance fields per object. As shown in Table in our global response and Fig 5 in the rebuttal PDF, the model’s performance significantly decreases. We speculate that using hypernetwork can provide stronger 3D geometric bias than directly using the slots for conditioning the radiance fields per object. This 3D-aware slot facilitates guiding the composition of all volumetric neural radiation fields.\n\n$\\textbf{Q4:}$ The limited technical contribution\n\nUnlike the previous work like uORF, our method has two unique contributions. $\\textbf{Firstly}$, we use a transformer-based module instead of the GRU module to extract object and background slots. This module is simple and easy to optimize without GRU block. When we substitute the transformer with slot attention, we observe that slot attention fails to achieve the decomposition task in our model, as shown in Fig 5 in the rebuttal PDF. Based on this comparison, we can conclude that our transformer-based module has a better scene decomposition than slot attention in our training setting. $\\textbf{Secondly}$, we propose a slot-guided scene composition module to recompose the slots to novel views. Compared to a conditional NeRF in uORF, this module uses a hypernetwork to transform a slot to its radiance field and utilizes explicit geometric bias to obtain density and color. This design performs better than using conditional NeRF, as shown in Table in our global response. We speculate that using hypernetwork can provide stronger 3D geometric bias than directly using the slots for conditioning the radiance fields per object. Besides, this module uses slots as a guidance to compose individual objects and background. This scheme can make slot features 3D-aware, which is useful for scene decomposition. As shown in section Composing Mechanism, this scheme outweighs the density-weighted mean used in uORF largely in FG-ARI metric.\n\n$\\textbf{Q5:}$ Is the number of slots known?\n\n$\\textbf{A5:}$ We know the maximum number of objects/slots in the scene, and ensure that the number of slots set is equal to or exceeds this maximum value.\n  \n$\\textbf{Q6:}$ I think that each NeRF is augmented with a local affine transformation defining its pose in 3D. Otherwise, it would not be possible to change the location of a specific slot to a new position.  \n\n$\\textbf{A6:}$ Yes, as you mentioned, in order to relocate the slot, an affine transformation is applied to the 3D sample points before passing them to the corresponding object NeRF.\n\n$\\textbf{Q7:}$ For composition mechanism, why both aggregation and attention modules are needed?\n\n$\\textbf{A7:}$ The aggregation block performs a cross-attention operation, which aggregates object representations $S$ with the 3D location $x$ as the query to obtain the corresponding feature $z$. The attention block computes the similarity between $S$ and $z$ after mapping them into the same space through a linear layer, thus obtaining the probability distribution of $x$ belonging to each slot. In the initial stages of our experiments, we observed that utilizing both modules simultaneously yielded superior results. We speculate that this improvement could be attributed to the increased difficulty of directly computing the similarity between the 3D points and the slot feature when they are not spatially aligned using $D^a$.\n\n$\\textbf{Q8:}$ Clarifing the meaning of \"we modify our reconstructed target view to equal with the input view\" in L264.\n\n$\\textbf{A8:}$ It means that we turns sVORF into a 2D image auto-encoder.\n\n$\\textbf{Q9:}$ The quality of the 3D geometry of the proposed model.\n\n$\\textbf{A9:}$ As shown in the rebuttal PDF Figure 3, we illustrate the depth maps of sVORF on different datasets. The results show that our method can learn a high quality of the 3D geometry."}, {"review_id": "QGLnHvGbS8", "reviewer": "Reviewer_3Ksj", "summary": "This paper presents a method called sVORF for learning 3D object centric representations. A transformer decompose a single input image into slots which are then each mapped to volumentric radiance fields by a hypernetwork. These object-radiance fields are then composed into a 3D scene. The effectiveness of sVORF is demonstrated on a set of simple synthetic datasets and two real world scenes. It achieves SOTA or at least competitive results on CLEVR567, CLEVR3D, Room-Chair, and Room-diverse datasets, compared to SlotAttention, uORF and COLF.", "questions": "- How many samples are used per scene during training. Appendix A mentions 64 rays with 64 coordinates in the coarse volume and 128 additional coordinates in the fine volume. What are the coarse and the fine volumes? \n- For how long (number of updates / epochs) was sVORF trained? How does that compare to uORF, COLF, OSRT?\n- The LLFF results use only two scenes. Does that mean the shown results are training images? Is sVORF trained on both scenes at once or as two separate models. In the latter setting it seems to me that sVORF is essentially equivalent to a simple volumetric NeRF, since the model could in principle learn to produce the correct target views during inference without even looking at the input view. Is there any evidence for generalization in this setting?", "limitations": "There is a brief discussion of limitations in the supplementary that touches on two important limitations of the method. \nIt does however not discuss any concerns regarding generalization to more complex real-world scenes.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The problem of unsupervised 3d object decomposition is quite challenging. The fact that sVORF performs so well with only a single input image, and no depth or segmentation supervision is remarkable.\n* The writing is clear and easy to follow. The figures look nice, are helpful and are easy to understand. \n* The individual parts of the method are well motivated and their individual contributions are established through systematic ablations.", "weaknesses": "* The paper stresses the low computational costs and memory footprint of sVORF, but fails to report any data on them in the main paper. Memory consumption is mentioned in the appendix, alongside training time per epoch. But information about the length of training is lacking so the real computational costs remain unclear. \n\n* The studied scenes are visually simple (yes even Room-diverse which is advertised as complex). The only exception are the two scenes from LLFF. It is encouraging that sVORF succeeds on these two scenes, but in this setting it appears to me that the model is effectively overfitting on the given scene, and there is no evidence for generalization to novel scenes in that setting. \n\n* The restriction to a single input image is rather limiting. The 3D structure of a general scene is highly underspecified from a single image, which should lead to large uncertainties (e.g. in occluded areas). For simple scenes like CLEVR and Room-Chair this is not a problem because none of the elements is ever fully occluded, and they come from a small set of known objects. But I expect this to become a severe limitation for scaling to more complex scenes. Especially since sVORF and its training objective are not designed to be probabilistic / generative, and its fidelity will thus likely suffer substantially from complex ambiguities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and helpful suggestions!\n\n$\\textbf{Q1:}$ Information about the length of training. How does that compare to uORF, COLF, OSRT?\n\n$\\textbf{A1:}$ For the CLEVR-567 and Room-Chair datasets, we train sVORF for approximately 7 hours using 8 Nvidia RTX 2080 Ti GPUs with batch size 16. The uORF and COLF models are trained on an Nvidia RTX V100 GPU for approximately 7 and 2 days, respectively, with a batch size of 1.\nFor the larger CLEVR3D datasets, sVORF is trained for approximately 2 days using 8 Nvidia RTX V100 GPUs with batch size 16, while OSRT is trained for approximately 1 day on 8 A100 GPUs with a batch size of 256.\n\n$\\textbf{Q2:}$ The studied scenes are visually simple.\n\n$ \\textbf{A2:}$ To validate our method on more challenging scenarios, we conduct a preliminary experiment on the MSN dataset. The MSN dataset comprises $\\textbf{11,733 distinct shapes}$, with each scene populated by 2-4 objects sourced from the ShapeNetV2 3D model dataset. The results are shown in below:\n\n| Model | Supervision |  ARI $\\uparrow$ | Fg-ARI $\\uparrow$ | PSNR $\\uparrow$|\n| ----------- | ----------- | ------------ | ----------- | ----------- |\n| ObSuRF | image+depth  | $\\bf{64.1}$ | 81.4 | 27.41 |\n| sVORF | image |  63.4 | $\\bf{84.1}$ | $\\bf{30.51}$ |\n\nCompared with ObSuRF, sVORF achieves significantly higher Fg-ARI and comparable ARI without using depth information, which demonstrates the model's ability to decouple more complex scenarios. Please refer to Fig 1 in the rebuttal PDF for visual results. \n\n$\\textbf{Q3:}$ The LLFF results use only two scenes. Does that mean the shown results are training images? On real-world datasets, there is no evidence that sVORF generalizes to new scenarios.\n\n$\\textbf{A3:}$ We train sVORF on both LLFF scenes as two separate models, but the results presented do not include the training images. Following the setup in NeRF-SOS[1], we divide the data of each scene into training and testing sets, ensuring there is no overlap between these two sets. \nOur experiment on LLFF aims to demonstrate the capability of sVORF to segment objects in real scenarios. For a simple volumetric NeRF, it can not segment objects in this setting like sVORF. Due to the limited availability of multi-view data from real scenes, it is challenging to show generalization of our method in this setting. However, it is an interesting topic and we plan to investigate it in future research.\n\n$\\textbf{Q4:}$ The restriction to a single input image is rather limiting. \n\n$ \\textbf{A4:}$ We acknowledge that the restriction to a single input image is rather limiting, particularly when dealing with more complex scenes. Incorporating multi-view images as inference may alleviate this problem. We consider this as a potential avenue for future research.\n\n$ \\textbf{Q5:}$ For how long (number of updates / epochs) was sVORF trained? How does that compare to uORF, COLF, OSRT?}\n\n$ \\textbf{A5:}$ Please refer to the comments in A1.\n\n$ \\textbf{Q6:}$ How many samples are used per scene during training?\n\n$ \\textbf{A6:}$ We follow the hierarchical sampling strategy in NeRF[2]. Specifically, we sample 64 points per ray through the coarse network and 64 + 64 = 128 points per ray through the fine network. The \"coarse\" and \"fine volume\" refers to the “coarse” and “fine” network.\n\n[1] Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any-view self-supervised object segmentation on complex scenes. arXiv preprint arXiv:2209.08776, 2022.  \n[2] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021."}, {"review_id": "Lo9Lfnb774", "reviewer": "Reviewer_thZt", "summary": "This paper proposes a method, sVORF, for unsupervised object-centric 3D representation learning. Given a single image as input, it decomposes the scenes into individual 3D objects and 3D backgrounds which thus support object segmentation, novel view synthesis (named scene generation in the paper), and scene editing (including object moving and background repainting). \nThe key idea of the paper is to combine object slot with object radiance fields together in a framework, in which the object slot is responsible for decomposing the scene into objects. In contrast, the object radiance fields are used to re-compose the scene for volumetric rendering. The only cue for training the network is the rendering loss between the re-composed and volumetric rendered image with the input single-view image. The overall performance of sVORF is plausible and thorough experiments and ablation studies well validated the effectiveness of the proposed method. ", "questions": "- Line 224, please provide the full name of FFN since this is the first time you use FFN. \n- In Table 2(a), it seems ObSURF works generally better on 3D segmentation, does this mainly benefit from the depth input? \n- Line 149, a->an", "limitations": "Yes. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+The task of reconstructing each object and the background in 3D given only a single RGB image is very challenging and with great importance for practical applications in robotics and CHI, etc. \n+The overall performance of the proposed method is strong when compared with existing approaches. \n+The idea of using the attention mechanism for scene decomposition and re-composition as well as the use of HyberNetwork for object-level NeRF generation is sound and practical. \n+The demonstration of segmentation on real captured dataset LLFF is important to help the readers to qualify the performance of the method. \n+The paper is well written and easy to follow. ", "weaknesses": "- The dataset used for validating the proposed method is quite simple, with obvious color or shape differences between objects and simple backgrounds, which is very convenient for segmentation and decomposition. In the LLFF Dataset, the segmented foreground color is also very different from the background. So maybe more complex scenes should be used to demonstrate the effectiveness of the proposed method. \n- The HyperNetwork produces MLP parameters of NeRF. However, the MLP representation of NeRF is very ambiguous, so learning such a hyberNetwork for NeRFs may not be very robust, please explain how to deal with this problem, especially for real-world scenes with complex objects in it. \n- In Fig.5, only segmentation results are provided on the LLFF dataset, is it possible to also provide the NVS results? why not? \n- The main paper should clarify the training strategies and key implementation details like how to set K for each dataset. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and helpful suggestions!\n\n$\\textbf{Q1:}$ Demonstrate the effectiveness of the proposed method on more complex scenarios.\n\n$\\textbf{A1:}$ To validate our method on more challenging scenarios, we conduct a preliminary experiment on the MSN dataset. The MSN dataset comprises $\\textbf{11,733 distinct shapes}$, with each scene populated by 2-4 objects sourced from the ShapeNetV2 3D model dataset. The results are shown in below:\n\n| Model | Supervision |  ARI $\\uparrow$ | Fg-ARI $\\uparrow$ | PSNR $\\uparrow$|\n| ----------- | ----------- | ------------ | ----------- | ----------- |\n| ObSuRF | image+depth  | $\\bf{64.1}$ | 81.4 | 27.41 |\n| sVORF | image |  63.4 | $\\bf{84.1}$ | $\\bf{30.51}$ |\n\nCompared with ObSuRF, sVORF achieves significantly higher Fg-ARI and comparable ARI without using depth information, which demonstrates the model's ability to decouple more complex scenarios. Please refer to Fig 1 in the rebuttal PDF for visual results.\n\n$\\textbf{Q2:}$ The robustness of hypernetwork.\n\n$\\textbf{A2:}$ To our knowledge, Shap$\\cdot$E[1] also learns a hypernetwork to produce MLP parameters. Shap$\\cdot$E can handle a large complex and diverse 3D assets, although it does not involve complex background. It is an indicator that a hypernetwork for NeRF may be enough for real-world scenes with complex objects in it. For the success of Shap$\\cdot$E, we speculate that a large dataset may be the key factor. Thus, to deal with the robustness of a hypernetwork, it may be an alternative to train with a large dataset. Certainly, we acknowledge the robustness problem of the hypernetwork. It is a future work to investigate and solve this problem.\n\n$\\textbf{Q3:}$ Provide the NVS results on the LLFF dataset.\n\n$\\textbf{A3:}$ As a result of limited page space, we provide the novel view synthesis results of LLFF in Appendix A.2. These results showcase the multi-view consistency of sVORF on the LLFF dataset. If possible, we will include these results in the main paper.\n\n$\\textbf{Q4:}$ Clarify the training strategies and key implementation details like how to set K for each dataset.\n\n$\\textbf{A4:}$ The implementation details are provided in Appendix. Furthermore, we employ additional training strategies as outlined below. We utilize the Adam optimizer with a learning rate of 0.0001, $\\beta_1$ = 0.9, and $\\beta_2$ = 0.999. Additionally, we implement learning rate warm-up for the initial 1,000 iterations. The value of K (set based on the number of objects in the scene) was customized separately as follows: K = 8, 7, 5, 5, 2, and 5 for the CLEVR-567, CLEVR-3D, Room-Chair, Room-Diverse, LLFF, and MSN datasets, respectively. For the CLEVR-567 and Room-Chair datasets, sVORF is trained with a batch size of 16  for approximately 7 hours on 8 Nvidia RTX 2080ti GPUs. For the larger CLEVR3D datasets, sVORF spends approximately 2 days using a batch size of 16 on 8 Nvidia RTX V100 GPUs.\n\n$\\textbf{Q5:}$ Provide the full name of FFN.\n\n$\\textbf{A5:}$ Thank you for catching that typo. The full name of FFN is feed-forward network.\n  \n$ \\textbf{Q6:}$ Influence of depth information on ObSuRF.\n\n$ \\textbf{A6:}$ Regarding image generation, ObSuRF points out that incorporating additional depth supervision can significantly reduce reconstruction errors. As for decomposition, although ObSuRF does not provide ablation experiments about the depth map, it is obvious that depth information plays a crucial role in separating the foreground and background, particularly for shadows and foreground objects.\n\n$ \\textbf{Q7:}$ Typo in L149\n\n$\\textbf{A7:}$ Thank you for catching that typo and we will fix it in final version.\n\n[1] Jun, Heewoo, and Alex Nichol. \"Shap-e: Generating conditional 3d implicit functions.\" arXiv preprint arXiv:2305.02463 (2023).\n"}, {"review_id": "aqFL7rtQ8n", "reviewer": "Reviewer_6zB4", "summary": "This paper presents slot-guided volumetric object radiance fields (sVORFa), a learning method for 3D object-centric representation. They proposed three key techniques, scene decomposition based on global pooling and slot-based representations, hyper-network to generate per-object radiance fields, and scene composition module. They have tested the method on various tasks on synthetic datasets, such as CLEVR and Room, showing superior performance to prior arts. Finally, they provided the simple experiment results on real images (LLFF), highlighting the broader effectiveness of the methods.", "questions": "1. In Table 2, why did you have to use different settings and re-evaluate the previous methods?\n\nOther questions are embedded in the weaknesses section.", "limitations": "I do not see any potential negative societal impact from this work.", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method improved the prior methods by a large margin, especially on Room-Chair and Room-Diverse datasets (segmentation task)\n2. The newly suggested modules seem very efficient. and I think the global pooling features combined with the slot-guided method is a reasonable and better approach than the previous slot-attention method, at least in these particular tasks.\n", "weaknesses": "1. From my understanding, three modules were newly adopted or introduced to object-centric representation learning tasks. They are 1. ‘global pooling’ with slot-guided scene decomposition module, 2. hypernetwork, and 3. scene composition modules. However, throughout the experiments, I do not see which components are actually effective compared to prior methods. For example, they could have introduced ‘hypernetwork’ to uORF, or they could have tested ‘slot-attention’ in the proposed sVORF to see the effectiveness of the 'hypernetwork' or ‘global-pooling’ slot-guide method. Hence, it is hard for the readers to know every component actually contributes to the final performance.\n2. It seems connectivity regularization (CR) is very important, I would like to see how much CR can improve the performance of the previous methods, such as uORF and others.\n3. Similarly, I believe we can easily incorporate spatial broadcast (SB) or slot mixers (SM) in the proposed method sVORF, which could clearly show the effectiveness of the proposed scene composition module.\n4. Although the newly introduced modules are well orchestrated into a single model and improve performance, I do not see significant technical novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and helpful suggestions!\n\n$\\textbf{Q1:}$ Need ablation studies on main components.\n\n$ \\textbf{A1:}$ We conduct some experiments to compare our proposed components with previous methods. $\\textbf{Firstly}$, we show the effectiveness of the transformer-based module. We substitute the transformer with slot attention and we observe that slot attention fails to achieve the decomposition task in our model, as shown in Fig 5 in the rebuttal PDF. Based on this comparison, we can conclude that our transformer-based module has a better scene decomposition than slot attention in our training setting.\n$\\textbf{Secondly}$, we validate our proposed slot-guided scene composition module. We replace the hypernetwork with the conditional NeRF used in uORF. As shown in Table of our global response, using a hypernetwork performs better than using conditional NeRF. We speculate that using hypernetwork can provide stronger 3D geometric bias than directly using the slots for conditioning the radiance fields per object. Besides, we demonstrate the efficacy of using slots as a guidance to compose individual objects and background. As shown in section Composing Mechanism, this scheme outweighs the density-weighted mean used in uORF largely in FG-ARI metric. This comparison proves that slot-guided composition can make slot features 3D-aware, which is useful for scene decomposition. \n\n$\\textbf{Q2:}$ How much CR can improve the performance of the previous methods, such as uORF and others.\n\n$\\textbf{A2:}$ The CR module is not the key to sVORF decoupling. It serves the purpose of mitigating the presence of semi-transparent clouds when the number of slots significantly exceeds the total number of objects in a scene, as discussed in section 3.4 of main paper. We also give an analysis of how this module affects the overall performance of the model in section 4.4. Based on the analysis, it is not one of the key components. Considering that the occurrence of semi-transparent clouds is infrequent in uORF, it is not necessary to provide experimental results of CR module on uORF.\n\n$\\textbf{Q3:}$ Incorporate Spatial Broadcast (SB) or Slot Mixers (SM) in the proposed method sVORF.\n \n$\\textbf{A3:}$ For SB method, we present the ablation results in the Composing Mechanism section of the main paper. The results show that our composing mechanism outperforms the density-weighted mean combination mode (SB), especially for small objects that may otherwise be segmented into attachments of other objects.  \nAs for the SM method, we have some modifications on the SM decoder[1]. First, we use a 3D point $x$ instead of the target ray as the query to aggregate the weighted slot feature. Second, we transform the weighted slot feature into the corresponding radiance field. Third, based on the radiance field, we can obtain the density and color of the 3D point $x$. In the experiment, we find that the composition performance of the SM method is lower than our proposed composition method. Specifically, the SM method exhibits 3D-inconsistent segmentation results, as shown in Fig 5 in the rebuttal PDF. It proves that the introduction of 3D geometric bias is really important for scene decomposition.\n\n$ \\textbf{Q4:}$ Limited significant technical novelty.\n\n$ \\textbf{A4:}$ Unlike the previous work like uORF, our method has two unique contributions. $\\textbf{Firstly}$, we use a transformer-based module instead of the GRU module to extract object and background slots. This module is simple and easy to optimize without GRU block. When we substitute the transformer with slot attention, we observe that slot attention fails to achieve the decomposition task in our model, as shown in Fig 5 in the rebuttal PDF. Based on this comparison, we can conclude that our transformer-based module has a better scene decomposition than slot attention in our training setting. $\\textbf{Secondly}$, we propose a slot-guided scene composition module to recompose the slots to novel views. Compared to a conditional NeRF in uORF, this module uses a hypernetwork to transform a slot to its radiance field and utilizes explicit geometric bias to obtain density and color. This design performs better than using conditional NeRF, as shown in Table in our global response. We speculate that using hypernetwork can provide stronger 3D geometric bias than directly using the slots for conditioning the radiance fields per object. Besides, this module uses slots as a guidance to compose individual objects and background. This scheme can make slot features 3D-aware, which is useful for scene decomposition. As shown in section Composing Mechanism, this scheme outweighs the density-weighted mean used in uORF largely in FG-ARI metric.\n\n$\\textbf{Q5:}$ In Table 2, why did you have to use different settings and re-evaluate the previous methods?\n\n$\\textbf{A5:}$ In the CLEVR-3D dataset, ObsSuRF and OSRT employ distinct training and test data divisions, as well as different test metrics. To ensure a fair comparison, we use the settings of ObsSuRF as default and re-evaluated OSRT accordingly.\n\n[1] Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. Advances in Neural Information Processing Systems, 35:9512–9524, 2022."}]

Author rebuttal:
We thank the reviewers for their helpful comments, according to the feedback and suggestions, we mainly added the following experiments:
- [uYMS, JyjT, 3Ksj, thZt] We conduct experiments on more challenging  MultiShapenet (MSN) dataset. The results demonstrate our model’s ability to decouple more complex scenarios.
- [uYMS, JyjT, 6zB4] We replace the hypernetwork with directly using the slots for conditioning the radiance fields per object, and observe that using hypernetwork can provide stronger 3D geometric bias than conditioning NeRF.
- [uYMS] We train sVORF with ResNet34 backbone on LLFF dataset, which produces a coarse segmentation and still segments foreground object from complex scenes.
- [JyjT, 6zB4] We replace the transformer with slot attention, and observe that our transformer decomposes scene better than slot attention.
- [6zB4] We incorporate Slot Mixers (SM) in our sVORF. The results prove that the introduction of 3D geometric bias in our slot-guided composition method is really important for scene decomposition.
- [uYMS] We discuss testing results on unseen grayscale version of CLEVR-567 dataset, indicating that our model really learns to decompose the scene intrinsically.
- [uYMS, JyjT] We provide some examples of depth maps for each dataset to show the geometry of our method.
- [uYMS] We give the visualization of learned object-level radiance field.

The results of the ablation experiments are recorded in the table below

| Model | NV-ARI$\\uparrow$  | FG-ARI $\\uparrow$ | 
| --- | ----------- | ----------- |
| sVORF~(w/o Hypernetwork) | 21.6 | 65.9 |
| sVORF~(w Slot-Attention) | 14.1 | 76.8 |
| sVORF~(w SM) | 28.4 | 71.2 |
| sVORF~(ours) | $\bf{81.5}$ | $\bf{92.0}$ |



Meta-review: This paper introduces Slot-guided Volumetric Object Radiance Fields (sVORF), a method that combines object-centric neural radiance fields with an attention-based encoder and hypernetworks to decompose an input image into individual object representations and render novel views of the underlying scene. The authors demonstrate that sVORF achieves competitive performance compared to established baselines in this area such as uORF or OSRT.

The paper is overall of high quality and addresses a challenging and interesting problem (3D scene decomposition) which is of relevance to the NeurIPS community. The reviewers appreciated that doing well on joint novel-view synthesis and (unsupervised) scene decomposition is very challenging, especially when doing so from only a single input view without using depth as auxiliary signal. sVORF shows clear advantages over prior work both in terms of decomposition / novel view synthesis (while not requiring depth or multi-view signals) as well as in terms of inference efficiency. 

A primary concern by some of the reviewers is related to the technical novelty of the method: sVORF can be viewed as a combination of existing techniques / established principles (hypernetworks, transformers, and object-centric radiance fields), which might appear as not sufficiently novel for a conference like NeurIPS. In my view, the novelty of this work goes beyond the technical novelty of the individual techniques used in the method: as highlighted by the authors in the rebuttal, the finding that a standard transformer encoder suffices to learn object-decomposed representations in this setting without supervision and without the use of a specialized module like Slot Attention is remarkable and surely of interest for the community. The use of hypernetworks in this setting is also novel -- the authors clearly demonstrate that this architectural choice results in several quantitative benefits compared to prior work. Finally, as highlighted by reviewer 3Ksj, the problem of unsupervised 3D scene decomposition is very challenging (while being a problem of significant interest for the NeurIPS community), and progress / new findings in this area should be judged accordingly.

Altogether, I believe that this paper will be a valuable addition to the NeurIPS conference program, even though the method itself largely relies on a combination of established architectural components. I would like to ask the authors to include the modifications recommended by the reviewers when preparing the camera-ready version of the paper.
Decision: Accept (poster)