SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Furthermore, to validate our method on more challenging scenarios, we conduct experiments on the MSN dataset and show the results in Table 4. Compared with ObSuRF, sVORF achieves significantly higher Fg-ARI and comparable ARI without using depth information, which demonstrates the model's ability to decouple more complex scenarios. Qualitative results are shown in Figure 2.
Figure 2: Qualitative results of sVORF on MSN.
Table 4: Comparison on MSN dataset.
Model	ARI*↑	Fg-ARI*↑	PSNR↑
ObSuRF [35]	64.1	81.4	27.41
sVORF	63.4	84.1	30.51
4.2 Novel View Synthesis
Setup For each test scene, we reserve one view as an input and use the other views to evaluate the quality of reconstruction.
Results In Table 3, sVORF outperforms all baselines on most metrics, despite not employing the coarse-to-fine training schedule. These findings suggest that our method is effective in producing high-quality images. The slightly lower LPIPS performance on Room-Chair and Room-Diverse can be attributed to the lack of perceptual loss in the sVORF training process, which does not explicitly optimize the LPIPS performance. Moreover, there exists a trade-off between structural and perceptual loss within the model. The excellent SSIM performance of our model indicates that the sVORF can generate complex object shapes. Specifically, as illustrated in Figure 3, our method generated diverse chair shapes with higher accuracy than COLF and uORF, even though it did not fully recover the background texture. In addition, the results presented in Table 2b indicate that sVORF outperforms OSRT in all metrics, despite using nearly half of the parameters as OSRT (48.73M v.s. 83.11M). Without the addition of depth information, our method synthesizes multi-view images with higher quality than obsurf, as shown in Table 4.
4.3 Scene Design and Editing in 3D
Setup We examine the potential of our method for basic scene editing on the Room-Chair dataset. Following uORF, our investigation involves two types of modifications, namely, moving objects and changing backgrounds. For editing the position of a foreground object, we move all query point coordinates on its object NeRF, based on the targeted movement. In order to relocate the slot, an affine transformation is applied to the 3D sample points before passing them to the corresponding object NeRF.
Figure 4: 3D scene manipulation for moving object and changing background.
Meanwhile, for changing the background, we substitute the original background texture by replacing the original background slot feature with that of the target image.
Figure 3: Qualitative Comparison. We compare the reconstructions of the input view, a novel view, as well as the novel view segmentation using the uORF [8], COLF [9], OSRT [3], and our method on four datasets. Our method produces a finer segmentation and more precise shapes.
Results Figure 4 shows that edited images maintain a harmonious quality while accurately executing object movement and background replacement, affirming the strong correlation between slots and 3D objects in sVORF, as well as the accuracy of our model in 3D scene segmentation.
In this section, we conduct ablation studies on the CLEVR-567 dataset to gain a deeper understanding of how the various parts contribute to the overall effectiveness of our approach. More ablation studies on architectures are detailed in Appendix C.
Table 5: Ablation studies on the CLEVR-567 dataset.
Model	NV-ARI ↑	FG-ARI ↑
sVORF (w/o NVS)	15.1	31.2
sVORF (w/o CR)	81.1	89.3
sVORF (density-weighted)	81.4	88.6
sVORF (w/o SA)	79.6	85.4
sVORF (ours)	81.5	92.0
Role of Novel View Synthesis To investigate the influence of the novel view synthesis setup on the training process, we modify our reconstructed target view to equal with the input view, i.e., we turns sVORF into a 2D image auto-encoder. As shown in Figure 5, the model divides images based on areas rather than objects. This division led to substantially lower ARI results, presented in Table 5. Above observations confirm the crucial role of viewpoint changes in scene decomposition. Specifically, the mapping between slots and 3D representations captures changes occurring in various regions due to differences in viewing angles between the target and source views. Changes that occur within regions belonging to the same object are more closely approximated, which allows the model to converge to K slots based on object clustering.
Connectivity Regularization As previously mentioned, novel view synthesis setup ensures distinctiveness between objects, while implementing Connectivity Regularization guarantees connectivity within the same objects. These two factors work together to achieve a clear decomposition of the scene. Figure 5 visualizes that our Connectivity Regularization ensures that points on an object belong to the same slot by preventing semi-transparent clouds from being modeled by object NeRFs.
Numerically, as shown in Table 5, the model's FG-ARI displays a significant increase from 89.3 to 92.0 when implementing connectivity regularization compared to the model without it.
Composing Mechanism As described in Section 3.3, our composing mechanism can perform well on small objects with few rays. To verify this, we implement the common density-weighted mean combination mode on our model and present the visualization in Figure 5. Our composing mechanism outperforms the density-weighted mean combination mode in decoupling effect, especially for small objects that may otherwise be segmented into attachments of other objects.
Self-Attention in Scene Decomposition If an object in the input image is occluded, it may be mistakenly segmented into the slot of another object that corresponds to the occluded area. This issue can be resolved by leveraging the self-attention layer in the decomposition module as it facilitates the interaction between slots. As illustrated in the Figure 5, removing the self-attention layer leads to incorrect division of the occluded object, highlighting the importance of inter-slot interaction in preventing the collapse of the corresponding slot of the occluded object.
Figure 5: Qualitative comparison of ablation studies on CLEVR-567 dataset. Specifically, we evaluate the impact of four techniques - Novel View Synthesis (NVS), Connectivity Regularization (CR), Composing Mechanism, and Self-Attention (SA).
4.5 Generalization
We conduct three sets of generalization experiments. The three subsets assess the ability of sVORF to generalize to unseen object appearances, unfamiliar spatial arrangements and grayscale images respectively.
Object Appearance For unseen object appearance, we follow the dataset in uORF, which is similar to CLEVR-567. The training set of this dataset excludes red cylinders and blue spheres, while the testing set includes only these types of objects. The results are presented in Table 6. It is noteworthy that our approach has never encountered the appearance of test objects but achieves comparable results to the models trained on a standard CLEVR-567 dataset.
Table 6: Object Appearance
Model	ARI ↑	Fg-ARI↑
Slot-Attention [43]	2.2	N/A
uORF [8]	85.5	N/A
sVORF	82.0	92.6
sVORF(567)	83.9	95.8
Spatial Arrangements To assess the model's ability to generalize to a greater number of objects with unseen, challenging arrangements, we train our method using 11 slots on the CLEVR-567 dataset and evaluate on the packed-CLEVR-11 dataset provided by uORF. During training, we randomly mask off four slots to prevent slot collapse, but we utilize all slots during testing. The
performance of our method on the unseen object arrangements test set is shown in Table 7 and is found to be reasonably well.
Table 7: Spatial Arrangements
Model	ARI↑	Fg-ARI↑
Slot-Attention [43]	5.7	N/A
uORF [8]	83.2	N/A
sVORF	81.0	85.5
RGB color To explore whether the sVORF mainly relies on RGB color for scene decomposition, we conduct an evaluation on a grayscale version of CLEVR-567 dataset. The model used in the evaluation is only trained on RGB CLEVR-567 dataset. The model achieves 87.5 FG-ARI on the grayscale test set, which is on par with 92.0 FG-ARI on the default RGB images. The evaluation results demonstrate that sVORF really learns to decompose the scene intrinsically. For qualitative results, see details in Appendix D.
4.6 Object Segmentation on Real Images
We demonstrate the effectiveness of our approach in real-world scenarios by validating it on LLFF datasets. To handle complex scenarios, we implement VIT-Base [47] as the backbone instead of ResNet34 [48] and train the network from scratch. Qualitative visualizations displayed in Figure 6 show that sVORF effectively segments the object within the scene, demonstrating the potential of our approach to understand general real scenarios.
Figure 6: Object segmentation on real images.
4.7 Training Speed and Memory Consumption
Setup We compare sVORF with uORF [8] and COLF [9] in terms of memory consumption and training speed. For all three methods, we train models on the CLEVR-567 dataset using a batch size of 1 on V100 and record their memory usage and time taken for one epoch. Since both COLF and uORF render and supervise images at  $64 \times 64$  resolution to learn the coarse structure first before supervising them at  $128 \times 128$  resolution, we record their performance during both stages.
Results As shown in Table 8, sVORF offers substantially shorter training time (3m24s vs. 9m2s) and lower memory consumption (4.6G vs. 23.7G) than the volumetric-decoder based uORF method, while displaying comparable performance to the light field-based COLF.
Table 8: Memory Consumption and Performance Comparison.
Model	Volumetric Field	Light Field
sVORF	uORF [8]	COLF [9]
coarse	fine	coarse	fine
Memory	4.6G	23.6G	23.7G	2.8G	3.8G
Training Time	3m24s	8m42s	9m2s	1m56s	2m22s
In terms of total training time,
for the CLEVR-567 and Room-Chair datasets, we train sVORF for approximately 7 hours using 8 Nvidia RTX 2080 Ti GPUs with batch size 16. The uORF and COLF models are trained on an Nvidia RTX V100 GPU for approximately 7 and 2 days, respectively, with a batch size of 1. For the CLEVR3D dataset, sVORF is trained for approximately 2 days using 8 Nvidia RTX V100 GPUs with batch size 16, while OSRT is trained for approximately 1 day on 8 A100 GPUs with a batch size of 256. These results demonstrate the effectiveness of sVORF in overcoming the challenges of volumetric decoders that demand extensive resources, while also contributing to enhanced training efficiency.
We present sVORF, a novel method for 3D object-centric representation learning. By adopting volumetric rendering to synthesis novel views and using object slots as a guidance to compose volumetric object radiance fields, sVORF precisely decompose 3D scenes into individual objects with limited training resources. It significantly outperforms existing SOTA methods, particularly in complex multi-object scenes. In addition, we demonstrate sVORF's ability for realistic scenario decomposition, indicating a promising direction for understanding the physical world.
[48] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
A Additional Training and Architecture Details
We utilize the Adam optimizer with a learning rate of 0.0001,  $\beta_{1} = 0.9$ , and  $\beta_{2} = 0.999$ . Additionally, we implement learning rate warm-up for the initial 1,000 iterations. The minimum number K of objects in each scene is customized separately as follows:  $K = 8, 7, 5, 5, 2$ , and 5 for the CLEVR-567, CLEVR-3D, Room-Chair, Room-Diverse, LLFF, and MSN datasets, respectively.
To allow training on a high resolution, such as  $256 \times 256$ , we render individual pixels instead of large-sized patches. Specifically, we randomly sample a batch of 64 rays from the set of all pixels in the dataset, and then follow the hierarchical volume sampling [4] to query 64 samples from the coarse network and 128 samples from the fine network.
In addition, we train our model from scratch, with the exception of the Room-Diverse dataset. The Room-Diverse dataset is more complex and requires an incremental learning approach. Specifically, we initialize our models for Room-Diverse using weights from a model that have previously been trained on the CLEVR-567 dataset.
A.2 Architecture details
Image encoder For the image encoder  $E$ , we utilize the well-known ResNet34 [48] architecture as the backbone, followed by three upsampling layers. Specifically, given the source image  $\mathbf{I} \in \mathbb{R}^{256 \times 256 \times 3}$ , we extract features  $\mathbf{F}_1 \in \mathbb{R}^{128 \times 128 \times 64}$ ,  $\mathbf{F}_2 \in \mathbb{R}^{64 \times 64 \times 128}$ , and  $\mathbf{F}_3 \in \mathbb{R}^{32 \times 32 \times 256}$  from the conv2, conv3 and conv4 layer of the ResNet34 architecture, respectively. Note that the max-pooling layer in conv2 is not used. Subsequently, these features are fed into the following three up-sampling layers within a U-net expansive path, resulting in the feature  $\mathbf{F} \in \mathbb{R}^{128 \times 128 \times 512}$ . The architecture of upsample layers is shown in Table 9.
Table 9: The architecture of upsample layers. All convolutional kernel sizes are  $3 \times 3$ . All activation functions for convolutional layers are ReLU. "+" indicates channel concatenation with a feature map of same resolution sourced from ResNet34.
Layer name	Input shape	Output shape	Stride
Conv1	32 × 32 × 256	32 × 32 × 512	1
Bilinear Upsampler	32 × 32 × 512	64 × 64 × 512
Conv2	64 × 64 × (512 + 128)	64 × 64 × 512	1
Bilinear Upsampler	64 × 64 × 512	128 × 128 × 512
Conv3	128 × 128 × (512 + 64)	128 × 128 × 512	1
Object NeRF To represent each object NeRF, we employ a simple 4-layer MLP that each layer has 128 channels and is followed by a ReLU activation. In addition, we use skip connection mechanism that add the first layer's activation to the third layer's activation. Note that the last layer outputs the RGB value with a sigmoid activation function.
Transformer-based module The efficient transformer module is a standard transformer decoder. Specifically, we firstly use each slot  $\mathbf{z}_i$  as query and interact with other object slots with multi-head self-attention operation. Then we employ multi-head cross-attention operation to attend into and aggregate features from the flattend image features  $E(\mathbf{I})$ . Finally, we pass the resulting slot features into a feed forward network (FFN) to get the final slots. This transformer module is simple and easy to train than the slot attention module, as it does not contain a Gated Recurrent Unit (GRU) block.
Composition module The aggregation block performs a cross-attention operation, which aggregates object representations  $S$  with the 3D location  $x$  as the query to obtain the corresponding feature  $z$ . The attention block computes the similarity between  $S$  and  $z$  after mapping them into the same space through a linear layer, thus obtaining the probability distribution of  $x$  belonging to each slot. In the initial stages of our experiments, we observed that utilizing both modules simultaneously yielded superior results. We speculate that this improvement may be attributed to a good feature space alignment between 3D points and slot features using the proposed aggregation block.
B Object Segmentation on Real Images
Setup In this study, we use VIT-Base [47] as the feature extractor for complex datasets of size  $1008 \times 756$  with intricate textures. The adopted backbone networks are trained from scratch, and the method strictly follows unsupervised learning. To conserve memory, we resize the source image to  $256 \times 256$  and select only 7168 rays from the target view during each training iteration. Moreover, we define the task as foreground-background segmentation. As such, we set the number of slots to 2 and map it to two NeRFs. Each NeRF consists of a simple 4-layer MLP.
Unlike the setup on other datasets, we train sVORF on both LLFF scenes as two separate models. Following the setup in NeRF-SOS, we divide the data of each scene into training and testing sets, ensuring there is no overlap between these two sets.
Results Figure 7 shows the complete segmentation results of sVORF on both Flower and Fortress datasets. The results demonstrate the potential of sVORF for 3D segmentation on non-object-centric real scenes with cluttered backgrounds. Additionally, we use ResNet34 as the backbone and provide the segmentation results in Figure 8. Unlike sVORF with ViT-Base, sVORF with ResNet34 produces a coarse segmentation and still segments foreground object from complex scenes.
Figure 7: Qualitative results of 3D segmentation on real images.
This section provides more ablation studies on architecture, assessing the efficacy of the transformer-based module (Section 3.2), hypernetwork (Section 3.3) and combination module (Section 3.3), respectively. The quantitative results are reported in Table 10.
Transformer-based module We substitute the transformer with slot attention and observe that slot attention fails to achieve the decomposition task in our model, as shown in the second row of Table 10.
Figure 8: Qualitative results of sVORF with ResNet-34 on LLFF dataset.
Table 10: Ablation studies on the CLEVR-567 dataset.
Model	NV-ARI ↑	FG-ARI ↑
sVORF (w/o Hypernetwork)	21.6	65.9
sVORF (w Slot-Attention)	14.1	76.8
sVORF (w SM)	28.4	71.2
sVORF (ours)	81.5	92.0
Based on this comparison, we can conclude that our transformer-based module has a better scene decomposition than slot attention in our training setting.
Hypernetwork We replace the hypernetwork with directly using the slots for conditioning the radiance fields per object like uORF [8]. As shown in the first row of Table 10, the model's performance significantly decreases. We speculate that using hypernetwork can provide stronger 3D geometric bias than directly using the slots for conditioning the radiance fields per object.
Our composition module v.s. Slot Mixers To compare our composition method with Slot Mixers decoder, we have some modifications on the SM decoder [3]. First, we use a 3D point  $x$  instead of the target ray as the query to aggregate the weighted slot feature. Second, we transform the weighted slot feature into the corresponding radiance field. Third, based on the radiance field, we can obtain the density and color of the 3D point  $x$ . In the experiment, we find that the composition performance of the SM method is lower than our proposed composition method. Specifically, the SM method exhibits 3D-inconsistent segmentation results, as shown in the third row of Table 10. It proves that the introduction of 3D geometric bias is really important for scene decomposition.
Depth maps We illustrate the depth maps of sVORF on different datasets in Figure 9. The results show that our method can learn a high quality of 3D geometry.
Figure 9: Reconstructed depth maps of sVORF on different datasets.
Object-level radiance fields. We provide the visualization of learned object-level radiance fields on CLEVR-567 dataset in Figure 10. It is further demonstrated that our method can achieve very clean scene decomposition.
Figure 10: Visualization of learned object-level radiance fields.
Grayscale images. We provide the multi-view results of the qualitative evaluation of the model on a grayscale version of the CLEVR-567 dataset in Figure 11.
Figure 11: Multi-view qualitative results on a grayscale version of the CLEVR-567 dataset.
There are three limitations in sVORF. First, although sVORF can generalize to more objects at test time, as shown in Section 4.7, the maximum slot number of test scenes is restricted by the training setting. In other words, we need know the maximum number of objects/slots in the scene, and ensure that the number of slots set is equal to or exceeds this maximum value. Second, like other 3D representation learning methods, the model's training process requires curated multi-view images, which are difficult to obtain and necessitate specialized equipment, particularly in real scenes. Finally, although our method performs well on some complex scenes, such as MSN, it is still a challenge for our method to decompose and understand real-world scenes like a human. Addressing the limitations related to the slot number, training dataset and the generalization capabilities on real scenes is a potential area for future research.