SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Di Qi
MEGVII Technology Inc. qidi@megvii.com
Tong Yang
MEGVII Technology Inc. yangtong@megvii.com
Xiangyu Zhang*
MEGVII Technology Inc.
We present a novel framework for 3D object-centric representation learning. Our approach effectively decomposes complex scenes into individual objects from a single image in an unsupervised fashion. This method, called slot-guided Volumetric Object Radiance Fields (sVORF), composes volumetric object radiance fields with object slots as a guidance to implement unsupervised 3D scene decomposition. Specifically, sVORF obtains object slots from a single image via a transformer module, maps these slots to volumetric object radiance fields with a hypernetwork and composes object radiance fields with the guidance of object slots at a 3D location. Moreover, sVORF significantly reduces memory requirement due to small-sized pixel rendering during training. We demonstrate the effectiveness of our approach by showing top results in scene decomposition and generation tasks of complex synthetic datasets (e.g., Room-Diverse). Furthermore, we also confirm the potential of sVORF to segment objects in real-world scenes (e.g., the LLFF dataset). We hope our approach can provide preliminary understanding of the physical world and help ease future research in 3D object-centric representation learning.
As humans, we can understand scenes, perceive discrete objects within it, and interact with these objects in a 3D environment. This object-centric, geometric understanding of the 3D world is a fundamental ability in human vision [1]. In computer vision, researchers attempt to replicate this fundamental ability in machine learning models with a high interest, due to its wide application ranging from robotics [2] to autonomous navigation. To this end, machine learning models should bear two characteristics: unsupervised representation learning manner and 3D-aware generative mode [3].
Recently, with the advances of neural radiance fields (NeRFs) [4] and representation learning [5, 6, 7], there are some works [8, 9, 3] to achieve these two characteristics. These works learn to decompose objects and understand 3D scene geometry from RGB supervision via novel view synthesis in an unsupervised manner. To this end, volumetric-based methods [8] utilize volume rendering mechanism to implement a 3D-aware differentiable generative process without supervision. To avoid high computational cost of volumetric-based methods, light-field based methods [9, 3] use light field formulation [10] to scale its application to large numbers of objects in a scene. However, existing works suffer from some limitations. Light-field based methods lack strict multi-view consistency [10] and fall into mask bleeding issues [3], which are harmful for object-centric representation learning. Besides, volumetric-based methods fail to decompose scenes caused by attention rank collapse [8] and do not perform well in complex multi-object scenes.
In this paper, we propose a novel framework for 3D object-centric representation learning, alleviating the issues of existing works. Our method, called slot-guided Volumetric Object Radiance
Fields (sVORF), adopts volumetric rendering to synthesis novel views and use object slots as a guidance to compose volumetric object radiance fields. Concretely, we firstly use an efficient transformer module to extract object slots from a single image and learn object-aware slot features with the help of self-attention mechanisms [11]. Then we utilize a hypernetwork to map these slots to volumetric object radiance fields. Finally, at each 3D location, we compose object radiance fields with the guidance of object slots for volumetric rendering. Thus, sVORF can avoid instinctive limits of light field formulation and resolve mask bleeding issues. Meanwhile, with the benefits from the guidance of object slots, sVORF can learn 3D-aware slot features and facilitate network optimization, alleviating the issues faced by existing volumetric-based methods. Moreover, instead of rendering whole image [8, 9], sVORF only render a small amount of image pixels during the training phase, which significantly reduces the demand for training resources.
To validate the effectiveness of sVORF, we conduct experiments on four synthetic datasets to assess the ability of scene decomposition (e.g., segmentation in 3D) and scene generation(e.g., novel view synthesis, scene editing in 3D). Our results demonstrate that sVORF can precisely decompose 3D scenes into individual objects and produce high-quality novel view images. Specially, sVORF outperforms other state-of-the-art methods by a significant margin in complex multi-object scenes. In ablation experiments, we also analyze the effects of core components and show their strength. Besides, we show the robustness of sVORF on unseen object appearance and unfamiliar spatial arrangements. Furthermore, we extend our validation process to complex real-world LLFF data, confirming that our approach can segment objects in complex scenes with high accuracy.
To sum up, our contributions are three fold. First, we introduce a novel approach for 3D-centric representation learning, named sVORF, that effectively decompose objects from a single image. Second, our slot-guided scene composition method avoids the shortcomings of existing methods and significantly reduces the memory requirements during training phase. Third, we validate the effectiveness of our proposed method on synthetic datasets and confirm the extendability of our approach on real-world scenes.
2.1 Neural Scene Representation and Rendering
Neural scene representations parameterize 3D scenes with a deep network that maps xyz coordinates to signed distance functions or occupancy fields. Equipped with differentiable rendering functions, they can be optimized using only 2D images, relaxing the requirement of 3D ground truth. In particular, Neural Radiance Fields (NeRFs) [4] can use an MLP to compute radiance values (color and density) for a given 5D coordinate (spatial location  $(x,y,z)$  and viewing direction  $(\theta ,\phi))$  and produce novel views with remarkably fine details. Numerous subsequent works have been introduced to address some its shortcomings and expand its applications, including rendering acceleration [12, 13, 14, 15, 16, 17, 18, 19, 20], NeRF with few images [21, 22, 23, 24], 3D reconstruction [25, 26, 27, 28, 29] and 3D scene semantic understanding [30, 31, 32, 33, 34]. However, these volumetric methods need hundreds of evaluations for a ray, leading to an expensive cost for rendering. To address this issue, Light Field Networks (LFN) [10, 24] directly map an input ray to an output color, making only a single evaluation of the MLP per ray.
2.2 3D Object-centric Representation Learning
Driven by the effectiveness of NeRF, recent research has attempted to combine 2D self-supervised object-centric models [5, 6, 7] with neural scene representations to decompose a 3D scene to individual objects. Earlier work [35] utilizes a slot-based encoder and NeRFs as 3D representations to decompose 3D scene with extra multi-view dense depth supervision. Likewise, uORF [8] explicitly model the separation of objects and background with only images in training to address complex scenes. In order to avoid expensive cost of volume rendering, OSRT [3] and COLF [9] further replace the volumetric parametrization with a light field formulation. However, existing methods face some limitations. ObSuRF [35] need dense depth as supervision and can not handle complex scenes, while uORF [8] encounters expensive computation cost during training. Although light field formulation method [3, 9] resolve computation cost issues, they lack strict multi-view consistency [10] and easily fall into mask bleeding issues [3].
Figure 1: sVORF overview. The image encoder  $E_{\theta}(\mathbf{I})$  processes the source view of a scene to generate 2D image features that serve as a prior. Next, these features are fed into the Scene Decomposition module to infer object and background slots. A hypernetwork then maps these slots to volumetric object radiance fields. Finally, the object slots provide guidance for the recombination of object radiance fields to render arbitrary views with 3D-consistent object decomposition.
Different from the above methods, our method adopts volume representation to parameterize 3D scenes, avoiding the limits of light field formulation. Also, our approach has low computation cost by only sampling a small amount of rays and avoids using depth supervision during training. Moreover, our method can be applied to object segmentation in real-world scenes (e.g. LLFF dataset) [36].
The aim of our method is to decompose a scene to a set of object-centric 3D representations given a single input image. An overview of our approach is provided in Figure 1. We firstly extract image features from an input image and decompose object slots from image features. Then we map these slots to volumetric object radiance fields and compose these object radiance fields with the guidance of object slots for novel view synthesis.
3.1 Preliminaries: NeRF
We begin by briefly reviewing the Neural Radiance Fields (NeRFs). A radiance field encodes a scene as a continuous volumetric radiance field  $f$ . The input to  $f$  is a location  $\mathbf{x} \in \mathbb{R}^3$  and a viewing direction  $\mathbf{d} \in \mathbb{R}^2$ , while the output is an RGB color value  $\mathbf{c} \in \mathbb{R}^3$  and a volume density  $\sigma \in \mathbb{R}^{+}$ . NeRFs parameterize  $f$  as an MLP  $\Phi_{\phi}$ . To make MLP learns high-frequency functions, the input coordinates  $\mathbf{x}$  and viewing directions  $\mathbf{d}$  are mapped into a higher dimensional space with sinusoids function  $\gamma(\cdot)$  before being passed into the MLP [37]. This process, known as positional encoding, allows the MLP to better capture high-frequency scene content. Therefore, the function  $\Phi_{\phi}$  can be formulated as:
$$
\Phi_ {\phi}: (\gamma (\mathbf {x}), \gamma (\mathbf {d})) \mapsto (\sigma , \mathbf {c}) \tag {1}
$$
Given  $\mathbf{N}$  sampled points along a ray  $\mathbf{r}$  and its predicted color and volume density  $\{(\mathbf{c_i},\sigma_i),i\in \{1,\ldots ,N\} \}$ , the expected color  $C(\mathbf{r})$  of camera ray  $\mathbf{r}$  can be derived from volume rendering:
$$
C (\mathbf {r}) = \sum_ {i = 1} ^ {N} T _ {i} \left(1 - \exp \left(- \sigma_ {i} \delta_ {i}\right)\right) \mathbf {c} _ {i}, T _ {i} = \exp \left(- \sum_ {j = 1} ^ {i - 1} \sigma_ {j} \delta_ {j}\right) \tag {2}
$$
where  $\delta_{i}$  indicates the distance between adjacent samples. The reconstruction loss between the rendered and true pixel colors is used during training to optimize the parameters  $\phi$  of MLP.
3.2 Scene Decomposition from 2D Prior
Following [35, 9], we define the scene as a combination of  $K$  entities, where the first  $K - 1$  represents the objects and the last one represents the background. To obtain these entities, we first extract image feature as a 2D prior using an encoder  $E(\mathbf{I})$  from a scene image  $\mathbf{I}$ . Instead of using slot attention module as existing methods, we adopt an efficient transformer module  $T$  to infer object and
background slots from image feature, termed as  $\mathbf{S} = \{\mathbf{s}_i\}_{i=1}^K$ . Compared to slot attention module, this transformer module is simple and easy to train without Gated Recurrent Unit (GRU) block. For this transformer module, we take a global image feature  $\bar{v} = \text{avg\_pool}(E(\mathbf{I}))$  as initial object slots  $\mathbf{Z} = \{\mathbf{z}_i\}_{i=1}^K$ . Combining with  $K$  learned positional encodings, these slots explicitly model all pairwise interactions between all slots and learn object-aware features via self-attention, which avoids inherent ambiguities in estimating color and geometry at occluded views, see Section 4.4 for details. Then these slots bind and explain the input image representation with cross-attention to image feature. Finally, the slots are transformed to  $\mathbf{S} = \{\mathbf{s}_i\}_{i=1}^K$  via feed forward network (FFN). The whole process can be formulated as follows:
$$
\mathbf {S} = T (\mathbf {Z}, E (\mathbf {I})) \tag {3}
$$
3.3 Scene Composition
Given the inferred slots  $\mathbf{S} = \{\mathbf{s}_i\}_{i=1}^K$ , we decompose these slots to novel views of the same scene, which is critical for achieving 3D object-centric representation learning. To achieve it, there are two forms: Spatial Broadcast (SB) [35, 8, 9, 3] and Slot Mixers (SM) [3]. SB and SM do scene composition on RGB space and feature space, respectively. SB can utilize 3D geometric bias (3D point or 3D ray) in the scene composition, facilitating the network optimization. Conversely, due to composition on feature space, SM is hard to optimize without 3D geometric bias. But SM can learn 3D-aware slot features, which is useful for scene decomposition. Combining the advantages of two forms, we propose a new method for scene composition. We transform slots into volumetric neural radiance fields to utilize explicit geometric bias and avoid the limits of light field formulation [10]. Then, we compose all volumetric neural radiance fields with the guidance of slots, making slot features 3D-aware.
Objects as Neural Radiance Fields To transform a slot to its radiance field, we utilize a hypernetwork [38, 39]  $H$  to map the slot  $\mathbf{s}_i$  directly to the parameters  $\phi_i$  of the associated object Neural Radiance Field. The mapping process is formulated as:
$$
\phi_ {i} = H \left(\mathbf {s} _ {i}\right) \tag {4}
$$
where  $i = 1,\dots ,K,K$  is the number of object slots.
With the radiance field  $\Phi_{\phi_i}$ , we can map a location  $\mathbf{x} \in \mathbb{R}^3$  with a viewing direction  $\mathbf{d} \in \mathbb{R}^2$  to a tuple of color  $\mathbf{c}_i$  and density  $\sigma_i$  of corresponding object:
$$
\Phi_ {\phi_ {i}}: (\gamma (\mathbf {x}), \gamma (\mathbf {d})) \mapsto (\sigma_ {i}, \mathbf {c} _ {i}) \tag {5}
$$
In this way, we transform feature space to RGB space, introducing 3D geometric bias for scene composition.
Composing Mechanism Given K Object NeRFs  $\{\Phi_{\phi_i}\}_{i = 1}^K$ , we compose their outputs  $\{\sigma_i\}_{i = 1}^K$  and  $\{\mathbf{c}_i\}_{i = 1}^K$  at a 3D location with the guidance of object slots. Specifically, we firstly leverage a feature aggregation block  $D^{c}$  to gather object slots and obtain an aggregate feature  $\mathbf{z}$  for query location  $\gamma (\mathbf{x})$ :
$$
\mathbf {z} = D ^ {c} (\gamma (\mathbf {x}), \mathbf {S}) \tag {6}
$$
where  $D^{c}$  is a cross-attention layer [40] network. Then we pass  $\mathbf{z}$  to an attention block  $D^{a}$  and compute a normalized dot-product similarity between  $\mathbf{z}$  and each slot feature in  $\mathbf{S}$ :
$$
\mathbf {m} = D ^ {a} (\mathbf {z}, \mathbf {S}) \tag {7}
$$
where  $\mathbf{m} = (m_1, \dots, m_K)$ ,  $D^a$  is a cross-attention layer without linear operation. Finally, we compute the combined density  $\bar{\sigma}$  and color  $\bar{\mathbf{c}}$  as follows:
$$
\bar {\sigma} = \sum_ {i = 0} ^ {K} m _ {i} \sigma_ {i}, \bar {\mathbf {c}} = \sum_ {i = 0} ^ {K} m _ {i} \mathbf {c} _ {i} \tag {8}
$$
Note that our model can train with a small amount of sample rays, leading to a significant reduction for computation and memory cost during training. We speculate that this characteristic benefits from our composing mechanism. In order to decompose small objects from a scene, it is essential to sample much more rays to cover the regions of these small objects, resulting in heavy training cost. However, our composing mechanism can perform well on small objects with a small amount of rays, addressing this large sampling cost, see Section 4.4.
Reconstruction Loss We train our model across multiple scenes and only take view images as the supervisory signal. The reconstruction loss is formulated as:
$$
\mathcal {L} _ {\text {r e c o n}} = \sum_ {\mathbf {r} \in \mathcal {R}} \| C (\mathbf {r}) - C _ {g t} (\mathbf {r}) \| ^ {2} \tag {9}
$$
where  $\mathcal{R}$  is the set of rays in each batch, and  $C_{gt}(\mathbf{r})$  is the groundtruth color.
Connectivity Regularization We observe that some object radiance fields exist semi-transparent clouds, especially when the number of slots is much larger than the total number of objects in a scene. To solve this issue, we apply a connectivity regularization  $\mathcal{L}_{\mathrm{connect}}$  to each  $\Phi_{\phi_i}$  by referring to the distortion loss presented in Mip-NeRF360 [41]:
$$
\mathcal {L} _ {\text {c o n n e c t}} (\mathrm {t}, \mathrm {w}) = \sum_ {i, j} w _ {i} w _ {j} \left| \frac {t _ {i} + t _ {i + 1}}{2} - \frac {t _ {j} + t _ {j + 1}}{2} \right| + \frac {1}{3} \sum_ {i} w _ {i} ^ {2} \left(t _ {i + 1} - t _ {i}\right) \tag {10}
$$
where  $\mathrm{w} = \{T_i(1 - \exp (-\sigma_i\delta_i))\}_{i = 1}^N$  is the weights along a ray and t is the normalized ray distance.
Total Loss The overall training loss function is formulated as follows:
$$
\mathcal {L} = \mathcal {L} _ {\text {r e c o n}} + \lambda_ {\text {c o n n e c t}} \mathcal {L} _ {\text {c o n n e c t}} \tag {11}
$$
where  $\lambda_{\mathrm{connect}}$  is the scale to balance the connectivity regularization  $\mathcal{L}_{\mathrm{connect}}$ , which is set to be 0.01 in our experiments.
Datasets Following uORF [8], we experiment on several datasets in increasing order of complexity.
CLEVR-567 [8]: The CLEVR [42] dataset is a widely used benchmark for evaluating object decomposition in computer vision. CLEVR-567 is a multicamera variant of this dataset with 1,000 scenes for training and 500 scenes for testing. Each scene consists of with 5-7 CLEVR objects that randomly positioned and oriented with a clean background. The objects in the scenes are comprised of three geometric primitives: cubes, spheres, and cylinders. We follow uORF's setup in using a "Rubber" material with largely diffuse properties.
CLEVR-3D [35]: This dataset is also a variant of CLEVR dataset, in which each scene consists of 3-6 basic geometric shapes of 2 sizes and 8 colors. In particular, each scene includes 3 fixed views: the two target views are the default CLEVR input view rotated by  $120^{\circ}$  and  $240^{\circ}$ , respectively. Following ObSuRF [35], we train 35k scenes and test on the first 320 scenes of each validation set [43, 6].
Room-Chair [8]: This dataset contains 1,000 scenes designated for training and 500 for testing. In this dataset, each scene includes 3-4 chairs of identical shape with 3 different textures background.
Room-Diverse [8]: This dataset is an upgraded Room-Chair. Each scene contains 4 distinct chairs, whose shape chosen randomly from ShapeNet [44] chair shapes, and a range of background that is selected from 50 unique textures. There are 5,000 scenes for training and 500 for testing.
MultiShapeNet (MSN) [35]: This dataset comprises 11,733 distinct shapes, with each scene populated by 2-4 objects with different categories sourced from the ShapeNetV2 3D model dataset.
Local Light Field Fusion (LLFF) [36]: This dataset includes real scene scenarios with complex foreground and background, making it highly challenging. We specifically utilize the forward-facing scenes Flower and Fortress from the LLFF dataset, with each scene consisting of 27 training images and 6 testing images.
Baselines We compare our model with a 2D object-centric learning method Slot-Attention [43] and four competitive 3D methods, namely uORF [8], COLF [9], ObSuRF [35] and OSRT [3], in terms of scene decomposition and novel view synthesis. Both COLF and OSRT are based on light field, while uORF and ObSuRF employ volumetric parameterization, which is similar to our approach.
Table 1: Scene segmentation results. Bold and Underline indicate state-of-the-art (SOTA) and the second best.
Model	CLEVR-567	Room-Chair	Room-Diverse
3D metric	2D metric	3D metric	2D metric	3D metric	2D metric
NV-ARI↑	ARI↑	Fg-ARI↑	NV-ARI↑	ARI↑	Fg-ARI↑	NV-ARI↑	ARI↑	Fg-ARI↑
Slot Attention [43]	N/A	3.5	93.2	N/A	38.4	40.2	N/A	17.4	43.8
uORF [8]	83.8	86.3	87.4	74.3	78.8	88.8	56.9	65.6	67.9
COLF [9]	46.6	59.5	92.6	83.5	83.9	92.4	54.5	70.7	71.7
sVORF	81.5	82.7	92.0	87.0	87.8	92.4	75.6	78.4	86.6
Table 2: Results on CLEVR-3D dataset.  ${}^{ \dagger  }$  Model is re-evaluated using the unified test set for a fair comparison. Bold and Underline indicate state-of-the-art (SOTA) and the second best.
(a) 3D Segmentation.
Model	Supervision	ARI* ↑	Fg-ARI*↑
ObSuRF [35]	image+depth	94.6	95.7
OSRT† [3]	image	42.7	97.0
sVORF	image	86.0	96.3
(b) Novel View Synthesis.
Model	LPIPS ↓	SSIM ↑	PSNR ↑
ObSuRF [35]	N/A	N/A	33.69
OSRT† [3]	0.0367	0.9719	36.74
sVORF	0.0258	0.9759	37.52
Metrics To evaluate the quality of novel view synthesis, we use Learned Perceptual Image Patch Similarity (LPIPS) [45], Structural Similarity Index (SSIM) [46], and Peak Signal-to-Noise Ratio (PSNR). For scene segmentation in 3D, we measure clustering similarity using Adjusted Rand Index (ARI). The ARI score ranges from 0 to 1, with a score of 0 indicating random segmentation and a score of 1 indicating perfect segmentation. For a fair comparison, we evaluate two types of 2D ARI metrics: ARI and FG-ARI, as well as three types of 3D ARI metrics: NV-ARI,  $\mathrm{ARI}^*$ , and FG-ARI*. Specifically, both ARI and FG-ARI are computed on the source image to facilitate comparison with 2D methods. The FG-ARI is further calculated solely on the foreground regions, using ground-truth data. Similarly,  $\mathrm{ARI}^*$  and  $\mathrm{FG - ARI}^*$  are calculated on all images. Lastly, the NV-ARI is computed on synthesized novel views.
4.1 Scene Segmentation in 3D
Setup Given the soft slot masks of 3D locations, 2D segmentation masks are inferred through volume rendering. To clarify, we begin by mapping the pixel  $\mathbf{p}$  in the rendered view to a ray  $\mathbf{r}$  for sampling N points along it. Then, we calculate the soft mask  $\mathbf{m}$  of each sampling point  $\mathbf{x}$  at all object NeRFs and derive the composite density  $\bar{\sigma}$  using the mask information. Finally, we render the segmentation mask along the ray based on the composite density to yield the segmentation of pixel  $\mathbf{p}$ .
Results We compare our method with uORF and COLF, and present the results in Table 1. The comparison reveals that our method outperforms all baselines in terms of NV-ARI and ARI values, particularly the NV-ARI, in both Room-Chair and Room-Diverse scenes. It provides evidence that sVORF can effectively identify 3D objects from a single image with better multi-view consistency. Furthermore, in the more complex Room-Diverse scene, our approach achieves comprehensive and distinct improvements over other methods, indicating the robustness of our design. Moreover, we report the performance on CLEVR-3D in Table 2a to ensure equitable comparisons with ObSuRF and OSRT. Compared with OSRT, sVORF achieves significantly higher  $\mathrm{ARI}^*$  and similar FG-ARI* without using depth information, which further proves that our volume parameterization can alleviate mask bleeding problems, as shown in Figure 3. However, we encounter exceptions in the ARI and NV-ARI scores on the CLEVR-567 dataset, and the  $\mathrm{ARI}^*$  on CLEVR-3D dataset. Our benchmark scores are slightly lower than those of uORF and ObSuRF. We attribute this to the penalty imposed on reconstructing shadows of foreground objects, which are not included in the ground truth object masks. This inadequacy is illustrated in Figure 3. Since the light source remains fixed in the CLEVR dataset, it is straightforward to model the shadow as a translucent layer belonging to the object. Nonetheless, the high FG-ARI indicates that our proposed method is proficient in forming factorized representations, which can segment objects in a scene effectively.
Table 3: Comparison on novel view synthesis from a single image.
Model	CLEVR-567	Room-Chair	Room-Diverse
LPIPS ↓	SSIM ↑	PSNR ↑	LPIPS ↓	SSIM ↑	PSNR ↑	LPIPS ↓	SSIM ↑	PSNR ↑
uORF [8]	0.0859	0.8971	29.28	0.0821	0.8722	29.60	0.1729	0.7094	25.96
COLF [9]	0.0608	0.9346	31.81	0.0485	0.8934	30.93	0.1274	0.7308	26.02
sVORF	0.0211	0.9701	37.20	0.0824	0.8992	33.04	0.1637	0.7825	29.41