Unsupervised Discovery of Object-Centric Neural Fields
Authors: Rundong Luo
Hong-Xing Yu
Jiajun Wu
Venue: Accepted by TMLR ()

Abstract: We study inferring 3D object-centric scene representations from a single image. While recent methods have shown potential in unsupervised 3D object discovery, they are limited in generalizing to unseen spatial configurations. This limitation stems from the lack of translation invariance in their 3D object representations. Previous 3D object discovery methods entangle objects’ intrinsic attributes like shape and appearance with their 3D locations. This entanglement hinders learning generalizable 3D object representations. To tackle this bottleneck, we propose the unsupervised discovery of Object-Centric neural Fields (uOCF), which integrates translation invariance into the object representation. To allow learning object-centric representations from limited real-world images, we further introduce an object prior learning method that transfers object-centric prior knowledge from a synthetic dataset. To evaluate our approach, we collect four new datasets, including two real kitchen environments. Extensive experiments show that our approach significantly improves generalization and sample efficiency and enables unsupervised 3D object discovery in real scenes. Notably, uOCF demonstrates zero-shot generalization to unseen objects from a single real image. We attach our code in the supplementary file, and the project page is available at https://red-fairy.github.io/uOCF/

Published in Transactions on Machine Learning Research (02/2025)
Unsupervised Discovery of Object-Centric Neural Fields
Rundong Luo Cornell University
rundonghaookes.coonlll.ee
Hong-Xing Yu' Stanford University
koven koweefacc..tannoordcoo edu
Jiajun Wu Stanford University
jiajumuuucss edu
Reviewed on OpenReview: https: // openreview. net/ forum? id= ScEu13W2f1
Abstract
We study inferring 3D object-centric scene representations from single image. While recent methods have shown potential in unsupervised 3D object discovery, they limited in generalizing to unseen spatial configurations. This limitation stems from the lack of translation invariance in their 3D object representations. Previous 3D object discovery methods entangle objects' intrinsic attributes like shape and appearance with their 3D locations. This entanglement hinders learning generalizable 3D object represontations. To lackle this bottleneck, we propose the unsupervised discovery of Object- neural Fields (uOCF), which integrates translation invariance into the object representation. To allow learning object-centric representations from limited rcal-world images, further introduce learning method that transfers object-centric prior knowledge from synthetic dataset. To evaluate our approach, collect four new datasets, including two real kitchen environments. Extensive experiments show that our approach significantly improves generalization and sample efficiency, and enables unsupervised 3D object discovery in real scenes. Notably, uOCF demonstrates zero-shot generalization unseen objects from a single real image. The 1o/uDCF/.
project page is available https://red-fate github.io/u0CF//
1 Introduction
Creating factorized, object-centric 3D scene representations fundamental ability in human vision and long-standing topic of interest in computer vision and machine learning. Some recent work has explored unsupervised learning of 3D factorized scene representations from images alone (Stelzner et al., 2021; Yu et al., 2022; Smith et al., 2023; Jia et al., 2023). These methods have delivered promising results in 3D object discovery and reconstruction from simple synthetic image.
However, existing methods fail to generalize to unseen spatial configurations and objects. A fundamental bottleneck that their representations lack the invariance the 3D positions of the objects. In particular, existing methods represent 3D objects as implicit functions in the viewer's coordinate frame, so that any change related the coordinate frame (e.g., slight changes in object's location or subtle camera movements) may lead to significant changes in the object representation even if the object remains the same. Therefore, existing methods do not generalize when an object appears at an unseen location during inference.
To address this fundamental bottleneck, we propose the unsupervised discovery of Object- Centric neural Fields (uOCF). Unlike existing methods, uOCF explicitly infers an object's 3D location, disentangling from the object's 's latent representation. This design builds translation invariance into the object representation, so
"Equal Contribution.
Work done while R. Lmo was visiting student Stanford University.

---

Published in Transactions on Machine Learning Research (02/2025)
Unseen
input image
Reconstruction (uORF)
Reconstruction (ours)
Rearrangement (ours)
Insertion (ours)
Figure 1: We propose the unsupervised discovery Object-Centric neural Fields (uOCF), which infers factorized 3D scene representations from unseen real image, thus enabling scene reconstruction and manipulation from novel views. We compare uOCF with the state-of-the-art method, uORF (Yu et al., 2022).
that the object's latent only represents the intrinsics of the object (c.g., shape and appcarance). This design significantly improves generalization. As showcased in Figure 1, uOCF can generalize to unseen real-world scenes. We train uOCF on sparse multi-view images without object annotations. During inference, uOCF takes in a single image and generates a set of object-centric neural radiance fields (NeRIFs) (Mildenhall et al., 2020) and a background NeRF.
Another advantage of our translation- innvaaiionnn 3D object representation that it facilitates learning 3D object priors from simple scenes and generalizes to more complex scenes with unseen spatial configurations and objects. This further boosts sample efficiency and thus it is particularly beneficial when deal with real scenes where training data is often limited. We introduce object prior learning method to this end.
To evaluate our approach, we introduce new challenging datasets for 3D object discovery, including two real kitchen datasets and two synthetic room datasets. The two real datasets feature real-world kitchen backgrounds and objects from multiple categories. The synthetic room datasets feature furniture with diverse, realistic shapes and textures. Across all these datasets, uOCF yields high-fidelity discovery of object-centric neural fields, allowing applications such unsupervised 3D object segmentation and scene manipulation from real image. uOCF shows strong generalization to unseen spatial configurations and high sample efficiency, and we showease that it even allows zero-shot 3D object discovery on a few simple real scenes with unseen objects. In summary, our contributions are threefold:
First, we highlight the overlooked role translation invariance in unsupervised 3D) object discovery. We instantiate the idea by proposing the unsupervised discovery of Object-Centric neural Fields (uOCF), which builds translation invariance to the object representation.
Second, we introduce 3D object prior learning method, which leverages uOCF's translation-invarian property to learn category-agnostie object priors from simple scenes and generalize to different object categorics and scene layouts.
Lastly, we collect four challenging datasets, Room-Texture, Furm--urni Kitchen-Matto, and Kitchen- Shiny, and show that uOCF significantly outperforms existing methods on these datasets, unlocking zero-shot, single-image object discovery. All code and data will be made public.
2 Related Works
Unsupervised 2D object discovery. Before the advent of deep learning, traditional methods for object discovery (often referred to co-segmentation) primarily focused locating visually similar objects across
2

---

Published in Transactions on Machine Learning Research (02/2025)
(6) Encoder
Input image
Feature map
Reconstruction
(Cu, Pa)
OF DINO VIT
Demsity eighted Comp.
Conv
Vollann Remdering
(1):)))
Novel view
pp. Famk
Moneennan Update
to
L [p Fron
Global residual
Cross-Ath
MLP
z'
Learaable
Intered object lasents
Inpul view rotation
Query point position
object queries
(b) Latent Inference Module
(c) Object NeRF Decoder
Figure 2: With single forward pass, nOCF processes a single image input to infer a set of object-centric radiance fields along with their 3D locations and background radiance field. sOCF is trained on sparse multi-view images from collection of scenes and uses a single image as input during inference.
collection of images (Sivic et al., 2005; Russell et al., 2006), where objects are defined visual words or clusters of patches (Grauman & Darrell, 2006; Joulin et al., 2010). This clustering concept was later incorporated into deep learning techniques for improved grouping results (Li et al., 2019; Vo et al., 2020). The incorporation of deep probabilistic inference propelled the field towards factorized scene representation learning (Eslami et al., 2016). These methods decompose a visual scene into several components, where objects are often modeled as latent codes that can be dccoded into image patches (Kosiorck et al., 2018; Crawford & Pineau, 2019; Jiang et. al., 2020; Lin al., 2020), phase values (Löwe el. al., 2022; Gopalakrishnan et al., 2024), scene mixtures (Greff et al., 2016; 2017; 2019; Burgess et al., 2019; Engelcke et al., 2019; Locatello et al., 2020; Biza et al., 2023; Didolkar et al., 2023), or layers (Monnier et al., 2021). Notably, Seitzer et al. (2023) leveraged DINO features to extract robust image representations, while Daniel & Tamar (2022; 2023) introduced approaches to disentangle object latents into 2D position and appearance attributes. Despite their efficacy in scene decomposition, they do not model the objects' 3D nature.
Unsupervised 3D object discovery. To capture the 3D nature of scenes and objects, recent works have explored learning 3D-aware representations from point clouds (Wang et al., 2022), videos (Henderson & Lampert, 2020), and multi-view images of either single scenes (Liang et al., 2022) or large datasets for generalization (Eslami et. al., 2018; Chen el al., 2020; Sajjadi et al., 2022). More recent research focuses on inferring object-centrin factorized scene representations from single images (Stelzner et al., 2021; Yu et al., 2022; Smith et al., 2023). Among these, Yu et al. (2022) proposed a mothod for the unsupervised discovery of object radiance fields (uORF) from single images. Follow-up works (Smith et al., 2023) improved rendering efficiency by replacing NeRF with light fields (Sitzmann et al., 2021) or enhanced segmentation accuracy through bi-level query optimization (Jia al., 2023). However, these representations often lack translation invariance, which limits their robustness and generalization capabilities. In this work, address this limitation by incorporating translation ion invariance, resulting in significant improvements in both generalization and sample efficiency. For a more comprehensive review of related methods, we refer readers to Villa- V Vásquez & Pedersoli (2024).
Object-centric 3D reconstruction. Decomposing visual scenes on an object-by-object basis and estimating their gooommtrii/oooiiicccccocoo attributes has been explored in several recent works (Wu et al., 2017; Yao et al., 2018; Kundu et al., 2018; Ost et al., 2021). Some approaches, such as AutoRF (Müller et al., 2022), successfully reconstruct specific objects (e.g., cars) from annotated images. Others decompose visual scenes into the background and individual objects represented by neural fields (Yang et al., 2021; Wu et al., 2022). Our work differs because of its emphasis on unsupervised learning. Another line of recent work focuses on lifting 2D segmentation to reconstructed 3D scenes (Fan et al., 2022; Cen et al., 2023a;b). In contrast, our work aims at. single-image inference, whereas these studies concentrate on multi-view reconstruction.
Generative neural fields. Neural fields have revolutionized 3D scene modeling. Early works have shown promising geometric representations (Sitzmann al., 2019; Park et al., 2019). The seminal work neural
3

---

Published in Transactions on Machine Learning Research (02/2025)
radiance fields (Mildenhall et al., 2020) has opened up burst of research on neural fields. We refer the reader to recent. survey papers (Tewari et al., 2020; Xie et al., 2022) for a comprehensive overview. In particular, compositional generative neural fields such as GIRAFFE (Niemeyer & Geiger, 2021) and others (Nguyen- Phuoc et al., 2020; Wang et al., 2023b) also allow learning object representations from image collections. Yet, they larget unconditional generation and cannot tackle inference.
3 Approach
Given a single input image, our goal to infer object-centric radiance fields (i.e., each discovered object is represented in its local object coordinate rather than the world or the viewer coordinates) and the objects' 3D locations. The object-centric design not only boosts generalizability due to representation invariance, but also allows learning object priors from scenes with different spatial layouts and compositional configurations. The following provides an overview of our approach and then introduces the technical details.
3.1 Model Overview
As shown Figure 2, uOCF consists of an encoder, a latent inference module, and a decoder.
Encoder. From input image I, the encoder extracts feature map f E RNC-C where N = H W is the spatial size of the feature map and C. represents the number of channels. We set it frozen DINOv2-ViT (Oquab et al., 2023) followed by two convolutional layers.
Latent inference module. The latent inference module infers the latent representation and position of the objects in the underlying 3D scene from the feature map. We assume that the scene is composed of background environment. and no more than K foreground objects. Therefore, the output includes a background latent zb R2ID and set of foreground object latent zf Izi" z 2X]" RRD with where p,dd E I denotes position in the world coordinate. Note that some object latent may be empty when the scene has < K objects.
their corresponding positions (p. K [p
Decoder. Our decoder employs the conditional NeRF formulation g(x|z), which takes the 3D location x and the latent z as input and generates the radiance color and densily for rendering. We use two MLPs, gb and g", for the background environment and the foreground objects, respectively.
3.2 Object-Centric 3D Scene Modeling
Object-centric latent inference. Our Latent Inference Module (LIM) aims at binding set of learnable object queries ( q = [q|" q T E RRK RKxD) to the visual features of each foreground object, and another query to the background features (q E RIxD). The binding is modeled via the cross-attention mechanism with learnable linear functions 1Ch K0, Lh, o Vh,
exp(M..) Aij Σn ex) where M
(D) VD ('(a) E Nx((K+))
1 -
(1)
We then calculate the update signals for queries via an iidenntion-eighhenn mean of the input:
nb = (W (..))" (P) 2xD; =WW()(2)) V'() E KXD
(2)
where W
Agj is the normalized attention map. Queries are then updated by: Σ, A1,
qd + uu,, of q + u': q + +b("))) a' +(
(3)
where t and of are MLPs. We repeat this procedure for T iterations, followed by concatenating the updated object queries with the corresponding attention- attention-weighted mean of the input feature map f (global residual), finally delivering the background latent z² and foreground latent {z/)KK
Our LIM is related to the Slot Attention (Locatello et al., 2020) while differs in several critical aspects. We discuss their relationship in Appendix C.I.
Object location inference. To infer objects' position along with their latent representation, we assign normalized image position p; img 1]2 initialized as zero to each foreground object query, then
4

---

Published in Transactions on Machine Learning Research (02/2025)
Stage 1: Learn 3D object prior
Stage 2: Learn to discover objects from scenes with diverse object category and spatial layout
from synthetic scenes with simple composition
Figure 3: Our object-centric design allows learning 3D object priors that generalize across different scene configurations. We first train our model to learn 3D object priors on simple synthetic scenes (e.g., single synthetic object), and then leverage the 3D object priors to learn discover objects in complex scenes with different object categories and spatial layouts. Note that no object annotation is needed in either stage.
iteratively update them by momentum m with the attention--eighter mean over the normalized 2D grid gabs E [
P
img
(W (..41))
Eabs (1- m) + P. img m
(4)
To incorporate the inferred positions, adopt the relative positional encoding (Biza et al., 2023) E img
concal((Eabe tm P; m Eabe]) RRxx, E (1,2, K}, and E = concal([Esbe T Ebabe),, where concat is the concatenation along the last dimension. Then, re-write M in Eq. (1) as:
(q)) (C h1 (Epoo) T ( Kf + hi (Epon) ))
M
(5)
VD
e (akk (C + hi (D
where 41 IR. > R is linear function.
Overall, LIM achieves a gradual binding between the queries and the objects the scene through an iterative update of the queries and their locations. To address potential issues of duplicate object identification, we invalidate one of two similar object queries with high similarity and positional proximity by the start of the last iteration. Finally, a small bias term is added to the position to handle potential occlusion, i.e., where scaling hyperparameter 0.2 and h2 EN + R² is linear
mm + p. + (WW function.
The 2D positions P, extend the rays by depth d s, where the depth estimated by a monocular depth estimator (Ranftl et al., 2022) and { are scaling terms predicted by a linear layer using the camera parameters and object latent as input.
ing are then unprojected into the 3D world coordinate to obtain p. To do this, we
Compositional neural rendering. The object positions allow us to put objects in their local coordinates rather than the viewer world coordinates, thereby oblaining object-centric neural fields. Technically, for each 3D point x in the world coordinate, we transform it to the 7th object's local coordinate by where R denotes the input camera rotation matrix. We then retrieve the color and density x in the foreground radiance fields as (c,, 0,) = g (x/2() and in the background radiance field as (c): (x(2")) These values are aggregated into the scene's composite density and color (c, o) using weightt means:
R (x - p.d),
Σ WiO; C Ewro, where Wi $20
T
Σj20 T
120
Finally, we compute the pixel color by volume rendering. Our pipeline is trivially differentiable, allowing
backpropagation through all paramcters simultancously.
Discussion on extrinsics disentanglement. An object's canonical orientation is ambiguous without assuming its category (Wang et al., 2019). Thus, we choose not to disentangle objects' orientation since we target catogory-agnostic object discovery. Further, we observe that uOCF has learnod meaningful representations that can smoothly interpolate an object's scale and orientation. Please refer Appendix B for visualization and analysis.
5

---

Published in Transactions on Machine Learning Research (02/2025)
(a) Room-Texture
(b)
(c) Kitchen-Matee
(d) Kitchen-Shinn -hiny
consist of synthetic images, Figure 4: Samples from our collected datasets, where Room- Texture and and Kitchen-Matte and Kirchen-Shiny consist of real photos.
3.3 Object Prior Learning
Unsupervised discovery of 3D objects in complex scenes is inherently difficult due multiple challenging ambiguities. A major ambiguity is what defines object. While existing methods define objects via visual appearance similarity (Yu et al., 2022) priors from 2D segments (Chen et al., 2024), they suffer from under-segmentation due to visual cluttering (Yu et al., 2022) or over-segmentation inherited from the 2D supervision (Chen et al., 2024).
We explore addressing this challenge by learning 3D object priors from synthetic data. Existing methods have difficultics learning generalizable 3D object priors, their object representation sensitive to spatial configurations: a minor shift in camera pose or object location, rather than the object itself, can lead drastic changes in the object representation. Thus, such learned object priors do not generalize when there are unseen spatial configurations.
Our 3D object-centric centric representation mitigates this issue by translation invariance. In particular, we introduce 3D object prior learning. We show an illustration in Figure 3. The main idea. is to pre-train uOCF on synthetic scenes that are constructed with a single object to ease the learning, similar to curriculum learning. After the pre- pre-training stage, we proceed training uOCF on the more complex scenes that may have different object categories and spatial layouts. Note that either training stage does not require any object annotation. The pre-training synthetic singlo-object dataset can be casily scaled up.
3.4 Model Training
Object-centric sampling. To improve the reconstruction quality, we leverage an object's local coordinates concentrate the sampled points proximity to the object. Specifically, we start dropping distant samples from the predicted object positions after few training epochs when the model has learned to distinguish the foreground objects and predict their positions. This approach enables to quadruple the number of samples with the same amount of computation, leading to significantly improved robustness and visual quality.
In both training stages, we train our model across scenes, each with calibrated sparse multi-view images. For each training step, the model receives an image as input, infers the objects' latent representations and positions, renders multiple views from the input and reference poses, and compares them to the ground truth images to calculate the loss. Model supervision consists of the MSE reconstruction loss freeon and the perceptual loss lpere (Johnson et al., 2016) between the reconstructed and ground truth images. addition, incorporate the depth ranking loss (Wang al., 2023a) with pre-trained monocular depth estimators and background occlusion regularization (Yang et al., 2023) to minimize common floating artifacts in few-shot.
NeRFs.
The overall loss function thus formulated as follows:
L= Erecon + Aperclpere + Adepth + Noceface-
(7)
We leave further architectural details and illustrations in Appendix C.1.
4 Experiments
We evaluate our method on three tasks: unsupervised object segmentation in 3D, novel view synthesis, and scene manipulation in 3D. Below, we briefly describe the data collection process and experimental configurations, with additional details provided in Appendices C.2 and C.3. Sample code and data are included in the supplementary material, and plan to release the full code and datasets for public use.

---

Published in Transactions on Machine Learning Research (02/2025)
Table Object segmentation and view synthesis on roxm-rextuure and Room- Koom-Furnitur Furniture.
Room-Texturs
Room- Furn-Fur
Object segmentation
Novel view syuthesis
Object segmentation
Novel view synthesis
Method
ARI FG-ARIT NV-ARI
PSNR1
SSIM LPIPSU
ARI FG-ARI NV-ARII
PSNR
SSIMI
LPIPSL
nORE (Yu et al., 2022)
0.670
0.093
0.578
24.23
0.711
0.254
0.686
0.497
0.556
27.49
0.780
0.258
BO-QSA (Jia al., 2023)
0.697
0.354
0.604
25.26
0.739
0.215
0.082
0.479
0.579
27.29
0.774
0.261
COLF (Smith et al,, 2023)
0.235
0.532
0.011
22.98
0.670
0.501
0.514
0.458
0.439
28.73
0.781
0.386
uOCF (ouns)
0.785
0.127
Input view
Novel view
Input view
N/A
Novel view
N/A
GT image
GT segment.
wOCF (ours)
uORF
BO-QSA
COLF
Figure 5: Scene segmentation qualitative results. Novel view images are for reference only.
Data. We collect two synthetic datasets and two real-world datasets to evaluate our method. Examples of these datasets are shown in Figure 4.
Room-Textune Room-Texture features 324 object models from the "armchair" category of the ABO(Collins et al., 2022) dataset. Each scene includes 2-4 objects arranged on backgrounds randomly selected from collection of floor textures. The dataset. comprises 5,000 scenes for training and 100 for evaluation. Each scene is rendered from four viewpoints centered on the scene.
Foom-Furnturre In Furniture, objects are selected from 1,425 ABO (Collins et al., 2022) models spanning seven categories, including "bed", "cabinet", "chair", "dresser", "ottoman", "sofa", and "plant pot". Other configurations match that of RRoomTTexture
Kitchen-Matte, This dataset includes scenes featuring single-color matte dinnerware set against two types of backgrounds: a plain tabletop a complex kitchen environment. The dataset contains 735 scenes for training and 102 for evaluation. Each scene includes 4 objects positioned randomly and is captured from three viewpoints (for tabletop scenes) or two viewpoints (for kitchen backdrops).
Kitchen-Shiny. This datasct contains scenes with textured, shiny dinnerware. Similar toKitchen-Matte, the first half of the dataset features plain tabletop, while the latter half includes kitchen background. The dataset consists of 324 scenes for training and 56 for evaluation.

---

Published in Transactions on Machine Learning Research (02/2025)
Table Novel view synthesis on Kitchen-Shiny with larger num- ber object queries K.
Table 2: Novel view synthesis on Kitchen-Shiny and Kitchen-Matte.
Sitchen-Shiny
Kitchen-Matte
Method
PSNR1
SSIM
I.PPS.
PSNR1
SSTM
I.PPSS
PSNR"
SSIMI
LPIPS,
uORE al., 2022)
0.602
26.07
0.808
K
28. 58.58
0.862
0.0-19
BO-QSA (Jia et al., 2023)
19.78
0.639
0.318
27.36
0.832
0.067
28.28
0.846
0.059
COLF (Smith et al,, 2023)
18.30
0.561
0.397
20.68
0.643
0.230
0.848
0.058
uOCF (ours)
28.58
0.862
0.049
29.40
0.867
0.043
K 10
28.20
0.840
0.065
Input view
Novel view
Input view
Novel view
GT
uOCF (ours)
uORF
BO-QSA
COLF
Figure 6: Novel view synthesis qualitative results Kitchen-Shiny (lop) and Room-Furniture (bottom).
Implementation details. To learn object priors, we generate synthetic dataset of over 8,000 scenes. Each scene contains one object, sampled from a high-quality subset of Objaverse-LVIS (Deitke et al., 2023), placod against room background. These objects span over 100 calegories. The synthetic datasel easy to generate and scalable, making ideal for learning object priors for all our experiments.
In the second stage, the number of foreground object queries is set K 4. We initialize the model with the pre-trained weights from the object prior learning stage and train it multi-object scenes. Once trained, our model can perform direct inference on images with spatial configurations that differ from those seen during training. Additionally, our model can adapt to unseen enviromments through efficient test-time optimization (see Sec. 4.2 for details).
Basclines. We compare our method against uORF (Yu ct al., 2022), BO-QSA (Jia et al., 2023), and COLF (Smith al., 2023). To ensure fair comparison, increase the latent dimensions and training iterations for all methods. For baseline models, retain their original implementation without incorporating our proposed object-centric learning stage to maintain consistency. Details on baselines enhanced with our object-centric learning stage can be found in Appendix D.
Metrics. We report the PSNR, SSIM, and LPIPS metrics for novel view synthesis. For scene segmentation, we use three variants of the Adjusted Rand Index (ART): the conventional ARI (calculated on all input image pixels), the Foreground ARI (FG-ARI, calculated on foreground input image pixels), and the Novel View ARI (NV-ARI, caleulated novel view pixels). All scores computed images of resolution 128 x 128.
8

---

Published in Transactions on Machine Learning Research (02/2025)
Table 4: Scene manipulation results on the Moom-TTextur dataset.
Object Translation
Object Removal
Method
PSNR1
SSIM
IPIPS!
PSNR1
SSIM
LPIPS]
nORF (Yu et al., 2022)
23.65
0.654
0.284
23.81
0.664
0.282
BO-QSA (Jia et al., 2023)
25.21
0.700
0.226
24.58
0.698
0.247
uOCF (ours)
27.66
0.774
0.156
28.99
0.802
0.136
Object removal
Object translation
Imput view
Novel view
Input image
uOCF (ours)
uORF
BO-QSA
#OCF (ours)
uORF
BO-QSA
Figure 7: Qualitative results of single-image 3D scene manipulation on the Kitchen-Shiny dataset.
4.1 Baseline Comparison on Multiple Tasks
Unsupervised object segmentation in 3D. evaluate the object discovery quality by object segmentation in 3D. We render a density map d' for each latent i and assign each pixel p segmentation label sp arg d f in the input view and novel views. We show our results in Table 1 and examples in Figure 5. From Table 1, we see that our uOCF outperforms all existing methods all metries. From Figure 5, we observe that no prior method can produce reasonable segmentation results in real-world Kitchen-Shiny scenes. Specifically, uORF binds all objects to the background, resulting in empty object segmentation; BO-QSA fails to distinguish different object instances; COLF produces meaningless results novel views. A fundamental issue in these methods is that they lack appropriate object priors to handle the ambiguity in disent multiple objects. In contrast, uOCF can discover objects in real-world scenes. Moreover, uOCF can handle scenes where objects occlude each other. We provide more visualization results in Appendix D.
Novel view synthesis. We evaluate the scene and object reconstruction quality by novel view senth For each test scene, we use single image as input and other views as references. We show our results Table and examples in Figure 6. We also show additional results in Appendix D. Our method significantly surpasses the baselines in all metrics. Importantly, while previous methods often fail to distinguish foreground objects and thus produce blurry reconstruction ion of objects, our approach consistently produces high-fidelity scene and object reconstruction and novel view synthesis results.
Scene manipulation in 3D. We further evaluate object discovery by single-image 3D scene manipulation. Since uOCF explicitly infers 3D locations of discovered objects, it readily supports: 1) object translation by modifying an object's position, and 2) object removal by excluding objects during compositional rendering.
For quantitative evaluation, create test set by randomly selecting an object in each of the Room- Texture scenes, and shift its position (object translation) remove it (object removal). During inference, we determine the object to manipulate by selecting the object with the highest IoU score with the ground truth mask. As shown in Table 4, uOCF outperforms baselines across all metrics in both object translation and object removal due to its better performance in object discovery. We further show qualitative examples from the Kitchen-Shiny dataset Figure 7. We observe that nORF merges all objects into the background, and thus the manipulation results are identical to the original reconstruction; BO-QSA fails to distinguish foreground objects, resulting in blurry manipulation results (we show more visualization in Appendix D).
9

---

Published in Transactions on Machine Learning Research (02/2025)
Objects
Reconstruction
Input image
Novel view
324 training
100 training scenes
50 training scenes
10 training
Figure & Qualitative results on sample efficiency. With fewer training scenes, our uOCF can still produce reasonable object discovery thanks the object-centric modeling and learned object priors.
AY
N/A
N/A
GT
WOCF (ours)
EORF
BO-QSA
uOCF (ours)
uORF
BO-QSA
(a) -oxturex > HM3D
(b) Room-Texture
Phone capture
Figure Zero-shot generalization results. load the model trained on one dataset and test on an image from another dataset after fast test-time optimization the input view only. First/sccond/Ulne /third row: scene reconstruction/non novel view/objects.
In contrast, our uOCF delivers much higher-quality manipulation results. We show additional visualization results the supplementary video.
4.2 Generalization Analysis
In the experiments above, all test scenes have unseen novel spatial configurations, where uOCF shows strong generalization. We further evaluate the sample efficiency on spatial generalization, and we showcase the generalization to unseen objects.
10

---

Published in Transactions on Machine Learning Research (02/2025)
Inpur view
Novel view
nORF
uOCF (variant 1) uOCF (variant 2) uOCF (variant 3)
uOCF (ours)
Figure 10: Ablation on translation invariance and object prior learning. Three variants: (1) without both translation
invariance and object prior learning, (2) without translation invariance, (3) without object prior learning.
Sample efficiency. We train uOCF with a small subset of (e.g,, only 10) the training scenes, and test it on the test set. As shown by the qualitative example in Figure 8, even when we only have a few training scenes, uOCF still demonstrates a good generalization ability discover objects. This is mainly due to the translation invariance and learned object priors, which reduce the dependence massive training scenes.
Generalization to unseen objects. We evaluate the zero-shot generalization ability of uOCF by training it on one dataset and test it on a single image of unseen background and objects. Specifically, we test our model on five real-world examples (one from the HM3D dataset (Ramakrishnan et al., 2021) and four captured with a phone) using model trained solely on the synthetic Room-Texture dataset. As shown Figures and existing methods struggle adapt to novel objects in unscen scttings. In contrast, uOCF demonstrates remarkable generalizability, requiring only lightweight single-image test-Lime optimization for 1000 iterations. This process takes approximately 3 minutes, fraction of the 6 days needed for the full training of the model. These results highlight uOCF 's ability to adapt effectively from synthetic training data to real-world scenarios. Details on this experiment can be found in Appendix C.3.
4.3 Ablation Study
Key technical contributions. We conduct ablation studies to analyze the impact of our key technical contributions: the translation-inwan invariant design and object prior learning.
As shown in Table and Figure 10, incorporating translation invariance dramatically reduces the LPIPS metric from 0.186 to 0.049. Similarly, leveraging object prior learning significantly decreases LPIPS from 0.125 to 0.049. These results demonstrate that both contributions are not only individually impactful but also complementary. Removing either one severely degrades performance, justifying their importance in achieving high-quality results.
Other technical improvements. We also evaluate the impact of other technical improvements through ablation studies. As shown in Table 6, the inclusion of DINO VIT and standard attention enhances overall performance. However, these components contribute relatively modestly; for instance, removing DINO or standard attention slightly increases LPIPS from 0.049 0.060 0.062, respectively. Excluding depth and occlusion losses also degrades visual quality, lcading noticeable drop in performance. Additionally, remov- ing the object-centric sampling strategy slightly reduces overall reconstruction quality, further highlighting the significance of these improvements.
Different K values. We evaluate the effectiveness of different K values and different scales of data used for 3D object prior learning. We evaluate our method's robustness to different values, and we show results in Table and Figure 11. From Table 3, we can see that even when we set K = 10 which is much higher than the number of possible maximal objects (i.c., 4), our model robust and gives comparable results. From Figure 11, we observe that even if there are more object queries than the number of objects in the scene, wOCF learns to generate "empty" object queries instead of over-segmenting the objects.
11

---

Published in Transactions on Machine Learning Research (02/2025)
Table 5: Ablation studies key technical contributions on Kitchen-Shiny.
Table Ablation study on model architecture and loss function Kitchen-Shiny.
Method
PSNR
SSTM
LPIPS
Method
PSNR1
SSIM
LPIPS4
w/o trans. invar. or object prior learning
0.645
w/o DINO
26.25
0.831
0.060
w/o trans. invar.
23.70
0.721
0.186
w/o standord attention
27.82
0.844
0.062
w/o object prior learning
26.81
0.806
0.125
w/o object-centric sampling
27.31
0.852
0.072
uOCF (ours)
28.58
0.862
0.049
w/o Edepsh and
26.79
0.819
0.081
uOCF (ours)
28.58
0.862
0.049
N/A
N/A
K=10
Input image
Reconstruction
Object
Object 2
Object 3
Object 4
Object
Object 6+
Figure 11: Qualitative results of uOCF with larger object queries K. The order of the object reconstructions is rearranged for better visualization.
5 Conclusion
We study the importance of translation invariance for unsupervised 3D object discovery, instantiated our model for the unsupervised discovery of Object-Centric neural Fields (uOCF). Our results show that our translation-invariant design and the 3D object prior learning can substantially improve the spatial generalization and sample efficiency. Our results demonstrate that unsupervised 3D object discovery can be extended to real scenes while obtaining satisfactory performances.
Limitations. Although uOCF shows promising unsupervised 3D object discovery results, is currently limited to simple real scenes such as the kitchen scenes. Extending more complex real scenes with complex spatial layouts and large number of objects from different categories important future direction. We leave more discussion technical limitations in Appendix E.
Acknowledgments
This work is in part supported by NSF RI #2211258 and #2338203, ONR MURI N00014-22-1-27400, and ONR YTP N00014-441-2117.
References
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualle and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In IEEE/CVP International Conference Computer Vision (ICCV), 2021.
Ondrej Biza, Sjoerd van Steenkiste, Mehdi SM Sajjadi, Gamaleldin F Elsayed, Aravindh Mahendran, and Thomas Kipf. Invariant slot attention: Object discovery with slot-centric reference frames. In International Conference on Machine Learning (ICML), 2023.
Christopher Burgess, Loie Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv:1901.11390 2019.
Jiazhong Cen, Jiemin Fang, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, and Qi Tian. Segment any 3d gaussians. arXiv:2312.00 2.00860, 2023a.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Wei Shen, Lingxi Xie, Xiaopeng Zhang, and Qi Tian. Segment anything in 3d with nerfs. In Advances in Newral Information. Processing Systems (NewrPPS), 2023b.
Chang Chen, Fei Deng, and Sungjin Ahn. Learning to infer 3d object models from images. arXiv:2006.06113 2020.
Honglin Chen, Wanhee Lee, Hong- Xing Yu, Rahul Mysore Venkatesh, Joshua B Tenenbaum, Daniel Bear, Jiajun Wu, and Daniel LK Yamins. Unsupervised 3d scene representation learning via movable object inference. Transactions on Machine Learning Research (TMLRR) 2024.
Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthicu Guillaumin, and Jitendra Malik. Abo: Dataset and benchmarks for real-world 3d object understanding. In IEEE/CVI Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In AAAI Conference on Artificial Intelligence (AAAI), 2019.
Tal Daniel and Aviv Tamar. Unsupervised image representation learning with deep latent particles. In International Conference on Machine Learning (ICML), 2022.
Tal Daniel and Aviv Tamar. Ddlp: Unsupervised object-centric video prediction with deep dynamic latent particles. Transactions on Machine Learning Research (TMLR), 2023.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsaui, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annolated 3d objects. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Aniket Didolkar, Anirudh Goyal, and Yoshua. Bengio. Cycle consistency driven object discovery. arXiv:2306.0220 2023.
Martin Engeleke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations 1rrXi.1190..00 2019.
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukeuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 2018.
Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any-view self-supervised object segmentation on complex scenes. orXiv:2209.0877 2022.
Anand Gopalakrishnan, Aleksandar Stanié, Jürgen Schmidhuber, and Michael Curtis Mozer. Recurrent complex-weite weighted autoencoders for unsupervised object discovery. In Advonces in Neural Information Processing Systems (NeurIPS), 2024.
Kristen Graunan and Trevor Darrell. Unsupervised learning of categories from sets of partially matching image features. In IEEE Conference on Compuler Vision and Pallern Recognition (CVPR), 2006.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hotloo Hao, Jürgen Schmidhuber, and Harri Valpola. Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems (NewrIPS), 2017.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Klaus Greff, Raphaäl Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning (ICML), 2019.
Paul Henderson and Christoph H Lampert. Unsupervised object-centric video generation and decomposition in 3d. In Advances in Newral Information Processing Systems (NeurIPS), 2020.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint Xrxiv:331 1. 1.04400, 2023.
Baoxiong Jia, Yu Liu, and Siyuan Huang. Improving object-centric learning with query optimization. In International Conference on Learning Represenlations (ICLR), 2023.
Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, and Sungin Ahn. Scalor: Generative world models with scalable object representations. In International Conference on Learning Representations (ICLR), 2020.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In European Conference on Computer Vision (ECCV), 2016.
Armand Joulin, Francis Bach, and Jean Ponce. Discriminative clustering for image co-segmentation. In IEEF/CVF Conference. on Computer Vision and Pattern Recognition (CVPR), 2010.
Adam R Kosiorek, Hyunjik Kim, Ingmar Posner, and Yee Whye Teh. Sequential attend, infer, repeat: Generative modolling of moving objects. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Abhijit Kundu, Yin Li, and James M Rehg. 3d-renn: Instance-level 3d object reconstruction via render-and- compare. In TEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Bo Li, Zhengxing Sun, Qian Li, Yunjie Wu, and Anqi Hu. Group-wise deep object co-segmentation with attention recurrent neural network. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Shengnan Liang, Yichen Liu, Shangzhe Wu, Yu- Wing Tai, and Chi-Keung Tang. Onerf: Unsupervised 3d object segmentation from multiple views. (xXi::2211.12935 2022.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference Learning Representations (ICLR), 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Sindy Löwe, Phillip Lippe, Maja Rudolph, and Max Welling. Complex-valued autoencoders for object discovery. Transactions on Machine Learning Research (TMLR), 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In European Conference Compuler Vision (ECCV), 2020.
Tom Monnier, Elliot Vincent, Jean Ponce, and Mathieu Aubry. Unsupervised layered image decompasition into object prototypes. In JEEE/ TVF International Conference on Computer Vision (ICCV), 2021.
Norman Müller, Andrea Simonelli, Lorenzo Porzi, Samuel Rota Bulo, Matthias Nicßiner, and Peter Kontschieder. Autorf: Learning 3d object radiance fields from single view observations. In IEEE/CVF Conference Computer Vision and Paltern Recognition (CVPR), 2022.
14

---

Published in Transactions on Machine Learning Research (02/2025)
Thu H. Nguyen-Phuoe, Christian Richardt, Long Mai, Yongliang Yang, and Niloy Mitra. Blockgan: Learning 3d object-aware scene representations from unlabelled images. In Aduances in Neural Information Processing Systems (NeurIPS), 2020.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Kii:2304.07190 2023.
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes, In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat-matterprl 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv:2109. 08238, 2021.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022.
Bryan C Russell, William T Freeman, Alexei A Efros, Josef Sivic, and Andrew Zisserman. Using multiple segmentations to discover objects and their extent in image collections. In IEEE Conference on Compuler Vision and Pattern Recognition (CVPR), 2006.
Mchdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjocrd van Steenkiste, Filip Pavetic, Mario Lucie, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al. Bridging the gap to real-world object-centric learning. In International Conference on Learning Representations (ICLR), 2023.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In Advances in Neural Information Processing Systems (NewrIPS), 2021.
Josef Sivic, Bryan Russell, Alexci A Efros, Andrew Zisserman, and William Freeman. Discovering objects and their location in images. In IEEE International Conference on Compuler Vision (ICCV), 2005.
Cameron Smith, Hong-Xing Yu, Sergey Zakharov, Fredo Durand, Joshua B Tenenbaum, Jiajun WI, and Vincent. Sitzmann. Unsupervised discovery and composition of object light fields. Transactions on Machine Learning Research (TMLR), 2023.
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXim:2104.011/6 2021.
Ayush Tewari, Ohad Fried, Justus Thics, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Bruadd Tomas Simon, Jason Saragih, Matthias Nießner, al. State of the art on neural rendering. Computer Graphics Forum, 2020.
15

---

Published in Transactions on Machine Learning Research (02/2025)
Villa-Vásquee and Marco Pedersoli. Unsupervised object discovery: A comprehensive survey and unified taxonomy. arXiv preprinl arXiv:24 11.:2411.0008000 1.00868, 2024.
Huy V Vo, Patrick Pérez, and Jean Ponce. Toward unsupervised, multi-object discovery in large-scale image collections. In European Conference Computer Vision (ECCV), 2020.
Guangcong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023a.
He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category-level 6d object pose and size est estimation. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Qian Wang, Yiqun Wang, Michael Birsak, and Peter Wonka. Blobgan-3d: A spatiaaly-ddisettngleed 3d-aware generative model for indoor scenes, arXiv:2303. 14 706, 2023b.
Tianyu Wang, Miaomiao Liu, and Kee Siong Ng. Spatially invariant unsupervised 3d object-centric learning and scene decomposition. In European Conference on Computer Vision (ECCV), 2022.
Jiajun Wu, Joshua Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Qianyi Wu, Xian Liu, Yucdong Chen, Kejic Li, Chuanxia Zheng, Jianfei Cai, and Jianmin Zheng. Object- compositional neural implicit surfaces. In European Conference on Compuler Vision (ECCV), 2022.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum, 2022.
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Jiawei Yang, Marco Pavone, and Yue Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Shunyu Yao, Tzu Ming Harry Hsu, Jun- Yan Zhu, Jiajun Wu, Antonio Torralba, William T Freeman, and Joshua Tenenbaum. 3d-aware scene manipulation via inverse graphics. In Advances in Neural Information Processing Systems (NewrIPS), 2018
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Hong- -ing Yu, Leonidas Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. In International Conference on Learning Representations (ICLR), 2022.
16

---

Published in Transactions on Machine Learning Research (02/2025)
A Appendix Overview
This supplementary document is structured as follows: We begin with the proof of concept Appendix B and provide the implementation details in Appendix C. Then, we discuss the limitations of our approach in Appendix E. and present additional qualitative results in Appendix D. Accompanying this document. is project page with overview video attached in the supplementary file.
B Proof of Concept
We conduct toy experiment (Figure 12) to demonstrate that our model has successfully learned object position, rotation, and scale. In this experiment, we begin with two images (input and input 2) of a chair placed at the scene's center, exhibiting different sizes (on the left) or rotation angles (on the right), all captured from the same viewing direction.
We extract the object latents from these images, interpolate them, and then send the interpolated latents the decoder. As shown between the two input images, observe smooth transition in object size and rolation, indicating that the latent representation has effectively captured the scale and rotation of objects.
In the second row, we placed the chairs in different positions. As shown on the right, we obtained a smooth transition again, proving that our model could discntangle object positions from the latent representation.
Input
Latent interpolation
Input
Input
Latent interpolation
Input 2
size: big
small
rotation; 0 - /2
Input
Rotation: π/ n/2 Position=(O. 0)
Input
Rotation: 0
Latent linear interpolation position translation
Position=(0, (0,00 0)
Figure 12: Proof of concept. We demonstrate that uOCF has effectively learned objects' scale and orientation along with the iranslation--nvvaiian object representation by interpolating the representation of two identical objects with different orientations and scales obtain transitional results.
C
Implementation
C.1 Model Architecture
Encoder. Our encoder module consists of frozen DINO encoder and two convolutional layers. We illustrate its architecture in Figure 13(.).
Latent inference module. While motivated by the background-aware slot attention module proposed by (Yu et al., 2022), our latent inference module exhibits three key differences: (1) The object queries are initialized with learnable embeddings instead of being sampled from learnable Gaussians, which enhances training stability; (2) We jointly extract object positions and their latent representations and add object- spocific positional encoding to utilize the extracted position information; (3) We remove the Gated Recurrent. Unit (GRU) and replace it with the transformer architecture to smooth the gradient flow.
C.2 Data Collection
This section introduces the details of our datasets.
17

---

Published in Transactions on Machine Learning Research (02/2025)
Resize 876*876
(Frozen) DINOv2 VIT 6676*876*3>>446444
3*3 conv, stride- ReLU (64*64"768 66 64
3+3 conv, stride-1 664*64"768 64 64
Dropped samples
Object bounding box
Remaining samples
Feature map f
(a) Encoder architecture
(b) Object-centric sampling
We drop the samples distant from Figure 13: (a) Architecture of encoder module. (b) Object-centric sampling: the predicted object position for efficient sampling..
Room-Texture Room-Texture, ure, objects are chosen from 324 ABO objects (Collins et al., 2022) from the "armchair" category. The single-object subsct contains four scenes for each object instance, resulting in 1296 scenes in total. The multiple-object subset includes 5, 000 scenes for training and 100 for evaluation, with each scene containing 2-4 objects set against background randomly chosen from collection of floor textures. Each scene is rendered from 4 directions toward the center.
Room-Purniture, In Room-Furniture, objects are chosen from 1, 425 ABO (Collins et al., 2022) object models, spanning across seven categories, including "bed", "cabinet", "chair", "dresser", "ottoman", "sofa", ", and "plant pot". Each scene contains 2-4 objects sot against a background randomly chosen from collection of floor textures. We render 5000 scenes for training and 100 scenes for evaluation.
Kitchen-Maten In Kitchen-Matte, objects diffuse and have texture. The dataset comprises 16 objects and 6 tablecloths in total. We captured 3 images for each tabletop scene and 2 for each kitchen scene. This dataset contains 735 scenes for training and 102 for evaluation, each containing 3-4 objects. We calibrate the cameras using the OpenCV library.
Kitchen-Shiny, Kitchen-Shiny, objects are specular, and the lighting is more complex. The dataset comprises 12 objects and 6 tablecloths, and the other settings are identical to Kitchen-Matte This dataset contains 324 scenes for training and 56 for evaluation, each containing 4 objects.
C.3 Training Configuration
This section discusses the training configuration of uOCF.
We employ Mip-NeRF (Barron et al., 2021) as our NeRF backbone and estimate the depth maps by MiDaS (Ranftl et al., 2022). An Adam optimizer with default seper-parameters and an exponential decay schoduler is used across all experiments. The initial lcarning rate 0.0003 for the first stage and 0.00015 for the second stage. Loss weights are set to Apere 0.006, Adepth 1.5, and Aner 0.1. The position update momentum is set to 0.5, and the latent inference module lasts T = 6 iterations. Most hyperparameters are inherited from Yu et al. (2022), while the loss weights of our proposed losses are chosen to ensure a similar scale between all loss terms. All experiments are run on a single RTTXA6000 GPU. Training lasts approximately days, including 1.5 days for object prior learning and another 4.5 days for training on multi-object scenes.
Coarse- To-see progressive training. We employ a coarsc-to-fine strategy in our second training stage to facilitate training at higher resolutions. Reference images downsampled to lower resolution (64 x 64)
18

---

Published in Transactions on Machine Learning Research (02/2025)
during the coarse training stage and replaced by image patches with the same size the low-resolution images randomly cropped from the high-resolution (128 x 128) input images during the fine training stage.
Locality constraint and object-centric sampling. We employ the locality constraint (a bounding box for foreground objects in the world coordinate) proposed by (Yu et al., 2022) in both training stages but only adopt before starting object-centric sampling. The number of samples along each ray before and after starting object-centric sampling set to 64 and 256, respectively. We provide an illustration of our object-centric sampling strategy in Figure 13(b).
Training configuration on Room-Texture. During stage 1, we train the model for 100 epochs directly on images of resolution 128 x 128. We start with the reconstruction loss only, add the perceptual 10th epoch, and start the object-centric sampling at the 20th epoch. During stage 2, we train the model for 60 epochs on the coarse stage and another 60 on the fine stage. We start with the reconstruction loss only, add the perceptual loss at the 10th opoch, and start the object-centric sampling from the 20th epoch.
Training configuration
Kitchen-Matte and Kitchen-Shiny. Both kitchen datasets share the same stage 1. During stage 2, train the model for 750 epochs, training configuration with pooo-TTxxtur in
where the fine stage starts at the 250th epoch. We add the perceptual loss at the 50th epoch and start the object-centric sampling from the 150th epoch.
Training configuration for zero-shot generalization. For the test-time adaptation, we fine-tune our model on the input view only using our proposed loss function (Eq. 7) for 1000 iterations on resolution 128 x 128. We use the Adam optimizer and the learning rate set to 1 10-4, This optimization takes about 3 minutes single A6000 gpu.
D Additional Experiments
Additional real-world dataset. We introduce an additional real-world dataset, named "Planters," " which features tabletop scenes containing four plant pots or vases arranged on tablecloths. The dataset includes 745 scenes for training and 140 scenes for evaluation, with each scene captured from three different camera poscs. As shown the quantitative results in Table and the qualitative results in Figure 17, our method significantly outperforms existing approaches. It. achieves superior scene reconstruction and novel view synthesis, delivering results with noticeably higher visual quality.
Method
PSNR1
SSTM
LPIPSJ
uORF
24.49
0.748
0.163
WORF-BO-QSA
28.09
0.847
0,108
COLF
19.22
0.588
0.461
nOCF (ours)
29.00
0.864
0.062
Table 7: Quantitative results on the Planters dataset.
Baseline performance with object-centric learning. To ensure fair comparisons, conducted additional experiments where object prior learning was incorporated into the baseline methods. The results in Table 8 demonstrate that even with the incorporation of object prior learning, uOCF significantly outperforms existing methods due to its translation-invariint object representation, which enhances generalization and data efficiency.
Additional zero-shot evaluation. We evaluate our method on two additional phone-captured scenes to further demonstrate the generalizability of uOCF, To ensure fair comparisons, we incorporate object prior learning into the baseline methods. However, shown in Figure 15, the baseline methods still struggle generalizing to unseen environments. In contrast, our method produces accurate object segmentation and novel view synthesis results, further validating its effectiveness.
Visualization on discovered objects. We visualize the discovered objects in Figure 16. Notably, uORF (Yu et al., 2022) puts all objects within the background, whereas BO-QSA (Jia et al., 2023) binds the same object to all queries, resulting in identical foreground reconstruction. In contrast, uOCF accurately differentiates between the foreground objects and the background.
19

---

Published in Transactions on Machine Learning Research (02/2025)
Method
LPIPS 4
SSTM
PSNR
uORF
0.336
0.602
19.23
nORF + object prior learning
0.193
0.714
22.78
BO-QSA
0.318
0.639
19.78
BO-QSA object prior learning
0.129
0.766
24.00
COLF
0.397
0.561
18.30
COLF + object prior learning
0.290
0.709
21.66
uOCF (ours)
0.049
0.862
28.58
Table 8: Baseline performance with object-centric learning. Our method maintains its superiority even when baselines methods employ proposed object-prio learning approach due to its Granslationinvariin representation.
Visualization on object segmentation in 3D. We show scene segmentation results on the kitchen datasets in Figure 18. Unlike compared methods that yield cluttered results, uOCF consistently yields high-fidelity segmentation results.
Additional novel view synthesis results. We show more qualitative results for novel view synt hesis in Figures 18, 19, and 20. Our method produces much better results than compared methods regarding visual quality.
E Limitations Analysis
Limitation reconstruction quality. Sceno-level generalizable NeRFs (Yu et al., 2021; Sajjadi al., 2022; Yu et al., 2022) commonly face challenges in accurately reconstructing detailed object textures. Our approach also has difficulty capturing extremely high-frequency details. As shown in Figure 14(a), our method fails to replicate the mug's detailed texture. Future research may benefit from stronger object priors learned from larger-scale datasets, such as Large Reconstruction Models (Hong et al., 2023).
Failure position prediction. Our two-stage training pipeline, despite robustness in many situations, is not immune errors, particularly in object position prediction. Due to the occlusion between objects, using the attention-weighted mean for determining object positions can sometimes lead to inaccuracies. Although bias term can rectify this in most instances (Figure 6), discrepancies persist under few conditions, depicted in Figure 14(b).
Training instability. Like other slot-based object discovery methods, our approach also faces challenges with training instability. For example, the model occasionally collapses within the first. few training epochs, even when using identical hyperparamoters. To address this, we perform multiple trials for each experiment, terminating early signs of collapse appear during the initial (raining stages. We observed that approximately half of the experiments fail at this stage. However, once the model progresses beyond this critical phase, the final results remain consistent across different trials. For bascline methods that struggle on our proposed complex datasets, we conduct at least five trials to ensure that their failure stems from limitations in their design rather than issues with random initialization.
20

---

Published in Transactions on Machine Learning Research (02/2025)
GT
Reconstruction
GT
Reconstruction
Imput view
Novel view
(a) Failure on reconstruct texture
(b) Error in position prediction
Figure 14: Failure case visualizations. Our method may fail to reconstruct intricate object texture or predict biased object position.
uOCF (ours)
uORE
Input image
BO-QSA
uOCF (ours)
WORF
Inpul image
BO-QSA
Reconstruction
Segmentation
Object 1
Object 2
Object
Object 4
Novel view
Figure 15: Qualitative zero-shot generalization results.
21

---

Published in Transactions on Machine Learning Research (02/2025)
uOCF (owrs)
EORF
Input image
BO-QSA
uOCF (ours)
wORF
Input image
BO-QSA
Background
Object
Object 2
Object
Object 4
Figure 16: Visualization on discovered objects on Kitchen-Shiny.
22

---

Published in Transactions on Machine Learning Research (02/2025)
Input view
Novel view
Input view
Novel view
Input view
Novel view
Input view
Novel view
GT
uOCF (ours)
uORF
BO-QSA
COLF
Figure Qualitative comparison results on the Planters datasel.
23

---

Published in Transactions on Machine Learning Research (02/2025)
Inpur view
Novel view
Input view
Novel view
Input view
Novel view
Input view
Novel view
GT
uOCF (ours)
uORF
BO-QSA
COLF
Figure 18: Additional segmentation and view synthesis results on the Room- cextTrex dataset.
24

---

Published in Transactions on Machine Learning Research (02/2025)
Input view
Novel view
Input view
Novel view
Input view
Novel view
Input view
Novel view
GT
uOCF (ours)
uORF
BO-QSA
COLF
Figure 19: Additional view synthesis results on the Kitchen-Matks dataset.
25

---

Published in Transactions on Machine Learning Research (02/2025)
Input view
Novel view
Input view
Novel view
Input view
Novel view
Input view
Novel view
GT
uOCF (ours)
uORF
BO-QSA
COLF
Figure 20: Additional view synthesis results the Kitchen-Shing dataset.
26


Reviews:
[{"review_id": "AqpgNhFHTR", "reviewer": "Reviewer_L2jR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nArchitecture - The proposal of an architecture explicitly addressing translation invariance is compelling, as a method for enhancing generalization across various scene configurations and layouts. The work also provides a clear ablation study, demonstrating the relative utility of the translation invariance aspect of the model.\n\nObject prior learning - The object prior learning schedule is an interesting proposal, demonstrating a successful approach to leveraging simple synthetic objects to better condition a model to perform on real data. The successful synthetic to real transfer is a nice example of how such easy to collect synthetic data can be leveraged. Zero-shot demonstration on unseen real world scenes is particularly compelling due to this reason.\n\nWeaknesses: \n\nLack of clarity around what data the baselines are able to have access to train on (e.g. is object prior learning applied for uORF?) For each of the 3 baselines, it would need to be very explicit - how does the training data provided compare to that for uOCF.\n\nLarge amount of pre-training - several training stages required (pre-training on single objects, training on the train set of scenes, training/optimization for purpose of discovery on unseen scene in the case of zero-shot.) \n\nClaim on zero-shot is not sufficiently supported - a primary practical application is in the zero-shot case - as for most scenes, there will not be 1000s of related scene configurations to learn from. However, the zero-shot case is only demonstrated for 2 scenes. A larger number of scenes, with a concrete diversity of real world items, would be more appropriate to support the claim on zero-shot.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work proposes Unsupervised Discovery of Object-Centric Neural Fields (uOCF), a method for explicitly identifying and representing 3D objects within a scene via learned 3D real world coordinates and a latent representation. Conditioning on each object's latent representation, a set of neural radiance field decoders learned per object (and additionally one for background) can be queried continuously in 3D space and consequently rendered to novel 2D views.  \n\nThe contributions are 3 fold:\n\n- The proposed architecture which addresses the issue of previous implicit object representations that did  not disentangle object position from object representation. The architecture in this work learns the two separately, allowing learned representations to be 3D position agnostic.\n- A method for improving generalization is provided by learning object priors from synthetic data. Specifically, the authors propose a series of training steps: first pre-training on vast number of simple easy to generate scenes of single objects from the ObjectVerse dataset, and then subsequently training on more complex scenes of synthetic or real data, enabling the model to develop an understanding of object representations before encountering a further unseen scene.\n- 4 datasets are provided: 2 synthetic (one consistent of armchair items with varying backgrounds and one of bedroom items), as well as 2 real datasets (one consistent of kitchen table top views with matte objects, and the other with shiny objects) \n\nuOCF is benchmarked against 3 baseline methods (uORF, BO-QSA, and COLF) on 3 demonstrative tasks of unsupervised object discovery in 3D (i.e. successful segmentation of items), novel view synthesis (visual reconstruction from unseen viewpoints), and 3D scene manipulation (object translation and removal). uOCF outperforms all baselines on these benchmark tasks.\n\nAdditionally the work explores the generalization capacity of uOCF via an evaluation of sample efficiency (training from as little as 10 scenes, iteratively up to the full dataset size, and demonstrating varying degrees of object recognition and novel view rendering capacity). The work also demonstrates the ability of uOCF to perform zero-shot adaptation to unseen objects (real world captures) leveraging only a single view from 2 new scenes. \n\nAblation studies are also conducted to confirm the relative contributions of the translation invariance and object prior pre-training aspects of the proposed method. The authors additionally confirm that the model is robust to the number of queried objects, and an excess of objects can be queried without leading to over segmentation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Include more explicit description of the training phases for uOCF (this is already briefly noted at the end of Object Prior Learning, but would be more clear if e.g. given a small thorough section within Model Training including such additional details such as, when test time optimization is used - such as for zero-shot case - and when it is not.) \n\n- Describe BO-QSA and COLF more explicitly within related works, and briefly explain why the 3 baselines were chosen for benchmarking against uOCF.\n\n- Should have a brief note on what exact data each of the 3 baselines are able to pre-train / train on & are given access to (for example, is object prior learning applied for uORF?) Looking at the uORF method, it is plausible that the full uOCF training scheme of object prior learning and subsequent training / evaluation on the given split, could be equally applied for training the uORF baseline as well. Making explicit in the paper, what exact data uORF had access to in this work, would greatly support ease of understanding. \n\n- For the claim of zero-shot demonstration on real world scenes to be kept - the experiment should be performed on a larger number of real world scenes than 2. Perhaps 3-5 scenes at least, to make a qualitative claim.  \n\n- Baselines should ideally be fairly benchmarked on the zero-shot task too, in order to demonstrate relative performance.\n\n\nMinor / Formatting:\n- Add space to \"Texturescenes\" on page 9", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EPEAX4jNzY", "reviewer": "Reviewer_agc3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.The paper studies a critical problem in object-centric learning without any human annotations, especially discovering objects in 3D space.\n2.Using posed multi-views as supervision signals to learn Nerf is interesting to provide more information and constraints for object discovery.\n\nWeakness:\n1. The experiments and results are limited to two set of environments, would like to see more diverse datasets\n2. Lack of ablation experiments to show how each part of the model contribute to the overall performance", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposed a method to predict 3D object centeric neural field using single RGB image input. The framework presented in this paper aims to create Nerf for each object instead of previous methods' approach where a single Nerf is used for the whole scene. To generalize to the real-world scenes, the paper focuses on disentangling the learning of object intrinsics and the extrinsic separately. The proposed approach significantly improves systematic generalization, enabling unsupervised learning of high-fidelity object-centric scene representations from sparse real-world images. The approach allows for the discovery of visually rich objects from a single real image, allowing for applications such as 3D object segmentation and scene manipulation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The experiments and results are limited to two set of environments, would like to see more diverse datasets\n2. Lack of ablation experiments to show how each part of the model contribute to the overall performance", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gc7USVFR5X", "reviewer": "Reviewer_q61i", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n* Improved reconstruction quality. Position/coordinates matter.\n* Improved sample efficiency (training with 10 scenes produces reasonable results).\n* Evaluated on real datasets.\n* Ability to edit 3D scenes.\n* The method generalizes (spatially) with little-to-no post-training effort.\n* (A promise of) open-source code.\n\n\n**Weaknesses**:\n* The method requires pre-training on synthetic single-object datasets before it can be trained for multi-object scenes. I think that compared to other methods, this is a limitation. In addition, there are several heuristics employed that are required for the reported performance (e.g., “dropping distant samples from the predicted object positions after a few training epochs“, “we incorporate the depth ranking loss with pre-trained monocular depth estimators and background occlusion regularization to minimize common floating artifacts in few-shot”). It is a weakness because it makes the method more complex and more complicated to train; however, making things work is also an important feat in this field, so I wouldn’t say it is a significant weakness.\n* Small number of objects (trained on $K=4$ and tested with $K$ up to $10$). What are the considerations required for scaling up for more objects?\n* It seems that many recent methods for unsupervised object discovery are based on the (slot-) attention mechanism (and I appreciate the comparison with slot-attention in the appendix); however, the idea of separating to background and foreground, and the object latent to explicit attributes, such as position and appearance, is not new and dates back to patch/glimpse-based object-centric approaches (e.g., SPACE and SCALOR which you cited), but also apparent in recent methods ([4], [5]). It seems that while the community is converging to the attention-based approach, the ideas from the other object-centric families (patch/glimpse, particles) still prove to be crucial. This paper adopts the position attribute (and maybe future work will discover that other attributes, such as depth or scale, can also help in generalization), and a more transparent discussion on the comparison for the 2D object discovery literature would help highlight the contribution of this paper.\n\n**Minor**:\n* Equation 1: what exactly is $D^s$ (specifically, what is $s$)?\n* Equation 4: can you explain what $E^{\\text{abs}}$ is? To my understanding, it is the absolute normalized coordinates of all the $N$ patches. Also, in Eq. 5 (the matrix), you say that $h_1$ takes in variables of dimension 4, but to my understanding  $E^{\\text{abs}}$ is of dimension 2.\n* Related work:\n  * Since you are using DINO as a pre-trained feature extractor for object discovery, I believe a citation of DINOSAUR [1] is due.\n  * Missing (early) related work (not asking to directly compare with): [2], [3]\n  * Missing related work for “unsupervised object discovery”: [4], [5], [6], [7]\n  * In general, a recent survey [8] of object-centric representations has categorized the different families (not expecting to cite all the works from there of course).\n\n\n\n[1] Seitzer, Maximilian, et al. \"Bridging the Gap to Real-World Object-Centric Learning.\" The Eleventh International Conference on Learning Representations. (https://arxiv.org/abs/2209.14860)\n\n[2] Wang, Tianyu, Miaomiao Liu, and Kee Siong Ng. \"Spatially invariant unsupervised 3D object-centric learning and scene decomposition.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022. (https://arxiv.org/abs/2106.05607)\n\n[3] Henderson, Paul, and Christoph H. Lampert. \"Unsupervised object-centric video generation and decomposition in 3D.\" Advances in Neural Information Processing Systems 33 (2020): 3106-3117. (https://arxiv.org/abs/2007.06705)\n\n[4] Daniel, Tal, and Aviv Tamar. \"Unsupervised Image Representation Learning with Deep Latent Particles.\" International Conference on Machine Learning. PMLR, 2022.  (https://arxiv.org/abs/2205.15821)\n\n[5] Daniel, Tal, and Aviv Tamar. \"DDLP: Unsupervised Object-centric Video Prediction with Deep Dynamic Latent Particles.\" Transactions on Machine Learning Research. (https://arxiv.org/abs/2306.05957)\n\n[6] Löwe, Sindy, et al. \"Complex-Valued Autoencoders for Object Discovery.\" Transactions on Machine Learning Research. (https://arxiv.org/abs/2204.02075)\n\n[7] Gopalakrishnan, Anand, et al. \"Recurrent Complex-Weighted Autoencoders for Unsupervised Object Discovery.\" arXiv preprint arXiv:2405.17283 (2024). (https://arxiv.org/abs/2405.17283v3)\n\n[8] Villa-Vásquez, José-Fabian, and Marco Pedersoli. \"Unsupervised Object Discovery: A Comprehensive Survey and Unified Taxonomy.\" arXiv preprint arXiv:2411.00868 (2024). (https://arxiv.org/abs/2411.00868)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an algorithm for unsupervised 3D object discovery from a single image (while being trained with more views), unsupervised Object-centric Neural Fields (uOCF), where the object representation is translation-invariant. This is achieved by disentangling the object’s latent representation (of its shape and appearance) from its 3D location, which is treated as an additional attribute. This enhancement enables the model to generalize to unseen spatial configurations (e.g., different placements of objects). The model is evaluated on real and synthetic scenes, and the results demonstrate improved visual reconstructions, generalization capabilities and better sample efficiency than prior methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Page 5: you mention the process of “unprojection”. I believe this requires more explanation. E.g., you mention the utilization of a monocular depth estimation, where did this come from?\n* Test-time adaptation: you mention “It requires only a minimal single-image test-time optimization to adapt from one synthetic dataset to real-world images with unseen objects.” and “a fast test-time optimization using a photometric reconstruction loss on the input view only.”. I couldn’t find any details about that, could you please explain the process here?\n* Hyper-parameters: the loss weights (reported in the appendix) seem oddly specific–is there intuition or just a sweep to find optimal values?\n* Limitations: I appreciate the current discussion on limitations. I would appreciate an additional discussion (can be in the appendix) on the training stability, as slot-attention based methods are prone to training instability (e.g., with the same hyper-parameters, different runs might fail in separating objects to slots). How consistent are the results between different runs with the same hyper-parameters?\n* See “Weaknesses” and “Minor”.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I am happy to recommend acceptance of the paper as is, given the extensive improvements that authors made (shown in https://openreview.net/forum?id=ScEv13W2f1&noteId=f7YvkZ6Kia) to address the comments of the reviewers.
Decision: Accept (poster)