Neural Radiance Field Codebooks
Authors: Matthew Wallingford
Aditya Kusupati
Alex Fang
Vivek Ramanujan
Aniruddha Kembhavi
Roozbeh Mottaghi
Ali Farhadi
Venue: ICLR 2023 poster (2023)
Keywords: Object-Centric Representation Learning
Representation Learning
Neural Radiance Fields

Abstract: Compositional representations of the world are a promising step towards enabling high-level scene understanding and efficient transfer to downstream tasks. Learning such representations for complex scenes and tasks remains an open challenge. Towards this goal, we introduce Neural Radiance Field Codebooks (NRC), a scalable method for learning object-centric representations through novel view reconstruction. NRC learns to reconstruct scenes from novel views using a dictionary of object codes which are decoded through a volumetric renderer. This enables the discovery of reoccurring visual and geometric patterns across scenes which are transferable to downstream tasks. We show that NRC representations transfer well to object navigation in THOR, outperforming 2D and 3D representation learning methods by 3.1\% success rate. We demonstrate that our approach is able to perform unsupervised segmentation for more complex synthetic (THOR) and real scenes (NYU Depth) better than prior methods (.101 ARI). Finally, we show that NRC improves on the task of depth ordering by 5.5% accuracy in THOR.

Published as a conference paper at ICLR 2023
NEURAL RADIANCE FIELD CODEBOOKS
Matthew Wallingford' Aditya Kusupati', Alex Fang', Vivek Ramanujan', Aniruddha Kembhavi", Roozbeh Mottaghi', Ali Farhadi'
University of Washington; 2PRIOR, Allen Institute for AI
ABSTRACT
Compositional representations of the world are a promising step towards enabling high-level scene understanding and efficient transfer to downstream tasks. Learning such representations for complex scenes and tasks remains an open challenge. To- wards this goal, we introduce Neural Radiance Field Codebooks (NRC), scalable method for learning object-centric representations through novel view reconstrue- tion. NRC learns to reconstruct scenes from novel views using a dictionary of object codes which are decoded through a volumetric renderer. This enables the discovery of reoccurring visual and geometric patterns across scenes which are transferable to downstream tasks. We show that NRC representations transfer well object navigation in THOR, outperforming 2D and 3D representation learning methods by 3.1% % success rate. We demonstrate that our approach is able to perform unsupervised segmentation for more complex synthetic (THOR) and real scenes (NYU Depth) better than prior methods (29% relative improvement). Finally, we show that NRC improves on the task of depth ordering by 5.5% accuracy in THOR.
INTRODUCTION
Parsing the world at the abstraction of objects is a key characteristic of human perception and reasoning (Rosch et 1976, Johnson al., 2003). Such object-centric representations enable us to infer attributes such as geometry, affordances, and physical properties of objects solely from perception (Spelke 1990). For example, upon perceiving a cup for the first time one can easily infer how to grasp it, know that it is designed for holding liquid, and estimate the force needed to lift it. Learning such models of the world without explicit supervision remains an open challenge.
Unsupervised decomposition of the visual world into objects has been long-standing challenge (Shi & Malik 2000). More recent work focuses on reconstructing images from sparse encodings as an objective for learning object-centric representations (Burgess et al.) 2019) Greff et al. 2019 Locatello et al. 2020 Lin et al., 2020; Monnier et al. 2021] Smirnov et al., 2021). The intuition is that object encodings which map closely to the underlying structure of the data should provide the most accurate reconstruction given a limited encoding size. Such methods have shown to be effective at decomposing 2D games and simple synthetic scenes into their parts. However, they rely solely on color cues and do not scale to more complex datasets (Karazija et al.
2021 Papa et al. 2022).
2021) have enabled learning geometric representations of objects from 2D images, Recent work has leveraged scene reconstruction from different views as a source of supervision for learning object-centric representations (Stelzner et al. 2021 Yu et al. 2021b; Sajjadi et al.) 2022a Smith et al., 2022). However, such methods have few key limitations. The computational cost of rendering scenes grows linearly with the number of objects which inhibits scaling to more complex datasets. Additionally, the number of objects per scene fixed and fails to consider variable scene complexity. Finally, objects are decomposed on per scene basis, therefore semantic and geometric information is not shared across object categories.
Advances neural rendering (Mildenhall al. 2021 Yang et al.
With this consideration we introduce, Neural Radiance Codebooks (NRC). NRC learns a codebook of object categories which are composed to explain the appearance of 3D scenes from multiple views. By reconstructing scenes from different views NRC captures reoccurring geometric and visual patterns to form object categories. This learned representation can be used for segmentation well as geometry- based tasks such as object navigation and depth ordering. Furthermore, NRC resolves the limitations of current 3D object-centric methods. First, NRC's method for assigning object

---

Published as a conference paper at ICLR 2023
Code 1 (Couch)
Code (Floor)
Code 3 (Fridge)
Figure I: Visualization of learned codes. The NRC codebook encodes reoccurring geometric and visual patterns. In the top row, couches of differing appearance are grouped by geometric structure. In the middle row, different textured floors are categorized based on their shared planar geometry. In the bottom row, NRC learns correspondences between fridges from different views and scenes.
codes to regions of the image enables constant rendering compute whereas that other methods scales with number of objects. Second, we introduce a novel mechanism for differentiably adding new categories which allows the codebook scale with the complexity of the data, Last, modeling intra-category variation in conjunction with the codebook enables sharing of semantic and geometric object information across scenes.
We evaluate NRC on unsupervised segmentation, object navigation, and depth ordering. For seg- mentation on indoor scenes from ProcTHOR (Deitke et al., 2022) we show 29.4% relative ARI improvement compared to current 3D object-centric methods (teelnner al. 2021 Yu al. 202 On world images (NYU Depth (Silberman et al. 2012h) we show promising qualitative results (Figure and 29% relative improvement. For object navigation and depth ordering, where geo- metric understanding relevant, we observe 3. % improvement in navigation success rate 5.5% improvement in depth ordering accuracy over comparable self self-supervised and object-centrie methods. Interestingly, we find qualitative evidence that the learned codes categorize objects by both visual appearance and geometric structure (Figure TD.
RELATED WORK
Object-Centric Learning Object-centric learning aims to build compositional models the world from building blocks which share meaningful properties and regularities across scenes. Prior works
2

---

Published as a conference paper at ICLR 2023
such MONet (Burgess et al. 2019,, IODINE Greff et al. 2019), Slot Attention (Locatello et al.) 2020), and Monnier et al. (0021) have demonstrated the potential for disentangling objects from images. Other work has shown the ability to decompose videos (Kabra et al., 2021 Kipt et al. 2021). In particular, Marionette (Smirnov et al. 2021) learns a shared dictionary for decomposing scenes of 2D sprites. We draw inspiration from MarioNette for learning codebooks, but differ in that we model the image formation process and intra-code variation, and dynamically add codes to our dictionary.
3D Object-Centric Learning Recent work has shown novel view reconstruction to be a promising approach for disentangling object representations, uORF (Yu et al. 2021b) and ObSuRF (Stelzner et al., 2021) combine Slot Attention with Neural Radiance Fields (Mildenhall et al. 2021 to
decompose scenes. COLF (Smith al. 2022 replaces the volumetrie renderer with light fields
to improve computational efficiency. NeRF-SOS Fan et al. (2022) uses contrastive loss for both geometry and appearance to perform object segmentation. SRT (Sajjadi et al. (0026)) encodes scenes into set of latent vectors which are used to condition light field. OSRT (Saijadi et al. 2022a)
extends SRT by explicitly assigning regions of the image to latent vectors. NeSF Vora et al. (202T) learns to perform 3D object segmentation using NeRF with 2D supervision. Although great progress has been made, these methods are limited to synthetic and relatively simple scenes. Our work differs from previous 3D object-centric works in that we learn reoccurring object codes across scenes and explicitly localize the learned codes. Additionally, our method can model an unbounded number of objects per scene compared to prior work which fixes this eyper-parameter priori. We show that approach generalizes to more complex synthetic and real-world scenes.
Neural Rendering Advances in neural rendering, in particular Neural Radiance Fields
(NeRF) (Mildenhall et al. 2021), have enabled host of new applications (Jang & Agapito 2021
Mildenhall et al. 2022 Pumarola et al. 2021] Park et al. [022 Lazova et al.) 2022] Niemeyer &
Geiger 2021 Zhi al.
2021). NeRF differentiably renders novel views of scene by optimizing a
continuous volumetric scene function given a sparse set of input views. Original formulation of NeRF
learned one representation for each scene; other works (Yu et al. 2021a [Jain et al. 2021 Kosiorek
et al. 2021 showed conditioning NeRFs on images enables generation of novel views of new scenes.
Dictionary/Codebook Learning Dictionary (codebook) learning OOshhussn & Field 1997) in- volves learning of a specific set of atoms or codes that potentially form a basis and span the input space through sparse combinations. Codebooks have been widely used for generative and discrimina-
tive tasks across vision (Elad& & Aharon 2006; Mairal et al. 2008). NLP (Mcauliffe & Blei 2007) and signal processing (Huang & Aviyente 2006). Learning sparse representations based codes enables large-scale methods which rely on latent representations. More recently, codebooks have been shown to be crucial in scaling discrete representation learning (Van Den Oord et al., 2017] Kusupati et al. 2021). Marionette (Smirnov et al., (021) is an object-centric representation learning method that relies on codebooks, unlike most other methods that are developed around set latent representations (Sajjadi et al.] [022a]bb Locatello et al.] 2020 Yu et al.) 20211)). Object-centric codebooks help semantic grounding for transfer between category instances and are important for large-scale representation learning across diverse scenes and objects.
METHOD
Our goal is to discover object categories without supervision, learn priors over their geometry and visual appearance, and model the variation between instances belonging to each group. Given multiple views of a scene, the objective is to explain all views of the scene given a set of object-codes. This learned decomposition can be used for segmentation and other downstream tasks that require semantic and geometric understanding such as depth ordering and object-navigation.
Figure 2 illlstratee the training pipeline. We begin by processing the image through convolutional network to obtain a spatial feature map. The feature map is then projected to a novel view using the relative camera matrix. Feature vectors from each respective region of the image are assigned to categorical latent codes from the finite-size codebook. The object codes and feature vectors are passed to convolutional network which transform the categorical codes to fineggrined instance codes. A volumetric renderer is then conditioned on the instance code, view direction, and positional encodings to render each region of the scene from the novel view. The rendered image from the
3

---

Published as a conference paper at ICLR 2023
Input View (f))
Novel View (A)
2 112
Fg
Variation Module
(g 12 we
A
MLP
05
Ha
Render (RGBr)
T
4
2. ellly-12
Positional
Codebook
Encoding
Code Assignment
An overview of NRC. We learn set of shared codes for decomposing scenes into objects. Each point the scene assigned one of n latent codes from the codebook. The variation module models the intra-code variation between objects by perturbing the code in latent space. A conditional NERF model renders the scene and is compared to the ground truth novel view for supervision.
Figure 2:
novel view is compared to the ground truth using L2 pixel loss. The categorical codes, assignment mechanism, and volumetric renderer are learned jointly in an end-to-end fashion.
Image Encoding and Camera Projection Given input frame and novel image, In I E Hxw respectively from scene Sn. we first encode In into a spatial feature map, fx E Rdx "/ex//s, using a convolutional network, Fg. We project each point, (I, y, z), in world co- ordinates of the novel view camera coordinates in the input frame, (z, y). using the relative camera pose. Given (z, y), we select the spatial feature fit E Rd from the patch that contains the projected coordinates. The spatial features vectors are then passed to the next stage where they are assigned to categorical object-codes.
Assigning + Learning Codes Our goal is to jointly learn a shared set of object categories and priors about their appearance and geometry. By mapping the spatial features from continuous vector space discrete, finite set of codes the model incentivized find reoccurring patterns in the images.
Given the features for point in the novel view, f assign code, l", chosen from the shared codebook, IL. We do so with an arg max n-nearest-neighbors during inference:
e 1111-/5-116 1/2
(STF) arg max
2 (x, v)
4s (E(M)
Σ 1= 114 11
The nearest-neighbor assignment used during inference is non- non- con-differentiable operation therefore propagating gradients to the encoder and codebook would be possible. To enable learning of the codebook elements we use a softmax relaxation of gearest-neighhors during the backward pass in conjunction with the trrighh-throug---stimtor (STE) Bengio et al. (2013):
14 1
Black (x, y)
Σ - 111--115
The number of codes should depend the complexity of the scenes they model. Learning when to add new codes is non-trivial because the number and selection of codes discrete and non-differentiable. To circumvent this problem, we a series of step functions with a haaighht-throu---ssmmmaa (STE) to sequentially add elements to the codebook. Each code gated according to the following:
Adding Categorical Codes
4
4 + TT ((/-/////)))))))))
4; T(a, = 1. a>t
(3)
0. stt
4

---

Published as a conference paper at ICLR 2023
T()) is a binarization function in the forward pass and lets the gradients pass through using STE in the back pass. (-) is the sigmoid function, À is a scaling hyperparameter, and s is a learnable scoring parameter whose magnitude is correlated with the overall capacity (number of codes) required to model the scenes accurately. A new code 1, is added when 8 exceeds the threshold i² /X. New codes initialized using standard normal distribution. Throughout training we keep k + 1 total codes where k is the current number of learnable codes. The extra code is used by the straight-throughe estimator to optimize for s on the backward pass. This formulation can be viewed as the discrete
analog of a gaussian prior over the number of elements, i in the codebook: P(k) = e "
Modeling Intra-Code Variation Once a categorical code has been assigned to region in the novel frame, the model must account for variation across instances. We model this variation in latent space using an encoder that takes in both the spatial feature, f and the categorical code 1" (ar, y). We rescale the norm of the variation vector by hyperparameter, to ensure the instance and categorical codes are close in latent space. The instance code is formulated as the following:
Go (1I" v). 17 Insaaace (z. v) (x, +e. ||G (I, y). fh. fit ")1l2"
(4)
We concatenate 1/"" and 1" (xx as input to the variation module, Gø, which we model as 3- layer convolutional network. Gø provides a d dimensional perturbation vector which models the intra-category variation and transforms the categorical code to an instance level code.
Decoding and Rendering Given the localized instance codes for a scene, we render it in the novel view and compare with the ground truth using L2 pixel loss. Intuitively, object categories which encode geometric and visual patterns should render the scene more accurately from novel views. Each region of the scene is rendered using an MLP conditioned on the instance codes and the volumetric rendering equation following the convention of NeRF Mildenhall et al. (202I):
Hg (Finstance (x., v): P. d) = (c,
(5)
Here p= (z. y, 2) is a coordinate in the scene, d Ra is view direction, c is the RGB value at p in the direction of cl and a is the volume density at that point. Recall that (I, 2) corresponds to (xr, y) in the input frame Tr. We can project (ar, y. into the camera coordinates of the novel view T', to get (a). This pixel (x', yy) in the novel view corresponds to (I, y) in the input frame, meaning they represent the same point in world coordinates. To get an RGB value for (Ir y). we use volume rendering along the ray from camera view In. into the scene, given by
C(r)= = ['' T(t) o(t) c(t) dt,
(6)
where T(1) exp (- (ss) ds) models absorbance and in and b are the near and far field.
Given target view with pose P. the ray to the target camera is given by r(t) + t d where d is unit direction vector which passes through (z, y). The volume rendering for particular pixel occurs along this ray. Let d' be the direction associated with the novel view (x', x²) and r'(1) = d'. The pixel intensity (a', ./) given by Ć' C(r') and our final loss
L(Tn, Th x. u) ||C² - T/( , 2l) +8,
where (x,, 1/) //)', is the ground pixel value at (x', v'). We penalize the scoring parameter, from section (i the loss to encourage learning a minimal number of codes.
NRC for Downstream Tasks Once the encoder, codebook, and MLP have been trained, evaluate the learned representation on various downstream tasks. To perform segmentation, we process each image through the trained encoder, Fo. to obtain the spatial feature map. Each feature vector, fzy in the spatial map is assigned to the nearest categorical code, I* in the learned codebook. The categorical codes are then designated to the corresponding pixel obtain a segmentation mask.
Traditionally in object navigation, frames from the embodied agent are processed by a frozen, pretrained network. The resulting feature vector is then passed to policy network which chooses an
5

---

Published as a conference paper at ICLR 2023
Input Image
Ground Truth
NRC
Figure 3: Unsupervised segmentation of real-world images. NRC segments scenes that have significant object category overlap with ProcTHOR. We show the first results for object-centric unsupervised segmentation of real-world scenes.
action. To assess the utility of the NRC representation, we replace the pretrained network with the NRC encoder and codebook. We process each frame obtain instance codes for each region of the image which are then fed to the policy network.
Depth ordering task consists of predicting which two objects is closer to the camera. To perform depth ordering with NRC we predict a segmentation mask and depth map. To predict the depth map we condition the trained MLP on the instance codes predict the density, a, along a given ray. We estimate the transmittance to predict the depth following the method of Yu et al. (2021a). Depth map and segmentation mask are combined to predict the average distance of each object from the camera.
4 EXPERIMENTS
We evaluate decomposition and representations several downstream tasks: unsupervised segmentation (real and synthetic), object navigation, and depth ordering. NRC shows improvement over baseline methods all three tasks. Prior works in object-centric learning have focused on unsupervised segmentation for measuring the quality of their decomposition. We show that NRC representations are also effective for downstream applications that require geometric and semantic understanding of scenes such as object navigation and depth ordering.
4.1 DATASETS
ProcTHOR & RoboTHOR THOR (Kolve et al. 2017) consists of interactive home environments built in the Unity game engine. We benchmark on the task of object navigation in RoboTHOR (Deitke al. 2020), variant of the THOR environment aimed sim2real benchmarking. Object navigation consists of an agent moving through different scenes to locate specified objects. RoboTHOR consists of 89 indoor scenes split between train, validation, and test. ProcTHOR (Deitke al. 2022) consists of procedurally generated indoor scenes similar to RoboTHOR, Examples of THOR scenes can be found in Appendix
CLEVR-3D CLEVR-3D (Johnson et al. 2017) is a synthetic dataset consisting of geometric primitives from multiple views and is used for unsupervised segmentation. Following the convention Stelzzer et al. (202T), we test on the first 500 scenes of the validation set and report foreground- adjusted random index (FG-ARI). Adjusted random index (ARI) Yeung & Ruzzo (2001) measures

---

Published as a conference paper at ICLR 2023
the agreement between two clusterings and is standard metric for unsupervised segmentation. In our case the two clusterings are the predicted and ground truth segmentations. Foreground adjusted random index only measures the ARI for pixels belonging to foreground objects. For comparison to prior works, we consider segmentations at both the class and instance level to be correct for CLEVR-3D, ProcTTOOR, and NYU Depth. Further details can be found in Appendix [
NYU Depth The NYU Depth Dataset (Silberman et al. 2012) consists of images from real-world indoor scenes accompanied by depth and segmentation maps. Methods are trained on the Proc dataset then evaluated on NYU Depth for segmentation. We chose NYU Depth because it has object categories and scene layouts that are similar to THOR. We report the adjusted random index (ARI).
Table 1:
Segmentation results (ARI) for NRC and comparable methods. We find that for more complex datasets, ProCTHOR and NYU Depth, NRC outperforms other methods.
Method
ProRTHOR (ART) NYU Depth (ARI) CLEVR-3D (FG-ARI)
MarioNette
.127
.035
uORF
.193
.115
.962
ObSuRF
.228
.141
.978
NRC
.295
.182
.977
4.2 UNSUPERVISED SEGMENTATION
Experimental Setup We evaluate NRC, ObSuRF, uORF, and MarioNette for unsupervised seg- mentation on ProcTHOR, CLEVR-3D, and NYU Depth. We compare with MarioNette because uses similar code mechanism for reconstruction. We report FG-ART on CLEVR-3D for comparison to prior works and ARI on the other datasets. For NYU Depth evaluation we use the representations trained on ProcTHOR and only consider classes that are seen in the training dataset.
Results We find that for NYU Depth and ProcTHOR which have more complex layouts and object diversity, NRC significantly outperforms other methods (Table Figure [ shows examples of the object codes learned by ProcTHÓR and Figure 3 shows segmentation examples of real-world images. To knowledge, this is the first object-centric method which has shown unsupervised segmentation results for complex real-world images. We find that NRC categorizes similar objects across scenes based on both geometry and visual appearance. In the top row of Figure [ we find that couches of similar shape are assigned to the same code despite differing visual appearance which indicates that NRC codes capture geometric similarity. In the same figure, we show further examples where floors of different texture are categorized by the same code. The improved segmentation performance and geometric categorization of objects indicates that NRC can leverage more than simple color cues which has been significant limitation for object-centric learning.
4.3 OBJECT NAVIGATION
Experimental Setup We design the object navigation experiments THOR to understand how well the learned representations transfer from observational data to embodied navigation (Anderson cet al., 2018; Batra et al. 2020). Object navigation consists of an embodied agent with the goal of moving through indoor scenes to specified objects. The agent can rotate its camera and move in discrete directions. At each step, agent is fed with the current RGB frame relayed by the camera.
For the representation learning component of the experiment collect observational video data from a heuristic planner (Appendix [B). which walks through procedurally generated ProcTHOR scenes. In total, the dataset consists of 1.5 million video frames from 500 indoor scenes. For further dataset details and example videos see Appendix
After training on ProcThor videos, we freeze the visual representations following standard prac- tice (Khandelwal et al. 2022). We train a policy using DD-PPO (Wijmans et al. 2019) for 200M steps the training set of RoboTHOR then evaluate on the test set. We report success rate (SR) and success weighted by path length (SPL). Success is defined as the agent signaling the stop action
E, within I meter of the goal object with it in the view. SPL is defined as & Σ S max where I is the shortest possible path, p3 is the taken path, & S is the binary indicator of success for episode i.
7

---

Experimental Setup We evaluate NRC on the task of ordering objects based on their depth from the camera. Understanding the relative depth of objects requires both geometric and semantic understanding of scene. For this task we evaluate on the ProcTHOR test dataset which provides Published as a conference paper at ICLR 2023
Table Results for object navigation on RoboTHOR object navigation. Visual representations are trained on observations from 500 scenes of ProcThor. A policy is learned on top of the frozen visual representations by training on the object navigation task in RoboTHOR training scenes. The results are obtained by evaluating on RoboTHOR test scenes.
Method
Success Rate (%)
SPL
uORF Yu et al.
2021
31.3
.146
ImageNet Pretramnnng
33.4
.150
ObSuRF (Stelzner et al. 2021
38.9
.167
Video MoCo (Feichtenhioter et al. 2021
43.9
.184
EmbCLIP (Khandelwal et al. 2022)
47.0
.200
NRC (Ours)
50.1
.239
We compare with the following baselines: ObSuRF, uORF, Video MoCo (Feichtenhofer et al. 2021 and EmbedCLIP (Khandelwal et al. 2022). ObSuRF and uORF are 3D, object-centric methods, and Video MoCo a contrastive video representation learning method. We include Video MoCo for comparison it was designed for large-scale, discriminative tasks, while ObSuRF and uORF were primarily intended for segmentation. See Appendix for further implementation details of baselines.
Results shows the performance of NRC and baselines on RoboTHOR object navigation. NRC outperforms the best baseline by 3% in success rate (SR) and by 20% relative improvement in SPL. These performance gains indicate that the geometrically-awar aware NRC representation provide an advantage over traditional representations such EmbCLIP and Video MoCo. Another key observation is that NRC has at least 18.8% improvement SR and relative SPL over the recent object-centric learning methods, uORF and ObSuRF. In particular, NRC performs better than ObSuRF and uORF when navigating near furniture and other immovable objects (see supplemental for object navigation videos). We hypothesize this is due NRC's more precise localization of objects.
4.4 DEPTH ORDERING
dense depth and segmentation maps. Following the convention Ehsani et al. (2018), we determine ground truth depth of each object by computing the mean depth of all pixels associated with its ground truth segmentation mask.
For evaluation, select pairs of objects in a scene and the goal is predict which object is closer. We take the segmentation that has the largest IoU with the ground truth mask as the predicted object mask. To determine the predicted object depth, associated with the predicted object mask. All representations are trained on the ProcT'HOR dataset and evaluated the ProcTHOR test set. In total evaluate 2,000 object pairs.
compute the mean predicted depth of cach pixel
Table 3:
We compare depth ordering results on RoboTHOR with other geometrically-awar represen- tations. Given pairs of objects in the scene, the model must infer which object is closer. We report the accuracy as the number of correct orderings over the total number of object pairs.
Method
Depth Order Acc. (%)
uORF Yu et al. 2021b
13.5
ObSuRF Stelzner et al. 2021 NRC (Ours)
18.3
23,8
Depth ordering requires accurate segmentation and depth estimation. Due to NRC's stronger segmentation performance and better depth estimation, we 5.5% and 10.3% depth ordering accuracy compared to ObSuRF and uORF respectively. The fine-grained localization of categorical latent codes in NRC allows for better depth ordering over existing object-centric methods. In particular, the other object-centric methods tend to assign object instances to the background which leads to large errors in estimating the depth of objects (Figure
Results
8

---

Published as a conference paper at ICLR 2023
4.5 ABLATION STUDY
Table 4: Ablation study for modeling the intra-code variation and learning the number of codes
evaluated on unsupervised segmentation in ProCTHOR. Default fixed number of codes is set to 25.
Method + (Ablation)
ProcTHOR (ARI)
NRC
.182
NRC +
Learned # of Codes
.197
NRC + Variation Module
.284
NRC + Variation Module + Learned # of Codes
.295
We present an ablation study on the ProcTHOR dataset determine the effect of the variation module and learnable codebook size on unsupervised segmentation performance. Quantitative results can be found in Table 4 We observe that performance improves by 9% when intra-class variation explicitly modeled. Intuitively, allowing for small variation between instances of the same category should lead to better representations and allow for greater expressiveness.
We also find that learning the number of codes moderately improves performance. However, if tuning, performance is matched (Table Nonetheless,
number of codes is found via differentiably learning the codebook size avoids computationally expensive hyper-parana tuning.
LIMITATIONS
Novel view reconstruction requires camera pose, which is not available for most images and videos. Some datasets such as Ego4D (Grauman et al., 2022) provide data from inertial measurement units that can be used to approximate camera pose, although this approach is prone to drift.
An incorrect assumption that NRC and most object-centric prior works make that scenes are static. However, it rarely is the case that scenes are free of movement due to the physical dynamics of our world. Recently, Kipf et al. (2021) Pumarola et al. (2021) made strides in learning representations from dynamic scenes.
Although NRC is relatively efficient compared to the other NeRF based methods, the NeRF sampling
procedure is compute and memory intensive. (Sajjadi et al. (2022a) and Smith et al. 2022 leverage object-centric light fields to reduce memory and compute costs. The efficiency improvement from modeling scenes as light fields is orthogonal NRC and can be combined.
final challenge inherent to novel view reconstruction is finding appropriate corresponding frames of videos. For example, if two subsequent frames differ by 60" rotation of the camera, then most of the scene in the subsequent frame will be completely new. Therefore, constructing the content in the novel view is ill-posed. Pairing frames with overlapping frustums is potential solution, although the content of the scene may not be contained in the intersecting volume of the frustums.
CONCLUSION
Compositional, object-centric understanding of the world is fundamental characteristic of human vision and such representations have the potential to enable high-level reasoning and efficient transfer to downstream tasks. Towards this goal, we presented Neural Radiance Field Codebooks (NRC), new approach for learning geometry-aware object-centric representations through novel view reconstruction. By jointly learning a shared dictionary of object codes through a differentiable renderer and explicitly localizing object codes within the scene, NRC finds reoccurring geometric and visual similarities to form objects categories. Through experiments, we show that NRC representations improve performance on object navigation and depth ordering compared to strong baselines by 3. % success rate and 5.5% accuracy respectively. Additionally, we find our method is capable of scaling to complex scenes with more objects and greater diversity. NRC shows relative ARI improvement over baselines for unsupervised segmentation by 29.4% ProcTHOR and 29.0% on NYU Depth. Qualitatively, NRC representations trained on synthetic data from ProcTHE THoC show reasonable transfer real-world scenes from NYU Depth.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank Kuo-Hao Zeng for technical advice regarding the THOR environment, Reza Salehi and Vishwas Sathish for helpful discussion, and Koven Yu and Mehdi Sajjadi for prompt correspondence regarding details of their methods. This work is in part supported by NSF IIS 1652052, IIS 17303 166, DARPA N66001-19 -99---1-9-- DARPA W911NF-15-1-06 and gifts from Allen Institute for Artificial Intelligence and Google.
REFERENCES
Peter Anderson, Angel Chang. Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint wwXiv:180.006733 2018.
Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects. arXiv preprint arXiv:2006. 13171, 2020.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville, Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308. 3432, 2013.
Christopher P Burgess, Loie Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.1133: 2019.
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, et al. Robothor: An open simulation-to-rcal embodied ai platform. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3164-3174, 2020.
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, et al. Procthor: Large-scale embodied ai using procedural generation. arXiv preprint arXiv:2206.069// 2022.
Kiana Ehsani, Roozbeh Mottaghi, and Ali Farhadi. Segan: Segmenting and generating the invisible. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6144-6153, 2018.
Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image processing, 15(12):3736-3755 2006.
Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any- view self-supervised object segmentation complex scenes, arXiv preprint arXiv:2209.08776 2022.
Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 3299-3309, 2021.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pp. 2424-2438 PMLR, 2019.
Ke Huang and Selin Aviyente. Sparse representation for signal classification. Advances in neural information processing systems, 19, 2006.
10

---

Published as a conference paper at ICLR 2023
Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5885-5894, 2021.
Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12949-12958, 2021.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Scott P Johnson, Dima Amso, and Jonathan A Slemmer. Development of object concepts in infancy: Evidence for early learning an eye-tracking paradigm. Proceedings of the National Academy of 2003.
Sciences,
Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matt Botvinick, Alexander Lerchner, and Chris Burgess. Simone: View-invariant, temporally-abstrace object representations via unsupervised video decomposition. Advances in Neural Information Processing Systems, 34:20146 20159, 2021.
Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevrtex: texture-rich benchmark for unsupervised multi-object segmentation. arXiv preprint arXiv:2111.1026; 2021.
Apoory Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: Clip embeddings for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14829-14838 2022.
Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. arXiv preprint arXiv:2111.12594 2021.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:
2017.
Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokrá, and Danilo Jimenez Rezende. Nerf-vae: geometry aware 3d scene generative model. In International Conference Machine Learning, pp. 5742-5752. PMLR, 2021.
Aditya Kusupati, Matthew Wallingford, Vivek Ramanujan, Raghav Somani, Jae Sung Park, Krishna Pillutla, Prateek Jain, Sham Kakade, and Ali Farhadi. Lle: Accurate, multi-purpose learnt low- dimensional binary codes. Advances in Neural Information Processing Systems, 012233000229991 2021.
Verica Lazova, Vladimir Guzov, Kyle Olszewski, Sergey Tulyakov, and Gerard Pons-Moll. Control-nerf: Editable feature volumes for scene rendering and manipulation. arXiv preprint arXiv:2204. 10850, 2022.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint arXiv:2001.024000 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:1 33:1 00:11525-1133 11538, 2020.
J Mairal, Bach, Ponce, G Sapiro, and A Zisserman. Learning discriminative dictionaries for local image analysis. In 26th IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8, 2008.
Jon Mcauliffe and David Blei. Supervised topic models. Advances in neural information processing systems, 20, 2007.
11

---

Published as a conference paper at ICLR 2023
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Ben Mildenhall, Peter Hedman, Ricardo Martin Baatln,, Pratul Srinivasan, and Jonathan Barron. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattrn Recognition, pp. 16190-16199, 2022.
Tom Monnier, Elliot Vincent, Jean Ponce, and Mathieu Aubry. Unsupervised layered image decom- position into object prototypes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11453-11464, 2021.
Bruno A Olshausen and David Field. Sparse coding with overcomplete basis A strategy employed by v1? Vision research, 37(23):331 11133311-335,,,, 1997.
Samuele Papa, Ole Winther, and Andrea Dittadi. Inductive biases for object- representations in the presence of complex textures. In UAI 2022 Workshop on Causal Representation Learning, 2022.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5865-5874, 2021.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10318-10327, 2021.
Eleanor Rosch, Carolyn B Mervis, Wayne D Gray, David M Johnson, and Penny Boyes-Braes Basic objects in natural categories. Cognitive psychology, 8(3):382 439,, 1976.
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučic, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object scene representation 2022a.
transformer. arXiv preprint arXiv:2206.0699
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučic, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6229-6238. 2022b.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattemn analysis and machine intelligence, 22(8):888-990) 00088-9050200 2000.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European conference computer vision. pp. 746-760. Springer, 2012.
Dmitriy Smimnv, Michaël Gharbi, Matthew Fisher, Vitor Guizilini, Alexei A. Efros, and Justin Solomon. MarioNette: Self-supervised sprite learning. In Advances in Neural Information Processing Systems, 2021.
Cameron Smith, Hong-Xing Yu, Sergey Zakharov, Fredo Durand, Joshua Tenenbaum, Jiajun Wu, and Vincent Sitzmann. Unsupervised discovery and composition of object light fields. arXiv preprint arXiv:2205.0392 2022.
Elizabeth S Spelke. Principles of object perception. Cognitive science, 14(1):29-56, 1990.
Karl Stelzner, Kristian Kersting, and Adam Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation, arXiv preprint 1r4iv:2 /48, 2021.
12

---

Published as a conference paper at ICLR 2023
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi SM Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes, arXiv preprint arXiv:2111 1...:22, 2021.
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-pes pear-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arXiv: 2019.
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning pojeec-compositionaa neural radiance field for editable scene rendering. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 13779-13788, 2021.
Ka Yee Yeung and Walter L. Ruzzo. Details of the adjusted rand index and clustering algorithms supplement the paper " an empirical study on principal component analysis for clustering gene expression data to appear in bioinformatics ). 2001.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4578 4487, 2021a.
Hong-Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. arXiv preprint arXiv:2107.07905, 2021
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison. In-place scene la- belling and understanding with implicit scene representation. Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 15838-158477 2021.
13

---

Published as a conference paper at ICLR 2023
A FURTHER EXPERIMENTS AND QUALITATIVE EXAMPLES
A.1 MORE COMPLEX REAL-WORLD SCENES (NYU-DEPTI))
We provide more segmentation examples of NRC on cluttered, real-world scenes (figure We find that NRC reasonably segments categories that overlap with those of THOR. Additionally we evaluate NRC and comparable methods on more difficult scenes of NYU Depth to see how complexity of the scene affects performance. Specifically we filter for scenes with 5 or more objects in them for evaluation on unsupervised segmentation. In general performance degrades (Ta though NRC performance decreases less compared to uORF and ObSuRF.
Image
Ground Truth
Predicted
Figure 4: Segmentation examples of cluttered scenes. Further segmentation examples of cluttered, real-world scenes from NYU Depth. NRC scales better to cluttered scenes compared other methods as the rendering and memory cost is constant the number of objects.
A.2 MATTERPORT-3D AND GIBSON EXPERIMENTS
We train NRC on scenes from MatterPort3D. Qualitative examples can be found in figure 6 and quantitative results be found in Table We find NRC performs slightly worse on MatterPort3D compared to on THOR likely due to the greater visual complexity. This is reinforced by the evidence that the learned codebook size is 67 whereas for THOR it is on average learned codes indicates that NRC requires more expressivity to accurately reconstruct Matterport3D scenes. For a given scene we find corresponding frames by filtering for images within 20 degrees viewing angle of each other.
53. greater number of
We evaluate NRC and ObSuRF on the Habitat Point Navigation Challenge. We train NRC and ObSuRF on data collected from random walks through the Habitat environment. We use ResNet-50 initialized with a MoCo backbone pretrained on ImageNet. We report success rate, shortest-pan path length, and distance to goal. Qualitative examples can be found in the supplemental. We find that NRC outperforms ObSuRF and ImageNet pretraining. Qualitatively we find that NRC is better at navigating around objects compared to ObSukF.
14

---

Published as a conference paper at ICLR 2023
Figure Predicted object depth maps for depth ordering. The predicted object masks are overlayed with the predicted depth map. We average the per pixel depth for each predicted object mask to infer which object closer. NRC's object localization leads to sharp object masks and less error when predicting average object depth.
B IMPLEMENTATION DETAILS
B. NRC
For the segmentation experiments decoder we use 3-layer with hidden dimension of 512. We train with learning rate of le 4 with the ADAM optimizer. We set the near field and far field to 5.5. We train from scratch, and from the video MoCo model trained on ProRTHOR. We found starting from the pretrained initialization reduced the number of training epochs required for convergence. For the model from scratch we train for 300 epochs. For the model from MoCo initialization we train for 100 epochs. While training we use sliding window of 5 frames determine corresponding images ProcTHOR. By corresponding images we mean the image given to the model and the ground truth novel view image. Given a sliding window of n frames we randomly select 2 images from this interval. We ablate over 2, 5, and 10 frames. We found 5 images led to the best segmentation results and therefore used that window size for nivicttnavigation.
the modified ResNet34 used by Yu et al. (2021b). For the
a ResNet50 for fair comparison to CLIP and Video MoCo. We train wit the same syper-parameter as discussed above for segmentation with the only change being the architecture. We train for 200M steps using PPO with default typer-parameters match Khandelwal et al. 2022) in RoboTHOR.
For
Table 5: Performance cluttered scenes of NYU Depth. We filter for scenes with 5 more objects and report the ARI NRC and comparable methods. NRC outperforms ObSuRF and uORF particularly on cluttered scenes, it can support unbounded number of objects.
Method
NYU Depth (Hard)
uORF
.046
ObSuRF
.092
NRC
.139
15

---

Published as a conference paper at ICLR 2023
Image
Ground Truth
Predicted
Figure 6: Segmentation examples for MatterPort3D. Segmentation examples on MatterPort3D, We train NRC on 190,000 images of MatterPort3d and learn 67 categorical codes. This increase in the learned number of codes is likely due to the increased visual complexity of MatterPort3D compared THOR. We find similar segmentation results to those THOR and NYU Depth.
B.2 UORF
We use the implementation available at the training of ProcTHOR which contains ~ 1.5 million images. We train three different variations of the uORF no GAN model: from scratch, fine-tune with encoder and decoder pre-trained on CLEVR-567, and fine-tune with all three components pre-trained on CLEVR-567. For variations that train the slot attention component from scratch, we try setting the number of slots to 8 (default) and 16. We train for 10 epochs when loading from a pre-trained model, and for 15 epochs when training from scratch; roughly half of the epochs are coarse epochs, and we set percept_ in to roughly one third of the epochs. We train for less epochs because our dataset much larger. Our images within scene batch size is 4, and we turn off shuffling to ensure that batches contain images that are close together within the scene. We pass in the fixed locality parameter, but also set the decoder's locality parameter to false. To compensate between differences between dataset and CLEVR-567, tune nss scale, object scale, near plane, and far plane. We train models on single NVIDIA A40.
com http:////////ee/////eeeee We train on
B.3 OBSURF
We use the official implementation provided by Stelzner et al. (2021) at bttpp:://ithub com/ stelznez obsurf We trained 3 variations of the ObSuRF model: onfrom scratch, from CLEVR-
Table 6: Unsupervised segmentation performance of NRC and ObSuRF MatterPort3D. We find NRC performs slightly worse on MatterPort3D compared to on THOR likely due to the greater visual complexity. This is reinforced by the observation that NRC leams 67 codes for MatterPort3D whereas 53 are learned for THOR.
Method
MatterPort3D
ObSuRF
.178
NRC
.237
16

---

Published as a conference paper at ICLR 2023
Table Habitat Point Navigation Challenge. The performance of various visual encoders on the point navigation challenge. We find that NRC outperforms other 3D-object centric methods due to its ability scale more objects and visually complex environments.
Method
SPL
SR
Goal Dist
ImageNet Pretrain
.82
.94
.73
ObSuRF
.77
.84
.81
NRC
.85
.96
.68
3D pretrained weights, and from MultiShapeNNet pretrained weights. For the pretrained models lowered the learning rate by a factor of 10 to le-5. 5. For training from scratch we used the default learning rate of le 4. For the number of slots tried the default number, 5, and 10. We report the results of the best model which was from scratch with 10 slots. For object navigation experiments in RobOTHOR,
used the model with the highest ARI which was training from scratch with 10 slots. We learn a policy network comprised of l-layer GRU with 512 hidden units which maps to a dimensional logit and scalar which are used for the actor-critic. We train with DD-PPO for 200M steps. For the encoder, we use ResNet-50 architecture.
To construct the feature vector fed the GRU we the output of the ResNet-50 which 2048 7x7, We pass this through a 2-convolutional layers to sample down to 32 x 7 7 then concatenate this with the trainable goal state embedding matrix which 32x7x7. This visual feature vector of shape 64 x 7 x 7 is then passed through another 2-layer convolutional network to be of size 32 x x 7.
B.4 OSRT
We do not compare with OSRT (Sajaadi al. (2022a) on data sets that were not evaluated in the original paper. We contacted the authors, but public implementation has not been released yet at the time of this work.
C DATASETS
C.I PROCTHOR
The ProcTHOR dataset Deitke et al. (2022) consists of 10,000 procedurally generated indoor rooms in the THOR environment. For our dataset we collect 10 video sequences of 300 actions from 500 randomly chosen training scenes. Each actions consisting of either rotation or stepping in specified direction. Actions are determined by a heuristic planner which moves throughout the scene. total we collect 1.5 millions image frames. See Figure for a sample of the dataset.
For evaluation of ARI we consider the 20 largest objects, floor, and ceiling. uORF and ObSuRF decompose scenes into 5-10 objects and therefore do not handle scenes with many small objects. We evaluate ARI only on the 20 largest objects THOR which are the following: (ArmChair, Bathtub, BathtubBasin, Bed, Cabinet, Drawer, Dresser, Chair, CounterTop, Curtains, Desk, Desktop, TableeTable, Dining Table, SideTable, Floor, Fridge, Television, TVStand, Toilet.)
C.2 CLEVR-3D
CLEVR-3D (Johnson et al. 2017) is synthetic dataset which consists of geometric primitives placed on a monochrome background. Following the convention of (Stelzner et al., (021) we evaluate on this benchmark for unsupervised object segmentation. Following convention evaluate the first 500 scenes of the validation set and report foreground adjusted random index. For class level labels consider objects with the shape to be of the same class.
17

---

Published as a conference paper at ICLR 2023
Figure 7: Snippets of Videos from ProcTHOR dataset used for training NRC.
Table 8: Performance of NRC with fixed codebook of various sizes on ProcTHOR.
Codebook Size
ProcTHOR (ARI)
10
.247
.256
30
.266
40
.279
50
.286
60
.293
70
.288
80
.295
90
.299
100
.273
18

---

Published as a conference paper at ICLR 2023
Input View
Novel View
Predicted
Figure 8: Reconstruction examples from NRC. For scenes and objects that are closer to the training
distribution, reconstruction quality is significantly better.
C.3 INSTANCE LEVEL ACCURACY
For comparison to the instance level classification performed by ObSuRF and uORF, we filtered for images in ProeTHOR which have one object of each class. In this setting, class and instance level classification are equivalent.
Table 9: The size of the learned codebook averaged over 5 runs on each dataset.
Codebook Size
CLEVR-3D ProcTHOR
15.6 + 1.2
53.8 + 2.4
19


Reviews:
[{"review_id": "wXzV1i_YlYq", "reviewer": "Reviewer_77Ao", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to marry the power of semantic codebooks with NeRFs, by conditioning the MLP on a fixed codebook (fixed after training, of course).\n\nWe are given two views - \"input view\" A and \"novel view\" B, as well as a shared codebook\nThe first stage is to run A through an encoder network, obtaining the (2D!) feature map F. \nNext, each 3D query point needed to render B is projected into A's 2D coordinates, essentially retrieving the relevant \"pixel feature\" f from F. We find the nearest neighbor atom m in the dictionary of the aforementioned \"pixel feature\" f, and add a small perturbation to account for intra class variability. This perturbation is generated by feeding both m and f into a small MLP.\nThe final value (atom+pertubation) is concatenated to the standard NeRF inputs, that is the view direction and the aforementioned 3D query point (transformed into positional encoding).\n\nThe total number of atoms is also learned with a special penalty which is added to the NeRF recunstruction loss\n\nThe method is tested on two synthetic and one real datasets, and present competitive resutls\nThe authors provide an ablation study to justify the two main modules, namely the dictionary size penalty and the perturbation MLP", "summary_of_the_review": "This seems like a promising method. If my concerns are addressed, I see no reason to accept this paper.\n\n---\n\nThe authors have addressed my concerns, and I think this paper is clearer now. \n", "strength_and_weaknesses": "### Strength \nThe manuscript is generally well written, and the method seems rather straightforward.\nThe experimental section presents impressive transfer from synthetic to real images.\n\n### Weaknesses\nThere are issues I hope the authors can clarify:\n\n1. It seems that the input, as in PixelNeRF, images are needed in inference time, otherwise - where is the feature map obtained from to query the dictionary?\nIf this is so - this is a major caveat compared to other NeRF methods that can render an image using only the MLP.\n\n2. Another possible discrepancy is the fact that the atom is generated from 2D coordinates of a view different from the novel one - but unlike \"PixelNeRF\", if an object is occluded in the \"input view\" we will get **a wrong atom** and hence the wrong class for the segmentation map - is this resolved by a low density for the rendering process as well?\n\n3. How exactly is a new atom added to the dictionary? Specifically, when the parameter 's' \"signifies when to add a new code\" - what value is the atom given?\n\nThere are some places in the text that can be made clearer:\n- I would recommend using a different notation for the two input images, rather than h and h'.\n- Figure 2 - It is not clear which part of the pipeline that are included in the MLP training, are not included in inference. For example:\n   - Where is the feature map F generated from?\n   - What feature values (or atoms) will we get for queries of regions out of the \"input view\"?\n- In addition, it was hard to follow how eq2 is incorporated into the flow - are discarded atoms kept as zero? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bBEGaWxgKmx", "reviewer": "Reviewer_GeqQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method to learn a shared set of object representations from paired views captured across 3D scenes, in a self-unsupervised manner. The core difference to existing 3D object-centric learning work is that it learns reoccurring objects across scenes and explicitly localize the learned objects, which are demonstrated in a set of downstream tasks, such as unsupervised image segmenation, object navigation, and depth ordering, that require the understanding of the scene layout and object goemetry and appearance.", "summary_of_the_review": "While I recognize the value of learning a codebook of 3D units over 3D scenes, my concerns expressed above make me leaning towards negative on this paper.", "strength_and_weaknesses": "Strength:\n\n1. I value the concept of learning of a codebook over 3D scenes, which could benefit many downstream tasks.\n\n2. this paper extends the technical route presented in the MarioNette for unsupervised dictionary learning on 3D scenes, with some novel technical contributions for working on 3D scenes, i.e., the modules for learning instance variation and the dynamic changing number of objects.\n\n\nWeakness:\n\nMy concerns over this paper are mainly on the evaluation/demonstration of the efficacy of the proposed learning setup.\n\n1. the paper ships itself that it learns reoccurring 3D objects across scenes, and explicitly localize the learned objects, but the demonstration of this is only done in a indirect way. Ideally, a successful model should be able to segment the 3D scene into meaningful 3D units, thus the evaluation of the learning could have focused on the 3D, i.e., measuring the cluster of predicted instances/objects against the GT cluster, a similar way as done for the unsupervised image segmenation as show in the paper, but with an emphasis in 3D. Moreover, more visual results of how 3D scenes are decomposed into meaningful units should be presented for qualitative examing the performance.\n\n2. the evaluation are mostly done on synthetic data. While this is not a fatal flaw, there is indeed no enough evidence to support that the proposed method can be applicable on more real scenarios. ScanNet and Matterport3D for example contain 3D scenes collected from the real world, with estimated camera poses available. I am cuorious about the performance on these kinds of real-world data, where the noisy camera estimates, difficulty in finding corresponding frames, and the imaging quality differ significantly from synthetic data.\n\n3. for the object navigation task, again, I think the evaluation can be conducted in a bit more realistic setting. What about evaluating on Gibson and Matterport3D scenes, where we can have more realistic scenes with camera poses available.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "ZLs-s2855lL", "reviewer": "Reviewer_PdhP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper decouples the scene representation into a set of learned codes. The codebook is object-level. The feature volume of a scene is encoded using the codebook before rendering with an MLP (NeRF). Experiments show the effectiveness of the proposed representation on various downstream tasks. The representation is evaluated using three tasks, object navigation in THOR, unsupervised segmentation in various datasets, and depth ordering estimation. These tasks demonstrate the effectiveness of the learned codebook.", "summary_of_the_review": "This paper presents an effective way to learn object-level codebooks through volume rendering. It shows the effectiveness of this codebook with various experiments. It is unclear whether this method is robust to cluttered scenarios, with more occlusions. I think overall it is a nice paper and should be valuable for the community to read.", "strength_and_weaknesses": "Strength:\n- Using an object-level codebook to represent a scene is not a new idea (MarioNette). But this paper makes the learned representation more effective with various techniques.\n- The experiment is through by comparing many related methods on three distinct tasks. The authors also show the method is not only useful for synthetic data but also for more complex real data.\n\nWeakness:\n- The majority of the experiment is done using synthetic datasets like THOR. The scene is relatively simple and has less clutter and occlusion than typical real-world scenarios. It is unclear how much the degradation will be for a more complex scenario.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-B7zePaveS", "reviewer": "Reviewer_BHLx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper target the learning of an object-centric scene representation through NeRF-wise volume rendering. The key component is inspired by Marionette (Smirnov et al., 2021) to utilize a dictionary of object codes. The modifications include a gated function that allows dynamic addition of new categorical codes, category-wise code to instance-wise code through a variation module. Experiments indicate that the learned representation can be applied to tasks such as unsupervised segmentation, object navigation, and depth ordering.", "summary_of_the_review": " Due to the issues mentioned in the Weakness section, I am leaning toward rejection currently. Though the experimental results in Tab. 4 indicate that the modification of the proposed method does bring benefits, major revision is required to make the method and the corresponding applications clear and intuitive. I would like to see how the authors respond and the opinions from other reviewers.", "strength_and_weaknesses": " Pros: \n+ The major differences between the proposed method and others are clearly presented and easy to follow.\n+ The paper tries to address a critical problem of learning object-centric neural scene representation\n\nCons:\n- Several key elements of the proposed method are missing. For instance, how the codebook is learned should be clarified even if it is similar to Marionette. Otherwise, it is hard to understand how the codebook in Fig. 2 is constructed during training. Besides, how the learned object-centric representation is applied to the three tasks should be detailed with illustrations. \n- The experimental results provide limited information regarding the performance of the proposed method. Without qualitative results of object navigation (video for instance) and depth ordering tasks, readers cannot intuitively understand how the proposed method performs. The analysis for results on view synthesis, unsupervised segmentation, object navigation, and depth ordering is also insufficient. \n\nMinors:\n1. The floor in Fig. 1 is placed in the middle row instead of the \"bottom row\"\n2. \"Additionally, we learn can model ...\" (page 3)\n3. The reused annotations lead to confusions: i \\in n indicates the index of scenes, while i \\in k indicates the index of codes; (x,y) indicates the pixel position while (x,y,z) indicates the world coordinate.  (page 3)\n4. The symbols such as O_h, f_h^{x,y}， G_\\theta, H_\\theta are encouraged to be added to Fig. 2. It is unclear where the concatenation of the relative position and the absolute position occur in Fig. 2.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster