Unsupervised Discovery of Object Radiance Fields

1 INTRODUCTION . Building factorized , object-centric scene representations is a fundamental ability in human vision and a constant topic of interest in computer vision and machine learning . We identify that such representations should bear three characteristics : they should be learned without supervision or prior knowledge about object categories , and therefore applicable to environments where object categories are unknown ; they should explain the image formation process , addressing questions like ‘ what if the object is not there ? ’ ; they should be 3D-aware , capturing geometric and physical object properties for navigation , interaction , and manipulation . For decades , researchers have attempted to solve the problems from various angles . Inspiring as they are , these methods each lack in one or more of the three aspects ( Table 1 ) . Computer vision research on unsupervised object discovery has achieved great success on deriving object segments from real images , but it doesn ’ t capture the image formation process , nor is it 3D-aware ( Rubinstein et al. , 2013 ; Zhu et al. , 2012 ) . Recent work on deep probabilistic inference for visual scene decomposition is unsupervised and generative ( Burgess et al. , 2019 ; Engelcke et al. , 2019 ; Locatello et al. , 2020 ) , though most still formulate the problem as 2D segmentation and work on simple scenes of geometric primitives , ignoring the complex 3D nature of realistic visual scenes . A few recent papers on ‘ scene de-rendering ’ have attempted to reconstruct 3D , object-centric representations by leveraging the forward rendering procedure ( Yao et al. , 2018 ; Ost et al. , 2021 ) ; they are however supervised , relying on annotations of specific object and scene categories , such as cars and road scenes . The fundamental challenge that prevents these systems from acquiring all three desired properties is that the image formation process from 3D to 2D is complex and non-differentiable ( e.g. , due to occlusion ) . Thus , for a long time , it has been unclear how it may be integrated with powerful deep inference schemes . But most recently , progresses in neural rendering ( Tewari et al. , 2020 ) have demonstrated that their continuous , implicit representation works well with gradient-based inference models , such as deep networks . In particular , Neural Radiance Fields ( NeRFs ) ( Mildenhall et al. , 2020 ) recover a 3D scene from a set of RGB images via differentiable volume rendering . Such encouraging advances in generative modeling suggest a promising route for inferring 3D , generative , and object-centric scene representations without supervision . In this paper , we propose unsupervised discovery of Object Radiance Fields ( uORF ) , integrating conditional NeRFs as 3D object representations with deep inference networks for unsupervised 3D scene decomposition . uORF infers a set of object radiance fields and a background radiance field ; thus , uORF represents a 3D scene as a composition of object radiance fields ( Figure 1 ) . During training , such radiance fields are neurally rendered in multiple views , with reconstruction losses in pixel space as training supervision ; during testing , uORF infers the set of object radiance fields from a single image . Learning uORF does not require explicit supervision of 3D geometry or object segmentation , but only sparse multi-view RGB images of training scenes . The integration of NeRFs allows us to work with more realistic scenes with complex object shapes and diverse background environments , beyond simple scenes such as those in multi-dSprites ( Greff et al. , 2019 ) and CLEVR ( Johnson et al. , 2017 ) , as considered by most current unsupervised scene decomposition methods . We further make two innovations to improve uORF ’ s performance . First , as background geometry and appearance can be quite different from foreground objects in 3D , we design uORF with explicit modeling of both components . This background-aware design not only facilitates learning on complex scenes , but also allows single-image scene editing including moving individual objects and changing background . Second , as volume rendering requires massive queries to render a single pixel for the recomposed scene , a practical challenge of learning uORF lies in the computational inefficiency . We tackle this issue by proposing a novel progressive coarse-to-fine training which improves representation quality while remaining affordable computational cost . We evaluate uORF on factorized scene representation learning ( e.g. , segmentation in 3D ) and scene generation ( e.g. , novel view synthesis , scene editing in 3D ) . Our evaluation is on three datasets with a gradually increasing complexity : first , CLEVR-like scenes with primitives foreground shapes ; second , room scenes with complex chair shapes and textured backgrounds ; third , more diverse room scenes with various foreground shapes and backgrounds . Our results show that uORF learns factorized representations that can segment 3D scenes into objects with fine shape details ( e.g. , thin chair legs ) and backgrounds with well-recovered appearance details ( e.g. , irregular textures of a wooden floor ) . In summary , our contributions are three-fold . First , we propose the problem of inferring an unsupervised , factorized , generative , and 3D-aware scene representation from a single image . Second , we introduce unsupervised discovery of Object Radiance Fields ( uORF ) that infers individual 3D object radiance fields from a single view for the proposed problem . Third , we demonstrate that uORF enables novel tasks such as scene segmentation and editing in 3D , and we show that it generalizes to novel scene arrangement and unseen combinations of object properties . 2 RELATED WORK Co-segmentation and object discovery . Our work is closely related to traditional computer vision methods on object discovery , which aims to locate ( visually similar ) objects in a collection of images . These methods typically model objects as visual words and adopted methods from topic modeling to localize objects ( Russell et al. , 2006 ; Sivic et al. , 2005 ; 2008 ) , or cluster and group image patches ( Grauman & Darrell , 2006 ; Joulin et al. , 2010 ; Rubio et al. , 2012 ; Vicente et al. , 2011 ; Rubinstein et al. , 2013 ; Cho et al. , 2015 ) . Recent works have integrated the clustering-based strategy with deep learning ( Li et al. , 2019 ; Vo et al. , 2020 ) . Nevertheless , they do not explain image formation process nor are they 3D-aware . Deep probabilistic inference for scene decomposition . Our method is also closely related to deep probabilistic inference for scene decomposition . Most works formulate the problem as compositional generative modeling , where a visual scene is represented by a set of latent codes that either correspond to localized object-centric patches ( Eslami et al. , 2016 ; Crawford & Pineau , 2019 ; Kosiorek et al. , 2018 ; Lin et al. , 2020 ; Jiang et al. , 2019 ) or scene mixture components ( Burgess et al. , 2019 ; Greff et al. , 2019 ; 2016 ; 2017 ; Engelcke et al. , 2019 ) . Recently , Locatello et al . ( 2020 ) proposed the Slot Attention module to simplify the inference by a slot-based encoder . Besides these inference models , Monnier et al . ( 2021 ) formulated scene decomposition as layered image decomposition and demonstrated it on real images . However , these methods do not account for the 3D nature of scenes . A few methods have recently been proposed for unsupervised 3D scene decomposition . Elich et al . ( 2020 ) infer object shapes from a single scene image , but they require pretraining on groundtruth shapes . Chen et al . ( 2020 ) extend Generative Query Network ( Eslami et al. , 2018 ) to decompose 3D scenes , but they require multi-view images during inference . The closest to our work is a concurrent work by Stelzner et al . ( 2021 ) which also utilizes a slot-based encoder and NeRFs as 3D representations . However , Stelzner et al . ( 2021 ) relies on groundtruth multi-view dense depth in addition to images in training . Moreover , we explicitly model the separation of objects and background to address various complex shapes and textured backgrounds , while they only demonstrate scenes with a single textureless background . Scene de-rendering . A few recent works have shown reconstructing 3D object-centric representations by incorporating forward image rendering process ( Wu et al. , 2017 ; Yao et al. , 2018 ; Kundu et al. , 2018 ; Ost et al. , 2021 ) . Yao et al . ( 2018 ) de-render an image into semantic segments and geometric object attributes , which enable 3D scene manipulation . Most recently , Ost et al . ( 2021 ) propose Neural Scene Graph to represent dynamic scenes into a scene graph where each node encodes object-centric information . However , these methods rely on manual annotations of specific objects ( such as cars ) and scene categories ( such as street scenes ) . Neural scene representations and rendering . Our method is related to recent progresses in neural continuous scene representations ( Sitzmann et al. , 2019 ) and neural rendering ( Tewari et al. , 2020 ) . Neural scene representations parameterize 3D scenes with a deep network ( Sitzmann et al. , 2019 ) . Combined with differentiable neural rendering techniques ( Kato et al. , 2020 ; Tewari et al. , 2020 ) , they can be learned from only 2D images ( Niemeyer et al. , 2020 ) . In particular , Neural Radiance Fields ( NeRFs ) ( Mildenhall et al. , 2020 ) have shown impressive novel view synthesis . Related follow-up works include those that infer NeRFs from a single image ( Yu et al. , 2020 ; Kosiorek et al. , 2021 ; Jang & Agapito , 2021 ) and those that incorporate NeRFs into generative models ( Schwarz et al. , 2020 ; Niemeyer & Geiger , 2021 ; Chan et al. , 2020 ) . Different from these works which cope with single objects or holistic scenes , we learn object NeRFs via decomposing a multi-object scene without segmentation annotations . GIRAFFE ( Niemeyer & Geiger , 2020 ) is also related to our work . It generates object NeRFs and thus compose 3D scenes in an adversarial framework . However , it targets at unconditional generation and can not tackle inference ( see Appendix E ) , while we focus on single-image inference of multi-object scenes . Thus , we address a fundamentally different problem compared to GIRAFFE . 3 APPROACH . Our goal is to infer from a single image a set of object-centric 3D representations to generate the underlying 3D scene . We show an illustration in Figure 2 . Our object representation is a conditional object radiance field . Thus , we firstly infer object-centric latents from a single image ( Figure 2- I ) . The inferred latents are used to condition a network to yield the 3D object and background radiance fields ( Figure 2-II ) , forming our 3D-aware , generative and factorized scene representation . In training , we compose all object and background radiance fields and render the recomposed scene from multiple views . We obtain supervision by comparing rendered images to reference images ( Figure 2-III ) without needing 3D geometry or segmentation annotations . We describe each of our model components in the following and leave implementation details in Appendix B . 3.1 OBJECT-CENTRIC LATENT INFERENCE Our goal is to infer latent object-centric representations from a single input image . We assume that an underlying 3D scene is composed of a background environment and no more than K foreground objects . Thus , the output of our object-centric latent inference process is a latent zb for background and a set of latents { zfi } Ki=1 for foreground objects ( empty objects are allowed ) . To encourage unsupervised object-wise factorization , we adopt a slot-based formulation ( Locatello et al. , 2020 ) . The assumption in this formulation is that objects should share a common prior latent space . The main idea include three steps . The first step is to sample all object latents ( i.e. , slots ) from the same prior distribution ( background is a special object ) to encourage representational uniformity across all slots ( “ sampling ” ) . Then each slot is bound to an object region via an attention module ( “ binding ” ) . In the last step each slot gets updated by the bound object features to specialize for that object ( “ updating ” ) . Locatello et al . ( 2020 ) have demonstrated success on segmenting 2D images . However , in 3D scenes , the geometry and appearance of the background are highly different from those of foreground objects . Modeling them indistinguishably often leads to object representations entangled with blurry background segments ( Burgess et al. , 2019 ; Locatello et al. , 2020 ) , which impedes applications such as scene editing and re-composition . Thus , we propose a backgroundaware slot attention module ( Figure 3 ) that separately models objects and environment to better capture the compositional structure of 3D scenes . Background-aware slot attention for sampling and binding . In the sampling step , we model the latent prior distribution of foreground objects by a Gaussian with learnable mean and variance , i.e. , we sample slotsf ∼ N f ( µf , diag ( σf ) ) ∈ RK×D for K objects . For latent prior of backgrounds , we learn another Gaussian and sample a single slot from it , i.e. , slotb ∼ N b ( µb , diag ( σb ) ) ∈ R1×D . To bind slots to image features , we let all the slots to compete for explaining the input image representation . To do this , we flatten the convolutional feature map ( we include details about convolutional encoder in Appendix B.1 ) into a set of N input feature vectors , feat ∈ RN×D . The slot competition is modeled by a key-query attention ( Bahdanau et al. , 2014 ) : attni , j : = exp ( Mi , j ) ∑ l exp ( Mi , l ) , where M : = 1√ D k ( feat ) · [ qb ( slotb ) qf ( slotsf ) ] T ∈ RN× ( K+1 ) . ( 1 ) Here k and qb/qf are learnable linear mappings RD→D for computing dot-product similarity ( Luong et al. , 2015 ) , and √ D is a fixed softmax temperature ( Vaswani et al. , 2017 ) . One can see this process as a soft K-means , where attni softly assigns a feature i to the slots ( centroids ) . The background slot is expected to capture the modality of background features and bind all of them , allowing foreground slots to focus only on the objects without explaining background segments ( Figure 3 ) . Besides the representation design , we further encourage disentanglement between background and foregrounds by two additional designs : ( 1 ) We represent and query foreground/background ( during the neural rendering process ) in different coordinate frames . ( 2 ) To discourage object slots from fitting background , we impose a locality constraint in early training . We set a foreground box and enforce that every foreground query point outside the box has zero density . We include details in Appendix B.2 . Updating slots to infer latents . With the attention weights , we form the update signal by aggregating input values via a weighted mean pooling updatesb : = W bT · vb ( feat ) ∈ R1×D , where W bi,1 : = attni,1/ ( ∑N l=1 attnl,1 ) , and updates f : = W fT · vf ( feat ) ∈ RK×D , where W fi , j : = attni , j+1/ ( ∑N l=1 attnl , j+1 ) . Slots are then updated using the update signals via a learnable rule parameterized by a Gated Recurrent Unit ( GRU ) ( Cho et al. , 2014 ) , so that slotsf ← GRUf ( slotsf , updatesf ) and slotb ← GRUb ( slotb , updatesb ) . We repeat the attention computation and updating for 3 iterations , and output all the slots as the final latents zb and { zfi } Ki=1 . We show pseudo-code of our background-aware slot attention in Appendix ( Alg . 1 ) .

Summary: This paper utilized the powerful NeRF. The authors present a new approach to learn the scene arrangement in an unsupervised way. The training is performed on unlabeled datasets of similar objects in different arrangements. The inference requires only one RGB image as input, and can correctly deduce the arrangement and the 3D geometry of the objects. The authors showed two supporting technical contributions to the system: (1) splitting background and foreground objects leads to better results, (2) a coarse-to-fine training to alleviate the space and time needs. The authors showed success on three synthetic datasets and various applications.