INFERNO: Inferring Object-Centric 3D Scene Representations without Supervision

1 INTRODUCTION . Inferring objects and their 3D geometry in a scene is a fundamental ability of biological visual systems ( Kahneman et al. , 1992 ; Roelfsema et al. , 1998 ; Spelke et al. , 1993 ) . Replicating this ability in machine is a promising step towards visual reasoning valuable to several applications involving object manipulation , navigation or forecasting . Recent works ( Jiang et al. , 2019 ; Locatello et al. , 2020 ; Burgess et al. , 2019 ) have shown that neural networks can learn object-centric representations from low-level perceptual features . They learn to recognize the objects in a visual scene from a singe image without relying on supervision . However , most of those approaches only consider the 2D structure of images and ignore the underlying 3D geometry of the visual scenes . On the other hand , Neural Radiance Fields ( NeRFs ) ( Mildenhall et al. , 2020 ) have demonstrated that differentiable renderers can be combined with gradient-based optimization to learn high-fidelity 3D scene reconstructions . NeRFs have been subsequently used to learn 3D-aware generative models , including compositional scene models ( Niemeyer & Geiger , 2021 ) . In this work , we leverage these recent advances in object-centric representation learning ( Locatello et al. , 2020 ) and 3D modelling through implicit functions ( Mildenhall et al. , 2020 ; Niemeyer & Geiger , 2021 ) and propose INFERNO , a model which infers a structured representation of objects and their 3D poses from a single image . Each object is represented by latent variables characterizing its shape and appearance , together with an explicit representation of their 3D poses ( translation , scale and rotation ) . The object representations are then decoded using implicit functions that are localized in the scene according to the objects poses and combined together to generate a 2D output view . Our model does not need supervision and instead is fitted through minimizing a reconstruction loss , akin to an auto-encoder . Disentangling the object appearance and pose in a scene representation allows for the model to manipulate a visual scene . In particular , we demonstrate that INFERNO learns interpretable object poses , which we can modify and render to alter the pose of an object in a scene . We also validate that our approach learns meaningful representations for object discovery and visual reasoning . More specifically , we show that our approach obtains competitive performance on the CLEVR6 object discovery benchmark ( Johnson et al. , 2017 ; Greff et al. , 2019 ) as well as for the snitch localization visual reasoning task of CATER ( Girdhar & Ramanan , 2019 ) . In summary our contributions are the following : • We propose a model able to infer and render 3D scene representations composed of multiple objects , each of them modeled by an implicit function and explicitly localized in the scene . • We show that the representations learned by the model are interpretable and amenable to manipulations . • We demonstrate that the inferred representations are useful for downstream tasks by showing competitive performance in object discovery and reasoning tasks . 2 METHOD We propose INFERNO ( Infer NeRF Objects ) . The goal of our method is to infer object-centric 3D scene representations from single 2D views . Given an image x ∈ RH×W×3 , we learn an inference function fθ that maps images to scene representations s = fθ ( x ) = ( o1 , o2 , ... , oK , obg , c ) . Scenes are composed of K objects oi , a background object obg and a camera location c. Each object is composed of three tensors oi = ( o shape i , o app i , o pose i ) . The object shape o shape i ∈ RDshape and object appearance oapp ∈ RDshape are tensors that respectively describe the 3D shape occupancy and color of an object with an implicit function . The object location oposei ∈ R4×4 is an affine matrix that describes the object 3D pose ( i.e . scale , translation and rotation ) in the scene . The background object obg = ( bgshape , bgapp ) only models 3D shape and color , and its location is fixed , encompassing the back-of-scene cube . We also define a camera matrix c ∈ R3×4 , that defines the location of the scene camera and determines a 2D projection of the 3D scene . To optimize our inference function we formulate an optimization problem in which we minimize a reconstruction loss over a dataset , similar to an auto-encoder . We define a rendering function gγ that takes as input a scene representation and generates a 2D view of that scene x̂ = gγ ( fθ ( x ) ) . We assume a isotropic Gaussian likelihood model with unit covariance and optimize the probability of the data under our model , which is equivalent to minimizing the mean squared error of inputs and reconstructions : γ∗ , θ∗ = argmin γ , θ p ( X|γ , θ ) = argmin γ , θ 1 N ∑ i≤N ( xi − gγ ( fθ ( xi ) ) ) 2 ( 1 ) In the following sections we describe in more detail our inference mechanism , our rendering pipeline and their implementation . 2.1 RENDERING PIPELINE . We represent 3D objects as Neural Radiance Fields ( NeRFs ) ( Mildenhall et al. , 2020 ) with a similar setup as that of GIRAFFE ( Niemeyer & Geiger , 2021 ) . A NeRF is a function gτ that defines a 3D shape implicitly . It takes as input a 3D location l = ( x , y , z ) and a 2D viewing direction d = ( ψ , φ ) and outputs an occupancy value σ and a color value a = ( r , g , b ) . NeRFs are usually implemented using fully connected neural networks . Additionally , the inputs are usually embedded into a higherdimensional space using positional encodings γ that embed locations and viewing directions into higher dimensional spaces RPl and RPd , respectively . gτ : RPl × RPd → R+ × R3 ; ( γ ( l ) , γ ( d ) ) → ( σ , a ) ( 2 ) To represent multiple 3D shapes with the same NeRF function , we can augment it with latent variables that determine which shape is being modeled ( Schwarz et al. , 2020 ) . NeRFs are usually augmented with two random variables : one random variable µ ∈ RDshape defines the shape of the entity being modeled , while υ ∈ RDapp models its appearance . In practice , this specialization is enforced by making the occupancy output a function of only the shape latent , while the color output is conditioned on the appearance latent . g′τ : RPl × RPd × RDshape × RDapp → R+ × R3 ; ( γ ( l ) , γ ( d ) , µ , υ ) → ( σ , a ) ( 3 ) In INFERNO , we share a single parametrization of a NeRF function across all objects . Each object specific shape and appearance is defined by the shape and appearance latent variables , which correspond to the object attributes oshape , oapp . The background is defined as another NeRF with separate parameters . The background NeRF also has shape and appearance latent variables to model different backgrounds . The pose of an object oi in the scene is determined by the affine transformation matrix o pose i . We denote the coordinate system of the NeRF function of an object as the object space , and the coordinate system of the scene ( and the background NeRF ) as scene space . Given an object pose , we can convert points from the scene space to the object space by applying the opose transformation matrix on those points , and we can transform points from object space to scene space by computing the inverse of the object pose matrix . To render a scene , we cast rays from each pixel in the 2D plane defined by a given camera to the 3D scene . We evaluate NeRFs at different points along a given ray , and integrate their occupancy and color outputs to determine pixel values . Rays might traverse multiple object NeRFs in addition to the background NeRF . To determine the occupancy and color of points described by multiple NeRFs , we first query each NeRF at those particular points . To query the object NeRFs , we first need to transform the points from scene space to the particular object space . Then , we compose the results of each NeRF with a pooling function C , which in our case is a weighted average : C ( l , d ) = ( σ = N∑ i=1 σi , 1 σ N∑ i=1 σiai ) ( 4 ) Since rendering with NeRFs as originally proposed is computationally expensive , we render output views at a fixed low resolution . Low resolution scenes are then upscaled to the desired output resolution using a convolutional neural network , keeping the entire rendering pipeline differentiable . Additionally , low resolution scenes are rendered with additional channels , allowing for more detailed upscalings beyond those possible when just rendering low resolution RGB outputs . For more details about NeRF rendering and compositional NeRF models refer to ( Mildenhall et al. , 2020 ; Niemeyer & Geiger , 2021 ) . 2.2 INFERENCE . Given the rendering pipeline , the goal of our method is to infer representations that reconstruct a given scene . Our inference mechanism computes image features through a neural network encoder and then extracts K object and a background slots . These slots are then mapped to our structured scene representation through learned neural networks . To extract image features for each object and background slots , we use Slot Attention ( Locatello et al. , 2020 ) . Slot Attention is a mechanism that maps a set of K entities , called slots , to image features without annotations . It extracts image features I ∈ RH×W×D from a given input using a resolution-preserving convolutional encoder . These features are then attended to by a set of K randomly sampled slots πj ∼ N ( µ , σ ) of dimensionD , where µ and σ are learnable parameters . We denote by π the matrix concatenating all the sampled slots . Slots attend spatial chunks of the input features I through soft-attention u = TQT , where T = k ( I ) and Q = q ( π ) are the embeddings of the inputs and slots respectively . The attention weights are normalized through a softmax operating on the slots axis , which makes slots compete among themselves and discourages multiple slots from attending the same input region w = softmax ( u ) . The weighted average of I according to the attention weights is then computed and fed to a GRU network , to update each slot value : πj = GRU ( wj ∗ I , πj ) ∀j ∈ 1 , K . Multiple rounds of soft attention are performed to iteratively refine the slots . For more details about Slot-Attention , please refer to ( Locatello et al. , 2020 ) . Object slots are unstructured tensors that result from aggregating image features . We map these slots to our structured scene representation through small fully-connected networks that operate on individual objects . More concretely , we map object slots to their 3D pose in the scene through a 2-layer MLP . To infer the object shape and appearance tensors we also use 2-layer MLPs , but we make them conditional on the predicted object pose through conditional normalization ( Dumoulin et al. , 2018 ) .

Summary: This paper proposes a model which is able to segment 3D scenes into objects by a combination of slot-attention (For inference) and a mixture of object NeRF functions which mix together (in 3D) to compose a scene. The method receives a single input image (with the camera coordinates though these are fixed) and extracts a set of slots - one slot for each object. These slots are decoded using a NeRF renderer: one part (the shape) generates the density, one (the appearance) generates the colours and one (the pose) transforms the points of the object to the appropriate pose in scene space. Results are demonstrated on CLEVR data as well as CATER (which is visually very similar) and some downstream tasks.