SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition

1 Introduction . The problem of unsupervised visual scene understanding has become an increasingly central topic in machine learning [ 1 , 2 ] . The attention is merited by potential gains to reasoning , autonomous navigation , and myriad tasks . However , within the current literature , different studies frame the problem in different ways . One approach aims to decompose images into component objects and object features , supporting ( among other things ) generation of alternative data that permits insertion , deletion , or repositioning of individual objects [ 3–6 ] . Another approach aims at a very different form of decomposition—between allocentric scene structure and a variable viewpoint—supporting generation of views of a scene from new vantage points [ 7–9 ] and , if not supplied as input , estimation of camera pose [ 10 ] . Although there is work pursuing both of these approaches concurrently in the supervised setting [ 11–13 ] , very few previous studies have approached the combined challenge in the unsupervised case . In this work , we introduce the Sequence-Integrating Multi-Object Net ( SIMONe ) , a model which pursues that goal of object-level and viewpoint-level scene decomposition and synthesis without supervision . SIMONe is designed to handle these challenges without privileged information concerning camera pose , and in dynamic scenes . Given a video of a scene our model is able to decouple scene structure from viewpoint information ( see Figure 1 ) . To do so , it utilizes video-based cues , and a structured latent space which separates 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . time-invariant per-object features from time-varying global features . These features are inferred using a transformer-based network which integrates information jointly across space and time . Second , our method seeks to summarize objects ’ dynamics . It learns to disentangle not only static object attributes ( and their 2D spatial masks ) , but also object trajectories , without any prior notion of these objects , from videos alone . The learnt trajectory features are temporally abstract and per object ; they are captured independently of the dynamics of camera pose , which being a global property , is captured in the model ’ s per-frame ( time-varying ) latents.1 Our model thus advances the state of the art in unsupervised , object-centric scene understanding by satisfying the following desiderata : ( 1 ) decomposition of multi-object scenes from RGB videos alone ; ( 2 ) handling of changing camera pose , and simultaneous inference of scene contents and viewpoint from correlated views ( i.e . sequential observations of a moving agent ) ; ( 3 ) learning of structure across diverse scene instances ( i.e . procedurally sampled contents ) ; ( 4 ) object representations which summarize static object attributes like color or shape , view-dissociated properties like position or size , as well as time-abstracted trajectory features like direction of motion ; ( 5 ) no explicit assumptions of 3D geometry , no explicit dynamics model , no specialized renderer , and few a priori modeling assumptions about the objects being studied ; and ( 6 ) simple , scalable modules ( for inference and rendering ) to enable large-scale use . 2 Related Work . Given the multifaceted problem it tackles , SIMONe connects across several areas of prior work . We describe its nearest neighbors from three scene understanding domains below : 1Animated figures are at https : //sites.google.com/view/simone-scene-understanding/ . Scene decomposition models . ( 1 ) Our work builds on a recent surge of interest in unsupervised scene decomposition and understanding , especially using slot structure to capture the objects in a scene [ 14 ] . One line of work closely related to SIMONe includes methods like [ 3–6 , 15 ] , which all share SIMONe ’ s Gaussian mixture pixel likelihood model . While these prior methods handled only static scenes , more recent work [ 16–18 , 12 ] has extended them to videos with promising results . Nevertheless , these approaches have no mechanism or inductive bias to separate view information from scene contents . Moreover , many of them are conditioned on extra inputs like the actions of an agent/camera to simplify inference . ( 2 ) Another family of decomposition models originated with Attend , Infer , Repeat ( AIR ) [ 19 ] . AIR ’ s recurrent attention mechanism does split images into components with separate appearance and pose latents each . Later work [ 6 , 20–23 ] also extended the model to videos . Despite their structured latents , these models do not learn to distill object appearance into a time-invariant representation ( as their appearance and pose latents are free to vary as a function of time ) . They also require separate object discovery and propagation modules to handle appearing/disappearing objects . In contrast , SIMONe processes a full sequence of images using spatio-temporal attention and produces a single time-invariant latent for each object , hence requiring no explicit transition model or discovery/propagation modules . Multi-view scene rendering models . Models which assume viewpoint information for each image like GQN [ 7 ] , SRNs [ 8 ] , and NeRF [ 9 ] have shown impressive success at learning implicit scene representations and generating novel views from different viewpoints . Recent work [ 24–27 ] has further extended these models to videos using deformation fields to model changes in scene geometry over time . In contrast to SIMONe , these models can achieve photorealistic reconstructions by assuming camera parameters ( viewpoint information ) . To allow a direct comparison , we use a view-supervised version of SIMONe in Section 4.1 . There is also recent work [ 28 , 29 ] that relaxes the known viewpoint constraint , but they still model single scenes at a time , which prevents them from exploiting regularities over multiple scenes . A more recent line of work [ 30–32 ] explored amortizing inference by mapping from a given set of images to scene latents , but they can not handle videos yet . Note that all of these models treat the whole scene as a single entity and avoid decomposing it into objects . One exception here is [ 33 ] , which represents objects with separate pose and appearance latents . However , this model is purely generative and can not infer object latents from a given scene . Another exception is [ 13 ] , which can in fact infer object representations , but nevertheless depends on view supervision . Simultaneous localization and mapping . The problem of inferring scene representations in a novel environment by exploring it ( rather than assuming given views and viewpoint information ) is well studied in robotics and vision [ 10 ] . Classic SLAM techniques often rely on EM [ 34 , 35 ] or particle filters [ 36 ] to infer viewpoint and scene contents jointly . While our problem is slightly simpler ( we can leverage shared structure across scene instances ; certain elements such as the shape of the room are held constant ; and we use offline data rather than active exploration ) , our approach of using a factorized variational posterior provides a learning-based solution to the same computational problem . Our simplified setting is perhaps justified by our unsupervised take on the problem . On the other hand , we don ’ t assume simplifications which may be common in robotics practice ( e.g . known camera properties like field of view ; or the use of multiple cameras or depth sensors ) . Most popular SLAM benchmarks [ 37–39 ] are on unstructured 3D scenes and hence it was not straightforward for us to compare directly to classic methods . But an encouraging point of overlap is that object-centric SLAM formulations [ 11 ] as well as learning-based solutions [ 40 , 41 ] are active topics of research . Our work could open new avenues in object-centric scene mapping without supervision . 3 Model . SIMONe is a variational auto-encoder [ 42 ] consisting of an inference network ( encoder ) which infers latent variables from a given input sequence , and a generative process ( decoder ) which decodes these latents back into pixels . Using the Evidence Lower Bound ( ELBO ) , the model is trained to minimize a pixel reconstruction loss and latent compression KL loss . Crucially , SIMONe relies on a factorized latent space which enforces a separation of static object attributes from global , dynamic properties such as camera pose . We introduce our latent factorization and generative process in Section 3.1 . Then in Section 3.2 , we describe how the latents can be inferred using a transformer-based encoder , significantly simplifying the ( recurrent or autoregressive ) architectures used in prior work . Finally , we fully specify the training scheme in Section 3.3 . 3.1 Latent Structure and Generative Process . Our model aims to capture the structure of a scene , observed as a sequence of images from multiple viewpoints ( often along a smooth camera trajectory , though this is not a requirement ) . Like many recently proposed object-centric models we choose to represent the scene as a set of K object latent variables O : = { ok } Kk=1 . These are invariant by construction across all frames in the sequence ( i.e . their distribution is constant through time , and expected to summarize information across the whole sequence ) . We also introduce T frame latents F : = { ft } Tt=1 , one for each frame in the sequence , that capture time-varying information . Note that by choosing this factorization we reduce the number of required latent variables from K ·T to K+T . The latent prior p ( O , F ) = ∏ kN ( ok | 0 , I ) ∏ tN ( ft | 0 , I ) is a unit spherical Gaussian , assuming and enforcing independence between object latents , frame latents , and their feature dimensions . Given the latent variables , we assume all pixels and all frames to be independent . Each pixel is modeled as a Gaussian mixture with K components . The mixture weights for pixel xt , i capture which component k “ explains ” that pixel ( 1 ≤ i ≤ HW ) . The mixture logits m̂k , t , i and RGB ( reconstruction ) means µk , t , i are computed for every component k at a specific time-step t and specific pixel location li using a decoder Dθ : m̂k , t , i , µk , t , i = Dθ ( ok , ft ; li , t ) ( 1 ) p ( xt , i | o1 , ... , oK , ft ; t , li ) = ∑ k mk , t , iN ( xt , i | µk , t , i ; σx ) ( 2 ) We decode each pixel independently , `` querying '' our pixel-wise decoder using the sampled latents , coordinates li ∈ [ −1 , 1 ] 2 of the pixel , and time-step t ∈ [ 0 , 1 ) being decoded as inputs . The decoder ’ s architecture is an MLP or 1x1 CNN . ( See Appendix A.3.1 for the exact parameterization as well as a diagram of the generative process ) . By constraining the decoder to work on individual pixels , we can use a subset of pixels as training targets ( as opposed to full images ; this is elaborated in Section 3.3 ) . Once they are decoded , we obtain the mixture weights mk , t , i by taking the softmax of the logits across the K components : mk , t , i = softmaxk ( m̂k , t , i ) . Equation 2 specifies the full pixel likelihood , where σx is a scalar hyperparameter .

Summary: Authors propose a structured generative model of video clips. This uses a factored latent space, with per-object latents that are global the video, and temporal latents that are not split between objects. The decoder uses a spatial GMM (as in many prior works), albeit with pixels given by per-coordinate MLPs not a CNN. The encoder uses a transformer on CNN features. Results on synthetic datasets show higher performance than one existing method (S-IODINE) on video segmentation, and higher performance than GQN and NeRF-VAE on novel view synthesis.