SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition

1 Introduction . The problem of unsupervised visual scene understanding has become an increasingly central topic in machine learning [ 1 , 2 ] . The attention is merited by potential gains to reasoning , autonomous navigation , and myriad tasks . However , within the current literature , different studies frame the problem in different ways . One approach aims to decompose images into component objects and object features , supporting ( among other things ) generation of alternative data that permits insertion , deletion , or repositioning of individual objects [ 3–6 ] . Another approach aims at a very different form of decomposition—between allocentric scene structure and a variable viewpoint—supporting generation of views of a scene from new vantage points [ 7–9 ] and , if not supplied as input , estimation of camera pose [ 10 ] . Although there is work pursuing both of these approaches concurrently in the supervised setting [ 11–13 ] , very few previous studies have approached the combined challenge in the unsupervised case . In this work , we introduce the Sequence-Integrating Multi-Object Net ( SIMONe ) , a model which pursues that goal of object-level and viewpoint-level scene decomposition and synthesis without supervision . SIMONe is designed to handle these challenges without privileged information concerning camera pose , and in dynamic scenes . Given a video of a scene our model is able to decouple scene structure from viewpoint information ( see Figure 1 ) . To do so , it utilizes video-based cues , and a structured latent space which separates 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . time-invariant per-object features from time-varying global features . These features are inferred using a transformer-based network which integrates information jointly across space and time . Second , our method seeks to summarize objects ’ dynamics . It learns to disentangle not only static object attributes ( and their 2D spatial masks ) , but also object trajectories , without any prior notion of these objects , from videos alone . The learnt trajectory features are temporally abstract and per object ; they are captured independently of the dynamics of camera pose , which being a global property , is captured in the model ’ s per-frame ( time-varying ) latents.1 Our model thus advances the state of the art in unsupervised , object-centric scene understanding by satisfying the following desiderata : ( 1 ) decomposition of multi-object scenes from RGB videos alone ; ( 2 ) handling of changing camera pose , and simultaneous inference of scene contents and viewpoint from correlated views ( i.e . sequential observations of a moving agent ) ; ( 3 ) learning of structure across diverse scene instances ( i.e . procedurally sampled contents ) ; ( 4 ) object representations which summarize static object attributes like color or shape , view-dissociated properties like position or size , as well as time-abstracted trajectory features like direction of motion ; ( 5 ) no explicit assumptions of 3D geometry , no explicit dynamics model , no specialized renderer , and few a priori modeling assumptions about the objects being studied ; and ( 6 ) simple , scalable modules ( for inference and rendering ) to enable large-scale use . 2 Related Work . Given the multifaceted problem it tackles , SIMONe connects across several areas of prior work . We describe its nearest neighbors from three scene understanding domains below : 1Animated figures are at https : //sites.google.com/view/simone-scene-understanding/ . Scene decomposition models . ( 1 ) Our work builds on a recent surge of interest in unsupervised scene decomposition and understanding , especially using slot structure to capture the objects in a scene [ 14 ] . One line of work closely related to SIMONe includes methods like [ 3–6 , 15 ] , which all share SIMONe ’ s Gaussian mixture pixel likelihood model . While these prior methods handled only static scenes , more recent work [ 16–18 , 12 ] has extended them to videos with promising results . Nevertheless , these approaches have no mechanism or inductive bias to separate view information from scene contents . Moreover , many of them are conditioned on extra inputs like the actions of an agent/camera to simplify inference . ( 2 ) Another family of decomposition models originated with Attend , Infer , Repeat ( AIR ) [ 19 ] . AIR ’ s recurrent attention mechanism does split images into components with separate appearance and pose latents each . Later work [ 6 , 20–23 ] also extended the model to videos . Despite their structured latents , these models do not learn to distill object appearance into a time-invariant representation ( as their appearance and pose latents are free to vary as a function of time ) . They also require separate object discovery and propagation modules to handle appearing/disappearing objects . In contrast , SIMONe processes a full sequence of images using spatio-temporal attention and produces a single time-invariant latent for each object , hence requiring no explicit transition model or discovery/propagation modules . Multi-view scene rendering models . Models which assume viewpoint information for each image like GQN [ 7 ] , SRNs [ 8 ] , and NeRF [ 9 ] have shown impressive success at learning implicit scene representations and generating novel views from different viewpoints . Recent work [ 24–27 ] has further extended these models to videos using deformation fields to model changes in scene geometry over time . In contrast to SIMONe , these models can achieve photorealistic reconstructions by assuming camera parameters ( viewpoint information ) . To allow a direct comparison , we use a view-supervised version of SIMONe in Section 4.1 . There is also recent work [ 28 , 29 ] that relaxes the known viewpoint constraint , but they still model single scenes at a time , which prevents them from exploiting regularities over multiple scenes . A more recent line of work [ 30–32 ] explored amortizing inference by mapping from a given set of images to scene latents , but they can not handle videos yet . Note that all of these models treat the whole scene as a single entity and avoid decomposing it into objects . One exception here is [ 33 ] , which represents objects with separate pose and appearance latents . However , this model is purely generative and can not infer object latents from a given scene . Another exception is [ 13 ] , which can in fact infer object representations , but nevertheless depends on view supervision . Simultaneous localization and mapping . The problem of inferring scene representations in a novel environment by exploring it ( rather than assuming given views and viewpoint information ) is well studied in robotics and vision [ 10 ] . Classic SLAM techniques often rely on EM [ 34 , 35 ] or particle filters [ 36 ] to infer viewpoint and scene contents jointly . While our problem is slightly simpler ( we can leverage shared structure across scene instances ; certain elements such as the shape of the room are held constant ; and we use offline data rather than active exploration ) , our approach of using a factorized variational posterior provides a learning-based solution to the same computational problem . Our simplified setting is perhaps justified by our unsupervised take on the problem . On the other hand , we don ’ t assume simplifications which may be common in robotics practice ( e.g . known camera properties like field of view ; or the use of multiple cameras or depth sensors ) . Most popular SLAM benchmarks [ 37–39 ] are on unstructured 3D scenes and hence it was not straightforward for us to compare directly to classic methods . But an encouraging point of overlap is that object-centric SLAM formulations [ 11 ] as well as learning-based solutions [ 40 , 41 ] are active topics of research . Our work could open new avenues in object-centric scene mapping without supervision . 3 Model . SIMONe is a variational auto-encoder [ 42 ] consisting of an inference network ( encoder ) which infers latent variables from a given input sequence , and a generative process ( decoder ) which decodes these latents back into pixels . Using the Evidence Lower Bound ( ELBO ) , the model is trained to minimize a pixel reconstruction loss and latent compression KL loss . Crucially , SIMONe relies on a factorized latent space which enforces a separation of static object attributes from global , dynamic properties such as camera pose . We introduce our latent factorization and generative process in Section 3.1 . Then in Section 3.2 , we describe how the latents can be inferred using a transformer-based encoder , significantly simplifying the ( recurrent or autoregressive ) architectures used in prior work . Finally , we fully specify the training scheme in Section 3.3 . 3.1 Latent Structure and Generative Process . Our model aims to capture the structure of a scene , observed as a sequence of images from multiple viewpoints ( often along a smooth camera trajectory , though this is not a requirement ) . Like many recently proposed object-centric models we choose to represent the scene as a set of K object latent variables O : = { ok } Kk=1 . These are invariant by construction across all frames in the sequence ( i.e . their distribution is constant through time , and expected to summarize information across the whole sequence ) . We also introduce T frame latents F : = { ft } Tt=1 , one for each frame in the sequence , that capture time-varying information . Note that by choosing this factorization we reduce the number of required latent variables from K ·T to K+T . The latent prior p ( O , F ) = ∏ kN ( ok | 0 , I ) ∏ tN ( ft | 0 , I ) is a unit spherical Gaussian , assuming and enforcing independence between object latents , frame latents , and their feature dimensions . Given the latent variables , we assume all pixels and all frames to be independent . Each pixel is modeled as a Gaussian mixture with K components . The mixture weights for pixel xt , i capture which component k “ explains ” that pixel ( 1 ≤ i ≤ HW ) . The mixture logits m̂k , t , i and RGB ( reconstruction ) means µk , t , i are computed for every component k at a specific time-step t and specific pixel location li using a decoder Dθ : m̂k , t , i , µk , t , i = Dθ ( ok , ft ; li , t ) ( 1 ) p ( xt , i | o1 , ... , oK , ft ; t , li ) = ∑ k mk , t , iN ( xt , i | µk , t , i ; σx ) ( 2 ) We decode each pixel independently , `` querying '' our pixel-wise decoder using the sampled latents , coordinates li ∈ [ −1 , 1 ] 2 of the pixel , and time-step t ∈ [ 0 , 1 ) being decoded as inputs . The decoder ’ s architecture is an MLP or 1x1 CNN . ( See Appendix A.3.1 for the exact parameterization as well as a diagram of the generative process ) . By constraining the decoder to work on individual pixels , we can use a subset of pixels as training targets ( as opposed to full images ; this is elaborated in Section 3.3 ) . Once they are decoded , we obtain the mixture weights mk , t , i by taking the softmax of the logits across the K components : mk , t , i = softmaxk ( m̂k , t , i ) . Equation 2 specifies the full pixel likelihood , where σx is a scalar hyperparameter .

Summary: This paper proposes SIMONe, a variational auto-encoder that decomposes a video into object latents and frame latents without any supervision. The object latents correspond to the time-invariant, object-level contents of the scene. The frame latents correspond to the global time-varying elemnts of the scene such as viewpoint. Each pixel is decoded independently by querying a pixel-wise decoder using sampled latents. To infer latents, SIMONe uses a transformer-based encoder that integrates information jointly across space and time. The experiments demonstrate SIMONe's capabilities in view syntheses and instance segmentation. Furthermore, the object latents are shown to be disentangled and summarize the trajectories of the objects while the frame latents are shown to contain viewpoint information.