Object-Centric Representation Learning with Generative Spatial-Temporal Factorization

1 Introduction . Object-centric representation learning promises improved interpretability , generalization , and dataefficient learning on various downstream tasks like reasoning ( e.g . [ 18 , 39 ] ) and planning ( e.g . [ 30 , 2 , 41 ] ) . It aims at discovering compositional structures around objects from the raw sensory input data , i.e . a binding problem [ 12 ] , where the segregation ( i.e . factorization ) is the major challenge ( e.g . [ 9 , 4 ] ) , especially in cases of no supervision . In the context of visual data , most existing focus has been on single-view settings , i.e . decomposing and representing 3D scenes based on a single 2D image [ 1 , 10 , 26 ] or a fixed-view video [ 23 ] . These methods often suffer from single-view spatial ambiguities and thus show several failures or inaccuracies in representing 3D scene properties . It was demonstrated by Nanbo et al . [ 31 ] that such ambiguities could be effectively resolved by multi-view information aggregation . However , current multi-view models are built upon a foundational staticscene assumption . As a result , they : 1 ) require static-scene data for training and 2 ) can not handle well dynamic scenes where the spatial structures evolve over time . This greatly harms a model ’ s potentials in real-world applications . In this work , we target an unexplored problem—unsupervised object-centric latent representation learning in multi-view-dynamic-scene scenarios . Despite the importance of the problem to spatial- 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . temporal understanding of 3D scenes , solving it presents several technical challenges . Consider one particularly interesting scenario where both an observer ( e.g . a camera ) and the objects in the scene are moving at the same time . To aggregate 3D object information from two consecutive observations , an agent needs not only to handle the cross-view object correspondence problem [ 31 ] but also to reason about the independent effects of the scene dynamics and observer motions . One can consider the aggregation as a process of answering two questions : “ how much has an object really changed in the 3D space ” and “ what previous spatial unclarity can be clarified by the current view ” . In this paper , we refer to the relationship between the scene spatial structures and the viewpoints as the temporal entanglement because the temporal dependence of them complicates the identification of the independent generative mechanism [ 35 ] . We introduce DyMON ( Dynamics-aware Multi-Object Network ) , a unified unsupervised framework for multi-view object-centric representation learning . Instead of making a strong assumption of static scenes as that in previous multi-view methods , we only make two weak assumptions about the training scenes : i ) observation sequences are taken at a high frame rate , and ii ) there exists a significant difference between the speed of the observer and the objects ( see Sec . 3 ) . Under these two assumptions , in a short period , we can transition a multi-view-dynamic-scene problem to a multi-view-static-scene problem if an observer moves faster than a scene evolves , or to a single-viewdynamic-scene problem if a scene evolves faster than an observer moves . These local approximations allow DyMON to learn independently the generative relationships between scenes and observations , and viewpoints and observations during training , which further enable DyMON to address the problem of scene spatial-temporal factorization , i.e . solving the observer-scene temporal entanglement and scene object decomposition , at test time . Through the experiments we demonstrate that : ( i ) DyMON represents the first unsupervised multiview object-centric representation learning work in the context of dynamic-scene settings that can train and perform object-oriented inference on multi-view-dynamic-scene data ( see Sec . 5 ) . ( ii ) DyMON recovers the independent generative mechanism of an observer and scene objects from observations and permits querying predictions of scene appearances and segmentations across both space and time ( see Sec . 5.1 ) . ( iii ) As DyMON learns scene representations that are factorized in terms of objects , DyMON allows single-object manipulation along both the space ( i.e . viewpoint ) and time axis—e.g . replays dynamics of a single object without interferring the others ( see Sec . 5.1 ) . 2 Background . Object-centric Representations Consider object-centric representation inference as the inverse problem of an observation generation problem ( i.e . the vision-as-inverse-graphics [ 40 ] idea ) . In the forward process , i.e . observation generation , we have a scene well-defined by a set of parameter vectors z = { zk } = { z1 , z2 , ... , zK } , where a zk ∈ RD specifies one and only one object in the scene . An observation of the scene x , e.g . an image x ∈ RM or an RGB image x ∈ RM×3 , can be taken only by a specified observer ( often defined as v ∈ Rd ) which is independent of the scene in the forward problem , using a specific mapping g : RD × Rd 7→ RM×3 . Assuming a deterministic process , an observation x is generated as x = g ( z , v ) , where v is often omitted in single-view scenarios ( e.g . [ 1 , 10 ] ) . With the forward problem defined , we can describe the goal of learning an object-centric representation as inferring the intrinsic parameters of the objects { zk } that compose a scene z based on the scene observation x . In other words , computing a factorized posterior p ( z|x ) = p ( z1 , z2 , ... , zK |x ) , even though it is computationally intractable . As the number of objects is unknown in the inverse problem , it is worth noting that i ) K is often set globally to be a sufficiently large number ( greater than the actual number of objects ) to capture all scene objects , and ii ) we allow empty “ slots ” . Temporal Entanglement The dynamic nature of the world suggests that the spatial configuration of a scene ( denoted by zt ) and an observer vt are bound to the specific time t that an observation is taken ( i.e . xt = g ( zt , vt ) ) . Let X = { ( xt , vt ) } 1 : T 1 represent a data sample , e.g . a sequence or set of multi-view image observations , from dataset D , where T is the number of the images in the sample . Assuming zt is given in the data sample for now , i.e . focusing on the generative process only , we augment a scene data sample as Xa = { ( xt , vt , zt ) } 1 : T . In general , we assume an independent scene-observer relation : zt ⊥ vt|∅ but they nevertheless become dependent when the 1We define ( · ) as a joint sample indicator that forbids independent sampling of the random variables wherein . corresponding observation is given : zt 6⊥ vt|xt . Under a static-scene assumption , we can treat an augmented data sample as Xa = { ( xt , vt ) , zt } 1 : T where zt and vt are separable ( i.e . can be sampled independently ) . In this case , to recover the independent generative mechanism ( i.e . train a g ( · ) ) w.r.t . scenes and observers from data , GQN [ 8 ] and MulMON [ 31 ] fix zt to z and intervene on the viewpoints vt. From a causal perspective , this can be seen as estimating p ( xt ′ |do ( vt = vt′ ) , zt = z ) , where ( xt ′ , vt ′ ) ∼ { ( xt , vt ) } 1 : T , implicitly under a causal model : zt → xt ← vt . However , in dynamic settings , the same estimation , i.e . sampling ( xt ′ , vt ′ ) ∼ { ( xt , vt ) } 1 : T independently of zt , is forbidden by the ( · ) indicator . Intuitively , an observer can not take more than one observations from different viewpoints at the same time t. In this paper , we refer to this issue as temporal entanglement in view of the temporal implication of the ( · ) indicator . 3 DyMON . Our goal is to train a multi-view object-centric representation learning model that recovers the independent generative mechanism of scene objects and their motions and observer motions from dynamic-scene observations . In this section , we detail how DyMON addresses these two presented challenges : 1 ) temporal disentanglement ( see Sec . 3.1 ) , and 2 ) scene spatial factorization ( see Sec . 3.2 ) . We discuss the training of DyMON in Sec . 3.3 . 3.1 Temporal Disentanglement . The key to resolving temporal entanglement , i.e . temporal disentanglement , is to enable sampling ( xt , vt ) independently of zt , or ( xt , zt ) independently of vt . This is seemingly impossible in the multiview-dynamic-scene setting as it requires to fix either zt ( static scene ) or vt ( single-view ) respectively . In this paper , we make two assumptions about the training scenes to ensure the satisfaction of the aforementioned two requirements without violating the global multi-view-dynamic-scene setting . Let us first describe the dynamics of scenes and observers with two independent dynamical systems : zt+∆t − zt = fz ( zt , t ) ∆t , vt+∆t − vt = fv ( vt , t ) ∆t , ( 1 ) where t and t+ ∆t are the times that two consecutive observations were taken , fz ( zt , t ) and fv ( vt , t ) , or simply fzt and fvt , are the average velocities of scene objects and the observer within [ t , t+ ∆t ] . Note that we use a zt to capture both the shape and pose information of an object . However , we do not consider shape changes in this work . With the dynamical systems defined , we introduce our assumptions ( which defines a tractable subset of all possible situations ) as : • ( A1 ) The high-frame-rate assumption ∆t→ 0 s.t . xt+∆t ≈ xt , • ( A2 ) The large-speed-difference assumption The data comes from one of two cases ( SCFO : Slow Camera , Fast Objects or FCSO : Fast Camera Slow Objects ) , that satisfy : | |fz||fv| | ≥ CSCFO or | |fz||fv| | ≤ CFCSO , where |velocity| computes a speed , and CSCFO and CFCSO are positive constants . A1 allows us to assume a nearly static scene zt or a fixed viewpoint vt for a short period . Consider an example where we assume a static scene , i.e . zτ−∆t ≈ zτ ≈ zτ+∆t , in [ τ − ∆t , τ + ∆t ] , A1 essentially allows us to extract zt out of a joint sample as : Xa = { ( xt , vt ) , zt } τ−∆t : τ+∆t . An intuitive way to define A2 is : |fz| |fv| or |fz| |fv| , which specify a large speed difference between scene speeds and observer speeds . These two assumptions enable us to accumulate instant changes ( velocities ) on one variable ( e.g . either zt or vt ) over a finite number of ∆t while ignoring the small changes of the other ( assumed fixed ) . We then treat a slow-camera-fast-objects ( i.e . SCFO ) scenario , where |fz| |fv| , as an approximate single-view-dynamic-scene scenario , and a fast-camera-slow-objects ( i.e . FCSO ) scenario , where |fz| |fv| , an approximate multi-view-static-scene scenario . Either case allows us to resolve the temporal entanglement problem . Importantly , to answer the question : “ is a given data sample an SCFO or FCSO sample ” , we need to quantitatively specify the two assignment criteria CSCFO and CFCSO . However , a direct calculation of these two constants is often difficult and does not generalize as : i ) |fz| is not available in unsupervised scene representation learning data , and ii ) the two constants vary across different datasets . In practice , we cluster the data samples into SCFO and FCSO clusters using only the viewpoint speed |fv| , i.e . assuming |fz| = 1 for training ( see Sec . 3.3 ) . In testing , DyMON treats them equally .

Summary: The following work proposes a method for learning the factorized latent structure of a scene. Specifically, while the primary prior work MulMon [28] assume static objects while the viewpoint varies through time, this work allows for both camera and object motion within the training data. To achieve their goal of factorizing object motion from camera motion, they assume high framerate and use clustering to determine cases where the camera is moving quickly and the objects slowly, versus cases where the camera moves slowly but the objects move quickly. Notably, once trained, they are able to resynthesize the scene's dynamic objects from novel viewpoints.