Object-Centric Representation Learning with Generative Spatial-Temporal Factorization
Year: 2022

Abstract:
Learning object-centric scene representations is essential for attaining structural understanding and abstraction of complex scenes. Yet, as current approaches for unsupervised object-centric representation learning are built upon either a stationary observer assumption or a static scene assumption, they often: i) suffer single-view spatial ambiguities, or ii) infer incorrectly or inaccurately object representations from dynamic scenes. To address this, we propose Dynamicsaware Multi-Object Network (DyMON), a method that broadens the scope of multi-view object-centric representation learning to dynamic scenes. We train DyMON on multi-view-dynamic-scene data and show that DyMON learns—without supervision—to factorize the entangled effects of observer motions and scene object dynamics from a sequence of observations, and constructs scene object spatial representations suitable for rendering at arbitrary times (querying across time) and from arbitrary viewpoints (querying across space). We also show that the factorized scene representations (w.r.t. objects) support querying about a single object by space and time independently.

heading: 1 Introduction
text: Object-centric representation learning promises improved interpretability, generalization, and dataefficient learning on various downstream tasks like reasoning (e.g. [18, 39]) and planning (e.g. [30, 2, 41]). It aims at discovering compositional structures around objects from the raw sensory input data, i.e. a binding problem [12], where the segregation (i.e. factorization) is the major challenge (e.g. [9, 4]), especially in cases of no supervision. In the context of visual data, most existing focus has been on single-view settings, i.e. decomposing and representing 3D scenes based on a single 2D image [1, 10, 26] or a fixed-view video [23]. These methods often suffer from single-view spatial ambiguities and thus show several failures or inaccuracies in representing 3D scene properties. It was demonstrated by Nanbo et al. [31] that such ambiguities could be effectively resolved by multi-view information aggregation. However, current multi-view models are built upon a foundational staticscene assumption. As a result, they: 1) require static-scene data for training and 2) cannot handle well dynamic scenes where the spatial structures evolve over time. This greatly harms a model’s potentials in real-world applications.
In this work, we target an unexplored problem—unsupervised object-centric latent representation learning in multi-view-dynamic-scene scenarios. Despite the importance of the problem to spatial-
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
temporal understanding of 3D scenes, solving it presents several technical challenges. Consider one particularly interesting scenario where both an observer (e.g. a camera) and the objects in the scene are moving at the same time. To aggregate 3D object information from two consecutive observations, an agent needs not only to handle the cross-view object correspondence problem [31] but also to reason about the independent effects of the scene dynamics and observer motions. One can consider the aggregation as a process of answering two questions: “how much has an object really changed in the 3D space” and “what previous spatial unclarity can be clarified by the current view”. In this paper, we refer to the relationship between the scene spatial structures and the viewpoints as the temporal entanglement because the temporal dependence of them complicates the identification of the independent generative mechanism [35].
We introduce DyMON (Dynamics-aware Multi-Object Network), a unified unsupervised framework for multi-view object-centric representation learning. Instead of making a strong assumption of static scenes as that in previous multi-view methods, we only make two weak assumptions about the training scenes: i) observation sequences are taken at a high frame rate, and ii) there exists a significant difference between the speed of the observer and the objects (see Sec. 3). Under these two assumptions, in a short period, we can transition a multi-view-dynamic-scene problem to a multi-view-static-scene problem if an observer moves faster than a scene evolves, or to a single-viewdynamic-scene problem if a scene evolves faster than an observer moves. These local approximations allow DyMON to learn independently the generative relationships between scenes and observations, and viewpoints and observations during training, which further enable DyMON to address the problem of scene spatial-temporal factorization, i.e. solving the observer-scene temporal entanglement and scene object decomposition, at test time.
Through the experiments we demonstrate that: (i) DyMON represents the first unsupervised multiview object-centric representation learning work in the context of dynamic-scene settings that can train and perform object-oriented inference on multi-view-dynamic-scene data (see Sec. 5). (ii) DyMON recovers the independent generative mechanism of an observer and scene objects from observations and permits querying predictions of scene appearances and segmentations across both space and time (see Sec. 5.1). (iii) As DyMON learns scene representations that are factorized in terms of objects, DyMON allows single-object manipulation along both the space (i.e. viewpoint) and time axis—e.g. replays dynamics of a single object without interferring the others (see Sec. 5.1).

heading: 2 Background
text: Object-centric Representations Consider object-centric representation inference as the inverse problem of an observation generation problem (i.e. the vision-as-inverse-graphics [40] idea). In the forward process, i.e. observation generation, we have a scene well-defined by a set of parameter vectors z = {zk} = {z1, z2, ..., zK}, where a zk ∈ RD specifies one and only one object in the scene. An observation of the scene x, e.g. an image x ∈ RM or an RGB image x ∈ RM×3, can be taken only by a specified observer (often defined as v ∈ Rd) which is independent of the scene in the forward problem, using a specific mapping g : RD × Rd 7→ RM×3. Assuming a deterministic process, an observation x is generated as x = g(z, v), where v is often omitted in single-view scenarios (e.g. [1, 10]). With the forward problem defined, we can describe the goal of learning an object-centric representation as inferring the intrinsic parameters of the objects {zk} that compose a scene z based on the scene observation x. In other words, computing a factorized posterior p(z|x) = p(z1, z2, ..., zK |x), even though it is computationally intractable. As the number of objects is unknown in the inverse problem, it is worth noting that i) K is often set globally to be a sufficiently large number (greater than the actual number of objects) to capture all scene objects, and ii) we allow empty “slots”.
Temporal Entanglement The dynamic nature of the world suggests that the spatial configuration of a scene (denoted by zt) and an observer vt are bound to the specific time t that an observation is taken (i.e. xt = g(zt, vt)). Let X = {(xt, vt)}1:T 1 represent a data sample, e.g. a sequence or set of multi-view image observations, from dataset D, where T is the number of the images in the sample. Assuming zt is given in the data sample for now, i.e. focusing on the generative process only, we augment a scene data sample as Xa = {(xt, vt, zt)}1:T . In general, we assume an independent scene-observer relation: zt ⊥ vt|∅ but they nevertheless become dependent when the
1We define (·) as a joint sample indicator that forbids independent sampling of the random variables wherein.
corresponding observation is given: zt 6⊥ vt|xt. Under a static-scene assumption, we can treat an augmented data sample as Xa = {(xt, vt), zt}1:T where zt and vt are separable (i.e. can be sampled independently). In this case, to recover the independent generative mechanism (i.e. train a g(·)) w.r.t. scenes and observers from data, GQN [8] and MulMON [31] fix zt to z and intervene on the viewpoints vt. From a causal perspective, this can be seen as estimating p(xt
′ |do(vt = vt′), zt = z), where (xt ′ , vt ′ ) ∼ {(xt, vt)}1:T , implicitly under a causal model: zt → xt ← vt. However, in dynamic settings, the same estimation, i.e. sampling (xt ′ , vt ′ ) ∼ {(xt, vt)}1:T independently of zt, is forbidden by the (·) indicator. Intuitively, an observer cannot take more than one observations from different viewpoints at the same time t. In this paper, we refer to this issue as temporal entanglement in view of the temporal implication of the (·) indicator.

heading: 3 DyMON
text: Our goal is to train a multi-view object-centric representation learning model that recovers the independent generative mechanism of scene objects and their motions and observer motions from dynamic-scene observations. In this section, we detail how DyMON addresses these two presented challenges: 1) temporal disentanglement (see Sec. 3.1), and 2) scene spatial factorization (see Sec. 3.2). We discuss the training of DyMON in Sec. 3.3.

heading: 3.1 Temporal Disentanglement
text: The key to resolving temporal entanglement, i.e. temporal disentanglement, is to enable sampling (xt, vt) independently of zt, or (xt, zt) independently of vt. This is seemingly impossible in the multiview-dynamic-scene setting as it requires to fix either zt (static scene) or vt (single-view) respectively. In this paper, we make two assumptions about the training scenes to ensure the satisfaction of the
aforementioned two requirements without violating the global multi-view-dynamic-scene setting. Let us first describe the dynamics of scenes and observers with two independent dynamical systems:
zt+∆t − zt = fz(zt, t)∆t , vt+∆t − vt = fv(vt, t)∆t, (1)
where t and t+ ∆t are the times that two consecutive observations were taken, fz(zt, t) and fv(vt, t), or simply fzt and fvt , are the average velocities of scene objects and the observer within [t, t+ ∆t]. Note that we use a zt to capture both the shape and pose information of an object. However, we do not consider shape changes in this work. With the dynamical systems defined, we introduce our assumptions (which defines a tractable subset of all possible situations) as:
• (A1) The high-frame-rate assumption ∆t→ 0 s.t. xt+∆t ≈ xt, • (A2) The large-speed-difference assumption The data comes from one of two cases (SCFO:
Slow Camera, Fast Objects or FCSO: Fast Camera Slow Objects), that satisfy: | |fz||fv| | ≥ CSCFO or | |fz||fv| | ≤ CFCSO, where |velocity| computes a speed, and CSCFO and CFCSO are positive constants.
A1 allows us to assume a nearly static scene zt or a fixed viewpoint vt for a short period. Consider an example where we assume a static scene, i.e. zτ−∆t ≈ zτ ≈ zτ+∆t, in [τ − ∆t, τ + ∆t], A1 essentially allows us to extract zt out of a joint sample as: Xa = {(xt, vt), zt}τ−∆t:τ+∆t. An intuitive way to define A2 is: |fz| |fv| or |fz| |fv|, which specify a large speed difference between scene speeds and observer speeds.
These two assumptions enable us to accumulate instant changes (velocities) on one variable (e.g. either zt or vt) over a finite number of ∆t while ignoring the small changes of the other (assumed fixed). We then treat a slow-camera-fast-objects (i.e. SCFO) scenario, where |fz| |fv|, as an approximate single-view-dynamic-scene scenario, and a fast-camera-slow-objects (i.e. FCSO) scenario, where |fz| |fv|, an approximate multi-view-static-scene scenario. Either case allows us to resolve the temporal entanglement problem. Importantly, to answer the question: “is a given data sample an SCFO or FCSO sample”, we need to quantitatively specify the two assignment criteria CSCFO and CFCSO. However, a direct calculation of these two constants is often difficult and does not generalize as: i) |fz| is not available in unsupervised scene representation learning data, and ii) the two constants vary across different datasets. In practice, we cluster the data samples into SCFO and FCSO clusters using only the viewpoint speed |fv|, i.e. assuming |fz| = 1 for training (see Sec. 3.3). In testing, DyMON treats them equally.

heading: 3.2 Spatial Object Factorization
text: DyMON tackles scene spatial decomposition in a similar way to MulMON [31] using a generative model and an inference model. The generative likelihood of a single image observation is modelled with a spatial Gaussian mixture [38, 11]:
pθ(x t|zt = {ztk}, vt) = M∏ i=1 K∑ k=1 pθ(C t i = k|ztk) · N (xtk,i; gθ(ztk, vt), σ2I), (2)
where i indexes a pixel location (M in total) and RGB values (e.g. xtk,i) that pertain to an object k are sampled from a Gaussian distribution N (xtk,i; gθ(ztk, vt), σ2I) whose mean is determined by the decoder network gθ(·) (defined in Sec. 2) with trainable parameter θ and standard deviation σ is globally set to a fixed value 0.1 for all pixels. The mixing coefficients pθ(Ci = k|zk) capture the categorical probability of assigning a pixel i to an object k (i.e. Ci = k). This imposes a competition over the K objects as every pixel has to be explained by one and only one object in the scene.
DyMON adapts the cross-view inference module [31] of MulMON to handle: i) the cross-view object correspondence problem, ii) recursive approximation of a factorized posterior, and iii) temporal evolution of spatial structures (which indicates the major difference between the inference modules of DyMON and MulMON). The decomposition and recursive approximation of the posterior is:
p(zt = {ztk}|x6t, v6t) ≈ qΦ(zt = {ztk}|x6t, v6t) = q(z0) ∏ t qΦ(z t|xt, vt, z<t), (3)
where qΦ(zt|xt, vt, z<t) denotes the approximate posterior to a subproblem w.r.t. an observation xt taken from viewpoint vt at time t, and assumes a standard Gaussian N (0, I) for the scene prior q(z0). The intuition is to treat a posterior inferred from previous observations as the new prior to perform Bayesian inference for a new posterior based on a new observation. We use zt to denote the inferred scene representations after observing xt, i.e. a new posterior, and z<t to denote the new prior before observing xt. Note that we can advance t either regularly or irregularly. The single-view (or within-view) inference is handled by DyMON using iterative amortized inference [27] with amortization function Φ (modelled with neural networks). Refer to Appendix B. for full details about the generative and inference models of DyMON.

heading: 3.3 Training
text: To enable DyMON to learn independently the generative relationships between scenes and observations, and viewpoints and observations during training, built upon MulMON’s architecture, we break a long moving-cam-dynamic-scene sequence into short sub-sequences (see Algo. 1) where sampling (xt ′ , vt
′ ) ∼ (xt, vt)1:T independently of zt is possible. Similar to MulMON [31], we then
train DyMON by maximizing the following objective function that linearly combines an evidence lower bound (abbr. ELBO) and the log likelihood (abbr. LL) of the querying views:
L =ELBO + β ·LLquery
= 1 |T | ∑ t∈T EqΦ(zt|·)[log pθ(x t|zt, vt)]− 1 |T | ∑ t∈T DKL[qΦ(zt|x6t, v6t)||qΦ(z<t|x<t, v<t)]
+ β · 1 |T | · |Q| ∑ t∈T ∑ tq∈Q EqΦ(zt|·)[log pθ(x q|zt, vq)], (4)
where T and Q record the times when DyMON performs inference and vt interventions (i.e. viewpoint-queried generation) and β is the weighting coefficient. We construct T by sampling t (either regularly or irregularly) with a random walk through [1, T ], where a uniform distribution U{∆t − 2,∆t + 2} of an expected value ∆t (> 2) is used as the step distribution. As shown in Algo. 1, by varying the updating periods of zt and vt (denoted as ∆tz and ∆tv respectively), DyMON imitates the behaviours of a multi-view-static-scene model and a single-view-dynamic-scene model to handle the SCFO and FCSO samples respectively. In addition, using different β for the SCFO and FCSO samples allows alternating the training focus between spatial reasoning (w.r.t. objects and viewpoints) and temporal updating.
Assignment Function and Batching As the samplers of T and Q behave differently for SCFO and FCSO data (see Algo. 1), we need to determine if a X ∼ D is an SCFO sample or an FCSO sample. Under A2, we consider any dataset consisting of only a mix of SCFO and FCSO samples (where a sample is a sequence of images). For a given dataset, we cluster all training samples of a dataset into two clusters w.r.t. the SCFO and FCSO scenarios. This then gives us an assignment function, assign(X;D) (as shown in Algo. 1. In practice, to avoid breaking parallel training processes with loading SCFO and FCSO samples into the same batch, we assign the training data beforehand instead of assigning every data sample on the fly during training. This allows to batch FCSO or SCFO samples independently at every training step.

heading: 4 Related Work
text: Single-View-Static-Scene The breakthrough of unsupervised object discovery based on a primary scenario, i.e. a single-view-image setting, lays a solid foundation for the recent rise of unsupervised object-centric representation learning research. Built upon a VAE [21], early success was shown by AIR [7] that searches for one object at a time on image regions. Because AIR and most of its successors (e.g. [22]) treat objects as flat pixel patches and the image generation process as “paste flat objects on canvas” using a spatial transformer [17], they often cannot summarize well scene spatial properties that are suitable for 3D manipulation: for example, they do not render smaller objects when the objects are “moved” further away from the camera. To overcome this, most recent advances [1, 10, 24, 6, 26, 5] model a single 2D image with a spatial Gaussian mixture model [38, 11] that allows explicit handling of background and occlusions. Although these models suffer from single-view ambiguities like occlusions or optical illusions, they have the potential for attaining
Algorithm 1: DyMON Training Algorithm Input: training data D Hyperparameters |Q|, (βFCSO, βSCFO), (∆t,∆τ) ; // ∆t > ∆τ > 2, |Q| = sizeof(Q) Initialize trained parameters Φ, θ, and latent prior λ0 = {(µk = 0, σk = I)}; repeat
Sample a sequence X = {(xt, vt)}1:T ∼ D ; // T (RGB images, viewpoints) if assign(X;D) == FCSO then
β, ∆tv, ∆tz = βFCSO, ∆τ, ∆t ; // ∆tv < ∆tz, update vt more often else
β, ∆tv, ∆tz = βSCFO, ∆t, ∆τ ; // ∆tz < ∆tv, update zt more often T = random_walk_t(s = 1, e = T, step_dist = U{∆tz − 2,∆tz + 2}) ; (x, v), t, λt, ELBO, LLquery, = X[1], 1, λ
0, 0, 0; while t 6 T do
(xt, vt) = X[t] ; if mod(t,∆tv) == 0 then
v = vt ; // update v if t ∈ T then
x = xt ; // update x ELBO(t), λt = iterative_inferenceΦ,θ(x, v,λt) ; zt ∼ N (zt; λt) ; // sample updated zt Q = {tq} = sample_query_t(dist = U{t−∆tz/2, t+ ∆tz/2}, size = |Q|); for tq ∈ Q do
(xq, vq) = X[tq]; LLquery+ = (1/(|Q| · |T |)) · log pθ(xq|zt, vq) ; // fix zt, do v = vq
ELBO+ = (1/|T |) ·ELBO(t); t+ = 1;

heading: L = ELBO + β ·LLquery ; // βFCSO > βSCFO
text: θ, Φ← optimizermax(L, θ,Φ);
until θ,Φ converge;
factorized representations of 3D scenes. Our work has close relationship to IODINE [10]: we handle the object-wise inference from an image observation at each time point using the iterative amortized inference [27] design and capture the compositional generative process with a spatial Gaussian mixture model.
Multi-View-Static-Scene A natural way of resolving single-view ambiguities is to aggregate information from multi-view observations. Although multi-view scene explorations do not directly facilitate object-level 3D scene factorization, Eslami et al. [8] demonstrated that they do reduce the spatial uncertainty and enable explicit 3D knowledge evaluation—novel-view prediction. As combining GQN [8] and IODINE [10], Nanbo et al. [31] showed that MulMON effectively leverages multi-view exploration to extract accurate object representations of 3D scenes. However, like GQN, MulMON can only train on static-scene samples and thus does not generalize well to dynamic scenes ROOTS [3] combines GQN and AIR’s merits to perform multi-view-static-scene object-centric representation learning whereas it requires camera intrinsic parameters to overcome AIR’s deficiency of 3D scene learning — it is thus camera-dependent hence less general. In our work, we propose DyMON as an extension of MulMON to dynamic scenes and a unified model for unsupervised multi-view object-centric representation learning.
Single-View-Dynamic-Scene A line of unsupervised scene object-centric representation learning research was established on the single-view-dynamic-scene setting [14, 22, 19], where they explicitly model and represent object dynamics based on video observations. However, as most of these works employ a similar image composition design to AIR, they deal with only flat 2D objects that are similar to MNIST digits and thus cannot model 3D spatial properties. A closely-related work is that of Lin et al. [23], i.e. GSWM, where they modelled relative depth information and pair-wise interactions of
3D object patches. In our work, the spatial-temporal factorization allows us to show the dynamics and depths of the objects from different viewpoints at different times.
Other Related Work As a multi-view-dynamic-scene representation learning framework, TGQN [36] represents the most closely-related work to ours. It models the spatial representation learning at each time step as a stochastic process (SP) and transitions between these time-stamped SPs with a state machine. However, a notable distinction between the problems that T-GQN and DyMON are targeting based on that: 1) T-GQN does not attain object-level scene factorization and 2) a typical T-GQN requires multi-view observations at each time step (as so-called “context”) to perform spatial learning so as to get rid of the temporal entanglement problem (which has been the core focus of our work). Our work is essentially dealing with disentangled representation learning problems, which are often formulated under the frameworks of causal inference [33, 37, 35] and independent component analysis (abbr. ICA) [16, 15]. Unlike traditional disentanglement representation learning works (e.g. [13, 20, 25]) that aims at feature-level disentanglement, in this work, we handle not only the object-level disentanglement that resides in the object-centric representation learning research, but also the time-dependent scene-observer disentanglement problem.Recent trend of neural radiance field (e.g. [29, 28, 34]) are relevant to our work in the sense of 3D scene representations
using multi-view images. However, from an vision-as-inverse-graphics [40] perspective, we do not consider them scene understanding models as they only aim to memorize the volumetric structure of a single scene during “training” thus cannot perform representation inference for unseen scenes.

heading: 5 Experiments
text: We used two simulated multi-view-dynamic-scene synthetic datasets, namely DRoom and MJCArm, and a real-world dataset, namely CubeLand (see Appendix C.3 for details), in this work. We conducted quantitative analysis on DRoom and show qualitative results on the other two datasets. The DRoom dataset consists of five subsets (including both training and testing sets): one subset (denoted as DR0-|fz|) with zero object motion (multi-view-static-scene data), one subset (denoted as DR0-|fv|) with zero camera motion (single-view-dynamic-scene data), and three multi-view-dynamic-scene subsets of increasing speed difference levels from 1 to 3 (denoted as DR-Lvl.1 ∼ 3). Each of the five subsets consists of around 200 training sequences (40 frames of RGB images per sequence) and 20 testing sequences (40 frames from 12 different views, i.e. 57.6k images). Although DyMON’s focus is on a more general problem, we nevertheless compare it against two recent and specialized unsupervised object-centric representation learning methods, i.e. GSWM [23], and MulMON [31], in two respective settings: single-view-dynamic-scenes, and multi-view-static-scenes. All models were trained with 3 different random seeds for quantitative comparisons. Refer to our supplementary material for full details on experimental setups, and ablation studies and more qualitative results.

heading: 5.1 Space-Time Querying
text: The recovery of the independent generative mechanism permits DyMON to make both viewpointqueried and time-queried predictions, i.e. querying across space and times, of scene appearances and segmentations using the inferred scene representations, which enables the below two demonstrations:
Novel-view Prediction at Arbitrary Times Recall that a scene observation x is the generative product of a specific scene (composed by objects) and observer at a specific time t with a welldefined generative mapping, i.e. x = g(z, v) (see sec. 2). Like previous multi-view object-centric representation learning models (e.g. MulMON [31]), we query from an arbitrary viewpoint v w.r.t. a scene of interest z by fixing z and manually setting the viewpoint v to arbitrary configurations. Similarly, we can query about the spatial state of a dynamic scene at time t from a specific viewpoint by fixing the viewpoint and manually inputting zt at arbitrary times t to the generative function. We trained a DyMON on the DR-Lvl.3 data and show qualitatively the prediction results that are queried by space-time tuples in Figure 2.
Dynamics Replay of Scenes & Objects From Arbitrary Viewpoints In this experiment, we give DyMON a sequence of image observations of a dynamic scene as input, and have it replay the dynamics from a novel viewpoint using the scene representations it infers from the observations. This is done by fixing the v to the desired values and querying about consecutive times. As the inferred scene representations are factorized in terms of objects, we show in Figure 3 (left) that, besides the complete scene dynamics, DyMON also allows to replay the dynamics of a single object independently of the others. We present the qualitative results on the MJC-Arm datasets in Figure 3 (right) where one can see that DyMON not only replays object dynamics as global position changes, it also captures object local motions.
Dynamics On Real-World Data To demonstrate that our model has the potential for real-world applications, we conduct experiments and show qualitative results on real images (i.e. CubeLand data). We refer the readers to Appendix D.4 for the results.

heading: 5.2 Versatile Evaluation
text: DyMON is designed to handle object-centric representation learning in a general setting—multi-viewdynamic-scenes. In this section, we experiment to evaluate how well DyMON handles the specialized settings.
DyMON vs. Dynamic Scenes We first evaluate DyMON’s performance in the multi-view-dynamicscene setting in comparison to MulMON. MulMON also learns the independent generative mechanism of scene objects and observer, but with a strict static-scene constraint. Note that both DyMON and
MulMON permit novel-view predictions of scene appearances and segmentations, this allows explicit quantification of the correctness and accuracy of the inferred scene representations. We use a meansquared-error (MSE) measure and a mean-intersection-over-union score (mIoU) measure. We train DyMON on the DR-Lvl.3 subset and MulMON on the DR0-|fz| subset (because it is UNABLE to train on dynamic-scene data) and conduct comparison across the three DRoom dynamic-scene subsets (i.e. DR-Lvl.1 ∼ 3). Table 1a shows that, although we train MulMON on a more strict dataset, i.e. the DR0-|fz| dataset, DyMON still outperforms MulMON on almost all the various indicators. We show the qualitative comparison results in Figure 2 and observe that MulMON’s performance declines along the temporal axis when large object motions appear. As neither DyMON nor MulMON impose any orders for object discovery, we used the Hungarian matching algorithm to find the best match that maximizes the mIoU score to handle the bipartite matching between the output and the Ground-truth masks.
DyMON vs. Static Scenes We evaluate how well it handles multi-view-static-scene scenarios in comparison with a specialized model, i.e. MulMON. We train and test both DyMON and MulMON on the DR0-|fz| subset w.r.t. reconstructions and segmentations of both the observed and unobserved views. Table 1b summarizes the results. They show that DyMON can handle this strict constraint
setting, even though it exhibits a slight performance gap compared with the specialized model. Also, it is worth noting that DyMON and MulMON produce high variances in segmentations. One possible reason is that both MulMON and DyMON employ stochastic parallel inference mechanisms that can sometimes infer duplicate latent representations and harm segmentations [32]. This experiment along with the DyMON-versus-dynamics-scenes experiment provides useful guidance for model selection in multi-view applications—use a specialized model in a well-controlled environment and DyMON to handle complex scenarios.
DyMON vs. Fixed-View Observations of Dynamic Scenes We assessed DyMON’s performance on handling single-view-dynamic-scene observations by comparing it with GSWM [23], which is a specialized object-centric representation model for this specific setting, although it is unable to achieve pixel-level segmentation. We train both DyMON and GSWM on the DR0-|fv| subset and measure the reconstruction quality of the observations. Table 1c shows that DyMON not only outperforms GSWM in observation reconstruction, but it also permits pixel-wise segmentation which the specialized model cannot. The qualitative results in Figure 4 show that GSWM learns better object appearances (especially for textures) than DyMON, whereas DyMON learns more accurate scene dynamics than GSWM. This is understandable as GSWM models object dynamics explicitly, which introduces risks of overfitting the observed motions. DyMON supports well temporal interpolations, i.e. dynamics replays (as shown in Figure 3 and 4), but it does not model the object dynamics nor interactions explicitly like GSWM. As a result, it does not provide readily extrapolatable features along the time (or dynamics) axis for predicting into the future.
DyMON vs. T-GQN T-GQN [36] is a closely related work as it targets unsupervised scene representation learning in the multi-view-dynamic-scene settings, even though it does not attain object-centric factorization in the latent space. Although T-GQN requires multi-view observations at each time step (as “context” information) to sidestep the temporal entanglement issue, we nevertheless train it on our DRoom data and show that it fails to represent the DRoom scenes (see Appendix D.3 for the results and discussions).

heading: 6 Conclusion
text: We have presented Dynamics-aware Multi-Object Network (DyMON), a method for learning objectcentric representations in a multi-view-dynamic-scene setting. We have made two weak assumptions that allows DyMON to recover the independent generative mechanism of observers and scene objects from both training and testing multi-view-dynamic-scene data—achieving spatial-temporal factorization. This permits querying the predictions of scene appearances and segmentations across both space and time. As this work focuses on representing the spatial scene configurations at every specific time point, i.e. DyMON does not model dynamics explicitly so it cannot predict the future evolution of scenes, which leaves space for future exploration.
Acknowledgments and Disclosure of Funding
The first authors would like to acknowledge the School of Informatics, the University of Edinburgh for providing his PhD scholarship. This research is partly supported by the Trimbot2020 project, which is funded by the European Union Horizon 2020 programme. The authors would like to thank Prof. C. K. I. Williams and Cian Eastwood for valuable discussions.

Review summaries:
The following work proposes a method for learning the factorized latent structure of a scene. Specifically, while the primary prior work MulMon [28] assume static objects while the viewpoint varies through time, this work allows for both camera and object motion within the training data. To achieve their goal of factorizing object motion from camera motion, they assume high framerate and use clustering to determine cases where the camera is moving quickly and the objects slowly, versus cases where the camera moves slowly but the objects move quickly. Notably, once trained, they are able to resynthesize the scene's dynamic objects from novel viewpoints. 

This paper proposes DyMON, an unsupervised object-centric representation learning model that can be applied to scenes with dynamic objects and multiple viewpoints. DyMON is trained to obtain object latents that are independent of the viewpoints. The model builds off of MulMON, extending it to work with scenes with dynamic objects. Two assumptions are made about the training scenes: 1) a high frame rate assumption and 2) the relative speed difference between the camera and the objects is large -- that is, either the camera moves much faster than the objects or the objects move much faster than the camera. With these assumptions, the problem can be reduced to either a multi-viewpoint, static objects problem or a single-viewpoint, dynamic objects problem. Through experiments, the authors demonstrate that DyMON can be used to query predictions of the scene at arbitrary times and viewpoints.