Object-Centric Representation Learning with Generative Spatial-Temporal Factorization
Authors: Li Nanbo
Muhammad Ahmed Raza
Hu Wenbin
Zhaole Sun
Robert Burns Fisher
Venue: NeurIPS 2021 Poster (2021)
Keywords: Object-centric Representation Learning
Deep Generative Models
Computer Vision

Abstract: Learning object-centric scene representations is essential for attaining structural understanding and abstraction of complex scenes. Yet, as current approaches for unsupervised object-centric representation learning are built upon either a stationary observer assumption or a static scene assumption, they often: i) suffer single-view spatial ambiguities, or ii) infer incorrectly or inaccurately object representations from dynamic scenes. To address this, we propose Dynamics-aware Multi-Object Network (DyMON), a method that broadens the scope of multi-view object-centric representation learning to dynamic scenes. We train DyMON on multi-view-dynamic-scene data and show that DyMON learns---without supervision---to factorize the entangled effects of observer motions and scene object dynamics from a sequence of observations, and constructs scene object spatial representations suitable for rendering at arbitrary times (querying across time) and from arbitrary viewpoints (querying across space). We also show that the factorized scene representations (w.r.t. objects) support querying about a single object by space and time independently. 

Object-Centric Representation Learning with Generative Spatial-Temporal Factorization
Li Nanbo School of Informatics University of Edinburgh nanbo. 1:10e @ed. ac. ackuk
Muhammad Ahmed Raza School of Informatics University of Edinburgh m. razaded ac.uk
Hu Wenbin
School of Informatics University of Edinburgh wenbin hu@ed ac uk
Zhaole Sun School Informatics University of Edinburgh zhaole. sun@ed. ac. uk
Robert B. Fisher School of Informatics University of Edinburgh rbf@inf ed. ac uk
Abstract
Learning object-centric scene representations essential for attaining structural understanding and abstraction of complex scenes. Yet, as current approaches for unsupervised object-centric representation learning are built upon either a static scene assumption, they often: i) suf-
stationary observer assumption fer single-view spatial ambiguities, or ii) infer incorrectly or inaccurately object representations from dynamic scenes. To address this, propose Dynamics- aware Kulti-Object Network (DyMON), a method that broadens the scope multi-view object-centric representation learning to dynamic scenes. We train Dy- data and show that DyMON learns-without supervision to factorize the entangled effects of observer motions and scene object dynamics from a sequence of observations, and constructs scene object spatial representations suitable for rendering arbitrary times (querying across time) and from arbitrary viewpoints (querying across space). We also show that the factorized scene representations (w.r.. objects) support querying about single object by space and time independently.
MON on mullii-view-dynmmi---ee
Introduction
Object-centric representation learning promises improved interpretability, generalization, and data- efficient learning various downstream tasks like reasoning (e.g. [18 (9] and planning (c.g. [30] 21) It aims at discovering compositional structures around objects from the raw sensory input data, i.e. binding problem [12,, where the segregation (i.e. factorization) is the major challenge (e.g. [9] especially in cases of no supervision. In the context of visual data, most existing focus has been single-vies view settings, i.e. decomposing and representing 3D scenes based
single 2D image [T 26 or a fixed-view video [23]. These methods often suffer from single-view spatial ambiguities and thus show several failures inaccuracies representing 3D scene properties. It was demonstrated by Nanbo et al. (31] that such ambiguities could be effectively resolved by multi-view information aggregation. However, current multi-view models built upon a foundational static- scene assumption. As result, they: 1) require static-scene data for training and 2) cannot handle well dynamic scenes where the spatial structures evolve over time. This greatly harms model's potentials in real-world applications.
In this work, target unexplored problem problem- unsupervieed object-centric latent representation scenarios. Despite the importance of the problem to spatial-
learning in wiei--viiww--iiiiee
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

temporal understanding of 3D scenes, solving it presents several technical challenges. Consider one particularly interesting scenario where both an observer (e.g. a camera) and the objects in the scene are moving at the same time. To aggregate 3D object information from two consecutive observations, an agent needs not only to handle the cross-view object correspondence problem [3]] but also to reason about the independent effects of the scene dynamics and observer motions. One can consider the aggregation as a process of answering two questions: "how much has an object really changed in the 3D space" and "what previous spatial unclarity can be clarified by the current view". In this paper, we refer to the relationship between the scene spatial structures and the viewpoints as the temporal entanglement because the temporal dependence of them complicates the identification of the independent generative mechanism [35]
Network). a unified unsupervised framework for view object-centric representation learning. Instead of making a strong assumption of static scenes as that in previous multi-view methods, we only make two weak assumptions about the training scenes: i) observation sequences are taken at a high frame rate, and ii) there exists a significant difference between the speed of the observer and the objects (see Sec. 3 Under these two assumptions, in a short period, we can transition a muult--vie-dyyammi--ee multi-view-stattic- stati problem if an observer moves faster than a scene evolves, or to a single-view- dynamic-scent problem if a scene evolves faster than an observer moves. These local approximations allow DyMON to learn independently the generative relationships between scenes and observations, and viewpoints and observations during training, which further enable DOMON to address the problem of scene -ential-eemporl factorization, i.c. solving the observer-scene temporal entanglement and scene object decomposition, at test time.
We introduce DyMON (Dynamics-award
problem to a
Through the experiments we demonstrate that: (i) DyMON represents the first unsupervised multi- view object-centric representation learning work in the context of dynamic-scene settings that can train and perform object-oriented inference on mulff-view-dynumii--ceee
data (see Sec. 5). (ii) DyMON recovers the independent generative mechanism of an observer and scene objects from observations and permits querying predictions of scene appearances and segmentations across both space and time (see Sec. Sec.5.II. (iii) As DyMON learns scene representations that are factorized in terms of objects, DyMON allows single-object manipulation along both the space (i.e. viewpoint) and time axis-e.g. replays dynamics of single object without interferring the others (see Sec. 5.I).
2 Background
Object-centric Representations Consider object-centric representation inference the inverse problem of an observation generation problem (i.e. the Dision-as-inverss graahicss [40] idea). In the forward process, i.e. observation generation, ). where a 2 E R²' specifies one and only one object in the scene. An observation of the scene x, e.g. an image æ R.M
have scene well-defined by set of parameter
vectors z = {2 = {2r,
RGB image E TR can be taken only by a specified observer (often defined as v E Rd)) which is independent of the scene in the forward problem, using a specific mapping g RD x I2d + M Assuming deterministic process, an observation x is generated as x = g(z, v), where u is often omitted in single-view scenarios (e.g. 10). With the forward problem defined, we can describe the goal of learning an object-centric representation as inferring the intrinsic parameters of the objects } that compose scene z based on the scene observation x. In other words, computing a factorized posterior p(z[x) p(21, 22.... ...,.............. R [x), even though it is computationally intractable. As the number of objects is unknown in the inverse problem, it is worth noting that i) K is often set globally to be a sufficiently large number (greater than the actual number of objects) to capture all scene objects, and ti) we allow empty "slots".
Temporal Entanglement The dynamic nature of the world suggests that the spatial configuration of a scene (denoted by z²) and an observer of are bound the specific time t that an observation is taken (i.e. x' = g(z", v²)). Let X = {(x', , /tr represent a data sample, e.g. a sequence or set of multi-view image observations, from dataset D. where T' is the number of the images in the sample. Assuming z' is given in the data sample for now, i.e. focusing on the generative process only, we augment a scene data sample as X, = {(x', v². z In general, we assume an independent scene-observer relation: z' 1 o² but they nevertheless become dependent when the
We define (.) as a joint sample indicator that forbids independent sampling of the random variables wherein.

---

vt
Viewpoint querryinttuuuiiivv Querying
u²
Camera trajectory Object trajectories
t
Z
V² 2
e t
t
Time Querying
t
A
Z Z Z
S Spatial-temporal Factorization
unobservable
Rec. & Seg. of the observations
setup. v with a time index superscript denotes the spatial configuration (e.g. position, orientation, etc.) of an observer at specific time. We highlight one particular interesting, yet unexplored, scenario where both an observer and scene objects are moving at the same time-which entangles the independent effects of the observer's and scene objects' motions on an scene observation, an image sequence (see bottom left). A latent variable z that is indexed by time describes the objects and their spatial configuration detailed definition). Right: DyMON decouples the generative effects of observer motions and scene object motions and enables: 1) reconstruction and factorization of the observed views (see bottom right), and 2) novel-view appearance and decomposition prediction for arbitrary times-querye across both space and time (see top right).
Figure 1: Top Left: mulli-view-dyneee
specific time (See Sec. for
corresponding observation is given: z' "0(x". Under a static-scene assumption, we can treat an augmented data sample as X, {xx, o²), z' hir where z' and of are separable (i.e. can be sampled independently). In this case, to recover the independent generative mechanism (i.e. train a g(-)) scenes and observers from data, GQN { and MulMON (31 fix z' and intervene the viewpoints o², From causal perspective, this can be seen estimating p(x" (do(v' =f )f', = z). where (st",, oo"") {(x', ):rr, implicitly under causal model: zt x² <- v.. However, in dynamic settings, the same estimation, i.e. sampling (a" w²") {(x", "²)] independently of z' is forbidden by the (.) indicator. Intuitively, an observer cannot take more than one observations from different viewpoints at the same time t. this paper, we refer to this issue as temporal entanglement in view of the temporal implication of the (-) indicator.
3 DyMON
Our goal to train multi-view object-centric representation learning model that recovers the independent generative mechanism of scene objects and their motions and observer motions from dynamic-scene observations. In this section, we detail how DyMON addresses these two presented challenges: temporal disentanglement (see Sec. 3.I. and scene spatial factorization (see
Sec. 3.2
We discuss the training of DyMON in Sec.
3.3
3.1
Temporal Disentanglement
The key to resolving temporal entanglement, i.e. temporal disentanglement, to enable sampling (at, w²) independently of x' or (x', x') independently of vf. This is seemingly impossible in the multi- new-dyyamic-scene setting as it requires to fix either 2 (statie scene) or v' (single-view) respectively. In this paper, make two assumptions about the training scenes to ensure the satisfaction of the

---

aforementioned two requirements without violating the global seaffi-view-yylee seeew-dynami-sceen setting. Let us first describe the dynamics of scenes and observers with two independent dynamical systems:
z - =' fa t)
v'+ - of Fo ,t)tt
(1)
where and ++ At are the times that two consecutive observations were taken, Fz .) and Fo .t,, or simply for and for, are the average velocities of scene objects and the observer within [t, t + At]. Note that we use 2f to capture both the shape and pose information of an object. However, we do not consider shape changes in this work. With the dynamical systems defined, introduce assumptions (which defines a tractable subset of all possible situations) as:
(A1) The high-frame-rate assumption At + 0 s.t. xt+At xe x²,
(A2) The large-speedddiffeenee assumption The data comes from one of two cases (SCFO:
Slow Camera, Fast Objects or FCSO: Fast Camera Slow Objects), that satisfy: F
CsCFO or Croso, where velocitty computes a speed, and CsCFO and Craso are positive constants.
A1 allows us to assume nearly static scene z' or fixed viewpoint of for a short period. Consider an example where we assume static scene, i.e. z"-Ar x z" R = in [r A + At). A1 essentially allows us to extract z' out of joint sample as: X. = {(x', o z' J An
intuitive way to define A2 is: [Ja] > Jol or [Ta] < [fo], which specify large speed difference between scene speeds and observer speeds.
These two assumptions enable us to accumulate instant changes (velocities) on one variable (c.g. either z' or v²) over a finite number of At while ignoring the small changes of the other (assumed fixed). We then treat a stow-camera-fast-objieee an approximate single-view-dyuumic scene scenario, and fjss-cameru-slov-ojjeee (i.e. FCSO) scenario, where [fx < [Ful. approximate cillii-view-sattic-o to resolve the temporal entanglement problem. Importantly, to answer the question: "is a given data sample an SCFO or FCSO sample", need to quantitatively specify the two assignment criteria CsCFO and Creso- However, a direct calculation of these two constants is often difficult and does not generalize as: i) [&| is not available in unsupervised scene representation learning data, and ii) the two constants vary across different datasets. In practice, we cluster the data samples into SCFO and FCSO clusters using only the viewpoint speed [/r] i.e. assuming [Tx = 1 for training (see In testing, DyMON treats them equally.
(i.e. SCFO) scenario, where |fal > [ful. as
scenario. Either case allows us
Sec. 3.3
3.2 Spatial Object Factorization
DyMON tackles scene spatial decomposition in a similar way to MulMON B1] using generative model and an inference model. The generative likelihood of a single image observation is modelled with a spatial Gaussian mixture [38] 11:
K
po(a?12' = (21 vz) II
Naak. uo(=k,u'), o
where i indexes pixel location (M in total) and RGB values (e.g. zf,0) that pertain to an object k are sampled from Gaussian distribution N( go u²), I²I) whose mean is determined by the decoder network (-))) (defined in Sec. 2 with trainable parameter 0 and standard deviation o is globally set to fixed value 0.1 for all pixels. The mixing coefficients po Po = klzx) capture the categorical probability of assigning a pixel i to an object (i.e. C, k). This imposes a competition over the K objects as every pixel has to be explained by one and only one object in the scene.
DyMON adapts the cross-view inference module (31) of MulMON handle: i) the cross-view object correspondence problem, ii) recursive approximation of a factorized posterior, and iii) temporal evolution of spatial structures (which indicates the major difference between the inference modules of DyMON and MulMON). The decomposition and recursive approximation of the posterior is:
p(z = Hz EI R gx
H St . = 9(2") II go (z lic", v0,2 =t))
(3)

---

where qu 1z']2", v²) denotes the approximate posterior to subproblem w.r.t. an observation x' taken from viewpoint r' at time and assumes a standard Gaussian (0 I) for the scene prior ((2)). The intuition is to treat posterior inferred from previous observations as the new prior to perform Bayesian inference for new posterior based on a new observation. We use z' denote the inferred scene representations after observing x², i.e. new posterior, and z to denote the new prior before observing x'. Note that we can advance t either regularly or irregularly. The single-view (or within-view) inference is handled by DyMON using iterative amortized inference [27] with amortization function & (modelled with neural networks). Refer to Appendix B. for full details about the generative and inference models of DyMON.
3.3 Training
To enable DyMON to learn independently the generative relationships between scenes and obser- vations, and viewpoints and observations during training, built upon MulMON's architecture, we break long moving-cam cam-dynamie-scem sequence into short sub-sequences (see Algo. I where sampling (xx w²) ~ (xc; "') independently of z' is possible. Similar to MulMON [3I, we then train DyMON by maximizing the following objective function that linearly combines evidence lower bound (abbr. ELBO) and the log likelihood (abbr. LL) of the querying views:
C -EELBO + 3. LL query
Σ
Eg (x (o
Σ EET
T
[
LET
+ +
Σ LET 1E
E ( (= Σ
(4)
T 12
where T and Q record the times when DyMON performs inference and of interventions (i.e. viewpoint-queried generation) and 8 is the weighting coefficient. We construct T by sampling t (either regularly or irregularly) with a random walk through TJ. where a uniform distribution Ut 2, At + 2} of an expected value At (> 2) is used as the step distribution. As shown in Algo. 1 by varying the updating periods of z' and of (denoted as At, and At, respectively), DyMON settiiiiew-sstaiii-e
model and handle the SCFO and FCSO samples respectively. In addition, using different for the SCFO and FCSO samples allows alternating the training focus between spatial reasoning (w.r.t. objects and viewpoints) and temporal updating.
imitates the behaviours of
gegle-view-dynmii--
model to
Assignment Function and Batching As the samplers of T and Q behave differently for SCFO and FCSO data (see Algo. [] we need to determine if a X ~ D an SCFO sample or an FCSO sample. Under A2 we consider any dataset consisting of only mix of SCFO and FCSO samples (where a sample is a sequence of images). For a given dataset, we cluster all training samples of a dataset into two clusters w.r.t. the SCFO and FCSO scenarios. This then gives an assignment function, assign X; D) (as shown in Algo. [ In practice, to avoid breaking parallel training processes with loading SCFO and FCSO samples into the same batch, we assign the training data beforehand instead of assigning every data sample on the fly during training. This allows to batch FCSO or SCFO samples independently at every training step.
4 Related Work
Soggle-iew--Sttti-Siiee The breakthrough of unsupervised object discovery based on a primary scenario, i.c. single-view-image setting, lays solid foundation for the recent rise of unsupervised object-centric representation learning research. Built upon VAE [21], carly success was shown by AIR [7] that searches for one object at a time on image regions. Because AIR and most of its successors (e.g. [22) treat objects as flat pixel patches and the image generation process as "paste flat objects on canvas" using a spatial transformer
[17), they often cannot summarize well scene spatial properties that are suitable for 3D manipulation: for example, they do not render smaller objects when the objects are "moved" further away from the camera. To overcome this, most recent advances [ 10 24 26 5 model single 2D image with spatial Gaussian mixture model [38 []] that allows explicit handling of background and occlusions. Although these models suffer from single-view ambiguities like occlusions optical illusions, they have the potential for attaining
5

---

Algorithm 1: DyMON Training Algorithm
Input: training data 'D
Hyperparameters 2), (Brcso, Bscro), (A,, AT) // A > Ar > 2, Q = sizeof (
Initialize trained parameters Φ 8, and latent prior X {(1x= = 0, ok = I));
repeat
Sample sequence X = {2=, . h'r D; if assign X; D) FCSO then
1/ T (RGB images, viewpoints)
L 8, Ar. Alz = Brcso, AT, At;
1/ Al, < Ala, update of more often
else
L 3, pt At, Bscro, At, ÂT;
1/ At, St., update z' more often
T= random factlo__wlkkttie 1,e T, step_tteee dist 2/(1(11, Al + +
vI, t, X; ELBO, LL gueuryy: [(1,, 1, 2o 0, 0:
while t < do
(a", v²) X
if (t. At,) = 0 then
0 vt;
// update
if T then
x = >x;; (LBO X iterative itcrative_ (2, X))
// update
z" ~ N(z'; x);
// sample updated z'
Q= [ty] = sample hample_query..ee query t(disttttitttttiii = Ui(t Ate/2, t At, /2), size = [2)):
for lg do
(x 09) = X(ty]:
+ = (1/(12) (TT)) log Log pog
11 fix z', do v ~
ELBO = (1/[T) ELBO(!;
1
ELBO + 3 Lggery:
Brcso > Ascro
0, $ < optimizer max max
(, D);
unil 0, th converge;
factorized representations of 3D scenes. Our work has close relationship IODINE [00:
we handle
the object-wise inference from an image observation each time point using the derative amortized inference [27] design and capture the compositional generative process with a spatial Gaussian mixture model.
Multi- View-Static-Seene A natural way resolving single-view ambiguities is aggregate in- formation from multi-view observations. Although multi-view scene explorations do not directly facilitate object-level 3D scene factorization, Eslami al. g demonstrated that they do reduce the spatial uncertainty and enable explicit 3D knowledge evaluation novel-view prediction. As combining GQN g and IODINE [0], Nanbo al. 31 showed that MulMON effectively leverages multi-view exploration to extract accurate object representations of 3D scenes. However, like GQN, MulMON can only train on static-scene samples and thus does not generalize well dynamic scenes seltiivvee--stti---seeee
object-centric representation learning whereas it requires camera intrinsic parameters to overcome AIR's 's deficiency of 3D scene learning it is thus camera-dependent hence less general. In our work, we propose DyMON as an extension of MulMON to dynamic scenes and a unified model for unsupervised multi-view object-centric representation learning.
ROOTS 3 combines GQN and AIR's merits to perform
Siale-iiew--Dnami--Seeeee A line of unsupervised scene object-centric representation learning research was established on the model and represent object dynamics based on video observations. However, as most of these works employ similar image composition design to AIR, they deal with only flat 2D objects that are similar MNIST digits and thus cannot model 3D spatial properties. closely-related work is that of Lin et al. (23). i.c. GSWM, where they modelled relative depth information and pair-wise wise interactions of
wwwieeewyaaicsseeeeee setting [14 22 19 where they explicitly

---

V
App
Sey
App.
Sog
App
Seg
App.
Sey
GT.
MM
DM
Viewpoint (Space) Querying
GT.
MM
DM
GT.
MM
Time Querying
t (z
Figure 2: Qualitative results of spatial-tempona factorization. The GT rows show the true scene. The "MM" and "DM" entries are the scene re-rendered from the corresponding models, i.e. MulMON and DyMON respectively. The vertical row pairs show the results from viewpoint changes and the horizontal direction shows the results different times. Note that train MulMON and DyMON on different datasets as MulMON cannot train on multi-view view-dynnami-scomm datasets. We also visualize MulMON's tendency of generating degenerated results along the temporal direction (marked with red arrows).
3D object patches. In work, the spatial-temports factorization allows us show the dynamics and depths of the objects from different viewpoints at different times.
representation learning framework, T- GQN (36) represents the most closely-related work to ours. It models the spatial representation learning each time step as stochastic process (SP) and transitions between these time-stamped
Other Related Work As
palli-view-ddnmmic--eee
SPs with a state machine. However, a notable distinction between the problems that T-GQN and DyMON are targeting based on that: 1) T-GQN does not attain object-level scene factorization and 2) a typical T-GQN requires multi-view observations at each time step (as so-called "context") to perform spatial learning so get rid of the temporal entanglement problem (which has been the core focus of our work). Our work is essentially dealing with disentangled representation learning problems, which are often formulated under the frameworks of causal inference [33] 37 351 and inde- pendent component analysis (abbr. ICA) [6 15]. Unlike traditional disentanglement representation learning works (e.g. [1] 20 25 that aims at feature-level disentanglement, this work, we handle not only the object-level disentanglement that resides in the object-centric representation learning research, but also the time-dependent scene-observer disentanglement problem. Recent trend neural radiance field (e.g. [29 28, 34) are relevant our work in the sense of 3D scene representations
7

---

using multi-view images. However, from an vision-as inweere--raapiiiiee [40] perspective, we do not consider them scene understanding models as they only aim to memorize the volumetric structure of a single scene during "training" thus cannot perform representation inference for unseen scenes.
Experiments
We used two simulated Arm, and a real-world dataset, namely CubeLand (see Appendix C.3 for details), in this work. We conducted quantitative analysis on DRoom and show qualitative results on the other two datasets. The DRoom dataset consists of five subsets (including both training and testing sets): one subset (denoted as DRO-[2) with zero object motion (multi-veev-static--ee with zero camera motion wanngle-vew-dynanii--eeee
synthetic datasets, namely DRoom and MJC-
data), one subset (denoted as DRO- data), and three mulfi-view--lyammic--eee subsets of increasing speed difference levels from 1 to 3 (denoted as DR-Lv1.1 ~ 3). Each of the five subsets consists of around 200 training sequences (40 frames of RGB images per sequence) and 20 testing sequences (40 frames from 12 different views, i.e. 57.6k images). Although DyMON's focus is on a more general problem, we nevertheless compare it against two recent and specialized unsupervised object-centri representation learning methods, i.e. GSWM (23), and MulMON [I], in two respective settings:
and multi-view wwww-wtttwe--eeeee static-scenes. All models were trained with 3 different random seeds for quantitative comparisons. Refer to our supplementary material for full details on experimental setups, and ablation studies and more qualitative results.
5.1 Space- Time Querying
The recovery of the independent generative mechanism permits DyMON to make both viewpoint- queried and time-queried predictions, i.c. querying across space and times, of scene appearances and segmentations using the inferred scene representations, which enables the below two demonstrations:
Novel-view Prediction at Arbitrary Times Recall that a scene observation x is the generative product of specific scene (composed by objects) and observer at specific time t with well- defined generative mapping, i.e. x = g(z, v) (see sec. 2 Like previous multi-view object-centric representation learning models (e.g. MulMON (3I), we query from an arbitrary viewpoint v w.r.t. a scene of interest z by fixing z and manually setting the viewpoint v to arbitrary configurations. Similarly, can query about the spatial state of dynamic scene at time / from a specific viewpoint by fixing the viewpoint and manually inputting 2' at arbitrary times to the generative function. We trained a DyMON on the DR-Lv1.3 data and show qualitatively the prediction results that are queried by space-time tuples in Figure 2
Dynamics Replay of Scenes & Objects From Arbitrary Viewpoints In this experiment, we give DyMON a sequence of image observations of a dynamic scene as input, and have it replay the dynamics from a novel viewpoint using the scene representations it infers from the observations. This is done by fixing the v to the desired values and querying about consecutive times. As the inferred scene representations are factorized in terms of objects, we show in Figure (left) that, besides the complete scene dynamics, DyMON also allows to replay the dynamics of a single object independently of the others. We present the qualitative results on the MJC-Arm datasets in Figure (right) where one can see that DyMON not only replays object dynamics as global position changes, also captures object local motions.
Dynamics On Real- World Data To demonstrate that our model has the potential for real-world applications, we conduct experiments and show qualitative results on real images (i.e. CubeLand data). We refer the readers to Appendix D.4 for the results.
5.2 Versatile Evaluation
DyMON is designed to handle object-centric representation learning in general setting-multi-ie multi-view- dynamic-scenes. In this section, experiment to evaluate how well DyMON handles the specialized settings.
DyMON vs. Dynamic Scenes We first evaluate DyMON's performance in the viy--id scene setting in comparison to MulMON. MulMON also learns the independent generative mechanism of scene objects and observer, but with strict static-scene constraint. Note that both DyMON and
8

---

v
DRoom
MJC-Arm
Obs.
Obs.
Scn.
v9
Rec,
Obj.
Scn.
v1
v9
Figure 3: Left: DyMON performing dynamics replays on the DRoom dataset, where the first row is the observation sequence input to DyMON, second and third rows show replays of the scene dynamics (all objects' original motions) and object dynamics (just the foreground green ball moves) respectively from an arbitrary viewpoint v², Right: DyMON replays local motions of robot arm from an arbitrary viewpoint (top: observation, middle: reconstruction, bottom: replay from a higher viewpoint).
MSEL
mloUt
Models
Rbs.
Nv.Obs.
Obs.Seg.
Nv.Seg.
MulMON
0.011 +: 0.001
0.019 0.002
0.511 0.001
0.461 1: 0.062
DyMON
0.004 +: 0.001
0.021 = 0.002
0.717 + 0.000
0.508 + 0.065
(a) DyMON vs. Multi View Son--Dyaamc--ern Scenes
MSEJ
mloUt
MSEJ
mloUt
Models
Rbs.Rcc.
Nv.Ots.
Obg
Nv.Seg
Models
Obs Rec.
Obs.Seg.
MuIMON
1: 0.001
0.012 1 0.005
0 0.583
0.080
0.538
GSWM
0.039 0.007
0.082
DyMON
0.0144 = 0.001
+ 1.007
0.529 0.00G
0L506
0.106
DyMON
0.014 + 0.011 0.682 + 0.107
(b) DyMON vs. Multi View-Static-Seeet
(c) DyMON vs. Single-View-De View-Veeeveeeeee Niew-Dynamic-Seee
Table 1: Quantitative comparisons of DyMON and two baseline models, i.e. GSWM and MulMON, in handling scenarios that the baseline models are specialized at. The models in table (a) are trained and tested on the DRO-IFo data, and those in (b) and (c) are trained and tested on the DRO-| data. "Obs." tags reconstructions and segmentations that computed for the observations and "Nv." tags those from novel viewpoints. Mean + stddev for 3 training seeds. 1 indicates higher is better and + indicates the opposite.
MulMON permit novel-view predictions of scene appearances and segmentations, this allows explicit quantification of the correctness and accuracy of the inferred scene representations. We use a mean- squared-error (MSE) measure and a mean intersection-over. DyMON the DR-Lvi.3 subset and MulMON the DRO- subset (because it is UNABLE to train on dynamic-scene data) and conduct comparison across the three DRoom dynamic-scene subsets (i.e. DR-Lvi.1 3). Table Ta shows that, although train MulMON on more striet dataset, i.c. the DRO- Fa dataset, DyMON still outperforms MulMON indicators. We show the qualitative comparison results in Figure 2 and observe that MulMON's performance declines along the temporal axis when large object motions appear. As neither DyMON nor MulMON impose any orders for object discovery, we used the Hungarian matching algorithm find the best match that maximizes the mIoU score to handle the bipartite matching between the output and the Ground-truth masks.
score (mloU) measure. We train
almost all the various
We evaluate how well it handles comparison with specialized model, i.e. MulMON. We train and test both DyMON and MulMON on the DRO-I/2 subset w.r.t. reconstructions and segmentations of both the observed and unobserved views. Table lb summarizes the results. They show that DyMON can handle this strict constraint
DYNON vs. Static Scenes
scenarios in

---

setting, even though it exhibits a slight performance gap compared with the specialized model. Also, is worth noting that DyMON and MulMON produce high variances in segmentations. One possible reason is that both MulMON and DyMON employ stochastic parallel inference mechanisms that can sometimes infer duplicate latent representations and harm segmentations [32]. This experiment along with the selection in multi-view applications use a specialized model in a well-controlled environment and DyMON to handle complex scenarios.
experiment provides useful guidance for model
use
GT.
DM
GSWM
Figure 4: Left: Qualitative comparisons of DyMON and GSWM on reconstructing the DRO- scenes. The GT rows show the actual observations of a dynamic scene, and the "DM" and "GSWM" rows show observation reconstruction results of DyMON and GSWM, respectively.
DyMON vs. Fixed-View Observations of Dynamic Scenes We assessed DyMON's performance observations by comparing it with GSWM (23), which is
on handling specialized object-centric representation model for this specific setting, although it is unable to achieve pixel-level segmentation. We train both DyMON and GSWM the DRO- subset and measure the reconstruction quality of the observations. Table [c shows that DyMON not only outperforms GSWM observation reconstruction, but it also permits pixel-wise segmentation which the specialized model cannot. The qualitative results in 4 that GSWM learns better object appearances (especially for textures) than DyMON, whereas DyMON learns more accurate scene dynamics than GSWM. This is understandable as GSWM models object dynamics explicitly, which introduces risks of overfitting the observed motions. DyMON supports well temporal interpolations, iee dynamics replays (as shown in Figure and 4) but it does not model the object dynamics nor interactions explicitly like GSWM. As result, does not provide readily extrapolatable features along the time (or dynamics) axis for predicting into the future.
MyMON vs. T-GQN T-GQN 36 is closely related work as it targets unsupervised scene represen- settings, even though it does not attain object-centric
tation learning in the mullt-vvew--dyammic.o factorization in the latent space. Although T-GQN requires multi-view observations each time step (as "context" information) to sidestep the temporal entanglement issue, we nevertheless train it on our DRoom data and show that it fails represent the DRoom scenes (see Appendix D.3 for the results and discussions).
6 Conclusion
We have presented Dynamics-aware Multi-Object Network (DyMON), method for learning object- centric representations in multi-view--dnnmmic--ee setting. We have made two weak assumptions that allows DyMON to recover the independent generative mechanism of observers and scene objects from both training and testing multi-view- mulli-view-dyynamicsseene data data-achieving spatial-temporal factorization. This permits querying the predictions of scene appearances and segmentations across both space and time. As this work focuses representing the spatial scene configurations every specific time point, i.e. DyMON does not model dynamics explicitly so it cannot predict the future evolution of scenes, which leaves space for future exploration.
10

---

Acknowledgments and Disclosure of Funding
The first authors would like to acknowledge the School of Informatics, the University of Edinburgh for providing his PhD scholarship. This research is partly supported by the Trimbot2020 project. which is funded by the European Union Horizon 2020 programme. The authors would like thank Prof. C. K. I. Williams and Cian Eastwood for valuable discussions.
References
[1] C. P. Burgess, L. Matthey, N. Watters, R. Kabra, Higgins, M. Botvinick, and A. Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 119901/139900 13300, 2019.
[2] D. Carlos, A. Cohen, and M. L. Littman. An object-oriented representation for efficient reinforcement learning. In International Conference on Machine Learning, pages
2008
[3] C. Chen, F. Deng, and Ahn. Object-centric representation and rendering 3d scenes. JMLR, 2021.
[4] A. Didolkar, A. Goyal, R. Ke, C. Blundell, P. Beaudoin, N. M. O. Heess, Mozer, and Y. Bengio, Neural production systems. In Advances in Neural Information Processing Systems, 2021.
M. Engeleke, O. P. Jones, and Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. arXiv preprint (tt:::2004.99900 2021.
[6] Engelcke, A. Kosiorek, Jones, and Posner. Genesis:
Generative scene inference and sampling with object-centric latent representations. International Conference on Learning Representations, 2019.
[7] S. Eslami, Heess, Weber, Tassa, D. Szepesvari, and E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pages 3225-3233, 2016.
2016.
[8] S. A. Eslami, D. J. Rezende, F. Besse, F. Viola, A. S. Morcos, M. Garnelo, A. Ruderman, A. A. Rusu, I. Danihelka, K. Gregor, et al. Neural scene representation and rendering. Science, 360K6394) 2018.
[9] A. Goyal, A. Lamb, P. Gampa, P. Beaudoin, S. Levine, C. Blundell, Y. Bengio, and M. Mozer. Object files and schemata: Factorizing declarative and procedural knowledge in dynamical systems. In International Conference on Learning Representations, 2020.
[10] K. Greff, R. L. Kaufman, R. Kabra, N. Watters, C. Burgess, D. Zoran, L. Matthey, M. Botvinick, and A. Lerchner. Multi-object representation learning with iterative variational inference. In Proceedings of the 36th International Conference on Machine Learning, pages 2424-2433, 2019.
[11] K. Greff, S. Van Steenkiste, and J. Schmidhuber. Neural expectation maximization. In Advances in Neural Information Procexsing Systems, pages 6691 1-7011, 2017.
[12] K. Greff, arXiv preprint arXiv:2012.05208. 2020.
Steenkiste, and Schmidhuher. On the binding problem in artificial neural networks.
[13] I. Higgins, L. Matthey, A. Pal, Burgess, X. Glorot, M. Botvinick, Mohamed, and A. Lerehner. beta-vae: Learning basic visual concepts with a constrained variational framework. International Conference on Learning Representations, 2017.
[14] J.-T. Hsieh, B. Liu, D.-A. Huang, L. F. Fei-Fei, and C. Niebles. Learning to decompose and disentangle representations for video prediction. In Advances in Neural Information Prucessing Systems, pages 517-526, 2018.
[15] A. Hyvarinen and H. Morioka. Unsupervised feature extraction by time-contrastiig learning and nonlinear ica. In Advances in Neural Information P Processing Systems, 2016.
[16] A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429 439, 1999.
[17] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. Spatial transformer networks. In Advances in Neural Information Processing Systems, pages 2017-2025, 2015.
[18] M. Janner, S. Levine, W. T. Freeman, J. B. Tenenbaum, C. Finn, and J. Wu. Reasoning about physical interactions with object-oriented prediction and planning. In International Conference on Learning Representations, 2019.
11

---

[19] M. Jaques, M. Burke, and T. Hospedales. Ptysics-as-inverse- -rrrraggiiieeeeeegahiissss Unsupervised physical parameter estimation from video. In International Conference on Learning Represenlations, 2020.
[20] H. Kim and A. Mnih. Disentangling by factorising. In International Conferencee on Muchine Leurning. pages 2649-2658, 2018.
[21] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
[22] A. Kosiorek, H. Kim, Y. W. Teh, and Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In Advances in Neural Information Processing Systems, pages 8606-8616, 2018.
[23] Z. Lin, Y.-F. Wu, S. Peri, B. Fu, J. Jiang, and S. Ahn. Improving generative imagination in object-centie world models. In International Conference on Machine Learning. 2020.
[24] Z. Lin, Y.-F Wu, S. V. Peri, Sun, G. Singh, F. Deng. J. Jiang, and S. Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations, 2019.
[25] F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf. and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning. pages 41 14-4124. PMIR, 2019.
[26] F. Locatello, D. Weissenborn, T. Unterthiner. A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems, 2020.
[27] J. Marino, Y. Yue, and S. Mandt. Iterative amortized inference. International Conference on Machine Learning, pages 3403-3412, 2018.
[28] R. Martin-Brualla, N. Radwan, M. S. M. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
[29] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405-421. Springer, 2020.
[30] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529 523-333,--3333310001 2015.
[31] L. Nanbo, C. Eastwood, and R. B. Fisher. Learning object-centric representations of multi-object scenes from multiple views. In Advances in Neural Information Processing Systems, 2020.
[32] L. Nanbo and R. B. Fisher. Duplicate latent representation suppression for multi-object variational autoencoders. In BMVC, 2021.
[33] J. Pearl et al. Causal inference in statistics: An overview. Statistics surveys, 3:96-146, 2009.
[34] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. arXiv preprint arXiv:2011..3900 2020.
[35] B. Schölkopf. F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5)x612-634, 2021.
[36] G. Singh, J. Yoon, Y. Son. and S. Ahn. Sequential neural processes. In Advances in Neural Information Processing Systems, 2019.
[37] R. Suter, D. Miladinovic, B. Schölkopf, and S. Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning. pages (056-60655. PMLR, 2019.
[38] C. K. Williams and M. K. Titsias. Greedy learning of multiple objects images using robust statistics and factorial learning. Neural Computation, 16(5); 1001-10092222 2004.
[39] J. Yang. J. Mao, J. Wu, D. Parikh, D. D. Cox, B. Tenenbaum, and C. Gan. Object-centric diagnosis of visual reasoning, arXiv preprint 2rXiv:20/2.15587 2020.
[40] A. Yuille and D. Kersten. Vision as bayesian inference: analysis by synthesis? Trends in cognitive sciences. 2006.
[41] A. Zadaianchuk, M. Seitzer, and G. Martius. Self-supervised visual reinforcement learning with object- centric representations. In International Conference Learning Representations, 2021.
12


Reviews:
[{"review_id": "niQNPo3u4a-", "reviewer": "Reviewer_BLv2", "summary": "This paper proposes a method that broadens the scope of multi-view object-centric representation learning to dynamic scenes. The model achieves spatial-temporal factorization by having two weak assumptions: high-frame-rate and large-speed-difference.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n1) Overall, this paper is well written, and the technical details are easy to follow.\n2) The main idea of unsupervised multi-view object-centric representation learning in the context of dynamic-scene settings is interesting.\n3) The factorization of space and time should be important. \n\n**Weaknesses:**\n\n**Assumptions.** The two assumptions of the high-frame-rate and the large-speed-difference are not holding in real-life scenarios. In real synthetic datasets (e.g., GTA and others), the observer is likely moving while the objects in the scene are also performing a movement. I agree the assumptions are valid on the DRoom and MJC-Arm datasets. However, my main concern is that this work would not generalize into real 3D datasets, unlike NRF, which can generalize novel-view synthesis.\n\n**Experiments.**  Table 1a shows that DYMON does not generalize well compared to MulMON[28] on the novel viewpoints. I find it quite surprising. Can the authors explain why that is? Also, why is there a gap between object reconstructions and segmentations?. Additionally, Table 1b is not clear to me. Why does the MulMON model perform better than DyMON? I would expect from a model that learns dynamics to generalize over static scenes.  \n\n\nMy main concern is I cannot see how this work generalizes over real 3D datasets and helps the field to move forward. I would expect the authors to add at least one real 3D dataset (or at least more realistic environments) since the dynamics play a critical factor in real-life scenarios more than on toy tasks. Furthermore, generalization and understanding of objects and observers' movements are vital for multiple tasks and domains, and thus I believe the spatial-temporal factorization should be leveraged on more realistic scenarios. On the other hand, I liked this paper, its contribution, and its motivation. I am open to the authors' feedback and other reviewers' opinions.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6-8"}, "extra_text": {}}, {"review_id": "iEA_AB4AxUz", "reviewer": "Reviewer_KAQi", "summary": "This work proposed a Dynamics-aware Multi-Object Network (DyMON), a model that captures object-centric representations in dynamic settings and enables querying novel viewpoints as well as single objects from the scene. DyMON builds on top of MulMON, a method that works in multi-view static settings. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) The problem setting is novel and has not been tackled from an object-centric representation point of view.\n\nWeaknesses:\n\n2) Is assumption A2 i.e having a large-speed difference between the camera and object realistic? Or is this just a trick to make the model learn? If so, then there must be some results shown on a dataset wherein both scene velocity and observer's velocity are perceivable.\n\n3) Baselines: The baselines used to compare are not fair. For example, T-GQN [Ref1] is a closely related work. This captures the model's dynamics in multi-view dynamic scenes (though not object-centric). It is surprising that the author(s) have missed this.\n\n4) Related works: Another closely related work for multi-view static setting is that of ROOTS [Ref2] which should be discussed in the related works section.\n\n5-a) Experiments: The author(s) claim that DyMON \"permits pixel-wise segmentation which the specialized model (GSEWM) cannot\". This is false. There are object-level masks in GSWM that essentially are the object segmentation masks. So, I don't see a reason why mIoU is N/A in Table 1(c).\n\n5-b) From my experience, GSWM has a very low MSE (in static view dynamic object scenes). Given that the DRoom dataset is similar to CLEVERER used in GSWM, I expect the reconstructions to be good. Can the author(s) provide the reconstruction comparison (either in the main paper or in the appendix)?\n\n6) Qualitative Results: Comparison against MulMON or GSWM in the qualitative results needs to be shown to visualize where DyMON is doing better. I'd urge the author(s) to include the qualitative comparison and give insights into where other models fail.\n\nSuggestions:\n7) The testing algorithm can be included in the main paper to make things clear. $\\beta_{\\text{FCSO}}$, $\\beta_{\\text{SCFO}}$ makes the reader think that they are also inferred during the test time, which is not the case.\n\n8) As mentioned in point 6, figures 2 and 3 need to improve. Fig. 2 occupies too much space (can reduce the number of examples and instead put MulMon and GSWM results).\n\n9) Abbreviations for Nv. Seg and Nv. Obs needs to be expanded in the caption of Table 1.\n\nOther minor details (not considered for decision):\n10) The paper seems to have some typos, please fix them. Otherwise in general the paper is well written.\n\nReferences:\n\n[Ref1] Sequential neural processes, G Singh, J Yoon, Y Son, S Ahn - In NeurIPS, 2019\n\n[Ref2] ROOTS: Object-Centric Representation and Rendering of 3D Scenes, Chang Chen, Fei Deng, Sungjin Ahn, JMLR, 2021\n\n\nJustification for Rating: Lack of proper baselines , comparisons with other methods and some confusion in the assumptions of the model are the reason why I'm currently giving the paper a  reject. I'd be looking forward for the rebuttal of the author(s) and change my score accordingly.\n\n-------\n\nEdit 1 (post rebuttal): The author(s) have addresses some of my concerns and I thank them for it. Hence, I've decided to increase my rating to \"5: Marginally below the acceptance threshold\" but cannot increase any further due to lack of proper baseline. \n\n----\nEdit 2 (post rebuttal and post T-GQN expt.) : Thanks to the author(s) for including the T-GQN baseline. Having addressed my concerns I'm increasing to the rating to a Weak accept. As mentioned earlier, the problem direction is quite novel and the proposed method is a good step in this direction, I'd hope more works come out which relax the assumptions and/or work in a much more realistic scenarios.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "a95AZC5dj6", "reviewer": "Reviewer_zpN5", "summary": "The following work proposes a method for learning the factorized latent structure of a scene. Specifically, while the primary prior work MulMon [28] assume static objects while the viewpoint varies through time, this work allows for both camera and object motion within the training data. To achieve their goal of factorizing object motion from camera motion, they assume high framerate and use clustering to determine cases where the camera is moving quickly and the objects slowly, versus cases where the camera moves slowly but the objects move quickly. Notably, once trained, they are able to resynthesize the scene's dynamic objects from novel viewpoints. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The ability to resynthesize dynamic scenes from novel viewpoints is impressive\n- Using clustering to identify cases of fast object slow camera and slow object fast camera seems like a highly practical solution to a difficult problem\n\nWeaknesses/Clarity concerns:\n- I think there's a lot of critical detail in the appendix, making the paper a bit difficult to follow. Specifically, I don't think I fully follow the details on how the data clustering for SCFO and FCSO works\n- There's a fair amount of prior art being left out here. Specifically there's a lot of works in unsupervised 2D landmark learning that are formulated very similarly.  [1], for example similarly disentangles dynamic objects from videos, falling under the \"slow camera fast object\" category specified in this work. [2] is another related work in the multi view setting.\n- I believe there are a number of additional unstated limitations that should be made clear here. What type of objects can we model here? By using spatial Gaussian mixtures, this typically assumes that the objects can be decomposed into rigid approximately convex shapes. \n- It appears that this model is capable of modeling the background as well. How do the spatial gaussians capture the background, considering the background spatially contains the foreground objects, and I assume the gaussians should not overlap too much?\n- What happens if the number of mixture components exceeds the number of objects by a lot? What happens if it's too small?\n\nOverall, I find the work pretty interesting despite the limitations. While it's quite possible that I've missed some critical details during my read-through, I would appreciate if the authors could point me to where the information I missed was, or clarify in their rebuttal.\n\n\n[1] Lorenz et al. Unsupervised Part-Based Disentangling of Object Shape and Appearance. CVPR 2019\n\n[2] Rhodin et al. Neural Scene Decomposition for Multi-Person Motion Capture. CVPR2019 \n\nPost rebuttal:\nI think the authors have adequately addressed most raised concerns and points of confusions. I agree with some other reviewers that the experimental setting is a bit too simple to be convincing, but I think the overall task is novel and the results are pretty interesting. For this reason, I maintain my original rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As stated in my main review, I don't think the technical limitaions are adequately addressed. I don't perceive any potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Fko6yWpCSRG", "reviewer": "Reviewer_dvA3", "summary": "This paper proposes DyMON, an unsupervised object-centric representation learning model that can be applied to scenes with dynamic objects and multiple viewpoints. DyMON is trained to obtain object latents that are independent of the viewpoints. The model builds off of MulMON, extending it to work with scenes with dynamic objects. Two assumptions are made about the training scenes: 1) a high frame rate assumption and 2) the relative speed difference between the camera and the objects is large -- that is, either the camera moves much faster than the objects or the objects move much faster than the camera. With these assumptions, the problem can be reduced to either a multi-viewpoint, static objects problem or a single-viewpoint, dynamic objects problem. Through experiments, the authors demonstrate that DyMON can be used to query predictions of the scene at arbitrary times and viewpoints.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-motivated, investigating the problem of learning object-centric representations in scenes with dynamic objects and multiple viewpoints without any supervision. This is an area previous works (that I know of) have not explicitly explored and is very important for modeling more realistic scenes.\n\nThe authors make several assumptions about the dataset and given those assumptions, the approach they take is reasonable and sound. However, while the authors claim to \"resolve\" (L127) the temporal entanglement problem, it seems that by making these assumptions, they have instead sidestepped the problem by restricting the dataset to a domain where the problem is not significant.\n\nThe experiments are encouraging, especially the ones querying novel viewpoints and times. However, I do not believe the DyMON vs. Dynamic Scenes section is very informative since MulMON is trained on a different dataset than it is evaluated on. I would also be curious how GSWM performs on the multi-view-dynamic-scene dataset. Although it does not explicitly take into account camera viewpoint, it can handle dynamic backgrounds (ie. context), so it may be able to model these scenes. Lastly, in addition to reporting MSE, it would be interesting to see comparisons of the reconstructions of DyMON vs. the baselines to see where the different models qualitatively fail.\n\nMy main concern with the paper is with the novelty. If I understand correctly, the model is almost identical to MulMON, but the difference is in the training procedure. While the DyMON training algorithm allows the model to factorize the camera dynamics from the object dynamics, this combined with the aforementioned assumptions made on the dataset limits the contributions of this paper. I encourage the authors to clarify the novelty of their approach over MulMON. I am willing to change my score if I have missed something.\n\nMinor issues / typos:\n\n- GSWM seems to be cited incorrectly (SPACE is cited twice)\n- In Equation (4), I'd recommend something other than q for the query since it is overloaded.\n- L23: These method -> These methods\n- L153: veiw -> view\n- L165 -> \"we have\" seems not necessary\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}]

Author rebuttal:

Meta-review: - The proposed method is tackling an important novel problem. The reviewers found the approach is reasonable and sound but technically not very novel. 
- Many concerns (including T-GQN) from the reviewers are addressed well by the rebuttal. These results should be included in the updated version.
- The strong assumptions the method takes to work are the main limitations of the approach but as the first work in the direction I think it still has some contribution. 
- Importantly, some key related works like T-GQN and ROOTS are missing and should be discussed with enough emphasis in the revision.

Decision: Accept (Poster)