SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition
Authors: Rishabh Kabra
Daniel Zoran
Goker Erdogan
Loic Matthey
Antonia Creswell
Matthew Botvinick
Alexander Lerchner
Christopher P. Burgess
Venue: NeurIPS 2021 Poster (2021)
Keywords: scene understanding
view invariance
object representations
unsupervised learning

Abstract: To help agents reason about scenes in terms of their building blocks, we wish to extract the compositional structure of any given scene (in particular, the configuration and characteristics of objects comprising the scene). This problem is especially difficult when scene structure needs to be inferred while also estimating the agent’s location/viewpoint, as the two variables jointly give rise to the agent’s observations. We present an unsupervised variational approach to this problem. Leveraging the shared structure that exists across different scenes, our model learns to infer two sets of latent representations from RGB video input alone: a set of "object" latents, corresponding to the time-invariant, object-level contents of the scene, as well as a set of "frame" latents, corresponding to global time-varying elements such as viewpoint. This factorization of latents allows our model, SIMONe, to represent object attributes in an allocentric manner which does not depend on viewpoint. Moreover, it allows us to disentangle object dynamics and summarize their trajectories as time-abstracted, view-invariant, per-object properties. We demonstrate these capabilities, as well as the model's performance in terms of view synthesis and instance segmentation, across three procedurally generated video datasets.     

SIMONe: View-Invariant, Temporally-Asstracted Object Representations via Unsupervised Video Decomposition
Rishabh Kabra', Daniel Zoran', Goker Erdogan', Loic Matthey?
Antonia Creswell', Matthew Botvinick', Alexander Lerehner', Christopher P. Burgess
'DeePMind, Wayve, "Work done at DeepMind
(rkabra, danielzoran, gokererdogan, Imatthey,
tonicreswell, botvinick, ) com, chrisburgessdwwyye..c
Abstract
To help agents reason about scenes terms of their building blocks, we wish to extract the compositional structure of any given scene (in particular, the con- figuration and characteristics of objects comprising the scene). This problem especially difficult when scene structure needs to be inferred while also estimating the agent's location/viewpoin,, as the two variables jointly give rise to the agent's s observations. We present unsupervised variational approach to this problem. Leveraging the shared structure that exists across different scenes, our model learns to infer two sets of latent representations from RGB video input: set of "object" latents, corresponding to the time-invariant, object-level contents of the scene, as well of "frame" latents, corresponding global time-varying elements such as viewpoint. This factorization of latents allows our model. SIMONe, to represent object attributes in allocentric manner which does not depend on viewpoint. Moreover, it allows us to disentangle object dynamics and summarize their trajec- tories abstracedd, view-invariant, per-object properties. We demonstrate these capabilities, as well as the model's performance in terms of view synthesis and instance segmentation, across three procedurally generated video datasets.
Introduction
The problem of unsupervised visual scene understanding has become an increasingly central topic in machine learning [1, 2]. The attention is merited by potential gains to reasoning, autonomous navigation, and myriad tasks. However, within the current literature, different studies frame the problem in different ways. One approach aims to decompose images into component objects and object features, supporting (among other things) generation of alternative data that permits insertion, deletion, or repositioning of individual objects [3-6]. Another approach aims at very different form of dccomposition--eetteee bbetween allocentric scene structure and a variable generation of views of a scene from new vantage points [7-9] and, if not supplied as input, estimation of camera pose [10]. Although there is work pursuing both of these approaches concurrently in the supervised setting [11-13], very few previous studies have approached the combined challenge in the unsupervised case. In this work, we introduce the Sequence-Integrating Multi-Object Net (SIMONe), model which pursues that goal of object-level and viewpoint- scene decomposition and synthesis without supervision. SIMONe is designed to handle these challenges without privileged information concerning camera pose, and in dynamic scenes.
Given a video of scene our model is able decouple scene structure from viewpoint information (see Figure 1). To do so, it utilizes video-based cues, and structured latent space which separates
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Scene sequence
(R
X' (K
lins
lajects
Dijjects latents from X
0 fo... oxj
Frame latents
Target cameeaa trajectory from X'
0
P (a.... fq)
F (fi.....
Segmantation Recondtruction
Reconttruction
tine
Hime
Figure 1: Decomposition (A): SIMONe factorizes a scene sequence into scene content ("object latents," constant across the sequence) and view/global content ("frame latents," per frame) without supervision. Its spatio-temporal attention bhased inference naturally allows stable object tracking (e.g. the green sphere is assigned the same segment across frames). Recomposition (B): Object latents of a given sequence can be recomposed with the frame latents different (i.i.d.) sequence X' to generate consistent rendering of the same scene (i.e. objects and their properties, relative arrangements, and segmentation assignments) from entirely different viewpoints. Notice that both camera pose and lighting transferred, evidenced by the wall corners in the background and the shadows of the green sphere.
time-invariant per-object features from time-varying global features. These features are inferred using transformer-based network which integrates information jointly across space and time.
Second, our method seeks to summarize objects' dynamics. learns to disentangle not only static object attributes (and their 2D spatial masks), but also object trajectories, without any prior notion of these objects, from videos alone. The learnt trajectory features are temporally abstract and per object; they captured independently of the dynamics of camera pose, which being global property, is captured the model's 's per-frame (time-varying) latents.'
Our model thus advances the state of the art in unsupervised, object-centric scene understanding by satisfying the following desiderata: (1) decomposition of multi-object scenes from RGB videos alone: (2) handling of changing camera pose, and simultaneous inference of scene contents and viewpoint from correlated views (i.e. sequential observations a moving agent): (3) learning of structure across diverse scene instances (i.e. procedurally sampled contents); (4) object representations which summarize static object attributes like color or shape, disssiisse properties like position or size, well time-abstracted trajectory features like direction of motion; (5) no explicit assumptions of 3D geometry, no explicit dynamics model, no specialized renderer, and few a priori modeling assumptions about the objects being studied; and (6) simple, scalable modules (for inference and rendering) to enable large-scale use.
2 Related Work
Given the multifaceted problem it tackles, SIMONe connects across several areas of prior work. We describe its nearest neighbors from three scene understanding domains below:
Annimtted figures are http://sss...ooeeecoeeeeeeeeeeee

---

Scene decompasition models. (1) Our work builds on recent surge of interest in unsupervised scene decomposition and understanding, especially using slot structure to capture the objects in a scene [14]. One line of work closely related to SIMONe includes methods like [3-6, 151, which all share SIMONe's Gaussian mixture pixel likelihood model. While these prior methods handled only static scenes, more recent work [16-18, 12] has extended them to videos with promising results. Nevertheless, these approaches have no mechanism or inductive bias to separate view information from scene contents. Moreover, many of them are conditioned on extra inputs like the actions of an agent/camera to simplify inference. (2) Another family of decomposition models originated with Attend, Infer, Repeat (AIR) [19]. AIR's s recurrent attention mechanism does split images into components with separate appearance and pose latents each. Later work [6, 20-23] also extended the model to videos. Despite their structured latents, these models do not learn to distill object appearance into a time-invariant representation (as their appearance and pose latents are free to vary as function of time). They also require separate object discovery and propagation modules to handle objects. In contrast, SIMONe processes a full sequence of images using spatio-temporal attention and produces single time-invariant latent for each object, hence requiring no explicit transition model or discovvery/roopagaionn modules.
Multi-view scene rendering models. Models which assume viewpoint information for each image like GQN [7], SRNs [8], and NeRF [9] have shown impressive success at learning implicit scene representations and generating novel views from different viewpoints. Recent work [24-27] has further extended these models to videos using deformation fields to model changes in scene geometry over time. In contrast to SIMONe. these models achieve photorealistic reconstructions by assuming camera parameters (viewpoint information). To allow a direct comparison, we use a view-supervised version of SIMONe in Section 4.1. There is also recent work [28, 29] that relaxes the known viewpoint constraint, but they still model single scenes at a time, which prevents them from exploiting regularities over multiple scenes. A more recent line of work [30-32] explored amortizing inference by mapping from a given set of images to scene latents, but they cannot handle videos yet. Note that all of these models treat the whole scene as a single entity and avoid decomposing it into objects. One exception here is [33]. which represents objects with separate pose and appearance latents. However, this model is purely generative and cannot infer object latents from given scene. Another exception is [13], which can in fact infer object representations, but nevertheless depends on view supervision.
Simultaneous localization and mapping. The problem of inferring scene representations in a novel environment by exploring it (rather than assuming given views and viewpoint information) is well studied in robotics and vision [10]. Classic SLAM techniques often rely on EM [34, 35] or particle filters [36] to infer viewpoint and scene contents jointly. While our problem slightly simpler (we can leverage shared structure across scene instances; certain elements such as the shape of the room are held constant; and we use offline data rather than active exploration). our approach of using a factorized variational posterior provides a learning- based solution to the same computational problem. Our simplified setting is perhaps justified by our unsupervised take on the problem. On the other hand, we don't 't assume simplifications which may be common in robotics practice (c.g. known camera properties like field of view; or the use of multiple cameras or depth sensors). Most popular SLAM benchmarks [37-39] are on unstructured 3D scenes and hence it was not straightforward for us to compare directly to classic methods. But an encouraging point of overlap is that object-centric SLAM formulations [1]] as well as learning-based solutions [40, 41] are active topics of research. Our work could open new avenues in object-centric scene mapping without supervision.
3 Model
SIMONe is variational auto-encoder [42] consisting of an inference network (encoder) which infers latent variables from a given input sequence, and a generative process (decoder) which decodes these latents back into pixels. Using the Evidence Lower Bound (ELBO), the model is trained to minimize a pixel reconstruction loss and latent compression KL loss. Crucially, SIMONe relies on a factorized latent space which enforces a separation of static object attributes from global, dynamic properties such as camera pose. We introduce our latent factorization and generative process in Section 3. Then in Section 3.2, we describe how the latents can be inferred using a transformer- significantly simplifying the (recurrent or autoregressive) architectures used in prior work. Finally, we fully specify the training scheme in Section 3.3.
encoder,
3

---

q(0
Conv
Temporal mean
Time
Spetial mean
Transtoreers
Cone
q(F Xg
Irpul
Spatiall
framos
mean
Figure Architecture of the SIMONe inference network &gg The transformers integrate information jointly across space and time to infer (the posterior parameters of) the object and frame latents.
3.1 Latent Structure and Generative Process
Our model aims capture the structure of scene, observed sequence of images from multiple viewpoints (often along a smooth camera trajectory, though this is not a requirement). Like many recently proposed object-centric models choose to represent the scene variables O = {ox K These are invariant by construction across all frames in the sequence (i.e. their distribution is constant through time, and expected summarize information across the whole sequence). We also introduce T frame latents F == {fe one for each frame in the sequence, that capture tmee-vryynng information. Note that by choosing this factorization we reduce the number of required latent variables from K T to K + T. The latent prior p(O, F) W[,:V(o 0, 1) Nf, 0, I) is a unit spherical Gaussian, assuming and enforcing independence between object latents, frame latents, and their feature dimensions.
set of K object latent
Given the latent variables, we assume all pixels and all frames be independent. Each pixel is modeled as a Gaussian mixture with K components. The mixture weights for pixel X1, capture which component & "explains" that pixel (1 S S HW). The mixture logits rig,,,4 and RGB (reconstruction) means Page, are computed for every component h at specific time-step t and specific pixel location using a decoder Dg:
iigai PRtt,i Do (5)4,,
p(x,,i 01, f,tt,1; = - M....NV(xt. ( (L.,0)))
k
We decode each pixel independently, "querying" our pixel-wise decoder using the sampled latents, coordinates E [-1, 1 of the pixel, and time-step I. [0, 1) being decoded as inputs. The decoder's architecture is an MIP or 1x1 CNN. (See Appendix A.3.1 for the exact parameterization as well as diagram the generative process). By constraining the decoder to work on individual pixels, we can use subset of pixels as training targets (as opposed to full images; this is elaborated in Section 3.3). Once they are decoded, we obtain the mixture weights my,t, by taking the softmax of the logits across the K components: mk,4ii softmax; (M,,,,.) Equation 2 specifies the full pixel likelihood, where a₂ is a scalar hyperparameter.
3.2 Inference
Given sequence of frames X [x:]T we wish to infer the corresponding object latents O and frame latents F. The exact posterior distribution POO, X) is intractable so we resort using Gaussian approximate posterior g(O, F X). The approximate posterior is parameterized as the output of an inference (encoder) network Eo (X) which outputs the mean and (diagonal) log scale for all latent variables given the input sequence.
SIMONe's s inference network is based on the principle that spatio-temporal data can be processed jointly across space and time using transformers. Beyond an initial step, we don't need the translation invariance CNN, which forces spatial features to interact gradually via widening receptive field. Nor do we need the temporal invariance of RNN which forces sequential processing. Instead,

---

let feature maps interact simultaneously across the cross- pross-product of space and time. See Figure 2 for an overview of our encoder architecture implementing this.
Concretely, each frame x, in the sequence is passed through a CNN which outputs IJ spatial feature maps each time-step (containing C' channels each). IJ can be larger than the number of object latents K. (For all results in the paper, we set I and J/ to 8 each, and K = 16.) The rest of the inference network consists of two transformers Ti and T2. Ti takes in all TIJ feature maps. Each feature map attends to all others as described. Ti outputs TJ transformed feature maps. When IJ > K, we apply a spatial pool to reduce the number of slots TK (see Appendix A.3.2 for details). These slots serve as the imput to Tz. which produces an equal number of output slots. Both transformers use absolute (rather than relative) positional embeddings. but these are 3D to denote the spatio-temporal position of each slot. We denote the output of T2 as ê This intermediate output is aggregated along separate axes (and passed through MI.ss) to obtain frame and object posterior parameters respectively. Specifically, Xo = m / Êx,z) while X mlp (1/ K Σx 6x,e). Using these posterior parameters we can sample the object latents Og ( exp X )1), and the frame latents f; ~ NN (xp({;)1))
3.3 Loss and Training
The model is trained end to end by minimizing the following negative-ELBO derivative:
Ta
a
logp(xes Oi, OK, fiit, +
K Σ
DRi(alos X) p(os))
TaHaWa
(3)
B1 Σ DKL X) P((())
We normalize the data log-likelihood by the number of decoded pixels (TaHaWa) to allow for decoding fewer than all input pixels (THW). This helps scale the size of the decoder (without reducing the learning signal, due to the correlations prevalent between adjacent pixels). Normalizing by I/TaHaaW ensures consistent learning dynamics regardless of the choice of how many pixels decoded. a is generally set to 1, but available to tweak in case the scale of 8. and By is too small to be numerically stable. Unless explicitly mentioned, set B, 3y. See Appendix A.3 for details.
4 Comparative Evaluation
To evaluate the model we focus on two tasks: novel view synthesis and video instance segmentation. On the first task (Section 4.1), we highlight the benefit of view information when it is provided as ground truth to a simplified version of our model (denoted "SIMONe-VS for view supervised), as well as baseline models like GQN [7] and NeRF-VAE [43]. On the second task (Section 4.2), deploy the fully unsupervised version of our model; we showcase not only the possibility of inferring viewpoint from data, but also its benefit extracting object-level structure in comparison methods like MONet [3]. Slot Attention [15], and Sequential IODINE [4].
Our results are based three procedurally generated video datasets of multi-object scenes. In increasing order of difficulty, they are: Objects Room 9 [44], CATER (moving camera) [45], and Playroom [46]. These were chosen meet number of criteria: we wanted at least 9-10 objects per scene (there can be fewer in view, or as many as 30 in the case of Playroom). We wanted a moving camera with randomized initial position (the only exception is CATER, where the camera moves rapidly but is initialized at a fixed position to help localization). We wanted ground-truth object masks to evaluate our results quantitatively. We also wanted richness in terms of lighting, texture, object attributes, and other procedurally sampled elements. Finally, we wanted independently moving objects in one dataset (to evaluate trajectory disentangling and temporal abstraction). and unpredictable camera trajectories in another (the Playroom dataset is sampled using an arbitrary agent policy, so the agent is not always moving). Details on all datasets are in Appendix A.2.
4.1 View synthesis (with viewpoint supervision)
We first motivate view-invariant object representations by considering the case when ground-truth camera pose is provided to our model (a simplified variant we call "SIMONe-VS") In this scenario,
5

---

Room layout and viewpeints
Target novel views
NeRF-VAE
GON
Input frames of 16)
segmentttion crediction
SIMONe
Figure 3: Comparison of scene representation and view synthesis capabilities between SIMONe- VS, NeRF- VAE, and GQN. All models partially observe a procedurally generated Playroom from given sequence of frames (we visualize 4 of the 16 input frames fed to the models). Then, we decode novel views on a circular trajectory around the room, with the yaw linearly spaced in [-т, x]. NeRF-VAE retains very little object structure, while GQN hallucinates content. SIMONe- VS can produce fine reconstructions objects that observes even partially or at distance (such as the bed or shelves in the scene). SIMONe- VS also segments the scene bonus. See Appendix A.5.1 for similar plots from different scenes/input sequences.
we don't infer any frame latents. Rather, the encoder and decoder are conditioned on the viewpoint directly. This view-supervised setting similar models like GQN and NeRF which represent the contents of scene implicitly and can be queried in different directions.
We compare three such models on view synthesis in the Playroom. The models are provided of 16 consecutive frames as context, partially revealing generated room. Having inferred scene representation from the input context, the models then tasked with generating unobserved views of the scene. This extrapolation task is performed without any retraining, and tests the coherence of the models' inferred representations. The task is challenging given the compositional structure of each Playroom scene, as well as the variation across scenes (the color, position, size, and choice of all objects are procedurally sampled per scene; only the L--haped layout of the room shared across scenes in the dataset). Because each model is trained on and learns to represent many Playroom instances, NeRF itself not directly suitable for the task. It needs to retrained on each scene, whereas we want to infer the specifics of any given room at evaluation time. NeRF-VAL addresses this issue and makes directly comparable to our model.
To set the comparison, we first trained SIMONe-VS and evaluated its log-likelihood on Playroom sequences. Then, we trained GQN and NeRF-VAL using constrained optimization (GECO [47]) to achieve roughly the same log likelihood per pixel. See Appendix A.5. for a comparison of the models in terms of the reconstruction comprrsssiono trade-off.
Qualitatively, the models show vast differences in their perceived structure (see Figure 3). NeRF-VAE blurs nearly all objects in the wall color. GQN produces more detailed reconstructions, but overfits to particular views and does not interpolate smoothly. SIMONe-VS on the other hand finely reproduces the object structure of the room. Even when it observes objects at a distance or up close, it places and sizes them correctly in totally novel views. This makes SIMONe-VS a powerful choice over NeRF-VAE and GQN-style models when the priority is to capture scene structure across diverse examples.
but understands the geometry of the room and is able infer
4.2 Instance segmentation (fully unsupervised)
Having shown the benefit of view information to inferring scene structure in the Section 4.1, we now turn to the added challenge of inferring viewpoint directly and simultaneously with scene contents (without any supervision).

---

Static ARI-F
Video ARI-F S-IODINE
MONet
SA
S-IODINE
MONet
SA
SIMONe
Objects Room 9
0.886
0.784
0.695
0.865
0.066
0.673
0.936
(1:0.661)
(2:0.138)
(1:0.007)
(1:0.00))
(1:0.11))
(1:0.0.002)
(100010)
CATER
0,937
0.923
0.728
0.412
0.073
0.668
0.918
(#0.004)
(=00076)
(+:00332)
(#000122)
(+000006)
(#:0.03))
(=0.036)
Playroom 0.647
0.653
0.439
0.442
0.059
0.356
0.800
(10.012)
(1.0.24))
(110.009)
(100.010)
(1:0.002)
(1.0.006)
(1000043)
Table 1: SIMONe segmentation performance (in terms of Adjusted Rand Index for foreground objects, ARI-F) compared to state-of-the-ar unsupervised baselines: two static-framn frame models (MONet and Slot Attention, SA) and video model (S-IODINE). We calculate static and video ARI-F scores separately. For static ARI-F, we evaluate the models per still image. For video ARI-F, we evaluate the models across space and time, taking an object's full trajectory thus penalizes models (especially Slot Attention) which fail to track objects stably. We report the mean and standard deviation of scores across 5 random seeds in each case.
single class. The video ARI-F
Hoyrsm
Figure 4: Segmentations and reconstructions produced by SIMONe on CATER and Playroom. SIMONe copes well with clutter and different-sized objects. It learns to use object motion segmentation signal on CATER, evident from the fact that an object's s shadow is correctly assigned to that object's 's segment
moves. This is true even when there's multiple shadows per object (due to multiple lights in the scene). SIMONe also overcomes volor-based cues to segment two-toned objects such beds in the Playroom as single objects. See Appendix A.5.2 compare with baseline models.
We compare SIMONe to a range of competitive but viewpoint-unaware scene decomposition ap- proaches. First, train two static-frame models: MONet and Slot Attention. MONet uses similar generative process and training loss to our model, achieving segmentation by modeling the scene as a spatial mixture of components, and achieving disentangled representations using a 3--weihhtd KL information bottleneck. On the other hand, it uses a deterministic, recurrent attention network to infer object masks. Slot Attention transformer-basee autoencoding model which focuses segmenta- tion performance rather than representation learning. Finally, we also compare against Sequential IODINE ("S-IODINE"), which applies a refinement network to amortize inference over time, separat- ing objects by processing them in parallel. It also uses a B-weighted KL loss to disentangle object representations. Note that S-IODINE is a simplified version of OP3 [17], which additionally attempts to model (pairwise) object dynamics using an agent's 's actions as inputs. SIMONe and S-IODINE both avoid relying on this privileged information. Table contains quantitative comparison of segmentation performance across these models, while Figure 4 shows qualitative results.
7

---

Feature traversal Itaret object)
feature traversal (target objecti
Color
bed
bowl
2-5120
camlaae
x-position
x-povition
shelves
green book
Lduntity
Identity
purple naiinerr
red rubber duck
Figure 5: Object attributes learnt by SIMONe. In each row, we manipulate a particular object
latent attribute for arbitrary target object (circled in red) in scenes. This reveals the attributes' relationship to interpretable object characteristics like color, size, position and identity.
5
Analysis
We take closer look at the representations learnt by model by decoding them in various ways. First, we manipulate latent attributes individually to assess the interpretability of object representations visually in Section 5.1. Next, exploit SIMONe's 's latent factorization render views of given scene using the camera trajectory of a different input sequence. These cross-over visualizations help identify how the model encodes object dynamics in Section 5.2. Finally, we measure the predictability of ground-trute camera dynamics and object dynamics from the two types of latents in Section 5.3. These analyses use single, fully unsupervised model per dataset.
5.1 Latent attribute traversals
We visualize the object representations learnt by SIMONe on Playroom to highlight their disentangle- ment, across latent attributes and across object slots, in Figure We seed all latents using a given input sequence, then manipulate one object latent attribute time by adding fixed offsets.
Note that object position and size are well disentangled in each direction. Aided by the extraction of view-specific information in the frame latents, SIMONe also learns object features corresponding identity. The decoder nevertheless obeys the biases in the dataset-for instance, shelves will slide along wall when their position latent is traversed. The rubber duck does not morph into a chest of drawers because those are always located against wall. This further suggests a well-structured latent representation, which the decoder can adapt to.
5.2 Object and frame latent cross-overs
We expect SIMONe to encode object trajectories and camera trajectories independently of each other. In fact, each object's 's trajectory should be summarized in its time-invariant latent code. To examine this, we recompose object latents from one sequence with frame latents from other sequences in the CATER dataset. The result, in Figure 6, is that multiple camera trajectories.
observe object motion trajectories from
Note the consistency of relative object positions (at any time-step) from all camera angles. In the single moving object case, its motion could in fact be interpreted as time-varying global property of the scene. Despite this challenge, SIMONe is able to encode the object's 's motion desired in its

---

Single object moving
Two objects moving
Input
Zoom out
0 x| x 0 P
Camera motion
Rotation
Camera mation 2
time
time
When encoding sequence with consistent (i.id.)) object dynamies, this information extracted in the object latents and is unaffected by changing frame latents (see green cone). Right: Movement events are sequenced correctly; object relative positions also remain consistent (see pattern of shadows on the floor circled in yellow). See Appendix A.5.3 for cross-over plots showing more object trajectories.
Figure 6: Separation of object trajectories from camera trajectories. Left:
specific time-invariant code. In Section 5.3, further confirm that object trajectories summarized in the object latents, which can be queried with time to recover allocentric object positions.
5.3 Camera pose and object trajectory prediction
We assessed SIMONe's frame latents by decoding the true camera position and orientation from them.
We trained linear and MLP regressors to predict the camera pose at time from the corresponding frame latent fr on a subset of CATER sequences. We also trained an MLP on the time-invariant object latents 01:K for the same task. We evaluated these decoders on held-out data. Table shows that frame latents describe the viewpoint almost perfectly.
We also assessed if the object latents contain precise information about allocentric object positions (to be clear, position information is not provided any form while training SIMONe). Table 3 shows that the correct object latent is predictive of the allocentric position of dynamic object (when queried along with the timestep). Adding the frame latent does not provide more information, and using
Linear(f.)
MLP(f.)
MLP(o Ok)
Camera location
0.832 + 0.0
0.949 + 0.002
0.044 + 0.026
Camera orientation (Rodrigues)
0.800 + 0.0
0.946 + 0.002
0.292 0.025
Table 2: Decoding camera pose. We show that ground-trute camera location or orientation is predictable from the corresponding frame latent, but cannot be predicted from all object latents put together. We report the test R2 score across 5 independently trained decoders per input type.
MLP(o,)
MLP(o,, t)
MLP(o,, t)
MLP((o, : * k))
Trained all objects
0.710 + 0.006
0.871 + 0.006
0.876 + 0.003
-0.062 + 0.006
Trained moving objects
0.724 + 0.007
0.894 + 0.004
0.898 + 0.005
-0.022 + 0.025
Table 3: Decoding object trajectories. We test MLP decoders predicting allocentric object positions (of moving object in unseen scenes) based on the following inputs: (a) the corresponding object latent, (b) the timestep well, and (c) the frame latent corresponding to that timestep well, and (d) remaining object latents from the scene (not pertaining to the object interest). The decoders trained arbitrary objects subset containing moving objects only. We report the test R² score across 5 independently trained decoders per input type.

---

localized the "wrong" objects (from the same scene) is completely uninformative. To perform this analysis, we needed align SIMONe's 's inferred objects with the ground-truth set of objects. We used the Hungarian matching algorithm on the MSE of inferred object masks and ground-truth object masks to perform the alignment. Given SIMONe's disentangling of object dynamics, its time-abstracted object representations could prove helpful for a variety of downstream tasks (e.g. "catch the flying ball!").
Taken together, Table and Table 3 show the separation of information that is achieved between the object and frame latents, helping assert our two central aims of view-invariant and temporally abstracted object representations.
6 Discussion and Future Work
Scalability. The transformer-basee inference network in SIMONe makes it amenable to processing arbitrarily large videos, just as transformer-basee language models can process long text. SIMONe could be trained on windows of consecutive frames sampled from larger videos (aka "chunks"" "). For inference over a full video, one could add memory slots which carry information over time from one window to the next. Applying SIMONe on sliding windows of frames also presents the opportunity to amortize inference at any given time-step if the windows are partially overlapping (so the model could observe every given frame part of two more sequences). Our use of the standard transformer architecture also makes SIMONe amenable to performance improvements via alternative implementations.
Limitations. (1) SIMONe cannot generate novel videos (e.g. sample natural camera trajectory via consecutive frame latents) in its current version. This could be addressed in similar fashion to the way GENESIS [5] built on MONet [3]-it should be possible (e.g. using recurrent networks) to learn conditional priors for objects in scene and for successive frame latents, which would make SIMONe fully generative. (2) We see another possible limitation arising from our strict latent factorization. We have shown that temporally abstracted object features can predict object trajectories when queried by time. This can cover a lot of interesting cases (even multiple object-level "events" over time), but will start to break as object trajectories get more stochastic (i.e. objects transition considerably/chaote through time). We leave it to future work to explore how temporal abstraction can be combined with explicit per-step dynamics modeling in those cases. For simpler settings, approach to encoding object trajectories (distilling them across time) is surprisingly effective.
7 Conclusion
We've presented SIMONe, a latent variable model which separates the time-invariant, object-level properties of a scene video from the time-varying, global properties. Our choice of scalable mod- ules such as transformers for inference, and a pixel-wise decoder, allow the model to extract this information effectively.
SIMONe can learn the common structure across variety of procedurally instantiated scenes, This enables it to recognize and generalize to novel scene instances from a handful of correlated views, as we showcased via 360-degree view traversal in the view-supervised setting. More significantly, SIMONe can Iearn to infer the two sets of latent variables jointly without supervision. Aided by cross- frame spatio-temporal attention, it achieves ofate-of-the-ard segmentation performance on complex 3D scenes. Our latent factorization (and information bottleneck pressures) further help with learning meaningful object representations. SIMONe can not only separate static object attributes (like size and position), but it can also separate the dynamics of different objects (as properties) from global changes in view.
We have discussed how the model can be applied to much longer videos in the future. It also has potential for applications in robotics (e.g. sim-to-real transfer) and reinforcement learning, where view-invariant object information (and summarizing their dynamics) could dramatically improve how agents reason about objects.
10

---

Acknowledgements
We thank Michael Bloesch, Markus Wulfmeier, Arunkumar Byravan, Claudio Fantacci, and Yusuf Aytar for valuable discussions on the purview of our work. We are also grateful for David Ding's support on the CATER dataset. The authors received no specific funding for this work.
References
[1] Jitendra Malik. Scene understanding in the era of deep learning, Jun 2015. URL ox. ac wtw.ac..///... ww/ /ammiinaar/xxtra2215555.5.. ____e_eedd___e__e___ee_________e a .dff.
[2] Farzad Husain, Babette Dellen, and Carme Torras. Chapter 20 Scene Understanding Using Deep Learning. pages 373-382. Academic Press, 2017. ISBN 978-0-12-811318-9 doi: https://lii.org/10..00 B978-0-12 11-0--12-111111111 11111119,000000000 9,00020-X.
Christopher Burgess, Loic Matthey, Nicholas Watters, Rishahh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised seene decomposition and representation. arXiv preprint arXiv: 1901. 11390, 2019.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pages 2424-2433. PMLR, 2019.
[5] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. International Conference on Learning Representations, 2020.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations,
202. 2020.
[7] S M Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garell,, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, David Reichert, Lars Buesing. Theophane Weber. Oriol Vinyals. Dan Rosenbaum, Neil Rabinowitz, Helen King. Chloe Hillier, Matt Botvinick, Daan Wienstra, Koray Kavukcuoglu, and Demis Hassabis. Neural scene representation and rendering. Science, (6006664))1124--110
June 2018.
[8] Vincent Sitzmann, Michael Zollhoefer, and Gordon Weizstein. Scene representation networks: Continuous 3D-Structure-Awaan neural scene representations. In H Wallach, H Larochelle, A Beygelzimer, F d' Alché Buc, E Fox, and R Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[9] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405 4251.. Springer, 2020.
[10] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, and J.J. Leonard. Past, present, and future of simultaneous localization and mapping: Towards the robust-perception age. IEEE Transactions on Rohotics, (2(6)):309-1133 2016.
[11] Binbin Xu, Wenbin Li, Dimos Tzoumanikas, Michael Bloesch, Andrew Davison, and Stefan Leutenegger. Mid-fusion: Octree-based object-level multi-instance dynamic slam. In 2019 International Conference on Robotics and Automation (ICRA), pages 5231 -2377. IEEE, 2019.
[12] Li Nanbo, Cian Eastwood, and Robert Fisher. Learning object-centric representations of multi-object scenes from multiple views. Advances Neural Information Processing Systems, 2020.
[13] Chang Chen, Fei Deng, and Sungjin Ahn. Object-centric representation and rendering of 3d scenes, arXiv preprint arXiv:2006. 1.30 2020.
[14] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuher. On the binding problem in artificial neural networks. arXiv preprini arXiv:2012.05208, 2020.
[15] Francesco Locatello, Dirk Weissenbom, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11525 11538, Curran Associates, Inc., 2020.

---

[16] Antonia Creswell, Rishabh Kabra, Chris Burgess, and Murray Shanahan. Unsupervised object-based transition models for 3d partially observable environments. arXiv preprint arXiv:2103.06693, 2021.
[17] Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. In Conference on Robot Learning, pages 1439-1456. PMLR, 2020.
[18] Polina Zablotskaia, Edoardo A Dominici, Leonid Sigal, and Andreas M Lehrmann. Unsupervised video decomposition using spatio-temporal iterative inference. arXiv preprint arXiv:2006. 14727, 2020.
[19] S M Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari. Koray Kavukcuoglu. and Geoffrey E Hinton. Attend, infer, repeat: fast scene understanding with generative models. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pages 3233-3244 Red Hook, NY, USA, December 2016. Curran Associates Inc.
[20] Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. Adv. Neural Inf. Process. Syst. 31, 2018.
[21] Eric Crawford and Joelle Pineau. Exploiting spatial invariance for scalable unsupervised object tracking. arXiv preprint arXiv: /tt:://91.1000 1910093 2019.
[22] Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. AAAI, 33(01))3412-3420 July 2019.
[23] Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, and Sungjin Ahn. SCALOR: Generative world models with scalable object representations. arXiv preprint 1rXiv:1910.00:00 October 2019.
[24] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer, D-NeRF: Neural radiance fields for dynamic scenes, arXiv preprint arXiv:201 1. 13166, November 2020.
[25] Keunhong Park, Utkarsh Sinha, Jonathan Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla, Nerfies: Deformable neural radiance fields. arXiv preprint ar244 2020.
[26] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for Space-Time view synthesis of dynamic scenes. arXiv preprint arXiv:201). November 2020.
[27] Yilun Du, Yinan Zhang, Hong-Xing Xing Yu, Joshua B Tenenbaum, and Jigjun Wu. Neural radiance flow for 4D view synthesis and video processing. arXiv preprint arXiv:20/2.09790, December 2020.
[28] Chen-Hsuan Lin, Wei-Chiu Ma. Antonio Torralba, and Simon Lucey. BARF: Bundle-Adjusting neural radiance fields. arXiv preprint arXiv:2/04.06400 April 2021.
[29] Zirui Wang, Shangzhe Wu, Weidi Xic, Min Chen, and Victor Adrian Prisacariu. NeRF-; Neural radiance fields without known camera parameters, arXiv preprint arXiv:2102.07064 February 2021.
[30] Adam Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Soña Mokrá, and Danilo Jimenez Rezende. NeRF-VAE: geometry aware 3D scene generative model. In International Conference on Machine Learning. PMLR, 2021.
[31] Alex Trevithick and Bo Yang. GRF: Learning a general radiance field for 3D scene representation and rendering. arXiv preprint arXiv:2010.04590 Octoher 2020.
[32] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. PixeINeRF: Neural radiance fields from one or few images. arxiv preprint ar ar1iv:2012.02/90, December 2020.
[33] Michael Niemeyer and Andreas Geiger. GIRAFFE: Representing scenes as compositional generative neural feature fields. arXiv preprint arXiv:2011.12100 November 2020.
[34] H. Durrant-Whine and T. Bailey. Simultancous localization and mapping: part IEEE Robotics Automation Magazine, 13(2):99- 110, 2006. doi: 10. 1..1109M/A22000 1638022.
[35] T. Bailey and H. Durrant- Durrant-Whyte. Simultaneous localization and mapping (slam): part ii. IEEE Robotics Automation Magazine, 13(3): 108-1108 2006. doi: 10.1 10.1109/RR..000 200......066...114...
[36] Josep Aulinas, Yan Petillot, Joaquim Salvi, and Xavier Llad6. The slam problem: A survey. In Proceedings of the 2008 Conference on Artificial Intelligence Research and Development: Proceedings of the 11th International Conference of the Catalan Association for Artificial Intelligence, page (31-371.
NLD, 2008 IOS Press. ISBN 9781 7781586032277.
12

---

[37] Michael Burri, Janosch Nikolic, Pascal Gohl, Thomas Schneider. Joern Rehder. Sammy Omari, Markus W Achtelik, and Roland Siegwart. The euroc miero aerial vehicle datasets. The International Journal of Robotics Research, 2016. doi: 10. 10.11770278364915660000 56200333.
[38] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Puttern Recognition (CVPR). 2012.
[39] J. Sturm, N. Engelhard. F. Endres, W. Burgard, and D. Cremers. A benchmark for the evaluation of rgb-d slam systems. In Proc. of the International Conference on Intelligent Robot Systems (IROS), Oct. 2012.
[40] Ruihao Li. Sen Wang. and Dongbing Gu. Deepslam: A robust monocular slam system with unsupervised deep learning. IEEE Transactions on Industrial Electronics, 68(4):357733587 2021. doi: 10.1 109/TIE. 2020. 2982096.
[41] Mingyang Geng, Suning Shang, Ding, Huaimin Wang, and Pengfei Zhang, Unsupervised learning- based depth estimation-aided visual slam approach. Circuits, Systems, and Signal Processing, 39(2): 543-570, 2020.
[42] Diederik P Kingma and Max Welling. An introduction to variational autoencoders. arXiv preprint arXiv: 190.02990600691, 2019.
[43] Adam R Kosiorek, Heiko Strathmann, Danicl Zoran, Pol Moreno, Rosalia Schneider, Soña Mokrá, and Danilo J Rezende. Nerf-vae: A geometry aware 3d scene generative model. arXiv preprint MrXiv:2104.00586 2021.
[44] Rishabh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datasets.
2019.
[45] Rohit Girdhar and Deva Ramanan. CATER: A diagnostic dataset for compositional actions & temporal reasoning. In International Conference Learning Representations, 2020.
[46] Josh Abramson, Arun Ahuja, Arthur Brussee, Federico Carnevale, Mary Cassin, Stephen Clark, Andrew Dudzik, Petko Georgiev, Aurelia Guy. Tim Harley, et al. Imitating interactive intelligence. arXiv preprint AXKi::0012.06622. 2020.
[47] Danilo Jimenez Rezende and Fabio Viola. Taming vaes. arXiv preprint arXiv:1810.00520 2018
[48] Vasu Parameswaran and Rama Chellappa. View invariance for human action recognition. International Journal of Computer Vision, 66(1):83 101, 2006.
[49] Yang Liu, Zhaoyang Lu, Jing Li. and Tao Yang. Hierarchically learned view-invarian representations for cross-view action recognition. IEEE Transactions on Circuits and Systems for Video Technology, 29(8): 2416-2430, 2018.
[50] Ting Liu, Jennifer J Sun, Long Zhao, Jiaping Zhao, Liangzhe Yuan, Yuxiao Wang. Liang-Chieh Chen, Florian Schroff, and Hartwig Adam. View iivarinv,r,ant, occlusion-robust probabilistic embedding for human pose. arXiv preprint arXiv:2010. 13321, 2020.
[51] Jennifer J Sun, Jiaping Zhao, Liang- Chinh Chen, Florian Schroff, Hartwig Adam, and Ting Liu. View- invariant probabilistic embedding for human pose. European Conference on Computer Vision, pages 53-70. Springer, 2020.
[52] Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Mare Pollefeys, and Cordelia Schmid. Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction. In CVPR, 2020.
[53] Emily L Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[54] Yizhe Zhu, Martin Renqiang Min, Asim Kaday, and Hans Peter Graf. S3VAE: Self-supervised sequential VAE for representation disentanglement and data generation. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2020.
[55] Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. arXiv preprint arXiv:1802.000 February 2018
[56] Ruben Villegas, Jimei Yang. Seunghoon Hong. Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
13

---

[57] Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In Jennifer Dy and Andreas Krause. editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5670-5679. PMLR, 2018.
[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. I. Guyon, U. Luxburg. S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Aditors, in Neural Information Processing Systems. volume 30, Curran Associates, Inc., 2017.
[59] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[60] Nicolas Carion, Francisco Massa, Gabricl Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagonuyko. End-to- tn- object detection with transformers. In Computer Vision ECCV 2020, pages 21 213-229. Springer International Publishing. 2020.
[61] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. Max-DeeppLab: End-to-End panoptic segmentation with mask transformers. arXiv preprint arXiv:2012200759, December 2020.
[62] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo. Yabiao Wang. Yanwei Fu, Jianfeng Feng, Tao Xiang. Philip S Torr, and Li Zhang. Rethinking semantic segmentation from a Sequence-to- Sequence perspective with transformers. arXiv preprint arXiv:2012. 15840, December 2020.
[63] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. 142944 2021.
[64] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is Space-Time attention all you need for vidco understanding? arXiv preprint 2rXiv:2102.05099 February 2021.
[65] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. arXiv preprint arXiv:2102.00719, February 2021.
[66] Anurag Arna,, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. VIVIT: A video vision transformer. arXiv preprint 20X:v:2103.1669, March 2021.
[67] Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-End video instance segmentation with transformers. arXiv preprint arXiv:2011. 14503, November 2020.
[68] Fabio Viola, Louise Deason, and Marcel Büschingg Datasets used to train generative query networks (ggns) in the 'neural scene representation and rendering' paper. http:///hhhbb.oom//eeem/////////
2018.
[69] Felix Hill, Olivier Tieleman, Tamara von Glehn, Nathaniel Wong, Hamza Merzic, and Stephen Clark. Grounded language learning fast and slow. arXiv preprint arXiv:2009.0/7/9 2020.
[70] Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint arXiv: / 19000771 2019.
14


Reviews:
[{"review_id": "XSDGGqkf9Fa", "reviewer": "Reviewer_RGhV", "summary": "Authors propose a structured generative model of video clips. This uses a factored latent space, with per-object latents that are global the video, and temporal latents that are not split between objects. The decoder uses a spatial GMM (as in many prior works), albeit with pixels given by per-coordinate MLPs not a CNN. The encoder uses a transformer on CNN features. Results on synthetic datasets show higher performance than one existing method (S-IODINE) on video segmentation, and higher performance than GQN and NeRF-VAE on novel view synthesis.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nModel etc.:\n\n++ The model is clearly described (albeit with fairly major aspects in the supplementary).\n\n++ The model structure is novel -- significantly different to other works in the same area, even though the components (transformers & per-pixel MLP decoding) are well-established elsewhere. Splitting a latent space into temporal and non-temporal components is also not a new idea (see below) -- this should be emphasised better. Overall I feel the novelty is sufficient.\n\n-- It is not clear where object motion is expected to be encoded, and why. The results (Sec.5.2) state post-hoc that encoding them in the object latents is 'correct', but from the model description, they could equally be in the temporal latents.\n\nRelated work:\n\n-- \"MoCoGAN: Decomposing Motion and Content for Video Generation\" [Tukyakov, CVPR 18] is relevant (it models videos and separates latents into static content and per-frame motion) and should be discussed (note various other follow-up works also used a similar factorisation).\n\n-- \"Unsupervised object-centric video generation and decomposition in 3D\" [Henderson, NeurIPS 20] is relevant (it forms an object-centric latent representation of videos, with temporal information factored) and should be discussed (and/or evaluated against). In particular, it would be expected to handle the case of Fig.8 correctly, as would other 3D-aware generative models.\n\nExperiments:\n\n++ Pixel reconstruction of scenes is very accurate, and camera motion is successfully encoded into (only) the per-frame latents.\n\n++ View-synthesis results are significantly better than baselines.\n\n-- Segmentation results are poor, both quantitatively and qualitatively.\n\n-- The model uses a novel encoder and decoder architectures (per-pixel MLP and transformer), but there is no ablation study, nor attempt to match parameter counts / architectures for the baselines. This means it is difficult to gain useful insights from the paper about what is important (e.g. maybe S-IODINE with a transformer encoder would be better? how much does the MLP decoder help vs. spatial broadcast CNN? how well does the same model work if the latent space is not factored?).\n\n-- Evaluation is only on simple synthetic datasets where a color-based segmentation (rgb clustering with smoothness) would likely work well. Other somewhat-similar works (e.g. RELATE and O3V) use more challenging data; it would be interesting to see how SIMONe performs on their datasets.\n\n-- Please discuss why qualitative GQN results look much worse than the original work.\n\n---\n\n## Post-rebuttal\n\nThe authors' response addresses several of my concerns, and I therefore am now mildly in favor of acceptance, provided missing discussion, references, etc. are added in the camera ready. Issues raised by other reviewers also seem to be adequately addressed. Note the authors' discussion in the responses here is more insightful than the original paper in some cases -- it would be good to incorporate as much as possible in the final version. The biggest remaining issues in my view are the simplistic nature of the datasets used in evaluation, and disappointing segmentation performance on them.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, both fine.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "TtLbXGZPWV", "reviewer": "Reviewer_Uh4X", "summary": "This submission presents a video scene model that decomposes a scene into object components, and frame components in an unsupervised way. The architecture improves over previous models that are static (MONET, IODINE, etc) and use videos (GENESIS) in that the model performs better and is conceptually simpler. The main idea is the use of a transformer model that stacks all feature representation generated from CNNs for every individual frame, then aggregate to object and frame latents. \n\nExperimental results are shown on synthetic images only. Comparison are done in two settings: knowing the camera locations (against GQN and Nerf-VAE) and fully unsupervised (compared against MONET and SlotAttention). Empirical results verify the model to be stronger.  ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Well written paper that is an extension of current work on unsupervised models that decompose scenes into its constituents. This model is novel in the way it uses transformers to decompose into frame and object latents. This makes for a conceptually simple model. Experimental results are sufficient, the model is tested in two scenarios (with and without known camera position) and performs better than reasonable chosen baselines. The submission presents some weak evidence of the disentangling of the object/frame latents in that the camera position is not recoverable from the objects but only from the frame representation.  \n\nThe main drawbacks are\n- This is no fully generative model. The submission comments on this in the future work and limitation section. I concur with the potential extension borrowed from IODINE. \n- Experimental results are shown on synthetic images only. The Nerf-VAE baseline is a fair competitor for this model and this type of date. The question whether these scene models will generalize to more complex environments is still open. This submission improves on the technical side but not on the experimental one.\n\nI recommend acceptance of this paper, I enjoyed reading it and the level of detail is sufficient to understand the details of the method.\n\nDid you experiment with the inclusion of optical flow into the model? Would that not potentially improve on the attention mechanism? What are the main obstacles for using optical flow?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are sufficiently addressed. Negative societal impact is abstract for this work and is shared with all models that improve on scene understanding in automated ways.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "O7Oq7kzSUnD", "reviewer": "Reviewer_5tHk", "summary": "This paper proposes SIMONe, a variational auto-encoder that decomposes a video into object latents and frame latents without any supervision. The object latents correspond to the time-invariant, object-level contents of the scene. The frame latents correspond to the global time-varying elemnts of the scene such as viewpoint. Each pixel is decoded independently by querying a pixel-wise decoder using sampled latents. To infer latents, SIMONe uses a transformer-based encoder that integrates information jointly across space and time. The experiments demonstrate SIMONe's capabilities in view syntheses and instance segmentation. Furthermore, the object latents are shown to be disentangled and summarize the trajectories of the objects while the frame latents are shown to contain viewpoint information.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This paper proposes a novel architecture that makes relatively few modeling assumptions about the scene when compared with previous works. The independence assumption in the decoder is also an interesting choice to allow the model to train on smaller inputs than the full image. It may be an interesting ablation to see the training speed tradeoff with this choice.\n\nQuality: The model is sound and makes reasonable assumptions. The claims are well supported in the experiments. In particular, Table 2 demonstrates the camera location information is encoded in the frame latents Table 7 in the supplementary demonstrates the object trajectories are encoded in the object latents. The view synthesis and segmentation results also show that SIMONe outperforms the baselines. Since the authors ran their model on the CATER dataset, it would be interesting to see if the factorized representations can be used in the CATER localization task.\n\nClarity: The paper is clearly written and well organized. However, I did find that I had to look in the Supplementary to get a full understanding of the model, especially for the decoder. The authors may consider moving parts of the appendix to the main text if space allows. One minor suggestion is to include the spatial pooling between the two transformers in Figure 2, since that is an important operation in obtaining the K object latents. As drawn, it is not clear that K and IJ may be different.\n\nSignificance: Given the current state of unsupervised visual scene understanding models, this paper provides a good direction for future research. I would be curious how SIMONe performs on more realistic scenes than the ones shown in the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "GULSi7c9qfj", "reviewer": "Reviewer_nXbF", "summary": "The authors devise a neural network architecture, SIMONe, for learning unsupervised representations of scenes viewed from multiple camera poses. The latent representation is factored into slots meant to represent time-invariant entities (i.e., physical objects) and time-varying \"frame latents,\" which can capture changes in the camera pose. They show that this method is able to capture the structure of scenes from three synthetic datasets by testing it on two tasks: novel view synthesis and (unsupervised) object instance segmentation. A number of qualitative results suggest that the learned latent representation successfully factors into time-invariant objects and time-varying viewpoint parameters.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work fits in the rapidly growing clade of unsupervised scene decomposition methods (also called \"object discovery\"), specifically those which try to factor scenes into K object slots plus background: MONet, IODINE, GENESIS, Slot-Attention, SPACE, SPAIR, OP3, and plenty of others. These methods are all meant to address a critical problem in AI and computer vision: how to infer and represent the structure of a scene when supervision isn't available. It's widely hypothesized (though not generally confirmed) that such object-centric representations, if they were to be used on downstream tasks, would have stronger generalization properties than unstructured representations. They're also better motivated from the standpoint of human psychology and behavior: most of our attention and goals are directed at _objects_, which can be picked out and referenced independently of the rest of a scene.\n\nDespite this area of research getting a lot of due attention, it seems like there's still a long way to go before any of these methods will be useful for dealing with real, complex scenes. Compared to **supervised** object instance segmentation methods, the object discovery methods work only on datasets and scenes of far lower complexity (and are usually only benchmarked on simple synthetic datasets.) It's been known at least since IODINE was proposed, for instance, that the unsupervised methods fail to identify objects with multiple complex textures and shapes rather than monochromatic spheres, cones, cylinders, and so forth.\n\nTo me, the present work mostly addresses a more peripheral deficiency in the previous object discovery methods, rather than tackling the core problems with them head on. The desiderata listed at the end of the Introduction are all surely nice to have, but I think most people who work on robotics, for instance, would rather have an unsupervised object discovery method that works on real videos and objects than one that can handle a sequence of camera poses. \n\nThe authors may have a different view of what's most important or interesting to address, and below I'll try to evaluate the work based on what I think the authors were aiming for. But I think it's important for them to argue more clearly (in the Introduction and Related Work) for taking the approach that they take and adding the enhancements to prior models that they add -- in particular, because other recent works **do** seem to aim more obviously toward the central problems with unsupervised object discovery, recognize what makes them difficult, and design their methods accordingly.\n\n_What makes unsupervised scene decomposition hard, and why doesn't this work try to address it?_\n\nI think it's mainly two things. First, realistic objects have complicated shapes, textures, and multi-part substructures. For this reason, methods that tend to segment objects as things with uniform color or texture or by their simple shape seem destined to badly underfit on realistic scenes and datasets. The present work doesn't really expose this challenge, because even the hardest of the datasets (\"Playroom\") has mostly monochromatic objects with simple shapes. It's not clear that SIMONe captures even these objects especially well (see below.) So I'm not sure how much of an advance this method represents; looking at more complicated and naturally textured objects (if they can be generated in Playroom) would be one way to show this more clearly.\n\nThe second challenging thing about object discovery is that reliable learning signals are hard to come by. Just from looking at a static scene, it's impossible to know for sure what its physical structure is -- i.e., which components (\"objects\") can move or exist independently of the rest of the scene. Developmental psychologists have long recognized that _object motion_ is a critical source of information about how people decompose scenes into objects, and may be the main cue through which infants _learn_ how to identify physical objects in static scenes [A]. Several recent unsupervised approaches have taken inspiration from this robust literature and prioritized object motion for learning to segment scenes [B,C,D]. This has led to some success on identifying objects in more realistic (or even real) scenes; it's therefore surprising that the authors don't really mention object motion as a critical learning cue or comment on others' progress in using it. Notably, object motion could allow for learning to segment objects without simple, uniform colors and textures, since complex parts could nevertheless be grouped by \"Common Fate.\"\n\nSo even if the authors don't think the present work is the appropriate place to solve these issues, I think the work really should at least bring them up and engage more with the literature here. If MONet, IODINE, and the like didn't already exist, there would be some logic in just developing a general approach; but at this point it's critical to identify and speak to the problems that are holding the field back.\n\n_Specific questions and concerns_\n\n1. The main quantitative evaluation presented is on unsupervised instance segmentation (Figure 4 and Table 1.) From what's shown, I don't quite follow how the authors are concluding that SIMONe sets a new \"state of the art\" on this task. Can SIMONe be tested on Static-ARI-F for direct comparison to MONet, etc. on the task for which the baselines were designed? I can see why the baselines wouldn't do as well on the Video ARI-F metric, since they're not explicitly designed to handle videos (except for S-IODINE -- but why create this \"ablation\" of OP3 rather than use the model that its authors intended?) Furthermore, the segmentations in Figure 4 from SIMONe look a lot \"blobbier\" than the ones from MONet shown in the Supplement; can the authors explain why SIMONe nevertheless has higher scores on this dataset?\n\n2. The qualitative analysis of the learned representations is a little cryptic. For example: \"The rubber duck does not morph into a chest of drawers because those are always located against a wall. This indicates a well-structured latent representation, which the decoder can adapt to\" (L236-241). Why does this indicate a well-structured latent? In other methods, would one expect the rubber duck to morph into a chest of drawers?\n\n3. I didn't really understand section 5.2 (\"Temporal abstraction\"). The authors seem to be making a point about how, in scenes where one or more object is moving, this motion is captured in the object latent rather than the frame latents, and that object motion is consistent across viewpoints. But what does it mean for an object trajectory to be captured in a \"time-invariant code?\" Is the idea that the interaction (in the Transformer modules) between the time-averaged object latents and the space-averaged frame latents can give rise to object motion? \n\n4. The authors point out that their architecture doesn't have an explicit dynamical model (e.g. a recurrent GNN) like OP3 or some others. But that seemed liked a strength of other models -- they could extrapolate object positions into the future (a little.) Would it be possible to do that with SIMONe after some modification?\n\n5. Related to my higher-level concerns above, the section on \"Scalability\" seems a little misdirected. What makes the authors think that SIMONe will scale to more complex scenes without major changes in the approach?\n\n6. Without at least some ablation studies, it's hard to know how important particular design choices are. Is the Transformer contributing significantly to this architecture, or could other latent state inference modules (like a recurrent GNN) serve just as well?\n\n\n[A] Spelke, Elizabeth S. \"Principles of object perception.\" Cognitive science 14.1 (1990): 29-56.\n\n[B] Veerapaneni, Rishi, et al. \"Entity abstraction in visual model-based reinforcement learning.\" Conference on Robot Learning. PMLR, 2020.\n\n[C] Bear, Daniel M., et al. \"Learning physical graph representations from visual scenes.\" arXiv preprint arXiv:2006.12373 (2020).\n\n[D] Du, Yilun, et al. \"Unsupervised Discovery of 3D Physical Objects from Video.\" arXiv preprint arXiv:2007.12348 (2020).\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, although I think the more relevant limitations are the ones I discussed above (complex objects and learning from object motion) rather than an inability to generate completely novel scenes, for instance.", "review_text": "", "extra_scores": {"ethical_concerns": "No, I don't have concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for learning disentanglement of scenes (as seen by an agent) into scene related content and object related content. It is based on transformers and a latent representation separated into time-varying and time-invariant slots.

The paper received 4 expert reviews, and in the discussion phase very quickly a consensus emerged on acceptance.	

Some weaknesses were raised, in particular with experiments, positioning and mid- and long-term objectives of this line work, and the advantages or disadvantages of key design choices (missing ablations). The authors provided a thorough and detail review, which answered most questions related to framing the method, and lead to increased ratings.

All reviewers agreed that this paper is of interest to the community and recommend acceptance.
The AC concurs.
Decision: Accept (Poster)