Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos
Authors: Gautam Singh
Yi-Fu Wu
Sungjin Ahn
Venue: NeurIPS 2022 Accept (2022)
Keywords: unsupervised object-centric learning
complex and naturalistic scenes
image transformer

Abstract: Unsupervised object-centric learning aims to represent the modular, compositional, and causal structure of a scene as a set of object representations and thereby promises to resolve many critical limitations of traditional single-vector representations such as poor systematic generalization. Although there have been many remarkable advances in recent years, one of the most critical problems in this direction has been that previous methods work only with simple and synthetic scenes but not with complex and naturalistic images or videos. In this paper, we propose STEVE, an unsupervised model for object-centric learning in videos. Our proposed model makes a significant advancement by demonstrating its effectiveness on various complex and naturalistic videos unprecedented in this line of research. Interestingly, this is achieved by neither adding complexity to the model architecture nor introducing a new objective or weak supervision. Rather, it is achieved by a surprisingly simple architecture that uses a transformer-based image decoder conditioned on slots and the learning objective is simply to reconstruct the observation. Our experiment results on various complex and naturalistic videos show significant improvements compared to the previous state-of-the-art.

Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos
Gautam Singh'
Yi-Fu Wu Rutgers University yifu. wu@gmail. com
Sungjin Ahn KAIST
Rutgers University s singh. weatggagtaaamtttrrrsccoom
sjn. ahn@gmail com
Abstract
Unsupervised object-centric learning aims represent the modular, compositional, and causal structure a scene as a set of object representations and thereby promises to resolve many critical limitations of traditional single-vector represen- tations such as poor systematic generalization. Although there have been many remarkable advances in recent years, one of the most critical problems in this direction has been that previous methods work only with simple and synthetic scenes but not with complex and naturalistic images videos. In this paper, we propose STEVE, an unsupervised model for object-centric learning in videos. Our proposed model makes significant advancement by demonstrating its ef- fectiveness on various complex and naturalistic videos unprecedented in this line of research. Interestingly, this is achieved by neither adding complexity the model architecture nor introducing a new objective or weak supervision. Rather, it is achieved by a surprisingly simple architecture that uses transformrr-base image decoder conditioned on slots and the learning objective is simply to recon- struct the observation. Our experiment results on various complex and naturalistic videos show significant improvements compared to the previous sfate-of-the-art google wwwpoool..../vv/////oooooooo.lee www.sooottttaaaoroeeee.eo.. for-
Introduction
The goal of object-centric representation learning is to represent the inherent structure of the physical world such as compositionality and causality as a set of representation vectors (and their relations) each corresponding to a conceptual entity module such
object (Greff et al. 2020 Schölkopf
ctt al.
2021). Many previous works have demonstrated the potential of this approach as a way of resolving the key limitations of traditional single-vector representations (Greff et al. [0200] Schölkopf
et al. 2021 Dittadi et al. 2021 Eslami et al.) 2016 Bapst et al., 2019) Mambelli et al. 2022 Ghasemipour et al. 2022). For example, considering object-centric representations independent mechanisms helps realize systematic and zero-shot generalization for image generation (Singh et al.)
2022 Chen al. 2021). Also, the ability decompose visual observation into set of discrete knowledge modules has shown to be useful for visual reasoning (Zhou et al. 2021 Wu et al. 202
One of the most critical challenges remaining in object-centric representation learning is to suc- cessfully apply it to complex and natural images or videos. Achieving this has primary importance because it opens up way to utilize the enormous amounts of images and videos available on the internet and thereby unleash the full potential of this method by applying at scale the machine learning community has recently observed the power of scale in large-scale language models (Brown et al. 2020 Bommasani et al. 2021)
"Correspondence singh. gwwtanrrutgers....moeo. edu and sjn. ahn@gmail com.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Recouituct Vidho Fremar
Image
Video Framse
Transfornus
Pied Micure Decoour
Piod Mixture Doooder
Transformes
Autoregessdet
Trarsforma
Stit
Atsention
Piseetion
Siots
Siots
State
Slote
Slots
Image
Video
Video
SLATE
Slot Attention Video
Slot-Transien Transformer for Videos
Figure Overview the proposed model. Left: SLATE auto-encoder combines the represen- tational bottleneck of slots with an auto-regressive transformer decoder and shows unsupervised object discovery visually complex images. However, the question of whether this framework can deal with complex and naturalistic videos is unexplored. Middle: Conventional object-centric video models such as Slot Attention Vidco deal with videos by applying slot attention recurrently on the frames and applying a pixel-mixture decoder to reconstruct the frames. However, in the fully unsupervised setting, these lack the ability to handle complex and naturalistic videos. Right: Our model Slot Transformer for Videos provides simple and minimal architecture leveraging ruto-reeressive transformer decoder that can effectively handle complex and naturalistic videos without any supervision.
However, despite significant amount of effort and many remarkable achievements in recent years, we do not yet have an unsupervised object-centric learning method that works well complex and naturalistic videos. fact, when considering the high complexity of natural images, the ultimate goal of making the concept of objects emerge without any supervision, i.e., only by observing, indeed remains quite elusive. As such, previous works have been shown to work only on toy or synthetic images/videos resorted utilizing some supervision such as optical flow or annotation the initial frame (Kipf et al. 202T).
In this paper, we propose unsupervised model for object-centric learning in videos, called STEVE Toot-TransfomEr for VidEos). Our proposed model makes significant advancement by demonstrat- ing its effectiveness for the first time on various complex and naturalistic videos unprecedented in this line of research. Interestingly, this achieved by neither adding much complexity to the model architecture nor by introducing new objective or weak supervision. Rather, it is achieved by surprisingly simple architecture, comparable to or simpler than most the previous methods that have not worked well on complex vidcos. Moreover, the learning objective is just to reconstruct. We believe this simplicity is one of the best strengths of our model.
Behind this success is the adoption of the transformer-base slot decoder recently proposed in SLATE (Singh et al., 2022). Because it is demonstrated the paper that this decoder is the key to dealing with some visually complex synthetic images, it is of deep interest to the community (i) whether and how it can be extended to temporal model utilizing dynamical observations, (ii) if it can eventually deal with complex naturalistic videos, and (iii) what lessons we can learn from this. To focus on investigating these central questions systematically, we intentionally choose not to pursue architectural novelty at the expense of adding complexity but to test minimal architecture that combines the SLATE decoder with a standard slot-level recurrence model. Our experiment results on various complex and naturalistic videos show that the proposed architecture significantly outperforms previous fate-of-the-- baseline models.
From these experiments, we discover important new knowledge to share with the community. We find that SLATE can deal with complex naturalistic images reasonably well despite not using temporal information. Although as static model, SLATE has a fundamental limitation that it cannot take advantage of temporal information, e.g., maintaining consistent object identity across time, if we only look at its frame-level segmentation ability, it already performs better than the previous state-of-the art temporal model that applies slot attention videos (Kipf et al. (02T). Crucially, we also find that SLATE alone is not enough since our model, STEVE, shows much better frame segmentation

---

than SLATE, while also providing consistent tracking of slots in videos, which SLATE cannot do. In particular, we find that on some complex videos such as MOVi-Tex, SLATE fails significantly. suggesting that learning jointly from both temporal information and the powerful SLATE decoder is significant and essential to realize the full potential of object-centric representation learning in videos. Furthermore, our analysis shows that our model is robust to challenges such as static objects and camera motion and that also generalizes well to out-of-distributinn number of objects and unseen textures. Lastly, we propose several new datasets that are significantly more complex than those tackled by the previous unsupervised models.
Preliminaries
In unsupervised object-centric representation learning, the goal is to learn to obtain the representation of an observation as a set of object vectors, referred to as slots (Locatello et al.) (020). A common approach for this is to auto-encode: slot-encoder fone (x) extracts set of slot representations s = (Si,... S from an image x and then a decoder (s) makes a reconstruction & from these slots, where x E " and Sy E IR. In the encoder, the slots compete with each other to learn about which area of the input will be explained by which slots. This provides a form of information bottleneck encouraging a slot to attend an area corresponding to a meaningful conceptual entity like an object. During training, reconstruction objective provides the learning signal for training the complete model. One nice property of this method is that it only requires reconstructing the image and does not require any weak supervision object-aware auxiliary loss terms. Ideally, want to maintain this simplicity in video models, but current state-of-the-art requires weak supervision and optical flow make it work on complex naturalistic videos (Kipf al. 202T
Mixture-basee Decoder. The emergence of object-centric slots is strongly dependent on what decoder is used. The traditional way that most of the previous methods have taken is the mixture- based decoder or some variant of it. In this approach, the decoder decodes each slot 8, to obtain an object image &,, and an alpha mask m,, via decoding functions gø g and gg guskk respectively. The decoded object images are then weighted-summet to obtain the full image x:
RGB 90
(s,)s
m2a
exp Go
mask mask(ss))) (Sn.)
X
m
X
Σm=1 exp 98 mask (Sm
n=1
These decoder networks implemented using CNN. The reconstruction objective is taken to be the squared error between the input and the reconstructed image i.e. Lmixture = | x|| A critical limitation of this approach is that it has never been successful in dealing with scenes with high visual complexity like natural images.
Autoregressive Not-Transsormer Decoder. Recently, (inghh et al. (0222) have challenged this traditional perspective that we need a mixture decoder for the emergence of objectness in slots. Arguing that mixture decoder severely limits the interaction among slots and the reconstruction quality-hene hence a problem in obtaining good learning signal for complex images images-they proposed a new architecture, SLATE, using a powerful autoregressive decoder based on transformer conditioned on the slots (Chen et al. 2020). However, many critical questions about this new approach still remain because the focus of the SLATE paper is on image generation ability. To get deeper understanding of the object representations that a model like SLATE produces, we would need to thoroughly investigate the quantitative evidence about its scene decomposition ability, its ability to deal with visual complexity close to that of natural images, and finally, the effect of extending this architecture temporal model. We aim to answer these in this paper.
3 STEVE: Slot Transformer for Videos
To study the effect of extending the alot-transformer decoder to a temporal model that can deal with videos, we choose to propose a minimal architecture. Specifically, our proposed model, STEVE, combines three main components: (1) CNN-based image encoder, (2) recurrent slot encoder that updates slots temporally with recurrent neural networks (RNNs), and (3) the slot-transformer decoder of SLATE. Although we may obtain additional performance gains by exploring further architectural novelty at the cost of additional complexity, we choose to investigate this minimal architecture to focus on goal of investigating slot-transformer decoder models for video.
3

---

Given a video consisting of frames Xi... Xy. we want to maintain N slots Se fst.1..... St.N for each time-step {1....T) }) by starting the temporal update from initial slots So- We apply recurrent slot encoder /Slot-mn at each step / to update the slot representations from S to S, using the information from the input frame X₃-
St fΦ fslot-rmn [ S(-)),
Sp initializee)..
It is expected that each slot would represent one object and track it consistently over time in a given video. For each frame, the slot representation 8, used to reconstruct the input frame x, using the slot-transformer decoder. For doing such reconstruction, each frame X, is treated discrete tokens provided by a discrete VAE encoder. Given the slots Sr, the slot-transformer decoder learns to predict this sequence of tokens auto-regressively by minimizing a cross-entropy loss,
sequence of
LCE = ΣΣ CE o1,)) 21,1 LLLL = JOVAF((X)) (x1), 11, 1=1-1
slt transsoonffrmrr d0
(8y; 2t,1 (t,1-1):
where -E((,-) denotes cross-entropy loss, (.......... L are L discrete tokens that act as prediction targets for the transformer at time t and or, contains the log-probabilities predicted by the transformer for the token at position for frame t. To train the discrete VAE, image reconstruction loss LIVAE is used as in SLATE.
LIVAE Σ +
X
Zt, Zt,L = f FdVAE (x1),
X
9o Zt, Zt,L).
The complete model is trained using LSTEVE LCE + LaVAE-
Recurrent Slot Encoder. Several recent works have implemented recurrent object-centric models by incorporating an RNN-based dynamics model to update slots over timesteps (Veerapaneni et al. 2019; Jiang et al. 2020 Kossen al., 2020 Lin et al. 2020a Stanić & Schmidhuber (201)). For our model, we leverage slot-based recurrent encoder that is also used as the backbone of Slot Attention Video (Kipf et al., 202T]. Our slot-based recurrent encoder works as follows. At the start of an episode, we provide initial slots So by randomly sampling from Gaussian distribution with learned mean and variance. For each input frame Xt, we compute an encoding in the form a feature map using backbone CNN. This feature map is flattened to obtain a set of input features e, = (er, e }. Next, the slots St-1 perform attention on the features e, and use the attention result to update the slots to Se. For this, the slots 5;-1 compute attention weights A = (A......., Ar, over the input features, where At,, are the attention weights for slot n. The attention weights are used to perform attention-weighe weighted sum of the input features. The attention result ri, is then used to update the respective slot using an RNN f RRNN
HW
Ar,,, v(e..) HW Σj=1 Arn..
((s(-1) (((t-1).((o)) k((,))
Btn
RNN (re.n S:-1,) Fen
Di=1
- sof
vD
Here, q. is and are linear projections and D is the output size of the projections. Finally, the slots are made to interact via an interaction network: 8, = f(nterec(ss) Following Kipf et al.] (2021), we the slots & before the interaction step for reconstruction. For complete details of the architecture, see Appendix B
Related Work
Unsupervised Object-Centric Representation Learning in Images. Unsupervised object-centric
learning methods for static scenes have received significant interest in recent years (Greff et al. 2016
Burgess al. 2019 Greff 2019 Locatello et al. 22000 Greff et al. 2017 Engelcke al. 2020
2021 Eslami et al. 2016 Crawford & Pineau| [019b]] [in et al. 2020b Jiang & Ahn 2020 Deng
al. 20h0n Chen et al.] 202i|\Ancuukevicuur et al.] 2020 yon Kügelgen et al. [2220] Greff et al.
2020p. These models learn through reconstruction, commonly adopting mixture based decoder.
Another line of work has pursued object discovery by minimizing the mutual information between the
predicted segments (Savarese et al. 2021 Yang et al. 2020). Approaches based on self- eeff-supeviieed
representation learning have also been investigated (Caron et al.] [202 Löwe al.) 2020 Wang et al.
2022). Recently, scene decomposition ability has been shown emerge in energy based models (Du
mased

---

et al. 2021a Yu et al. 2021). Complex- Complex-valued valued neural networks have also shown such emergence
Lowe et al. (2022). Closely related to our work. Lambb et al. (202T) and [Singh et al. (2022) have
shown that a slot-based encoder combined with a transformer decoder can enable object emergence in visually complex images. However, all of these approaches deal only with static images unlike
ours which can be applied to videos.
Unsupervised Object- Centrrc Representation Learning in Videos. A large body of work has approached fully-unsupervised video segmentation and tracking by combining recurrent slot-based encoders with a reconstruction objective. In this class, (Kosiorek al. 2018 Staaić & Schmidhuber
2019 Jiang et al. 2020 Crawford & Pineau 2019a; Lin et al.) 2020a; Wu et al. | [Singh et al.
2021; He et al. 2019) use bounding boxes for tracking. A parallel line of research learns to localize
objects via segmentation masks (Greff et al. 2017] Van Steenkiste et al.
2018:
Veerapanem ct al.
2019] Watters et al. 2019; Weis et al. 2020 Du et al. al.] Kipf et al.
2021
Kabra et al. 2021
Zoran et al. [021]] Beshinar & Frossard]
Creswell et al. 2020 (02T). Our work lies along
this line of research. Other approaches have considered using contrastive losses (Kipf et al., 2019
Carvalho et al. 2020) while some works have explored unsupervised object-centric learning in 3D
scenes (Chen et al. [2.22221 Henderson & Lampert 2020; Crawford & Pineau| 2020; Stelzner et al. 2021; Du et al. 202 Kabra et al.] (021). While dealing with 3D scenes would be an important future direction for our work, it is orthogonal to our current focus. However, unlike ours, all of the above works have only been successful on visually simple videos. In Appendix D we discuss the related work that deals with unsupervised video object segmentation using motion cues and unsupervised segmentation propagation.
5 Experiments
Datasets. We evaluate our model on datasets. These include 6 procedurally generated datasets: CATER (Girdhar& & Ramanan (020). CATERTex, MOVi-Solid, MOVi-Tex, MOVi-D, and MOVi-E (Grff et al. 2022); and 2 natural datasets: Traffic and Aquarium. Out of these, 5 datasets are our contributions in this work which we now describe. i) CATERTex. We generated it by combining objects and textures of CLEVRTex (Karazija et al.) 2021) with the object motion of CATER (Girdhar & Ramanaa 2020). ii) MOVi-Solid. We used Kubric (Greff et al. 2022) to generate this dataset and introduced textured backgrounds and more complex shapes to the MOVI dataset proposed by Kipl et al. (2021) iii MOVi-Tex. We added the textures from the Describable Textures dataset (Cimpoi et al. 2014 as materials to the MOVi-Solid dataset to make this dataset. By applying the same material on all surfaces, the objects and their borders become significantly harder to distinguish, resulting in much higher visual complexity over MOVi-Solid. iv) Traffic and Aquarium. We created these two natural datasets by collecting a 6-hour long video stream from Youtube. For training, all models use only the raw videos as input with no other supervision or input whatsoever. For evaluation, we use the ground truth instance-level masks. For sample video frames and more details, see Appendix [A] We will release all the proposed datasets which, we believe, will facilitate future research. For existing benchmarks i.c. CATER, MOVi-D and MOVi-E, we use the standard train and test splits as prescribed by their respective authors. We also note that in the MOVi benchmark i.e. MOVi-A to E, versions D and are the two most challenging versions. Thus, we consider these sufficient for evaluating our hypothesis.
Baselines. We compare our performance with three baselines which are state-of-the-art in unsu- pervised object-centric scene decomposition. These include two baselines that use mixture-based reconstruction: Slot Attention Video (Kipf et al., (02T) and OP3 (Veerapaneni al. 2019); ; and one baseline which uses transformer-based reconstruction: SLATE (Singh et al. (022). For Slot Attention Video, we use the fully unsupervised version that is trained only using the image reconstruction objective without optical flow or label information in the first frame. For simplicity, we will refer to this unsupervised Slot Attention Video as SAVi. Comparison with SAVi is important because our model is most similar to it with one main difference: our model uses transformer based decoding while SAVi uses mixture- based decoding. Therefore, comparison of our model with SAVi will provide a clear insight into the effect of the decoding approach. For the baseline SLATE, we use the same CNN backbone as our model for a fair comparison. In line with the previous works, for SAVi and OP3, we take their decoding masks to be their predicted segmentation. Since our model STEVE and the baseline SLATE are based on transformer decoder, they do not provide decoding masks. Therefore, we use their input attention masks our predicted segmentation. Note that this
5

---

CATER
CATERTex
MOVi-Solid
MOVi-Tex
MOVI-D
MOVIEE
100
100
100
Image FG-ARI (in %)
80
10
541
40
50
00
1A1
00
030
-0
STEVE SLATE
SAVI
OP3
2
3
3 Past Frames
Pact
Framos
Past Framos
Past Frames
8f Past Frames
17 Past Frames
Figure 2: Unsupervised Image Segmentation. On the y-axis, we plot Image FG-ARI of per-frame segmentation. Along the z-axis, we show how the segmentation affected as more frames are seen in video. The FG-ARI of SLATE is computed on one-frame 'videos' and is broadcasted along the x-axis to facilitate comparison with other models.
difference gives an advantage to the baselines but not to our model as in section 5.3 our analysis will show that the decoding masks perform better than the input attention masks.
5.1 Unsupervised Image Segmentation
In this section, evaluate how effective our model is for unsupervised image segmentation in videos. We report Image FG-AR/ which measures how well the predicted segmentation matches the ground truth segmentation of a given single image. In line with the previous works (Locatello al. 2020) Kabra et al. 2021: Kipf et al. 22221, we consider only the foreground segments of the ground truth we plot the Image FG-ARI of per-frame segmentation.
In Figure
Benefit over Mixture-based Methods. Comparing STEVE with SAVi and OP3, we see that STEVE achieves significantly higher FG-ARI compared with the baselines in 5 out of 6 datasets. All of these 5 datasets are characterized by complex textures. Therefore, our superior performance shows that our model is more effective in dealing with textured scenes compared to the baselines. Because the main difference of STEVE with SAVi is the reconstruction approach, this result provides strong evidence that the transformer-based reconstruction is the key driver of this improvement. Analyzing the qualitative results in Figures 4 (left. and reveaals that the baselines fail completely in CATERTex, MOVi-Tex, MOVi-D, and MOVi-E, most commonly by splitting the image into fixed patch regions instead of meaningful object regions. Unlike simple datasets used in prior works such as CLEVR or CATER that provide strong color contrast between objects, our datasets such as CATERTex and MOVi-Tex provide no such color cues. Despite this, our model can remarkably discover the objects. With the offe-of--the-rr baselines completely failing, it is for the first time in this line of research that such complex datasets have been effectively handled. In one dataset i.e. CATER, our performance is comparable to OP3 and slightly worse than SAVi. As CATER does not have complex textures, is not surprising to see baselines performing well on this dataset. In Figure 2 we also find that as the models see more frames, the segmentation tends to improve.
Benefit of Temporal Learning. We also analyze whether training STEVE on videos has any benefit in image segmentation compared to SLATE. As SLATE is applicable only for static images, we train and test it on one frame "videos' of our datasets. Comparing STEVE with SLATE, we note that STEVE performs consistently better in all datasets. Noteworthy is the gap in MOVi-Tex which is especially large. This suggests that training on temporal data i.e. videos can be helpful for learning to segment. We conjecture that due to the similar texture in the background and foreground. inferring correct object regions from a static image can be harder than inferring them when the objects are moving. This may explain the significantly larger gap with SLATE in MOVi-Tex compared to the other datasets. Note that this gap exists even when evaluating STEVE with zero past frames (i.e. static image), indicating that training on temporal data helps STEVE infer segments on static images.
5.2 Unsupervised Video Segmentation
In this section, we evaluate how well our predicted segmentation tracks the ground truth segmentation in both space and time without switching identities. For this, we compute Video FG-ARI which considers the full trajectory of segment over the video length one cluster. In line with the previous

---

CATER
MOVI-Solid
MOVHD
MOV-E
Video PG-ARI (in %)
80
20
80
50
60
STEVE
00
60
54
SLATE
OP3
11 Video Longth
15
10
18 Length
17
Longth
Length
Langth
Video Longth
Figure 3: Unsupervised Video Segmentation. On the y-axis, plot Video FG-ARI and compare our performance with the baselines. Along æ-axis, we show how the performance affected by the length of the video test time.
(a) Unseon Number of Objects
(In
T Test Set
BOOOD Test Set
20
CATERTox
MOVi-Solid
MOVI- Tex
(b) Unseen Materials
ARI %)
80
BMil Test Set
1OOOD Test Set
40
20
MOVi-Tex
MOViDD
MOVI-
Natural Scenes
Ours
Video FG-ARI
SAVi
40
IOOP
OP3
Aquarium
Figure Left: Visualization of Unsupervised Video Segmentation in CATERTex and MOVi- We show the video frames and their true masks followed by foreground and background segmentation of each model. We separated the visualization of the foreground segments to make the result easier to interpret by applying threshold project website https::p// : :/sstess...e.eeeeesseesseeeeettt .owgle. e httpll...c..aao wwt./viwwww/eeeeeeeeeeee slot cransformeet for for-vie vidos for more visualizations. Right: Unsupervised Video Segmentation in OOD and Natural Datasets. We plot the Video FG-ARI. In (a) and (b), we compare the performance of our model on IID and OOD test sets. In (c), evaluate our performance in unsupervised video segmentation natural scenes.
the segment area. See the supplementary material and the
works Locatello et al. (0000); Kabra et al. (2021); Kipf et al. (022), we consider only the foreground segments of the ground truth.
Video Segmentation Performance. In Figure 3 see that in 5 out of datasets, all of which are textured, STEVE significantly outperforms the baselines. In CATER, the baselines are able to
perform comparably or slightly better than ours. As with the segmentation result of Section [5.] this is not surprising given that CATER is visually simple enough for the baselines to handle well. We also observe the one major drawback of SLATE these results i.e. SLATE cannot provide aligned slot representations for video. This is because SLATE can only be applied to individual frames and the slots of each frame would be randomly permuted. also show how the performance affected by the length of the video. The models were trained on 3-length vidcos (6-length for CATER) while
7

---

Traffic
Youtube Aquarium
Tou
Visualization of Unsupervised Video Segmentation in MOVi-Solid, MOVi-E, and Natural Scenes. The rows (top bottom) show the input video, the true segmentation followed by the predicted segments of the models. See the supplementary material and the project website https://sites google. com www/vieww//eeo slot- transformer- for-videos for more visualizations.
Figure 5:
we evaluate on up to 24-length videos. As the likelihood of switching identities should increase with the video length, all models show some downward slope. Going from video length 6 to 24 in CATERTex, MOVi-D, and MOVi-E, our model deteriorates noticeably less than the baselines.
Out-of- Distrriitiintinn Generalization. In Figure 4 (a) and (b), we show how well our trained model generalize at test time out-of-distriburs objects and materials. We see that our model generalizes well and retains a similar performance as that the IID test set. One exception is MOVi-Tex in which our model generalizes well more objects but suffers on novel textures. Due to the modular structure of the recurrent encoder, each slot binds input objects quite independently. We conjecture that this modularity is responsible for good generalization to more objects. However, for new materials, generalization depends on how well the backbone CNN can generalize. As the training set of MOVi-Tex was generated using library of 240 textures, it would not be surprising if this CNN provides poor features of unseen textures. This, we believe, is affecting the performance of our model.
videos which have more number of objects and unseen
Natural Scenes. In Figure 4 (c), we evaluate video segmentation on two natural datasets taken from real-world videos: Traffic and Aquarium. We that our model performs significantly better than the baselines, approximately doubling the FG-ART in the Traffic dataset and tripling the FG-ARI in the Aquarium dataset. Noteworthy is the visual complexity of the Aquarium dataset in which the color and texture of fish are strongly camouflaged against the background and seeing them requires careful inspection even for a human (see Figure 5). Despite this, note that our model is able to deal with this challenge effectively.
5.3 Analysis
Diagnosing Slot Representation in STEVE. In previous sections, we focused on segmentation quality and used input attention masks of STEVE. However, input attention masks do not provide direct indication of the information content of slots. We would like to diagnose how well the slots represent the object-centric structure of the scene. For this, we take the slots from the pre-trained encoder of STEVE and, without propagating the gradient to the encoder, we train a mixture- based decoder to reconstruct the observation from the slots. We call this model STEVE- DDVV----iistcccc. In 6 (a), we compare the performance of unsupervised video segmentation with the baselines using the decoding masks of STEVE-Diagnostic. The decoder of STEVE- diaggnosi is made to be exactly the same that of SAVi for a fair comparison. We find that STEVE-Diagnosee better than the baselines all textured datasets, line with our earlier results Figures 2 and This shows that the slots in STEVE represent the object-centric structure better than the basclines. In
Figure
performs

---

(a) Diagnosis of Slot Representation of STEVE
(b) Effect of DVAE Patch Size in STEVE
ISTTVE
DaSTEVE-Daggoetic
RRPSS
B616616
32
FC-481 (ts
60
40
20
CATER
CATERTI
MOVi-Sali
MOVi-Txx
MOV-D
MOV-EE
CATER
MOVI-Sdid
MOVSE
(c) Using Attention Masks in SAVI for Unsupervised Video Segmentation
(d) Effect Deconvalution based Decoder on S/Vi
Video FGIARI N)
100
IBSTEVE ISSAV Decoding Masks
SaS/ Attention Masks
100
S
[ [sSAvi-Come Transoose
80
80
60
40
CATER
CATERTex
MOVI Solid
MOVI Tex
MOVI D
MOVI
CATERTox MOVI Soldd
MOVI Tex
MOVI D
MOVI E
Figure 6: Analysis. Based on the analyses described in Section 5.3 models. In all plots, we report Video FG-ARI computed on videos of length 6.
we evaluate various aspects the
Table 9 in Appendix C we further analyze the slot contents by training linear regressor predict the object positions and find that STEVE performs favorably relative to the baselines.
Effect of Granularity of dVAE on STEVE. In this section, we analyze the effect of granularity of discrete VAE on the segmentation performance of STEVE. In the default configuration of our model, the discrete VAE divides the image uniformly into patches of size 4 4 and represents each patch with one discrete token. That is, an image of size 128 x 128 would be represented by a sequence of length 1024. In this analysis, we test the effect on segmentation performance dVAE represents the image using larger patches i.e. 16 x 16 and 32 x 32. In our results shown in Figure 6 we note that using larger patches results in worsening of the performance. This suggests a possible scaling law that smaller patches lead to better performance. However, due to the prohibitive memory needs of the transformer on longer sequences, we were not able to experiment with smaller patches.
Effect of Transformer Capacity on STEVE. In Fig. 7 in the appendix, we analyze whether our
segmentation performance is sensitive to the capacity of the decoder. For this, test the effect of increasing the number of layers in the transformer decoder. We find that our segmentation performance is unaffected by this change, showing that model is robust.
Comparison between Decoding Masks and Input Attention Masks in SAVi. In previous sections on unsupervised image and video decomposition, we used input attention masks of STEVE while for the baselines, we used the decoding masks. We would like to test how different are the performances of input attention masks and decoding masks. While OP3 only provides decoding masks, fortunately, SAVi provides both input masks and decoding masks which we compare in Figure (c). We find that input attention masks tend to be worse than decoding masks of SAVi. Therefore, the attention masks of STEVE outperform both the input and decoding masks of the baseline SAVi.
Mixture- decoding without Spatial Broadcast. Mixture-based decoding requires CNN to decode the object image and the mask from each slot. To implement this CNN, the baselines prescribe using a Spatial Broadcast decoder, type of CNN decoder that limits the expressiveness of the decoder to facilitate the discovery of objects. We would like to test if more flexible CNN implemented completely using transposed convolutions would be enough to improve the performance (d),
or not. In Figure (d), our results on the 5 textured datasets show that this change leads to a similar or slightly higher performance than the original decoder. However, the model still fails by splitting the image into fixed patches similarly to before and still performs significantly worse than our model. This shows that simply using more flexible CNN with mixture-based decoding is not enough and the expressiveness of an nuto-regressive transformer is important for achieving significant gains.
Computational Requirements of STEVE. In Table 6 in the Appendix, we empirically compare the computational requirements of STEVE and SAVi. We assess by how much the computational requirement change in going from the mixture-based to the mansformer-based approach. We find that in videos with image size 64 x 64, STEVE requires similar resources as SAVi and about twice for image size 128 x 128. Given the quadratic memory complexity of transformers, the larger memory

---

demand of STEVE is not surprising. However, one should also consider that the mixture- based baselines fail almost completely on datasets like CATERTex, MOVi-Tex, MOVi-D, and MOVi-E. unlike ours. We also provide a big-O analysis of the memory costs in Appendix B.T
Effect of Camera Motion. Our segmentation results MOVi-D and MOVi-E in Figures and provide insight into the effect of camera motion as the MOVi-E dataset introduces a moving camera MOVi-D. We observe that camera motion in MOVi-E leads to slight improvement performance over MOVi-D. We conjecture that this is because moving camera causes relative motion between objects and can help better in bringing out patterns that tend to remain stable under motion.
Static Objects. We also note that our model is effective even when static objects are present in the scene such as in CATER, CATERTex, MOVi-D, and MOVi-E. In methods that rely purely optical flow for discovering objects can suffer significantly with static objects. For MOVi-D and MOVi-E, [rreff et al. (2022) report FG-ARI of 19.4 and 2.7 using SAVi trained with optical flow supervision. Compared to this, our model achieves a significantly better 47.67 and 52.15 15 on these datasets, respectively, despite the static objects and without using any supervision whatsoever.
Conclusion
In this paper, we study the effect of a powerful transformer- slot decoder for unsupervised object-centric video learning. For this, we propose a simple model, STEVE, and demonstrate that it can deal with various complex and naturalistic videos without any supervision by significantly outperforming previous state-of-the-art baseline models. experiments, we also find that although the slot-transformer decoder alone is an important component, for videos it is essential to learn also from temporal information for dealing with complex videos. The results of this paper and the SLATE paper jointly suggest using a powerful reconstruction decoder for unsupervised object-centric learning which is contrary to the traditional view advocating the use of weak mixture decoders. This raises question to the community: is accurate reconstruction from slots all we need to capture the objectness? ? In the future, it seems worth investigating further in this direction. Also, because we choose to investigate minimal architecture in this paper, would be interesting could obtain additionally by exploring more advanced architectures. Lastly, it would be interesting to investigate applying this method large-scale datasets. We further elaborate on these in Appendix F
what gains
Acknowledgments and Disclosure of Funding
This work is supported by Brain Pool Plus (BP+) Program (No. 2021HID3422031006// and Young Researcher Program (No. (022RIC1C100944) through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT.
References
Anciukevicius, T., Lampert, C. H., and Henderson, P. Object-centric image generation with factored depths, locations, and appearances. urXiv preprint arXiv:2004. 00642, 2020.
Bao, Z., Tokmakov, P., Jabri, A., Wang, Y.XX., Gaidon, A., and Hebert, M. Discovering objects that can move. ArXiv, abs/2203. 10159, 2022.
Bapst, V., Sanchez-Gonzallzz, A., Doersch, C., Stachenfeld, K. L., Kohli, P., Battaglia, P. W., and Hamrick, J. B. Structured agents for physical construction. arXiv preprint 19X4:39944:0377 2019.
Besbinar, B. and Frossard, P. Self-supervision by prediction for object discovery in videos. arXiv preprint arXiv:2103.05669 2021.
Bideau, P. and Leannd-Miller, E. G. It's moving! probabilistic model for causal motion segmenta- tion in moving camera videos. ArXiv, abs/16044.0 2016.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg. J., Bosselut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint 278.022108..7758, 2021.
10

---

Brown, T. B., Mann. B., Ryder. N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan. A.. Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Brox, T. and Malik, J. Object segmentation by long term analysis of point trajectories. In ECCV, 2010.
Burgess, P., Matthey, L., Watters, N., Kabra, R., Higgins I., Botvinick, M., and Lerchner, A. Monet: Unsupervised scene decomposition and representation. arXiv preprini arXiv: 1rXiv:1901. 13390, 2019.
Caron, M., Touvron, H., Misra, L,, Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in supervidd vision transformers. In ICCV 2021 International Conference Computer Vision, 2021.
Carvalho, W. Liang, A., Lee, K., Sohn, S., Lee, H., Lewis, R. L., and Singh, S. Reinforcement learning for sparse-reward object-interaction tasks first-person simulated 3d environments. arXiv preprint arXhv:2010.15199 2020.
Chen, C., Deng, F., and Ahn, S. Roots: Object-centric representation and rendering of 3d scenes. Journal of Machine Learning Research, 22(259):1 2021.
Chen, M., Radford, A., Child, R., Wu, K., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pixels. In ICML 2020: 37th International Conference on Machine Learning. volume 1, pp. 1691-1703, 2020.
Cho. K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. Learning phrase representations using encoder-decoder for statistical machine translation. arXiv preprint arXiv: /406 1078, 2014.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, , and Vedaldi, A. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Crawford, E. and Pineau, J. Exploiting spatial invariance for scalable unsupervised object tracking. arXiv preprint arXiv: 191::9911.000 1911.0003, 2019a.
Crawford, E. and Pineau, J. Spatially invariant unsupervised object detection with convolutional neural networks. Proceedings of AAAI, 2019b.
Crawford, E. and Pineau, J. Learning 3d object-oriented world models from unlabeled videos. In Workshop Object-Oriented Learning ICML, 2020.
Creswell, A., Nikiforou, K., Vinyals, O., Saraiva, A., Kabra, R., Matthey, L., Burgess, C., Reynolds, M., Tanbur, R., Garnelo, M., et al. Alignnet: Unsupervised entity alignment. arXiv preprint arXiv:2007.08973, 2020.
Creswell, A., Kabra, R., Burgess, C., and Shanahan, M. Unsupervised object-based transition models for 3d partially observable environments. arXiv preprint arXiv:2103.0480 033694 2021.
Deng, F., Zhi, Z., Lee, D., and Ahn, S. Generative scene graph networks. In International Conference on Learning Representations, 2020.
Dittadi, A., Papa, S., De Vita, M., Schölkopf, B., Winther, O., and Locatello, F. Generalization and
robustness implications in object-centric learning. arXiv preprint arXiv:2107.0000 (07.00637, 2021.
Du, Y., Smith, K., Ulman, T., Tenenbaum, J., and Wu, J. Unsupervised discovery of 3d physical objects from video. arXiv preprint 2rXiv:2007.123444 2020.
Du, Y., Li, S., Sharma, Y., Tenenbaum, J., and Mordatch, I. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems, 34, 2021a.
Du, Y., Smith, A., Ulman, T., Tenenbaum, J. B., and Wu, Unsupervised discovery of 3d physical objects from video. ArXiv, abs/2007. 2021b.

---

Engelcke, M., Kosiorek, A. R., Jones, O. P., and Posner, I. Genesis: Generative scene inference and sampling with object-centric latent representations. ICLR 2020 Eighth International Conference on Learning Representations, 2020.
Engelcke, M., Jones, O. P., and Posner, I. Genesis-v2: Inferring unordered object representations without iterative refinement. arXiv preprint arXiv:2104. 09958, 2021.
Eslami, S. A., Heess, N., Weber, T., Tassa, Y., Szepesvari, D., and Hinton, G. E. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pp. 3225 3225-3233 -2333, 2016.
Ghasemipour, S. K. S., Freeman, D., David, B., Kataoka, S., Mordatch, I,, et al. Blocks assem- ble! learning to assemble with large-scale structured reinforcement learning. arXiv preprint arXiv:2203. 13733, 2022.
Girdhar, R. and Ramanan, D. CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning. In ICLR, 2020.
Greff, K., Rasmus, A., Berglund, M., Hao, T., Valpola, H., and Schmidhuber, J. Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems, pp. 4484-4492, 2016.
Greff, K., van Steenkiste, S., and Schmidhuber, J. Neural expectation maximization. In Advances in Neural Information Processing Systems, pp. 6691-6701, 2017.
Greff, K., Kaufmann, R. I.., Kabra, R., Watters, N., Burgess, C.,, Zoran, D., Matthey, L.,, Botvinick, M., and Lerchner, A. Multi-object representation learning with iterative variational inference. arXiv preprint arXiv: 0ee:::/03000540 2019.
Greff, K., van Steenkiste, S., and Schmidhuber, J. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.052008 2020.
Greff, K., Belletti, F., Beyer, L., Doersch, C., Du, Y., Duckworth, D., Fleet, D., Gnanapragasam, D., Golemo, F., Herrmann, C., Kipf, T., Kundu, A., Lagun, D., Laradji, H., Liu, H.-T., Meyer, H., Miao, Y., Nowrouzezahrai, D., Oztireli, C., Pot, E., Radwan, N., Rebain, D., Sabour, S., Sajjadi, M. S. M., Sela, M., Sitzmann, V., Stone, A., Sun, D., Vora, S., Wang, Z., Wu, T., Yi, K. M., Zhong, F, and Tagliasacchi, A. Kubric: A scalable dataset generator. ArXiv, abs/2203.03570, 2022.
He, Z., Li, J., Liu, D., He, H., and Barber, D. Tracking by animation: Unsupervised learning of multi-object attentive trackers. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 13 1318-1327, 2019.
Henderson, P. and Lampert, C. H. Unsupervised object-centric video generation and decomposition in 3d. ArXiv, abs/2007 06705, 2020.
Janai, J., Güney, F., Ranjan, A., Black, M. J., and Geiger, A. Unsupervised learning of multi-frame optical flow with occlusions. In ECCV, 2018.
Jiang, J. and Ahn, S. Generative neurosymbolic machines. arXiv preprint arXiv:2010.122552 2020.
Jiang, J., Janghorbani, S., de Melo, G., and Ahn, S. SCALOR: generative world models with scalable object representations. In International Conference on Learning Representations, 2020.
Jonschkowski, R., Stone, A., Barron, J. T., Gordon, A., Konolige, K., and Angelova, A. What matters in unsupervised optical flow. In ECCV, 2020.
Kabra, R., Zoran, D., Erdogan, G., Matthey, L.. Creswell, A., Botvinick, M., Lerchner, A., and
Burgess, C. Simone: View-invariant, temporally-abstraated object representations via unsuper- vised video decomposition. arXiv preprint arXiv:2106.03849 2021.
Karazija, L., Laina, I., and Rupprecht, C. Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation. arXiv preprint arXiv:2111.1026 2021.
Kipf, T., van der Pol, E., and Welling, M. Contrastive learning of structured world models. arXiv preprint arXiv: 1271. 2019.
12

---

Kipf, T., Elsayed. G. F., Mahendran, A., Stone, A., Sabour, S., Heigold. G., Jonschkowski, R., Dosovitskiy, A.. and Greff, K. Conditional Object- Centric Learning from Video. arXiv preprint w.Xiv:2111.12544 2021.
Kosiorek, A., Kim, H., Teh, Y. W., and Posner, I. Sequential attend, infer, repeat: Generative modelling of moving objects. In Advances in Neural Information Processing Systems, pp. 8606- 8616, 2018
Kossen, J., Stelzner, K., Hussing, M., Voelcker, C., and Kersting, K. Structured object-aware physics prediction for video modeling and planning. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:://ppenrevieex net/forummid-llee http:/ppprrrrrrtee.eeerorrrreere kxSKDH
Lai, Z. and Xie, W. Self-supervised learning for video correspondence flow. ArXiv, abs/1905.00875, 2019.
Lai, Z., Lu, E., and Xie, W. Mast: memory-augmented self-supervised tracker. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 6478-6487, 2020.
Lamb, A., He, D., Goyal, A., Ke, G., Liao, C.-F., Ravanelli, M., and Bengio, Y. Transformers with competitive ensembles of independent mechanisms. In arXiv e-prints, 2021.
Lin, Z., Wu, Y.-F., Peri, V., Jiang, and Ahn, Improving generative imagination in object-centric world models. In International Conference on Machine Learning, pp- 4114-41224 2020a.
Lin, Z., Wu, Y.-F., Peri, S. V., Sun, W., Singh, G., Deng, F., Jiang, J., and Ahn, S. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations, 2020b.
Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention, 2020.
Löwe, S., Greff, K., Jonschkowski, R., Dosovitskiy, A., and Kipf, T. Learning object-centric video models by contrasting sets. arXiv preprint urXiv:2011.102877 2020.
Lowe, S., Lippe, P., Rudolph, M. R., and Welling, M. Complex-van valued autoencoders for object discovery. ArXiv, abs/2204.00075. 2022.
Mambelli, D., Träuble, F., Bauer, S., Schölkopf, B., and Locatello, F. Compositional multi-object reinforcement learning with linear relation networks. arXiv preprint arXiv:2201. 230,:22011,3888, 2022.
Oh, S. W., Lee, J.-Y., Xu, N., and Kim, S. Video object segmentation using space-time memory networks. 2019 JEEE/CVF International Conference on Computer Vision (ICCV), pp. 9234-9234, 2019.
Ren, Z., Yan, J., Ni, B., Liu, B., Yang, X., and Zha, H. Unsupervised deep learning for optical flow estimation. In AAAI, 2017.
Savarese, P., Kim, S. Y., Maire, M., Shakhnarovich, G., and McAllester, D. Eiformation-theorette segmentation by inpainting error maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4029-4039, 2021.
Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio, Y. Toward causal representation learning. Proceedings of the IEEE, 109(5):612 634, 2021.
Singh, G., Peri, S., Kim, J., Kim, H., and Ahn, S. Structured world belief for reinforcement learning in pomdp. In International Conference on Machine Learning, pp. 9744-9755. PMLR, 2021.
Singh, G., Deng, F., and Ahn, S. Illitraae dall-e learns to compose. In International Conference on Learning Representations, 2022. URL httppp///openreeeeeeeeeeeeeeeeee htt//forrum//dd1000000033.
Stanié, A. and Schmidhuher, J. R-sqair: relational sequential attend, infer, repeat. arXiv preprint arXiv:1910.05231, 2019.
13

---

Stelzner, K., Kersting, K., and Kosiorek, A. R. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.01 148, 2021.
Stone, A., Maurer, D., Ayvaci, A., Angelova, A., and Jonschkowski, R. Smurf: Self-teaching multi-frame unsupervised raft with full-image warping. 2021 IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3886-3895, 2021.
Tangemann, M., Schneider, S., von Kügelgen, J., Locatello, F., Gehler, P., Brox, T., Kümmerer, M., Bethge, M., and Schölkopf, B. Unsupervised object learning via common fate. arXiv preprint arXiv:2 10. 10.06562, 2021.
Van Steenkiste, S., Chang, M., Greff, K., and Schmidhuber, J. Relational neural expectation maximiza- tion: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv:1802.1035: 2018.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Veerapaneni, R., Co-Reyes, J. D., Chang, M., Janner, M., Finn, C., Wu, J., Tenenbaum, J. B., and Levine, S. Entity abstraction in visual model-based reinforcement learning. arXiv preprint a91iv: 12827, 2019.
von Kügelgen, J., Ustyuzhaninov, I,, Gehler, P., Bethge, M., and Schölkopf, B. Towards causal generative scene models via competition of experts. arXiv preprint arXiv:2004. 12906, 2020.
Vondrick, C., Shrivastava, A., Fathi, A., Guadarrama, S., and Murphy, K. Tracking emerges by colorizing videos. In ECCV, 2018.
Wang. N., Song, Y., Ma, C., gang Zhou, W., Liu, W., and Li, H. Unsupervised deep tracking. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 1308 1317, 2019a.
Wang, X., Jabri, and Efros, A. Learning correspondence from the cycle-consistency of time. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2561-2571, 2019b.
Wang, Y., Yang. Y., Yang, Z., Zhao, L., and Xu, W. Occlusion aware unsupervised learning of optical flow. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4884-4893, 2018.
Wang, Y., Shen, X., Hu, X., Yuan, Y., Crowley, J. L., and Vaufreydaz, D. Self-supervised transformers for unsupervised object discovery using normalized cut. ArXiv, abs/2202.11539, 2022.
Watters, N., Matthey, L., Bosnjak, M.. Burgess, C. P., and Lerchner, A. Cobra: Data-efficient basel-based through unsupervised object discovery and curiosity-drive exploration. arXiv 2019.
preprint /905.
Weis, M. A., Chitta, K., Sharma. Y., Brendel, W., Bethge, M., Geiger, A., and Ecker, A. Unmasking the inductive biases of unsupervised object representations for video sequences. arXiv preprint Xiv:20066000034 2, 2020.
Wu, Y.F., Yoon, J., and Ahn, S. Generative video transformer: Can objects be the words? In International Conference on Machine Learning, pp. 11307-11318. PMLR, 2021.
Yang, C., Lamdouar, H., Lu, E., Zisserman, A., and Xie, W. Self-supervised video object segmentation by motion grouping. arXiv: Computer Vision and Pattern Recognition, 2021.
Yang, Y., Loquercio, A., Scaramuzza, D., and Soatto, S. Unsupervised moving object detection via contextual information separation. 2019 JEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2019.
14

---

Yang, Y., Chen, Y., and Soatto, S. Learning to manipulate individual objects in an image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6558 6567, 2020.
Yu, P., Xie, S., Ma, X., Zhu, Y.. Wu, Y.N. and Zhu, S.-C. Unsupervised foreground extraction via deep region competition. In NeurlPS, 2021.
Zhou, H., Kadav, A., Lai, F., Niculescu- A., Min, M. R., Kapadia, M., and Graf, H. P. Hopper: Multi-hop transformer for spatiotemporal reasoning. arXiv preprint arXiv:2103. 2021.
Zhu, F., Zhang. L.,, Fu, Y., Guo, G., and Xie, W. Self-supervised video object segmentation. ArXiv, abs/2006. 12400, 2020.
Zoran, D., Kabra, R., Lerehner, A., and Rezende, J. Parts: Unsupervised segmentation with slots, attention and independence maximization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(C) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes] In sup- plementary material, we provide the architectural details and the hyperparameters used for our experiments. We will share the code and our proposed datasets at https://sites: google. cwm/vimei veem/view/ee http:/slt--oooe sttsssloottssttttteee transfranffffo ormer-
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] We provide the amount of resources used in Section B. in Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
We will share them upon acceptance.
(d) Did you discuss whether and how consent was obtained from people whose data you're 'rer
using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
15

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "rgGvej2xE_0", "reviewer": "Reviewer_88Gq", "summary": "The paper proposes a new method for unsupervised object discovery and tracking in videos called STEVE. This model combines an encoding backbone, a recurrent slot encoder and a transformer-based decoder to achieve impressive object discovery results on complex, textured multi-object datasets. Additionally, the model provides useful insights into its predecessor, SLATE, which worked on static images.\n", "questions": "Q.1) From Figure 1, it becomes apparent that the model improves in performance when a couple of initial frames have been processed. Does this effect still hold when objects move very fast in between frames? Is there some kind of maximum speed of objects between frames that the model can handle?\n\nQ.2) SLATE's big selling point was to be able to generate new multi-object images. Since STEVE is using a very similar architecture, would it be possible to generate multi-object videos with this model as well?\n", "limitations": "Limitations are adequately addressed in the paper.\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nS.1) The writing style is great, and the paper is a pleasure to read. It is easy to follow and contains a lot of useful insights. In particular, I enjoyed the summarizing statements at the end of many paragraphs in the experiments section.\n\nS.2) The proposed method is comparatively simple, yet very effective. It achieves impressive object discovery performance, especially on textured multi-object datasets.\n\nS.3) The paper provides extensive and insightful experiments across 8 datasets, 5 of which are introduced by the paper itself. \n\n\n# Weaknesses\n\nW.1) The description of the model in l.119-128 was somewhat unclear to me. Is the dVAE an entirely separate model from the slot-encoder, transformer-decoder architecture? How exactly do the dVAE and transformer decoder interact, and why? Additionally, since this is heavily based on the SLATE model, it would be nice to explicitly mention the architectural differences between the two models.\n\n# Others\nThe references need some clean-up: there are several papers that appear twice in the list.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "H5vkzuPn8LA", "reviewer": "Reviewer_kCr8", "summary": "Learning the compositional structure (e.g. scene graph) in dynamic visual scenes without labels (e.g. object masks) has proven challenging.  Slot-based models leveraging motion cues have recently made progress in learning to represent, segment, and track objects without direct supervision. However, these methods only work on synthetic data and fails on complex real-world multi-object videos.  This paper proposes STEVE, an unsupervised model for object-centric learning in videos. The paper clams that it made a significant progress by demonstrating its effectiveness on various complex and naturalistic videos unprecedented. This is enabled by a simple architecture that uses a transformer-based image decoder conditioned on slots and the learning objective is simply to reconstruct the observation.  Experiment results on several videos show improvements compared to the previous state-of-the-art, SAVi, SLATE and OP3.", "questions": "The paper uses discrete VAE encoder to discretize each frame as a sequence of discrete tokens. The authors may want to try Vision-Transformer-based VQGAN (ViT-VQGAN) which has found successes in Parti (Scaling Autoregressive Models for Content-Rich Text-to-Image Generation, https://arxiv.org/abs/2206.10789).\n\nThe paper should provide more details on the two natural datasets. Furthermore, it should consider evaluating on existing benchmarks, e.g. Waymo Open Dataset.\n\nThe paper should provide more insights and characterize on when STEVE works and when it fails. Identify failure cases is important for future work.\n", "limitations": "The paper does not address its limitations. A section clearly articulating the technical limitations is important. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe paper proposes a a simple architecture that uses a transformer-based image decoder conditioned on slots and the learning objective is simply to reconstruct the observation. The transformer-based image decoder uses the one proposed in prior work, SLATE (Singh et al., 2022).\n\nWeaknesses\n\nDespite the claims on its effectiveness on complex and naturalistic videos, the datasets seem to be very limiting.  6 of them are procedurally generated datasets, CATER (Girdhar & Ramanan, 2020), CATERTex, MOVi-Solid, MOVi-Tex, MOVi-D, and MOVi-E (Greff et al., 2022). 2 natural datasets are only collected for this paper: Traffic and Aquarium. The paper does not provide details on the complexity and diversity of these datasets. \n\nThe paper seems to lack novelty as the design seems to be a straightforward combination of (1) a CNN-based image encoder, (2) a recurrent slot encoder that updates slots temporally with recurrent neural networks (RNNs), and (3) the slot-transformer decoder of SLATE.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DpD8Qj0U1l-", "reviewer": "Reviewer_84xq", "summary": "-- Post rebuttal update ---\nThe rebuttal addresses my concerns with new experiments and a new limitations section in the manuscript. I continue to support the acceptance of this paper. Rating: 8.\n\n-- End of post rebuttal update---\n\nThis paper proposes a model, called STEVE, for object centric learning for video data. STEVE is a combination of a recurrent slot encoder (like SAVi) and the SLATE slot-transformer decoder. The combination works surprisingly well on relatively complex synthetic data and even in some real world video datasets. STEVE demonstrates better object centric decomposition when compared against baselines on the FG-ARI metric. It also generalizes OOD to unseen number of objects and unseen materials.", "questions": "No major limitations and questions. The paper is a solid submission. I have discussed questions and concerns in the previous section itself.", "limitations": "Societal impact has been discussed.\n\nLimitations can be elaborated further. For instance, the memory complexity of SLATE has been discussed in the paper; but the time complexity of sampling an image has not been discussed. Since transformer decoders need auto-regressive token-by-token generation they tend to be very slow during inference (not a problem during training). Maybe this becomes a big issue when running this at high resolution (512x512 or 1080p)?", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The proposed method is simply yet very effectively. STEVE does not use extra annotations in the first frame, nor does it need optical flow or other signal in its loss function. Purely using dVAE token reconstruction as the loss function (indireclty just pixel reconstruction), STEVE is able to decompose textured videos, handle static objects (MOVi-D dataset), and moving camera (MOVi-E dataset).\n\n- The method has been tested on a wide variety of datasets. This gives more weight to the claims. The comparisons show that the SLATE decoder is significantly better than a spatial broadcast decoder in both per frame and video metrics. The paper thus also provides insights about the SLATE decoder. SLATE itself is very powerful for object centric learning.\n\n- The analysis covers obvious concerns such as using encoder attention maps as opposed to decoder attention maps, and relevant ablations  such as dVAE granularity are included.\n\n### Weaknesses\n* Minor issues:\n  - Missing legend in figure 3(a), please note how many objects were used in the in-distribution and OOD settings.\n\n  - If possible include STEVE in figure 5(a). What's the performance of STEVE relative to STEVE-Diagnostic?\n\n  - I think you mean \"convolution transpose\" not \"Deconvolution\" in page 9. Please describe the exact architecture of this convolution transpose network. The spatial broadcast decoder also does some convolution transpose operations. How is this decoder different?\n\n  - An explicit big-O analysis would be great in the \"computational requirements\" paragraph.\n\n* A model figure is missing. I have read SAVi paper so I understand this method but people new to STEVE would likely benefit from a nice figure.\n\n* FG-ARI ignores background pixels. This does not penalize leaking of object segments into the background. Relying only on this metric is dangerous. Is it possible to report readout based metrics in these datasets? For instance the slot attention paper reported property prediction for CLEVR.\n\n* It is still not clear why the transformer decoder, which mixes information from all the slots in the decoding step unlike the mixture decoder which is applied to each slot independently, still leads to object decompositions? Does one need to control transformer capacity to make this work? What is the most sensitive hyper-parameter here?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4FsBZFIXWQe", "reviewer": "Reviewer_WXnF", "summary": "This work attempts to learn improved slot-based object representations and video segmentations with the help of the SLATE/Image GPT [2] decoder. The model is evaluated on several datasets included ones generated by the authors. ", "questions": "**Claims**:\n1. Page 2, line 87: “The emergence of object-centric slots is strongly dependent on what decoder is used.” Could you please provide supporting evidence for this.\n2. Page 3, line 92. “The reconstruction objective is taken to be the squared error between the input and the reconstructed image.” This is not always the case and I would strongly caution against this over-simplification. Typically you would use the mixture likelihood $p(x) = \\sum_n m_n \\odot p(x; \\hat{x_n}, \\sigma)$ rather than an MSE wrt to $\\hat{x} = \\sum_n m_n \\odot \\hat{x_n}$, which is a lossy point estimate that defeats the purpose of a mixture model.\n3. Page 9, line 209: “Therefore, in all the comparisons, the decoding masks gave an advantage to the baselines but not to our model.” This claim is far from obvious. If SAVi’s decoder masks are better than its own attention masks, that is due to either the smoothing effect of the decoder or representation quality/information bottleneck in SAVi specifically. A model with a different loss/architecture/set of hyperparameters may behave entirely differently. For example: if we shrank the decoder size (number of layers/parameters) to make it underfit, the decoded segmentation performance will likely undershoot the attention mask performance. Hence, whether STEVE is at an advantage or disadvantage cannot be ascertained by looking at SAVi's performance alone.\n\n**Methodology**:\n1. Why not use an existing natural image/video dataset? It is impossible to judge the quality of Traffic and Aquarium especially given the segmentations were derived by manual annotation. \n2. OP3 is not the best choice of baseline. Its performance is not meaningfully different from SAVi’s (except on MOVi-E where it suffers from not being able to handle camera movement). Its inclusion does not shed any light on STEVE’s performance. Could you consider at least an improved version of OP3 (e.g. PARTS [1])?\n3. Figure 3: “We separated the visualization of the foreground segments to make the result easier to interpret by applying a threshold on the segment area.” This makes the use of the ARI-FG metric somewhat disingenuous. It is an acceptable measure so long as the discarded background pixels correspond to a single or separately modeled (e.g. in MONet or SPACE respectively) component. Can you reliably extract the background components from STEVE? Is your threshold fine-tuned per dataset? Do you use the same threshold across the baseline models?\n\n**Results**:\n1. Figures 3 and 4: In both CATERTex and MOVi-Tex, the model tends to split the textured background into multiple components. But it tends to produce a solid background on both Traffic and Aquarium data. Could you explain this difference in behavior, despite the obviously textured background in the natural datasets?\n2. movi-d-002.gif: the falling object takes at least 4 different slots over time (green, blue, yellow, pink). Surely this is far from desirable? There’s no camera motion in this video. For human eyes, the moving object is trivially trackable.\n3. catertex-001.gif: there’s a single slot (pink) for multiple object shadows, even when the objects have different dynamics. If the model took its cues from object dynamics (rather than colors), surely you would expect each object’s shadow to be associated with that object’s slot?\n\n**Writing**:\n1. There’s a lot of repetition between the abstract and introduction. Moreover, sentences like the following (Page 2, line 59) are superfluous in a technical submission to a research conference: “From these experiments, we discover important new knowledge to share with the community.”\n3. Page 3, line 124: could you please clarify the dimensionality of each $z_{t, l}$? Are they one-hot vectors indexing into the discrete codebook? There’s other model details which would be useful to include in the main paper.\n4. Figure 1: what are the error bars computed on?\n5. Could you please cite [2] which is clearly the inspiration for the decoder architecture? It is not sufficient to cite SLATE alone.\n\n**References**:\n\n[1]  Zoran, D., Kabra, R., Lerchner, A., & Rezende, D. J. (2021). PARTS: Unsupervised segmentation with slots, attention and independence maximization. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ (pp. 10439-10447).\n\n[2] Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D. &amp; Sutskever, I.. (2020). Generative Pretraining From Pixels. _Proceedings of the 37th International Conference on Machine Learning<_, in _Proceedings of Machine Learning Research_ 119:1691-1703 Available from https://proceedings.mlr.press/v119/chen20s.html.\n", "limitations": "There was no attempt to explain the limitations of the model.", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The results have dubious merit, the claims are poorly supported, and the writing needs a lot of improvement. There is little innovation in experimental methodology. On the plus side, the paper conducts a wide evaluation over several datasets, but the central claim that STEVE works on naturalistic images is not convincing at all. It is also unclear whether STEVE is any different than SAVi with a SLATE/Image GPT [2] decoder.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept