VONet: Unsupervised Video Object Learning With Parallel U-Net Attention and Object-wise Sequential VAE
Authors: Haonan Yu
Wei Xu
Venue: ICLR 2024 poster (2024)
Keywords: Unsupervised learning; Video object learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A new state-of-the-art unsupervised video object learning approach incorporating parallel U-Net attention, object-wise sequential VAE, and transformer-based decoding.

Abstract: Unsupervised video object learning seeks to decompose video scenes into structural object representations without any supervision from depth, optical flow, or segmentation. We present VONet, an innovative approach that is inspired by MONet. While utilizing a U-Net architecture, VONet employs an efficient and effective parallel attention inference process, generating attention masks for all slots simultaneously. Additionally, to enhance the temporal consistency of each mask across consecutive video frames, VONet develops an object-wise sequential VAE framework. The integration of these innovative encoder-side techniques, in conjunction with an expressive transformer-based decoder, establishes VONet as the leading unsupervised method for object learning across five MOVI datasets, encompassing videos of diverse complexities. Code is available at https://github.com/hnyu/vonet.

Published as a conference paper at ICLR 2024
VONET: UNSUPERVISED VIDEO OBJECT LEARNING WITH PARALLEL U-NET ATTENTION AND OBJECT- WISE SEQUENTIAL VAE
Haonan Yu
Wei Xu
Horizon Robotics Cupertino, CA 95014, USA haonan. www.an.iu@hrrii..cc
Horizon Robotics Cupertino, CA 95014, USA wei. www.xuuuoriiccc...
ABSTRACT
Unsupervised video object learning seeks to decompose video scenes into struc- tural object representations without any supervision from depth, optical flow, or segmentation. We present VONet, an innovative approach that is inspired by MONet. While utilizing a U-Net architecture, VONet employs an efficient and effective parallel attention inference process, generating attention masks for all slots simultaneously. Additionally, enhance the temporal consistency of each mask across consecutive video frames, VONet develops object-wise se- quential VAE framework. The integration of these innovative encoder-side tech- niques, in conjunction with an expressive transformer-basee decoder, establishes VONet as the leading unsupervised method for object learning across five MOVI datasets, encompassing videos of diverse complexities. Code is available at https: / / thub. com httm///yy//voetteteee
INTRODUCTION
Unsupervised video object learning has garnered increasing attention in recent years. It focuses on the extraction of structural object representations from video sequences, without the aid of any super- vision, such as depth information, optical flow, or labeled segmentation masks. The goal is to enable machines to automatically learn to discern and understand objects within video streams, an essential capability with wide-ranging applications in fields such as autonomous robotics (Veerapaneni al., 2020; Creswell et al., 2021), surveillance (Jiang et al., 2019), and video content analysis (Zhou et al., 2022). The utilization of such object-centric representations could lead to improved sample efficiency, robustness, and generalization novel tasks (Greff et al., 2020).
Slot attention methods (Locatello et al., 2020; Kipf et al., 2021; Elsayed et al., 2022; Singh et al., 2022b) have recently demonstrated significant successes in video object learning. They typically utilize a CNN to extract a feature map from an input image. This feature map is spatially flattened into a sequence of features, which are then queried by each slot latent generate attention mask. Our observation is that slot attention often encounters dilemma, referred to as "granularity versus continuity"". To generate fine-grained attention masks, it is necessary to select a large spatial shape for the feature map. However, doing so makes it challenging to ensure the smoothness of the mask due to the nature of the Key-Query- Value attention mechanism (Locatello et al., 2020). Sometimes the absence of smoothness may result in significant mask quality degradation.
This paper introduces VONet for unsupervised video object learning. Inspired by MONet (Burgess et al., 2019) for image object learning, we posit that the inductive bias for spatial locality, as seen in the U-Net (Ronneherger al., 2015) of MONet, offers a solution to the dilemma. However, MONet's recurrent attention generation. forwarding the same U-Net multiple times sequentiallyy is very inefficient when handling large number slots, and consequently impedes its further application to video. Our first key innovation is efficient and effective parallel attention inference process (Figure 1, b) that generates attention masks for all slots simultaneously from a U-Net. It can sustain a nearly constant inference time the number of slots increases within reasonable range.
Furthermore, to achieve temporal consistency of objects between adjacent video frames, VONet incorporates an object-wise sequential VAE framework. This framework adapts the conventional

---

Published as a conference paper at ICLR 2024
sequential VAE (Kingma & Welling, 2013; Hafner et al., 2019) to the context of multi- inter- action and dynamics. The minimization of the KLD between the posterior and a forecasted prior models the dynamic interaction and coevolvement of multiple objects in the scene. This encourages the emergence of slot content that is temporally predictable and thus consistent in a holistic manner. By adjusting the weight of the KLDD, we are able control the importance of temporal consistency relative to video reconstruction quality.
To further bolster its capabilities, VONet employs an expressive transformer-based decoder (Singh et al., 2022b) that empowers itself to successfully derive object representations from complex video scenes. To showcase the effectiveness of VONet, we conduct extensive evaluations across five MOVI datasets (Greff et al., 2022) encompassing video scenes of varying complexities. The evaluation results position VONet as the new state-of-the-an unsupervised method for video object learning.
2 RELATED WORK
Numerous prior studies, such as Burgess et al. (2019); Greff et al. (2019); Locatello et al. (2020); Engelcke al. (2021); Singh al. (2022a); Zoran et al. (2021); Emami et al. (2021); Hénaff et al. (2022); Seitzer et al. (2022), explored unsupervised object learning in single images. For unsuper- vised video object learning, applying these image-based methods video frames independently is not a viable approach, as it would likely result in slot masks lacking temporal consistency. A conventional strategy for transitioning from image object learning to video object learning entails inheriting and modifying the slot content from the preceding time step. For instance, AIR (Eslami et al., 2016), SQAIR (Kosiorek et al., 2018), STOVE (Kossen et al., 2019), and SCALOR (Jiang et al., 2019) all employed a propagation process in which a subset of currently existing objects is propagated to the next time step; TBA (He et al., 2019). ViMON (Weis et al., 2021), SAVI (Kipf et al., 2021), SAVI+ (Elsayed et al., 2022), STEVE (Singh et al., 2022b), VideoSAUR (Zada- ianchuk et al., 2023), and RSM (Nguyen et al., 2024) initialized slots for the current step using the output of forward predietor/tracker applied to the preceding slots. Another technique for ensuring temporal consistency is to model constant object latents across time, as demonstrated in Kabra et al. (2021). These object latents remain invariant across all frames by design, enabling stable object tracking. Alternatively, an explicit temporal consistency loss could be incorporated. Creswell et al. (2021) proposed an alignment loss which ensures that each object is represented in the same slot across time; Bao et al. (2022) encouraged similarity between the feature representations of slots in consecutive frames. Our approach inherits preceding slot content while also introducing a KLD loss of sequential VAE to further enhance temporal consistency.
Due to the absence of supervision signals, most existing methods could only handle uncomplicated video scenes with unfformly-colored objects or objects with simple sizes and shapes. For instance, in an unsupervised setting, Creswell al. (2021); Kipf et al. (2021); Elsayed et al. (2022) demon- strated effectiveness primarily on uniformly-colored objects with clean backgrounds. Similarly, while SCALOR (Jiang et al., 2019) showcases an impressive capability in discovering and tracking dozens of simple objects of similar sizes in certain videos, it exhibits sensitivity to hyperparameters related to object scale and size ratio. As a result, it performs inadequately when dealing with textured objects of diverse shapes and sizes. STEVE (Singh et al., 2022b), the other hand, successfully im- proved performance in complex videos through the introduction of an expressive transformer- bassed decoder. Nevertheless, it faces significant issues of over-segmentation and object identity swapping in less complex videos. Our method adopts the same transformer-batt decoder to handle complex video scenes, while still being able maintain superior performance in simple scenes.
A substantial body of related work in the field of video segmentation (Zhou et al., 2022) relies on supervision signals such as segmentation masks, depth information. optical flow, and more. Our problem also bears relevance to the domain of video object tracking (Ciaparrone et al., 2020). where object locations or bounding boxes are specified in the initial video frames and tracked across sub- sequent frames. Since we do not assume these additional learning signals, due to space constraints, we will not delve into detailed discussions of these two topics here. Notably, certain previous works, such as Zadaianchuk et al. (2023), employ a vision backbone pretrained on large-scale datasets for feature extraction from video frames, aiding in the process of object learning. While their objective remains unsupervised, the use of pretrained features integrates valuable prior knowledge of every- day object appearances. Consequently, the results they reported, though surpassing what typical unsupervised methods including ours could achieve, hold limited reference value in this context.
2

---

Published as a conference paper at ICLR 2024
generator
All
All Net
Paraliai atiention network
(a)
(b)
Figure 1: Attention processes of MONet (a) and VONet (b) on a single image. Red arrows represent sequential operations ("scp" stands for the MONet scope), while blue arrows at the same horizontal level represent parallel operations. The dependency on the input image has been omitted for clarity.
VONet shares similarity with VIMON (Weis et al.. 2021) in that both extend MONet (Burgess et al., 2019) from images to video. However, several major differences exist between the two: i) VONet parallelizes the original MONet architecture for efficient inference while ViMON still generates slots recurrently. ii) VONet seeds the attention network with context vectors while ViMON does this using previous attention masks. iii) VONet formulates object-wise sequential VAE to promote temporal consistency, whereas ViMON ignores this by using a typical intra-step VAE.
PRELIMINARIES
MONet for unsupervised image object learning. The Multi-Object network (MONet) (Burgess et al., 2019) is scene decomposition model designed for single images. A forward pass of MONet generally consists of two stages: mask generation and image reconstruction with a component VAE. Given input RGB image generate K attention masks my E [o, 1| with K representing the predefined number of slots. A slot represents either an object at each step, the mask is determined as m, = ax_ 8k-1) where the scope Sp-1 E [o. 1] 1|0x x represents the remaining attention for each pixel, with the initial scope being So 1. The atten- tion network, denoted as a, is implemented as U-Net (Ronneberger et al., 2015), to predict the amount of attention the k-th step will consume from the current scope Sk-1. The scope is then updated as S = Sk-1 - (k-)).. At the last step, MONet directly sets mæ Sk-1, which ensures that the entire image is explained by all slots E k=1 my 1). Conditioned on the predicted attention mask, each slot undergoes independent processing through a VAE to (par- tially) reconstruct the input image. The VAE first encodes each slot into compact embedding by Zy ~ {z x |x, mg) and then derives a decoded image distribution o(x[z+) from this embedding. To do so. pixels {xa} are decoded independently from each other conditioned on the slot embedding, namely o(x|zx) = II, o(x,, (z.). Finally, to consolidate the reconstruction results from different slots, MONet formulates a mixture of components decoder distribution in its training loss !:
R #xxxx MONet utilizes a recurrent procedure sequentially
background region. The procedure comprises K steps, and
#xW
LONNE =
Σ log Σ ((Xalzz)ma.n + Σ DKi (((zllx mx) (p(zn). k=1
m
(I)
where the prior p(zx) is unit Gaussian. Intuitively, each slot only needs to encode/decode image regions that has been selected by its attention mask. Figure (a) depicts the simplified attention process of MONet, emphasizing the data flow during forward pass. MONet is deterministic and its masks have only unidirectional dependencies. Due to the recurrent nature of mask generation, as K increases, the inference time for a forward pass will become prohibitive.
Unsupervised video object learning. In the context of unsupervised video object learning, each input sample is sequence of RGB frames fx1..... xr). The goal is determine a set of attention masks. {m,, 1k=1- | for each frame x, as in the single-image case. Additionally, the mask sequence, (m.,))=1. associated with a specific slot k, should focus on consistent set of objects. Because MONet was originally proposed for single images, there is no guarantee that directly applying to two adjacent video frames will result in temporally coherent object masks, even though the two video frames might look similar. It is yet to be determined how MONet can be extended to facilitate video object learning with temporal consistency.
'OONet included third loss term to encourage the geometrical simplicity of the attention masks m. by having the decoder also reconstruct these masks. This loss is not discussed here.
3

---

Published as a conference paper at ICLR 2024
4/.1
Inage
mardoomm
Image
LiNi down
Net
4
b
9/_K
9-11X
Figure 2:
Diagram of the parallel attention network. Except for the transformer and softmax opera- tor, it is possible to parallelize the executions related to the U-Net components. The skip connections between the U-Net downsampling and upsampling layers have been omitted for clarity.
VONET FOR UNSUPERVISED VIDEO OBJECT LEARNING
Parallel U-Net attention. VONet begins by eliminating the concept of "scope" in MONet and introduces context, a compact embedding vector expected to encompass prior object information for each slot to be generated. Let the context vector of slot time step be Ct-1k Instead of recurrent mask generation, VONet generates all masks at step t at once (Figure 1, b):
m ParallelAttt(x, X,, 1-11,1.-... C.-1kk)
(2)
The parallel attention network operates by simultaneously applying the same U-Net architecture to the K context inputs in parallel, while establishing communication among the slots the U- Net bottleneck layer. To achieve this, the output of the U-Net downsampling path is first flattened, and the outputs from different slots are pooled to create a sequence. This sequence is then input into decoder-only transformer that produces a sequence of latent mask embeddings. Each mask embedding further projected and reshaped to a 2D feature map, which then serves as the input for the U-Net upsampling path. In the final step, the output logits maps undergo pixel-wise softmax to derive the ultimate attention masks (Figure 2). Formally,
9r Kt,K Transformermask (h..1.... k.,k). b = doown (Xt, Cc-1,),
(3)
1e,A Ump(
1 m K 1o k.
Due to the single shared downsampingguusampingg path among the slots, practice, Udown and Uup can be executed efficiently by rearranging the slot dimension into the batch dimension.
Calculating the contexts. One may pose the question: how can the context vectors be acquired?
As previously mentioned, context vector for specific slot should encapsulate prior information regarding the aspects of the scene that this particular slot is intended to encode. Indeed, this prior information can naturally be derived from the content of that slot at the preceding time steps. For each slot k we derive its context Cr (used by + 1) based on the history the slot up to step t:
Yt = SlotEnc(Xx, m,,s), Ft.k = RNN(Y It-1,k), Cr = MLLexx((.)))
(4)
where the per-frame slot latent Yt.k extracted through a slot encoder that takes both the image X, and the generated attention mask m the inputs. Meanwhile, Ft, can be viewed as the per- trajectory slot latent, as it accumulates previous per-frame slot latents via recurrent network. We initialize the per-trajectory slot latent by transforming a collection of noise vectors independently drawn from a unit Gaussian:
ro. Fo,K = Transformer,aaf
Ej NO,,
This initialization signifies a stochastic process where the slots are collaboratively seeded prior to any exposure to the video content. Figure 9 (Appendix A.2) illustrates the temporal unrolling of VONet during forward pass video.
Object-wise sequential VAE. Finally, we compute the slot posterior distribution directly based on its most recent -rajerajyy slot latent at as
24,k q( (T,,))
(6)
4

---

Published as a conference paper at ICLR 2024
Context vector
Atlention mask
encoding
Per-trjjctory
Parsiles
m
rt,*
Z
TranDDoc
1t-1+
MLP
Dec tosa
KLD
Popt
Predicted por-trjectory elot latent Tt.
prior
Figure 3: VONet's architecture. The dependency the input image x, has been omitted for clarity.
in the subscripts represents the collection of K (K = 2 here) slots in parallel.
Meanwhile, we derive the slot prior distribution using all the K per-trajectory slot latents up to t ě-1 1,
p ,(..,,,..,,
which in fact results in object-wise sequential VAE. In essence, for proper learning of the prior, VONet must anticipate how each slot will evolve in interaction with the other K 1 slots. A straightforward approach to instantiating the prior would involve utilizing transformer to predict future slot latent for each slot. This prediction can then serve as the basis for computing the prior:
Y r.. r/,K Transformerpaie ( (f:-1,1:-..; rt-1,k), -(-1,, Zt,k p(Z),x))r...)
(7)
For reconstruction, we opt for the transformer based decoder (Singh et al., 2022a;b), owing to its remarkable performance observed in handling complex images. Thus our overall training loss is
LVONe
log PrrmnsDe (x, Ze......... Zt.K)
(8)
Σ DEL (gllze//Taaa
Note that our KLD term, which integrates a learned prior, plays pivotal role in strengthening the temporal consistency of individual slots across consecutive video frames. The rationale behind this enhancement lies in the fact that only slot representations that exhibit temporal consistency can exhibit predictability, consequently resulting in reduced KLDD loss.
An overview of the architecture of VONet is illustrated in Figure 3.
IMPLEMENTATION
Backbone. In the very initial stage of the encoding step, use CNN backbone (Appendix A.2) to extract a feature map from each input image x, The output from the backbone is shared between the parallel attention network (Eq. and the slot encoder (Bq. 4), serving as the actual image input. Through parameter sharing, this backbone effectively reduces the total number of model parameters. is trained from scratch using VONet's training loss.
Parallel attention network. Directly training the attention network formulated in Eq. could be challenging due to the very depth of the U-Net. each individual frame, this challenge stems from the intricate path that the gradient signal must traverse, starting from the VAE decoder, progressing through the slot encoder, then navigating the U-Net, and ultimately reaching the context vectors. Consequently, the generated masks m, may be trapped in trivial solution, where each pixel receives equal attention values from the K slots. To address this issue, we first convolve each context vector across the backbone feature locations, yielding estimated attention mask for each slot. This operation analogous to slot attention (Locatello et al., 2020; Kipf et al., 2021). Subsequently, the U-Net is tasked with generating only delta changes (in log space) atop the estimated mask. This effectively establishes a special skip connection between the U-Net's 's input and output layers. We found that this skip connection plays crucial role in learning meaningful attention masks.
Slot encoder. We adopt simple architecture for the slot encoder. The attention mask is first element-wise multiplied with the backbone feature map, with broadcasting in the channel dimension.

---

Published as a conference paper at ICLR 2024
MOVI-A
MOVI-B
MOVI-C
MOVI-D
MOVI-E
Figure 4: Example video frames of the MOVI datasets, A,B,C contain up to 10 objects while D,E contain up to 23 objects in each video.
Then, the per-frame slot latent Yt, is obtained by performing average pooling on the masked feature map across the spatial domain. While there may be other possible implementations of the slot encoder, we have found this straightforward approach to be highly effective and easy to train.
Computing the per-trajectory slot latent. We instantiate the RNN for calculating rr, (Eq. 4) as a GRU (Cho et al., 2014) followed by an MLP. Specifically, it is defined as
f/, GRU(Y, Ir,,,, Fr LayerNorm(r',, + (
LayerNorm (Ba et al., 2016) ensures that the latent values will not explode even for long video.
Computing the KLD. We model both the posterior and prior distributions as diagonal Gaussians. When minimizing the KLD loss in Eq. 8, two primary influences come into play: the posterior is regularized by the prior, and conversely, the prior is shaped towards the posterior. These dynamics jointly contribute to the learning of slot representations. Following Hafner et al. (2020), we use KL balancing coefficient > 0.5 to accelerate the learning of the prior relative to the posterior. This done to prevent the regularization of the posterior by a poorly trained prior. Mathematically,
KLD = k Dx-x Dxz ( + (1 - x)) [Ku((]]Stoprrad()))
(10)
EXPERIMENTS
Benchmarks. We assess VONet on five well-establishes public datasets MOVI-(A.B,CA. (Greff al., 2022), which include both synthetic and naturalistic videos. MOVI-A and MOVI-B consist of simple scenarios featuring nearly uniform backgrounds and objects with varying colors but minimal texture. MOVI-B exhibits greater complexity compared MOVI-A, owing to the inclusion of additional object shapes. MOVI-(C,D,E) stand out as the most challenging, featuring realistic, intricately textured everyday objects and backgrounds. While MOVI-(A,B,C) include up to 10 objects in given scene, MOVI-D and MOVI-E include up to 23 objects. Each video within the five datasets comprises 24 frames, lasting 2 seconds. We adhere to the official dataset splits, except that the validation split is employed as the test split, following Kipf al. (2021). Some example frames of the five datasets are shown in Figure
Metrics. We assess the quality of 3D slot segmentation masks generated from slot attention masks across the full length 24 video frames (details in Appendix A.2). Two common metrics for video object learning are used: FG-ARI (Greff et al., 2019) and mloU. FG-ARI serves similarity metric, measuring the degree to which predicted segmentation masks align with ground- manner. It only considers foreground pixels, where each
clustering
truth masks in permutation-innaaian cluster corresponds to foreground segmentation mask. To also evaluate background pixels, mloU calculates the mean Intersection-over- Union by first employing the Hungarian algorithm find the optimal bipartite matching (in terms of IoU) between predicted and groundtruth masks, and then dividing sum of the optimal IoUs by the number of groundtruth masks. Both FG-ARI and mloU demand temporal consistency and penalize object identity switches at any video frame. Nonetheless, neither of them is perfect. For comprehensive evaluation, a combined interpretation is necessary.
General training setup. VONet learns 11 slots on MOVI-(A,BCC and 16 slots on MOVI-(D,E). We use a mini-batch size of 32 for MOVI-(A,B,C) and 24 for MOVI-(D.E). Each sample a batch has a sequence length of 3. A replay buffer (detailed in Appendix A.2) is employed enable training from past slot states while preserving the i.i.d. assumption of training data. All video frames are resized to 128 128. In all subsequent experiments, the learning is entirely unsupervised. meaning that no additional supervision signals, such as depth, optical flow, or segmentation, are provided. For each experiment, run three random seeds report the mean and standard deviation of metric

---

Published as a conference paper at ICLR 2024
FG-ARI
mioU
Method
MOVI-A
MOVI-B
MOVI-C
MOVI-D
MOVI-E
MOVI-A
MOVI-B
MOVI-C
MOVI-D
MOVI-E
SAVT
45.121.
31.811.2
22.
29.4
36.023.
32.2117
31.214.3
15.310.7
15.6103
15.812.2
SIMONe
50.226.
36.5.0.5
19.5.0.1
34.8=6.3
41.3-003
37.621.
35.5+0.
20.2a0.1
22.720.1
20.8a0.3
SCALOR
68.1+14
15. 3.0.6
21.322
33.5 33.5=22
39.6=0.5
59.3+0..
16.6+0.1
14.3+0,8
1/1.211.1
12.1
VIMON
62.8+2.5
26.2+4.4
18.0=2.1
22.5=6.7
17.7=1.5
34.4+5.0
27.1+0.9
19.6+15
STEVE
47.818.2
29.6..03
38.1:0.3
42.9.2.
49.7=1.0
$3.315..1
42.74..
30.51:022
23.813.7
26 26.211.3
VONet
91.0115
60.6.0.
45.3.104
50.741.
50.310.
60.5_22
59.7.22
42.8.0.5
37.7.0.
36.4
Table 1: Results of VONet and the five baselines, in an unsupervised learning setting.
values. Additional architectural specifics and hyperparameter details are provided in Appendix A.3 for reference. It takes about 36 hours to train VONet on 4x 3090 GPUs on each of the five datasets, for total number of 150k gradient updates.
6.1 EFFICIENCY OF THE PARALLEL ATTENTION
Att inference time (sec)
0.07
MONet
We start with measuring how efficient our parallel attention is compared to the recurrent attention of MONet. With the same U-Net architec- ture (Appendix A.2). we report the average time of generating attention masks for varying number of slots on an image of size 128 x 128. The results are plotted Figure VONet demonstrates a strong advantage of being able to maintain nearly constant inference time regardless of the increasing number of slots. Conversely, MONet's 's time grows linearly with respect the slot number, which is expected as MONet generates one attention mask after another. The superior efficiency of VONet on images forms the basis for it being extended to video frame sequences.
0.04
0.01
0.00
5
11
16
24
Number of Slots
Figure 5: Comparison of the attention inference efficiencies.
6.2 COMPARISON WITH BASELINES
Baselines. We conduct a comparative analysis of VONet against five representative unsupervised video object learning baselines: SCALOR (Jiang et al.. 2019), ViMON (Weis al., 2021), SI- MONe (Kabra et al., 2021). SAVI (Kipf et al., 2021; Elsayed et al., 2022), and STEVE (Singh et al., 2022b). A crucial criterion for baseline selection is the accessibility of an official or public implementation. We employ official implementations of SCALOR, VIMON, SAVI, and STEVE, and re-implementation of SIMONe (Appendix A.4). SCALOR, VIMON, SIMONe, and STEVE were originally designed for unsupervised learning, with their reconstruction targets being RGB video frames. SAVI, on the other hand, relies on object bounding boxes initial video frames and reconstructs supervision signals like depth or optical flow. For fair comparison, we eliminated the bounding box conditioning and only allowed SAVI to reconstruct RGB frames. We adhered to best practices from the literature for configuring the hyperparameters of the five baselines, selecting values recommended either by official codebases or by hyperparameter search results.
Results. In Table 1, it is evident that VONet surpasses all the baselines in terms of both FG-ARI and mloU metrics across all five MOVI datasets. Furthermore, the low std values imply its stability, an important characteristic often lacking in unsupervised video object learning methods. It can also be seen that MOVI-(C.D,E) are indeed much more challenging than MOVI-(A,B), due to the rich textures and complex backgrounds contained in these datasets. SAVI, SCALOR, and VIMON face challenges when dealing with complex video scenes. Surprisingly. while STEVE performed admirably on MOVI-(C,D,E), its FG-ARI performance on MOVI-(A.B) was unexpectedly low. Upon closer examination of its visualized results, STEVE tends to over-segment foreground objects and swap object identities, when provided with redundant slots simpler scenarios (see examples in Figure 7). This behavior significantly diminishes its FG-ARI performance.
6.3 HOW CRITICAL ARE DIFFERENT COMPONENTS TO VONET?
We conduct an ablation study to quantify the contribution of various components to the effectiveness of VONet. We select several key ingredients for ablation, resulting in four variants of VONet.
a) wo- wo- UNer ablates the U-Net for attention mask generation by removing it, along with the mask transformer at its bottleneck layer, from the parallel attention pipeline. Instead, it directly outputs the estimated attention masks (Section 5). This variant aims to investigate whether the mask refinement provided by the U-Net is indispensable.
7

---

Published as a conference paper at ICLR 2024
MOVIA
MODIIC
MOVIA
MOVIE
FCARI misi DS2
wo-l
Figure 6: Ablation study results. Each error bar represents the std of three random seeds.
MOVI-A
MOVI-B
MOVI-C
MOVE-D
MOVI-E
(PR)) GF Video
(P3)
55
Figure 7: Example segmentation masks of STEVE (S) and VONet (V). Each column showcases an example video from one of the five datasets. Three key frames are presented in a row for each video. Each unique color represents the mask for specific slot. S/V (FG) shows foreground-only segmentation after background pixels being masked out.
b) KL- & reduces the importance of the KLD term the training loss, where 3 the KLD weight used by VONet in the experiments. We set W to four values: 20, 200, 20k, and co, where do corresponds to completely eliminating the KLD.
wo-Replay removes the replay buffer technique and trains from fresh states for each sampled short video segment, as commonly seen in prior works (Kipf et al., 2021; Singh et al., 2022b). d) Wo- excludes the KI. balancing technique and calculates a standard KLD loss directly.
All four variants were trained using three random seeds on the five MOVI datasets. For each variant, the remaining training configurations are exactly the same those used for VONet.
Results. Figure shows the impact on the performance if a specific component is removed or weakened in VONet. Notably, without KI. balancing. the outcome is very poor MOVi-A. In this case, VONet struggles to acquire any meaningful mask, as each pixel receives uniform attention from all slots. It is obvious that the posterior has been heavily forced into resembling the prior before the latter is adequately learned. Interestingly, the replay buffer is far more important on MOVI-(A.B) than on the other three. One plausible explanation that these two datasets can have objects with similar appearances in video, which makes object tracking more challenging. Training with replayed slot states enhances the long-term object tracking ability of VONet. As for the U-Net architecture, the mask refinement it offers proves to be particularly crucial for MOVI-(C.D.E), This aligns with our expectation that its inherent spatial locality bias is beneficial for handling complex videos. Lastly, we have observed that the complete removal of the KLD term (W oc) consistently results in unstable training and eventual crashes, and thus its results were not plotted. Apart from this, do observe performance declines the weight decreases, especially MOVI-(B,C,DDF In summary, the four components all prove to be essential. The removal of any of them results in catastrophic outcomes on at least one dataset.
6.4 VISUALIZATION AND ANALYSIS
Object masks. Figure 7 shows example object masks for VONet and STEVE. While VONet occa- sionally splits the background, it excels at preserving the structural integrity of moving foreground objects. On the contrary, STEVE tends to over-segment foreground objects, and its object masks exhibit temporal shifts over time. VONet generates smoother and more compact foreground object masks, whereas STEVE produces jagged ones with irregular shapes. We believe that this difference
8

---

Published as a conference paper at ICLR 2024
Ispui
Rr:
Prun Roc
Impui
Posamim Roc
Prin Re:
StarD
0.4
0.2
10:3
Time 590p
R.D
03
02
10 18.19 20 22
Time stop
Figure 8: Left: Reconstruction results from the posterior and prior. Right: KLD curves of slot 0 on two example videos. The KLD value has been divided by the slot embedding dimension (128).
arises partially from the strong inductive bias of spatial locality of VONet's U-Net architecture for attention masks. This bias is absent in slot-attention methods like STEVE.
VAE prior. One can evaluate the quality of the VAE prior modeling in VONet by reconstructing slots. We randomly sample multiple video frames and utilize the predicted priors for their reconstruction. The results are shown in Figure 8 (left). We can see that the prior is effectively modeled, since the decoded images generated from the prior closely resembles reconstructed image derived from the posterior slots.
input frames using
KLD visualization. One can also inspect how the per-slot KLD loss in Eq. 8 varies from frame frame. We anticipate that when a slot exhibits greater temporal variation at specific frames, the corresponding KLD losses will also rise. Figure (right) illustrates two example videos in which the KLD curves are plotted for slot O. It is evident that the KLD effectively measures the degree of variation in slot content. When the content undergoes significant changes, the KLD loss rises, whereas when the object represented by the slot remains stable, the KLD loss stays at a low level.
Failure modes. The first failure mode is over-segmentation (Zimmermann et al., 2023), happening when the video scene's object count is much lower than the pre-allocated slot number. Without extra constraints priors, VONet aims to use all available slots to encode the video scene for reduced decoder loss. This causes some objects attended to by multiple slots or the background fragmented into parts (Figure 7, MOVI-A). To address over-segmentation, the model needs to deactivate "re- dundant" slots. Extra losses may be needed to penalize the use of too many slots. The second failure mode is incomplete object understanding due to the absence of objectness priors. Although we use temporal information for object discovery in videos, the overall learning system remains under-constrained. When object exhibits multiple texture regions, the model faces challenges in discerning whether it represents a single entity with visually distinct components in motion or multi- ple distinct objects moving closely together (Figure 7, MOVI-D), Integrating pretrained knowledge about the appearances of everyday objects could help (Zadaianchuk et al., 2023). Lastly, the en- forcement of slot temporal consistency may occasionally prove unsuccessful. In certain instances, even when a slot appears to lose temporal consistency, upon closer examination, its KLD loss re- mains low (Figure 7, the dark-green background slot in MOVI-B), simply because the VAE prior is accurately predicted. This suggests opportunity for improving our KLD loss. The temporal consistency might also benefit from using long-term memory model (Cheng Schwing, 2022) as opposed to the current short-term GRU memory which might get expired under long-time bong-time occlusion.
7 CONCLUSION
We have presented VONet, sute-of-the-ar approach for unsupervised video object learning. VONet incorporates a parallel attention process that simultaneously generates attention masks for all slots from U-Net. The strong inductive bias of spatial locality in the U-Net leads to smoother and more compact object segmentation masks, compared to those derived from slot attention. Fur- thermore, VONet effectively tackles the challenge of temporal consistency in video object learning by propagating context vectors across time and adopting an object-wing sequential VAE framework. Across five datasets comprising videos of diverse complexities, VONet consistently demonstrates superior effectiveness compared to several strong baselines in generating high-quality object repre- sentations. We hope that our findings will offer valuable insights for future research in this field.

---

Published as a conference paper at ICLR 2024
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 06450, 2016.
Zhipeng Bao, Pavel Tokmakov, Allan Jabri, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert. Discovering objects that can move. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11789-11798, 2022.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint arXiv:1901.1133 2019.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference machine learning, pp- 1691- 1703. PMLR, 2020.
Ho Kei Cheng and Alexander G. Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In ECCV, 2022.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcchre. Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1406. 1078, 2014.
Gioele Ciaparrone, Francisco Luque Sánchez, Siham Tabik, Luigi Troiano, Roberto Tagliaferri, and Francisco Herrera. Deep learning in video multi-object tracking: A survey. Neurocomputingg, 381:61-88, 2020.
Antonia Creswell, Rishabh Kabra, Chris Burgess, and Murray Shanahan. Unsupervised object- based transition models for 3d partially observable environments. Advances in Neural Information Processing Systems, 34:27344
2021.
Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. Advances in Neural Information Processing Systems, 35:28940-28954 2022.
Patrick Emami, Pan He, Sanjay Ranka, and Anand Rangarajan. Efficient iterative amortized in- ference for learning symmetric and disentangled multi-object representations. In International Conference on Machine Learning, pp. 1977-1981. PMLR, 2021.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement, Advances in Neural Information Processing Systems, 34:8085-8094, 2021.
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in neural information processing systems, 29, 2016.
Rohit Girdhar and Deva Ramanan. Cater: A diagnostic dataset for compositional actions and tem- poral reasoning. arXiv preprint arXiv: 2019.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pp. 1424-2433 PMLR, 2019.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem artificial neural networks. arXiv preprini arXiv:20/2.0521 2.000088 2020.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji. Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sizzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang.
10

---

Published as a conference paper at ICLR 2024
Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong. and Andrea Tagliasacchi. Kubric: scalable dataset generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR),
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1eXiv:1912.01600 2019.
Danijar Hafner, Timothy Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02193 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Zhen He, Jian Li, Daxue Liu, Hangen He, and David Barber. Tracking by animation: Unsuper- vised learning of multi-object attentive trackers. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1318-1327, 2019.
Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisser- man, João Carreira, and Relja Arandjelović. Object discovery and representation networks. In European Conference on Computer Vision, pp. 123-143. Springer, 2022.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. arXiv preprint arXiv: 2019.
Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loie Matthey, Antonia Creswell, Matt Botvinick, Alexander Lerchner, and Chris Burgess. Simone: View-invariant, temporally-abstracteed object representations via unsupervised video decomposition. Advances Neural Information Process-
ing Systems, 44-20146-20159) 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv: /4/2. 6980, 2014.
arXiv preprint
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1312v 14, 2013.
Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. arXiv preprini urXiv:2111.12555 2021.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systems, 31, 2018.
Jannik Kossen, Karl Stelzner, Marcel Hussing, Claas Voelcker, and Kristian Kersting. Struc- tured object-aware physics prediction for video modeling and planning. arXiv preprint (5t:::990002225..0 2019.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in Neural Information Processing Systems, 13:11525-11538. 2020.
Trang Nguyen, Amin Mansouri, Kanika Madan, Khuong Duy Nguyen, Kartik Ahuja, Dianbo Liu, and Yoshua Bengio. Reusable slotwise mechanisms. Advances in Neural Information Processing Systems, 36, 2024.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assisted Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part III 18, pp. 234-241. Springer, 2015.
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann FaboneGa Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al. Bridging the gap real-world object-centric learning. arXiv preprint arXiv:2209. 228, 2022.
11

---

Published as a conference paper at ICLR 2024
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns compose. In ICLR, 2022a.
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for com- plex and naturalistic videos. Advances in Neural Information Processing Systems, 35:18181- 18196, 2022b.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint 107.v0267.,8022,, 2016.
Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. In Conference Robot Learning, pp. 1439-1456. PMLR, 2020.
Marissa Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger, and Alexander S Ecker. Benchmarking unsupervised object representations for video sequences. The Journal of Machine Learning Research, 22(1):8253-83) 8313, 2021.
Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Object-centric learning for real-world videos by predicting temporal feature similarities. arXiv preprint arXiv:2306.04822 2023.
Tianfei Zhou, Fatih Porikli, David Crandall, Luc Van Gool, and Wenguan Wang. A survey deep learning technique for video segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):7099--122) 2022.
Roland S Zimmermann, Sjoerd van Steenkiste, Mehdi SM Sajjadi, Thomas Kipf, and Klaus Gr- eff. Sensitivity of slot-based object-centric models to their number of slots. arXiv preprint arXiv:2305. 2023.
Daniel Zoran, Rishabh Kabra, Alexander Lerchner, and Danilo J Rezende. Parts: Unsupervised seg- mentation with slots, attention and independence maximization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10439-10447, 2021.
12

---

Published as a conference paper at ICLR 2024
Image FG-ARI
Image mloU MOVI-D
Method
MOVI-C
MOVI-D
MOVI-E
MOVI-C
MOVI-E
MONet
21.8+0.1
22.0+177
18.8
3
17.4=20
13.6216
VONet
47.012.8
51.0a1.2
50.323.6
36.2a3.0
28.5=006
18.1
Table 2: Results of VONet and MONet in the single- frame scenario.
Transformer
Fo,
Co.k
mikk
Y1.k
Time 5t000
6 N(O,
SpEnn
RNN
CLA
Y2.k
Skotm
11,k
Figure 9: Illustration of the unrolling process for VONet on video sequence. Only slot k is depicted in the diagram, with the other slots following similar flows. Slot computations can be parallelized within each step, except when inter-slot interaction required (e.g. initial slot transformer and parallel attention). For simplicity, we omit depicting the dependencies on the input frames X,.
A APPENDIX
A.1 SINGLE-FRAM SCENARIO
We compared VONet with MONet in the single-frame frame- scenario, as MONet was designed for object discovery in individual images only. We configured the U-Net architecture of MONet to match that of VONet, and set MONet's 3 1 and = 2 after a brief exploration within reasonable parameter range. VONet was modified to train with sequence length of and excluded training from past slot states. For the training and evaluation data, we utilized video frames from MOVI- (C,D.E), treating them as independent images with no temporal relationship. FG-ARI and mloU are calculated on individual frames, referred to as Image FG-ARI and Image mloU, The results are shown in Table 2. They suggest that VONet is not only more efficient than MONet, but also more competent to discover objects from complex images.
A.2 MORE DETAILS OF VONET
Backbone for image features. We train a backbone based small ResNet (He et al., 2016) to extract a feature map from each input frame X,, The backbone architecture includes five residual blocks, each employing a 3 kernel and a stride of 1, except for the first block which has stride of 2. All the blocks have an output channel number of 64. Following the ResNet, there is a concluding convolution layer with a
kernel and linear activation, with output channel number of 128. Finally, position embedding network (Kipf et al., 2021) is applied to provide 2D location information. With this backbone, the output feature map has a spatial shape that half of that of the input frame. This feature map is shared by the parallel attention network and the slot encoder as the input.
U-Net and mask transformer. The U-Net used by the parallel attention network follows a standard architecture closely resembling that MONet (Burgess al., 2019). There are blocks on either the upsampling or downsampling path, with M taking the value of 5 for datasets MOVI- MOVI-(A,B (A.B) and for datasets MOVI-(C.D,E). Each block comprises the following layers in order: (a) a 3 x bias- free convolution with stride of I and padding of 1, (b) instance normalization (Ulyanov al., 2016) layer with learnable affine parameters, (c) a ReLU activation, and finally (d) a max pooling layer of size 2 for the downsampling path and a nearest upsampling layer of factor 2 for the upsam- pling path. No max pooling (upsampling) is applied to the first (last) block of the downsampling (upsampling) path. A skip connection is established between the output of the m-th downsampling
13

---

Published as a conference paper at ICLR 2024
block and the output of the (M - m 1)-th upsampling block, for each 2 S S M. The non-skip bottleneck connection is an MLP with two layers of sizes (512, 512), both activated with ReLU. The output of this MLP will be projected and reshaped to match the output of the downsampling path, concatenated with it, and input to the upsampling path. After the upsampling path, a 1 x convolu- tion is applied with an output channels of 1. The final output is a 2D map that retains the same spatial dimensions as the input. The values within this map represent unnormalized attention logits. For the downsampling blocks, we set their output channels {32, 64, 64, 128, 128) for MOVI-(A.B), with an additional output channel number of 128 for MOVI-[C.D.E].
To prepare the input the U-Net for slot's attention mask, we first convolve its context vector across the backbone feature locations, yielding a roughly estimated attention mask for that slot. Then its context vector spatially broadcast to the feature locations, forming a vecation-invariant context map. Finally, the backbone feature map, the estimated attention mask, and the context map, are concatenated in the channel dimension to form the input to the U-Net.
When there are K slots whose attention masks are being computed in parallel, right after the U-Net bottleneck MLP, we use a mask transformer to model the interaction among the K masks in the latent space. The transformer consists of three decoder-only transformer blocks, and each block can be briefly described as:
u' v MLP(LayerNorm(v)) V Mulliieeadttm(LaayeNoomt
where represents the K mask latent vectors and u' are the updated latent vectors. We configure each block with three attention heads. To maintain permutation invariance for the slots, no position encoding is incorporated in any of the transformer blocks.
Initial slot latent. The initial slot latent transformer in Eq. 5 shares a similar architecture with the U-Net mask transformer, differing only in terms of layer sizes. It also has three decoder-only transformer blocks and three attention heads per block. We set both the per-trajectory slot latent size and the per-frame slot latent size be 128.
VAE prior and posterior. To compute the prior, again we use a transformer (Eq. 7) that has a similar architecture with the U-Net mask transformer, except for different layer sizes. The trans- former has only two blocks with three attention heads in each block. Its output r', is independently projected (via an MLP of one hidden layer of size 128) to generate the mean and log variance of Gaussian the prior distribution. The posterior is obtained by projecting the updated per-trajectory slot latent y to generate its mean and log variance with a similar MLP.
Decoder. Singh al. (2022a) identified the mixture of components decoder in MONet as a weak decoder, pinpointing two primary limitations: the slot-decoding dilemma and pixel independence. These limitations pose challenges the encoder's capacity achieve effective scene decomposi- tion. As a remedy, they proposed using a more powerful transformer-based decoder (Chen et al., 2020) instead. This decoder attends all slots simultaneously and decodes the image in an auto- regressive manner. At a high level, the decoder is formulated as:
PrransDee (x (x|z1, ...,... zR) II P(x(m)) (1. ZK, x(1, x(m (1)- 1)).
2x,
(11)
Here, x(m) represents the m-th patch on the image in the row-major order. Singh et al. (2022b) applied this decoder to learning objects from complex videos and achieved a notable performance enhancement.
We directly reused the transformer decoder implementation from the official code² of STEVE (Singh et al., 2022b). For the discrete VAE model, we re-implemented our own, but closely following the official implementation. All the decoder-related hyperparameters used in our experiments kept the same with those of STEVE.
Segmentation mask. For FG-ARI and mloU evaluation, an integer segmentation mask is generated by consolidating the K slot attention masks m,, (Eq. 3), where each mask contains floating- -oint value within the range of [0, 1] at each pixel location. This value represents the probability of the pixel being assigned to the corresponding slot. If the maximum probability at the pixel is smaller than 0.3 (no slot is confident enough), we assign that pixel to special "null" slot. Otherwise, the
Phtpps:/ / thub. com/ wwg.ggggggaam/mmmmmmm steve
14

---

Published as a conference paper at ICLR 2024
pixel is assigned to the slot with the maximum probability. As result, each segmentation mask contains at most K + 1 distinct integer values, where K is the number of pre-allocated slots.
Training from replayed video segments. Training VONet entire videos is impractical due to high memory usage caused by the large computational graph unrolled over time. Traditional video object learning methods, like Jiang et al. (2019); Kipf et al. (2021); Singh et al. (2022b), train models on short video segments, assuming that each mini-batch is initialized with fresh slots. During testing, they expect models generalize to longer videos. VONet also trained on short video segments, but it stores past states in a replay buffer, where each state includes per-trajectory slot latents {ro k= and a boolean flag indicating the initial frame of video. When sampling a video segment from the replay buffer, we initiate the training of VONet with the state of the first step of that segment, and only reset the state if the flag true. Despite potential state obsolescence, this replay-based training approach proves effective if small replay buffer length is used. Using a replay buffer allows training short video segments from past slot states, while still preserving the i..d. assumption with a random replay strategy. Without it, online training from previous slot states requires sequential training through entire videos, introducing temporal correlation in the training data.
During training, we create a replay buffer with a length of 10k frames and a width of 16 vidcos. For each gradient update, first unroll the same copy of VONet each of 16 videos for 2 steps, store the unrolled states in the replay buffer, and randomly extract a mini-batch of size B with a length of 3 (in total 3B frames) from the buffer to compute the gradient. After the update, the unrolling continues until the videos are finished (after 24 = 12 updates), when we randomly select another 16 videos from the dataset to replace them. We set B 32 for MOVI-(A.B.C) while B 24 for MOVI-(D.E).
A.3 HYPERPARAMETERS
We provide a brief description of the key hyperparameters used for VONet in our experiments. For the complete set of hyperparameters, we encourage the reader to refer to our code
The input video frame x, is resized 128 128. Unlike some prior methods, we did not perform any data augmentation on the frames. We set the lengths of all the following vectors to 128:
i) Noise vector 6g (Eq. 5);
ii) Per-frame slot latent y (Eq. 4);
iii Per-trajectory slot latent rt. (Eq. 4);
iv) Context vector Ct (Eq. 4);
v) VAE posterior embedding 2t,k (Eq. 6).
(We also explored smaller length of 64 for these vectors, but obtained slightly worse results.) For any transformer, its model size (d_ @__model), key size (d_k), and value size (d_v) are all set to be equal the query size, whereas its hidden size (d_ff) is twice the query size.
We rolled out 16 workers in parallel for collecting video data in the replay buffer. Each worker utilizes the most recent VONet model for inference on 2 consecutive video frames, stores the inputs and states in the replay buffer, pauses for the trainer to perform a gradient update, and then resumes data collection for another frames, following this iterative process. The replay buffer length was set to 10k, resulting in maximum number of time steps 16 x 10k = 160k in the buffer.
We trained 11 slots for MOVI-(A.B.C) while 16 slots for MOVI- (D.)). reflecting the increased maximum number of objects in the latter two datasets. Accordingly, to ensure consistent GPU memory consumption across all training runs, we employed mini-batch size* of 32 for MOVI- {A,B,C} and 24 for MOVI-(D,E). We sampled video segments of length 3 for training on all datasets. This training configuration ensures that any training job can be executed on system equipped with 4 NVIDIA GeForce RTX 3090 GPUs or an equivalent amount of GPU memory.
For the optimization, we set the KI. balaneing coefficient x (Fq. 10) to 0.7. The KI. loss weight / (Eq. 8) was linearly increased from 0 to D
20 in the first 50k updates, where D 128 is the posterior
'https: / // thub. com / hnyu/ / vonet In our paper, when we refer to a sample within mini-hatch we referring video segment. Therefore, when we specify mini-batch size of N, it indicates that the batch consists of N video segments.
15

---

Published as a conference paper at ICLR 2024
dimension. We used the Adam optimizer (Kingma & Ba. 2014) with a learning rate schedule as follows. The learning rate was increased linearly from 10-5 to 10-4 in the first 5k training updates, maintained the value until 100k updates, and finally was decreased linearly back to 10 in another 50k updates. Thus in total, we trained each job for 150k gradient updates. In each update step, we normalized the global parameter norm to 0.1 if the norm exceeds this threshold.
A.4 BASELINE DETAILS
All baseline methods except SIMONe (reason explained below), were trained with 11 slots for MOVI-(A,B.C) and 16 slots for MOVI-(C.D). All baseline methods, with the exception of SCALOR, employed the same methodology for obtaining segmentation masks was utilized in VONet. SCALOR, on the other hand, possesses its own robust method for deriving segmentation masks from attention maps, and thus we did not replace it. Below are their detailed training settings.
SCALOR. We used the official implementation of SCALOR at https: : pttpps///qiii /github.. com/ JindongJianee /SCALRR. We followed the suggestions in SAVI (Kipf et al., 2021) to config- ure its hyperparameters, with several exceptions as follows. The training video segment length was reduced from 10 6 to reduce memory consumption. We used an 8 8 grid instead of the 4 x 4 grid in SAVI as we found the former produced much better results on HO.I-(A.B). We used a learning rate of 10-4 speed up the training convergence.
VIMON. We used the official implementation of VIMON at https: / /tggt/ggitt////// thub. com/ ecker- lab /
cencct--eentrre cintrric-- represent www.eeettttiionbbnnemmmaeaae benchmark. We made adjustments to its default hyperparameters to adapt its training to the MOVI datasets. by setting the VAE latent dim to 64 and the training video seq length 6. We also explored different values for y, but found the default value 2 is already good enough. All other hyperparameters remain unchanged.
SAVI. We made several changes to the official implementation of SAVI at https::/gátth com/ homm/goglee..o resereeeerccceeeeeeeeeeeeeeeceee slot-at tent ion-vided for our experiments. First, SAVI was trained in an entirely unsupervised manner to reconstruct RGB video frames only, without the object bounding box conditioning in the first video frame. The slots were initialized as unit Gaussian noises as in VONet. Second, to mitigate GPU memory usage, we halved the original training batch size, reducing from 64 to 32. We also reduced the training video segment length from 6 to 3 MOVI- (D.E). Remarkably, these size adjustments did not yield any discernible impact on the training results according to our observations on several trial training jobs. Finally, conducted training with a medium-sized SAVI model, featuring a 9-layer ResNet as the encoder backbone which has comparable size to the backbone of VONet. Specifically, the ResNet architecture was created by assigning a size (number of residual blocks) of I to each of the four ResNet stages, employing the class ResNetWithBasicc..
implemented by SAVI.
SIMONe. We used the re-implementation of SIMONe at https: / /tggtttlllba////.////////...ball com/ imerrall-intteligeee / iimone, which successfully reproduced the performance on CATER (Girdhar& & Ramanan, 2019). In contrast to other bascline methods and VONet, SIMONe imposes a strict requirement on the number of learnable slots. This number must be equal to the product of the height and width of the feature map resulting from its CNN encoder and transformer. Consequently, we employed a fixed slot number of 16 across all MOVI datasets for SIMONe. We set the mini-batch size to 40 and the video segment length to 24 (full length). All other hyperparameters remained unaltered.
STEVE. Since STEVE has been extensively evaluated MOVI-(D.E) by its authors, we largely retained its original hyperparameters. However, we explored one exception: experimenting with two different values for its slot size: 64 and 128. Our observation revealed that larger slot size of 128 consistently yielded no better or even sometimes slightly inferior results across all five datasets. Consequently, settled on a slot size of 64 for STEVE in our final configuration.
A.5 DATASET DETAILS
We refer the reader to the official MOVI datasets website for details: https: / com/google-ree www.llerreeecchheeeeeeee wwarehh//kbiiie t/tttre//me ceee/mman//cii www.a///aaaaannnn/eaaaaea /novi. We did not make any change to the five datasets. The official training/validation splits were used in our experiments.
16

---

Published as a conference paper at ICLR 2024
A.6 MORE VISUALIZATION RESULTS
Figure 10: Additional segmentation results of VONet. In these results, each video is presented with every other frame displayed. The boundaries of object segments marked with white curves. No post-processing was performed on the masks.
17

---

Published as a conference paper at ICLR 2024
25
Figure 11: Additional segmentation results of VONet. In these results, each video is presented with every other frame displayed. The boundaries of object segments are marked with white curves. No post-processing was performed on the masks.
18

---

Published as a conference paper at ICLR 2024
Figure 12: Additional segmentation results of VONet. In these results, each video is presented with every other frame displayed. The boundaries of object segments are marked with white curves. No post-processing was performed on the masks.
19


Reviews:
[{"review_id": "8AA5UFQJkT", "reviewer": "Reviewer_v4jo", "summary": "This paper presents VONet, an innovative approach to unsupervised video object learning inspired by Monet. Specifically, Monet implements an efficient and effective parallel attention inference process that simultaneously generates attention masks from U-Net for all slots. In addition, the temporal consistency necessary to track objects across video frames is achieved by integrating an object-wise sequential VAE framework. Experiments demonstrate that the approach achieves competitive performance on several challenging object-centric video prediction benchmarks.", "questions": "For writng, the introduction section is short. It is hard to understand the main idea of the whole work.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.The parallel attention inference process proposed in this paper greatly improves the slot generation efficiency of MoNet and creates conditions for its further application.\n\n2.This paper proposes the KLD loss, which utilizes the principle that \"only slot representations that exhibit temporal consistency can exhibit predictability\" to cleverly achieve temporal consistency in unsupervised video object learning.\n\n3.The paper is relatively easy to follow, with good mathematical formulations and diagrams.", "weaknesses": "1.VoNet is a work based on MoNet and is highly similar to ViMON, a comparison of results with MoNet and ViMON should be reported in the experimental phase to demonstrate the advantages of VoNet.\n\n2.Can the method in this paper correctly handle objects appearing in the middle of the video or reappearing after being occluded and maintain temporal consistency? The MOVI dataset does not seem to be able to model this situation, consider adding experiments in natural scenes.\n\n3.Existing video object-centric learning methods based on slot attention are developing rapidly, e.g.DINOSAUR(https://arxiv.org/abs/2209.14860), VideoSAUR(https://arxiv.org/pdf/2306.04829.pdf), and perform well on natural scene datasets such as YouTube-VIS, please explain the advantages of the method in this paper compared to methods based on slot attention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KpBK9zt4aP", "reviewer": "Reviewer_9HJm", "summary": "VONet introduces two key innovations: 1) a parallel attention network that employs the same U-Net architecture simultaneously on K context inputs, and 2) an object-wise sequential VAE framework, aimed at improving the temporal consistency in unsupervised video object learning. Notably, VONet significantly outperforms the baseline by a substantial margin.", "questions": "Please see Weaknesses section.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Within this paper, the \"object-wise sequential VAE\" is introduced, which is a novel and highly effective representation for exploring temporal dependencies in video frames. \n2. The experimental results in this paper are impressive, surpassing previous methods by a large margin.", "weaknesses": "1. In order to show the advantages of parallel processing, it would be beneficial that a comprehensive latency/accuracy comparison with MONet could be provided in the single-frame scenario.\n\n2. It would be beneficial to have a comparative analysis between the object-wise sequential VAE and other temporal dependency networks, particularly the memory network mentioned in reference [1]. These methods excel in modeling temporal information and such a comparison would greatly enhance the paper's overall comprehensiveness.\n\n[1] XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model ECCV 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fgxbFsdXwn", "reviewer": "Reviewer_x2B3", "summary": "The paper proposes a framework for unsupervised video object learning, VONet. The uniqueness of VONet is primarily the parallel attention process that is capable of generating attention masks for all slots with the consideration of temporal consistency which utilizes context propagation across time and object-wise sequential VAE framework. Their results on 5 MOVI datasets show that the proposed method significantly outperforms previous methods as measured by two popular metrics namely FG-ARI and mIoU.", "questions": "Can the authors provide any insights on how to potentially combine multiple slots to prevent such overfitting?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method addresses unsupervised video object learning which can be paralleled and temporally consistent given the slot numbers. Both temporal consistency and parallel segmentation instead of a sequential learning are crucial for learning objects in a video. This paper solves these very important first steps.", "weaknesses": "As already mentioned in the paper when the predefined slot numbers is larger than the actual number of objects, an unwanted side-effect if over-segmentation. Can the authors provide any insights on how to potentially combine multiple slots to prevent such overfitting?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FsgRKp3nFU", "reviewer": "Reviewer_751B", "summary": "This paper proposes an unsupervised video object representation learning framework, namely VONet. Taking an image-based method as per-frame baseline, the proposed VONet builds temporal attention to learn correspondence in high-level space, resulting in significant improvement against previous video-based methods.", "questions": "See weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The results look good with large improvement against previous methods.", "weaknesses": "1. The current title is too broad, which makes the readers hard to understand the specific contributions of this paper.\n\n2. The motivation of each contribution of the proposed method is not clearly clarified in Introduction, especially the difference or new insights w.r.t previous  methods.\n\n3. It is hard to refer to the expression of the symbols in Figure 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)