Multi-Scale Fusion for Object Representation
Authors: Rongzhen Zhao
Vivienne Huiling Wang
Juho Kannala
Joni Pajarinen
Venue: ICLR 2025 Poster (2025)
Keywords: Object-Centric Learning (OCL)
Variational Autoencoder (VAE)
Multi-Scale
Unsupervised Object Segmentation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: By processing the same image in different sizes, we obtain representations in multiple scales. And high quality object super-pixels in one scale can augment the low quaity super-pixels of the same object in other scales.

Abstract: Representing images or videos as object-level feature vectors, rather than pixel-level feature maps, facilitates advanced visual tasks.
Object-Centric Learning (OCL) primarily achieves this by reconstructing the input under the guidance of Variational Autoencoder (VAE) intermediate representation to drive so-called slots to aggregate as much object information as possible.
However, existing VAE guidance does not explicitly address that objects can vary in pixel sizes while models typically excel at specific pattern scales.
We propose Multi-Scale Fusion (MSF) to enhance VAE guidance for OCL training. 
To ensure objects of all sizes fall within VAE's comfort zone, we adopt the image pyramid, which produces intermediate representations at multiple scales;
To foster scale-invariance/variance in object super-pixels, we devise inter/intra-scale fusion, which augments low-quality object super-pixels of one scale with corresponding high-quality super-pixels from another scale.
On standard OCL benchmarks, our technique improves mainstream methods, including state-of-the-art diffusion-based ones.
The source code is available on https://github.com/Genera1Z/MultiScaleFusion.

Published as a conference paper at ICLR 2025
MULTI-SCALE FUSION FOR OBJECT REPRESENTATION
Rongzhen Zhao', Vivienne Wang', Juho Kannala Joni Pajarinen'
"Department of Electrical Engineering and Automation, Aalto University, Finland
DDeparmeett of Computer Science, Aalto University, Finland
*Center for Machine Vision and Signal Analysis, University of Oulu, Finland
Frongzhen.zhag vivienne.wang kannala, joni.pajjaineenJeaalee
ABSTRACT
Representing images or videos as object-level feature vectors, rather than pixel- level feature maps, facilitates advanced visual tasks. Object-Centric Learning (OCL) primarily achieves this by reconstructing the input under the guidance of Variational Autoencoder (VAE) intermediate representation drive so-called slots to aggregate as much object information as possible. However, existing VAE guidance does not explicitly address that objects can vary in pixel sizes while models typically excel at specific pattern scales. We propose Multi-Scale Fusion (MSF) to enhance VAE guidance for OCL training. To ensure objects of all sizes fall within VAE's comfort zone, we adopt the image pyramid, which produces in- termediate representations at multiple scales; foster scale-invarianceevvarincc in object super-pixess, we devise inter/intra-scale fusion, which augments low- quality object super super-pixels one scale with corresponding high high-quality super- pixels from another scale. On standard OCL benchmarks, technique improves mainstream methods, including state-of-the-art diffusion-based ones. The source code is available on https://giihub..co//Gneenna Mow.eeeeeeeee//Mllssscccclliii.a
INTRODUCTION
According to vision cognition research, humans perceive a visual scene objects and relations among them for higher-level cognition such as understanding, reasoning, prediction, planning and decision-making (Bar 2004 Cavanagh [011] Palmeri & Gauthier 2004). In computer vision, Object-Centric Learning (OCL) is promising way to achieving similar effeets Images or video frames represented sparse object-level feature vectors, known slots, rather than dense (siper-)pixel-leeel feature maps under weak or self-supervision (Greff et al., 2019) Burgess et al.) 2019) Meanwhile, segmentation masks of objects and the background are generated as byproducts, intuitively reflecting the corresponding object representation quality of these slots.
Object-level representation learning is highly influenced by textures, Early OCL methods, such as the mixture-based (Locatello et al. 20200 Kipf et al.) 20222] Elsayed et al. 2022), which directly reconstruct the input pixels as supervision, often struggle with objects that have complex textures. To address this issue, mainstream OCL methods, such as transformer-baa
(Singh et al.] 2022a c) foundation- mundatioon-based (Seitzer et al. 2023] Zadaianchuk et al. 2024) and diffusion-basee (Jiang et al., 2023 Wu et al. 2023b), leverage well-learnt intermediate representation for guided reconstruction, so as to drive slots to aggregate as much object-level information as possible.
Objects appear at different scales in images or videos due to imaging distances and actual sizes. Encoders/decoders usually perform well on certain pattern scales, necessitating "divide-and-rule" Zou et al. 2023 Minace et al. 2021). Mainstream OCL, however, overlooks such issue.
Suppose three objects of different sizes the input image or video frame. As shown in Fig. uup per left, current single-scale VAE encoding/decoding are likely only good at some specific scale of patterns, and only represent object in high quality or confidence, leaving the other two being rep- resented in low quality. Thus such single-scale VAE representation could not provide good guidance to OCL training on objects 2 and 3. But, as shown in Fig. [ lower left and right, if resize the input into three scales, ensure all the objects fall within the comfort-zone of the VAE encoder/decoder.

---

Published as a conference paper at ICLR 2025
Frimsry
VAE Excado
Intestact
suggoarted
Figure I: Multi-scale fusion is necessary in VAE guidance OCL training. Upper right: simplified architecture of OCL with VAE guidance. Upper left: single-scale VAE encoding/decoding only represent patterns of some certain scale in high quality/confidence. Lower left and right: by resizing the input, patterns of different scales all fall within the VAE encoder/decoder's comfort-zone; low- quality super-pixes pixels in one scale can be augmented by the high-quality in other scales.
Moreover, as shown in Fig. [lower left center, for scale the low-quality super-pixels of object and 3 can be augmented by their corresponding representations in scale 2 and 3, respectively.
We propose Multi-Scale Fusion (MSF) upon VAE representation, to guide OCL of both the transformer-bases classics and diffusion-based state-of-the-arts (i) We utilize technique image pyra- mid by resizing the input into different scales, to ensure objects of various sizes all fall within VAE encoder/decoderr' 's confort zones. (ii) We devise inter-/intra-scat quantized fusion by quantizing VAE representation of different scales with shared/specified codebooks, to foster cclle-inariance/- variance in object representation. (iii) Our technique augments VAE discrete representation with better object separability and, in turn, provides better guidance for OCL training.
2 RELATED WORK
VAE & OCL. Pretrained VAE (Im Im et al. 2017 Van Den Oord et al. (017), like dVAE or VQ-VAE. VAE,
can provide representations that highlight object separability in the input by suppressing texture redundancies for OCL. With such guidance, transformer- based OCL methods (Singh et al. 2022a c
like SLATE and STEVE, decode slots, which extracted by SlotAttention (Locatello et al.) 2020)
from input images or video frames, into the input via a transformer decoder. Similarly, diffusion- based methods (Jiang et al. 2023 Wu al., 20235), like SlotDiffusion, decode slots via diffusion model. Slots are driven to aggregate as much object information for good object representation. We improve these two lines of work. Foundation-basee methods (Seitzer et al. 2023 Zadaianchuk et al. 2024), like DINOSAUR, are also included for complete comparison.
Channel Grouping. Widely adopted in CNNs (Zhang et al. 2018) and ViTs (Gu al. 2022
grouping features along the channel dimension helps learn expressive representations. Grouping can performed on features (Krizhevsky et al., 2012 Huang al.. 2018 Chen et al. 2019). ) or
on weights (Zhao et al., 2021 2022). SysBinder (Singh et al., 2022b) explores this in OCL by grouping slots queries to aggregate different object information, for emerging attributes under extra supervision. The most recent GDR (Zhao et al. (202)) groups the VAE codebook from features into combinatorial attributes, achieving better generalization and convergence in oberessen Our work also groups it but for inlleiinvrrinnce and -variance.
Multi-Scale, Objects exist at different scales due actual sizes and imaging distances. CNN (Zhang et al. 2018) and ViT (Gu et al. 2022 typically have their comfort zones the scale distribution of objects in training data is always nonuniform. In computer vision, like detection (Zou et al. 2023) and segmentation (Minaee al. 2021), the multi-scale issue is significant, leading to the
2

---

Published as a conference paper at ICLR 2025
wide adoption of "divide-and-rule" The image pyramid (Adelson et al., 1984) produces multi-scale
representations by resizing; Feature pyramid network (Lin et al.) 2017) and variants mix features of different layers by channel-concat and element-wisee wise-sum. We the first to realize multi-scale fusion on VAE, augmenting every scale with the high-quality representation in other scales.
3 PROPOSED METHOD
We propose Multi-Scal Fusion (MSF), to explicitly address the multi-scale issue in VAE interme- diate representation, and thereby to guide Object-Centrie Learning (OCL) better. Our technique is applicable to both transformer-based OCL (Singh et al. 2022a c) classics and diffusion-based OCL (Wu et al. 2023b) ttate-of-the-arts, which are busis methods. Note that two types of methods use dVAE Im Im et al., 2017) and VQ-VAE (Vnn Den Oord et al. 2017) respectively, but we unify them with VQ- VAE like that GDR (Zhao et 2024).
3.1 BACKGROUND: OCL WITH VAE GUIDANCE
Assume there is an input image or video frame T Xuoxxo where typically hu wy 128 and co = 3.
Initially, the primary encoder encodes the input into feature map. Φe T F. where F ghxuxxe and Φ. is CNN (He et al. 2016) or ViT (Caron et al. 2021) module. Typically, h = w 32 and c= 256.
Then, the Slot Attention (SA) (Locatello et al. 2020 is used to aggregate the dense feature map F into sparse object-level feature vectors, i.e., slots S E Ra along with the byproduct, i.e., segmentation masks M E I2*x under query vectors So. d.n (So. F) + (S, M). which basically an iterative Query-Key-Value attention (Bahdanau et al., 2015) mechanism with So as the query and F the key and value. Typically, number objects plus the background in an image or video sample of a dataset.
predefined number, usually the maximum
Meanwhile, pretrained VAE (Im Im et al. 2017: Van Den Oord et al. 2017 converts T into contin- Z. where Z E R and typically h = w = 32 and 256. This continuous representation is quantized into discrete representation by matching continuous representation with codebook and selecting matched codes to form discrete representa- tion. fn fm Ss (Z, (P, I, X), where codebook C mm matching probabilities P xm matched indexes TR and discrete representation X Rhxwxe, The match- ing fm and selection fs are implemented as below, respectively
uous intermediate representation. $.
P = softmax, ( (-1 - -C(l)). argmax,, (P)
x = select,
(2)
where each super-pixel ((d..) E R means the matching probabilities of the m template features in C. So the index of the maximum element in p(ij-)) i.e., ((65) is the index of the most matched code, which can quantize Z (d.ji) into X (A.j.) = (...)
Lastly, with S as the condition, the primary decoder reconstructs the input guided by VAE discrete intermediate representation conditioned on S, which drives every slot to aggregate as much object information as possible, &a (X, S) X', where X' E kxxwxx For transformer-based OCL, a transformer decoder (Vaswani et al. 2017) with internal masking performs the reconstruction as classification. For diffusion- based OCL, diffusion model, typically conditional UNet (Rombach et al. 2022), carries out the reconstruction as regression the noise added to X.
For VAE pretraining, there is also VAE decoding from discrete representation to the input Φ x - T', where T" E X And the supervision signal comes from Mean-Squared Error (MSE) between T and T'. For OCL training. the pretrained VAE module is frozen, and no VAE decoding occurs. Remaining parts are trained under the supervision signal of either Cross Entropy (CE) between X and X', or Mean-Squared Error (MSE) between the ground-trum truth noise added to X and the reconstructed noise.
3

---

Published as a conference paper at ICLR 2025
clevrtex
COCO
VDC
movi_c
movidd
moviee
COCO
VOC
image
truth
slotdilfoz
State
Figure 2: Qualitative results of OCL unsupervised segmentation. The last row MSF.
3.2 NAIVE MULTI-SCALE TRAINING
There can be objects of different sizes in images or videos, while models usually have best-at pat- terns scales due non-uniform data distributions. This makes image pyramid (Adelson et al. 1984) an effective technique, where the input is resized into different sizes and processed in parallel. Such multi-scale representations make the pattern scale distribution more uniform for training, and sures different sized objects to fall within models' comfort zone for testing.
Input image video frame Ti T is down-sampled with typical bi-linear interpolation into Tn E R
2o-1 XCu where I...N means different spatial scales and we use N 3.
Multi-scal VAE encoding is Φ² Tn - Zn where Zn 2 g and typically h w 32 and 256.
In, where P. Here codebook C is shared over all N scales so as to facilitate scale-invariant pattern learning that differentiates among different ob- ject types. But this strong inductive bias lose scale-variant information that differentiates among different object instances of the same type; Besides, no information exchange among the scales is kind of information waste The object pattern that is well represented in one scale should augment its poorly represented patterns in the other scales. We address this in the next subsection.
Multi-scale VAE quantization is fq = o CZ, - xm I, <R= and X R x E
Multi-scale VAE decoding Φi X, Th where TV E R
During VAE pretraining, there are N sets of reconstruction and quantization losses. During OCL training, only scale n = is used as guidance, thus the supervision is no different from the former subsection. Although multi-scale guidance
be used, it always harmful in practice.
By following such recipe, hopefully at least one of the N scales of representation provides high- quality representation some sized objects.
Note: The original VQ- IQ-VAE has only n = scale, which is a special case our method.
3.3 INTER/INTRA-SCALL QUANTIZED FUSION
Based on the above multi-scale quantization, we further devise our novel fusion among the scales. Inter-scale fusion utilizes scale-invariance to augment poorly represented features in some scales with the well-represented features in other scales; Intra-scale fusion supplements with scale-variant details in the corresponding scale. Scale-invariant patterns differentiate among different object types, while scale-variance distinguishes among different object instances of the same type.
Firstly, project VAE continuous representation of certain scale Z,, into two copies, Z for scale-imes invariance and Z for scale-variance, respectively. These two will finally be combined back, thus we employ the invertible pooject-up-project--don design proposed in Zhao al. (2024) to
4

---

Published as a conference paper at ICLR 2025
maintain the inductive biases added in between.
7 2 chunk,.(Z,, Dinv(W)
where pin is pseudo-inverse of the project-down matrix in Eq. 5 for project-up here; chunkg2()) splits the tensor into two along the channel dimension.
Secondly, for the scale-variant continuous representation, conduct quantization by codebook specific to the current scale. fq = fm fs (Z C, (PAY F x These non-sharing codebooks enforces scale-variant representation learning.
Meanwhile, for the scale-invariant continuous representation, we conduct inter-scale fusion over scale-invariant quantization of all scales. fit {(Z,) C= - {(Pai jsl ) where n = 1...N. The shared codebook enforces scale-invariant representation learning.
Now that all scales {x:) consist of codes a shared codebook C matching probabilities (P:)) for the same (super-)pixel region under different scales indicate how confident the model is about the representation quality. We utilize this to augment the low-quality representation in some scales with the high-quality other scales. Take scales n 1, n and + as an example:
1. Given any super-pixel z == Z (ا.+)) Re in any scale find the corresponding region/super-pixe(()) in other scales, and mean them along all spatial dimensions as vot- ing. For scale 1, we choose 2. 222.2222444222222-222
22222x and mean it into vector
2el ((((())))+))))
2 E IR:; For scale 1, we choose 2
R.
2. Match {=i-11 = =n+1} respectively with codebook C in Eq. then we have corre- sponding matching probabilities (Pn PM: p++1) E {RT These probabilities can be taken as the super-pixel quality of the same region in scales [n - 1, n, n + 1}.
3. Select the most matched code with the highest probability among all scales as the quanti- zation (t..) of the super-pixel Z (i..). Specifically, for scales {n 1, n, 1}, the most matched code index is
(L.) argmax, (max, (fpsl P. P p
(4)
n
with which we can determine (i..)( as in Eq.
Lastly, for each scale n, conduct the intra-scale fusion between the fused scale-invariant represen- tation and scale-variant representation. far (X:i n2 x=)) + X,, This combines the augmented
scale-invariant and scale-variant copies back into the final discrete intermediate representation.
X, concat,
W
(5)
where c-ncate()) is channel concatenation; project-down matrix W E 22exe is trainable.
The above processes can be parallelized using PyTorch, shown in Algo. pseudo code.
3.4 RESOURCE CONSUMPTION ANALYSIS
Codebook-related parameters. The basis methods codebook of size (),c) 4096 256, which amounts to 1. 1.08MM. We use (1) invertiable projection to project the continuous representation into two copies (and project them back), up x 2c 2"7; (2) four groups of codebooks of size (m. c) (64, 256), one shared for N 3 scale-invarian scales, three specifically for N = 3 variant scales, up 0.5 x x (1 + N) 216 Thus our codehook-related parameters are 0.196M we have 81. 25% fewer codebook parameters than the basis methods.
Computation and memory consumption. The exact calculation relates with the VAE encoder/decoder layers, like Conv2d, GroupNorm, Mish, etc.. thus has too many details. But due to the O(n²) complexity nature of convolution layers, we can quickly get the estimation in ratio. Denote the computation of the original VAE as unit, then our N 3 scales' computations would be 1, and 16 since the latter two has 2x and 4x down sampling rates. Thus in VÁE pretraining. our technique requires roughly 31. 25% more computation. The memory consumption increase is similar.
5

---

Published as a conference paper at ICLR 2025
ART
ARIg
mloU
mBO
ARI
ARIg
mloU
mBO
ClevrTex
MOVI-C
SLATE,
20.56
68.14
34.11
34.57
STEVE
52.20
31.16
16.74
19.05
#SysBind@2
23.27
71.27
35.63
36.62
+SysBind@2
51.87
34.64
17.36
19.12
+GDR@.² g2
34.46
73.38
37.42
36.69
+GDR@g2
60.14
35.79
20.01
21.95
+MSF
32.70
80.70
40.61
41.48
+MSF
60.94
36.22
20.33
22.74
COCO
MOVi-D
SLATE,
24.18
24.54
21.37
21.76
STEVE.
35.71
50.24
19.10
20.88
+SysBind@2 @
25.71
24.97
21.46
22.01
+SysBind@2
35.34
52.16
19.46
21.53
+GDR@ g2
30.37
29.95
23.47
22.98
+GDR@92 g2
40.37
52.47
20.42
22.64
+MSF
30.95
30.47
23.33
23.85
+MSF
43.20
55.64
21.21
23.14
VOC
MOVi-E
SLATE,
11.64
15.65
15.64
14.99
STEVE.
28.00
52.06
18.78
20.48
SysBind @
11.75
16.04
15.73
15.01
SSssBind@2
28.47
55.46
18.95
20.50
+GDR @ g2
13.20
17.49
16.46
16.65
+GDR@gt
34.17
53.21
19.47
20.76
+MSF
12.17
16.54
16.74
16.69
+MSF
36.70
54.28
20.39
22.37
Table 1:
Transformer-basee image OCL perfor- mance on synthetic and real-world datasets.
Table 2: Transformer-base video OCL syn- thetic datasets.
ARI
ARIg
mloU
mBO
Aiter-Cluster'
Intra-Clusted
ClevrTex
VQ-VAE
VQ-VAE +MSF
SlotDiffuz,
64.21
26.50
31.51
32.44
ec255
+GDR @2
69.21
37.83
34.74
34.03
ClevrTex
0.534
0.786
0,043
0.038
+MSF
71.47
38.08
35.06
35.67
COCO
0.746
1.095
0.184
0.093
DINOSAUR
60.74
45.75
30.48
32.56
VOC
0.633
0.978
0.191
0.101
COCO
MOVi-C
0,506
0.842
0.092
0.090
SlotDiffuz,
36.54
35.67
22.08
22.75
MOVi-D
0.511
0.725
0,089
0.088
+GDR@0 92
37.68
36.33
22.73
22.25
MOVi-E
0.499
0.719
0.103
0.094
+MSF
37.63
36.99
22.32
22.87
@c4
DINOSAUR
33.24
33.35
22.01
21.93
ClevrTex COCO
0.857
1.473
0.103
0.099
VOC
1.283
1.745
0.218
0.114
SlotDiffuz,
16.97
14.33
15.71
16.02
VOC
1.046
1.659
0.196
0.132
+GDR@g2
18.20
15.59
16.11
17.04
+MSF
19.40
15.69
16.37
16.76
Table 4: Object separability of VAE guidance. "Inter-Cluster" the mean distance of super- pixels within cluster, while "Intra-Cluster" is that among different clusters. "@c256" and @ "@c4" are VAE intermediate dimensions, for SLATE/STEVE and SlotDiffusion, respectively.
DINOSAUR
16.00
18.48
15.94
16.37
Table 3: Diffusion-based image OCL on syn- thetic and real-world datasets. Subssript "r" and "c" stand for random and condition query initialization, respectively.
4 EXPERIMENTS
We experiment the following points: (i) Our technique MSF augments performance of mainstream OCL methods that are either transformer-basel or diffusion-based. (ii) MSF fuses multi-scale infor- mation into VAE discrete representation and guides OCL better. (ii)) How the composing designs of MSF contribute to its effectiveness. Results are mostly averaged over three random seeds.
4.1 OCL PERFORMANCE
To evaluate the quality of OCL object representations, i.e., slots, we use the accuracy of OCL's byproduct (unsupervised) segmentation as an intuitive measurement. Metries including ARI (Ad- justed Rand Index] ARI, (foreground), mloU (mean Intersection-over-URee
? and mBO (mean
Best Overlap) (Caron et al. 2021) are used measure OCL's byproduct segmentation accuracy.
http:////wtttiemnnnaaaaaooooooee http:aaammmmrrrii.aiiiiiai adjusted rand scare.html
http://///dddiianaaaaaaaaaaaaaee wttp:aaawmmttttsscccaeacooo score.html

---

Published as a conference paper at ICLR 2025
Datasets are either synthetic images, ie., ClevrTex real-world images, i..., cOco and voc or synthetic videos, i.c., MOVi-C/D/E
To evaluate how MSF boosts OCL, for images and STEVE (Singh et al. 2022c) for videos; use diffusion- sfate-of-the-ane the-arts, Slot-
transformer- classics, SLATE (Singh et al. 2022a)
Diffusion (Wu al. 2023b) for images. The foundation-basee) DINOSAUR (Seitzer et al. 2023)
and competitors SysBinder (Singh et al.) 2022b) and GDR (Zhao et al. 2024 are compared too. All
primary encoders are unified DINO (Caron al. 2021 form strong baselines.
Our MSF is a general improver to both transformer-basee classics and inffusion-based state-of-ther arts, as shown in Tab. m2 and On complex synthetic dataset ClevrTex, MSF improves SLATE significantly, up to 12% in ARlig. On challenging real-worke dataset COCO. MSF still manages to improve SLATE by 6% in ARIgg. MSF even boosts SlotDiffusion, which is state-of-the-arr by 3% in ARI on VOC. Compared with other improvers, i.e., SysBinder g2 and GDR @ @g our MSF also beats or at least ties them on SLATE, while beats them consistently on STEVE. On SlotDiffusion, our MSF beats the strong competitor GDR e2 and defeats DINOSUAR.
By the way, our MSF boosts OCL performance on COCO and VOC less than on other datasets. We attribute this to those object sizes are often out of the coverage of scales 1'3. Namely, many humans, trees and grass lands in those real-world images take up the whole scene, and either down sampling 4x (scale 1) 16x (scale 3) cannot produce scale-inee scale-invariant patterns for inter-scale fusion.
4.2
VAE GUIDANCE
To investigate how MSF fuses multi-scale information into VAE guidance, we analyze object separa- bility (Stanic et al. 2024 Lowe et al. 2024). Empirically, better object separability in VAE guidance contributes to better OCL performance. We cluster the VAE guidance X,== then calculate inter- fintra-cluster distances; We also visualize (Caron et al.) (021) object separability of invariinvariant n inter-scale fusion (Xal) and representations after intra-scale fusion (X,).
representations {Xx') and scale-variant representations {X*),, J. as well as representations before
As shown in Fig. B and Tab. our MSF contributes to significant object separability boosts both qualitatively and quantitatively upon basis methods. In contrast, the object separability of basis methods is consistently inferior to ours.
4.3 ABLATION STUDY
We use SLATE on ClevrTex as an example to evaluate how different designs contribute to our MSF. We use ARI+ARI; as the metrics because ARI is mostly dominated by background segmentation performance while ARlg only measures the foreground. We employ naive CNN (Kipf' et al. 2022) as the primary encoder to reduce experiment time.
Inter/intra-see scaalerrr-scaeeeeeeeeeeee fusion: with vs without, As shown in Tab. 5 uppeer left, both types of fusion are crucial performance. Yet using inter-scale fusion alone can be even worse than disabling both, meaning that inter-scale fusion enforces inveeinveriannee at the cost of losing too much information, wheras intra-scale fusion can recover it. Therefore, enabling both types of fusion is the best setting.
Scale-ine Scale-invariant/vee invariant/variant codebook: shared vs specified. Firstly, using different codebooks as the ccale-invariant codebook is meaningless because the corresponding matching probabilities for the same super-pixel cannot be compared. As shown in Tab. [ uppee right, sharing one codebook over different scales for the ccale-variance is harmful to the performance. Hence different sets of codes necessary to foster diverse patterns that different among the scales.
Number of scales: 2, 3 or 4. As shown in Tab. flowe left, the default number of scales = 3 is the best setting. By contrast, N = 2 includes too less explicit scales; As for N 4, its last scale, i.e., scale 4, down samples too much and loses too much spatial information to support the inter-scale fusion of our MSF technique.
7

---

Published as a conference paper at ICLR 2025
intta--caaee fusion
scale valeiinarioee rop..
husedial sircpi
Xn
Xii
X!
X
X
scale1
scale2
Scale3
inter-scale fusion-
Figure 3: Qualitative object separability VAE guidance with MSF. Take the upper left as an example. The circled cube in column X scalel is vague and noisy, while its corresponding representation b in scale2 is very clear, thus b can augment a, yielding clearer e in column X row 1cale1. As e only captures scale-invariant information, d in column % row scalel supplements scale-variant information, which has too many details and is not object-like. By combining e and d, we obtain in column X, row scale1, making this cube much more separable from the background than in column X the corresponding area. Notations are detailed in Sect. 4.2 beginning.
inter-scale fuz
intra-scale fuz
ARI+ARI
scale- variant codchook(s) specified
100.71
shared
ARI+ARI
90.85
100.71
95.64
92.28
91.03
number of guidance
number of scales
ARI+ARIg
scale scale
ARI+ARIg
2
95.83
100.71
100.71
96.42
98.96
89.59
Table 5: Ablation study of SLATE+MSF on ClevrTex using naive CNN as the primary encoder. (Upper left) effects inter-scale fusion and intra-scale fusion; (Upper right) effects of specified or shared codebooks for scale-variance; (Lower left) effects of number of scales being used in VAE pretraining; (Lower right) effects of number of VAE guidance being used for OCL training.
Number of guidance: or 2. After MSF in VAE, we have three augmented scales. But as shown in Tab. lower right, using the augmented scale I as VAE guidance is the best choice, compared with either using both augmented scale and 2, or using augmented scale solely. This is because the augmented scale 1 representation has the largest resolution, i.e., the most spatial details, and also multiple scales information. Although the other two scales are also multi-scale fused, the information incompleteness due their low resolution is harmful to guide OCL training.
8

---

Published as a conference paper at ICLR 2025
5 CONCLUSION
We propose a technique named MSF to improve existing OCL methods, including both transformer- based classics and diffusion-based state-of-the-arts. Our technique is realized by integrating the naive image pyramid with our novel inter-scale quantized fusion and intra-scale quantized fusion. Our technique ensures different sizes of objects are all processed in VAE models' comfort zone, and fosters scale-invariance and scale-variance in object representation. This is achieved at acceptable costs of extra computation and memory. We evaluate our technique via comprehensive experiments, and also interpret the multi-scale fusion processes by visualizing it. Such multi-scale fusion is also worth exploring in the primary encoder and slot attention part, or, in VAE designing for generative models, which are suggested for future work.
ACKNOWLEDGMENTS
We acknowledge the support of the Finnish Center for Artificial Intelligence (FCAI) and the Re- search Council of Finland through its Flagship program. Additionally, we thank the Research Coun- cil of Finland for funding the projects ADEREHA (grant no. 353198). BERMUDA (362407) and PROFI7 (352788). We also appreciate CSC - IT Center for Science, Finland, for granting access the LUMI supercomputer, owned by the EuroHPC Joint Undertaking and hosted by CSC (Fin- land) in collaboration with the LUMI consortium. Furthermore, we acknowledge the computational resources provided by the Aalto Science-IT project through the Triton cluster.
REFERENCES
Edward H Adelson, Charles H Anderson, James R Bergen, Peter Burt, and Joan M Ogden. Pyramid Methods in Image Processing. RCA engineer, 49,6):33-4,, 1984.
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. International Conference on Learning Representations, 2015.
Moshe Bar. Visual Objects in Context. Nature Reviews Neuroscience, 58)):117-629, 2004.
Christopher Burgess, Loic Matthey, Nicholas Watters, al. MONet: Unsupervised Scene Decom- position and Representation. arXiv preprint arXivv1901. 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervise Vision Transformers. In Proceedings of the IEFF/CVF international conference on computer vision, pp. 1660-9660, 2021.
Patrick Cavanagh. Visual Cognition. Vision Research, 51(13))
2011. 2011.
Yunpeng Chen, Haoqi Fan, Bing Xu, et al. Drop Octave: Reducing Spatial Redundancy in Convolutional Neural Networks With Octave Convolution. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14435-3444 2019.
Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd Van Steenkiste, al. SAVi++: Towards End-to- End Object-Centric Learning from Real-World Videos. Advances in Neural Information Process- ing Systems, 35:28940028854 2022.
songwei Ge, Shlok Mishra, Simon Korblith, Chun-Liang Li, and David Jacobs. Hyperbolic Con- trastive Learning for Visual Representations Beyond Objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6840-6849, 2023.
Klaus Greff, Raphael Lopez Kaufman, Rishabh Kabra, et al. Multi-Object Representation Learning with Iterative Variational Inference. In International Conference on Machine Learning, pp. 2424- 2433. PMLR, 2019.
Jiaqi Gu, Hyoukjun Kwon, Dilin Wang, Wei Ye, Meng Li, Yu-Hsin Chen, Liangzhen Lai, Vikas Chandra, and David Z Pan. Multi-Scall Righ-Resslutton Vision Transformer for Semantic Seg- mentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- fion, pp. 12094-1 12103, 2022.

---

Published as a conference paper at ICLR 2025
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778,788 2016.
Gao Huang, Shichen Liu, Laurens Van der Maaten, and Kilian Weinberger. CondenseNet: An Efficient DenseNet Using Learned Group Convolutions. In Proceedings of the IEEE Conference Computer Vision and Pattem Recognition, pp- 2752-27611 2018.
Daniel Im Im, Sungjin Ahn, Roland Memisevic, and Yoshua Bengio. Denoising Criterion for Vari- ational Auto-Ennte Eetto-nn Framework. In Proceedings of the AAAI conference on artificial intelli- gence, volume 31, 2017.
Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-Centric Slot Diffusion. Advances in Neural Information Processing Systems, 2023.
Thomas Kipf, Gamaleldin Elsayed, Aravindh Mahendran, et al. Conditional Object-Centric Learn- ing from Video. International Conference on Learning Representations, 2022.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet Classification with Deep Convo- lutional Neural Networks. Advances in Neural Information Processing Systems, 25, 2012.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming Hc, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2117-2125, 2017.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, et al. Object-Centric Learning with Slot Attention. Advances in Neural Information Processing Systems, 33:11525--15588 2020.
Sindy Lowe, Phillip Lippe, Francesco Locatello, and Max Welling. Rotating Features for Object Discovery. Advances in Neural Information Processing Systems, 36, 2024.
Shervin Minace, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Ter- zopoulos. Image Segmentation Using Deep Learning: A Survey. IEEE transactions pattern analysis and machine intelligence, 44(7):3523-3544 2021.
Mahyar Najibi, Bharat Singh, and Larry S Davis. Autofocus: Efficient Multi-Scale Inference. In Proceedings of the IEEE/CV international conference on computer vision, pp. 9745-9755, 2019.
Thomas Palmeri and Isabel Gauthier. Visual Object Understanding. Nature Reviews Neuroscience, 5(4):291-303, 2004.
Yongri Piao, Yongyao Jiang, Miao Zhang, Jian Wang, and Huchuan Lu. PANet: Patch-Aware Network for Light Field Salient Object Detection. IEEE Transactions Cybernetics, 53(1): 379-391, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High- Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, et al. Bridging the Gap to Real- World Object- Centric Learning. International Conference Learning Representations, 2023.
Bharat Singh and Larry Davis. An Analysis of Scale Invariance in Object Detection SNIP. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3578-3587, 2018.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALL-E Learns Compose. International Conference on Learning Representations, 2022a.
Gautam Singh, Ycongbin Kim, and Sungjin Ahn. Neural Systematic Binder. International Confer- on Learning Representations, 2022b.
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos. Advances in Neural Information Processing Systems, 35: 18181-18196, 2022c.
10

---

Published as a conference paper at ICLR 2025
Aleksandar Stanic, Anand Gopalakrishnam, Kazuki Irie, and Jurgen Schmidhuber. Contrastive Train- ing of Complex- Valued Autoencoders for Object Discovery. Advances Neural Information Processing Systems, 36, 2024.
Mingxing Tan, Ruoming Pang, and Quoc V Le. EfficientDet: Scalable and Efficient Object Detec- tion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10781-10790, 2020.
Keyu Tian, Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual Autoregressive Mod- eling: Scalable Image Generation via Next-Scale Prediction. arXiv preprint arXiv:2404.02905, 2024.
Aaron Van Den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural Discrete Representation Learning. Advances in Neural Information Processing Systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, al. Attention Is All You Need. Advances in Neural Information Processing Systems, 30, 2017.
Xudong Wang, Ishan Misra, Ziyun Zeng, Rohit Girdhar, and Trevor Darrell. VideoCutler: Sur- prisingly Simple Unsupervised Video Instance Segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 22755 -27644, 2024.
Ziyi Wu, Nikita Dvorikk, Klaus Greff, Thomas Kipf. and Animesh Garg. SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centte Models. International Conference on Learning Representations, 2023a.
Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. SlotDiffusion: Object- Centric Generative Modeling with Diffusion Models, Advances in Neural Information Processing Systems, 36:50932-50958 2023b.
Lijun Yu, Yong Cheng, Zhiruo Wang. Vivek Kumar, Wolfgang Macherey, Yanping Huang, David Ross, Irfan Essa, Yonatan Bisk, Ming-Hsuan Yang, et al. SPAE: Semantic Pyramid Autoencoder for Multimodal Generation with Frozen LLMs. Advances in Neurul Information Processing Sys- fems, 36, 2024.
Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Object-Centtie Learning for Real- World Videos by Predicting Temporal Feature Similarities. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848 6856, 2018
Rongzhen Zhao, Zhenzhi Wu, and Qikun Zhang. Learnable Heterogeneous Convolution: Learning both Topology and Strength. Neural Networks, 141:270-280, 2021.
Rongzhen Zhao, Jian Li, and Zhenzhi Wu. Convolution of Convolution: Let Kernels Spatially Col- laborate. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- 2022.
fion, pp.
Rongzhen Zhao, Vivienne Wang. Juho Kannala, and Joni Pajarinen. Grouped Discrete Representa- tion for Object-Centric Learning. arXiv preprint arXiv:2411.02299 2024.
Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object Detection in 20 Years: A Survey. Proceedings of the IEEE, 111(3)2257-27,, 2023.
A APPENDIX
A.1 EXTENDED RELATED WORK
Multi-scale & image/feature pyramid. Existing multi-scale methods (i) Either employ an image pyramid (Adelson al. 1984 Singh & Davis 2018 Najibi et al. 2019) without information fusion
11

---

Published as a conference paper at ICLR 2025
among pyramid levels, wasting useful information among different scales; (ii) Or rely feature pyramids, namely, FPN and numerous variants (Lin et al.) 2017; Tan et al. 2020 Pio et al. 2021 where features of different DNN depths are fused by channel-concat element wise- sum, mixing both low- and high-quality representations together. In contrast, our MSF is the first to enable fusion of multiple scales VAF representations. By leveraging codebook matching (Van Den Oord al.) 2017), we selectively fuse high-quality information among scales, rather than mixing them together. As shown in Tab. our MSF is superior to naive channel-concat or element-wise-sum,
channel-concat
clement-wise-sum our msi
ARI+ARI
92.63
89.18
100.71
Table 6: Effects of different fusions among scales. Model is SLATE; dataset is ClevrTex.
Multi-scale & VAE. Recent works like VAR (Tian et al., 2024 and SPAE Yu et al. 2024) also build the multi-scale representations upon VAE. But they are different from our MSF as follows. (i) VAR vs MSF: VAR auto-regresses from smaller scales of VAE representation to the larger, but with no information fusion among scales. contrast, our MSF realizes fusion among different scales on VAE for the first time. (ii) SPAE MSF: SPAE relies multi-modal foundation model CLIP, while our MSF does not. SPAE clement-wisely averages multiple scales into one, mixing both low- and high-quality representations together, i.e., no fusion among scales. Our MSF augments any scaled representation with all other scales' high-quality representations, i.e., fusion among scales.
Relationships among SSLseg, OCL and WMs. As shown in Fig. (1) SSLseg -Self-SSpervised Learning segmentation), e.g., HCL (Ge et al. 2023) and VideoCutLEi (Wang et al. 2024), focuses extracting segmentation masks; (2) OCL (Object-Centric Learning), e.g., SLATE (Singh al., 2022a) and SlotDiffusion (Wu et al. (023b), represents each object as a feature vector, with seg- mentation masks byproducts, kind of overlapping with SSLseg; (3) WMs (World Models), e.g., SlotFormer (Wu et al. (2023a), upon OCL, address downstream tasks like visual reasoning, planning and Decision-making
(1) object segmentation masks
represent objects as festure vectors respectively
visual reasoning, planning, decision-making
SSLseg HCL, CutLER
OCL SLATE/SlotDifueion +MSF
WMs SlotFormer
Figure 4: SSLseg vs OCL vs WMs.
A.2 EXTENDED EXPERIMENTS
MSF's effect on different sized objects. We follow COCO's small/medium/large size splits to evaluate our MSF's performance. Results of resolution 128x 128 are shown below Tab, 7 Our MSF does show better performance on small objects.
mloUs
mloUs
mloU,
overall
SLATE +MSF
8.57
26.65
34.57
26.94
12.63
28.14
34.83
29.57
Table 7: How MSF peforms different sized objects. Dataset is COCO instance segmentation.
MSF on OpenImages subset with higher resolution. We use input resolutions 128 and 256 to evaluate on OpenImages subset Ge et al.) (2023). Under resolution 128, we use n=3 and 4; under resolution 256, n=3, 4 and 5. Results shown below Tab. g We use ARI+ARIfg as
12

---

Published as a conference paper at ICLR 2025
the metrics because ARI mostly reflect how well the background is segmented and ARlfg only measures foreground objects. According to the results, under resolution 128, n=3 the best choice; under resolution 256, n=4 is the best choice.
resolution
128x 128
256x256
ARI+ARIg
62.07
60.93
64.82
67.58
62.75
Table 8: Effects input resolution and #scales n. Model SLATE+MSF; dataset is Openlmages subset.
A.3 MODEL ARCHITECTURE
The overall model structure is depicted in Fig. upper right. It consists of primary encoder, SlotAttention and primary decoder, as well as VAE encoder, codebook, and VAE decoder.
Primary Encoder
Unlike exiting works, which used a naive CNN or small ResNet, we use pretrained foundation model DINO as the primary encoder on all datasets, that setup strong baseline to show MSF's effectiveness.
To save computation while maintain strong performance for all methods, we adopt the big-little design used in (Zhao et al. 2024), which takes small resolution 128 as input while still produce high resolution (128) feature maps. We employ the DINO vl, tiny, with downsampling rate of 8, as presented in (Caron et al. 2021). We integrate this into a big-little architecture by running DINO parallel with the previously mentioned naive CNN. The output of DINO then projected, upsampled, and combined element-wis with the output of the CNN. The architecture is briefly represented (k33)) potooooooosppppappppppeeeeeee upsamplee(88) + naivecnn(). Notably, the weights of DINO are frozen, which retains its pretrained features while allowing the CNN to focus refining high spatial details.
Hence, no spatial downsampling occurs within this module. This part remains consistent across all experiments.
SlotAttention
We adhere to the standard design outlined (Locatello al. 2020). This design is consistent across all experiments utilizing this module.
Primary Decoder
This module can be either transformer decoder or a diffusion model.
For transformer-base based OCL, e.g., SLATE and STEVE, we follow the architecture described in (Singh et al.) 2022a), unified across all experiments involving this component. This decoder op- erates on VAE intermediate feature tensors, which undergo 4x4 spatial downsampling relative to the input. SlotAttention-derrive slots are provided as the conditioning input. The input channel dimension for this module is set to 256.
For diffusion-based OCL, c.g., SlotDiffusion, follow the architecture described in (Wu et al. 20236). This decoder take VAE representation with noise as input, taking slots, time step as con- dition, finally produce the reconstruction of the noise being added to the input. The input channel dimension for this module is set 4.
VAE Encoder & Decoder
The architecture of this module is in accordance with the design described in (Singh et al. (022a), and it is unified for all experiments incorporating this module. The VAE encoder applies 4x4 spatial downsampling, and the VAE decoder performs corresponding 4x4 upsampling.
Codebook
This module distinguishes between dVAE and VQ- and is utilized in both transformer-bas based and diffusion-based methods.
13

---

Published as a conference paper at ICLR 2025
Algorithm Implementation of inter/intra-scale quantized fusion in PyTorch.
def ns fuulff codebook: : ModuleList, encodes: list, project: Module) list:
len(dncse assert len(self 1en(self_codebood) codebook)
shared#1 specified*s
b, c, h, encodes[e]. shape
encodes [project(_, pinv-True) for in encodes] # (bbb,,,hh
zidx1s, encodels, quantis [], [1, [] for range(s):
encodel_ - []
for j1, encode in enumerate(enccoee):
if j1 == 11: pass
elif j1 <
el_ avg poooo2ddeenccdee ic, ", ;), 2 (i1 j1))
else:
el_ upsample(encode[;
:1, faato_factor=2 ** (j1 i1), mode="nearest")
encodel app) (b,,,,,,,w)
encode1_ = stack (encodee1
(s,b,,,h,w)
zsofti_, zidx1_ self @elf_codebooo_000 tet.e
match(eencoee1 1))
zsoft1 zsofti_ .nftiatenfllttee(( [s, b]) (stb:mmh.w)=>555b.mmbb
# zidx1 zidx1 unflatten(0, [s, b]) (s*b,hh))-=(ssbb,h,,))
2soft11, zidx11 2softl_.axx(2 (s)b:,)
zidx12 argax(1 (,,b,,,,))-((bhhp)))
zidx1 zidx11.gather(6. zidx12[None, i, i, :])[0]
# (s,bb,chh,w->(bbc,hh.,w)
encodel encodel..gathere zidx12(None, None, :] exxpanddd--1, -1, -1, -1))[0]
quant1 [ell__oddeooo(e]]] (zidxx)) (zickt)..errutee(() permute(e, 3, 1, 2)
z 2loxts. a s*bb,,,w)
sppodees.appenddenccoo quant1s.appende (quant))) # (b,,,,h,,)
(b,,,,,,,)
zidx2s, encode2s, quant2s [], [], []
for 12 in range(s):
encode2 encodes c:, ;, :]
zsoft2, zidx2 self [olf_codebooo[1111 (22ccoe2 quant2 self_codebook[1 (eeexxxx)).peruuteee 3, 1, 2)
zidx2s. (zze222x)pe))))idk2)) h*b,,, (cccdee2s..peendeencceeeee (b,,,,,,,,) quant2s. apppld2uatpp))( a s*(b,c,h,w)
zidxs [stack([u, v], 1) u, zip(zidx1s, zidx2s)] (b,,,,, encodes [cat([u, v], 1) for u, zip(encode1s, encode2s)] (b,2,,,,,,)) quants [cat([u, v], 1) for u, v in zip(quantis, quant2s)] # (bb,,c,,,,w)
quants (project(_) for in quants] sbb,,,, return zidxs, quants
For the non-grouped codebook design, we adhere to the typical d'VEE and VQ-VAE implementations as described in (Singh et al. (202aa) and (Wu et al.] 2023b), respectively. For the grouped codebook take the GDR as the basis, by applying with
design used in SLATE/STEVE and SlotDiffusion, our design as mentioned in main body of this paper. Thus except what we describe the main body, we also adopt annealing residual connection and Gumbel sampling code matching stabilize training.
Implementation of our MSF described in Sect. 3.3 is described in Code T
This module operates on VAE intermediate feature tensors with a 4x4 spatial downsampling relative to the input.
14

---

Published as a conference paper at ICLR 2025
The non-grouped codebook contains 4096 codes; Similarly, the grouped codebook contains 4096 codes combined from scale-invariant and scale-variant codebooks, for which we introduce code replacing trick, instead of utilization loss, to improve code utilization. In transformer- the code dimension (VAE intermediate channel size) is 256, whereas in diffusion-han -hased models, the code dimension is 4.
models,
Basis Methods
For SLATE/STEVE, primarily adhere to the original designs for transformer- based models, SLATE and STEVE. Their primary encoder, SlotAttention, decoder, and VAE conform to the re- spective designs mentioned earlier. SLATE is tailored for image-based OCI tasks, while STEVE extends SLATE to video-based OCL tasks by adding an additional transformer encoder block to bridge current and subsequent queries.
For SlotDiffusion, we also follow its original design for diffusion-based methods. Its primary en- coder, SlotAttention, decoder, and VAE conform to the designs outlined earlier. For video-based OCL tasks, we extend SlotDiffusion by adding an extra transformer encoder block to connect cur- rent and future queries. For the diffusion decoder, we use default noise scheduler and beta values.
DINOSAUR
DINOSAUR is included as foundational method for OCL, serving primarily as a reference. We adopt its original design but unify its primary encoder and SlotAttention with those described above.
Competitive Improvers
SysBinder is one of the competitors. This module enhances the SlotAttention module in SLATE/SEVV. We follow its original design while keeping the other components consistent with the carlier-described modules. We use group number two.
GDR is another competitors. We use group sizes of two, and leave all the other settings identical to the original paper.
A.4 DATASET PROCESSING
The datasets used for image OCL include ClevrTex, COCO, and VOC, with the latter two being real-world datasets. For video OCL, the datasets are MOVi-C, D, and E, all of which are synthetic. While the processing is mostly consistent across datasets, there are a few differences.
Shared Preprocessing
To speed experiments, we implement a dataset conversion approach: converting all datasets into the LMDB database format and storing them on an NVMe RAM disk to reduce I/O overhead and maximize throughput.
In particular, all images or video frames, along with their segmentation masks, are center-cropped and resized to 128x128. These are then stored in the uint8 data type to save space. For images and videos, we apply the default interpolation mode, whereas for segmentation masks, we use the NEAREST-EXACT interpolation mode.
During training, for both images and videos. we use an input and output spatial resolution of 128x128. Input images and vidcos normalized by subtracting 127.5 and dividing by 127.5, en- suring all pixel values fall within the range of to 1. For videos, we apply random strided temporal cropping with fixed window size of 6 to speed up training and improve generalization.
During testing, image processing remains identical to the training phase. However, for videos, we omit the random strided temporal cropping and instead use the full 24 time steps.
CleyrTex, COCO & VOC
For ClevrTex, each image contain up to 10 objects, slot representing the background.
10 1 slot queries, with the extra
Microsoft COCO and Pascal VOC are both real-world datasets. For COCO, mentation annotations, as OCL favors panoptic segmentation over instance or semantic segmen- tation. However, for VOC, rely semantic segmentation annotations due to the absence of
panoptic seg-
15

---

Published as a conference paper at ICLR 2025
panoptic or instance segmentation labels. Given that images contain numerous objects, includ- ing very small ones, and considering the 128x128 resolution, we filter out images with more than 10 objects or objects smaller than 16 16-pieel bounding box area. As result, objects (plus stuff in COCO) per image, corresponding to slot queries.
maximum of 10
MOVi-C/D/E
These datasets provide rich annotations, such as depth, optical flow, and camera intrinsic/extrinsic parameters. However, for simplicity and comparison purposes, limit our to segmentation masks. Additionally, we retain the bounding box annotations for all objects across all time steps, this conditioned query initialization is crucial and widely used in video OCL. Given that these datasets contain 10, 20, and 23 objects, respectively, we use 10+1, 20+1, and 20+1 slot queries.
A.5 TRAINING SCHEME
In line with established practice, our training process comprises two stages. The first stage is pre- training. during which VAE modules trained on their respective datasets to acquire robust dis- crete intermediate representations. In the second stage, OCL training utilizes the pretrained VAE representations to guide object-centric learning.
Pretraining VAE
Across all datasets, conduct 30,000 training iterations, with validation every 600 iterations. This gives us roughly 50 checkpoints for each OCL model on every dataset. To optimize storage, we retain only the final 25 checkpoints. This approach is uniformly applied across datasets.
For image datasets, the batch size set to 64, while for video datasets, it is 16, a configuration maintained across both training and validation phases. This is consistent for all datasets. We employ 4 workers for multi-processing, applicable to both training and validation. This configuration is the same for all datasets.
We use the Adam optimizer with an initial learning rate of 2e-3, adjusting the learning rate through cosine annealing scheduling, with linear warmup over the first 1/20 of the total steps. This config- uration is standardized across all datasets.
Automatic mixed precision is utilized, leveraging the PyTorch autocast API. In tandem with this, we use PyTorch's 's built-in gradient scaler to enable gradient clipping with a maximum norm of 1.0. This setting uniform across all datasets.
Initializing Slot Queries
The query initializer provides the initial values to aggregate the dense feature map of the input into slots representing different objects and the background.
For image datasets like ClevrTex, COCO, and VOC, we use random query initialization. This method involves learning a set of Gaussian distributions and sampling from them. There are two parameters involved: the means of the Gaussian distributions, which are trainable and of dimen- sion and the shared sigma (a scalar value), which is non-trainable and follows cosine annealing schedule, fixed at 0 during evaluation. Both parameters are of the same dimension as the slot queries.
For video datasets such MOVi-C/D/E, initialization takes different approach. Prior knowledge, such as bounding boxes, is projected into vectors. The bounding boxes are normalized by the input frame dimensions and flattened into 4-dimensional vector corresponding to the number of slots. This vector then processed through a two-layer MLP, using GELU activation, to map the bounding boxes into the channel dimension consistent with the slot queries.
Training the OCL Model
On this stage, we load the pretrained VAE weights to guide the OCL model, where the VAE part is frozen.
For all datasets, about 50 checkpoints per OCL model for each dataset. To reduce storage demands, only the last 25 checkpoints saved. This procedure is applied across all datasets.
50,000 training iterations, validating every 1,000 iterations. This results in
16

---

Published as a conference paper at ICLR 2025
The batch size for image datasets is set to 32 for both training and validation. For video datasets, the batch size is 8 for training and 4 for validation, to account for the increased time steps during video validation. This setting is shared across datasets.
We utilize 4 workers for multi-process operations, consistent across both training and validation phases, and applied uniformly across datasets.
The metrics used for evaluation include ARI, mloU, and mBO, which calculate the panoptic seg- mentation accuracy for both objects and background. These metrics are applied consistently across all datasets.
We the Adam optimizer with initial learning rate of 2e-4, following cosine annealing sched- ule and a linear warmup over the first 1/20 of the total steps. This configuration is standardized across datasets.
We employ automatic mixed precision using the PyTorch autocast API. Alongside this, we use the PyTorch gradient scaler apply gradient clipping, with a maximum of 1.0 for images and 0.02 for videos.
For random query initialization, adjust the value the learned non-shared Gaussian distribu- tion to balance exploration and exploitation. On multi-object datasets, a starts at and decays to by the end of training using cosine annealing scheduling. On single-object datasets, a remains O throughout training. During validation and testing, this value is set to O to ensure deterministic performance.
17


Reviews:
[{"review_id": "gcrg9XQHBy", "reviewer": "Reviewer_a2Kd", "summary": "The paper proposes multi-scale fusion (MSF) for object-centric learning. Specifically, MSF extracts VAE representations corresponding to the same input, downsampled to varying resolutions. The output representations are then upsampled and downsampled as necessary and then combined as a way to augment each set of features. The augmented features can then be fed as input to the general OCL decoder (along with the slot attention outputs and the matching entries from the discrete codebook). This intends to allow for better, more separable, semantically distinct representations of objects at varying scales. The quality is demonstrated with varying metrics corresponding to unsupervised segmentation on the masks that are generated as a byproduct of this OCL pipeline.", "questions": "What downstream tasks could this method help with? It doesn't seem to be a SAM alternative, so what is the potential practical impact?\n\nWhat would change were this applied to higher-resolution images? MSF would seem to have some promise for small objects, but those are essentially all filtered out by the pre-processing.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[S1] The method seems to consistently outperform competitors for the OCL benchmarks, and is even somewhat competitive with the foundation model baseline (DINOSAUR).\n\n[S2] The approach is well-motivated.\n\n[S3] Figure 3 effectively demonstrates, qualitatively, how the MSF accomplishes better OCL, that is, features that are better connected to the objects themselves.", "weaknesses": "[W1] The impact of the work seems limited. The ideas about multi-scale representation and fusion themselves are not novel (see for example FPN in object detection literature), but the implementation and application to this task are. However, the implementation neglects potentially more impactfully redesigns of the primary encoder-decoder pipeline or the slot attention mechanism itself. \n\n[W2] Additionally, there are no results for any downstream tasks. Thus, while interesting, it is hard to project the impact of the work beyond this niche.\n\nMinor: The notation and its interaction with Figure 3 is unpleasant and difficult to follow. In particular, the differences between scale-variant, scale-invariant, and representation before inter-scale fusion are quite slight and seem slightly arbitrary. \n\nMinor: Algorithm 1 would fit in the main paper, and help add clarity in the flow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AWhgp9ldxs", "reviewer": "Reviewer_CqKd", "summary": "This paper addresses Object-Centric Learning (OCL), which aims to capture comprehensive object information by reconstructing inputs using intermediate representations from a Variational Autoencoder (VAE). The focus is on multi-scale training, acknowledging that objects may appear at various scales in images or videos due to changes in imaging distance or intrinsic size differences. The authors propose Multi-Scale Fusion (MSF) applied to VAE representations to guide OCL for both transformer-based approaches and state-of-the-art diffusion models. The method demonstrates promising performance on datasets such as COCO, ClevrTex, and VOC.", "questions": "Overall the paper is good, but the results specially on COCO seems a bit incremental.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) The overall intuition and direction of the paper is good and deal with important problems of Object-Centric Learning.\n2) The multi-scale training makes a lot sense since dealing with objects is an important problem and generally has not been solved fully in the field.\n3) The paper shows good analysis and shows object separability of VAE guidance in Fig3. Fig2 quantitative results also look good.", "weaknesses": "1) Results on real world datasets especially COCO in Table 1 are really incremental. It’s not really clear how much advantage is by adding MSF.\n2) Analysis of varying object sizes can show results on scale understanding better than overall IoU improvement.\n3) Is there any intuition as to why the value of n is 3? Can we do a similar experiment on OpenImages and see if this holds true across datasets? For Openimages, if it’s easier you can try using the smaller subset of open images curated in this paper [1].\n4) Discussion on the area of unsupervised semantic segmentation and object detection using SSL methods should be added. Papers like [1,2,3] should be discussed. \nReferences:\n1) Hyperbolic Contrastive Learning for Visual Representations beyond Objects CVPR2023\n2) VideoCutLER\n3) MOST: Multiple Object localization with Self-supervised Transformers for object discovery", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h4OJ7qUasF", "reviewer": "Reviewer_7kQN", "summary": "This paper addresses the lack of multi-scale object representation in VAE-guided OCL problems by constructing a multi-scale VAE model. This enables the VAE-generated results to represent multi-scale objects, thereby improving performance across various tasks.", "questions": "I am not familiar with this field, so please refer to the weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clear writing and easy to follow.", "weaknesses": "1. I think the paper's novelty is very limited. Although I am not familiar with object-centric learning, I believe that constructing multi-scale VAE features has already been applied in many generative tasks[1, 2]. Therefore, the core innovation of this paper, in my opinion, does not warrant a standalone publication. Can the author tell me about the difference between MSF and other methods?\n\n2. Since I am not familiar with this field, the Area Chair (AC) and the authors may disregard my opinion if it appears to be biased.\n\n[1]Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction, arxiv2404.02905\n[2]SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs, NeurIPS2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j8Q7zPCRFQ", "reviewer": "Reviewer_8Piy", "summary": "This paper is motivated by Object-Centric Learning (OCL) AND introduces a multi-scale fusion scheme to vanilla VAE and improves its representation of different scales of objects. The motivation is convincing, but the technical novelty is limited and multi-scale fusion is a widely used technique in many models and computer vision tasks. Though this paper proposes some specific design modules, like multi-scale features augmentation, they are straightforward and improvements are limited according to the ablation study.\nThough the technical novelty is limited, this paper investigate an interesting view, OCL, to VAE, should be benefit the comunity.", "questions": "Can the proposed method be applied to state-of-the instance segmentation models, like MaskDINO, to improve their results (~50 mask AP) on MS-COCO for instance segmentation.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The MSF is straightforward and easy to understand. According to the experimental results, it works well and improves a strong basline model DINO.\nOCL is an interesting and promising direction and this paper outperform previous OCL methods.\nThe method can be generalized to virous computer tasks and model to improve their representation of objects.", "weaknesses": "According to Sec. 4.1, the improvements from MSF are limited, just 1-2 points in many metrics. \nThe overall presentation is not satisfactory and difficult to understand for readers not familiar with OCL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)