Bootstrapping Top-down Information for Self-modulating Slot Attention
Authors: Dongwon Kim
Seoyeon Kim
Suha Kwak
Venue: NeurIPS 2024 poster (2024)
Keywords: object-centric learning
object discovery
slot attention
Primary area: machine_vision
TL;DR: We propose a object-centric learning framework incorporating a top-down pathway.

Abstract: Object-centric learning (OCL) aims to learn representations of individual objects within visual scenes without manual supervision, facilitating efficient and effective visual reasoning. Traditional OCL methods primarily employ bottom-up approaches that aggregate homogeneous visual features to represent objects. However, in complex visual environments, these methods often fall short due to the heterogeneous nature of visual features within an object. To address this, we propose a novel OCL framework incorporating a top-down pathway. This pathway first bootstraps the semantics of individual objects and then modulates the model to prioritize features relevant to these semantics. By dynamically modulating the model based on its own output, our top-down pathway enhances the representational quality of objects. Our framework achieves state-of-the-art performance across multiple synthetic and real-world object-discovery benchmarks.

Bootstrapping Top-down Information for Self-modulating Slot Attention
Dongwon Kim Dept. of CSE, POSTECH!
Seoyeon Kim
Suha Kwak
Graduate School of AI, POSTECH
(kdwon, syeonkim07, suha, www.kkksstthhhhac.... kr
Abstract
Object-centric learning (OCL) aims to learn representations of individual objects within visual scenes without manual supervision, facilitating efficient and effec- tive visual reasoning. Traditional OCL methods primarily employ bottom-up approaches that aggregate homogeneous visual features to represent objects. How- ever, in complex visual environments, these methods often fall short due to the heterogeneous nature of visual features within an object. propose novel OCL framework incorporating a top-down pathway. This pathway first bootstraps the semantics of individual objects and then modulates the model to prioritize features relevant to these semantics. By dynamically modulating the model based on its own output, our top-down pathway enhances the representa- tional quality of objects. Our framework achieves state-of-the- performance across multiple synthetic and real-world object-discovery benchmarks.
address this, we
1 Introduction
Object-centric learning (OCL) is the task of learning representations of individual objects from visual scenes without manual labels. The task draws inspiration from the human perception which naturally decomposes scene into individual entities for comprehending and interacting with the real world visual environment. Object-centric representations provides improved generalization and robustness 99, and have been proven to be useful for diverse downstream tasks such as visual reasoning simulation 45 and multi-modal learning 24 this context, OCI which learns such representations without labeled data has gained increasing attention.
successful line of OCL builds upon slot attention (30] This method decomposes an image into a set of representations, called slots, that iteratively compete with each other to aggregate image features. Reconstructing the original image from the slots, they are encouraged to capture entities constituting the scene. This simple yet effective method has been further advanced by novel encoder or decoder architectures [3 [9 46 42), optimization technique [88 6). and new query initialization
strategies [18 24]
It is worth noting that all these methods are fundamentally considered bottom-up models, they rely on aggregating visual features without incorporating high-level semantic information from the beginning. This bottom-up approach assumes that visual features within an object are homogeneous and can be clustered in the feature space, which only holds for simplistic objects that can be identified using low-level cues such color (20). In complex real-world scenarios where visual entities of the same semantics exhibit diverse appearances, this homogeneity often breaks down, leading to suboptimal object representations 222 47 Thus, we take an approach different from the previous line of research: introducing top-down information into slot attention, such as object categories and semantic attributes.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Codebook
Top-down
Attention Map
Reconstruction
Decoder
O
Quantization
Slot Attention
MLP
Centering
Channel Modulation
Spatial
Encoder
Input Values
Modulation
Modulated Values
The overall pipeline of our framework. A top-down pathway is introduced into slot attention utilize top-down information. The pathway consists of two parts: bootstrapping top-down knowledge and exploiting them. Firstly, semantic information is bootstrapped from slot attention outputs by mapping slots to discrete codes from learned codebook through vector quantization. Secondly, slot attention is modulated using these codes and its attention maps, transforming it into self-modulating module. Inner activations are modulated across channels with codes and across space with centered attention maps. Slot attention is then repeated with these modulated activations, yielding more representative slots.
Figure 1:
Incorporating top-down information enables slot attention to specialize in discerning objects within specific semantic categories. For instance, identifying vehicles in complex urban environment can be challenging due the diverse and cluttered nature of the scene. Top-down information can guide the model to prioritize vehicle-specific features, such wheels and windows. This inhibits the contributions of irrelevant features when computing slots, and enhances the aggregation of visual features of individual vehicles into slots. Nevertheless, devising such top-down approach is not straightforward since OCL assumes an unsupervised setting without any labeled data, making it hard to identify and exploit the high-level semantics typically obtained from annotated datasets.
We propose novel framework that incorporates top-down pathway into slot attention provide and exploit top-down semantic information; Fig. Fig.[illlster [illustrates of our framework. The pathway consists of two parts: bootstrapping semantics and exploiting them for better representations. Firstly, semantic information is bootstrapped from the output of slot attention itself, by mapping continuous slots to discrete codes selected from a finite learned codebook. Such an approach allows codebook to learn prevalent semantics in the dataset, with each code representing specific semantic concept. Thus, semantic information can bootstrapped without any object-level annotations and used to provide top-down semantic information. Secondly, slot attention is modulated using bootstrapped top-down cues obtained from the first phase, which we call self-modulation. top-down pathway dynamically guides the slot attention by re-scaling its inner activations based on the top-down information. This self-modulation process enables the model to focus on feature sub-spaces where object homogeneity is more consistent, thereby improving its performance in diverse and realistic settings.
this phase, the
Our contributions are threefold:
We introduce method bootstrap top-down semantic information from the output of slot attention, without requiring any object-level annotations. This allows the extraction of high-level semantic cues from an unsupervised learning process.
We propose self-modulation scheme that dynamically guides the slot attention's inner acti- vations to enhance object representation, successfully incorporating the top-down semantic cues extracted.
By integrating the proposed top-down puthway into slot attention, we demonstrate that the performance object discovery largely improved on various OCL benchmarks.
2 Related Work
Object-centric learning OCI aims to learn representations of individual objects within an image.
The 'object-centric' dimension is orthogonal to the conventional representation learning which learns representations independent of the composition of the image. The structured nature of

---

object-centric representations offers improved generalization 19. making it valuable for various
applications, including visual reasoning [44, dynamics simulation 45 and multi-modal learning 23 [2]]. A foundational method in this field is slot attention 30 which introduced simple yet
effective framework that employs a competitive attention mechanism between slots. Following slot attention, many recent works have proposed improvements by introducing novel encoder or
decoder formulations [33] 19 46 35), optimization techniques [T8 6.. additional slot refinement modules [37 25 3) and expansions to video modality (27 II 36). These methods are primarily
bottom-up models, while our approach proposes to bootstrap and incorporate top-down information.
Incorporating top-down information The human visual system perceives scenes by leveraging both top-down and bottom-up visual information 44 [ Top-down information represents task- driven contextual cues, such as high-level semantics and prior knowledge about the scene. In contrast, bottom-up information is derived directly from the sensory input. Inspired by this dual- processing mechanism of the human visual system, several studies 34 model this approach within deep learning, achieving significant improvements across various tasks. Our work follows in similar direction, specifically focusing on introducing top-down information into the representative OCL method, slot attention. By incorporating top-down semantic and spatial information, we aim to enhance the performance of slot attention in diverse visual environments, addressing the limitations of previous bottom-up methods
48 have attempted to
Discrete representation learning Discrete representations within neural networks are considered effective for modeling discrete modalities [50 and tackling generation tasks [3 28
Particularly, the pioneering work, VQ-VAE (38), introduced method for learning discrete latent representations through vector quantization. This model uses a discrete codebook, where the encoder maps input data to discrete codes using nearest-neighhor lookup and the decoder reconstructs the input from the codes. Another notable approach is the Gumbel-softman trick [17 32), which provides a differentiable approximation of sampling from categorical distribution. Recent advancements have aimed to incorporate a more sophisticated formulation of codebooks (49] or improve codebook utilization [T6] to better handle discrete representations. Recent work by Wallingford et al. 40) is related to our research in terms of using vector quantization for segmentation task. However, our method differs in that the quantized codes are used to modulate bottom-up slot attention, while in Wallingford et al.
[40] the codes are used solely for segmentation labeling.
3 Method
We propose an OCL framework that incorporates top-down semantic information, such as object categories and semantic attributes, into slot attention through lop-down pathway. Fig. Dog-[illlnne illustrates the overall pipeline of our framework. Firstly, slot attention is applied to visual features extracted from an image encoder to output slots (Sec. 3.1). Then, top-down pathway leverages the slots to identify semantics in the input image and modulate slot attention. The pathway consists of two parts: bootstrapping top-down semantic information from a learned codebook and attention maps (Sec. 3.2
and modulating the inner activations of slot attention with this semantic information (Sec. 3.3
During the self-m iffdlloullaion stage, slot attention is repeated with the modulated activations, resulting in more representative slots.
3.1 Slot Attention
Slot attention is recurrent bottom-up module that aggregates image features into slots through an iterative attention process, where each of the resulting slots represent an entity in the input image. Within our framework, these slots are used to bootstrap top-down information in the later stage (Sec. 3.2
The module takes in the initial slots, S RKDD and visual features extracted from an image encoder, x DEs. The initial slots are obtained by sampling K vectors from learnable Gaussian distribution using reparameterizztion trick. The slots S = [s1, $2..... sk] E I2 K D where each represents an individual object the image, are computed by iteratively updating the initial slots T times as
S ST where s slot_ slot_attn(x, S*)
(I)
In each iteration, the slots attend to the visual features, refining their representations through series of attention-based updates. Let g(-), k(-), and v(-) represent linear projections from dimension d to
3

---

d,. Then, the attention map A E RKxN is computed as
A, P Σ ePu
where P= 1((-)1)((x) dd
Unlike the original attention introduced in transformer [39] which normalizes across the keys, the attention in slot attention is normalized across the slots. Such a distinct normalization scheme makes the slots compete with each other to aggregate the visual features, encouraging each slot to represent a distinct object in the scene. Then, the computed attention map A normalized across the rows into A and used to update the slots as
S Supdata (U, where U Au(xx)
such that U is the weighted sum of the visual features ux).. The function fupdate processes U with consecutive GRU (7) and Multi-Layer Perceptron (MLP) and residually sums it to the previous slots, S to produce the updated slots, S For further details, refer to Locatello et al. [30].
3.2
Bootstrapping Top-down Information
Our idea to bootstrap top-down information without annotations is based on our observation that the slots S, which are outputs of the nottom-up attention module, contain rough semantic information about objects. We leverage this coarse information to bootstrap both the semantic and spatial top- down information about the objects coarsely represented by the slots. Top-down semantic information pertains to the specific semantic categories or attributes of the objects (whar), while top-down spatial information indicates the locations or regions within the image where these objects are located (where). Incorporating such knowledge can guide slot attention to focus on the features most relevant to the objects expected to appear, enabling it to accurately capture objects that are obscured or have high intra-object variance, such as people with different hairstyles clothing.
Firstly, we extract the "what" information from the slots S using Vector Quantization (VQ), which maps each slot to one of the semantic concepts learned throughout training. Specifically, each slot S₁ is mapped the nearest code in a finite codebook. The mapped code c e RD is considered top-down semantic cue for the slot sg. Formally, this quantization process can be written as
C1, C2..... Cgl R Fx with size E.
C arg min ||0k - cll2.
eEC
Since the arg min operation is non difffeeeeettible, we use the trriigh-tthouuge estimator 2 for backpropagation. During training. the codebook learns store distinct semantic patterns recurring within the dataset by quantizing continuous slot embeddings into limited number of discrete embeddings. Thereby, each code can act as automatically discovered top-down semantic information.
Secondly, we obtain the "where" information from the attention maps of the last layer of slot attention. For each slot 8k, the k-th row vector of the attention map A, denoted as ag E R used to aggregate visual features and update 8y. This attention map provides useful spatial prior information about where each extracted top-down semantic information located in the image.
3.3 Self-modulating Slot Attention
In the original slot attention (Sec. 3.I.. p. the slot updates are driven purely by visual features extracted from the input without incorporating higher-level semantic information that can provide additional context. To address this limitation, we introduce iofuuaouut slot attention, which modulates the computation of the slot updates based on the top-down information obtained in the bootstrapping stage (Sec. (3.2). This bootstrapped top-down information is used dynamically amplify or inhibit specific channel dimensions or regions of the value-projecte parameters unchanged. Formally, self-modulating slot attention can be represented by conditioning slot attention with the vector quantized slots cj, and their corresponding slot-wise attention map ag:
visual features, while keeping the model
$ == --ST. where St+1 = slot_ poot_attnxx.. S¹ [c]]] 1m [a,]k]]]
(5)

---

Algorithm Self-modulating Slot Attention. The module inputs visual features extracted from an encoder, initial slots sampled from a learned Gaussian distribution, and K vector quantized slots and their corresponding K slot-wise attention maps output from the original slot attention. The number of iterations, T, set to three.
Input: Visual features x, initial slots $ vector quantized slots [c: A and slot-wise attention maps [a.] Output: Slots after -iteeration of self-modulating slot attention S"
I: for 1 to K in parallel do
2: mi MLP(c))
// Compute channel -wise modulation vector // Compute spatial -ise modulation vector Compute modu Indulation map
3: mi 1 1+ (ak ax)
4: Mr m m
for to do
6: A = coftmax(1(5 (xx) /VDa)
7: S = fupdaro (122, U2 u...... uk). 1" where un = Au(Ma O ox))
// Modulated slot update
8: return S
where S represents the slots from the self moflllttdu slot attention, different from the slots of the original slot attention denoted by S. Note that the original slot attention (Sec. 3.I and molf--oouulaing slot attention share parameter weights and initial slots, such that S 50
Specifically, we modulate slot attention with a modulation map M2 E RN computed from ej and ag. Each element of the modulation map represents the relevance score between the corresponding visual feature element and the top-down information of the expected object. This modulation map can be used to guide the update of each slot S₁ (Eq. 3) by prioritizing specific value elements with the high relevance scores. In the self-modulating slot attention, computation of the slot update U replaced with:
[1, U2 .......,........................ ux] ERRXD u Am o v(x);
(6)
where @ represents Hadamard product. Such re-scaling of the value features with the modulation map ensures that the specific channel dimension or regions contribute more to the update of each slot, based on the semantics or locations encoded in bootstrapped top-down information.
The modulation map M& is computed by taking the outer product between channel -oinne and spatial- wise modulation vectors, which are predicted using ct, and ag., respectively:
Mi mg m E R.
For predicting channel- modulation vector m quantized slot cj is used, which tells "what" the object appearing in the image is. The channel-wise scaling is designed to enforce the model to focus certain feature subspaces closely correlated to the semantic concept identified. Specifically, channel-wise modulation vector mf. can be obtained by feeding quantized slot oj to the MLP, which represented as:
m MLP((() R..
(8)
The spatial-wise modulation vector m can be obtained by further processing the attention map of each slot, which contains the top-down information on the location of the semantic concept:
m + (as - a.) E R
(9)
where ag for the average the attention score of as. Using an attention map as for modulation will make all values down-scaled, while some regions likely to contain the object should be highlighted for effective incorporation of the spatial top-down information. Thus, we use the attention map shifted to have mean value of I for the spatial-wist modulation map.
3.4 Training
Slot attention trained within an autoencoding framework, using a decoder that reconstructs visual features output by the image encoder (35 33 or the original image (30) from the slots. In this paper, we choose the visual feature reconstruction as our training objective since known to provide more robust training signals for real-world datasets [33]. We also employ vector quantization objective
5

---

for the codebook C only, which thereby learns to minimize the mean- mrean-squared error between the slot and the sampled codes. The reconstruction objective Lraan and vector quantization objective Lvg are given by
recon ||Dec(S; - 211
Lve ||g 1
(10)
where sg(-) represents stop gradient operation and C* c ..... ckl E RX For the decoder, we utilized the autoregressive slot decoder 35 33 The reconstruction objective ensures that the learned slot representations capture essential information about the objects in the scene, while the vector quantization objective encourages the codebook to capture recurring semantic concepts in the
dataset.
4 Experiments
4.1 Experimental Settings
Datasets To verify the proposed method in diverse settings, including synthetic and authentic datasets, we considered four object-centric learning benchmarks: MOVI-C (15), MOVI-E (13). PASCAL VOC
2012 [14, and MS COCO 2017 [29] MOVI-C and MOVI-E are synthetic datasets, adopted for validating our method in relatively simple visual environments. MOVI-C contains 87,633 images for training and 6,000 images for evaluation, while MOVI-E contains 87,741 and 6,000, respectively. To evaluate the proposed model in real-world settings, we leverage the VOC and COCO datasets. Following DINOSAUR B33, we use the trainaug variants, containsing 10.582 training images, for VOC dataset. For the evaluation, we use the validation split containing 1,449 images. The COCO dataset consists of 118287 training images and 5,000 images for evaluation. While the VOC dataset includes some images with single object, images of the COCO dataset always contain 2 or more objects, making it the most challenging. MOVI datasets are licensed under apache license 2.0 and COCO is licensed under CC-BY-4.0.
Metrics We evaluate our method with three metrics: foreground adjusted random index (FG-ARI), mean best overlap (mBO), and mean intersection over union (mloU). The FG-ARI is the ARI metric computed for foreground regions only (objects), which measures the similarity between different clustering results. The mBO and mIoU are both IoU-based metrics, computed for all regions including the background. The mBO computes the average IoU between ground truth and prediction pairs, obtained by assigning each prediction to the ground truth mask with the largest overlap. The mloU is computed as the average IoU between ground truth and prediction pairs obtained from Hungarian matching. For COCO and VOC, mBO' and mBO indicate the mBO metric computed using semantic segmentation and instance segmentation ground truth. We use the instance segmentation ground truth for other metrics. Following previous work (33) the internal attention maps of the autoregressive decoder are used as the mask prediction results of the slots.
Implementation details To assess the effectiveness of the proposed top-down pathway, our model implemented based on DINOSAUR [33), a representative slot-based OCL method. For the encoder and decoder, we use DINO (5] pretrained ViT-B/16 [00 and an autoregressive transformer decoder 35 33 respectively. The model trained using an Adam optimizer 26 with an initial learning rate of 0.0004, while the encoder parameters are not trained. The number of slots K is set to 11, 24, 7, and 6 for MOVI-C, MOVI-E, COCO, and VOC, respectively. The codebook size E is set to 128 for synthetic datasets (MOVI-C and MOVI-E) and 512 for authentic datasets (COCO and VOC). The model trained for 250K iterations on VOC and for 500K iterations on the others. For the ablation study and analysis, models are trained for 200K iterations on COCO, as this was enough to reveal overall trends given limited computational resources. Full training of the model takes 26 hours using a single NVIDIA RTX 3090 GPU.
Codebook size E' selection The performance of the proposed top-down pathway depends on code- book size E (Sec. 4.4), necessitating principled selection method. We determine E automatically by monitoring the perplexity of code usage distribution during training. requiring only the training set without validation data. Perplexity- -te exponent of entropy intropy-indicates indicates how uniformly the codes are being used. While perplexity typically increases with codebook size, it plateaus when exceeds the number of distinct semantic patterns in the data, some codes become unused [16] 49 28 To find the optimal size, we start with = 64 and double it until the perplexity plateaus after 250K iterations. For example, COCO, the perplexity when the codebook size is 256, 512, and 1024 are

---

Table I: Comparison with DINOSAUR 33
synthetic datasets: MOVI-C 15] and MOVI-E [15
We include both the reported and reproduced performance of DINOSAUR for fair comparison.
MOVI-C
MOVI-E
Method
FG-ARI
mBO
mloU
FG-ARI
mBO
mloU
DINOSAUR
55.7
42,4
DINOSAUR reprod
54.7++4.1
41.9+1188
41.0+
53.8+2.1
3.,
33.6+1.99
Ours
58.915.11
46.8.1244
45.912.5
59.7.13.1
39.3118
38.3.1199
Table 2: Comparison with DINOSAUR include both the reported and reproduced performance of DINOSAUR for fair comparison.
33
on real-world datasets: COCO 29 and VOC
[1.. We
COCO
VOC
Method
FG-ARI
mBO
mBO
mloU
FG-ARI
mBO'
mBC
mloU
DINOSAUR
34.1+1.0
31.6+H.77
39.740.9
24.84-2.2
44.0+1.9
51.2+1.9
DINOSAUR reprod 33
34.1=0.9
31.4+0.5
39.5+0.1
29.4
27.0+:3,2
41,2#3.4
48.2+3.8
39.0+3.66
Ours
37.=0.00
33.01:0.3
40.340.2
31.2:0.3
26.7::4.77
43,942.6
51.022.5
42.0:2.8
176.9, 253.9, and 242.8, respectively, where 512 was chosen as the final size. This procedure enables efficient hyperparameter selection using only training data, eliminating the need for validation set tuning. Following this approach, we set E = 128 for synthetic datasets (MOVI-C and MOVI-E) and E = 512 for real-world datasets (COCO and VOC).
4.2 Quantitative Analysis
DINOSAUR 333 is the first successful OCI, method that scales slot attention to real-world datasets by introducing the use of a self-supervised image encoder [5), an autoregressive decoder, and feature reconstruction objective. Notably, DINOSAUR uses the vanilla slot attention mechanism without modifications, making it perfect baseline for validating the effectiveness of proposed top-down pathway. Thus, we adopt DINOSAUR as a baseline and compare its performance with and without proposed method. For fair comparison, report both the reported and reproduced performance of DINOSAUR.
Tab. [ demonstrates that incorporating the proposed top-down pathway into DINOSAUR largely improves performance every metric. Specifically, our method improves FG-ARI by 5.9 on MOVI-E, which is the most challenging synthetic dataset. In Tab. performances authentic datasets, COCO and VOC, are reported. Our method largely surpasses the reproduced baseline on most metrics. The only metric for which our method does not show improvement is the FG-ARI on VOC. We hypothesize that this is because VOC images frequently contain single objects only, and FG-ARI is computed solely with foreground pixels so that the performance is less affected by the top-down information.
Tab. 3 presents a comparison between our method and recent state-of thetthe-rrt methods. It is notable that our proposed method achieves competitive performance even recent methods using advanced diffusion-based decoders [19 46). Moreover, our approach focuses on incorporating top-down information into slot attention, which orthogonal to the line of work advancing decoders to provide better training signals to slot attention.
4.3 Qualitative Results
Codebook visualization To validate whether the proposed codebook learns meaningful semantic concepts, we present visualizations of the codebook in Fig. 2 The index of the code and the mask prediction obtained from the slots modulated by the code presented together, revealing the semantic entity each code represents. Visualization demonstrates that the codebook successfully discovers and stores distinct semantic concepts without using any annotations. Moreover, the codes are mapped to objects with various appearances and layouts, which demonstrates that the codes learn high-level semantic information and not low-level structural or positional information.
Prediction visualization In Fig. 3 we visualize the original image, mask predictions, and slot atten- tion maps A before and after self-modulation. Results demonstrate that the modulation dynamically
7

---

Table 3: Comparison with state of the arts +
COCO
29
VOC 14
MOVI-C 15 and
MOVI-E 15..
Performances of Slot Attention on MOVI-C and -E are reproduced by 33 and that of
SLATE by 19
Those COCO and VOC are from 46
COCO
VOC
MOVI-C
MOVI-E
Method
FG-ARI
mBO'
FG-ARI
mB0
FG-ARI
mBO'
FG-ARI
mBO
Slot Attention
30
214
17.2
12.3
24.6
43.821.33
26.2+110 39.410.8 42,4
45.0%L77 46.0613.3
24.0:11.2
SLATE [35
32.5
29.1
15.6
35.9
49.54=144
30.24:1.7
DINOSAUR 33
34.1=1.0
31.6:20.7
24.8#22.
44.0+11.
55.7
Rotating Features
11
40.7++0.1
SlotDiffusion
46
37.2
31.0
17.8
50.4
30.2
LSD 19
35.0
30.4
52.0
45.6
52.2
39,0
Ours
0..4
33.3+:0.3
26.7=4.7
43.9:12.6
58.9115.1
46.8:12.4
59.7433.1
39.3:11.8
Code 124
Code 496
207
Code 341
Code 352
Code 359
Code 343
The results show that the codebook learns to capture recurring semantic concepts the dataset, such as 'pizza' (code 124), 'sign' (code 496), "clock" (code 235), 'zebra' (code 207), 'motorcycle' (code 341), 'surfer' (code 352), 'dog' (code 359), and 'skier' (code 343).
Figure 2: Visualization of the codebook C COCO 29
refines the attention maps, depending how well they have captured the scene. For example, when the attention map well-structured but coarse, the modulation process refines the boundaries of the attention map without changing the overall layout (first row). However, if the attention maps fail to delineate objects, the modulation process recomposes the attention maps to differentiate objects (second to fourth row). By providing top-down semantie and spatial information via self- self-modulation, the attention maps are enhanced to capture the object within complex real-world environments.
4.4 In-depth Analysis
Effect of codebook size In our framework, vector quantization maps slots to distinct top-down semantic information stored in the codebook, as shown in Fig. 2 In Tab, we report the performances across different codebook sizes, E. The results show that codebook size too large (E 1024) or small (E < 256) leads to performance degradation. When the codebook size is too small, the codes cannot sufficiently learn distinct semantic information, which degrades the quality of the bootstrapped top-down semantic information. On the other hand, codebook size too large may cause the codes to capture irrelevant details such as appearance variance or positional information rather than meaningful semantic concepts. However, we determine the optimal codebook size automatically using the perplexity of codebook usage during training (Sec. (4.). eliminating the need for extensive hyperparameter tuning using validation split and benchmark metrics.

---

Image
Prediction
Slot Attention Map Before and After Self-Modulation
Defore
After
Before
After
Before
After
Before
After
Figure 3:
Visualization of the input image, predicted object mask, and attention maps of slot attention
before and after self- aeff-modultiin on COCO 29
Lighter the color, higher the attention score.
Table 4: Comparison between different code-
Table Comparison with DINOSAUR 3
uS-
book sizes on COCO [29].
ing six iterations in slot attention on COCO 29
128
256
512
1024
FG-ARI
mBO
FG-ARI
33.1
32.6
37.3
DINOSAUR 6-iter
28.5
28.2
mBO
30.2
30.5
32.7
29.8
Ours
37.3
32.7
Impact of increased iterations Since our method requires repeating slot attention with self- modulation, we investigate whether our performance improvement simply comes from the increased iterations of slot attention. In Tab. 5 we compare the results of our model with the DINOSAUR baseline model using six iterations of slot attention, which twice as many iterations as the default setting. Notably, both of iterations. The results show that the DINOSAUR model with six itcrations actually performs worse compared to the default 3 iterations. This indicates that merely increasing the number of iterations does not guarantee improvement. Our method's 's superior performance is thus attributed to the eeff-modulation mechanism rather than the increased number of iterations.
model and the DINOSAUR 6-iteration model leverage the same number
Computation overhead of top-down pathway While our model requires one more forward pass for slot attention, the additional computation cost is negligible. In DINOSAUR, slot attention accounts for only 0.64% of the total FLOPs, compared to 71. .266% for encoder and 28. 10% for the decoder. Consequently, our model requires 47.62 GFL.OPs versus 47.32 GFL .OPs of DINOSAUR- increase below 1 %.. In practice, processing the entire COCO 2017 val split on a single NVIDIA RTX 3090 GPU takes 71.4 seconds for our model compared to 70.5 seconds for DINOSAUR. demonstrating minimal impact on inference time.
mere
Codes representing broader semantics While most codes in our codebook consistently represent single object categories, we observed interesting cases where codes capture broader concepts, as shown in Fig. 4 Some codes are trained to represent supercategories for instance, grouping different animal species (code 468) or various human parts into shared codes (code 328). This suggests codebook can flexibly adapt to different levels of semantic abstraction when beneficial. We also discovered an edge case where certain codes (e-g., code 223) specialize capturing top-left patches of images. This behavior appears be influenced by the autoregressive decoding process, which

---

Code 468
Code 328
223
236
Code 133
Code 508
The results show that the codebook learns capture broader semantics other than single object categories, such supercategory (code 468, 328), top-left patch (code 223), and background (code 236, 133, 508).
Figure 4: Visualization of the codebook C on COCO 29
must reconstruct the top-left patch first without surrounding context. However, these specialized positional codes are rare (1-2 out of 512 codes) and have minimal impact on overall performance. Additionally, we found that certain codes specialize in capturing background elements common in natural scenes. For example, code 236 represents sky regions, code 133 captures sports fields, and code 508 represents crowd scenes.
Ablation study In Tab. 6 we present an ablation study of our method on COCO for channel-wise modulation, vector quantization, spatial-wise modulation, and at- tention map shifting. In the first row, the result with no modules presented, which equivalent to the baseline DINOSAUR model. The last row is the per- formance of using all four modules, equivalent to our proposed method. The second and third rows for the ablation of vector quantization and channel-wise wise mod- ulation, demonstrating that incorporating top-down se- mantic information significantly improves performance. In the fourth and fifth rows, the ablation of attention map shifting and spatial-wise modulation is presented. The results show that both operations eritical for the effective exploitation of top-down spatial information.
Table 6: Ablation studies on COCO dataset for each module consisting of the proposed top-down pathway: channel-wise modula- tion (m²), vector quantization (VQ), spatial- wise modulation (m²), and shifting atten-
tion map (shift).
m
VQ m shift
FG-ARI
mBO
DINOSAUR
34.8
30.5
36.3
32.3
35.1
32.5
35.2
31.7
/ V
36.0
31.9
37.3
32.7
5 Conclusion
In this paper, we introduced an OCL framework that incorporates top-down information into the slot attention mechanism through a lop-down pathway. In this pathway, the output of the slot attention used to bootstrap high-level semantic knowledge and rough localization cues for existing objects. Using the bootstrapped top-down knowledge, slot attention is modulated to focus features most relevant to the objects the scene. Consequently, by incorporating the proposed top-down pathway into slot attention, achieved state-of-thed -hertt-rrt performance various OCL benchmarks, including challenging synthetic and authentic datasets.
Limitation
The proposed top-down pathway has limitation in that its overall performance relies on the quality of the codebook learned during training. As shown in Tab. an incorrect choice of codebook size can result in the codes failing to learn distinct semantic concepts capturing irrelevant details. While we mitigate this limitation through perplexity-basee automatic codebook size tuning, the more principled codebook design that eliminate the need for pre- pre-defined hyperparameter, such as dynamically expanding codebook during learning [40), will be promising future research direction.
10

---

Acknowledgments and Disclosure of Funding
This work supported by IITP grants funded by the Korea government (MSIT) Artificial Intelligence Graduate School Program (POSTECH); RS-2024-00457882 AI Research Hub Project; RS-2024-00509258 Global Al Frontier Lab).
References
[1] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In JEEE Conference Computer Vision and Pottern Recognition (CVPR), pages 6077-6086, 2018.
[2] Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint /38.:33308.3433 2013.
O. Biza, S. Van Steenkiste, S. Sajjadi, G. F. Elsayed, A. Mahendran, and T. Kipf. Invariant slot attention: Object discovery with slot-centric reference frames. 2023.
[4] T. J. Buschman and E. Miller. Top-down versus bottom-up control of attention the prefrontal and posterior parictal cortices. science, 31565820;1860-1888 2007.
[5] Caron, H. Touvron, I. Misra, Jégou, Mairal, P. Bojanowski, and A. Joulin. Emerging Properties in Self-Supervised Vision Transformers. In IEEF International Conference on Computer Vision (ICCV), 2021.
[6] M. Chang, T. Griffiths, and S. Levine. Object representations as fixed points: Training iterative refine- ment algorithms with implicit differentiation. In Conference Neural Information Processing Systems (NeurlPS), 2022.
[7] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1412.3555, 2014.
Corbetta and G. Shulman. Control of goal- coal-directed and stimulus-drive attention in the brain. Nature reviews neuroscience, 3(3):201-22 215, 2002.
[9] A. Dittadi, Papa. M. De Vita. B. Schölkopf, Winther, and F. Locatello. Generalization robustness implications in object-centric learning. In International Conference on Machine Learning (ICML). 2022.
[10] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbomn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. 2021.
[11] G. Elsayed, A. Mahendran, S. Van Steenkiste, K. Greff, M. C. Mozer, and T. Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. In Conference Neural Information Processing Systems (NeurIPS), volume 35, pages 28940-28954 2022.
[12] M. Engelcke, A. Kosiorek, Jones, and Posner. Genesis: Generative scene inference and sampling with object-centric latent repressttttts,, International Conference on Learning Representations (ICLR), 2019.
[13] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[14] M. Everingham, I. Van Gool, C. K. T. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal- html.
network. wawp:/lnnggee///ooooo/22ooooooo
[15] K. Greff, F. Belletti, L. Beyer, C. Doersch, Y. Du, D. Duckworth, Fleet, D. Gnanapragasam. F. Golemo, C. Herrmann, et al. Kubric: scalable dataset generator. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3749 3761, 2022.
[16] M. Huh, B. Cheung, Agrawal, and P. Isola. Straightening out the straight-through estimator: Overcoming optimization challenges in vector quantized networks. In International Conference on Machine Learning (ICML). PMLR, 2023.
[17] E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. In International Conference Learning Representations (ICLR), 2017.

---

[18] B. Jia, Y. Liu, and S. Huang. Improving object-centric learning with query optimization. In International Conference on Learning Representations (ICLR), 2023.
[19] J. Jian,, F. Deng, G. Singh, and S. Ahn. Object-centric slot diffusion. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
[20] J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2901-2910, 2017.
[21] I. Kakogeorgiou. S. Gidaris, K. Karantzalos, and N. Komodakis. Spot: Self-training with patch-order permutation for object-centric learning with autoregressive transformers. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 22776-22786, 2024.
[22] L.. Karazija, I. Laina, and Rupprecht. Clevitex: texture-rich benchmark for unsupervised multi-ohject segmentation. In Conference on Neural Information Processing Systems (NeurIPS). 2021.
[23] D. Kim, N. Kim, and S. Kwak. Improving cross-modal retrieval with set of diverse embeddings. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2023. doi: 10.1109/ vvpr52729.2023.022 2023,0223,. URL htttp://dx doi. org/ 10. LL09/CVPR52229... 2023. .22433
[24] D. Kim, N. Kim, C. Lan, and S. Kwak. Shatter and Gather: Learning Referring Image Segmentation with Text Supervision. In IEEE International Conference Computer Vision (ICCV), 2023.
[25] J. Kim, J. Choi, H.-J. Choi, and S. J. Kim. Shepherding slots to objects: Towards stable and robust object-centric learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 19198 19207, 2023.
[26] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6950. 2014.
[27] T. Kipf, G. F. Elkayed, A. Mahendran, A. Stone, S. Sabour, G. Heigold, R. Jonschkowski, A. Dosovitskiy, and K. Greff. Conditional object-centric learning from video. In International Conference on Learning Representations (ICLR), 2022.
[28] D. Lee, C. Kim, S. Kim, M. Cho, and W.-. Han. Autoregressive Image Generation using Residual Quantization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[29] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and L. Zitmick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.
[30] F. Locatello, D. Weissenhom, T. Unterthiner, Mahendran, G. Heigold, Uszkoreit, A. Dosovitskiy, and T. Kipf. Object-centric learning with slot attention. In Conference on Neural Information Processing Systems (NeurIPS), volume 33, pages 1525-11538, 2020.
[31] S. Löwe, P. Lippe, F. Locatello, and M. Welling. Rotating features for object discovery. Conference on Neural Information Processing Systems (NeurlPS), volume 36, 2024.
[32] C. J. Maddison, A. Mnih, and Y. W. Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations (ICLR), 2017.
[33] M. Seitzer, M. Horn, Zadaianchuk, D. Zietlow, Xiao, C.-J. Simon-Gabriel, T. He, Z. Zhang, B. Schölkopf, T. Brox, and F. Lacatello. Bridging the gap to real-world object-centric learning. In International Conference on Learning Representations (ICLR), 2023. TRI..ttpss: httpss://ppereriieeee net. forum?id=b9tus
[34] B. Shi, T. Darrell, and X. Wang. Top-down visual attention from analysis by synthesis. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 2102-2112, 2023.
[35] G. Singh, F. Deng, and S. Ahn. Illiterate dall-e learns to compose. In International Conference on Learning Representations (ICLR), 2021.
[36] G. Singh, Y.-F. Wu, and S. Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. volume 35, pages 18181-18196, 2022.
[37] G. Singh, Y. Kim, and S. Ahn. Neural systematic binder. In International Conference on Learning Representations (ICLR), 2023.
[38] A. Van Den Oord, O. Vinyals, al. Neural discrete representation learning. 2017.
12

---

[39] A. Vaswani. N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. In Conference Neural Information Processing Systems (NeurIPS), 2017. 2017.
[40] M. Wallingford, A. Kusupati, A. Fang, V. Ramanujan, A. Kembhavi, R. Mottaghi, and A. Farhadi. Neural radiance field codebooks. 2023.
[41] X. Wang, R. Girdhar, S. X. Yu, and I. Misra. Cut and learn for unsupervised object detection and instance segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3124-3134, 2023.
[42] Z. Wang, M. Z. Shou, and M. Zhang. Object-centric learning with cyclic walks between parts and whole. In Conference on Neural Information Processing Systems (NeurIPS), volume 36, 2023.
[43] N. Watters, L.. Matthey, C. P. Burgess, and A. Lerchner. Spatial broadcast decoder: simple architecture learning disentangled representations in vaes. 2019.
[44] T. Webb, S. Mondal, and J. D. Cohen. Systematic visual reasoning through object-centric relational abstraction. volume 36, 2024.
[45] 7. Wu, N. Dvorikk K. Greff, T. Kipf, and A. Garg. Slotformer: Unsupervised visual dynamics simulation with object-centrie models. 2023.
[46] Z. Wu, J. Hu, W. Lu, I. Gilitschenski, and Garg. Slotdiffusion: Object-centric generative modeling with diffuusin models. In Conference on Neural Information Processing Systems (NeurIPS), volume 36, pages 50932-50958, 2023.
[47] Y. Yang and Yang, Promising or elusive? unsupervised object segmentation from real-world single images. In Conference on Neural Information Processing Systems (NewrIPS), volume 35, pages 4722-4735. 2022.
[48] 7. Yin, P. Wang, F. Wang, X. Xu, H. Zhang, H. Li, and R. Jin. Transfgu: a top-down approach to fine- tne-grained unsupervised semantic segmentation. In European Conference on Computer Vision (ECCV), pages 73 89. Springer, 2022.
[49] J. Yu, X. J. Y. Koh, H. Zhang, R. Pang, J. Qin, A. Ku, Y. Xu, J. Baldridge, and Y. Wu. Vector-quantized image modeling with improved vggan. In International Conference on Learning Representations (ICLR), 2021.
[50] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
13

---

Appendices
A Additional Qualitative Results
Fig. A6 demonstrates the additional visualizations of original image, mask predictions, and slot attention maps before and after self-modulation on the COCO dataset.
B Details of Autoregressive Decoder
The proposed top-down pathway is implemented based on the DINOSAUR baseline [3] using an autoregressive slot decoder. Singh et al. (35] first proposed to such autoregressive decoding scheme for slot attention training. Autoregressive decoder is known to provide better training signal leading to improved performance, compared to the MLP-based broadcast decoder 43 used by the slot attention originally. Autoregressive decoder is the simple variant of the transformer decoder (39], which takes input visual feature x and slots S. The decoder consists of multiple decoding blocks. Let multi-head attention be denoted as MHA Q; K; V), where Q. K, and V is for query, key, and value, respectively. Then, the decoding block can be represented
DecBlock(x; S)
FFN &; S;))
(11)
X MHA BOS) X X X X
(12)
where FFN denotes feedforward layer with MLP and residual connection, M+) represents multi-head self-attention with causal masking, and X|Bos] represents the visual feature sequence with learnable [BOS] token appended at the start of the sequence. By using multiple decoding blocks, we can compute the autoregressive reconstruction of the visual feature x, which is consequently used for the computing reconstruction objective. Following DINOSAUR, decoder with four decoding blocks. The number of heads for multi-head attention set to 8.
autoregressive
C Additional Experiments
Top-down pathway without DINO and autoregressive decoder We have mainly built a top-down pathway with the DINO (5) pretrained weight and autoregressive decoder (framework proposed in DINOSAUR 33 since it the best working object-centric learning framework in a real-world setting. To see if these settings
essential for the top-down pathway, we have implemented our self-modulation technique with the orig- inal slot attention setting [30], which includes training encoders from scratch and using an image reconstruction objective with spa-
Slet attention
tial broadcast decoder [43]. Tab. A7 summarizes the results of the
CLEVR6 dataset. We observe significant improvement in mBO, showing that our self-modulation technique is applicable slot tention and provides complementary benefits. Although FG-ARI decreased, mBO is considered more robust when evaluating model performance 22 12
Slot attention wi top-down puthway
21 33). We have also included qualitative re- sults in Fig. A5 which demonstrates that self modulation markedly improves segmentation. These results indicate our method's is effec- tiveness with different encoder configurations and training objec-
Figure A5: Visualization of the predicted object mask on CLEVR6 20
tives.
Comparison MaskCut [41] While object-centric learning (OCL) and unsupervised instance segmentation share the goal of discovering objects without supervision, their ultimate objectives differ. OCL aims learn object-wise representations that support downstream tasks requiring compositionality and systematic generalization, whereas unsupervised instance segmentation focuses primarily on obtaining accurate object masks. Nevertheless, we can directly compare methods from both tasks on their object discovery capabilities. We evaluate our method against MaskCut, pseudo-mask generation algorithm underlying CutLER [4], on the COCO dataset. As shown in Tab. A8 our method significantly outperforms MaskCut across all metrics, demonstrating its effectiveness for object discovery even when compared to specialized unsupervised segmentation approaches.
14

---

Table A7: Comparison with slot attention on CLEVR6
30
Table A8: Comparison with MaskCut 41 on
COCO
FG-ARI
mBO
FG-ARI
mBO,
mloU
Slot atto (reprod.) 30
98.7
21.2
MaskCut 41
31.5
28.9
26.7 31.2 0.3
Slot attn (reprod))
+ lop down pathway
88.7
61.9
Ours
37.4
33.0=
Table A9: Comparison with SPOT
21 on COCO 29). VOC 14, MOVI-C 15 and MOVI-E [5]
COCO
VOC
MOVI-C
MOVI-E
Method
FG-ARI
mBO'
FG-ARI
mBO'
FG-ARI
mBO'
FG-ARI
mBO'
SPOT
21
36.6 10.3
34,7 20.1
19.4 10.7
48.1 10.4
52.1 23.3
47.0 =12
56.4 14.1
39.9
#1.1
Ours
37.4.0.0
33.3.10.3
26.7
43.912.6
58.915.1
46.812.4
59.7.13.1
39.3.188
Comparison with SPOT 21 In Tab. A9 we present the comparison with SPOT [2] a recent state- of-the-art object-centric learning method. SPOT proposed various ideas that can improve the quality of object-centric representation, such as patch order permutation within autoregressive decoder and self-distillation. We want to emphasize that these ideas are all orthogonal to the proposed top-down pathway and can be used together for further improvement, which we will leave as future work.
15

---

Image
Prediction
Slot Attention Map Before and After Self-
Before
After
Before
After
Before
After
Before
After
Before
After
Before
After
Before After
Figure A6: Visualization the input image, predicted object mask, and attention maps of slot attention before and after self-modulation COCO (29). Lighter the color, higher the attention score.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We explain that we tackle object-centric learning (OCL) with a novel top- down pathway that bootstraps top-down information and transforms slot-attention into self-modulatin module. We also explain that we obtain impressive results on major OCL benchmarks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We explain the limitations of our approach in the last paragraph of the conclu-
sion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
17

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe the model architecture in our method section and also the imple- mentation details in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
18

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Code will be made public for open access after publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. ec/
sublic/guidess httpii/ggiiiee//Ceeubbbimiae
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips cc/ pttppppllllllppieee https///oooooooassssooooooloee.e
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include these in the experiments section: data splits in datasets and hyperparameters implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide the standard deviation of the reported performances in table 1, 2, and 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include the specifications of the GPU we used to run our experiments in the experiments section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics sttiiis/tttt:////tt
Answer: [Yes]
Justification:
We comply with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
To our best knowledge, work does not have any direct societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model does not have high risk for misuse and we do release dataset.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite the baseline used (DINOSAUR). We also cite the datasets and provide their licenses.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We not submitting datasets, code, models weights.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We do not conduct crowdsourcing experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
There were no study participants.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "OKwWZT036D", "reviewer": "Reviewer_3ckb", "summary": "This work proposes a novel unsupervised object-centric learning method. In particular, the proposed approach enhances the slot-attention mechanism by incorporating a \"top-down pathway\" that highlights features relevant to objects in the image. \n\nThe method first employs a standard self-attention mechanism to extract slot vectors and the corresponding attention masks. These initial slot vectors and attention masks are then fed into another round of the slot-attention mechanism. In this round, they modulate the value maps of the cross-attention mechanism separately for each slot. This modulation is channel-wise for slot vectors and spatial-wise for attention masks, enabling the slot-attention mechanism to emphasize to features relevant to the slots discovered on the first slot-attention round. \n\nFurthermore, the method quantizes the slot vectors from the first slot-attention round using the VQ approach before utilizing them for modulation in the second round.\n\nTo evaluate the proposed self-modulated-based slot-attention mechanism, the authors integrate it into the DINOSAUR framework and apply it to the COCO, VOC, MOVIE-E, and MOVIE-C datasets for the downstream task of object segmentation. The results demonstrate improvements over the vanilla slot-attention mechanism. Ablation studies are conducted on the COCO dataset.", "questions": "- My primary concern regarding this work (mentioned as (W2) in the weaknesses section) is that the channel-wise and Vector Quantization (VQ) components unnecessarily complicate the method without demonstrating significant improvements in the ablation results. I would appreciate it if the authors could provide compelling arguments, such as additional experimental evidence on other datasets (e.g., VOC or MOVIE-E/C) or different settings (e.g., longer training), that show (if it is the case) the benefits of channel-wise modulation with VQ.\n\n- Please address the concerns (W1), (W3), and (W4) mentioned in the weaknesses section.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "STRENGTHS:\n- The idea of modulating the value maps for each slot, using slots found from a previous self-attention round, is interesting. This technique allows the subsequent self-attention round to focus more on features that are relevant to the objects discovered in the initial round. Essentially, it's an additional iterative improvement for the extracted slots, supplementing the existing iterative mechanism in the standard slot-attention.\n- The proposed method has been demonstrated to improve object segmentation results on all tested datasets. Furthermore, the authors utilized challenging datasets, such as COCO, to validate the effectiveness of their method, rather than resorting to simpler one, which is often among other object-centric works. \n- The paper has strong results and detailed ablation studies.", "weaknesses": "- **(W1)** It's unusual that the mIoU results in the MOVIE-C and MOVIE-E datasets surpass the mBO results.  The mBO is computed by assigning each ground truth mask the predicted mask with the largest overlap, and then averaging the IoUs of the assigned mask pairs. On the other hand, the mIoU metric is more strict and employs Hungarian matching (rather than greedy matching) to assign predicted masks to the ground truth masks. Consequently, mBO results should be either greater than or equal to the mIoU results. Therefore, it's highly likely that there's a bug in the computation of the mBO or the mIoU metrics on the MOVIE-C and MOVIE-E datasets. Or a typo / mistake when copying the results to the paper maybe.\n- As a relevant side note for the upcoming comments, it's important to mention that the FG-ARI metric is generally viewed as unreliable for assessing unsupervised object-centric methods. This is because it only takes foreground pixels into account, which can provide a deceptive understanding of segmentation quality by disregarding the localization accuracy of predicted masks [A, B, C, 30, 42].\n-  **(W2)** The motivation behind the Vector Quantization (VQ) component of the method is unclear. In particular it's uncertain why quantizing the slot vectors is essential for self-modulation. In fact, Table 4 indicates that the method is sensitive to the codebook size (also acknowledged by the authors in the limitations section). The mBO performance for sub-optimal codebook sizes 128, 256, and 1024 (30.2%, 30.5%, and 29.8%, respectively) is either the same or worse than the reproduced DINOSAUR mBO performance (30.5% from Table 6). Conversely, removing VQ achieves 32.3%, which is only marginally worse than the 32.7% obtained with VQ and the optimal codebook size. Therefore, VQ unnecessarily complicates the proposed method. As mentioned earlier, FG-ARI is unreliable, and the higher differences in this metric (36.3% w/o VQ vs 37.7% w/ VQ) should be disregarded.\n- It's also somewhat concerning that Table 6 shows only a minimal impact on mBO performance when using channel-wise modulation (from 32.5% to 32.7%). However, this does not really increase the complexity of the method, as VQ does.\n- **(W3)** Lacks discussions on how the proposed method affects the training and test time\n- **(W4)** Table 3 is missing comparisons with related works, such as Rotating Features [D] and SPOT [C], which also demonstrate strong results.\n\n[A] Genesis: Generative scene inference and sampling with object-centric latent representations, ICLR 2020.\n[B] Unsupervised Layered Image Decomposition into Object Prototypes, ICCV 2021.\n[C] SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers, CVPR 2024\n[D] Rotating features for object discovery. NeurIPs 2023.\n\n[30] Bridging the gap to real-world object-centric learning, ICLR 2023.\n[42] Slotdiffusion: Object-centric generative modeling with diffusion models, NeurIPs 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Validity of mIoU scores in MOVI-C/E\nThank you for your astute feedback. We sincerely apologize for the mistake in the mIoU calculations on the MOVI datasets and any confusion it may have caused. As you correctly pointed out, the mBO should be greater than or equal to the mIoU. The reported mIoU is inflated due to omitting the first ground-truth mask. We have provided the corrected mIoUs in Table R1 of the attached PDF. Importantly, despite this error, our revised results still demonstrate that our method outperforms DINOSAUR, substantiating our original claim about the benefits of our top-down pathway. We greatly appreciate your diligence in identifying this issue and allowing us to correct it.\n\n## Necessity & effect of VQ\nThank you for your insightful comment regarding Vector Quantization (VQ). We explain the necessity of VQ by addressing the inquiries below:\n\n**(1) Necessity of VQ for Self-Modulation**\n\nVQ is fundamental to our method since it allows the model to capture recurring semantic patterns from rough object representations produced by the first slot attention. Similar to online clustering, VQ maps a continuous slot representation to the nearest discrete code. Throughout this process, codes in the codebook are updated to encode only the semantic information from the noisy object representations; this is consistent with clustering being used to capture high-level semantic knowledge from noisy inputs in various tasks [D, 44]. Therefore, incorporating VQ into the top-down pathway enables our model to focus on core semantic concepts of the dataset, which are bootstrapped from noisy representations without extra supervision. This leads to more refined object-centric representations.\n\n**(2) Sensitivity to Codebook Size**\n\nWe acknowledge the performance dependence on the codebook size, as noted in our limitations section. This characteristic is indeed common in clustering-based methods.\nHowever, we would like to emphasize that the optimal codebook size was determined automatically using the training set only (without the validation set). To choose the codebook size, we monitor the perplexity of the code usage distribution during training. Perplexity—the exponent of entropy—indicates how uniformly the codes are being used and typically increases as the codebook becomes larger. However, when the codebook size exceeds the number of semantic patterns in the data, some codes become neglected, causing low code utilization [E, 45, 26]. Hence, the ideal codebook size leads to the maximum perplexity.\nTo find the optimal codebook size, we start with a size of 64 and monitor the perplexity of the codebook over training. After sufficient iteration (e.g., 250K), the codebook size is doubled repeatedly until the perplexity reaches a plateau. For example, on COCO, the perplexity when the codebook size is 256, 512, and 1024 are 176.9, 253.9, and 242.8, respectively, where 512 was chosen as the final size. Thereby, an adequate codebook size could be efficiently determined without using the validation set. \n\n**(3) Difference in FG-ARI and mBO in VQ Ablation**\n\nWhile we agree that mBO is the more reliable metric, FG-ARI can provide complementary information when mBOs are similar. FG-ARI is considered deceptive because it does not penalize under-segmentations that include background pixels [A], sometimes resulting in misguidedly high FG-ARIs despite low mBOs. However, in Table 6 of the main paper, VQ improves both metrics. The increased mBO, which accounts for background pixels, suggests that the FG-ARI increase is not due to under-segmentation but due to improved foreground object segmentation. Thus, we respectfully claim that the increase in FG-ARI with VQ should be considered as valid evidence for the effectiveness of VQ, especially considering the slight improvement in mBO.\n\n**(4) VQ Ablation with Full Training Schedule**\n\nWe further validate the effectiveness of VQ with an experiment under a full training schedule (500K iterations) on COCO as suggested. The results demonstrate consistent improvement in both FG-ARI and mBO, clearly showing that VQ plays a crucial role in our model:\n|  | FG-ARI | mBO$_i$ |\n|----------|----------|----------|\n| Ours wo/ VQ | 36.3 $\\pm$ 0.5 | 32.4 $\\pm$ 0.1 |\n| Ours | 37.4 $\\pm$ 0.0 | 33.0 $\\pm$ 0.3 |\n\nIn summary, while we acknowledge the sensitivity to codebook size, we believe the benefits of VQ outweigh this limitation. Additionally, an appropriate codebook size can be efficiently determined by monitoring perplexity during training.\n\n## Computation cost analysis\nWhile our model requires one more forward pass for slot attention, the additional computation cost is negligible. Slot attention accounts for only 0.64% of the total FLOPs, compared to 71.26% for the encoder and 28.10% for the decoder. Thus, our model requires 47.62 GFLOPs while DINOSAUR needs 47.32 GFLOPs. Also, processing the entire COCO 2017 val split takes 71.4 seconds for our model, while 70.5 seconds in the DINOSAUR.\n## Missing comparison with recent work\n\nThank you for pointing out the missing related work. We will add comparisons with Rotating Features [C] and SPOT [B] in the revision. We would like to kindly remind the reviewer that SPOT was published at CVPR 2024, after our submission deadline. Moreover, SPOT's self-training and sequence permutation techniques are completely orthogonal to our work on top-down pathways, suggesting potential for future integration of these complementary approaches.\n\n---\n[A] Genesis: Generative Scene Inference and Sampling with Object-centric Latent Representations, ICLR, 2020.\n\n[B] SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers, CVPR, 2024 \n\n[C] Rotating Features for Object Discovery, NeurIPS, 2023\n\n[D] Deep Clustering for Unsupervised Learning of Visual Features, ECCV, 2018\n\n[E] Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks, ICML, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MIcs5zNvud", "reviewer": "Reviewer_Yith", "summary": "This paper proposes a modification to Slot Attention incorporating top-down information into the algorithm. After an iteration of Slot Attention, the slots are quantized into a learned codebook. The quantized slots and attention maps are then used in another iteration of Slot Attention, refining the representations. The algorithm is evaluated in the DINOSAUR setting on the MOVi, Pascal VOC, and COCO datasets, showing improved segmentation quality over vanilla DINOSAUR. Visualizations of the codebook show that meaningful semantic concepts are learned and several ablations are performed.", "questions": "1. Since the slots also contain position information (”where” information), it seems possible that this gets captured in the learned codebook. Is this something the authors observed?\n2. How cherry-picked is Figure 2? Are there cases where these codes do not correspond to the same semantic concept?\n3. In Figure 2, it seems that sometimes, multiple objects with similar semantics (e.g. multiple zebras or signs) are being captured by one slot. Did you notice your proposed method does more semantic grouping instead of instance grouping compared with vanilla DINOSAUR?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method is well-motivated and seems to show an improvement over previous methods. The use of top-down semantic information from a learned codebook is novel, from my understanding. Overall, the paper is well-written and the authors provide a good analysis of their method including ablations of the different design choices.", "weaknesses": "1. The experiments are only performed in the DINOSAUR setting with pretrained, frozen ViT features as the input and reconstruction target. It would be very informative to also include experiments with the original image as the input and reconstruction target since that is also a common use case for Slot Attention. By only evaluating in the DINOSAUR setting, it is unclear how reliant the performance of the proposed model is on the pretrained ViT features.\n2. In Figure 3 and 5.B, it seems the slots are sometimes reassigned before and after the self-modulation step? For example, the skater in Figure 3 is captured by the 4th slot before the modulation and the 5th slot after the modulation. The 5th slot before the modulation seems to capture part of the background, not any part of the skater. It is unclear to me why this would happen during the modulation update.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Experiment on models other than DINOSAUR\nWe appreciate your suggestion for additional experiments. We have implemented our self-modulation technique with the original slot attention setting [28], which includes training encoders from scratch and using an image reconstruction objective. The following table summarizes the results on the CLEVR6 dataset:\n|  | FG-ARI | mBO |\n|----------|----------|----------|\n| Original slot attention | 98.7 | 21.2 |\n| Original slot attention + SelfMod | 88.7 | 61.9 |\n\nWe observe a significant improvement in mBO, showing that our self-modulation technique is applicable to slot attention and provides complementary benefits. Although FG-ARI decreased, mBO is considered more robust and should be prioritized when evaluating model performance, as detailed in our response to reviewer 3ckb (\"Necessity & effect of VQ: (3) Difference in FG-ARI and mBO in VQ Ablation\"). We have also included qualitative results in Figure R3 of the attached PDF, which demonstrates that self-modulation markedly improves segmentation. These results indicate our method's effectiveness with different encoder configurations and training objectives, suggesting it does not heavily rely on pretrained features. We will add the results and discussion in the revision.\n## Slots reassigned to other object after modulation\nThank you for your acute observation. We sincerely apologize for the discrepancy in Figures 3 of the main paper and B.5 of the appendix, which resulted from inadvertently pairing slot attention maps from different samples: the slot attention maps after self-modulation are correct but attention maps before self-modulation were wrongly paired. We have provided the updated figures as Figure R1 and R5 in the attached PDF.\n\nAs the reviewer correctly pointed out, slots are modulated based on the information captured during the first slot attention. In the corrected Figure 3 (Figure R1 of the attached PDF), row 3, the 5th slot, which initially captures the skater with less certainty, is modulated to capture the skater with higher confidence by leveraging the semantic and spatial information from the initial slot.\n\nWe would like to reaffirm that our original observation—“ modulation dynamically refines the attention maps, depending on how well they have captured the scene”—remains valid. For instance, in the 2nd row, the 1st slot initially captures both the plates and the orange, but after modulation, it is associated exclusively with the orange, yielding the plates to the 7th slot which more accurately captures the plates during the first slot attention. Similarly, in the 4th sample, the 3rd slot, which initially associates with the straw in a trivial manner, is modulated to identify and locate the straw more accurately. On the other hand, the attention maps of the 1st row are refined at the boundaries only since the initial attention maps were well structured.\n\nOnce again, we apologize for any confusion caused by this error. We appreciate your understanding and the opportunity to clarify our findings.\n\n## Does the codebook also contain positional information?\nWhile slot representations can indeed contain positional information due to the reconstruction objective, our observations suggest that the codebook primarily encodes semantic content. Performing $K$-means clustering on slot representations from DINOSAUR, we observed that the clusters are primarily grouped by semantic categories rather than spatial positions, as shown in Figure R6 of the attached PDF. This indicates that semantic information dominates in slot representations. Figure 2 in the main paper further supports this, showing objects with similar semantics but different positions mapped to the same code. Thus, while present, positional information likely occupies a small subspace in slot representations, with semantic content being the primary factor captured by our codebook.\n\n## Question about codebook visualization/semantics\nFigure 2 is not heavily cherry-picked. Most codes consistently represent a single semantic concept, with a few codes that portray supercategories such as animals and humans (people and their hands), as shown in Figure R4 of the attached PDF.\n\nAlso, we did notice an interesting edge case where some codes capture only the top-left-most patch, as shown in Figure R2 of the attached PDF. We attribute this to the autoregressive decoding process. During decoding, the top-left patch must be reconstructed first without any context from previous patches. Consequently, in images with less visual complexity, some slots appear dedicated to capturing these top-left patches. However, these special codes are minimal (1 or 2 out of 512 codes) and have limited impact. We will include more codebook visualizations and a detailed discussion in the revision.\n\n## Do modulated slots perform semantic grouping more than instance grouping?\nThank you for the insightful observation. To quantitatively assess whether our method favors semantic grouping over instance grouping compared to DINOSAUR, we conducted an analysis using two metrics:\nInstance Precision: The precision score with instance-level GTs \nSemantic Recall: The recall score with semantic segmentation GTs \nIf our model was biased toward semantic grouping, we would expect to see a lower instance precision (as semantic segmentation ground truth masks often have larger sizes) and a higher semantic recall compared to DINOSAUR. However, our analysis on the COCO dataset shows that our method improves both metrics, as shown in the table below.\n|  | Instance Precision | Semantic Recall |\n|----------|----------|----------|\n| DINOSAUR | 41.1 | 75.0 |\n| Ours | 42.3 | 75.3 | \n\nThese results suggest that the proposed top-down pathway enhances identifying both individual instances and semantic categories rather than favoring one.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uMgIVHsVfi", "reviewer": "Reviewer_qEia", "summary": "Slot Attention is a popular component for object-centric learning methods. In this\npaper, the authors propose an extension of Slot Attention named \"top-down pathway\".\nAfter the last iteration of Slot Attention, the slots are mapped to a discrete, learnable\ncodebook. Jointly with the final attention map, this information is used in a second\niteration of Slot Attention to modulate the QKV-like attention mechanism. Previous\nmodels are consistently outperformed on established benchmarks. Moreover, a more\ndetailed analysis of the learned codebook shows that distinct semantic concepts are\nrepresented by different codebook entries.", "questions": "- A recent approach to object-centric learning which is not based on Slot Attention is\n  CutLER (Wang et al. 2023). How does the proposed method compare to CutLER?", "limitations": "Limitations are very briefly discussed. An additional point could be the dependence on\nthe pretrained feature encoder, which might not work well in all scenarios.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed extension of Slot Attention is clearly explained. The original Slot\n  Attention is described at an adequate level of detail, which helps to make the paper\n  accessible.\n- The model consistently outperforms DINOSAUR, from which all design decisions are\n  derived. This demonstrates the effectiveness of the novel mechanism.\n- The analysis of the learned codebook is insightful and confirms the motivation of the\n  proposed method.", "weaknesses": "I am not fully convinced by the term \"top-down pathway\". Only the recurrent Slot\n  Attention module has been extended, the rest of the model is still bottom-up. From\n  the initial description, I would have expected top-down modulation reaching to the\n  encoder.\n\n  Did you consider a variant of the model that continues with the slot embeddings from\n  the first iteration instead of starting from scratch? If this performs similarly well,\n  it might indiciate that the quantization due to the codebook is driving the improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Clarification on term “top-down pathway” / modulating encoder\nWe appreciate the reviewer's comment on our use of the terminology, \"top-down pathway.\" We respectfully explain that our terminology is appropriate for the following reasons:\nIn our context, \"top-down\" refers to using higher-level task-relevant information to guide lower-level processing. Our method achieves this by: a) Extracting semantic information via vector quantization to enforce the focusing on certain features. b) Using spatial information from the previous slot attention to guide object localization.\nBoth of the above mechanisms bootstrap top-down information beyond pure bottom-up processing, without requiring additional supervision. While our top-down pathway is confined to the slot attention, it still introduces a meaningful top-down influence on the features by modulating values in slot attention.\nWe agree that extending the top-down modulation to the encoder is an interesting direction for future research. However, our current approach strikes a balance between effectiveness and efficiency, as modulating the encoder would significantly increase computational cost (the encoder accounts for over 71% of total compute). Our method achieves meaningful improvement without this additional overhead. For a detailed discussion of our method's efficiency, please refer to our response to Reviewer 3ckb (“Computation cost analysis”).\n\n## Value modulation vs. slot initialization\nWe appreciate the insightful suggestion. We did consider such a variant that initializes the slots with the quantized representations from the previous slot attention. However, this approach did not perform well in our experiments.\nGiven that slot attention consists of 3 iterations, initializing the second slot attention with quantized representations from the first is equivalent to inserting VQ in the middle of 6 iterations. To evaluate this, we insert VQ into DINOSAUR's slot attention with 6 iterations, after the third iteration. Results on COCO with 200K training schedule are presented below, together with the results presented in the Table 5 of the main paper:\n|  | FG-ARI | mBO$_i$ |\n|----------|----------|----------|\n| DINOSAUR 6-iter  | 28.5 | 28.2 |\n| Ours  | 37.3 | 32.7 |\n| DINOSAUR 6-iter + VQ | 17.4 | 12.9 |\n\nIntroducing VQ to DINOSAUR with 6 iterations leads to a performance decrease of 11.1%p and 15.3%p in FG-ARI and mBO, respectively (row 1 vs. 3 of the table). The severe performance degradation is likely due to the disruption in slot attention's convergence process. As shown in [6], slot representations naturally converge over its recurrent iterations. Discretizing in mid-process introduces a sudden shift in the slot representation, which will harm the overall optimization and convergence. Our method avoids this issue by modulating inner value activations, which can modulate the recurrent slot update process without directly changing the slot representation. By doing so, we can exploit the top-down information without hindering convergence of the slot attention.\n\n## Comparison to CutLER\nWe appreciate the suggestion to compare with CutLER [A], a recent approach to unsupervised object segmentation. While CutLER's full pipeline includes self-training of Mask R-CNN, we focus on comparing our method to CutLER's MaskCut algorithm for pseudo-mask generation, as this is the most direct comparison to our approach. Our method’s predictions can also be used as pseudo labels for training Mask R-CNN. We reproduced MaskCut using the official CutLER repository and evaluated it on the COCO dataset:\n|  | FG-ARI | mBO$_i$ | mIoU |\n|----------|----------|----------|----------|\n| MaskCut (CutLER )| 31.5 | 28.9 | 26.7 |\n| Ours | 37.4 $\\pm$ 0.0 | 33.0 $\\pm$ 0.3 | 31.2 $\\pm$ 0.3 |\n\nOur method significantly outperforms MaskCut across all metrics, demonstrating its effectiveness for object discovery. We will add these results and discussion in the revision. \n\n## Dependence on the pretrained feature encoder\nWe appreciate the insightful suggestion. Regarding the inquiry, we conducted an experiment with the settings of the original slot attention [28], which trains the encoder from scratch (response to reviewer Yith, PJTV: “Experiment on models other than DINOSAUR”). Interestingly we observe substantial performance gain even in this setting, demonstrating that our method's improvements does not solely arise from using a pretrained feature encoder. For more detailed results and discussion, please refer to the corresponding response. We will discuss this experiment in the future revision as well.\n\n\n---\n[A] Cut and Learn for Unsupervised Object Detection and Instance Segmentation, CVPR, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zVkkutedoU", "reviewer": "Reviewer_PJTV", "summary": "This paper proposes a method that improves the performance of object-centric learning by modulating Slot Attention with semantic and location information obtained based on the output slots and attention maps of Slot Attention. For a given output slot, the semantic information is chosen as the vector closest to this slot in a codebook that is learned from all output slots of the entire dataset via vector quantization, and the location information is chosen as the shifted attention map with a mean value of 1. The proposed method is implemented based on DINASOUR and is compared with DINASOUR and other methods on two synthetic and two real datasets. The proposed method outperforms the compared method in most cases when the size of codebook is chosen appropriately.", "questions": "1. How is the best size of codebook chosen?\n2. Is it possible to learn a codebook that is independent of the locations of objects in the image?", "limitations": "The authors have addressed the limitation in the conclusion part, and there is no potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Improving the performance of object-centric learning with top-down information is an important and interesting research direction.\n2. The proposed method outperforms the compared methods when the size of codebook is chosen appropriately.", "weaknesses": "1. My main concern about the paper is that the performance of the proposed method is very sensitive to the size of codebook. If the size of codebook is not chosen appropriately, the performance can be even worse than the original DINASOUR method. Moreover, how to choose the best size of codebook is not described clearly.\n2. From my understanding, the core of the proposed method is a self-modulation module (along with a codebook) that is compatible with all the object-centric learning methods using the slot attention mechanism. However, the proposed method is only implemented based on DINASOUR. The quality of the paper could be significantly improved if the proposed method is implemented based on at least another object-centric learning method that is developed based on slot attention.\n3. The proposed self-modulation includes both semantic and spatial modulations. However, the codebook used in the semantic modulation also contains spatial information. This design can be more elegant, and the codebook can be much more useful if it only contains semantic information. In this way, whether two objects belong to the same category can be determined automatically.\n4. Some data in the experimental part are inconsistent. For example, the mBO^i of the proposed method on COCO is 33.0 in Table 2, 33.3 in Table 3, and 32.7 in Table 4.\n5.  The authors attribute both semantic modulation and spatial modulation as top-down modulations. I agree that the semantic modulation is a top-down modulation because the codebook contains knowledge learned from all images (not just the inferred image). However, I don’t think that the spatial modulation can be considered top-down because it just uses attention maps computed in a bottom-up manner (only based on the content of one image).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Sensitivity to codebook size\nWe acknowledge the performance dependence on the codebook size, as noted in our limitations section. \nHowever, we would like to emphasize that the optimal codebook size was determined automatically using the training set only (without the validation set). To choose the codebook size, we monitor the perplexity of the code usage distribution during training. Perplexity—the exponent of entropy—indicates how uniformly the codes are being used and typically increases as the codebook becomes larger. However, when the codebook size exceeds the number of semantic patterns in the data, some codes become neglected, causing low code utilization [D, 45, 26]. Hence, the ideal codebook size leads to the maximum perplexity.\n\nTo find the optimal codebook size, we start with a size of 64 and monitor the perplexity of the codebook over training. After sufficient iteration (e.g., 250K), the codebook size is doubled repeatedly until the perplexity reaches a plateau. For example, on COCO, the perplexity when the codebook size is 256, 512, and 1024 are 176.9, 253.9, and 242.8, respectively, where 512 was chosen as the final size. Thereby, an adequate codebook size could be efficiently determined without using the validation set. \n\n## Experiment on models other than DINOSAUR\nWe appreciate your suggestion and have implemented our self-modulation technique with the original Slot Attention setting [28], which trains the encoder from scratch and uses an image reconstruction objective. The following table summarizes the results on the CLEVR6 dataset:\n|  | FG-ARI | mBO |\n|----------|----------|----------|\n| Original slot attention | 98.7 | 21.2 |\n| Original slot attention + SelfMod | 88.7 | 61.9 |\n\nWe observe a significant improvement in mBO, showing that our self-modulation technique is applicable to slot attention and provides complementary benefits. Although FG-ARI decreased, mBO is considered more robust and should be prioritized when evaluating model performance, as detailed in our response to reviewer 3ckb (\"Necessity & effect of VQ: (3) Difference in FG-ARI and mBO in VQ Ablation\"). We have also included qualitative results in Figure R3 of the attached PDF, which demonstrates that self-modulation markedly improves segmentation. These results indicate our method's effectiveness with different encoder configurations and training objectives, suggesting it does not heavily rely on pretrained features. We will add the results and discussion in the revision.\n## Does the codebook also contain positional information?\nOur codebook primarily encodes semantic content, largely independent of the object position (Figure 2 of the  main paper). We hypothesize that this is because semantic information dominates slot representations, with positional information occupying a small subspace. This is supported by $K$-means clustering of DINOSAUR slot representations, which automatically groups semantic categories using simple L2 distance (Figure R6 of the attached PDF). While our approach empirically extracts mainly semantic information, explicitly disentangling positional and semantic information remains an interesting future direction, with work like [3] offering promising leads.\n## Inconsistent numbers between tables\nWe apologize for the confusion. Our mBO$_i$ on COCO of 33.3 in Table 3 is a typo, and the correct number is 33.0. We will revise the paper to include the correct measurement. Despite this error, our method still outperforms the LSD [18] by 2.6%p, showing the effectiveness of our approach.\n\nAdditionally, as mentioned in line 234 on page 6, the in-depth analysis and ablation studies (Tables 4, 5, and 6) were conducted with models trained for 200K iterations, rather than the full 500K iterations, due to computational constraints. This explains the differences between Tables 1-3 and Tables 4-6. We found that the models mostly converged at 200K iterations and thus was enough to reflect the performance trends of models trained for the full 500K iterations. To improve clarity, we will update the captions of Tables 4-6. We appreciate your attention to these details.\n## Can spatial modulation be called “top-down”?\nWe appreciate the reviewer’s comment on our top-down approach. We respectfully clarify our reasons for calling spatial modulation a top-down approach.\n\nA top-down approach is defined by the use of prior knowledge (contextual or task-relevant) to guide visual processing, while a bottom-up approach relies solely on sensory data. As the reviewer noted, our semantic modulation using vector quantization aligns with this definition, leveraging contextual knowledge learned from the dataset.\n\nAlthough the spatial information is derived from the same image, the spatial modulation provides rough object saliency maps to the subsequent slot attention, which is crucial task-relevant information for object discovery. It is task-relevant because it explicitly guides the model to discover objects within the provided saliency map, narrowing the search space. Such spatial cues are recognized as a form of top-down knowledge in human vision research [A, B, C], guiding attention to regions likely containing objects of interest.\n\nIn conclusion, while our semantic and spatial modulations differ in their information sources, both embody top-down processing principles by providing crucial task-relevant guidance to the visual processing pipeline.\n\n---\n[A] SUN: Top-down saliency using natural statistics, Visual cognition, 2009\n\n[B] Components of visual orienting, Attention and performance X: Control of language processes, 1984\n\n[C] Oculomotor strategies for the direction of gaze tested with a real-world activity, Vision research, 2003\n\n[D] Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks, ICML, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for the constructive and insightful comments. Our work introduces a novel top-down pathway for object-centric learning that consistently improves performance across multiple benchmarks. Reviewers highlighted several strengths of our work, including the clarity of the manuscript, proposed method’s motivation and novelty, consistent improvements over the baseline across challenging real-world datasets, and extensive analyses. 

In each response to the reviewer, we have carefully addressed every comment and question to the best of our ability, providing additional results, clarifications, and analyses where requested. Key points we elaborate on include the clarification for vector quantization, comparisons to additional baselines, and further insights into our method's behavior. We also provide additional results and visualizations in the attached PDF. 

Due to the word limit, we referenced **papers cited in the main paper with numbers** and **newly cited ones with alphabetical letters**. 

Thank you again, and we look forward to a constructive interaction in the following discussion period!

Meta-review: Slot attention has emerged as a popular approach for unsupervised object discovery, and the improvements presented in this paper have been appreciated by all the reviewers. Given that the work is both timely and well-executed, I recommend accepting it.

The key highlights include the use of a visual codebook to self-modulate internal activations by defining the "what" and "where" information based on the attention maps. As noted by reviewer qEia, the model consistently outperforms the underlying DINOSAUR model on which this technique is built. Both the qualitative and quantitative evaluations are impressive. Reviewer YiTH and PJTV has offered valuable suggestions on how the paper could be enhanced by incorporating insights from related work. I hope the authors take these into consideration while preparing the final version.
Decision: Accept (poster)