Bootstrapping Top-down Information for Self-modulating Slot Attention

Authors: dongwon  kim
seoyeon  kim
suha  kwak
dept of cse
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:42:32

Abstract: Object-centric learning (OCL) aims to learn representations of individual objects within visual scenes without manual supervision, facilitating efficient and effective visual reasoning. Traditional OCL methods primarily employ bottom-up approaches that aggregate homogeneous visual features to represent objects. However, in complex visual environments, these methods often fall short due to the heterogeneous nature of visual features within an object. To address this, we propose a novel OCL framework incorporating a top-down pathway. This pathway first bootstraps the semantics of individual objects and then modulates the model to prioritize features relevant to these semantics. By dynamically modulating the model based on its own output, our top-down pathway enhances the representational quality of objects. Our framework achieves state-of-the-art performance across multiple synthetic and real-world object-discovery benchmarks.

Main paper: 
 introduction: 
 Object-centric learning (OCL) is the task of learning representations of individual objects from visual scenes without manual labels. The task draws inspiration from the human perception which naturally decomposes a scene into individual entities for comprehending and interacting with the real world visual environment. Object-centric representations provides improved generalization and robustness [9] , and have been proven to be useful for diverse downstream tasks such as visual reasoning [44] , simulation [45] , and multi-modal learning [24] . In this context, OCL which learns such representations without labeled data has gained increasing attention.A successful line of OCL builds upon slot attention [30] . This method decomposes an image into a set of representations, called slots, that iteratively compete with each other to aggregate image features. Reconstructing the original image from the slots, they are encouraged to capture entities constituting the scene. This simple yet effective method has been further advanced by novel encoder or decoder architectures [33, 19, 46, 42] , optimization technique [18, 6] , and new query initialization strategies [18, 24] .It is worth noting that all these methods are fundamentally considered bottom-up models, as they rely on aggregating visual features without incorporating high-level semantic information from the beginning. This bottom-up approach assumes that visual features within an object are homogeneous and can be clustered in the feature space, which only holds for simplistic objects that can be identified using low-level cues such as color [20] . In complex real-world scenarios where visual entities of the same semantics exhibit diverse appearances, this homogeneity often breaks down, leading to suboptimal object representations [22, 47] . Thus, we take an approach different from the previous line of research: introducing top-down information into slot attention, such as object categories and semantic attributes. The overall pipeline of our framework. A top-down pathway is introduced into slot attention to utilize top-down information. The pathway consists of two parts: bootstrapping top-down knowledge and exploiting them. Firstly, semantic information is bootstrapped from slot attention outputs by mapping slots to discrete codes from a learned codebook through vector quantization.Secondly, slot attention is modulated using these codes and its attention maps, transforming it into a self-modulating module. Inner activations are modulated across channels with codes and across space with centered attention maps. Slot attention is then repeated with these modulated activations, yielding more representative slots.Incorporating top-down information enables slot attention to specialize in discerning objects within specific semantic categories. For instance, identifying vehicles in a complex urban environment can be challenging due to the diverse and cluttered nature of the scene. Top-down information can guide the model to prioritize vehicle-specific features, such as wheels and windows. This inhibits the contributions of irrelevant features when computing slots, and enhances the aggregation of visual features of individual vehicles into slots. Nevertheless, devising such a top-down approach is not straightforward since OCL assumes an unsupervised setting without any labeled data, making it hard to identify and exploit the high-level semantics typically obtained from annotated datasets.We propose a novel framework that incorporates a top-down pathway into slot attention to provide and exploit top-down semantic information; Fig. 1 illustrates of our framework. The pathway consists of two parts: bootstrapping semantics and exploiting them for better representations. Firstly, top-down semantic information is bootstrapped from the output of slot attention itself, by mapping continuous slots to discrete codes selected from a finite learned codebook. Such an approach allows the codebook to learn prevalent semantics in the dataset, with each code representing a specific semantic concept. Thus, semantic information can be bootstrapped without any object-level annotations and used to provide top-down semantic information. Secondly, slot attention is modulated using bootstrapped top-down cues obtained from the first phase, which we call self-modulation. In this phase, the top-down pathway dynamically guides the slot attention by re-scaling its inner activations based on the top-down information. This self-modulation process enables the model to focus on feature sub-spaces where object homogeneity is more consistent, thereby improving its performance in diverse and realistic settings.Our contributions are threefold:• We introduce a method to bootstrap top-down semantic information from the output of slot attention, without requiring any object-level annotations. This allows the extraction of high-level semantic cues from an unsupervised learning process. • We propose a self-modulation scheme that dynamically guides the slot attention's inner activations to enhance object representation, successfully incorporating the top-down semantic cues extracted. • By integrating the proposed top-down pathway into slot attention, we demonstrate that the performance of object discovery is largely improved on various OCL benchmarks.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We explain that we tackle object-centric learning (OCL) with a novel topdown pathway that bootstraps top-down information and transforms slot-attention into a self-modulating module. We also explain that we obtain impressive results on major OCL benchmarks.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Object-centric learning OCL aims to learn representations of individual objects within an image. The 'object-centric' dimension is orthogonal to the conventional representation learning which learns representations independent of the composition of the image. The structured nature of object-centric representations offers improved generalization [9] , making it valuable for various applications, including visual reasoning [44] , dynamics simulation [45] , and multi-modal learning [23, 24] . A foundational method in this field is slot attention [30] , which introduced a simple yet effective framework that employs a competitive attention mechanism between slots. Following slot attention, many recent works have proposed improvements by introducing novel encoder or decoder formulations [33, 19, 46, 35] , optimization techniques [18, 6] , additional slot refinement modules [37, 25, 3] , and expansions to video modality [27, 11, 36] . These methods are primarily bottom-up models, while our approach proposes to bootstrap and incorporate top-down information.Incorporating top-down information The human visual system perceives scenes by leveraging both top-down and bottom-up visual information [4, 8] . Top-down information represents taskdriven contextual cues, such as high-level semantics and prior knowledge about the scene. In contrast, bottom-up information is derived directly from the sensory input. Inspired by this dualprocessing mechanism of the human visual system, several studies [34, 1, 48] have attempted to model this approach within deep learning, achieving significant improvements across various tasks.Our work follows in a similar direction, specifically focusing on introducing top-down information into the representative OCL method, slot attention. By incorporating top-down semantic and spatial information, we aim to enhance the performance of slot attention in diverse visual environments, addressing the limitations of previous bottom-up methods Discrete representation learning Discrete representations within neural networks are considered effective for modeling discrete modalities [50, 38] and tackling generation tasks [13, 28] . Particularly, the pioneering work, VQ-VAE [38] , introduced a method for learning discrete latent representations through vector quantization. This model uses a discrete codebook, where the encoder maps input data to discrete codes using nearest-neighbor lookup and the decoder reconstructs the input from the codes. Another notable approach is the Gumbel-softmax trick [17, 32] , which provides a differentiable approximation of sampling from a categorical distribution. Recent advancements have aimed to incorporate a more sophisticated formulation of codebooks [49] or improve codebook utilization [16] to better handle discrete representations. Recent work by Wallingford et al. [40] is related to our research in terms of using vector quantization for segmentation task. However, our method differs in that the quantized codes are used to modulate bottom-up slot attention, while in Wallingford et al. [40] , the codes are used solely for segmentation labeling.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We explain the limitations of our approach in the last paragraph of the conclusion.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 We propose an OCL framework that incorporates top-down semantic information, such as object categories and semantic attributes, into slot attention through a top-down pathway. Fig. 1 illustrates the overall pipeline of our framework. Firstly, slot attention is applied to visual features extracted from an image encoder to output slots (Sec. 3.1). Then, a top-down pathway leverages the slots to identify semantics in the input image and modulate slot attention. The pathway consists of two parts: bootstrapping top-down semantic information from a learned codebook and attention maps (Sec. 3.2) and modulating the inner activations of slot attention with this semantic information (Sec. 3.3).During the self-modulation stage, slot attention is repeated with the modulated activations, resulting in more representative slots.Slot attention is a recurrent bottom-up module that aggregates image features into slots through an iterative attention process, where each of the resulting slots represent an entity in the input image. Within our framework, these slots are used to bootstrap top-down information in the later stage (Sec. 3.2).The module takes in the initial slots, S 0 ∈ R K×D , and visual features extracted from an image encoder, x ∈ R N ×Dfeat . The initial slots are obtained by sampling K vectors from a learnable Gaussian distribution using a reparameterization trick. The slots S = [s 1 , s 2 , . . . , s K ] ∈ R K×D , where each represents an individual object in the image, are computed by iteratively updating the initial slots T times as S := S T , where S t+1 = slot_attn x, S t .(1) In each iteration, the slots attend to the visual features, refining their representations through a series of attention-based updates. Let q(•), k(•), and v(•) represent linear projections from dimension d to d h . Then, the attention map A ∈ R K×N is computed asA i,j = e Pi,j K i=1 e Pi,j, whereEQUATIONUnlike the original attention introduced in transformer [39] which normalizes across the keys, the attention in slot attention is normalized across the slots. Such a distinct normalization scheme makes the slots compete with each other to aggregate the visual features, encouraging each slot to represent a distinct object in the scene. Then, the computed attention map A is normalized across the rows into Ã and used to update the slots asEQUATIONsuch that U is the weighted sum of the visual features v(x). The function f update processes U with consecutive GRU [7] and Multi-Layer Perceptron (MLP) and residually sums it to the previous slots, S t-1 , to produce the updated slots, S t . For further details, refer to Locatello et al. [30] .Our idea to bootstrap top-down information without annotations is based on our observation that the slots S, which are outputs of the bottom-up attention module, contain rough semantic information about objects. We leverage this coarse information to bootstrap both the semantic and spatial topdown information about the objects coarsely represented by the slots. Top-down semantic information pertains to the specific semantic categories or attributes of the objects (what), while top-down spatial information indicates the locations or regions within the image where these objects are located (where). Incorporating such knowledge can guide slot attention to focus on the features most relevant to the objects expected to appear, enabling it to accurately capture objects that are obscured or have high intra-object variance, such as people with different hairstyles or clothing.Firstly, we extract the "what" information from the slots S using Vector Quantization (VQ), which maps each slot to one of the semantic concepts learned throughout training. Specifically, each slot S k is mapped to the nearest code in a finite codebook, C = [c 1 , c 2 , . . . , c E ] ∈ R E×D with size E. The mapped code c * k ∈ R D is considered a top-down semantic cue for the slot s k . Formally, this quantization process can be written asEQUATIONSince the arg min operation is non-differentiable, we use the straight-through estimator [2] for backpropagation. During training, the codebook learns to store distinct semantic patterns recurring within the dataset by quantizing continuous slot embeddings into a limited number of discrete embeddings. Thereby, each code can act as automatically discovered top-down semantic information.Secondly, we obtain the "where" information from the attention maps of the last layer of slot attention. For each slot s k , the k-th row vector of the attention map A, denoted as a k ∈ R N , is used to aggregate visual features and update s k . This attention map provides useful spatial prior information about where each extracted top-down semantic information is located in the image.In the original slot attention (Sec. 3.1), the slot updates are driven purely by visual features extracted from the input without incorporating higher-level semantic information that can provide additional context. To address this limitation, we introduce self-modulating slot attention, which modulates the computation of the slot updates based on the top-down information obtained in the bootstrapping stage (Sec. 3.2). This bootstrapped top-down information is used to dynamically amplify or inhibit specific channel dimensions or regions of the value-projected visual features, while keeping the model parameters unchanged. Formally, self-modulating slot attention can be represented by conditioning slot attention with the vector quantized slots c * k and their corresponding slot-wise attention map a k :EQUATIONAlgorithm 1 Self-modulating Slot Attention. The module inputs visual features extracted from an encoder, initial slots sampled from a learned Gaussian distribution, and K vector quantized slots and their corresponding K slot-wise attention maps output from the original slot attention. The number of iterations, T , is set to three.Input: Visual features x, initial slots Ŝ0 , vector quantized slots [c * i ] K i=1 , and slot-wise attention maps [ai] K i=1 . Output: Slots after T -iteration of self-modulating slot attention ŜT .1:for k = 1 to K in parallel do 2: m c k = MLP(c * k )// Compute channel-wise modulation vector 3:m s k = 1 + (a k -a k )// Compute spatial-wise modulation vector 4:M k = m s k ⊗ m c k // Compute modulation map 5: for t = 1 to T do 6: Ã = softmax q Ŝt-1 k x ⊤ / √ D h 7: Ŝt = fupdate([u1, u2, . . . , uK ], Ŝt-1 ), where u k = Ãk (M k ⊙ v x ) // Modulated slot update 8: return ŜTwhere Ŝ represents the slots from the self-modulating slot attention, different from the slots of the original slot attention denoted by S. Note that the original slot attention (Sec. 3.1) and self-modulating slot attention share parameter weights and initial slots, such that S 0 = Ŝ0 . Specifically, we modulate slot attention with a modulation map M k ∈ R N ×D computed from c * k and a k . Each element of the modulation map represents the relevance score between the corresponding visual feature element and the top-down information of the expected object. This modulation map can be used to guide the update of each slot Ŝk (Eq. 3), by prioritizing specific value elements with the high relevance scores. In the self-modulating slot attention, computation of the slot update U is replaced with:EQUATIONwhere ⊙ represents Hadamard product. Such re-scaling of the value features with the modulation map ensures that the specific channel dimension or regions contribute more to the update of each slot, based on the semantics or locations encoded in bootstrapped top-down information.The modulation map M k is computed by taking the outer product between channel-wise and spatialwise modulation vectors, which are predicted using c * k and a k , respectively:EQUATIONFor predicting channel-wise modulation vector m c k , quantized slot c * k is used, which tells us "what" the object appearing in the image is. The channel-wise scaling is designed to enforce the model to focus on certain feature subspaces closely correlated to the semantic concept identified. Specifically, channel-wise modulation vector m c k can be obtained by feeding quantized slot c * k to the MLP, which is represented as:EQUATIONThe spatial-wise modulation vector m s k can be obtained by further processing the attention map of each slot, which contains the top-down information on the location of the semantic concept:EQUATIONwhere a k is for the average of the attention score of a k . Using an attention map as is for modulation will make all values down-scaled, while some regions likely to contain the object should be highlighted for effective incorporation of the spatial top-down information. Thus, we use the attention map shifted to have a mean value of 1 for the spatial-wise modulation map.Slot attention is trained within an autoencoding framework, using a decoder that reconstructs visual features output by the image encoder [35, 33] or the original image [30] from the slots. In this paper, we choose the visual feature reconstruction as our training objective since it is known to provide more robust training signals for real-world datasets [33] . We also employ a vector quantization objective for the codebook C only, which thereby learns to minimize the mean-squared error between the slot and the sampled codes. The reconstruction objective L recon and vector quantization objective L VQ are given byEQUATIONwhere sg(•) represents stop gradient operation andC * = [c * 1 , c * 2 , . . . , c * K ] ∈ R K×D .For the decoder, we utilized the autoregressive slot decoder [35, 33] . The reconstruction objective ensures that the learned slot representations capture essential information about the objects in the scene, while the vector quantization objective encourages the codebook to capture recurring semantic concepts in the dataset.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We include the specifications of the GPU we used to run our experiments in the experiments section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
experimental settings: 
 Datasets To verify the proposed method in diverse settings, including synthetic and authentic datasets, we considered four object-centric learning benchmarks: MOVI-C [15] , MOVI-E [15] , PASCAL VOC 2012 [14] , and MS COCO 2017 [29] . MOVI-C and MOVI-E are synthetic datasets, adopted for validating our method in relatively simple visual environments. MOVI-C contains 87,633 images for training and 6,000 images for evaluation, while MOVI-E contains 87,741 and 6,000, respectively. To evaluate the proposed model in real-world settings, we leverage the VOC and COCO datasets. Following DINOSAUR [33] , we use the trainaug variants, containsing 10,582 training images, for VOC dataset. For the evaluation, we use the validation split containing 1,449 images. The COCO dataset consists of 118,287 training images and 5,000 images for evaluation. While the VOC dataset includes some images with a single object, images of the COCO dataset always contain 2 or more objects, making it the most challenging. MOVI datasets are licensed under apache license 2.0 and COCO is licensed under CC-BY-4.0.Metrics We evaluate our method with three metrics: foreground adjusted random index (FG-ARI), mean best overlap (mBO), and mean intersection over union (mIoU). The FG-ARI is the ARI metric computed for foreground regions only (objects), which measures the similarity between different clustering results. The mBO and mIoU are both IoU-based metrics, computed for all regions including the background. The mBO computes the average IoU between ground truth and prediction pairs, obtained by assigning each prediction to the ground truth mask with the largest overlap. The mIoU is computed as the average IoU between ground truth and prediction pairs obtained from Hungarian matching. For COCO and VOC, mBO i and mBO c indicate the mBO metric computed using semantic segmentation and instance segmentation ground truth. We use the instance segmentation ground truth for other metrics. Following previous work [33] , the internal attention maps of the autoregressive decoder are used as the mask prediction results of the slots.Implementation details To assess the effectiveness of the proposed top-down pathway, our model is implemented based on DINOSAUR [33] , a representative slot-based OCL method. For the encoder and decoder, we use a DINO [5] pretrained ViT-B/16 [10] and an autoregressive transformer decoder [35, 33] , respectively. The model is trained using an Adam optimizer [26] with an initial learning rate of 0.0004, while the encoder parameters are not trained. The number of slots K is set to 11, 24, 7, and 6 for MOVI-C, MOVI-E, COCO, and VOC, respectively. The codebook size E is set to 128 for synthetic datasets (MOVI-C and MOVI-E) and 512 for authentic datasets (COCO and VOC). The model is trained for 250K iterations on VOC and for 500K iterations on the others. For the ablation study and analysis, models are trained for 200K iterations on COCO, as this was enough to reveal overall trends given limited computational resources. Full training of the model takes 26 hours using a single NVIDIA RTX 3090 GPU.Codebook size E selection The performance of the proposed top-down pathway depends on codebook size E (Sec. 4.4), necessitating a principled selection method. We determine E automatically by monitoring the perplexity of code usage distribution during training, requiring only the training set without validation data. Perplexity-the exponent of entropy-indicates how uniformly the codes are being used. While perplexity typically increases with codebook size, it plateaus when E exceeds the number of distinct semantic patterns in the data, as some codes become unused [16, 49, 28] . To find the optimal size, we start with E = 64 and double it until the perplexity plateaus after 250K iterations. For example, on COCO, the perplexity when the codebook size is 256, 512, and 1024 are Table 1 : Comparison with DINOSAUR [33] on synthetic datasets: MOVI-C [15] and MOVI-E [15] . We include both the reported and reproduced performance of DINOSAUR for fair comparison. 176.9, 253.9, and 242.8, respectively, where 512 was chosen as the final size. This procedure enables efficient hyperparameter selection using only training data, eliminating the need for validation set tuning. Following this approach, we set E = 128 for synthetic datasets (MOVI-C and MOVI-E) and E = 512 for real-world datasets (COCO and VOC).DINOSAUR [33] is the first successful OCL method that scales slot attention to real-world datasets by introducing the use of a self-supervised image encoder [5] , an autoregressive decoder, and a feature reconstruction objective. Notably, DINOSAUR uses the vanilla slot attention mechanism without modifications, making it a perfect baseline for validating the effectiveness of our proposed top-down pathway. Thus, we adopt DINOSAUR as a baseline and compare its performance with and without our proposed method. For a fair comparison, we report both the reported and reproduced performance of DINOSAUR.Tab. 1 demonstrates that incorporating the proposed top-down pathway into DINOSAUR largely improves performance in every metric. Specifically, our method improves FG-ARI by 5.9 on MOVI-E, which is the most challenging synthetic dataset. In Tab. 2, performances on authentic datasets, COCO and VOC, are reported. Our method largely surpasses the reproduced baseline on most metrics. The only metric for which our method does not show improvement is the FG-ARI on VOC. We hypothesize that this is because VOC images frequently contain single objects only, and FG-ARI is computed solely with foreground pixels so that the performance is less affected by the top-down information.Tab. 3 presents a comparison between our method and recent state-of-the-art methods. It is notable that our proposed method achieves competitive performance even to recent methods using advanced diffusion-based decoders [19, 46] . Moreover, our approach focuses on incorporating top-down information into slot attention, which is orthogonal to the line of work advancing decoders to provide better training signals to slot attention.Codebook visualization To validate whether the proposed codebook learns meaningful semantic concepts, we present visualizations of the codebook in Fig. 2 . The index of the code and the mask prediction obtained from the slots modulated by the code are presented together, revealing the semantic entity each code represents. Visualization demonstrates that the codebook successfully discovers and stores distinct semantic concepts without using any annotations. Moreover, the codes are mapped to objects with various appearances and layouts, which demonstrates that the codes learn high-level semantic information and not low-level structural or positional information.Prediction visualization In Fig. 3 , we visualize the original image, mask predictions, and slot attention maps A before and after self-modulation. Results demonstrate that the modulation dynamically Table 3 : Comparison with state of the arts [33] on COCO [29] , VOC [14] , MOVI-C [15] , and MOVI-E [15] . Performances of Slot Attention on MOVI-C and -E are reproduced by [33] and that of SLATE by [19] . Those on COCO and VOC are from [46] . refines the attention maps, depending on how well they have captured the scene. For example, when the attention map is well-structured but coarse, the modulation process refines the boundaries of the attention map without changing the overall layout (first row). However, if the attention maps fail to delineate objects, the modulation process recomposes the attention maps to differentiate objects (second to fourth row). By providing top-down semantic and spatial information via self-modulation, the attention maps are enhanced to capture the object within complex real-world environments.MethodEffect of codebook size In our framework, vector quantization maps slots to distinct top-down semantic information stored in the codebook, as shown in Fig. 2 . In Tab. 4, we report the performances across different codebook sizes, E. The results show that a codebook size too large (E = 1024) or small (E ≤ 256) leads to performance degradation. When the codebook size is too small, the codes cannot sufficiently learn distinct semantic information, which degrades the quality of the bootstrapped top-down semantic information. On the other hand, a codebook size too large may cause the codes to capture irrelevant details such as appearance variance or positional information rather than meaningful semantic concepts. However, we determine the optimal codebook size automatically using the perplexity of codebook usage during training (Sec. 4.1), eliminating the need for extensive hyperparameter tuning using validation split and benchmark metrics. Impact of increased iterations Since our method requires repeating slot attention with selfmodulation, we investigate whether our performance improvement simply comes from the increased iterations of slot attention. In Tab. 5, we compare the results of our model with the DINOSAUR baseline model using six iterations of slot attention, which is twice as many iterations as the default setting. Notably, both our model and the DINOSAUR 6-iteration model leverage the same number of iterations. The results show that the DINOSAUR model with six iterations actually performs worse compared to the default 3 iterations. This indicates that merely increasing the number of iterations does not guarantee improvement. Our method's superior performance is thus attributed to the self-modulation mechanism rather than the increased number of iterations.Computation overhead of top-down pathway While our model requires one more forward pass for slot attention, the additional computation cost is negligible. In DINOSAUR, slot attention accounts for only 0.64% of the total FLOPs, compared to 71.26% for the encoder and 28.10% for the decoder. Consequently, our model requires 47.62 GFLOPs versus 47.32 GFLOPs of DINOSAUR-a mere increase below 1%. In practice, processing the entire COCO 2017 val split on a single NVIDIA RTX 3090 GPU takes 71.4 seconds for our model compared to 70.5 seconds for DINOSAUR, demonstrating minimal impact on inference time.Codes representing broader semantics While most codes in our codebook consistently represent single object categories, we observed interesting cases where codes capture broader concepts, as shown in Fig. 4 . Some codes are trained to represent supercategories -for instance, grouping different animal species (code 468) or various human parts into shared codes (code 328). This suggests the codebook can flexibly adapt to different levels of semantic abstraction when beneficial. We also discovered an edge case where certain codes (e.g., code 223) specialize in capturing top-left patches of images. This behavior appears to be influenced by the autoregressive decoding process, which Ablation study In Tab. 6, we present an ablation study of our method on COCO for channel-wise modulation, vector quantization, spatial-wise modulation, and attention map shifting. In the first row, the result with no modules is presented, which is equivalent to the baseline DINOSAUR model. The last row is the performance of using all four modules, equivalent to our proposed method. The second and third rows are for the ablation of vector quantization and channel-wise modulation, demonstrating that incorporating top-down semantic information significantly improves performance.In the fourth and fifth rows, the ablation of attention map shifting and spatial-wise modulation is presented.The results show that both operations are critical for the effective exploitation of top-down spatial information. 
conclusion: 
 In this paper, we introduced an OCL framework that incorporates top-down information into the slot attention mechanism through a top-down pathway. In this pathway, the output of the slot attention is used to bootstrap high-level semantic knowledge and rough localization cues for existing objects.Using the bootstrapped top-down knowledge, slot attention is modulated to focus on features most relevant to the objects in the scene. Consequently, by incorporating the proposed top-down pathway into slot attention, we achieved state-of-the-art performance on various OCL benchmarks, including challenging synthetic and authentic datasets.Limitation The proposed top-down pathway has a limitation in that its overall performance relies on the quality of the codebook learned during training. As shown in Tab. 4, an incorrect choice of codebook size can result in the codes failing to learn distinct semantic concepts or capturing irrelevant details. While we mitigate this limitation through perplexity-based automatic codebook size tuning, the more principled codebook design that can eliminate the need for a pre-defined hyperparameter, such as dynamically expanding codebook during learning [40] , will be promising future research direction. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: To our best knowledge, our work does not have any direct societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our model does not have high risk for misuse and we do not release a dataset.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We properly cite the baseline we used (DINOSAUR). We also cite the datasets and provide their licenses.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Fig. A6 demonstrates the additional visualizations of original image, mask predictions, and slot attention maps A before and after self-modulation on the COCO dataset.The proposed top-down pathway is implemented based on the DINOSAUR baseline [33] , using an autoregressive slot decoder. Singh et al. [35] first proposed to use such an autoregressive decoding scheme for slot attention training. Autoregressive decoder is known to provide better training signal leading to improved performance, compared to the MLP-based broadcast decoder [43] used by the slot attention originally. Autoregressive decoder is the simple variant of the transformer decoder [39] , which takes input visual feature x and slots S. The decoder consists of multiple decoding blocks. Let multi-head attention be denoted as MHA(Q; K; V ), where Q, K, and V is for query, key, and value, respectively. Then, the decoding block can be represented as:EQUATIONEQUATIONwhere FFN denotes a feedforward layer with MLP and residual connection, MHA < (•) represents multi-head self-attention with causal masking, and x [BOS] represents the visual feature sequence with a learnable [BOS] token appended at the start of the sequence. By using multiple decoding blocks, we can compute the autoregressive reconstruction of the visual feature x, which is consequently used for the computing reconstruction objective. Following DINOSAUR, we use an autoregressive decoder with four decoding blocks. The number of heads for multi-head attention is set to 8.Top-down pathway without DINO and autoregressive decoder We have mainly built a top-down pathway with the DINO [5] pretrained weight and autoregressive decoder (framework proposed in DINOSAUR [33] ) since it is the best working object-centric learning framework in a real-world setting. To see if these settings are essential for the top-down pathway, we have implemented our self-modulation technique with the original slot attention setting [30] , which includes training encoders from scratch and using an image reconstruction objective with spatial broadcast decoder [43] . Tab. A7 summarizes the results of the CLEVR6 dataset. We observe a significant improvement in mBO, showing that our self-modulation technique is applicable to slot attention and provides complementary benefits. Although FG-ARI decreased, mBO is considered more robust when evaluating model performance [25, 12, 21, 33] . We have also included qualitative results in Fig. A5 , which demonstrates that self-modulation markedly improves segmentation. These results indicate our method's effectiveness with different encoder configurations and training objectives.Comparison to MaskCut [41] While object-centric learning (OCL) and unsupervised instance segmentation share the goal of discovering objects without supervision, their ultimate objectives differ. OCL aims to learn object-wise representations that support downstream tasks requiring compositionality and systematic generalization, whereas unsupervised instance segmentation focuses primarily on obtaining accurate object masks. Nevertheless, we can directly compare methods from both tasks on their object discovery capabilities. We evaluate our method against MaskCut, the pseudo-mask generation algorithm underlying CutLER [41] , on the COCO dataset. As shown in Tab. A8, our method significantly outperforms MaskCut across all metrics, demonstrating its effectiveness for object discovery even when compared to specialized unsupervised segmentation approaches. Comparison with SPOT [21] In Tab. A9, we present the comparison with SPOT [21] , a recent stateof-the-art object-centric learning method. SPOT proposed various ideas that can improve the quality of object-centric representation, such as patch order permutation within autoregressive decoder and self-distillation. We want to emphasize that these ideas are all orthogonal to the proposed top-down pathway and can be used together for further improvement, which we will leave as future work. NeurIPS Paper ChecklistJustification: We comply with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: Slot attention has emerged as a popular approach for unsupervised object discovery, and the improvements presented in this paper have been appreciated by all the reviewers. Given that the work is both timely and well-executed, I recommend accepting it.

The key highlights include the use of a visual codebook to self-modulate internal activations by defining the "what" and "where" information based on the attention maps. As noted by reviewer qEia, the model consistently outperforms the underlying DINOSAUR model on which this technique is built. Both the qualitative and quantitative evaluations are impressive. Reviewer YiTH and PJTV has offered valuable suggestions on how the paper could be enhanced by incorporating insights from related work. I hope the authors take these into consideration while preparing the final version.

Reviews: review: paper_summary: paper_summary: This work proposes a novel unsupervised object-centric learning method. In particular, the proposed approach enhances the slot-attention mechanism by incorporating a "top-down pathway" that highlights features relevant to objects in the image. 

The method first employs a standard self-attention mechanism to extract slot vectors and the corresponding attention masks. These initial slot vectors and attention masks are then fed into another round of the slot-attention mechanism. In this round, they modulate the value maps of the cross-attention mechanism separately for each slot. This modulation is channel-wise for slot vectors and spatial-wise for attention masks, enabling the slot-attention mechanism to emphasize to features relevant to the slots discovered on the first slot-attention round. 

Furthermore, the method quantizes the slot vectors from the first slot-attention round using the VQ approach before utilizing them for modulation in the second round.

To evaluate the proposed self-modulated-based slot-attention mechanism, the authors integrate it into the DINOSAUR framework and apply it to the COCO, VOC, MOVIE-E, and MOVIE-C datasets for the downstream task of object segmentation. The results demonstrate improvements over the vanilla slot-attention mechanism. Ablation studies are conducted on the COCO dataset.
strength_weakness: strength_weakness: - **(W1)** It's unusual that the mIoU results in the MOVIE-C and MOVIE-E datasets surpass the mBO results.  The mBO is computed by assigning each ground truth mask the predicted mask with the largest overlap, and then averaging the IoUs of the assigned mask pairs. On the other hand, the mIoU metric is more strict and employs Hungarian matching (rather than greedy matching) to assign predicted masks to the ground truth masks. Consequently, mBO results should be either greater than or equal to the mIoU results. Therefore, it's highly likely that there's a bug in the computation of the mBO or the mIoU metrics on the MOVIE-C and MOVIE-E datasets. Or a typo / mistake when copying the results to the paper maybe.
- As a relevant side note for the upcoming comments, it's important to mention that the FG-ARI metric is generally viewed as unreliable for assessing unsupervised object-centric methods. This is because it only takes foreground pixels into account, which can provide a deceptive understanding of segmentation quality by disregarding the localization accuracy of predicted masks [A, B, C, 30, 42].
-  **(W2)** The motivation behind the Vector Quantization (VQ) component of the method is unclear. In particular it's uncertain why quantizing the slot vectors is essential for self-modulation. In fact, Table 4 indicates that the method is sensitive to the codebook size (also acknowledged by the authors in the limitations section). The mBO performance for sub-optimal codebook sizes 128, 256, and 1024 (30.2%, 30.5%, and 29.8%, respectively) is either the same or worse than the reproduced DINOSAUR mBO performance (30.5% from Table 6). Conversely, removing VQ achieves 32.3%, which is only marginally worse than the 32.7% obtained with VQ and the optimal codebook size. Therefore, VQ unnecessarily complicates the proposed method. As mentioned earlier, FG-ARI is unreliable, and the higher differences in this metric (36.3% w/o VQ vs 37.7% w/ VQ) should be disregarded.
- It's also somewhat concerning that Table 6 shows only a minimal impact on mBO performance when using channel-wise modulation (from 32.5% to 32.7%). However, this does not really increase the complexity of the method, as VQ does.
- **(W3)** Lacks discussions on how the proposed method affects the training and test time
- **(W4)** Table 3 is missing comparisons with related works, such as Rotating Features [D] and SPOT [C], which also demonstrate strong results.

[A] Genesis: Generative scene inference and sampling with object-centric latent representations, ICLR 2020.
[B] Unsupervised Layered Image Decomposition into Object Prototypes, ICCV 2021.
[C] SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers, CVPR 2024
[D] Rotating features for object discovery. NeurIPs 2023.

[30] Bridging the gap to real-world object-centric learning, ICLR 2023.
[42] Slotdiffusion: Object-centric generative modeling with diffusion models, NeurIPs 2023.
questions: questions: - My primary concern regarding this work (mentioned as (W2) in the weaknesses section) is that the channel-wise and Vector Quantization (VQ) components unnecessarily complicate the method without demonstrating significant improvements in the ablation results. I would appreciate it if the authors could provide compelling arguments, such as additional experimental evidence on other datasets (e.g., VOC or MOVIE-E/C) or different settings (e.g., longer training), that show (if it is the case) the benefits of channel-wise modulation with VQ.

- Please address the concerns (W1), (W3), and (W4) mentioned in the weaknesses section.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes a modification to Slot Attention incorporating top-down information into the algorithm. After an iteration of Slot Attention, the slots are quantized into a learned codebook. The quantized slots and attention maps are then used in another iteration of Slot Attention, refining the representations. The algorithm is evaluated in the DINOSAUR setting on the MOVi, Pascal VOC, and COCO datasets, showing improved segmentation quality over vanilla DINOSAUR. Visualizations of the codebook show that meaningful semantic concepts are learned and several ablations are performed.
strength_weakness: strength_weakness: 1. The experiments are only performed in the DINOSAUR setting with pretrained, frozen ViT features as the input and reconstruction target. It would be very informative to also include experiments with the original image as the input and reconstruction target since that is also a common use case for Slot Attention. By only evaluating in the DINOSAUR setting, it is unclear how reliant the performance of the proposed model is on the pretrained ViT features.
2. In Figure 3 and 5.B, it seems the slots are sometimes reassigned before and after the self-modulation step? For example, the skater in Figure 3 is captured by the 4th slot before the modulation and the 5th slot after the modulation. The 5th slot before the modulation seems to capture part of the background, not any part of the skater. It is unclear to me why this would happen during the modulation update.
questions: questions: 1. Since the slots also contain position information (”where” information), it seems possible that this gets captured in the learned codebook. Is this something the authors observed?
2. How cherry-picked is Figure 2? Are there cases where these codes do not correspond to the same semantic concept?
3. In Figure 2, it seems that sometimes, multiple objects with similar semantics (e.g. multiple zebras or signs) are being captured by one slot. Did you notice your proposed method does more semantic grouping instead of instance grouping compared with vanilla DINOSAUR?
limitations: limitations: Yes
paper_summary: paper_summary: Slot Attention is a popular component for object-centric learning methods. In this
paper, the authors propose an extension of Slot Attention named "top-down pathway".
After the last iteration of Slot Attention, the slots are mapped to a discrete, learnable
codebook. Jointly with the final attention map, this information is used in a second
iteration of Slot Attention to modulate the QKV-like attention mechanism. Previous
models are consistently outperformed on established benchmarks. Moreover, a more
detailed analysis of the learned codebook shows that distinct semantic concepts are
represented by different codebook entries.
strength_weakness: strength_weakness: I am not fully convinced by the term "top-down pathway". Only the recurrent Slot
  Attention module has been extended, the rest of the model is still bottom-up. From
  the initial description, I would have expected top-down modulation reaching to the
  encoder.

  Did you consider a variant of the model that continues with the slot embeddings from
  the first iteration instead of starting from scratch? If this performs similarly well,
  it might indiciate that the quantization due to the codebook is driving the improvement.
questions: questions: - A recent approach to object-centric learning which is not based on Slot Attention is
  CutLER (Wang et al. 2023). How does the proposed method compare to CutLER?
limitations: limitations: Limitations are very briefly discussed. An additional point could be the dependence on
the pretrained feature encoder, which might not work well in all scenarios.
paper_summary: paper_summary: This paper proposes a method that improves the performance of object-centric learning by modulating Slot Attention with semantic and location information obtained based on the output slots and attention maps of Slot Attention. For a given output slot, the semantic information is chosen as the vector closest to this slot in a codebook that is learned from all output slots of the entire dataset via vector quantization, and the location information is chosen as the shifted attention map with a mean value of 1. The proposed method is implemented based on DINASOUR and is compared with DINASOUR and other methods on two synthetic and two real datasets. The proposed method outperforms the compared method in most cases when the size of codebook is chosen appropriately.
strength_weakness: strength_weakness: 1. My main concern about the paper is that the performance of the proposed method is very sensitive to the size of codebook. If the size of codebook is not chosen appropriately, the performance can be even worse than the original DINASOUR method. Moreover, how to choose the best size of codebook is not described clearly.
2. From my understanding, the core of the proposed method is a self-modulation module (along with a codebook) that is compatible with all the object-centric learning methods using the slot attention mechanism. However, the proposed method is only implemented based on DINASOUR. The quality of the paper could be significantly improved if the proposed method is implemented based on at least another object-centric learning method that is developed based on slot attention.
3. The proposed self-modulation includes both semantic and spatial modulations. However, the codebook used in the semantic modulation also contains spatial information. This design can be more elegant, and the codebook can be much more useful if it only contains semantic information. In this way, whether two objects belong to the same category can be determined automatically.
4. Some data in the experimental part are inconsistent. For example, the mBO^i of the proposed method on COCO is 33.0 in Table 2, 33.3 in Table 3, and 32.7 in Table 4.
5.  The authors attribute both semantic modulation and spatial modulation as top-down modulations. I agree that the semantic modulation is a top-down modulation because the codebook contains knowledge learned from all images (not just the inferred image). However, I don’t think that the spatial modulation can be considered top-down because it just uses attention maps computed in a bottom-up manner (only based on the content of one image).
questions: questions: 1. How is the best size of codebook chosen?
2. Is it possible to learn a codebook that is independent of the locations of objects in the image?
limitations: limitations: The authors have addressed the limitation in the conclusion part, and there is no potential negative societal impact.

Comments: title: Reply to Rebuttal
comment: We sincerely appreciate your careful review of our responses and your reconsideration. We're glad our explanations about codebook size selection and VQ benefits clarified your concerns. In our future revision, we will include discussions and experiments about VQ, which will significantly improve the paper's quality. Thank you for your valuable feedback!
I thank the authors for the detailed responses to my comments. Their response  addresses my main concerns regarding the necessity of VQ: the codebook size selection is automatic and uses the training split, and VQ offers bigger improvement for longer training. Therefore, and after reading the other reviews and the respective rebuttals, I am going to increase my score.
We sincerely appreciate your detailed response and thoughtful consideration of our rebuttal. We will address your feedback as follows:

- We will elaborate on the term "top-down pathway" in the paper to clarify its usage and context.
- We're pleased you found the additional comparisons to MaskCut helpful. We will include this in the future revision.
- Regarding codebook concerns: a) While determining optimal codebook size requires multiple training runs, perplexity is measured at half the full training schedule (250K), reducing computational burden. b) On the necessity of vector quantization, we kindly refer you to our response to reviewer 3ckb, specifically points (1), (3), and (4) under **"Necessity & effect of VQ"**, which provide further insights into VQ's role in our method's performance.

We truly appreciate your valuable feedback and will incorporate your suggestions to improve our paper's final version. Thank you for your support!
We appreciate your thorough review and time evaluating our rebuttal. We're pleased our updated figures and CLEVR experiments have resolved your concerns. We acknowledge your suggestion and will include CLEVR results from multiple seeds in the final paper. Thank you for your constructive feedback and recognizing the strength of our work.
We sincerely appreciate your thoughtful reconsideration of our work. We are pleased that our clarifications regarding codebook size determination and the CLEVR6 experiment have addressed your concerns. We will improve the paper in future revision based on your feedback. Thank you for your valuable feedback and updated assessment.
Thank you for the rebuttal and running additional experiments. The updated figures for the slots before and after modulation make much more sense now. The additional results with vanilla Slot Attention on CLEVR are also promising, although I would encourage the authors to run additional seeds and/or make sure the hyperparameters are correct, as I've seen better results with vanilla slot attention on CLEVR (looking at the qualitative results it seems the background is being split). For what it's worth, I think it would still be a useful contribution if the proposed approach only improves the metrics on more complex datasets (since as a field, we should be generally pushing towards more complex datasets), but it is important to know the effect on simpler datasets as well so researchers are aware of the limitations. I would encourage the authors to include these results in the final version of the paper. After reading the rebuttal as well as the other reviews and rebuttals, I have decided to increase my score to 7.
Thank you very much for the detailed response.

- I am still not entirely convinced about the tmerin "top-down pathway", however I think
  this not a major issue that speaks against accepting the paper.
- The additional comparisons to MaskCut and other methods in the responses to other
  reviewers are very helpful and confirm the consistent improvements of the proposed
  model.
- I agree with the other reviewers on the concerns regarding the codebook size and the
  unclear role of the vector quantization. (1) While the codebook size can be chosen
  without supervision, it seems that the required multiple training runs substantially
  increase the computational cost. (2) The impact of the codebook on the performance
  seems to be small and heavily decreases the performance when used alone, as shown by
  the additional ablation study. While I still think the analysis of the codebook is
  interesting, I am not sure to which extent it is necessary for the performance
  improvement.

Overall, I still think the paper should be accepted, since the proposed method is
interesting and consistently outperforms previous methods. Due to the remaining concerns
I keep my rating for now. But I am looking forward to read the other reviewers comments
on these points and happy to discuss further.
Thanks for the detailed response. Since the codebook size is determined solely based on the training set in an unsupervised way and applying the proposed self-modulation to Slot Attention also improves the performance on the CLEVR6 dataset, my concerns about technical flaws disappear and the rating is increased.