Improving Object-centric Learning with Query Optimization
Authors: Baoxiong Jia
Yu Liu
Siyuan Huang
Venue: ICLR 2023 poster (2023)
Keywords: unsupervised object-centric learning

Abstract: The ability to decompose complex natural scenes into meaningful object-centric abstractions lies at the core of human perception and reasoning. In the recent culmination of unsupervised object-centric learning, the Slot-Attention module has played an important role with its simple yet effective design and fostered many powerful variants. These methods, however, have been exceedingly difficult to train
without supervision and are ambiguous in the notion of object, especially for complex natural scenes. In this paper, we propose to address these issues by investigating the potential of learnable queries as initializations for Slot-Attention learning, uniting it with efforts from existing attempts on improving Slot-Attention learning with bi-level optimization. With simple code adjustments on Slot-Attention, our model, Bi-level Optimized Query Slot Attention, achieves state-of-the-art results on 3 challenging synthetic and 7 complex real-world datasets in unsupervised image segmentation and reconstruction, outperforming previous baselines by a large margin. We provide thorough ablative studies to validate the necessity and effectiveness of our design. Additionally, our model exhibits great potential for concept binding and zero-shot learning. Our work is made publicly available at https://bo-qsa.github.io.

Published as a conference paper at ICLR 2023
IMPROVING OBJECT-CENTRIC LEARNING WITH QUERY OPTIMIZATION
Baoxiong Jia 1ia Yu Liu 2,3 Siyuan Huang"
UCLA "Tsinghua University 3 National Key Laboratory of General Artificial Intelligence, BIGAI
ABSTRACT
The ability to decompose complex natural scenes into meaningful object-centric abstractions lies at the core of human perception and reasoning. In the recent culmination of unsupervised object-centric learning, the Slot-Attention module has played an important role with its simple yet effective design and fostered many powerful variants. These methods, however, have been exceedingly difficult to train without supervision and are ambiguous in the notion of object, especially for com- plex natural scenes. In this paper, propose to address these issues by investigat- ing the potential of learnable queries as initializations for Slot-Attention learning, uniting it with efforts from existing attempts on improving Slot-Attention learning with bi-level optimization. With simple code adjustments on Slot-Attentin, our model. Bi-level Optimized Query Slot Attention, achieves state-of -te--etertttt results on 3 challenging synthetic and 7 complex real-world datasets in unsupervised image segmentation and reconstruction, outperforming previous baselines by a large margin. We provide thorough ablative studies to validate the necessity and effectiveness of our design. Additionally, our model exhibits great potential for concept binding and zero-shot learning. Our work is made publicly available at https: //bo-////////// qsa. github.
INTRODUCTION
Objects, and their interactions, are the foundations of human cognition (Spelke & Kinzler, 2007), The endowment on making abstractions from perception and organizing them systematically empowers humans the ability to accomplish and generalize across a broad range of tasks, such as scene modeling (Bear et al., 2020), visual reasoning (Yi et al., 2020), and simulating interactions (Bear et al., 2020). The key to such success lies in the emergence of symbol- mental representations of object concepts (Whitehead, 1928). However, important as it is, disentangling object-centric concepts from visual stimuli is an exceedingly difficult task to accomplish with limited supervision (Greff et al., 2020) and requires proper inductive biases (Schölkopf et al., 2021).
Motivated by the development of symbolic thought in human cognition, blot- representations, instance (Greff et al., 2017; 2019; Locatello et al., 2020), sequential (Gregor et al., 2015; Burgess al., 2021; Goyal et al., 2021), or spatial (Crawford & Pineau, 2019; Lin
et al., 2019; Engeleke et al., 2020; Jiang et al., 2019), have been the key inductive bias to recent advances in unsupervised object-centric learning. Among them, the Slot-Atennton module has received tremendous focus given its simple yet effective design (Locatello et al., 2020). By leveraging the iterative attention mechanism, Slot-Attention learns to compete between slots for explaining parts of the input, exhibiting soft- clustering effect on visual signals. It is later proven to be more memory and training efficient as alug-and-play module for unsupervised object-centric learning (Locatello al., 2020) and fostered powerful variants in understanding images (Singh et al., 2021; Xu et al., 2022), 3D scenes (Yu et al., 2022; Sajjadi al., 2022a) and videos (Kipf al., 2022; Elsayed al., 2022; Singh et al., 2022).
However, as revealed by recent studies, the Slot-Attentin module comes with innate discrepancies for object-centric representation learning. First, with slots randomly initialized each time, the object- centric representations obtained by these models do not necessarily bind to object concepts (Kipf et al., 2022). Intuitively, such randomness leads to undesired scenarios where slots with similar
$ Equal contribution. Work done during internship BIGAL.

---

Published as a conference paper at ICLR 2023
initializations compete for objects on different images. Such randommess challenges the iterative refinement procedure as it now needs to project sets of potentially similar representations to indepen- dent constituents of the input. As discovered by Chang et al. (2022), differentiating through such recurrences contributes to various training instabilities with growing spectral norm of Slot-Attention weights. This leads the second and perhaps least desired property of Slot-Attention; it relies heavily on hyper-parameter tuning, including gradient clipping, learning rate warm-up, etc., and further hurts the flexibility of Slot-Attention in adapting broader applications with more complex signals.
To this end, we propose an extension of the Slot-Attention module, Bi-level Optimized Query Slot Attention (BO-QSA), to tackle the aforementioned problems. First, we follow the bi-level optimiza- tion framework proposed by Chang et al. (2022) for easing the training difficulty in Slot-Attention More importantly, instead of sampling from a learnable Gaussian distribution, we propose to directly learn the slot initializations as queries. With these learnable representations, we eliminate the am- biguous competitions between slots and provide a better chance for them to bind to specific object concepts. We improve the training of query-initializee Slot-Attention with straight-through gradient estimator (STE) by connecting our method with first-order approaches (Finn et al., 2017; Nichol & Schulman, 2018; Geng et al., 2021) in solving bi-level optimization problems. The experimental results show that the proposed BO-QSA can achieve state-of of-thheartr results on both synthetic and real-world image datasets with simple code adjustments the original Slot-Attention module.
With our model significantly outperforming previous methods in both synthetic and real domains, we provide thorough ablative studies demonstrating the effectiveness of our model design. We later show that our BO-QSA possesses the potential of binding object concepts to slots. To validate this potential, we design zero-shot transfer learning experiments to show the generalization power of our model on unsupervised object-centric learning. As the experiments suggest (see Sec. 5), our model could potentially be a principle approach for unsupervised object-centric learning and serve as a general plug-and-pla module for a broader range of modalities where variants of Slot-Attention prosper. We hope these efforts can help foster new insights in the field of object-centric learning.
Contributions In summary, our main contributions are three-fold:
We propose BO-QSA, a query-initializedd Slot-Attention model that unites straight- throuuhthhrughhhhh gradient updates to learnable queries with methods on improving Slot-Attention with bi-level optimization. We show that, with simple code adjustments on Slot-Attention, the proposed BO-QSA achieves state-of-the-ard results on several challenging synthetic and real-world image benchmarks, outper- forming previous methods by a large margin.
We show the potential of our BO-QSA being a better approach to concept binding and learning generalizable representations with qualitative results and zero-shot transfer learning experiments.
2
PRELIMINARIES
2.1 OBJECT-CENTRE REPRESENTATION LEARNING WITH SLOT-ATTENTION
Slot-Attention (Locatello et al., 2020) takes set of N input feature vectors E N* Dune and maps them to a set of K output vectors (i.e., slots) s e R K It leverages an iterative attention mechanism first map inputs and slots to the same dimension D with linear transformations k(-), q()) and v(-) parameterized by @ast At each iteration, the slots compete to explain part of the visual input by computing the attention matrix A with softmax function over slots and updating slots with the weighted average of visual values:
Aij IN
(xx)) q(s)
s= farm (s,x)
(xx) where A softmax
NNK
Av.
VD
The slots are initialized from learnable Gaussian distribution with mean f and variance o. They are refined iteratively within the Slot-Attention module by passing the updates into a Gated Recurrent Unit (GRU) (Cho et al., 2014) and MLP parameterized by dupdate for T iterations:
(1+1) hamer (5()) g()), s ~~~ ( diag(o)), (T)
The final prediction can be treated as the learned object-centric representation w.r.t. to input features In the image domain, take as input a of images I and encode them with fone to obtain
2

---

Published as a conference paper at ICLR 2023
features RBW Dapu, After obtaining & through the iterative refinement procedure with hgnnae, images could be decoded from these object-centric representations with a mexxture-based decoder autoregressive cransfomer-bassee decoder. We refer the readers to Appendix A.1 for details on different decoder designs and their ways of visualizing learned object concepts.
2.2 IMPROVING SLOT-ATTENTION WITH BI-LEVEL OPTIMIZATION
The problem of livel optimization embeds the optimization of an inner objective within the outer objective. Normally, a bi-level optimization problem can be formulated as:
min f .)) s.t. A arg min g(0', +), 9,6
#r
where we call f(0, ()) the outer objective function and g(8, Φ) the inner objective function. To jointly optimize both objectives w.r.t. parameters 0 and Φ, a straightforward approach solving Eq. (2) is to represent the inner solution of 0 as a function of Ø, i.e., (Ø) = arg ming g(0", 6). Then we can optimize the outer objective with gradient descent by approximating Vof( (Ø), Φ) as a function of 6. When the inner optimization objective could be solved by fixed point iteration & = Fs(() (Amos Kolter, 2017; Bai et al., 2019), the bi-level optimization problem could be solved by
é ( (b). o) (T))
F (0*) 80
(F*( dá
d
800
For efficiency concerns, recent methods often use the first-order approximation of the infinite Neumann's 's series (Shaban al., 2019; Geng al., 2021) for updating Given that Slot-Attention is, in essence, an iterative refinement method that falls into the same framework, Chang et al. (2022) adapted this technique to improve Slot- Attentinn training and obtained significant improvement both in model performance and training stability. We provide more discussions on this in Sec. 3.2 and also other bi-level optimization methods for approximating V of (0)) Φ) in Appendix A.2.
METHOD
3.1
QUERY SLOT ATTENTION
As mentioned in Sec. 1, the Slot-Attention module adopts a random initialization of slots and conducts iterative refinement to obtain object-centric representations argued by Kipf et al. (2022), such random initializations provide no hint on the notion of object and no means for controllably probing concepts from the model. As shown by Chang al. (2022), this random initialization plays minimal role and could be detached from training. This indicates that the estimation of relies heavily on the task specifee iterative refining of slots over data, leaving a limited possibility for slots to bind to specific concepts and be leveraged as generalizable representations.
in Eq. (1). However, as
To address this issue, we focus on the Query Slot Attention (QSA), which initializes the slots in the Slot-Attention module with learnable queries So Ginit, Such design is motivated by the success of recent query-based based networks (Van Den Oord et al., 2017; Jaegle al., 2021b). facilitates an object- centric model to learn general symbolic-like representations that could be quickly adapted by refining over task-specific requirements, as discussed in Sec. and Kipf et al. (2022). Meanwhile, in contrast to the use of learnable queries in other encoder-decoder structures (e.g. discrete VAE (dVAE)), the slot initializations Sy are not necessarily required to encode image features since they were designed for separating them. This resembles recent discoveries in query networks (Carion et al., 2020; Yang et al., 2021) where queries could be generalizable probes for input properties. Despite the good properties and potentials QSA presents, it is shown detrimental to initialize slots independently in Slot-Attention under unsupervised settings (Locatello et al., 2020).
3.2 RETHINKING BI-LEVEL OPTIMIZATION METHODS FOR QUERY SLOT ATTENTION
To improve the learning of QSA, rewind the idea of improving the learning of the vanilla Slot-Autention module with bi-level optimization (Chang et al., 2022). Under this formulation, Alot-Attentinn could be treated as solving the following objectives:
M
min
L(x; Si, D)
s.t.
s = arg min Letuster (x,, +).
(4)
3

---

Published as a conference paper at ICLR 2023
where x; and S; denote the input feature from the i-th image and its corresponding slots, and = f 6ith,, dupdate) denotes parameters for assigning input features x to different slots. Under this setting, the outer objective L is usually reconstruction objective and the inner objective could be viewed as a soft-clustering objective (Locatello et al., 2020). Next, the inner objective is solved by iterative refinement, which could be formulated as solving for fixed-points (Chang al., 2022) of
(5) where F.(...) -) is fixed-point operation. As introduced by Chang et al. (2022) in Implicit Slot- Attention (I-SA), with Eq. (3), the instabilities through the iterative updates could be avoided by detaching gradients, treating slots in the final iteration as approximation of s/ and computing first-order gradient approximations for updating with s.. However, we demonstrate in Tab. 7 that this design is only beneficial for randomly initialized slots and detrimental for query-initialized Slot-Autention architectures since it relies heavily on the good approximation of the solution to the inner objective. With no randomness in slot initializations or gradient during training, starting from a fixed set of initialization points puts challenges on the learning of Slot-Attention update F& as it will be difficult provide a good approximation of sf with only a fixed number of iterations (see in Appendix B.2). This urges the need for information flow to the slot initialization queries.
8 = Rigpotne (s, $) = hgge (s, f (s, x)) = Fs x),
3.3 BI-LEVEL OPTIMIZED QUERY SLOT ATTENTION
We propose BO-QSA to address the learn- ing problem of QSA. As shown in Algo- rithm 1, we initialize slots with learnable queries in BO-QSA and perform T steps of Slot-Attention update to obtain an approxima- tion of s/ These near-optimal solutions of the inner objective are passed into one addi- tional Slot-Attention step where gradients to all previous iterations are detached. In contrary to I-SA, use STE (Bengio et al., 2013; Van Den Oord et al., 2017) to backpropagate gradi- ents and also to slot initialization queries. Such designs help find good starting points for the inner optimization problem on clustering, alleviating the problem of bi-level optimization with QSA mentioned in Sec. 3.2. Similar to dVAE, the STE adds bias to the gradient of the initialization queries. However, since these learnable queries are meant for disentangling image features, they do not have to maintain information about the approximated s*. Such bias could lead to learned queries which are better pivots for separating different image features, similar to anchors, or filter queries Iearned for different tasks (Carion ct al., 2020; Zhang et al., 2021). Note that we do not add constraints the consistency between So and (e.g. ||sg($) ()) soll3) as done in d'VAE since we find such constraints lead to a mean-representation of datasets that forbids better concept binding (see in Appendix B.3). As shown in Tab. 7 and Fig. 3, our learned slot initialization queries do fulfill this goal by providing more separable initialization space and can significantly facilitate model learning.
Algorithm 1:
BO-QSA
Input: input features input, learnable queries ini c. number of iterations T
Output: object-centric representation slots Modules :stop gradient module SG(-). slot attention module SA(;))
s slots = init for t 1, 1,... .T do I slots = SAlsssSA(slsss. inputs) slots = SG(slots) init SGGinit) slots SA(slots. inputs) return slots
4 RELATED WORK
Unsupervised Object-Centric Learning Our work falls into the recent line of research on unsu- pervised object-centric learning on images (Greff al., 2016; Eslami et al., 2016; Greff et al., 2017; 2019; Burgess et al., 2019; Crawford & Pineau, 2019; Engelcke et al., 2020; Lin et al., 2020; Bear et al., 2020; Locatello et al., 2020; Zoran et al., 2021). A thorough review and discussion on this type of method can be found in Greff et al. (2020). One critical issue of these methods is on handling complex natural scenes. Singh al. (2021); Lamb al. (2021) leverages transformer-based decoder with Slot-Attention for addressing this problem. Similar attempts have also been made by exploiting self-supervised contrastive learning (Choudhury et al., 2021; Caron et al., 2021; Wang et al., 2022: Hénaff et al., 2022) and energy-based models (Du et al., 2021; Yu et al., 2022). Our work builds upon Slot- Attentitni by extending it with learnable queries and a novel optimization method for learning. Our compelling experimental suggests our model could potentially serve as general plug-and and and-play module for a wider range of modalities where variants of Slot-Attention prosper (Kipf et al., 2022; Elsayed et al., 2022; Singh et al., 2022; Yu al., 2022; Sajjadi et al., 2022a;b).
4

---

Published as a conference paper at ICLR 2023
Query Networks Sets of latent queries commonly used in neural networks. These methods leverage permutation equivariant network modules (e.g. GNNs (Scarselli et al., 2008) and attention modules (Vaswani et al., 2017)) in model design for solving set-related tasks such as clustering (Lee et al., 2019), outlier detection (Zaheer et al., 2017; 7: Zhang et al., 2019), etc. These learned latent queries have been shown to have good potential features for tasks like contrastive learning (Caron et al., 2020), object detection (Carion et al., 2020), and data compression (Jaegle et al., 2021a;b). In contrast the recent success of query networks in supervised or weakly-supervised learning (Carion et al., 2020; Zhang et al., 2021; Kipf et al., 2022; Elsayed et al., 2022; Xu et al., 2022), Locatello et al. (2020) demonstrates the detrimental effect of using independently initialized slots in Slot-Attention learning. However, we show that our BO-QSA method successfully overcomes this issue and generalizes the success of query networks to the domain of unsupervised object-centric learning.
Bi-level Optimization Our work is closely related to bi-level optimization methods with iterative fixed update rules for solving the inner objective. Specifically, methods are designed with implicit differentiation (Amos & Kolter, 2017: Bai et al., 2019) to stabilize the iterative update procedure. Similar formulations are also found when combined with meta-learning where Madan et al. (2021) train queries through recurrence in meta-learning fashion and Rajeswaran et al. (2019) provides a unified view of the optimization problem with implicit gradients. Concurrent work from Chang al. (2022) formulate the Slot-Attention learning from implicit gradient perspective with gradient stopping derived from first-order hyper-gradient methods (Geng et al., 2021). However, they ignore the important role of slot initializations in generalization and concept binding. As our experiments suggest, such gradient-stopping methods do not guarantee superior performance compared to the original Slot-Attenton. We leave the details to Sec. 5.3 for an in-depth discussion.
EXPERIMENTS
In this section, we aim address the following questions with our experimental results:
How good is our proposed BO-QSA on both synthetic and complex natural scenes?
How important the query and the optimization method in BO-QSA?
Does BO-QSA possess the potential for concept binding and zero-shot transfer?
We provide details in the following sections with thorough comparative and ablative experiments and leave the details model implementation and hyperparameter selection to Appendix A.3. Here we clarify the datasets and metrics selected for evaluating our model on each domain:
Synthetic Domain For the synthetic domain, we select three well-established challenging multi- object datasets Shapestacks (Groth al., 2018), ObjectsRoom (Kabra et al., 2019), and CLEVRTEX for evaluating our BO-QSA model. Specifically, we consider three metrics to evaluate the quality of object segmentation and reconstruction. Adjusted Rand Index (ARI) (Hubert & Arabie, 1985) and Mean Segmentation Covering (MSC) (Engelcke al., 2020) for segmentation and Mean Squared Error (MSE) for reconstruction. Following the evaluation setting recent works, we report the first two segmentation metrics over foreground objects (ARI-FG and MSC-FG). Additionally, we conduct extra experiments on more datasets and leave the discussion to Appendix B.
two tasks (1) unsupervised foreground extraction and (2) unsupervised multi-object segmentation for evaluating our method. Specifically, we select Stanford Dogs (Khosla et al., 20120 1), Stanford Cars (Krause al., 2013), CUB200 Birds (Welin- der et al., 2010) 0),0), and Flowers (Nilsback & Zisserman, 2010) as our benchmarking datasets for foreground extraction and YCB (Calli al., 2017), ScanNet (Dai et al.. 2017), COCO (Lin et al., 2014) proposed by Yang & Yang (2022) for multi-object segmentation. We use mean Intersection over Union (mloU) and Dice as metrics for evaluating the quality of foreground extraction and use the evaluation metrics adopted by Yang & Yang (2022) for multi-object segmentation.
Real-world Images For the real image domain,
5.1 OBJECT DISCOVERY ON SYNTHETIC DATASETS
Experimental Setup We explore our proposed BO-QSA with two types of decoder designs, mixture-base based and transformer-bast based, as discussed Sec. 2.1 and Appendix A.1. We follow the decoder architecture in Slot-Attentin (Locatello al., 2020) for mixture-based decoders and
5

---

Published as a conference paper at ICLR 2023
Table 1: Multi-object segmentation results on ShapeStacks and ObjectsRoom. We report ARI-FG and MSC-FG of all models with (mean variance) across experiment trials. We visualize the best results bold.
ShapeStacks
ObjectsRoom
Model
ARI-FG
MSC-FG
1 ARI-FG
1 MSC-FG
MONet-G (Burgess et al., 2019)
0.70
0.57:10.12
0.54
0.33-10.01
GENESIS (Engelcke al., 2020)
0.70++0.05
0.67:t0.02
0.63+-0.03
0.53:00.07
Slot-Attenton (I locatello al., 20200
0.76:10.01
0.70:10.05
0.79:1:002
0.64
GENSIS-V2 (Engeleke et al,, 2021)
0.81110.11
0.67-1:0.0
0.86:10.01
0.59
SLATE (Singh et ul., 2021)
0.65
0.63-t:005
0.57:+0.03
0.30
1-SA (Chang et al., 2022)
0.90
0.85
0.85
0.76
Ours (transformer)
0.68:10.02
0.70110.02
0.68
0.72-10.03
Ours (mixture)
0.93++00011
0.89
0.87
0.80
Table 2: Multi-object segmentation results on CLEVRTEX. We report ARI-FG (%) and MSE of all models in the form of (mean + variance) across 3 experiment trials. We visualize the best results in bold.
CLEVRTEX-FULL
CLEVRTEX
CLEVRTEX CAMO
Model
1 ARI-RG
MSE
1 TAI-FG
MSE
1 ARI--G
[ MSH
MONe: (Burgess 2019)
1907
146-7
37.29:1.04
4091:
31.52=0.87
265
Slee-Ateetion locatello 2020)
62.40=2.38
254-8
18.45=1.87
16
57.54=1011
215=7
GENSIS-V2 V2 (Engelcke al., 2021)
11.1941241
315=106
29.04-111.33
139
29.60::2.884
278-75
ETTI (Monnier ol., 2021)
79.90= 1.37
438-222
73.67:00.98
59012-4
72.90=1189
377=17
(Chang el
3.886
0.888
241:14
17.20111.228
295-30
Ours (mixture)
20.47:2.499
268:22
86.50:00119
2654:25
63.71=6.11
280:27
SLATE (Singh et al., 2021) for transformer-bast decoders. For both types of models, we use the Slot-Attention module with CNN image encoder and initialize slots with learnable embeddings.
Results We report multi-object segmentation results on synthetic datasets in Tab. I and visualize qualitative results in Fig. As shown in Tab. 1, our BO-QSA achieves the saaee-ofthhe-e results with large improvements over previous object-centrid learning methods on all metrics in ShapeStacks and ObjectsRoom. We also observe more stable model performance, i.e. smaller variances results, across different trials of experiments. Our model with mixture-based decoders obtains the best overall performance on all datasets. More specifically, our mixture- based BO-QSA significantly outperforms the vanilla Slot- dlot-Atteniion model (~15%) with minimal architectural differences. This validates the importance of the learnable queries and our optimization method. We will continue this discussion in Sec. 5.3. As shown in Tab. 2, our model also achieves state-of-the-an art results the unsupervised object segmentation task in CLEVRTEX with consistent improvement over Slot-Attention on the CAMO and OOD generalization split. Interestingly, our model (1) shows larger reconstruction errors, (2) generalizes well in outto--distribution scenarios, and (3) shows marginal improvement in camouflaged images. We attribute (1) and (3) to the simple architecture of encoders/decoders currently adopted and provide insights on (2) in Sec. 5.4.
Mixture-based vs. Transformer-basee De- coder We observe inferior segmentation but superior reconstruction performance of transformer-basee variants of Slot-Attention on synthetic datasets. Specifically, we compare the MSE of models on ShapeStacks and Object- sRoom. As shown in Tab. 3, transformer based methods provide better reconstruction results. We attribute the low segmentation performance to mask prediction in these methods, which relies the attention matrix computed over input features. This leads to coarse object masks as a result of image tokenization. Nonetheless, we observe consistent improvement by applying our slot encoder to both mixture and transformer decoders.
Table 3: Reconstruction results on ShapeStacks and ObjectsRoom (. We compare mixture- based and transformer-bas based decoder designs.
Model
ShapeStacks
ObjectsRoom
Slot-Attentin (mixture)
80.8
20.4
ours (mixture)
72.0
8.1
SLATE (Iranfforme)
52.3
16.3
ours (transformer)
49.3
14.7
5.2 OВJЕCT DISCOVERY ON REAL DATASETS
Experimental Setup For real-world experiments, we use the same slot encoder design used in Sec. 5.1 with 4-layer CNN image encoder and initialize slots with learnable queries. For

---

Published as a conference paper at ICLR 2023
Table 4: Unsupervised multi-object segmentation results on YCB, ScanNet, and COCO variant proposed by Yang Yang (2022). We use the same evaluation metries as in Yang Yang (2022) and report all models' results with (mean (variance)) over 3 experiment trials. We visualize the best results in bold.
YCB (AP//RQ/ Peo Roci
ScatNee (AP/PQ/ Red]
COCO
Model
INQ Pho Rrec
(Eslami
(0
7.
2.7
MoNer (lhrggss 2019)
7TD 29 (30
01 28 346-1.60
(1.9 12.00 12.50/11511 O 1Y
TODINE (Grellf et 2019)
(0.2) (L.3) 26 22.00 (2.0)23333333 (1.9))
10.1 22.9)/ 13.7 (2.5) (&B
24A
4.0(12)/63 (1.22//66.2 (L8)) (0.
Skn-Anemiion clacslllla et al. amto
9-2.0.6) 13.5 (0/5) N1 3/ 26.30000
5.7 (0:31 ( 57 12.4 12.51 (3.7
0.800 3.50 71 ( (2.33
1SA (Charg of 2022)
31.5 (15.2) 25.6 (P/5 36,1 (12.5) 40.55/440222 (11.9)
21.4 (6.80 23 (3.)))))) 29L1 (7.8)) (7.00
12.8 (4.3)) 13.7 (4-5) 20A (6.0)/22..0001 (7..6
Our transformet
(1.3)/ 99A (1:11/ (377)
(27,3 43.4
(8/60 17.6 (1,6) 25.3 /
Table Unsupervised foreground extraction results CUB200 Birds (Birds), Stanford Dogs (Dogs), Stanford Cars (Cars), and Caltech Flowers (Flowers). visualize the best results in bold.
Binks
Cars
Flowers
Model
IoUU
[ Dicc
1 IoU
Dioc
1 |oOU
Dice
1 loU
1 Dice
ReDO (Chen et all 2019)
46.5
60.2
55.7
70.3
52.5
76.4
IODINE (Grett et al., 2019)
30,9
44.6
54.4
67.0
51.7
67.3
OneGAN (Benny Wolf, 2020)
55.5
69.2
71,0
81.7
71,2
Slot-Artention (1 (Locatello et ol. 2020)
35.6
51.5
39.6
55.3
41.3
58.3
30.8
45.9
Voynov al. (2020)
68.3
54.0
DRC (Yu et al., 2021)
56.4
70.9
71.7
83.2
72.4
83.7
Melas Kyriazi
(2021)
SLATE (Singh et al,, 202
36.1
51.0
62.3
76.3
75.5
85.9
68.1
79.1
1.5A (Chang
2022)
80.6
85.9
75.0
Ours (mixiure)
25.1
39.2
36.8
53.6
69.1
81.5
36.1
51.6
Ours (trnsffommr)
71.0
82.6
82.5
90.3
87.5
93.2
78.4
86.1
ShapeStacks
Stanford Dogs
CUR200 Rirds
Scan
Irpa
Peol
Reon
aT 5eg
AL
Ped.
Sep.
ObjectsRoom
Stanford Cars
YCB
COCO
Figure 1: Visualization of our predicted segmentation and reconstruction results on synthetic and real images. We color the predicted mask that has maximum intersection with the ground ground-truth background in black.
unsupervised foreground extraction, we follow Yu et al. (2021) and report the best model performance on all datasets. During the evaluation, we select the slot's mask prediction that has maximum intersection with the ground-truth foreground mask as our predicted foreground. For unsupervised multi-object segmentation, we follow Yang & Yang (2022) and report the models' performance on all datasets across trials with different random seeds.
Table Unsupervised segmentation results on Birds (mloUf). *Contrastive learning methods are pre-trained on ImageNet and segment with K-means clustering.
Results We show quantitative experimental results in Tab. 5 and Tab. 4. We also visualize qualitative re- sults Fig. 1. For multi-object segmentation, as shown in Tab. 4, our model outperforms existing object-centric learning baselines by large margin, especially on the YCB dataset where the segmented objects have clear se- mantic meanings. For foreground extraction, as shown in Tab. 5, our method significantly outperforms all existing baselines on the task foreground extraction, achieving new state-of-ther -ottr---rttttttto on all datasets. We recognize the dis-
Model
Birds
MoCo v2 (Chen et al., 2020)
63.5
BYOL (Grill et al., 2020)
56.1
R20 (Gokul al., 2022)
71.2
ours (BO-QSA+transfommee)
71.0
7

---

Published as a conference paper at ICLR 2023
Table 7: Ablative experiments on slot initialization and optimization methods. We visualize the best results in bold and underline the second-bes best results. N*Note that SA represents Slot-Attentin with our encoder-decoder design and different the original one reported Tab. 5.)
75.0
70.0
ShapeStacks
60.0
SA
Method
1-5A
IoU
Dice
ARI-PG(%)
MSC-FG((%)
B0-SA
QSA
I-SA
80.8
89.2
88.1
76.8
50.0
I-QAA
QSA
64.5
72.9
88.1
76.1
45.0
B0-QSA
BO-QSA (ours)
82.5
90.3
92.9
89.2
Number of iterations
Figure 2: Effects iterative updates in testing.
Ap
SA
I-SA
I-QSA
BO-QSA
Figure 3: Visualization of learned slot initializations and post-iteration slots after the first iteration of Slot-Attention on ShapeStacks (we use dots for initialization vectors and inverse triangles for post-iteration slots). We show our BO-QSA provides the best overall separation as well as correspondence between initialization vectors and post-iteration slots. For I-SA, there exist mismatches between initialization vectors and post-iteration slots (yellow and red). The same optimization method is also not effective for I-QSA, leading to mixing post-iteration slots similar SA for slot initializations (best viewed in color and with zoom-in).
crepancy of mixture-basee decoders in both Slot-Attention and our mixture- -bseed design in modeling real-world images, reflecting similar discoveries from recent works (Singh et al., 2021) that mixture- based decoder struggles modeling real-world images. the other hand, our transformer-basel model shows significant improvements over the vanilla version. Notably, our method outperforms broad range models, including GAN-based generative models (i.e. OneGAN, Voynov et al. (2020)), and large-scale pre-trained contrastive methods (i.e. MoCo-v2. BYOL, R20). As shown in Tab. 6, our method achieves comparable results with state-of-the-art self-supervised contrastive learning methods without large-scale pre-training and data augmentation. This result sheds light the potential of object-centric learning as pre-training task for learning general visual representations.
5.3 ABLATIVE STUDIES
Experimental Setup We perform ablative studies over our designs by comparing them with different design variants ShapeStacks and Stanford Dogs. For slot initialization, consider (1) the original Slot-Attention module's sampling initialization (SA), and (2) initializing with learnable queries (QSA). For optimization, we consider (I) the original optimization Slot-Attention (i.e. w/o detach or STE), (2) the I-SA optimization where gradients to slots in iterative updates are detached (i.e. w/ detach only), and (3) our optimization where we both detach the gradients into iterative refinement, and pass gradient to the initialization queries with STE (i.e. w/ detach and STE). For simplicity, we term these variants with prefixes (I-) for I-SA and (BO-) for our full method. We run all ablations each dataset with the same decoder architecture.
We show experimental results Tab. and Fig. 2. First, from Tab. we observe that BO-QSA significantly outperforms other variants. For sample-based slot initializations, our method shows a similar effect compared with I-SA on improving Slot-Attentio learning. For query-based slot initializations, we validate the difficulty in training query-base Slot-Attention with its inferior performance. We further show the ineffectiveness of I-SA for query-based Slot-Attention. The experiments query- based Slot-Attention prove that both of our design choices necessary and effective for superior performance. To study the effect of learned queries, we visualize in Fig. where set different numbers of iterative updates of Slot-Attentio during inference the Stanford
Results
8

---

Published as a conference paper at ICLR 2023
Trazake
Trasder
Tinadi
COKOO Scnnia
Figure 4: Visualization of learned concepts and attention maps in zero-shot transfer. At the top, we visualize the per-slot reconstruction of our model trained on ShapeStacks (left), Birds (middle), and YCB (right). At the bottom, we show that learned slot initialization queries bind the experiments (f.e. color in ShapeStacks to CLEVRTEX, contours Birds to Dogs and Cars, and spatial positions in YCB to ScanNet and COCO) by visualizing attention maps of slot initialization queries over input images. *Note that for the ShapeStacks experiment(left, alternate object colors in CLEVRTEX with seen colors for better qualitative evaluations, and do perform such operations for quantitative evaluations.
concepts in zero-shot transfer
Dogs dataset. We can see that our BO-QSA significantly outperforms other variants with only one iteration. This indicates that query-based design can help ease training difficulties. In Fig. further visualize the learned initializations and post-iteration slots in the same feature space using I-SNE (Van der Maaten & Hinton, 2008). Our initializers provide a more separable space when differentiating image features, which validates the desired model behaviors mentioned in Sec. 3.3.
5.4 ADDITIONAL ANALYSES
In this section, we provide additional analyses on the potential of our BO-QSA as a concept binder for generalizing to examples. First, qualitatively visualize our learned con- tent for each slot (without additional clustering) in ShapeStacks, Birds, and YCB in Fig. We observe high similarity within the learned content of each slot, indicating similar concepts learned by specific slots. This shows the potential of the slots in our BO-QSA for binding specific concepts object properties (e.g. colors, contours, and spatial positions). Although we can not control which concepts to learn, these results are important indicators that our learned initialization queries could potentially be generalizable concept probes. We further
provide quantitative evaluations where we use models trained on dataset X for zero-shot in- ference on dataset Y. We term this transfer as (X-Y)) shown Tab. 8, when adapting models trained YCB to zero-shot inference on ScanNet and COCO, our method outper- form I-SA and also the majority of fine-tuned methods shown in Tab. 4. Due to the page limit, we show in Appendix B. that this superior transfer capability is general across datasets when compared Slot-Attention variants.
Table Zero-shot transfer results unsupervised multi- object segmentation on real images.
YCB SconNer (AP: Pre Ree)
YCB COC0
Model
(AP/PQ/ Pre Rec)
1.37
16.35 31.811.811
1.20/ 4.97 66.75
1:A
34.19
18.39 35.79
QS-QSSA (ouns)
CONCLUSIONS
We introduce BO-QSA for unsupervised object-centric representation learning. We initialize Slot-Attention with learnable queries, and combine bi-level optimization and straight-through gra- dient estimators to ease the difficulty in query- query-based Slot-Attention learning. With simple code adjustments on Slot-Attention, we obtain state-of-the-art model for unsupervised object segmentation in both synthetic and natural image domains, outperforming previous baselines by a large margin. More importantly, our learned model exhibits concept-binding effects where visual concepts are attached specific slot queries. With fixed number of initialized slots, our model limited to handling a fixed maximum number of objects in the inputs. However, our queries could be learned to bind object attributes, which leads to meaningful segmentation of images by grouping similar proper- ties (e.g. color, position, etc.). As future direction, this connects our method with weakly-supervised contrastive learning methods that learn grounded visual representations with language.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We gratefully thank all colleagues from BIGAI for fruitful discussions. We would also like to thank the anonymous reviewers for their constructive feedback. This work reported herein was supported by National Key R&D Program of China (20212D0150200),
REFERENCES
Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization layer in neural networks. In Proceedings of International Conference on Machine Learning (ICML), pp. 136-145 145, 2017. 3, 5
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019. 3,5
Daniel Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li F Fei-Fei, Jiajun Wu, Josh Tenenbaum, et al. Learning physical graph representations from visual scenes. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020. 1,4 4
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3445 2013. 4
Yaniv Benny and Lior Wolf. Onegan: Simultaneous unsupervised learning of conditional image generation, foreground segmentation, and fine-gra aie-ggained clustering. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:/901. 1390, 2019. 1,4, 7
Berk Calli, Arjun Singh, James Bruce, Aaron Walsman, Kurt Konolige, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. Yale-cmu-berkeley dataset for robotic manipulation research. International Journal of Robotics Research (IJRR), 2017.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. Proceedings of European Conference
Computer Vision (ECCV), 2020. 5,
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of International Conference Computer Vision (ICCV), 2021. 4
Michael Chang, Thomas Griffiths, and Sergey Levine. Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022. 2, 5,
Mickaël Chen, Thierry Artières, and Ludovic Denoyer. Unsupervised object segmentation by redrawing. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS),
2019. 7
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04200 042997 2020.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. In Proceedings of the conference on Empirical Methods Natural Language Processing (EMNLP). 2014.
10

---

Published as a conference paper at ICLR 2023
Subhabrata Choudhury, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Unsupervised part dis- covery from contrastive reconstruction. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2021. 4
Eric Crawford and Joelle Pincau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of AAAI Conference Artificial Intelligence (AAAI), 2019. 1,
Angela Dai, Angel Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Nießner. Scannet:
Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2021.
Gamaleldin F Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022. 1, 4,
Martin Engelcke, Adam R Kosiorek. Oiwi Parker Jones, and Ingmar Posner. Genesis: Genera- tive scene inference and sampling with object-centric latent representations. Proceedings of International Conference Learning Representations (ICLR), 2020. 4, 5,
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2 Inferring unordered object representations without iterative refinement. In Proceedings of Advances in Neural Information
Processing Systems (NeurIPS), 2021. 1,
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2016. 4,7
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. In Proceedings of International Conference on Machine Learning (ICML), 2017. 2017.
2, 17
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learinn. In Proceedings of International Conference Machine Learning (ICML), 2018. 17
Zhengyang Geng, Xin-Yu Zhang, Shaojie Bai, Yisen Wang, and Zhouchen Lin. On training implicit models. In Proceedings Advances in Neural Information Processing Systems (NeurIPS), 2021.
2, 3,5
Akash Gokul, Konstantinos Kallidromitis, Shufan Li, Yusuke Kato, Kazuki Kozuka, Trevor Darrell, and Colorado Reed. Refine and represent: Region-to-obyee representation learning. arXiv preprint ar arXiv:2208.1 11800, 2022. 7
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. In Proceedings of International Conference on Learning Representations (ICLR), 2021. 1
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harr Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2016.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2017. 1,
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. Proceedings of International Conference on Machine Learning (ICMI), 2019. 1, 7,
11

---

Published as a conference paper at ICLR 2023
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05508,, 2020. 1,4
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra, Draw: A recurrent neural network for image generation. Proceedings of International Conference on Machine Learning (ICML), 2015.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach self-supervised learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020. 7
Oliver Groth, Fabian B Fuchs, Ingmar Posner, and Andrea Vedaldi. Shapestacks: Learning vision- based physical intuition for generalised object stacking. In Proceedings of European Conference on Computer Vision (ECCV), 2018. 5
Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisser- man, João Carreira, and Relja Arandjelović. Object discovery and representation networks. In Proceedings of European Conference on Computer Vision (ECCV), 2022. 4
Yining Hong, Li Yi, Josh Tenenbaum, Antonio Torralba, and Chuang Gan. Ptr: A benchmark for part-based conceptual, relational, and physical reasoning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2021. 21
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 2(1):193-218, 1985.
Andrew Jaegle, Sebastian Borgeaud, Jean-Raptiste Alayrac, Carl Doersch, Catalin Ioneseu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint wxXiv22107.17795, 2021a.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In Proceedings of International Conference on Machine Learning (ICML), 2021b. 3,5
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. In Proceedings of International Conference on Learning Representations (ICLR), 2019.
Rishabh Kabra, Chris Burgess, Loie Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datasets. https:/giitubbcom//ccpmm////i/n object-datasets/, 2019.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grad fine-grained image categorization: Stanford dogs. In Proc. CVPR workshop on ime-grained visual categorization (FGVC). 2011. 5
Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. In Proceedings of International Conference on Learning Representations (ICLR), 2022. 3, 4,5
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of International Conference Computer Vision Workshops (ICCVW), 2013. 5
Alex Lamb. Di He, Anirudh Goyal, Guolin Ke, Chien-Feng Liao, Mirco Ravanelli, and Yoshua Bengio. Transformers with competitive ensembles of independent mechanisms. arXiv preprint arXiv:2103.00336, 2021. 4
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: framework for attention- permutation-ines invvaiian neural networks. In Proceedings of International Conference on Machine Learning (ICML), 2019.
12

---

Published as a conference paper at ICLR 2023
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of European Conference on Computer Vision (ECCV), 2014. 5
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang. and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. Proceedings of International Conference on Learning Representations (ICI.R, 2020. 1,4
Francesco Locatello, Dirk Weissenborn Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020. 1, 2,
5, 6. 7, 18
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2020. 17
Kanika Madan, Nan Rosemary Ke, Anirudh Goyal, Bernhard Schölkopf, and Yoshua Bengio. Fast and slow learning of recurrent independent mechanisms. In Proceedings of International Conference on Learning Representations (ICLR), 2021.
Luke Delas-Kyriazi, Christian Rupprecht, Laina, and Andrea Vedaldi. Finding an unsupervised image segmenter in each of your deep generative models. arXiv preprint arXiv:2105.08127, 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 2021. 22
Tom Monnier, Elliot Vincent, Jean Ponce, and Mathieu Aubry. Unsupervised layered image decom- position into object prototypes. In Proceedings of International Conference on Computer Vision (ICCV), 2021. 6
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint 1733099803,02999 2018. 2, 17
Maria-Elena Nilsback and Andrew Zisserman. Delving deeper into the whorl of flower segmentation. Image and Vision Computing, 1849)1004- 2010. 5
Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In Proceedings of International Conference on Machine Learning (ICML), 2016. 17
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2019. 5,
17
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proceedings of International Conference on Machine Learning (ICML), 2021. 16
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučić, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS),
2022a. 1,4
Mehdi SM Sajjudi. Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, al. Scene representation transformer: Geometry-free novel view synthesis through let-latent scene representations. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-80, 2008.
13

---

Published as a conference paper at ICLR 2023
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. In Proceedings of the IEEE, 2021.
Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated pack-propagation for bilevel optimization. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2019. 3
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. Proceedings of
International Conference on Learning Representations (ICLR), 2021. 1, 4, 6, 7, 8, 16, 18
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022. 1, 4, 16
Elizabeth S Spelke and Katherine D) Kinzler. Core knowledge. Developmental science, 10(1):89-96. 2007.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2017. 3, 4, 16
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine
Learning Research (JMLR), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2017. 5
Andrey Voynov, Stanislav Morozov, and Artem Babenko. Big gans are watching you: Towards unsupervised object segmentation with off-the-shelf generative models. 2020. 7,8
Yangtao Wang. Xi Shen, Shell Xu Hu, Yuan Yuan, James L Crowley, and Dominique Vaufreydaz. Self- supervised transformers for unsupervised object discovery using normalized cut. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 4
Nicholas Watters, Loie Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint arXiv:1901.07017 2019. 16
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010. 5
Alfred North Whitehead. Symbolism: Its meaning and effect. Journal of Philosophical Studies, (12), 1928.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1,
Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. In Proceedings of International Conference Computer Vision (ICCV), 2021.
Yafci Yang and Bo Yang. Promising or clusive? unsupervised object segmentation from real-world single images. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022. 5,7
Mao Ye, Bo Liu, Stephen Wright, Peter Stone, and Qiang Liu. Bome! bilevel optimization made easy: A simple first-order approach. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022. 17
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. In Proceedings of International Conference Learning Representations (ICI.), 2020. 1
14

---

Published as a conference paper at ICLR 2023
Hong- Xing-Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. In Proceedings of International Conference on Learning Representations (ICLR), 2022. 1, 4, 22
Peiyu Yu, Sirui Xie, Xiaojian Ma, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu, Unsupervised fore- ground extraction via deep region competition. In Proceedings Advances in Neural Information Processing Systems (NeurIPS), 2021.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets, Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2017. 5
Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Temporal query networks for fine-grained video understanding. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 5,5
Yan Zhang, Jonathon Hare, and Adam Prugel-Bennett. Deep set prediction networks. In Proceedings of Advances in Neural Information Processing Systems (NeurlPS), 2019. 5
Daniel Zoran, Rishabh Kabra, Alexander Lerchner, and Danilo J Rezende. Parts: Unsupervised segmentation with slots, attention and independence maximization. In Proceedings of International Conference Computer Vision (ICCV), 2021. 4
15

---

Published as a conference paper at ICLR 2023
A MODEL ARCHITECTURE AND DESIGN
A.1 DESIGN OF DECODERS
In this section, follow the notations used Sec. 2.1 and describe two common approaches, mixture-based and transformer-bascd for decoding images from the learned slot representations.
Mixture- Decoder The mixture-based decoder (Watters et al., 2019) decodes each slot into img an object image x; and mask m, with decoding functions 9gde and maakk which are implemented using CNNs. The decoded images and masks are calculated by:
img 9 giize (S:),
exp
9
mask
L.
m.
j==1
exp
C (8₁)
During training, a reconstruction objective is employed for supervising model learning. Despite its wide usage, mixture- mixture-based decoders showed limited capability at handling natural scenes with high visual complexity (Singh et al., 2021).
Autoregressive Transformer Decoder Recently, Singh al. (2021; 2022) reveal the limitations of mixture decoder and leverage transformers and dVAEs (Van Den Oord et al., 2017; Ramesh et al., 2021) for decoding slot-based object-centric representations. To obtain decoded images 1, they learn a separate dVAE for first encoding into a sequence of L tokens = [z..... ... 2 with dVAE encoder faNAE, Next, they use transformer decoder dlte granslormer to auto-regressively predict image tokens with learned slot representation S:
transformer g
S; ) where
ddVAE Ar
To train the entire model, we have the reconstruction objective supervising the learning of z with dVAE decoder g Next, the objective for object-centrie learning relies the correct prediction from the ruto-regressive transformer for predicting correct tokens:
C = LdVAE + LCE where LaVAE = 11gghe (z) -112, LCE
dVAF
Σ CrossEntropy(zz, (zi, oi)
Under this setting, the model does not predict additional masks and relies on the attention A within the Slot-Attention module for obtaining slot-specific object masks. Although such models can achieve competitive results on real-world synthetic datasets, as our experiments suggest, they can be inferior to mixture-basee decoders on segmentation in synthetic datasets. We suspect that this originates from the low resolution when discretizing images into tokens.
A.2 BI-LEVEL OPTIMIZATION AND LEATNILG
Recall the bi-level optimization problem we introduced in Sec. 2.2.
min f(, Φ) s.t. A arg min g(0'))(()))))) Φ),
0.0
where call ((0, Φ) the outer objective function and g(0, Φ) the inner objective function. To jointly optimize both objectives w.r.t. parameters 0 and 6, a straightforward approach to solving Eq. (6) is to represent the inner solution of as function of o, i.e., 8* (*()) (Ø) arg ming g(0', Φ). Then we can optimize the outer objective with gradient descent:
Vof(0* ()). v0 (o) (0), 6) + (o).
However, the difficulty of this method lies in the calculation of V (0) where we need to solve linear equation from implicit gradient theorem:
V1 0* (0), Φ) V 0 (6) + 22 (0), Φ) 0.
If (0* Φ) is invertible, solve for V V (Φ) and obtain the gradient update d:
Φk+1 Φb E (V2Jk
(V,29k) - v
16

---

Published as a conference paper at ICLR 2023
Initialization Querles
glait
Input Image
Reconstruction
EnC
Dec
Learned Slots
Features
galn gpdaats
Figure 5: An illustrative visualization of our proposed BO-QSA slot-encoder. During the backward pass, BO-QSA STE to backpropagate gradients directly o ini & amt and o "pdate without gradients into the itcrative process.
where ViSk (0x)) Φk) and Vife
V1 ()(()()))) (b)), Various methods have been proposed to approximate the solution (Pedregosa, 2016; Lorraine et al., 2020), and we refer the authors to Ye al. (2022) for a thorough review of related methods.
Bi-level optimization is closely related to meta-leaming, In meta-learning, we have meta-training tasks which comes in N different collections of datasets D (D, Dj UD The inner and outer objectives in Eq. (6) are substituted by averaging training and validation errors over multiple tasks (Franceschi al., 2018):
Ji=1
N Σ 0, s.t. 0, min Σ 0
min f0, 8.0
o;, o;
where L: represents dask-dependent error D, The final goal of meta-leaming aims seeking the meta-parameter & that is shared between tasks which later enables few-shot learning and fast adaptation. With its connections with bi-level optimization, the previously mentioned optimization methods are broadly adapted for solving meta-learning problems (Finn et al., 2017: Nichol & Schulman, 2018; Rajeswaran al., 2019). From the meta-learning perspective, our attempt shares similar insights with first-order meta-learning methods (Finn et al., 2017; Nichol & Schulman, 2018), where we use the gradient at some task-specific optimal solution S of the inner optimization for optimizing slot initialization queries which are shared across datasets on the outer objective. This meta-learning perspective also indicates the potentials of our BO-QSA for fast adaptation and generalization.
A.3 IMPLEMENTATION DETAILS
We provide visualization of our designed slot-encoder in Fig. 5 and discuss the implementation details for different experimental settings the following sections.
A.3.1 SLOT INITIALIZATION
We initialize all models with the number of slots shown in Tab. 13, During training, we add a small perturbation to the queries by sampling from a zero-mean distribution with variance a as we found it empirically helpful for better performance. We perform annealing over to gradually eliminate the effect of this random perturbation during training. We adopt the cosine annealing strategy such that starts from I and gradually anneals to 0 after N, training steps, where N₂ is a hyperparameter that controls the annealing rate of o. In our experiments, we use N₂ on Cars and Flowers and N₂ 30000 the rest of the datasets.
17

---

Published as a conference paper at ICLR 2023
Layer
Kernel Size
Stride
Padding
Channels
Activation
Conv
5x5
1(2)
64
ReLU
Conv
5x5
1
2
64
ReLU
Conv
5x55
64
ReLU
Conv
5x5
64
ReLU
Table Configuration of CNN encoder used in our model. The values parentheses are adopted for CLEVRTex
and StapeStacks
Layer
Kernel Size
Stride
Padding
Channels
Activation
TransConv
5x5
2
2
64
ReLU
TransConv
5x5
2
2
64
ReL.U
TransConv
5x5
2
2
64
ReLU
TransConv
5x5
2(1)
2
64
ReLU
TransConv
5x5
1
2
64
ReLU
TransConv
3x3
1
1
4
None
Table 10: Configuration of mixture decoder used in our model. ObjectsRoom
The values in parentheses are adopted for
A.3.2 BO-QSA WITH MIXTURE-BASE DECODERS
For mixture-based decoders, we use the same Slot-Attentio architecture as in Locatello et al. (2020) with slots initialized by learnable queries. Given an input image, Slot-Attention uses CNN encoder to extract image features. After adding positional embedding, these features are input into the Slot-Attention module slot updates. Finally, these slots are decoded by the mixture decoder to reconstruct the input image. We provide the details of our image encoder in Tab. 9. For the mixture- based decoder, we use six transposed convolutional layers with ReLU activations following Locatello et al. (2020). We visualize the details of our mixture-based decoder design in Tab. 10. We train our model for 250k steps with batch size of 128 and describe training configurations and hyperparameter selection Tab. 11.
Batch Size 128
LR 4e-4
Slot Dim 64
MLP Hidden Dim
128
Warmup Steps
Decay Steps 50k
Max Steps
Sigma Down Steps
250k
30k
Table 11: Training configuration for mixture-based model
A.3.3 BO-QSA WITH TRANSFORMER-BAN DECODER
For transformer- decoders, we adopt the transformer architecture proposed by SLATE (Singh et al., 2021), For the transformer-basee BO-QSA, unlike SLATE, we use the same CNN as in mixture-based BO-QSA (instead of dVAE encoder) to extract features from the image as input to the Slot-Attention module prediction mentioned in Sec. 5.1. Next, we use the same overall architecture of dVAE as mentioned in SLATE Singh al. (2021). However, we change the kernel size of the dVAE encoder from 1 to since we find that such changes can help increase model performance when decomposing scenes. We train model for 250k steps with batch size of 1 128, and all the training configuration in experiments is described in Tab. 12.
find such changes help solve the problem coarse object boundary
A.3.4 BASELINES
The reproduction of Slot-Attention and SLATE follows the architecture and hyperparameter selection mentioned in their paper. Similar to models, train all baseline models with 250K steps all datasets. For SLATE, we use the input image size of 96 on the ShapeStacks dataset as we find that the image size of 128 will cause all objects to be divided into the same slot, resulting in low
18

---

Published as a conference paper at ICLR 2023
batch size
128
warmup steps
10000
Training
learning rate
lc-4
max steps
250k
vocabulary size
1024
Gumbel-Softmax annealing range
1.0 to 0.1
dVAE
Gumbel- -oomee-Softmax annealing steps
30000
Ir-dVAE(no warmup)
3e-4
layers
4
heads
Transformer Decoder
dropout
0.1
hidden dimension
256
slot dimension
256
Slot Attention Module
iterations
annealing steps
30000(0)
Table 12: Training configuration for transformer baasd model. The values in parentheses adopted for Cars and Flowers dataset
ARI and MSC. For fair comparison with numbers reported SLATE's paper, we report the MSE of models by first computing per-pixe enrors and then multiplying it by the total number of pixels. For CLEVRTEX, we follow the same experimental setting (BO-QSA+mixune) and set the number of slots to 11. For YCB, ScanNet, and COCO, we follow the same experimental setting of -0o-SSAAtrranfoommer) for birds and set the number slots
for ShapeStacks
Model
Shapestacks
ObjectsRoom
Birds
Dogs
Flowers
Cars
Slot-Attention SLATE
2
2
2
2
# of slots
BO-QSA +Mixture
2
2
BO-QSA +Transformer
12
6
2
2
Image Size
128
64
128
128
128
128
Table 13: The number of slots and image size used for each dataset
B ADDITIONAL EXPERIMENTS
B. ZERO-SHOT TRANSFER
In this section, we continue the discussion in Sec. 5.4 and provide additional zero-shot transfer results. Similarly, we use the notation (X Y) to denote the zero-shot adaptation models trained unsupervisedly on dataset X to new datasets Y.
For unsupervised multi-object segmentation, we report transfer results from ScanNet and COCO to all other real-image multi-object segmentation datasets in addition to the results on YCB (mentioned Sec. 5.4). As shown in Tab. 14, our model shows consistent improvement over Slot-Attention and I-SA during zero zero-shot transfer.
Table 14: Zero-shot transfer results unsupervised multi-ohject segmentation on real images.
Model
COCO
ScanNet
(AP/ Pie Ree)
(AP/ Pre Roc)
(AP PQ Pie Rec) 1
(AP PQ Pre Roc)
19.63/19.24 28 36/4133
14.86 22.06 26.74
26.53/23.00 35
20.99 22.08 32,14
1-SA
18.66 / 18, 56 28 97 30.82
11.83/ 14.14 20.70 25.42
26.72. 22.2//22.000 90 35 89 37
19.34 20.00 00 44 38
BO-QSA (ouns)
21.85 19.96 31.51 33.45
13.95 16.04 23.35 28.49
31.21 / 25.44 38.90 41.35
24.21 23.59 34,07 38.49
For unsupervised foreground extraction, we report transfer results from Stanford Dogs and CUB200 Birds all other real-image foreground extraction datasets. As
from Tab. 15, model
19

---

Published as a conference paper at ICLR 2023
achieves the overall best results compared with other powerful Slot-Attention variants (models that achieve best or second- best results in our ablation studies as in Tab. 7) except for (Birds-Cars)) However, our optimization method still helps improve zero-shot transfer for randomly initialized Slot-Autention.
Table 15: Zero-shot transfer results on unsupervised foreground extraction (mloU ().
Model
Dogs -+ Cars
Dogs -
Flowers
Dogs -- Birds
Birds
Dogs
Birds - Cars
Birds
Flowers
SA I-SA
$7.96
$7.96
45,06
74.68
58,79
62.02
58.05
58,06
48.88
71.16
69.90
BO-SA
58, 10
58.10
47.96
71.81
70.75
67.95
BO-QSA (ours)
75,50
63.43
52.49
76.66
66.74
70.74
B.2 ANALYSIS NUMBER OF SLOT-ATTENTION ITERATIONS
As described in Sec. 3.2, we study whether a fixed point s* could be reached by a fixed number of iterations during training. Since we hypothesized that the low performance of I-QSA in Sec. 5.3 originated from the insufficient number of starting points for fixed-point approximation, we conduct experiments on increasing the number of Slot-Attentin iterations during training for I-QSA on the Dog dataset. As shown in Tab. 16, increasing the number of Slot-Attention iterations during training for I-QSA significantly improves its performance. However, we found that adding more iterations after threshold (i.e. 7 in this case) does not further improve the overall performance. This verifies the need for learning slot initialization vectors for better approximating the fixed point solution the inner soft-clustering objective in Slot-Attention.
Table 16: Increasing the number of iterations during training for I-QSA.
Dogs
Model # of Training Iterations
IoU
Gain
Dice
Gain
I-QSA I-QSA
3
59.3
77.6
7
80.5
+35.8%
88.9
+14.6%
Ours
3
82.5
90.3
B.3 DESIGN CHOICES ON SLOT INITIALIZATION
As described in Sec. 3.3, our method is connected with recent works on dVAE. However, we do not require the initialization queries to maintain information about the post-iteration slots $ as we found such constraints lead to the learning of the mean representation of datasets which forbids disentanglement and concept binding. In this section, we provide experimental results to verify this argument. Specifically, we consider three different ways update slot initialization queries in addition to our proposed method: 1) using the running mean of the post-iteration slots as initialization queries (RunningMean), 2) running K-Means clustering on post-iteration slots and updating the initialization queries using re-clustered centers by Hungarian matching (KMeans), 3) adding consistency loss between initialization queries and post-iteration slots as done in VQ-VAE (VQ-constraint) For (I) and (2), empirically found such designs to be suffering from frequent updates and therefore use momentum updates to stabilize their training. We term these variants with the suffix (-M).
Table 17: Comparison between update methods for slot-initialization queries.
Metrics
RunningMean RunningMean-M
KMcans
KMcans-M
VQ-constraint
Ours
ARI-FG (ShapeStacks)
7.5
51.4
21.0
70.6
88.6
92.9
MSC-FG (ShapeStacks)
3.7
15.4
4.2
60.4
85,3
89,2
As shown in Tab. 17, model achieves the best overall performance compared to other initialization methods. Specifically, we found that using the runming mean of post-iteration slots or K- K-Means cluster centers re-clustered from post-iteration slots to be harmful to model performance. We attribute this
20

---

Published as a conference paper at ICLR 2023
RunningMean
RunningMean-M
KMeans
KMcins-M
VQ-constraint
Ours
Figure 6: Visualizations per-slot reconstruction for different update methods. show that Running and KMcans suffer at decomposing the image, even with momentum updates. For VQ-constraint, though the model variant achieves similar but slightly inferior effect segmentation, they can not preserve the same filtered property for each slot across images.
effect to the learning of the mean-representation datasets. This is further proved in experiments with VQ-VAE loss on consistency between slot initializations and post-iteration slots (i.e. ||sg($) soll ) where the VQ-constraint variant showed inferior performance. We also found that the weight of this additional loss needs to be carefully tuned for the model to decompose objects. Empirically, most configurations of this hyperparameter will lead to bad reconstructions except for certain small weights (e.g. 0.01 reported here). Above all, we believe these experimental results verify the effectiveness of our design choices on initialization query learning. We provide additional visualizations on the learned contents of slots for each update method in Fig. 6.
B.4 EXPERIMENTS ON ADDITIONAL DATASETS
In addition datasets considered in Sec. 5, we conduct experiments other synthetic datasets and visualize qualitative results. More specifically, we test our model on PTR (Hong et al., 2021). PTR is synthetic dataset of 3D objects from PartNet with rendering variations. We our BO-QSA with the same configuration mentioned in Appendix A.3 previously. We compare our method with the vanilla Slot-Attention module multi-object segmentation. We report ARI-FG and MSC-FG scores of our model compared with the vanilla Slot-Attention on the PTR validation set.
Table 18: Multi-object segmentation results on PTR. We visualize the best results bold.
PTR
Model
ARI-FG 1
MSC-FG
Slot-Attention ours (BO-QSA+mixtur))
0.72
0.21
0.75
0.61
As we can see from Tab. 18, our model achieves similar performance compared with Slot-Attention on ARI-FG and significantly outperforms it on MSC-FG. We attribute this result to the capability of precisely segmenting objects. As ARI-FG applies masks each slot prediction for calculating results, it does not require models to precisely segment the object from the background. However, MSC-FG uses a mloU-like measure that requires the model to precisely predict the object boundaries. This indicates that our model is better precisely segmenting objects without noise. Similarly, we observe the binding certain slots to scene backgrounds, but with more complex concepts, the binding of slots concepts is not straightforward as in StapeStacks and CUB200 Birds.
21

---

Published as a conference paper at ICLR 2023
To further investigate the effectiveness and generality of our method, we adapt BO-QSA to the recent 3D object-centric learning model, uORF (Yu et al., 2022), and test on 3D datasets including CLEVR- 567, Room- Chair, and Diversee uORF can decompose complex 3D scenes from single image by combining NeRF (Mildenhall et al., 2021) with Slot-Attention. We only modify the initialization and optimization method of the Slot-Attenton module in uORF, leaving all other hyperparameters unchanged. As we can see from Tab. 19, with our method, the uORF model that trained with 600 epochs can achieve a similar even superior result compared the original model trained with 1200 epochs. Additionally, when the dataset complexity increases (e.g., in Room- Diverse), our method demonstrates significant improvement. Please refer to uORF (Yu al., 2022) for more details about the model, datasets, and evaluation metrics.
Table 3D-ohject segmentation results on CLEVR-50 Room-Chair, and noom-Diverse. We visualize
best results in bold and underline the second-best results. "indicates reimplemented results.
Dataset
Model
Train-epoch
NV-ARI"
ARI
ARI-FGI
LPIPS]
SSIM
PSNR
600
66.8
73.8
81.0
0.1249
0.8763
27.84
OORF
1200
84.4
87.4
85.3
0.0869
0.8985
29.32
CLEVR-567
600
74.5
82.9
89.1
0.0783
0.9153
30.07
uORF-BO-QSA
1200
77.7
86.9
89.5
0.0711
0.9223
30.64
600*
37.9
39.4
18.8
0,2932
0.7734
25.08
uORF
1200
77.9
80.3
91.8
0.0845
0.8762
29.66
Room-Chair
600
76.9
79.8
94.6
0.0821
0.8850
30.13
WORF+BO-QSA
1200
80,5
83.2
93.8
0.0733
0.8938
30.61
120*
51.2
60.1
62.0
0.2139
0.6905
25.21
oORF
240
56.6
68.5
66.7
0.1820
0.7146
25.92
Room-Diverse
120
60.4
70.0
75.1
0.1657
0.7137
26.38
uORF-BO-QSA
240
63.0
72.8
76.6
0.1533
0.7378
26.85
C LIMITATIONS AND FUTURE WORK
We discuss all limitations of work found in the experiments. First, observed strong correlation between the powerfulness of encoder-decoder architectures and model performance. However, in contrast supervised learning, more powerful encoders/decoders do guarantee superior performance. Gaining insights from how contrastive learning methods have shown the effect of concept emergence with large-scale pretraining, we can also incorporate such representations learned by self-supervised learning into object-centric learning to unite the best of both worlds. Second, our work is primarily limited by the fixed number of slot initialization vectors. In contrast to the vanilla Slot-Attent Alte-ntttntion that could generalize to new number of objects, our model can not casily generalize to scenarios with new concepts since our model learns fixed of separating spaces that best disentangle different parts of the image. This problem is also frequently met in semantic segmentation and object classification, where we can only use existing concepts to interpret novel objects/semantic entities. Although solutions to this proposed in supervised classification and segmentation, we leave the exploration of this problem in object-centric learning to future work. Finally, the current learned slot initialization vectors do not explicitly bind towards concepts and need to be mined by humans. We believe this is an important next step in our current work combine unsupervised object-centric learning with semantic alignments from language for concept grounding. This opens future research directions on learning finer-level organization of object concepts under more complex scenarios (e.g. hierarchical grouping) with weak supervision of correspondence.
problem have been
22

---

Published as a conference paper at ICLR 2023
D ADDITIONAL VISUALIZATIONS
We provide more qualitative results of our model on different datasets in the following pages.
Figure 7: Unsupervised Multi Multi-Object Segmentation CLEVRTEX.
Figure 8: Unsupervised Multi Aulti-Object Segmentation on PTR.
23

---

Published as a conference paper at ICLR 2023
Figure 9: Unsupervised Multi-Object Segmentation on ShapeStacks.
24

---

Published as a conference paper at ICLR 2023
Figure 10: Unsupervised Multi-Object Segmentation ObjectsRoom, In contrast ShapeStacks, we observe
consistent binding of slots ground, wall, sky, and also objects in the front.
25

---

Published as a conference paper at ICLR 2023
Figure 11: Unsupervised Foreground Extraction CUB200 Birds.
26

---

Published as a conference paper at ICLR 2023
Figure 12: Unsupervised Foreground Extraction Stanford Dogs.
27

---

Published as a conference paper at ICLR 2023
Figure 13: Unsupervised Foreground Extraction on Stanford Cars.
28

---

Published as a conference paper at ICLR 2023
Figure 14; Unsupervised Foreground Extraction Caltech Flowers.
29

---

Published as a conference paper at ICLR 2023
Figure 15: Unsupervised Multi-Ohject Segmentation on YCB.
30

---

Published as a conference paper at ICLR 2023
*> #1200 4XX
Figure 16: Unsupervised Multi-Objec Segmentation on ScanNet.
31

---

Published as a conference paper at ICLR 2023
A
Figure 17: Unsupervised Multi-Ohject Segmentation COCO.
32


Reviews:
[{"review_id": "uTZkKkqWuo", "reviewer": "Reviewer_G5Uw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes two design choices to improve unsupervised object-centric representation learning of Slot Attention. The two design choices are to learn the initial slot embeddings instead of sampling them from a Gaussian distribution, and to skip the gradients through the iterative slot refinement by using a straight-through gradient estimator. This setup is extensively evaluated in experiments on unsupervised foreground segmentation, object-centric representation learning, and zero-shot transfer of foreground segmentation.", "summary_of_the_review": "The proposed design choices are simple yet effective changes to a popular object-centric representation learning method. While easily understandable, the paper could be improved by having a clearer connection to its (intuitively) bi-level optimization objective and more discussions on out-of-distribution results. Overall, I think it is a good paper that is of interest to a larger community.", "strength_and_weaknesses": "## Strengths\n\n- The paper proposes two simple, but yet very significant improvements in Slot Attention considering the shown experimental results. \n- The paper presents the two design choices clearly. In terms of re-producing the architecture, it requires changing only a few lines in the original Slot Attention model. Hence, the model could easily be adapted by many works in the field and be reproduced.\n- To my understanding, the design choices do not introduce any additional hyperparameters and overall stabilizes Slot Attention. This is also important for future adaptations.\n- The paper conducts extensive experiments on various datasets. I highly appreciated the variety of datasets, and it gives confidence that the method does not outperform Slot Attention just 'by luck' on a few cherry-picked settings.\n\n## Weaknesses\n\n- The paper spends a considerable amount of space on discussing the possible motivation behind the bi-level optimization, but it seems to me that the bi-level optimization is rather just an intuition on how one could connect the proposed setup to first-order meta-learning methods. For instance, what is the precise formulation of $\\mathcal{L}_{\\text{cluster}}$ that the model (implicitly) tries to solve? In Section 3.2, motivations are discussed for why it might be related to k-means, but it remains again a bit ambiguous. For the space used, I would have expected a clearer connection to the bi-level optimization.\n- The paper misses a bit out on discussing the potential disadvantages of its method. For instance, by learning an initialization of the slots, the authors mention that they specialize to certain concepts. This would suggest that it may not work too well in out-of-distribution datasets that differ more in the 'concepts' learned. The zero-shot study is one step towards it, but the concepts remain quite similar in terms of foreground vs background. A more challenging situation is the out-of-distribution and CAMO evaluation datasets of CLEVRTEX. Since the authors already provide training results on the CLEVRTEX dataset, I believe it would be also important to report the evaluation scores for the OOD and CAMO part, even if the results may not outperform previous baselines. \n- Additionally, since slots specialize on concepts, what happens if a test set image has a larger amount of objects of the same concept than seen in the training set, e.g. 10 red objects in ShapeStacks?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TndDuuuQf6", "reviewer": "Reviewer_HyQy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper draws a connection between slot attention and bi-level optimization.\nAs a consequence, the authors suggest to learn the initial query slots rather than sampling them as in previous work. Furthermore, the design choice of Chang 2022 w.r.t. gradient propagation is adopted and validated.\nEmpirical results show the potential of this method for unsupervised instance segmentation on synthetic datasets and foreground/background segmentation on real-world datasets.\n", "summary_of_the_review": "On the first pass through the paper, I was initially excited to read about the connection between bi-level optimization and slot attention, also thanks to the higher reported scores on several benchmarks. However, I then realized that this connection is not well-motivated and does not add any new insight. The message of the paper could have been simply \"learn the query vectors with a straight-through estimator\".\n\nThe paper is well written and the authors conducted experiments on several datasets, but I think the paper does not contribute novel knowledge. Surely, the empirical numbers are higher, but the choice of experiments is not on point and does not justify acceptance. I recommend rejecting the paper, but I remain open to changing my mind upon reading the rebuttal and discussing with the other reviewers.", "strength_and_weaknesses": "Strengths:\n\n- In terms of implementation, the paper proposes two straightforward design changes on top of regular slot attention: 1) initial slots are learned rather than sampled as in Locatello 2020, 2) gradient updates are skipped in intermediate steps as Chang 2022 and are propagated to the initial slots using a straight-through estimator.\n\n- I appreciate the thorough experiments which cover both synthetic and real datasets and different decoder designs. The proposed method achieves higher scores on all datasets considered, which is a point in its favor.\n\nWeaknesses:\n\n- In a side-by-side comparison with the implementation of \"Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation\" (Chang 2022), this paper makes only one slight change in the gradient propagation. Specifically, gradients from the last step of the clustering loop are used to update the initial slots (straight-through estimator). This is a quite minor change, which may be seen as an ablation study of Chang 2022. Apart from the empirical experiments that show a performance improvement in some benchmarks, which I discuss below, I don't see any interesting theoretical insight related to this change. Sure, there is a lengthy introduction on bi-level optimization in section 3.2, but it feels artificial and disconnected from the practical change proposed in section 3.3. If this paper studies the connection between slot attention, k-means clustering, and bi-level optimization, what theoretical insights can be drawn from this previously unexplored connection? I think this is the main question that the paper should answer rather than focusing on the empirical results.\n\n- Since the main contribution of this work is the bi-level optimization of the initial query vectors, I expected more experiments studying the effect of the bi-level optimization on the final performance. What are the implication of propagating gradients to the initial query vectors? In particular, does the straight-through estimator introduce a bias in the gradients? How different are the learned queries from the post-clustering slots? Do the intermediate clustering steps become redundant due to the straight-through estimator, essentially turning this method in a VQ-VAE? How would other methods for learning initial queries behave, e.g. a running mean of the post-clustering slots or some actual bi-level update mechanism? I think these are interesting questions that the paper should address.\n\n- In Locatello 2020, the original slot attention is introduced as soft unsupervised clustering method that can be initialized with a variable number of random vectors. One of the most interesting experiments was the ability to train on images containing a certain number of objects and generalizing to a larger number of objects simply by increasing the number of sampled slots. Learning the slot initialization precludes this possibility because the number of learnable queries must be set as a hyperparameter, and also requires a certain a priori domain knowledge. Discussion of this drawback is lacking in the paper, instead, empirical results are presented on synthetic data where the number of object is known and on foreground/background segmentation where two slots suffice.\n\n- The choice of how many slots to use for each dataset is not made explicit in the main text (only in the appendix). Setting the number of slots to 2 for most foreground/background tasks is very convenient but does not convince about the generality of the method. For the birds dataset it is set to 3 for some unclear reason.\n\n- Two points regarding the ablation studies:\n  - First, ablation studies usually focus on hyperparameter choices of the proposed method, e.g. number of slots or how to optimize them. Instead the ablation studies presented in the paper are a comparison with previous methods. This comparison should arise from the main experiments, not from the ablation studies.\n  - Second, I am confused by some combinations of slot initialization and optimization procedure. Why were these two combinations chosen?\n    - I-QSA: in QSA the initial slots are initialized at random and then learned, but they receive no gradient due to the stop-gradient operation, which makes this combination equivalent to I-SA. Unless there are other differences, I would attribute the lower performance of I-QSA wrt I-SA to an unfortunate initialization.\n    - BO-SA: in SA the slots are sampled at random so propagating gradients to them (BO) should have no effect and should be equivalent to I-SA. The results are in fact very similar.\n\n- Section 5.4 the statement that BO-QSA has the potential to be used for learning concepts is overstated in my opinion. The only supporting evidence is that slots tend to specialize on colors for ShapeStacks in figure 3. However, this behavior has been observed in other slot-based methods too, and can be explained simply by noting that even a traditional pixel clustering algorithm would capture RGB colors. I would not draw any conclusion about the ability to learn concepts from these color-based observations. The other example given in figure 3 uses the birds dataset where the number of slots has been conveniently set to 3 for foreground/background segmentation. Since the birds dataset contains images of a single subject over a blurry textured background, it's rather obvious that the slots will specialize on the subject and the background. This is not a very convincing example of concept learning.\n\n- Related to the point above, the conclusion also contains an overly bold statement \"By further verifying our findings under zero-shot transfer learning settings, our model shows great potential as a principle approach for learning generalizable object-centric representations\". Though I acknowledge the transfer learning results, I would highlight that the experiment in question is foreground/background segmentation on rather simple datasets. I would not extrapolate these observations to truly multi-object real-world datasets like COCO, LVIS, or PASCAL. \n\nMinor points:\n\n- Why the connection with meta-learning? Bi-level optimization has many applications and meta-learning is just one of them. Nowhere else in the paper the connection with meta-learning is leveraged to justify or enhance the proposed method. I think the entire section 2.2 could be removed without hindering the message of the paper.\n\n- Appendix A.1, check the paragraph names", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ABjtzuKLVu", "reviewer": "Reviewer_tD3C", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes two tricks to optimize the training of slot attention. First, it initializes the query with learnable embedding instead of sampling from a learnable Gaussian distribution. Second, it applies bi-level optimization to the training. In practice, the slot binding process serves as the optimization of an inner optimization thus the gradient caused by this inner loop stops flowing backward. However, the learnable queries can still be updated with a straight-through estimator. The experiments on synthetic datasets (ShapeStacks and  ObjectsRoom) and real datasets (CUB200 Birds, Stanford Dogs, Stanford Cars, and Caltech Flowers) show that the proposed method achieves competitive performance. Moreover, the ablation study dissects the two tricks and proves both are indispensable. ", "summary_of_the_review": "In short, it is a good paper that takes simple tricks to boost the slot attention module. One possible improvement would be the theoretical explanation. Thus, I recommend accepting the paper.", "strength_and_weaknesses": "Strength:\n+ The idea is neat and well-presented. Especially, table 5 clearly demonstrates the effectiveness of the method.\n+ The two tricks complement each other, which strengthens the contribution of the paper. For instance, bi-level optimization makes the gradient backpropagate to slot initialization queries. \n+ The method considerably boosts the performance on almost all the benchmarks. \n\n\nWeakness:\n+ The theoretical analysis is not sufficient. Despite the superior experimental results, the theoretical explanation is somewhat missing.\n+ Learnable query is not novel. Previous work [1] has adopted this variant to stabilize the training.\n\n\n\n[1] Self-supervised Video Object Segmentation by Motion Grouping. ICCV 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "0KhUF7k_Xy", "reviewer": "Reviewer_9uz4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an extension of SlotAtt (NeurIPS 2020) by adopting learnable queries as slot initializations. The learning process is formulated as a bi-level optimization problem, where image reconstruction is the outer objective and soft clustering of image feature is the inner objective. In order to stabilize the training procedure, this paper detaches gradients to the recursive updates, only keeping gradients for last iteration and slot initialization queries.\n\nThe proposed method is evaluated for unsupervised object discovery task on synthetic datasets and unsupervised foreground extraction task on real datasets. It demonstrates improved performance on both tasks compared with state-of-art unsupervised object-centric models, especially the vanilla SlotAtt.\n\nAdditionally, this paper shows the potential to bind object concepts to its learned slots with experiments on zero-shot transfer learning.", "summary_of_the_review": "This paper is a simple but effective extension of Slot Attention. The structure and presentation are great. It demonstrates remarkable improvements on unsupervised object discovery and foreground extraction. However, the major concerns are: 1) deeper technical insights/analysis/discussion are missing, 2) more convincing experiments are needed.\n\nConsider the simple and effective design and the excellent performance, the reviwer is happy to increase the score if the concerns can be well addressed or explained. ", "strength_and_weaknesses": "Strength:\n\n1. The structure of the paper is very clear. The motivation and related research are well-explained. The technical details are concise.\n\n2. The proposed module is simple and effective, which could serve as a plug-and-play module for many models.\n\n3. It explicitly learns object concepts from datasets as slot initializations, which paves the way for object-centric learning in more challenging cases.\n\n\nWeakness:\n\n1. More insights on learned slot initializations are expected:\n\nFigure 3 has shown slot contents for a given input image after iterative updating. But what’s more special in this paper is the learned slot initializations, which are shared among all images in the dataset. It is expected that the learned slot initializations are the object concepts abstracted from the dataset. You may convert the learned slot initializations as images for visualization, or perform feature space analysis such as T-SNE to provide more insights.\n\n2. More discussions on the effectiveness of new design are needed:\n\nAlthough ablation experiments in Section 5.3 have demonstrated the effectiveness of the proposed module, it is still unclear why learnable queries as slot initializations can bring these large improvements. You may include more analysis and comparison with SlotAtt especially in the experiment part. For example, SlotAtt mentions that its typical solution is to distribute the background equally over all slots. However, from the visualization of this paper, it seems all background pixels are separately assigned into a single slot. A more detailed and analytical comparison with SlotAtt are needed to provide more insights on your contribution.\n\n3. More challenging experiments are expected:\n\nThe experiment parts demonstrate remarkable results on a set of synthetic and real datasets. But for all synthetic datasets including those in the appendix, the objects are simple-colored and mostly simple-shaped, where a color-based bias may already work very well. It is suggested to evaluate on more complex synthetic datasets such as ClevrTex [1]. \n\nFor the real datasets, all images only have a single foreground object, and the task becomes a simple binary classification. It’s more convincing to evaluate on multi-object real images, as also discussed in a very recent paper [2]. \n\n[1] ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation, NeurIPS 2021.\n[2] Promising or Elusive? Unsupervised Object Segmentation from Real-world Single Images, NeurIPS 2022.\n\n4. Zero-shot transfer learning experiments:\n\nSuccessful zero-shot transfer learning experiments suggest generalizable representations are learned. However, if a slot learned on dog dataset can be easily transferred to flower dataset. Does it imply the slots are not necessarily binding with object concept, at least not a specific type of object? You need to investigate what object concept is learned. Are they objects such as cat and dog, or a set of properties\nthat defines objects?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster