Neural Systematic Binder
Authors: Gautam Singh
Yeongbin Kim
Sungjin Ahn
Venue: ICLR 2023 poster (2023)
Keywords: object-centric learning
unsupervised representation learning
disentanglement
concept learning

Abstract: The key to high-level cognition is believed to be the ability to systematically manipulate and compose knowledge pieces. While token-like structured knowledge representations are naturally provided in text, it is elusive how to obtain them for unstructured modalities such as scene images. In this paper, we propose a neural mechanism called Neural Systematic Binder or SysBinder for constructing a novel structured representation called Block-Slot Representation. In Block-Slot Representation, object-centric representations known as slots are constructed by composing a set of independent factor representations called blocks, to facilitate systematic generalization. SysBinder obtains this structure in an unsupervised way by alternatingly applying two different binding principles: spatial binding for spatial modularity across the full scene and factor binding for factor modularity within an object. SysBinder is a simple, deterministic, and general-purpose layer that can be applied as a drop-in module in any arbitrary neural network and on any modality.  In experiments, we find that SysBinder provides significantly better factor disentanglement within the slots than the conventional object-centric methods, including, for the first time, in visually complex scene images such as CLEVR-Tex. Furthermore, we demonstrate factor-level systematicity in controlled scene generation by decoding unseen factor combinations.

Published as a conference paper at ICLR 2023
NEURAL SYSTEMATIC BINDER
Gautam Singh', Yeongbin Kim² & Sungjin Ahn"
Rutgers University KAIST
ABSTRACT
The key to high-level cognition is believed to be the ability to systematically manipulate and compose knowledge pieces. While token-like structured knowledge representations are naturally provided in text, it is elusive how to obtain them for unstructured modalities such as scene images. In this paper, we propose a neural mechanism called Neural Systematic Binder or SysBinder for constructing a novel structured representation called Block-Slot Representation. Representation, object-centric representations known as slots are constructed by composing a set of independent factor representations called blocks, to facilitate systematic generalization. SysBinder obtains this structure in an unsupervised way by alternatingly applying two different binding principles: spatial binding for spatial modularity across the full scene and factor binding for factor modularity within an object. SysBinder is a simple, deterministic, and ceneral-purposs layer that be applied as drop-in module in any arbitrary neural network and on any modality. In experiments, we find that SysBinder provides significantly better factor disentanglement within the slots than the conventional object-centric methods, including, for the first time, in visually complex scene images such as CLx Furthermore, we demonstrate factor-level systematicity in controlled scene generation by decoding unseen factor combinations. wttp:///ew.ooo//eeeee/e/eeeeee systematic binder
Block-Slot
INTRODUCTION
One of the most remarkable traits of human intelligence is the ability to generalize systematically. Humans are good at dealing with distr--distiitution situations because of the ability to understand them composition of pre-acquired knowledge pieces and knowing how to manipulate these pieces (Lake et al. [0177; Fodor & Pylyshyn, 1988). This also seems to be foundational for broad range of higher-level cognitive functions, such as planning, reasoning, analogy-making, and causal inference. However, realizing this ability in machines still remains a major challenge in modern machine learning Vee et al.) (0055) Bottou 2014 Schölkopf et al. (021).
This problem is considerably more difficult for unstructured modalities such as visual scenes or speech signals compared to modalities such as language. In language, we can consider embeddings of words or other forms of tokens as modular knowledge pieces. However, for unstructured modalities, we would first need to obtain such tokens, e.g., by grouping relevant low-level features, through a process called binding (Greff et al. 2020a). Yet it is quite elusive what should be the appropriate structure and granularity of these tokens to support systematic generalization and how to obtain them, particularly in the unsupervised setting where the model should learn this ability only by observing.
In visual scenes, binding has recently been pursued by object-centrre learning methods through the spatial binding approach (Locatello al. 2020; Singh et al.) 2022a). Spatial binding aims to divide scene spatially into smaller areas so that each area contains a meaningful entity like an object. The information in each area is then grouped and aggregated to produce representation of an object i.c., a slot, resulting in a set of slots per scene. These slots can be seen as independent modular knowledge pieces describing the full scene. However, the main limitation of the current methods based on spatial binding is that slot is an entangled vector and not a composition of independent
"Correspondence s sing.. gawttam gww.taamuut.grrr......e wuw.grrr.errr................... and a ajn. ahn ahn@gmai com.

---

Published as a conference paper at ICLR 2023
Block-Sio Repromettaiin
Representation
Slot
Block
Black
Block
Block
Nower
Noran
Mowey C
C
C:
VYN
mm
Neural Systematic Binder
Factor Binding
XN
Spatial Binding
Inpur Featurer
We propose novel binding mechanism, Neural Systematic Binder, that represents object as a slot constructed by concatenating multi-dimeenionnal factor representations called blocks. Without any supervision, each block learns to represent a specific factor of the object such color, shape, position. Right: Neural Systematic Binder works by combining two binding principles: spatial binding and factor binding. In spatial binding, the slots undergo a competition for each input feature followed by iterative refinement, similar Slot Attention. In factor binding, unlike Slot Attention, for each slot, the bottom-up information from the attended input features is split and routed M independent block refinement pathways. Importantly, each pathway provides representation bottleneck by performing dot-product attention on memory of learned prototypes.
Figure I: Overview. Left::
modular representations. As such, systematically novel object would map to an unfamiliar slot vector rather than modular combination of familiar factor tokens, such color, shape, and position.
In this paper, we propose the Neural Systematic Binder, or SysBinder for short. SysBinder generalizes the conventional binding process by combining spatial binding and factor binding. While spatial binding provides spatial modularity across the full scene, factor binding provides factor modularity within an object. More specifically, given input, SysBinder produces set of vector representations, called slots, via spatial binding. However, unlike typical object-centric learning, each of these slots is constructed by concatenating a set of independent factor representations called blocks obtained via factor binding. SysBinder uses iterative refinement to learn the slots by applying spatial binding step and factor-bil bintor- steps alternatingly during the refinement process. the spatial binding step, slots compete with each other to find an input attention area per slot. The information in each area is then grouped and used to refine the blocks. Crucially, to achieve factor binding, each block is refined in a modular fashion by applying an independent RNN and soft information bottleneck on each block. We train SysBinder in a fully unsupervised manner by reconstructing the input from the slots using the decoding framework of SLATE (Singh et al. 2022a b).
The contributions of this paper are as follows. First, SysBinder is the first deterministic binding mechanism produce disentangled factors within slot. This deterministic nature of SysBinder is remarkable since, conventionally, probabilistic modeling has been considered crucial for the emergence of factors within slots (Higgins al. 2017 Greff al., 2019). Second, in probabilistic frameworks, the representation of a factor is a single dimension of the slot, while in SysBinder, the multi-dimensionn
representation of factor to represent a factor. Third, similar to Slot Attention, SysBinder is a simple, deterministic, and ceneral-purpont layer that can be applied drop-in module any arbitrary neural network and on any modality. In experiments, we show that i) SysBinder achieves significantly better factor disentanglement within the slots than conventional object-centric methods, including those based on probabilistic frameworks. ti) Notably, for the first time in this line, we show factor emergence in visually complex scene images such as CLEVR-Tex (Karazija et al. 2021b). iii) Using the emergent factor blocks obtained from SysBinder, generation by decoding unseen block combinations. iv) Lastly, we provide an extensive analysis by evaluating the performance of several key model variants.
block. providing more flexible and richer way
demonstrate factor-level systematicity in controlled scene
2

---

Published as a conference paper at ICLR 2023
2 METHOD
2.1 BLOCK-SLOT REPRESENTATION
We propose and define Block-Slot Representation set of slots where each slot is constructed by concatenating a set of factor representations called blocks. For images, a slot would represent one object in the scene, similar to the conventional slots. However, unlike the conventional slots which either i) do not seek intra-slot factors at all (Locatello et al. 2020) or ii) seek to make each individual dimension a factor (Greff et al. 2019), we seek make each multi-dimensional block factor. Each block should represent at most one factor of the object, and, conversely, each factor should be fully represented by single block. Formally, by denoting the number of slots by N, the number of blocks per slot by M, and the dimension of block by d, we denote the collection of N slots by matrix E R XMdd Each slot Sn E Mdd a concatenation of M blocks. We denote the m-th block of slot n as m,,m Rd
2.2 SYSBINDER: NEURAL SYSTEMATIC BINDER
We propose Neural Systematic Binder or simply SysBinder as novel binding mechanism to encode a set of L imput vectors E into Block-Sle Slot Representation S. SysBinder is general purpose mechanism that, in principle, can be used for any input modality (e.g., speech signals) as long as it is provided as set of vectors. It can therefore be seen neural architecture. SysBinder is based on two binding principles: spatial binding and factor binding. In spatial binding, each slot S, performs competitive attention on the input features E find a spatial attention area. In factor binding, each of the M blocks is updated modularly using the information from the attended spatial area by applying an independent RNN and soft information bottleneck on each block. We first initialize the slots by randomly sampling them from a learned Gaussian distribution. We then refine the slots by alternatingly applying the spatial binding and factor binding steps multiple times. We now describe single refinement iteration in detail.
general-purpose
drop-in module that can be plugged into any arbitrary
2.2. SPATIAL BINDING
In the spatial binding, the slots S E N first perform competitive attention (Locatello et al. 2020) on the input features E IR Lx to attend to local area. For this, we apply linear projection q on the slots to obtain the queries, and projections k and v on the inputs to obtain the keys and the values, all having the same size Md. We then perform dot-product between the queries and the keys to get the attention matrix A E RNXI. In A, each entry A is the attention weight of slot n for attending over the input vector We normalize A by applying softmax across slots i.e., along the axis N. This implements form of competition among slots for attending to each input 7.
We now seek to group and aggregate the attended inputs and obtain the attention readout for each slot. For this, we normalize the attention matrix A E R X along the axis L, and multiply it with the input values v(E) E RR Lx Md This produces the attention readout in the form of a matrix U E R N where each row un E RB is the readout corresponding to slot n.
(Q()) -(E))
An,
A- softmax N
A
A=A.oBB (E),
Mdd
11=1 And
2.2.2 FACTOR BINDING
In factor binding, we seek to use the readout information obtained from spatial binding and update each block in a modular manner. This is achieved via the following two steps:
Modular Block Refinement. For this, we split each slot's readout vector u,, into M vector segments of equal sizes (denoting the m-th segment as un E Rd). We then apply a refinement RNN independently on each block sp,m in parallel by providing its corresponding readout segment u, as the input. The RNN is implemented using a GRU and a residual ML? In our design, we choose maintain separate parameters on for each as this design can potentially scale better for complex large-scale problems where the number of blocks can be large. However, depending on the context, it is also a sensible design to share the parameters Om & for all m.
S mmm CRU (Sn,m, Up
Sum += MLP p... (LN (sn,))
3

---

Published as a conference paper at ICLR 2023
Block Bottleneck. Next, apply soft information bottleneck on each block by making each block retrieve a representation from a concept memory. A concept memory Cm E R xxd is a set of K learnable vectors or K latent prototype vectors associated with the particular factor m. Each block 8n.m performs dot-product attention on its corresponding concept memory Cm,, and the retrieved vectior is used as the block representation:
Bn,m
C vd
]]
Sp,m
so softmax
Cm-
The spatial binding and the factor binding together form one refinement iteration. The refinment iterations are repeated several times and the slots obtained from the last iteration are taken to be the final Block-Slot Representation for providing downstream. We provide a pseudo-code of our method in Algorithm]
2.3 UNSUPERVISED OBJECT-CENTREE LEARNING WITH SYSBINDER
In this section, we propose unsupervised object-centric learning method for images using the proposed binding mechanism, SysBinder. For this, we adopt an auto-encoding approach in which apply SysBinder to encode the image and then apply autoregressive transformer decoder to reconstruct the image.
Encoding,
Given image E Hx we first obtain of image features using backbone encoder network, e.g., a CNN in our experiments. In this CNN, positional encodings are added in the second-last layer and the final output feature map is flattened form set of input vectors We then apply SysBinder on E to obtain slots S.
represented in a matrix form E E LxD
Block Coupling, One crucial benefit realized by Block-Slot Representation is that the relationships and interactions among the representations can be modeled explicitly at the level of blocks rather than the level of slots. To obtain this benefit, we would like to provide all block vectors together as set of tokens {s } to the downstream transformer decoder. However, because a set is order-less, is difficult for the transformer (i) to know which concept each block is representing and (ii) to identify to which slot each block is belonging. Following the traditional approach for dealing with the order issue in Transformer, one possible approach to this problem to provide two positional embeddings to each block indexed by (n, m.) one for slot membership and the other for the block index m. However, this would mean that the decoder should take the burden of learning invariance to the slot order which is not built-in into the design. Furthermore, it could be difficult for the decoder to generalize well when many more slots than the number seen during training are given at test time.
To avoid these challenges, we propose the following method. To each block, add positional encoding phlock that depends only on m but not on n. We then let the blocks belonging to the same slot interact via l-layer transformer, termed block coupler, as follows:
8p,m Sm + Pm
block
Sp,l. Sn,M BlockCoupler(S...... S..,.,,,......,,)
After this, we provide all the returned vectors to the transformer decoder as a single set S (S..m) }. With this approach, the decoder is informed about the role of each block via the positional encoding pllcck Because the blocks in the same slot have interacted with each other, the information about which blocks belong to the same slot is also implicitly available to the decoder. Furthermore, the decoder treats the slots as order-invariant and can also deal with more slots than shown in training.
Autoregressive Decoding, Conditioned on S obtained via block coupling, a transformer decoder learns to reconstruct the image. Similarly to the decoder of SLATE (Singh et al. 2022a) and STEVE (Singh et al.) 2022b), the transformer does not directly reconstruct the pixels of the image. Instead, we reconstruct a discrete code representation Z1,..., Zz of the given image x obtained from a discrete VAE (dVAE). In autoregressive decoding, the transformer predicts each code 2, at position I by looking at all the previous codes Zi,...; , 21-1 and the input blocks S. For this, first retrieve learned embedding for each discrete code Zi...., 2;-1. To these, we add positional encodings and provide the resulting embeddings ei...., el-1 the transformer predict the next token:
Gy Dictionary, (2:) + P
token
O Transformer-Deooderffee---- (1-1;;;
where o the predicted logits for the token at position l. The training is done by learning to predict all tokens in parallel using causal masking via the learning objective L = Σ CrossEntropy o,). The dVAE may be pre-trained or trained jointly with the model. For more details, see Appendix D.4
4

---

Published as a conference paper at ICLR 2023
3 EVALUATING BLOCK-SLAN REPRESENTATION
In this section, we introduce the method to quantitatively evaluate Block-Slot Representations based on the DCI framework (Eastwood & Williams, 2018). The DCI framework is multi-faceted and evaluates multiple aspects of the quality of a representation vector in terms of Disentanglement (D), Completeness (C), and Informativeness (I). Although the original DCI framework is designed for single vector representation, Nanbo et al. (2020) proposed method to make it applicable to set of slots by first matching the slots with the true objects using IoU before applying the DCI procedure.
Specifically, let S = (8 (S1..... so) be set of slots, with slot S, E R Also, for each object, let there be K ground-truth factors of object variation such as color, shape, and position. We train K probe functions gp ..... 9K gxx one for each factor, to predict the factor label y.¹ from the slot vector So- trees (Locatello et al. 2019). Using
The probes are classifiers implemented using gradient-boosse the trained probes, we can obtain feature importance matrix R (Rx.;) E Ig x Each entry Ri.j denotes the importance of feature dimension j in predicting the factor and is obtained from the corresponding probe gk ohtibe Using the importance matrix, the disentanglement-scor D, for specific feature dimension j and the completeness-scor Cj for specific factor & are computed as follows:
Dj = 1 - Hx (R..),
C2 = 1 Ha(
where (+) denotes entropy with log-base K and Ha(-) denotes entropy with log-base d. From these, the final disentanglement-score D and completeness-scon sccore C are computed as the weighted average of Dj over all the dimensions = I,...,d d and similarly for C2 by averaging over k 1. 1...., K. The informativeness-scor score is the average of the prediction accuracies of all the probes.
For evaluating the Block-Slot Representation, above to obtain the importance matrix R = (Re.j) E R xMd, where M is the number of blocks, d is the block size, and Md is the size of the slot. To evaluate the representation in the setting where each block is an independent factor, we obtain an importance score per block by summing the importance scores across the dimensions belonging R block = (IRbloch) E IR XMM Using these block importance values, we compute the disentanglement- scores and the completeness-scont in the same way as described above.
first apply the same probing approach as described
the same block, giving us block importance matrix
4 RELATED WORK
Our work belongs in the line of unsupervised object-centric learning methods (Greff et al.
2016
Buugges et al. 2019, Greff al. 2019 Locatello et al. 2020 Greff et al., 2017 EEnellcee et al. 2019
Engelcke et al. 2202 Anciukeviiin et al. von Kügelgen et al. 2220 Greff et al. 2020b
Singh
et 2022al Chang et al. 2022). In this line, intra-slot disentanglement has been explored inside
the VAE framework (Greff et al. 2019 Burgess et al. 2019 Nanbo et al.) 2020 Zoran et al. 2021).
However, unlike these, our work achieves this goal without taking the complexity of probabilistic modeling. Another parallel line of work pursues disentanglement of object properties by adopting
spatial transformer decoder - a special decoder designed to accept explicit values of object size and
position (Jaderberg al. 2015 Eslami et al.) 2016 Crawford & Pineau 2019b Lin et al.) 2020b]
Jiang & Ahn, 2020) Deng et al. 2020 Chen et al., 2021). However, these cannot scale to other factors of variation such as color or material. Praabhudesai al. (202 seek object and factto-level
disentanglement but require the use of 3D viewpoint, occupancy labels, and an auxiliary consistency loss. Learning neural networks composed of independent competing modules has been explored by
Goyal et al. (2021b] 2020); Lamb et al. (2021); Goyal et al. (2021a). Our prototype learning is related
to van den Oord et al. (2017): Liu et al. 2021): Caron et al. (020)) and our concept memory attention is related to (Ramsauer et al. 2021). However, these works do not focus on object-centric learning or intra-slot disentanglement.
Disentangling single-vector scene representations has been pursued in Chen et al. (2016); Higgins ct al. (2017); Kim & Mnih (2018); Kumar et al. (0017); Chen et al. (2018); Locatello et al. (2019); 9):;
Montero et al. (2021). Parallel to these, Hu et al. (2022 Zhou et al. (2020) pursue block-level disentanglement but on 3D mesh data and in single object scenes. Stammer et al. (202 use prototypes to learn block-wise disentangled codes. However, this is not unsupervised and does not deal with multiple objects, unlike ours. Several metrics have also been proposed alongside the above works to measure disentanglement in single vector representations (Higgins al. 2017, Kim & Mnih
5

---

Published as a conference paper at ICLR 2023
Ridgeway & Mozer 2018 Kumar et al. 2018 Andreas 2019). 9). For slot representations, Several metrics have been proposed but these do not apply to bloek-level disentanglement within slots (Racah & Chandar 2020; Xie et al. 2022; Nanbo et al. 2020). The metric of (DangNhin (2022) applies to block-level disentanglement, However, it involves simultaneously optimizing slot permutation and factor prediction which could be unstable. Also, it requires building a much larger fcature-importance matrix which can be extremely slow.
2018, Chen et al.
2018 Eastwood & Williams 2018
5 EXPERIMENTS
Datasets. We evaluate our model on three datasets: CLEVR-Easy, CLEVR-Hard, and CLEVR-Tex, These are variants of the original CLEVR dataset (Johnson et al. (017) having large object sizes to make object properties such as shape or texture clearly visible in the image. These datasets have different difficulty levels: i) In CLEVR-Fasy, there are three ground-truth factors of variation, i.e., color, shape, and position while the material and the size are fixed. As in the original CLEVR, the color takes values from small palette of 8 colors. ii) In CLEVR-Hard, all 5 properties i.c. color, shape, position, material, and size are the factors of variation. Furthermore, the color takes values from much larger palette containing 137 colors. iii CLEVR-Tex is derived from the original CLEVR- Tex dataset (Karazija et al., 2021a). Here, the objects and the backgrounds can take textured materials from total of 57 possible materials and is much more complex than the previous two datasets. In addition to the material, the object shapes and positions are also varied. Our model only sees the raw images input and no other supervision is provided. For evaluation, we use ground-truth object masks and factor labels. See Fig. 5 for sample images of these datasets. We release our code and the datasets here
Baselines. We compare our model with three state-of-the-art unsupervised object-centric representa- tion models: IODINE (Greff et al. 2019). Slot Attention (Locatello al. 2020), and SLATE (Singh et al. 2022a). IODINE represents the class of VAE-based object-centric models while Slot Attention and SLATE are two deterministic object-centric models. The number of factor representations per slot for the baseline models is their slot size since every dimension acts factor representation. Following their original papers, we set this to 64. our model, the number of factor representations per slot is the number of blocks M. We use 8, 16, and 8 blocks for CLEVR- Easy, CLEVR- -ardd, and CLEVR-Tex. respectively. Importantly, having such few factor representations, c.g., 8 or 16, is desirable characteristic of our model because it leads to better interpretability within slot. As such, we also test how IODINE would perform with such few factor representations. We test this by training IODINE by setting its slot size equal to the number of blocks M of our model (termed as 1ODINE-M). Because using such small slot sizes deviates from IODINE's original hyperparameters, also analyze a wide range of other slot sizes for IODINE in Fig. 6 and find that it does not lead to much improvement in IODINE's performance.
5.1 OBJECT AND FACTOR DISENTANGLEEMENT
To compare how well the models decompose the image into slots and then the slots into the factors of object variation, we use four metrics. To evaluate object segmentation, we adopt the Adjusted Rand Index applied to the foreground objects (FG-ARI), as done in prior works (Greff et al. 2019
Locatello et al. 2020). FG-ARI has also been shown to correlate with downstream task performance 2022 To evaluate intra-slot disentanglement, we use DCI introduced in Section 3
DDitadi et al.
Quantitative Results. We show a quantitative evaluation of object-level and factor-level disentan- glement in Fig. We note that our model significantly outperforms the baselines in all 3 datasets, nearly doubling the disentanglement and completeness--cores in CLEVR-EaSy and CLEVV- CLEVR-Tes the other models except for SLATE completely fail in handling the visual complexity of the textured scenes while our model remains robust. To the best of our knowledge, it is for the first time in this line of research that factor-level disentanglement has been achieved on this level of textural complexity. The informativeness-sctt in factor prediction and object segmentation to the other methods. In Fig. 3 we visualize the feature importance matrices used for computing the DCI scores in CLEVR- We note that our importance matrix more sparse than other methods. We also note that deterministic baselines like SLATE and
In
and FG-ARI also suggest that our model is superior
https://sitte http:///siee...gglleee/ wttpoo/le..oomm/e//eeeeeee/// http://wwwwenuurrrasssstttteeeee systemat binérrer

---

Published as a conference paper at ICLR 2023
CLEVE-Eaoy
CLEVR-Hard
CLEvR-Tex
Noddl Noweee
cone
0.8
0.8
0.8
06
00
0.6
FG-ARI
FC-AR
FR
Figure Slot Learning and Intra-Slot Disentanglement. We compare our model with the baselines in terms of Disentanglement (D), Completeness (C), Informativeness (I,, and FG-ARI.
Shape
Color
Neural Systematic Binder with blocks in a sioe
IODINE with dimensions siot
Postice
IODINE with 54 dimensions slot
Shape
Color
Slot Attention with dimensions in slot
Shapo
Color
Podlice
SLATE with 64 dimensions in slot
Figure 3: Feature Importances in CLEVR-Easy, We visualize the intra-slot feature importance matrices of the evaluated models. Each row corresponds to ground-truth intra-object factor and each column corresponds to a factor representation. our model, a factor representation is a block while in the baselines, factor representation is single dimension of the slot.
Slot Attention have significantly more active dimensions than the IODINE baseline due to the effect of the VAE prior in IODINE. This observation also correlates with lower completeness-score of Slot Attention and SLATE relative to IODINE. Such effect the VAE prior was also noted in Dang- Nhu (2022). Surprisingly, unlike the conventional methods, in SysBinder, such disentanglement emerges with simple deterministic model and without applying a VAE framework.
Qualitative Results. In Fig. and 10 we visualize the learned representation space of each block. For this, we feed a large batch of images to SysBinder, collect the m-th blocks, and then apply k-means clustering on the collected block representations. By visualizing the objects belonging to various clusters in each m-th block, we observe the emergence of semantically meaningful and abstract factor concepts. For CLEVR-Easy an example, in m 3 we can see shape concepts like sphere or cylinder; in m color concepts like cyan and blue; and in m 5 the position concepts. remarkable that this emergence also occurs the visually complex CLEVR- the first time in this line of research. It remarkable to see all the spheres of CLEVR-Tes dataset are clustered together despite the fact that their low-level surface appearances and textures are very different from each other.
dataset-foo for
5.2 FACTOR-LEVEL SAALLEEELLSSSEEEE SYSTEMATICITY IN SCENE COMPOSITION
Since SysBinder provides knowledge blocks at the level of intra-object factors, we test whether these blocks can be composed to generate novel scenes with systematically novel factor combinations. We do this by first representing a given image as Block Representation using SysBinder. We then choose two slots, select the blocks that capture a specific object factor e.g., color, and swap the selected blocks across the two slots. The manipulated slots are then provided to the decoder to generate the scene. Fig. we show that our model can effectively generate novel images this way. This shows that our blocks are indeed modular, allowing us manipulate specific factor without affecting the other properties. We also found the blocks to be interpretable and compact which enabled us easily identify which blocks swap in order swap specific factor. Furthermore. our model demonstrates this ability not only in simple scenes but also complex textured scenes of CLEVR-TE
7

---

Published as a conference paper at ICLR 2023
CLEVR- Tex
represents shapes
Block represents materials
Block 4 represents positions
CLEVR- ELEVR-Easy Easy
Block represents shapes
Block represents colors
Block represents positions
Figure 4: Visualization of object clusters obtained by applying k-means on specific blocks in CLEVR-Tex and CLEVR-Easy. Each block learns specialize to specific object factor e.g.. shape or material, abstracting away the remaining properties. See Fig. TO for clusters of all datasets.
Inpur
Shope Swap
Color Swop
Poskit bon Swap
Irput Image
Shape Seap
Color Swap
Material Swop. Porkioo Swap
Sioz Swup
CLEVR Easy
Irpat
Shape Swap Material Swap Pastian Swap
CLEVR-Tex
CLEVR-HAL Hard
Figure Visualization of Factor-level Scene Manipulation. manipulate a given scene by swapping specific factor of two objects in the scene. For given scene, we choose two objects and swap their shape, color, position, material, and size. White arrows shown on the input images point to two objects in each scene whose properties swapped. See also, Figures 11 12
and 13
5.3 ANALYSIS
In this section, analyze several model variants and hyperparameters. We perform these evaluations on the CLEVR-Fasy dataset and report the results in Fig. 6
Role of Architectural Components of SysBinder. In Fig. 6 (a), analyze the role of various architectural components of our model. First, we compare the effect of sharing the parameters of the RNN used during block update (i.e. the GRU and the MLP) relative having separate RNN parameters for each block. While we do not find a significant difference in performance, nevertheless, having separate RNN per block may serve as a more flexible and modular design for more complex large-scale problems. Second, we test the performance by removing the RNN used during the block refinement. We find that this has significant adverse effect on segmentation quality as well as the DCI performance, showing that the RNN is crucial component. Third, we test the role of having the concept memory bottleneck by removing it. We find that while this does not affect the segmentation and informativeness of the slots, however, it significantly reduces the disentanglement and the completeness-ee scooess,, showing the importance block bottleneck in enabling specialization of blocks to various intra-object factors. Fourth, test the role of the block coupling used the interface between SysBinder and the transformer decoder. We test this by training a variant of our model in which provide all the blocks as single set to the decoder. We find that performance
8

---

Published as a conference paper at ICLR 2023
(a) Analysis of Architectural Components
Effect of Number Prototypes
Effect Number of Blocks
Shan- No of
BB
1 boar
Mitssu Cxcoder
0.8
0.8--1
0.4
0.4-
0.4
0.2
0.2
FG.ARI
FG-ARI
FG-ARI
(d) Effect of Block Size
(e) Effect of on IODINE-M
(f) Effect of Slot Size on IODINE
0.8
0.8
0.6
0.6
0.4
0.4
0.2
FC-ARI
FG-ARI
FG-ARI
Figure 6: Analysis. We evaluate the role of various design choices and hyperparameters for our model and the baseline IODINE. In all plots, report the DCI and the FG-ARI.
drops severely with some seeds failing completely. This is because without knowing the role of each block and their membership information, the decoder is unable reconstruct well.
Role of Transformer Decoder. In Fig. (a), we also analyze the importance of the transformer decoder in our model and whether using the mixture-based decoder, a common choice in several object-centric learning methods, can also provide a similar performance or not. To implement this variant, we adopt the decoder design of Locatello et al. (2020). We decode each object component by concatenating the blocks of slot and providing it to spatial broadcast decoder Watters et al. (2019). Interestingly, we find that this leads to a large drop in the disentanglement and the completeness scores, showing the important role of our expressive decoder.
Effect of Number of Prototypes, Number of Blocks, and Block Size. In Fig. (b), (c), and (d), we study the effect of the number of prototypes K, the number of blocks M, and block size d on the DCI performance. First, with the increasing number of prototypes, we note improvement, especially in the completeness-score. This suggests that more prototypes increase the expressiveness of each block and thus fewer blocks are needed to represent single factor. Second, for the number of blocks M = 4, see low disentanglement and completeness scores. Because this dataset has factors of object variation, this suggests that 4 blocks may be too few for them to train well. However, going to M and M 16, the performance improves and then saturates. Third, with increasing block sizes, we note a steady increase in performance.
Analysis of IODINE. In Fig. (e) and (f), we analyze IODINE. We test the effect of adopting the 3-VAE objective (Higgins et al. 2017) and the effect of varying the slot size. While larger 3 leads to slight increase the disentanglement score and the completeness-see score, it still remains significantly worse than ours. Large / also leads to slight drop in the informativeness--onn score due to the greater regularizing effect of the KL term. Increasing the slot size d which is equivalent to increasing the number of factor representations in IODINE slightly improves performance but still remains worse than our method. We also note that while we test IODINE with a slot size as large as 256, having few factor representations, e.g.. 8 or 16, provides better interpretability within slot.
6 CONCLUSION
We propose a new structured representation, Block-Slot Representation, and its new binding mecha- nism, Neural Systematic Binder (SysBinder). Unlike traditional Slot Attention, SysBinder learns to discover the factors of variation within a slot. Also, SysBinder provides the factor representations in a vector form, called block, rather than a single-dimensional variable. Using SysBinder, we also introduce
unsupervised object-centric learning method for images. In achieving this, also note the importance autoregressive transformer baseed decoding as well as our novel block-coupling approach. Our results show various benefits in disentanglement produced by the proposed model.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We release all the resources used in this work, including the code and the datasets, at the project link:
https://sites. httpa//ggog..o.oo/////////aaaaaa
ETHICS STATEMENT
Future extensions and applications arising from our work should be mindful of the environmental impact of training large-scale models. They should actively avoid its potential misuse in surveillance or by generating synthetic images with malicious intent. However, it is unlikely that the model in its current form would lead to such an impact in the near future. Our model also has the potential for making positive impact in areas such scene understanding, roboties, and autonomous navigation.
ACKNOWLEDGEMENT
This work is supported by Brain Pool Plus (BP+) Program (No. 2021H1D342A03 and Young Researcher Program (No. (022ICIC1109443) through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT. The authors would like to thank the members Agent Machine Learning Lab for their helpful comments.
REFERENCES
Titas Anciukevicius, Christoph H Lampert, and Paul Henderson. Object-centric image generation with factored depths, locations, and appearances. arXiv preprint arXiv:2004.00642, 2020.
Jacob Andreas. Measuring compositionality in representation learning. ArXiv, abs/1902.00788 2019.
Federico Baldassarre and Hossein Azizpour: Towards self-supervised learning of global and object- centric representations. ArXiv, 0tt/2203.059977 2022.
Beril Besbinar and Pascal Frossard. Self-supervision by prediction for object discovery in videos. arXivv preprint arXiv:2103.05660
2021.
Léon Bottou. From machine learning to machine reasoning. Machine learning, 94(2): (022:133-22) 13--149, 2014.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.1 11.001111100 11390, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. ArXiv, abs/2006.09883 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV 202/ International Conference on Computer Vision, 2021.
Michael Chang, Thomas L Griffiths, and Sergey Levine. Object representations as fixed points: Train- ing iterative refinement algorithms with implicit differentiation. arXiv preprint arXiv:2207.00787, 2022.
Chang Chen, Fei Deng, and Sungjin Ahn. Roots: Object-centric representation and rendering of 3d scenes. Journal of Machine Learning Research, 22(259):1 2021.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Newral Information Processing Systems, pp. 2610-2620 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180. 2016.
10

---

Published as a conference paper at ICLR 2023
Eric Crawford and Joelle Pineau. Exploiting spatial invariance for scalable unsupervised object tracking. arXiv preprint arXiv:1911.0903: 2019a.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. Proceedings of AAAI, 2019b.
Eric Crawford and Joelle Pineau. Learning 3d object-oriented world models from unlabeled videos. In Workshop Object-Oriented Learning ICML, 2020.
Antonia Creswell, Kyriacos Nikiforou, Oriol Vinyals, Andre Saraiva, Rishabh Kabra, Loic Matthey, Chris Burgess, Malcolm Reynolds, Richard Tanburn, Marta Garnelo, et al. Alignnet: Unsupervised entity alignment. arXiv preprint urXiv:2007.08973 2020.
Antonia Creswell, Rishabh Kabra, Chris Burgess, and Murray Shanahan. Unsupervised object-based transition models for 3d partially observable environments. arXiv preprint arXiv:2103.04692 2021.
Raphaël Dang-Nhu. Evaluating disentanglement of structured representations. In ICLR, 2022.
Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn. Generative scene graph networks. In International Conference on Learning Representations, 2020.
Andrea Dittadi, Samuele Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Locatello. Generalization and robustness implications in object-centric learning. In ICML, 2022.
Yilun Du, Kevin Smith, Tomer Ulman, Joshua Tenenbaum, and Jiajun Wu. Unsupervised discovery of 3d physical objects from video. arXiv preprint arXiv:2007.12338. 2020.
Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems, 34, 2021a. a.
Yilun Du, Kevin A. Smith, Tomer Ulman, Joshua B. Tenenbaum, and Jiajun Wu. Unsupervised discovery of 3d physical objects from video. ArXiv, abs/2007.12345 2021b.
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In ICLR, 2018.
Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Curtis Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. ArXiv, abs/2206.07764, 2022.
Martin Engeleke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations, 2019.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. arXiv preprint arXiv:2104.0990 2021.
SM Ali Eslami. Nicolas Heess, Theophane Weber, Yuval Tassa. David Szepesvari, and Geoffrey E Hinton, Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.
Jerry Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: critical analysis. Cognition, 28 1-11)-333711, 1988.
Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Sergey Levine, Charles Blundell, Yoshua Bengio, and Michael C. Mozer. Object files and schemata: Factorizing declarative and procedural knowledge in dynamical systems. ArXiv, abs/2006. 2ttp22006.12255... 2020.
Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Manfred Otto Heess, Michael C. Mozer, and Yoshua Bengio. Neural production systems. ArXiv, abs/2103.01937, 2021a.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. ArXiv, abs/1909. 10893, 2021b.
11

---

Published as a conference paper at ICLR 2023
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber.
Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems, pp. 4484 4492, 2016.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems, pp. 6691-6701, 2017.
Klaus Greff, Raphaël Lopez. Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. arXiv preprint arXiv:/903.0044 004500 2019.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.052206 2020a.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint 2rXiv:20/2.05208 2020b.
Zhen He, Jian Li. Daxue Liu, Hangen He, and David Barber. Tracking by animation: Unsupervised learning of multi-object attentive trackers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1318-1327, 2019.
Olivier H' d'effff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zis- serman, João Carreira, and Relja Arandjelovi'c. Object discovery and representation networks. ArXiv, abs/2203.087777 2022.
Paul Henderson and Christoph H. Lampert. Unsupervised object-centric video generation and decomposition in 3d. ArXiv, abs/2200.00670) 2020.
Irina Higgins, Loie Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Xue Hu, Xinghui Li, Benjamin Busam, Yiren Zhou, Ale Leonardis, and Shanxin Yuan. Disentangling 3d attributes from single 2d image: Human pose, shape and garment. ArXiv, 1bs/2208.03167, 2022.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In Advances in neural information processing systems, pp. 2017-2025, 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatiion with gumbel-softmax. arXiv preprint 161iv:161 1.0/ 144, 2016.
Jindong Jiang and Sungjin Ahn. arXiv:2010.12152, 2020.
Generative neurosymbolic machines. arXiv preprint
Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, and Sungjin Ahn. SCALOR: generative world models with scalable object representations. In International Conference Learning Representations, 2020.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-2910, 2017.
Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matthew Botvinick. Alexander Lerchner, and Christopher P Burgess. Simone: View-invan niww-invrriant, aemsorally-abstracteed object representations via unsupervised video decomposition. arXiv preprint 2xX.v:2/060038499 2021.
Laurynas Karazija, Iro Laina, and C. Rupprecht. Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation. ArXiv, abs/21 1.11665,, 2021a.
Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevitex: texture-rich benchmark for unsupervised multi-object segmentation. arXiv preprint arXiv:2111. 10265, 2021b.
12

---

Published as a conference paper at ICLR 2023
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv: arXiv:1802.05983 2018.
Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. arXiv preprint S.ti:22111.12553 2021.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In Advances in Neural Information Processing Systems, pp. 8606-8616, 2018.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disen- tangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2017.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. ArXiv, abs/171 0ttps/711..0080 1.00488 2018.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Alex Lamb, Di He, Anirudh Goyal, Guolin Ke, Chien-Feng Liao, Mirco Ravanelli, and Yoshua Bengio. Transformers with competitive ensembles independent mechanisms. In arXiv e-prints, 2021.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Jindong Jiang, and Sungjin Ahn. Improving generative imagination in object-centric world models. In International Conference on Machine Learning, pp. 41 14244,, 2020a.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference Learning Representations, 2020b.
Dianbo Liu, Alex Lamb, Kenji Kawaguchi, Anirudh Goyal, Chen Sun, Michael Curtis Mozer, and Yoshua Bengio. Discrete-valued neural communication. In NeurIPS, 2021.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of dis- entangled representations. In International Conference on Machine Learning, pp. 4124- 2019.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention, 2020.
Sindy Löwe, Klaus Greff, Rico Jonschkowski, Alexey Dosovitskiy, and Thomas Kipf. Learning object-centric video models by contrasting sets. arXiv preprint arXiv:2011.10287, 2020.
Sindy Lowe, Phillip Lippe, Maja R. Rudolph, and Max Welling. Complex-valued autoencoders for object discovery. ArXiv, abs/2204.02075. 2022.
Mary B MoVee, Kailonnie Dunsmore, and James R Gavelek. Schema theory revisited. Review of educational research, 75(4):531-566 2005.
Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bow- ers. The role of disentanglement in generalisation. In International Conference on Learning Representations, 2021. URL http::tttt///t |tttttts:: //tttp:ss//ppeeevieew nnt wwwpe//oooriidooooooeeeeee wttp://wwww.e.oooooooooooa//1110
Li Nanbo, Cian Eastwood, and Robert B Fisher. Learning object-centric representations of multi- object scenes from multiple views. In Advances in Neural Information Processing Systems, 2020.
Mihir Prabhudesai, Shamit Lal, Darshan Patil, Hsiao- Yu Fish Tung, Adam W. Harley, and Katerina Fragkiadaki. Disentangling 3d prototypical networks for fhw-shot concept learning. ArXiv, abs/201 1030011.333 1.03367, 2021.
13

---

Published as a conference paper at ICLR 2023
Evan Racah and A. P. Sarath Chandar. Slot contrastive networks: A contrastive approach for representing objects. ArXiv, abs/2007.09294, 2020.
Hubert Ramsauer, Bernhard Schafl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Milena Pavlovi's c, Geir Kjetil Sandve, Victor Greiff, David P. Kreil, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. Hopfield networks is all you need. ArXiv, abs/2008.02217, 2021.
Karl Ridgeway and Michael C. Mozer. Learning deep disentangled embeddings with the f-statistic loss. ArXiv, abs/ 180205332, 2018.
Mehdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučié, Leonidas J. Guibas, Klaus Greff, and Thomas Kipf. Object Scene Representation Transformer. NeurIPS, 2022.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings the IEEE, 10965x612-634, 2021.
Gautam Singh, Skand Peri, Junghyun Kim, Hyunseok Kim, and Sungjin Ahn. Structured world belief for reinforcement learning in pomdp. In International Conference on Machine Learning, pp. 9744-9755, PMLR, 2021.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. In International Conference on Learning Representations, 2022a. URL
1d=h0oYVOWe30:
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. ArXiv, aaw.a200..1000000 2022b.
Wolfgang Stammer, Marius Memmel, Patrick Schramosski, and Kristian Kersting. Interactive disentanglement: Learning concepts by interacting with their prototype representations. ArXiv, abs/2112.00299 2021.
Aleksandar Stanić and Jürgen Schmidhuber. R-sqair: relational sequential attend, infer, repeat. arXiv preprint arXiv:1910.052.11 2019.
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.0/ /48, 2021.
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems, volume 30, pp. 6306-6315, 2017.
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery objects and their interactions. arXiv preprint 1033:3380,.,0333,, 2018.
Rishi Veerapaneni, John Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua B Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. arXiv preprint 1910v: 12827, 2019.
Julius von Kügelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, and Bernhard Schölkopf. Towards causal generative scene models via competition of experts. arXiv preprint arXiv:2004.12906, 2020.
Yangtao Wang. Xiaoke Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufreydaz. Self-supervised transformers for unsupervised object discovery using normalized cut. ArXiv, abs/2202.11539,, 2022.
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P Burgess, and Alexander Lerchner. Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven exploration, arXiv preprint 1r0iv 1905.0925, 2019.
14

---

Published as a conference paper at ICLR 2023
Nicholas Watters, Loic Matthey, Christopher P. Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint arXiv:1901.07017, 2019.
Marissa Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger, and Alexander S Ecker. Unmasking the inductive biases of unsupervised object representations for video sequences. arXiv preprint arXiv:2006. 07034, 2. 2020.
Xin Wen, Bingchen Zhao, Anlin Zheng, Zhang, and Xiaojuan Qi. Sulf-supe visual represen- tation learning with semantic grouping. ArXiv, abs/2205. 2022.
Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Generative video transformer: Can objects be the words? In International Conference on Machine Learning, pp. 11307-11318, PMLR, 2021.
Sirui Xie, Ari S. Morcos, Song-Chun Chun Zhu, and Ramakrishna Vedantam. Coat: Measuring object
compositionality in emergent representations. In ICML, 2022.
Peiyu Yu, Sirui Xie, Xiaojian Ma, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Unsupervised foreground extraction via deep region competition. In NeurlPS, 2021.
Keyang Zhou, Bharat Lal Bhatnagar, and Gerard Pons-Moll. Unsupervised shape and pose disentan- glement for 3d meshes. In ECCV, 2020.
Daniel Zoran, Rishabh Kabra, Alexander Lerehner, and Danilo Rezende. Parts: Unsupervised seg- mentation with slots, attention and independence maximization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10439-10447, 2021.
15

---

Published as a conference paper at ICLR 2023
A PSEUDO-CODE
Algorithm Neural Systematic Binder. The algorithm receives the set of input features E E TR LxD the number of slots N; the number of blocks M; and the block size d. The model parameters include: the linear projections k, 4, v with output dimension Md; GRU and MLP networks for each m Le. GRU,, ... ORU and MLP dj ... MMP the learned concept memories Cl...., CM R Rxxd and a Gaussian distribution's 's mean and diagonal covariance μu, a E RMdd
01: Tensor (V, Md)
02: (P, σ))
03: E LayerNorm (E)
04: for 1...7 sequentially:
05:
LayerNorm(S)
06:
A Softmax (8) k()),, axis='slote"
07:
A A Sum 'inputs', keepdim-True) U x(E)
08:
09:
for n 1...N and m m=
parallel:
10:
Sn,m CRUsm (state=s,.m) update=u
11:
Sp,m += MLP (LayerNozm(sh..))
12:
Sp,m Softmax ( 8n,m C axis= 'prototypss" C.n.
13:
return
B ADDITIONAL RELATED WORK
Unsupervised Object-Centric Learning. Object-centric learning has also been explored extensively
in dynamic scenes (Kosioeek et al., 2018; Stanić & Schmidhuber 2019:
Jiang et al. 2020
Crawford
Pineau
2019a Lin al. [2020a: Wu et | 2021 (Singh et al.
2021 He et al. 2019: Greft et al.
2017; Van Steenkiste et al.
2018:
Veerapaneni et al. 2019 Watters et al. 20199) Weis et al. 2020
Du al. [0200] Kipf et al.
2021 Kabra et al.] 2021 Zoran al. [021] Besbinar & Frossard 202T
Creswell et al. 2020 202
Elsayed et al. (2022) and in 3D scenes (Chen et al. 202 Henderson
Lamperi 2020 Crawford & Pineau 2020 Stelzner et al. 2021 Du al. 2021b Kabra al.
2021 Sajjadi et al. 2022] Nanbo et al. 2020 Singh et al.) 2022b). However, adopting Block-Slot
Representation dynamic and 3D scenes is a future direction and orthogonal to our current focus. Aside from these nuttoeencoding-basee
methods, alternative frameworks have also shown promise, notably compositional energy-based scene modeling (Du et al.) 2021a] Yu et al.) (02T), complex-valued neural networks Lowe et al.] (2022). and reconstruction-fres representation learning
(Caron et al. 2021; Löwe et al. 2020 Wang et al. 20222] H cnafff et al.) 2022 Wen et al. 2022 Baldassarre & Azizpour 2022).
However, these methods do not disentangle object properties.
C ADDITIONAL RESULTS
C.1 EXTENDED ANALYSIS OF OUR MODEL
In this section, we perform additional ablations of our model that shed further light on it.
In Fig. 7 we show all the ablation results together, including those that we could not include in the main paper due to a limitation of space. Below, we analyze these ablation results.
Sharing the concept-memory across blocks. We test what would be the effect of sharing not just the RNNs across the blocks but also sharing the concept memory across the blocks. We can see that as the level of weight-sharing increases, the 'completeness' score decreases. In other words, with more weight-sharing among blocks, the model tends use more blocks on average to represent a single factor. However, as this not drastic degradation, we can say that not the major factor responsible for the emerged disentanglement. What is erucial for getting good disentanglement doing each block's prototype attention and refinement independently.
is
Concatenating blocks to construct slot instead of Block-Coupling, In this ablation, test an alternative to the coock-couplngg approach: what if we simply concatenate the blocks in a slor and directly provide the of slots input to the transformer decoder? We find that this variant
16

---

Published as a conference paper at ICLR 2023
Analysis of Architectural Components (Extended)
Ours Seporats RMNs
Ous Shared RNAss
Ours Shared RMis Sharod Memories
B Concatenate Blocks
NN--RN
No- Coo-Connepttaiie Library
To Clock-Couplng
Mixture Decoder
[Cnnapt- Mener--rterr----h-- Projecccti
0.8
0.6
0.2
0
D
FG-ARI
Figure 7: Extended Analysis of Our Model. We report Disentanglement (D), Completeness (C), Informativeness (I), and FG-ARI.
achieves a significantly worse performance in the FG-ARI score while also achieving a slightly worse performance in the DCI scores.
Linear projections in concept-memory attention. In this ablation, we test the effect of applying linear projections on the query, key, and value when performing attention on the concept memory as follows:
(I (Sn,)) kim C..) "m
Sn,
softmax K
Vd.
U/ 10 (Cm))
Here, Ims ki and Um are the per-block linear projections for computing the queries, the keys, and the values. We find that this variant of our model is much more unstable and performs significantly worse than our default architecture.
C.2 ANALYSIS OF CONCEPT MEMORY AND PROTOTYPES
In this section, we qualitatively analyze several questions regarding the concept-memory bottleneck and the prototypes.
Does concept memory attention perform hard selection of single prototype? We analyze the attention weights of a block on the concept memory in Fig. g We note that the weights are not one-hot and thus the model is not making single hard selection of prototype. Rather, it is performing a soft selection of multiple activated prototypes. Thus, its capacity not limited to the K prototype vectors but be much more flexible.
How is the coverage of prototype usage? We analyze the average activation of each prototype across large batch of input images in the CLEVR-Eas dataset. We show these average activations in We find that a large majority of prototypes seem to be active.
Fig. 9
C.3 BLOCK-WISE OBJECT CLUSTERS
C.4 COMPUTATIONAL REQUIREMENTS
In T we compare the memory requirements of our model with respect to IODINE, our main baseline that pursues intra-slot disentanglement. Here, we can see that our memory consumption is better than that of IODINE. We also note that our training speed is slightly slower than that of IODINE. However, we must interpret these results with care and take into account the fact that IODINE performs significantly worse than model when it comes disentanglement performance.
17

---

Published as a conference paper at ICLR 2023
Block
Block
Block
Concapt Prototypes
Figure 8: Prototype Attention Weights. Three instances of blocks attending to the prototypes in the concept memory. cell (m, k) in the visualized matrices corresponds a block m = 1,.... 8 and prototype vector k = 1, ... 64 from the concept memory of the corresponding block. We note that the attention patterns are not one-hot selection of a single prototype. Instead, it a soft and flexible linear combination of multiple prototypes.
0
5
10
20
30
40
50
60
Figure 9: Prototype Coverage. We show the attention weight of each prototype averaged over batch of 300 input images of CLEVR-EES Easy dataset. Each cell (m, k) in the visualized matrix corresponds to a block m 1,...,& 8 and prototype 1,.., 64 from its corresponding concept memory C..-
IODINE
Our Model
Memory Consumption
37.30 GiB
31.01 GiB
Time per Training Iteration
0.84s
1.01s
Table 1: Comparison of computational requirements between IODINE and our model. For fair comparison, all values are measured on training runs with the same batch size of 40.
18

---

Published as a conference paper at ICLR 2023
CLEVR Tex
Block represents shapes
represents
Block represents positions
CLEVR Easy
Block represents shapes
Block represents colors
Block represents positions
CLEVR Hard
Block 6 represents shapes
Block 9 represents colors
Block 10 represents material
Figure 10: Visualization of object clusters obtained by applying k-means specific blocks in CLEyR-Easy, CLEVR-Hard, and CLe Each block learns to specialize specific object
factor e.g., shape color, abstracting away the remaining properties.
19

---

Published as a conference paper at ICLR 2023
Input Image
Recon.
Shape Stoap
Color Smap
Position Seap
Objjcts chosen for swapping
Figure 11: Additional samples of factor-swapping in CLEVR-East
20

---

Published as a conference paper at ICLR 2023
Input Image
Recon.
Shape
Color Snop
Material Swap
Position Swap
Size Objocts chosen for swapping
Figure 12: Additional samples of factor-swapping in CLEVR-Hard.
21

---

Published as a conference paper at ICLR 2023
Image
Stape Snop
awapping
Figure 13: Additional samples of factor-swapping in CLEVR-Tex.
22

---

Published as a conference paper at ICLR 2023
Input Images
Extracted Object Block-Slots
Take
Take
Take
Take
Take
Shape
Color
Material from
Position
Size
from
from
from
from
Object
Object 2
Property-love
Novel Composition
Take
Take
Take
Take
Take from
Shape Material
Position
from
from
from
from
Object 1
Object 2
Property-level
Novel Scene Composition
Take
Take
Take
Take
Take Size from
Shape
Colar
Material from
Position from
from
from
Object
Object 2
Property-leved
Novel Scene Composition
Figure 14: Systematic Scene Generation CLEVR-Hard. are given input images from which we extract slots. Using these extracted slots, we compose new objects by combining object properties into novel combinations. By decoding these composed novel slots, we show that we can generate novel scenes.
23

---

Published as a conference paper at ICLR 2023
Input Images
Extracted Object Block-Stots
Take Shape
Take
Take
Color
Position
from
from
from
Object
Object
Property- -eper
Novel Scone Composition
Take
Take
Take
Shape
Color From
Position
from
from
Object
Object
Property-lavd
Novel Scene Composition
Take
Take
Take
Shape
Color
from
from
from
Object
Object
Property-level
Novel Scone Composition
Figure 15: Systematic Scene Generation in CLEVR-Easy. We are given 8 input images from which we extract slots. Using these extracted slots, we compose new objects by combining object properties into novel combinations. By decoding these composed novel slots, we show that we can generate novel scenes.
24

---

Published as a conference paper at ICLR 2023
D ADDITIONAL IMPLEMENTATION DETAILS
Dataset
Module
Hyperparameter
CLEvR-Easy
CLEVR-Hard CLEVR-Tex
General
Batch Size
40
40
40
Training Steps
200K
200K
400K
SysBinder
Block Size
256
128
256
# Blocks
16
# Prototypes
64
64
64
# Iterations
# Slots
4
6
Learning Rate
0.0001
0.0001
0.0001
Transformer Decoder
# Decoder Blocks
8
8
8
# Decoder Heads
Hidden Size
192
192
192
Dropout
0.1
0.1
0.1
Learning Rate
0.0003
0,0003
0,0003
dVAE
Learning Rate
0.0003
0,0003
0,0003
Patch Size
4 x 4 pixels
4 4 pixels
4 x 4 pixels
Vocabulary Size
4096
4096
4096
Temperature Start
1.0
1.0 30000
1.0
Temperature End
Temperature Decay Steps
30000
30000
Table 2: Hyperparameters of our model used in our experiments.
D.1 BACKBONE IMAGE ENCODER
The backbone image encoder consists CNN encoder that encodes the image into a feature map. The specifications of this CNN encoder described in Table 3 For CLEVR-Tes dataset, found useful to use a deeper CNN network described in Table [ After applying the CNN, positional encodings are added the feature map. Layer-normalization applied on the resulting feature map followed by a 2-layer MLP with hidden dimension 192. The resulting feature map is flattened along the spatial dimensions to produce
of input features that are then provided the SysBinder module. The positional encodings are learned in a similar way as in Slot Attention by mapping 4-dimensional grid of position coordinates
92-dimensional vector via a linear projection.
Layer
Kernel Size
Stride
Padding
Channels
Activation
Conv
5 X 5
2
2
512
ReLU
Cony
5
2
512
ReLU
Conv
5 X 5
1
2
512
ReLU
Conv
5
192
None
Table 3: Specifications of CNN layers of the backbone image encoder in CLEVR-Easy and CLEVR-
Hard datasets.
D.2 RECURRENT BLOCK UPDATE
The block update is performed using a GRU network and 2-layer MLP network with dimensions equal to the block size. For maintaining the separation of GRU and MLP model parameters between blocks and to apply GRU and MLP on all the M blocks in parallel, we adopt the implementation of Goyal al. 2222bb)
25

---

Published as a conference paper at ICLR 2023
Layer
Kernel Size
Stride
Padding
Channels
Activation
Conv
X 5
512
ReLU
Conv
X 5
2
512
ReLU
Conv
5 X 5
2
2
512
ReLU
Conv
5
2
512
ReLU
Conv
5 X
2
192
None
Table 4: Specifications CNN layers of the backbone image encoder in CLEVR-Tex dataset.
D.3 CONCEPT MEMORY
To facilitate the training of the concept memories Cl..... Cm we parameterize prototypes as an MLP projection of learned vector: Cm,,k ML (Em,k), (em,)) where each cm.k is a learned vector of dimension equal to the block size. The MLP has 4 layers with hidden dimensions times the block size and the output dimension
be equal to the block size.
D.4 IMAGE TOKENIZER AND TRANSFORMER
We use discrete VAE, or simply dVAE, as our image tokenizer. We adopt the same architecture as that used in SLATE (Singh al. 2022a). Like SLATE, the temperature of Gumbel-Softmax (Jang et al.) 2016) is decayed using cosine-annealing from 1.0 to 0.1 in the first 30K training iterations. Like SLATE, the dVAE trained simultaneously with the other parts of the model. We use a patch size of sequence of 1024 discrete
4 x 4, meaning that 128 x 128-sized image would be represented tokens. To learn to generate these tokens rugo-regressively we adopt the same transformer decoder and configurations SLATE without any architectural significant hyperparameter changes.
D.5 TRAINING
Similar to SLATE (Singh et al. 2022a), we split the learning rates of the dVAE image tokenizer, the SysBinder encoder, and the transformer decoder. The learning rates and the full set hyperparameters are provided in Table 2 During training, the learning rate is linearly warmed up to its peak value in the first 30K training iterations and then decayed exponentially with half-life of 250K iterations.
ADDITIONAL. EXPERIMENTAL DETAILS
E. VISUALIZATION OBJECT CLUSTERS USING K K-MEANS ON BLOCKS
In this section, we provide more detailed description of the visualizations of Fig. 1O To generate these visualizations, we first take large collection images. On these, we apply BSA and extract S = (S,,, x[tm, x= a collection of slots paired with its corresponding object image x of the obtained by multiplying the input image with the attention mask the slot s,. From this collection, take m-th blocks as Sn, [(st., ((.tm..."m)) x:"")) for each block index m. We then apply K- K-means the block vectors in S., to obtain clusters blocks. We use K = 12. In Fig. 10, we visualize the blocks inside some of the clusters by showing the object that those blocks
representing. In the top- of Fig. [0] we show two of the clusters for the block index for CLEVR-Tex by showing the object images of blocks that lie in those clusters. shows that all the sphere objects had representation of block 6 close to each other. Similarly, all the cylinder objects also have block 6 representations that are close each other. This shows which concept each block is capturing and which it abstracting away. That is, in this case, block 6 is capturing the shape factor of the object but abstracting away other properties such as material or position.
E.2 SELECTING BLOCKS FOR SWAPPING
We selected the blocks representing specific factor by first applying K K-means each m-th block as in the qualitative analysis of Fig. [0] We then manually inspected the resulting object clusters, and identified, for each block, which factor block is specializing in.
26

---

Published as a conference paper at ICLR 2023
F LIMITATIONS AND DISCUSSION
The proposed model also shows some limitations. First, although our model significantly outperforms the baselines, the absolute performance still far from perfect. Second, the vector-formed factors in our model make slots larger and thus take more computation. Lastly, our transformer decoder can be computationally more expensive than the mixture decoders used in the conventional approaches.
A few future directions are as follows. First, it would be interesting to extend the model to make it work on more complex natural scenes even though the current dataset already much more complex than those in similar line of research (Greff et al., 20199] Locatello et al., (020). Second, it would be interesting to study the multi-modal aspect of the proposed model, particularly with language. Finally, an extension to video learning is also interesting.
27


Reviews:
[{"review_id": "vTkeRlh2GFE", "reviewer": "Reviewer_jDiA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel object-centric representation called Block-Slot Representation, which, unlike the conventional slot representation, provides concept-level disentanglement within a slot, such as color, texture, and position. In comparison with the previous methods, the approach demonstrated significantly better disentanglement of object properties, including complex textured scenes. Finally, the proposed method improves the results on a number of synthetic datasets (CLEVR).\n", "summary_of_the_review": "I wrote above my concerns. I am open to the authors' feedback and other reviewers' opinions.\n", "strength_and_weaknesses": "**Strengths:**\n\n(1) Overall, this paper is well written, and the technical details are easy to follow. \n\n(2) The main idea of representing a factor of object variation, such as color, texture, and position, is nice and appealing.\n\n(3) The main contribution of this paper is a new method for binding block factors within a slot.\n\n(4) The experiment results support the proposed approach, including the evaluation of the DCI framework, which is a valuable addition to the paper.\n\n\n**Weaknesses:**\n\n**Experiments.** I am mostly concerned with the fact that this approach is mainly proved to be working on synthetic datasets. Thus, I wonder how it would generalize to a real domain. I think the idea is appealing, but I am honestly not sure it will work on real-life datasets.\n\n\n**Novelty.** First of all, I would like to note that the main idea of this paper maintains the high standards of the conference. However, the authors should also know that I had indeed considered this idea important and implemented it 14 months ago, when the first slot-attention paper was published. The method of disentangling the structure within a slot in order to represent a factor of four object variations, such as color, shape, texture, and position, appears to be working entirely on synthetic datasets (CLEVR-like). Due to the fact that it did not work on real domains, as well as the simple nature of the contribution of the main idea (which is, to me, way too simple in this case, although there is a compelling story behind it), I decided to abandon this research direction. Despite acknowledging the importance of this paper, we should also be aware that its contributions could not be substantial enough, especially if we look at the results on real domains.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bteEDYJKXg", "reviewer": "Reviewer_NtVS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a novel structured representation of scenes with several levels of hierarchy. The representation is divided into slots which provide object compositionality, and slots have several blocks of latent dimensions which are aimed to disentangle factors of variations for each object. To obtain this representation, the encoder architecture can be seen as a generalization of the well known Slot Attention component to the Block-Slot setting. The model is trained in a standard autoencoding setting, with a decoder inspired from recent work in compositional generative modeling for scenes (SLATE, STEVE), also generalized to Block-Slots via original additions to the architecture. An experimental evaluation on different variations of the CLEVR dataset clearly demonstrates the advantages of the design choices for representation compositionality, quantified as pixel-level object separation and disentanglement. ", "summary_of_the_review": "This is a novel and original contribution building on the well known Slot Attention component. The technical description is clear, but I have raised a number of concerns about the high-level presentation and phrasing of claims. I am looking forward to discussion with the authors in order to make a stronger acceptance recommendation.", "strength_and_weaknesses": "The contribution is well scoped, and obtaining good multi layer disentanglement is a very important challenge. Results in Figure 2 and 3 evidence benefits of the proposed method with a clear gap in reported metrics compared to prior work. I however have some concerns about clarity of the presentation and some experimental choices. I believe that these concerns can be addressed during the rebuttal phase. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "XOoFufbote", "reviewer": "Reviewer_v6kB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a modification of Slot Attention to make it learn sub-components which are independent of each other (blocks), built out of prototypes which are attended to be independent RNNs. The model is trained through reconstruction (with a transformer+dVAE decoder like recent works) and performs very well on several extensions of the CLEVR dataset, including one with textures (current techniques struggle with non-trivial textured datasets). The results are strong and the presentation is very clear.", "summary_of_the_review": "Overall, I found this paper extremely interesting, presenting several key novel ideas which appear simple in hindsight which is always a great sign. Results are strong, the evaluation is good with good ablations and the paper demonstrates what I wanted to see and understand.\n\nAs a practitioner of this field, this is absolutely something I will try to leverage and build upon, hence I would recommend for acceptance.", "strength_and_weaknesses": "\nOverall, I really liked this paper in this current form so I have very little extra to request:\n\n\n1. The Abstract and Introduction are very strong, easy to follow yet entirely clear, it was really well done. Figure 1 conveys nearly the entirety of the model in one single diagram which is fantastic.\n   1. The only nitpick I would have is to indicate that there are M blocks in total, this is the only missing symbol currently. \n2. The Concept Memory bottleneck is potentially both the biggest novelty or drawback of the technique in its current form.\n   1. It was surprising to me that you would directly attend into it by multiplying s_nm with Cm. Somehow I would have expected something more akin to a query/key setup with additional Linear layers. Did you try something like that?\n   2. The choice of making the prototypes the direct outputs is rather strong. \n      1. I feel like this may be too constraining and brittle, for example in complex situations? \n      2. I had to find how many prototypes K you used in the Appendix. You seem to use K=64 throughout, which is rather small. This seems rather much more constraining than what is done VQ-VAE/GAN or GroupVIT-style models.\n      3. How “delta-ish” were the softmax attention in general? If the softmax becomes extremely peaky, this means you considerably restrict the “space” of representation (i.e. in the limit, you would only represent 64 possible values, given your choice of K)\n      4. What happens if the dictionary size is much larger? What about if you share it between groups?\n      5. Did you try outputting the “distance” to the prototypes instead, to have more flexibility and expressive power?\n   3. Didn’t you have to use more flexible attention mechanisms, ala Gumbel, to train these prototypes well enough? I would expect the gradients to die off rather quickly in unused prototypes.\n3. The Block Binding is a very interesting choice, and it seems rather useful to me so I will probably give it a try in my own work.\n   1. However, I would have expected you to try the simplest choice of “just” concatenating all the groups together per slot first? I.e. provide $S=\\lbrace s_n \\rbrace$, where $s_n = Concat_m(s_{n, m})$.\n   2. You never compared against that, even in Section 6.2? I would never have expected “the bag of blocks” to be an appropriate representation, provided as a set to the decoder. You do have the block-slots directly, so why don’t you just keep that structure? How does Figure 6a change with that choice?\n4. The results are strong, and the improvement to the metric is well executed. Figure 2 is clear and very impactful.\n5. Figure 4 and the methodology for obtaining it is not clear enough.\n   1. How did you “mask out” part of the images? Did you use the attention weights?\n   2. The corresponding section on 6.1 at the end of Page 7 needs to be expanded to provide more details, as this is a very interesting assessment which is the only way to assess what the groups and prototypes really learn.\n   3. How many prototypes end up being learnt/used? Can we get some kind of “coverage” metric for their relative importance / specialization?\n6. Figure 5 is strong and very interesting to see.\n   1. It might be worth flagging that the model gets the reflection right (e.g. last row, second to last column on CLEVR-Hard). Do you know if this is thanks to your model, or “simply” due to the Transformer+dVAE decoder being really good at handling these?\n   2. How exactly did you select the groups? You say it was easy to identify and select them, but you do not precisely say how.\n7. Even though you target more complicated versions of CLEVR, this is still “just” CLEVR, which is rather easy to get results on.\n   1. Did you try targeting Kubric [1] (MOVi-C onwards, discarding the temporal video aspect) or other more complex yet still manageable datasets?\n\n\n[1] https://arxiv.org/abs/2203.03570 ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "08AjqE4yYj6", "reviewer": "Reviewer_HPVD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a block-slot object-centric representation learning scheme\nto use a set of sub-slot representations, aka., block representations, to explicitly encode different properties that\nare conventionally encoded with only one slot representation.\nThe authors show the proposed disentanglement scheme performs better and enjoys better interpretability.\n\nThis idea also seems related to the Part-and-Sum detection Transformer[1] used to perform visual relationship\ndetection, where the authors also propose to use part queries to explicitly\nencode the different partial information that is originally encoded with one sum query.\nThe authors are highly encouraged to discuss whether the proposed method can be applied to\nbroader vision tasks based on DETR such as object detection and segmentation.\n\n[1] Visual relationship detection using part-and-sum transformers with composite queries, CVPR2021", "summary_of_the_review": "How to perform within-slot disentanglement to real-world images in an unsupervised manner is a grand challenge. This work presents some encouraging results in this direction. However, the experiments are relatively weak and the authors are encouraged to extend the proposed approach to more challenging benchmarks, which might be necessary to attract broader attention from the research community. I would like to increase the ratings if the authors could well address the above concerns.", "strength_and_weaknesses": "> Strengths\n\n✅ The presented idea is interesting and the illustration is clear.\n\n✅ The approach section is well-organized into two subsections including (i) the details of the proposed block-slot attention and (ii) how to\nperform object-centric learning with a block-slot attention scheme.\n\n\n> Weaknesses\n\n❎ According to Figure 1 and the Approach section, we can see that the proposed approach might bring significant extra computation overhead compared\nto the original baseline approach. The authors should report detailed complexity comparisons such as the number of parameters and GFLOPs.\n\n❎ The authors only verify the proposed approach on the very toy benchmarks like CLEVR-Easy, CLEVR-Hard, and CLEVR-Tex.\nIt would be great if the authors could extend the approach to some real-world benchmarks following very recent works such as SAVi[1] and SAVi++[2]. Besides,  the authors are encouraged to discuss more details and compare the proposed approach to GENESIS-V2[3] empirically if possible.\n\n[1] SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models\n\n[2] SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos\n\n[3] GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement\n\n❎ The emergence of semantically meaningful abstract concepts in blocks shown in Figure 4 is very impressive. The authors are encouraged to conduct detailed\nablation experiments to investigate which component is the key to ensuring the emergence of semantically meaningful abstract concepts and whether this conclusion\nstill holds on some real-world benchmarks such as ImageNet and COCO.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster