Quantized Disentangled Representations for Object-Centric Visual Tasks
Authors: Daniil Kirilenko
Alexandr Korchemnyi
Alexey Kovalev
Aleksandr Panov
Venue: Submitted to ICLR 2023 (2023)
Keywords: quantised representation
disentangled representation
object-centric task

Abstract: Recently, the pre-quantization of image features into discrete latent variables has helped to achieve remarkable results in image modeling. In this paper, we propose a method to learn discrete latent variables applied to object-centric tasks. In our approach, each object is assigned a slot which is represented as a vector generated by sampling from non-overlapping sets of low-dimensional discrete variables.
We empirically demonstrate that embeddings from the learned discrete latent spaces have the disentanglement property. The model is trained with a set prediction and object discovery as downstream tasks. It achieves the state-of-the-art results on the CLEVR dataset among a class of object-centric methods for set prediction task. We also demonstrate manipulation of individual objects in a scene with controllable image generation in the object discovery setting.

Under review as a conference paper at ICLR 2023
QUANTIZED DISENTANGLED REPRESENTATIONS FOR OBJECT-CENTRIC VISUAL TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, the pre-quantization of image features into discrete latent variables has helped to achieve remarkable results image modeling. In this paper, we propose method learn discrete latent variables applied object-centric tasks. In our approach, each object is assigned a slot which is represented as a vector generated by sampling from non-overlapping sets of low-dimensional discrete variables. We empirically demonstrate that embeddings from the learned discrete latent spaces have the disentanglemen property. The model trained with set prediction and object discovery as downstream tasks. It achieves the state-of-the-e results the CLEVR dataset among a class of object-centric methods for set prediction task. We also demonstrate manipulation of individual objects in scene with controllable image generation in the object discovery setting.
INTRODUCTION
The known problem of existing neural networks is that they cannot generalize at the human level (Lake et al. (2016); Greff al. (2020)). assumed that the reason for this the inability of current neural networks to dynamically and flexibly bind information distributed throughout it. This called the binding problem. This problem affects the ability of neural networks 1) to construet meaningful representations of entities from unstructured sensory inputs; 2) to maintain obtained separation of information at representation level; 3) reuse these representations of entities for new inferences and predictions. One way to solve this problem is to constrain the neural network for learning disentangled object-centric representations of a scene (Burgess et al. (2019); Greff et al.) (2019): Yang et al. (2020b)).
The disentangled object-centric representation may potentially improve generalization and explainabil- ity in many machine learning domains such structured scene representation and scene generation (El-Nouby et al. (2019); Matsumori et al. (202T); Kulkarni et al. (2019)). )) reinforcement learning (Keramati et al. (2018); Watters et al. (2019a) 9a); Kulkarm et al. (2019 9 Berner et al, (2019); [Sun et al.)
(2019), planning (Migimatsu & Bohg (2020)) reasoning (Yang et al. (2020a)), and object-centric visual tasks (Groth al. (2018); Yi et al. (2020); [iinhh al. (202 However, recent research has focused either on object-centric or disentangled representation and has not paid enough attention to combining them. There are just several works that consider both objectives (Burgess et al. (2019):
Greff et al. 2019); Li et al.
2020);; Yang et al. (2020b)).
We propose method that produces the disentangled representation of objects by quantization of the corresponding slot representation. We call it Vector Quantized Slot Attention (VQ-SA). VQ-SA obtains object slots an unsupervised manner (Locatello al. (2020) and then perform quantization. The slot quantization involves two steps. At the first step, we initialize several discrete latent spaces each corresponding to the one of potential generative factors in the data. At the second step, we initialize each latent space with separate embeddings for potential values of corresponding generative factor. This two-step quantization allows the model to assign a particular generative factor value particular latent embedding.
The proposed object-centric disentangled representation improves the results of the conventional model from Locatello al. (2020) object-centric visual tasks such as set prediction compared to Light-weighter specialized models (Locatello et al. (2020); Zhang et al. (2019)). We demonstrate through extensive experiments on the CLEVR dataset (Johnson et al. (017))

---

Under review as a conference paper at ICLR 2023
To measure degree of disentanglement the commonly used disentanglement metrics are VAE score (Higgins et al. (2017a)), MIG (Chen et al. (2018)) DCI disentanglement (Eastwood & Williams (2018a)), SAP score (Kumar al. (2018a)), and FactorVAE score (Kim & MMmh][2015
These metrics are based on the assumption that disentanglement is achieved on a vector coordinate level, i.e. each coordinate corresponds to the generative factor. In our approach, generative factors expressed by vectors, and separate coordinates are not interpretable. Thus, metrics listed above are not suitable and the problem of quantitative evaluation of disentanglement in the case of vector representation of generative factors remains an open question for future studies. Nevertheless, we propose DQCF-micro and DQCF-macro methods that qualitatively evaluate the disentanglement in the object discovery task. Original Slot Attention based model achieves remarkable results in the object discovery task, but our model allows not only to separate distributed features into object representations but also to separate distributed features of the objects themselves into representations of their properties.
We first give an overview of the proposed model VQ-SA (Section 2.1 Then, we provide detailed explanation of the slot quantization approach (Section 2.3) we use to represent objects from an image. We conduct experiments on the CLEVR dataset (Johnson al. (2017)) for the set prediction task (Section [.1] and show that our model achieves the state-of-the-aas results in some settings and performs comparably well in others. We also conduct experiments for the object discovery task (Section (.2) and show quality results for the CLEVR dataset (Johnson et al. (2017)). We conduct ablation studies (Section | and provide results of modified versions of the proposed model to confirm our design choices. The learned discrete latent spaces possess the disentangled property. We qualitatively demonstrate this (Section 4 by analyzing set prediction results. Finally, we position our work relative to other approaches (Section (b) and discuss the obtained results, advantages, and limitations of our work (Section 7
Our main contributions are follows:
We propose a discrete representation (quantization) of object-centric embeddings (Section 2.3) that maps them to several latent spaces.
The quantization produces disentangled representation (Section 4 there the disentanglement achieved on the level of latent embeddings rather than embedding coordinates.
Learned discrete representations allow us to manipulate individual objects in a scene and generate scenes with objects with given attributes by manipulation in the latent space
(Section 3.2
The proposed model VQ-SA achieves state-of-the-ar results on the set prediction task on the CLEVR dataset (Section 3.1 among class of object-centric methods.
We propose DQCF-micro and DQCF-macro methods that qualitatively evaluate the disen- tanglement of the learned discrete variables, when they are represented by vectors rather by vector coordinates.
2 METHOD
2.1 OVERVIEW
To obtain valuable object representations, we should first discover objects in the image and then transform their representations into desired ones. We discover objects in an unsupervised manner with the use of slot attention mechanism (Locatello et al. (020)). The idea of slot representation is to map an input (image) to a set of latent variables (slots) instead of a single latent vector (Kingma & Welling 20 Rezende et al. (2014)) such that each slot will describe a part of an input (Locatello et al. 22200; Engelcke et al. (2020) 202T)). We assign each object to a slot of a dimension d.. Further, we transform each slot to a desired latent representation. We draw inspiration from the discrete latent representation proposed in van den Oord et al. (2017) and apply its modification to each slot. We use multiple latent spaces with small embedding dimension dy (dj < d,)) and the small number of embeddings in each latent space instead of using single discrete latent space to map slots. The small dimension of vectors in the latent spaces enables us to construct the resultant slot representation by concatenation. That could be seen constructing a new vector of factors from given ones.
2

---

Under review as a conference paper at ICLR 2023
The main assumption behind this design choice is that each object is generated by fixed number of generative factors. Thus, is possible to represent each object (slot) as a combination of embeddings corresponding values of generative factors. As most of the generative factors have discrete nature, the proper choice of the generative distribution would he the categorical one. Particularly, we choose the number of categorical distributions equal to that of generative factors with the number of possible categories equal to that of values of a corresponding generative factor. The overall architecture of the proposed model VQ-SA depicted in Fig.
a) Praprocsasing stop
Empty
Quantization
Prop. Prodiction Object Matching
Set Prediction
CondF
Image
sumpre
Filled
Slot Attention
Gumbel Softmak
CatGF Projectio
Property Prediction
Otyy
Encoder
Entelliing Spices
Match
Ott
Rutten
Postionna Embbeoding
Preprocassing step
Empty Stots
(b) Slot Quantization
Decoding & Reconstruction
Object Discovery
Reconstruche Image
Filled Slets
Attention
Stot
Gumbel Softmax
CatoF Projectio
Siet
Encoder
Eitloling Spaees
flmen
Embedhhing
Figure Overall architecture of the proposed VQ-SA mode. Top part corresponds to the set prediction task and bottom part corresponds to the object discovery task. We assign objects to slots in an unsupervised manner. The preprocessing and slot quantization part are the same in both tasks. To process categorical generative factors (CatGF), i.e., shape, color, size, material, perform quantization by multiple discrete latent spaces with a Gumbel-Softmax trick. For continuous generative factors (ConGF), i.e., x, y, coordinates, the case of the object discovery task, we treat all generative factors as categorical. For continuous factors, this corresponds to splitting it into bins. For the set prediction task, process categorical and continuous generative factor separately. Then we concatenate both representations and predict object properties.
estimate their values (ConGF Estimate). In
The VQ-SA pipeline for set prediction task consists of three stages. At the first preprocessing step (Fig. [h), the image is flattened by Encoder, combined with positional embeddings, and objects are assigned to slots by an iterative slot attention mechanism. At the second stage (Fig. [b). we perform separate processing of categorical generative factors (CatGF) and continuous generative factors (ConGF). We use slot quantization by multiple discrete latent spaces to represent categorical generative factors and estimate values continuous generative factors with ConGF Estimate module. At the final, third stage, (Fig. [T)) we combine both types of generative factors and predict object properties. Then, we match the predicted objects with the ground truth using the Hungarian algorithm (Kuhn 1955)).
For the object discovery task, the pipeline differs at the third stages. We process continuous generative factors similar to generative factors (Fig. (T) and use the same decoding strategy as in the original
Slot Attention Locatello al. 2020 (Fig. b).
2.2 SLOT REPRESENTATION
We follow the procedure proposed in Locatello et al. 2020)) to represent image as of slots S,, I...N, where the slots are vectors dimension d,. Each slot S, is initialized randomly
S μ t a. +N, 1) and is used assign object O 1... M or background from
3

---

Under review as a conference paper at ICLR 2023
image in an iterative procedure. The main feature of slot attention mechanism is that slots competing with each other for assigning an object, i.c., attention coefficients are normalized over dot-product attention (Lunng al. (2013))
the slots. Following Locatello Locateloo et al. (2020), attention mechanism for slots and aggregate the resultant slot representation by Gated Recurrent Unit (GRU) (Cho et al. (2014) (Fig.
2.3 SLOT QUANTIZATION
After assigning the objects to the slots, each slot S; is represented discrete latent spaces Lys k = 1... K, K is the number of latent spaces corresponding the number of categorical generative factors (K = 4 in the case of the CLEVR dataset). Each latent space L3 is initialized with embeddings ef .. 7ig. with dimension di (di < d,). T&p the number of embeddings in the latent space Li, is equal to the number of categories of a corresponding categorical generative factor. We linearly project slot S, lower dimension dj: S = .MS; (S₁ R4 M Rd S, Rd.x1). Then, we construct new representation of slot S' in each discrete latent space L, by the Gumbel-Softma trick (ang et al. (2016); Maddison al. (0066). First, calculate the similarity between the slot S and each embedding ej in latent space L4 to get posterior distributions q(e)) ((5) (S)) by normalization with Softmax function:
sim = (S)) = (2, mull
exp(simtt
(1)
a(c)1s)) Σn exp(simf)
To get the continuous approximation y of the me--hot-encoded representation of the discrete variable ef we use Gumbel-Softmax trick with a constant temperature parameter
(notlogia(e)155)
exp
(2)
(g, (g,+log(g(e"1s)))
E, exp
where g denotes random samples from Gumbel distribution.
The resultant representation at of slot St in the discrete latent space Le a weighted sum of all embeddings ex E LA from L3 with weights y [yt v2. ..... v
c
(y ve.... sh Iiet eg. e
(3)
Then, representations of S from all discrete latent spaces Li, k 1. K are concatenated:
SP further is used as quantized representation of the slot Sy. Concatenation could be construction of new vector representation S:? from separate generative factors é! ,.,...,
2.4 COORDINATES REPRESENTATION
We use discrete latent spaces to represent generative factors with categorical nature. However, such encoding of continuous generative factors, e.g., coordinates, may lead to significant errors due to quantization. Also, there is no unique way to represent a continuous value by a discrete segment, as the length of segment could be viewed hyperparameter.
In the set prediction task, we explicitly split generative factors into categorical (shape, color, size, material) and continuous (z, y, 2 coordinates). We use slot quantization (Section 2.3) to represent categorical generative factors and obtain a quantized slot representation S." We use multilayer perceptron (MIP) with two layers predict object coordinates.
4

---

Under review as a conference paper at ICLR 2023
In the object discovery task, we also process coordinates separately but in the manner similar to categorical generative factors (Section 2.3). The only difference is that we use a single latent space rather than multiple latent spaces.
2.5 ENCOURAGING DISENTANGLEMENT
In order to aid disentanglement between discrete latent spaces, we add the following well-known term the loss function:
Σ (L(g(La\S")ll((a))
(5)
p(Lx) is a true prior uniform categorical distribution over discrete latent space Lx, posterior categorical distribution over discrete latent space predicted by neural network. Using this loss term, we force posterior distributions over each latent space q (L= )") to be independent and closer to prior distributions, which results in better disentanglement between spaces.
a
EXPERIMENTS
In this section, demonstrate the two advantages of using the additional vector quantization module. First, it improves performance on the set prediction task without drastically changing the overall model architecture and transforming it into a highly specialized set prediction model (Section Second, unsupervised training for the object discovery task makes it possible to learn representations that could be used to edit individual objects in the image in an interpretable manner (Section 3.2).
For all tasks, we first trained the original Slot Attention (Lccaeelo et al. (2020) for about 600K iterations and used the learned encoder and slot attention weights to initialize the corresponding modules of the VQ-SA model. We also experimented with end-to-end learning (Appendix A, Table and Fig. 6). The model is trained in this mode, but converges more slowly, which complicates the setting of multiple experiments.
3.1 SET PREDICTION
In the set prediction task, the model receives an image as an input and predicts target features as the unordered set of object vectors. The vectors of predicted and target features are matched using the
Hungarian algorithm (Kuhn (1955). ). To quantify the quality of the model, the Average Precision (AP) metric with a certain threshold is used. A detected object considered being true positive the of its predicted properties exactly matches the ground truth object and the position of the detected object predicted within a threshold t relative to the ground truth object. The threshold oo (APx>) means we do not use the distance threshold. One of the difficult aspects of this task is the invariance of sets to permutations. This specific property is explicitly modeled in the architecture using the Slot Attention module (Locatello et al. (2020). which turns distributed representation of a whole scene into set of object representations. The goal of our method to model the distribution of object features. The number of vector spaces use is equal to the implied number of generative factors. From each latent space, we sample one vector and concatenate them into the final object representation. As shown in Table I compared to the original Slot Attention based model, our approach allows the model make more accurate predictions.
also worth noting that the proposed model significantly improves the results for small thresholds, more than twice for the 0.25 threshold and more than and a half times for the 125 threshold. Compared to a highly specialized model for the set prediction task, iDSPN (Zhang et al. (2022)), our proposed model VQ-SA demonstrates comparable results for the thresholds that are greater than one and moderate results for smaller thresholds. The reason is that the architecture our model was not specially tuned to the set prediction task and could be used for other tasks, e.g., object discovery. Another key difference is that the learned discrete representations are disentangled (Section 4 and could be used to generate images with individual objects edited in an interpretable manner (Section 3.2
Setup To make correct comparisons with the Slot Attention model, we use the same hyperparam- batch size of 512, three iterations of Slot Attention, and 150,000
eters during training:
5

---

Under review as a conference paper at ICLR 2023
Table 1: Performance on the CLEVR object property set prediction task. For DSPN (Zhang al. (2019)) Slot MLP and Slot Attention we use results from Locatello et al. (2020) (mean std for five seeds). The results of iDSPN are reported following Zhang et (2022) For the proposed model, mean and std are calculated for four seeds.
Model
AP (%)
AP1 (%)
APo.5 (%)
APo.25 (%)
125.125 (%)
Slot MLP
19.8 + 1.6
1.4 0.3
0.3 0.2
0.0 0.0
0,0 + 0.0
DSPN T=30
85.2 + 4.8
81.1 5.2
47.4 + 17.6
10.8 9.0
0.6 0.7
DSPN T=10
72.8 + 2.3
59.2 + 2.8
39.0 + 4.4
12.4 2.5
1.3
Slot Attention
94.3
86.7 1.4
56.0
3.6
1
0.9 0.2
VQ-SA (ours)
96.1
91.2 - 0.5
71.8 + 2.3
22.2 - 2.1
2.4 + 0.2
iDSPN
98.8 + 0.5
98.5 + 0.6
98.2 + 0.6
95.8 + 0.7
76.9 + 2.5
training iterations. The model trained with the Adam (Kingma & Ba (2013) optimizer with learning rate of 0.0004. We also use learning rate warmup with an exponential decay schedule after that. The number slots is equal to 10, as we use the CLEVR (Johnson et al. (207)) dataset, except the images with than 10 objects. The encoder architecture is shown in Appendix A, Table The remaining model hyperparameters are presented in the Appendix in Tables 4a and 4b
3.2 OBJECT DISCOVERY
In the object discovery task, the model receives image, separates distributed scene features into individual object features, and uses them to reconstruct objects, combining them into the original scene image. We use the Slot Attention module, which transforms input convolutional features into a set of vectors, and a convolutional decoder (Watters et al. (2019h)). which decodes each slot independently into the 4-channel image, where the first three channels are RGB color channels and the fourth channel is an unnormalized mask. To merge the reconstructed objects into the scene, the masks are normalized across all slots using Softmax and used as mixture weights. The original Slot Attention-base model achieves remarkable results in the object discovery task, while our extension allows the model not only to separate distributed features into object representations separate distributed features of the objects themselves into representations of their properties.
also to
Fig. 2 demonstrates the ability manipulate individual objects in a scene with our model: each column corresponds the column is obtained by changing only one discrete component the object representation to the discrete latent variable from the same space.
change in one attribute (except the fourth column). Each next image in
x
color
sibe and thope
Setup We use the same training setup the one the original Slot Attention- based model: the main term in loss is the mean squared image reconstruction error, the batch size is equal to 64, and the optimizer and learning rate setup are the same in Section 3.1 In this case, we utilize the CLEVR (Johnson et al. (2017) dataset without images with more than six objects, so the number of slots is equal to 10. We also use an additional loss term, which aids disenten- glement. It is described in Section 2.5 The encoder and decoder architectures are shown Appendix in Tables 5 and respectively. The remaining model hyperparameters are presented in the Appendix A Tables 4a and 4c We also conducted experiments with the Tetrominocs Kabra et al. (2019) dataset for the object discovery task (Fig. 8 in Appendix A).
6
Figure 2: Learned discrete representations enable the manipulation of individual objects in the scene.

---

Under review as a conference paper at ICLR 2023
4 DISENTANGLED REPRESENTATIONS
To qualitatively evaluate the disentanglement of the trained discrete variables, we propose methods DQCF-micro and DQCF-macro (DQCF stays for Disentanglement Quality of Categorical generative Factors) that work when the value of generative factor is represented by the vector rather than particular position in a vector. DQCF-micro evaluates disentanglement with respect to all other vectors from all discrete spaces, while DQCF-macro evaluates it the level of discrete spaces. For DQCF-micro, we calculate each latent vector e, as the most similar vector. This statistics with Hungarian matching gives us frequency probabilities conditioned on the sampled latent vectors p(propp = value, (e,).. The example of this value for first latent space L is presented in Fig. 3 A strong closeness of the distribution of values of one property (e.g., values "small" and "large") ) to a uniform distribution means that this vector is not specific to objects with certain values of that property and does not contain information that is unique to that property.
every set of objects from the validation data frequency of sampling
each property propp. = 1... ....P P with values value,, 0 = 1...
0.8
voctor number
vector
vector
0.6
vector
0.4
0.2
0.0
large
nubber
sphere
cylindar
gray
yellow
red
green
purple
cyan
properly
Figure 3: Results DQCF-micro. Example of p(propp valueyle? (et)) for embeddings from the first space. The probability is calculated as the frequency of objects with value, of property propp for which the vector ey was sampled.
Therefore, for DQCF-macnt we further calculate the standard deviation over all values of corre- sponding properties and get the mean value over vectors from the same space. The obtained values presented in Fig. Intuitively, these values show how much the change in the object properties affects the change in the distribution over a particular embeddings space. High values indicate that latent variables from this space contain information specific this property, while low values indicate the opposite. It can be seen that the information specific to the property "color" is contained in the vectors of both the third and fourth spaces, i.e. they entangled, while information about other properties is distributed over vectors of single spaces.
To deal with this entanglement,
additional KI-
64
term in loss during training, described in Section 2.5 The difference in resulting disentanglement between models with and without this additional constraint is shown in
There are three images: the original and two with modi- fied cylinder. Both modifications are obtained by changing one latent variable responsible for the size of the object, but in the case of the model trained without an additional KL-term, this led to a change in color, while in the other case, only the size changed.
popperty
Figure 4: Results of DQCF-macro. Re- lation between variations in object prop- erties and changes the probability dis- tribution over sets of latent variables.
without KL-term
with KL-term
ABLATION STUDIES
As ablation studies, provide additional quantitative results on how constraints and extensions affect model performance in the supervised set prediction task.
Figure 5: Example of the increasing ob-
7
ject size with models: one trained without using a KL-term and the other- using it.

---

Under review as a conference paper at ICLR 2023
We investigated how increasing the number of embeddings in each space affects the results. We did not get a noticeable change in the metrics by increasing the number of vectors in each space to eight (32 in total) and 16 (64 in total). Another modification to use only single common space that also does not affect the performance. The results for different settings are shown in T
Table 2: Performance comparison of the Slot Attention model (Locatello et al. (2020)) our proposed VQ-SA model, and its modifications with additional constraints and extensions. The number in parentheses, e.g., VQ-SA(32), indicates the total amount of embeddings we used in discrete latent spaces, i.e., eight embeddings in each for four latent spaces. "Cspace" means that used only single latent space with a given number of embeddings. Ablation studies' results are calculated for five seeds.
Model
AP (%)
AP1 (%)
APo.s (%)
APo.25 (%)
AP0.125 (%)
Slot Attention
94.3 = 1.1
86.7 + 1.4
56.0 = 3.6
10.8 + 1.7
0.9 + 0.2
VQ-SA (ours)
96.1 = 0.4
91.2 = 0.5
71.8 2.3
22.2 2.1
2.4 + 0.2
VQ-SA(8, espace)
41.8 = 27.5
38.2 +: 26.5
27.6 18.3
7.0 4.4
0.7 + 0.4
VQ-SA(16, espace)
83.3 = 7.7
79.3 = 7.2
58.8 4.5
15.3 1.2
1.5 0.1
VQ-SA(32, espace)
92.3 = 1.4
88.1
66.0 = 2.4
17.7 1.6
1.8 0.2
VQ-SA(64, espace)
94.2 0.6
90.3
69.2
19.1 - 2.2
1.9
VQ-SA(32)
95.9 = 0.2
92.1
70.4 1.6
18.9 1.4
1.9 0.2
VQ-SA(64)
96.1 0.1
92.1 0.3
69.6 1.1
18.5
1.8 0.1
6 RELATED WORK
Disentanglement There is no conventional definition of disentangled representation. The intuitive idea behind the term "disentanglement" is the about representations that capture and separate generative factors in the data (Bengio et al. (2013): Higgins et al.) (2018).. Generative factors of the dataset are set of independent factors that can describe any sample from the dataset. Metries of disentanglement such as Beta-VAE (Higgins et al. (20176), FactorVAE (Kim & Mnih (208)), ),
DCI (Eastwood & Williams (0188)), and SAP (Kumar al. (2018b)) were designed reflect the definition of disentanglenan representations from Bengio et al. (2013).
Most of the recent approaches to obtaining disentangled representations are based on the Variational Autoencoder framework, which consists of an encoder, that maps the data samples into their latent representations, and decoder that maps given representations into source data space. VAE tends to match the distribution of the latent representation of the input data and the standard Gaussian distribution. Thus, each representation reflect the discrete nature of some generative factors. BAt (Higgins al. (2017b); Burgess et al.) (2018) and Factor- VAE (Kim & Mnih (2018)) as modifications of original VAE use additional constraints during training enforce better disentanglement. As an architecturally different approach, InfoGAN (Chen et al. (2016)) maximizes mutual information between a subset of latent variables and the generated samples aid disentanglement.
generated from continuous distribution and may not
Vector quantized representations Using discrete latent representation can be seen as way to gain some advantages in modeling discrete generative factors of data. Assuming that the dimensionality of the discrete factors is known, it is possible to enhance disentanglement by inducing semantic factor biases from known factors (Locatello al. (2019)). Vector Quantized Variational Autoencoder (VQ-VAB) (van den Oord et al. (2017)) is an approach to learn discrete latent variables by splitting encoded inputs into fixed-size embeddings and assigning them the nearest vectors from learnable codebook. The prior distribution of discrete representations is modeled by separate autoregressive model. VQ-GAN (Esser et al. (2020)) takes the idea of VQ-VAE and extends it through the use of transformers. discriminator and perceptual loss, achieving remarkable image generation results.
One of the crucial challenges of Vector Quantized models is achieving high codebook usage, since typically, there is noticable part of vectors that are rarely used. Yu et al. (2021) claims that mapping all latent variables on a sphere by normalization and using oiwee-iimensional lookup space can significantly improve the codebook usage with training stability and reconstruction quality. (Shin et al.
8

---

Under review as a conference paper at ICLR 2023
(2021) investigates how forcing latent variables to be orthogonal brings translation equivariance in the quantized space with increased performance in image-to-text and text-to-image generation tasks. In contrast to the improvements described in Yu al. (202I), this technique reduces the number of latent vectors used.
Set prediction Neural networks for sets are applied in various fields: object detection (Carion al. (2020)) point cloud generation (Achlioptas et al. (2017)) molecule generation (Simonovsky & Komodakis (2 and speaker diarization (Fujita et al. (2019). Although the set structure is natural for use in many data, traditional deep learning models are not inherently suitable for representing sets. There are some approaches that are built to reflect the unordered nature of sets: the Deep Set Prediction Network (DSPN) (Zhang et al. (2099) reflects permutation symmetry by running an inner gradient descent loop that changes set encode more similarly to the input; iDSPN Zhang et al.) (2022) is an improved version of DSPN with approximate implicit differentiation that provides better optimizations with faster convergence and state-of-the-ar performance on the CLEVR dataset. Slot Attention (Locatello et al. (2020)) and TSPN (Kosiorek et al. (2020) use set-equivariant self-attention layers to represent the structure of sets.
Object discovery The discovery of objects in a scene an unsupervised manner is a crucial aspect of representation learning and a desirable part of binding problem solution. Works as MONET (Burgess et al. (2019),, IODINE (Greff et al. (019)), and GENESIS (Engeleke al. (2200) are built upon the Variational Autoencoder (VAE) framework (Kingma & Welling (2014); Rezende ict al. (2014). MONET uses the attention network that generates masks and conditions VAE on these masks. IODINE models an image as spatial Gaussian mixture model to jointly infer the object representation and segmentation. Compared to MONET and IODINE, GENESIS explicitly models dependencies between scene components that allow the sampling of novel scenes. MONET, IODINE, and GENESIS use multiple steps to encode and decode an image, while Slot Attention (and its sequential extension for video (Kipf et al. (202T) ) uses one step but performs procedure inside this step. The useful property of Slot Attention is that it produces the set of output vectors (slots) with permutation symmetry. Slots group input information and could be used in unsupervised tasks (object discovery) and supervised tasks (set prediction). GENESIS-v2 (Engelcke al. (221)) ). a development of the GENESIS model, uses attention masks similarly to Locatello et al. 2020). Another approach utilizes Generative Adversarial Networks (GAN) (Goodfellow et al. (2014)) by structuring the generative process (van Steenkiste et al. (2018)). That enables learning about individual objects without supervision. Our work is closely related to Slot Attention, but we move further and improve slot representation by quantization.
iterative
7 CONCLUSION AND DISCUSSION
In this paper, we propose the VQ-SA model that utilizes the idea of slot object representation and models non-overlapping sets of low-dimensional discrete variables, sampling one vector from each to obtain the latent representation of the object. Such representation allows one not only to separate scene distributed features into object representations but also separate distributed features of the objects themselves into representations of their properties. Our model achieves state-of-the -ffth--r---- results among the class of object-centric methods on set prediction task. We also demonstrate that through manipulation with learned discrete representations, we can generate scenes with objects with the given property. To qualitatively evaluate the disentanglement in the object discovery task we propose DQCF-micro and DQCF-mace methods.
The important feature of our model is that we use the number of latent discrete spaces equal to the number of generative factors in the data. Thus, our model cannot be applied out of the box to data with a different number of generative factors. As with most object-oriented models, we show results on relatively simple data. Modifying from simulators, e.g., A12Thor (Kolve et al. (2017)), is an interesting and challenging task and will be considered in future works.
model for more complex scenes both from real life and

---

Under review as a conference paper at ICLR 2023
REFERENCES
Panos Achlioptas, Olga Diamanti, Toannis Mitliagkas, and Leonidas J. Guibas. Representation learning and adversarial generation of 3d point clouds. ArXiv, abs/1707.02392. 2017.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35:1798-1828 2013.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray. Catherine Olsson, Jakub W. Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, llya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. 2019.
ArXiv, abs/1912.066m
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nicholas Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling beta-vae. arXivv Machine Learning, 2018.
Christopher Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew M. Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. ArXiv, abs/1901. 1390, 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ArXiv, abs/2005. 12872, 2020.
Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Biandee and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL ht http:////oooceeeeee ing. ng.. paper 2018/ fil http///eeeleeeeddddeeee 0 64 3577992239999922222222
neur ps.pss. cc/ pdf
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Pro- ceedings of the 30th International Conference Neural Information Processing Systems, NIPS' 16, pp. 2180-2188, 18, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781 1781511 1118888119.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLLP), pp. 1724-1735 1734, Doha, Qatar, October 2014, Association for Computational Linguistics. doi:
URL
org/D14-1179
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018a. / /ttp::///peeeeeeeieee net met/oorum?dd-By-110
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In ICLR, 2018b.
Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W.Taylor. Tell, draw, and repeat: Generating and modifying images based on continual linguistic instruction. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10303-10311, 2019.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. ArXiv, abs/1907. 130522 2020.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. In NeurIPS, 2021.
10

---

Under review as a conference paper at ICLR 2023
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. 2020.
Yusuke Fujita, Naoyuki Kanda, Shota Horiguchi, Kenji Nagamatsu, and Shinji Watanabe. End-to-end neural speaker diarization with permutation-fre objectives. In INTERSPEECH, 2019.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In 7. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (cds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URLhttps: httttpsss//ppoeeenggs....t
cC
Klaus Greff, Raphael Lopez Kaufman, Rishabh Kabra, Nicholas Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew M. Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. ICML, 2019.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. ArXiv, abs/2012.0522 2020.
Oliver Groth, Fabian B. Fuchs, Ingmar Posner, and Andrea Vedaldi. Shapestacks: Learning vision- based physical intuition for generalised object stacking. ArXiv, (bs//804.08008 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations,
2017a. URL /RLL:ttps///ppeeeeiiii net http:///www.eeet/oomoeeeoo---e/
Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017b.
Irina Higgins, David Amos, David Pfau, Sébastien Racanière, Loic Matthey, Danilo Jimenez Rezende, and Alexander Lerchner. Towards definition of disentangled representations. ArXiv, abs/1812.022300, 2018
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, 2016. URL
abs/ 01144
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. CVPR, 2017.
Rishabh Kabra, Chris Burgess, Loie Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm
Reynolds, and Alexander Lerchner. Multi-object datasets. http://githu..comdepnnnn//mmm/ object-datasets/ 2019.
Ramtin Keramati, Jay Whang, Patrick Cho, and Emma Brunskill. Fast exploration with simplified models and approximately optimistic planning in model based reinforcement learning. arXiv: Artificial Intelligence, 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2649-2658. PMLR, 10-15 Jul 2018. URL https: :ttps:///prcceedings ml www.sssw....p...eeeeememem wwwress/v8//iine 8b. html
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015. h // //arxiv... org/ abs /1//211.6666 6980
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. CoRR, abs/1312.61 2014.
11

---

Under review as a conference paper at ICLR 2023
Thomas Kipf, Gamaleldin F. Elsayed. Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. ArXiv, abs/2111.. 1.22944, 2021.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Enviromment for Visual AI. arXiv, 2017.
Adam R. Kosiorek, Hyunjik Kim, and Danilo Jimenez Rezende. Conditional set generation with transformers. ArXiv, 168420 2020.
H. W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83 97, 1955. doi: 7/onl won.ineeeiban wiwarrrrrnrrrr...mmmmm wi wileyy-.. com/ Booi/bbs/ie 1000//0000.0002/.30300022
URL https
Tejas D. Kulkarni, Ankush Gupta, Catalin lonescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. ArXiv, abs/1906. 1883, 2019.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In 6th International Conference on Learning Represen- tations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 201 8a. httttsss::////tt /ttttps:://ppeteeeeeww net. Boborum?d==11k77CG
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. ArXiv, 0tbs//711.00844 2018b.
Brenden M. Lake, Tomer David Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40, 2016.
Nanbo Li, Cian Eastwood, and Robert Fisher. Learning object-centric representations of multi-object scenes from multiple views. Advances in Neural Information Processing Systems, 33:5656-5666, 2020.
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled represen- tations. ArXiv, abs/18 1. 2019.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. arXiv preprint arXiv:2006.15055 2020.
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention- neural machine translation. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pp. 1412- 1421, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10.18653/11/D15-1111 URL URLnttts:///llattooo
org/D15-1165
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. CoRR, abs/161 1,00012,, 2016. URL httt:http://aaa/
org/ 611 00712
Shoya Matsumori, Kosuke Shingyouchi, Yukikoko Abe, Yosuke Fukuchi, Komei Sugiura, and Michita Imai. Unified questioner transformer for descriptive question generation in goal-oriented visual dialogue. 202 IEEF/CVF International Conference on Computer Vision (ICCV), pp. 1878-1887, 2021.
Toki Migimatsu and Jeannette Bohg. Object-centric task and motion planning in dynamic environ- ments. IEEE Roboties and Automation Letters, 5:844-851 2020.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic back- propagation and variational inference in deep latent gaussian models. ArXiv, abs/1401.4088 2014.
12

---

Under review as a conference paper at ICLR 2023
Won Young Shin, Gyubok Lee, Jiyoung Lee, Joonseok Lee, and E. Choi. cuunslaaioo-equivaiian image quantizer for bi-directional image-text generation. ArXiv, abs/21 12.00384, 2021.
Martin Simonovsky and Nikos Komodakis, Graphvae: Towards generation of small graphs using variational autoencoders. In ICANN, 2018.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. ArXiv, abs/2110.114000 2021.
Chen Sun, Per Karlsson, Jiajun Wu, Joshua B. Tenenbaum, and Kevin P. Murphy. Stochastic prediction of multi-agent interactions from partial observations. ArXiv, Abss1902.09641.. 2019.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing- In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 6309-6318, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 978151086096..
Sjoerd van Steenkiste, Karol Kurach, and Sylvain Gelly. A case for object compositionality in deep generative models of images. ArXiv, abs/18 Abs//810.1000 10118000000,, 2018.
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P. Burgess, and Alexander Lerchner. Cobra: Data eaft-effficeet model- based rl through unsupervised object discovery and curiosity-driven exploration. ArXiv, abs/1905, abs/1905.09275, 2019a.
Nicholas Watters, Loic Matthey, Christopher Burgess, and Alexander Lerchner. Spatial broad- cast decoder: A simple architecture for learning disentangled representations in vaes. ArXiv, Abs//990.070111 2019b.
Jianwei Yang. Jiayuan Mao, Jiajun Wu, Devi Parikh, David Cox, Joshua Tenenbaum, and Chuang Gan. Object-centric diagnosis of visual reasoning. ArXiv, abs/2012. 11587, 2020a.
Yanchao Yang, Yutong Chen, and Stefano Soatto. Learning manipulate individual objects in an image. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6557-6566, 2020b.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. ArXiv, abs/1910.01442, 2020.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. ArXiv, 1bs/2110.04627, 2021.
Yan Zhang, Jonathon Hare, and Adam Prugel-Bennett. Deep set prediction networks. In H. Wallach, H. Larochelle, A. Beyyellimer, F. d'Alché-Bue, E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems. volume 32. Curran Asso- ciates, Inc., 2019. URL htttps://proceeeinggn .etpeisseerrppp.pp pwtpurips.ccpppper.eee https//aaapppp/1////1e.... pdf
Yan Zhang, David W Zhang, Simon Lacoste-Julien, Gertjan J. Burghouts, and Cees G. M. Snoek. Multiset-equivariant set prediction with approximate implicit differentiation. International Conference Learning Representations, 2022. URL RtLptt::: http///ppneeweeeeeeeeeee net fer.euuuuwee//ttm forum?
Id=5K7RRq2E:0S
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
Table 3: Performance comparison the Slot Attention model (Locatello et al. (2020), ) our proposed VQ-SA SA model and VQ-SA model trained for 150K steps with (VQ-SA-p 150K) and without (VQ-SA 1 150K) pretrained Slot Attention Module in the supervised set prediction task.
Model
AP (%)
API (%)
APo.5 (%)
AP (%)
1 (%)
Slot Attention
94.3
1.1
86.7
56.0
3.6
10.8 1.7
0.9 0.2
VQ-SA (ours)
96.1 + 0.4
91.2 + 0.5
71.8 = 2.3
22.2 + 2.1
2.4 + 0.2
VQ-SA 150K
87.0
82.2
63.3
19.7
2.1
VQ-SA-p 150K
96.0
91.7
78.3
30.4
3.8
Validation MSE
End-to-ond
0.02
0.015
0.005
trsine/globbl. step
20k
40k
50k
B0k
100k
Figure 6: Learning curve for the object discovery task model with end-to-en learning.
Validation
pretrained beta
pretrained
pretrained
0.02
0.015
0.01
trainer/global step
20k
40k
60k
100k
Figure 7: Learning curves for different 3 values for model the object discovery task with pretrained slot attention weights initialization.
14

---

Under review as a conference paper at ICLR 2023
Table 4: Hyperparameters of the VQ-SA model for the set prediction and the object discovery tasks.
(a) Shared hyperparameters.
Name
Value
(b) Hyperparameters for the set prediction task.
(c) Hyperparameters for the
AdamW: 91
0.9
object discovery task.
Adam W: 32
0.999
Name
Value
Name
Value
AdamW:
le-08
Adam W: learning
0.999
Batch size
512
Batch size
64
Adam W: weight decay
0.01
Train steps
150K
Train steps
500K
OneCycleLR: pet. start
0.05
Slot dim.
64
Table 5: Architecture of the CNN encoder for the experiments on CLEVR dataset for set property prediction and object discovery tasks. Set prediction uses stride of in the layers with while object discovery model uses stride of I in these layers.
Layer
Channels
Activation
Params
Conv2D
64
ReLU
stride:
Conv2D 5 5
64
ReLU
stride: 1/2*
Conv2D
64
ReLU
stride: 1/2*
Conv2D 5 x
64
ReLU
stride:
Position Embedding
absolute
Flatten
dims: w, h
LayerNorm
Linear
64
ReLU
Linear
64
Table 6: Spatial broadcast decoder for object discovery task for CLEVR dataset
Layer
Channels/Size
Activation
Params
Spatial Broadcast
8 x
Position Embedding
absolute
Conv SonnTrasspos22 x 5
64
ReLU
stride: 2
Conv SoonTraanspos2D> x 5
64
ReLU
stride: 2
Cony DonnTrranspse2D 5
64
ReLU
stride: 2
Conv Transpose2D x 5
64
ReLU
stride: 2
Conv TonnTranspose2D 5
64
ReLU
stride: I
Conv Sonn/raanspos2D> x 3
4
stride: 1
Split Channels
RGB (3), mask (1) Softmax on masks (slots dim)
Combine components
Table 7: Spatial broadcast decoder for object discovery task for Tetrominoes dataset
Layer
Channels/Size
Activation
Params
Spatial Broadcast
35 35
Position Embedding
absolute
Conv SonnTrasppose2D 3
ReLU
stride: 1
Conv Tronsraaspose2D 3 x 3
32
ReLUU
stride: I
Conv Tronsraanppsee2D 3
ReL.U
stride:
Conv Transpose2D 3 x
stride: I
Split Channels
RGB (3), mask (1)
Softmax on masks (slots dim)
Combine components
15

---

Under review as a conference paper at ICLR 2023
Image
Figure 8: Visualization of per-slot reconstructions and alpha masks in the object discovery training setting for Tetrominoes dataset.
16


Reviews:
[{"review_id": "ijJ4v9Vyxqe", "reviewer": "Reviewer_1Nh6", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a version of Slot Attention with vector-quantized representations, focussing on object- as well as feature-level disentanglement. The paper also proposes a pair of techniques (DQCF-micro and DQCF-macro) to look at disentanglement when generative factors are encoded as vectors.", "summary_of_the_review": "The paper is certainly not ready for publication. The results need fleshing out and the writing needs a few more iterations.\n\nHere are some questions to help the paper in future iterations:\n1. __End-to-end__? Figure 1 shows the “three stages” of the VQ-SA pipeline. Could you confirm whether the stages are run one after the other (freezing weights as you go), or if the pipeline is run end to end?\n2. __Prior work__: This isn’t the first work that uses discrete representations with Slot Attention. How does this work relate to Singh et al. 2021 where they also use a discrete VAE?\n3. __Traversals__: Figure 2: what exactly do you mean by manipulating a particular attribute in your model? Since each slot representation $S^D$ is a weighted sum of embeddings (concatenated across latent spaces), I don’t understand how you can manipulate the weighted sum. Do you replace the weights $y^i$ with a one-hot lookup?\n4. __KL term__: I assume Figure 4 corresponds to the model trained “without KL-term” (shown in Figure 5). Could you also share DQCF-macro results for the model which achieves better disentanglement (“with KL-term”)? What is the effect of boosting the weight of the KL loss?\n5. __Multiple latent spaces__: In Table 2, it appears that using a single latent space with 32 embeddings yields better set prediction performance than 4 different latent spaces. Why is that? Is disentanglement the only rationale of using multiple latent spaces?\n", "strength_and_weaknesses": "Pros:\n- The set prediction results show a marginal improvement over Slot Attention.\n- Code is available in the supplementary materials.\n\nCons:\n- The results are pretty basic on CLEVR alone. Only a single example image is used throughout the paper (for the object discovery and disentanglement results). There are no additional images in the Supplementary material.\n- The proposed evaluation for disentanglement (DQCF-micro and DQCF-macro) is purely qualitative and involves inspecting histograms (for DQCF-mico, this could be over several latent spaces). Surely deviation from uniform could be quantified as a metric?\n- The paper is not clear on several technical aspects (see questions below). It reads like an early draft at the moment.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "bRoSxi7-3M", "reviewer": "Reviewer_Eih1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper combines Slot Attention with VQ-VAE (although in a stepwise training fashion) and shows results on CLEVR and on set prediction tasks.\n", "summary_of_the_review": "Overall, I think this paper tries to do something interesting, but it makes several arbitrary and limiting choices, which seriously hinder the usefulness of the model, for example the decision to train SlotAttention first, then the VQs second. It also assumes quite a lot of knowledge about the task in various places, which reduces its use as a real unsupervised method. Finally, the current presentation is not as clear as it could be and I found it hard to follow. \n\nHence in this current form I do not believe this work reaches the standard expected by ICLR.\n", "strength_and_weaknesses": "1. The presentation of the model as 2 independent tasks and entirely different methods is not very helpful. I would have expected the model to be trained on image reconstruction only, and demonstrate that one can make use of the discrete representations to perform the set prediction post-hoc.\n   1. Figure 1 shows the setup as one single model diagram, even though they are done entirely separately and in different training stages.\n   2. Section 1 and 2 are not very clear in presenting this choice.\n   3. The Set Prediction task is not presented well enough. What are these thresholds?\n   4. Overall I found the paper hard to follow, even though the idea is rather simple.\n2. Combining VQ-VAE with SlotAttention is a good idea, and I think some of the choices they made in how to do so makes sense, however this should be done end to end and with less assumptions to be really impactful.\n   1. What happens when you learn everything together? In particular if this was made to work when trying to do reconstruction, this would be a valuable piece of research.\n   2. The number of categorical distributions being fixed to the number of generative factors, and the number of categories to be equal to the number of values per factor is too much supervision. \n      1. What happens if you use more?\n   3. The fact that continuous variables are handled independently and entirely differently from categorical ones is too much supervision.\n      1. What happens if you just use VQ for everything? Obviously you would need to use many for this to make sense.\n3. The proposed disentanglement metrics were confusing and I would have assumed that computing the discrete mutual information would have directly done the same?\n   1. Can you comment on how these differ?\n   2. Figure 3 and 4 were not very clear to me, and feel like they belong to the Appendix? A table could replace Figure 4 and be more informative.\n4. Nits:\n   1. The math in section 2.3 uses cross products instead of dot products. This should be changed.\n   2. I could not see an Appendix with details of the architecture and training setup. There are not enough details about the model in the main text to reproduce this work.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EagY2TSmmj", "reviewer": "Reviewer_vXo7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposed to combine the idea of slot-attention and vector quantization to learn discrete object-centric representation of visual scenes. The proposed model utilizes slot-attention to decompose the image into a set of object-centric slots, and then transform each inferred slot into a few concatenated vectors by vector quantization from a learned codebook. Both set prediction and object discovery tasks are evaluated to show the effectiveness of the proposed model, while particular efforts are made to show the ability to discover disentangled subspace in the latent space partitioned by the discrete quantization.", "summary_of_the_review": "This work proposes an approach for learning discrete object-centric representation by combining existing ideas, I believe more thorough evaluation and better clarity are needed in this reviewing cycle.", "strength_and_weaknesses": "Strengths:\n- The proposed work seems to be a reasonable extension of the slot-attention model, and the empirical results on set prediction show some extent of improvements over the baseline.\n\nWeaknesses:\n- The writing is generally not as clear as it could be. The most important part of the paper should be Sec. 2.3, however, it's unnecessarily hard to follow, and Figure 1 is not very helpful here. The notations of introduced variables are not introduced in a more natural way, for example, the $e_j^i$ term is not described in a clear way in its first appearance.\n- In Sec 2.5, the authors propose to use beta-VAE style loss to encourage the latent space to be disentangled. However, there's no mention and discussion on the weighting effect, i.e. $\\beta>1$, which plays a crucial role.\n- In object discovery tasks, only qualitative results are provided, and no quantitative evaluation is included, which is not convincing enough, especially when only CLEVR dataset is considered.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-ktNXl2JP_", "reviewer": "Reviewer_88V8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use a latent space to quantize object-centric representations learned with slot attention for better disentanglement. The authors adopted a similar idea of VQ-VAE and initialized learnable codebooks for each slot representation generated from slot attention to obtain the vector-quantized representation of slot representations.  With experiments, the authors show that their methods can outperform slot attention on set prediction task in CLEVR.", "summary_of_the_review": "Given that the current experimental results can not fully justify the claims of the authors, I'm recommending rejection and suggest the authors design better tasks/settings for illustrating the effectiveness of the proposed VQ-SA, not only from a qualitative perspective on the analysis of latent spaces. The authors might also want to step further from CLEVR to more complex datasets (both real and synthetic) for evaluating the learned representations.", "strength_and_weaknesses": "[+] The idea of vector quantizing slot representations for better disentangling representations is new.\n\n[+] The resulting model does show better performance compared with the vanilla slot attention model.\n\n[-] The major concern of this paper lies in the justification of claims in this paper and experiments. The authors are motivated to perform vector quantization to better learn disentangled object-centric representations. However, they only tested the resulting model on the CLEVR dataset on the set-prediction task which shows the limited significance of the design. The disentangling mechanism (KL) is also not quantitatively evaluated in ablative studies. This makes the overall claims of the proposed VQ-SA not fully addressed and justified.\n\n[-] The authors might want to elaborate more on disentangled representations as it is a critical factor in this paper and show their significance. The current sec.4 does not fully show the uniqueness of learning discrete spaces and does not make a direct comparison of disentanglement with ones that do not leverage a quantization module. The description of DQCF-micro and DQCF-macro is also a bit hard to follow in the text, especially given notations are not properly defined before the illustrations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject