Neural Language of Thought Models
Authors: Yi-Fu Wu
Minseung Lee
Sungjin Ahn
Venue: ICLR 2024 poster (2024)
Keywords: unsupervised object-centric learning
discrete representations
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce a model that can obtain semantic discrete representations of a visual scene.

Abstract: The Language of Thought Hypothesis suggests that human cognition operates on a structured, language-like system of mental representations. While neural language models can naturally benefit from the compositional structure inherently and explicitly expressed in language data, learning such representations from non-linguistic general observations, like images, remains a challenge. In this work, we introduce the Neural Language of Thought Model (NLoTM), a novel approach for unsupervised learning of LoTH-inspired representation and generation. NLoTM comprises two key components: (1) the Semantic Vector-Quantized Variational Autoencoder, which learns hierarchical, composable discrete representations aligned with objects and their properties, and (2) the Autoregressive LoT Prior, an autoregressive transformer that learns to generate semantic concept tokens compositionally, capturing the underlying data distribution. We evaluate NLoTM on several 2D and 3D image datasets, demonstrating superior performance in downstream tasks, out-of-distribution generalization, and image generation quality compared to patch-based VQ-VAE and continuous object-centric representations. Our work presents a significant step towards creating neural networks exhibiting more human-like understanding by developing LoT-like representations and offers insights into the intersection of cognitive science and machine learning.

Published as a conference paper at ICLR 2024
NEURAL LANGUAGE OF THOUGHT MODELS
Yi-Fu Wu', Minseung Lee", Sungjin Ahn?* Rutgers University "KAIST
ABSTRACT
The Language of Thought Hypothesis suggests that human cognition operates structured, language-like system of mental representations. While neural language models can naturally benefit from the compositional structure inherently and explicitly expressed in language data, learning such representations from non-linguistic general observations, like images, remains a challenge. In this work, introduce the Neural Language of Thought Model (NLoTM), a novel approach for unsupervised learning of LoTH-inspired representation and genera- tion. NLoTM comprises two key components: (1) the Semantic Vector- curtto--uattieed Variational Autoencoder, which learns hierarchical, composable discrete represen- tations aligned with objects and their properties, and (2) the Autoregressive LoT Prior, an autoregressive transformer that learns to generate semantic concept tokens compositionally, capturing the underlying data distribution. We evaluate NLoTM several 2D and 3D image datasets, demonstrating superior performance in generalization, and image generation quality
downstream tasks, compared to patch- based VQ- VQ-VAE and continuous object-centric representations. Our work presents a significant step towards creating neural networks exhibiting more human-like understanding by developing LoTT-like representations and offers insights into the intersection of cognitive science and machine learning.
INTRODUCTION
The Language of Thought Hypothesis (LoTH) (Fodor et al. 1975) suggests that human cognition is based on a structured, language-like system of mental representations, often referred as "Mentalese", Mentalese comprises word-like units that form sentence- -ike structures, which convey meaning. The meaning of these mental "sentences" is systematically determined by the meanings of their constituent "words" and their specific arrangement. From a computational viewpoint, while neural language models (Bengio et al. 2000; Brown et al., 20200 Bommasani et al. 202T) can benefit from the compositional and symbolic structure inherently expressed in the language data they trained on, remains unclear how we can learn such -- structure from non- non-linguistic general observations, such images, videos, and audio signals. The significance of this ability is further highlighted by the fact that infants learn these structures from observing objects and events before they acquire language skills (Spelke 2022).
How can we create neural networks that learn to develop such language of thought representations in unsupervised way? To address this, we outline the following three properties the desired characteristics of a neural language of thought model.
First, when perceiving visual scene, humans do not simply represent it monolithic vector of features. Instead, we view the scene structurally and semantically, recognizing it as a composition of meaningful components such as objects and their attributes, including shape, color, and position (Palmer 1977; Singer 2007: Spelke & Kinzler 2007). Our observation here is that in line with the LoTH, these visual attributes can be likened words, objects sentences, and the scene to paragraph. Recent works, particularly those focused on object-centric representations (Greff et al. 2020), have demonstrated that this structural decomposition facilitates the benefits associated with the LoTH such as relational reasoning (Wu et al. [02 Yoon et al. 2023 Webb et al. [203a b) and out- of-distribution generalization (Dittadi et al. 2022 Yoon et al. 2023) due to increased compositional
generalization.
"Correspondence sungjin.ahn @ kaist.ac.kr

---

Published as a conference paper at ICLR 2024
Table 1: Desiderata for Neural Language of Thought Models and Related Models
VAE
VQ-VAE
Slot Attention
SysBinder
NLoTM (Ours)
Compositionality
Factor
x
Object
Object & Factor
Object & Factor
(Semantic Scene Decompusttion
Symbolic
x
/ (Patch Concept)
x
x
(Semantic Concept)
(Discrete Concept Abstraction)
Pruductivity
(Patch Stichiing
x
(Semantic Composition)
(Probabillstic Compositional Generation)
Moreover, these structured and semantic representations can be categorized and conceptualized,
resulting in symbol-like discrete concept abstraction. Such an ability is critical for organizing and
comprehending the complexity of the environment, e.g., via language, well for implementing
modularity (Andreas et al., 2016) or symbolic reasoning (Lake et al. 2016)) Such discrete representa- tions are also useful to leverage powerful generative models like autoregressive transformers (Vaswani et al. 2017). One of the most popular models for discrete representation learning is VQ- VAE (van den Oord et al. 2017). has been shown be beneficial for image generation (Razavi et al., 2019; Esser et al. 2021) and probability density modeling Van den Oord et al. 2016). However, VQ- (AEV and its
variants, such as dVAE (Ramesh et al.) 2021 Singh 2022a) and VQ-GAN (Esser et al. 202T represent scene grid of small patches, lacking the capability capture the scene's 's holistic
structure and semantics.
Besides, the ability to compositionally and probabilistically generate samples that adhere to the distribution of prior beliefs, constructed from observation data, is crucial for endowing Al with the capabilities to imagine and simulate. These abilities are essential for tasks such planning (Mattar & Lengyel 2022; Hafner et al. (2199) and reasoning. However, this ability, related to probabilistic Language of Thought (PLoTT) (Goodman et al. 2015) and productivity, is supported by only certain class of representation learning models. While models like Slot Attention (Locatello et al.) (0200) and SysBinder (Singh al. (203) offer structured, object-centric compositional representations, in their original form it is unclear how they support density-based sampling. In contrast, VAE-based models support this ability to sample from a prior distribution. However, they either do not provide object-centric structures or are limited to batch-based discrete abstractions (AE-.
In this work, we present the Neural Language of Thought Model (NLoTM), the first model that satisfies the aforementioned criteria summarized in Table NL.oTM comprises two novel compo- nents: the Semantic Vector- Qurito--uantzzedd (SVQ) Variational Autoencoder and the Autoregressive LoT Prior (ALP). SVQ achieves discrete semantic decomposition of scene by learning hierarchical, composable factors that closely align with the objects and the properties of objects in visual scenes, similar to the role of words and sentences in LoTH. ALP, analogous to language models trained on text tokens, is an autoregressive transformer trained to learn a probabilistic generation of semantic concept tokens. However, unlike VQ-VAE which stitches a grid of patches, ALP composes semantic concepts such as objects and their attributes to represent a scene.
In our experiments, we demonstrate the following practical benefits of our method. First, find that for multi-object scenes, NLoTT is able to model the prior distribution better than patch- based methods, as measured by the quality of the samples generated. Second, we find that NLoTM represen- tations outperform patch-based VQ representations on downstream tasks that require knowledge of the different properties of the objects in the scene. We also find evidence that NLoTM representations can generalize better to out-of- of-diitribution tasks compared to patch- VQ representations and SysBinder continuous representations. Lastly, we show that despite introducing a discrete bottleneck. NLoTM can work on the challenging CLEVRTex (Karazija et al., 2021) dataset, one of the most complex datasets used in recent unsupervised object-centric representation learning models.
Our contributions are as follows: First, we introduce NLoTM, the first neural network model implementing the LoTH for unstructured observations. Second, we propose the SVQ model to obtain object-centric semantic neural discrete representations. Third, we propose the ALP to model the probabilistic prior capable of capturing the underlying data distribution and compositional generation of new samples. Lastly, we evaluate model on several 2D and 3D datasets including the challenging CLEVRTex dataset, showing superior downstream task performance and image generation quality.
2

---

Published as a conference paper at ICLR 2024
2 BACKGROUND
2.1
VECTOR-QUANTIZER VARIATIONAL AUTOENCODER (VQ-VAE)
The VQ-VAE (van den Oord et al. 2017) is model that learns to compress high-dimensional data into a discretized latent space. The latent space maintained by a codebook of prototype vectors e E I R xd where K is the size of the codebook and d is the dimensionality of each prototype vector. An input is first passed through encoder E(x) to obtain latents ze E IR nearest-neighbor lookup between 2. and each of the prototype vectors in the codebook yields quantized representation
zn = Quantize(Ze) = ej where & = arg min,/ | -eyll2. The decoder then uses Z, to reconstruct
the input: = D(z,). The model is trained with the following loss:
L
x
1 +
||sg[zl] 2, +B lIze sg |2, 1,,111
Reconstruction
Codebook
Commitment
The first term is a reconstruction loss and is used to train the encoder and decoder. A straight-through estimator (Bengio et al. 2013) is used to estimate the gradients through the quantization step by copying the gradients from Z, to Ze. The second term is the codebook loss which encourages the prototype vectors in the codebook to be close to the output of the encoder. The third term, scaled by a constant hyperparameter B, is the commitment loss and helps to stabilize the training by encouraging the output of the encoder to not deviate too much from the chosen prototype vectors. Instead of the codebook loss, we use exponential moving average (EMA) updates the codebook, which found to speed up training in our experiments (Razavi et al. 2019: Dhariwal et al.
2020 Yan et al. 2021).
When VQ- VA-E are applied to images x R H the encoder E(x) is typically implemented as a convolution encoder, outputting a feature map of latents Ze E IR "R, XW, xd This means that each latent corresponds to a local area represented by a convolutional feature cell and thus can only capture information in a local receptive field (Figure ( However, images typically contain multiple objects, and the discrete factors underlying visual scenes typically correspond to different properties of the objects in the scene, such shape, color, type, and so on. The local patches from convolutional feature maps are inadequate to capture this rich structure.
2.2 OBJECT- CENTRIC REPRESENTATIONS
The goal of unsupervised object-centric representation learning is to decompose scene into set of representations each capturing a different object the scene. It shown that this structural decomposition, matching to the true factor structure of the world, facilitates some high-level cognition abilities such as relational reasoning (Wu et al. 2021 Yoon et al., 2023 Webb et al. 2023a b) and out-of-distribution generalization (Dittadi et al. 2022] Yoon et al. (023)) We build on top of Slot Attention (Locatello et al. 2020), a spatial attention based ceject-centric representation method.
Given image R. XC slot attention learns set of slots, {s1 ................................ SN) }. where s,, Rd and N is the total number of slots. An encoder is applied to x and, after adding a positional encoding, the result is flattened to an L-length input feature vector F R4xd=_ Then, an iterative attention mechanism is used to spatially group the input features F to the slot representations. s. First, the slots are randomly initialized from a Gaussian distribution with learned parameters. Then, in each iteration, the slots are used as queries in an inverted version of dot-product attention (Tsai et al. 2020) with the input features F as the keys and values. Instead of normalizing over the keys as is done in traditional dot-pne -ro-prrd attention, normalization is done over the queries (ie. slots). Additionally, a weighted mean is used to aggregate the values instead of the normal weighted sum, which is shown to stabilize training. The result is then used to update the slots with a per-slot GRU (Chung et al.) 2014) followed by a per-slot residual MI.P, both with shared parameters across the slots.
The slot representations are then used in a decoder to reconstruct the image and the entire model is trained with an image reconstruction loss. The original formulation of slot attention used a spatial broadcast decoder (Watters et al. 2019b) to create masked images per slot which are then combined to form a final reconstructed image. Recently, (Singh et al., 2022a) proposed using a transformer decoder to reconstruct the image while attending to the slots with cross attention. This method was shown to scale to more complex scenes than the spatial broadcast decoder (Singh et al. 2022b) and is what we choose to use in our model.
3

---

Published as a conference paper at ICLR 2024
Color
Latent
Latent
Latent
VQ-VAE
Quantized Slots
SVQ
Codebook
Codebook
Codebook
Image
Image
Image
(a)
(b)
(c)
Figure Comparison between VQ-VAE, Quantized Slots, SVQ. (a) VQ-VAE quantizes scene at local patch level and may not capture the semantic structure of the scene. (b) Quantized Slots (QS) would quantize the scene at the slot level but require separate code for every possible configuration of an object. (c) SVQ quantizes the block level, representing each factor (such as color shape) code. In this example, to represent all possible object configurations, SVQ requires only codebook entries at the block level while QS requires
3 NEURAL LANGUAGE OF THOUGHT MODEL
3.1 SEMANTIC VECTOR QUANTIZATION
Given slot attention encoder that obtain set of representations of the objects in scene, one may think of a hypothetical method, applying vector quantization to the slot representation itself to obtain set of semantic discrete representations (Figure Ip). While these representations would indeed correspond to the different objects in scene, this scheme would require one codebook entry per possible object configuration and be insufficient for anything beyond trivially simple scenes.
For example, consider a simple scene containing single object in a fixed position that only varies by color and shape. Assume there are possible colors and s possible shapes for the object. With slot-level quantization, in order to represent all the potential objects, the codebook would require at least e entries. This is because each slot representation is single entangled representation, so each combination of factors needs to be represented by separate code. If, instead, we were able to disentangle the object-level representations into factor-level representations repruttiooos-rrrpeeettttttooon that align with the underlying latent factors of variation of each object-we could describe the potentially large combinatorial space of each object with a much smaller number of discrete factors. In the above example, if we had fully disentangled representation of the color and the shape, could represent all possible scenes with c + s codes (Figure Tee.. See Appendix B.2 for further discussion.
This observation motivates to factor representations that reflect the underlying discrete factors the objects in the scene, and to perform vector quantization these factor representations. Under this scheme, each object representation would be composed of multiple discrete factors, and each factor would have its own codebook that can be shared across objects. The resulting model, the Semantic Vector-Quantiza Variational Autoencoder (SVQ), is depicted in Figure 2a and described below.
design architecture that further disentangles slot representations
To obtain factored representations, we follow an approach motivated by Neural Systematic Binder (SysBinder) (Singh et al.) (023). where binding mechanism is introduced to produce disentangled factors within slot. Specifically, the following modifications we maintain M codebooks C E Mx x each with K discrete prototype vectors of dimension de = d. blocks, each of which will represent one factor. We denote the full set of block representations as E R N xd. Crucially, we replace the slot-level GRUs and residual MLPs with block-level equivalents that have shared parameters across blocks corresponding to the same factor. the end of each slot attention iteration, we apply vector quantization for each block using its corresponding codebook to obtain a set of quantized blocks Zy IR xde For r E [], m E [1.
applied slot attention: First,
M Then, we split each of the N d,-dimensional slot representations into M equally-sized
by
n.
= Cm,k where is = arg min
R. Cm.5
j
4

---

Published as a conference paper at ICLR 2024
Ze
Z
Quantization
RHN
Target
Stoc
Slot update
query
Decode
Autoregressive LoT Prior
S
Image
Reconstruction &
Codebook
(a)
(b)
Figure Overall architecture of NLoTM. (a) The Semantic Vector- Surtto--uattzeedd (SVQ) Variational Autoencoder. We maintain learned codebooks and split each slot into blocks. Afler each Slot Attention iteration, we apply vector quantization to each block representation obtain set discrete codes for cach slot. Each block ends up specializing to different underlying factors of variation for the objects in the seene. (b) The Autoregressive LoT Prior (ALP), We train autoregressive prior the discrete latent codes from SVQ. Sampling from this prior allows generate image object at time, based on their properties.
where z.,. denotes the m-th block in the n-th slot and Cm.& is the k-th prototype vector in the m-th codebook, By sharing the codebook for each block across all of the slots, each block ends up specializing in different underlying factors of the objects the scene, such as color, shape, and position. Thus, these quantized representations are semantic in the sense that they contain factor-level representations mapping to the underlying structure of the scene.
To reconstruct the image, we use the autoregressive transformer decoder described in Section 2.2 and condition on Zy via cross attention. Similar to Singh et al. (2023), we first let the blocks within a slot interact with a single-layer transformer and then add block-level positional encoding before inputting the representations as cross attention the transformer decoder. We train the model with the reconstruction loss, the VQ-VAE commitment loss, and update the codebooks with EMA updates. To prevent codebook collapse, we also incorporate random restarts for the embeddings, similar previous work (Dhariwal et al., (2020). To achieve this, keep a count of the usage of each code in the codebooks and randomly reset it to be near one of the encoder outputs of the current batch if its usage falls below threshold.
3.2 AUTOREGRESSIVE LANGUAGE OF THOUGHT PRIOR
Given these semantic discrete codes representing the different objects in the scene, we can now freeze the SVQ and train prior p(z,,) over these codes obtain a generative model of the underlying data that captures the structure and semantics of the data, analogous to language models that are trained over tokens. We can sample from this prior, called Autoregressive LoT Prior (ALP), to obtain codes for new scenes and use these codes in the SVQ decoder to generate new images. Compared patch-based VQ methods that generate patch tokens corresponding to a local region of an image, ALP generates representation of scene one object at time, based on their properties (Figure 2p)
We implement the prior using a simple autoregressive transformer decoder. First, we flatten Z, along the slot and block dimensions to vector with dimensions NM x dg. We then apply positional encoding across all slots and blocks and input the resulting vector transformer decoder with an objective of predicting the discrete code of the next block. Although slot attention does not guarantec any specific ordering of the slots, the blocks within the slots are arranged predefined order. Therefore, the positional encoding is important in providing information about the ordering of the blocks as well as which block belongs to which slot.
Note that generating the latents of one image requires sampling N blocks, but does not depend on the size of the image. This is different than VQ- VAE, which scales with the size of the feature map and may become expensive for high-resolution images.
5

---

Published as a conference paper at ICLR 2024
4 RELATED WORK
Neural Discrete Representation Learning. Our work builds on top of neural discrete representation learning, which has played a pivotal role in the advancement of generative models for images in recent years (van den Oord et al., 2017 Razavi et al.) 2019. Ramesh et al. [02T]] Esser et al. 202T
Yu et al. 2022). These methods typically follow a two-stage approach. First, an image is encoded into a CNN feature map, which is then tokenized using vector quantization (Gray 1984) into a set of
discrete latent variables. In the second stage, a powerful autoregressive prior is then trained to model the distribution of these discrete tokens, allowing for sampling new images from this distribution. Our model also follows this two-stage approach, except our latents correspond to the properties of objects instead of cells in CNN feature map.
Unsupervised Object-Centric Learning. Recent unsupervised object-centric learning methods have been shown to decompose image or video into set of latents, each representing an object in the
scene (Burgess et al. 2019 Greff et al.
2019
Anciukevicius et al. 2020;
Locatello et al.
2020
Greff al.]
2017] Engelcke et al. 22200]
2022
von Kügelgen al., 2020: Du et al.
2021
Kabra
ct al.
2021 Zhang et al. [022]] Eslami et al.
2016; Lin et al. |020b] Jiang & Ahn
2020; Chen
al. 202 Deng al. [022 Lin et al. 2020b a. Singh al. [023]] Kipf al. 2022 Singh et al.
2022b
Gopalakrishnan et al. 2022, Seitzer et al. 2022 Hénaff et al.) 2022 Wang et al. [203a
Wu et al., 2021] Wen et al. 2022] Zoran et al. 2021). While most of these methods result a
distributed representation per object, there have been several attempts at learning more structured
or disentangled representations, such as those methods that decompose the latents into what and
where components (Eslami et al. 2016; Crawford & Pineau 2019b Tiang et al. 2019 Jiang & Ahn
2020 Lin ct al. 2020b a. Chen et al. (021) or those that learn disentangled latents via a VAE (Greff 2019 Zoran et al. 2021). Closely related to our work, recent methods have been designed
ct al.
to learn factor-level disentanglemas
(Singh et al. 2023 Kirilenko et al.) (023). However, these methods still operate with continuous latents instead of discrete tokens and do not support sampling new images. While there are several object-centric learning methods that do support sampling new
images (Engeleke al., 2020; 2022. Jiing & Ahn 2020 Wang et al. 2023b). these also do not use
Wang
semantic discrete latents as we do in our work.
EXPERIMENTS
Datasets. We evaluate our model on two variants of 2D Sprites dataset Watters al. 2019a
Yoon et al. (023) and three variants of the CLEVR dataset (Johnson et al.) 2017D, CLEVR SLLEVR-East Easy, CarvR-Hard, CLEVR-Tex. In the 2D Sprites datasets, objects of varying shapes and colors are placed in a scene. In total, there are 7 possible colors and 12 possible shapes. In each image, one object has a single property that unique from the other objects. All other properties are shared by at least two objects. This structure allows to evaluate if the prior correctly models the dependencies between the properties of the scene. We test versions of this dataset with and without textured backgrounds (Cimpoi al. 2014). CLEVR-Easy, CLEVR-Hard and CLEVR-TeX were previously used in (Singh et al. 2023) and are modified from the original CLEVR (Johnson et al.) 2017) and CLEVR-Tex (Karazija et al. 2021) datasets to have larger objects so properties such as shape and texture are more clearly visible. In CLEVR-Easy, objects may differ by only shape, color, and position. In this dataset, there are 3 possible shapes and possible colors. In Card objects may differ by shape, color, position, size, and material. There are 3 possible shapes, 137 possible colors, and 2 possible materials (shiny or matte). In CLEVR- Tex, there are possible shapes and 58 possible textures for the objects and background.
Baselines. We compare our model with several pateh-based quantization methods: VQ-VAE (van den Oord et al. (017) with PixelCNN (Van den Oord et al., 2016) prior, and dVAE (Ramesh et al. 2021 Singh et al. 2022a) with a transformer decoder prior. For the dVVE baseline, we use the dVAE weights that are trained along with the SVQ. This provides more direct ablation comparing the ALP of NLoTM with the patch-based transformer decoder prior since the dVAE decoder is shared across these models and will not contribute to differences in image quality. We also compare with GENESIS- v2 (Engelcke et al., (202)), a continuous latent object-centric prior that can also generate samples.
with an autoregressive
6

---

Published as a conference paper at ICLR 2024
4-Object NLoTm (ours)
4-Ohject w/ BG
Dataset
VQ-VAE
cVVEE
Figure 3: Generated samples for the 4-object 2D Sprites and 4-object Sprites with background datasets.
5.1 GENERATING SAMPLES WITH THE AUTOREGRESSIVE LoT PRIOR
5.1.1 2D SPRITES
We show the sample generations for the 21 Sprites datasets in Figure 3 and the FID results in Table 2] We additionally calculate generation accuracy by manually inspecting 128 images per model to check if the generated images follow the constraints of the dataset. That is, each image must have exactly one object that has unique property, All other properties in the scene will have at least one duplicate among the other objects.
We see that for the simplest dataset with 3 objects and no background, NLoTM achieves the lowest FID of the models and comparable generation accuracy to dVAE, generating about 75% of the scenes correctly. This setting may be simple enough that dVAFE with a transformer prior can capture the structure of the scene with patch-based discrete latent. As the scene complexity increases with more objects and textured background, NLoTM starts to outperform the baselines in terms of generation accuracy. Inspecting the qualitative results, we see that in the dataset with the background, VQ-VAE and d start generating occasional blurry objects, whereas NLoTM maintains clean- looking objects that match the ground truth dataset. This may be because NLoTM can segment the background into its own slot and factor the texture into a discrete latent, cleanly separating the representation of the objects from the background. The patch-based methods, however, may have a harder time separating the foreground from the background resulting in messier generations. Interestingly, despite the blurry shapes, VQ-VAE achieves the lowest FID score on the 2D Sprites dataset with background. We hypothesize this may be because the model spends more capacity modeling the background correctly instead of the foreground, which may produce a better FID score, but not necessarily better generation accuracy. This is confirmed by the low generation accuracy of the VQ- VAE model this dataset, only generating 19.5% of the scenes correctly.
Table 2: FID and Generation Accuracy the 2D Sprites datasets. For Generation Accuracy, 128 samples were inspected manually to determine they matched the constraints the scene (ie. exactly one unique property among all the shapes). Underlined numbers indicate a minor difference from the best value.
FID 1
Generation Accuracy (in %)
Dataset
VQ-VAR
dVAE
NL.TM (ours)
VQ-VAE
dVAE
NL.TM (ours)
2D Sprites (3 obj)
14.81
7.26
6.61
28.91
75.78
75.00
2D Sprites (4 obj)
26.35
19.15
17.93
21.88
62.50
66.41
2D Sprites w/ bg (4 obj)
58.14
66.08
58.50
19.53
30.47
42.19
5.1.2 CLEVR
In Figure 4 we show sample generations after training the models on the CLEVR-Easy. CLEVR- Hard, and CLEVR-Tex datasets. We report the Frechet Inception Distance (FID) in Table 3 We find that compared to the other models, GENESIS-v2 generates very blurry images and completely fails on CLEVR-Tex, resulting a high FID. While VQ-VAE produces sharper images, several of the generated shapes are malformed or have mixed colors. The dVAE--enerated images look closer to the ground truth dataset, but still have some errors such as overlapping objects (first image) and generating scenes with more objects than seen in the training set (third image). NLoTM has the lowest FID for all of these datasets and the generated images look very close to the ground truth dataset, indicating the usefulness of the ASP for generating these multi-object scenes.
7

---

Published as a conference paper at ICLR 2024
In Appendix B.1
show additional analysis of the learned codebook the CLEVR-Easy dataset.
Dataset
GENESIS
VQ-VAL
dVAE
NLoTM (ours)
CLEVR-Fasy
CLEvR-Hard
Figure 4: Generated samples the CLEVR -ass,, CLEVR-Hard, and CLEVR-Tex Datasets.
Table 3: FID for the various models on the CLEVR datasets.
FID 4
Datasel
GENESIS-v2
VQ-VAE
dVAE
NLoTM (ours)
CLEVR-Easy
115.56
57.06
40.30
32.50
CLEVR- Hard
93.01
73.33
65.89
43.12
CLEVR-Tex
225.08
178.59
112.80
84.52
5.2 DOWNSTREAM TASKS
5.2.1 ODD-ONE-OUT
We first evaluate on a downstream supervised learning task on the 2D Sprites dataset. We modify the dataset by first dividing each image into four quadrants and ensuring exactly one object will be in each quadrant. As in our previous experiments, one object has a single property that is unique from the other objects. The goal of the task We first pretrain the baseline models on a dataset containing all 12 possible shapes and 7 possible colors. Then, we freeze the underlying model and train a downstream model on top of the learned representations with the supervised objective. The downstream model is trained only contains possible shapes and possible colors. We then evaluate on both the in in-distribution (ID) dataset and out-of-distribution (OOD) dataset that consists of the remaining 3 shapes and colors. addition to dVAE and VQ-VAE, we use SysBinder as a baseline for this task, to compare its continuous representation with NLoTM's discrete representation. For the latent representation of NLoTM, we include variants that use the codebook indices in the SVQ (NLoTM Indices) and the codebook prototype vectors in the SVQ (NLoTM Codebook).
identify the quadrant of the odd-one-out object.
dataset that
Ssoos the results of our experiments. Since all models can solve the task when evaluated on the ID dataset, we report the number of steps to reach 98% accuracy on the validation dataset. We find that SysBinder and NLoTM Codebook learn the quickest in the ID setting. For the OOD setting, we find that d'VAE and VQ-VAE fail, not performing better than randomly guessing, showing that the patch-based discrete latent is insufficient for OOD generalization in this task. SysBinder can partially solve the task in the OOD setting, while the NLoTM Codebook seems to be able to solve the task, achieving 99% accuracy. This indicates that the compact latent space offered by the discrete code provides better OOD generalization abilities for this particular task. One possible explanation for this that since this is odd-one-out task, the downstream network needs to do comparisons between the properties of the objects and this may be easier to do with SVQ's codebook vectors that are fixed. SysBinder's continuous latents, the other hand, offer greater variations for the same concept. This
8

---

Published as a conference paper at ICLR 2024
Table 4: Results for the downstream odd-one- out task. Since the models can solve the in- distribution (ID) task, we report the number of steps to 98% ID Accuracy and out-of-distribution (OOD) accuracy.
Table 5: Results for the downstream CLEVR-Hal
Property Comparison task.
ID Ace. % (1)
OOD Acc. % (1)
dVAF Discrete
27.52
19.87
Steps 98% ())
OOD Acc. % (f)
dVAE Continuous
24.51
20.51
dVAE Discrete
37,000
26.7
VQ-VAE Indices
24.53
17.74
dVAE Continuous
32,000
29.5
VQ-VAE Codebook
23.73
18.80
VQ-VAE Indices
77,000
24.0
SysBinder
79.60
70.09
VQ-VAE Codebook
54,500
55.6
NLoTM Indices
68.21
64.53
SysBinder
27,000
67.6
NLoTM Codebook
75.86
71.15
NLTTM Indices
77,000
46.8
NLTTM Codebook
27,000
99.1
increases the potential for the downstream network to learn spurious correlations in the data, which negatively impact OOD performance. NLoTM Indices is also only able partially solve the task. This makes sense because in the out-of distriiiittiittionnn case, the model does not have any way of knowing two codebook indices are for the same property value (e.g. if two codebook vectors both correspond to the color blue). Since NLoTM Codebook uses the prototype vectors, it does not have this problem because the similarity be determined by the vector representation.
5.2.2 CLEVR- HARD PROPERTY COMPARISON
For CLEVR-Hard, construct downstream task that assigns number to each image follows: First, we assign a number for each possible shape, color, and material present the dataset. Then, for given image, we identify the maximum number for each of these three properties. Lastly, the max numbers for each of the properties to arrive at one integer label per image. We formulate the problem classification problem to correctly identify the number for each image. For example, suppose we have scene containing a matte red cylinder and a shiny blue sphere. Assume we assign the following numbers the different property values: matte O. shiny 1, red 5. blue cylinder = 4, sphere = 6. Thus the two objects are represented by the numbers (0, 5, 4) and (1.3, 6). The max numbers for each of the properties (1, 5, 6) and the final integer label is Solving this task requires understanding the property values of each object in the scene.
+ 6 12.
We train the underlying models on the entire dataset consisting of all the possible property values. Then we randomly select 50 objects for OOD dataset. Since our task relies on knowing the numerical value of each property, the ID dataset we train on may still contain property values of objects in the property values is present in the OOD dataset. Thus, when evaluating on the OOD dataset, we are testing the model on novel combinations of property values, even if those property values were individually observed during training. We show the ID and OOD results in Table We see that SVQ outperforms the patch-based methods and performs comparably SysBinder in both IDD and OOD settings. This shows that despite adding discretization bottleneck, the latents in SVQ are still useful for downstream tasks that rely on the properties of the objects in the scene.
dataset, but it will not contain objects where the combination of
CONCLUSION
In this paper, introduce the Neural Language of Thought Model (NLOTM). This is the first model that satisfies our proposed desiderata of a neural network model of the LoTH: (1) Compositional se- mantic representation, (2) Symbol-like discrete concept abstraction, (3) Probabilistic neural grammar to generate samples compositionally. The NLoTM consists of two main components: Semantic Vector Quantized Variational Autoencoder, which learns hierarchical and composable discrete representa- tions aligned with objects and their properties; and Autoregressive Language of Thought Prior, which learns to generate semantic concept tokens compositionally in probabilistic manner. We evaluated NLoTM several 2D and 3D image datasets. It demonstrated superior performance in downstream tasks, out-of-distribution generalization, and image generation quality compared to patch- based VQ-VAE and continuous object-centric representations. By further developing and refining models like NLoTM, we believe that we can continue to make progress in creating Al systems that exhibit more human-like like understanding and generalization capabilities.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
The scope of our study was restricted to visually simple, procedurally generated scenes and in its current form does not pose any immediate ethical concerns. Future work, however, that extends the capabilities of our model to work on more complex scenes may have the potential to generate fake, ceclittic-looking images. The semantic discrete latent may allow users to control scenes in ways that were not previously explored. While this may serve to enhance productivity, such as for artists and graphic designers, it could also be used maliciously in the hands of a bad actor. Future researchers pursuing this direction should do so under strong ethical standards and be cognizant of the potential misuse this technology.
REPRODUCIBILITY STATEMENT
In addition to details about our model described in Section 3.1 we provide additional implementation details in Appendix C including detailed hyperparameters used our experiments. We will also release the source code upon acceptance of the paper.
ACKNOWLEDGMENTS
This work is supported by Brain Pool Plus Program (No. 202HHID3A2A0310336) and Young Researcher Program (No. (NRF) funded by the Ministry Science and ICT. We thank Gautam Singh for insightful discussions and help with the CLEVR datasets. We also thank Sjoerd van Steenkiste for valuable feedback an earlier draft of this paper.
through the National Research Foundation of Korea
REFERENCES
Titas Anciukevicius, Christoph H Lampert, and Paul Henderson. Object-centric image generation with factored depths, locations, and appearances. arXiv preprint arxiv:2004.006442 2020.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 39-48, 2016.
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural information processing systems, 13, 2000.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, hts/1308.3432. 2013. URL
http://arxiv..xxt wttps//rrraabbb/13.3e3 abs//30ss.130//.. 3432
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07258. 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Christopher P Burgess, Loie Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 1390, 2019.
Chang Chen, Fei Deng, and Sungiin Ahn. ROOTS: Object-centric representation and rendering of 3D scenes. Jourmal of Machine Learning Research, 22(259):1-36, 2021. URL http::/jmmmm
2tte//ppperr//22220-1177
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling- arXiv preprint 1rXiv:1412.3555 2014.
10

---

Published as a conference paper at ICLR 2024
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Eric Crawford and Joelle Pineau. Exploiting spatial invariance for scalable unsupervised object tracking. arXiv preprint arXiv:191..0033, 2019a.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of AAAI, 2019b.
Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn. Generative scene graph networks. In International Conference on Learning Representations, 2021. URL /attpss//ppereeview
id 3mcPm9m3tnn
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever.
Jukebox: A generative model for music. CoRR, abs/2005. 00341,, 2020. URL
org/abs/ 2005. 00341
Andrea Dittadi, Samuele S. Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Locatello. Generalization and robustness implications in object-centric learning. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 5221552115551 -2255. PMLR, 2022. URL https: / pww.eedingss...c.oooo.p........e mlr prwsa/ v1 6262/ wwtttaiiiaa... htmmml
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B. McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items, 2022. URL htttttt/p//:aatttt org/ abs//204. 11918
Yilun Du, Kevin Smith, Tomer Ulman, Joshua Tenenbaum, and Jiajun Wu. Unsupervised discovery of 3d physical objects from video, 2021.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: generative scene inference and sampling with object-centric latent representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httptttttt:/ppeeeeee wet http://www..eeoooeee/ee//ea/aara
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object
representations without iterative refinement, 2022.
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2021, virtual, June 19-25, 2021, I, pp. 12873-12883. Computer Vision Foundation / IEEE, 2021. doi:
2A.1099/CVPPR44447..0011..111 CVPR2021/ttm/Esser Taming Transformers for Reghh-eesoltion.
ht..st::::: htthhttss:///penettt
Image_ Sange__Snttesiis. CVCPP CVPR 2021
Jerry A Fodor et al. The language of thought, volume 5, Harvard university press Cambridge, MA, 1975.
Noah D Goodman, Joshua B Tenenbaum, and Tobias Gerstenberg. Concepts in a probabilistic language of thought. 2015.
Anand Gopalakrishnan, Kazuki Irie, Jürgen Schmidhuber, and Sjoerd van Steenkiste. Unsupervised learning of temporal abstractions with slot-based transformers, arXiv preprint 13x1: 2022.
Robert M. Gray. Vector quantization. JEEE ASSP Magazine, 1:4-29, 1984. URL https://api
semant www.hccchlaa........ www.chollr..or/oorruuuu... ID: 14T::::447777
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schnidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems, pp. 6691-6701, 2017.
11

---

Published as a conference paper at ICLR 2024
Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. arXiv preprint arXiv:1903.00450 2019.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208, 2020.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01600 2019.
Olivier Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelović. Object discovery and representation networks. In ECCV, pp. 123- 143. Springer, 2022.
Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines. In Advances in Neural Information Processing Systems, 2020.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world mod- els with scalable object representations. International Conference on Learning Representations, 2019.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings the the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-2910, 2017.
Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matthew Botvinick, Alexander Lerchner, and Christopher Burgess. Simone: View-invariant, temporally-abstracee object representations via unsupervised video decomposition. arXiv preprint arXiv:2106.03849, 2021.
Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation. In Proceedings of the Neu- ral Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021. December 2021, virtual, 2021. URL https
wwweercccarrssppooceeedn...eeeee pttp://pps..ccpppp/////////////a hatpapprrr222/haaee 2021/hash/
html
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. otochasticootiimizaiion In 3rd International Conference Learning Representations, ICLR 2015, San Diego CA. USA. May 7-9, 2015, Conference Track Proceedings, 2015. URL http: http:///axiv.vv wrg.vv.orrrgaaao. aba /
wwwpgg/aa////aaa..a////////aaaaa 1412. 6980
Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. In International Conference on Learning Representations (ICLR), 2022.
Daniil Kirilenko, Alexandr Korchemnyi, Alexey Kovalev, and Aleksandr Panov. Quantized disentan-
gled representations for object-centric visual tasks, 2023. URL
Forum? Siprum?iddJIptuwane
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. CoRR, abs/1604.0023 00229, 2016. URL http::////ax org/abs / 6604. 00289
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Jindong Jiang, and Sungjin Ahn. Improving generative imagination in object-centric world models. In International Conference Machine Learning, pp. 4124, 2020a.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference Learning Representations, 2020b.
12

---

Published as a conference paper at ICLR 2024
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention, 2020.
Marcelo G Mattar and Máté Lengyel. Planning in the brain. Neuron, 110(6))914-334, 2022.
Stephen E. Palmer. Hierarchical structure in perceptual representation. Cognitive Psychol- ogy. 9()):441477 1977. ISSN 0010-0285. doi: https///oi.or//10000000000000002
90016-0. URL
001 0010028511 070000799700000 900160
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. In Proceedings of the 38th International
Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831. PMLR, 2021. URL http: //proceedings.oo mlr press /9/ ramesh2la html
Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with
VQ-VAE-2. In Advances in Neural Information Processing Systems 32: Annual Conference
Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC,
Canada. pp. 14837-14847, 2019. URL httt::::/proceeeedeeo www.:gs...uuiis... cctpppssccc.ppaaeeeeeeeeeeeeeeee wwpppppp/ppppppppppppcpp......ap
2019 22g.://aee..eaeaaa0000aaaa11111.
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Scholkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning. arXiv preprint arXiv:2209.14860, 2022.
Wolf Singer. Binding by synchrony. Scholarpedia, 2:1657, 2007.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALL-E learns to compose. The Tenth Inter- national Conference Learning Representationss ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022a. URL/stpps:/pppeeeeeeee.. net
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object- centric learning for complex and naturalistic videos. In NeurIPS, 2022b. http:: mips. cc/paper haales/paper/202222
URL
html
Gautam Singh, Ycongbin Kim, and Sungjin Ahn. Neural systematic binder. In The Eleventh International Conference Learning Representations, 2023. htttttttt//ppeeeeeeeee
Elizabeth S Spelke. What babies know: Core knowledge and composition volume 1, volume 1. Oxford University Press, 2022.
Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1):89-96. 2007.
Yao-Hung Hubert Tsai, Nitish Srivastava, Hanlin Goh, and Ruslan Salakhutdinov. Capsules with inverted dot-product attention routing. In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL htttps://openeet ew. net www.ee//ooouutee (oo.ommmdd-HJeeuAA.e....
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. In Advances in neural information processing systems, pp. 17990-4798, 2016.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukeuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6306 6315, 2017. URL / neuripnssssiinnnnseeeee...... cc/ wwtpppppppp.p///pcaaaaaaap 7 f 7e 7
.htll
13

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attentionobjcc--cetrie image generation is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Julius von Kügelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, and Bernhard Schölkopf. Towards causal generative scene models via competition of experts, 2020.
Xudong Wang, Rohit Girdhar, Stella X. Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation, 2023a.
Yanbo Wang, Letao Liu, and Justin Dauwels. Slot-vae: Object-centric scene generation with slot attention. In International Conference Machine Learning, ICML 2023, 23-29 July 2023, Hon- olulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 36020-36035. PMLR, 2023b. URL http::tttppooceddd..ttt
mmlr. presssresss..... v202 22222///wgg23z html
Nicholas Watters, Loic Matthey, Sebastian Borgeaud, Rishabh Kabra, and Alexander Lerehner. Spriteworid: A flexible, configurable reinforcement learning environment. 2019a. URL
http:uu..oommeeeeeeeeeeeeee deepmind/
spriteworid/
Nicholas Watters, Loic Matthey, Christopher P. Burgess, and Alexander Lerchner. Spatial broad- cast decoder: A simple architecture for learning disentangled representations in vaes. CoRR, (bs/1900.070117 2019b. URL/attp://arxxi. wrg. abs/ 1901. 07017
Taylor W. Webb, Shanka Subhra Mondal, and Jonathan D. Cohen. Systematic visual reasoning through object-centric relational abstraction. CoRR, abs/2306.02500, 2023a. doi: 10.48550/arXiv. 2306.02500. URL httptttts://oo.oopppppp/
T0..C.485550 48550/arXiv 2306. 02500
Taylor W. Webb, Shanka Subhra Mondal, and Jonathan D. Cohen. Systematic visual reasoning through object-centric relational abstraction. CoRR, abs/2306.025000 2023b. doi: 10.48550/arXiv. 2306.02500. URL
02500
Xin Wen, Bingchen Zhao, Anlin Zheng, X. Zhang, and Xiaojuan Qi. supf-supe visual represen- tation learning with semantic grouping. arXiv preprint arXiv:2205. 2222:22205.12288, 2022.
Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Generative video transformer: Can objects be the words? In International Conference on Machine Learning, pp. 11307-11318. PMLR, 2021.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using VQ- VQE and transformers. CoRR, abs/2104.10157, 2021. /tt/sttss//arxiiv.ooo wtggaaiii......a.....o
2104.10133 10157
Jaesik Yoon, Yi-Fu Wu, Heechul Bae, and Sungjin Ahn. An investigation into pre-training object- centric representations for reinforcement learning. CORR, abs/2300.004419, 2023. doi: 10.48550/ MrXiv.2302.04419 URL htllhhtts://eee www.:ooo..rrg.l.. wrg/10.g....... 48540 war.//aaXivvv.. 044012204444 04419
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net 2022. URL /ttphttss///opeeeiiiee net fww.mmormmo/ did= pfNyExj72i
Ruixiang Zhang, Tong Che, B. Ivanovic, Renhao Wang, Marco Pavone, Yoshua Bengio, and Liam Paull. Robust and controllable object-centric learning through energy- based models. arXiv preprint arXiv:2210.05519, 2022.
Daniel Zoran, Rishabh Kabra, Alexander Lerchner, and Danilo Rezende. Parts: Unsupervised seg- mentation with slots, attention and independence maximization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10439-10447,, 2021.
14

---

Published as a conference paper at ICLR 2024
A LIMITATIONS
While our method can learn semantic discrete representations and is capable of using these repre- sentations generate images of higher visual fidelity than previous object-centric methods such as GENESIS (Engelcke al.) 2020 2022), it is still only shown work well synthetic datasets with similar visual complexity as previous work (Singh et al. 2023). Although scaling unsupervised object-centrie models to more realistic datasets not focus of this work, further improving model so that it can work well on more realistic scenes is an important avenue of future research. Another limitation of our model is that our latent representations world does consist of many discrete concepts, factors such as position and pose are continuous. would be interesting explore ways combine continuous and discrete factors better model realistic scenes.
all discrete. Although our visual
B ADDITIONAL EXPERIMENTAL RESULTS
Slots
Image
Recon
Figure 5: Sample scene we use our codebook analysis.
B.1 CODEBOOK ANALYSIS
Latent Traversal. In this section, we qualitatively analyze the codebook for a sample scene. Figure 5 shows the sample we will use in our analysis. First, the image through the pretrained SVQ encoder to obtain a set of semantic discrete latents. Each latent represents one block from one slot and is provided by a prototype vector in the corresponding codebook for that block. To investigate the effect of traversing through the codebook, we replace each block with a different code in the codebook while keeping all other latents fixed. We then reconstruct the scene with the SVQ decoder and dVAF, essentially generating new image that only differs from the original image by one discrete latent.
Figure 6 shows the results for several sample blocks for the first slot (which corresponds to the teal ball) and the fourth slot (which corresponds to the gray cylinder). For each block, we choose the same set of 16 prototype vectors to display. First, object level-changing one block in one slot does not affect the other objects. We also see that the different blocks specialize in different factors. Block 1 corresponds to the left and right placement of the object. Block also corresponds to the placement of the object, but seems to also control the forward and backward placement of the object, as well the size of the object. We notice that in this particular case, the factors of position and size are not completely disentangled. This may be because this scene, the size depends on the placement of the object (e.g. closer objects are bigger). Block controls the color of the object, We see that the same prototype vector seems to produce the same color, although there are some inconsistencies such the disappearing cylinder in the bottom left. The color also seems to be cleanly disentangled from the other factors cchanging the color does not affect other factors like shape, size, or position.
see that the slots are disentangled at the
Block Analysis. Next, to further explore the representation captured in the codebook, we visualize the objects that are attended to for different prototype vectors. To achieve this, we run the pretrained SVQ on 1000 images obtaining the semantic discrete latents and slot attention segmentation maps for the objects in the images. Then, for each prototype vector in the codebook, we find and visualize the corresponding slots that are utilizing that code in one of its blocks. Note that unlike Singh et al. (2023), we do not need to do any k- k-means clustering to obtain this visualization since our representations are discrete representations in the codebook. Figures and & show sample objects corresponding to three different prototype vectors for block and block 7. We see that block 3 corresponds to object size and block 7 corresponds to object color. These results are consistent with the previous latent traversal
15

---

Published as a conference paper at ICLR 2024
Slot 1 (Teal Ball)
(Gry Cylinder)
Block
Block 3
Block
Figure 6: Latent traversal changing one latent in one block at time while keeping all other latents fixed. The image is then reconstructed with the single changed latent.
experiments. Furthermore, the three prototype vectors chose for block 7 correspond with the first three latents in Figure (right), showing that these three prototype vectors represent gray, purple, and teal, respectively.
Figure 7: Objects attended to when the latent for block 3 is set to three different prototype vectors.
Figure 8: Objects attended when the latent for block is set to three different prototype vectors.
16

---

Published as a conference paper at ICLR 2024
B.2 COMPARISON WITH SLOT-LEVEL QUANTIZATION
As discussed in Section 3.1 we hypothesize that slot-level discretization would struggle with complex scenes due to the combinatorial nature of the underlying factors of the objects. We test this hypothesis by running experiments on 2D Sprites and CLEVR-EaS -asy where we set the number of blocks M to and tune the size of the codebook, essentially doing slot-level quantization. In Figures we show the masked attention of each slot on the input image as well the image reconstruction. We find that with slot-level quantization, the model completely fails on the CLEVR-Easy dataset, unable that
to cleanly attend to the objects and reconstruct the image. On the 2D sprites dataset, with slot discretization, one slot ends up attending to the foreground objects and the model still cannot reconstruct the input image correctly. These results point to the importance of our choice to do block-level discretization.
Slots
Image
Recon
Slot Discretization
Block Discretization
Slots
Image
Recon
Slot Discretization
Block Discretization
Figure 9: Comparison of slot discretization and block discretization CLEVR-Easy (top) and 2D sprites
(bottom).
B.3 PRIOR MODEL CAPACITY FOR DVAE
In order evaluate whether or not larger capacity prior may improve the results for the patch-based dVAE baseline, we ran ablations using larger transformers for the dVAE prior on the CLEVR- dataset. The results are presented in Table 6 We see that while increasing the size of the transformer for the prior does slightly improve the FID for dVAE, it still underperforms when compared to NLoTM, indicating that simply scaling the dVVEE prior may not be sufficient to match NLoTMM performance.
Table 6: Effect of increasing the dVAE prior model capacity on FID on the CLEVR-Hard dataset.
Prior Model
FID
dVAE (8-layer)
65,89
dVAE (12-layer)
61.74
dVAE lay-layer)
60.75
NLoTM (8-layer)
43.12
B.4 NUMBER OF BLOCKS ABLATION
Table 7 shows the results of changing the number blocks in SVQ on the 2D Sprites (3 obj) dataset. We that when the number of blocks is too small, the model performs poorly and fails generate
17

---

Published as a conference paper at ICLR 2024
scenes corresponding the data distribution. For sufficiently large number of blocks, the model able to segment the scene, but Generation Accuracy decreases when the number of blocks is too large. We suspect this to be because with more blocks, the model may require higher capacity prior, which we kept fixed in this ablation.
Table 7: Effect of changing the number of blocks in SVQ on the 2D Sprites (3 obj) dataset.
Number Blocks
FID
Generation Accuracy (in %
465.60
0.00
80.71
1.56
7.76
75.78
6.61
75.00
16
7.17
55.47
32
8.74
54.69
B.5 CODEBOOK SIZE ABLATION
Table 8 shows the results changing the codebook size in SVQ on the 2D Sprites (3 obj) dataset. We had also tried smaller codebook sizes of 4 and 16, but found that for codebook sizes smaller than 32, the SVQ model did not converge well, resulting in black reconstructions. Similar to the number see that for larger codebook sizes, the FID scores are similar, but generation accuracy decreases for codebook sizes larger than 64, This again may be because the larger codebook sizes require higher capacity prior, which fixed in these ablations.
of blocks ablations,
Table 8: Effect of changing the codebook size in SVQ the 2D Sprites obj) datasct.
Codebook Size
FID
Generation Accuracy (in %)
32
7.31
76.56
64
6.61
75.00
96
6.64
51.56
128
7.58
48.44
256
8.73
32.81
B.6 FG-ARI SEGMENTATION RESULTS
Ssoows the Foreground Adjusted-Rande Rand--nexx (FG-ARI) metric for the CLEVR datasets for different slot-based models. We that when compared to SysBinder, NLoTM performs similarly in terms of FG-ARI on CLEVR-Easy and CLEVR-Hard and slightly underperforms on CLEVR-Tex. Compared to vanilla Slot Attention, NLoTM achieves higher FG-ARI on all 3 datasets.
Table 9; FG-ART results CLEVR datasets.
Slot Attention
SLATE
SysBinder
NLoTM
CLEVR
85,85
91.65
92.58
91.37
CLEVR- Hard
81.29
76.79
90.43
90.48
CLEVR-Tex
24.67
73.85
78.12
70.93
B.7 EXPERIMENTS ON GOOGLE SCANNED OBJECTS
In order to evaluate NLoTM on a more realistic dataset, we use a dataset where the objects are taken from the Google Scanned Objects (Downs et al.) (002). Specifically, the "Shoe" and "Bottles and Cans and Cups' categories. We evaluate on two versions of NLoTM: Mmmal- uses the same hyperparameters as used for the CLEVR--ard dataset (see Table
the objects from
18

---

Published as a conference paper at ICLR 2024
11 and -LoTM-large increases the codebook size to 256 and increases the size of ALP model from layers, 4 heads, model size 192 to 16 layers, 8 heads, model size 512.
We present the FID results in Table 10 and qualitative samples in Figure 10 TL-TM-small is able to generate objects from the dataset, the objects are smoothed out, resulting in high FID score. Increasing the model size NLoTM- significantly improves the quality of the generated scenes and the FID score, providing some evidence that NLoTM can be scaled to work on more realistic datasets.
We see that while
Table 10: FID for the Google Scanned Objects dataset.
Model
FID
NMoTM-small
114.16
NLoTM-large
72.68
Dataset
NLoTM-small
NLoTM-large
Samples for NLTTM on the Google Scanned Ohjects dataset. Scaling to larger model size noticeably improves the quality of the samples.
Figure 10:
C IMPLEMENTATION DETAILS
C.I TRAINING AND IMPLEMENTATION DETAILS.
We use input images of 64x64 resolution for the 2D Sprites datasets and 128x128 for the CLEVR datasets. Each model is trained NVIDIA Quadro RTX 8000 GPUs with 48GB memory and use half pref-srrcision floating-poin point format. We train SVQ for 400k iterations which takes around 80 hours for the CLEVR datasets and 50 hours for the 2D datasets. We then train the ALP prior for million iterations which takes around 40 hours. For the 2D Sprites dataset, similar to (Yoon et al. 2023), we first train the underlying models on dataset of random shapes without any relationship between the objects. We then train the prior models on the odd-one-out datasets.
C.2 HYPERPARAMETERS
Table II shows the hyperparameters used for the different datasets in our experiments with SVQ. For the d'VAE and Transformer Decoder, we follow the hyperparameters, architecture, and training procedure provided in Singh al. (2023) for CLEVR-Easy and CLEVR-Hard. For the 2D Sprites datasets, we use the same hyperparameters as we do for CLEVR- -asy for those components. All models are trained with the Adam optimizer (Kingma Ba 2015 with 31 0.9 and f3 0.999.
C.3 PRIOR MODELS
For the DVAE and ALP prior models, we use a transformer architecture with layers, 4 heads, model dimension 192, feedforward dimension 768, and dropout probability of 0.1. We use learning rate
19

---

Published as a conference paper at ICLR 2024
Dataset
Module
Hyperparameter
CLEVR-Fasy
CLEVR-Hard
2D Sprites
2D Sprites w/ BG
General
Batch Size
40
40
40
40
Training Steps
400K
400K
400K
400K
Image Size
128
128
128 x 128
64 x 64
64 x 64
SVQ
Codebook Dimension
256
128
256
32
# Blocks
16
8
8
Codebook Size
64
64
64
128
# Iterations
3
3
3
# Slots
6
8
8
50
50
50
50
Learning Rate
0.0001
0.0001
0.0001
0.0001
Table 11: Hyperparameters of our model used in our experiments.
of 0.0003 and 30,000 warmup steps. For VQ-VAE, we use a 20-layers PixelCNN prior, as proposed in the original paper (van den Oord et al. 2017).
C.4 DOWNSTREAM MODELS
For the 2D Sprites downstream experiments, we use a transformer architecture with 3 layers, 8 heads, model dimension 192, feedforward dimension 768. and a dropout probability of 0.1 for all models. We use the Adam optimizer with a learning rate of 0.0003.
For the CLEVR- Hard downstream experiments, we use transformer architecture with layers, heads, model dimension 192, feedforward dimension 768, and a dropout probability of 0. for all models. We use the Adam optimizer with learning rate of 0.0001.
20


Reviews:
[{"review_id": "b8mPL4ZmN6", "reviewer": "Reviewer_cC8S", "summary": "The paper proposed a novel image-quantized method, called semantic vector-quantized variational autoencoder. The SVQ constructs representations hierarchically from low-level discrete concept schemas to high-level object representation. The author conducts experiments on various 2D and 3D object-centric datasets and validates the effectiveness of the SVQ.", "questions": "Refer to the weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "i) Compared with widely used VAE and VQ-VAE, SVQ models the discrete abstraction and object-level representations simultaneously. \n\nii) The proposed semantic prior based on discrete latent codes can be directly used in generation tasks, and the visualization results show it is superior compared with the baseline methods.", "weaknesses": "I am not an expert in the quantization area, my main works are related to self-supervised learning. So I want to ask some questions about the application of self-supervised learning.\n\ni) The patch-level quantization method VQ-VAE can maintain the geometry structure. So it is widely used in mask-image-modeling methods, such as BEiT.  Can SVQ also maintain such a geometry structure? \n\nii) If the SVQ can be applied in the general 2D-image generation methods, such as stable diffusion? I think it is a promising application of the quantization method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uyZ7EGZ9Zs", "reviewer": "Reviewer_gjT4", "summary": "This paper proposes an approach to perform semantic-based vector quantization and learn a corresponding codebook as well as a generative model under object-centric learning scenarios. It uses a slot-attention-based encoder to obtain N object-centric features for each image, where the hidden dimension is divided by M groups and each group shares the parameters. The M denotes the number of attributes. Further, it learns a codebook with MxK codes to quantize the features from the encoder. For every slot, each attribute would be quantized to the code with the closest L2 distance among the corresponding K codes, resulting in the final NxM quantized results for the reconstruction task. Experiments are conducted on various synthetic datasets to showcase the effectiveness of the proposed model. The results show that it is capable to capture high-level information beyond patch and generate better results.", "questions": "1. In Tab.4, the authors only compare with dVAE and VQ-VAE Indices, which makes the comparison unfair as SVQ Indices also perform bad. Why not the authors also showcase the VQ-VAE code (like Tab.5) as well as dVAE feature (using the code id to index the weight of the first layer in the decoder)?\n2. The Tab.5 seems not a fair comparison as well. VQ-VAE / dVAE are not designed for the object-centric tasks so it is expected they will not perform well on this task, on the other hand, SysBinder shows comparable and even better performance on this task.\n3. Following 3, the paper lacks of ablation studies, which makes the reader hard to understand which part plays the critical role and is useful to support the author's claim, for example, what would happen when M varies and becomes smaller than the actual number of attributes?\n4. The authors have shown some interesting codebook analysis results, would be great if the authors could also showcase whether the learned codebook is composable and can perform controllable generation (other than based on reconstruction) and by solely manipulate the code.\n5. Recently there is a paper present similar idea (group level quantization https://arxiv.org/abs/2309.15505), which may give the authors some inspiration as well, note that this is a concurrent work so I'm not asking the authors to compare with them.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The writing is clear and easy to follow.\nThe motivation looks good to me, and the proposed idea is interesting, the codebook analysis results are impressive. The performance looks good on these synthetic datasets.", "weaknesses": "I'm not up-to-date to the latest slot-attention models and their performance on synthetic datasets such as CLEVR. It seems that the authors only choose a few baselines (GENESIS-v2, VQ-VAE, dVAE) for the generation quality evaluation as well as other downstream baselines. The VQ-VAE/dVAE model may requires larger model capacity to handle this tasks as they works on the local patch level. The generation results in Fig.4 is surprisingly bad, given the fact that they actually perform well on real data generation (VQGAN/DALL-E), it should be easy to reconstruct/generate good results when the hyperparameters are appropriate. I'm not sure whether the current results is convincing enough.\n\nI also have several questions in the following section, it would be great if the authors could address them. Overall I think this paper shows some interesting results and could inspire people. But the current experiments only show results on synthetic data which is somewhat weak.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YtGkIxyzt9", "reviewer": "Reviewer_qSSD", "summary": "The paper proposes a method for decomposing scenes hierarchically from low level factors such as color and shape to objects. The method learns a prior over these concepts which allows for generating data from the data distribution which most object-centric papers cannot do. In the the experiments they show that the representation can be used to solve downstream tasks which require reasoning about the properties of different objects in the scene. Additionally they show that the inductive biases proposed by the work leads to better FID score compared to VQ-VAE and similar generative models.", "questions": "- How sensitive are the results to the codebook size and number of blocks? It seems that for CLEVR we can a priori know these hyper-parameters, but for real-world or more complex datasets how would we set these?\n\n- It seems natural to compare the ARI with slot-attention or a variant of slot-attention. Is there a reason why the method shouldn't be compared in this manner?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes an interesting idea and the experimental setup for the downstream tasks are well done. Evaluating whether the model has learned underlying generative rules of learning is a neat and, to my knowledge, novel way of probing the model. I think the idea would be impactful and the community would be interested if it works for more realistic and larger scale datasets. \n\n- The qualitative results in figure 6 where the color, position and size are disentangled are impressive even for a simple, synthetic dataset. \n\n- The paper is well written and figures are illustrative of the main idea. \n\n- Overall the experiment are well done, with reasonable baselines and clear analysis of the results.", "weaknesses": "- The experiments are only conducted on synthetic, simple datasets. The variants of CLEVR are procedurally generated based on disentangled factors such as shape, color, and texture so it's unclear if the factor based approach proposed by this work generalizes to more realistic scenes. \n\n- The main metric used to evaluate the method and baseline is FID. This metric seems orthogonal to the goal of object-centric decomposition. Typically object-centric papers such as Slot-Attention measure segmentation with adjusted random index (ARI). I ask for clarification on why FID is the primary metric for evaluating object-centric models. I think if generation is the primary goal, then the work should compare to generative models and VQ-VAE on more realistic generative datasets. \n\n- I would be happy to improve my score if my concerns are addressed, particularly if the method works on real, more complex datasets than CLEVR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cGsakfg5bP", "reviewer": "Reviewer_41Cm", "summary": "This paper presents a modification of VQ-VAE called Semantic Vector-Quantized Variational Autoencoder (SVQ), which provides semantic discrete object representation by learning block-level factors, leveraging advances in object-centric learning. Different from patch-based VAE models, SVQ first applies slot attention to obtain object-level codebooks, then splits the slots into blocks following a binding mechanism to represent semantic factors. In comparison with previous methods, the model demonstrates better image sampling quality considering object fidelity. The proposed representation also shows promising performance on several downstream tasks in the out-of-distribution setting.", "questions": "My questions are listed in weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "（1）The paper is overall well-written and the idea of learning semantic representation is well-motivated.\n（2）Extensive and thorough experiments have been carried out to verify the effectiveness of the approach on multiple datasets and settings.", "weaknesses": "Weaknesses:\n（1）\tNovelty: The major limitation of this paper is the lack of theoretical novelty, since it seems the idea of disentangling object-level slots into blocked-level factors as well as the autoregressive transformer decoder for image reconstruction have been introduced in prior work SysBinder. It’s not elaborated on how the proposed approach is developed further in representing discrete semantics, especially when there’s not a large margin between the performance of SVG and other baselines.\n（2）\tSemantic illustration: Throughout the paper (including the name of the proposed model), the authors emphasize that the model is capable of learning semantic representation of objects such as shapes, colors, etc. However, how and what semantics are actually learned with unsupervised learning in the codebook is never illustrated or visualized, which makes the statement less convincing.\n（3）\tExperiment: First, the specification of multiple hyper-parameters (e.g. number of blocks, dimension of vectors) of SVG and other implementation details is never found. Second, the artificial designs of the constraints of the scene (metric Generation Accuracy) as well as the odd-one-out downstream task are not explained. Third, as the authors mentioned in the section of Limitations, the evaluations are restricted within synthetic datasets with simple scenarios. It’s unclear whether the proposed method will work in realistic, complex datasets, especially when the representation is object-related whose effectiveness is likely to be related to the size of the codebook.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)