Robust and Controllable Object-Centric Learning through Energy-based Models
Authors: Ruixiang ZHANG
Tong Che
Boris Ivanovic
Renhao Wang
Marco Pavone
Yoshua Bengio
Liam Paull
Venue: ICLR 2023 poster (2023)

Abstract: Humans are remarkably good at understanding and reasoning about complex visual scenes. The capability of decomposing low-level observations into discrete objects allows us to build a grounded abstract representation and identify the compositional structure of the world. Thus it is a crucial step for machine learning models to be capable of inferring objects and their properties from visual scene without explicit supervision. However, existing works on object-centric representation learning are either relying on tailor-made neural network modules or assuming sophisticated models of underlying generative and inference processes. In this work, we present EGO, a conceptually simple and general approach to learning object-centric representation through energy-based model. By forming a permutation-invariant energy function using vanilla attention blocks that are readily available in Transformers, we can infer object-centric latent variables via gradient-based MCMC methods where permutation equivariance is automatically guaranteed. We show that EGO can be easily integrated into existing architectures, and can effectively extract high-quality object-centric representations, leading to better segmentation accuracy and competitive downstream task performance. We empirically evaluate the robustness of the learned representation from EGO against distribution shift. Finally, we demonstrate the effectiveness of EGO in systematic compositional generalization, by recomposing learned energy functions for novel scene generation and manipulation.

Robust and Controllable Object-Centric Learning through Energy-based Models
ROBUST AND CONTROLLABLE OBJECT-CENTRIC LEARNING THROUGH ENERGY-BASED MODELS
Ruixiang Zhang?
Tong Che
Boris Ivanovic
Renhao Wang
Marco Pavone
Yoshua Bengio
Liam Paull
"Nvidia Research
Stanford University
Mila, Université de Montréal
(ruixiang zhang, tong. umont
ABSTRACT
Humans are remarkably good at understanding and reasoning about complex vi- sual scenes. The capability decompose low-level observations into discrete objects allows us to build a grounded abstract representation and identify the compositional structure of the world. Accordingly, it is crucial step for chine learning models to be capable of inferring objects and their properties from visual scenes without explicit supervision. However, existing works on object- centric representation learning either rely on tailor-made neural network modules or strong probabilistic assumptions in the underlying generative and inference processes. In this work, we present EGO, a conceptually simple and general ap- proach to learning object-centric representations through an energy-based model. By forming permulation-invarian energy function using vanilla attention blocks readily available Transformers, we can infer object-centric latent variables via tradient-based MCMC methods where permutation equivariance is automatically guaranteed. We show that EGO can be easily integrated into existing architectures and can effectively extract high-quality object-centric representations, leading better segmentation accuracy and competitive downstream task performance. Fur- ther, empirical evaluations show that EGO's learned representations against distribution shift. Finally, we demonstrate the effectiveness of EGO in sys- tematic compositional generalization, by re-composing learned energy functions for novel scene generation and manipulation.
robust
INTRODUCTION
The ability to recognize objects and infer their properties and relations a scene is a fundamental capability of human cognition. The central question of how objects discovered and represented the brain has been subject intense research for decades, and has prompted the field of cog- nitive science (Spelke) 1990) ask how we might develop intelligent machine agents to learn to represent objects in the same way humans do, without being explicitly taught what those objects are. Developing artificial agents capable of decomposing complex scenes into discrete objects can be a crucial step for many applications in robotics, vision, reasoning, and planning. Learning such object-centric representations can further help to identify the relational and compositional structure among objects and enables the agent to reason about novel scene composed of new objects by leveraging knowledge from previously- representations of similar objects.
In recent years, many works have been proposed to learn object-centric representations from visual scenes without human supervision. A variety models, in the form structured generative mod- els (Greff et al. 2019: Burgess et 2019 Engelcke al. 2020 Lin al. 2020) specifically designed neural network modules (Locatello et al. (2020). have been proposed to tackle the problem of visual scene decomposition and generation. On the other hand, recent progress in large language models Vaswani et al. 2017: Brown et al.) 2020) and visual- Lasuul-anguugg models (Radford et al. [202]]
Ramesh et al. 2022) shows the huge potential of training expressive neural network models with

---

Robust and Controllable Object-Centric Learning through Energy-based Models
Lx
Attention
MLP
MLP
X
Z2
Figure 1: Architecture EGO-Attention, the variant of EGO used in experiments. x is the input image and 2; are the object-centrie representations. In each block, EGO attends to the latent variables to refines the hidden scene representation using cross-attention mechanism between and 2, to measure the consistency between image input and latent representation.
minimal hand-designed inductive biases. In a similar spirit, we ask whether we can learn object- centric representations with minimal human assumptions and task-specific architectures.
Contributions In this work, introduce EGO (EnerGy-based Object-centric learning), ceptually simple yet effective approach to learning object-centric representations without the need for specially-tailored neural network architectures strong (typically parametric) assumptions data generating process. Based on the Energy-based Model (EBM) framework, we propose to learn an energy function that takes as input visual scene and a of object-centric latent variables and outputs scalar value that measures the consistency between the observation and the latent repre- sentation (Section 2}. We minimally assume permutation invariance among objects and embed this assumption into the energy function by leveraging the vanilla attention mechanisms from the Trans- former (Vaswani al. 2017) architecture (Section 2.1). In essence, our method makes models act as segmentation annotators, aiming to iteratively improve their annotations by minimizing our energy function. We use gradient- based Markov chain Monte Carlo (MCMC) sampling to efficiently sam- ple latent variables from the EBM distribution, which automatically yields permutation-equiin -qquivrrinn update rule for the latent variables (Section 2.2)). This stochastic inference procedure also addresses the inherent uncertainty in learning object-centric representations; models learn represent scenes containing multiple objects and potential occlusions a probabilistic and multi-modal man- We demonstrate the effectiveness of approach variety of unsupervised object discovery tasks and show both qualitatively and qualitatively that our model can learn to decompose complex scenes into highly accurate and interpretable objects, outperforming state-of-the-art methods on seg- mentation performance (Section 4. We also show that we can reuse the learned energy functions for controllable scene generation and manipulation. which enables systematic compositional gen- eralization to novel scenes (Section 4.2). Finally, we demonstrate the robustness of our model to various distribution shifts and hyperparameter settings (Section
2 ENERGY-BASED
REPRESENTATION LEARNING
The goal of object-centric representation learning is to learn a mapping from a visual observation Px to a vectors {zæ), where each vector Zд E D. describes an individual object (or background) in In this work, we make use of an EBM E(x, 0), parameterized by 0, to learn a joint energy function which assigns low energy to regions where the visual observation x and latent object descriptors are consistent, where = [2 k= are set of K object-centric latent variables. To implement the mapping from a visual scene to its constituting objects, we can sample from the posterior distribution z ~ p(z[x: x: 0) - e- by using any efficient MCMC sampling algorithm, such the stochastic gradient Langevin dynamics method (Parisi 1981 Welling & Teh 2011 and Hamiltonian Monte Carlo (Duane al. 1987 Neal et al. 201 Accordingly, the EBM E can be used as generic module for object-centric representation learning, offering great flexibility in which neural network architectures can be used and the functional form of the energy function.
2.1 PERMUTATION INVARIANT ENERGY FUNCTION
One fundamental inductive bias in object- representation learning is encoding the permutation invariance of of objects into model learning. In this section, introduce formulations
2

---

Robust and Controllable Object-Centric Learning through Energy-based Models
of the energy function Exx, z: 0) that are permutation invariant with respect to the order of the object-centric latent variables }
EGO by composing individual energy functions First, we consider simple formulation of the energy function E(x, x: 0) that decomposed into set of individual energy functions E(x, Zx; 0k), followed by permutation invariant aggregation function Φ as E(x, 2) & ({F((x*))))
The individual energy function Exx, z,: 0x) RR x RD + R can be any function that takes the observation x and single latent variable 25 input, and outputs a scalar energy value which quantifies the belief that an object with representation Zς is present in the visual scene x. We share the parameters 0& across all the individual energy functions 8, Ak, such that it can generalize to an arbitrary number objects without breaking symmetry.
The aggregation function @ is a permutation cermuutationinvariinn invariant function with respect to the set {x Zi. We can use any function & that is invariant to the order of inputs, such as the sum (Du et al. 2020 2021], minimum (Parascandolo et al. 2018), , parameterized transformations (Zaheer al. 2017). Throughout this work, we use the sum as the aggregation function, which is effective in encouraging the model to learn to decompose the input into discrete objects and local variations, explored in (Du et al. 202
Zhang et al. 2020). We call the resulting EBM formulation EGO-Sun Sum, given by
F((, 2; =
Σ
F(xx 2g; 0)
EGO from permutation equivariant/ivvaiintt transformations We introduce another formu- lation of the energy function F(x,, 2; 0) that composes multiple permutation equivariant//nvarian transformations on top of the set { and x. In particular, use vanilla attention blocks (Vaswani ct al. 2017), such as cross-attention and self-attention, to build up these differentiable mappings.
We first encode the data input E Dx into higher-level representation h R.M,x,, where Nn = Width,, x Height, is the number of vectors in the 2D feature map h when using a convolu- tional neural network (CNN) as the backbone encoder.
We then use a stack of L standard transformer blocks with a cross-attention layer to fuse the infor- mation in h and the object-centric latent variables z (z,) K Each block consists of a multi-head cross-attention layer, followed by a position-wise, fully connected feed-forward network. In each cross-attention layer, linear transformation is applied to the image feature map h to produce queries over the set of latent variables. Cross-attention weights are then computed between the queries and projected keys and values from the set of latent variables {z The stacked transformer
blocks allow the model to sufficiently capture information from {z by attending to the most rel-
evant subset of latent variables each image feature location, such that the model can learn to tell whether the set of latent variables fully explains each object.
We then use an average pooling layer to aggregate the final output of the transformer blocks h Nu Da,, into single vector, which is then passed through calle-connected layer to produce the scalar energy term.
We provide an overview of our architecture in Figure and call the resulting EBM formulation EGO EGO-Attentioon on, given by:
hg Encoder(x)
(2)
h/ CrossAttaattoonLLLaeeeoommb.... LayerNorm
(avo + h
i = 1...... L
(3)
h MLP((ayyeNorr(())) + h/
{= 1...L L
(4)
E MIPPAVgPPol(hz:)
(5)
2.2 LEARNING AND INFERENCE
For tasks requiring geometric understanding of visual scene and reasoning over entities, model can be used as a plug-and-play module to be integrated into existing architectures for encod- ing structured object-centric representations. Owing the great flexibility of EGO's energy function
3

---

Robust and Controllable Object-Centric Learning through Energy-based Models
Algorithm 1: Training procedure of EGO for unsupervised object discovery.
Input: Image data x E DR.x, number of latent variables K, number of MCMC iterations T, step size c
Parameters: EGO Exx, 0), decoder Decoder(z; <))
Output: Training loss for unsupervised object discovery
Infer object-centr latent variables by Langevin MCMC sampling
Draw random initialization z ~ I),
for f
do
# #Using
energy functions from Eq.
Or
Eq.
" (0,11)
z 1-1 = z' + + z'; 2en''
Decode the latent variables to create reconstructions
X Decoder(z*; ;))
Compute the reconstruction loss
return Lee = D x
and learning objective formulation, we can customize the model adapt to a wide range of task contexts and learning desiderata.
Among many possible training objective choices (e.g., maximum likelihood training with MCMC sampling or Contrastive Divergence (Hinton 2002)) to learn the model monolithic generative model, akin to the family of existing approaches (Engelcke et al. 2020; Greff et al., 2019 Burgess et al. 2019) for visual scene understanding and generation, focus investigating the potential of EGO as a generic standalone module for extracting object-centric representations, similar to (Lo- catello et al. 2020). To this end, we adopt encoder-decoder architecture, with our EGO module serving as the encoder to transform the unstructured observation into structured object represen- tations, which are then decoded by a separate decoder into reconstructions or other task-specific predictions.
Encoding object-centric representations by MCMC sampling To infer the set of object-centric latent variables from the input x, use gradient-based MCMC sampling methods to sample from the posterior distribution ~ p(z) x) x e E(x,2;0) Specifically, in this work we utilize the Langevin MCMC (Parisi) 1981 WWlling & Teh 2011) method. Starting from a random initialization z" drawn from simple prior distribution, we iteratively update the latent variables by simulating the Langevin diffusion process for steps, with step size e, as follows:
z = + EV + 2en'. t 0, . - n/' ~ ~(O.I), z" ~ ~N(O..) (6)
where z' denotes the latent variables at the 1-th iteration. When e -> O and 7 - oc, the sampling process converges to the true posterior distribution p(z|x) under some regularity conditions.
Though running MCMC sampling until convergence can be computationally expensive, we are sim- ulating the Langevin dynamics the latent space
RKxD, rather than the high-dimensional pixel space, in contrast to previous works (Du al. 2021 Du & Mordatch 2019). We also only run Langevin dynamics for a relatively small number of iterations (I 10) and find that it sufficient to produce good latent variable samples z" in our experiments. This allows us to make use of the gradient-based MCMC sampling much more efficient manner, even comparable to amortized inference methods (Eslami et al. 2016) Greff et al. 2017 2019: Burgess et al. 2019).
Training procedure We outline the detailed training procedure in Algorithm I taking the un- supervised object discovery task as example. Given the encoded structured representation z" ~ p[z]x; A) from MCMC sampling, we use a decoder to map the set of latent variables to task-specific predictions. For unsupervised object discovery, we follow prior approaches and use spatial broadcast decoder (Watters et al. 2019: Greff et al. [019 Locatello et al. (020) to decode each latent variable z₁ separately into an alpha mask Qg E 0, 1x and input reconstruction X;. which are then combined to produce the final output x = Ek Softmax( (x The model can be trained end-to-end to minimize the reconstruction loss. For other tasks, can use corresponding
4

---

Robust and Controllable Object-Centric Learning through Energy-based Models
decoders (Kosiorek et al. 2020 Zhang et al. 2019: Zaheer et al. 2017) map latent variables to predictions and train the model end-to-end to minimize task-specific losses.
3 RELATED WORK
Object-centric learning Object-centric representation learning (Greff et al. 2015; 2016; Eslami (ct al., 2016; Greff et al., 2017: 2019; Lin et al. 2020 Xie et al. 2021 plays an important role in scene understanding, visual reasoning, and compositionll generalization. Many research works like MONet (Burgess et al.] 2019), IODINE (Greff et al., 2019), GENESIS (Engelcke et al.) 2020 2021, and SPACE (Lin et 2020) propose various probabilistic models build spatial mixture model of visual scenes and use variational inference to learn object-centric latent variables. Slot aftention (Locatello al., 2020) proposed novel inverted attention mechanism iteratively sign objects to slots, by introducing competition among slots via attention weight normalization. (Kipf al. 202T) further applied the slot attention module video data, and (Chang et al. 2022)
improved the approach by using fixed points as object representations, yielding better stability. Be- sides reconstruction- baasd object-centric learning. various works (Racah & Chandar 2220 Kipf
eet al. 2020 Löwe et al., 2020 Pirk et al., 2019 Baldassarre & Azizpour 2022) tackle the problem from the perspective contrastive learning and self-supervised learning. Object-centric represen- tations are also explored in other tasks such visual question answering (Huang al., 2020 Dang ct al. 202I, visual reasoning (Assouel et al. 2022), and neural scene rendering (Guo et al. 2020).
Set generation Set generation (Vinyals et al. 2016; Rezatofighi et al., 2018 Zhai et al. 2020 Rezatofighi et al.) (022)) has received much attention in computer vision and natural language processing tasks in the past few years. (Kosiorek et al., 2020) explored using a cardinality- conditioned transformer for set prediction. The DETR (Carion al. 2020) model also adopted a transformer to predict a set of objects for the object detection task. The Deep Set Prediction Network (DSPN) (Zhang et al., 2019) proposed set prediction approach by using gradient opti- mization inner-loop with a permutation-invariann encoder to iteratively update the representation of the target set prediction by minimizing the distance between the input and output induced by the encoder. Though the bi-level optimization framework is similar to ours, the DSPN model uses the deterministic gradient descent procedure to optimize a pre-defined loss function, while our formulation uses stochastic MCMC sampling to optimize learned energy function. Our formulation brings more flexibility to the optimization process, and the introduced stochasticity also results in more robustness and diversity in both inference and generation (Section
latent space
Energy- models Energy-based models (EBMs) (LeCun et al. 2006) have been widely used in learning abstract concepts (Du al. 2020) and controllable generation (Nie al.) 202 Che et al. 2020), due to the compositionality nature of the energy function (Andreas) 2019). Mordatch
2018 proposed an EBM- based framework for learning abstract concepts from observations, which
enables compositional generalization by test-time optimization. (Zhang et al., 2020) explored the approach of using EBM to replace the hand-crafted permutation-invant -ivvattiionnnaiinennneeee loss functions which aim to optimize the prediction output towards observable ground- truth data, it differs from work in that we focus on learning the latent object-centric representations from the EBM for unsupervised object discovery and controllable scene manipulation. (Du al., (02T)) also proposed to EBM to learn both local and global factors of variations from image data, where they proposed to train the model by a nested gradient descent optimization in the sigh-dimensional pixel space, which can be much more computationally expensive compared to our model which runs Langevin dynamics in the lower-dimensional latent space.
4 EXPERIMENTS
4.1 UNSUPERVISED OBJECT DISCOVERY
We quantitatively and qualitatively evaluate our proposed model on the task of unsupervised object discovery, with the goal of decomposing the visual scene into set of objects without any human supervision. As we will show, our approach able to consistently segment images into highly interpretable and meaningful object masks.
5

---

Robust and Controllable Object-Centric Learning through Energy-based Models
Table 1: ARI scores for unsupervised object discovery on CLEVR-6, Multi-dSprites, Tetrominoes datasets.
CLEVR6
Multi-dSprites
Tetrominoes
Slot MLP (I Locatello al.
2020
60.4 + 6.6
60.3 + 1.8
25.1 + 34.3
MONet (Burgess 2019
96.2 0.6
90.4 0.8
Slot-Attenton (Locatello al.
2020
98.8
91.3 0.3
99.5 - 0.2
IODINE (Gref et al. 2019
98.8 1 0.0
76.7 4: 5.6
99.2 1 0.4
BG0-Sum
96.1 + 0.4
82.1 + 0.7
99.1 + 0.3
EGO Atttetiin
98.9 1 0.5
93.8 1 0.5
99.6 - 0.2
Datasets In line with previous state-of-the-art works on object discovery, we use the follow- ing three multi-object datasets (Kabra et al.) [019); CLEVR (Johnson et al.] (017). Multi- dSprites (Matthey et al. 2017). and Tetrominoes (Greff et al. 2019). We also use variant of the CLEVR dataset which filters out scenes with more than 6 objects, referred to as CLEVR-6. Same as IODINE (Greff al., 2019) and Slot- Alot-Attetionn (Locatello al.) 2020), the first 70K samples from the CLEVR-6 dataset and the first 60K samples from the Multi-dSprites and Tetrominoes datasets are used for training. Evaluation is performed on 320 test data examples.
Implementation For the EGO model, first encode the image input using CNN backbone. We use the same CNN architecture from Slot-Attention (Locatello et al. 2020), which is aug- mented with positional embeddings for all results in this section. To condition our EGO- Sum model F((x, 2, O) on the latent variables, we use an MLP to project z to a vector, replicating it to match the spatial dimension of the image features and concatenating it along the channel dimension. We process the joint features through set of self-attention layers, followed by a global average pool- ing layer and a fully-connected layer to produce the final energy value. For the EGO-Attt tentttttttt model, we obtain the energy value from Equation 5 in Section. 2. We do not use dropout in our self-attention and cross-attentio layers. To decode the inferred latent variables into reconstructions, we apply spatial broadcast decoding and use the same architecture from IODINE (Greff et al. 2019). We use D2 64 for the latent variable dimension, as baseline methods. We use K variables for CLEVR-6, K 6 for Multi-dSprites, and K 4 for Tetrominoes, which is more than the maximum number of objects in the corresponding datasets. In Langevin MCMC sampling, set the step size 0.1 and the number of Langevin steps T 5. We train the model using the Adam optimizer (Kingma & Ba 2015) with a learning rate of 0.0002 for 500K iterations, with a batch size of 128. Additional implementation details and results can be found in the Appendix.
latent
Segmentation accuracy Following the evaluation protocol in existing literature, we use the Ad- justed Rand Index (ARI) (Hubert& Arabie 1985) metric, which measures how accurately the model can decompose the scene into discrete objects. We compare our model against a variety of base- al., 2020). IODINE (Greff et al., 2019). and MONet (Burgess et al., 2019). Similar to IODINE and Slot-Attention, we take the alpha masks generated by the spatial broadcast decoder for each latent variable, compute the clustering similarity with the ground truth masks using the ARI metric, and exclude the background in the evaluation. We report the ARI scores across different datasets in Table [ As can be seen, EGO consistently outper- forms the baselines, achieving near-peet serffct segmentation accuracy on CLEVR-6 and Tetrominoes, and substantially better results on Multi-dSprites compared to the existing state-of-the-arr. Notably, there are more occlusions among objects presented in Multi-dSprites, which makes the task more challenging and demonstrates the effectiveness of our model in handling more complex scenes.
line methods, including Slot Attention (Locatello
Downstream prediction To investigate the usefulness and quality of the learned representations, We evaluate our learned object-centric model on downstream object property prediction tasks. Sim- ilar to IODINE, we probe pre-trained models' learned representations by training a linear model on top of the latent variables to predict associated object properties, such as color, shape, size, and position. Thanks to the object-centric nature of baseline methods and our model, where object rep- resentations share a common format, we can train a single probing model to independently extract properties from each object-centric latent variable. We train the probing model by using the Hun- garian algorithm (Kuhn 1955) to match the latent variables to the ground-truth objects. Following the same training and evaluation procedure described in (Dittadi et al.] (022), we compare our pre- trained AGO-Atteet:. i on model against baseline approaches across different datasets in Figure

---

Robust and Controllable Object-Centric Learning through Energy-based Models
CLEVR
Sprites
Tetrominoes
0.8
Accurscy
0.6
0.2
0.0
size
mhae
color
shape
color
scale
shape
color
EG0
Slot: Attention in
MONET
GENESIS
Downstream object property prediction results on CLEVR, Multi-dSprites, and Tetromi- noes. The metric is accuracy for categorical properties or R² for numerical properties.
Figure 2:
We see that the learned representations from our model are highly informative for predicting object properties and comparable or outperform the competitive baseline methods all datasets.
4.2 SCENE DECOMPOSITION AND MANIPULATION
We next qualitatively study the learned object-centric representations by inspecting scene decom- positions and visualizing the inference procedure. We also demonstrate the ability of our model to manipulate the scene by recomposing learned energy functions dynamically manipulate scenes.
Scene decomposition We visualize per-latent variable reconstruction results from the trained EGO-Attent model across different datasets in Figure 3[a). The examples show that our model is able to decompose the scene into highly-interpretablle segmentations, which align well with the ground-truth objects. Extra latent variables are assigned to the background when there are more latent variables than the number of objects in the scene. Our model learns to spread objects across latent variables without explicit supervision, and object properties are also well-preserved in the inferred representations. Meanwhile, when multiple objects occlude each other, the model is able to infer the correct parts of objects by leveraging other clues such as shape and color. We further visualize scene reconstructions at each MCMC sampling step in Figure 3e., showing that the inferred scenes are iteratively refined within a few steps. We also plot the energy function values evaluated at each Langevin dynamics iteration from a trained model with T = 10 steps on the Tetro- minoes dataset in Figure b). As can be seen, both energy values are monotonically decreasing, indicating that the model can infer the latent variables by optimizing the energy function efficiently and stably.
Scene manipulation With trained EGO models, learned energy functions can be used to dynami- cally manipulate a scene's 's constituent objects. To show the controllable scene manipulation ability of EGO, in Figure aa), we show that we can combine arbitrary objects from different visual obser- vations (X1 and x₂) together into a novel scene, by sampling the latent variables from the joint EBM E(x, z; 9) Exx,, z; 0) E(x2, z; 0), known as product-of-experts (Hinton 2002), and recon- structing the scene from the inferred latent variables. A visualization of the scene reconstructions and predicted masks associated with each latent variable is also included in the figure, showing that, starting from the first few iterations, the model captures object components from both images and combines them across the latent variables to generate the complete scene. We additionally show another example in Figure [b), where we can remove any specific object from the scene by reusing learned energy functions. As described in (Du et al., 2020), we form new energy function as E(x, 2; 0) = E z: O) E Exx,, 2; 0) to remove the objects shown in X2 from the scene X1. Sim- ilarly, we also illustrate the intermediate results at each sampling step, where we can see that in the first few steps of the sampling procedure, the model recovers the complete scene from X1, and then gradually removes the objects in X2 by optimizing the latent representations towards the region where Exx,, z; 0) is higher. These results clearly demonstrate that the EBM formulation of EGO allows us to control the scene composition combinatorically. leading to systematic generalization to unseen object combinations.
7

---

Robust and Controllable Object-Centric Learning through Energy-based Models
535
198
-2030
LEWENEL
2535
1930
hansbora
Energy evaluation during sampling.
(a) Scene decomposition across datasets.
(c) Scene reconstructions each step.
Figure 3: (a) Per-latent variable reconstructions and masks on CLEVR-6 (top), Multi-dSprites (mid- dle), and Tetrominoes bottom. (b) The progression of energy function evaluations during Langevin sampling. (c) Scene decomposition and reconstruction visualization at each sampling step.
CLEVI
Necoranndees
Met
Xi
X1
Unseen num. of objects.
X>
(a) Scene combination.
(b) Scene subtraction.
(d) Ablation on and T.
Figure 4: (a) Scene manipulation for combining objects, full version in Figure 8
(b) Scene manip-
ulation for removing objects, full version in Figure (c) ARI scores of pre-trained EGO on CLEVR dataset when generalizing to higher numbers of objects than seen during training. We use 'C' to denote CLEVR for brevity. (d) Model ablation results.
4.3 ROBUSTNESS AND GENERALIZATION EVALUATION
Finally, evaluate the generalization capability of EGO to out-of-distribution (OOD) visual scenes and investigate its robustness with respect to model hyperparameters and component choices in an ablation study.
Unseen object style We first study the generalization capability of our learned object-centric rep- resentations to unseen object styles by altering the visual styles of objects presented in the test data distribution, including color, shape, and texture. Following (Dittadi et al.) 2022). we apply a random random object the scene CLEVR and Tetrominoes.
color jitter transformation to the color Unseen object textures are simulated by applying random neural style transfer (Gatys et al.) 2015) to a random object in CLEVR and Multi-dSprites. We use the previously-traing pervised object discovery task and evaluate it on these modified datasets test whether the learned representations are able to generalize to unseen object styles. ARI scores of our model and other baseline methods shown in Figure 5 a). Our model achieves strong generalization performance
model from the unsu-
8

---

Robust and Controllable Object-Centre Learning through Energy-based Models
CLEVI
0.2
0.6
0.4
0.11
Original
Texture
Color
Original
Shape
Testure
Original
Texture
Color
Orggal
Shape
Tachure
EGO
MONel
GENESI
6.1
60
(b) Effect of stochasticity.
(a) ARI scores on test data with unseen object styles.
Figure 5: (a) Segmentation results of pre-trained model on OOD test dataset. (b) Model ablation on the effect of the noise variable in MCMC sampling.
to unseen object colors, textures, and shapes on CLEVR and Multi-dSprites, consistently providing high-quality segmentation masks in the presence of OOD object styles. We provide more details about the datasets and evaluation in the appendix.
Increasing the number of objects To test how our model generalizes when more objects are present in the scene (compared to the training data), we increase the number of objects at test time. We construct sequence of datasets by filtering CLEVR to only contain scenes with at most [7, 8, 9, 10] objects, referred to as CLEVR-[7-10) respectively. We use EGO-At Attentteo ior trained on CLEVR-6 with K 7 latent variables, and test on the newly-constructed datasets by increasing the number of latent variables to one more than the maximum number of objects in the correspond- ing dataset. We compute the ARI scores for each dataset, and report the results in Figure ((c). We use Slot-Attention with 3 iterations (same as T = 3 in our model) as the baseline. As can be seen, the segmentation quality of our model remains robust to unseen numbers of objects in the test dataset, and suffers less from the increase in the number of objects than the baseline approach.
Model ablation We perform ablation studies to evaluate the robustness of our approach to different hyperparameters and study the effect of different components of our model, particularly the number of MCMC sampling steps T, the step size c, and the noise in Langevin dynamics. We first run grid search over the hyperparameters, including E {0.01, 0.05, 0.1), E {3, 5, 10), number of attention blocks E {1, 2,3). and dropout rate E {0.0, 0.1}, and report the ARI scores on the Multi-dSprites dataset in Figure dd.. Our model is quite robust to wide range of the values of both the step size and the number of sampling steps, and we obtain npar-optimal segmentation performance with each combination of the hyperparameters. We then investigate the role of the noise used in the Langevin dynamics sampling by introducing weight term to rescale the noise variable in Equation (6] We train RGO-At tenttttetttt...............t on variants with different rescaling weights {1.0, 0.1, 0.0} on the Multi-dSprites dataset and evaluate their ARI scores in both the original dataset and the modified OOD dataset with unseen object styles. The results in Figure demonstrate that the stochasticity brought by the noise in Langevin dynamics gives our model more robustness and generalization to novel scene configurations.
5 CONCLUSION
In this work we present EGO, a novel energy-based object-centric learning model. EGO successfully combines three essential ingredients of object-centric learning: (i) flexible energy-based models without strong assumptions on data generating process, (ii) minimal usage of potentially unexpres- sive specifically-desineed neural modules, and (iii) explicitly modeling randomness allow one-to- objects. We empirically demonstrate
many mappings to reason about occluded or partially-observe that EGO can achieve state-of-the-arr performance on various unsupervised object discovery tasks and excels at generalizing to OOD scenes. Meanwhile, controllable scene generation and manipu- lation are also feasible by reusing learned energy functions. These advantages make EGO strong candidate for scaling unsupervised object-centric learning to real-world datasets, as a promising next step for future investigation.

---

Robust and Controllable Object-Centric Learning through Energy-based Models
REPRODUCIBILITY STATEMENT
In order to make the experimental results easily reproducible, made following efforts: i) Source code necessary to reproduce results of our model made available in the supplementary material. ii) We provided the detailed training procedure in Algorithm[[] and described all model architectures and hyperparameters in main text (Section 4 and appendix (Section
REFERENCES
Jacob Andreas. Measuring compositionality in representation learning. In 7th International Confer- ence on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenRe- view.net, 2019. URL https: bettps://openreviie net/ forum? d=HJz0500qks 0
(Cited on
5
Rim Assouel, Pau Rodriguez, Perouz Taslakian, David Vazquez, and Yoshua Bengio. Object-centric compositional imagination for visual abstract reasoning. In ICLR2022 Workshop on the Elements of Reasoning: Objects. Structure and Causality, 2022. (Cited on 5
Federico Baldassarre and Hossein Azizpour. Towards supervupe learning of global and object- centric representations. arXiv preprint arX(v:2203.059997 2022. URL httph:ttp//aaat /arpiii////xv...//////e/// org/
abs/2203 05997 (Cited on
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnee and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL http:
//qithub. wwt.///./ooooo qoooe//jool//ax
(Cited on
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Vosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse. Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL httpp::/pprcceeeeee 1457c0dbb 4b4 96741 8bfb8ac1 - html
www.iperrpp..... wttpsssccppppp.e 2020/has0/ (Cited on
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint /9I.:/901..1190 2019. URL 11390
(Cited on
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-221 Springer, 2020. (Cited 55
Michael Chang. Thomas I. Griffiths, and Sergey Levine. Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation. arXiv preprint arXiv:2207.008 00787, 2022.
http://awww...og/////////a 2btp://bb2b222007877 (Cited on 5
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your GAN secretly an energy-based model and you should use discriminator driven latent sampling. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Informattio Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL (tRLlitttps::/ppoceeeeeee ott.////oooooooo///oo/oo/ooooooo http://wwwcceea.oo......eeeeeeee
CC/ pappp/aa//////aaaa pappr/2220/hass. html
(Cited on
10

---

Robust and Controllable Object-Centric Learning through Energy-based Models
Long Hoang Dang. Thao Minh Le, Vuong Le, and Truyen Tran. Object-centric representation learn- ing for video question answering. In 2021 International Joint Conference on Neural Networks (LJCNN), pp. 1-8. IEEE, 2021. (Cited on
Andrea Dittadi, Samuele S. Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Locatello. Generalization and robustness implications in object-centric learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Mary- land, USA, volume 162 of Proceedings of Machine Learning Research, pp. 5221-5285. PMLR, wtt.://www.........sssseeee.e...
2022. URL
(Cited
on 6 16
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alcéé-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Infor- mation Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp- 3603-3613, 2019. URL
neurip http:eerrrrpppeepeeeeeeee////e/e pttppppee/219///aeeeee.ee (Cited on
html
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL http:tt://pocccedi.iiie 19856ed44 F5e
htm]
(Cited on
35
Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems, 34:15608- 15620, 2021. (Cited on
5
Simon Duane, Anthony D Kennedy, Brian J Pendleton, and Duncan Roweth. Hybrid monte carlo. Physics letters B, 195(2):216225)) 222, 1987. (Cited on
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: generative
scene inference and sampling with object-centric latent representations. In 8th International Con- ference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net net, 2020. URL https:://peenree
Bid net. wttp//ooooun...-xxfafaaaaaaaaae
(Cited on
Martin Engeleke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. Advances in Neural Information Processing Systems, 34:8085-8094, 2021. (Cited on 5)
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E. Hinton. Attend, infer, repeat: Fast scene understanding with gener- ative models. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Ro- man Garnett (eds.), Advances in Neural Information Processing Systems 29; Annual Conference on Neural Information P Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3225-3233, 2016. URL http:: /ttp:::///rccceeee
neur wwtpccppppppppeeee httpppppr/////hass//p (Cited on
html
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. A neural algorithm of artistic style. arXiv preprint arxiv: 2015. URL htt.:ttpsr/aaaiiv.rrr/////////// on
(Cited
Klaus Greff, Rupesh Kumar Srivastava, and Jürgen Schmidhuber. Binding via reconstruction clus- tering. arXiv preprint arXiv: 1604188, 2015. URL. httpp:::/aa.ivvorrrbbb////////// 06418
(Cited on
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hotloo Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In Daniel D. Lee,
11

---

Robust and Controllable Object-Centric Learning through Energy-based Models
Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neu- ral Information Processing Systems 2016. December 5-10, 2016. Barcelona, Spain, pp- httppaperr2216/hhaee
4484 4492, 2016.
new. -bsp::aa0000eeebbbaaaaaaatteeeee html (Cited on 5)
0
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation max- imization. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wal- lach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference
Neural Informa- tion Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pp. 6691 67011 2017. URL https: /ttetts:://rrceeee http:///ccceeeeggseeeee hww.eeedddinnn.eeii.e..ee wwtpeepp.....pppree///222222 hocppapprr/201//aaa .tmll (Cited on 45
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9- 15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research. pp. 2424-24223 PMLR, 2019. greffl9a. Satf199..ba. html
URL wtw.:htppp//. httt:::/prooeecinggggg mlr press/v97/ wttpeeedddnna.m.m...rrssss/////
(Cited on I 5
6
16
Michelle Guo, Alireza Fathi, Jiajun Wu, and Thomas Funkhouser. Object-centric neural scene ren- dering. arXiv preprint arXiv:2012.08503, 2020. URL htttps://axx....org/bbb/// 08503 (Cited 55
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zee. Flax: A neural network library and ecosystem for JAX, 2020. URL http:://
(Cited on [7)
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural
computation, 14(8):1771-1800) 2002. (Cited on
Qian Huang, Horace He, Abhay Singh, Yan Zhang, Ser-Nam Lim, and Austin R. Benson. Better set representations for relational reasoning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Process- ing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httttss::ppooceeiinggggooo nwwrrnuuip..... inerreiepp cc// paper 2220//ashh/ 09cc 303399990eeeaa f425d5 9 Abst html (Cited on 5)
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 2(1):193-218
1985. (Cited on 6)
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 1988- 1997. IEEE Computer Society, 2017. doi: 10. 10.1109/CVPR.2017.211 URL https: /oi. org/ 10. 11t.//111....../ 2017. 2115
(Cited
on 6 16
Rishabh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datasets. http:://jthbb.co//cemmm/////// object-datasets/, 2019. (Cited on
T6
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. U //arxiv. wwwxii.co wwwp:iwwwrggga//o/ee. /112. 6980
(Cited on 6 17
Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric
12

---

Robust and Controllable Object-Centric Learning through Energy-based Models
learning from video. arXiv preprint arXiv:2111112259, 2021. URL https::/axxiv.orr focs/2111..
12594 211.11111.111191
(Cited on
Thomas N. Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. In 8th International Conference on Learning Representations, ICLR 2020. Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:// http:ss///pperrvieee net/
forum? d== gax6VtDB. (Cited on 5)
Adam R Kosiorek, Hyunjik Kim, and Danilo J Rezende. Conditional set generation with trans- formers. arXiv preprint arXiv:2006. 16841, 2020. URL http://://aaxxi..o//abbbb////
16841
(Cited on 5)
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics
quarterly, 2(1-2):83-97, 1955. (Cited on 6
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. tutorial on energy-based learning. Predicting structured data, 1(0), 2006. (Cited 5
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jin- dong Jiang, and Sungjin Ahn. SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition. In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
net oog.gpareeeeieeeeeeoooooooooooaa
(Cited on 5
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in Neural Information Processing Systems, 33:1 1525 11538, 2020. (Cited on I
6
16
17
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 060800333,,, 2016. (Cited on [7)
Sindy Löwe, Klaus Greff, Rico Jonschkowski, Alexey Dosovitskiy, and Thomas Kipf. Learning object-centric video models by contrasting sets. arXiv preprint arXiv:2011.10287, 2020. URL
https: org/ htcp//bb/220111 10287 (Cited on
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement 2017. (Cited on 6 16
testing sprites dataset.
Igor Mordatch. Concept learning with energy-based models. arXiv preprint arXiv: 18ki: 1et::8811...244 1.04486 2018. BRLLattps///axxxvo//////// /001.//0/11111111561111 02486 (Cited 5)
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011. (Cited on
Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and compositional generation with latent-space energy-based models. Advances in Neural Information Processing Systems, 34: 34:13997-13510, 2021. (Cited on
Giambattista Parascandolo, Niki Kilbertus, Mateo Rojas-Carulla, and Bernhard Schölkopf. Learn- ing independent causal mechanisms. In Jennifer G. Dy and Andreas Krause (eds.), Proceed- ings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm. Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Re- search, pp. 4033 4041. PMLR, 2018. URL http://proccedings. ml r.pesss///8 parascandol 8a. html] (Cited on 3
Giorgio Parisi. Correlation functions and computer simulations. Nuclear Physics B, 180(3):378- 384, 1981. (Cited on
Sören Pirk, Mohi Khansari, Yunfei Bai, Corey Lynch, and Pierre Sermanet. Online object rep- resentations with contrastive learning. arXiv preprint arXiv:1906.04312, 2019. URL https: w/aaxiv.orggaatt 906. 04312 (Cited on 5)
13

---

Robust and Controllable Object-Centric Learning through Energy-based Models
Evan Racah and Sarath Chandar. Slot contrastive networks: A contrastive approach for representing objects. arXiv preprint arXiv:2007.09229, 2020. URL hattps://arxiiii wwt.i/v.rrg/b///.o/e/o//o/oo abs//007.. (Cited on 5
09294
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pp. 8748-8763. PMLR, 2021. (Cited on !
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06123 2022. URL (Cited on
hittps://axxe org/ /ttg/b//////ooooog///g/////// 2204 2211222244.2222555
Hamid Rezatofighi, Tianyu Zhu, Roman Kaskman, Farbod T Motlagh, Qinfeng Shi, Anton Milan, Daniel Cremers, Laura Leal-Taixé, and Ian Reid. Learn predict sets using feed-forward neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. (Cited on 5)
S Hamid Rezatofighi, Roman Kaskman, Farbod T Motlagh, Qinfeng Shi, Daniel Cremers, Laura Leal-Taixé, and Ian Reid. Deep perm-set net: Learn predict sets with unknown permutation and cardinality using deep neural networks. arXiv preprint arXiv: arXiv: 2018. URL https: (Cited on 55)
00613
Elizabeth S Spelke. Principles of object perception. Cognitive science, 14(1):29-56, 1990. (Cited on I
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL https::/pproeeedngggsss neurips cc/paper 2017/hash/ hta.e22557ccae11bb0333cccaaaaaaa
html
(Cited on 123
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence sequence for sets. In Yoshua Bengio and Yann LeCun (cds.), 4th International Conference on Learning Representa- fions, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL httth:t/taaiii// org/ abs///11..... 06391 (Cited on
Nicholas Watters, Loie Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint arXiv:1901.07017 2019. URL btLhttp:: http//aaxivvv.. wwwp://wwwrraab/////ooaa 07017 (Cited on 4
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Lise Getoor and Tobias Scheffer (eds.), Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 July 2, 2011, pp. 681 688. Omnipress. 2011. htRL:ttps: /tte///cmll..///me//ee///eeeeeee htt.l//c//occpppraee./ 398 immlpaper.......... pdf
(Cited on 4
Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Unsupervised object- level representation learning from scene images. Advances in Neural Information Processing Systems, 30..4444842288888 2021. (Cited on 5)
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander Smola. Deep sets. In Isabelle Guyon, Ulrike von Luxburg, Samy Ben- gio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach. CA. USA. pp. 3391-3401, 2017 URL http:::tt//ppttt htttp::://ppooeednggg... www.uinggn.eurip.pp httpp//.g....iipp....c/..ppppp wttpppcrpppppcceeeee /207//aa1/ F22e aa
Absteerrrreeeeeeaaaaatata. html (Cited on 3
Shuangfei Zhai, Walter Talbott. Miguel Angel Bautista, Carlos Guestrin, and Josh M Susskind. Set distribution networks: a generative model for sets of images. arXiv preprint arXiv:2006.1077
2020. URL http:::/tsaatt wrw./orggooooo /200 10705 (Cited 5n
14

---

Robust and Controllable Object-Centric Learning through Energy-based Models
David W Zhang. Gertjan J Burghouts, and Cees GM Snoek. Set prediction without imposing structure as conditional density estimation. arXiv preprint arXiv:2010.04109, 2020. URL
https://arxiv otggabs///o0.../ooo 04109 (Cited on
Yan Zhang, Jonathon S. Hare, and Adam Prügel-Bennett. Deep set prediction networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, neuriss CC cc/paper/ (Cited on
BC, Canada, pp. 3207-3217 2019, http:///00...///..////oo/oo//oor
201 20199/ash/6e/// 6e http/////////oeaaaaaeeaaaaaaaaaa
S
15

---

Robust and Controllable Object-Centric Learning through Energy-based Models
A IMPLEMENTATION DETAILS
A.1 DATASETS AND PREPROCESSING
We describe more details about the datasets used in our experiments, and the preprocessing steps we applied during training and evaluation.
CLEVR We use the CLEVR JJohnson et al., 2017) dataset from the Multi-Object Datasets li- Each data example in the CLEVR dataset contains rendered image
brary (Kabra al. 2019 ! of a scene with set of up to 10 objects. The multiple objects in the scene can possibly occlude each other, and each object can be rendered with different shapes (e.g., cube, sphere, cylinder). col- ors (Red, Cyan, Green, Blue, Brown, Gray, Purple, Yello), sizes (small, large), materials (rubber, metal), and positions (æ and coordinates). Following (Greff et al. 2019 Locatello et al. 2020), center crop of the image followed by resizing to 128 128 is applied first, then we transform the RGB values to the range [-,
Multi-dSprites We use the Multi-dSprites dataset from the dSprites (Matthey et al. 2017) dataset. More specifically, we use the variant with grayscale background and colored sprites done (Greff et al., 2019 Locatello et al. 2020). Each data example in the Multi-dSprites dataset contains a scene with up objects, and each object can be rendered with different shapes (ellipse, square, hear), colors (HSV space), scales (in [0.5, 1)), positions (x and coordinates, in [0, 1)) and orientations. Following (Greff et al. 2019
Locatello al. 2020), we keep the resolution 64 x 64 and transform
the RGB values to the range
Tetrominoes We use the Tetrominoes dataset from the Multi-Object Datasets library (Kabra al., 2019). Each data example in the Tetrominoes dataset contains scene with exactly 3 tetris pieces with a black background. The tetris pieces can be rendered with different shapes (19 different shapes), colors (Yellow, Purple, Red, Blue, Green, Cyan), and positions (x and coordinates). Following (Greff et al. [019 Locatello al. 2020), we keep the resolution 35 x 35 and transform the RGB values to the range
11
OOD dataset variants For the evaluation of our model constructed OOD datasets with unseen object styles and colors, we use the library tadi et al.] 2022).. To apply the random color transformation to the data exam- wttpaviisiionttsfoooo.o..m
2 from the benchmark (Dit-
ples, a color jittering torchvision. transforms ColorJitter (brightness=O. randomly selected object
contrast=0.5 5, saturation-0.5 5, hue=0. 5) is applied in the scene. To apply the random style transformation, neural style transfer is used to transfer the first 100K samples from all datasets. Please refer to Figure. 1 in (Dittadi al., 2022) for qualitative evaluation of the introduced visual effects of object style and color. For more detailed descriptions of the transformations used for constructing the OOD dataset variants, please refer to (Dittadi et al., 2022).
A.2 EGO MODEL ARCHITECTURE
We introduced the detailed neural network architecture of our model used in our experiments here.
Image Encoder We use the same CNN backbone in the Slot-Attention work (Locatello et al., 2020) to encode the visual scene input into a spatial feature map. For CLEVR dataset, the CNN net- work consists of convolutional layers with 5 x kernel size, [64, 64, 64, 64] channels, 1, 1] zero-padding size respectively, and each convolutional layer followed by a ReLU activation fune- tion. Positional encoding is applied to the encoded feature map to provide the representation with positional information for better modeling the consistency between images and latent variables. More details about the CNN backbone architecture on Multi-dSprites and Tetrominoes datasets can be found in (Locatello et al. 2020).
16

---

from the figure Robust and Controllable Object-Centric Learning through Energy-based Models
EGO Module As illustrated in Figure 1 our model consists of several vanilla attention blocks. For CLEVR and Multi-dSprites datasets, we use steps, 0.1 step size and K latent variables. We use attention head, and do not any dropout in the attention layer or the MIPP layer. For Tetrominoes dataset, we found using = cross-attention block would also lead near-perfect results. We optionally learn the parameters of the initial Gaussian distribution of z and found it helps in Tetrominoes dataset. We use D₂ = 64 for all models across datasets, and all MLP layers have the same number hidden units with ReLU activation function.
3 cross-attention blocks, = 3 sampling
Image Decoder We use the spatial broadcast decoder decode the inferred latent variables into scene reconstruction. The decoder consists of several deconvolutional layers, augmented with po- sitional encoding. For detailed description of the deconvolutional layers, please refer to (Locatello)
et al. 2020
A.3 MODEL TRAINING AND EVALUATION
We implemented our model in Jax (Bradbury et al. 2018 and Flax (Heek et al. 2020 We train all models with batch size 128 using the Adam optimizer (Kingma & Ba, 2015) with learning rate 0.0002 for 500K steps. We use the cosine learning rate schedule (Loshchilov Hutter (006) with warmup steps 2500. We clip the gradients to maximum global norm 1, though we generally found that the training process quite stable without gradient clipping. We train our models on 8 Nvidia A100 GPUs, and the training time CLEVR dataset is about 1 day, and within few hours Multi- -Strite and Tetrominoes datasets.
B ADDITIONAL EXPERIMENTS
We provide additional experimental results scene decomposition, scene manipulation, and model ablation.
B. SCENE DECOMPOSITION
We illustrate examples of visualizing the scene decomposition and reconstruction results accompanying the results shown in Figure] 3
each sampling step across datasets in Figure 6 7
We additionally consider visualizing the per-step decomposition results in more extreme case when we have much more sampling steps T. In doing so, we train a model with T = 10 on Tetrominoes dataset and visualize the per-iteration reconstruction results in Figure [3] We
that after 6 iterations, the model already reconstructed the almost complete scene, which can be seen as qualitative evidence for the sufficiency of using a small number of sampling steps in the latent space.
Besides the ARI scores for evaluating the segmentation performance in unsupervised object dis- covery task, we additionally evaluate our model in terms of other metrics, mean squared error in reconstruction (MSE) and mean segmentation covering (mSC). Detailed description of the metrics are introduced in (Dittadi et al. 2022). We report the results in Figure. TO) We see that EGO achieves comparable performance in terms of the reconstruction quality compared to other methods, and the segmentation accuracy in terms of mSC remains across different datasets consistent with the observation on ARI scores, where EGO shows competative or superior performance.
B.2 SCENE MANIPULATION
We also show the per-iteration reconstruction results for the examples shown in Figure (a.b). In the first few iterations, the model recovers the global scene and then seeks for the updates to satisfy the constraints of latent representation given by the joint energy function.
B.3 MULTI-MODALITY AND MULTI-STABILITY
We present qualitative visualization of the multi-modaa posterior in EGO. Similar to (Greff et al. 2019) construct visual scene input with multiple possible explanations, and display the per-
17

---

Robust and Controllable Object-Centric Learning through Energy-based Models
Latent Lotent
Var.nt
Reconstr action
Mask
Latent Var.
Latent Var. 2
Latent Var.
Latent Var.
Latent Var.
Latent Vor.
Latent Var.
fer
Figure 6: Scene decomposition and reconstruction results at each sampling step on CLEVR.
latent variable reconstructions in Figure. We can see that without distinguishable clues like color and position, we sample a of different segmentation configurations explain the ambiguous visual scene input, showing the multi-modality in the learned posterior.
B.4 MODEL ABLATION
We include more model ablation results, studying the effeets of magnitude of noise in sampling, number of attention blocks L, dropout rate in EGO MLP, dropout in EGO attention layer, and gradient clipping norm on Tetrominoes. We plot the distribution of the ARI scores for each configu- ration of the considered hyperparameters, grouped by the number of sampling steps T in z-axis. We show the results in Figure [2 As can seen from the results, our proposed model is robust to almost any configurations of the hyperparameters, where near-optimal performance be attained in each setup.
18

---

Robust and Controllable Object-Centriy Learning through Energy-based Models
Reconstruction
Latent
Latant
Itr
tar
to 4
Reconstruction
Mask
Latent Var.
Latent Var:
Latent Var: 3
Latent Var.
Latent Var.
Latent Var. 6
Kar
tor
Her
Figure Scene decomposition and reconstruction results at each sampling step on Multi-dSprites
19

---

Robust and Controllable Object-Centric Learning through Energy-based Models
Secoretuctin
Vanc
Latrer
Var
Lawe
Laberr
Linert
Figure 8: Scene manipulation: Combining objects from different images into novel scene using learned energy function.
Figure 9: Scene manipulation: Removing specific object from scene using learned energy function.
1e:2
1.0
0.8
MSE
mSc
0.6
CLEVR
Mult-dSprites Tetrominoes
CLEVR
Muti-dSprites Tetrominoes
EGO
Slot-Altention
MONet
GENESIS
EGO
Slot-Allention
MONet
GENESIS
Figure 10: Evaluation of reconstruction quality in MSE and segmentation accuracy in mSC.
Figure 11: Visualization of samples from multi-modal posterior from EGO on Tetrominoes.
20

---

Robust and Controllable Object-Centric Learning through Energy-based Models
1.0
1.0
0.9
0.9
0.7
0.7
weights
0.01
0.0
0.5
0.4
0.4
10
10
number steps
number of steps
(a) Number of sampling steps T.
(b) Magnitude of noise.
1.0
1.0
0.9
0.9
88
0.7
0.7
0.6
0.6
MLP dropout rate
0.5
0.0
0.1
0.4
0.4
10
10
number stops
number of stops
(c) Number attention blocks L.
(d) Dropout rate MLP.
1.0
0.9
0.9
88
0.7
0.6
0.6
Attention dropout rate
gradient norm
10.0
0.4
0.4
10
3
10
number steps
number of steps
(e) Dropout rate attention layer.
(l) Gradient clipping norm.
Figure 12: Model ablation results: ARI scores by varying individual hyperparameter choices, grouped by the number sampling steps.
21

---

Robust and Controllable Object-Centric Learning through Energy-based Models
Reconstruction
Mask
Latent Var.
Latent Var. 2
Latent Var.
Latent Var. 4
Iteer
ntr 1
ster7
iter 10
Figure 13: Scene decomposition and reconstruction results at each sampling step on Tetrominoes.
22


Reviews:
[{"review_id": "e833YDQJR7V", "reviewer": "Reviewer_DPAM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to learn object-centric representation with an energy-based model. This energy model takes a visual scene and a set of object-centric latent variables as input. Latent variables are inferred from visual observations through gradient-based MCMC sampling, where the gradient is derived from the energy function.\n\nTo formulate a permutation-invariant energy function over a set of objects, this paper proposes two variants. The first one is to simply sum over individual energy functions from the set of latent variables. Another is to compose them with an attention mechanism.\n\nThe proposed model is firstly evaluated on the unsupervised object discovery task, where it demonstrates good segmentation performance. It also exhibits the ability to perform scene manipulation given the compositionality nature of the energy function. Lastly, the proposed model shows robustness to different hyper-parameters and generalization to unseen data.", "summary_of_the_review": "This paper formulates a permutation-invariant energy model to effectively learn object-centric representation from visual scenes.  The proposed method is inspiring by modelling object-centric learning from a different perspective. The major concern is its structural similarity with a previous method and its marginal performance gain over baselines. It is strongly suggested to demonstrate its capacity for more challenging scenarios using more informative metrics. \n", "strength_and_weaknesses": "Strength:\n\n(1) The energy-based modelling of object-centric learning minimizes assumptions made on the generative process, relieving neural networks from complicated designs to model visual scenes.\n\n(2) The compositionality nature of the energy function makes it possible to perform scene manipulation with learned object-centric latent variables.\n\nWeakness:\n(1) In the Introduction, there is a lack of clear motivation to develop energy-based models compared with existing object-centric learning frameworks such as VAE based (MONet, IODINE, etc) or Attention based (SlotAtt). What are the key advantages of energy-based models over others? \n\n(2) The proposed method somehow takes a similar formulation with SlotAtt, especially EGO-Attention. In experiments, the performance gain is also marginal compared with SlotAtt.\n\n(3) The datasets used for unsupervised object discovery are quite simple. Previous approaches have already achieved very high ARI scores on them. Actually, the ARI metric seems not informative because it’s very easy to get high scores. More general metrics such as AP are suggested, as also pointed out in a very recent work [1]. \n\nIn addition, evaluation and comparison on more challenging datasets such as ShapeStacks [2], ObjectsRoom [3], or even real-world datasets are very desirable. Otherwise, it’s hard to demonstrate the clear advantages of EGO over other baselines. \n\n[1] Promising or Elusive? Unsupervised Object Segmentation from Real-world Single Images, NeurIPS 2022.\n[2] ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking, ECCV 2018.\n[3] Neural Scene Representation and Rendering, Science 2018.\n\n(4) In the evaluation of generalization capability on unseen object styles, it is claimed that the created OOD data have altered colours, textures and shapes. However, it is not demonstrated qualitatively or quantitatively how and to what extent colour jitter transformation and neural style transfer can alter the colour, texture and shape distributions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "dfxwZuBflN", "reviewer": "Reviewer_1fbA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel encoder that, given an image, learns object-centric slot representations using the EBM framework. The encoder computes an image-conditioned energy function $E(\\mathbf{z}_1, \\ldots, \\mathbf{z}_K, \\mathbf{x}; \\boldsymbol{\\theta})$ that assigns an energy value to slot representations $\\mathbf{z}_1, \\ldots, \\mathbf{z}_K$. It then applies gradient-based MCMC sampling on the energy function to obtain the slots. Crucially, the proposed energy function $E(\\mathbf{z}_1, \\ldots, \\mathbf{z}_K, \\mathbf{x}; \\boldsymbol{\\theta})$ is invariant to the ordering of the slots $\\mathbf{z}_1, \\ldots, \\mathbf{z}_K$. This order-invariance is achieved via two approaches: 1) by summing a per-slot energy function, OR, 2) by adopting a transformer that treats the slots as a set. The learning signal for training the model parameters comes from a reconstruction objective where a pixel-mixture decoder reconstructs the image by decoding the learned slots.", "summary_of_the_review": "I think the paper is novel and interesting and should be shared with the community. I support the acceptance of the paper.", "strength_and_weaknesses": "### Pros\n\n1. Energy-based modeling is an important framework and the question of whether this approach can be used to learn slots is an interesting one. In this regard, the experiment results are useful — showing that it can perform comparably or somewhat better than the previous methods — and thus has the potential to spark a new line of exploration in the object-centric learning community.\n2. The proposed encoder is almost entirely a transformer. I believe this is a good thing given that the transformer is currently a default model in the community and is perhaps more well-understood than a specifically-tailored architecture. This may also make their encoder applicable more generally beyond the image domain unlike some previous models like IODINE.\n3. Useful ablations showing the model to be robust to various hyper-parameters.\n4. The scene addition and subtraction via summing/subtracting the energy functions is interesting/surprising.\n5. Generalizes well to OOD number of objects and unseen object style.\n6. Has a probabilistic interpretation and thus the slots may capture the uncertainty.\n\n### Weaknesses/Questions\n\n1. While I believe in the promise of EGO’s probabilistic interpretation (i.e. modeling uncertainty in the partially-observable environments) there is a lack of some experiment that explores this aspect. Authors may consider showing a qualitative experiment of multi-stability in Tetris dataset similar to that shown in the IODINE paper.\n2. Some unclear aspects:\n    1. In the conclusion section, there is a line saying “minimal assumptions on the generative process…”. This line is a bit unclear to me. Is this referring to the fact that the model does not apply a prior on the latents like IODINE does? \n    2. Slot-attention performs an explicit spatial attention and mean-pooling to bind low-level information into object slots. The proposed model does this implicitly via the transformer. What are the implications of this design difference?\n    3. How does the gradient flow backward via the MCMC sampling steps and to the EGO’s parameters.\n3. Another unexplored question is whether the encoder can be adopted to deal with complex scenes in the same way as slot attention could be adopted in SLATE or SAVi.\n\n### Minor Comments/Questions\n\n1. I am wondering why IODINE is not shown in the property prediction experiment and also why GENESIS is omitted from the ARI result in Table 1.\n2. The bar colors of various models can be made consistent across Fig. 2 and Fig 5(a).\n3. The word ‘controllable’ in the title could be confusing to some because it might imply steerable/conditional slots such as that learned by SAVi. As I understand, ‘controllable’ seems to be about controlling scene generation and not controlling the slot representation itself.\n4. In the section on ‘Increasing the number of objects’, it seems that the referred figure should be 4(c) and not 4(b). Also, in Fig. 4(c), the Y-axis range may be set to 0-100 to better highlight the fact that the performance deterioration is not large when testing on more objects.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ULODxdrtpY", "reviewer": "Reviewer_tTco", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces an energy-based model object-centric learning pipeline.\nEnergy functions are designed to be permutation invariant.\nExperiment results show that scenes are decomposed reasonably well.\nEnergy function algebra leads to meaningful scene editing results.", "summary_of_the_review": "  - While the EBM model is not as versatile as the generative object-centric learning model, I agree that it could be a strong object-centric learning candidate.\n  - However, I hope the author can explain more about their assertion about their main advantages over previous works.", "strength_and_weaknesses": "  - Strength\n    - The use of EBM is novel.\n    - The segmentation performance is better than slot-attention\n  - Weakness\n    - The paper mainly focuses on segmentation performance leaving the reconstruction quantity untouched.\n    - While the generative object-centric learning model can sample objects with various appearances, EBM does not support the generation of any form (at least not trivial).\n    - While the beta-VAE-based model encourages feature disentanglement, EBM does not explicitly encourage such behavior.\n    - And the above weakness are not discussed.\n    - It is not entirely clear to me the meaning of \"need for specifically-tailored neural network architecture or excessive generative modeling assumption\".\n    Could the author explain, for example, how IODINE is specifically-tailored or has excessive generative modeling assumptions?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MPz0recPWh", "reviewer": "Reviewer_vs6t", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a EGO, a neural module for object-centric representation learning. EGO casts the inference of object-centric latent variables as energy-based modeling. It uses Langevin MCMC to sample latent variables consistent with the given input image. Simplicity of the neural network architecture is emphasized. Empirical evidence of the validity of the approach is provided in the form of segmentation and property prediction on the Multi-Object Datasets environments. Additional experiments evaluating hyperparameter choices, robustness, and generalization are provided as well. Overall, the performance of EGO appears to be comparable to the most similar baseline Slot Attention.\n", "summary_of_the_review": "Overall, I believe this work is not yet ready for publication due to various incorrect/unjustified claims and limited novelty of the contributions (as described under Weaknesses). More justification and evidence is needed as to A) why this EBM-based approach constitutes a promising new direction for object-centric representation learning and B) whether EGO is truly a generic module for learning object-centric representations in other architectures/contexts as claimed. I believe with these improvements, the work could be a valuable contribution.\n\n---\nUpdate after rebuttal: I have increased my score from 3 --> 6 to reflect my new perspective of this work after discussion and improvements. See comment thread below for more details. ", "strength_and_weaknesses": "Strengths\n========\nThe core idea, which I believe is casting slot inference as sampling slots consistent with an image as scored by an energy function, seems to be a novel and interesting approach to this task.\n\nThe presentation of the paper is solid.\n\nWeaknesses\n========\nI have concerns with various claims made throughout the paper.\n\n- **[EGO is…] “conceptually simple”/“Without the need for specially-tailored neural network architectures”** This is provided as one of the main contributions of the paper. I disagree that EGO is any simpler than the key baseline, Slot Attention. The “variant” of EGO explored in this paper (Algorithm 1) and Slot Attention are fairly similar methods in terms of the neural modules employed and the use of an iterative inference algorithm.\n\n- **“Without the need for … excessive generative modeling” / “Minimal assumptions on the generative process”** When comparing EGO with previous generative object-centric models like GENESIS and GENESIS-v2 [1], it is important to consider that these models support unconditional generation of novel scenes because they have a prior distribution over slots. EGO sacrifices this capability in favor of a simpler approach to slot inference. Also, Slot Attention, the most relevant baseline, does not make any generative modeling assumptions either.\n\n- **“We show that EGO can be easily integrated into existing architectures”** This is also provided as one of the main contributions of EGO, but it is not actually shown outside of the specific encoder-decoder object representation learning context. No other examples of how EGO can be used are provided. In fact, EGO's ability to discover objects seems to hinge on the use of the spatial broadcasting decoder (SBD) as an inductive bias to encourage each latent slot to fixate on a distinct scene object, in the same manner as the baselines (see [2]). Demonstrating how EGO can be used in a variety of diverse contexts to learn slots that fixate, without supervision, on objects would be one way to improve the paper. \n\nI struggle to see the novel conceptual insights or new capabilities enabled by EGO. One supposedly new capability --- compositional scene editing --- is explored in the experiments, but I believe the baseline object-centric methods can also support this to some degree? It would be good to compare against the performance of at least the key baseline, Slot Attention, at this. Also, I believe IODINE is capable learning a multi-modal distribution over slots (like EGO)? This capability is not empirically explored in the paper, however.\n\nIssues of instability [3] and difficulty with more realistic scenes [4] for object-centric representation learning methods (particularly those based on the SBD) have been demonstrated in the literature, yet these issues are not discussed in this work. I could not find any discussed limitations.\n\nOverall, the segmentation results seem to be comparable to Slot Attention, with only marginal demonstrated improvements in segmentation on Multi-dSprites. Perhaps conducting experiments on more challenging datasets would reveal a larger gap in performance with respect to baselines?\n\nReferences\n========\n1. Engelcke, Martin, Oiwi Parker Jones, and Ingmar Posner. \"Genesis-v2: Inferring unordered object representations without iterative refinement.\" Advances in Neural Information Processing Systems 34 (2021): 8085-8094.\n2. Singh, Gautam, Fei Deng, and Sungjin Ahn. \"Illiterate dall-e learns to compose.\" International Conference on Learning Representations. 2021.\n3. Chang, Michael, Thomas L. Griffiths, and Sergey Levine. \"Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation.\" arXiv preprint arXiv:2207.00787 (2022).\n4. Karazija, Laurynas, Iro Laina, and Christian Rupprecht. \"Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation.\" arXiv preprint arXiv:2111.10265 (2021).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster