Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Models
Authors: Amir Mohammad Karimi Mamaghan
Samuele Papa
Karl Henrik Johansson
Stefan Bauer
Andrea Dittadi
Venue: ICLR 2025 Poster (2025)
Keywords: Object-centric Learning
Foundation Models
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We evaluate and clarify the benefits and trade-offs of object-centric models compared to alternative approaches including foundation models, and analyze the benefits of object-centric bias

Abstract: Object-centric (OC) representations, which model visual scenes as compositions of discrete objects, have the potential to be used in various downstream tasks to achieve systematic compositional generalization and facilitate reasoning. However, these claims have yet to be thoroughly validated empirically.
Recently, foundation models have demonstrated unparalleled capabilities across diverse domains, from language to computer vision, positioning them as a potential cornerstone of future research for a wide range of computational tasks.
In this paper, we conduct an extensive empirical study on representation learning for downstream Visual Question Answering (VQA), which requires an accurate compositional understanding of the scene. 
We thoroughly investigate the benefits and trade-offs of OC models and alternative approaches including large pre-trained foundation models on both synthetic and real-world data, ultimately identifying a promising path to leverage the strengths of both paradigms. 
The extensiveness of our study, encompassing over 600 downstream VQA models and 15 different types of upstream representations, also provides several additional insights that we believe will be of interest to the community at large.

Published as a conference paper at ICLR 2025
EXPLORING THE EFFECTIVENESS OF OBJECT-CENTRIS REPRESENTATIONS IN VISUAL QUESTION ANSWERING: COMPARATIVE INSIGHTS WITH FOUNDATION MODELS
Amir Mohammad Karimi Mamaghan' Stefan Bauer
Samuele Papa"
Karl Henrik Johansson 1.3
Andrea Dittadi 1.5,6
'KTH Royal Institute Technology Universityy of Amsterdam 'Helmholtz AI
Digital Futures MPI for Intelligent Systems, Tübingen
Techhical University of Munich
ABSTRACT
Object-centric (OC) representations, which model visual scenes as compositions of discrete objects, have the potential to be used in various downstream tasks to achieve systematic compositional generalization and facilitate reasoning. However, these claims have yet to be thoroughly validated empirically. Recently, foundation models have demonstrated unparalleled capabilities across diverse domains, from language computer vision, positioning them potential cornerstone of future research for wide range of computational tasks. this paper, we conduct an extensive empirical study on representation learning for downstream Visual Ques- tion Answering (VQA), which requires an accurate compositional understanding of the scene. We thoroughly investigate the benefits and trade-off's of OC models and alternative approaches including large pre- pre-trained foundation models on both synthetic and real-world data, ultimately identifying a promising path to leverage the strengths of both paradigms. The extensiveness of our study, encompassing over 600 downstream VQA models and 15 different types of upstream representations, also provides several additional insights that we believe will be of interest to the community at large.
INTRODUCTION
Object-centric (OC) learning aims to represent the physical world's inherent structure, assuming visual scenes consist entities or objects and employing this as an inductive bias for neural networks (Goyal et al., 2019; Locatello et al., 2020; Lin al., 2020b; Burgess al., 2019; Singh et al., 2022a;b; Seitzer et al., 2023; Wu et al., 2023; Jiang et al., 2023; Löwe et al., 2023). Applied in various domains like visual reasoning (Chen et al., 2021b; Ding al., 2021a; Wu al., 2022; Ding al., 2021b; Santoro et al., 2017; Webb et al., 2023; Mondal et al., 2023) and image and video generation (Chen al.. 202 la; Singh al., 2022a;b; Elsayed al., 2022; Jabri al., 2023), these representations play crucial role in capturing compositional and causal structures, with the potential to improve the generalizability and interpretability of Al algorithms (Lake et al., 2017; Schölkopf et al., 2021: Brady et al., 2023; Kim al., 2023b; Jung al., 2024). Breaking down scenes into conceptual elements corresponding to causal factors aligns with the idea that causal models play crucial role in achieving human-level generalization (Pearl, 2009; Peters al., 2017; Mansouri et al., 2023; Liu et al., 2023).
While OC representations thus provide way of representing the state of visual scene, comprehensive understanding of these representations is still an ongoing exploration. Recently, there have been several works on evaluating OC representations. Some studies evaluate object-centric models in terms of reconstruction and segmentation accuracy, and quantify the quality and information content of object representations via a downstream object property prediction task (Dittadi et al., 2022; Papa et al., 2022). Arguing that major goal of representation learning is to facilitate downstream tasks, Yoon et al. (2023) focuses on the evaluation the representations in reinforcement learning, which requires thorough understanding of the environment in terms of objects and the relations between them. However, a more direct quantification of the role of object-centric representations for reasoning is still missing.

---

Published as a conference paper at ICLR 2025
In the rapidly evolving landscape of deep learning, foundation models, often characterized by self- supervised and large-scale pre- pre-training, have demonstrated unparalleled capabilities in generalization and zero-shot learning, showcasing their prowess in tasks across diverse domains from natural language processing to computer vision (Kirillov et al., 2023; Wen et al., 2023; Touvron et al., 2023; Brown et al., 2020; Chowdhery et al., 2023; Huang et al., 2023; Borsos et al., 2023; Yang et al., 2023; Jumper et al., 2021; ; Rombach et al., 2022). Despite their widespread success, foundation models have not been comprehensively analyzed and compared with OC models.
In this paper, we aim to take one step towards understanding the relevance of object-centric represen- tations in the era of foundation models by evaluating them on visual reasoning tasks through simple framework. More specifically, our main contributions are the following:
We conduct a large empirical study on representation learning for downstream Visual Question Answering (VQA) (Antol al., 2015; Johnson et al., 2017) on three synthetic and two real-world multi-object datasets. In our extensive evaluation, we train overall 684 downstream transformer models for VQA, involving 15 different types of upstream representation models, ranging from VAEs to state-of-the-an art OC methods to large pre-trained foundation models.
We identify and investigate the trade-offs between large foundation models and OC models. We observe that, without any fine-tuning or hyperparameter adjustment, foundation models perform comparably to the top pop-performing OC models. On the other hand, they typically require more compute and larger downstream models. We find that applying the OC inductive bias to foundation models effectively achieves the best of both worlds, reducing the downstream computational needs while achieving comparable or better performance and obtaining more explicit representations.
We present several additional insights regarding, among other things, the correlation between performances on VQA and a simpler downstream task, the relationship between upstream and downstream performance of OC models, the effect of training set size on VQA performance, the difference between different question types, and a deeper analysis of the global (single-vector) representations of traditional VAEs.
RELATED WORKS
Object-Centric Learning. Object-centric (OC) learning has gained attention over the past few years (Goyal et al., 2019; Singh et al., 2021; 2022a;b; Seitzer et al., 2023; Wu et al., 2021; 2023; Jiang et al., 2023; Eslami et al., 2016; Crawford & Pineau, 2019; Kosiorek et al., 2018; Jiang et al., 2019; Dittadi & Winther, 2019; Engelcke al., 2019; 2021; Lin et al., 2020a;b; Greff et al., 2017;
Gregor et al., 2015; Yuan et al., 2019; Locatello et al., 2020; Burgess et al., 2019; Jabri et al., 2023;
Chen et al., 2021a; Kipf al., 2019; 2022; Elsayed et al., 2022; Löwe al., 2023; Kori al., 2023; Sajjadi et al., 2022; Daniel & Tamar, 2022). OC models aim to learn visual representations without supervision by treating each image as a composition of objects. Among them, Slot Attention (Locatello et al., 2020) stands out as a popular model and a crucial component in several recent state-of-the of-te-of-th models. Numerous enhancements have been proposed, including improvements of the Slot Attention module (Biza et al., 2023; Jia et al., 2023; Majellaro & Collu, 2024) or adding additional modules on top (Kim et al., 2023a), using a transformer decoder instead of the original mixture-based decoder (Singh et al., 2021; 2022b), replacing the CNN backbone with pre-trained model (Seitzer et al., 2023), and integrating diffusion models with Slot Attention (Jiang et al., 2023; Wu et al., 2023; Jabri et al., 2023).
Evaluation of Oejeet-Centric Representations. OC methods have been applied in several works in visual reasoning (Chen et al., 2021b; Ding et al., 2021a; Wu al., 2022: Ding et al., 2021b; Santoro et al., 2017; Webb et al., 2023; Mondal et al., 2023; Driess et al., 2023) and some of these works try to address the Visual Question Answering task itself. Ding et al. (2021a) propose new method to address the VQA in videos and run a transformer over slots obtained from pre-trained MONet (Burgess et al., 2019), and text tokens of the question, and applies an MIP on top to predict the answer. The method proposed by Wu et al. (2022) reasons over the object representations of Slot Attention to model spatiotemporal relationships, and predicts future object states. Their framework is also applied to VQA downstream task.
In addition, a few works focus more specifically on the evaluation of OC representations. Weis et al. (2021) designs benchmark over only OC video models and analyzes their performance over different tracking scenarios relevant to natural videos. Yang & Yang (2024) evaluates OC representations and shows their shortcomings in segmenting objects in real-world dataset. Dittadi et al. (2022) evaluates
2

---

Published as a conference paper at ICLR 2025
the representations indirectly in the context of reconstruction loss, segmentation quality, and object property prediction, and analyzes their generalization and robustness. Papa et al. (2022) uses the same evaluation metrics dataset with complex textures. Yoon et al. (2023) evaluates the representations on more practically relevant downstream tasks in reinforcement learning and includes a wider range of methods compared to the previous works. Finally, Driess al. (2023) demonstrates the suitability of OC representations in planning and VQA tasks within a robotic environment. However, the assessment done on single OC baseline in the presence of Large Language Model (LLM) and the VQA setup is restricted to particular scenarios, In our work, we are interested in investigating the suitability of different types of representation, including object-centric ones, for reasoning tasks. To this end, we opt to more directly assess the suitability of representations for reasoning through VQA.
EXPERIMENTAL SETUP
In this section, we provide an overview of our experimental setup. First, we introduce the downstream task used in our experiments evaluate representations. We then outline the upstream representation models, the datasets and metrics, and the concrete setup for learning the downstream task.
3.1 VISUAL. QUESTION ANSWERING
In this paper, evaluate the performance attainable Visual Question Answering (VQA) task (Antol et al., 2015) from different representations of the visual scenes. With questions that can involve any number of objects from just one to all the objects in an image, VQA presents more demanding challenge compared to object-level tasks. It requires a thorough understanding of the image and complex reasoning about objects and their relationships. We therefore choose VQA as benchmark to directly assess the suitability of different representations for reasoning.
Given an image, the task is to answer a natural language question such as "How many tiny green objects are made of the same material the purple cube?". The questions are usually about how many objects there are, whether an object with a specific attribute exists, and what properties they have in relation another of objects in an image. The possible answers include "yes", "no", and various numerical and categorical values. Further details are provided in Appendix B.2.
Our framework, summarized in Fig. 1, consists of: (1) an upstream model that provides high-level representations of an image, (2) fixed pre-trained text embedding model that converts question in natural language to text embeddings, and (3) a downstream model that takes as input the image representation and the text embedding and outputs the answer to the question. We will elaborate on each part in the following sections.
3.2 UPSTREAM MODELS
To investigate OC representations, we consider three types of representations: global, fixed-region, and object-centric. Global representations encode the image into a single vector which contains high-level information about the image. Fixed-region representations consist of a fixed number of vectors, each loosely corresponding to specific region within the image. OC representations consist of set of vectors, each ideally corresponding
single object.
The evaluated models are summarized in Table 1. As OC baselines, we use MONet (Burgess et al., 2019), SPACE (Lin et al., 2020b), and Slot Attention (SA) (Locatello et al., 2020). We also include ResNer SA (Biza et al., 2023), an improved version of the standard SA autoencoder with the following modifications: the backbone is re- placed by ResNet34 (He et al., 2016) without pre-training; a larger feature map resolution is used in both the encoder and the decoder; and the slot initializations are learnable. We also consider STEVE (Singh et al., 2022b), a state-of-the-ar OC video model for complex and naturalistic videos. STEVE is a more robust ver- sion of SLATE (Singh al., 2022a) combining the SLATE decoder with standard slot-level recurrent model.
Table 1: Summary of models included in our study.
Model
Representation Type
Training Regime
DINDx2 (Oqub et al., 2023)
Fined-Rteiion
Pre-tmining
MAE (Ile al. 2022) 2022)
Fixed Region
Pre-traniing
CLIP (Ralford
Fixed Region
Pre-criningg
YQ-AB (Rombach
2022)
Region
Pre cr-irinnng
(Rombach ReoNe:50 (Il et al., 2016)
2022)
Pised-Rtcjion
Pre-tmining
Fixed Region
Pre-traning
CNN (zambaldi et al., 2018)
Pixed Megion
End-co-Ean Training
MuliCNNN (Kipf al., 2019)
Object-Cnnric
End-a-Fas Tmining
Skoe Attention (Locatello ot al., 2000)
Chjean-Centie
Dataset-Spciilee Piestmining
RecNe: Slot Attention (Biza 2023)
Object- Cenett-Centiic
Dataset- Specific Pre-sruningg
MONe (Burgess
2019)
Ubjec--Camnie
Ditaset Specitic Pre Pre-trinnng
SPACE in 20206)
Objean-Cemrie
Ditaves Snacit-Specilic Pre-tmining
STEVE (Singh
2022b)
Object
Dataset Specific Pre craining
DINOSAURV2 (Saizer et 2023)
Cbjert42auric
Ditases- Ditase-Specile Pie-triining
VAE (Winters et al., 2019)
Global
Ditaset Specitic Pre craining
3

---

Published as a conference paper at ICLR 2025
No
Output
Output MLP
Transformer Encoder
Proyected Representation tatisssttiiins
Projection Leyer
Layer Positional Encoding
Reprevent
Upetream Model
Text Embedding Module
Input
gray large
some material?
Figure I: An overview of framework. Starting from an image and question, first extract image and question representations, by applying the upstream model and the text embedding module, respectively. The obtained representations are then passed to the projection layer and then, positional encoding is applicd to the text representations. Next, both are concatenated and a transformer model is applied to the combined sequence. Finally, The answer the question is obtained produces probability vector over all possible answers.
MLP that takes the transformed value of the CLS token and
To adapt STEVE to images, we simply consider images as I-frame videos, following the authors' recommendation. Furthermore, as the last OC baseline, we consider DINOSAUR (Seitzer et al., 2023), state-of-the-art OC image model, and replace its pre-trained DINO (Caron et al., 2021) backbone with DINOv2 (Oquab et al., 2023) we refer to this model as DINOSAURv2. Following previous work, we also consider multiple CNNs (Kipf al., 2019; Watters al., 2017; Yoon al., 2023), each CNN being expected to capture one object in the image, and train all of them end-to-end together with the downstream model-wa refer to this approach as MuliCCNN
As classic benchmark for fixed-region representations (Yoon et al., 2023; Seitzer et al., 2023), we include a pre-trained ResNer50 (He et al., 2016). We also utilize two pre-trained autoencoders from Latent Diffusion Models (LDM) (Rombach et al., 2022). one with KL regularization and the other one with a vector quantization layer, both with scaling factor of 16. We refer to them KL-AE and VQ-AE, respectively. Additionally, we use pre-trained versions of DINOv2 (Oquab et al., 2023; Darcet et al., 2023), Masked Autoencoder (MAE) (He et al., 2022), and CLIP (Radford al., 2021), all of which have achieved outstanding performance of tasks. Following previous works (Santoro et al., 2017; Zambaldi et al., 2018; Yoon et al., 2023), we also implement& a simple CNN, and train it end-to-end with the downstream model. As bascline providing global representation, we follow Dittadi et al. (2022) and consider variation of vanilla variational autoencoders (VAE) (Kingma al., 2014; Rezende et al., 2014) with a broadcast decoder (Watters et al., 2019). Finally, to better understand the models' performances, we include, for each dataset, baseline trained only on questions, without any information from the corresponding images.
backbone in diverse array
Training. Regarding the training of upstream models, we have 3 different types of models: pre- trained foundation models, pre-trained dataset-specific models, and end-to-end models. Pre-trained foundation models have been trained on large-scale datasets and tasks, serving as the basis for transfer learning in various applications. DINOv2, MAE, CLIP, VQ-AE, KL-AE, and ResNet50 belong to this category that we use off-the-shelf without fine-tuning, all experiments. Dataset-specific pre-trained models are first trained with autoencoding objective (only using images, disregarding the questions) on the same dataset that will be used for VQA with their original training procedures and hyperpa- rameter choices. They subsequently frozen, similarly to foundation models. MONet, SPACE, SA, ResNet SA. STEVE, DINOSAURV2. and VAE are in this category. Finally, end-to-end models are trained from scratch alongside the downstream model solve the VQA task directly. CNN and Mul- tiCNN belong to this category. For more information about the upstream models, see Appendix A.1.
3.3 DATASETS
Synthetic. We utilize three popular multi-object datasets in our experiments: Multi Sulti-dSprites (Matthey et al., 2017), variation of CLEVR (Johnson et al., 2017) with 6 objects known CLEVR6
4

---

Published as a conference paper at ICLR 2025
(Greff et al., 2019; Locatello et al., 2020; Dittadi et al., 2022), and CLEVRTex (Karazija al., 2021) which is a variation of CLEVR featuring synthetic scenes with diverse shapes, textures and mhpped materials. This dataset is closer to real-world datasets in terms of visual complexity. To analyze the effect of training data size, we consider 4 different training data sizes in Multi-dSprites with 40k, 80k, 160k, and 320k unique images, with the 320k version as the default version. Each image in the multi-object datasets consists of a background with fixed color and set of objects with different properties. Originally, only CLEVR contains questions associated with each image. To make the other datasets applicable to the same VQA task, we augment them with several questions (roughly 40-50) for each image, by adapting the question generation mechanism of Johnson et al. (2017) to each dataset. We use this to generate different types of questions, with possible answers including "yes", "no", natural numbers up to the maximum number of objects, and all possible values of object properties. For more details about the datasets and question generation, see Appendix B.
Real-World. Additionally, we extend our results to real-world scenarios with the VQA-12 (Goyal et al., 2017; Antol et al., 2015) and GQA (Hudson & Manning, 2019) datasets. VQA-v2 consists of open-ended questions about images sourced from MS COCO 2014 (Lin al., 2014), real-world multi-object dataset. Recently, COCO has been increasingly utilized in object-centric literature (Seitzer et al., 2023; Jiang et al., 2023; Wu et al., 2023), marking a significant advancement in complexity compared to datasets typically used to evaluate object-centric models. GQA is a large- scale dataset designed for visual question answering, focusing on compositional reasoning over real-world scenes. VQA-v2 features a diverse range of possible answers. To align with the same classification pipeline used for synthetic datasets, we limit the questions to yes/no and questions with numeric answers ranging from 0 to 14, This results in total of 17 possible answers. For more details about the dataset and the preprocessing, see Appendix
3.4 METRICS
Following previous works (Ding et al., 2021a; Wu et al., 2022), we measure performance in our VQA downstream task by average accuracy. As metrics for the upstream OC models, we use the Mean Squared Error (MSE) of the reconstructions, and 3 segmentation metries: the Adjusted Rand Index (ARI) (Hubert & Arabie, 1985), Segmentation Covering (SC) (Arbelaez et al., 2010), and mean Segmentation Covering (mSC) (Engelcke et al., 2019). All of these metrics have been extensively used in previous studies (Locatello et al., 2020; Dittadi et al., 2022; Singh et al., 2022b; Biza et al., 2023). See Appendix C for more details about the metrics.
3.5 FRAMEWORK SETUP
Our VQA framework, depieted in Fig. 1, closely follows Ding et al. (2021a). Given a pair (1r, 4) where E R3xH*W denotes image of height H and width W, and q denotes question, the task is to select the correct answer from the set of all possible answers. Since the number of answers in each dataset is relatively small, it is not necessary generate text tokens the answers, and similarly to Ding et al. (2021a), we stick to the simpler case of predicting a probability vector over all possible answers in the dataset. Another key aspect consider is that our primary focus is on evaluating representations while the method for generating questions and the format of the answers hold less significance in this context.
Image and Text Representations. Given a data pair (ar, q), the upstream model computes the image representation z. In global representations, 2 is a vector of size Dytos- In OC models, Natots x Doc matrix where Nalots is the number of slots in the OC model. In fixed-region representations, z is a feature map of size Pи x Pw x Dj, where the first two dimensions correspond to the feature map sizes and the third dimension is the size of the representation. For more details about obtaining image representations from the upstream models, see Appendix A.I.
To embed the question q from text format to word embeddings, we use the Text-to-Text Transfer Transformer (T5; Raffel et al., 2020) which outputs a matrix of size Niokens x Demb representing the embeddings of the tokens in the question where the dimensions correspond to the number of tokens and the embedding size, respectively. See Appendix A.2 for more details.
Unifying Image Representations. In order to use different types of image representations in the downstream model which follows transformer architecture and will be explained later on, it is necessary to unify the format of representations and convert them to sequence. We use & as it is for OC representations since each slot corresponds an object, and can be separately used an item in
5

---

Published as a conference paper at ICLR 2025
CLEVR
CLEVRTeX
Mults-dpprites
Accuracy (Overail)
Reed-Reglor
Object-Centric COO Global
40
20
26
DINOv2
MMB
CUP
RoAE
Resfeddo
CNN
M
NONet
SPACE
STEVE
DINeSAulRo
VAE
Base tne
DINON
CUP
KL-AE
Basteedo
CNN
MuRCOW
SA
Restdan SA
SPACE
STEVE
DINOSAURVE VkF Busein
DINOX MAE
CUP
Vo-KE
K0-AE
Rustes0
CNN
Muniic
SA
Rusthn SA
MONU
SPACE
STrV
DINOSAiliv AilsAilli2
Baselin
Figure 2: Average accuracies on the VQA downstream task for different upstream representation models on synthetic datasets, when using T-15 as the downstream model. The bars indicate means and 95% confidence intervals with 3 random seeds, when available.
the sequence. We reshape fixed-region representations by flattening the spatial dimensions, obtaining a matrix of size Pn Pw x Drc.
For global representations, we split the single vector into K vectors of size Dotob/ K. Here, K roughly corresponds to the number of slots in an OC model. In other words, we treat as a sequence of length K with a latent size of D globb/ K. While we observed this to be the most effective option in terms of downstream performance, we also considered three alternative approaches. The first applies 2-layer MIP to and subsequently splits the output similarly to what described above; the second method treats the single vector as one token in a sequence of length 1; the third splits z into D Dolob sequences of size 1. All these approaches showed poorer downstream performance, and in addition, the last one is computationally expensive due to a large sequence length.
Downstream Model. Following previous works on VQA (Ding et al., 2021a; Devlin et al., 201 8; Lu et al., 2019), we use a transformer-basee architecture (Vaswani et al., 2017). Having t and the refor- matted text and image representations, we apply separate linear layer on each to make the latent size and the embedding size equal, and we get t' and 2', respectively. Then, to inform the downstream model about the order of words, following Ding et al. (2021a), we augment each vector in 2' and with a 2-dimensional one-hot vector indicating whether the input is from the image representation both will become D Modelt- We introduce trainable vector CLS E TR. Dmodel, akin to the CLS token in BERT (Devlin et al., 2018), to generate classification results. In the final step, we concatenate z', t', and the CLS token and pass this sequence through a transformer with Nr layers. An MLP classifier then takes the transformed CLS token and outputs a probability vector over all possible answers.
apply a sinusoidal positional encoding layer to t'. Additionally,
the text, and the latent size for
3.6 LIMITATIONS
While goal is to execute robust and informative experimental study address the research questions identified in Section 1, it's important to acknowledge inherent limitations related to datasets, models, and evaluations. The foundation models in our study are trained with different objectives and on datasets that differ in size and characteristics, making direct comparisons with OC models more difficult. However, is important emphasize that this is first and foremost a pragmatic study aimed at deriving practical, actionable insights into representation learning for downstream reasoning tasks. To achieve this, we empirically investigate a diverse range of approaches directly available in the literature, without significant modifications, and evaluate their effectiveness for these tasks.
4 EXPERIMENTAL RESULTS
Our key findings are presented in this section. In our main set of experiments, assess how different model representations perform on the Visual Question Answering (VQA) downstream task defined in Section 3. We primarily focus on results from synthetic datasets where we have unified question- generation procedure and access to underlying ground-truth factors. Our downstream models are transformer encoders with 2, 5, and 15 layers, which we refer as T-n with n the number of layers. We train all combinations of upstream representation models and downstream classifiers, which amounts to 684 downstream models, with the cross-entropy loss. We provide all implementation details in Appendix A and additional experimental results in Appendix D.
Reerrducing our experimental study requires approximately 13 GPU years Nyidia 1100 GPUs.

---

Published as a conference paper at ICLR 2025
CLEVRTex
dSpritsss
D/NO22
MAE
Accuracy (Overali)
-4
03
CUP
KLIVE
VO-AE
ResheeS0
70
Reshict SA
64-
MONel
SANIE
STEVE
D-NOSAURR
0.0 1:21 Downstream GFLOPs
1:03
VAE
Downstream GFLOPs
Downstream GFLOPs
Figure 3: Average accuracies of different models vs, downstream GFOS across different datasets. Points along the x-axis represent T-2, T-5, and T-15, respectively. For pre-trained models, only I seed is available. For other models, the results are averaged over 3 random seeds and the shaded areas indicate 95% confidence intervals.
We carried out an extensive set of experiments with numerous baselines. To improve clarity, we have organized the results into key points and summarized the main takeaways. In the following, we report average results and confidence intervals over 3 random seeds, except for foundation models, where only seed is available. We omit MONet's results on CLEVRTex due to its suboptimal performance, consistent with similar experimental results by Papa et al. (2022). When extending to real-world datasets, we keep only the pre-trained foundation models and top-performing OC models, excluding other upstream representation models due to their poor performance. Additionally, we report the results on VQA-v2 only with T-2 as the downstream model due to a degradation in performance observed when increasing the number of transformer layers (see Appendix A.3 for more details). Finally, unless explicitly mentioned, the Multi-dSprites version featured in the plots the one comprising 320k unique images.
4.1 MAIN FINDINGS
Performance of Large Foundation Models. Fig. shows the overall accuracy for different upstream models across different synthetic datasets with T-15 as the downstream model, which generally achieves the best performance across synthetic datasets and upstream models. We observe that large foundation models, i.e., DINOv2, CLIP, and MAE, without any fine-tuning perform comparably well or the best on all datasets, although not by a large margin. However, when considering compute requirements, the picture appears more nuanced. In Fig. 3. which shows overall accuracy against the GFLOPs used for downstream training, we observe that some OC models achieve comparable performance to large foundation models with significantly less compute, making them more appealing under a limited compute budget.
It's important to emphasize the differences in model sizes and training data between foundation models and OC models. As shown in Table 2 in Appendix A. best-performing OC models like STEVE and ResNet SA are much smaller than their counterparts in the foundation model group and are specifically trained on the datasets studied in this work, which are significantly smaller than those used for training foundation models. Additionally, foundation models require substantial computational resources and significant engineering for pre-training, which are beyond our control. Therefore, carefully analyzing the effects of these factors in studies like ours is challenging.
Efffcct of Object-Centric Bias. DINOSAURV2 (Seitzer et al., 2023), which consists of a pre-trained DINOv2 with Slot Attention applied downstream, allows to explore the effect of applying the OC bias on a foundation model. Comparing the results of DINOv2 and DINOSAURv2 in Figs. and on CLEVRTex and Multi-dSprites, we observe that DINOSAURv2 outperforms DINOv2 while requiring significantly less downstream compute. However, on CLEVR, we do not observe the same patterns, as DINOSAURE performs slightly worse than DINOv2. After experimenting with different hyperparameter choices on CLEVR, we found that DINOSAURV2 is highly sensitive to these choices, which likely explains this suboptimal performance.
Additionally, by looking at Fig. 14 (Appendix D.3) which shows the overall accuracies on different downstream model sizes, we observe that on T-2, DINOv2 exhibits inferior performance compared to DINOSAURv2 CLEVRTex, Multi-dSprites, and GQA. However, as we scale up the downstream model, starting from T-5, DINOv2 almost matches DINOSAURV2. This indicates that DINOv2 representations do contain the relevant information for the downstream task, but they seem to be less explicit and less readily usable, necessitating a larger downstream model compared to DINOSAURV2 to extract the required information effectively (Eastwood et al., 2023).
7

---

Published as a conference paper at ICLR 2025
GQA
GQA
VQA-12
Accuracy (Overall)
60
50
5.1
5.0
VQ-ÉE
48
40
410
ResNedSo
46
30
STEVE
44
DINOSARR22
Ragiin
Cipect-Cantrc Global
40
0.5
1.0
1.5
DINOvZ
MAE
CUP
VQ-AE
KL. AE
ReeNe:50
ResNet SA
STEVE
DINOSAURV2
Baseline
DINOV2
MAE
CUP
VQ AE
KL-AE
ResNet5o
Restint SA
STEVE
DINDSAURV2
Baseline
Downstream GFLOPs
Figure 4: Left: Average accuracies of different models w.r.t. downstream GFLOPs GQA. Points along the x-axis represent T-2, T-5, and T-15, respectively, For pre-trained models, only one seed is available. For other models, the results are averaged over 3 random seeds and the shaded areas indicate 95% confidence intervals. Middle & Right: Average accuracies GQA and VQA-v2 for different upstream models, with T-15 and T-2 as the downstream models, respectively. The bars indicate means and 95% confidence intervals with 3 random seeds, when available.
Performance of Other Upstream Models. In Fig. discernible pattern emerges among upstream models. Generally, OC models consistently outperform other models except large foundation models. Smaller pre-trained models (VQ-AE, KL-AE, and ResNet50) tend to perform worse. Notably, on CLEVRTex, this trend is less pronounced, as most OC and pre-trained models struggle due to the dataset's 's complexity. Consistent with prior studies (Yoon al., 2023), End-to-end CNN and MultiCNN models consistently score the lowest, and are followed by the global representation of VAES. Additionally, on CLEVR and CLEVRTex, several models show only slight improvement over the baseline, which relies solely on the question without any image-related information.
Within foundation models, DINOv2 and MAE consistently outperform others, with CLIP ranking as the third-best model probably due to its relatively smaller size. Looking at Table 2, we observe that while the good performance DINOv2, MAE, and CLIP can likely be attributed the size their backbone, there appears to be no clear trend explaining the performance gap among smaller models.
Real-world Data. To investigate whether our findings hold in real-world scenarios, we conduct the same experiments the GQA (Hudson & Manning, 2019) and VQA-v2 datasets (Goyal et al., 2017; Antol et al., 2015), two well-establishe benchmarks for the Visual Question Answering task. Fig. left shows the overall accuracy of different upstream models on GQA vs. downstream GFLOPs, and the overall accuracy on GQA and VQA-v2 with T-15 and T-2 as the downstream models, respectively, Comparing DINOSAURV2 with DINOv2, we observe the same pattern as in synthetic datasets. Additionally, the performance trends across different models are consistent with those observed in synthetic datasets, further validating our primary conclusions and suggesting that the findings are robust across both real-world and synthetic datasets.
Takeaway. While large foundation models can perform comparably to the best-performing OC models without any fine-tuning or hyperparameter adjustments, they generally require larger downstream models and more compute, presumably because their representations are less explicit than OC representations. On the other hand, the performance more complex datasets. Learning OC representations top of foundation model (see, e.g.. DINOSAURv2) can be a viable solution to get the best of both worlds.
many OC models drops on
4.2 ADDITIONAL INSIGHTS
Property Prediction vs VQA. We additionally evaluate the representations on property prediction, much simpler downstream task wherein the objective is to predict object properties from the representations. We adopt the same setup as Dittadi et al. (2022) (see Appendix A.4 for further details). In Fig. 5, observe a strong correlation between accuracy of this simple task most properties, and downstream VQA performance. This demonstrates that models capable of accurately predicting object properties excel on more challenging tasks like VQA. Therefore, performance simple tasks like property prediction can be a useful evaluation metric for model selection. For the complete correlation results, see Appendix D.I.
8

---

Published as a conference paper at ICLR 2025
CLEVRTIX
MuttidSpriee
Comreatton
SPACE
24
Figure 5: Correlation between property prediction accuracy (reported separately by object property) and overall VQA accuracy.
Figure 6: Overall VQA accuracy with T-15 as down- stream model, plotted against ARI and MSE for dif- ferent OC models on CLEVRTex.
Takeaway. Overall, performance on object property prediction strongly correlates with VQA performance, indicating that much simpler tasks can be used to guide model selection.
Upstream vs. Downstream Performance. Fig. 6 depicts the relationship between upstream perfor- mance metrics and downstream VQA accuracy of OC models when using T-15 as the downstream model on CLEVRTex. Notably, STEVE exhibits the worst reconstruction MSE among OC models but achieves the second-best accuracy on VQA. This is not necessarily surprising: while Dittadi et al. (2022) observed a negative correlation between MSE and downstream performance, Papa et al. (2022) later showed this to no longer hold in the presence of textured objects. However, higher ARI was shown to be predictive of better downstream performance. This appears not to hold in our case, as ResNet Slot Attention attains the second-best ARI but does not perform well in the VQA downstream task, while STEVE has a poor segmentation performance while achieving high accuracy. Further investigations needed to shed more light on these trends, allowing for more robust upstream model selection strategies. For results on more upstream metrics and other datasets, see Appendix D.2.
Takeaway. Upstream metrics such as ARI (segmentation) and MSE (reconstruction) are not good predictors of downstream performance on our VQA task.
Effeet of Training Size. Fig. 7 shows the percentage decrease in the overall error rate of different models on Multi-dSprites when the training size increases from 40k to 320k unique images. Notably, with approximately 8x more data, most upstream models exhibit similar improvements, typically around 20-40%, regardless of their initial performance. CLIP is notable exception, showing an increase in overall error rate of up to 50%. Additionally, end-to-end models (CNN and MultiCNN) and VQ-AE show only minimal improvement compared to the other models. See Appendix D.4 for further results, including raw accuracies and additional dataset sizes between 40k and 320k.
Takeaway. Except for end-to-end models which show minimal improvements, all other models generally exhibit similar performance gain with larger downstream training sizes.
Consistency of the Results Across Question Types. The average Spearman rank correlation between VQA accuracy different question categories is 0.96 for CLEVR, 0.98 for CLEVRTex, 0.88 for Multi-dSprites, and 0.92 for VQA-v2. This suggests that the average VQA accuracy results shown in Figs. and 4 are consistent across question categories. In Appendix D.5, Fig. 17 shows that for these datasets, the rank correlations are consistently high for all pairs question categories, and Figs. 18 28 illustrate the complete results separately by question category. In contrast, on GQA, the correlation between accuracies of different question types is notably weaker. In particular, Compare and Logical questions exhibit low correlation with other categories. This likely due to their reduced dependency on visual representations and heavier reliance on linguistic cues (Liu et al., 2022).
Delving deeper into the results for each category, it becomes apparent that on VQA-v2. Number questions, which require recognizing quantities in the image, are harder for all the models compared to YexNoo questions. On GQA, Query questions which demand complex, open-ended reasoning, consistently prove to be the most difficult, whereas Verify questions are the easiest. Moreover, models perform similarly on Logical and Compare questions, indicating that these question types depend much less the visual information provided by the models. On synthetic datasets, we observe that Count questions, which necessitate an understanding of the existence of multiple objects with specific properties, are generally the most challenging for almost all models. In contrast, Exist questions the easiest, which expected because they check for the existence of a single object with specific properties. Among Compare Integer questions, Equal questions appear to be the most challenging.

---

Published as a conference paper at ICLR 2025
Mulli-S5prtts-320K
Decrease
heddFaggnn Ocpeccentry
16
Accuracy (Dverall)
34-408
D-NOVG-AR
WX 220k
DINOv2
MAE
CUP
Vo-M
Rosteso
o
SPACE
MINSSAUI VA
T-32
Training Time (hours)
Figure 7: Average % decrease in VQA error rate (means and 95% CIs with 3 random seeds) for dif- ferent upstream models on Multi-dSprites. when increasing the training set size from 40k 320k, using T-15 as the downstream model.
Figure 8: VQA accuracy downstream training time Multi-dSprites. Slot Attention and DI- NOv2 are trained on 40k images, while VAE is trained on 320k images for million steps (5x more than the other models).
requiring an exact count two of objects. Finally, Attribute questions, Size questions emerge as the easiest, while there is no specific discernible pattern among other object attributes.
Takeaway. While some question categories are on average more difficult than others, we observe a strong correlation between accuracies across different question categories.
Evaluation Global Representations. Here we further investigate whether the global representa- tions of VAE can match the performance of OC representations when given a significant advantage terms of data and training budget. For this purpose, we train a downstream transformer encoder model with 20 layers (T-20) top of the VAE on Multi-dSprites with 320k unique images for 3 million steps and compare the result with other models trained on Multi-dSprites with the smallest training size (40k), with the smallest downstream model (T-2) trained for the default number of training steps (600k). From Fig. 8, is evident that the performance of T-20 trained on top of the VAE cannot match the performance of T-22 trained top of Slot Attention and DINOv2. In conclusion, even with larger downstream model, more training steps, and a larger training dataset size, global representations of VAEs cannot match the performance of OC models and therefore do not seem ideal for downstream tasks related to objects.
Takeaway. Even with significantly more training data and compute, global representations such those of standard VAEs are far from competitive. This corroborates the common assumption that such representations are not suitable for object-related downstream tasks.
CONCLUSION AND DISCUSSION
In this study, we systematically assess OC representations on downstream reasoning tasks by compar- ing them with foundation models and various benchmarks across three synthetic and two real- world multi-object datasets. Our primary focus is the VQA task, which requires a precise compositional understanding of images, objects. and their relationships. Our findings indicate that foundation mod- els perform comparably to OC models without requiring fine-tuning or hyperparameter adjustments. However, they are significantly larger and demand greater computational resources. Overall, this points to complex trade-off between model classes. Still, one can benefit from both worlds by applying the OC bias to foundation models. Beyond performance comparisons, we emphasize the importance of downstream evaluations, they provide quality and align well with the goals of OC learning particularly capturing the compositional properties of scenes. We encourage future research to adopt such evaluations to better understand the strengths and limitations of different approaches.
pragmatic assessment of representation
While our study covers commonly used OC datasets, they are limited to static images. Future work could extend these analyses to video data, where dynamic scene understanding presents additional challenges. Additionally, further research could explore the effects of fine-tuning foundation models, both with and without OC inductive biases, as well as systematically study the generalization capabilities of OC models in other tasks, such causal inference.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like to thank Thomas Kipf, Max Horn, and Sindy Löwe for the helpful discussions and comments. This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP), funded by the Knut and Alice Wallenberg Foundation, and by the Helmholtz Foundation Model Initiative, supported by the Helmholtz Association. A.D. acknowledges support from G-Research, The computations were enabled by the Berzelius resource, provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre, and by the Gauss Centre for Supercomputing e.V. www.gauss-centre.coo which provided the required computing time through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUPITER IJUWELS (Jülich Supercomputing Centre, 2021) Jülich Supercomputing Centre (JSC).
ETHICS STATEMENT
In this work, we are taking a step in the direction of systematically analyzing and understanding the reasoning capabilities of deep learning systems with a particular focus on object-centric models, which benefits both the research community and society. We do not see a negative societal impact of this work beyond what is brought about by general advances in machine learning.
REPRODUCIBILITY STATEMENT
All reproduction details regarding the models, hyperparameters, implementation. and training and evaluation procedures are provided the main text and appendix. Specifically, information about the upstream models, downstream VQA pipeline, and downstream property prediction pipeline available in Appendix A. We use publicly available implementations of the models, which are properly referenced in the paper. All datasets are publicly available, and details about each dataset, as well as the question generation procedure, can be found in Appendix B. Finally, the metrics are described and properly referenced in Appendix C.
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In IEEF International Conference on Computer Vision, pp. 2425-2433, 2015.
Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5):898-9166 2010.
Ondrej Biza, Sjoerd van Steenkiste, Mehdi S. M. Sajjadi, Gamaleldin Fathy Elsayed, Aravindh Mahendran, and Thomas Kipf. Invariant slot attention: Object discovery with slot-centric reference frames. In International Conference on Machine Learning, 2023.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: A language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
Jack Brady, Roland S Zimmermann, Yash Sharma, Bernhard Schölkopf, Julius von Kügel- gen, and Wieland Brendel. Provably learning object-centric representations. arXiv preprint arXiv:2305. 14229, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. In Advances in Neural Information Processing Systems, pp. 1877-1901, 2020.
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:190..11390 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE International Conference Computer Vision, pp. 9650-9660, 9660, 2021.
11

---

Published as a conference paper at ICLR 2025
Chang Chen, Fei Deng, and Sungjin Ahn. Roots: Object-centric representation and rendering of 3d scenes. Journal of Machine Learning Research, (2(1):11770-11 2021a.
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan- Yee K Wong, Joshua B Tenenbaum, and Chuang Gan. Grounding physical concepts of objects and events through dynamic visual reasoning. In International Conference Learning Representations, 2021b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In AAAI Conference on Artificial Intelligence, pp. 3412-3420, 2019.
Tal Daniel and Aviv Tamar. Unsupervised image representation learning with deep latent particles. arXiv preprint arXiv:2205. 2022.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309. /6588, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matt Botvinick. Attention over learned object embeddings enables complex visual reasoning, In Advances in Neural Information Processing Systems, pp. 91 12-91224, 2021a.
Mingyu Ding, Zhenfang Chen, Tao Du, Ping Luo, Josh Tenenbaum, and Chuang Gan. Dynamic visual reasoning by learning differentiable physics models from video and language. In Advances in Neural Information Processing Systems, pp. 887-899, 2021b.
Andrea Dittadi and Ole Winther. Lavae: Disentangling location and appearance. arXiv preprint arXiv:/909, 1813, 2019.
Andrea Dittadi, Samuele S Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Locatello. Generalization and robustness implications in object-centric learning. In International Conference Machine Learning, pp. 5221-5285, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. /929, 2020.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzuan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378. 2023.
Cian Eastwood, Andrei Liviu Nicolicioiu, Julius Von Kügelgen, Armin Kekić, Frederik Träuble, Andrea Dittadi, and Bernhard Schölkopf. Dci-es: An extended disentanglement framework with connections to identifiability. In International Conference on Learning Representations, 2023.
Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. Savi+t: Towards end-to-end object-centric learning from real-world videos. In Advances in Neural Information Processing Systems, pp. 28940-28954, 2022.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Gener- ative scene inference and sampling with object-centric latent representations. arXiv preprint arXiv:1907.13005 /3052, 2019.
12

---

Published as a conference paper at ICLR 2025
Martin Engeleke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. In Advances in Neural Information Processing Systems, pp. 5085-8094, 2021.
M. Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, 2016.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint arXiv:1909. 10011199990993,,. 2019.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In IEEE Conference Computer Vision and Pattern Recognition, pp. 1904-6913, 2017.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization, In Advances in Neural Information Processing Systems, 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pp. 2424-2433, 2019.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In International Conference on Machine Learning, pp. 1462-149 1471, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Ziyan Huang, Haoyu Wang, Zhongying Deng, Jin Ye, Yanzhou Su, Hui Sun, Junjun He, Yun Gu, Lixu Gu, Shaoting Zhang, et al. Stu-net: Scalable and transferable medical image segmentation models empowered by large-scale supervised pre-training. arXiv preprint arXiv:2304.06711 2023.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 2:193-218 1985.
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Puttern Recognition, pp. 6700-6709, 2019.
Allan Jabri, Sjoerd van Steenkiste, Emiel Hoogeboom, Mehdi S. M. Sajjadi, and Thomas Kipf. Dorsal: Diffusion for object-centric representations of scenes al. arXiv preprint arXiv:2306.0800 2023.
Baoxiong Jia, Yu Liu, and Siyuan Huang. Improving object-centric learning with query optimization. In International Conference on Learning Representations, 2023.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world mod- els with scalable object representations. International Conference on Learning Representations, 2019.
Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. In Advances in Neural Information Processing Systems, 2023.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-2910, 2017.
13

---

Published as a conference paper at ICLR 2025
Jülich Supercomputing Centre. Juwels cluster and booster: Exascale pathfinder with modular supercomputing architecture at juelich supercomputing centre. JLSRF, 7(A138), 2021. doi: 10.17815/jksff77-1883
John Jumper. Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn T'unyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-558) 2021.
Whie Jung, Jaehoon Yoo, Sungjin Ahn, and Seunghoon Hong. Learning to compose: Improving object-centric learning by injecting compositionality. arXiv preprint arXiv:2405.00066. 2024.
Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevrtex: texture-rich benchmark for unsupervised multi-object segmentation. arXiv preprint arXiv:2111.10206 2021.
Jinwoo Kim, Janghyuk Choi, Ho-Jin Choi, and Seon Joo Kim. Shepherding slots to objects: Towards stable and robust object-centric learning. In IEEE Conference Computer Vision and Pattern Recognition, pp. 19198-19207, 2023a.
Yeongbin Kim, Gautam Singh, Junyeong Park, Caglar Gulcehre, and Sungjin Ahn. Imagine the unseen world: A benchmark for systematic generalization in visual world models. arXiv preprint arXiv:2311.09064 2023b.
Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes. International Conference on Learning Representations, 2014.
Thomas Kipf, Elise Van der Pol, and Max Welling. Contrastive learning of structured world models. arXiv preprint arXiv:1911. 12244, 2019.
Thomas Kipf, Gamaleldin Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. International Conference on Learning Representations, 2022.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv preprint
2023.
Avinash Kori, Francesco Locatello, Fabio De Sousa Ribeiro, Francesca Toni, and Ben Glocker. Grounded object-centric learning. In International Conference Learning Representations, 2023.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In Advances in Neural Information Processing Systems, 2018.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv preprint 181: 1.0982,, 2018
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision, pp. 740-755. 2014.
Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving generative imagination in object-centric world models. In International Conference Machine Learning, pp. 6140-6149, 2020a.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference Learning Representations, 2020b.
14

---

Published as a conference paper at ICLR 2025
Chen Liu, Jonas Pfeiffer, Anna Korhonen, Ivan Vulié, and Iryna Gurevych. Delving deeper into cross-lingual visual question answering. arXiv preprint arXiv:2202.07630, 2022.
Yuejiang Liu, Alexandre Alahi, Chris Russell, Max Horn, Dominik Zietlow, Bernhard Schölkopf, and Francesco Locatello. Causal triplet: An open challenge for intervention-centric causal representation learning. arXiv preprint arXiv:2301.088 1660,, 2023.
Francesco Locatello, Dirk Weissenbom, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems, pp. 11525- 11525-11538, 2020.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and- siio--and-language tasks. In Advances in Neural Information Processing Systems, 2019.
Sindy Löwe, Phillip Lippe, Francesco Locatello, and Max Welling. Rotating features for object
discovery. In Advances in Neural Information Processing Systems, 2023.
Riccardo Majellaro and Jonathan Collu. Explicitly disentangled representations in object-centric learning. arXiv preprint 10X14:2401.10148, 2024.
Amin Mansouri, Jason Hartford, Yan Zhang, and Yoshua Bengio. Object-centric architectures enable efficient causal representation learning. arXiv preprint arXiv:2310. 19054 2023.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset, 2017.
Shanka Subhra Mondal, Taylor Whittington Webb, and Jonathan Cohen. Learning to reason over visual objects. In International Conference Learning Representations, 2023.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo. Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po- Huang, Hu Xu, Vasu Sharma, Shang-Wes Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
Samuele Papa, Ole Winther, and Andrea Dittadi. Inductive biases for object-centric representations in the presence of complex textures. arXiv preprint arXiv:2204.084799 2022.
Judea Pearl. Causality. Cambridge University Press, 2009.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pp. 8748 1188-8763,,45 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(1):5485-5511 2020.
Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. In Advances in Neural Information Processing Systems, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, pp. 1278-1286, 2014.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 10684-10695. 2022.
15

---

Published as a conference paper at ICLR 2025
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. Advances in Neural Information Processing Systems, pp. 9512-9524, 2022.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems, 2017.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5)6612-63))) 2021.
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabrins Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the gap real-world object-centric learning. arXiv preprint wrXv:2209.14860, 2023.
Gautam Singh, Skand Peri, Junghyun Kim, Hyunseok Kim, and Sungjin Ahn. Structured world belief for reinforcement learning in pomdp. In International Conference Machine Learning, pp. 9744-9755, 2021.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. In International Conference on Leurning Representations, 2022a.
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. In Advances Neural Information Processing Systems, 2022b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 008.02087 09288, 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning a physics simulator from video. In Advances in Neural Information Processing Systems, 2017.
Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint arXiv:1901.00017, 2019.
Taylor W Webb, Shanka Subhra Mondal, and Jonathan D Cohen. Systematic visual reasoning through object-centric relational abstraction. arXiv preprint arXiv:2306.02500 2023.
Marissa A Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger, and Alexander S Ecker. Benchmarking unsupervised object representations for video sequences. Journal of Machine Learning Research, 22(1): -311233333333,,038 2021.
Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. arXiv preprint arXiv:23/2.08344 2023.
Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Generative video transformer: Can objects be the words? In International Conference on Machine Learning, pp. 1 11307-11318, 2021.
Ziyi Wu, Nikita Dvomik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.05880 2022.
Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. Slotdiffusion: Object-centric generative modeling with diffusion models. In Advances in Neural Information Processing Systems, pp. 50932-50958. 2023.
16

---

Published as a conference paper at ICLR 2025
Yafei Yang and Bo Yang. Benchmarking and analysis of unsupervised object segmentation from real-world single images. International Journal of Computer Vision. pp. 1-37, 2024.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. AAAI Conference Artificial Intelligence, pp. 3081-3089, 2022.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Li- juan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309. 17421, 2023.
Jaesik Yoon, Yi-Fu Wu, Heechul Bae, and Sungjin Ahn. An investigation into pre-training object- centric representations for reinforcement learning. arXiv preprint arXiv:2302.04419, 2023.
Jinyang Yuan, Bin Li, and Xiangyang Xue. Generative modeling of infinite occluded objects for compositional scene representation. In International Conference on Machine Learning, pp. 7222-7231, 2019.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. Relational deep reinforcement learning. arXiv preprint 186.:01806 2018.
17

---

Published as a conference paper at ICLR 2025
A MODELS AND IMPLEMENTATION DETAILS
Here we elaborate on the upstream and downstream models included in this study along with details on the training, the implementation, and hyperparameter choices.
A.1 UPSTREAM MODELS
Here we elaborate all the upstream models we use in experiments and provide details on the implementation, training, and hyperparameter choices.
Implementation & Training Details. Our code is based on the implementations of object-centric models of Dittadi et al. (2022) and we use their implementation of Slot Attention, MONet, SPACE, and VAE. For these models, we use the same set of recommended hyperparameters on CLEVR and Multi-dSprites, and apply the same hyperparameters for CLEVRTex as used in CLEVR. All other models are either re-implemented or adapted from available code. Unless explicitly stated otherwise, except for pre-trained foundation models, we train all the other models with default batch size of 32 for 3 random seeds with Adam optimizer. The training finishes after 500k steps on synthetic datasets and 250k steps on real-world datasets. The reported metrics are then averaged over the seeds to provide comprchensive assessment. Additionally, more information about pre- pre-trained models and the size of the models are shown in Table 2. The details of each model are explained below.
DINOv2. DINOV2 (Oquab et al., 2023), an enhanced version of the DINO (Caron et al., 2021). stands out as a self supervuue ViT-based model designed for training perfoorrrrmm computer vision models without the need for extensive labeled data. It serves as a versatile backbone for diverse tasks, including image classification, video action recognition, semantic segmentation, and depth estimation. Trained on carefully curated dataset comprising 142 million images with a discriminative self-supervised method, DINOv2 excels in producing versatile visual features that transcend specific image distributions and tasks without the necessity for fine-tuning.
Similar to DINO, DINOv2 follows transformer architecture, with a patch size of 14, and is trained with 1B parameters with a self-supervised learning objective, and distilled into series of smaller models that generally surpass the other best available all-purpose features. There are 4 distinct backbone versions, each varying in the number of transformer layers. After experimenting with all 4, considering a balance between the performance and downstream training time, we selected the
variant, denoted as ViT-L/14. We employ this backbone without fin-tuning, and uniformly resize images from all datasets dimensions of 224 x 224 and pass them to the model, generating fixed-region 16 x 16 representations with a channel size of 1024 for each patch. We flatten the spatial dimensions and pass matrix of size 256 x 1024 to the downstream model.
MAE. The Masked Autoencoder (MAE) (He al., 2022) is simple approach for reconstructing an original signal from a partial observation. In this approach, the image is divided into non-overlapping patches, with 75% of them randomly masked. These patches are then fed into ViT-based encoder, which converts the partially observed input into a latent representation. Next, a lightweight decoder, which uses the representation and mask tokens, reconstructs the original image. The model is trained on ImageNet- KK (Deng et al., 2009) by minimizing the Mean Squared Error (MSE) between the reconstructed and original input in the pixel space.
There are various pre-trained MAES available. differing in model sizes. In our framework, we utilize the pre-trained ViT-L/16. We resize the input images to 224 224 and pass them to the encoder without masking. This generates a fixed-region representation sequence of length 197 (196 corresponding different regions in the image and one corresponding to the output of the CLS token) with a latent size of 1024 for each sequence. We pass the obtained representation matrix without any modifications to the downstream model.
The Contrastive Language-Iman Pretraining (CLIP) (Radford et al., 2021) a multimodal model that learns to associate images and corresponding text descriptions. CLIP uses VIT to extract a feature vector that represents the visual content of the image. Similarly, for the text, CLIP uses a transformer-basee model to generate a feature vector that represents the semantic content of the text. These two feature vectors are then projected into a shared embedding space and the cosine similarity between the two vectors is calculated. The model is trained on dataset of 400 million (image, text) pairs collected from a variety of publicly available sources on the Internet, using a contrastive loss function which is symmetric cross-entropy loss over the similarity scores, that encourages the
CLIP.
18

---

Published as a conference paper at ICLR 2025
Table 2: Model sizes and training information of the models.
Model
Model Architecture
# of Paramst
Pre-training Dataset
Dataset Size
DINOv2
ViT-L/14
304M
LVD-142M INet- Ik
142M
MAE
ViT-L/16
303M
1.2M
CLIP
VIT-B/32
87M
WTT-400M
400M
VQ-AE
34.5M
OpenImages v4
9M
KL-AE
34.5M
OpenImages v4
9M
ResNet50
23M
INet-21k
14M
CNN
0.IM
MultiCNN
1.4M
SA
0.9M
ResNet SA
22,4M
MONet
1.6M
SPACE
5.3M
STEVE
17.1M
DINOSAURV2
4M 304M
VAE
19.2M
# of parameters of pre-trained DINOv2 is shown separately. Except for foundation models, of parameters are reported CLEVR.
similarity between the image and text feature vectors be high when they are matching pair and low when they are not.
In our experiments, we utilize pre-trained CLIP image encoder with a ViT architecture denoted as ViT-B/32. Similar DINOv2 and MAE, we resize the input images for all datasets 224 224 and pass them to the encoder. The encoder produces representation of size 50 x 768 (one corresponding the output of the CLS token and the rest corresponding regions in the image) which we utilize directly in the downstream model.
VQ-AE & KL-AE. VQ-AE and KL-AE are two pre-trained autoencoders of the latent diffusion model (LDM) (Rombach et al., 2022). In LDM, they don't 't directly use a diffusion model in pixel space. Instead, to facilitate training quality and flexibility, they apply the models in the latent space of a powerful autoencoder pre-trained on OpenImages (Kuznetsova et al., 2018) with 9M images in adversarial manner. The autoencoder consists of an encoder that downsamples the images by factor f, and a decoder that reconstructs the original image. In order to avoid arbitrarily high-variance latent spaces, they apply two different regularizations: one imposes KL penalty towards a standard normal on the learned latent (KL-AE), similar to a VAE, and the other one uses a vector quantization layer (Van Den Oord et al., 2017) within the decoder (VQ-AE).
constrained computational resourees without compromising
Various pre- tre-trained autoencoders with different downsampling factors are available. In our experi- ments, we explore multiple models with varying downsampling factors and find that factor of 16 is the balancing point between performance and training speed. Therefore, we adopt models with this factor for further analysis. our experiments, we utilize the encoder these two autoencoders off-the-shelf. By applying the encoders, we get a vector of size W/16 H/16 x Djr where Dyr is 16 and for KL-AE and VQ-AE, respectively. Similar to DINOv2, we flatten 2d feature maps and feed them into the downstream model.
ResNet50. ResNet50 He al. (2016) is a deep neural network with 50 layers that has been pre- trained on ImageNet-21k (Deng et al., 2009). It is well-known for its residual learning blocks and serves as baseline in our study. In our experiments, we employ the off the-f ResNet50 model and remove the pooling and fully connected layers at the end. We apply the default ResNet50 transformations on the input image, pass it to the model, and obtain a vector of size W/32 x H/32 x 2048 which we flatten the spatial dimensions and pass to the downstream model.
CNN. CNN (Zambaldi et al., 2018) is small comvolutional neural network that is commonly used in the literature. It consists of few convolutional layers with ReLU activation functions in between, and is trained end-to-end with the downstream model the downstream task. It produces fixed-size
19

---

Published as a conference paper at ICLR 2025
Table 3: Hyperparameters of CNN.
CNN
Dataset
Kernel Size
Stride
Output Channels
Activation Function
CLEVR & CLEVRTex
8
4
32
ReLU
4
2
64
ReLU
2
64
ReLU
3
1
64
ReLU
Multi-dSprites
8
4
32
ReLU
4
2
64
ReLU
3
1
64
ReLUU
fixed-region representation of size 4 x 4 x 64 which we flatten the spatial dimensions and produce vector of size 16 64 and pass it to the downstream model. The hyperparameters of CNN are shown in Table 3.
MultiCNN MultiCNN (Kipf et al., 2019) is an object-centric model consisting of Naiors CNNs, each dedicated to detecting single object within an image. These CNNs operate with non-shared parameters, processing each input image independently. MultiCNN shares the same dataset-specific hyperparameters as the CNN baseline, and similar to CNN. is trained end-to-end using the same training hyperparameters and loss function as the downstream model. The output of each CNN un- dergoes complete flattening, followed by shared linear layer of size 64, resulting in a representation of size Nsiots x 64, which is then passed to the downstream model.
Slot Attention. Slot Attention (Locatello et al., 2020) has become the primary representative for object-centric (OC) learning in recent years. It follows autoencoder setup and begins with a Con- volutional Neural Network (CNN) and is followed by the Slot Attention module. This module refines the initial image features through multiple iterations, turning them into distinct slots representing objects. Each slot is updated using Gated Recurrent Unit (GRU) that takes the current slot and attention information as inputs. After refining, these slots are used to reconstruct the appearance and mask of each object, which are then combined to reconstruct the original image. The model is trained by minimizing the MSE reconstruction loss.
We also employ an improved version of Slot Attention introduced in Biza et al. (2023) which we refer to as ResNer SA. In this improved version, the CNN backbone is replaced by a ResNet34 (He et al., 2016) without pre-training, and larger feature map resolution of 16 on synthetic datasets and 7 on VQA-v2 and GQA is used in both the encoder and the decoder of the model. Furthermore, the initial slots changed into learnable slots. Both the original and improved models trained each dataset with a batch size of 64, a learning rate of 0.0004, a learning rate warmup of 10k steps, and an exponential learning rate decay with a half-life of 100k steps. For ResNet SA, we additionally clip the gradient norm at 0.05 to stabilize training. We follow the same architecture for ResNet SA as in Biza et al. (2023) on synthetic datasets. On VQA-v2 and GQA, we modify the architecture by replacing the initial convolutional layer of ResNet34 with convolutional layer featuring kernel size of 7 and a stride of 4. Additionally, in the decoder, we incorporate two additional transpose convolutional layers at the beginning, each configured with the same hyperparameters as the existing transpose convolutional layers. After training, the learned slot vectors size Nalots X 64 are used as representations in the downstream model for both versions.
The Multi-Object Network (MONet) (Burgess et al., 2019) consists of a recurrent segmen- tation network that generates attention masks that represent the probability of each pixel belonging to each object. For each slot, VAE (the component VAE) encodes the image and the current attention mask, and decodes the latent representation to an image reconstruction the slot and the slot mask. To create the final reconstructed image, the reconstructed images are combined using the attention masks obtained from the segmentation network. The model is trained by an objective function comprising reconstruction loss defined as the negative log-likeliie likelihoodd of spatial Gaussian mixture model (GMM) with one component per slot, where each pixel is modeled independently, and a KL divergence of the component VAE, and additional mask reconstruction loss for the component VAE. The mean of the GMM for each component is used as the representation of each object. The learned representation is vector of size Natats 16 and is directly used for the downstream task.
MONet.
20

---

Published as a conference paper at ICLR 2025
Table 4: Hyperparameters of STEVE.
STEVE
Module
Hyperparameter
Hyperparameter Value
Encoder
Corrector Iterations
2
Slot Size
192
MLP Hidden Size
192
# Predictor Blocks
1
Predictor Heads
Learning Rate
0.0001
Transformer Decoder
# Decoder Blocks
8 4
# Decoder Heads
Hidden Size
192
Dropout
0.1
Learning Rate
0.0003
DVAE
Learning Rate
0.0003
Patch Size
4 x 4 pixels
Vocabulary Size
4096
Temperature Start
1.0
Temperature End
0.1
Temperature Decay Steps
30k
SPACE. Spatially Parallel Attention and Component Extraction (SPACH) (Lin et al., 2020b) pro- vides a unified probabilistic modeling framework that combines the best of spatial attention and scene-mixture approaches. Foreground objects are identified using bounding boxes computed in parallel spatial attention process, and background elements are modeled using a mixture of compo- nents. The model is trained by optimizing the Evidence Lower Bound (ELBO) of the probabilistic model. An additional boundary loss is introduced to penalize the splitting of objects across bounding boxes, addressing potential under- or over- seg--etemetttio issues. SPACE representations are vectors of size 69 x 38 where 69 is the number of slots that is determined by the grid size, and the latent representation of each slot obtained by concatenating all the latent variables which will have a dimension of 38. We utilize this representation directly in the downstream model.
STEVE. STEVE (Singh et al., 2022b) is simple object-centric video model achieving remarkable performance over various complex and naturalistic videos. is a more robust version of SLATE (Singh et al., 2022a), state-of state-of-the-art the-art object-centric model. The model contains two reconstruction paths. The first path uses a discrete VAE encoder to convert the input image into discrete tokens, and then discrete VAE decoder to reconstruct the original image. This path is trained using MSE reconstruction loss. The second path uses a CNN-based image encoder on the input, and the output is fed into recurrent slot encoder that updates slots over time using recurrent neural networks. Finally, slot-transformer decoder, similar to SLATE, is applied to the produced slots to predict and reconstruct the discrete tokens of the input image. This path is trained by minimizing the cross-entropy loss between the original tokens produced by the discrete VAE encoder, and the predicted tokens of the toottrranf decoder.
We incorporate the original implementation of STEVE into our framework. STEVE works videos, considering the input to be a sequence of image frames. Following the authors' recommendation, to utilize it on images, datasets, we train STEVE for 500k steps with an exponential learning rate decay with a half-life of 250k steps and with 30k warm-up steps. For, we employ the same training hyperparameters but reduce the number of steps to 250k. After the training, we use slots of size Notots x 192 obtained from the recurrent slot encoder in the downstream model.
treat images 1-frame videos and pass them to the model. On synthetic
A summary of the model's hyperparameters on all datasets is shown in Table 4. We maintain the original architecture for the CNN backbone and discrete VAE encoder/decoder for synthetic datasets. However, for VQA-v2 and GQA with image sizes of 224 x 224, reduce the feature map dimensions of both the discrete VAE encoder and CNN backbone by half. This adjustment includes changing the stride of the first convolutional layer of the CNN backbone to 4. Additionally, in the discrete
21

---

Published as a conference paper at ICLR 2025
Table 5: Hyperparameters of DINOSAURV2.
DINOSAURv2
CLEVR & CLEVRTex VQA-v2 (COCO) &
Hyperparameter
Multi-dSprites
GQA
Training Steps
250k
500k
250k
Batch Size
64
64
LR Warmup Steps
10k
10k
10k
Peak LR
0.0004
0.0004
0.0004
Exp. Decay Half-Life
100k
100k
100k
VIT Architecture
VIT-L
VIT-L
VIT-L
Patch Size
14
14
14
Feature Dim.
1024
1024
1024
Gradient Norm Clipping
1.0
1.0
1.0
Image Size
224
224
224
Cropping Strategy
Full
Full
Full
Image Tokens
256
256
256
Type
MLP
MLP
MLP
Decoder
Layers
4
MLP Hidden Dim.
1024
512
2048
Iterations
3
Slot Attention
Slot Dim.
256
256
256
MLP Hidden Dim.
1024
1024
1024
VAE encoder, we insert a convolutional layer with a 2 x 2 kernel and stride 2, followed by ReLU activation, after the initial convolutional layer. To accommodate these feature map changes in the discrete VAE decoder, we duplicate the four convolutional blocks and the pixel shuffling layer before final convolutional block.
It is noteworthy that the training of STEVE is not entirely stable, performance metrics begin to degrade after some training time (after roughly 20-40k steps). However, observe that it performs well the downstream task.
observed that upstream
when training the model for longer, Nevertheless, in some seeds, the training fails as the upstream metrics are significantly worse than in other seeds. Therefore, only consider the seeds that are stable and perform the best in terms of reconstruction and segmentation quality.
Additionally, we experimented with a modified version of STEVE using a pre-trained DINOv2 as the CNN encoder backbone. However, it performed similarly or worse than the original STEVE on the downstream VQA. We suspect this may be due to the chosen hyperparameters or an architectural bottleneck in the discrete VAE. As result, decided not to include it in our reported results.
DINOSAUR. DINO and Slot Attention Using Real-world data (DINOSAUR) (Seitzer et al., 2023) an object-centric model designed to bridge the gap between object-centric models and real-world data. It consists of encoder that extracts features from the input data, a slot attention module that groups the extracted features into slots, and a decoder that reconstructs the extracted features. Their approach can be considered similar to SLATE (Singh et al., 2022a) and STEVE (Singh al., 2022b), but with the difference of reconstructing global features from a pre-trained Vision Transformer (Dosovitskiy et al., 2020) instead of local features from VQ-VAE (Van Den Oord et al., 2017).
We adapt the original implementation of DINOSAUR into our framework and replace the pre- pre-trained DINO backbone with pre-trained DINOv2 (Oquab et al., 2023), which we refer to as DINOSAURV2. Similar to the original training procedure of DINOSAUR, the input images are resized to 224 x 224 and we train DINOSAURV2 for 500k steps on CLEVRTex, and 250k steps on CLEVR. Multi- dSprites, COCO images of VQA-v2, and GQA images. The training uses a learning rate of 0.0004, a learning rate warm-up of 10k optimization steps, and an exponentially decaying learning rate schedule. Furthermore,
clip the gradient norm at 1 to stabilize training. After the training, use the slots of size Nalots x 256 obtained from the Slot Attention module in the downstream model. The full hyperparameters of the model provided in Table 5.
22

---

Published as a conference paper at ICLR 2025
VAE. We train variational autoencoder (VAEs) (Kingma et al., 2014) with broadcast decoder (Watters et al., 2019) as a baseline that learns global representations. We use the broadcast VAE implementation of Dittadi et al. (2022) with the same architecture and hyperparameter choices. For CLEVRTex, we use the same hyperparameters as in CLEVR. The latent size is selected to be 64 times the number of slots used when training an object-centric model the same dataset. As explained in Section 3.5, we divide the flat representation vector into Natota vectors of size 64, and pass them to the downstream model.
A.2 TEXT EMBEDDING MODULE
As our text embedding module, we use Text-to-Text Transfer Transformer (T5) (Raffel et al., 2020), a transformer-batt language model developed by Google AI Language. It is capable of performing wide range of natural language processing tasks such text classification, question answering, summarization, and translation. The model is trained on the colossal, cleaned version of Common Crawl's s web crawl corpus (C4), #06-gigabyte corpus of text data using pretext task called Text- to-Text Transfer Transformer (TS), which involves converting a given input text to a target output text. We utilize the available implementation in Hugging Face's Transformers library. T5 comes in different sizes and we use the T5-base tokenizer and encoder which produces the representations of size 768 for each token.
A.3 DOWNSTREAM VQA SETUP
Architecture and Hyperparameters. We use transformer-based architecture (Vaswani et al., 2017) which is the standard downstream architecture for the VQA task (Ding et al., 2021a; Devlin et al., 2018; Lu et al., 2019). We utilize the original PyTorrc implementation of the transformer. We first project the image and the text representations with two separate linear layers with a 126 size and a dropout of 0.1. Then, we augment the image and text vectors with a 2-dimensional one-hot vector indicating whether the input is from the image representation or the text embeddings. A sinusoidal positional encoding is then added to the text embeddings. We introduce a trainable vector CLS E R 128, akin to the CLS token in BERT (Devlin et al., 2018), to generate classification results. The image and text representations, along with the CL.S token, are concatenated and passed through a transformer encoder with a d m_deel of 128 and a hidden dimension of 128. The transformed value of the CLS token is passed through a classifier MLP that generates a probability vector over all possible answers in each dataset. The MLP consists of 2 linear layers of size 128, A normalization layer, dropout of 0.1, and a ReLU activation function are applied in between the layers.
Training. On synthetic datasets, all downstream models trained with batch size of 128 and learning rate of 0.0001 for 600k training steps, with the cross-entropy loss. However, when using DINOv2 and MAE as upstream models, it is infeasible to keep the current batch size with only one GPU due to the substantial sequence length of the representations. To ensure a fair comparison, gradients are accumulated, and the optimizer is applied every training step with reduced batch size of 32. Consequently, the downstream model is trained for 2.4 million steps, 4 times the default number of steps.
On GQA, the downstream models are trained for 900k steps with a batch size of 32 and a learning rate of 0.0001, using the same loss function as for synthetic datasets. On VQA-v2, we train all downstream models with the same batch size and loss function for 300k steps. Furthermore, on VQA-v2, we use a learning rate of 0.0001 for T-2 and T-5, and 0.00005 for T-15. However, we find that T-2 outperforms the other downstream models, with performance degrading significantly as the number of layers increases which is due to overfitting caused by the much smaller training size of VQA-v2 compared to other datasets in our study. As a result, only report results for T-2 on VQA-v2.
A.4 DOWNSTREAM PROPERTY PREDICTION SETUP
Here we assess representations by training downstream models to predict ground-truth object proper- ties from these representations. Following the approach outlined by Dittadi et al. (2022) for object property prediction on synthetic datasets. in summary, we employ a single downstream model f to predict the properties of each object independently. To be more specific, for OC models, we apply if on each slot representation, i.e. ÿN = f where 25 is the kth slot representation and ÿx is the predicted properties of the kth slot. Similarly, for models with fixed-region representations, we treat each region as a slot and apply the same approach as with OC models. We find this approach to be effective for these models. Lastly, for models with global representations, since the representations of
23

---

Published as a conference paper at ICLR 2025
individual objects not readily available, adopt the same strategy demonstrated in Dittadi et al. (2022) which has proven to be working. We predict the properties of all objects using the downstream model f and split it into K vectors {Ūh k== where K roughly corresponds to the number of slots in an OC model.
Since the predicted properties might not correspond to the objects in the same order the ground- truth objects, and the number of slots can exceed the number of objects in the image, we follow Dittadi al. (2022); Locatello et al. (2020) in using the same loss-matching algorithm match ÿN with its corresponding ground-truth vector. We define f as an MLP with hidden layer of size 256, and we utilize cross-entropy loss for categorical properties and MSE for numerical properties. We train f for one seed on 10000 images of each dataset using Adam optimizer with a learning rate of 0.001 and a batch size of 64 for 6000 steps. For a randomly selected test set of 2000 images, we calculate the accuracy for categorical properties and the adjusted R² for numerical properties and report the results.
B DATASETS
We work with 3 existing multi-object datasets: Multi-dSpriies, CLEVR, and CLEVRTex. We use the common format for these datasets as outlined in Dittadi et al. (2022). We primarily focus on these datasets due to their unified ground-truth object properties. Additionally, we utilize VQA-v2 and GQA extend our study to real-world settings. All datasets are summarized in Table 6. More details about each dataset are provided in the next subsections.
procedure and complete access to all underlying
B.I OVERVIEW OF DATASETS
CLEVR CLEVR (Johnson al., 2017) comprises 128 x 128 images depicting 3D scenes with a plain gray background featuring up to 10 objects that may partially occlude one another. Objects vary in colors (8 options in total), materials (rubber or metal), shapes (sphere, cylinder, cube), sizes (small or large), and positions (x and y) as well as rotations. Following previous works (Locatello et al., 2020; Dittadi et al., 2022: Greff et al., 2019), we utilize the CLEVR6 variant to learn object-centric representations in which the number of objects is limited to 6. The dataset has been cropped and resized according to the procedure detailed originally by Burgess et al. (2019).
The CLEVRTex dataset extends the original CLEVR (Johnson et al., 2017) by incorpo- rating textures on object surfaces, introducing a more visually complex environment. Each scene in CLEVRTex consists 3-10 objects with distinct shapes (cube, cylinder, sphere, monkey head), sizes (small, medium, large), and textures (60 in total), contributing to the diversity of visual features. The backgrounds also present complex textures compared to the plain gray ones CLEVR. The dataset is designed to facilitate the exploration of models' abilities in handling textured objects, providing valuable resource for evaluating the performance of vision-related tasks in the context of rich visual scenes.
CLEVRTex
Multi-dSprites This dataset is derived from the dSprites dataset (Matthey et al., 2017) of 64 x 64 synthetic images. Following prior research (Dittadi al., 2022; Locatello et al., 2020; Greff et al., 2019), we utilize the Multi-dSprites variant, featuring colored sprites set against grayscale background where the intensity of the uniform grayscale background randomly determined for each image. Each scene consists of 2-5 objects with randomized attributes, including shapes (ellipse, square, heart), sizes (selected from discrete values in [0.5, 1]. and converted to small and large with threshold of 0.8), x and y positions, orientation, and color (randomly sampled in HSV space). Objects might occlude one another, with certain objects being nearly entirely concealed by others in specific images. Consequently, another object, leaving only those with clearer visibility of individual objects. We use different training sizes of this dataset which is demonstrated in Table 6.
eliminate images where an object is significantly obscured by
VQA-v2 VQA-v2 (Goyal et al., 2017; Antol et al., 2015) pairs open-ended questions with images from the MS COCO 2014 dataset (Lin et al., 2014). It serves as a benchmark for evaluating how well models can comprehend and reason about visual information in real-world scenarios. The questions cover a wide range of topics and require a detailed understanding of the image content, spanning from basic inquiries about object presence to more complex queries about relationships and attributes within the scene. Answers are categorized into Yes/No, Number, and Other types. Each question has 10 grouhd-truth answers, with the most frequent ground-truth answer considered correct in our
24

---

Published as a conference paper at ICLR 2025
Table 6: Dataset splits for upstream and downstream training, number of slots used in training slot-based object-centric models, and count of unique answers to questions each dataset.
Dataset Splits
Dataset Name
Slots"
Answers
Train Size
Validation Size
Test Size
CLEVR6 CLEVRTex
7
24
2314980 (49000)
70702 (1500)
70997 (1500)
11
20
1489005 (40000)
55977 (1500)
55646 (1500)
1545444 (320000)
57816 (1500)
57557 (1500)
3089425 (320000)
57540 (1500)
57557 (1500)
Multi-dSpritest
6
22
6181799 (320000)
58289 (1500)
57557 (1500)
12365042 (320000)
57792 (1500)
57557 (1500)
VQA-v2 (COCO)
7
17
215553 (82753)
10000 (10000)
103717 (40504)
GQA
7
1834
943000 (72140)
132044 (10234)
12576 (398)
Wc define it as the maximum number of objects plus one additional slot for the background.
The values in parentheses denote the size of the corresponding image split used for the upstream model.
For MSpiisdSprites, We use 4 different training sizes.
evaluation framework. We filter out specific questions in the dataset based on their answer type, as explained in detail in Appendix B.2. We train downstream models using the filtered train split of the dataset. However, since answer types for the test split are not available, filtering process to evaluate on this split properly. Therefore, we utilize the val split as our test set for evaluation, and additionally select subset of 10k questions from it our validation set during training.
cannot apply the same
The MS COCO (Microsoft Common Objects in Context) dataset (Lin et al., 2014) a widely used collection of images designed for object detection, segmentation, and captioning tasks. It contains images sourced from everyday scenes that are diverse in content, encompassing various objects, activities, and environments. The dataset includes over 200k labeled images across 80 object categories, such as people, animals, vehicles, and indoor objects, with a significant emphasis on diversity in scenes and object appearances. Following Seitzer al. (2023), resize all the images to 224 x 224 without any cropping, while ignoring the aspect ratio. We use the original split of the dataset when training and evaluating the upstream models. Furthermore, during upstream training, we augment the dataset by horizontally flipping images with a probability of 0.5.
GQA GQA (Hudson & Manning, 2019) is a benchmark for visual reasoning, featuring open-ended questions about images that require a detailed understanding of object relationships and spatial reasoning. it provides both questions and corresponding scene graphs, making it particularly suited for evaluating relational and compositional reasoning multi-object environments. This makes GQA a challenging and insightful benchmark for assessing object-centric and foundation models alike. The dataset is available in both balanced and unbalanced versions. The balanced version ensures a uniform distribution of answers across question categories, focusing on reasoning over memorization, while the unbalanced version reflects more natural distributions of answers. In our work, we utilize the balanced version, with the train split used for training and the testdev split used for evaluation. Furthermore, we use the same image preprocessing and augmentations as for VQA-v2.
B.2 QUESTION GENERATION & PREPROCESSING
Synthetic. Originally, excluding CLEVR, the synthetic datasets exclusively comprise images without associated questions. Consequently, for their transformation into Visual Question Answering (VQA) datasets, we employ a question generation mechanism based on Johnson et al. (2017), adapted for all datasets. The process involves utilizing a script that takes as input JSON file that contains scene information and object features in the dataset. The script outputs a JSON file containing questions for each image. To generate questions for each image, there are 9 different question templates that take the features the objects in each image and generate a question based on the template. These templates are adjusted for each dataset, producing maximum of 50 questions for each image in each dataset. However, due to some images featuring only a few objects, the average number of questions per image tends to be less than 50 in each dataset. Additionally, we refrain
25

---

Published as a conference paper at ICLR 2025
CLEVR
CLEVRTex
Multi-dSprites
VQA-v2
GQA
80
77-22
Percentage (%)
60
40
75%
30,204
20
13.12%
12,60
14332 RIN
16.2%
20 60%
112
Count
Exict
Compare Integer
Compare Attribute
Query Attribute
Count
Exist
Compare Integer
Compare Attribute
Query Altribute
Count
Exist
Compare Integer
Compare Attribute
Query Artributo
Number
Yes/No
Query
Werlty
Choose
Logical
Compare
Figure 9: Distrihution of question categories per dataset.
from utilizing pre-existing questions for CLEVR. Instead, we generate questions for it to establish standardized guestion--enerationn
generation process for all datasets.
Question templates can be categorized into categories:
1. Counting:
Counting questions ask for the number of objects meeting specific criteria (e-g. "How many other things are there of the same size as the tiny metal block?").
2. Existence: Existence questions ask if an object with certain properties is present in the image (e.g. "Are there any other things that are the same color as the cylinder?").
3. Integer Comparison: Integer comparison questions ask about the relative sizes of two sets of objects (e.g. "Are there fewer spheres than large cylinders?"".
4. Comparing Attributes: Attribute comparison questions check two objects have the same value for attribute (e.g. "Do the red cube and the green cylinder have the same size?").
5. Querying Attributes: Query questions ask about an attribute of a particular object (e-g- "What shape the object at the right of the green cube?")
The set of possible answers includes yes, numerical values between 0 and the maximum number of objects the dataset, and all possible values of object properties. We have different number of templates per each question category in our question-generatiin process. The proportion of each question category in each dataset is outlined in Fig.
For CLEVR, questions are generated based on the attributes of each object, including shape, color, size, material, and their relative position in comparison to other objects. In the case of Multi-dSprites, where objects lack material attribute, questions focus the shape, color, size, and relative position of the objects. Regarding CLEVRTex, questions are generated considering only shape, size, and relative position, color absent in the dataset and is replaced with 60 materials, each having specific color. The material is not included as feature for the questions because the names of the materials are not suitable for accurate processing by a text embedding module.
Real-World. The VQA-v2 dataset includes questions from three categories: Yes/No, Number, and Other, with open-ended answers that are not limited to specific words or numbers. Therefore, in order use the same framework employed for synthetic datasets, we only keep a subset of questions with specific answers. We omit questions with "Other" answer types and keep "Yes/No" questions that have "yes" or "no" answers. Additionally, we retain questions with numerical answers ranging from 0 to 14, resulting in a total of 17 possible answers in the "Yes/No" and "Number" categories. Before filtering, the train and test sets contain 443,757 and 21 214,354 questions, respectively. After filtering, they are reduced to 215,553 and 103,717.
The GQA dataset contains questions from five structural categories: Verify, Logical, Compare, Query, Choose. Verify questions yes/no queries that check for the presence attributes of objects. Logical questions demand inference and multi-step reasoning. Compare questions require evaluating two more objects against each other. Query questions open-ended, asking for descriptive details
26

---

Published as a conference paper at ICLR 2025
about the scene. Finally, Choose questions present two alternatives from which the answer must be selected. Furthermore, unlike VQA-v2, on GQA, the number of unique answers is not significantly large, making it possible to use the pipeline as-is without any deletion.
C METRICS
Upstream. evaluate the performance of upstream models trained on the datasets used in our study, use MSE reconstruction eπror and Adjusted Rand Index (ARI) (Hubert & Arabic, 1985), which are two commonly used metrics in the literature (Locatello et al., 2020; Dittadi et al., 2022; Singh al., 2022b; Biza et al., 2023). Consistent with prior work, calculate the ARI considering only foreground objects. Additionally, following Dittadi et al. (2022), we include Segmentation Covering (Arbelaez al., 2010) and mean Segmentation Covering (Engelcke et al., 2019) as evaluation metrics. For more information on the upstream metrics, we refer to Dittadi et al. (2022).
Downstream VQA. To assess the performance of representations in the VQA downstream task, following previous works on applying the VQA task to object-centric models (Ding et al., 2021a; Wu et al., 2022; Johnson et al., 2017) and other works on VQA in general (Antol et al., 2015; Goyal et al., 2017; Ren et al., 2015; Yang et al., 2022), we use accuracy balanced accuracy which takes into account the class imbalances and is defined as the average of recall obtained in each class, and F1 score alternative metries but results revealed consistent trends across all metrics. Consequently, we focus on presenting results based on the accuracy metric.
main metric. We also analyzed
D ADDITIONAL RESULTS
In this section, we report additional results that did not fit into the main part.
D.1 PROPERTY PREDICTION vs VQA
Fig. 10 shows Spearman's 's rank correlation between downstream property prediction performance of the models for each property, and downstream VQA performance for each question category. Overall, we observe a strong correlation between the performance of the models in two downstream tasks. It's important to note that training for property prediction takes significantly less time than VQA, by around 2 orders of magnitude. Hence, property prediction can be a helpful guide when selecting a model for downstream task.
CLEVR
CLEVRTeX
Multi-dSprites
Sizw Shape Natorial
Color
00
19
21
22
76
77
82
Mater
Shap
43
26
93
95
96
60
28
29
89
92
P3
90
Color
50
Ovzat
Compare Coler
Compaee Matorta
ouery Matoha
query Shape
Compane Shape
Compare Esua
Count
Ovco1
Compere Equa
Comcare Groate
Compere Lnw
Cound
Exttt
Query Stap-
Query Sieo
Ovura
Compara Color
Compa Shape
Query Ste
Gampare Campare Equal
Count
Exter
Query toko
Quany Shape
Query
Comoore
Compare
Compare
Query
Compane
Comuere
Compare
Compor
Figure 10: Spearman's 's rank correlation between downstream property prediction performance and downstream VQA performance of the models. The correlations are color-coded only when p < 0.05.
D.2 UPSTREAM VS DOWNSTREAM PERFORMANCE
Here we analyze the relationship between the upstream and downstream VQA performance of OC models. We exclude VQA-v2 and GQA results due to only having three well- well-performing OC models for this dataset. Figs. 11 to 13 depict the upstream performance of object-centric models in comparison to their downstream performance when using T-15 as the downstream model. Generally, there no strong correlation between the two performances, and several outliers are observed in each plot.
More specifically, on CLEVR and Multi-dSprites, only ARI on CLEVR shows a slight correlation with VQA accuracy. All other upstream metrics do not correlate with the downstream performance. On CLEVRTex, however, we observe a weak correlation between ARI, mSC, and SC with VQA accuracy. At the same time, contrary to expectations, we observe a positive correlation between MSE and overall VQA accuracy which suggests that models with higher MSE values tend to perform better on the downstream task. These require further investigation which is beyond the scope of this work.
27

---

Published as a conference paper at ICLR 2025
Additionally, when we look the outliers, STEVE is the main one among OC models in all datasets, showing higher MSE and lower ARI, but better downstream performance. Also, one seed STEVE tends perform poorly in upstream metrics but achieves downstream performance comparable to other seeds. Moreover, DINOSAURY2 consistently performs poorly on mSC and SC across all datasets despite performing well downstream. Additionally, SPACE typically has the best MSE but the worst downstream performance.
In conclusion, observe that upstream metrics are a good indicator of the downstream perfor- mance of different models and thus, are not reliable for upstream model selection.
It is noteworthy that DINOSAURV2 does not reconstruct the input but instead reconstructs the latent features of the input. Thus, it is not included in the plots showing reconstruction MSE. Furthermore, we also calculated the Spearman rank correlations between upstream and downstream metries but due to the high p-value of most of the correlations, we chose not to report them.
900
104
ABI
VISE
Figure 11: Upstream performance of object-centric models against the overall VQA accuracy when using T-15 the downstream model on CLEVR.
MSE
Figure 12: Upstream performance of object-centric models against the overall VQA accuracy when using T-15 as the downstream model on CLEVRTex.
213
623
133
AR1
1150
5C
VISE
Figure 13: Upstream performance of object-centric models against the overall VQA accuracy when using T-15 the downstream model on Multi-dSrrites.
28

---

Published as a conference paper at ICLR 2025
D.3 EFFECT OF DOWNSTREAM MODEL SIZE
Fig. 14 depicts the overall accuracy of the models across different datasets with a downstream model having a varying number of layers. As expected, the downstream performance increases with the downstream model size. Furthermore, we observe that DINOSAURv2 performs better than DINOv2 on CLEVRTex, Multi-dSprites, and GQA when using T-2 the downstream model. However, as the downstream model size increases, DINOv2's performance matches DINOSAURV2. This indicates that while DINOv2 representations do have the necessary information for object-related tasks, the information is less obvious and needs larger downstream model to be effectively extracted. However, on CLEVR, we don't It see the same pattern, as DINOSAURY2 performs slightly worse than DINOv2. After experimenting with various hyperparameter sets, we found the results of DINOSAURv2 on CLEVR to be highly sensitive to the selected hyperparameters, which likely explains its suboptimal performance.
Multi-SSprites
DRO62
Vi-a
OMM
RelfeeSA
SPACE
Downstream Model
Downstream Model
Downstream Model
Downstream Model
We
Figure 14: Average accuracies of different models w.r.t. downstream model size across different datasets. For pre-trained models, only one seed available. For other models, the results are averaged over 3 random soods and the shaded arcas indicate 95% confidence intervals.
D.4 EFFECT OF TRAINING SIZE
Fig. 15 depicts the overall accuracy of different models on Multi-dSprites with varying training sizes of 40k, 80k, 160k, and 320k unique images, and Fig. 16 shows the average percentage of decrease in error rate when increasing the dataset size from 40k to the respective size. With increased data, most upstream models show similar performance improvement across different training sizes regardless of their initial performance. CLIP the only exception, showing a decrease in overall error rate of up to 50%. Furthermore, the performance of the end-to-end CNN and MultiCNN models show minimal improvement compared to the other models.
MuttidSprites-e
Muti-SSprite
Nubi-dSprttss 100k
MubtidSprites 320k
Accorncy (Overll)
5
Oback Cartric Gebal
DIOCU2
VO
Baok
DINC
N41
VON
HONAL
CENOSAUR
Book
HONA
Figure 15: Average accuracies on the VQA downstream task for different models on Multi-dSprites with different training sizes when using T-15 as the downstream model. The hars indicate means and 95% confidence intervals with 3 random seeds.
D.5 CONSISTENCY OF THE RESULTS ACROSS DIFFERENT QUESTION TYPES
Fig. 17 shows the Spearman's 's rank correlation between the performances of models each question category when using T-15 for synthetic datasets and GQA, and T-2 for VQA-v2. Additionally, Figs. 18 28 illustrate the accuracy of different question types of all upstream model representations across different datasets. Except for GQA, we observe strong correlations between the performance of different question categories which indicates that the trend in the overall accuracy in Fig, matches the trend in the accuracy of each question category and the results are consistent across different question categories.
29

---

Published as a conference paper at ICLR 2025
Error Race Decreate
60
Multi-d5piits-80k
pritt-65rrites-600k
Multi-dSprite-3220
60
40
Fixed Region
Object Centric
18
8
DINoV2
MAE
CuP
Resheddo
CMN
Munsccn
MORe
smnce
STEVE DINOSAURA
VSE
MMB
cui
KL AG
Reshe
CuN
MulicIN
MON
SPACE
STEVe
DINOSIURG
VAE
DINO
CUP
vo-A
K--t
RecNee50
CHN
Mumrcan
RedNer SA
MONoe
SPACE
STEYS
DINOSAUING
We
Resten
Figure 16: Average % decrease in VQA error rate for different upstream models on Multi-dSprites, when
increasing the training size from 40k to larger sizes, using T-15 as the downstream model. The bars indicate mcans and 95% confidence intervals with 3 random sceds.
Prik
VIJA
pay
Figure 17: Spearman's 's rank correlation of model performances for each question category using T-15 as the downstream model.
CLEVR
CLEVRTeX
Multi-Spories
Rxed-Region Deject-Cetticc Clobal
Accuracy
DIN002
MAR
CLIP
VO-A0
Reshit50
MublcAN
SA
Restee
SPACE
STEVE
DINOSAURO m02
VIE
DIKOu2
MA
CLIP
MQ-4E
KI-AF
Restiesso
MulICAN
SA
Restee S6
SPACE
STEVE
DINOSAUR/)
DINOv2
MAE
CLiP
RusNNrSo 550
MuliCAN
S4
Restet SA
Moxe!
SINCE
DINOSAURI
Figure 18: Average accuracies of Exist questions for different upstream representation models when using T-15
the downstream model. The bars indicate means 95% confidence intervals with random seeds.
CLEVR
CLEVRTex
Multi-dSprites
100
100
1.0
Accuracy (Count)
60
We Ce
lixed-Region Object Centric
DINO.2
MAE
CLP
Vg-u
NE
Restedio
OWN
MUKIONN
SA
Restiet SA
SPACE
STEVE
DINOSAURV2
DIKO
MAX
CLP
Rediebo
CNN
MunioN
SA
SPACE
STEVE
DIVOSAURI2
DINO-2
MAE
CLP
RustuiSo
MunicNN
54
RusNit 54
MONe
SINCE
STEVE
DINOSAURI2
Figure 19: Average accuracies of Count questions for different upstream representation models when using T-15
as downstream model. The hars indicate means 95% confidence intervals with random seeds.
30

---

Published as a conference paper at ICLR 2025
CLEVR
CLEVATex
dSprites
Accuracy (Comparn
Rxed Rogion Object Centric Global
DINDV2 MAG
CLPP
Vg
RedNedo
CWN
MurtiCN
SA
Restee SA
SPACE
STEVE
DINOSAURG
VAE
DINDvE
MAN
CUP
VQ-AE
KL-AF
ResNesso
MuticNN
SA
RasNes SA
SPACE
STEVE
DINOSAURR
VAE
DINOV!
CUP
VD- 40
XL-AF
RecteeSo
MuticNN
SA
Rushte SA
Morer
SPACE
STRVE
DINOSAII
Figure 20: Average accuracies of Compare Integer (Less) questions for different upstream representation models
when using T-15 as the downstream model. The bars indicate means and 95% confidence intervals with 3 random seeds.
Accuracy (Compare Greater)
CLEVR
CLEVATex
Multi-dSprites
000
:00
120
60
Riee--tegion Object Global
DINOY2
MW
CLIP
Vo--e
Reshees0
CNN
MURICWN
SA
STEVE
DINOSAURS
DIND
MA
CLIP
VQ-AE
Resiketo
CHN
MuRIENN
Restle SA
SPACE
STEVe
DINOSAURRV
WE
DROO
NAE
CLIP
vo-we
RecNee
CHN
MuRiCNA
SA
Moxel
SINCE
STEVE
DINOSAURVE
SM
RL
Redter
Figure 21: Average accuracies of Compare Integer (Greater) questions for different upstream representation
models when using T-15 as the downstream model. The bars indicate means and 95% confidence intervals with 3 random seeds.
Accuracy (Compare Equal)
CLEVR
CLEVATeX
Multi-dSorites
200
100
120
80
liced Region Dejert-Centric
DINO
MAE
CUP
VD-AI
Reshic0
ENN
SA
STEVE
DINDSUUR2
VAE
DINOv2
MAI
CUP
VO-4E
Ki-Af
ResNeto
CNN
MURICNN
SPAGE
STEVE
DINDSAURO
DINDV?
MAE
CUP
VD MD
Ki-AE
CNN
SA
Restet SA
MONel
SPACE
STEVE
Muis
Reshee
Figure 22: Average accuracies Compare Integer (Equal) questions for different upstream representation
models when using T-15 as the downstream model. The bars indicate means and 95% confidence intervals with 3 random seeds.
Accuracy (Compare Shape)
CLEVR
CLEVRTex
Multi-dSprites
1no
1.0
122
60
Fixed Region CentiCC
DINOV2
MAE
CUP
Reste50
ENN
MuR/CON
5A
Reshet SA
STEVE
DINOSAURRO
VA
DINDVZ
MAM
CUP
VO-E
Resteto
CUN
MuRiCIN
ResNet
SPACE
STEVE
DINEISAURV)
WE
CINOVO
MAE
CLP
VD-M
RasNeiEO
CAN
MuRIEIO
SA
Reshie SA
MoMel
SPACE
STEVE
DINOSAURVV
Figure 23: Average accuracies Compare Auribute (Shape) questions for different upstream representation
models when using T-15 as the downstream model. The bars indicate means and 95% confidence intervals with 3 random seeds.
31

---

Published as a conference paper at ICLR 2025
CLEVR
CLEVRTOX
Multi-dSprite Stte
200
.00
120
Accuracy (Comparn
Rxed-Ragion Ceject-Cartric
MAE
CUP
VC-An
Reswe50
CNN
MUDCWW
SK
Reshie SA
STEVE SPACE
DNO DROSKURVZ
DINO
MAB
CUP
KL-At
Reste500
CNN
MUBION
SPACE
STEVE
DINOSAURI2
DROW?
MAE
CUP
VO-AB
Ri-At
Ruster50
CNN
MUMCHN
SA
Rushet SA
SIACE
STEVE
BINOSUURZ2 Vi
KL
Figure 24: Average accuracies Compare Attribute (Size) questions for different upstream representation
models when using T-15 as the downstream model. The bars indicate means and 95% confidence intervals with 3 random seeds.
Accuracy (Query Shape)
CLEVR
CLEVATex
Multi-dSpp SSprits
100
1.00
120
Fxed Region
Dejedt-Crrticc
13
DINOV2
MMC
CUP
Vo
ME
Restedo
CNN
MultiWN
SA
Restiet SA
STEVE DINOSUUN2
VIE
DINova
MMt
CUP
KL-PP
Resteeto
CNN
CNN
SA
SPACE
STEVE
DIVOSAUIV2
viz
DINova
MAE
cuP
Vovee
RastuuSo
CAN
Mult CNN
SA
Rechet SA
MONe
SPACE
STEVE
DINOSMURN
Figure 25: Average accuracies of Query Attribute (Shape) questions for different upstream representation models
when using T-15 as downstream model. The bars indicate means and 95% confidence intervals with random seeds.
CLEVR
CLEVRTeX
Multi-dSorites
Accuracy (Query Size)
Region Object Centric Global
+
DINDIG
HAEE
CLIP
VD o0
Restec0
MuISIONN
SA
SPACE
STEVE
DINDSAURVO
DINDO
MA
CLIP
VQ-éE
KLAF
Resteci
MURICON
SA
SA
SPACE
STEVE
DINOSAURO2)
DINDOO
CLIP
VD-A Al
KL-AF
BasNeiSo
MURICIN
SAT
Reshie SA
SPNCE
DINOSRURI S
EL
STRVE
Restee
Figure 26: Average accuracies of Query Astribute (Size) questions for different upstream representation models
when using T-15 as the downstream model. The bars indicate means and 95% confidence intervals with 3 random seeds.
VQA-v2
VQA-v2
70
70
Accuracy (Number)
60
Accuracy (Ye//No)
60
50
50
40
40
ZIZ
Fixed- TixedReegion
Object-Centric
30
30
20
20
10
1
DINOV2
MAE
CLIP
VO-AE
KL-AE
ResNet50
ResNet SA
STEVE
DINOSAURV2
DINOV2
MAE
CLIP
VQ-AE
KL-AE
ResNet50
ResNet SA
STEVE
DINOSAURV2
Figure 27: Average accuracies of different question types for different upstrcam representation models when
using T-2 as the downstream model VQA-v2. The bars indicate means and 95% confidence intervals with random seeds.
32

---

Published as a conference paper at ICLR 2025
GQA
GQA
GQA
Accuracy (verify)
Accuracy (query)
Accuracy (choose)
80
40
40
28
31
DINOV2
MAE
CUP
VQ-AE
ResNet SA
STEVE
DINOSAURV2
MAE
CUP
VQ VO
KL-4E
ResNet
DINOSAUIV22
DINOV2
VQ -0
KL-AE
Resivet50
RedWitt SA
STEVE
DINOSAURR2
GQA
GQA
Accuracy (logical)
Accuracy (compare)
60
40
Fxed-RRgion
Object-Centric
20
DINOv2
MAE
CUP
VQ-AE
KL-AE
ResNe150
ResNet SA
STEVE
DINDSAURV2
CUP
VQ AQ
KLAEE
ResNei50
DINOSAURV2
Figure 28: Average accuracies of different question types for different upstrcam representation models when
using T-15 as the downstream model on GQA. The bars indicate means and 95% confidence intervals with random seeds.
33

---

Published as a conference paper at ICLR 2025
D.6 FULL RESULTS
The complete VQA accuracies of different upstream models across all datasets are presented in Tables to 14.
Table Average accuracies on CLEVR when using T-15 as the downstream model. For pre-trained models, only one seed is available. For other models, the results are aggregated over random seeds.
Model
Overall
Exist
Count
Compare Integer
Compare Attribute
Query Attribute
Less
Greater
Equal
Shape
Color
Material
Size
Shape
Color
Material
Size
DINOv2
98.9
99.4
97.4
99.2
98.9
97.4
99.7
100.0
97.4
99.6
99.5
99.7
99.3
99.8
98.2
99.5
99.0
97.3
99.4
99.9
98.2
99.2
99.6
99.5
99.7
CLJP
96.5
97.9
92.2
97.7
96.8
94.1
95.8
98.6
96.2
98.0
96.9
98.2
98.1
99.1
VQ-AE
52.7
64.3
40.3
75.8
73.9
66.0
50.9
52.1
49.9
56.6
43.2
33.2
58.5
63.5
KL-AE
62.0
84.2
82.0
68.7
80.0
80.6
77.4
77.4
84.6
84,9
ResNe150
85.4
91.2
73.1
93.2
92.4
85.2
79.6
K5.1
82.3
92.0
86.3
84.9
89.6
94.7
MultiCNN
48.1
64.0
42.2
74.8
72.2
65.3
51.3
52.7
51.4
53.1
33.3
12.6
50.3
52.9
86.9
92.3
84.6
88.9
93.5
95.8
97.6
ResNet
96.2
97.6
92.8
97.3
95,9
93.0
96.0
96.4
96.3
988.5
96.8
96.4
98.U
98.7
SPACE
96.8
98.3
91.6
97.7
97.1
93.9
98.0
98.0
97.5
98.2
98.2
98.5
98.8
98.9
STEVE
96.5
93.0
98.3
97.7
95.3
96.7
96.9
96.2
98.4
96.5
97.7
98.8
DINOSAURY2
94.3
92.8
86.6
95.9
94.9
87.2
95.2
76.3
95.1
97.8
96.8
63.0
97.7
99.2
Table 8: Average accuracies on CLEVRTex when using T-15 as the downstream model. For pre-trained models, only seed is available. For other models, the results are aggregated over random seeds.
Model
Overall
Exist
Count
Compare Integer
Compare Attribute
Query Attribute
Less
Greater
Equal
Shape
Size
Shape
Size
DINOV2
97.1
96.2
MAE
91.1
95.4
82.3
96.5
95.4
90.0
94.0
94.5
94.7
95.8
81.9
VQ-AE
51.5
69.6
33.3
73.3
73.0
63.1
$4.1
58.8
37.1
48.7
KL-AE
53.5
71.7
35.6
75.1
74.3
65.9
55.2
58.5
41.7
49.7
ResNe150
60.8
75.8
40.4
79.4
78.5
67.0
61.7
65.2
60.2
CNN
51.7
71.0
33.5
74.5
73.9
64.5
$3.5
57.6
36.9
47.4
MaliCNN
51.3
70.8
32.8
74.9
73.5
64.7
$4.0
57.6
36.2
46.2
SA
55.3
73.0
36.2
76.5
75.5
65.7
53.9
60.6
44.0
$7.0
ResNet SA
67.6
81.7
46.8
82.3
81.0
71.2
68.6
73.7
69.8
76.9
SPACE
54.2
71.9
35.6
75.7
74.8
65.8
54.3
58.7
44.9
52,3
STEVE
86.8
92.3
DINOSAUR<2
93.9
97.3
87.1
97.5
97.2
94.5
96.6
96.0
96.5
96.8
VAF
51.9
69.7
33.9
74.9
74.1
64.4
$2.6
57.5
37.3
49.5
34

---

Published as a conference paper at ICLR 2025
Table 9: Average accuracies on Multi-dSpritts with 40k unique training images when using T-15 as the downstream model. For pre-trained models, only one seed is available. For other models, the results are aggregated over 3 random seeds.
Model
Overall
Exist
Count
Compure Integer
Compare Attribute
Query Attribute
Less
Greater
Equal
Shape
Color
Size
Shape
Color
Size
DINOV2
91.3
96.4
96.1
91.7
96.9
97.6
96.1
MAE
95.9
96.9
95.2
98.1
98.2
94.8
98.2
91.0
96.7
98.8
89.9
98.2
CLIP
91.0
86.6
92.8
93.1
86.1
92.3
84.8
95.0
80.8
92.1
VQ-AE
63.1
72.8
62.2
82.6
83.2
71.8
53.5
58.8
59,9
48.6
41.6
71.5
KL-AE
68.5
75.4
67.9
83.2
84.2
72.1
56.9
66.4
65.0
54.2
76.5
ResNe150
74.4
79.9
73.7
87.8
87.3
78.8
73.4
60.9
70.0
77.4
53.5
79.8
CNN
65.3
74.0
65.9
82,3
82.6
71.6
56.3
62.1
59.9
49.8
47.4
72.5
MuliCNN
61.7
72.6
62.6
80.7
80.7
71.3
58.3
59.0
58.0
46.6
32.2
68.4
SA
94.2
95.4
93.1
97.2
96.4
92.2
96.2
87.8
94.7
97.4
89.2
97.2
ResNet SA
87.4
90.6
87.2
95.5
93.6
89.0
94.4
65.0
90.4
94.7
65.3
95.7
MONet
90.0
92.4
88.2
95.6
93.9
88.5
84.1
88.6
88.0
86.4
89.8
94.6
SPACE
89.3
93.5
STEVE
96.6
97.5
95.5
98.0
97.1
93.5
97.6
92.6
97.9
98.5
95.1
98.8
DINOSAURV2
91.1
91.0
87.8
95.3
94.2
89.4
94.3
64.8
92.8
95.8
63.3
95.9
VAE
68.0
74.5
68.9
83.8
83.3
71.0
59.3
60.9
68.7
57.7
47.7
75.6
Table 10: Average accuracies Multi-dSprites with 80k unique training images when using T-15 the downstrcam model. For pre-trained models, only one seed is available. For other models, the results are aggregated over random seeds.
Model
Overall
Exist
Count
Compare Integer
Compare Attribute
Query Attribute
Less
Greater
Equal
Shape
Color
Size
Shape
Color
Size
DINOv2
93.1
95.3
92.9
97.0
96.0
91.6
97.7
81.2
94.7
98.2
$1.1
97.4
MAE
97.1
97.7
96.6
98.2
98.3
96.5
98.3
93.0
98.7
98.7
93.5
99.1
CLIP
91.7
94.3
90.5
95.2
95.6
89.5
95.3
84.0
89.0
95.8
83.9
95.2
VQ-AE
65.5
73.0
64.6
83.3
83.5
73.4
57.8
62.8
62.5
50.0
49.0
73.0
KL-AE
70.6
76.9
67.5
84.3
85.1
70.8
60.8
70.7
70.1
57.1
80.3
ResNet50
78.8
82.7
77.6
90.6
89.4
83.5
76.5
66.3
74.5
81.7
62.5
85.0
67.4
75.1
MuliCNN
62.8
72.7
63.6
81.2
82.0
72.4
59.5
61.3
59.9
48.2
34.0
68.9
SA
94.3
95.4
93.6
97.8
97.0
93.4
97.1
89.0
91.8
97.7
96.0
ResNet SA
90.2
93.1
89.7
96.0
95.4
91.0
96.0
71.9
93.3
95.5
73.5
96,3
84.9
94.7
SPACE
89.3
90.1
88.3
89.6
89.2
77.3
93.9
89.1
87.4
93.3
87.6
91.9
STEVE
97.9
98.4
97.3
98.9
98.3
95.8
99.3
95.8
99.0
99.1
96.4
99.3
DINOSAUR<2
92.8
93.6
90.6
96.1
96.0
91.7
97.0
74.5
95.0
96.9
75.3
97.2
Table 11: Average accuracies on Multi-dSprites with 160k unique training images when using T-15 as the downstream model. For traitrained models, only seed is available. For other models, the results aggregated over random seeds.
Model
Overall
Exist
Count
Compare Integer
Compare Attribute
Query Altribute
Less
Greater
Equal
Shape
Color
Size
Shape
Color
Size
DINOV2
94.2
95.6
93.7
97.2
95.4
91.3
98.9
84.9
96.2
98.2
84.8
98.6
MAE
97.0
97.6
96.7
98.4
98.8
96.0
98.3
92.7
98.7
99.2
91.9
98.9
CLIP
93.4
95.5
91.6
96.6
96.5
91.1
96.7
90.5
91.5
97.1
86.9
96.0
VQ-AE
73.2
65.6
82.5
84.2
72.5
58.1
66.6
64.8
50.1
74.2
KI-AF
72.9
78.4
71.0
84.8
85.5
74.0
61.5
72.8
72.9
58.2
67.3
81.7
ResNet50
80.0
92.8
90.8
85.3
78.2
76.4
84.2
87.1
CNN
67.8
75.4
67.4
83.8
83.8
72.8
60.0
65,3
62.7
51.2
54.8
75.8
MultiCNN
61.8
72.6
63.1
81.2
81.7
72.5
$9.0
61.0
$8.9
42.8
33.2
67.9
SA
95.3
96.0
94.33
98.2
97.8
94.7
97.5
91.0
94.5
98.0
91.0
97.2
ResNet SA
93.1
90.6
96.7
96.0
91.3
95.8
92.9
96.0
96.2
MONet
91.0
92.8
89.3
95.9
95.3
91.3
85.6
91.4
88.6
87.4
91.3
94.7
SPACE
90.1
90.6
89.6
89.6
90.0
77.6
94.8
89.8
89.0
93.8
88.0
92.6
STEVE
98.1
97.77
98.77
98.6
95.9
98.8
96.1
99.0
99.3
96.8
99,4
DINOSAURV2
93.2
93.3
90.4
96.2
95.7
91.5
96.5
73.2
95.2
96.8
76.0
97.1
VAE
76.3
81.9
75.6
89.0
87.8
79.4
68.0
70.9
74.6
65.3
67.2
83.2
35

---

Published as a conference paper at ICLR 2025
Table 12: Average accuracies on Multi-dSprites with 320k unique training images when using T-15 as the downstream model. For pre-trained models, only seed is available. For other models, the results aggregated over random seeds.
Model
Overall
Exist
Count
Compure Integer
Compare Attribute
Query Attribute
Equal
Shape
Color
Size
Shape
Size
DINO₂2
94.2
95.6
93.7
97.2
95.4
91.3
98.9
84,9
96.2
98.2
84.8
98.6
MAE
97.1
97.7
96.8
98.4
98.1
95.1
98.8
92.5
98.8
99.2
99.1
CLIP
94.1
95.2
92.6
96.9
96.8
91.9
97.0
90.3
93.8
97.7
89.3
96.4
VQ-AE
KL-AE
73.3
78.9
72.6
84.2
85.1
73.2
61.9
72.9
75.5
57.2
67.4
81.3
ResNet50
78.7
87.1
CNN
68.2
75.5
68.0
84.1
84.1
73.1
59.6
66.2
64.8
51.2
$5.0
75,9
MuliCNN
59.1
47.0
68.3
SA
95.4
96.1
94.4
98.0
97.0
93.5
97.4
90.4
95.3
98.0
91.2
97.8
ResNet
91.3
97.0
96.3
96,9
MONet
90.9
92.7
89.3
95.9
95.3
90.5
85.1
91.4
89.4
87.0
91.4
94,8
SPACE
90.0
90.7
89.5
89.8
90.1
77.4
94.5
88.7
88.7
93.6
87.4
92.6
STEVE
97.9
98.8
97.9
95.5
98.8
96.4
98.7
99.0
99.2
DINOSAURV2
94.4
94.7
92.7
97.6
97.5
93.3
98.5
78.0
96.8
98.3
78.0
98.6
VAE
76.8
82.5
76.3
89.8
87.6
79.9
66.2
71.5
74.9
65.3
68.8
83.7
Table 13: Average accuracies VQA-v2 when using T-2 the downstream model. For pre-trained models, only one seed is available, For other models, the results are aggregated over random seeds.
Model
Overall
Number
Yes/No
MAE
56.5
36.8
62.2
CLIP
57.2
36.6
VQ-AE
53.2
30.0
59.9
KLEAF
54.2
30.0
61.2
ResNet50
53.3
30.1
59.9
ResNet SA
52.6
30.1
59.1
STEVE
53.8
29.5
60.8
DINOSAURV2
61.3
41.7
67.0
Table Average accuracies on GQA when using T-15 as the downstream model. For pre-trained models, only one seed is available. For other models, the results are aggregated over 3 random seeds.
Overall
Verify
Query
Logical
Compare
DINOV2
51.3
71.9
39,2
66.2
61.6
59.3
66.9
32,9
CLIP
47.3
69.9
32,6
57.0
60.8
62.0
VQ-AE
64.1
21.3
KL-AE
41.7
64.7
24.3
56.7
62.5
55.5
ResNet50
ResNet SA
40.8
66.2
22.3
55.6
61.3
59.7
25.1
DINOSAURV2
51.1
70.6
37.4
66.2
63,0
56.6
36


Reviews:
[{"review_id": "EkTGAYZNbW", "reviewer": "Reviewer_rPSW", "summary": "This paper presents a comprehensive empirical study comparing object-centric (OC) representations with foundation models on Visual Question Answering tasks. Through extensive experiments involving 15 different upstream models, the authors demonstrate that combining object-centric bias with foundation models can achieve strong performance while reducing computational costs compared to using foundation models alone.", "questions": "- How would the proposed approach perform on more challenging real-world datasets like GQA or CRIC? \n- How would the findings about the benefits of object-centric representations translate to more challenging scenarios in real datasets?\n- Why did the authors choose to focus on transformer encoder architectures for downstream models? Would incorporating decoder-based architectures potentially lead to different conclusions?\n- Could the authors provide more detailed analysis of experiments with 640 downstream models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Extensive empirical evaluation across different upstream and downstream models and datasets\n- Experimental results thoroughly support and validate each insight presented", "weaknesses": "- While the study includes VQA-v2, it primarily relies on synthetic datasets. The evaluation would be more compelling if it included additional established real-world VQA datasets such as GQA or CRIC. Each image in the GQA and CRIC datasets is associated with a scene graph describing the image's objects, attributes, and relations. Moreover, questions in these datasets involve multiple reasoning skills, spatial understanding, and multi-step inference, making them ideal for analyzing object-centric models.\n- The majority of claims and findings are based on synthetic datasets, which feature simpler scenes with clear object-background separation. This raises concerns about the generalizability of the findings to more complex real-world scenarios. It would be great if the author can provide additional analysis on the challenging datasets such as GQA and CRIC\n- The downstream architecture is limited to BERT-style transformer encoders. The authors should explore decoder-based transformer architectures, which have become increasingly popular and achieved more favorable results than BERT-style encoders on VQA tasks in recent research.\n- While the authors claim they evaluated 640 downstream models, the paper lacks sufficient detail and analysis regarding these experiments. They should provide comprehensive information about these models in the paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nXMxtXNsnF", "reviewer": "Reviewer_7see", "summary": "This paper presents a comprehensive empirical study of Object-Centric (OC) representations in VQA task. The study evaluates 15 different upstream models across three synthetic datasets and one real-world dataset. The authors draw conclusions on the comparison foundation models and OC models, the relation of VQA performance and intermediate tasks or metrics, the training data efficiency and so on.", "questions": "Minor: I wonder how is the \"640\" computed on Line 315? And the \"640 models\" is slightly misleading, as they are actually different are experimental configurations rather than distinct model architectures.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I appreciate the authors' great efforts (and also their GPU cluster's) of this empirical study. The paper shows empirical rigor through its extensive experimental design, with multiple trials across various models and datasets. The experiments brings credibility to the conclusions, while the conclusions themselves could provide valuable insights for the field.", "weaknesses": "- Some findings align with existing intuitions. For example, it is not suprising that large foundation models perform comparably to specialized OC models, and that their combination yields better performance. While valuable, these conclusion is intuitive and somehow obvious to the maching learning field.\n- The correlation between property/attribute accuracy and overall VQA performance may be due to the characteristics or preference of the selected 4 datasets, rather than a fundamental relationship. This correlation might not generalize well to relational or reasoning or logical questions, potentially make the conclusion less convincing.\n- While there are many real-world QA datasets like VCR, GQA, DAQUAR which would capture more general data intrinsics of QA question, only one real-world dataset (VQA-v2) was used in this work. The three synthetic datasets is quantitatively extensive, but might exhibit construction biases or preferences and not capture the full complexity of real-world scenarios. Focusing on more diverse real-world datasets would strengthen the conclusions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TXhXChDNiO", "reviewer": "Reviewer_HU9k", "summary": "This is an experimental paper that examines how object-centric (OC) representations, which treat scenes as compositions of objects, compare with large pre-trained foundation models in Visual Question Answering (VQA) tasks.", "questions": "See weakness, I would like to see the responses to the above three weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "Using object features to represent images to solve VQA has become quite popular since the bottom-up attention (from Peter Anderson et al.). However, there have been many debates about whether the object-level feature makes it work or whether it is just because the image representation model used in BUTD was better trained (with more data). This paper tried to solve this problem by running many experiments.", "weaknesses": "1. The question that this paper wants to answer is: whether OC, fixed-region or global representation is better for VQA. To answer this question, the authors need to provide a model with the same architecture, model size and pre-training data. The only variant in the model is the feature representation way, i.e., OC, fixed-region or global. However, I didn't find this experiment in the paper. Maybe I missed it. Please point it out and highlight it in the rebuttal.\n\n2. Synthetic data such as CLEVR and CLEVRTex are not challenging enough since the objects can be easily detected (and even hard-coded). Many symbolic methods have solved these datasets with 100% accuracy.\n\n3. I am not sure how these takeaway messages could contribute to the community. Some conclusions are already known such as : ' Consistency of the Results Across Question Types.'", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nQprll8IH0", "reviewer": "Reviewer_feeJ", "summary": "This paper explores the effectiveness of object-centric (OC) representations in the Visual Question Answering (VQA) task and compares them with traditional large-scale pretrained foundation models.", "questions": "1. What makes me curious is what the performance would be after fine-tuning the foundation models on the specific VQA tasks, since other models have been fitted on the data. I suppose this would make it more fair for the comparison of different models and may inspire more insights.\n2. Could you give some more insights on the application of the findings in the paper? I'm wondering if the conclusions would be applied in other domains out of VQA.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The experimental design is rigorous, considering the impact of multiple factors on the task.\n2. The writing is fluent, and the conclusions are clear.\n3. The finding that object-centric representations are effective in VQA tasks is insightful for the community.", "weaknesses": "1. As stated in the paper, the datasets would have included more real-world data.\n2. Some analysis of the application of the findings is lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)