Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Models

Authors: Amir Mohammad Karimi Mamaghan, Samuele Papa, Karl Henrik Johansson, Stefan Bauer, Andrea Dittadi
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Object-centric Learning, Foundation Models

Abstract:
Object-centric (OC) representations, which model visual scenes as compositions of discrete objects, have the potential to be used in various downstream tasks to achieve systematic compositional generalization and facilitate reasoning. However, these claims have yet to be thoroughly validated empirically.
Recently, foundation models have demonstrated unparalleled capabilities across diverse domains, from language to computer vision, positioning them as a potential cornerstone of future research for a wide range of computational tasks.
In this paper, we conduct an extensive empirical study on representation learning for downstream Visual Question Answering (VQA), which requires an accurate compositional understanding of the scene. 
We thoroughly investigate the benefits and trade-offs of OC models and alternative approaches including large pre-trained foundation models on both synthetic and real-world data, ultimately identifying a promising path to leverage the strengths of both paradigms. 
The extensiveness of our study, encompassing over 600 downstream VQA models and 15 different types of upstream representations, also provides several additional insights that we believe will be of interest to the community at large.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730463203736

thread_type:
official_review

<thread_object>
creator: reviewer_feej

<content>
summary:
This paper explores the effectiveness of object-centric (OC) representations in the Visual Question Answering (VQA) task and compares them with traditional large-scale pretrained foundation models.

strengths:
1. The experimental design is rigorous, considering the impact of multiple factors on the task.
2. The writing is fluent, and the conclusions are clear.
3. The finding that object-centric representations are effective in VQA tasks is insightful for the community.

weaknesses:
1. As stated in the paper, the datasets would have included more real-world data.
2. Some analysis of the application of the findings is lacking.

questions:
1. What makes me curious is what the performance would be after fine-tuning the foundation models on the specific VQA tasks, since other models have been fitted on the data. I suppose this would make it more fair for the comparison of different models and may inspire more insights.
2. Could you give some more insights on the application of the findings in the paper? I'm wondering if the conclusions would be applied in other domains out of VQA.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730624378206

thread_type:
official_review

<thread_object>
creator: reviewer_hu9k

<content>
summary:
This is an experimental paper that examines how object-centric (OC) representations, which treat scenes as compositions of objects, compare with large pre-trained foundation models in Visual Question Answering (VQA) tasks.

strengths:
Using object features to represent images to solve VQA has become quite popular since the bottom-up attention (from Peter Anderson et al.). However, there have been many debates about whether the object-level feature makes it work or whether it is just because the image representation model used in BUTD was better trained (with more data). This paper tried to solve this problem by running many experiments.

weaknesses:
1. The question that this paper wants to answer is: whether OC, fixed-region or global representation is better for VQA. To answer this question, the authors need to provide a model with the same architecture, model size and pre-training data. The only variant in the model is the feature representation way, i.e., OC, fixed-region or global. However, I didn't find this experiment in the paper. Maybe I missed it. Please point it out and highlight it in the rebuttal.

2. Synthetic data such as CLEVR and CLEVRTex are not challenging enough since the objects can be easily detected (and even hard-coded). Many symbolic methods have solved these datasets with 100% accuracy.

3. I am not sure how these takeaway messages could contribute to the community. Some conclusions are already known such as : ' Consistency of the Results Across Question Types.'

questions:
See weakness, I would like to see the responses to the above three weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644275546

thread_type:
official_review

<thread_object>
creator: reviewer_7see

<content>
summary:
This paper presents a comprehensive empirical study of Object-Centric (OC) representations in VQA task. The study evaluates 15 different upstream models across three synthetic datasets and one real-world dataset. The authors draw conclusions on the comparison foundation models and OC models, the relation of VQA performance and intermediate tasks or metrics, the training data efficiency and so on.

strengths:
I appreciate the authors' great efforts (and also their GPU cluster's) of this empirical study. The paper shows empirical rigor through its extensive experimental design, with multiple trials across various models and datasets. The experiments brings credibility to the conclusions, while the conclusions themselves could provide valuable insights for the field.

weaknesses:
- Some findings align with existing intuitions. For example, it is not suprising that large foundation models perform comparably to specialized OC models, and that their combination yields better performance. While valuable, these conclusion is intuitive and somehow obvious to the maching learning field.
- The correlation between property/attribute accuracy and overall VQA performance may be due to the characteristics or preference of the selected 4 datasets, rather than a fundamental relationship. This correlation might not generalize well to relational or reasoning or logical questions, potentially make the conclusion less convincing.
- While there are many real-world QA datasets like VCR, GQA, DAQUAR which would capture more general data intrinsics of QA question, only one real-world dataset (VQA-v2) was used in this work. The three synthetic datasets is quantitatively extensive, but might exhibit construction biases or preferences and not capture the full complexity of real-world scenarios. Focusing on more diverse real-world datasets would strengthen the conclusions

questions:
Minor: I wonder how is the "640" computed on Line 315? And the "640 models" is slightly misleading, as they are actually different are experimental configurations rather than distinct model architectures.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703490771

thread_type:
official_review

<thread_object>
creator: reviewer_rpsw

<content>
summary:
This paper presents a comprehensive empirical study comparing object-centric (OC) representations with foundation models on Visual Question Answering tasks. Through extensive experiments involving 15 different upstream models, the authors demonstrate that combining object-centric bias with foundation models can achieve strong performance while reducing computational costs compared to using foundation models alone.

strengths:
- Extensive empirical evaluation across different upstream and downstream models and datasets
- Experimental results thoroughly support and validate each insight presented

weaknesses:
- While the study includes VQA-v2, it primarily relies on synthetic datasets. The evaluation would be more compelling if it included additional established real-world VQA datasets such as GQA or CRIC. Each image in the GQA and CRIC datasets is associated with a scene graph describing the image's objects, attributes, and relations. Moreover, questions in these datasets involve multiple reasoning skills, spatial understanding, and multi-step inference, making them ideal for analyzing object-centric models.
- The majority of claims and findings are based on synthetic datasets, which feature simpler scenes with clear object-background separation. This raises concerns about the generalizability of the findings to more complex real-world scenarios. It would be great if the author can provide additional analysis on the challenging datasets such as GQA and CRIC
- The downstream architecture is limited to BERT-style transformer encoders. The authors should explore decoder-based transformer architectures, which have become increasingly popular and achieved more favorable results than BERT-style encoders on VQA tasks in recent research.
- While the authors claim they evaluated 640 downstream models, the paper lacks sufficient detail and analysis regarding these experiments. They should provide comprehensive information about these models in the paper

questions:
- How would the proposed approach perform on more challenging real-world datasets like GQA or CRIC? 
- How would the findings about the benefits of object-centric representations translate to more challenging scenarios in real datasets?
- Why did the authors choose to focus on transformer encoder architectures for downstream models? Would incorporating decoder-based architectures potentially lead to different conclusions?
- Could the authors provide more detailed analysis of experiments with 640 downstream models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731664038302

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We thank all reviewers for their helpful feedback. We are happy to see that the reviewers find our paper well-written (feeJ), with important insights for the community (7see, feeJ) supported by extensive experimentation (rPSW, 7see, HU9k, feeJ).

Common concerns were raised regarding the need for experiments on additional real-world datasets. We appreciate this valuable feedback and fully acknowledge its importance. To address this, we have started integrating the GQA dataset into our framework. Given the effort and computational resources needed, we expect to have the initial results ready a few days before the discussion deadline. We will post an update here and hope the reviewers will consider this in their assessment.

In the meantime, we will address your other concerns and questions in our responses to each review.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733300864842

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Discussion Outcomes and Manuscript Improvements

comment:
Dear AC and Reviewers,

As the rebuttal period approaches its end, we would like to offer a brief summary of the discussion outcomes and highlight the improvements made to our manuscript based on the valuable feedback we received:

* **Request for an additional real-world dataset (raised by all reviewers as a major concern)**: We added experimental results on GQA during the discussion period, which fully supported our claims in the paper. The new results and the required effort were acknowledged by all the reviewers, however only reflected in the scores of rPSW and HU9k.
* **Suggestion of using transformer decoders instead of transformer encoders as the downstream model (rPSW)**: We explained why the current pipeline is suitable and sufficient for our goal. This explanation was acknowledged by rPSW and reflected in their updated score.
* **Some of the conclusions, whether it be major or minor, are intuitive or well-known (7see and HU9k)**:
    * **7see raised concerns about the intuitive nature of conclusions, such as the combination of the OC inductive bias and a foundation model improving performance over the foundation model alone**: We emphasized that while intuitive to some, this has not been systematically tested for OC models in downstream reasoning tasks like VQA. Additionally, by analyzing performance, explicitness of representations, and computational requirements, we provide insights beyond performance alone to enhance understanding of OC representations and their applicability.
    * **Later, 7see raised a broader concern about the intuitiveness of most conclusions and maintained their score.**
    * **HU9k raised concerns about the intuitiveness of specific conclusions, such as "Consistency of the Results Across Question Types"**: We explained that while it is a minor takeaway, this conclusion has not been definitively demonstrated for OC models. Additionally, the differences in model training approaches make it plausible that certain representations might perform better on specific question types.
    * **HU9k later expressed concerns about the intuitiveness of some or most of the overall conclusions in the paper, referencing the final concern raised by 7see**: We further raised two main points: (1) it is not sufficient to overlook the analysis of an idea simply because it is intuitive, and (2) intuition does not always align with reality in the literature after proper investigation, as demonstrated e.g. in the disentanglement literature. This response was acknowledged by HU9k, as reflected in their updated score. While we were responding to HU9k, we believe this response also effectively addresses the concern raised by 7see, although this was not explicitly acknowledged.
* **Confusion about the core question of our study (HU9k)**: We clarified the core question of our study, which was accepted by HU9k, as indicated by their increased score.
* **Missing analysis of the applications of the findings (feeJ)**: We explained the main contributions of our study and the takeaway messages for the community. According to feeJ’s response, this addressed the reviewer’s concern to some extent but did not affect the final score.
* **Effect of fine-tuning the foundation model on the downstream VQA task to make comparisons more fair (feeJ)**: We described our initial experiments of fine-tuning DINOv2 on VQA and explained that while downstream performance improved slightly, the gains were not significant and introduced substantial computational costs, making comparisons with OC models less fair in terms of compute requirements. Although the reviewer acknowledged our efforts, this did not lead to an increase in the final score.

We believe we have addressed all major concerns raised by the reviewers during the rebuttal. Reviewers rPSW and HU9k notably increased their scores from 3 to 6 and from 5 to 6, respectively, recognizing the improvements made to our manuscript and expressing a positive inclination towards its acceptance. On the other hand, reviewers 7see and feeJ maintained their initial scores, without raising any further concerns or questions.

In conclusion, we are deeply grateful to the Area Chair for their guidance and support throughout the review and discussion process. We also sincerely thank the reviewers for their valuable time and constructive feedback, which have significantly improved our manuscript. We hope this summary provides helpful context regarding the efforts and progress made during the rebuttal phase.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission conducts an empirical study on the impact of object-centric representations for visual question answering, and aims to put the observations into perspective with "foundation models". The original submission primarily relied on synthetic datasets for their analysis, and the authors provided additional results on GQA, on which the main observations align with the original claims. After rebuttal, the submission received four borderline accept (6) ratings. All reviewers appreciate the GQA experiments as demonstration on the generalizability of the empirical study on real world benchmarks. While some of the observations might be intuitive or partially demonstrated by prior work, as pointed out by reviewer HU9k, the AC agrees with the authors' response that the systematic study presented by the submission, and the take-away messages summarized by the authors (in response to the question "I am not sure how these takeaway messages could contribute to the community") are valuable to the ICLR research community. The AC therefore agrees with the consensus reached by the reviewers and recommends acceptance of the submission to ICLR 2025.