Mind the GAP: Glimpse-based Active Perception improves generalization and sample efficiency of visual reasoning

Authors: Oleh Kolner, Thomas Ortner, Stanisław Woźniak, Angeliki Pantazi
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: visual reasoning, active vision, out-of-distribution, generalization, sample efficiency, relational features, brain-inspired, neuro-inspired

Abstract:
Human capabilities in understanding visual relations are far superior to those of AI systems, especially for previously unseen objects. For example, while AI systems struggle to determine whether two such objects are visually the same or different, humans can do so with ease. Active vision theories postulate that the learning of visual relations is grounded in actions that we take to fixate objects and their parts by moving our eyes. In particular, the low-dimensional spatial information about the corresponding eye movements is hypothesized to facilitate the representation of relations between different image parts. Inspired by these theories, we develop a system equipped with a novel Glimpse-based Active Perception (GAP) that sequentially glimpses at the most salient regions of the input image and processes them at high resolution. Importantly, our system leverages the locations stemming from the glimpsing actions, along with the visual content around them, to represent relations between different parts of the image. The results suggest that the GAP is essential for extracting visual relations that go beyond the immediate visual content. Our approach reaches state-of-the-art performance on several visual reasoning tasks being more sample-efficient, and generalizing better to out-of-distribution visual inputs than prior models.

Decision: accept (poster)
Reviewer scores: 8, 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730444222926

thread_type:
official_review

<thread_object>
creator: reviewer_kwe3

<content>
summary:
This paper discusses how to improve the performance of visual reasoning by simulating the active perception of humans, especially when dealing with unknown objects. By expressing the relationship between different image parts with the position generated by glimpse-based active perception and the visual parts around them, this approach reaches a better sampling efficiency and generalization performance.

strengths:
1. It is really inspiring to simulate human's active perception to improve the performance of visual reasoning, which is a contribution to the community of visual reasoning.
2. The approach achieves better sampling efficiency and generalization performance.
3. The authors provide extensive experiment results and analysis.

weaknesses:
1. For the discussion of generalization ability, the paper focuses on the OOD data of the specific benchmarks. However, what we hope to achieve is the ability that models can deal with totally different tasks like humans, although it is really though. This paper lacks the analysis of it.
2. Recently, many LLM-based approaches are gaining higher performance on the reasoning tasks like VQAs. More analysis of the relationship between pure abstract reasoning like this approach and LLM-based reasoning is desirable.

questions:
1. What concerns me most is the generalization ability of this approach, although it is inspiring and fancy. Could you give me some clues that this approach can deal with tasks across different domains to reach the authentic reasoning ability?
2. Many people assume that LLM-based reasoning is closest to the generalization due to its rich knowledge. I suppose this approach would also be helpful to the LLM community. Would you conduct some analysis on the relationship between this approach and LLM-based reasoning?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730498267348

thread_type:
official_review

<thread_object>
creator: reviewer_8myj

<content>
summary:
The paper presents an architecture that performs visual perception based on local glimpses. It uses a saliency map to determine glimpse positions, and then feeds an encoding of the appearance and of the location of the glimpses to an existing downstream architecture - a Transformer or the recently proposed Abstractor. The glimpse extraction is hardcoded while the downstream architecture can be trained. The architecture is evaluated on reasoning tasks that rely on local structure and spatial relations.

strengths:
The paper explores an architecture that make explicit the positions and not just appearance of local image regions. These types of architecture are interesting because of the potential capabilities they can enable, and because of their relation to human vision. The paper is well written and fairly easy to follow. The proposed architecture is very simple.

weaknesses:
The proposed architecture can be viewed as a local feature extractor that encodes local image regions and their positions. There is very extensive prior work in this area, which makes the novelty somewhat limited. The proposed architecture is evaluated on two types of toy task (including some variants of those tasks): SVRT with/without OOD setting and ART/CLEVR-ART. All tasks use simple synthetic images with highly local structure in entirely uncluttered scenes, and their solutions seems to strongly rely on positions of objects. One would expect any architecture that makes those positions explicit to do exceptionally well on these specific tasks. Training sets for these tasks are also extremely small and will thereby favor architectures that are based on hardcoded (not trained) feature extraction. Given the extensive prior work on foveated vision, how do existing methods perform on these tasks?

questions:
I'm not sure about the term "active", since the architecture relies on a fixed (and not adaptive or recurrent) scheme to extract local features? 

Why is the dataset size for results in Table 1 restricted to 500/1000 training examples? This seems arbitrary. Figure 4 seems to suggest that the training set is larger? 

Can the SVRT #1-OOD experiment be extended to other subsets than SVRT #1? 

Is there a reason why the results in Table 3 (on a subset of ART) do not include all models considered previously (e.g. Attn-ResNet?) 

It would be very beneficial to include a modern vision-language model in the evaluation, as these models have become very good at solving similar reasoning tasks. 

Some additional related work: 

"Learning to combine foveal glimpses with a third-order Boltzmann machine", Larochelle et al. 2010 

"Multiple Object Recognition with Visual Attention", Ba et al. 2014 

"Show, Attend and Tell: Neural Image Caption Generation with Visual Attention", Xu et al. 2015
</content>
</thread_object>

</thread>
<thread>
creation date:
1730654101328

thread_type:
official_review

<thread_object>
creator: reviewer_1rv1

<content>
summary:
The authors develop a system equipped with a novel Glimpse-based Active Perception (GAP) that sequentially glimpses at the most salient regions of the input image and processes them at high resolution. Their approach reaches state-of-the-art performance on several visual reasoning tasks being more sample-efficient, and generalizing better to out-of-distribution visual inputs than prior models.

strengths:
1. The paper introduces a novel "Glimpse-Based Active Perception" (GAP) model inspired by human active vision. 
2. The proposed approach reaches state-of-the-art performance on several visual reasoning tasks being more sample-efficient, and generalizing better to out-of-distribution visual inputs than prior models.

weaknesses:
1.  The images in the four selected datasets seem relatively simple, with very clean backgrounds. Have you considered comparing your proposed model with baseline models on more realistic image datasets, such as the COCO dataset?
2. Given that the GAP mechanism involves multiple steps (e.g., saliency map generation, inhibition of return), have you compared its computational performance with other baseline models?
3. Why can GAP-Abstractor improve OOD generalization of same-different relation and more abstract relations?

questions:
1. Have you explored alternative methods for computing the saliency map, given that GAP's effectiveness largely depends on the map's accuracy in identifying key image regions?
2. What is the impact of using a hard versus a soft mask M(x_t)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694482244

thread_type:
official_review

<thread_object>
creator: reviewer_xwhi

<content>
summary:
This paper proposes a novel method for visual reasoning, called Glimpse-based Active Perception (GAP). Based on that the human eye selectively concentrates on salient and/or task-relevant parts of a scene, the authors devise a method that extracts salient regions of an image and feeds them into downstream architectures to enforce perception of salient regions only. Firstly, a saliency map is built based on error neurons, which marks salient regions as those that differ significantly with their surrounding neighbours. Secondly, salient regions are extracted from the image, using the saliency map, with either the multi-scale or log-polar glimpse sensor. Lastly, these salient regions and their locations are fed into either Transformer or Abstractor to perform visual reasoning tasks, each dubbed as GAP-Transformer and GAP-Abstractor. By forcing the downstream architectures to concentrate on salient regions only, the GAP-Abstractor achieves SOTA or comparative-to-previous-SOTA performance on the conventional visual reasoning benchmarks, OOD benchmarks, and real-image-based OOD benchmarks.

strengths:
- Authors propose a well-motivated method: 
  - They explain the motivation behind GAP logically, linking back to the human perception model throughout their method section. 
  - Ie. They explain that they concentrate on salient regions since "humans use active vision"; They explain that they use the concept of error neurons since "the activity of neurons is influenced ... also by stimuli that come from the surroundings of those receptive fields". 

- The proposed method attains strong performance on benchmarks:
  - Especially compared to previous work that do not use pretraining as GAP, GAP-Abstractor outperforms them by large margins. 
  - Authors also infer on four datasets, each with different purposes (testing visual reasoning only vs OOD generalization too) and different domains (binary figures vs real-world images). 

- The paper is well-written and easy to follow with both textual and visual explanations. 

- This paper introduces a novel and logical method to field of Visual Reasoning, which is of great importance since visual reasoning is required for a range of real-life tasks, and thereby contributes to the community.

weaknesses:
- Authors do not make it explicit or clear that different sensors (multi-scale or log-polar) have been used across different datasets for the same downstream architectures (ViT/Abstractor). Ie. For SVRT, GAP-Abstractor used the multi-scale sensor but for SVRT #1-OOD it used the log-polar sensor. This should be made clear in the main paper, instead of referring to the Appendix. 
  - Also, in Appendix D, authors do not explain why each sensor works better or worse for each downstream architecture-dataset pair. Since the performance depends on the type of sensor and since authors have proposed to use both, authors should explain where each sensor may be a better choice. 

- References are missing in Tables and Figures. 
- Explain what the tasks RMTS and ID are, at least briefly, in Section 5.4 before explaining the results.

questions:
- Can the authors provide an explanation to why GAP-Abstractor perform worse on 'straight lines' and 'rectangles' compared to other classes? 
- Since the authors explained that "glimpsing behavior can be distracted by spurious salient locations such as edges of shades or regions of high reflection", would performing GAP (finding salient regions) on feature maps, instead of images, where these spurious features are probably less influential, increase performance? Can authors provide results? 
- Please explain why each sensor works better or worse for each downstream architecture-dataset pair.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730958460893

thread_type:
official_review

<thread_object>
creator: reviewer_yjpp

<content>
summary:
The paper focuses on understanding visual relations. This remains a challenging problem for current vision models. To deal with this challenge the paper leverages active vision where the learning of visual relations is grounded in actions that we take to fixate objects and their parts by moving our eyes. The proposed approach with glimpse-based active perception demonstrates promising performance on a range of visual reasoning tasks.

strengths:
* The paper provides interesting insights into visual reasoning problems.
* The proposed glimpse-based active perception is relatively novel and interesting.
* The proposed approach shows promising performance on a range of visual reasoning problems.
* The paper considers diverse visual "sensors" and downstream architectures.

weaknesses:
* While the proposed approach is somewhat novel it is similar to prior work such as "AdaGlimpse: Active Visual Exploration with
Arbitrary Glimpse Position and Scale, ECCV 2024" which also focus on what and where to look.

* Current state of the art vision-language models such as LLaVA (NeurIPS 2024) keep the visual features from the target image in the context window. This means that they can actively attend to the image as many times are necessary to extract visual features. It would be interesting to compare the proposed approach to current SOTA VLMs such as LLaVA.

* Prior work such as "Look, Remember and Reason: Grounded reasoning in videos with language models, ICLR 2024" used surrogate tasks to decide what and where to look at. It would be beneficial to include a discussion of this related work.

* Synthetic data: Three out of the four datasets used for evaluation are based on synthetic datasets. It would be beneficial to include more real world datasets such as GQA or Super-Clevr.

questions:
* The paper should include a broader discussion of related work (see above).
* The paper should better motivate the choice of evaluation datasets.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732270231499

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Study on LLMs’ capabilities of solving same-different task

comment:
To address the common questions of reviewers about how well visual LLMs (VLMs) solve the visual reasoning tasks considered in our work, we conducted an experiment with Pixtral 12B model (Agrawal et al., 2024). We chose this recently introduced open-source model due to its superior performance in various visual reasoning benchmarks.

We evaluated Pixtral on the same-different task from the SVRT dataset where the model has to determine whether two shapes are the same up to translation. Below we list the accuracy for different prompts and report the best performance in a table along with the performance of our GAP-Abstractor model on these images for comparison (this value corresponds to the first bar on the left in Figure D.6).

- prompt 1: 62%
	- Answer the following question using only information from the image with "yes" or "no". Question: Are the two shapes the same up to translation?
- prompt 2: 63%
	- Are the two shapes in the image the same up to translation? Answer with "yes" or "no".
- prompt 3: 67%
	- Are the two shapes in the image the same? Please answer "yes" or "no".
- prompt 4: 71%
	- Is the second shape in the image an identical copy of the first shape just drawn at a different location? Please answer "yes" or "no".
- prompt 5: 67%
	- You see an image from SVRT dataset that tests visual reasoning capabilities. Answer the following question using only information from the image with "yes" or "no". Question: Are the two shapes the same up to translation?
- prompt 6: 58%
	- Given an image with two shapes, you have to say 'yes' if they are the same up to translation or 'no' if they are different. The first example image contains two shapes that are the same and the second example contains shape that are different. Given now the following image with two shapes, are they the same up to translation? Please answer 'yes' or 'no'
		- Note: This prompt included two images as examples for the “same” and “different” cases

|              Model               | Accuracy |
| :------------------------------: | :------: |
|          GAP-Abstractor          |   99%    |
| Pixtral (best performing prompt) |   71%    |

Importantly, a related work mentioned by one of the reviewers (Bhattacharyya et al., 2024) shows that “guiding” a VLM to sequentially attend to the relevant parts of the visual input can improve its reasoning capabilities. Hence, our approach has the potential to be combined with VLMs to provide such guidance through the glimpsing.

References
- Agrawal, Pravesh, Szymon Antoniak, Emma Bou Hanna, Devendra Chaplot, Jessica Chudnovsky, Saurabh Garg, Theophile Gervet et al. "Pixtral 12B." arXiv preprint arXiv:2410.07073 (2024).
- Bhattacharyya, Apratim, et al. "Look, Remember and Reason: Grounded Reasoning in Videos with Language Models." _The Twelfth International Conference on Learning Representations, 2024_.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732270518224

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Experiment with more realistic objects and alternative saliency maps

comment:
We evaluated our model on more realistic objects. Additionally, we show how different saliency maps may improve the model’s performance in this case.  

In particular, we used the objects from the Super-CLEVR dataset (Li et al., 2023), including cars, planes and motorbikes, to generate a new test set for the Identity Rules (ID) task from the CLEVR-ART dataset. The task is to determine whether three objects in the top row are arranged under the same abstract rule as the objects the bottom row. For example, objects “bus”, “car” and “bus” in one row are arranged under the abstract rule ABA just as objects “car”, “bus” and “car.  We considered four models trained on the original CLEVR-ART train set that consists only of simple cubes (Figure 6), and evaluated them on the new dataset, referred to as Super-CLEVR test set. The example images from the Super-CLEVR test set are shown in Figure E.8 in the revised manuscript.

Simultaneously, we explored using different saliency maps. The first model is our GAP-Abstractor model with the original error neurons applied on the raw image to extract the saliency map, we refer to this model here as GAP-Abstractor (EN). The second model, referred to as GAP-Abstractor (CNN+EN), is a modification of GAP-Abstractor that extracts the saliency map by applying the error neurons on the feature maps computed by the first layer of ResNet-18 that was pre-trained on ImageNet. As suggested by one of the Reviewers, computing the saliency map from the feature maps should make the glimpsing behavior less distracted by spurious salient locations such as edges of shades or regions of high reflection. The third model, referred to as GAP-Abstractor (IK), employs an alternative way to extract the saliency maps using a more complex bio-plausible model proposed by (Itti and Koch, 1998). The example saliency maps are shown in Figure E.8.

The table below shows the corresponding results for five runs with the best and second-best accuracy being in bold and italics, respectively.  Slot-Abstractor is the best performing prior art providing the baseline.
| Model                       |  original  test set | Super-CLEVR  test set  |
| :-------------------------- | :----------------------: | :-------------------------: |
| Slot-Abstractor (prior art) | 91\.61 ± 0.2             | 24\.1 ± 0.8                 |
| GAP-Abstractor (EN)         | **93\.4 ± 1.3**          | 72\.3 ± 2.3                 |
| GAP-Abstractor (CNN+EN)     | _92\.7 ± 1.8_              | _80\.2 ± 1.9_                 |
| GAP-Abstractor (IK)         | 90\.2 ± 2.1              | **82\.1 ± 2.2**             |

As it can be seen, Slot+Abstractor fails completely on the unseen Super-CLEVR objects while the performance of our GAP-Abstractor does not degrade so severely. Moreover, we see that GAP-Abstractor (CNN+EN) performs better on Super-CLEVR test set while performing comparably on the original test set. Therefore, applying the error neurons on top of features maps rather than on top of the raw images results in a saliency map that can better drive the glimpsing behavior. 
Lastly, we see that GAP-Abstractor (IK) with an alternative bio-plausible way to extract the saliency map reaches the best performance on the Super-CLEVR test set while being slightly worse on the original test set. A possible reason for this difference is that this saliency map extractor was designed to handle real-world scenes which are of much higher visual complexity compared to simple shapes. Overall, these results indicate a possible direction of enhancing our approach for visual reasoning by integrating more powerful saliency maps. 

References
- Li, Zhuowan, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L. Yuille. "Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14963-14973. 2023.
- Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(11), 1254–1259. IEEE Transactions on Pattern Analysis and Machine Intelligence.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733242800250

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank the reviewers for their time and effort in thoroughly evaluating our submission. We greatly appreciate their thoughtful feedback and constructive suggestions, which have significantly contributed to improving the quality of our paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a Glimpse-based Active Perception (GAP) system that sequentially focuses on salient image regions, leveraging spatial information from eye-movement-inspired actions to represent visual relations, achieving state-of-the-art performance in visual reasoning tasks with improved sample efficiency and out-of-distribution generalization.

The manuscript is well-written, presenting an innovative method with convincing results that surpass the current state-of-the-art, while offering valuable insights into plausible mechanisms for visual reasoning on object relations.

Key concerns, including the use of more realistic image datasets and alternative saliency map computations, were satisfactorily addressed during the rebuttal.

Following the rebuttal, all reviewers agreed to accept the paper.