Mind the GAP: Glimpse-based Active Perception improves generalization and sample efficiency of visual reasoning
Authors: Oleh Kolner
Thomas Ortner
Stanisław Woźniak
Angeliki Pantazi
Venue: ICLR 2025 Poster (2025)
Keywords: visual reasoning
active vision
out-of-distribution
generalization
sample efficiency
relational features
brain-inspired
neuro-inspired
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Human capabilities in understanding visual relations are far superior to those of AI systems, especially for previously unseen objects. For example, while AI systems struggle to determine whether two such objects are visually the same or different, humans can do so with ease. Active vision theories postulate that the learning of visual relations is grounded in actions that we take to fixate objects and their parts by moving our eyes. In particular, the low-dimensional spatial information about the corresponding eye movements is hypothesized to facilitate the representation of relations between different image parts. Inspired by these theories, we develop a system equipped with a novel Glimpse-based Active Perception (GAP) that sequentially glimpses at the most salient regions of the input image and processes them at high resolution. Importantly, our system leverages the locations stemming from the glimpsing actions, along with the visual content around them, to represent relations between different parts of the image. The results suggest that the GAP is essential for extracting visual relations that go beyond the immediate visual content. Our approach reaches state-of-the-art performance on several visual reasoning tasks being more sample-efficient, and generalizing better to out-of-distribution visual inputs than prior models.

Published as a conference paper at ICLR 2025
MIND THE GAP: GLIMPSE-BASED ACTIVE PERCEP- TION IMPROVES GENERALIZATION AND SAMPLE EFFI- CIENCY OF VISUAL REASONING
Oleh Kolner1.2, Thomas Ortner', Stanislaw Wožniak' & &Angeliki Pantazi'
IBM Research Europe- -Zurich, Switzerland 2 Institute of Machine Learning and Neural Computation, Graz University of Technology, Austria ol olkezurich, ibm. com
ABSTRACT
Human capabilities in understanding visual relations far superior those of Al systems, especially for previously unseen objects. For example, while Al systems struggle determine whether two such objects are visually the same or different. humans can do so with ease. Active vision theories postulate that the learning of visual relations grounded in actions that we take to fixate objects and their parts by moving our eyes. In particular, the low-dimensional spatial information about the corresponding eye movements is hypothesized to facilitate the representation of relations between different image parts. Inspired by these theories, we develop a system equipped with a novel Glimpse-bart based Active Perception (GAP) that se- quentially glimpses the most salient regions of the input image and processes them at high resolution. Importantly, our system leverages the locations stem- ming from the glimpsing actions, along with the visual content around them, represent relations between different parts of the image. The results suggest that the GAP is essential for extracting visual relations that go beyond the immedi- ate visual content. Our approach reaches state-of-the-ar performance on several visual reasoning tasks being more sample-effcciett and generalizing better to out- of-distribution visual inputs than prior models.
INTRODUCTION
Humans use vision both to detect objects and to analyze various relations between them. For ex- ample, to successfully grasp an object from a table, we should first see whether some other object is not lying top of it. Such relations can be spatial, like in the aforementioned example, more abstract, e.g. if two objects are the same or different. While humans can easily recognize many kinds of relations, even between previously unseen objects (Fleuret et al. 2011 Kotovsky & Gen- tner 1996). growing body of literature suggests that artificial neural networks (ANNs) struggle with this (Kim et al. 2018; Ricci al. 2021 Puebla & Bowers 2022 2024). Typically, ANNs pro- cess visual information holistically with each part of an image being processed in parallel with the others. In contrast, humans use active vision by selectively moving their eyes glimpse at salient and/or task-relevant parts of the image and to process those parts in high resolution (Yarbus) 1967] Hayhoe Ballard 20055 Land 2009). Moreover, by moving the eyes, the brain is able to form rep- resentations of the corresponding glimpse locations. This process can also be seen as factorization of the visual scene into its visual (""hatt") and spatial ("where")) contents (Goodale & Milner 1992, Behrens et al.) 2018). It is hypothesized that the low-dimensional representation and the relational geometry between those locations form scaffold for representing the image structure, i.e. relations between different image parts (Summerfield et al.) 2020
Currently, the best-performing ANNs, proposed for visual reasoning, use completely different mech- anisms compared to humans (Webb et al.) 2024a Mondal et al. 2024). Specifically, they combine an object-centre representation using slot attention (ocaaeelo et al.] 2020 and a special inductive bias called relational bottleneck (Webb et al.) 2024b). The former aims decomposing an image into set of representations corresponding to its objects and the latter encourages the model to focus on the relations between objects' representations instead of the representations themselves. While showing

---

Published as a conference paper at ICLR 2025
compelling performance on several visual reasoning tasks, these ANNs struggle to generalize the learned rules to out-of-distribution (OOD) objects (Puebla & Bowers 2024). A likely reason is that the methods for object-centric representation cannot reliably capture the relations between different parts of objects. Instead, they merely perform perceptual segregation through clustering (Mehrani & TSotsos 2023). Inspired by active vision, alternative model, proposed by /aiihnav & Serre (2023), processes images iteratively using a soft attention mechanism controlled by memory mod- ule. However, in every iteration, this model still processes the image in its entirety only masking specific regions. Moreover, it lacks a www-dieessional spatial representation of task-relevant regions.
Drawing more inspiration from humans's active vision, and the aforementioned theories behind it, we introduce a model equipped with novel Glimpse-base basse Active Perception (GAP) that sequen- tially glimpses at the most salient parts of the image. The GAP produces dual sequence of glimpse locations and glimpse contents around them, promoting spatial "where" and visual "what" path- ways. Importantly, the glimpse locations in the "where" pathway convey purely spatial information that is complementary to the visual contents of the "what" pathway. In particular, the geometrical relations between those locations provide a basis for extracting structural aspects of the image that also be present in image with different visual contents. The GAP can be paired with various downstream architectures that can process the dual sequence to solve the task at hand. In this work, use Transformer (Vaswani et al. 2017) and its recently introduced extension, called Abstractor (Altabaa et al. (2024). We evaluate our model on four visual reasoning datasets testing OOD gen- cralization capabilities and sample efficiency. Achieving Kate-offtheearr performance, our model provides computational support for the active vision theories.
Our contributions can be summarized as follows:
We present a brain-inspired active vision system for visual reasoning that extracts structural aspects of images by sequentially glimpsing at their salient parts.
We show that our system outperforms state-of-the-art models on several benchmarks in terms of sample efficiency and OOD generalization.
Our model is amenable to the integration of more advanced components for both saliency detection and downstream architecture which allows further scaling and extension. Our results support cognitive science theories postulating that the brain factorizes images into their visual and spatial contents with the latter being crucial for OOD generalization.
APPROACH
We introduce a model that learns to extract relations between different parts of an image by fac- torizing it into its visual ("what") and spatial ("where") contents. This is achieved by sequentially glimpsing at the most salient image parts that comprise the visual content whereas the locations of those parts provide the spatial content. Our approach is illustrated in Figure Given an input image, our model employs a novel concept of error neurons to extract a saliency map where stronger activ- ities of the error neurons correspond to more salient parts, such as corners of the shapes presented in the image. The saliency map is then used to guide the glimpsing process by producing locations to the highest saliency, referred to as glimpse locations, in a winner-takes-all (WTA) fashion. To prevent re-visiting previous glimpse locations, inhibition-of-retum (IoR) mechanism updates the saliency map by masking out regions around the previously visited locations. These two steps are inspired by (Itti et al., 1998). Each glimpse location is passed to glimpse sensor that extracts the corresponding visual content around that location, referred to as glimpse content. The described process repeats for pre-defined number of iterations, producing sequence of glimpse contents and a sequence of glimpse locations. These two sequences represent the factorization of the image into its complementary "what" and "where" contents. Both sequences are processed by the downstream architecture that makes the final decision for the given the task.
2.1 GLIMPSE-BASED ACTIVE PERCEPTION
Glimpse- Climpse-based Active Perception (GAP) is a process of acquiring visual and spatial information by steering glimpse sensor to different parts of the visual scene. The steering is facilitated by saliency map that highlights the most salient image parts. While there are many ways to compute such saliency map, we use simple but efficient concept of error neurons (Figure a)) described
2

---

Published as a conference paper at ICLR 2025
Glimpse-based Active Perception
Glimpse contents
Glimpse sensor
"What"
Image
91
92
93
94
9a
Glimpse location
Downstream Architecture
Error neurons
Glimpse locations
(i) (+2 au) (Kl, ats) (rt, sal (3a, an)
Saliency map
"Where"
WTA
Inhibition-of-relurn
Figure 1: Architecture overview. Based on the input image, the layer of error neurons produces the saliency map where salient image parts are highlighted (brighter regions). Through WTA combined with inhibition-of-retura regions. For each such location, the glimpse sensor extracts the glimpse content from the region around it. The sequences of glimpse locations and glimpse contents are processed by the down- stream architecture to make the final decision about the presented task.
sequence of glimpse locations is produced that covers the most salient
Based on the saliency map E Rhs x hg, ws denote the saliency map's height and width, the model glimpses the most salient regions for a pre-defined number of iterations T. Specifically, at each iteration i, the next glimpse location 22 E N² is selected through WTA competition returning the 2D location of the highest saliency:
in detail Section 2.1.1
2t (T) arg max (St.g))
(I)
where S, is the saliency map at iteration t. To prevent the same location from being selected at subsequent iterations, the loR mechanism applies a mask M (x++) E xassws around this location, updating the saliency map:
(2) The mask M can be either hard or soft one. The hard mask consists of ones except for a round region of zeroes around the location æt as shown in red in Figure (b. The size of that region is a hyper-parameter. In the soft mask, the values at each mask location i, are computed via the radial basis function with the Gaussian kernel -))(())) with c being a hyper-parameter and ()) E N2 a vector location i.j.
S = St C (xt).
Given glimpse location tracted by sampling pixels from the area around it. This sampling process is conceived as obtaining information from a glimpse sensor that provides limited view mimicking thereby human eye that fixates different parts of the visual surroundings. The main goal of such a glimpse sensor is to strike balance between the size of the glimpse content and the amount of information be captured from the image. In this work, we implement two kinds of sensors: a multi-scale sensor and a log-polar sensor. First, the multi-scale sensor, introduced in (Mnih al.) 2014), extracts several patches of the same size hg x Wg. but with different resolutions, around the glimpse location, as illustrated in Figure ( (three patches are pictured). The resulting glimpse content Dt E xh x xer the concatenation of those patches where $g is the number of different scales supported by the multi-scale sensor and is the number of channels in the input image. The patch with the highest resolution corresponds to the patch cut out from the input image at the glimpse location in the orig- inal resolution. The other patches are of decreasing resolution and correspond to larger areas of the input image that are down-scaled to h, X Wy allowing to capture coarse information of the surround of the glimpse location. Second, the log-polar glimpse sensor is inspired by the human foveal vision (Schwartz 1977 Weiman & Chaikin 1979 Javier Traver & Bernardino 2010) and samples pixels according distribution defined by log-polar grid, illustrated in Figure 2 d). This means that more pixels are sampled closer to the glimpse location, and fewer pixels are sampled farther from it, producing a glimpse content 9t. RR X Xc; whose size equal single image patch.
(Ir, yı) E N2, the corresponding region glimpse content is ex-
3

---

Published as a conference paper at ICLR 2025
(a)
(b)
Saliency map extraction with error neurans
Saliency map
Inhibition of return O
Glimpsing process
P
Image
time
(d)
Multi-scale glimpse sensor
resolution
Log-polar glimpse sensor
Medium
resolution
High resolution
Figure 2: Components of Glimpse-based Active Perception. (a) Error neurons: sample neuron, marked by the blue circle in the saliency map on the right, computes the difference between the visual content in its central receptive field (blue square) and the visual content in its surroundings (area shaded in red). (b) Glimpsing process: at each iteration the WTT mechanism selects location of the highest value in the saliency map (corresponding to its brightest regions) that is passed the glimpse sensor extract the corresponding glimpse content (blue circles). Afterward, the inhibition- of-return mechanism masks out the selected location in the saliency map (shown by red circles) so that it does get selected again. (c) Multi scale glimpse sensor outputs concatenation of several patches (three are shown) of the same size each of which encompasses a region of different size and resolution around the glimpse location on the left (depicted by three grids). (d) Log- Log-polar glimpse sensor samples pixels according to the log-polar grid shown on the left providing high resolution towards its central region and low resolution towards the periphery. After the sampling process, the polar grid is warped into regular rectangular grid which results in the glimpse content on the right.
Using these procedures, the GAP ultimately produces a sequence of glimpse contents 91 Ju......... .T and a sequence of glimpse locations I1, 22........ æT, that are further passed to the downstream architecture described in Section 2.2
2.1.1 SALIENCY MAP EXTRACTION WITH ERROR NEURONS
In the primary visual cortex, it is observed that the activity of neurons is influenced not only by stimuli from their local receptive fields but also by stimuli that come from the surroundings of those receptive fields (Vinje & Gallant 2000 Keller al. 2020). From theoretical view, this influence can make the same visual stimulus be perceived as more or less salient, depending on how signifi- cantly it differs from its local surroundings (Treisman Gelade 1980p. Inspired by this, we propose a computational block called error neurons, illustrated in Figure aa),. to compute the saliency map S R x ws, Specifically, each location i,j of the saliency map S the corresponding error neuron computes the saliency as
S Agg [D(PL P.))] 1.KESuI((i.))
where P.J E Rhpp-pp-es is a flattened patch obtained from the input image I E Bhy around location j.j. The patch size, hp x wr, a hyper-paramente and hys wI,cr denote the input image's height, width, and the number of channels. Unless specified otherwise, h, = hy and wy Ws. D(.,.) is a distance function, such as sum minimum over distances between the patch at location i, and the patches from surrounding locations k E surr(i.j). The set of surrounding locations, ()) can defined arbitrarily, comprising, for example, all available image locations. In this work, assume the 8- connected neighborhood of location j. Intuitively, each value S;j represents how much the visual content around location i, differs (or stands out) from the surrounding visual content.
this work, set to L2 distance. Aggl-] is an aggregation operator
4

---

Published as a conference paper at ICLR 2025
2.2 DOWNSTREAM ARCHITECTURE
As detailed above, the GAP provides sequence of glimpse contents g and sequence of the cor- responding 2D glimpse locations æ to the downstream architecture. Importantly, our approach is flexible with respect to the choice of the downstream architecture. In this work, we explore two options: Transformers (Vaswani et al., 2017) and Abstractors (Altabaa et al. 2024).
Transformers have emerged over recent years as popular state-of-the-art architectures for a variety of tasks, including the vision domain with the Vision Transformers (ViT) Dosovitskiy et al. (22TT) Apart from their successes in machine learning tasks, there have also been works suggesting poten- tial links between ViT and biology (Pandey al.) 2023). The ViT processes sequence of image patches, augmented with positional encodings, to solve the given task. In our case, we provide the sequence of glimpse contents augmented with the respective glimpse locations as inputs.
The Abstractor has been recently introduced as extension of the Transformer, with the key dif- ference of the so-called Relational Cross Attention (RCA) module. It was designed to disentangle relational information from the perceptual one to enable generalization from limited data. This is achieved by modifying the self-attention so that the queries and the keys are produced based on per- ceptual inputs whereas the values are provided by separate independent set of representations. The latter, in our case, is substituted by the sequence of glimpse locations whereas the perceptual inputs comprise the sequence of glimpse contents. Hence, adopting the Abstractor allows for the explicit processing of spatial information so that the visual information carries only a modulatory effect in form of attention. The formal description of the Abstractor is provided in Appendix A
RELATED WORK
Inspired by human active vision, several models for learning visual relations have been introduced. By decomposing input images into a sequence of image regions obtained from synthetic eye move- ments, Wožniak et al. (023)) demonstrated that visual reasoning tasks can be solved with high accuracy using relatively compact networks. However, the synthetic eye movements were based on hard-coded locations, whereas in case, the glimpsing process is controlled by saliency map. The approach from (Larochelle & Hinton 2010) and its ANN-based extension (Mnih et al.] 2014) use reinforcement learning (RL) train a glimpsing policy to determine the next glimpse location either in a continuous 2D space or in a discrete grid-based space of all possible glimpse locations. While this approach was evaluated mainly on simple image classification and object detection tasks, Ba et al. (2014); Xu et al. (2015) extended it to more complex images and image captioning tasks. However, having been evaluated on several visual reasoning tasks by Vaishnav Serre (202)), the RL-based approaches could not achieve reasonable performance. This is likely caused by learn- ing inefficiency in exploring the large space of possible glimpse locations. Nevertheless, the RL-based approaches that use more efficient RL techniques such (Pardyl et al. (222) to learn complex glimpsing policies are relevant to our work as they can integrated into our model to enhance its capabilities of dealing with real-world images. Our approach, in contrast, leverages the concept of saliency maps to determine the next glimpse location which significantly reduces the space of glimpse locations to the most salient ones. Gregor et al. (2015) proposed variational auto-encoder that uses an attention mechanism for an iterative (re- cconstructionn of complex images by attending to their different parts. Its extension proposed by Adeli al. (2023) was designed to tackle visual reasoning tasks where it outperformed baseline CNNs. However, being trained by image reconstruction, the model does not always generalize well OOD visual inputs. Vaishnav & Serre (2023) introduced a recurrent soft attention mechanism guided by memory module that masks specific parts of the image where the processing should be focused. At each iteration, the model can thereby attend to multiple regions of the image simultaneously which is different from our approach where only one region can focused at time.
Another family of models combines object-centric representations (Greff et al., 2019 Locatello ctt al. 2020) with relational inductive biases (Webb et al., 2021 Kerg et al. 2022; Webb et al. 2024b). The former decomposes an image into objects present in it while the latter constrains the processing to relations between objects instead of their specific properties. Webb et al. (022a)) used pre-trained slot attention (Locatello et al., 2020) to segment objects into sequence of slots that contain disentangled representations of objects' visual features and their positional informa- tion. Each pair of slots is then processed by relational operator and the resulting sequence of
5

---

Published as a conference paper at ICLR 2025
relational representations is processed by the Transformer (Vaswani al. 2017). The successor model. Slot-Abstractor (Mondal et al. 2024), processes the sequence of slots with a recently intro- duced Abstractor module (Altabaa et a (0244) that proved to be especially effective in reasoning about relations between objects using their disentangled visual and positional features. Similar to our approach, there are two streams of information: one based on positional information. However, the latter stream consists of positional embeddings that are learned during the pre-training process which may make the model more fragile to OOD inputs, In contrast, our approach uses the raw 2D locations of salient regions. Interestingly, in the domain of visually extended large language models (LLMs), Bhattacharyya et al. (2024) showed that forcing these models via surrogate tasks to collect relevant pieces of information before solving the actual task improves the final performance. While using a different approach compared to ours to achieve that, this work points to the potential of integrating our conceptual approach into LLMss
visual features and one based
EXPERIMENTS
Datasets Our approach was evaluated on four visual reasoning datasets illustrated in Figure 3 We start with an evaluation on all 23 tasks of the SVRT dataset (Fleuret et al.] 2011] to see how effi- ciently the model can learn spatial and similarity relations between simple shapes, see an example in Figure a).. Next, we test the model's OOD capabilities to generalize learned relations to previ- ously unseen shapes. Specifically, we use an extended version of the task #1 from the SVRT dataset (Puebla & Bowers] (022), referred to as SVRT #1 #1-OOD, see Figure B(). It contains test sets with OOD shapes coming from 13 different distributions. Additionally, we use the ART dataset (Webb) et al. 2021), which contains more abstract relations, see Figure 3(c). Finally, we test the model's s potential to scale to more complex images by testing it on the CLEVR-ART dataset (Webb et al, 2024a), see Figure It contains two tasks from ART but with more realistic images. Each dataset is described in detail in Appendix B While all tasks described above consist of simple images, they are still challenging, as illustrated by the limited accuracy of the baselines described below.
Baselines We compare the impact of our approach with the plain downstream architectures that receive entire images as inputs. The images get split into non-overlapping patches and each patch gets augmented by its positional information also tested those models where the patches overlapped but did not observe any significant perfor- mance differences. Therefore, we only report the standard approach with non-overlapping patches. We refer to these models simply as Transformer and Abstructor. The comparison with these base- lines allows us to demonstrate the importance of processing only the relevant patches obtained from the glimpsing process rather than all patches that constitute an image. We also compare our ap- proach with GAMR (Vaishnav & Serre 2202), ResNet (He et al. 2015), its extension Resn-R Vaishnav et al. 2022), OCRA (Webb et al.) 2024a) and Slot-Abstractor (Mondal et al. 2024 To
is done by ViTs (Dosovitskiy et al. 2021). ). We
(a)
SVRT
(b)
SVRT #1-00D
(c)
ART
(d)
CLEVR-ART
Category
Category 2
Training set
Test sets
Filled
Lines
Arrows
Same
L
Different
Arig
Is
b
Figure 3: Four datasets used in our experiments. (a) SVRT dataset consists of 23 binary classifica-
tion tasks where the model has to determine whether shapes are in a certain spatial and/or similarity relation to each other. Shown are images from the task of determining whether the same triplets of shapes exist in each image pair. (b) SVRT #1-OOD instantiates the first task from SVRT where has to be determined whether two shapes are the same up to translation. While this dataset contains training images from the original SVRT dataset, it contains 13 additional test sets (only three are shown here) with novel shapes to evaluate OOD generalization. (c) ART datasct contains four tasks where the model has to determine whether shapes in each two rows are arranged under the same abstract rule. (d) CLEVR-ART contains two tasks from ART but with more complex images.

---

Published as a conference paper at ICLR 2025
the best of our knowledge, these baselines the best-perforning models on the tasks that consider.
5 RESULTS
In all experiments, report the performance of GAP combined with two downstream architectures, Transformer and Abstractor. The corresponding models are referred to as GAP-Transformer and -As--bbtracttr respectively. Details about hyper-parameters and training process described in Appendix [C, One important hyper-parameter is the type of glimpse sensor (multi-scale or log- polar) that we select for each of the four considered datasets. We report performance for different glimpse sensors in Appendix glimpse--bae -aseed---aad---ee- act aet aet.vve-eeeeeetieee.....eee. perception
D Our code is publicly available at hnttps://githtii www.ccm///M..
5.1 VISUAL REASONING PERFORMANCE AND SAMPLE EFFICIENCY
We evaluate the visual reasoning capabilities using the SVRT dataset that consists of 23 tasks, 11 of which require to reason about same-different relations (SI) tasks) and the remaining 12 tasks require to reason about spatial relations (SR tasks). Consistent with (Vaishnav & Serre [023] Webb al. 2024a Mondal et al. 2024), Table shows accuracy performance for models trained with 1000 and 500 samples. Our perfooorn model, the GAP-Abstractor, outperforms the prior art models. In particular, it achieves very high accuracy the SD tasks, considered to be more challenging than
Table 1: Test accuracy for SVRT evaluated for common dataset sizes of 1000 and 500 samples. The results obtained from the best trained models are averaged over SD, SR and over (SD SR) tasks. The standard deviation is calculated over accuracies for each individual task. Prior art results are listed in the top section whereas results obtained from our evaluations are listed in the two bottom sections. The highest and the eecond-highest mean accuracy in each column is marked in bold and underlined, respectively. The models appended by contain a module pre-trained on the train sets from all SVRT tasks to segment the shapes.
SD tasks accuracy [%]
SR tasks accuracy [%]
All tasks accuracy [%]
Model
Dataset size
Dataset size
Dataset size
Dataset size
Dataset size
Dataset size
1000
500
1000
500
1000
500
ResNet
2.5
5..9
94.9
94.8 +
76.7 2.0
70.7 70.7
Attn-ResNet
68.8 + 4.4
62.3 3.5
97.7 = 0.7
1.4
83.9
79.3 1 2.4
GAMR
$ 8.4
79.9 + 8.7
98.7 = 0.3
97.4
90.8 4.2
87.6
OCRA"
90.3 +: 4.0
95.0
2.5
92.8
84,8 3.5
Slot-Abstractor**
82.2 +.7
97.3 1.1
91.8
94.7
87,2
Transformer
51.9
51.6 1.7
64,8 13.7
59.9 11.0
$6.4 10.3
55.9 8.9
GAP-Transformer
83.3 13.2
78.4 13.5
98.2 2.2
2.7
91.1
88.3 13.4
Abstractor
5..
51.0 1.9
60.9 14.3
57.0 11.0
9.76
54.1 8.5
GAP-Abstractor
93.1 13.1
90.5 14.5
98.5 1.9
96.6 +
95.4
(a)
(b)
tasks
100
100
GAP-Abslractor Slot-Abstractor GAMR
90
Accuracy %
90
80
Altn-ResNel
ResNet
80
70
Tranffranfoomme
Abstractor
Transformer
70
50
10000
5000
1000
500
250
100
10000
5000
1000
290
100
Dataset size
Dalaset size
Figure 4: Sample efficiency evaluation for SVRT beyond common dataset sizes 1000 and 500 samples. Average test accuracy all 23 SVRT tasks depending the size of the training dataset. Panel (a) compares our best model with prior art. Panel (b) compares our glimpse-basek models with their ablated counterparts.

---

Published as a conference paper at ICLR 2025
Table 2: OOD performance for SVRT #1- OOD. The accuracy results are averaged over all OOD test sets for 10 trained models. Prior art results are listed in the top section whereas results obtained from our evaluations are listed in the two bottom sections. The highest and the second- second-highest mean accu- racy is marked in bold and underlined. respec- tively. The models appended by contain module pre-trained the train set segment the shapes.
irregular
regular
connected
circles
open
80
connected squares
70
60
wider line
straight lines
scrambled
rectangles
Accuracy [%] (averaged over runs
random
color
arrows
Model
filled
lines
GAMR
and OOD datasets)
Slot- Slot-Abstractor
ResNet
71.2 + 17.7
GAP- Transformer
GAP-Abstractor
GAMR
79.8 + 13.1
OCRA*
73.9 = 17.2
Figure 5: Accuracy on each of the SVRT #1- OOD test sets. The accuracy results (in %) are averaged over 10 trained models. Slot-Abstractor could not be properly evaluated on 'random color' test set that contains RGB images because its pre- trained component handles only grayscale images.
Slot-Abstractor*
72.8 + 16.0
Transformer
50.0
GAP- Tra-Tranfformer
76.3 18.4
Abstractor
66.2 + 16.0
GAt-Abstractorr
89.6 = 8.7
the SR tasks, for which it performs comparably with the prior art. It should be noted that although for the SR tasks, the rest-performing model is GAMR, its performance on SD tasks significantly lower compared with all other models. Detailed accuracy for each individual task is provided in Appendix, Figure [.6] Furthermore, Figure 4 shows an extensive evaluation of sample efficiency, in which we trained the models with different amounts of training data beyond those reported in Table D and in prior works. Compared with the results we obtained for the prior art models, our best model exhibits higher sample efficiency, especially in scenarios with fewer than 1000 training samples, see Figure a). Moreover, we observe favorable trends and improvements when applying GAP to either of the downstream architectures, see Figure[[b) More detailed results with separate performance for SD and SR tasks are provided in Appendix, Figure D.7
5.2 OUT-OR-DISTRIBUTION GENERALIZATIN OF SAMB-DIFFERENT RELATION
While the SVRT dataset allows to whether model can learn visual relations from limited amount of training data, does not allow us to test the model's ability to generalize those relations OOD inputs. We evaluate the OOD generalization of single same-different relation using the SVRT #1-OOD dataset (Appendix B.2) where a model has to determine whether two shapes are the same or different up to translation. Importantly, the models are evaluated test with OOD shapes coming from 13 different distributions (see Figure B.2 for examples). Table 2 shows the performance averaged over OOD test sets and Figure shows the performance on each of the test sets. Outperforming the prior art models, our model, GAP-Abstractor, performs quite consis- tently on test sets being slightly less accurate on only two sets, 'straight lines' and 'rectangles'. Figure B.3 illustrates the reason why these two sets are more challenging.
5.3 OUT-OF-DISTRIBUTION GENERALIZATION OF MORE ABSTRACT RELATIONS
Going beyond evaluations of OOD generalization on a single task, we expand the evaluations to four tasks from the ART dataset, three of which contain more abstract relations (see Appendix(B.3 Moreover, the evaluations on this dataset validate also the insights on sample efficiency since the models are required to be trained on very few samples. From Table 3 it can be seen that the GAP- Abstractor outperforms all other models demonstrating superior capabilities to generalize abstract relations to novel visual inputs. Crucially, this performance improvement can be observed even though no component of our model was tre-trained, as opposed to OCRA or Boot-bbstrcctor that have been pre-trained segment objects.
8

---

Published as a conference paper at ICLR 2025
Table 3: Test accuracy for ART. The accuracy results are averaged over 10 trained models. Prior art results are listed in the top section whereas results obtained from our evaluations are listed in the two bottom sections. The highest and the second-highest mean accuracy each column is marked in bold and underlined. respectively. The models appended by contain a module to segment the shapes that was pre-trained on images with shapes distinct from those in the ART dataset.
Task accuracy [%]
Model
SD
RMTS
Dist3
GAMR
83.5 1.4
3.0
68.6 1.8
66.2 + 4.3
ResNet
66.6 1.5
49.9
50.1 1.3 1.3
54.8 2.4
OCRA*
87.9
86,4 2.0
92.8
cottAbbtractor*
9.4
91.6
9..2
9.1
Transformer
5.1
29.3 1.2 0.1
1.7
73.7
50.7 7.3
55.5
Abstractor
6.6
51.3 0.2
58,9 = 3.8
47.9
GAP-Abstractor
97.7
96.3 # 1.0
98.4 0.5
96.8 + 1.8
5.4 SCALING POTENTIAL FOR MORE COMPLEX VISUAL SCENES
We test our approach on two tasks from ART ollatoonllmatcch-tossmma
Table 4: Test accuracy for CLEVR-ART. The accuracy results are averaged over 5 trained models. Prior art results are listed in the top section whereas results obtained from our evaluations are listed in the two bottom sec- tions. The highest and second-highest mean accuracy in each column is marked in bold and underlined. respectively. The models ap- pended by contain module to segment the shapes that was pre-trained on images contain- ing shapes from both the train and test datasets.
(RMTS) and identity rules (ID) but with more realistic images us- ing the CLEVR-ART dataset. In the RMTS task the model has to determine whether the objects in the top row are in the same "same/differeet" re- lation as the objects in the bottom row. In the ID task the model has determine whether the bot- tom row contains objects that follow the same ab- stract pattern (ABA, ABB or AAA) as the objects in the top row (see Figure B.5). While the images from other datasets considered binary and contain only 2D objects, the CLEVR- ART images are colored and contain 3D objects with such real-world features as shades and re- flections. Hence, the glimpsing behavior can be distracted by spurious salient locations such as edges of shades or regions of high reflection. Table highlights that the GAP-Abstractor per- forms on with the state-of-the- model, Slot- Abstractor, being marginally better in one task and marginally worse in the other. However, Slot-
this work are
Task accuracy [%]
Model
RMTS
ID
GAMR
7..4
74.2
OCRA*
93.3 1.0
77.1 0.7
Slot-Abstractor*
9.5
9.2
Transformer
6..9
4.3
GrP-Transforeer
76.4
6.67
Abstractor
77.8
8.82
GAP-Abstractor
9..9
93.4 1.3
Abstractor was pre-trained on all shapes from CLEVR-ART being thereby potentially exposed to objects from both train and test sets. This is in contrast to models, which were not pre- traitrd and were exposed only to the objects from the train set. To compare our models with Slot-
Training set cubes
Original OOD test set cubes /sphere
New O0D-1 test set v /spheres Xpyramids
New OOD-2 test set Xpyramids Xtoruses
Slots were pre-trained
X No pre-training
Figure 6: Extension CLEVR-ART for more thorough evaluation of OOD generalization. Examples for the RMTS task shown here. The two new OOD test sets contain objects on which neither the models was pre-trained: OOD-I test set contains novel object (pyramid) and OOD-2 contains 2 novel objects (pyramid and torus).

---

Published as a conference paper at ICLR 2025
RMTS lask
ID task
Accuracy %
80
06
95 9
08.94
94.6
86
88
73.4
87.2
6B 68
60
Original
OOD-1
OOD-2
Original
OOD-1
OOD-2
Slot-Abstractor
GAP-Abstractor
Figure 7: Extended OOD evaluation CLEVR-ART. While the Slot-Abstractor was pre-trained on the original test set, the OOD-1 and OOD-2 test sets contain shapes on which neither of the evaluated models was pre-trained. The accuracy results averaged over 5 trained models.
Abstractor on a more equal footing, we generated two additional OOD test sets shown in Figure The new OOD-1 test set contains one novel type of shape -pyammid on which the Slot-Abstractor was not pre-trained. Similarly, the new OOD-2 test set contains 2 such novel shapes: pyramids and toruses. The results are shown in Figure 7 based model compared with Slot-Abstractor. In addition, we provide preliminary experiments with more realistic objects and more complex saliency map extraction Appendix E
demonstrating superior OOD generalization of our
DISCUSSION
Our results suggest that factorizing an image into its complementary "what" and "where" contents plays an essential role. The low-dimensional "where" content (glimpse locations) is crucial since does not contain any visual information and, in turn, allows learn relations that are agnostic to specific visual details. To capitalize on that, it is important to process the "where" content explicitly, disentangling from its "what" complement. We implemented this by using the recently introduced relational cross-attention mechanism (employed by the Abstractor downstream architecture) where "what" content used to compute the attention scores for the processing of the "where" content. In contrast, implicitly mixing the "whatt" and the "where" contents, in the case of the Transformer downstream architecture, weakens the generalization capabilities. This can be seen by comparing the performance between GAP-Transffrmar and GAP-Abstractor. To further support this, provide supplementary results in Appendix D.2 showing the importance of using both the "what" and the "where" contents instead of just one of them for task solving.
Another important aspect of our model is that processes only the most salient image regions while ignoring the unimportant ones. The inferior performance of models where the downstream archi- tectures receive full information (i.e. all patches that constitute the image) suggests that supplying superfluous amount of information may distract the models hindering the learning process. Ad- ditional results provided in Appendix [D33] elucidate this further. Specifically, we show that it is insufficient to naively reduce the amount of information provided the downstream architecture, by discarding uninformative image parts. Instead, it has to be ensured that the supplied informa- tion well structured.
context of GAP, means that glimpse contents have to be centered around salient image regions. For example, the image patches of the glimpse contents should contain objects' edges in their central parts rather than somewhere in the periphery.
CONCLUSION
Drawing inspiration from human eye movements and theories of active vision, we proposed a system equipped with the novel Glimpse-based Active Perception (GAP). The system selectively glimpses the most salient image parts and processes them high resolution. The relational geometry be- tween the corresponding glimpse locations complemented by the glimpse contents provides basis for reasoning about image structure, i.e. relations between different image parts. We evaluated our approach on four visual reasoning benchmarks and achieved state-of-the-art performance in terms of sample efficiency and the ability to generalize to OOD visual inputs without any pre-training. Our approach allows for further extension by integrating more advanced components. Particularly, the saliency map extraction can be substantially enhanced by either extending our concept of error neurons integrating more powerful components.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like to thank Wolfgang Maass for the helpful comments and fruitful discussions.
REFERENCES
Hossein Adeli, Seoyoung Ahn, and Gregory J Zelinsky. A brain-inspired object-based attention network for multiobject recognition and visual reasoning. Journal of Vision, 23(5):16-16, 2023.
Awni Altabaa, Taylor Whittington Webb, Jonathan D. Cohen, and John Lafferty. Abstractors and relational cross-attention: An inductive bias for explicit relational reasoning in transformers. In The Twelfth International Conference on Learning Representations, 2024.
Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual attention. arXiv preprint arXiv: 44 141.7755, 2014.
Timothy E.J. Behrens, Timothy H. Muller, James C.R. Whittington, Shirley Mark, Alon B. Baram, Kimberly L. Stachenfeld, and Zeb Kurth-Nelson. What is a cognitive map? organizing knowledge for flexible behavior. 100(2)490-550, 2018. ISSN 08966273.
Apratim Bhattacharyya, Sunny Panchal, Reza Pourreza, Mingu Lee, Pulkit Madan, and Roland Memisevic. Look, remember and reason: Grounded reasoning in videos with language models. In The Twelfth International Conference on Learning Representattonss 2024.
G. Bradski. The OpenCV Library. Dr. Dobb's 's Journal of Software Tools, 2000.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
François Fleuret, Ting Li, Charles Dubout, Emma K. Wampler, Steven Yantis, and Donald Geman. Comparing machines and humans on a visual categorization test. 108(43):17621 11625, 2011. ISSN 0027-8424, 1091-6490.
Melvyn A Goodale and A David Milner. Separate visual pathways for perception and action. Trends in Neurosciences, 15(1):20-25 1992.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-ohject representation learning with iterative variational inference. International Conference on Machine Learning. pp. 2424-2433. PMLR, 2019.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In International Conference on Machine Learning, pp. 1462-1471. PMLR, 2015.
Mary Hayhoe and Dana Ballard. Eye movements in natural behavior. Trends in cognitive sciences, 9(4):188-194, 2005.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, abs/15 1w/.15122.3385. 2015.
L. Itti, C. Koch, and E. Niebur. A model of saliency-based visual attention for rapid scene analysis. 20( [011] 12:144--22,9, 1998. ISSN 1939-3539. Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.
V. Javier Traver and Alexandre Bernardino. A review of log-polar imaging for visual perception in robotics. (8(4):378-398) 2010. ISSN 09218890.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
11

---

Published as a conference paper at ICLR 2025
Andreas Keller, Morgane M Roth, and Massimo Scanziani. Feedback generates second receptive field in neurons of the visual cortex. Nature, 582(7813):545-544 549,, 2020.
Giancarlo Kerg, Sarthak Mittal, David Rolnick, Yoshua Bengio, Blake Aaron Richards, and Guil- laume Lajoie. Inductive biases for relational tasks. ICLR2022 Workshop on the Elements of Reasoning: Objects, Struciure and Causality, 2022.
Junkyung Kim, Matthew Ricci, and Thomas Serre. Not-so-clevr: learning same different relations strains feedforward neural networks. Interface focus, 8(4):2018001, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.
Laura Kotovsky and Dedre Gentner. Comparison and categorization in the development of relational similarity. Child development, 67(6):2777-2822) 1996.
Michael F Land. Vision, eye movements, and natural behavior. Visual Neuroscience, 26(1):51-62, 2009.
Hugo Larochelle and Geoffrey E Hinton. Learning to combine foveal glimpses with third-order boltzmann machine. Advances neural information processing systems, 23, 2010.
Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma. Benjamin Van Durme, and Alan Yuille. Super-clevr: virtual benchmark to diagnose domain robust- ness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14963 14973, 2023.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion, 2020.
Paria Mehrani and John K Tsotsos. Self-attention in vision transformers performs perceptual group- ing, not attention. Frontiers in Computer Science, 5:1 5:1178450, 2023.
Volodymyr Mnih, Nicolas Heess, Alex Graves, and koray kavukcuoglu. Recurrent Models of Visual Attention. In Advances in Neural Information Processing Systems, volume 27, Curran Associates, Inc., 2014.
Shanka Subhra Mondal, Jonathan D. Cohen, and Taylor Whittington Webb. Slot abstractors: To- ward scalable abstract visual reasoning. In Proceedings of the 41st International Conference on Machine Learning. PMLR, 21-27 Jul 2024.
Lalit Pandey, Samantha Wood, and Justin Wood. Are vision transformers more data hungry than newborn visual systems? Advances in Neural Information Processing Systems, 2023.
Adam Pardyl, Michat Wronka, Maciej Wolezyk, Kamil Adamczewski, Tomasz Trzciáski, and Bar- tosz Zieliński. Adaglimpse: Active visual exploration with arbitrary glimpse position and scale. In European Conference on Computer Vision, pp. 112-129. Springer, 2025.
Guillermo Puebla and Jeffrey S. Bowers. Can deep convolutional neural networks support relational reasoning in the same-different task? 22(10):11, 2022. ISSN 1534-7362.
Guillermo Puebla and Jeffrey S. Bowers. The role of object-centric representations, guided attention, and external memory on generalizing visual relations, 2023.
Guillermo Puebla and Jeffrey S. Bowers. Visual reasoning in object-centric deep neural networks: A comparative cognition approach, 2024.
Matthew Ricci, Rémi Cadène, and Thomas Serre. Same-different conceptualization: machine
vision perspective. Current Opinion in Behavioral Sciences. 37:47-55, 2021.
Eric Schwartz. Spatial mapping in the primate sensory projection: analytic structure and relevance to perception. Biological cybernetics, 1814)118 1977.
12

---

Published as a conference paper at ICLR 2025
Christopher Summerfield, Fabrice Luyckx, and Hannah Sheahan. Structure learning and the poste- rоor parietal cortex. Progress in Neurobiology, 184:101717, January 2020. ISSN 03010082.
Anne M Treisman and Garry Gelade. A feature-integration theory of attention. Cognitive psychol- ogy, 12(1):97-136, 1980.
Mohit Vaishnav and Thomas Serre. GAMR: guided attention model for (visual) reasoning. In The Eleventh International Conference on Learning Representations, 2023.
Mohit Vaishnav, Remi Cadene, Andrea Alamia, Drew Linsley, Rufin VanRullen, and Thomas Serre. Understanding the computational demands underlying visual reasoning, Neural Computation, 34 (5):0055-1099, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, 30, 2017.
William Vinje and Jack L. Gallant. Sparse coding and decorrelation in primary visual cortex during natural vision. Science, 28764456 2000.
Taylor Webb, Zachary Dulberg, Steven Frankland, Alexander Petrov, Randall O'Reilly, and Jonathan Cohen. Learning representations that support extrapolation. In International Conference Machine Learning. pp. 10136-10146. PMLR, 2020.
Taylor Webb, Shanka Subhra Mondal, and Jonathan D Cohen. Systematic visual reasoning through object-centric relational abstraction. Advances in Neural Information Processing Systems, 36, 2024a.
Taylor W. Webb, Steven M. Frankland, Awni Altabaa, Simon Segert, Kamesh Krishnamurthy, De- clan Campbell, Jacob Russin, Tyler Giallanza, Randall O'Reilly, John Lafferty, and Jonathan D. Cohen. The relational bottleneck as an inductive bias for efficient abstraction. Trends in Cognitive Sciences, May 2024b. ISSN 1364-6613.
Taylor Whittington Webb, Ishan Sinha, and Jonathan Cohen. Emergent symbols through binding in external memory. In International Conference on Learning Representations, 2021.
Carl FR Weiman and George Chaikin. Logarithmic spiral grids for image processing and display. Computer Graphics and Image Processing, 11(3):197-226, 1979.
Stanistaw Wožniak, Hlynur Jónsson, Giovanni Cherubini, Angeliki Pantazi, and Evangelos Eleft- heriou. On the visual analytic intelligence of neural networks. Nature Communications, 14(1): 5978, 2023.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual at- tention. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2048-2057, Lille, France, 07-09 Jul 2015. PMLR.
Alfred L.. Yarbus. Eye Movements and Vision. Springer US, 1967. ISBN 978 489953797.
13

---

Published as a conference paper at ICLR 2025
A ABSTRACTOR
To describe the Abstractor-based downstream architecture and the way it is applied, we follow (Mondal et al. 2024). Specifically, given sequence of glimpse contents g (g1 9...99........ 9r) and their locations a = (01, 22....; Tr), the Abstractor computes a sequence of the correspond- ing relational representations correspond to the glimpse locations æ. Each subsequent layer / updates the relational represen- tations using multi-head relational cross-attention between the glimpse contents g (or their latent representations, if they are further pre- pre-processed by neural network):
series of L layers. The initial relational representations ri=o
(gWa)"
F/ softmax
ri-1
(4)
Vd
= RCA(C, r1-1) =
(5)
9: Wh. Wh Rdxd (d is the number of features of each head), the linear projection matrices used by the ht head to generate queries, keys, and values respectively; Fa is the result of relational cross-attention in the ht head; W, are the output weights through which the concatenated outputs of all H heads are passed. Importantly, in RCA the queries and keys are produced based on glimpse contents, Le. based visual information, that the resulting inner product in the softmax of Eq. retrieves the visual relations disentangled from the specific visual features. At the same time, the values are generated from the glimpse locations that stripped of any visual information. Hence, the overall processing in RCA driven only by the relations between visual features (but the visual features themselves) that modulate the processing of structural information (glimpse locations). Similar to (Mondal et al.) (2024), in each layer, RCA was appended with feedforward networks and standard self-attention Vaswani et al., 2017) as well as residual connections between each of those components (Figure A.1
where W.,
Relation embeddings
Glimpse contents g Glimpse locations T(-0
Abstractor Layer
Abstractor Layer
Abstractor Layer
Abstractor Layer
Relational
Feedforward
Self-Attention
Feedforward
Cross- Attention
Figure A.1: Abstractor downstream architecture: consists of several layers each consisting of the relational cross-attention (RCA), feedforward networks, the standard self-attention (SA), and residual connections between those. The most important component, the RCA, differs from the SA in that the queries and keys are generated based on glimpse contents (or their visual features) whereas the values are generated based on the glimpse locations. The figure is adapted from (Mondal et al. 2024).
B DATASET DESCRIPTIONS
B. SVRT
The Synthetic Visual Reasoning Test (SVRT) dataset, introduced by (Fleuret et al. (011). comprises 23 binary classification tasks. Each task involves set synthetic 2D shapes with an underlying re- lation between them. Following (Vaishnav & Serre 2023), these tasks are divided into two families: those defined by same/different relations (SD) and those defined by spatial relations (SR). SD tasks: 1, 5, 6. 7, 13, 16. 17, 19, 20, 21, 22; SR tasks: 2, 3, 4, 8, 9, 10. 11. 12, 14, 15, 18, 23. The main challenge with these tasks lies training with very few samples, thus testing the models' induc- tive biases for learning the relational aspects of image. The most commonly used dataset splits
14

---

Published as a conference paper at ICLR 2025
consider 500 1000 examples per task. The validation and test sets contain 4k and 40k samples respectively.
B.2 SVRT #1-0OD
This dataset introduced by (Puebla Bowers 2022) and built around the same-different task #1 from SVRT, in which the model has to make binary decision whether two shapes in the image are the same up to a translation. While this dataset contains the same training images as in the original SVRT dataset, it introduces 13 test sets with novel OOD shapes that significantly differ from those the train (Figure B.2) The dataset contains 28k and 5.6k images for training and validation respectively. Each OOD test set contains 11.2k images.
Same
Diflerent
Same
Different
Original (SVRT
Filled
4
D
Irregular
Lines 5
Regular
Arrows
L
Open
Rectangles
Straight lines
Wider
7
Connected squares
Scrambled
8
Random
Connected circles
color
8
Figure B.2: SVRT #1-0OD dataset. The goal is to determine whether 2 shapes are the same (up to a translation) or different. The train set consists of images from the original SVRT dataset. The test includes subsets each containing different type of out-tf-distribuution shapes. Taken from (Puebla & Bowers 2023).
B.3 ART
The ART (Abstract Reasoning Tasks) dataset, proposed by (Webb et al. 2021) comprises four visual reasoning tasks each having distinct underlying abstract rule to be detected: same/different (SD), (RMTS), distribution-of-3 (Dist-3) and identity rules (ID). Figure [..]] illustrates each of the tasks. The dataset was constructed using 100 Unicode character objects, and it includes generalization regimes of varying difficulty based on the number of unique objects used during training. Following (Mondal et al. 2024), we focused on the most challenging generalization regime, where the train set involves problems created from 5 out of the 100 possible objects, while the test problems use the remaining 95 objects. This setup tests systematic generalization, requiring abstract rule learning from small set of examples with minimal perceptual overlap between train
15

---

Published as a conference paper at ICLR 2025
Figure B.3: Difficult cases of SVRT #1-OOD: The shown images are from the 'straight lines' and 'rectangles' OOD test sets. The images contain different shapes that might be difficult distinguish even for humans.
and test sets. SD, RMTS, Dist-3, and ID tasks contain 40, 480, 360, and 8640 training examples respectively while all of them contain 20K and 10K validation and test examples respectively.
Relational sample
Distribution-of-3
Identity rules
Same / different
Figure B.4: Abstract Reasoning Tasks (ART) dataset consists of 4 tasks. In the same/different task, the model has determine whether image contains 2 the same (up translation only) shapes. In the sirioonllmmacchtto-seee pair candidates presented in the bottom row that contains objects in the same relation (either the 'same' or differentt)) as the objects in the upper row. In the distribution-off task, the model has to pick an object so that the lower row contains 3 objects that comprise the same set of objects as that of the upper row. In the identity rules task, the model has to select an object so that the resulting lower row contains objects that follow the same abstract pattern (ABA, ABB or AAA) as the objects in the upper row. Each instance of each task (except the same/different task) was presented with multiple images each containing separate candidate object (or object pair) so that the model has to pick the one with the correct pair. Taken from (Webb et al. 2021).
task, the model has to select pair of objects out of two
B.4 CLEVR-ART
The CLEVR-ART dataset Figure B.5 proposed by Webb et al. 2024a), utilizes photorealistic syn- thetic 3D shapes from CLEVR (Johnson al. 2017). includes two visual reasoning tasks from ART:
match-to-sampll and identity rules. The train set consists of images created using small and medium-sized rubber cubes four colors (cyan, brown, green, and gray). In contrast, the test set features images generated from large-sized metal spheres and cylinders in four differ- ent colors (yellow, purple, blue, and red). Hence, the object features in the train and test sets are entirely distinct, challenging the systematic generalization of learned abstract rules previously unseen object characteristics.
C EXPERIMENTAL DETAILS
For all tasks, the images were resized to 128 x 128 and the pixels normalized to the range [0, 1]. For SVRT tasks, random horizontal and vertical flips were applied during the training following
(Vaishnav & Serre 2023)
16

---

Published as a conference paper at ICLR 2025
Relational match-to-sample
Identity rules
Figure B.5: CLEVR-ART dataset consists of two tasks from ART dataset. Relational match-to- sample task: example problem involving the relation of difference. The correct answer choice (left image) contains pairs of different objects in the back and the front rows. The incorrect answer choice (right image) contains a pair of different objects in the front row but the same objects in the back row. Identity rules: example problem involving ABA rule. The correct answer choice (top left image) involves ABA rule in both back row and front row of objects. The object in the front row on the right in the other three images (incorrect choices) violates this rule. Taken from (Webb al. 2024a).
At each location of the saliency map, the corresponding eтror neuron compares small (central) image patch of size 5 that in receptive field with 8 surrounding patches of the same size. The surrounding patches are obtained by sliding window from the neuron's receptive field with cardinal directions so that the surrounding patches have a high overlap with the
the stride of central patch. The aggregation operator takes the minimal difference between the central patch and each the surrounding patches.
The number of iterations 7' in the glimpsing process is set to 15 for SVRT (and SVRT #1-OOD), 20 for SD and RMTS tasks from ART, and 35 for all other tasks from ART and CLEVR-ART. For all datasets, the IoR mechanism used the hard mask (with radius of 5 pixels) except for CLEVR-ART where its soft version (with 450) was used. For all datasets, the multi-scale glimpse sensor provided glimpses at 3 scales each of which was of size 15 x 15, where the first scale corresponded
17

---

Published as a conference paper at ICLR 2025
to the original resolution and the other two downsampled the regions of size 30 30 and 45 x 45. For the log-polar sensor, we use OpenCV (Bradski 2000) formulation and implementation of the log-polar sampling setting the radius of the log-polat grid 21 for SVRT (and SVRT #1-OOD) and ART tasks and to 15 for CLEVR-ART.
to 48. The glimpse size set to
In glimpse-based models, feeding it to the downstream architecture. For glimpse contents produced by the multi-scale sensor, use 7-layer CNN where each layer has the kernel size of 3 3, padding and ReLU activation function. For glimpse contents produced by the log-polar sensor, we use almost the same CNN but with 6 layers where the first 4 layers have the kernel size of 5 5. The number of channels per layer is the same in both cases being 9 for SVRT (and SVRT #1 -OOD and 64 for the rest of the datasets. Each glimpse location was re-scaled to MLP with with two layers of size 32 and 64, As in (Mondal et al. 2024), we applied temporal context normalization (TCN) (Wehb et al.) 2020) on both sequences of pre-processed glimpse contents and locations. TCN normalizes sequence along its temporal dimension.
small CNN to pre-process each glimpse content before
range - and was pre-processed by an
For all datasets, the Abstractor module was instantiated with attention 64-dimensional heads, 24 layers, and 2-layered MLPs with the hidden size of 256. The 50% dropout for the MLPs and attention used only for SVRT +1-OOD. To generate the final output for the task, we took the mean of the sequence of final representations TL and passed it to single linear unit.
For the training, we used ADAM optimizer (Kingma & Ba 2013) with the learning rate - 4, batch size of 64. The whole model trained end-to-end with the binary cross-entropy loss.
D ABLATIONS AND ADDITIONAL RESULTS
Table D.1: Test accuracy (in %) for SVRT dataset using different glimpse sensors. The accuracy results are averaged over SD and SR tasks obtained from the best trained model.
SD tasks
SR tasks
Model
Glimpse sensor
Dataset size
Dataset size
Dataset size
Dataset size
1000
500
1000
500
GAP- Tra-sfonsfrrmer
multi-scale
80.8 = 13.5
68.0 14.3
97.9 1..922.55
96.5 1 3.1
GAP- TAP-Transformer
log-polar
83.3 + 13.2
78.4 + 13.5
98.2 2.2
97.3 # 2.7
GAP-Abstracton
multi-scale
93.1 13.1
90.5 14.5
98.5
96.6 + 2.1
GAP-Abstractor
log-polar
92.5 12.3
90.6 13.3
98.2
97.0 + 3.4
Table D.2: OOD performance (in %) for test sets from SVRT #1-OOD using different glimpse sensors. The accuracy results are averaged over all OOD test sets for 10 trained models.
Accuracy (averaged over runs and OOD datasets)
Model
Glimpse sensor
Granffooorrr
multi-scale
76.3 18.4 74.7 + 16.0
Tra-Transformer
log-polar
GAP- Abstracctt
multi-scale
84.4 15.3
GAP-Abstractor
log-polar
89.6 + 8.7
Table D.3: Accuracy (in %) for tasks from the ART dataset using different glimpse sensors. The accuracy results averaged over 10 trained models.
Task
Model
Glimpse sensor
SD
RMTS
Dist3
Gr--Transformer
multi-scale
76.4 13.7
+
50.7 6.1
55.5 5,8
GAP- Transsfforerrr
log-polar
64.3 + 9.1
52.6
51.3 # 13.7
57.0 + 8.8
Absttraca
multi-scale
95.8 0.8
95.3
93.9
91.1 4.5
GAP-Abstrace
log log-polar
97.7 + 2.0
96.3 + 1.0
98.4 0.5
96.8 + 1.8
18

---

Published as a conference paper at ICLR 2025
Table D.4: Accuracy (in %) for tasks from the CLEVR-ART dataset using different glimpse sensors. The accuracy results are averaged over 5 trained models.
Task
Model
Glimpse sensor
RMTS
ID
GAP-Transfomme
multi-scale
76,4 + 4.3
64.7 5.6
GAP-Transformer
log-polar
70.0 3.5
63.0
Ght-Aastr
multi-scale
9.4
9.3
GAP-Abstractor
log-polar
9.0
92.5 # 1.5
D.1 PERFORMANCE DIFFERENCES BETWEEN GLIMPSE SENSORS
We evaluated different glimpse sensors for each of the four datasets considered in this work, see Table D D.4 and Figure D.6 For the Abstractor-based downstream architecture, we observe comparable performance for both sensors in most cases. The most significant performance differ- ence is observed for SVRT #1-OOD and two tasks from ART dataset (Table D.2 where the log-polar sensor achieves around 5% better accuracy. For the Transformer-basee downstream archi- tecture, the performance difference between two glimpse sensors averaged over all tasks is around 5%. However, there are tasks where the multi-scale sensor results in around 10% worse accuracy, see Table D.I. and 10% better accuracy, see Table D.3 Overall, based on the performance data, there is no clear quantitative trend favoring either of the two glimpse sensors. We ascribe this to qualitative advantages and disadvantages of both sensors. In particular, the log-polar sensor benefits from the properties the tog-polar space where the rotation and scaling in the Cartesian space be- come translations, e.g. (Javier Traver & Bernardino 2010). However, the warping into log-polar space may make it difficult to capture information about image parts that are far from the glimpse locations. The multi-scale sensor, in contrast, captures the distant information more faithfully ing downsampled image patches of different sizes. The disadvantage of this, however, is that the resulting glimpse content contains additional dimensions for the different scales.
Dataset size 1000
100
90
Accuracy %
80
70
60
50
40
Dataset size 500
100
80
Accuracy
70
60
40
5
6
7
16
17
18
21
Task
multi-scale
log-polar
Figure D.6: Test accuracy for each of SVRT tasks using different glimpse sensors for the GAP- Abstractor model.
19

---

Published as a conference paper at ICLR 2025
Table D.5: Sample efficiency on the SVRT dataset for 11 same-different (SD) tasks and 12 spatial relations (SR) tasks depending on the size of the training dataset. The results report the test accuracy [%] for the best trained models, the standard deviation is computed over SD and SR tasks. Only our models are listed. The prior art models are shown in Fig. D.7
SD tasks TranTransfrrrer
Dataset size
GAP-Abstractor
Abstractor 52.2 + 4.8
Transformer
10000
96.2 + 6.7
95.3 = 6.2
57.4 14.5
5000
95.4 8.1
92.7 = 9.8
51.7 3.9
56.5 14.2
1000
93.1 13.1
82.5 = 15.1
2.1
5.7
500
90.5 = 14.5
76.0 13.4
1.9
51.6 1.7
250
86.8 16.0
69,4 15.3
50.9
51.3 # 1.2
100
74.3 # 17.2
58.7 # 6.2
5
51.0 # 0.8
SR tasks
Dataset size
GAP-Abstractor
GAP-Transformer
Abstractor
Transformer
10000
99.7 = 0.7
99.7 + 0.5
93.5 + 13.9
82.9 19.2
5000
99.6 + 0.5
99.6 + 0.8
86.5 + 17.3
75.7 20.3
1000
98.5 1.9
98.2 2.2
60.9 + 14.3
64.8 13.7
500
96.6 = 1.2
97.2
$1.0
59.9 11.0
250
93.4 + 6.3
95.1 + 3.5
5.84
58.1
100
86.2 + 13.4
86.9 11.2
5.3
5.15
SD tasks
tasks
90
GAP-Abstractor
Slot-Abstractor OCRA
Accuracy %
GAP-Transformer
GAMR
70
Altn-ResNet
Abstractor
Transformer
50
10000
5000
1000
500
250
100
10000
5000
1000
500
250
100
Dalasel
Dataset size
Figure D.7: Sample efficiency on the SVRT dataset for 11 same-different (SD) tasks and 12 spatial relations (SR) tasks depending on the size of the training dataset. The results report the test accuracy [%] for the best trained models, the standard deviation is computed over SD and SR tasks.
D.2 EVALUATING IMPORTANCE OF GLIMPSE CONTENTS AND GLIMPSE LOCATIONS
To measure the extent which glimpse contents ("what") and glimpse locations ("where") important, we trained our models passing only one of those two sequences to their downstream ar- chitectures. The training done SD tasks from SVRT dataset using 1000 training samples. For the Transformer- downstream architecture, the input sequence consisted of either glimpse con- tents or glimpse locations, instead of the concatenation thereof. For Abstractor-based downstream architecture, we used fixed set of trainable embeddings from which the RCA produced values while the keys and the queries were produced from either of the aforementioned input sequences. As it can be seen from Table D.6] the combination of glimpse contents and glimpse locations yields the highest performance compared to cases where only one of those sequences used.
D.3 EVALUATING THE ROLE OF PROCESSING ONLY THE RELEVANT INFORMATION
The superior performance of our GaP-based models compared to the ablated downstream architec- tures that process entire images indicates the importance of processing only the relevant information. The relevance of information in the context of GAP entails features explained below.
20

---

Published as a conference paper at ICLR 2025
Table D.6: Test accuracy [%] averaged over SVRT SD tasks (11 in total) for models trained with 1000 samples depending on information passed to the downstream architecture.
Inputs to the downstream architecture
Model
glimpse contents
glimpse locations glimpse contents and locations
GAP-Abstractor
81.0 21.9
66.3 = 14.2
93.1 = 13.1
GAP-Trans -raassssffrr
77.4 12.1
72.8 = 17.5
83.3 = 13.2
Table D.7: Test accuracy [%] averaged over SVRT SD tasks (11 in total) for models trained with 1000 samples depending on information passed to the downstream architecture. See Section D.3 for more information.
Downstream architecture
All patches
ViT patches (#patches=64)
GAP-regular (#patches=15) 15)
GAP (#patches=15) 15)
sopatches=12990
Abstractor
50.2 1.3
51.3 + 2.5
58.6 + 14.8
93.1 = 13.1
Transformer 50.9 1.9
51.9 = 2.6
53.8 + 6,1
83.3 = 13.2
First, GAP discards all uninformative parts of the image providing thereby much less information to the downstream architecture than is available. This, in turn, improves the learning process preventing the models from being distracted by superftuous information. To elucidate the importance of this feature, we show the comparison of our GAP approach to the downstream architectures that process i) all image patches obtained by the sliding window approach with the stride of one (such patches almost entirely overlap and include glimpse contents obtainable by GAP); ii) all non-overlapping image patches obtained in ViT-like fashion based on the regular grid. The cases i) and ii) are referred to as all patches and ViT patches respectively.
Second, each glimpse content is focused meaning that it is centered around a salient image region. This is, for example, stark contrast to the patching process taking place in ViTs where the image is split into non-overlapping patches based on a regular grid. In that case, even if there were method to determine only a subset of the most relevant patches, those patches would not necessarily contain the best portions of salient regions. To show the importance of focusing, we compare GAP to its modification where glimpse locations are tweaked to be in a set of "coarse" locations defined by the VIT's regular grid. Importantly, this set of locations is only subset of all possible image locations that the GAP can glimpse at. The resulting glimpse contents are obtained from the corresponding tweaked glimpse locations. We refer this modification of GAP as GAP-regular.
All models described above were trained on SVRT SI) tasks (11 tasks in total) using 1000 training samples. The size of the patches was the same the size of the glimpse contents (15 x 15) and the number of glimpses for GAP and GAP-regular was set to 15. The total number of patches obtained in the cases of all patches and VIT patches is 12996 and 64 respectively.
Table D.7 shows the corresponding results. It can be seen that neither of the two aforementioned features alone is sufficient solve the tasks. Specifically, providing all information available in face of either all patches or ViT-patches makes the downstream architecture not able to solve any task. The same can seen the case of providing less information but not well focused on salient regions (although the performance of GAP- GAP-regular is slightly above the 50% chance level). In other words, it is important to process only the relevant information by providing only subset of available information and making this subset contain information focused on the most salient image parts.
E EXPERIMENT WITH MORE REALISTIC OBJECTS AND ALTERNATIVE SALIENCY MAPS
We evaluated our model on more realistic objects. Additionally, we show how different saliency maps may improve the model's performance in this case.
In particular, we used the objects from the Super- CLEVR dataset (Li et al. 2023). including cars, planes and motorbikes, to generate new test set for the Identity Rules task from the ART dataset. The task is to determine whether three objects in the top row are arranged under the same abstract rule the objects the bottom row. For example, objects "bus", "car" and "bus" in row
21

---

Published as a conference paper at ICLR 2025
arranged under the abstract pattern ABA just objects "car", "bus" and "car. We considered four models trained on the original CLEVR-ART train set that consists only of simple cubes (Figure 6 and evaluated them on the new dataset, referred to from the Super-CIEVR test set are shown in Figure E.8
test set. The example images
Simultaneously, explored using different saliency maps. The first model is GAP-Abstractor model with the original error neurons applied on the raw image to extract the saliency map, we refer this model here GAP-Abstractor (EN). The second model, referred to as GAP-Abstractor (CNN+EN), is modification of GAP-Abstractor that extracts the saliency map by applying the error neurons on the feature maps computed by the first layer of ResNet-18 that was pre-trained on ImageNet. Computing the saliency map from the feature maps should make the glimpsing behavior less distracted by spurious salient locations, such as edges of shades or regions of high reflection. The third model, referred to GAP-Abstractor (IK), employs an alternative way to extract the saliency maps using more complex bio- bio-plausible model proposed by Itti et al. (1998). The example saliency maps are shown in Figure E.8 Table E.8 shows the corresponding results, where Slot- Abstractor is the best performing prior-art model providing the baseline.
Saliency maps
Super-CLEVR lest set
EN
CNN EN
IK
Figure E.8: Different saliency maps for images with more realistic objects. The images were gen- erated using objects from the Super-CLEVR dataset for the ID task defined by the CLEVR-ART dataset.
Table E.8: Accuracy [%] on ID task from CLEVR-ART on the original test set and the OOD test with more realistic objects from Super-CLEVR. The accuracy results are averaged over trained models. The highest and the second-highest mean accuracy in each column is marked in bold and underlined, respectively.
original test set
Super-CLEVR CLEVR
Model
test
Slot-Abstractor (prior art)
91.61 + 0.2
24.1 # 0.8
Gbr-Abstractorr (EN)
93.4 1.3
72.3 2.3
GAP-Abstractor (CNN+EN)
92.7 + 1.8
80.2 + 1.9
Abstraat (IK)
90.2 + 2.1
82.1 2.2
As it can be seen, Slot-Abstrace Aotttbbttactor fails completely on the unseen Super-CLEVE performance of our Abs-Abstractor does not degrade severely. Moreover, objects while the that GAP- Abstractor (CNN+EN) performs better on Super-CLEVR test set while performing comparably on the original test set. Therefore, applying the error neurons top of features maps rather than
22

---

Published as a conference paper at ICLR 2025
top of the images results in saliency map that can better steer the glimpsing behavior. Lastly, we see that GAP-Abstractor (IK) with an alternative bio-plausible way to extract the saliency map reaches the best performance original test set. A possible reason for this difference is that this saliency map extractor was designed to handle real-world scenes that of much higher visual complexity compared simple shapes. Overall, these results indicate a possible direction of enhancing our approach for visual reasoning by integrating more powerful saliency maps.
the Super-CLEVR test set while being slightly worse on the
23


Reviews:
[{"review_id": "QjVajMCP0Q", "reviewer": "Reviewer_YJPp", "summary": "The paper focuses on understanding visual relations. This remains a challenging problem for current vision models. To deal with this challenge the paper leverages active vision where the learning of visual relations is grounded in actions that we take to fixate objects and their parts by moving our eyes. The proposed approach with glimpse-based active perception demonstrates promising performance on a range of visual reasoning tasks.", "questions": "* The paper should include a broader discussion of related work (see above).\n* The paper should better motivate the choice of evaluation datasets.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper provides interesting insights into visual reasoning problems.\n* The proposed glimpse-based active perception is relatively novel and interesting.\n* The proposed approach shows promising performance on a range of visual reasoning problems.\n* The paper considers diverse visual \"sensors\" and downstream architectures.", "weaknesses": "* While the proposed approach is somewhat novel it is similar to prior work such as \"AdaGlimpse: Active Visual Exploration with\nArbitrary Glimpse Position and Scale, ECCV 2024\" which also focus on what and where to look.\n\n* Current state of the art vision-language models such as LLaVA (NeurIPS 2024) keep the visual features from the target image in the context window. This means that they can actively attend to the image as many times are necessary to extract visual features. It would be interesting to compare the proposed approach to current SOTA VLMs such as LLaVA.\n\n* Prior work such as \"Look, Remember and Reason: Grounded reasoning in videos with language models, ICLR 2024\" used surrogate tasks to decide what and where to look at. It would be beneficial to include a discussion of this related work.\n\n* Synthetic data: Three out of the four datasets used for evaluation are based on synthetic datasets. It would be beneficial to include more real world datasets such as GQA or Super-Clevr.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N2MazL3QZj", "reviewer": "Reviewer_xwHi", "summary": "This paper proposes a novel method for visual reasoning, called Glimpse-based Active Perception (GAP). Based on that the human eye selectively concentrates on salient and/or task-relevant parts of a scene, the authors devise a method that extracts salient regions of an image and feeds them into downstream architectures to enforce perception of salient regions only. Firstly, a saliency map is built based on error neurons, which marks salient regions as those that differ significantly with their surrounding neighbours. Secondly, salient regions are extracted from the image, using the saliency map, with either the multi-scale or log-polar glimpse sensor. Lastly, these salient regions and their locations are fed into either Transformer or Abstractor to perform visual reasoning tasks, each dubbed as GAP-Transformer and GAP-Abstractor. By forcing the downstream architectures to concentrate on salient regions only, the GAP-Abstractor achieves SOTA or comparative-to-previous-SOTA performance on the conventional visual reasoning benchmarks, OOD benchmarks, and real-image-based OOD benchmarks.", "questions": "- Can the authors provide an explanation to why GAP-Abstractor perform worse on 'straight lines' and 'rectangles' compared to other classes? \n- Since the authors explained that \"glimpsing behavior can be distracted by spurious salient locations such as edges of shades or regions of high reflection\", would performing GAP (finding salient regions) on feature maps, instead of images, where these spurious features are probably less influential, increase performance? Can authors provide results? \n- Please explain why each sensor works better or worse for each downstream architecture-dataset pair.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Authors propose a well-motivated method: \n  - They explain the motivation behind GAP logically, linking back to the human perception model throughout their method section. \n  - Ie. They explain that they concentrate on salient regions since \"humans use active vision\"; They explain that they use the concept of error neurons since \"the activity of neurons is influenced ... also by stimuli that come from the surroundings of those receptive fields\". \n\n- The proposed method attains strong performance on benchmarks:\n  - Especially compared to previous work that do not use pretraining as GAP, GAP-Abstractor outperforms them by large margins. \n  - Authors also infer on four datasets, each with different purposes (testing visual reasoning only vs OOD generalization too) and different domains (binary figures vs real-world images). \n\n- The paper is well-written and easy to follow with both textual and visual explanations. \n\n- This paper introduces a novel and logical method to field of Visual Reasoning, which is of great importance since visual reasoning is required for a range of real-life tasks, and thereby contributes to the community.", "weaknesses": "- Authors do not make it explicit or clear that different sensors (multi-scale or log-polar) have been used across different datasets for the same downstream architectures (ViT/Abstractor). Ie. For SVRT, GAP-Abstractor used the multi-scale sensor but for SVRT #1-OOD it used the log-polar sensor. This should be made clear in the main paper, instead of referring to the Appendix. \n  - Also, in Appendix D, authors do not explain why each sensor works better or worse for each downstream architecture-dataset pair. Since the performance depends on the type of sensor and since authors have proposed to use both, authors should explain where each sensor may be a better choice. \n\n- References are missing in Tables and Figures. \n- Explain what the tasks RMTS and ID are, at least briefly, in Section 5.4 before explaining the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iTRn8oHj9B", "reviewer": "Reviewer_1Rv1", "summary": "The authors develop a system equipped with a novel Glimpse-based Active Perception (GAP) that sequentially glimpses at the most salient regions of the input image and processes them at high resolution. Their approach reaches state-of-the-art performance on several visual reasoning tasks being more sample-efficient, and generalizing better to out-of-distribution visual inputs than prior models.", "questions": "1. Have you explored alternative methods for computing the saliency map, given that GAP's effectiveness largely depends on the map's accuracy in identifying key image regions?\n2. What is the impact of using a hard versus a soft mask M(x_t)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel \"Glimpse-Based Active Perception\" (GAP) model inspired by human active vision. \n2. The proposed approach reaches state-of-the-art performance on several visual reasoning tasks being more sample-efficient, and generalizing better to out-of-distribution visual inputs than prior models.", "weaknesses": "1.  The images in the four selected datasets seem relatively simple, with very clean backgrounds. Have you considered comparing your proposed model with baseline models on more realistic image datasets, such as the COCO dataset?\n2. Given that the GAP mechanism involves multiple steps (e.g., saliency map generation, inhibition of return), have you compared its computational performance with other baseline models?\n3. Why can GAP-Abstractor improve OOD generalization of same-different relation and more abstract relations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xZVefyTFQk", "reviewer": "Reviewer_8myJ", "summary": "The paper presents an architecture that performs visual perception based on local glimpses. It uses a saliency map to determine glimpse positions, and then feeds an encoding of the appearance and of the location of the glimpses to an existing downstream architecture - a Transformer or the recently proposed Abstractor. The glimpse extraction is hardcoded while the downstream architecture can be trained. The architecture is evaluated on reasoning tasks that rely on local structure and spatial relations.", "questions": "I'm not sure about the term \"active\", since the architecture relies on a fixed (and not adaptive or recurrent) scheme to extract local features? \n\nWhy is the dataset size for results in Table 1 restricted to 500/1000 training examples? This seems arbitrary. Figure 4 seems to suggest that the training set is larger? \n\nCan the SVRT #1-OOD experiment be extended to other subsets than SVRT #1? \n\nIs there a reason why the results in Table 3 (on a subset of ART) do not include all models considered previously (e.g. Attn-ResNet?) \n\nIt would be very beneficial to include a modern vision-language model in the evaluation, as these models have become very good at solving similar reasoning tasks. \n\nSome additional related work: \n\n\"Learning to combine foveal glimpses with a third-order Boltzmann machine\", Larochelle et al. 2010 \n\n\"Multiple Object Recognition with Visual Attention\", Ba et al. 2014 \n\n\"Show, Attend and Tell: Neural Image Caption Generation with Visual Attention\", Xu et al. 2015", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper explores an architecture that make explicit the positions and not just appearance of local image regions. These types of architecture are interesting because of the potential capabilities they can enable, and because of their relation to human vision. The paper is well written and fairly easy to follow. The proposed architecture is very simple.", "weaknesses": "The proposed architecture can be viewed as a local feature extractor that encodes local image regions and their positions. There is very extensive prior work in this area, which makes the novelty somewhat limited. The proposed architecture is evaluated on two types of toy task (including some variants of those tasks): SVRT with/without OOD setting and ART/CLEVR-ART. All tasks use simple synthetic images with highly local structure in entirely uncluttered scenes, and their solutions seems to strongly rely on positions of objects. One would expect any architecture that makes those positions explicit to do exceptionally well on these specific tasks. Training sets for these tasks are also extremely small and will thereby favor architectures that are based on hardcoded (not trained) feature extraction. Given the extensive prior work on foveated vision, how do existing methods perform on these tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nUcwjIJanF", "reviewer": "Reviewer_KWe3", "summary": "This paper discusses how to improve the performance of visual reasoning by simulating the active perception of humans, especially when dealing with unknown objects. By expressing the relationship between different image parts with the position generated by glimpse-based active perception and the visual parts around them, this approach reaches a better sampling efficiency and generalization performance.", "questions": "1. What concerns me most is the generalization ability of this approach, although it is inspiring and fancy. Could you give me some clues that this approach can deal with tasks across different domains to reach the authentic reasoning ability?\n2. Many people assume that LLM-based reasoning is closest to the generalization due to its rich knowledge. I suppose this approach would also be helpful to the LLM community. Would you conduct some analysis on the relationship between this approach and LLM-based reasoning?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It is really inspiring to simulate human's active perception to improve the performance of visual reasoning, which is a contribution to the community of visual reasoning.\n2. The approach achieves better sampling efficiency and generalization performance.\n3. The authors provide extensive experiment results and analysis.", "weaknesses": "1. For the discussion of generalization ability, the paper focuses on the OOD data of the specific benchmarks. However, what we hope to achieve is the ability that models can deal with totally different tasks like humans, although it is really though. This paper lacks the analysis of it.\n2. Recently, many LLM-based approaches are gaining higher performance on the reasoning tasks like VQAs. More analysis of the relationship between pure abstract reasoning like this approach and LLM-based reasoning is desirable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)