Mind the GAP: Glimpse-based Active Perception improves generalization and sample efficiency of visual reasoning

## Abstract

Abstract Human capabilities in understanding visual relations are far superior to those of AI systems, especially for previously unseen objects.
For example, while AI systems struggle to determine whether two such objects are visually the same or different, humans can do so with ease.
Active vision theories postulate that the learning of visual relations is grounded in actions that we take to fixate objects and their parts by moving our eyes.
In particular, the low-dimensional spatial information about the corresponding eye movements is hypothesized to facilitate the representation of relations between different image parts.
Inspired by these theories, we develop a system equipped with a novel Glimpse-based Active Perception (GAP) that sequentially glimpses at the most salient regions of the input image and processes them at high resolution.
Importantly, our system leverages the locations stemming from the glimpsing actions, along with the visual content around them, to represent relations between different parts of the image. The results suggest that the GAP is essential for extracting visual relations that go beyond the immediate visual content. Our approach reaches state-of-the-art performance on several visual reasoning tasks being more sample-efficient, and generalizing better to out-of-distribution visual inputs than prior models.

## 1 Introduction

Humans use vision both to detect objects and to analyze various relations between them.
For example, to successfully grasp an object from a table, we should first see whether some other object is not lying on top of it.
Such relations can be spatial, like in the aforementioned example, or more abstract, e.g. if two objects are the same or different.
While humans can easily recognize many kinds of relations, even between previously unseen objects (Fleuret et al., ; Kotovsky & Gentner, ), a growing body of literature suggests that artificial neural networks (ANNs) struggle with this (Kim et al., ; Ricci et al., ; Puebla & Bowers, ; ).
Typically, ANNs process visual information holistically with each part of an image being processed in parallel with the others.
In contrast, humans use active vision by selectively moving their eyes to glimpse at salient and/or task-relevant parts of the image and to process those parts in high resolution (Yarbus, ; Hayhoe & Ballard, ; Land, ).
Moreover, by moving the eyes, the brain is able to form representations of the corresponding glimpse locations.
This process can also be seen as a factorization of the visual scene into its visual (“what”) and spatial (“where”) contents (Goodale & Milner, ; Behrens et al., ).
It is hypothesized that the low-dimensional representation and the relational geometry between those locations form a scaffold for representing the image structure, i.e. relations between different image parts (Summerfield et al., ).

Currently, the best-performing ANNs, proposed for visual reasoning, use completely different mechanisms compared to humans (Webb et al., ; Mondal et al., ).
Specifically, they combine an object-centric representation using slot attention (Locatello et al., ) and a special inductive bias called relational bottleneck (Webb et al., ).
The former aims at decomposing an image into a set of representations corresponding to its objects and the latter encourages the model to focus on the relations between objects’ representations instead of the representations themselves.
While showing compelling performance on several visual reasoning tasks, these ANNs struggle to generalize the learned rules to out-of-distribution (OOD) objects (Puebla & Bowers, ).
A likely reason is that the methods for object-centric representation cannot reliably capture the relations between different parts of objects.
Instead, they merely perform perceptual segregation through clustering (Mehrani & Tsotsos, ).
Inspired by active vision, an alternative model, proposed by Vaishnav & Serre (), processes images iteratively using a soft attention mechanism controlled by a memory module. However, in every iteration, this model still processes the image in its entirety only masking specific regions.
Moreover, it lacks a low-dimensional spatial representation of task-relevant regions.

Drawing more inspiration from humans’s active vision, and the aforementioned theories behind it, we introduce a model equipped with a novel Glimpse-based Active Perception (GAP) that sequentially glimpses at the most salient parts of the image.
The GAP produces a dual sequence of glimpse locations and glimpse contents around them, promoting spatial “where” and visual “what” pathways.
Importantly, the glimpse locations in the “where” pathway convey purely spatial information that is complementary to the visual contents of the “what” pathway.
In particular, the geometrical relations between those locations provide a basis for extracting structural aspects of the image that can also be present in an image with different visual contents.
The GAP can be paired with various downstream architectures that can process the dual sequence to solve the task at hand.
In this work, we use a Transformer (Vaswani et al., ) and its recently introduced extension, called Abstractor (Altabaa et al., ).
We evaluate our model on four visual reasoning datasets testing OOD generalization capabilities and sample efficiency.
Achieving state-of-the-art performance, our model provides computational support for the active vision theories.

Our contributions can be summarized as follows:

- •
We present a brain-inspired active vision system for visual reasoning that extracts structural aspects of images by sequentially glimpsing at their salient parts.
- •
We show that our system outperforms state-of-the-art models on several benchmarks in terms of sample efficiency and OOD generalization.
- •
Our model is amenable to the integration of more advanced components for both saliency detection and downstream architecture which allows further scaling and extension.
- •
Our results support cognitive science theories postulating that the brain factorizes images into their visual and spatial contents with the latter being crucial for OOD generalization.

## 2 Approach

We introduce a model that learns to extract relations between different parts of an image by factorizing it into its visual (“what”) and spatial (“where”) contents. This is achieved by sequentially glimpsing at the most salient image parts that comprise the visual content whereas the locations of those parts provide the spatial content.
Our approach is illustrated in Figure 1.
Given an input image, our model employs a novel concept of error neurons to extract a saliency map where stronger activities of the error neurons correspond to more salient parts, such as corners of the shapes presented in the image.
The saliency map is then used to guide the glimpsing process by producing locations to the highest saliency, referred to as glimpse locations, in a winner-takes-all (WTA) fashion.
To prevent re-visiting previous glimpse locations, an inhibition-of-return (IoR) mechanism updates the saliency map by masking out regions around the previously visited locations.
These two steps are inspired by (Itti et al., ).
Each glimpse location is passed to a glimpse sensor that extracts the corresponding visual content around that location, referred to as glimpse content.
The described process repeats for a pre-defined number of iterations, producing a sequence of glimpse contents and a sequence of glimpse locations.
These two sequences represent the factorization of the image into its complementary “what” and “where” contents.
Both sequences are processed by the downstream architecture that makes the final decision for the given the task.

Figure: Figure 1: Architecture overview. Based on the input image, the layer of error neurons produces the saliency map where salient image parts are highlighted (brighter regions). Through WTA combined with inhibition-of-return, a sequence of glimpse locations is produced that covers the most salient regions. For each such location, the glimpse sensor extracts the glimpse content from the region around it. The sequences of glimpse locations and glimpse contents are processed by the downstream architecture to make the final decision about the presented task.
Refer to caption: x1.png

### 2.1 Glimpse-based Active Perception

Glimpse-based Active Perception (GAP) is a process of acquiring visual and spatial information by steering a glimpse sensor to different parts of the visual scene. The steering is facilitated by a saliency map that highlights the most salient image parts.
While there are many ways to compute such a saliency map, we use a simple but efficient concept of error neurons (Figure 2(a)) described in detail in Section 2.1.1.
Based on the saliency map $\bm{S}\in\mathbb{R}^{h_{S}\times w_{S}}$, $h_{S},w_{S}$ denote the saliency map’s height and width, the model glimpses at the most salient regions for a pre-defined number of iterations $T$.
Specifically, at each iteration $t$, the next glimpse location $\bm{x_{t}}\in\mathbb{N}^{2}$ is selected through WTA competition returning the 2D location of the highest saliency:

$$ $\bm{x_{t}}=\text{WTA}(\bm{S_{t}})\stackrel{{\scriptstyle\text{def}}}{{=}} \operatorname*{arg\,max}_{ij}(S_{t,ij}),$ (1) $$

where $\bm{S_{t}}$ is the saliency map at iteration $t$.
To prevent the same location from being selected at subsequent iterations, the IoR mechanism applies a mask $\bm{M}({\bm{x_{t}}})\in\mathbb{R}^{h_{S}\times w_{S}}$ around this location, updating the saliency map:

$$ $\bm{S_{t+1}}=\bm{S_{t}}\odot\bm{M}({\bm{x_{t}}}).$ (2) $$

The mask $\bm{M}({\bm{x_{t}}})$ can be either a hard or a soft one.
The hard mask consists of ones except for a round region of zeroes around the location $\bm{x_{t}}$ as shown in red in Figure 2(b). The size of that region is a hyper-parameter.
In the soft mask, the values at each mask location $i,j$ are computed via the radial basis function with the Gaussian kernel $e^{-\epsilon||\binom{i}{j}-\bm{x_{t}}||_{2}}$ with $\epsilon$ being a hyper-parameter and $\binom{i}{j}\in\mathbb{N}^{2}$ a vector of location $i,j$.

Figure: Figure 2: Components of Glimpse-based Active Perception. (a) Error neurons: a sample error neuron, marked by the blue circle in the saliency map on the right, computes the difference between the visual content in its central receptive field (blue square) and the visual content in its surroundings (area shaded in red). (b) Glimpsing process: at each iteration the WTA mechanism selects a location of the highest value in the saliency map (corresponding to its brightest regions) that is passed to the glimpse sensor to extract the corresponding glimpse content (blue circles). Afterward, the inhibition-of-return mechanism masks out the selected location in the saliency map (shown by red circles) so that it does not get selected again. (c) Multi-scale glimpse sensor outputs a concatenation of several patches (three are shown) of the same size each of which encompasses a region of different size and resolution around the glimpse location on the left (depicted by three grids). (d) Log-polar glimpse sensor samples pixels according to the log-polar grid shown on the left providing high resolution towards its central region and low resolution towards the periphery. After the sampling process, the polar grid is warped into a regular rectangular grid which results in the glimpse content on the right.
Refer to caption: x2.png

Given a glimpse location $\bm{x_{t}}=(x_{t},y_{t})\in\mathbb{N}^{2}$, the corresponding region – glimpse content – is extracted by sampling pixels from the area around it.
This sampling process is conceived as obtaining information from a glimpse sensor that provides a limited view mimicking thereby a human eye that fixates different parts of the visual surroundings.
The main goal of such a glimpse sensor is to strike a balance between the size of the glimpse content and the amount of information to be captured from the image.
In this work, we implement two kinds of sensors: a multi-scale sensor and a log-polar sensor.
First, the multi-scale sensor, introduced in (Mnih et al., ), extracts several patches of the same size $h_{g}\times w_{g}$, but with different resolutions, around the glimpse location, as illustrated in Figure 2(c) (three patches are pictured). The resulting glimpse content $\bm{g_{t}}\in\mathbb{R}^{s_{g}\times h_{g}\times w_{g}\times c_{I}}$ is the concatenation of those patches where $s_{g}$ is the number of different scales supported by the multi-scale sensor and $c_{I}$ is the number of channels in the input image.
The patch with the highest resolution corresponds to the patch cut out from the input image at the glimpse location in the original resolution.
The other patches are of decreasing resolution and correspond to larger areas of the input image that are down-scaled to $h_{g}\times w_{g}$, allowing to capture coarse information of the surround of the glimpse location.
Second, the log-polar glimpse sensor is inspired by the human foveal vision (Schwartz, ; Weiman & Chaikin, ; Javier Traver & Bernardino, ) and samples pixels according to a distribution defined by a log-polar grid, illustrated in Figure 2(d). This means that more pixels are sampled closer to the glimpse location, and fewer pixels are sampled farther from it, producing a glimpse content $\bm{g_{t}}\in\mathbb{R}^{h_{g}\times w_{g}\times c_{I}}$ whose size is equal to a single image patch.

Using these procedures, the GAP ultimately produces a sequence of glimpse contents $\bm{g_{1}},\bm{g_{2}},\dots,\bm{g_{T}}$ and a sequence of glimpse locations $\bm{x_{1}},\bm{x_{2}},\dots,\bm{x_{T}}$, that are further passed to the downstream architecture described in Section 2.2.

#### 2.1.1 Saliency map extraction with error neurons

In the primary visual cortex, it is observed that the activity of neurons is influenced not only by stimuli from their local receptive fields but also by stimuli that come from the surroundings of those receptive fields (Vinje & Gallant, ; Keller et al., ).
From a theoretical view, this influence can make the same visual stimulus be perceived as more or less salient, depending on how significantly it differs from its local surroundings (Treisman & Gelade, ).
Inspired by this, we propose a computational block called error neurons, illustrated in Figure 2(a), to compute the saliency map $\bm{S}\in\mathbb{R}^{h_{S}\times w_{S}}$.
Specifically, at each location $i,j$ of the saliency map $\bm{S}$ the corresponding error neuron computes the saliency as

$$ $\bm{S}_{ij}=\underset{l,k\in\text{surr}(i,j)}{\text{Agg}}\big{[}D(\bm{P}^{\bm{ I}}_{ij},\bm{P}^{\bm{I}}_{lk})\big{]},$ (3) $$

where $\bm{P}^{\bm{I}}_{ij}\in\mathbb{R}^{h_{P}\cdot w_{P}\cdot c_{I}}$ is a flattened patch obtained from the input image $\bm{I}\in\mathbb{R}^{h_{I}\times w_{I}\times c_{I}}$ around location $i,j$. The patch size, $h_{P}\times w_{P}$, is a hyper-parameter, and $h_{I},w_{I},c_{I}$ denote the input image’s height, width, and the number of channels. Unless specified otherwise, $h_{I}=h_{S}$ and $w_{I}=w_{S}$. $D(\cdot,\cdot)$ is a distance function, in this work, set to L2 distance. $\text{Agg}[\cdot]$ is an aggregation operator such as sum or minimum over distances between the patch at location $i,j$ and the patches from surrounding locations $l,k\in\text{surr}(i,j)$.
The set of surrounding locations, $\text{surr}(i,j)$, can be defined arbitrarily, comprising, for example, all available image locations. In this work, we assume the 8-connected neighborhood of location $i,j$.
Intuitively, each value $\bm{S}_{ij}$ represents how much the visual content around location $i,j$ differs (or stands out) from the surrounding visual content.

### 2.2 Downstream architecture

As detailed above, the GAP provides a sequence of glimpse contents $\bm{g}$ and a sequence of the corresponding 2D glimpse locations $\bm{x}$ to the downstream architecture. Importantly, our approach is flexible with respect to the choice of the downstream architecture. In this work, we explore two options: Transformers (Vaswani et al., ) and Abstractors (Altabaa et al., ).

Transformers have emerged over recent years as popular state-of-the-art architectures for a variety of tasks, including the vision domain with the Vision Transformers (ViT) Dosovitskiy et al. ().
Apart from their successes in machine learning tasks, there have also been works suggesting potential links between ViT and biology (Pandey et al., ).
The ViT processes a sequence of image patches, augmented with positional encodings, to solve the given task.
In our case, we provide the sequence of glimpse contents augmented with the respective glimpse locations as inputs.

The Abstractor has been recently introduced as an extension of the Transformer, with the key difference of the so-called Relational Cross Attention (RCA) module.
It was designed to disentangle relational information from the perceptual one to enable generalization from limited data.
This is achieved by modifying the self-attention so that the queries and the keys are produced based on perceptual inputs whereas the values are provided by a separate independent set of representations.
The latter, in our case, is substituted by the sequence of glimpse locations whereas the perceptual inputs comprise the sequence of glimpse contents.
Hence, adopting the Abstractor allows for the explicit processing of spatial information so that the visual information carries only a modulatory effect in form of attention.
The formal description of the Abstractor is provided in Appendix A.

## 3 Related work

Inspired by human active vision, several models for learning visual relations have been introduced.
By decomposing input images into a sequence of image regions obtained from synthetic eye movements, Woźniak et al. () demonstrated that visual reasoning tasks can be solved with high accuracy using relatively compact networks.
However, the synthetic eye movements were based on hard-coded locations, whereas in our case, the glimpsing process is controlled by a saliency map.
Aiming primarily at computationally efficient image classification, Mnih et al. () used reinforcement learning to train a glimpsing policy. The policy iteratively determines the next 2D glimpse location based on the history of previously visited locations and the corresponding glimpse contents.
However, having been evaluated on several visual reasoning tasks by Vaishnav & Serre (), this model could not achieve reasonable performance. This is likely caused by learning inefficiency in exploring the space of all possible 2D glimpse locations.
Our approach, in contrast, leverages the concept of salience maps to determine the next glimpse location which significantly reduces the space of glimpse locations to the most salient ones.
Gregor et al. () proposed a variational auto-encoder that uses an attention mechanism for an iterative (re-)construction of complex images by attending to their different parts.
Its extension proposed by Adeli et al. () was designed to tackle visual reasoning tasks where it outperformed baseline CNNs. However, being trained by image reconstruction, the model does not always generalize well to OOD visual inputs.
Vaishnav & Serre () introduced a recurrent soft attention mechanism guided by a memory module that masks specific parts of the image where the processing should be focused. At each iteration, the model can thereby attend to multiple regions of the image simultaneously which is different from our approach where only one region can be focused at a time.

Another family of models combines object-centric representations (Greff et al., ; Locatello et al., ) with relational inductive biases (Webb et al., ; Kerg et al., ; Webb et al., ). The former decomposes an image into objects present in it while the latter constrains the processing to relations between objects instead of their specific properties.
Webb et al. () used pre-trained slot attention (Locatello et al., ) to segment objects into a sequence of slots that contain disentangled representations of objects’ visual features and their positional information.
Each pair of slots is then processed by a relational operator and the resulting sequence of relational representations is processed by the Transformer (Vaswani et al., ).
The successor model, Slot-Abstractor (Mondal et al., ), processes the sequence of slots with a recently introduced Abstractor module (Altabaa et al., ) that proved to be especially effective in reasoning about relations between objects using their disentangled visual and positional features.
Similar to our approach, there are two streams of information: one based on visual features and one based on positional information.
However, the latter stream consists of positional embeddings that are learned during the pre-training process which may make the model more fragile to OOD inputs.
In contrast, our approach uses the raw 2D locations of salient regions.

Figure: Figure 3: Four datasets used in our experiments. (a) SVRT dataset consists of 23 binary classification tasks where the model has to determine whether shapes are in a certain spatial and/or similarity relation to each other. Shown are images from the task of determining whether the same triplets of shapes exist in each image pair. (b) SVRT #1-OOD instantiates the first task from SVRT where it has to be determined whether two shapes are the same up to a translation. While this dataset contains training images from the original SVRT dataset, it contains 13 additional test sets (only three are shown here) with novel shapes to evaluate OOD generalization. (c) ART dataset contains four tasks where the model has to determine whether shapes in each two rows are arranged under the same abstract rule. (d) CLEVR-ART contains two tasks from ART but with more complex images.
Refer to caption: x3.png

## 4 Experiments

##### Datasets

Our approach was evaluated on four visual reasoning datasets illustrated in Figure 3.
We start with an evaluation on all 23 tasks of the SVRT dataset (Fleuret et al., ) to see how efficiently the model can learn spatial and similarity relations between simple shapes, see an example in Figure 3(a).
Next, we test the model’s OOD capabilities to generalize learned relations to previously unseen shapes.
Specifically, we use an extended version of the task #1 from the SVRT dataset (Puebla & Bowers, ), referred to as SVRT #1-OOD, see Figure 3(b). It contains test sets with OOD shapes coming from 13 different distributions.
Additionally, we use the ART dataset (Webb et al., ), which contains more abstract relations, see Figure 3(c).
Finally, we test the model’s potential to scale to more complex images by testing it on the CLEVR-ART dataset (Webb et al., ), see Figure 3(d). It contains two tasks from ART but with more realistic images.
Each dataset is described in more detail in Appendix B.

##### Baselines

We compare the impact of our approach with the plain downstream architectures that receive entire images as inputs. The images get split into non-overlapping patches and each patch gets augmented by its positional information as is done by ViTs (Dosovitskiy et al., ).
We also tested those models where the patches overlapped but did not observe any significant performance differences. Therefore, we only report the standard approach with non-overlapping patches.
We refer to these models simply as Transformer and Abstractor.
The comparison with these baselines allows us to demonstrate the importance of processing only the relevant patches obtained from the glimpsing process rather than all patches that constitute an image.
We also compare our approach with GAMR (Vaishnav & Serre, ), ResNet (He et al., ), its extension Attn-ResNet (Vaishnav et al., ), OCRA (Webb et al., ) and Slot-Abstractor (Mondal et al., ).
To the best of our knowledge, these baselines are the best-performing models on the tasks that we consider.

## 5 Results

In all experiments, we report the performance of GAP combined with two downstream architectures, Transformer and Abstractor. The corresponding models are referred to as GAP-Transformer and GAP-Abstractor respectively.
Details about hyper-parameters, such as the type of glimpse sensor (multi-scale or log-polar), and training process are described in Appendix C.
Moreover, extensive performance reports for different glimpse sensors are included in Appendix  D.

**Table 1: Test accuracy for SVRT. The results obtained from the best trained models are averaged over SD, SR and over all (SD + SR) tasks. The standard deviation is calculated over accuracies for each individual task. Prior art results are listed in the top section whereas results obtained from our evaluations are listed in the two bottom sections. The highest and the second-highest mean accuracy in each column is marked in bold and underlined, respectively. The models appended by * contain a module pre-trained on the train sets from all SVRT tasks to segment the shapes.**
| Model | SD tasks accuracy [%] | SR tasks accuracy [%] | All tasks accuracy [%] |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dataset size<br>1000 | Dataset size<br>500 | Dataset size<br>1000 | Dataset size<br>500 | Dataset size<br>1000 | Dataset size<br>500 |  |
| ResNet | 56.9 ± 2.5 | 54.9 ± 2.2 | 94.9 ± 1.6 | 85.2 ± 4.3 | 76.7 ± 2.0 | 70.7 ± 3.3 |
| Attn-ResNet | 68.8 ± 4.4 | 62.3 ± 3.5 | 97.7 ± 0.7 | 94.8 ± 1.4 | 83.9 ± 2.5 | 79.3 ± 2.4 |
| GAMR | 82.1 ± 8.4 | 76.8 ± 8.7 | 98.7 ± 0.3 | 97.4 ± 0.7 | 90.8 ± 4.2 | 87.6 ± 4.6 |
| OCRA* | 90.3 ± 4.1 | 79.9 ± 4.5 | 95.0 ± 2.4 | 89.3 ± 2.5 | 92.8 ± 3.2 | 84.8 ± 3.5 |
| Slot-Abstractor* | 91.9 ± 4.0 | 82.2 ± 4.7 | 97.3 ± 1.1 | 91.8 ± 2.2 | 94.7 ± 2.5 | 87.2 ± 3.4 |
| Transformer | 51.9 ± 2.6 | 51.6 ± 1.7 | 64.8 ± 13.7 | 59.9 ± 11.0 | 56.4 ± 10.3 | 55.9 ± 8.9 |
| GAP-Transformer | 83.3 ± 13.2 | 78.4 ± 13.5 | 98.2 ± 2.2 | 97.3 ± 2.7 | 91.1 ± 11.8 | 88.3 ± 13.4 |
| Abstractor | 51.3 ± 2.5 | 51.0 ± 1.9 | 60.9 ± 14.3 | 57.0 ± 11.0 | 54.6 ± 9.7 | 54.1 ± 8.5 |
| GAP-Abstractor | 93.1 ± 13.1 | 90.5 ± 14.5 | 98.5 ± 1.9 | 96.6 ± 1.2 | 95.4 ± 8.9 | 93.9 ± 10.1 |

Figure: Figure 4: Sample efficiency evaluation for SVRT. Average test accuracy on all 23 SVRT tasks depending on the size of the training dataset. Panel (a) compares our best model with prior art. Panel (b) compares our glimpse-based models with their ablated counterparts.
Refer to caption: x4.png

### 5.1 Visual reasoning performance and sample efficiency

We evaluate the visual reasoning capabilities using the SVRT dataset that consists of 23 tasks, 11 of which require to reason about same-different relations (SD tasks) and the remaining 12 tasks require to reason about spatial relations (SR tasks).
Consistent with (Vaishnav & Serre, ; Webb et al., ; Mondal et al., ), Table 1 shows accuracy performance for models trained with 1000 and 500 samples.
Our best-performing model, the GAP-Abstractor, outperforms the prior art models. In particular, it achieves very high accuracy on the SD tasks, considered to be more challenging than the SR tasks, for which it performs comparably with the prior art.
It should be noted that although for the SR tasks, the best-performing model is GAMR, its performance on SD tasks is significantly lower compared with all other models.
Detailed accuracy for each individual task is provided in Appendix, Figure D.5.
Furthermore, Figure 4 shows an extensive evaluation of sample efficiency, in which we trained the models with different amounts of training data beyond those reported in Table 1 and in prior works.
Compared with the results we obtained for the prior art models, our best model exhibits higher sample efficiency, especially in scenarios with fewer than 1000 training samples, see Figure 4(a). Moreover, we observe favorable trends and improvements when applying GAP to either of the downstream architectures, see Figure 4(b). More detailed results with separate performance for SD and SR tasks are provided in Appendix, Figure D.6.

### 5.2 Out-of-distribution generalization of same-different relation

While the SVRT dataset allows us to see whether a model can learn visual relations from a limited amount of training data, it does not allow us to test the model’s ability to generalize those relations to OOD inputs.
We evaluate the OOD generalization of a single same-different relation using the SVRT #1-OOD dataset (Appendix  B.2) where a model has to determine whether two shapes are the same or different up to translation.
Importantly, the models are evaluated on test sets with OOD shapes coming from 13 different distributions (see Figure B.2 for examples).
Table 5.2 shows the performance averaged over all OOD test sets and Figure 5 shows the performance on each of the test sets.
Outperforming the prior art models, our model, GAP-Abstractor, performs quite consistently on all test sets being slightly less accurate on only two sets (’straight_lines’ and ’rectangles’).

Figure: Table 2: OOD performance for SVRT #1-OOD. The accuracy results are averaged over all OOD test sets for 10 trained models. Prior art results are listed in the top section whereas results obtained from our evaluations are listed in the two bottom sections. The highest and the second-highest mean accuracy is marked in bold and underlined, respectively. The models appended by * contain a module pre-trained on the train set to segment the shapes.
Refer to caption: x5.png

**Table 3: Test accuracy for ART. The accuracy results are averaged over 10 trained models. Prior art results are listed in the top section whereas results obtained from our evaluations are listed in the two bottom sections. The highest and the second-highest mean accuracy in each column is marked in bold and underlined, respectively. The models appended by * contain a module to segment the shapes that was pre-trained on images with shapes distinct from those in the ART dataset.**
| Model | Task accuracy [%] |  |  |  |
| --- | --- | --- | --- | --- |
| SD | RMTS | Dist3 | ID |  |
| GAMR | 83.5 ± 1.4 | 72.2 ± 3.0 | 68.6 ± 1.8 | 66.2 ± 4.3 |
| ResNet | 66.6 ± 1.5 | 49.9 ± 0.2 | 50.1 ± 1.3 | 54.8 ± 2.4 |
| OCRA* | 87.9 ± 1.3 | 85.3 ± 2.0 | 86.4 ± 1.3 | 92.8 ± 0.3 |
| Slot-Abstractor* | 96.4 ± 0.4 | 91.6 ± 1.6 | 95.2 ± 0.4 | 96.4 ± 0.1 |
| Transformer | 55.9 ± 2.1 | 51.3 ± 0.1 | 29.3 ± 1.2 | 33.3 ± 1.7 |
| GAP-Transformer | 176.4 ± 13.7 | 62.2 ± 7.3 | 50.7 ± 6.1 | 55.5 ± 5.8 |
| Abstractor | 70.9 ± 6.6 | 51.3 ± 0.2 | 58.9 ± 3.8 | 47.9 ± 1.4 |
| GAP-Abstractor | 97.7 ± 2.0 | 96.3 ± 1.0 | 98.4 ± 0.5 | 96.8 ± 1.8 |

### 5.3 Out-of-distribution generalization of more abstract relations

Going beyond evaluations of OOD generalization on a single task, we expand the evaluations to four tasks from the ART dataset, three of which contain more abstract relations (see Appendix B.3).
Moreover, the evaluations on this dataset validate also the insights on sample efficiency since the models are required to be trained on very few samples.
From Table 3 it can be seen that the GAP-Abstractor outperforms all other models demonstrating superior capabilities to generalize abstract relations to novel visual inputs.
Crucially, this performance improvement can be observed even though no component of our model was pre-trained, as opposed to OCRA or Slot-Abstractor that have been pre-trained to segment objects.

### 5.4 Scaling potential for more complex visual scenes

**Table 4: Test accuracy for CLEVR-ART. The accuracy results are averaged over 5 trained models. Prior art results are listed in the top section whereas results obtained from our evaluations are listed in the two bottom sections. The highest and the second-highest mean accuracy in each column is marked in bold and underlined, respectively. The models appended by * contain a module to segment the shapes that was pre-trained on images containing shapes from both the train and test datasets.**
| Model | Task accuracy [%] |  |
| --- | --- | --- |
| RMTS | ID |  |
| GAMR | 70.4 ± 5.8 | 74.2 ± 4.0 |
| OCRA* | 93.3 ± 1.0 | 77.1 ± 0.7 |
| Slot-Abstractor* | 96.3 ± 0.5 | 91.6 ± 0.2 |
| Transformer | 64.9 ± 3.4 | 52.2 ± 4.3 |
| GAP-Transformer | 76.4 ± 4.3 | 64.7 ± 5.6 |
| Abstractor | 177.8 ± 12.2 | 82.2 ± 3.8 |
| GAP-Abstractor | 95.9 ± 1.4 | 93.4 ± 1.3 |

We test our approach on two tasks from ART (RMTS and ID) but with more realistic images using the CLEVR-ART dataset.
While the images from other datasets considered in this work are binary and contain only 2D objects, the CLEVR-ART images are colored and contain 3D objects with such real-world features as shades and reflections.
Hence, the glimpsing behavior can be distracted by spurious salient locations such as edges of shades or regions of high reflection.
Table 4 highlights that the GAP-Abstractor performs on par with the state-of-the-art model, Slot-Abstractor, being marginally better in one task and marginally worse
in the other.
However, Slot-Abstractor was pre-trained on all shapes from CLEVR-ART being thereby potentially exposed to objects from both train and test sets.
This is in contrast to our models, which were not pre-trained and were exposed only to the objects from the train set.
To compare our models with Slot-Abstractor on a more equal footing, we generated two additional OOD test sets shown in Figure 6.
The new OOD-1 test set contains one novel type of shape – pyramid – on which the Slot-Abstractor was not pre-trained.
Similarly, the new OOD-2 test set contains 2 such novel shapes: pyramids and toruses.
The corresponding results are shown in Figure 7 demonstrating superior OOD generalization of our GAP-based model compared with Slot-Abstractor.
Notably, the latter performs significantly worse, if the images contain multiple objects on which it was not pre-trained.

Figure: Figure 6: Extension to CLEVR-ART for more thorough evaluation of OOD generalization. Examples for the RMTS task are shown here. The two new OOD test sets contain objects on which neither of the models was pre-trained: OOD-1 test set contains 1 novel object (pyramid) and OOD-2 contains 2 novel objects (pyramid and torus).
Refer to caption: x6.png

Figure: Figure 7: Extended OOD evaluation on CLEVR-ART. While the Slot-Abstractor was pre-trained on the original test set, the OOD-1 and OOD-2 test sets contain shapes on which neither of the evaluated models was pre-trained. The accuracy results are averaged over 5 trained models.
Refer to caption: x7.png

## 6 Discussion

Our results suggest that factorizing an image into its complementary “what” and “where” contents plays an essential role.
The low-dimensional “where” content (glimpse locations) is crucial since it does not contain any visual information and, in turn, allows to learn relations that generalize to OOD images.
To capitalize on that, it is important to process the “where” content explicitly, disentangling it from its “what” complement, so that the latter carries only a modulatory effect in form of attention.
This is achieved by employing the recently introduced Abstractor as our downstream architecture.
In contrast, implicitly mixing the “what” and the “where” contents, as in the case of the Transformer downstream architecture, weakens the generalization capabilities.
This can be seen by comparing the performance between GAP-Transformer and GAP-Abstractor.
To further support this, we provide supplementary results in Appendix D.1 showing the importance of using both the “what” and the “where” contents instead of just one of them for task solving.

Another important aspect of our model is that it processes only the most salient image regions while ignoring the unimportant ones.
The inferior performance of models where the downstream architectures receive full information (i.e. all patches that constitute the image) suggests that supplying a superfluous amount of information may distract the models hindering the learning process.
Additional results provided in Appendix D.2 elucidate this further.
Specifically, we show that it is insufficient to naively reduce the amount of information provided to the downstream architecture, by discarding uninformative image parts.
Instead, it has to be ensured that the supplied information is well structured. In the context of GAP, it means that the glimpse contents have to be centered around salient image regions instead of, for example, being picked from a set of non-overlapping patches obtained by splitting the image using a regular grid (as is done in ViT-based models).

## 7 Conclusion

Drawing inspiration from human eye movements and theories of active vision, we proposed a system equipped with the novel Glimpse-based Active Perception (GAP).
The system selectively glimpses at the most salient image parts and processes them in high resolution. The relational geometry between the corresponding glimpse locations complemented by the glimpse contents provides a basis for reasoning about image structure, i.e. relations between different image parts. We evaluated our approach on four visual reasoning benchmarks and achieved state-of-the-art performance in terms of sample efficiency and the ability to generalize to OOD visual inputs without any pre-training.
Our approach allows for further extension by integrating more advanced components. Particularly, the saliency map extraction can be substantially enhanced by either extending our concept of error neurons or integrating more powerful components.

## Appendix A Abstractor

To describe the Abstractor-based downstream architecture and the way it is applied, we follow (Mondal et al., ).
Specifically, given a sequence of glimpse contents $\bm{g}=(\bm{g_{1}},\bm{g_{2}},\dots,\bm{g_{T}})$ and their locations $\bm{x}=(\bm{x_{1}},\bm{x_{2}},\dots,\bm{x_{T}})$, the Abstractor computes a sequence of the corresponding relational representations $\bm{r}$ over a series of $L$ layers.
The initial relational representations $\bm{r}_{l=0}$ correspond to the glimpse locations $\bm{x}$.
Each subsequent layer $l$ updates the relational representations using multi-head relational cross-attention between the glimpse contents $\bm{g}$ (or their latent representations, if they are further pre-processed by a neural network):

$$ $\bm{\tilde{r}}_{h}=\text{softmax}\left(\frac{(\bm{g}\bm{W_{q}}^{h})^{T}(\bm{g} \bm{W_{k}}^{h})}{\sqrt{d}}\right)\bm{r}_{l-1}\bm{W_{v}}^{h},$ (4) $$

$$ $\bm{r}_{l}=\text{RCA}(\bm{g},\bm{r}_{l-1})=\text{concat}(\bm{\tilde{r}_{h=1}}, \dots,\bm{\tilde{r}_{h=H}})\bm{W_{o}}$ (5) $$

where $\bm{W_{q}}^{h},\bm{W_{k}}^{h},\bm{W_{v}}^{h}\in\mathbb{R}^{d\times d}$ ($d$ is the number of features of each head), are the linear projection matrices used by the $h^{\text{th}}$ head to generate queries, keys, and values respectively; $\bm{\tilde{r}}_{h}$ is the result of relational cross-attention in the $h^{\text{th}}$ head; $\bm{W_{o}}$ are the output weights through which the concatenated outputs of all $H$ heads are passed.
Importantly, in RCA the queries and keys are produced based on glimpse contents, i.e. based on visual information, so that the resulting inner product in the softmax of Eq. 4 retrieves the visual relations disentangled from the specific visual features.
At the same time, the values are generated from the glimpse locations that are stripped of any visual information.
Hence, the overall processing in RCA is driven only by the relations between visual features (but not the visual features themselves) that modulate the processing of structural information (glimpse locations).
Similar to (Mondal et al., ), in each layer, RCA was appended with feedforward networks and standard self-attention (Vaswani et al., ) as well as residual connections between each of those components (Figure A.1).

Figure: Figure A.1: Abstractor downstream architecture: consists of several layers each consisting of the relational cross-attention (RCA), feedforward networks, the standard self-attention (SA), and residual connections between those. The most important component, the RCA, differs from the SA in that the queries and keys are generated based on glimpse contents (or their visual features) whereas the values are generated based on the glimpse locations. The figure is adapted from (Mondal et al., ).

## Appendix B Dataset descriptions

### B.1 SVRT

The Synthetic Visual Reasoning Test (SVRT) dataset, introduced by (Fleuret et al., ), comprises 23 binary classification tasks. Each task involves a set of synthetic 2D shapes with an underlying relation between them.
Following (Vaishnav & Serre, ), these tasks are divided into two families: those defined by same/different relations (SD) and those defined by spatial relations (SR). SD tasks: 1, 5, 6, 7, 13, 16, 17, 19, 20, 21, 22; SR tasks: 2, 3, 4, 8, 9, 10, 11, 12, 14, 15, 18, 23.
The main challenge with these tasks lies in training with very few samples, thus testing the models’ inductive biases for learning the relational aspects of an image.
The most commonly used dataset splits consider 500 or 1000 examples per task.
The validation and test sets contain 4k and 40k samples respectively.

### B.2 SVRT #1-OOD

This dataset is introduced by (Puebla & Bowers, ) and is built around the same-different task #1 from SVRT, in which the model has to make a binary decision whether two shapes in the image are the same up to a translation.
While this dataset contains the same training images as in the original SVRT dataset, it introduces 13 test sets with novel OOD shapes that significantly differ from those in the train set (Figure B.2).
The dataset contains 28k and 5.6k images for training and validation respectively. Each OOD test set contains 11.2k images.

Figure: Figure B.2: SVRT #1-OOD dataset. The goal is to determine whether 2 shapes are the same (up to a translation) or different. The train set consists of images from the original SVRT dataset. The test set includes 13 subsets each containing a different type of out-of-distribution shapes. Taken from (Puebla & Bowers, ).
Refer to caption: extracted/5889755/figures/svrt_ood_full.png

### B.3 ART

The ART (Abstract Reasoning Tasks) dataset, proposed by (Webb et al., ) comprises four visual reasoning tasks each having a distinct underlying abstract rule to be detected: same/different (SD), relational-match-to-sample (RMTS), distribution-of-3 (Dist-3) and identity rules (ID). Figure B.3 illustrates each of the tasks.
The dataset was constructed using 100 Unicode character objects, and it includes generalization regimes of varying difficulty based on the number of unique objects used during training. Following (Mondal et al., ), we focused on the most challenging generalization regime, where the train set involves problems created from 5 out of the 100 possible objects, while the test problems use the remaining 95 objects. This setup tests systematic generalization, requiring abstract rule learning from a small set of examples with minimal perceptual overlap between train and test sets.
SD, RMTS, Dist-3, and ID tasks contain 40, 480, 360, and 8640 training examples respectively while all of them contain 20K and 10K validation and test examples respectively.

Figure: Figure B.3: Abstract Reasoning Tasks (ART) dataset consists of 4 tasks. In the same/different task, the model has to determine whether an image contains 2 the same (up to a translation only) shapes. In the relational-match-to-sample task, the model has to select a pair of objects out of two pair candidates presented in the bottom row that contains objects in the same relation (either the ’same’ or ’different’) as the objects in the upper row. In the distribution-of-3 task, the model has to pick an object so that the lower row contains 3 objects that comprise the same set of objects as that of the upper row. In the identity rules task, the model has to select an object so that the resulting lower row contains objects that follow the same abstract pattern (ABA, ABB or AAA) as the objects in the upper row. Each instance of each task (except the same/different task) was presented with multiple images each containing a separate candidate object (or object pair) so that the model has to pick the one with the correct pair. Taken from (Webb et al., ).
Refer to caption: extracted/5889755/figures/art_full_dataset.png

### B.4 CLEVR-ART

The CLEVR-ART dataset Figure B.4, proposed by (Webb et al., ), utilizes photorealistic synthetic 3D shapes from CLEVR (Johnson et al., ). It includes two visual reasoning tasks from ART: relational-match-to-sample and identity rules. The train set consists of images created using small and medium-sized rubber cubes in four colors (cyan, brown, green, and gray). In contrast, the test set features images generated from large-sized metal spheres and cylinders in four different colors (yellow, purple, blue, and red). Hence, the object features in the train and test sets are entirely distinct, challenging the systematic generalization of learned abstract rules to previously unseen object characteristics.

Figure: Figure B.4: CLEVR-ART dataset consists of two tasks from ART dataset. Relational match-to-sample task: example problem involving the relation of difference. The correct answer choice (left image) contains pairs of different objects in the back and the front rows. The incorrect answer choice (right image) contains a pair of different objects in the front row but the same objects in the back row. Identity rules: example problem involving ABA rule. The correct answer choice (top left image) involves ABA rule in both back row and front row of objects. The object in the front row on the right in the other three images (incorrect choices) violates this rule. Taken from (Webb et al., ).
Refer to caption: extracted/5889755/figures/clevr_art_illustration.png

## Appendix C Experimental details

For all tasks, the images were resized to $128\times 128$ and the pixels were normalized to the range $[0,1]$.
For SVRT tasks, random horizontal and vertical flips were applied during the training following (Vaishnav & Serre, ).

At each location of the saliency map, the corresponding error neuron compares a small (central) image patch of size $5\times 5$ that is in its receptive field with 8 surrounding patches of the same size.
The surrounding patches are obtained by sliding a window from the neuron’s receptive field with the stride of 1 in 8 cardinal directions so that the surrounding patches have a high overlap with the central patch.
The aggregation operator takes the minimal difference between the central patch and each of the surrounding patches.

The number of iterations $T$ in the glimpsing process is set to 15 for SVRT (and SVRT #1-OOD), 20 for SD and RMTS tasks from ART, and 35 for all other tasks from ART and CLEVR-ART.
For all datasets, the IoR mechanism used the hard mask (with radius of 5 pixels) except for CLEVR-ART where its soft version (with $\epsilon=450$) was used.
For all datasets, the multi-scale glimpse sensor provided glimpses at 3 scales each of which was of size $15\times 15$, where the first scale corresponded to the original resolution and the other two downsampled the regions of size $30\times 30$ and $45\times 45$.
For the log-polar sensor, we use OpenCV (Bradski, ) formulation and implementation of the log-polar sampling setting the radius of the log-polar grid was set to 48.
The glimpse size was set to 21 for SVRT (and SVRT #1-OOD) and ART tasks and to 15 for CLEVR-ART.

In our glimpse-based models, we use a small CNN to pre-process each glimpse content before feeding it to the downstream architecture.
For glimpse contents produced by the multi-scale sensor, we use a 7-layer CNN where each layer has the kernel size of $3\times 3$, no padding and a ReLU activation function. For glimpse contents produced by the log-polar sensor, we use almost the same CNN but with 6 layers where the first 4 layers have the kernel size of $5\times 5$. The number of channels per layer is the same in both cases being 9 for SVRT (and SVRT #1-OOD) and 64 for the rest of the datasets.
Each glimpse location was re-scaled to be in range $[-1,1]$ and was pre-processed by an MLP with with two layers of size 32 and 64.
As in (Mondal et al., ), we applied temporal context normalization (TCN) (Webb et al., ) on both sequences of pre-processed glimpse contents and locations. TCN normalizes a sequence along its temporal dimension.

For all datasets, the Abstractor module was instantiated with 8 attention 64-dimensional heads, 24 layers, and 2-layered MLPs with the hidden size of 256.
The 50% dropout for the MLPs and attention is used only for SVRT #1-OOD.
To generate the final output for the task, we took the mean of the sequence of final representations $\bm{r}_{L}$ and passed it to a single linear unit.

For the training, we used ADAM optimizer (Kingma & Ba, ) with the learning rate of $1e-4$, batch size of 64.
The whole model was trained end-to-end with the binary cross-entropy loss.

## Appendix D Ablations and additional results

**Table D.1: Test accuracy (in %) for SVRT dataset using different glimpse sensors. The accuracy results are averaged over SD and SR tasks obtained from the best trained model.**
| Model | Glimpse sensor | SD tasks | SR tasks |  |  |
| --- | --- | --- | --- | --- | --- |
| Dataset size<br>1000 | Dataset size<br>500 | Dataset size<br>1000 | Dataset size<br>500 |  |  |
| GAP-Transformer | multi-scale | 80.8 ± 13.5 | 68.0 ± 14.3 | 97.9 ± 2.5 | 96.5 ± 3.1 |
| GAP-Transformer | log-polar | 83.3 ± 13.2 | 78.4 ± 13.5 | 98.2 ± 2.2 | 97.3 ± 2.7 |
| GAP-Abstractor | multi-scale | 93.1 ± 13.1 | 90.5 ± 14.5 | 98.5 ± 1.9 | 96.6 ± 2.1 |
| GAP-Abstractor | log-polar | 92.5 ± 12.3 | 90.6 ± 13.3 | 98.2 ± 2.6 | 97.0 ± 3.4 |

**Table D.2: OOD performance (in %) for test sets from SVRT #1-OOD using different glimpse sensors. The accuracy results are averaged over all OOD test sets for 10 trained models.**
| Model | Glimpse sensor | Accuracy<br>(averaged over runs<br>and OOD datasets) |
| --- | --- | --- |
| GAP-Transformer | multi-scale | 76.3 ± 18.4 |
| GAP-Transformer | log-polar | 74.7 ± 16.0 |
| GAP-Abstractor | multi-scale | 84.4 ± 15.3 |
| GAP-Abstractor | log-polar | 89.6 ± 8.7 |

**Table D.3: Accuracy (in %) for tasks from the ART dataset using different glimpse sensors. The accuracy results are averaged over 10 trained models.**
| Model | Glimpse sensor | Task |  |  |  |
| --- | --- | --- | --- | --- | --- |
| SD | RMTS | Dist3 | ID |  |  |
| GAP-Transformer | multi-scale | 76.4 ± 13.7 | 62.2 ± 7.3 | 50.7 ± 6.1 | 55.5 ± 5.8 |
| GAP-Transformer | log-polar | 64.3 ± 9.1 | 52.6 ± 0.8 | 51.3 ± 13.7 | 57.0 ± 8.8 |
| GAP-Abstractor | multi-scale | 95.8 ± 0.8 | 95.3 ± 2.0 | 93.9 ± 1.3 | 91.1 ± 4.5 |
| GAP-Abstractor | log-polar | 97.7 ± 2.0 | 96.3 ± 1.0 | 98.4 ± 0.5 | 96.8 ± 1.8 |

**Table D.4: Accuracy (in %) for tasks from the CLEVR-ART dataset using different glimpse sensors. The accuracy results are averaged over 5 trained models.**
| Model | Glimpse sensor | Task |  |
| --- | --- | --- | --- |
| RMTS | ID |  |  |
| GAP-Transformer | mult-scale | 76.4 ± 4.3 | 64.7 ± 5.6 |
| GAP-Transformer | log-polar | 70.0 ± 3.5 | 63.0 ± 8.0 |
| GAP-Abstractor | multi-scale | 95.9 ± 1.4 | 93.4 ± 1.3 |
| GAP-Abstractor | log-polar | 94.6 ± 2.0 | 92.5 ± 1.5 |

Figure: Figure D.5: Test accuracy for each of 23 SVRT tasks using different glimpse sensors for the GAP-Abstractor model.
Refer to caption: x9.png

Figure: Figure D.6: Sample efficiency on the SVRT dataset for 11 same-different (SD) tasks and 12 spatial relations (SR) tasks depending on the size of the training dataset. The results report the test accuracy [%] for the best trained models, the standard deviation is computed over SD and SR tasks.
Refer to caption: x10.png

**Table D.5: Sample efficiency on the SVRT dataset for 11 same-different (SD) tasks and 12 spatial relations (SR) tasks depending on the size of the training dataset. The results report the test accuracy [%] for the best trained models, the standard deviation is computed over SD and SR tasks. Only our models are listed. The prior art models are shown in Fig. D.6**
| Dataset size | SD tasks |  |  |  |
| --- | --- | --- | --- | --- |
| GAP-Abstractor | GAP-Transformer | Abstractor | Transformer |  |
| 10000 | 96.2 ± 6.7 | 95.3 ± 6.2 | 52.2 ± 4.8 | 57.4 ± 14.5 |
| 5000 | 95.4 ± 8.1 | 92.7 ± 9.8 | 51.7 ± 3.9 | 56.5 ± 14.2 |
| 1000 | 93.1 ± 13.1 | 82.5 ± 15.1 | 51.2 ± 2.1 | 51.9 ± 2.7 |
| 500 | 90.5 ± 14.5 | 76.0 ± 13.4 | 51.0 ± 1.9 | 51.6 ± 1.7 |
| 250 | 86.8 ± 16.0 | 69.4 ± 15.3 | 50.9 ± 1.5 | 51.3 ± 1.2 |
| 100 | 74.3 ± 17.2 | 58.7 ± 6.2 | 50.6 ± 0.9 | 51.0 ± 0.8 |

| Dataset size | SD tasks |  |  |  |
| --- | --- | --- | --- | --- |
| GAP-Abstractor | GAP-Transformer | Abstractor | Transformer |  |
| 10000 | 99.7 ± 0.7 | 99.7 ± 0.5 | 93.5 ± 13.9 | 82.9 ± 19.2 |
| 5000 | 99.6 ± 0.5 | 99.6 ± 0.8 | 86.5 ± 17.3 | 75.7 ± 20.3 |
| 1000 | 98.5 ± 1.9 | 98.2 ± 2.2 | 60.9 ± 14.3 | 64.8 ± 13.7 |
| 500 | 96.6 ± 1.2 | 97.2 ± 2.5 | 57.0 ± 11.0 | 59.9 ± 11.0 |
| 250 | 93.4 ± 6.3 | 95.1 ± 3.5 | 52.4 ± 3.8 | 58.1 ± 8.8 |
| 100 | 86.2 ± 13.4 | 86.9 ± 11.2 | 52.2 ± 3.3 | 54.5 ± 5.1 |

### D.1 Evaluating importance of glimpse contents and glimpse locations

To measure the extent to which glimpse contents (”what”) and glimpse locations (”where”) are important, we trained our models passing only one of those two sequences to their downstream architectures.
The training was done on SD tasks from SVRT dataset using 1000 training samples.
For the Transformer-based downstream architecture, the input sequence consisted of either glimpse contents or glimpse locations, instead of the concatenation thereof.
For Abstractor-based downstream architecture, we used a fixed set of trainable embeddings from which the RCA produced values while the keys and the queries were produced from either of the aforementioned input sequences.
As it can be seen from Table D.6, the combination of glimpse contents and glimpse locations yields the highest performance compared to cases where only one of those sequences is used.

**Table D.6: Test accuracy [%] averaged over SVRT SD tasks (11 in total) for models trained with 1000 samples depending on information passed to the downstream architecture.**
| Model | Inputs to the downstream architecture |  |  |
| --- | --- | --- | --- |
| glimpse contents | glimpse locations | glimpse contents and locations |  |
| GAP-Abstractor | 81.0 ± 21.9 | 66.3 ± 14.2 | 93.1 ± 13.1 |
| GAP-Transformer | 77.4 ± 12.1 | 72.8 ± 17.5 | 83.3 ± 13.2 |

### D.2 Evaluating the role of processing only the relevant information

The superior performance of our GAP-based models compared to the ablated downstream architectures that process entire images indicates the importance of processing only the relevant information.
The relevance of information in the context of GAP entails two features explained below.

First, GAP discards all uninformative parts of the image providing thereby much less information to the downstream architecture than is available.
This, in turn, improves the learning process preventing the models from being distracted by superfluous information.
To elucidate the importance of this feature, we show the comparison of our GAP approach to the downstream architectures that process i) all image patches obtained by the sliding window approach with the stride of one (such patches almost entirely overlap and include glimpse contents obtainable by GAP); ii) all non-overlapping image patches obtained in ViT-like fashion based on the regular grid. The cases i) and ii) are referred to as all patches and ViT patches respectively.

Second, each glimpse content is focused meaning that it is centered around a salient image region.
This is, for example, in stark contrast to the patching process taking place in ViTs where the image is split into non-overlapping patches based on a regular grid.
In that case, even if there were a method to determine only a subset of the most relevant patches, those patches would not necessarily contain the best portions of salient regions.
To show the importance of focusing, we compare GAP to its modification where glimpse locations are tweaked to be in a set of “coarse” locations defined by the ViT’s regular grid. Importantly, this set of locations is only a subset of all possible image locations that the GAP can glimpse at.
The resulting glimpse contents are obtained from the corresponding tweaked glimpse locations. We refer to this modification of GAP as GAP-regular.

All models described above were trained on SVRT SD tasks (11 tasks in total) using 1000 training samples. The size of the patches was the same as the size of the glimpse contents ($15\times 15$) and the number of glimpses for GAP and GAP-regular was set to 15. The total number of patches obtained in the cases of all patches and ViT patches is 12996 and 64 respectively.

Table D.7 shows the corresponding results.
It can be seen that neither of the two aforementioned features alone is sufficient to solve the tasks.
Specifically, providing all information available in face of either all patches or ViT-patches makes the downstream architecture not able to solve any task.
The same can be seen in the case of providing less information but not well focused on salient regions (although the performance of GAP-regular is slightly above the 50% chance level).
In other words, it is important to process only the relevant information by providing only a subset of available information and making this subset contain information focused on the most salient image parts.

**Table D.7: Test accuracy [%] averaged over SVRT SD tasks (11 in total) for models trained with 1000 samples depending on information passed to the downstream architecture. See Section D.2 for more information.**
| Downstream<br>architecture | All patches<br>(#patches=12996) | ViT patches<br>(#patches=64) | GAP-regular<br>(#patches=15) | GAP<br>(#patches=15) |
| --- | --- | --- | --- | --- |
| Abstractor | 50.2 ± 1.3 | 51.3 ± 2.5 | 58.6 ± 14.8 | 93.1 ± 13.1 |
| Transformer | 50.9 ± 1.9 | 51.9 ± 2.6 | 53.8 ± 6.1 | 83.3 ± 13.2 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * The paper should include a broader discussion of related work (see above).
* The paper should better motivate the choice of evaluation datasets.
soundness: 3
strengths: * The paper provides interesting insights into visual reasoning problems.
* The proposed glimpse-based active perception is relatively novel and interesting.
* The proposed approach shows promising performance on a range of visual reasoning problems.
* The paper considers diverse visual "sensors" and downstream architectures.
summary: The paper focuses on understanding visual relations. This remains a challenging problem for current vision models. To deal with this challenge the paper leverages active vision where the learning of visual relations is grounded in actions that we take to fixate objects and their parts by moving our eyes. The proposed approach with glimpse-based active perception demonstrates promising performance on a range of visual reasoning tasks.
weaknesses: * While the proposed approach is somewhat novel it is similar to prior work such as "AdaGlimpse: Active Visual Exploration with
Arbitrary Glimpse Position and Scale, ECCV 2024" which also focus on what and where to look.

* Current state of the art vision-language models such as LLaVA (NeurIPS 2024) keep the visual features from the target image in the context window. This means that they can actively attend to the image as many times are necessary to extract visual features. It would be interesting to compare the proposed approach to current SOTA VLMs such as LLaVA.

* Prior work such as "Look, Remember and Reason: Grounded reasoning in videos with language models, ICLR 2024" used surrogate tasks to decide what and where to look at. It would be beneficial to include a discussion of this related work.

* Synthetic data: Three out of the four datasets used for evaluation are based on synthetic datasets. It would be beneficial to include more real world datasets such as GQA or Super-Clevr.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Can the authors provide an explanation to why GAP-Abstractor perform worse on 'straight lines' and 'rectangles' compared to other classes? 
- Since the authors explained that "glimpsing behavior can be distracted by spurious salient locations such as edges of shades or regions of high reflection", would performing GAP (finding salient regions) on feature maps, instead of images, where these spurious features are probably less influential, increase performance? Can authors provide results? 
- Please explain why each sensor works better or worse for each downstream architecture-dataset pair.
soundness: 3
strengths: - Authors propose a well-motivated method: 
  - They explain the motivation behind GAP logically, linking back to the human perception model throughout their method section. 
  - Ie. They explain that they concentrate on salient regions since "humans use active vision"; They explain that they use the concept of error neurons since "the activity of neurons is influenced ... also by stimuli that come from the surroundings of those receptive fields". 

- The proposed method attains strong performance on benchmarks:
  - Especially compared to previous work that do not use pretraining as GAP, GAP-Abstractor outperforms them by large margins. 
  - Authors also infer on four datasets, each with different purposes (testing visual reasoning only vs OOD generalization too) and different domains (binary figures vs real-world images). 

- The paper is well-written and easy to follow with both textual and visual explanations. 

- This paper introduces a novel and logical method to field of Visual Reasoning, which is of great importance since visual reasoning is required for a range of real-life tasks, and thereby contributes to the community.
summary: This paper proposes a novel method for visual reasoning, called Glimpse-based Active Perception (GAP). Based on that the human eye selectively concentrates on salient and/or task-relevant parts of a scene, the authors devise a method that extracts salient regions of an image and feeds them into downstream architectures to enforce perception of salient regions only. Firstly, a saliency map is built based on error neurons, which marks salient regions as those that differ significantly with their surrounding neighbours. Secondly, salient regions are extracted from the image, using the saliency map, with either the multi-scale or log-polar glimpse sensor. Lastly, these salient regions and their locations are fed into either Transformer or Abstractor to perform visual reasoning tasks, each dubbed as GAP-Transformer and GAP-Abstractor. By forcing the downstream architectures to concentrate on salient regions only, the GAP-Abstractor achieves SOTA or comparative-to-previous-SOTA performance on the conventional visual reasoning benchmarks, OOD benchmarks, and real-image-based OOD benchmarks.
weaknesses: - Authors do not make it explicit or clear that different sensors (multi-scale or log-polar) have been used across different datasets for the same downstream architectures (ViT/Abstractor). Ie. For SVRT, GAP-Abstractor used the multi-scale sensor but for SVRT #1-OOD it used the log-polar sensor. This should be made clear in the main paper, instead of referring to the Appendix. 
  - Also, in Appendix D, authors do not explain why each sensor works better or worse for each downstream architecture-dataset pair. Since the performance depends on the type of sensor and since authors have proposed to use both, authors should explain where each sensor may be a better choice. 

- References are missing in Tables and Figures. 
- Explain what the tasks RMTS and ID are, at least briefly, in Section 5.4 before explaining the results.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Have you explored alternative methods for computing the saliency map, given that GAP's effectiveness largely depends on the map's accuracy in identifying key image regions?
2. What is the impact of using a hard versus a soft mask M(x_t)?
soundness: 3
strengths: 1. The paper introduces a novel "Glimpse-Based Active Perception" (GAP) model inspired by human active vision. 
2. The proposed approach reaches state-of-the-art performance on several visual reasoning tasks being more sample-efficient, and generalizing better to out-of-distribution visual inputs than prior models.
summary: The authors develop a system equipped with a novel Glimpse-based Active Perception (GAP) that sequentially glimpses at the most salient regions of the input image and processes them at high resolution. Their approach reaches state-of-the-art performance on several visual reasoning tasks being more sample-efficient, and generalizing better to out-of-distribution visual inputs than prior models.
weaknesses: 1.  The images in the four selected datasets seem relatively simple, with very clean backgrounds. Have you considered comparing your proposed model with baseline models on more realistic image datasets, such as the COCO dataset?
2. Given that the GAP mechanism involves multiple steps (e.g., saliency map generation, inhibition of return), have you compared its computational performance with other baseline models?
3. Why can GAP-Abstractor improve OOD generalization of same-different relation and more abstract relations?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: I'm not sure about the term "active", since the architecture relies on a fixed (and not adaptive or recurrent) scheme to extract local features? 

Why is the dataset size for results in Table 1 restricted to 500/1000 training examples? This seems arbitrary. Figure 4 seems to suggest that the training set is larger? 

Can the SVRT #1-OOD experiment be extended to other subsets than SVRT #1? 

Is there a reason why the results in Table 3 (on a subset of ART) do not include all models considered previously (e.g. Attn-ResNet?) 

It would be very beneficial to include a modern vision-language model in the evaluation, as these models have become very good at solving similar reasoning tasks. 

Some additional related work: 

"Learning to combine foveal glimpses with a third-order Boltzmann machine", Larochelle et al. 2010 

"Multiple Object Recognition with Visual Attention", Ba et al. 2014 

"Show, Attend and Tell: Neural Image Caption Generation with Visual Attention", Xu et al. 2015
soundness: 3
strengths: The paper explores an architecture that make explicit the positions and not just appearance of local image regions. These types of architecture are interesting because of the potential capabilities they can enable, and because of their relation to human vision. The paper is well written and fairly easy to follow. The proposed architecture is very simple.
summary: The paper presents an architecture that performs visual perception based on local glimpses. It uses a saliency map to determine glimpse positions, and then feeds an encoding of the appearance and of the location of the glimpses to an existing downstream architecture - a Transformer or the recently proposed Abstractor. The glimpse extraction is hardcoded while the downstream architecture can be trained. The architecture is evaluated on reasoning tasks that rely on local structure and spatial relations.
weaknesses: The proposed architecture can be viewed as a local feature extractor that encodes local image regions and their positions. There is very extensive prior work in this area, which makes the novelty somewhat limited. The proposed architecture is evaluated on two types of toy task (including some variants of those tasks): SVRT with/without OOD setting and ART/CLEVR-ART. All tasks use simple synthetic images with highly local structure in entirely uncluttered scenes, and their solutions seems to strongly rely on positions of objects. One would expect any architecture that makes those positions explicit to do exceptionally well on these specific tasks. Training sets for these tasks are also extremely small and will thereby favor architectures that are based on hardcoded (not trained) feature extraction. Given the extensive prior work on foveated vision, how do existing methods perform on these tasks?

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. What concerns me most is the generalization ability of this approach, although it is inspiring and fancy. Could you give me some clues that this approach can deal with tasks across different domains to reach the authentic reasoning ability?
2. Many people assume that LLM-based reasoning is closest to the generalization due to its rich knowledge. I suppose this approach would also be helpful to the LLM community. Would you conduct some analysis on the relationship between this approach and LLM-based reasoning?
soundness: 3
strengths: 1. It is really inspiring to simulate human's active perception to improve the performance of visual reasoning, which is a contribution to the community of visual reasoning.
2. The approach achieves better sampling efficiency and generalization performance.
3. The authors provide extensive experiment results and analysis.
summary: This paper discusses how to improve the performance of visual reasoning by simulating the active perception of humans, especially when dealing with unknown objects. By expressing the relationship between different image parts with the position generated by glimpse-based active perception and the visual parts around them, this approach reaches a better sampling efficiency and generalization performance.
weaknesses: 1. For the discussion of generalization ability, the paper focuses on the OOD data of the specific benchmarks. However, what we hope to achieve is the ability that models can deal with totally different tasks like humans, although it is really though. This paper lacks the analysis of it.
2. Recently, many LLM-based approaches are gaining higher performance on the reasoning tasks like VQAs. More analysis of the relationship between pure abstract reasoning like this approach and LLM-based reasoning is desirable.

Meta-review: The paper proposes a Glimpse-based Active Perception (GAP) system that sequentially focuses on salient image regions, leveraging spatial information from eye-movement-inspired actions to represent visual relations, achieving state-of-the-art performance in visual reasoning tasks with improved sample efficiency and out-of-distribution generalization.

The manuscript is well-written, presenting an innovative method with convincing results that surpass the current state-of-the-art, while offering valuable insights into plausible mechanisms for visual reasoning on object relations.

Key concerns, including the use of more realistic image datasets and alternative saliency map computations, were satisfactorily addressed during the rebuttal.

Following the rebuttal, all reviewers agreed to accept the paper.

Decision: Accept (Poster)