Improved Efficiency Based on Learned Saccade and Continuous Scene Reconstruction From Foveated Visual Sampling
Authors: Jiayang Liu
Yiming Bu
Daniel Tso
Qinru Qiu
Venue: ICLR 2024 spotlight (2024)
Keywords: Biological inspired high performance energy efficient vision system
data efficient training
energy saving sensoring
learned saccade
reinforcement learning
foveated visual sampling
continuous scene reconstruction.
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: High accuracy, low latency and high energy efficiency represent a set of contradictory goals when searching for  system solutions for image classification and detection. While high-quality images naturally result in more precise detection and classification, they also result in a heavier computational workload for imaging and processing, reduce camera refresh rates, and increase the volume of data communication between the camera and processor. Taking inspiration from the foveal-peripheral sampling mechanism, saccade mechanism observed in the human visual system and the filling-in phenomena of brain, we have developed an active scene reconstruction architecture based on multiple foveal views. This model stitches together information from foveal and peripheral vision, which are sampled from multiple glances. Assisted by a reinforcement learning-based saccade mechanism, our model reduces the required input pixels by over 90\% per frame while maintaining the same level of performance in image recognition as with the original images. We evaluated the effectiveness of our model using the GTSRB dataset and the ImageNet dataset. Using an equal number of input pixels, our study demonstrates a 5\% higher image recognition accuracy compared to state-of-the-art foveal-peripheral vision systems. Furthermore, we demonstrate that our foveal sampling/saccadic scene reconstruction model exhibits significantly lower complexity and higher data efficiency during the training phase compared to existing approaches.

Published as a conference paper at ICLR 2024
IMPROVED EFFICIENCY BASED ON LEARNED SAC- CADE AND CONTINUOUS SCENE RECONSTRUCTION FROM FOVEATED VISUAL SAMPLING
Jiayang Liu', Yiming Bu', Daniel Ts'o", Qinru Qiu'
1 DDpartment of Electrical Engineering and Computer Science, Syracuse University
2 SUNY Upstate Medical University [jliu206, ybu104, qiqiu) @syr.edu, [tsod) @ @upstate.edu
ABSTRACT
High accuracy, low latency and high energy efficiency represent set of conflict- ing goals when searching for system solutions for image classification and de- tection. While high-quality images naturally result in more precise detection and classification, they also result in a heavier computational workload for imaging and processing, reduced camera frame rates, and increased data communication between the camera and processor. Taking inspiration from the foveal-periphert sampling mechanism, and saccade mechanism of the human visual system and the filling-in phenomena of brain, we have developed active scene reconstruc- tion architecture based on multiple foveal views. This model stitches together information from sequence of foveel--prrphher multiple glances. Assisted by a reinforcement learning- based saccade mechanism, our model reduces the required input pixels by over 90% per frame while main- taining the same level of performance in image recognition as with the original images. We evaluated the effectiveness of our model using the GTSRB dataset and the ImageNet dataset. Using an equal number of input pixels, our model demonstrates a 5% higher image recognition accuracy compared to state-of- the art loveal-peripherr based vision systems. Furthermore, we demonstrate that our foveal sampling/saccatoe scene reconstruction model exhibits significantly lower complexity and higher data efficiency during the training phase compared to ex- isting approaches. Code is available at (Githb
views, which are sampled from
INTRODUCTION
High resolution, low latency and high energy efficiency represents set of conflicting requirements for cameras in image sensing and processing system. While utilizing high resolution images usually leads to more precise detection and classification, reduces frame rate, extends the read time and increases the amount of data communications of the camera. Biological visual systems outperform artificial optical sensing and processing systems in precision, responsiveness, and energy efficiency. This partly comes from the fovea of the retina, the saccadic eye movements that permit high resolution sampling of a visual scene, and how the brain utilizes the information obtained from foveal and peripheral region.
The visual field of the human eye spans approximately 140 degrees of arc, where only degree in its central are observed in high resolution called fovea, and the rest of the visual field sampled at diminishing resolution out to the periphery (Bull 2014). Compared to today's camera images, this foveal/peripheral visual architecture has high sparsity and requires much less input data. has also been demonstrated to have superior adversarial and occlusion robustness Deza & Konkle, 2000))
Gant et al., 2021| Vuyyuru et al. 2020 Harrington & Deza 2022; Wang et al.) 2021). Sparse input can effectively reduce computation and memory requirements, however, it also accompanies with significant accuracy loss (Lukanov et al., 2021 Fang et al.) 2020 Zhang et al. 2023 Kaplanyan let al. 2019). The accuracy degradation can partly be mitigated by having high resolution foveal view and low resolution peripheral view. The distribution of information carried by differing portions of an image is highly nonuniform and regionally imbalanced. Placing the foveal center at

---

Published as a conference paper at ICLR 2024
high information region and the peripheral vision in the low information region will result in better perception without significant increase in the amount of imput data and computation.
Although humans receive visual information with drastically reduced resolution in the periphery, we seem to see sharply and clearly. This is because our eyes frequently and rapidly move the foveal centers to place them on different regions, and our brain stitches them together and fills in the rest in a gross, impressionistic way based upon what we know and expect (Ehinger et al.) 2017)) When humans observe their environment with purpose, like during object recognition or visual search, prioritizing a saccade towards an informative fixation further enhances both the speed and accuracy of scene understanding (Shepherd et al., [986] Moore & Fallah) 200T) The brain's ' filling-in capability allows us to complete missing information across the physiological blind spot, and across natural and artificial scotomata, yielding visual perception that continuous and seamless. The sensing behavior and the information collected are closely coupled to each other, together delivering incomparable performance and energy efficiency.
202T).
Inspired by the foveation and saccade mechanisms of the human visual system, and the filling- in phenomena of brain, we present an artificial vision system designed for energy-effiiient and low cost sensing and processing. The system employs a foveal/peripheral vision-inspired image sampling incorporating saccadic control to reduce the amount of data required from the camera. The incoming stream of foveal and peripheral inputs is processed for scene prediction and reconstruction where the missing pixels are filled-in to form a smooth and semantically consistent image. The scene reconstruction model is trained using self-supervised learning maximize the structural similarity and minimize the mean square error between the constructed and original images. Based on the received information, the saccade controller chooses the next foveation target such that the scene recognition can be completed with the minimum amount of input data and the highest accuracy. While there are many forms of scene recognition, without loss of generality, we focus on image classification. Our experimental results show that the scene reconstruction improves the accuracy of image classification by 64.8% on GTSRB dataset and 35.2% on ImageNet dataset. And the controlled saccades provide an additional 2.9% and 11.1% improvement for GTSRB and ImageNet respectively. Overall we can reach a similar image classification results at 70% less pixel usage.
To the best of our knowledge, this is the first work that merges multiple foveal/peripheral based vision samples controlled by saccades into a reconstructed image. While some of the existing works adopts foveal/peripheral vision-inspired image sampling (Lukanov et al.] 2021 He et al. 2021 Uzkent& & Emnon| 2020; Cheung et al., 2016). they do not consider sequential saccades and neither do they perform scene reconstruction. Instead, new classifiers are trained directly on the pixels received from foveal/peripheral regions, which requires a significantly amount of labeled data and tuning efforts. Compared to those works, important advantage of our approach is high data efficiency during the training phase. The self-supervised training of scene reconstruction model does not require any labeled data. The training of the RL based saccade model can be done without the labels if the goal is to maximize the similarity of the reconstructed and the original images. The training process does require labeled data if the goal is maximize the classification accuracy. However, we also found that the saccade model is highly transferable. In other words, the pattern of saccades moving the foveal center similar under different scenes. Using the training data that covers only 50% of classes in the ImageNet dataset, we can train the saccade model that works cqually well as the model trained using all training data. Overall, our model and sensing mechanism can casily be used to replace the front-end of cxisting artificial visual systems without the necessity of modifying the back-end image classification or object detection model. The contribution of this work can be summarized as follows:
This is the first framework that integrates foveal-peripheral sampling and saccade control with continuous scene reconstruction. Experimental results show that the proposed sam- pling and pre-processing framework achieves similar image recognition performance with at least 70% fewer pixels compared to systems with conventional imaging front-end.
The self-trained scene reconstruction model restores the original scene from highly sparse input and achieves average of 0.88 structure similarity on ImageNet dataset. This pro- cess restores missing information to the sparse input and improves the classification accu- racy by at least. 35.2%.
2

---

Published as a conference paper at ICLR 2024
Trained using actor-critic reinforeement learning, our saccade controller additionally im- proves the scene classification accuracy by more than 10%. And is highly transferable. It works effectively scenes that are different from the training classes.
Compared to the state-of-the-ar foveal-peripheral based vision systems, with the similar amount of input pixels, our framework gives 5% higher scene classification accuracy while requires 50% less training data.
In the rest of the paper, we use the term pereppleerpperall view to refer to the pixels sampled from the foveal and peripheral region. We use glance and glimpse to refer to the action of taking a sample. We use the term sevvallperipherde sampling to refer to the sampling strategy (i.e., high density sampling in the foveal region and low density sampling in the peripheral region.)
BACKGROUND AND RELATED WORK
2.1 FOVEAL-PPERPPHERA INPUT IN HUMAN VISION SYSTEM
Human eyes typically capture much more information than our brains are capable of processing (Borji et al. 2019). To improve the efficiency of information transmission and processing, the human visual system has develop multiple mechanisms to ensure the eyes move to where they most needed. There are primarily two types of photosensitive cells in the eyes responsible for retinal image capture. They distributed non-uniformly in the retina, with the cones providing color and high acuity central vision, while the rods specializing in low-light vision (Roorda & Williams) 1999). There are about 6 million cones and 125 million rods in each human eye. More remarkably, this retinal information is highly processed and compressed, culminating to only 1.2 million separate outputs (axons) in each optic nerve. While we possess high acuity only a very small portion of our entire visual field, we rarely suffer from any obvious deficiency due to our foveal and saccadic mechanisms(Hirsee & Curcio 1989).
The fovea centralis is small pit structure in the eye, composed of closely packed cones. People can see objects clearly only in this small region which 1-2 degrees in the field of vision. Outside the fovea centralis, the spatial resolution of human vision drops dramatically. To continuously collect useful information, our neural system processes the visual information at the current fixation site and moves the foveal center to new locations where further information might be found. Such process is referred to as a saccade. The correlation between foveation and saccade has been extensively studied and confirmed by many studies (Poletti et al. 2013 Shepheedd et al., 1986 Moore & Fallah 2001 Henderson & Ferreira 1990). Biological studies prove that saccade planning is related to attentional signals generated by the brain (Shepherd al., [986] MMoore& & Fallah, (001)) based on the information provided in periphery. Experiments also indicates that saccade planning helps the brain process visual stimuli more efficiently.
2.2 MACHINE LEARNING FOR ARTIFICIAL FOVEATION AND SACCADE
Many artificial visual systems learned from the biological foveation and saccade processes to reduce
data processing and transmission overhead and to enhance image understanding (Zhao et al. 2018
Elsayed et al. 2019: Uzkent & Ermon 2020 Oord et al. 2018 Deza & Konkle| 2020) Wang & Cottrell 2017). For example, (Jaramillo-viila & Anderson 2019) showed that using the foveated sampling, the image size can be reduced to 1/16 of its original value which doubles the frame rate of object detection. At the same time the percentage recall of object detection in the foveal center drops from 34% to 24% while the percentage precision increases from 25% to 30%. Overall, the performance of object detection on foveated image is much better than that on a uniformed subsampled image. However, the recall and precision of object detection out in periphery drops rapidly to only 10% of the baseline. This work simply imposes a fovea centralis to the center of the image without saccades. The authors of (He et al. 2021) subsample an image by taking some random patches, which can be viewed as a set of foveal views. They then reconstruct the image using a masked autoencoder trained to minimize the MSE error of the reconstruction. Although the autoencoder is trained via superviee learning, they do need to extensively train and fine tune the classifier. This work does not consider how to sample the patches for better performance.
3

---

Published as a conference paper at ICLR 2024
Determining the best location of the foveal center is similar to locating optimal region for at- tention. A simple way is to move the foveal center to the regions with the most salient image features Wloka al. 2018). However, such saccade model ignores any top-down influences of attention. Lukanov et al. (2021) mitigates this limitation by averaging the bottom-up saliency with top-down saliency from the activation map of classifier. However, the beginning of the visual process, the classification is likely to be wrong, which will lead to incorrect saccade movements. Schwinn et al. (2022) trained a model predict the visual scanpath by minimizing the error of the guide application. They assumed foveation-bil -lur model for the input, which is not sparse. (Elsayed al., 2019) considers foveal center as region with hard attention. applies labeled supervised training to an attention model and moves the foveal center to the region with the highest attention. However, no peripheral vision included in this work and the attention map is generated solely based on the foveal view. As the result, it is not able to díscover interesting regions outside the foveal center and it usually take quite a few number of glimpses in order to gather enough informa- tion. (Chen 202T) learns the strategy of saccades using reinforcement learning. In each decision step, it selects a quadrant of the current input image, refine its resolution, and set the selected quad- rant current image. This process continues until it reaches the desired resolution of the foveal center. The drawback of this approach is the latency. It takes several decision steps to find the location of the foveal center and, for each image, only one foveal center is identified.
For all of the aforementioned works, a new backend model must be trained on the received foveal- peripheral views for classification or object detection. To reach state-of-the-ang performance, large number of labeled data is required. In this work, we present framework that mimics the foveal- peripheral vision and saccade process. actively reconstruct the original image from a sequence of foveal and peripheral views and at the same time decides the location of next foveation target. The reconstructed image resembles with high similarity the original image, hence no new back-end model needs to be trained. Any existing classification object detection model can continue to be used as-is.
G(+1)
Nexl
Neni
action
artion
Actor(Resnet188)
Acton(Rasnet13)
Crllc/ReesNe188)
Convlstm
Crlls|RaeNel1@8
Conv
Come
Resmer101
Resnat101
mtU
((++))
Rensoyt)
Recun(++))
Stap?
Stop?
Prodictive reconstucciion mod dec
Reward
Policy module
Fvaluation modile
Figure I: Architecture Overview
METHOD
The overall architecture of the proposed artificial vision system is comprised of three core compo- nents: the predictive reconstruction module, the evaluation module, and the policy module. Figure shows the data flow between the system components. The predictive reconstruction module receives a sequence of foveal-peripheral views. Its primary function is to merge the received information to generate a high-resolution rendition of the original scene. The policy module controls the sac- cades. It leverages information obtained by the predictive reconstruction module to strategically direct foveal center to a new position. After that, a new foveal-periphent view is sampled from the environment. This entire process recurs until the termination condition is fulfilled.
4

---

Published as a conference paper at ICLR 2024
3.1 FOVEAL-PERIPHERAL VISION
We apply foveal-periphen sampling to reduce the amount of input for a lower transmission and processing cost. We divide the image into N N regions of equal size. For each sampling process, one of them will be selected the foveal center and the rest will serve as the peripheral region. The pixels the foveal center will be sampled with probability 1 and the pixels in the peripheral region will be sampled with very small probably μ. A binary mask M E {0, 1]xxx will be generated based on the sampling probability to select the pixels, where and are the dimension of the original image. The mask is generated independently for every frame, that the system can accumulate more context over time. An example of a sequence of foveal ppriiheel views can be seen in Figure 2 (c).
timestep
timestep
timestep
timestep
timestep
(b) Continous reconstructions
timestop
timestep
timestep
timestep
timestep
1
LV
(a) Original image
(c) Fovea-peripheral sampling
Figure 2: An example of model input(c), model output(b), and original image(a)
3.2 PREDICTIVE RECONSTRUCTION MODEL
We adopt three-layer ConvLSTM as the structure of the predictive reconstruction model, for its superb ability in memorizing spatial-temporal patterns. This model is trained independently to the other two models in a self-supervised manner. Given the foveal-peripheral sampled input as de- scribed in section 3.1, the model trained reconstruct the original image. During training, we randomly select the foveal center to prevent any bias in the training that the trained model can work with any saccade controller. More details about the model and data flow are discussed in appendix A.1.
Previous studies (Lotter et al., 2016 He et al. 2021) that aim to reconstruct the original image from sub-sampled image seek to minimize the mean squared error (MSE) of the reconstruction, simple and inexpensive to compute. However, MSE error is computed on local differences. When calculate MSE for pixel at coordinates (x.y), it only considers the differences between the original and predicted image at location (x.y). Given the substantial sparsity in the input data (only 8% of pixels per frame), the limitations of using MSE loss for image reconstruction is obvious. does not favor predictions with better spatial coherence and higher smoothness. Our experimental results show that the model trained solely on the MSE of reconstructed image yields unsatisfactory results.
is
We have chosen use instead the Local Structural Similarity Index(SSIM) Wang et al. 2004 2003), which takes into account the structural information and luminance of an image, and aligns better with human perception of image quality. It measures the similarity based on patches taken from the same location of two images being compared. For selected patches x and it compares: the similarity of the local patch luminance (brightness values), the similarity the local patch contrast, and the similarity of the local patch structure.
(2pzly + C2) xy + +C2) SSIM(x,)) (M2 13 Ci)(o + of + C2)
Where N Ex, (z) - (y)) and are mean and variance of patches to be compared. In this work, we trained the predictive reconstruction model to minimize a hybrid loss of MSE and (1-SSIM). We found that the hybrid loss gives superior result than using either MSE
5

---

Published as a conference paper at ICLR 2024
SSIM alone.
LosShybrid(z; y) = MSE(2, y) + A|1 SSIM(a,))
(2)
Figure 2 (b) gives an example of the sequence of reconstructed image from the corresponding foveal- peripheral views.
3.3 EVALUATION MODULE
The evaluation module assesses the performance of scene reconstruction. Its output is used to gen- erate reward for the training of the saccade controller. Different evaluation strategies can be applied. For example, SSIM or MSE can be used as evaluation functions to measure the quality of the recon- struction. The evaluation module could also be an existing image classifier, and the accuracy of the classification can be used to measure the quality of the reconstruction. In either cases, the evaluation module does not require additional training.
3.4 SACCADE CONTROL WITH ADVANTAGE ACTOR-CRITIC MODEL
We formulate our problem multi-step episodic Markov Decision Process (MDP). The original image is equally divided into several non-overlapping patches, and those patches form the action space of saccade control Σob = {Jr,, 31, I",,..; vn.. The selected foveal center plus the sparse periph- eral vision form foveal-peripheral view of one glimpse. The controller observes the environment through the the reconstruction model. In each decision step, the hidden state, 81:t. of the reconstruc- tion model, which integrates the present and historical foveal-periphee views, is presented to the saccade controller to choose the next action, ag { We define the saccad control policy model parameterized by pp. as:
(10 8p) = p(ac|s1:;;
Where ( 8,) is a function that maps the observations (i.e., $1:)) to probability distribution the patch sampling actions dz.
The saccade controller is implemented and trained as an advantage actor critic (A2C) model. Both the actor and critic networks adopt the Resnet18 architecture. The saccade controller is trained after the predictive reconstruction model. During the training of the saccade controller, the predictive reconstruction model remains frozen. The training follows the policy gradient method:
Vo, J(pp) E Σ Vo, logToog (L:t) a:)]
(4)
where A(s1:t, ax) is the advantage function, calculated as the following:
A(81a, a:) + VV + V(t)
(5)
The reward is generated by the evaluation module. If SSIM is used as the quality metric then can directly be used as the reward. If the classification result is used as the quality metric, then set Fr to be the top-1 softmax classification probability if the classification returns correct label, otherwise rz is set to be the negated top-1 softmax classification probability as a penalty. (ff) the predicted value from critic model given the state vector $1:t, and V + 1) represents the predicted value after agent take the action Ge- The reason that we train the saccade controller using the gradient of the advantage function instead of the absolute reward is because the expected reward of sequence of random sampled actions may have a very large variance, which will lead to unstable training. Using advantage function A(s1a: ax) can help to reduce the variance.
To collect more information in a limited number of glimpses, we define the actions of moving the foveal center to location that has been selected before invalid actions. This achieved by using an invalid action mask, invalidM ask E R". The ith entry of the vector is M, where 0 < M < 1, action x; is an invalid action, otherwise, it 1. We adjust the action probability using the invalid mask as the following:
p' (a 0p) Equation 6 is differentiable. Hence it be included part of the controller without affect the policy gradient flow as shown in Equation[4] [ For each training and testing image, the initial location of the foveal center is randomly selected.
(s::; 0p) + log (ogiiaaaiddaa
(6)

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
4.1 EXPERIMENT SETUPS
We apply the proposed framework on GTSRB and ImageNe((ILSVRC 2012) datasets. To evaluate its performance, we measure the SSIM and classification accuracy of the reconstructed image. We also report the amount of pixels sampled from the camera in order to achieve the accuracy.
The GTSRB (German Traffic Sign Recognition Benchmark) dataset consists of 43 classes of images. We preprocess the images by resizing to a dimension of 1 112 12 without applying any normaliza- tion. We refer these resized images as our original input. Each image is divided equally into 4x4 patches, where each patch serves as a potential region of 28x28 foveal vision. The peripheral sampling portion varies from 1% to 2%. We evaluation model. It achieves 92.1% 1% top-1 accuracy on the original input.
classifier trained with the original input as the
The ImageNe((ILVRR) 2012) dataset consists of 1000 classes. The original input has size 224 x 224. Two different sizes of foveal regions are tested. The larger foveal region has the size 56x56, which is obtained by dividing the image into 4x4 patches. The smaller foveal region has the size x2x which is obtained by dividing the image into 7x7 patches. A Resnet101 trained using the original input is used as the evaluation model. The model has 77.2% top-1 accuracy on original inputs.
4.2
IMPACT OF FOVEAL-PERIPHIEE SAMPLING AND SCENE RECONSTRUCTION
100
90
70
original
1/16,
1/16, 1.5%
70
1/16,
original
50
1//16, 2% no recon
3/30.
uniForm no recon
50
3/36,
uniform
unlliom
20
20
0
3st gimpss
2nd glimpse
3rd glimpse
4th glimpso
glimpsu
1st glimpse
2nd glimpse
3rd glimpse
4th glimpse
5th glimpse
(a) GTSRB
(b) ImageNet
Figure 3: Comparisons of classification accuracy without trained saccades. x-axis is number of glimpse, y-axis is the classification accuracy. All datapoints are classification results of recon- structed images unless specified as "no recon". The data points labeled "uniform" are obtained from uniformly sampled images without any foveal center. The pixel usage of those images are set to be 8.25% per glimpse.
In the first set of experiments, we evaluate how foveal-peripheral sampling and scene reconstruction can improve scene understanding. Random saccades are used in this experiment. For both datasets, we set the foveal region be 1/16 of the original image and vary the peripheral sampling proportion from 1% to 2%. The foveal centers are randomly picked. After each glimpse, the reconstruction model merges the sampled peraal-peripheral view with the previous input to generate reconstructed image, which will be classified using the evaluation model. The classification accuracy is reported in Figure3uusing red, grey and orange data points. The blue line the figure classification accuracy of the original image, which is also the upper bound of classification.
To demonstrate the impact of scene reconstruction, we directly combine all poveel-peripheral views together without reconstruction and apply the classification model on the combined image, and the result is shown in Feeereere by green triangle data points. As we can see, for both GTSRB and ImageNet dataset, the predictive reconstruction provides significant improvement in terms of top-1 classification accuracy. For 2% peripheral sampling proportion, after scene reconstruction, the object recognition accuracy the ImageNet dataset increases from 4.1% to 12.1% at the first glance. At the fifth glimpse, this number is further improved from 27.4% to 62.6%. The fact that the accuracy improvement resulted from scene reconstruction gradually increases more glimpses
7

---

Published as a conference paper at ICLR 2024
cast indicates that the benefit of predictive reconstruction accumulates. Similar level of accuracy improvements are observed in the GTSRB dataset.
To demonstrate the efficiency of foveal-peripheral sampling, we compared it with uniform sampling. We uniformly sample 8.25% pixels per glimpse such that it has approximately the same number of pixels the peveal-peripheral sampling where the fovea size is 1/16 of the original scene and the peripheral sampling proportion is 2%. The classification results on the uniformly sampled images after 5 glimpses are represented in Figure 3 using black data points. Without scene reconstruction, we can see that the foveal-peripheral sampled vision achieves 24.0% and 16.1% higher classification accuracy than the uniformly sampled image for ImageNet and GTSRB datasets respectively. After scene reconstruction, the improvement reduces to 5.6% and 13.6% for ImageNet and GTSRB. The reason that the benefit of periphera sampling drops for ImageNet is because the images in this dataset have more complicated scenes. Randomly selected foveal centers may miss some important regions compared uniformly sampled input.
100
90
70
70
30
orisinal
40
original
40
3/10,
1/16,
1/15, policy
1/16, 1.5%polyy
20
1/16, 2%
1/16,
1/25, 2%polic
1/16, 2%pollcy
0
glmpse glimpse
glimpse
glimpse
glimpse
1st glimpoe
2nd glimpse
3rd glimpse
4th glimpse
5th glimpse
(a) GTSRB
(b) ImageNet
Figure 4: Classification result W or W/O saccade control. The datapoint labeled as "policy" repre- sents the results using the learned saccade control. Size of foveal region is 1/16 of the original image and the peripheral sampling proportion varies from 1.5% % to 2%.
4.3 IMPACT OF SACCADE CONTROL
In the second set of experiments, we compare the performance of learned saccade controller and a random saccade controller. As shown in Figure 4 with the same number of glimpses, random saccades give lower accuracy compared to learned saccades on both GTSRB (Figure 44) and Ima- geNet(Figure 4p). With learned saccades, the classification accuracy for GTSRB achieved 91. % and 91 188% when we sample 1.5% and 2% pixels for peripheral view respectively. Compared to the results with random saccades, using learned saccades increase the accuracy by 8.6% and 1.9% when the peripheral sampling proportions are 1.5% and 2% respectively. For ImageNet, compared with random saccades, learned saccades improve the top-1 object recognition accuracy by 14.4% and 11.1% respectively when the peripheral sampling proportions 1.5% and 2%. The results show that the improvement introduced by the learned saccades increases when the peripheral view gets sparser. This agrees with intuition: when the information from the peripheral view reduces, the location of foveal center becomes more important.
Model
Test accuracy[ %]
Pixel usage rate %)
Average glimpses
Bio-FCG Saccader
65.17
27
2
70.0
70
7
Ours 7x
70.9
28
7
Ours 4x4
73.7
41
5
Ours7 7early stop
66.4
22
5,4
Ours4x 4early stop
69.4
30
3.5
Table 1: Comparing our results with two STOA foveal-peripheral vision system. 7x7, and 4x4 indicate the action space of fovea. The peripheral sampling proportion is 2%.
8

---

Published as a conference paper at ICLR 2024
We further compare our model with sfate-of-the-arr poveal-peripheral visual systems Bio- FCG(nu et al., 2021) and Saccader(Elsayed et al. 2019). In this experiment, we also in- troduce carly-stop mechanism aiming at further reduced pixel usage. If the evaluation model confidently predicts the same label in two consecutive glimpses, the process will stop. In the exper- iment, the confidence threshold is set to be 0.5. As shown in Table our 7x7 model achieved 5% improvement in accuracy compared to Bio-FCG with similar pixel usage. The Saccader samples least 70% of the pixels in the original image over 7 glimpse to attain a 70% accuracy, whereas our 7 model can achieve slightly better accuracy using only 28% of the pixels in the original image. Moreover, bolstered by the early-stop mechanism, our 4 model achieves 69.4% accuracy with only an average of 3.5 glimpses. We need to point out that both Bio-FCG and Saccader have re-trained or fine-tuned their classifier using the loveal-peripheral view, while our framework did not. Re-train the classifier obviously will boost the classification accuracy, however, it also requires significant amount of training data and effort. While our solution allows the users to keep their existing back-end classifier image processing model as-is.
4.4 DATA EFFICIENT POLICY TRAINING
In the third experiment, compare the performance of the saccade controller trained in different ways. The results are shown in Table 2. It includes a random controller, a controller trained to maximize the structural similarities between the reconstructed and the original images, and set of controllers that are trained using only a small portion of the training set of the ImageNet. More specifically, we randomly select 100, 200, and up to 500 classes of training data the ImageNet to train the saccade controller and test the resulting model on the testing data of both the trained classes and 1000 classes. Finally, we use all 1000 classes training data to train the saccade controller.
We observed that all of the RL trained controllers outperform the random controller. Secondly, the controller trained to maximize the SSIM does gives the highest SSIM value, however, slightly lower classification accuracy than other RL trained controllers. Thirdly, training the controller using data from only half the classes in the training set performs almost equally well as training the model using the entire training set. The result shows that the saccade strategy learned from one type of images can be applied to other types of images. Overall, our approach has high data efficiency during the training phase.
training classes
acc on all classes
acc on training classes
SSIM
Random SSIM
62.6
0.85
66.3
0.90
100
70.3
77.2
0.87
200
71.5
75.3
0.87
300
72.4
74.7
0.87
400
73.1
74.2
0.88
500
73.6
73.8
0.88
1000
73.7
0.88
Table 2: Transferable data efficient learning result. Random is the top1 accuracy for reconstruction from random multiple saccades.
CONCLUSIONS AND FUTURE DIRECTIONS
In this paper, we present novel framework that merges multiple loveal-peripheral views controlled by saccades into a reconstructed image. Our work is an casy-to-use and energy-saving front-end sensing system. can easily be integrated with any existing back-end processing models, such as image classification and object detection. It reduces 70% of pixel usage to achieve similar classification accuracy as the original image and requires no effort to re-train or fine-tune the back- end classification model. It also has 5% more top-1 accuracy compared to state-of-the-art foveal- peripheral based vision systems. Our saccade model is highly transferable and data-efficient. works effectively scenes that differ from the training classes. Our future efforts will focus better periphery sampling techniques and investigating the differences between the trained saccade model versus human visual scanpaths(Kümmeres & Bethge 2021 Kümmerer et al. 2022)

---

Published as a conference paper at ICLR 2024
AKNOWLEDGEMENT
This work is partially supported by the National Science Foundation I/UCRC ASIC (Alternative
Sustainable and Intelligent Computing) Center (CNS-1822165).
REFERENCES
Ali Borji, Ming-Ming Cheng, Qibin Hou, Huaizu Jiang, and Jia Li. Salient object detection: A survey. Computational visual media, 5:117-150, 2019.
David R. Bull. Chapter 2 - the human visual system. In David R. Bull (ed.), Communicating
Pictures, pp. 11-61. Academic Press, Oxford, 2014. ISBN 978-0-12-405906-1. doi: https://doi.
12- 12-401900-091111 1.00000-7 URL https: www htttps://www.sieenneeeeeeeee sclencedrrrrr................... com/
Sci Stt.eenc//arrdccl///1////7200210
0C0027
Gang Chen. Where to look: A unified attention model for visual recognition with reinforcement learning. arXiv preprint arXiv:2 (1.0711,, 2021.
Brian Cheung, Eric Weiss, and Bruno A. Olshausen. Emergence of foveal image sampling from learning to attend in visual scenes. CoRR, abs/1611.0045333 2016. URI. http:tttp//xiiv.rrrrt
abs 1////111.... 09430
Arturo Deza and Talia Konkle. Emergent properties of foveated perceptual systems. CoRR, abs/2006.0793 17991, 2020. URL https://aaxxi......o.mm
07991
Benedikt Ehinger, Katja Häusser, José Ossandón, and Peter König. Humans treat unreliable filled-in percepts more real than veridical ones. 03 2017. doi: 10.1101/066217.
Gamaleldin Elsayed, Simon Kornblith, and Quoc V Le. Saccader: Improving accuracy of hard
attention models for vision. Advances in Neural Information Processing Systems, 32, 2019.
Haowen Fang, Amar Shrestha, Ziyi Zhao, and Qinru Qiu. Exploiting neuron and synapse filter dynamics in spatial temporal learning of deep spiking neural network. CoRR, abs/2003.02944, 2020.
orw.:iiiioooraaoooooe abs 2003. 02944
Jonathan M Gant, Andrzej Banburski, and Arturo Deza. Evaluating the adversarial robustness of foveated texture transform module in a cnn. In SVRHM 2021 Workshop@ NeurIPS, 2021.
Anne Harrington and Arturo Deza. Finding biological plausibility for adversarially robust features via metameric tasks. In Published at The Tenth International Conference on Learning Represen- tations, 2022. URL
net/ forum www.umiddddyy_yey_eeee
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked au- toencoders are scalable vision learners. CoRR, abs/2111.00637, 2021. URL /tttps://///xxiv.
org/ atd//bs/22111 06377
John M Henderson and Fernanda Ferreira. Effects of foveal processing difficulty on the perceptual implications for attention and eye movement control. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16(3):417, 1990.
span in reading:
Joy Hirsch and Christine A Curcio. The spatial resolution capacity of human foveal retina. Vision research, 2995: 1989.
Uziel Jaramillo-Avin and Sean R Anderson. Foveated image processing for faster object detection and recognition in embedded systems using deep convolutional neural networks. Conference on Biomimetic and Biohybrid Systems, pp. 193-204 Springer, 2019.
Jeff Johnson. Chapter 5 our peripheral vision is poor. In Jeff Johnson (ed.), Designing with the Mind in Mind (Third Edition), pp. 57-78. Morgan Kaufmann, third edition edition, 2021. ISBN 978 0- 12- 211222888222 8202-4, doi:
12-00-0------2222-------0-- 2000055 URL hetps:
w/www. i encedi rect. com htt.://weeeecceecceeecoeooeeeeaa 28 28182024000052
10

---

Published as a conference paper at ICLR 2024
Anton S Kaplanyan, Anton Sochenov, Thomas Leimkühler, Mikhail Okunev, Todd Goodall, and Gizem Rufo. Deepfovea: Neural reconstruction for foveated rendering and video compression using learned statisties of natural videos. ACM Transactions on Graphics (TOG), 38(6):1-13, 2019.
Matthias Kümmerer and Matthias Bethge. State-of-the-art in human scanpath prediction. arXiv preprint arXiv:2102. 10:::299922229 2021.
Matthias Kümmerer, Matthias Bethge, and Thomas SA Wallis. Deepgaze ii:: Modeling free-viewing human scanpaths with deep learning. Journal of Vision, 22(5):7-7, 2022.
William Lotter, Gabriel Kreiman, and David D. Cox. Deep predictive coding networks for video prediction and unsupervised learning. CoRR, abs/1605.08104, 2016. URL http:://axiv.
08104
Hristofor Lukanov, Peter König, and Gordon Pipa. Biologically inspired deep learning model for
efficient poveal-peripheral vision. Frontiers Computational Neuroscience, 15:746204, 2021.
Tirin Moore and Mazyar Fallah. Control of eye movements and spatial attention. Proceedings of the National Academy of Sciences, 98(3):1273 1276, 2001.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1801 1807003748, 2018.
Martina Poletti, Chiara Listorti, and Michele Rucci. Microscopic eye movements compensate for nonhomogeneous vision within the fovea. Current Biology, 23((7):1691-1695 2013.
Austin Roorda and David R Williams. The arrangement of the three cone classes in the living human eye. Nature, 397(6719):520 1999.
Leo Schwinn, Doina Precup, Björn Eskofier, and Dario Zanca. Behind the machine's gaze: Biolog- ically constrained neural networks exhibit human-like visual attention. 04 2022.
Martin Shepherd, John M Findlay, and Robert Hockey. The relationship between eye movements and spatial attention. The Quarterly Journal of Experimental Psychology Section A, 38(3):475- 491. 1986.
Burak Uzkent and Stefano Ermon. Learning when and where zoom with deep reinforcement learning. In Proceedings of the IEEF/CVF conference on computer vision and paltern recognition, pp. 12345-12354, 2020.
Manish Reddy Vuyyuru, Andrzej Banburski, Nishka Pant, and Tomaso Poggio. Biologically in- spired mechanisms for adversarial robustness. Advances Neural Information Processing Sys- tems, 33:2135-2146, 2020.
Binxu Wang, David Mayo, Arturo Deza, Andrei Barbu, and Colin Conwell. On the of cor- tical magnification and saccades as biological proxies for data augmentation. arXiv preprint arXiv:2/12.07113, 2021.
Panqu Wang and Garrison W Cottrell. Central and peripheral vision for scene recognition: neu- rocomputational modeling exploration. Journal of vision, 17(4):9-9, 2017.
Z. Wang, E.P. Simoncelli, and A.C. Bovik. Multiscale structural similarity for image quality as- sessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems Computers, 2003, volume 2, pp. 1398-1402 Vol.2, 2003. doi: 10.1 10.1199/ASSS...03.12292.
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, 2004. doi: 10.1 819861.
Calden Wloka, Iuliia Kotseruba, and John K. Tsotsos. Active fixation control to predict saccade sequences. In 2018 JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 3184-3193, 2018. doi: 00.1109/CVPR.2218.0005

---

Published as a conference paper at ICLR 2024
Zhenhang Zhang, Jingang Jin, Haowen Fang, and Qinru Qiu. Neuromorphic online learning for spatiotemporal patterns with a forward-only timeline, 2023.
Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. Psanet: Point-wise spatial attention network for scene parsing. In Proceedings of the European conference on computer vision (ECCV), pp. 267-283, 2018.
A
APPENDIX
A.1 IMPLEMENTATION DETAILS FOR PREDICTIVE RECONSTRUCTION MODEL
The predictive reconstruction model has three Convolutional-LSTT layers and one convolutional layer. As shown in figure I, a loveal-periphers view ,G,.is received and processed by the predictive reconstruction model in each time step based on the following equations:
b Convlstm (Ge, -1)
(7)
h Conulstm (h) h
(8)
S1d Conulstm (h),, $1:-11
(9)
Recon, = C0nv(81::)
(10)
The variables f1).. Ai and 8142 are the hidden states of the ConvLSTM layers, while Recon; is the
predictive reconstruction outcome time t. our model, the initial two layers of the ConvLSTM
gradually increase the depth of feature maps from 3, to and 16, while the last layer compresses all the generated feature maps into channel dimension 3.
A.2 HYBRID RECONSTRUCTION LOSS AND NUMBER OF GLIMPSES
When more glimpses are obtained, the SSIM of the reconstructed image increases, while MSE and hybrid reconstruction loss decrease. In figure 5, we show how hybrid loss, SSIM, and MSE change with the number of obtained glimpses when the controlled saccade is used.
(a) Hybrid
(b) SSIM
(c) MSE
Figure 5: Hybrid Loss SSIM, and MSE after each glance. Size of foveal region is 1/16 of the original image and the peripheral sampling proportion varies from 1% to 2%.
A.3 RECONSTRUCTION UNDER DIFFERENT PERIPHERAL SAMPLING PROPORTION
In this section, examples under different peripheral sampling settings are provided. For all examples. the size of the foveal region is 1/16 of the original image size and the peripheral sampling proportion ranges from 1%, 2% 5%. In each figures of Figures 6-11, there are 6 rows of images, which can be divided into three groups corresponding to the 3 peripheral sampling settings arranged in the ascending order of sampling proportion.
each group, the first row gives the sequence of input loveal-peripher views, and the second row gives the corresponding reconstructed images. All foveal centers are selected by the saccade controller, except the one at timestep 0, which is selected randomly. The image with green bounding box is the one that is classified correctly. Please note that the image of Monalisa does not belong to the ImageNet, therefore, has no classification result. We also that some of the cases have fewer glimpses than other. This is because of the early stop mechanism. If the same classification result is recived with sufficience confidence consecutively, then the whole process will stop.
12

---

Published as a conference paper at ICLR 2024
timestep 0
timestep 1
timestep
timestep 3
timestep 4
Poreae-peripheral views and Sequential reconstructions when peripheral smapling proportion 1%
timestep 0 timestep
timestep
timestep 3
timestep 4
(b) Poveal-peripheral views Sequential reconstructions when peripheral smapling proportion 2%
timestep 0
timestep 1
timestep 2
timestep 3
timestep 4
(c) perppelperipheral views Sequential reconstructions when peripheral smapling proportion 5%
Figure 6: Monalisa reconstruction result when peripheral sampling proportion is 1%, 2% and 5%
respectively.
13

---

Published as a conference paper at ICLR 2024
timestep 0
timestep 1
timestep 2
timestep 3
timestep 4
(a) Poveal-peripheral views Sequential reconstructions when peripheral smapling proportion 1%
timestep 0
timestep 1
timestep 2
timestep 3
timestep 4
(b) Foveal-peripheral views and Sequential reconstructions when peripheral smapling proportion is 2%
timestep 0
timestep 1
timestep 2
timestep 3 timestep 4
(c) foveal-peripheral views and Sequential reconstructions when peripheral smapling proportion 5%
Figure 7: ImageNet reconstruction result when peripheral sampling proportion is 2% and 5% respectively
14

---

Published as a conference paper at ICLR 2024
timestep 0
timestep 1
timestep 2
timestep 3
timestep 4
(a) peripe-peripheral views Sequential reconstructions when peripheral smapling proportion 1%
timestep 0
timestep 1
timestep 2
timestep 3
timestep 4
(b) Poweal-paripheral views Sequential reconstructions when peripheral smapling proportion 2%
timestep 0
timestep 1
timestep 2
timestep 3 timestep 4
(c) Foveal- periphepe views and Sequential reconstructions when peripheral smapling proportion is 5%
Figure 8: ImageNet reconstruction result when peripheral sampling proportion is 1%, 2% and 5%
respectively
15

---

Published as a conference paper at ICLR 2024
timestep 0
timestep 1
timestep 2
timestep 3
timestep 4
(a) Poreal-peripheral views Sequential reconstructions when peripheral smapling proportion 1%
timestep 0
timestep 1
timestep 2
timestep 3
timestep 4
(b) pereae-pl views and Sequential reconstructions when peripheral smapling proportion is 2%
timestep 0
timestep 1
timestep 2
timestep 3
timestep 4
(c) Foveal- pereal-peripheral views and Sequential reconstructions when peripheral smapling proportion is 5%
Figure 9: ImageNet reconstruction result when peripheral sampling proportion is %, 2% and 5% respectively
16

---

Published as a conference paper at ICLR 2024
timestep 0
timestep 1
timestep 2
timestep 3
timestep 4
(a) Poreal-peripheral views and Sequential reconstructions when peripheral smapling proportion 1%
timestep O
timestep 1
timestep 2
timestep 3
timestep 4
(b) Foveal-peripheral views and Sequential reconstructions when peripheral smapling proportion 2%
timestep 0
timestep 1
timestep 2
timestep 3
timestep 4
(c) Foveal peripherr views and Sequential reconstructions when peripheral smapling proportion is 5%
Figure 10: ImageNet reconstruction result when peripheral sampling proportion 1%, 2% and 5% respectively
17

---

Published as a conference paper at ICLR 2024
timestep 0
timestep 1
timestep timestep 3 timestep 4
(a) Foveal pereppearripe views and Sequential reconstructions when peripheral smapling proportion is 1%
timestep 0 timestep
timestep timestep 3 timestep 4
perihe-pa views and Sequential reconstructions when peripheral smapling proportion 2%
timestep 0
timestep 1
timestep timestep 3
timestep 4
(c) Foveal peripheral views Sequential reconstructions when peripheral smapling proportion 5%
Figure 11: ImageNet reconstruction result when peripheral sampling proportion is 1%, 2% and 5% respectively
18


Reviews:
[{"review_id": "WXa16Jej34", "reviewer": "Reviewer_5v5k", "summary": "This paper presents a novel application of spatially-varying computation (foveation) coupled with eye-movements towards the goal of image reconstruction. The authors introduce an active sensing model that takes into account all of the image information in a spatially varying way and continually updates the visual stimulus until it is near perfectly reconstructed. Authors introduce a novel loss function and show small toy experiments that prove their claims.", "questions": "I am open to changing my mind about this paper. There are a lot of missing papers, but the idea seems interesting. I am fan of papers that explore non-intuitive applications or theories of foveation but I am still not there yet to give this paper a clear accept.\n\nI'm also struggling to know what is $t_0$? Is it a blank image? Is it a corrupted image? Is it only a fraction/glimpse of an image?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- This paper presents a novel application of foveal-peripheral vision tailored towards image reconstruction.\n- The paper has shown and presented a set of experiments that seem to support their claimed contribution\n- The paper has references many other works in perceptual psychology and neuroscience -- though many more of these papers are missing, and the field has moved forward quite a lot (see Weaknesses below), thus potentially impacting the novelty of the paper.", "weaknesses": "I think the main weakness this paper has is I am confused on how the system is trained to do reconstruction. Is it doing the reconstruction from the same image and \"testing on the training set\"? Otherwise, I am surprised the first auto-completion of the image is surprisingly quite well without any prior knowledge of the underlying geometry of the visual stimulus. If indeed it is testing on the training set, what would be the contribution/application of such system? A compression engine that works better than JPEG, or would the contribution here really be more of an intellectual one of saying that reconstruction through foveation is indeed possible.\n\n-------\nThere are a set of missing papers that the authors should add and/or discuss in this work. While none of these papers directly attack the problem of using foveation as a tool for reconstruction, many of such works discuss the complimentary theory of foveation having a representational goal in addition to purely optimizing for metabolic cost (and thus limiting the impact of the authors through this paper)\n\nKey Missing Critical References:\n- Deza & Konkle. ArXiv, 2021. Emergent Properties of Foveated Perceptual Systems.\n- Wang & Cottrell. Journal of Vision, 2017. Central and peripheral vision for scene recognition: A neurocomputational modeling exploration.\n- Cheung, Weiss & Olshausen. ICLR 2017. Emergence of foveal image sampling from learning to attend in visual scenes\n\nSecondary, but also important References:\n- Gant, Banburski & Deza. SVRHM, 2022. Evaluating the adversarial robustness of a foveated texture transform module in a CNN.\n- Reddy, Banburski, Pant & Poggio. NeurIPS 2020. Biologically inspired mechanisms for adversarial robustness\n- Wang, Mayo, Deza, Barbu & Conwell. SVRHM, 2021. On the use of Cortical Magnification and Saccades as Biological Proxies for Data Augmentation\n- Harrington & Deza. ICLR, 2022. Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks\n\nIn addition the original SSIM paper:\n- Wang, Bovik, Sheik & Simoncelli. IEEE TIP, 2004. Image quality assessment: from error visibility to structural similarity (SSIM).\n\nand Foveation paper that introduce the idea of texture-based computation in the periphery:\n- Freeman & Simoncelli. Nature Neuroscience, 2011. Metamers of the Ventral Stream.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l4oymZThIA", "reviewer": "Reviewer_cXLY", "summary": "The authors present an innovative solution for image classification and detection that addresses the trade-off between image quality and computational efficiency. They introduce an active scene reconstruction architecture that leverages foveal and peripheral views, along with a reinforcement learning-based saccade mechanism, reducing input pixels by over 90% per frame while maintaining image recognition performance.", "questions": "- Can you provide more details about the computation overhead of your model and possible complications in real world applications?\n- Can you better frame your contribution, and compare it to the literature in saccade modeling?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper introduces an innovative concept inspired by the human visual system, combining foveal and peripheral views with a saccade mechanism in image reconstruction. This approach has potential applications in various fields.\n- A 90% reduction in required input pixels per frame has practical implications for real-time image processing\n- paper is easy to read", "weaknesses": "- Although the paper addresses the trade-off between image quality and computational efficiency, it would be valuable to provide insights into the computational overhead of implementing the proposed model, particularly in terms of hardware and energy requirements.\n- The paper totally fails to mention a whole branch of literature in saccade modeling. See for example [1], [2], or  [3]. In particular, [2] also uses reconstruction as a guiding task. It seems true that none of the mentioned approaches focused on performance in terms of image reconstruction, but I think it is relevant to at least position the current contribution compared to those. I imagine, some of these saccade models could potentially be used in the same framework proposed by the authors here.\n\n[1] Wloka, C., Kotseruba, I., & Tsotsos, J. K. (2018). Active fixation control to predict saccade sequences. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3184-3193).\n[2] Schwinn, L., Precup, D., Eskofier, B., & Zanca, D. (2022). Behind the Machine’s Gaze: Neural Networks with Biologically-inspired Constraints Exhibit Human-like Visual Attention. Transactions on Machine Learning Research.\n[3] Assens, M., Giro-i-Nieto, X., McGuinness, K., & O'Connor, N. E. (2018). PathGAN: Visual scanpath prediction with generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops (pp. 0-0).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dqAyHkLSVb", "reviewer": "Reviewer_rEUv", "summary": "This paper aims to reconstruct the original image from multiple subsampled views, using reinforcement learning and neural network models for scan control and image reconstruction, respectively. The paper conducts numerous experiments to demonstrate that the proposed algorithm can maintain detection task accuracy, reasonable saccade control, and high reconstruction quality under high data efficiency. However, the motivation for the work is not well-founded, and there are possible improvements in the experiments.", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The task addressed in the paper is novel, as it is the first in the industry to reconstruct an image from continuous central foveal subsampled images. Other methods focus on single-sample images and proceed directly to downstream tasks without reconstructing the original image, making this work unique.\n2. The methods used are innovative, employing an actor-critic model for saccade control, which can achieve near-original image classification accuracy in just five scans.\n3. The writing style of the paper is easy to understand, especially in describing the proposed methods.", "weaknesses": "1. While the task is novel, it lacks a convincing real-world application, as it simulates the process of multiple eye samplings without addressing practical problems.\n2. The experimental comparisons are not entirely fair. The uniform control group uses an 8% sampling probability, while the 1/16+2% group differs by 0.25%, indicating an unequal amount of information that might affect performance.\n3. Using classification model metrics to assess the quality of reconstruction is questionable, as classification tasks do not focus on texture details. If this method was to downsample the original image with the same number of sampled pixels, how much better is the method in terms of performance compared to this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2PQHB1UZba", "reviewer": "Reviewer_7Zf9", "summary": "This paper presents a new algorithm for sequential foveated visual sampling of an image.\n\nThe main claims of the paper are that \n\n- the required input pixels per frame are reduced by 90% without losing image recognition performance\n- 5% higher recognition accuracy compared to existing foveal sampling models with matching pixel number input\n- higher data efficiency in training\n\nI find the algorithm to be interesting and novel, and that the second and third claims above are supported.\nI am confused where to find evidence for the first claim.\n\nOverall I think this paper is a borderline accept.", "questions": "- I would like to see how the hybrid reconstruction loss changes over timestep, and not just classification accuracy.\n- The sampling of the periphery of individual pixels with small probability is not very like human vision. Effectively this is providing low pass information. Have the authors considered how the sampling density could be approximated more plausibly (e.g. [6])?\n- Have the authors considered comparing scanpath strategies learned in this model to human scanpaths (e.g. [3, 4])?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I find the method simple and useful, with interesting potential application. \nIt is appealing that the method seems to be suitable for existing classification models (no retraining).", "weaknesses": "## Major\n\nI am confused how the image information from the sequential glimpses is passed and integrated in the predictive reconstruction model. Much more space is spent on the background to the hybrid loss function than actually making explicit how the sequential image information is used to improve reconstruction.\n\nIn addition, the abstract states \"our model reduces the required input pixels by over 90% per frame while maintaining the same level of performance in image recognition as with the original images.\" I don't understand where to find support for this claim in the results. For example, in Figure 3, all subsampled models perform worse than the original. The data in Figure 4 are coming closest to the original; is this what is meant?\n\nAlso, please clarify whether the experiments in Figure 3 are conducted with the trained saccade control model (which one?). \n\n\n## Minor\n\n- Instead of \"continuous saccades\" a better terminology would be \"sequential saccades\" or \"scanpaths\". See e.g. [2, 3, 4]\n- There are now known to be three types of photosensitive cells: rods, cones and intrinsically-photosensitive ganglion cells [1, 8]\n- You use SSIM but the relevant paper(s) are not cited (e.g. [7]).\n- Heading 3.1 \"Periphrl\"\n\n## Literature\n\n1. Do, M. T. H., & Yau, K.-W. (2010). Intrinsically Photosensitive Retinal Ganglion Cells. Physiol Rev, 90.\n\n1. Hoppe, D., & Rothkopf, C. A. (2019). Multi-step planning of eye movements in visual search. Scientific Reports, 9(1), 144. https://doi.org/10.1038/s41598-018-37536-0\n\n1. Kümmerer, M., & Bethge, M. (2021). State-of-the-Art in Human Scanpath Prediction (arXiv:2102.12239). arXiv. http://arxiv.org/abs/2102.12239\n\n1. Kümmerer, M., Bethge, M., & Wallis, T. S. A. (2022). DeepGaze III: Modeling free-viewing human scanpaths with deep learning. Journal of Vision, 22(5), 7. https://doi.org/10.1167/jov.22.5.7\n\n1. Rosenholtz, R. (2016). Capabilities and Limitations of Peripheral Vision. Annual Review of Vision Science, 2(1), 437–457. https://doi.org/10.1146/annurev-vision-082114-035733\n\n1. Watson, A. B. (2014). A formula for human retinal ganglion cell receptive field density as a function of visual field location. Journal of Vision, 14(7), 15. https://doi.org/10.1167/14.7.15\n\n1. Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2003). Multiscale structural similarity for image quality assessment. The Thirty-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, 1398–1402. https://doi.org/10.1109/ACSSC.2003.1292216\n\n1. Zele, A. J., Feigl, B., Adhikari, P., Maynard, M. L., & Cao, D. (2018). Melanopsin photoreception contributes to human visual detection, temporal and colour processing. Scientific Reports, 8(1), 3842. https://doi.org/10.1038/s41598-018-22197-w", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)