Emergent Properties of Foveated Perceptual Systems
Authors: Arturo Deza
Talia Konkle
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Hybrid Perceptual Systems
Foveation
Spatially-Adaptive Computation
Vision
Representation Learning

Abstract: The goal of this work is to characterize the representational impact that foveation operations have for machine vision systems, inspired by the foveated human visual system, which has higher acuity at the center of gaze and texture-like encoding in the periphery. To do so, we introduce models consisting of a first-stage \textit{fixed} image transform followed by a second-stage \textit{learnable} convolutional neural network, and we varied the first stage component. The primary model has a foveated-textural input stage, which we compare to a model with foveated-blurred input and a model with spatially-uniform blurred input (both matched for perceptual compression), and a final reference model with minimal input-based compression. We find that: 1) the foveated-texture model shows similar scene classification accuracy as the reference model despite its compressed input, with greater i.i.d. generalization than the other models; 2) the foveated-texture model has greater sensitivity to high-spatial frequency information and greater robustness to occlusion, w.r.t the comparison models; 3) both the foveated systems, show a stronger center image-bias relative to the spatially-uniform systems even with a weight sharing constraint. Critically, these results are preserved over different classical CNN architectures throughout their learning dynamics. Altogether, this suggests that foveation with peripheral texture-based computations yields an efficient, distinct, and robust representational format of scene information, and provides symbiotic computational insight into the representational consequences that texture-based peripheral encoding may have for processing in the human visual system, while also potentially inspiring the next generation of computer vision models via spatially-adaptive computation.

Emergent Properties of Foveated Perceptual Systems
Anonymous Author(s)
Affiliation Address email
Abstract
The goal of this work is to characterize the representational impact that foveation operations have for machine vision systems, inspired by the foveated human visual system, which has higher acuity at the center of gaze and texture-like encoding in the periphery. To do so, we introduce models consisting of first-stage fixed image transform followed by second-stage learnable convolutional neural network, and we varied the first stage component. The primary model has a foveated- textural input stage. which we compare model with foveated-blurre input and model with spatially-unifong compression), and a final reference model with minimal input-based compression. We find that: 1) the dnveated-texture model shows similar scene classification accuracy as the reference model despite its compressed input, with greater i.i.d. generalization than the other models; 2) the deveated-texturr model has greater sensitivity to high-spatial frequency information and greater robustness occlusion, w.r.t the comparison models; 3) both the foveated systems, show a stronger center image-bias relative to the spatially-unifomm systems even with a weight sharing constraint. Critically, these results are preserved over different classical CNN architectures throughout their learning dynamics. Altogether, this suggests that foveation with peripheral texture-based based computations yields efficient, distinct, and robust representational format of scene information, and provides symbiotic computational insight into the representational consequences that texture-basee peripheral encoding may have for processing in the human visual system, while also potentially inspiring the next generation of computer vision models via spatially- adaptive computation.
2
6
blurred input (both matched for perceptual
11
13
15
17
19
21
23
Introduction
24
25
In the human visual system, incoming light sampled with different resolution across the retina, a stark contrast to machines that perceive images at uniform resolution. One account for the nature of this foveated (spatially-varying constraints) (Land & Nilsson 2012 Eckstein 2011). e.g., there is only finite amount of retinal ganglion cells (RGC) that can relay information from the retina to the Lateral Geniculate Nucleus (LGN) constrained by the thickness of the optic nerve. Thus is "more efficient" have moveable high-acuity fovea, rather than a non- non-moveable uniform resolution retina when given limited number of photoreceptors suggested in Akbas Eckstein (2017). Machines, however do have such wiring/resource constraints and with their already proven success in computer vision (LeCun et al.) 2015) this raises the question if a foveated inductive bias necessary for vision at all.
27
array in humans is related purely to sensory efficiency (biophysical
28
29
30
31
32
33
34
35
However, it is also possible that foveation plays a functional role at the representational level, which may confer perceptual advantages as most computational approaches have mainly focused on saccade planning (Geisler et al., 2006 Mnnih et al. 20144 Elsayed et al.) 2019) Daucé et al. 2020). This idea has remained elusive in computer vision, but popular in vision science, and has been explored both psychophysically (Loschky et al. 2019) and computationally (Poggio et al. 2014
36
37
38
39
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

Image
Image Transfor:: fa(e)
Foveated (Texture-based) Image
Figure 1: A cartoon illustrating how a biologically ionspic foveated image (texture- based) is rendered resembling& a human visual metamer via the foveated fced-forward style transfer model of Deza et al.) (2019). Here, each receptive field is locally perturbed with noise in its latent space in the direction of their equivalent texture representation (blue arrows) resulting in visual crowding effects that warp the image locally in the periphery (Balas et al.] 2009 Freeman & Simoncelli] [001] Rooenholtz (2016). These effects are most noticeable far away from the navy dot which is the simulated center of gaze (foveal region) of an observer under certain viewing conditions.
40
2017, Han al. 2020). Other works that have suggested representational advantages of foveation include the work of Pramod et al. (2018), where blurring the image in the periphery gave an increase in object recognition performance of computer vision systems by reducing their false positive rate. In Wu et al. (2018)ss 's GistNet, directly introducing dual-stream
Cheung et al.
41
42
43
pathway in neural network boosted object detection performance via scene gist and contextual cueing. Relatedly, the most well known example of work that has directly shown the advantage of peripheral vision for scene processing in humans is Wang & Cottrell (2017)'s dual stream CNN that modelled the results of Larson & Loschky (2009) with log-polar transform and adaptive Gaussian blurring (RGC- convergence). Taken together, these studies present support for the idea that foveation has useful representational consequences for perceptual systems. Further, these computational examples have symbiotic implications for understanding biological vision, indicating what the functional advantages of foveation in humans may be, via functional advantages in machine vision systems.
44
45
46
47
48
50
51
52
Importantly, none of these studies introduce the notion of texture representation in the periphery- -a key property of peripheral computation as posed in Rosenholtz (2016). What functional consequences does this well-known rexture-based coding in the visual periphery have, if any, on the nature of later stage visual representation? Here we directly examine this question. Specifically, we introduce perceptual systems: as two-stage models that have an image transform stage followed by deep convolutional neural network. The primary model class of interest possesses first stage image transform that mimies texture- based foveation via visual crowding (Levi] 201T Pelli 2008 Doerig 2019b in the periphery shown in Figure T (ezaa et al. 2211)) rather than Gaussian blurring Wang & Cottrell 2017 |ramod et al., 2018 Malkin et al. 2020) or compression (Patncy ct al. 2016 Kaplanyan et al. 2219)). These rendered images capture image statistics akin to those preserved in human peripheral vision, and resembling texture computation at the stage of area V2, as
53
54
55
56
57
Pelli
58
59
et al.,
60
61
62
argued in Freeman & Simoncelli (2011); ( Rosenholtz 2016 Wallis et al. (2019).
63
64
Our strategy is thus to compare in terms of generalization, robustness and bias these foveation -exturre models to three other kinds of models. The first comparison model class baass-foveurte -oooeaatioonbblle models- uses the same spatially-vayying foveation operations but uses blur rather than texture based input. The second class- uniform-blur models uses blur operation uniformly over the input, with the level of blur set to match the perceptual compression rates of the foveation-texture nets. Finally, the last comparison model class is the reference, which has minimal distortion, and serves as perceptual upper bound from which to assess the impact of these different first-stage transforms.
65
66
67
68
70
71 Note that our approach is different from the one taken by Wang & Cottrell (201)) who have built
72 foveated models that fit results to human behavioural data like those of Larson & Loschky (2009).
73 Rather, our goal is to explore the emergent properties in CNNs with eattre-based foveation fovedtaattiooon on scene
74 representation compared to their controls agnostic to any behavioural data or expected outcome. Naturally, the results of our experimental paradigm is symbiotic as it can shed light into both
76 the importance of texture-based peripheral computation in humans, and could also suggest new inductive bias for advanced machine perception in scenes.

---

Input
Image Transform
CNN-backbone
Scene Classification
Algorishm: Fowested Rowatad-TTxxtuee Texture Transform: f.fe)
Given perturbation coeficients (a)
Note: fo(a) compptted V {ocs
Given euoodee-deeoee poiz(E, D)
Reference
Ju(a
Given the Stylo-Transter Operator:
AdaII(Style Content)
Given sot of pooling regions: (0,0)
s(o)
1: Define Architecture A (9,0)
2: T =(()
for each E
4: for each E
5. sample Noise AV
Qe. AdalNN(Ts, N)
A.(o)
T: Mo. as Qo, ss. (1 ar,,) No..s
Foveation
Viaual Crmnding
8: l' P(A))
Iopor: Imaago T, outpat Fovested Image /'
g(o)
Figure A. Two of the four perceptual systems: Reference (top row) and Foveation- Texture (bottom row), where each system receives image input, applies image transform (f(o)), which is then relayed to a CNN architecture (g(o)) for scene classification. Reference provides an undistorted baseline perceptual upper-bound, while Foveation- uses a visual crowding model that distorts the image with spatially-varyiae texture computation (shown on right) B. The algorithm of how the biologically inspired Foveation- Texture transform works which enables effects of visual crowding in the periphery (mainly steps 5-7).
Perceptual Systems
78
We define perceptual systems two-stage models with image transform (stage 1, f(o) R R RD), that is relayed to a deep convolutional neural network (stage 2, g(o) IR Rd.. Note that the first transform stage a fixed operation over the input image, while the second stage has learnable parameters. In general, the perceptual system S()), with retinal image input 7 RR defined as:
81
S(T) = ((f())
(1)
85
Such two-stage models have been growing in popularity, and the reasons these models are designed to not be fully end-to-end differentiable is mainly to force one type of computation into the first-stage of system such that the second-stage stage g(o) must figure out how to capitalize on such forced transformation and thus assess its f(o) representational consequences (See Figure 2). For example, Parthasarathy & Simoncelli (2020) successfully imposed VI-like computation in stage to explore the learned role of texture representation in later stages with self-supervised objective, and Dapello et al. (2020) found that fixing VI-like computation also at stage aided adversarial robustness. At a higher level, objective is similar where would like to force texture- peripheral coding mechanism (loosely inspired by V2: Ziemba et al. 2016) at the first stage to check if the perceptual system (now foveated) will learn pick-up this newly made representation through g(o) and make 'good' of it potentially shedding light on the functionality hypothesis for machines and humans.
84
85
87
89
91
88
94
2.1
Stage 1: Image Transform
95
To model the computations of a texture- based foveated visual system, we employed the model of Deza et al. (2019) (henceforth Foveuted- -eetueedd-Txxuree Texture Transform). This model is inspired by the metamer synthesis model Freeman Simoncelli (201T), where new images are rendered have locally matching texture statistics (Portilla & Simoncelli] 2000 Balas et al. 2009) in greater size pooling regions of the visual periphery with structural constraints. Analogously. the Deza et al. (2019) Foveation Transform uses a foveated feed-forward style transfer (Huang & Belongie 2017) network to latently perturb the image in the direction of its locally matched texture (see Figure L).. Altogether, IR + _ is convolutional auto-encoder that is non-foveated when the latent space is un- perturbed: fI( (T)( but foveated (cg) when the latent space is perturbed via localized style transfer: f.) (T),, for given encoder-decoder (E, D) pair.
96
97
99
101
103
105
Note that with proper calibration, the resulting distorted image can visual metamer (for a human), which is a carefully perturbed image perceptually indistinguishable from its reference image (Freeman Simoncelli] 2011] Rosenholtz et al.] 20022] Feather al.) 2019 Vacher et al., 2020). However, importantly in the present work, exaggerated the strength of these texture-driven distortions
107

---

A.
Trifrreno Computation
B. Rate- Distortion Optimization
Peserpttul Upper Bound Reference
Texturu Fovestion- Texture
Biar Foveation- Blur
Equalize Exposted Rates:
Uniform- Blur
B[R
l B[A
R SSIM
R : SSSM
for given Distortion: (D) for the whole transformed imzage
Matchee Reconroee
Equalize Expected Rates:
BB
SSIM (O.
SSIM (O.0)
for a given Distartion: (D) retinal coceutricity
fn(=) (.to)
folo) f*t>)
Figure A. Two perceptually matched-resoure controls foveation-Texture are introduced. Middle-Rige Right, orchid: uniform blurring emulating matched-resoure, non-foveated visual system (Uniform-Biie -Burr;;; Far-Right, seagreen: adaptive gaussian blurring (Foveation- -Blur) emulating a matched resource blur-based foveated system. B. A Rate-Distortion Optimization procedure is summarized where we find the hyper-parameters of the new matched-resource image transforms fL(o())) to Goveation-Textue via expected SSIM matching over the validation set.
(beyond the metameric boundary), of texturized peripheral input on later stage representations (e.g. following a similar approach as Dapello et al. (2020).. By having an extreme manipulation, reasoned this would accentuate the consequences of these distortions, making them more detectable in our subsequent experiments.
aim here is to understand the implications of this kind
110
112
2.2
Stage 2: Convolutional Neural Network backbone
The transformed images (stage I) are passed into standard convolutional neural network architecture. Here we tested two different base architectures: AlexNet (Krizhevsky et al. 2012), and ResNet18 (He) et al. 2016). The goal of running these experiments on two different hierarchically local architectures is to let us examine the consequences across all image transforms (with our main focus towards texture-basse foveation) that robust these different network architectures. Further, this CNN backbone (g: R + IR should not be viewed in the traditional way of an end-to-end input/output system where the input is the retinal image (7), and the output is one-hot vector encoding d-class- label in ddd Rather, the CNN (g) acts as a loose proxy of higher stages of visual processing (as it receives input from f), analogous to the 2-stage model of Lindsey et al. (2019).
115
117
119
120
122
123
2.3
Critical Manipulations: Foveated vs Non-Foveatee Perceptual Systems
124 Now, we can define the first two of the four perceptual systems that will perform 20-way scene categorization:
125
irreation-Texture receives an image input, applies the foveation transform f- (o), and relays it through the CNN g(o). Similarly, Reference performs non foveeted transform
126
fo(o), where images are sent through the same convolutional auto-encoder D(E of fo)),, but with the parameter that determines the degree of texture style transfer set to 0- producing an upper- bounded, compressed and non-foveated reference image then relayed through the CNN g(o). Both of these systems are depicted in Figure (A). As the foveation-texture model has less information from the input, relative to the reference networks, we next designed two further comparison models which have comparable amount of information after the input stage, but with different amounts of resources systems, our broad approach was to
127
129
131
133
blurring in the stage I operations. To create a Rate-Distortin (RD) optimization procedure (Ballé et al. 2016) to match information between the stage operations, given the SSIM (Wang et al. 2004) image quality assessment (IQA) metric.
135
136
Specifically, to create matched- Uniform-Bi Blur, we identified the standard deviation of the Gaussian blurring kernel (the 'distortion' D), such that we could render a perceptually resource- matched Gaussian blurred image w.r.t Reference that matches the perceptual transmission 'rate' R of Foveation- Texture via the SSIM perceptual metric (Wang et al. 2004). This procedure yields a
137
139
model class with uniform blur across the image, but with matched stage information content the

---

141
Foveation-Texeee And, to create matched-resource resource Foveation-Blir we carried our this same RD optimization pipeline per eccentricity ring (assuming homogeneity across pooling regions the same eccentricity), thus finding set of blurring coefficients that vary as a function of eccentricity. This
142
143
144
procedures yielded a different matched-resoure model class, this time with spatially-varying blur.
145
Figure 3 (B) summarizes our solution to this problem. Details of the RD Optimization are presented in Appendix [
146
147
Ultimately, it is important to note that the selection of the perceptual metric (SSIM in our case), plays a role in this optimization procedure, and sets the context in which we can call a network pessurce-mmachedd
148
149
We selected SSIM given its monotonic relationship of distortions to human perceptual judgements, symmetric upper-bounded nature, sensitivity to contrast, local structure and spatial frequency, and popularity in the Image Quality Assessment (IQA) community. However to anticipate any possible discrepancy the interpretability of our future results, we additionally computed the Mean Square Error (MSE), MS-SSIM, and 11 other IQA metrics as recently explored Ding et al. 2020) to compare all other image transforms to the Reference on the testing set. Our logic is the following: if the MSE is greater(t) for Foveation-Texture compared to Foveation- Blur and maiform-Blur, then the current distortion levels place eeveation-Texturr at a resource disadvanttaee" relative to the other transforms, and any interesting results would not only hold but also be strengthened. This same logic applies to the other IQA metrics contingent on their direction of greater distortion. Indeed, these patterns of results were evident across IQA metrics except those
150
151
152
153
154
156
158
160
tolerant to texture such as DISTS (Ding et al. 2020) as shown in Table and Appendix C
Imean Eiii))
SSIM (Mtched)
MS MS-SSIM (1)
MSE (1)
Motual Information (4)
NLPD (T)
DISTS
Reterence
1.0
0.54
Fovcation- Texture
0.58 1 0.11
0.20 1: 0.03
976. 78
522 22
1.40 1: 0.42
0.75 1 0.16
0.20
0.03
Unifomm-BBu Blur
0.57
1 0.15
0.36 0.00
458.67 2.7. 13
1.56 1: 0.58
0.40 0.09
0.36 1: 0.03
Fovesticee-Rlur
0.58 + 0.15
0.36 0.03
507.35 302.71
1.84 + 0.56
0.45 0.11
0.35 0.03
Table I: Comparing Image Transforms wri Reference. Arrows indicate direction of greater distortion.
3 Experiments
161
182
Altogether, the 4 previously introduced perceptual systems help us answer three key questions that we should have in mind throughout the rest of the paper: 1) Foveation- Texture vs Reference will tell us how a texture-based foveation mechanism will compare to its perceptual upper- bound shedding light into arguments about computa- tional efficiency. 2) Foveation
Image Dataset: Mini-Plaoos (2il-ray Sermn C'ttegariaation Tusk)
Reference
DoocationTexture Uniform Blur Foveation Blu
163
184
165
166
167
168
vs Foveation- -Buur will tell us if any potentially interesting pattern of results is due to the type/stage of foveation. This will help us measure the contributions of the adaptive texture coding
169
170
171
172
vs adaptive gaussian blurring: 3) Foveation-Texe Uniform- Blur will tell us how do these perceptual systems (one foveated, and the other one not) behave when allo- cated with a fixed number of perceptual resources under certain assumptions potentially shedding light on why biological organisms like humans have foveated texture- based computation in the visual field instead of uniform spatial processing like modern machines.
173
175
177
178
179
180
All previously introduced models were trained to perform 20-way scene categorization. Scene categories were selected from the Places2 dataset (Zhou et al.] 2017). and were re- re-partitioned into a new 4500 images per cate- gory for training. 250 per category for validation, and 250 per category for testing. The categories included were: aquarium, badlands, bedroom, bridge, campus, corridor, forest path, highway, hospital, industrial area, japanese garden, kitchen, mansion, mountain, ocean, office, restaurant, skyscraper, train interior, waterfall. Samples of these scenes coupled with their image transforms can be seen in Figure 4
Dataset:
(Only same classes shown Figers)
181
Figure 4: Five example images from the 20 scene categories are shown, after be- ing passed through the first stage of each perceptual system.
182
183
185
196
187
188
189
Networks: Training: Convolutional neural networks of the stage 2 of each perceptual system were trained which resulted in 40 image-transform based networks per architecture (Ale))))
5

---

LId Generalization
B.
Generalization
b.D.d Generalization
0.0.d Generalization
+ 10.50 B605
00 100%
2 1327922188
56.00 0. pug
S60 L.5%
22:02: 31.77 LIME
2 1.55
+ 10:33:226
22:4
12:33:22.700
38.11 2.37
2:334 1.555
2:1 54 LIRR
28.72 BAR
0 1337
BL16 1.37
$4-22 1.5
80
80
Top-1 Accuracy (%)
Top-1 Accuracy (%)
70
ResNet18 g(o)
AlexNet as g(o)
60
60
50
50
40
40
30
20
10
10
0
Poweette
Fowediion
Unfor
Fourstion
Teeurs
Bla
Sku
Rolure
Birr
Bixr
Figure 5: Scene Categorization Accuracy of AlexNet and ResNet18 as g(o). We observe the following: Foveation- Texture has greater i.i.d. generalization than other matched-resourg systems across both network architectures; Uniform- Blur's o.o.d generalization interacts with the architecture (performing worse for Re but highest for AlexNet); Foveation- maintains high o.o.d. generalization independent of network architecture. Confusion Matrices can be seen in Appendix [
191 10 Foveation- Texture, 10 Reference, 10 Uniform-Blur, 10 Foveation-Blur; totalling 80 trained networks to compute relevant error bars shown in all figures (standard deviations, not standard errors) and to reduce effects of randomness driven by the particular network initialization. All systems were paired such that their stage 2 architectures g(o) started with the same random weight initialization prior to training. Testing: The networks of each perceptual system were tested on the same type of image distribution they were trained on. Learning Dynamics: Available in Appendix H
193
195
196
197
3.1
Texture-based foveation provides greater i.i.d. generalization than Blur-based foveation
How well does the doveation-textuus stage classify scene images (i.i.d generalization) compared to the other matched-resource models that use blurring and the reference? The results can be seen in Figure 5 Each bars' height reflects overall accuracy for each of the 10 neural network backbone runs (g())) per system, with a square marker at the top indicating the i.i.d. accuracy. We found that Inneaaiio-Textunn had similar i.i.d. performance the Reference which is the the undistorted perceptual upper bound, and greater performance than both Uniform- Blur and Foveation-Blus Thus the compression induced by foveated-texture generally maintains scene category information.
199
201
202
203
204
205
We next performed a contrived experiment where we tested how well each perceptual system could classify the stage outputs of the other models. For example, showed set of foveated blurred images to a network trained on foveated texture images. This experiment is in essence a test of out-of -iit----dittibttionn (o.o.d.) generalization. The results of these tests are also shown in Figure 5 For each model, the classification accuracy for the inputs from the other stage images is indicated by the height of the different colored diamonds, where the color corresponds to the stage 1 operation.
207
209
210
211
This experiment yielded a rather complex set of patterns, that even differed depending on the architecture (AlexNet vs ResNer18 as g(o)). Generally, the iiveation-Texture model had a similar profile of generalization as the Reference model. However, the networks trained with different types of blur (Uniform-Bit & siveated-Blury in some cases showed very high o.o.d. generalization though once again this is contingent on g(o).
213
214
215
218
Unraveling the underlying causes to understand this last set of results sets the stage for our experiments the rest of this section. So far seems like Foveation-Textus has learned to properly capitalize the texture information in the periphery and still out- net-perform all other matched-
217
218
systems even if heavily penalized under several IQA metrics (Table highlighting the critical differences in texture
219
220
vs blur for scene processing. As for the interaction of Uniform- falforr--Blur with g(o), is is likely that the residual connections are counter-proce produttepr to o.o.d. generalization (or it has overfit). Interestingly,
221
humans have combination of texture and -aapsive-gaussian based peripheral computation (Ehinger Rosenholtz 2016), so future work should look into the effects of continual learning, joint-training or combined image transform (Texture Blur) to merge gains of both i.i.d and o.o.d generalization.
223

---

A1.
AlexNet as g(a)
ResNet18 as g(o)
Spatial Frequency Sensitivity
Spatial Frequency Sensitivity
High Pass Frequency
Low Pass Frequency
High Frequency
Low Pass Frequency
Top-1 Accuracy (%)
Top-I Accuracy (%)
80
30
40-
30
20
0.5
35
35
logg standard deviation Gaussian
long of star daad deviation Goussian
Jong of standard decation of Gaussian
Sunget stancard deviation Goussian
A2.
Reference
Foveation-Textui
Uniform-Blur
Faveation-Blur
Referance
Foveatioo-Texttre
Unitorm Blur
Foveation- Blur
Figure 6: Foveation DeveationTTexture has greater sensitivity to high pass spatial frequency filtered stimuli than the Reference (contingent on the architecture for g(o) - See AI). though both of these systems present notably higher sensitivity to high spatial frequencies than Uniform-Blut and Leveation-Blur This pattern is reversed for low pass frequency stimuli applied to both color and grayscale filtered images (Appendix K). Visualizations of the first convolutional layer of Alex and ResNet18 8 as g(o) (A..,B2.) shows strong similarities of learned filters despite -isttre-ditoortion for Foveation-Texxe to Reference preserving high spatial frequency Gabors; Uniform-Blu shows a strong predominance of low spatial frequency Gabors for AlexNet and low spatial frequency center-surround filters for Re and Foveation-Blur mixture of high-low spatial frequency tuned filters.
3.2
Texture-based foveated systems preserve greater high-spatial frequency sensitivity
226
We next examined whether the learned feature representations of these models are more reliant on low or high pass spatial frequency information. To do so, we filtered the testing image set at multiple levels to create both high pass and low pass frequency stimuli and assessed acene-classifican
227
228
performance over these images for all models, as shown in Figure 6 Low pass frequency stimuli were rendered by convolving a Gaussian filter of standard deviation
229
[0, 3, 7, 10, 15, 40] pixels on the foveation transform (fo, for fa. f-) outputs. Similarly, the high pass stimuli
computed by subtracting the reference image from its low pass filtered version with a = 3, 1.5, 1, 0.7, 0.55, 0.45, 0.4] pixels and adding a residual. These are the same values used in the experiments Geirhos et al.) (2019).
232
233
234
We found that Texxure and Reference trained networks were more sensitive to High Pass Frequency information, while Foveation-Blur and Uniform-Blur were selective to Low Pass Frequency stimuli. Although one may naively assume that this is an expected result as both Foveation- Blur and Uniform-BIi networks are exposed to a blurring procedure it is important to note that: 1) the foveal resolution has been preserved between Foveation- Texture and Foveation Blur (See Fig. 4 thus high spatial frequency sensitivity could have still predominated in Roveation-Blum but did not (though see Fig. 66 A2/B2 where these high pass Gabors are still learned, implying that higher layers in g(o) overshadow their computation); and 2) Foveation- -Txturr could have peripheral distortion, but this was not the case (likely due to the weight sharing constraint embedded in the
236
238
240
242
also learned to develop low spatial frequency sensitivity given the growding/texture-l/
243
244
CNN architecture Elsayed et al. 2020). Finally, the robustness to low-pass filtering of Foveation-Blu suggests that foveation via adaptive gaussian blurring may implicitly contribute scale-invariance as also shown in Poggio et al. (2014); Cheung et al. (2017); Han et al. (2020).
245
246
247
3.3
Texture-based foveation develops greater robustness occlusion
248
We next examined how all perceptual systems could classify scene information under conditions of visual field loss, either from left to right (left2right), top to bottom (top2bottom), center part of
7

---

Robustness to Occlusion
Left2Right
Tottom
Scotoma
Glaucoma
AlexNet as g
Top-1 Accuracy (96
70
Uniform-Bhl
Fovestion-
Fovearien-Blir
50
20
20
ResNet18 as g
Top-2 Accuracy (%)
80
s0
20
Occluded Area Ratio (/166
Occluded Area Ratio (116)
Occluded Area Ratio (/16)
Occluded Area Ratio (/16)
Figure Roveation-Textur Texture has greater robustness than both Foveation Beuur and Uniform-Bil Blur while roughly preserving a performance similarity to Reference (the upper bound) beyond the i..d. regime. The asymmetry in performance of the Scotoma vs Glaucoma conditions for foveated models also suggests they have learned to weigh spatial information differently in the fovea the periphery despite a weight sharing constraint imposed through g(c).
250
the image (scotoma), or the periphery (glaucoma). This manipulation lets us examine the degree which learned representations relying on different parts of the image classify scene categories. Critically, here we apply the occlusion after the stage operation. The results are shown in Figure 7
252
Overall found that, across all types of occlusion the Foveation-Textur modules have greater bustness to occlusion than both the Foveation- -Bur and Uniform- Blur models. Further, the Foveation- Texture models have nearly equivalent performance to the Reference. In contrast, both models with blurring, whether uniformly or a spatially-varyiin way, were far worse at classifying scenes under conditions of visual field loss. These results highlight that the texture-based information content captured by the foveation- texture nets preserves scene category content in dramatically different way than simple lower-resolution sampling perhaps using the texture-bias (Geirhos al. (019) their favor; as humans too use texture as their classification strategy for scenes (Renninger & Malik 2004).
254
256
258
259
280
In addition, the Foveation- Texture model is not overfitting. As recent work has suggested an Accuracy vs Robustness trade-off where networks trained to outperform under the i.i.d. generalization condition will do worse under other perceptual tasks mainly adversarial (Zhang al., 2019) did not observe such trade-off and a greater accuracy did not imply lower robustness to occlusion.
282
264
265
Foveated systems learn a stronger center image bias than non-foveated systems
possible that foveated systems weight visual information strongly in the foveal region than the 286
peripheral region as hinted by our occlusion results (the different rate of decay for the accuracy curves in the Scotoma and Glaucoma conditions). To resolve this question, we conducted an experiment where we created windowed cue-conffict stimuli where with one image category in the fovea, and another one in the periphery (all aligned with a different class systematically; ex: aquarium with badlands). We also had an additional condition where the conflicting cue was now square-like and uniformly and randomly paired with conflicting scene
268
re-rendered
set of testing images
270
272
class and more finely sampled. We then systematically varied the porea-eeriphery visual area ratios re-examined classification accuracy for both the foveal and peripheral scenes (Figure83
274
We found that the Prreation-Texture and Foveation-BB Blu transform imposed the networks g(o) to learn to weigh information the center the image stronger than Reference Uniform-Blur for scene categorization. A qualitative way of seeing this foveal-bias is by checking the foveal/peripheral
276

---

AlexNet as g(o)
ResNet18 g(o)
Window Cue-Conflict
Square Cue-Conflict
Window Cue-Conflict
Square Cue-Conffict
(+) Perpheral Scene Accuraty (hg
8 00.44
80 10.99
00 0.42
50 |1.39
0:35
0 0.04
0:535 0.33
8 0.42
10 00:35
0:433
@ 0:35
20 0.400
0.33
50 0.306
0.30
C2
R2
06
04
B2
06
Rato Corrul trage Aroa with Poveal Coss
Roo0 of Cont Iroge Area with hovost ess
hot certall trogs Prio with Foveal Gass Kolo Cerbu mrogs Areo w(h PONOa Class
Figure 8: Foveated Perceptual Systems- independent of the computation type (Foveation-Textuu) Foveation-Blur show stronger biases classify hybrid scenes with the foveal region; a result also observed in humans (Larson & Loschky. 2009).
278
ratio where these two accuracy lines cross. The more leftward the cross-over point (@), the higher the foveal bias (highlighted through the vertical bars). This result was unexpected as we initially predicted that g()) would weigh the peripheral information stronger as it has been implicitly regularized through a distortion. However this was not the case and our findings are similar to Wang & Cottrell (2017) who showed this foveal bias on foveated system with adaptive blur with a dual-stream neural network. Thus, these results indicate that the sputially varying computation from center to periphery is mainly responsible for the development of center image bias even with weight sharing constraint. Furthermore, it is possible that one of the functions of any spatially-varying in the visual field is to enforce the perceptual system to attend the foveal region avoiding the shortcut of learning to attend the entire visual field unnecessary (Geirhos et al. 2020).
279
280
281
285
285
coding mechanisms
287
Discussion
288
289
The present work was designed to probe the impact of foveated texture-based input representations in machine vision systems. To do this we specifically compared the learned perceptual signatures in the second-stage of visual processing across a set of of networks trained on other image transforms. We found that when comparing foveation-Textue to their matched-resource models that differed in (foveated w/ adaptive gaussian blur) and Uniform-Blu (non-foveated w/ uniform blur) that peripheral texture encoding did lead to specific representational signatures, particularly greater i.idd generalization, preservation of high-spatial frequency sensitivity, and ro- bustness to occlusion even high its perceptual upper bound (Reference). We also found that foveation (in general) seems to induce a focusing mechanism, servicing the foveal/central region whereas neither a perceptually upper-bounded system (Reference) or non-foveated compressed system (Uniform-Bil -lurr) did not develop as strongly.
290
291
293
computation: Foveation-Bie
295
297
299
300
The particular consequences of our foveation stage raises interesting future directions about what computational advantages could arise when trained on object categorization (Pramod et al. 2018)
301
coupled with eye-movements (Akbas& & Eckstein) 2017 Deza et al.) 2017), as objects are typically centered in view and have different hierarchical/composttion
303
priors than scenes (Zhou et al. (0014
Deza et al. (2020) in addition to different processing mechanisms (Renninger & Malik 2004);
305
Ehinger& & Rosenholtz (2016)). We are currently exploring the impact of these foveated texture- based
representational signatures on shape texture bias for object recognition similar to Geirhos et al.
307
2019) and Hermann et al. (2020), and assessing their interaction with scene representation.
308
Further, a future direction is investigating the effects of texture-based foveation to adversarial robustness. Motivated by the recent work of Dapelll et al. (2020) which has shown promise of adversarial robustness via enforcing stochasticity and V1-like computation by obeying the Nyquist sampling frequency these filters w.r.t the image (Serre et al. 2007) in addition to a natural gamut of orientations and frequencies as studied in De Valois et al. (1982). J it raises the question of how much we can further push for robustness in hybrid perceptual systems like these, drawing on even more biological mechanisms. Works such as [uo et al. (2015) and recently Reddy et al. 2020); Kiritani &
309
311
313
315
Ono (2020) have already taken steps in this direction by coupling fixations with a ppatiall-vvarring retina. However, the representational impact of texture-based based foveation adversarial robustness,
317
and its symbiotic implication for human vision still remains an open question.

---

References
318
319
Akbas, and Eckstein, M. Object detection through search with a foveated visual system. PLoS computational biology, 13(10)=e005743, 2017.
Balas, B., Nakano, L.,, and Rosenholtz, R. A summary-statistic representation in peripheral vision explains visual crowding. Journal of vision, 912):13-13, 2009.
322
323
Ballé, J. Laparra, V., and Simoncelli, E. P. End-to-end optimized image compression. arXiv preprint arXiv: 1611017711 2016.
324
325
Cheung, B., Weiss, E., and Olshausen, B. Emergence of foveal image sampling from learning to
328
attend visual scenes. International Conference on Learning Representations (ICLR), 2017.
327
Dapello, J., Marques, T., Schrimpf, M., Geiger, F., Cox, D. D., and DiCarlo, J. J. Simulating primary visual cortex at the front of cnns improves robustness to image perturbations. BioRxiv,
328
329
2020.
330
Daucé, E., Albiges, P., and Perrinet, L. U. A dual foveal-peripheral visual processing model implements efficient saccade selection. Journal of Vision, 20(8):22-22, 2020.
331
332
De Valois, R. L., Yund, E. W., and Hepler, N. The orientation and direction selectivity of cells in macaque visual cortex. Vision research, 22(5) 1982.
338
Deza, A. and Eckstein, M. Can peripheral representations o improve clutter metrics on complex scenes? In Advances in Neural Information Processing Systems, pp. 2847-2855. 2016.
335
336
Deza, A., Peters, R., Taylor, G. S., Surana, A., and Eckstein, M. P. Attention allocation aid for visual search. In Proceedings of the 2017 CHI Conference Human Factors in Computing Systems, pp. 220-231, 2017.
338
339
Deza, A., Jonnalagadda, A., and Eckstein, M. Towards metamerism via foveated style transfer. In International Conference on Learning Representations, 2019. URL httpps:/ .eet Forum2idd-J2bb220cF
341
342
Deza, A., Liao, Q., Banburski, A., and Poggio, Hierarchically local tasks and deep convolutional networks. CBMM Memo, 2020.
344
Ding, K., Ma, K., Wang, S., and Simoncelli, E. Image quality assessment: Unifying structure and texture similarity, IEEE Trunsactions on Puttern Analysis and Machine Intelligence, 2020.
345
346
Ding, K., Ma, K., Wang, S., and Simoncelli, E. P. Comparison of Image Quality Models for Optimization of Image Processing Systems. arXiv e-prints, art. arXiv:2005.01333 May 2020.
347
348
Doerig, A., Bornet, A., Choung, O. H., and Herzog, M. H. Crowding reveals fundamental differences in local vs. global processing in humans and machines. bioRxiv, 2019a. doi: 10. 1011/44444
349
350
URL https: http://wwwwbbbboiiooo/ooo biorxiv. org/ /
351
Doerig, A., Bornet, A., Rosenholtz, R., Francis, G., Clarke, A. M., and Herzog, M. H. Beyond bouma's 's window: How to explain global aspects of crowding? PLoS computational biology, 15
352
353
(0):e000 2019b.
354
Eckstein, M. P. Visual search: A retrospective. Journal of vision, 1155):1-14, 2011.
355
Eckstein, M. P., Koehler, K., Welbourne, L. E., and Akbas, E. Humans, but not deep neural networks, often miss giant targets in scenes. Current Biology, 27(18):287722822 2017.
356
Ehinger, K. A. and Rosenholtz, R. A general account of peripheral encoding also predicts scene perception performance. Journal of Vision, 16(2):13-33, 2016.
358
359
Elsayed, G., Kornblith, S., and Le, Q. V. Saccader: Improving accuracy of hard attention models for vision. In Advances in Neural Information Processing Systems, pp. 700-712, 2019.
Elsayed, G., Ramachandran, P., Shlens, J., and Kornblith, S. Revisiting spatial invariance with
362
low-rank local connectivity. International Conference on Machine Learning, pp. 2868-2879 PMLR, 2020.
10

---

384
Feather, J., Durango, A., Gonzalez, R., and McDermott, J. Metamers of neural networks reveal diver-
365
gence from human perceptual systems. In Wallach, H., Larochelle, H., Beygelzimer, A., d' d'Alché-
Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems
367
32. pp. 10078-10089 Curran Associates, Inc., 2019. URL httpt///:://paperrs wis. httpppprrr.iipssccpppperr.e
368
1198-metamers-- of -eteral--nttorrs--eeae-------- pdf
human- hww.an--eerrepppualssyeeeeeee
369
370
Freeman. J. and Simoncelli, E. Metamers of the ventral stream. Nature neuroscience, 14(9):
371
1195-1201, 2011.
372
Fridman, I.,, Jenik, B., Keshvari, S., Reimer, B., Zetzsche, C., and Rosenholtz, R. Sideeye: A genera- tive neural network based simulator of human peripheral vision. arXiv preprint arXiv: 7706 04568, 2017.
374
Gatys, L. A.,, Ecker, A. S., and Bethge, M. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2414-2423, 2016.
376
378
Geirhos, R., Temme, C. R., Rauber, J., Schüttt, H. H., Bethge, M., and Wichmann, F. Generalisation in humans and deep neural networks. In Advances in Neural Information Processing Systems, pp.
379
380
3538-7550 2018.
Geirhos, R.. Rubisch, P., Michaelis, C., Bethge, M.. Wichmann, F. A., and Brendel, W. Imagenet- trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019. URL net//forum?.ddByye8jjjok.
382
383
384
385
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. Shortcut learning in deep neural networks. arXiv preprint arXiv:2004.07780, 2020.
387
Geisler, W. S. and Perry, J. Real-time foveated multiresolution system for low-bandwidth video communication. In Human vision and electronic imaging III, volume 3299, pp. 305. Interna- tional Society for Optics and Photonics, 1998.
389
390
Geisler, W. S., Perry, J. S., and Najemnik. J. Visual search: The role of peripheral information measured using gaze-contingent displays. Journal of Vision, 6(9):111, 2006.
381
382
Han, Y., Roig, G., Geiger, G., and Poggio, T. Scale and translation-invariance for novel objects in human vision. Scientific Reports, 10(1):1-13, 2020.
383
394
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
396 Hermann, K. L., Chen, T., and Kornblith, The origins and prevalence of texture bias in convolutional neural networks. Neural Information Processing Systems, 2020.
398
Huang, X. and Belongie, S. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1501-1510, 2017.
399
Kaplanyan, A. S., Sochenov, A., Leinkühler, T., Okunev, M., Goodall, T., and Rufo, G. Deepfovea: neural reconstruction for foveated rendering and video compression using learned statistics of natural videos. ACM Transactions Graphics (TOG), 38(6):1-13, 2019.
401
403
Kiritani, T. and Ono, K. Recurrent attention model with log-polar mapping is robust against adversarial attacks. arXiv preprint arXiv:2002.05388, 2020.
405
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
406
Land, M. F. and Nilsson, D.-E. Animal eyes. Oxford University Press, 2012.
408
Laparra, V., Ballé, J., Berardino, A., and Simoncelli, F. P. Perceptual image quality assessment using normalized laplacian pyramid. Electronic Imaging, 2016(16):1-6,, 2016.

---

410
Larson, A. M. and Loschky. L. C. The contributions of central versus peripheral vision to scene gist
411
recognition. Journal of Vision, 9(10),6 6, 2009.
412
Larson, E. C. and Chandler, D. M. Most apparent distortion: full-reference image quality assessment and the role of strategy. Journal of electronic imaging, 19(1):01 1006, 2010.
414
LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature, 521 2015.
415
Levi, D. M. Visual crowding. Current Biology, 21(18):R678 R678, 2011.
416
Lindsey, J., Ocko, S. A., Ganguli, S., and Deny, S. The effects neural resource constraints on
early visual representations. In International Conference Learning Representations, 2019. URL http:////ppppeeeeeeeeeeerorrrrre
418
419
Loschky, C., Szaffarezyk, S., Beugnet, C., Young, M. E., and Boucart, M. The contributions of central and peripheral vision to scene-gist recognition with 180 visual field. Journal of Vision, 19 15) 2019.
421
422
Luo, Y., Boix, X., Roig, G., Poggio, T., and Zhao, Q. Foveation- based mechanisms alleviate adversarial examples. arXiv preprini arXiv: 1ttl:::511.0999 2015.
424
Malkin, E., Deza, A., and tomaso poggio. [CUDA]-optimizee real-time rendering of a foveated
425
visual system. In NeurlPS 2020 Workshop SVRHM, 2020. URL http:::// net/ Forun?id=2MsqUuana
426
427
Mnih, V., Heess, N., Graves, A., et al. Recurrent models of visual attention. In Advances in neural
428
information processing systems, pp. 2204-2212, 2014.
Parthasarathy, N. and Simoncelli, E. Self-supervised learning of biologically-inspired visual texture model. arXiv preprini arXiv:2006. 16976, 2020.
430
431
Patney, A., Salvi, M., Kim, J., Kaplanyan, A., Wyman, C., Benty, N., Luebke, D., and Lefohn, A.
Towards foveated rendering for grae-tracked virtual reality. ACM Transactions Graphics (TOG), 35(6):179, 2016.
433
434
Pelli, D. G. Crowding: A cortical constraint on object recognition. Current opinion in neurobiology. 18(4):445-455 2008
Poggio, T., Mutch, J., and Isik, L. Computational role of eccentricity dependent cortical magnification. arXiv preprint arXiv: 1406. 1770, 2014.
437
438
Portilla, J. and Simoncelli, E. parametric texture model based on joint statisties of complex
wavelet coefficients. International journal of computer vision, 40(1):49-70, 2000.
440
Pramod, R. T., Katti, H., and Arun, S. P. Human peripheral blur is optimal for object recognition.
441
arXiv preprint arXiv: 0188070.477 2018
442
Reddy, M. V., Banburski, A., Pant, N., and Poggio, T. Biologically inspired mechanisms for adversarial robustness. arXiv preprint arXiv:2006. 16427, 2020.
443
444
Renninger, I. W. and Malik, J. When is scene identification just texture recognition? Vision research, 44(19)13011-211, 2004.
445
Rosenholtz, R. Capabilities and limitations of peripheral vision. Annual Review of Vision Science, 2:
447
437-457, 2016.
448
Rosenholtz, R., Huang, J., Raj, A., Balas, B. J., and Ilie, L. A summary statistic representation in peripheral vision explains visual search. Journal of vision, 12(4):14-144 14, 2012.
Russakovsky, O., Deng, J., Su, H.. Krause, J.. Satheesh, S., Ma. S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition challenge. International jouraaaaannoouunaall of computer vision, 115(3):21 1--222, 2015.
451
Serre, T., Wolf, L., Bileschi, S., Riesenhuber, M., and Poggio, T. Robust object recognition with like mechanisms. IEEE transactions on pattern analysis and machine intelligence, 29(3): 411-426, 2007.
454
12

---

458
Sheikh, H. R. and Bovik, A. C. Image information and visual quality. IEEE Transactions on image processing. 15(2):430 444, 2006.
457
458
Shumikhin, M. M. A. Quantitative measures of crowding susceptibility in peripheral vision for large
datasets. PhD thesis, Massachusetts Institute of Technology. 2020.
Vacher, J., Davila, A., Kohn, A., and Coen-Cagli, R. Texture interpolation for probing visual perception. Advances in Neural Information Processing Systems, 33, 2020.
461
462
Wallis, T. S., Funke, C. M., Ecker, A. S., Gatys, L. A., Wichmann, F. A., and Bethge, M. Image content is more important than bouma's 's law for scene metamers. eLife, 8:c42512, 2019.
463
484
Wallis, T. A., Funke, C. M., Ecker, A. S., Gatys, L. A.. Wichmann, A., and Bethge, M. A
485
parametric texture model based on deep convolutional features closely matches texture appearance for humans. Journal of Vision, 17(12), Oct 2017. doi: 10.1 10.1167/17.12.5, URL http::/doo org/
466
10. 1261161777.77 1 12.5
Wang, P. and Cottrell, G. W. Central and peripheral vision for scene recognition: A neurocomputa- tional modeling exploration. Journal of vision, 17(4):9-9) 2017.
489
470
Wang, Z. and Simoncelli, E. P. Translation insensitive image similarity in complex wavelet domain. In CiccceddingssCAASSPPP Processing, 2005., volume 2, pp. ii-53.. IEEE, 2005.
IEEE International Conference on Acoustics, Speech, and Signal
472
473
Wang. Z. Simoncelli, E. P., and Bovik, A. C. Multiscale structural similarity for image quality assessment. In The Seventhh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pp. 1402. leee, 2003.
474
475
476
Wang. Z., Bovik, A. C.,, Sheikh, H. R., and Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4)600-661,, 2004.
477
478
Wu, K., Wu, E., and Kreiman, G. Learning scene gist with convolutional neural networks to improve object recognition. In 2018 52nd Annual Conference on Information Sciences and Systems (CISS), pp. 1-6. IEEE, 2018.
480
481
Xue, W., Zhang, I.., Mou, X., and Bovik, A. C. Gradient magnitude similarity deviation: A highly efficient perceptual image quality index. IEEE Transactions on Image Processing, 23(2):684 695, 2013.
483
484
Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L.,, and Jordan, M. Theoretically principled trade- off between robustness and accuracy. In International Conference on Machine Learning, pp. 7482-7482. PMIR, 2019.
485
Zhang, L., Zhang, L., Mou, X., and Zhang, D. Fsim: A feature similarity index for image quality assessment. IEEE transactions on Image Processing, 20(8):2378-2338 2011.
488
489
Zhang, L., Shen, Y., and Li, H. Vsi: A visual saliency-induced index for perceptual image quality assessment. IEEE Transactions Image processing, 23(10):4270 12811 2014.
491
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
492
493
494
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. Object detectors emerge in deep scene cnns, 2014.
Zhou, B., Lapedriza, A.. Khosla, A., Oliva, A.. and Torralba, A. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence. 40(6): 1464, 2017.
497
489
Ziemba, C. M., Freeman, J., Movshon, J. A., and Simoncelli, E. P. Selectivity and tolerance for visual texture in macaque v2. Proceedings of the National Academy of Sciences, 113((2):E3110- 144-EE399,, 2016.
500
13

---

Checklist
502
1. For all authors...
504
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We have focused our experiments on implementing a two-stage model that has a texture-based foveation transform and compared it to reference model (a perceptual upper bound), and two matched resource systems: one foveated with blur and another one uniformly blurred.
505
506
507
(b) Did you describe the limitations of your work? [Yes] At the end of each Experiments Sub-Section we provide a mini-discussion of our work and how it fits or does not fit the literature. Mainly we provide limitations in the Discussion at the end (See Section 4
510
512
(c) Did you discuss any potential negative societal impacts of your work? [No] To our knowledge, there are none.
513
514
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
515
2. If you are including theoretical results...
517
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We include only one supplementary theoretical result and proof in the AppendixB
518
519
(b) Did you include complete proofs of all theoretical results? [Yes] See above.
520
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] See Supple- mentary Material (that provides access to a URL)
622
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] These are reported brielfy in Section 3 and in more detail through-out the Appendix.
525
528
527
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] All experiments were ran with paired initial noise seeds to control for matched initial conditions derived from SGD (though the order in which the networks were exposed to images was different.. All errorbars report standard deviation, and these can be seen throughout Sectionss.233.31 3.3
528
$29
530
531
532
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] These are specified in the Appendix.
534
535
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use re-partition of the Places2 Dataset which is cited.
537
538
(b) Did you mention the license of the assets? [No] Given that to our knowledge the Places2 dataset is widely known and free use.
539
540
(c) Did you include any new assets either in the supplemental material or as a URL? [No] As everything in the Supplementary Material/URL. has been created/derived by us.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We did not run any experiments with humans.
543
544
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] We did not run any experiments with humans, and the scene classes we used were all publicly known and non-offensive places: e.g. ocean.
545
546
547
548
5. If you used crowdsourcing or conducted research with human subjects...
549
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] No human subjects were used.
550
551
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] No human subjects were used.
552
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] No human subjects were used.
554
14


Reviews:
[{"review_id": "sQN1pRyVRlP", "reviewer": "Reviewer_qxyi", "summary": "The authors address hypotheses about the nature of peripheral biological vision, specifically the Foveation-Texture theory stating that peripheral representations in primate vision are not simply blurred (subsampled) but rather extract more informative texture statistics. The empirical study comparing the proposed model vs several baseline model corroborates their proposal.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work addresses an important question: Is foveated vision an artifact of constrained resources or does it endow primate vision with an advantage in terms of learning more efficient representations?\nTheir empirical studies corroborate the notion that texture-foveation mechanism offers greater generalization, preserves more high spatial frequency information in periphery and develops greater robustness to occlusion than the blur-based model of peripheral representations.\nThe paper is well written and the claims are convincingly supported by empirical tests. \nHowever, the line of argumentation seems a bit confounded: while the authors state in the introduction that they focus on representational consequences of foveation, they are less interested in general such consequences of any foveation model as compared to non-foveated alternatives, but rather differences between specifically texture-foveation model vs other foveation models. It would be interesting to also address a model where the texture representation is used without the foveation model for testing these factors independently.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors of the study seems to have made a peculiar choice that might be undermining their conclusions: the first (texture processing) system that imitates foveation (and texture-level processing) actually acts as an encoder-decoder that generates processed images for the second stage, implemented as a standard CNN. However, a  more natural for both ML and for modeling a biological vision system would be to actually use the learnt latent codes of the autoencoder as the input to the CNN. So it would be interested to see how (if at all) the reaults would be affected by such processing pipeline.\n\nImpact: better understanding of human vision with implications for medicine and novel inductive biases for computer vision models.", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "7rHpk_WVhYV", "reviewer": "Reviewer_ThVH", "summary": "The paper aims to explore the effects on representation that emerge when a CNN is equipped with a foveated visual system. The authors build a model whereby images are artificially rendered into a form that captures foveation which are then fed into standard CNN architectures. Two different types of foveation are considered: one where the periphery is blurred (greater blurring as a function of eccentricity), and another where texture statistics are preserved over local regions which increase in size as a function of eccentricity.  These are compared against each other and two baselines:\n\n- Reference: images are transformed by pushing them through a convolutional AE\n- Foveation-texture: images are transformed by pushing them through a convolutional AE with a foveated texture style transfer\n- Uniform-blur: each image is blurred uniformly, with the amount of blurring set such that the SSIM between the blurred image the its corresponding reference was the same as the SSIM between the corresponding foveation-texture image and reference.\n- Foveation-blur: as above, but with blur additionally increasing with eccentricity\n\nClearly the reference images contain more information than the foveation-texture images; the SSIM matching on the is done to resource match the blurred images for comparison to foveation-texture, and the reference serves as an upper bound. Experiments are performed on a scene classification task. ", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n-----------\n\nThe paper is original in the sense that it's trying to give a new understanding of the potential benefits of a foveated visual system. I believe it is adequately cited and is complementary in its aims to prior work.\n\nQuality\n-------\n\nClearly a lot of work has gone into this paper (just from the the shear number of experiments performed for example), however I have a number of questions/issues which I'll describe below. It is of course possible that I've misinterpreted some parts of the paper, and if so, I look forward to a discussion with the authors during that phase of the review process.\n\n__The model:__\n\nTwo-stage models like the one proposed (and e.g. Lindsay (2019) as mentioned, but also recent work looking at pre-filtering the network input with Gabor filters or similar) have become popular ways of exploring how different inductive biases affect representations in artificial visual systems. To the best of my knowledge however, all the prior work has involved using an input that is matched with the inductive biases built into the network - that is the spatial resolution of the input is constant and thus matched to the convolutions that follow. In this work, the foveated networks effectively vary spatial resolution across spatial position, so it is not in anyway clear that applying fixed convolutions (with respect to the space between taps) across this input in anyway makes sense. Have the authors considered this? Why wasn't an alternative (perhaps multiple streams of fixed spatial resolution in the 1st stage? or maybe not using shared weights? or...).\n\nWith respect to the resource matching, it isn't clear to me why matching using a perceptual metric is the correct approach. Surely the aim should be to match the amount of information that is transmitted to the second stage? in which case why wouldn't you e.g. match based on entropy?\n\n\n__The results:__\n\nThe authors have chosen to do their experiments with a variant of the Places2 dataset for a scene categorization task, but there isn't really any discussion of why this was chosen in the paper, or of any of the potential pitfalls of using this dataset in the evaluation that is performed. By its very nature a scene categorization task requires information from across the entire image, at least in expectation over the entire dataset; for example, many (the majority?) of the images have very human biases which affect their overall composition - e.g.:\n\n- photos with a person in the foreground, often near the center of the image, where the presence of a person is unrelated to the scene (e.g. ocean)\n- images where if you relied just on the center you would be unlikely to predict the 'correct' scene (the badlands image in the paper is a good representative example)\n\nThese factors need to be considered when analysing results of experiments, because they will inherently cause a bias. In this case before doing any experiments, one might reasonably expect that if the periphery of the image is corrupted in some way performance would be lower. Further, because early work on scene classification (e.g. using things like bags of visual words with quantised dense SIFT descriptors, which were basically capturing local texture) actually worked rather well on these tasks, one would inherently expect that texture is going to be an important factor. With this in mind, I question if many of the results in the paper actually tell you more about the nature of the data rather than about foveation?\n\nI've written comments/questions on each experimental section below:\n\n_\"Texture-based foveation provides greater i.i.d generalisation than Blur-based foveation\"_\n\n- The above claim could possibly be inferred from the results, but it could just be that you've shown that the task you've chosen is more easily solved with high-frequency/texture information as I suggested previously? Do the differences between uniform and foviated -blur just indicate that for this task there is _relatively more_ information to solve the task in the periphery (which shouldn't really be surprising given the nature of the dataset)?\n- The claim that foveation-texture has \"similar i.i.d performance\" to reference seems like something of an overstatement; it's closer than blur, but still seems worse (particularly for the resnet) unless I'm missing something?\n- Following on, isn't it obvious that texture is likely to be more informative given what is already known about this type of dataset?\n- The reference images applied to foveated-texture showing good generalisation seems intuitive, however the uniform blur differences across networks are odd. When I looked at this I the question that arose is \"why is alexnet so good (particularly w.r.t reference - you'd expect foviated-blur to be better)?\"; the authors clearly thought that problem was instead with ResNet (which I'd argue has more intuitive performance with respect to the ordering of each input type) - I wonder if the authors could elaborate on this?\n\n_\"Texture-based foveated systems preserve greater high-spatial frequency sensitivity\"_ \n\n- I don't understand the argument in note 1: yes, the foveal bit is the same, but the texture model clearly has more high frequencies, which for this task are more likely to be important. Point 2 again just suggests that higher frequencies are more important for this task.\n\n_\"Texture-based foveation develops greater robustness to occlusion\"_ \n\n- So the takeaway is again that the solution to the task is dominated by higher frequency information, and clearly there is relevant high-frequency information in the periphery that helps significantly? Again, given the data is this really surprising?\n\n_\"Foveated systems learn a stronger center image bias than non-foveated systems\"_\n\n- I'm completely failing to understand what is surprising about this (or why the authors hypothesised it would be different); surely any learning system given a mixture of true and slightly corrupted data is going to have a preference for weighting the uncorrupted data higher?\n\n\nClarity\n-------\n\nFrom a technical perspective I believe enough information is given to be able to reproduce the experiments if one were minded. I did however struggle with understanding the motivation of the work (largely with respect to the model choices as outlined above). I also felt that the introduction was not sufficiently self-contained - for example the reader needs to be fully aware of the contents Rosenholtz (2016) to really understand some of the specific motivation for the _foveation-texture_ model (I had to read it; perhaps it is \"well known\" in the core vision sciences community, but this is a paper submitted to NeurIPS where the audience is much broader (particularly with respect to people in the computer vision community, as well as those more generally interested in understanding how different biases affect representation). I would suggest that this bit of the paper needs rewriting to make it much more self contained.\n\n\nSignificance\n------------\n\nCommenting on the significance is tough: having read the paper, do I know things that I didn't before? yes. Are they useful? that's completely unclear to me. As I've outlined above it's not clear to me that the model itself makes sense, or that the results actually tell us anything other than that the data used to train/evaluate the models had an inherent 'human' bias which can be exploited by foveation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are not adequately addressed as I've described above. It's probably fair to say that there are no major potential negative societal impacts, although I'd note that the energy cost in training all of these models must have been quite large (in terms of CO2 equivalent).  \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "1QNu4-FgNi0", "reviewer": "Reviewer_89Wr", "summary": "In this manuscript the authors test how DNNs trained on classifying scene categories react to texture based foveation, which they compare to undisturbed images, a foveated blur and a uniform blur matched in their SSIM to the original image. In overall accuracy the texture model performs close to the performance of the undistorted images and clearly better than the other two distortions. Going beyond accuracy the authors then test the effects of the input distortions on processing filtered images, occlusion, and cue conflicts between center and periphery.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I am torn on this one. On the one hand, the authors did a lot of work and I don’t have many technical issues with their work. On the other hand, I do not see at all what the relevance of the performed experiments is or what could be learned from them. The texture based foveation the authors apply appears quite different from the way we think the human visual system implements this, which limits any interpretation towards human comparisons and the observed effects of the foveation seem rather trivial. This leaves me with the question: yes, but why?\n\nTo elaborate on the human comparison problem: This kind of peripheral texturization of images is a method for testing human peripheral perception, which is thought not to differentiate these texturized images from each other due to computing summary statistics at midlevel processing steps. Using such distorted images as input to a non-foveated processing system really does not capture the human foveated processing well in my opinion. Also this implementation provides none of the advantages of foveated processing like reduced computational power requirements for the periphery.\n\nAnd on the observations not being particularly interesting:\n\nFigure 6: Networks trained with blurred inputs react more to lower frequencies. Although, the authors dismiss this as a naive idea (in 3.2), I still find this unsurprising. Even if some higher frequency information in available for the foveated blur, it certainly sees much more low frequency content.\n\nFigure 7: In principle, better performance for occlusions is an independent observation, but after observing a generally better performance of the Foveation-texture model this mostly looks like this advantage is simply preserved across occlusions.\n\nFigure 8: Models trained with more information in the center weigh central information slightly stronger. That is definitely expected and if anything the effect is surprisingly small due to the non-foveated processing in the models.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not comment on their limitations and societal impact. Besides the sampling biases of the used databases I don’t think there are important societal impacts associated with the content of this paper though.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received one positive review and two negative reviews (including one borderline reject). The positive reviewer did not engage in follow-up discussion which is based only on the two negative reviewers. There seems to have been some difficulty in communicating between the reviewers and the authors. As they stated in their final comments, I think the authors are right "perhaps there is a confusion on the goal (rather than the method) of our paper?". I agree that the reviewers understand what the methods is. However, it is pretty clear that they do not see the benefit of doing so either from an ML or neuroscience perspective. Even the positive reviewer calls the validity of the approach into question. Unfortunately it seems that no amount of revision will address this issue. For all these reasons, the AC recommends this paper being rejected.
Decision: Reject