Emergent Properties of Foveated Perceptual Systems
Authors: Arturo Deza
Talia Konkle
Venue:  (2021)
Keywords: Hybrid Perceptual Systems
Foveation
Visual Crowding
Texture
Two-stage models

Abstract: We introduce foveated perceptual systems -- a hybrid architecture inspired by human vision, to explore the role of a \textit{texture-based} foveation stage on the nature and robustness of subsequently learned visual representation in machines. Specifically, these two-stage perceptual systems first foveate an image, inducing a texture-like encoding of peripheral information -- mimicking the effects of \textit{visual crowding} --  which is then relayed through a convolutional neural network (CNN) trained to perform scene categorization. We find that these foveated perceptual systems learn a visual representation that is \textit{distinct} from their non-foveated counterpart through experiments that probe: 1) i.i.d and o.o.d generalization; 2) robustness to occlusion; 3) a center image bias; and 4) high spatial frequency sensitivity. In addition, we examined the impact of this foveation transform with respect to two additional models derived with a rate-distortion optimization procedure to compute matched-resource systems: a lower resolution non-foveated system, and a foveated system with adaptive Gaussian blurring. The properties of greater i.i.d generalization, high spatial frequency sensitivity, and robustness to occlusion emerged exclusively in our foveated texture-based models, independent of network architecture and learning dynamics. Altogether, these results demonstrate that foveation -- via peripheral texture-based computations -- yields a distinct and robust representational format of scene information relative to standard machine vision approaches, and also provides symbiotic computational support that texture-based peripheral encoding has important representational consequences for processing in the human visual system.


Under review as a conference paper at ICLR 2021
EMERGENT PROPERTIES OF FOVEATED PERCEPTUAL SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce foveated perceptual systems - a hybrid architecture inspired by human vision, to explore the role of a texture-based foveation stage on the nature and robustness of subsequently learned visual representation in machines. Specifically, these two-stage perceptual systems first foveate an image, inducing texture- like encoding of peripheral information mimicking the effects of visual crowding which is then relayed through a convolutional neural network (CNN) trained to perform scene categorization. We find that these foveated perceptual systems learn visual representation that is distinct from their nov-aoeete counterpart through experiments that probe: 1) iid and o.o.d generalization; 2) robustness ocelusion; 3) center image bias; and 4) high spatial frequency sensitivity. In addition, we examined the impact of this foveation transform with respect to additional models derived with a rate-distortion optimization procedure compute matched-resource systems: lower resolution non-foveated system, and a foveated system with adaptive Gaussian blurring. The properties of greater i..d generalization, high spatial frequency sensitivity, and robustness to occlusion emerged exclusively in our foveated texture-based models, independent of network architecture and learning dynamics. Altogether, these results demonstrate that foveation via peripheral texture-based computations yields a distinct and robust representational format of scene information relative to standard machine vision approaches, and also provides symbiotic computational support that texture-based peripheral encoding has important representational consequences for processing in the human visual system.
INTRODUCTION
In the human visual system, incoming light sampled with different resolution across the retinal area, stark contrast to machines that perceive images at uniform resolution. One account for the nature of this foveated (spatially-vayinng array humans is related purely to sensory efficiency (biophysical constraints) (Land & Nilsson 2012 Eckstein 2011). c.g.. there is only finite amount of retinal ganglion cells (RGC) that can relay information from the retina to the LGN constrained by the flexibility and thickness of the optic nerve. Thus it is "more efficient" to have moveable high-acuity fovea, rather than a non-moveable uniform resolution retina when given a limited number of photoreceptors suggested in Akbas Eckstein (2017). Machines, however do not have such wiring/resource constraints and with their already proven success in computer vision (LeCun et al.) 2015 this raises the question if foveated inductive bias is even necessary for vision at all.
However, it is also possible that foveation plays a functional role at the representational level, which can confer perceptual advantages as has been explored in humans. This idea has remained elusive in computer vision, but popular in vision science, and has been explored both psychophysically (Loschky et al. 2019) and computationally (Poggio et al., 2014 Cheung et af., 2017 Han et al.] (020). There are several symbiotic examples arguing for the functional advantages of foveation in humans, via functional advantages machine vision systems. For example, in the work Pramod al. (2018), blurring the image in the periphery gave increase in object recognition performance of computer vision systems by reducing their false positive rate. Wu et al. (2018) 's GistNet, directly introducing dual-stream siveall-eripheril pathway in neural network boosted object detection performance via scene gist and contextual cucing. Relatedly, the most well known example of work that has directly shown the advantage of peripheral vision for scene processing in humans is Wang & Cottrell

---

Under review as a conference paper at ICLR 2021
Image
Foveation Transtom: fo(o)
Foveated Image
Figure 1: A cartoon illustrating how a foveated image rendered resembling human visual metamer via the foveated locally perturbed with noise in its latent space the direction of their equivalent texture representation (blue arrows) resulting in visual crowding effects in the periphery. These effects most noticeable far away from the navy dot which is the simulated center of gaze (foveal region) of an observer.
style transfer model of Deza et al. (2019). Here, each receptive field is
theirequivalent
(2017)'s dual stream CNN that modelled the results of Larson & Loschky (2009) with a log-polar transform and adaptive Gaussian blurring IRGC-convergence). Taken together, these studies present support for the functional hypothesis of a foveated visual system.
Importantly, none of these studies introducing the notion of texture representation in the periphery a key property of peripheral computation as posed in [Rosenholtz (2016). Testing whether this texture-based coding of the visual periphery is functional in any perceptual system still an open question. Here we address this question directly. Specifically, we introduce foveated perceptual systems: these are two-stage (hybrid) systems that have texture-based foveation stage followed by deep convolutional neural network. In particular, we will mimic foveation over images using a transform that simulates visual crowding (Levi. 2011 Pelli 2008 Doerig al. 2019h a) in the periphery as shown in Figure 1 (Deza et al., 2019), rather than Gaussian blurring (Pramod et al.) 2018. Wang & Cottrell 2017) or compression (Patney et al., 2016 Kaplanyan et al. 2019). These rendered images capture image statistics akin to those preserved in human peripheral vision, and resembling texture computation at the stage of area V2, as argued in Freeman & Simoncelli (201T);
Rosenholtz (016);; Wallis et al. (2019).
Thus, our strategy in this paper is to compare these hybrid models' perceptual biases to their non-foveated counterpart through a set of experiments: generalization, robustness to occlusion, image-region bias and spatial frequency sensitivity. A difference from Wang & Cottrell (2017) is that our goal is not implement foveation with adaptive gaussian blurring to fit known results to data (Larson & Loschky 2009); but rather to explore the emergent representational consequences on scene representation following textare-based foveation. While it is certainly possible that in these machine vision systems that only need to categorize scenes, there may be little to no benefit of this texture-based computation: the logic of our approach however, is that any benefits or relevant differences between these systems can shed light into both the importance of texture-base peripheral computation humans, and could suggest a new inductive bias for advanced machine perception.
FOVEATED PERCEPTUAL SYSTEMS
We define perceptual systems as two-stage with a foveation transform (stage 1, f(o) RR - RD,, that is relayed to a deep convolutional neural network (stage 2, g(o) R - RR... Note that the first transform stage is a fixed operation over the input image, while the second stage has learnable parameters. In general, the perceptual system S(o), with retinal image input / IR_ is defined as:
S(I) = g(f(())
(I)
Such two stage models have been growing in popularity, and the reasons these models (including our own) are designed to not be fully end-to-end differentiable is mainly to force one type of computation into the first-stage of a system such that the second-stage g(o) must figure out how to capitalize on such forced transformation and thus assess its f(o) representational consequences. For example, Parthasarathy & Simoncelli 2020) successfully imposed V1-like computation in stage 1
2

---

Under review as a conference paper at ICLR 2021
A.
Input
Foveation Transform
CNN-backbone
Scene Classification
Algorishm:
Fowettio
(f(()
Given pestrraatinn coefficients [o]
fo(o) computed {ar)
Foveation
Gaven the earoder decoder pain[E, D)
Gaven Style-Traudde Opeeatoo:
(Styl, Content)
Given the set pooling, regicna: (6,0)
1: Define Architecture A:
2: 7 E(T)
6, 4: b. for nach v3
Standard-NWe -etrr-d--ettt
5. samppe Noion
Jofe
6: Qa.a, ATa.N(Ts...... NM
7: Ms..o. an + an
Virzal Crowding
8: D(M)
Tapet lingge outggut Fovested Imaaee I'
Figure 2: A. Two key Perceptual Systems S: Foveation-Ner (top row) and Standard-Ner (bottom row), where each system receives image as input, applies foveation transform (f(o)). which then relayed to a CNN architecture (g(o)) for scene classification. Foveation-Ne uses a visual crowding model with texture computation (shown right), while Standard-Nets provides baseline for perceptual upper-hound for all perceptual systems. B. The algorithm of how a foveated image is generated that enables visual crowding (mainly steps 5-7).
to explore the learned role of texture representation in later stages with a self-supervised objective, and [Dapello et al. (2020) found that fixing VI-like computation also at stage aided adversarial robustness. At a higher level, our objective is similar where we would like to force texture-based peripheral coding mechanism (V2) at the first stage to check if the perceptual system through g(o) will learn to pick-up on this newly made representation and make 'good' use of it potentially shedding light on the functionality hypothesis for machines and humans. This leads us to further motivate our choice for single-stream
deveated-periphert pathway (vs dual-stream Wang & Cottrell (2017); Wu et al. (018)), as we would like to focus on how g(o) will differ purely based on the learning dynamics driven by SGD rather than architectural constraints (Deza al.) (020) for the newly transformed input space (foveated: f- vs non-foveated. fo)) to perform a 20-way scene categorization task.
2.1 STAGE 1: FOVEATION TRANSFORM (f)
To model the computations a foveated visual system, we employed the model of Deza et al. (2019) (henceforth Foveation Transform). This model is inspired by the metamer synthesis model Freeman & Simoncelli (2011), where new images are rendered to have locally matching texture statisties (Portilla & Simoncelli 22000; Balas al., 2009) in greater size pooling regions of the visual periphery with structural constraints. Analogously, the Deza et al. (2019) Foveation Transform uses foveated feed-forward style transfer (Huang & Belongie 2017 network to latently perturb the image in the direction of its locally matched texture (see Figure Altogether, RD is convolutional auto-encoder that non-foveated when the latent space un un-perturbed: fo(T) = (T), but foveated (ox: when the latent space is perturbed via localized style transfer: f.(T) = (T)), for given encoder-decoder (E, D) pair.
RD
Note that when calibrating the distortions correctly, the resulting procedure can yield visual metamer (for human), which is carefully perturbed image perceptually indistinguishable from its reference image (Feather et al., 2019). However, importantly in the present work, we exaggerated the strength of these texture-driven distortions (beyond the metameric boundary), as our aim here is to understand the implications of this kind of texturized peripheral input later stage representations (e.g. following a similar approach as Dapello et al. (2020)). By having an extreme manipulation, we reasoned this would accentuate the consequences of these distortions, making them more detectable in subsequent experiments.
2.2 STAGE CONVOLUTIONAL NEURAL NETWORK BACKBONE (g)
The foveated images (stage 1) are passed into a standard neural network architecture. Here we tested two different base architectures: AlexNet (Krizhevsky et al. 2012), and ResNet18 (He et al.) 2016). The goal of running these experiments on two different architectures is to let us examine the consequences of foveation that robust to these different network architectures. Further, this CNN
3

---

Under review as a conference paper at ICLR 2021
A.
Matched Resource Foveation-Nets
B. Sate-Diittottion Optimization
C. Foveation Transform
Equalize Ratew R
Ada Ada-Gauss- Net
Foveated
R
SSIM
SSIM
for given Distortion: (D)
Texture
Adaptive Blu
for each rotinal ccomtricity
Equalize Rates: R
Matched-Niet Net
Nom con-Foveatedd
RO
SSIM
fo
R
SSIM
for given Distortion: (D)
Blud for the whole transformed image
Figure A. Two matched-resource controls Foveation-Net are introduced. Top, seagreen: adaptive gaussian blurring Adda-Gauss-Net emulating RGC convergence- foveation; Bottom, orchid: uniform blurring emulating sattched-resuure non-foveated visual system (Matched-Net). B. A Rate- DistorrDtt Optimization procedure is summarized where we compute the matched-resource images that will define the new foveation transforms: {f., fo). C. The foveation transforms of both perceptual systems Ada-Gauss-Ne (top) and Matched-Net (bottom) do not model visual crowding.
backbone (g= RRD. IR should not be viewed in the traditional way of end-to-end input/output system where the input the retinal image (7), and the output is a one-hot vector encoding d-class- label in IRd Rather, the CNN (g) acts proxy of higher stages of visual processing (as receives input from f), analogous to the 2-stage model of Lindsey et al. (2019)
2.3 CRITICAL MANIPULATIONS: FOVEATED NON- NON-FOVEA PERCEPTUAL SYSTEMS
Now, we can define two perceptual systems as the focus of our experiments that must perform 20-way scene categorization: Foveation-Nel, receives image input, applies the foveated transform ʃ- fo()), and later relays it through the CNN g(o); and Standard-Ner that performs a non-foveated transform fo(o), where images are sent through the same convolutional auto-encoder (I)(.). but with the parameter that determines the degree of texture style transfer set to O. Both of these systems are depicted Figure 2 (A).
Anticipating the potential differences between Foveation- -eets and Standard-Nets, a natural next question arises: Will these differences be found with any type opatiall-vvayying processing, or are the effects specific to type of foveation transformation (texture-based coding as linked to area V2 vs adaptive gaussian blurring given retinal constraints). Thus, we created matched-resoume adaptive blurred foveation transform using tatt--Disootion (RD) optimization framework inspired from Ballé cct al. (0016). Figure 3 (B) summarizes our goal and approach: we will have to find a set standard deviation of the gaussian blurring kernel per retinal eccentricity (the 'distortions' D), such that can render a perceptually matched adaptive gaussian blurred image in reference to the compressed image from Standard-Nets that matches the perceptual transmission 'rate' R of Foveation-Nets via the SSIM perceptual metric [Wang et al., 2004) also in reference to Standard-Nets.
Perceptual Systems trained on these images (which define new function f-), provide matched-resource control Foveation- Net that we dub Ada-Gauss-Net. This same optimization pro- cedure can be extended to the entire image to also develop matched-resoue
E R
Rate (R)
0.7
Matched-Resouree
E R
control for non-foveated visual system (Matched-Net.. fo: Figure 4D, as one could argue that Standard- Nets may be at an unfair advantage to Foveation-Nees since their foveal regions are identical and the retinal sensors have not been properly re-distributed (Cheung et al., (017) & Figure (A).
0.5
30
Distortion (D)
Figure 4: RD-Optimization.
Altogether these 4 perceptual systems help answer three key questions shown in Figure (A.): 1) Foveation-Ner vs Standard-No will tell us how a texture-based foveation mechanism will compare to its undistorted perceptual upper-bound, 2) Foveation-Nets Ada-Gauss- will tell
4

---

Under review as a conference paper at ICLR 2021
Q
How does Fovastion mechomism sflect Jearaed Reprosmntatios!
B.
Image Dataees Min--Plooss Places (20 way Semne Categotiintion Thek)
Fiveation-Ker Net
Novellon-Met
Maycheee-td
Ace Saans-
VIS
aadiands
Systems
advantage Foweated Systems??
Fovestton-heet
Hattttedi-bin
Adb-CGuussMMe
Malethecitble
Q Fivrated Systemss perkeet wark pronegteed Uppez Bound?
(Kon Ressave Masika
Povestion-Neu Not
Standard Not
Ada Gauss Not
Standard het
Carpue
VS
VS
1.00 Felt
(Duly 5 scaue clesss slawn Figuas)
Figure 5: A. Three critical questions that our conditions will help us answer pertaining the role of foveated (spatially-varyingg) perceptual systems and their comparisons to their non-foveated counterparts, B. Five example images from the 20 scene categories are shown, after being passed through the first stage of each perceptual system.
us if any potentially interesting pattern of results is due the type/stage of foveation. This will help us measure the contributions of the crowding mechanism and texture computation vs adaptive gaussian blurring; 3) Foveation-Net vs Matched-Net will tell how do these perceptual systems (one foveated, and the other one not) behave when allocated with a fixed number of perceptual resources. In addition our supplementary material includes an extended analysis that includes 2 more highly relevant systems such as Foveation-Aue Aug- Net (a foveated system with the crowding operator coupled with eye-movements non-foveated system trained with popular data-augmentation mechanisms).
biological proxy of fata-auummettttio)) and Standard- -uuurd---u--eet (a
3 METHODS
All models were trained to perform 20-way scene categorization. The scene categories were selected from the Places2 dataset (Zhou al. 2017). with 4500 images per category in the training 250 per category for validation, and 250 per category for testing. The categories included were: aquarium, badlands, bedroom, bridge, campus, corridor, forest path, highway, hospital, industrial area, japanese garden, kitchen, mansion, mountain, ocean, office, restaurant, skyscraper, train interior, waterfall. Samples of these scenes coupled with their foveation transforms can be seen in Figure (B.).
Training: Convolutional neural networks of the stage 2 of each perceptual system were trained which resulted in 40 networks per architecture: Foveation- Nets, 10 Standard-Nets, 10 Matched- Net, 10 Ada-Gauss-Nec totalling 80 trained networks to compute relevant error bars shown in all figures (standard deviations, not standard errors) and to reduce effects of randomness driven by the particular network initialization. All systems were paired such that their stage 2 architectures g(o) started with the same random weight initialization prior to training. Other parameters in training via backpropagation were: learning rates: n 0.001 AlexNet, 17 0.0005 ResNet18 with no learning rate decay for both architectures, and batch size: 128. We trained the AlexNet architectures for each perceptual system (g(o)) up 360 epochs and each ResNet18 architecture up to 180 epochs. In the main body of the paper we will show our results for AlexNet at 270 epochs. All effects reported are reproduced across both architectures (AlexNet, ResNet18) and multiple epochs (180,270,366) (90, 220, 180) respectively in the Supplementary Material which suggests that these effects are independent of learning dynamics.
Testing: The networks of each perceptual system were tested on the same type of image distribution they were trained on.
5

---

Under review as a conference paper at ICLR 2021
A. Penseiual Uyper Bord Standard-Nee Net
Dilleni Comyutatin
B.
843 o.o.d Generalization
i.i.d Generalization
V2 Teetzze Coonding Foveation-Nes Net
RGC Uhs Acmyt Lowe Ada Gauss-Net
Matched Net
à 2,205
00.3 5.155
6337 -333
80
Top-1 Accuracy (%)
70
50
SCatcheo Rescuras
40
30
20
(e(-)
A-(+)
(nle)
f.le)
0
Figure 6: A. A legend illustrating the 4 key perceptual systems in all our experiments with their differences and similarities: Standard-Net, Foveation-Net, Matched-Net and Ada-Gauss, -etsseuss-et. B. Scene Categorization Accuracy: Foveation-Net has the highest i.i.d generalization while Ada-Gauss- Net has the greatest o.o.d generalization both perceptual systems are foveated (spatially-varyingg
4 EXPERIMENTS
In the following section we will discuss results found with regards to i.i.d. & o.o.d. generalization, robustness to occlusion, central image bias and spatial frequency sensitivity. All results shown in this section use AlexNet as g() for the second stage of each perceptual system. The same pattem of results are shown with a ResNet18 as stage 2 - these extended results coupled with learning dynamics, as well as extra calculations and additional controls, can be accessed in the Supplementary Material.
4.1 GENERALIZATION
We first examined the generalization capacity for our 4 network types. Given that Foveation-Nets receive distorted inputs in the periphery one could expect them to do worse compared to Standard- Nets (which operate over full resolution, untexturized images) and on par with Matched- Nets (which have with perceptually matched resources as Foveation-Nets). On the other hand, is possible that the texture-based image encoding could confer functional advantage, in which case Foveation-Nets would do better. If this is indeed the case, we would also see an advantage of Foveation-Ner over Ada-Ga Causs-Net; but if their performance is the same, we can conclude that any type of spatially- varying processing operator can enhance scene categorization performance. Additionally, a somewhat contrived but interesting experiment, which each perceptual system is shown stimuli type that they never encountered during training in particular stimuli that is the input following the first stage of the other perceptual systems.
will also conduct is the -fitt--distiiuutin (o.o.d) case in
Result 1: Figure ( shows that a foveated system with texture-based computation in the periphery (Foveation-Net) has greater i..d generalization than its matched resource non-foveated system (Matched-Ne Net) and a foveated system with adaptive gaussian blurring foveated system (Standard-Nerl For an o.o.d performs on par with an un- n--matched resource generalization task where systems are shown 0.o.d images from another system, we find that drops in performance for foveated perceptual systems are lower compared to their non-foveated counter-part (Foveation-Nee (1) vs Standard-Ne (J); ada-Gaass-Nee (f) vs Matched-Net (4)); and that overall Ala-Gauss-Nee has greater o.o.d generalization compared to all other systems, while Foveation-Ner struggles to cross-generalize to low pass spatially transformed images compared to Ada- -aaGGuus--Net a result that ties to the experiments of Section
and also
4.2 ROBUSTNESS TO OCCLUSION
We next examined how all perceptual systems could classify scene information under conditions of visual field loss, either from the center part of the image (scotoma), or the periphery (glaucoma). This manipulation lets examine the degree to which the representations are relying on central vs peripheral information to classify scene categories. For the loveal-occlusion conditions. we superimposed central gray square on each image from the testing image set, with 8 levels of

---

Under review as a conference paper at ICLR 2021
Robustness Occlusion
Parceptual System
Inpur Visualization
A.
Occluding stimulus Fovestion "ransform
Ooduuingg stimuuss Fost Fovestion Transform
aUU e O1) fenden
(L
Standard o(/12 OO)
Faveation
Standard--g
st/()) ())
Foveation Net
Scatoms
det
do(o)
[.(0)
fa(o)
5.(0)
Visual Input
Visual Input Occiusion is superimposed transtormed images
Scotoma
Metched Heel
Ada- Male-Guus--le -eel
Matched-- Net
Ade-Geuss-Ke Nel.
(ot-)
[.(+)
(o(=)
f+(+)
Standard-Net
Fovestion
Standard-Net
Foveation-Nes
Glauconn
The rec 1o
fo(o)
Visual Input
(+(+)
Su(=)
Visual Input
Glaucome
Occiusion
superimpased transtormed images
Matched-Net
Ada- Gaa----uss-eett
Matched--ae Not
Ada-Gauss Net
fo(o)
f.(+)
fo(-)
Fabio
Occluded Area (4/81
Occluded Area (4/8)
Figure A. The Robustness curves for image occlusion (o @ O) when the Foveation Transform is On (as the stimuli is shown before the transform), and when the Foveation Transform is Off (stimuli shown post-transform). Foveation-Ner
performs has greater robustness to occlusion across all matched- resource systems, and also bumps it's performance due to visual crowding. B. A visualization of how the visual input is transformed for each perceptual system given their respective Foveation transform (J). Left: The effects of crowding are noticeable for Foveation-Nets (but not Ada-Gauss-Nett as it increases visual area. Right: Even without the aid of visual crowding, the learned texture representation yields greater robustness with matched occluded areas for Foveation-Nets.
increasing size. For the triihheal--oclusiie condition, we superimposed a gray box over the outter image boundaries, with area- geaame levels of occlusion. These images were then passed through all of the trained models to compute scene categorization performance. Accuracy was measured at each level of occlusion and area under the curve was computed as an index of robustness to occlusion.
Figure 7 (top) shows summary of these results. Overall Foveation showed more robustness to central/scotoma occlusion to all other systems: un matmate non-foveated "Standard-Netts matched- resource foveated system with adaptive gaussian blurring (Ada-Gauss-Nett) and matched-resoure non-foveated systems (Matched-Ned This result similar to peripheral/glaucome being par with non-resource matched Standard-Net To further explore these results, we visualized what the image inputs look like after the foveation stage, across all systems. Figure (B. left) reveals critical information:
occlusion with
the exception of Foveation
Result 2a: The effect of the stage 1 crowding operation (o)) of Foveation-Net exclusively reduces the occluded area in the periphery stark contrast compared to all other systems. By filling in the image with texture statistics due to the foveation stage (f.) - Foveation-Net (but not Ada-Gauss-Net able boost classification performance by using greater visual information and/or the peripheral texturized representations that Standard-Nets & Matched-Nett do not have access to.
But is this the main factor accounting for the higher robustness of the foveated systems? In other words: is it driven by greater computable visual area that has been filled in from the foveation stage; Or the benefit driven by the learned texturized representations in the subsequent deep convolutional neural network? To examine this question, we occluded either the center periphery of the images, but after the stage 1 computation (see Figure 7 (A., right)). In this way, the area of occlusion was matched for the second stage classification task.
area condition: Foveation- Net has greater robustness to occlusion than Matched-Net and ada-Gauss-Net. This puzzling result for both scotomal and glaucomal occlusion, not only suggests that the learned represen- tations in Foveation-Net is different than Ada-Gauss-Net, but that Foveation-Net has learned locally distributed representation vs a globally distributed representation (Ada-Gauss-)) Gauss-Net), In addi- tion Foveation-No Nett and Gauss-Net only differ in peripheral computation, this suggests that a cesture-based encoding mechanism provides a representational advantage than an adaptive blurred representation when information is unavailable, occluded removed.
Result 2b: Across the matched- resouurce systems and
7

---

Under review as a conference paper at ICLR 2021
Perceptual System Input Visualization
Window Cue-Conflict
(e) Peripheral Scone Accuracy (1s
Window Cue-Conflict
Wedsarng Soee Foveation Theratom
Foveal Scene Accuratty (%)
Standard Net
(((()) 8 (02) Fovcation Net
Peripheraa Accuracy
72
4209
Matchweb-lont
Adl-Cousss-liet
3el-2
4,19
1.1
Windowing Composition Procedure
Ratio image Arma with Fooera
Centrall Image Aena hoowel
Figure 8: A. A visualization of the stimuli type that is shown to each perceptual system consists of a smoothed composition of a foveal image (e.g. badlands) mixed with a peripheral image (e.g. aquarium). B. The composition procedure relies on using the same log-polar pooling windows over which the foveated images were rendered. C.,D. Foveated Perceptual Systems independent the computation type (Ada- Netss,, Foveation-Net) show stronger biases classify scenes with the foveal region; a result also observed in humans (Larson & Loschky 2009 Loschky et al. (019).
4.3 WINDOW CUE- CONFLICT EXPERIMENT
is possible that foveated systems weight visual information strongly in the foveal region than the peripheral region as hinted by our occlusion results (the different rate of decay for the accuracy curves in the Scotoma and Glaucoma conditions) - and due to the fact that the information the periphery is texturized/blurred To assess such difference, we conducted an experiment where we created a windowed cue-conflict stimuli where we re-rendered our set of testing images with one image category in the fovea, and another one in the periphery. All cunfconfii images were paired with different class (ex: aquarium with badlands). We then systematically varied the fovea-periphery visual area ratio & examined classification accuracy for both the foveal and peripheral scenes (Figure 8).
We found that Foveation-Nets and Ada-Gause Gauss-uuseee Nets maintained higher accuracy for the foveal scene class than do Standard-Nets and Matched- as the amount of competing peripheral information increased- - and vice versa for the peripheral scene class. A qualitative way of seeing this foveal-bias is by checking the foveal/peripheral ratio where these two accuracy lines cross. The more leftward the cross-over point (8), the higher the foveal bias. Thus, Foveation-Nets have learned to weigh information in the center of the image more when categorizing scenes a similar finding to Wang & Cottrell (2017) indeed as Ada- ada-Gauss-Net vs Matched-Ner shows the exact same pattern of results. Thus, these results indicate that the spatially varying computation from center to periphery (evident in both Foveation-Nets and Ada-Gauss-Nety is mainly responsible for the development of center image bias. is possible that one of the functions of any ppatialy-vvariing coding mechanisms in the visual field is to enforce the perceptual system to attend on the foveal region avoiding the shortcut of learning to attend the entire visual field (Geirhos et al. 2020).
Result 3:
4.4 SPATIAL FREQUENCY SENSITIVITY
We next examined differences of learned feature representations that are more reliant on low or high spatial frequency information, at the second stage of visual processing (post-foveation) across all systems. To do so, we filtered the testing image set multiple levels to create both high pass and low pass frequency stimuli and assessed ccene-lassification performance over these images for all models, shown in Figure 9 (A.). Low pass frequency stimuli (FL) were rendered by convolving Gaussian filter of standard deviation o = [o, 1, 5, 7, 10, 15, 40] pixels on the foveation transform (Jo. f., outputs. Similarly, the high pass stimuli (Fn) was computed by subtracting the reference image from its low pass filtered version with o = (,,, 1.5, 1, 0.7. 0.55, 0.45, 0.4) pixels and adding residual. These are the same values used in the experiments of Geirhos et al. (2019).
Result 4: We found that Foveation-Nets and Standard-Nets were more sensitive to High Pass Fre- quency information, while Ada- -eatGaass-Nets and Matched-Nets were sensitive to Low Pass Frequency stimuli. This suggests that foveation via adaptive gaussian blurring MAda-Gauss-Na contribute scale-invee invariance also shown in PPggioo et al. (2014); Cheung al. (2017); Han et al.) (2020). Additionally, an intriguing possibility is that Foveation Nets' peripheral texture representation may potentially support aallooccally--lassiii mechanism of shape bias argued in Figure g
may implicitly
8

---

Under review as a conference paper at ICLR 2021
Perceptual System Input Visualization
Spatial Frequency Sensitivity
A.
Fitering cooon Post Fovcation Transform
High Pass Frequency
Low Pass Frequency
Standard-Net
(FFx())))
Foveation-Ned
Maanaalhe Facualss-et Aes Cassa led
fele
1 Accuraic (%)
Visual Input
Matched-AMe Nett
Ada- Get-Gauss-Net
fu(o)
High Pass
05 15 35 25 a5 K5
luge condert devirtion of Gountiion
Ing crastaedd SPANSON Gaunda
Fitering (F)
Post Fovcation Transform
Standard-Nee Met
(FFT))
Foveation Net
Visual Input
Matched Not
Ados Gaussa Not
22
foto)
15 05 so Logg High Pass Frequency
de -10
6 filee stardard deviation Low Pass Frequency
Low Pass
A. Sample images from the foveated and non-foveated images and how they change as a function of spatial frequency filtering at the post-foveation stage. B. Foveation-Nets have nominally greater sensitivity to high spatial frequency filtered stimuli than Standard -ets, and both of these systems present notably higher sensitivity to high spatial frequencies than Matched-Nee -ett and Ada- Geuss-Nets. Conversely, this pattern is reversed for low pass frequency stimuli. C. This suggests Foveation-Nets's crowding-like computation may naturally enforce shape-bias given it's high pass spatially frequency tolerance as these Spatial Frequency curves show similar trends as Geirhos et al. (2019)'s 's Stylized ImageNet (SIN) that were trained on objects (adapted and redrawn).
Figure 9:
DISCUSSION
The present work was motivated by the broad question of the functional role of a eexureebbasd peripheral representation. We specifically examined whether a texture-based peripheral encoding mechanism yielded any perceptual advantages and distinctive representational signatures in second- stage deep neural networks. We found that when comparing Foveation-Nets to their matched-resource models that differed in computation: Ada-Gauss- Nets (foveated w/ adaptive gaussian blur) and Matched- Nets (non-foveated w/ uniform blur) that peripheral texture encoding did lead to specific computational signatures, particularly in robustness to occlusion. We also found that foveation (in general) seems to induce focusing mechanism, servicing the foveal/central region, while the texture-based computation still preserves high spatial-frequena selectivity - this last result is likely due to weight- that does not give room to low-spatial frequency tuned filters to naturally emerge as in Wang & Cottrell (017))
The particular consequences of foveation stage raises interesting future directions about what computational advantages could arise when trained on object categorization, as objects are typically centered in view and have different hierarchicall/compssiiinn priors than scenes (Zhou al. (2014);
Deza et al. (2020)) in addition to different processing mechanisms (Renninger & Malik (2004)
Specifically, one intriguing possibility is that our foveated representational signatures may induce
more shape sensitivity for object recognition (rather than texture sensitivity; Geirhos et al. (2019);
Hermann et al. (2020)) and may amplify the perceptual differences we identified across foveated and non-foveated systems.
Finally, future direction is investigating the effects of crowding to adversarial robustness. Motivated by the recent work of Dapello et al.] (2020) which has shown promise of adversarial robustness via enforcing stochasticity and V1-like computation by matching the Nyquist sampling frequency of these filters (Serre et al., 2007) in addition to natural gamut of orientations and frequencies as studied in [e Valois et al. (1982). it raises the question of how much we can further push for robustness in hybrid perceptual systems like these, drawing on more biological mechanisms. Both Luo et al. (2013) and Reddy et al. (2020) have already taken steps in this direction by coupling fixations with -arriin retina. However, the computational impact of visual crowding and texture-based based computation on adversarial robustness is an open question.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Emre Akbas and Miguel P Eckstein. Object detection through search with foveated visual system.
PLoS computational biology, 13(10)ee0055433, 2017.
Benjamin Balas, Lisa Nakano, and Ruth Rosenholtz. summary-statistic representation peripheral vision explains visual crowding. Journal of vision, 9(12):33-13, 2009.
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. arXiv preprint arXiv: 161.00704, 2016.
Brian Cheung, Eric Weiss, and Bruno Oishausen. Emergence of foveal image sampling from learning to attend in visual scenes. International Conference on Learning Representations (ICLR), 2017.
Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James DiCarlo. Simulating a primary visual cortex at the front of cnns improves robustness to image perturbations. BioRxiv, 2020.
Russell De Valois, William Yund, and Norva Hepler. The orientation and direction selectivity of cells in macaque visual cortex. Vision research, 22(5):531-544, 1982.
Arturo Deza and Miguel Eckstein. Can peripheral representations improve clutter metrics complex scenes? In Advances in Neural Information Processing Systems, pp. 2847-2855, 2016.
Arturo Deza, Aditya Jonnalagadda, and Miguel P. Eckstein. Towards metamerism via foveated
style transfer. International Conference on Learning Representations, 2019. URL https
//openreview net
Arturo Deza, Qianli Liao, Andrzej Banburski, and Tomaso Poggio. Hierarchically local tasks and
deep convolutional networks. CBMM Memo, 2020.
A. Doerig, A. Bornet, O. H. Choung, and M. H. Herzog. Crowding reveals fundamental differences in local vs. global processing humans and machines. bioRxiv, 2019a. doi: 10. 101/744268. URL https: www wwwpwwwwwwww...oowwwo.twwwpwpwww www.biiiiiii................ii.. wttp:///onteeetteaaa//21/.
7442688
Adrien Doerig, Alban Bornet, Ruth Rosenholtz, Gregory Francis, Aaron M Clarke, and Michael H Herzog. Beyond bouma's window: How explain global aspects of crowding? PLoS computa- tional biology, 15(5)ee100588) 2019b.
Miguel P Eckstein. Visual search: A retrospective. Journal of vision, 11(5):14-14, 2011.
Jenelle Feather, Alex Durango, Ray Gonzalez, and Josh McDermott. Metamers of neural networks reveal divergence from human perceptual systems. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. dAlché-Bue, E. Fox, and R. Garnet (eds.), Advances in Neural Information Processing Systems 32, pp. 10078-10089, Curran Associates, Inc.. 2019. URL http::/ wwweerrssippps. httppsscccppppe////pppppppppp pdf
J Freeman and EP Simoncelli. Metamers the ventral stream. Nature neuroscience, 14(9):1195-
1201, 2011.
Lex Fridman, Benedikt Jenik, Shaiyan Keshvari, Bryan Reimer, Christoph Zetzsche, and Ruth Rosenholtz. Sideeye: A generative neural network based simulator of human peripheral vision. arXiv preprint arXiv:1706. 045608 2017.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2414-2423, 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019. http://////eeeeeeeeeeeeeeeeeeeee http://wwwerreeeeeeee///oooonnoa
10

---

Under review as a conference paper at ICLR 2021
Robert Geirhos, Homn-errik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. arXiv preprint 00Xh::2004.07780, 2020.
Wilson Geisler and Jeffrey S Perry. Real-time foveated multiresolution system for low-bandwidth video communication. In Human vision and electronic imaging III, volume 3299, pp. 294-305. International Society for Optics and Photonics, 1998.
Yena Han, Gemma Roig, Gad Geiger, and Tomaso Poggio. Scale and translation translationinnvrrannee invariance for novel objects in human vision. Scientific Reports, 10(1):1-13, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
John M Henderson and Andrew Hollingworth. High-level scene perception. Annual review of psychology, 50(1):243-271, 1999.
Katherine Hermann and Simon Korblith. Exploring inductive biases: Shape texture. 1st Workshop of Shared Visual Representations in Human and Machine Intelligence (SVRHM) at NeurlPS, 2019.
Katherine L. Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence texture bias in convolutional neural networks. Neural Information Processing Systems, 2020.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1501-1510, 2017.
Anton S Kaplanyan, Anton Sochenov, Thomas Leimkühler, Mikhail Okunev, Todd Goodall, and Gizem Rufo. Deepfovea: neural reconstruction for foveated rendering and video compression using learned statistics of natural videos. ACM Transactions on Graphics (TOG), 38(6):1-13, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Michael F Land and Dan-Eric Nilsson. Animal eyes. Oxford University Press, 2012.
Adam Larson and Lester C Loschky. The contributions of central versus peripheral vision to scene gist recognition. Journal of Vision, 910):6-6, 2009.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553)::36, 2015.
Dennis M Levi. Visual crowding. Current Biology, 21118)) 2011.
Jack Lindsey, Samuel A. Ocko. Surya Ganguli, and Stephane Deny. The effects of neural resource con- straints on early visual representations. In International Conference on Learning Representations. 2019. URL httpttttt.//peeeeeeeeeeee
fort//frumm? d=$1 xx335xtt3
Lester Loschky, Sebastien Szaffarezyk, Clement Beugnet, Michael E Young, and Muriel Boucart. The contributions of central and peripheral vision to scene-gist recognition with 180 visual field. Journal of Vision, 1955):1--15, 2019.
Yan Luo, Xavier Boix, Gemma Roig, Tomaso Poggio, and Qi Zhao. Foveation-based mechanisms alleviate adversarial examples. arXiv preprint arXiv: 1. 1,06992, 2015.
Aude Oliva and Antonio Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. International jouraaal of computer vision, 42(3):145-175, 2001.
Nikhil Parthasarathy and Eero P Simoncelli. Sele-super learning of biologically-insping visual texture model. arXiv preprint arXiv:2006. 16976. 2020.

---

Under review as a conference paper at ICLR 2021
Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. Towards foveated rendering for gaze-tracked virtual reality. ACM Transactions Graphics (TOG), 35(6):179, 2016.
Denis G Pelli. Crowding: A cortical constraint on object recognition. Current opinion in neurobiology. 18(4):445-451, 2008.
Tomaso Poggio, Jim Mutch, and Leyla Isik. Computational role of eccentricity dependent cortical magnification. arXiv preprint arXiv: 1406. 1770, 2014.
Javier Portilla and Ecro P Simoncelli. A parametric texture model based on joint statistics of complex wavelet coefficients. International journal of computer vision, 40(1):49-700 2000.
R. T. Pramod, Harish Katti, and S. P. Arun. Human peripheral blur is optimal for object recognition. arXiv preprint arXiv:1807.08477 2018.
Manish V Reddy, Andrzej Banburski, Nishka Pant, and Tomaso Poggio. Biologically inspired mechanisms for adversarial robustness. arXiv preprint arXiv:2006. 16427, 2020.
Laura Walker Renninger and Jitendra Malik. When is scene identification just texture recognition?
Vision research, (1119923301-21111 2004.
Ruth Rosenholtz. Capabilities and limitations of peripheral vision. Annual Review of Vision Science, 2:437-457, 2016.
Ruth Rosenholtz, Jie Huang, Alvin Raj. Benjamin J Balas, and Livia Ilie. A summary statistic representation in peripheral vision explains visual search. Journal of vision, 12(4):14-14, 2012.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 252, 2015.
Thomas Serre, Lior Wolf, Stanley Bileschi, Maximilian Riesenhuber, and Tomaso Poggio. Robust object recognition with cortex-like mechanisms. IEEE transactions on paltern analysis and machine intelligence, 29(3):411 (133):41-4400 1-426, 2007.
T. S. A. Wallis, C. M. Funke, A. S. Ecker, L. A. Gatys, F. A. Wichmann, and M. Bethge. A parametric texture model based on deep convolutional features closely matches texture appearance for humans. Journal of Vision, 17(12), Oct 2017. doi: 10.1167/17.12.5 12.5. htthttp:://dde ttg///..oooooooooo 116/7 11677 12.5
Thomas SA Wallis, Christina M Funke, Alexander S Ecker, Leon Gatys, Felix A Wichmann, and Matthias Bethge. Image content is more important than bouma's 's law for scene metamers. eLife, 8: e42512, 2019.
Panqu Wang and Garrison W Cottrell. Central and peripheral vision for scene recognition: A neurocomputational modeling exploration. Journal of vision, 17(4):9 2017.
Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4):600-612, 2004.
Kevin Wu, Eric Wu, and Gabriel Kreiman. Learning scene gist with convolutional neural networks to improve object recognition. In 2018 52nd Annual Conference on Information Sciences and Systems (CISS), pp. 1-6. IEEE, 2018.
Bolci Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cnns, 2014.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 4056):452-1144, 2017.
12

---

Under review as a conference paper at ICLR 2021
fo(
f.(1)
For the following Fixed Perceptusl Transmission Rate R: R SSIM( (.(4)
6.
E
Mastcher- Rasioorre
Find the distortion D: that
matches the resouroes: R (alM( fu(
Distortion
(o(o)
0=1
==3
10 15
40
Figure 10: A full explanatory diagram of the Cate-Distortion Optimization Procedure inspired from both Ballé et al. (2016) and Deza et al. (2019). The goal is to find the equivalent 'perceptual transmission rate' for a given distortion a to find a matched-resouse resource perceptual input for Foveation- Nets that is non-foveated (Figure D. This optimization produces Matched-Ner a perceptual system that receives as input uniformly blurred images as way to quantify the expense of computing lower frequencies given retinal ganglion cell re-distribution in if it were occur in humans.
SUPPLEMENTARY MATERIAL
6.1 DESCRIPTION OF FOVEATED PERCEPTUAL SYSTEMS
Foveation-Net: We adjusted the parameters of the foveation transform to have stronger distortions in the periphery that can consequently amplify the differences between a foveated and non-foveated system. This done setting the rate of growth of the receptive field size (scaling factor) s 0.4. Thus, when foveation is on at the previous scaling factor, we will abbreviate f as f=.
This value was used instead of 0.5, given that experiments of Freeman & Simoncelli (201 ));
Deza et al. (2019) has shown that this scaling factor yields a match with physiology but only when human observers are psychophysically tested between pairs of synthesized/renderee image metamers. Works of Wallis et al. (2017) 2019): Deza et al. (2019) have suggested that the when comparing non-foveated reference image it's foveated crowded version. the scaling factor is actually much smaller than 0.5 (0.24, or in some cases small as 0.20; See Table []. We thus selected a smaller factor of s = 0.4 (that still metameric to human observer between synthesized pairs), as smaller scaling factors significantly reduced the crowding effects. Ultimately, the selection of this value not critical in our studies as we are not making any comparative measurements to human psychophysical experiments (See Deza & Eckstein 2016) for example where matching scaling factors is critical).
We use the same foveation transform at the foveation stage for Standard-Net but set the scaling factor set to = 0. In this way, any potential effects of the compression/eepaasiio operations of the foveation stage in the image are matched between the Foveation-Nets and Standard- Nets. Thus, the only difference after stage is whether the image statistics were texturized in increasingly large pooling windows (Fovcation-Netts) or not (Standard-Nets) Extending our notation, no foveation stage one will be abbreviated as fo-
Standard-Net:
Note however, that Standard-Nets are not matched-resouree non-foveated controls - Standard- Nets only provide the control for comparing purely the effects of crowding (Foveation-Net) while potentially using more resources as both the foveal regions of both transforms remain intact, and matched resource control should redistribute the retinal ganglion cell convergence uniformly as explored in Cheung et al. (2017). In fact, the matched-resource control that is also non-foveated is Matched-Net as described earlier in the paper, and more detail as follows.
Matched-Net: Matched-Net provides a non-foveated resource matched control with respect to Foveation-Ner. This perceptual system essentially computed via finding the optimal standard deviation of the filtering kernel G. as shown in Figure 10 This distortion image is computed via the convolution
(0) of the image (T) with the low-pass filter: Go. Here, Wang et al. (004))s s SSIM is our idea candidate as it will take into consideration the luminance, contrast and structural
changes locally for the entire image and pool them together for an aggregate perceptual score (and also the rate R) that is upper bounded by and correlated with human perceptual judgments. As
13

---

Under review as a conference paper at ICLR 2021
(2)
(P())
For the following Fixed Pecceptual
Tranmuaiion Rate R:
0: = 0:40
02. 0.86
R S (M). (1)
Rate (N)
E
Find distortion o t that
E E|
resouaces: R SSIM @ (1) G. (P00) vo
Rate (82
(.(c)
e
e o=7
o
e-16 15
05 6.16
dig - 5.12
9 (O) 04 with contepanding:
(d), (2
Poripheral Residnal
Ost
Rate (R0
Distartion (2)
Distortion (2)
Recall: Gg is the loves
all infortmation preserved
Figure 11: A. The full explanatory diagram (continued) of the catt--Distotion Optimization Procedure from Figure 10 adapted for Ada-Gauss-NNer Nets.. B. The goal is to find the equivalent 'perceptual transmission rate" for a given distortion to find a matched-resource perceptual input for Foveation- Nets that is foveated but with adaptive gaussian blurring, i.e. we must find the standard deviation of the gaussian blurring kernel which is computed over a set of eccentricity rings that have been windowed with cosine functions. C. The full aate-Distortion curves as a function of retinal eccentricity.
SSIM operates on the luminance of the image, all validation images over which the RD curve (right) was computed were transformed grayscale find the optimal standard deviation (a 3.4737).
Ada-Gauss-Net: Is a foveated perceptual system that receives cate-Distortion optimized images that have been blurred with different standard deviations of the gaussian kernel Go as a function of retinal eccentricity. We picked the same eccentricity rings (or pooling regions) as Foveation-Ne Nets given that we did want include a potential effect that is driven by differences in receptive fields rather than differences in type of computation. Figure 11 shows the full set of distortion strengths (a) of each receptive field ring to match the transmission rate of the Foveation Transform (J)( that computes crowding.
There are other alternatives to potentially find the set of standard deviation coefficients that are not driven by rate-distortion optimization procedure. One possibility could have been to find mapping between pixels and degrees of visual angle as done in Pramod et al. (2018) and derive the coefficients by fitting contrast sensitivity function given the visual field. While this approach is appealing, the coefficients for object recognition (ImageNet Russakovsky er al. (2015)) can not be extended to scenes (Places Zhou al. (2017)) In addition, the coupling of the RDDoptimization with SSIM provides a perceptual guarantee to compare Ada-Gause GaaaGasss-Net with Foveation-Net or Matched-Net
Foveation-Aug-NV (Supplementary Control): Eye-movements can be seen as a biologically moti- vated type of data-augmentation strategy. More generally, the goal of this additional control is to test if eye-movements coupled with texture-based foveated operator will vary the strength of our results in any possible dimension: generalization, robustness, center bias, spatial frequency sensitivity. To test this, we created an enhanced version of -oveation-Net where the deep neural network stage receives foveated image of a variable fixation point from one of 9 points from fixed 3 x 3 grid. A schematic of the type of inputs Foveation-Aug-NVe Nets receives can be seen in Figure T2]
Statistical testing (paired t-test, n.s.) revealed strong differences between Foveation-Nets and Foveation-Aug-Neee likely due to the fact that scene recognition can be done independent of fixation point (Henderson & Hollingworth 1999 Oliva & Torralba| (001). However there are nominal positives differences for Foveation Aug-Nett Nett over Foveation- Nets, and it is possible that the contribution of meeemmeeentss in learning new representation is redundant in scene recognition, and that re- re-comput these experiments for an object classification task may yield stronger perceptual biases.
14

---

Under review as a conference paper at ICLR 2021
Input
Foveation Transform
CNN-backbone
Scene Classification
Iiveation-AugNNe
Standard-Aup Auod--Au---oe -Net
Polo
Figure 12: Two additional Perceptual Systems, that serve as augmmmttmmttiioonn controls: Foveation- Aug-Net (top row) and Standard-Aug- (bottom row), where each system receives an image as an input, applies a foveation transform (f'(o)) coupled with data-augmentation procedure such as eye-movements (Foveation-Aee AecpationAAugNeee Aug-Nets) or random cropping + resizing + horizontal mirroring (Standard-Auge AugtNesss Once again, these newly transformed image representations are then relayed to a CNN architecture (g(o)) for scene classification.
Standard-Aug-Net (Supplementary Control): In analogy to the previous augmentation condition, Standard-Aug-Ne enhances our Standard-Ner model with artificial data-augmeetattin regimes such as random cropping (0.7 1.0 of area), resizing, and 0.5 chance of horizontally flipping the image at training. Standard- -andard--Au-Netss was motivated by 2 main questions: 1) How will actual computer vision systems trained with common data-augmentations behave in reference to other perceptual systems?; 2) Is an augmented non-foveated system asymptotically equal to a foveated system? A schematic of the type of inputs Stagdard-Aug-aa Aug-Nets receives can be seen in Figure 12
Several experiments as we will show through-out the Supplementary section, have shown remarkable perceptual similarities between Standard-Aug-No and Matched-Nets. This should come at surprise given that Standard-Aug-Ner received randomly cropped and up-sampled images (thus simulating a type of blur through loss of resolution in the up-sampling stage). What is perhaps most interesting however is that random cropping, up-sampling and horizontal mirroring did not contribute to stronger center bias to high spatial frequency sensitivity which is correlated to shape-bias (See Figure 24 from the extended results of Spatial Frequency sensitivity). These are findings that differ from the results of Hermann & Kornblith (2019); Hermann et al.] (2020) that suggest that data-augmentation enhances shape bias for object recognition. However, recall that scene recognition and object recognition processing mechanisms differ (Renninger & Malik 2004 Zhou al. 2014; Deza et al. 2020 our findings may in fact be complimentary. This is encouraging as it supports our initial conjectures of the impact of foveation on scenes vs objects, and that stronger pattern of results may emerge when training such different image input distributions, as mentioned the Discussion (Section 55)
15

---

Under review as a conference paper at ICLR 2021
6.2 METHODS (EXTENDED)
All models were trained to perform 20-way scene categorization. The scene categories were selected from the Places2 dataset (Zhou et al. 2017), with 4500 images per category in the training set, 250 per category for validation, and 250 per category for testing. The categories included were: aquarium, badlands, bedroom, bridge, campus, corridor, forest path, highway, hospital, industrial area, japanese garden, kitchen, mansion, mountain, ocean, office, restaurant, skyscraper, train interior, waterfall.
Samples of these scenes coupled with their foveation transforms for each perceptual system can be seen in Section66.4
Training: Convolutional neural networks of the stage of each perceptual system were trained which resulted in 60 networks per architecture: 10 Foveation-Nets, 10 Standard-Nets, 10 Matched-No Net,, 10 Ada-Gauss-Nett 10 Foveation-Aug-Nee to compute relevant error bars shown in all figures (standard deviations, not standard errors) and to reduce effects of randomness driven by the particular network initialization. All systems were paired such that their stage 2 architectures g()) started with the same random weight initialization prior to training, thus the randomness was purely imposed by SGD. This procedure also facilitated our statistical testing via paired t-tests the model systems began their visual diet with the same set of random weights, thus reasonably 'pairing' them up for statistical analysis when necessary. Other parameters in training via backpropagation were: (learning rates: 1) 0.001 AlexNet, r 0.0005 ResNet18 - with no learning rate decay for both architectures), and batch size: 128. We trained the AlexNet architectures for each perceptual system (g(o)) up to 360 epochs and each ResNet18 architecture up to 180 epochs to study the learning dynamics that can be seen throughout the entire Supplementary Material showing that the main effects and patterns of results reported in the paper are preserved throughout different epochs in training.
10 Standard-Aug-Nett totalling 120 trained networks
Validation: Validation images were used exclusively to: 1) visualize the convergence of the loss function for each perceptual system (See Figure 13): 2) compute quantities in our experiments that if otherwise may introduce biases. Examples of these quantities are: the rate-distortion optimization curves for Matched-Nets and Ada-Gauss-Nets, the average channel intensity for each image across all perceptual systems as a sub-step to compute the residual in high-pass spatial frequency filtering (See Section 6.9).
Testing: Foveation-Nett were only evaluated on center fixation images at inference time. Standard- Nets were tested non-foveated images (the output of the foveation transform with scaling factor set to zero). Matched-No -ett were evaluated on uniformly blurred images that were gaussian filtered with the same standard deviation coefficient that they were trained on. Ada-Gauss-Nets were tested on adaptive gaussian blurred images that held the same collection of standard deviation coefficients over retinal eccentricities that they were trained on. Standard-Aug-Ne images as Standard-Nets. images Foveation-Nets were tested on).
were tested on the same Aeeaation-AugNNae were tested only on center fixation images (the same
No data-augmentatiin or multi-crop averaging was performed at inference for any of the networks. No data-augment aataa-augmenttaioon was performed training for each network either (except for Goveation-Aug-Neee + Standard- Augeed--AA---a Aug-Nets).
All networks performed mean and standard contrast normalization both at training and testing with values set to mean: (0.485, 0.456, 0.406), std: (0.229, 0.224, 0.225).
16

---

Under review as a conference paper at ICLR 2021
6.3 RESULTS ARE INDEPENDENT OF LEARNING DYNAMICS OF g(o)
To quantify the robustness of the results reported in the main body of the paper, we re-ran our experiments different epochs during training of all 6 perceptual systems. Critically, the 270-th epoch for AlexNet was chosen to be displayed in the main body of the paper given that it is the approximate epoch after which the validation loss begins to diverge from the training loss for least one of the perceptual systems. This promotes our choice of picking relevant snapshot for AlexNet as two more points before and after the 270-th epoch, suggesting the 180-th (before 270) and 360-th epoch (after 270). The same effect (of diverging validation loss of at least one system) can be seen for epoch 120 of R8.Net18. Similarly, this encourages us to pick epoch 90 (before) and epoch 180 (after) other reference snapshots to evaluate the interaction of learning dynamics with our pattern of results.
g(+) AlexNet Loss Function Convergence
g(o) ResNet18 Function
Standard-Net
Standard-Net
Fovestioo-Net
Fovestion-Net
Marched-Net
Matched-Net
LO05
13 2-
Ade- Geuss-Net
Ade-GGeuss Net
LO75
Standard Aug Not
Standard Aug Net
1o15
2:
Jins
Foveation Aug Net
Foveation Aug Net
Number of Epochs
Number of Epochs
Figure 13: Learning: An averaged visualization of loss function convergence as function of epochs for all 6 perceptual systems: Standard-Nets, Foveation-Nets, Matched-Nets, Ada-Gauss-Ned Standard- Surrrd-Au-NNetss Foveation-Aug-Nett Each point in the plot is the average across the 10 different network runs from the locally averaged/smooth loss function per each 9 epochs. The epoch snapshots we show our analysis are: AlexNet: 180, 270 (reported in main body paper), 360; ResNet18: 90, 120, 180.
The collection of all the extended results for AlexNet + ResNet18 architectures at multiple epochs is shown in Section (66 (Generalization Extended), Section 6.7 (Robustness Occlusion Extended), Section 6.8 (Window Cue-Conflict Extended) and Section 6.9 (Spatial Frequency Senstitivity Ex- tended).
17

---

Under review as a conference paper at ICLR 2021
6.4 FOVEATION TRANSFORM SAMPLE VISUALIZATIONS
Standard-Net
Foveation-Net
Standard-Aug-Net Foveation-Aug-Net
Matched-Net
Ada-Gauss-NNet
Aquarium
Badlands
Bedroom
Bridge
Campus
Figure 14: Sample Testing Image Mosaics (Part 1, not cherry picked).
18

---

Under review as a conference paper at ICLR 2021
Standard-Net Standard-Aug-Nee Doveation-Aug-Nee
Foveation-Net
Matched-Net
Ada-Gauss-Ner
Corridor
Forest Path
Highway
Hospital
Industrial Area
Figure 15: Sample Testing Image Mosaic (Part 2, not cherry picked).
19

---

Under review as a conference paper at ICLR 2021
Standard-Net Standard-Aug-Ne Foveation-Aug-Net
Foveation-Net
Matched-Net
Ada-Gauss-Ne
Japanese Garden
Kitchen
Mansion
Mountain
Ocean
Figure 16: Sample Testing Image Mosaic (Part 3, not cherry picked).
20

---

Under review as a conference paper at ICLR 2021
Standard-Net Standard-Aug-Net
Foveation-Net Foveation-Aug-Net
Matched-Net
Sta-Gauss--ett
Office
Restaurant
Skyscraper
Train Interior
Waterfall
Figure 17: Sample Testing Image Mosaic (Part 4, not cherry picked).
21

---

Under review as a conference paper at ICLR 2021
6.5 DIFFERENCES ACROSS DIFFERENT FOVEATION MODELS AND RELEVANT WORK
There are currently 4 foveation models that implement texture-like computation in the peripheral field of view as shown in Table [T] We selected the Foveation Transform model given that it is computationally tractable to render foveated image dataset (100 at rate of image/second (rather than hours Freeman & Simoncelli (20TT) or minutes Wallis et al. (2017)). We did not use the highly accelerated model of Fridman al. (2017) (order of miliseconds, that was based on the Texture- Tiling Model of Rosenholtz et al. (2012) because it was: 1) Not psychophysically tested with human observers thus there is no guarantee of visual metamerism via the choice of texture statistics; 2) it does not provide an upper- bound computational baseline (similar to Standard-Net when all the perturbatioos coefficients are off, given that the system is closed and lossy; 3) the code was/is unavailable at the time. However we think that re-running our experiments and testing them with other foveated models such as the before-mentioned is a direction of future work
would be curious to see the replicability of our pattern of results across other texture- basre- peripheral models.
Model
Preeman & Simoncelli 2011)
Wallis al. 2019
Fridman et al. 2017
Deza et al 2019
Food Foeward
Input
Noise
Noise
Image
Imsige
Mulii Resclution
Texture Statisties
Steemble Paramid
VGG19 com-
41, 51
Stecrable Peramid
VGG19 refel Fluang Belongie 2017
Style Trandfer
Portilla Simomeell
2000
Galys
2016
Resenbokze al 2012
Fovested Pooling
(Imptica meamen//mongrels
Decoder (trained on)
images
Movesble Foves
Use Noise
Initialization
Initializaiion
Pertudbstion
Non Non-Daeeninnitic
Dinet Computable Inverse
(Implicit via FCN)
Renderng Time
hours
minutes
millscconds
secunds scetie
Image type
sceness
sceressttettrre
woenes
Critical Staling (vs Symth)
(1.45
[0.39/0.44
Not Requirad
0.5
Critical Scaling (1s Reference)
Not Available
(0.2/0. /0,35)
Not Required
0.24
Experimental design
Oddhall
ARX
Reference Image in Exp.
Motamer
Original
Comprossed Deooder
Number of Images tested
400
10
Trials per observers
1000
1000
3000
Table 1: Metamer Model comparison. Redrawn from Deza et al. (2019).
There are several works that have used foveation to show a type of representational advantage over non-foveated systems. Mainly Pramod et al.] (2018) with adaptive gaussian blur, and Wu et al. (2018) with scene gist, that have been targeted towards a computational goal in increasing object recognition performance. For scene recognition, only Waan & Cottrell (2017) has successfully modelled known behavioural results of Larson & Loschky (2009) via dual-stream neural network that uses adaptive gaussian blurring and a log-polar transform. One key difference however is that we are interested in exploring the effects of visual crowding that is a phenomena linked to area V2 in the primate ventral stream (rather than retinal as in Wang & Cottrell (2017) which resembles our control condition: Ada-Gauss- Nets). In general, we are taking complimentary approach where we priori do not know of functional role of rexture-based computation or prime ourselves to fit our model to reference behavioural result. Thus we explore what functionality it may have in comparison to a non-foveated system (Matched-Nett Standard-Nets) foveated system that only implements adaptive gaussian blurring (Ada-Gate Gatss-Nets). Table 2 highlights key similarities & differences between these papers and ours.
Model
Wong & Coarrll 2017
Wa al 2015
Pranod et al. 2018
(Ours)
Image tngut tyge Single/Dail Stream
abjeets
abjects
Dual Cating
Dual Concalenation
Single
Single
Role Single/Dual Stream
Coupling the fovea + peripbery log polar adagtive gusssaan blurtigg
Contextual modulation (soene gist)
Serializing the (single) twu-stage model
Foveated Transfom (ETT
Regian Selection
adaptive gaussian bluring
Visual Metamer wi texture distertion
texture-distrrtion
Stochastic F.T.
Deza al. 2019
Reprosentatioal Stuge
retinal Geisler Perry 1998
Attention"
Geisler Perry
Freehom
Moveable Fovea
Explores the cule of Eye Movements
(Supplement)
Aucounts for pooling regions
Implicii aduptive gaussimn blurring
Implicit via scaptive gaussian blurring
Accounts visual coowding
Accoumt for retinal cceentricry
Implicit via cropping
Accounts for loss of visual aculty
Implicit vis visunl crowding 5.9 deg Usstimated from Fig V
Citiicl Rutius Lasson Loschly 2009
dog
Not Applicable (Uojeces)
Oui of Distribatonn Gemeratzaioon
Rohhsttnsss Distortion Type
Blurring
Bluring
Ocelusion
Spotial Frequency Preference
High (Favea), (Poriphery)
Low (Gilohal)
High (Fovea), Low (Periphery) CenterFowen
High (Gilabal)
Wrighedd Bin Emerges
CenterrFuvee
Center/ooees
CentenFooees
Goal of loveal Peripeeral Architicture
Pot Denavioural Results
Increase Recognition Accuracy
Lispoore Perceptual Properties Hybrid
Model System Focus
Human
Machine
Humaa
Table 2: A summary set of Foveal- Periihherriperall CNN model characteristics.
22

---

Under review as a conference paper at ICLR 2021
6.6
GENERALIZATION (EXTENDED)
Li Generalization
0.0.d Generalization
AlexNet @ 180 epochs
AlexNet @ 270 epochs
AlexNet @ 360 epochs
80
70
Top-1 Accuracy (%)
50
40
30
20
10
ResNet18 @ 90 epochs
ResNet18 @ 120 epochs
ResNet18 180 epochs
80
70
Top-1 Accuracy (%)
60
50
40
20
10
Lidd Generalization
0.o.d Generalization
AlexNet @ 180 epochs
AlexNet @ 270 epochs
AlexNet @ 360 epochs
80
70
60
Top-1 Accuracy
50
40
30
20
10
18 @ epochs
ResNet18 @ epochs
ResNet18 epochs
80
70
Top-1 Accuracy (%)
50
40
30
20
10
Figure 18: Generalization: The full i..dd Generalization and o.o.d Generalization plots for AlexNet and ResNet18 across multiple epochs of training. We observe that our results do not vary as function of training epoch or network architecture: Foveation-Nets with greater i.i.d generalization across matched-resource systems, and kda-Gauss-Nee with greater o.o.d generalization across all systems. This suggests spatially-varyy iidd and o.o.d setting contingent on the type of foveated computation (texture blur). Future work should evaluate combining both computations. A. Our 4 main perceptual systems: Standard-Net Foveation-Net, Matched-Net, Ada-Gauss-Net; B. Standard-Net (re-plotted from above), Foveation- Net (re-plaa re-ppltted from above). Standard- -ugrrd--A----ee -et (supplementary)) Foveation- -eerattoo-Au-Nee Aug-Net (supplementary). Notice that only gold and navy dots are shown for o.o.d plots because Standard-Ar Maeaard--u--eett were tested on Standard- Net inputs, and Foveation-Aug-Ne
computation provides representational benefit in both the
tested Foveation Nets inputs.
23

---

Under review as a conference paper at ICLR 2021
Foveation after Occlusion: g(f + 0))
Left2Right
Top2Bottom
Glaucoma
Scotoma
AlexNet 180 epochs
80
Top-1 Accuracy (96
60
40
20
AlexNet @ 270 epochs
70
Top-1 Accuracy
30
30
15
AlexNet 360 epochs
70
Top-1 Accuracy (85)
60
30
20
Occluded Area Ratio (/8)
Occluded Area Ratio (/8)
Occluded Area Ratio (B)
Occluded Area Ratio (/B)
Figure 19: Robustness to Occlusion of All Perceptual Systems (Extended Part 1/4).
6.7 ROBUSTNESS TO OCCLUSION (EXTENDED)
We decided extend the robustness to occlusion analysis with 2 additional conditions: Left2Right and Top2Bottom, which are both occluding conditions where the stimuli occluded with a gray patch from "left to right" and from "top to bottom" respectively. Altogether we find that the same pattern of results is sustained where Foveation are near the perceptual upper-bound provided by Standard-Nett -etts. Both matched-resouree systems: Matched-Net (spatially uniform and lower- resolution) and Ada-Gauss-Net (spatially varying with adaptive gaussian blurring) are less robust to lateral, vertical, scotomal and peripheral occlusion than Foveation-Net (spatially-varying with texture-based peripheral encoding) across both AlexNet and ResNet18 architectures as g()), and across different epochs (learning dynamics). Furthermore the reported findings in the main body of the paper with regards to the impact of the visual crowding mechanism (exclusively fo)) as filling-in operator, is sustained across both AlexNet and ResNet18 architecture as g(o) and number of epochs (learning dynamics). These extended results are plotted in addition to the new controls Roveation-Aug-WW -Ness and Standard-Aur Aug-ard-- in Figures 19 21 22 In these plots we once again show that the contribution of eye-movements in training (Foveation-Aug- improvements robustness to Foveation-Nets for both the pre and post foveation to occlusion conditions. On the hand, Blandard-Aug-ae show pattern of results very similar to combination of Matched-Ner and Standard-Nett This should not be a surprise as Standard-Aug-Ner trained with multi-resolution images. Somewhat surprisingly, Standard- Baaard-Aug--a Aug-Nets do not exceed the robustness of Standard-Nets -eets, showing a particularly counter-ing intuitive limitation to aatta-augmentation.
provides marginal
have been
24

---

Under review as a conference paper at ICLR 2021
Foveation before Occlusion: g(f(1) O 0)
Left2Right
Top2Bottom
Glaucoma
Scotoma
AlexNet @ 180 epochs
Zandurs-ii
Top-1 Accuracy (%)
Notth os
Plandur de
40
25
AlexNet 270 epochs
70
1op-1 Accuracy (85)
20
18
AlexNet 360 epochs
Top-1 Accuracy (96)
60
40
20
Occluded Area Ratio (/8)
Occluded Area Ratio 1/8)
Occluded Area Ratio (/8)
Occluded Area Ratio (/8)
Figure 20: Robustness to Occlusion All Perceptual Systems (Extended Part 2/4).
25

---

Under review as a conference paper at ICLR 2021
Foveation after Occlusion: g(f + O))
Left2Right
Top2Bottom
Glaucoma
Scotoma
ResNet18 @ 90 epochs
80
Zandurs-ii
Top-1 Accuracy (%)
Notto os
Plandce do
40
ResNet18 120 epochs
70
Top-1 Accuracy (85)
20
ResNet18 180 epochs
Top-1 Accuracy (9))
60
40
20
Occluded Area Ratio (/8)
Occluded Area Ratio 1/8)
Occluded Area Ratio (/8)
Occluded Area Ratio (/8)
Figure 21: Robustness to Occlusion All Perceptual Systems (Extended Part 3/4).
26

---

Under review as a conference paper at ICLR 2021
Foveation before Occlusion: g(f(1) O 0)
Left2Right
Top2Bottom
Glaucoma
Scotoma
ResNet18 @ 90 epochs
Top-1 Accuracy (%)
Notto es
Plandue de
40
20
ResNet18 120 epochs
70
Top-1 Accuracy (85)
20
ResNet18 180 epochs
Top-1 Accuracy (96)
60
40
20
Occluded Area Ratio (/8)
Occluded Area Ratio 1/8)
Occluded Area Ratio (/8)
Occluded Area Ratio (/8)
Figure 22: Robustness to Occlusion All Perceptual Systems (Extended Part 4/4).
27

---

Under review as a conference paper at ICLR 2021
6.8
WINDOW CUE-CONELICT (EXTENDED)
AlexNet Epoch 180
ResNet18 8 Epoch 90
Perpheaa Scene Accurscy 1S1 (e Pemphers scere Accurssy 196)
Standrrkip
3
Epoch 270
Epoch 120
Reriherai Soone Accuncy 1
Epoch 360
Epoch 180
82
04
Figure 23: Window Cue Conflict Experiment: The pattern or results with regards a greater foveal bias for any foveated perceptual system non-foveated perceptual systems remains independent of the network architecture (g())) and the epoch. This can be verified by finding the cross-over points for Foveation-Nets, Ada-Gauss-Nel Standard-Nets, Matched-Nett and Standard-Aug-Nett These results are independent of potential perceptual differences at testing time i.e. baseline. For example, see AlexNet @ 360 epochs, ResNet18 120 or 180 epochs, where the cross-over point for Standard-Ners, Matched-Nets and Baandad-AAug-eer is still shifted more biased towards the right than Foveation-Nets. Ada-Gauss- Nets and Foveation-Aug-Nett subjective equality (PSE). A final note on the interpretability of these results is that this foveal bias is being tested after the foveation transforms are computed (similar to our post-foveation occlusion experiments), such that no changes in area are driving the revealed biases, and thus the bias driven purely by the learned representation.
and Foveation-Aug-Ne
being placed more leftwards than
implying greated need for foveal area to arrive to the point of
28

---

Under review as a conference paper at ICLR 2021
6.9
SPATIAL FREQUENCY SENSITIVITY (EXTENDED)
Spatial Frequency Sensitivity
Spatial Frequency Sensitivity
High Pass Frequency
Low Pass Frequency
High Pass Frequency
Low Pass Frequency
AlexNet 180 epochs
ResNet18 epochs
Standard--t Favestion Net
70
Top-1 Accuracy (%6
60
Standard
Pavcoltons dag Met
10
AlexNet epochs
ResNet18 @ epochs
Top-1 Accuracy (%6
30
AlexNet @ 360 epachs
ResNet18 @ 180 epochs
Top-1 Accuracy
30
-00
05 15 25 35 455 5.5
000
0.5
15
25
3.5 45 5.5
Jop, stancard deviation
deviation
Ing,
Figure 24: Spatial Frequency Sensitivity for AlexNet and ResNet18 across multiple training epochs:
There are no notorious differences for high pass frequency sensitivity across network architecture and epochs in comparison the results reported in the main body of the paper. Specifically, these patterns are: no differences between Foveation-Nets Foveation- Foveation-Aug-Nets Aug-Nets and Standard-Nets (all differences between (a greater bias to high pass spatial frequency sensitivity in Standard- -Ness). These systems are notably also more biased high pass spatial frequency than Ada-Gan Gaa-GaussSNN Net and Matched- -eet in an orderly fashion. The opposite pattern of results hold for low pass frequency sensitivity.
three statistically tested with paired t-tests against each other, n.s.). There Standard-Nar and Standard-Aug-Ne
Images shown to each network were the respective images from each training-testing distribution pair. Foveation-Nets were shown foveated images with a center fixation, Standard-Net were shown non- foveated images, Matched-Nets were shown matched-resoure non-foveated (yet blurred) images, and Alaa-Guss-Aa -ets were shown foveated images with adaptive gaussian blurring. Thus, both High Pass Spatial Frequency and Low Pass Spatial Frequency experiments were conducted at the post-foveation stage to directly examined the learned representations of go)): the second stage of each perceptual system.
29

---

Under review as a conference paper at ICLR 2021
The size of all shown images 256 256 x 3, thus the units of the gaussian filters specified from Section 4.4 are in pixels. For given Gaussian filtering operation Go for a given standard deviation ø, low pass spatial frequency (LF) images were computed via:
=F(1))=Gam GO
for each channel C. In the main body of the paper we used the summarized notation FL()) for low-pass frequency filtering operator. Similarly, High Pass Spatial Frequency (HF) image stimuli were computed via:
=
mean C
where mean.al (which we call the residual in the main body of the paper) is the average of image intensity over the held-out validation for each channel C, small extension from Geirhos al. (2019) as our image stimuli is in color vs grayscale. In the main body of the paper we used the summarized notation Fu (o) for the high pass frequency filtering operator.
30


Reviews:
[{"review_id": "tz0Fc4NlkZm", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Update after discussion\nThe discussion has reinforced my concerns. I cannot fully follow the logic of the paper and am not convinced the controls are useful. Therefore  I stand by my original assessment.\n\n\n# Original Review\n\nThe paper proposes to use texturization of the periphery in images as a proprocessing step for scene classification and claims \"greater iid generalization, high spatial frequency sensitivity and robustness to occlusion\" for such models. The model is essentially a concatenation of the synthesis procedure for metameric images by Deza and colleagues (2019) followed by AlexNet as a classification head.\n\n\nStrengths:\n+ Tries to address an interesting question: why does human vision treat the periphery mainly as texture?\n+ In-depth investigation of the model and three controls\n\n\nWeaknesses:\n- Unclear if the premise of the paper is well-conceived\n- No evidence provided that \"foveated\" images are actually metameric\n- \"Matched-resource\" controls are somewhat questionable\n- Definition of \"foveation\" seems strange\n\n\nDetails on major issues (all of which would need to be addressed in a convincing manner for my score to change):\n\nMy main issue with the paper is that I don't understand its logic. The results by Rosenholtz, Simoncelli, Wallis and others show that some information about the image is discarded in human peripheral vision and, to some extent, peripheral vision \"cares\" only about the texture statistics of an image rather than the detailed composition. The hypothesis is that this is the same effect that has been described as crowding.\n\nNow, the paper seems to put forward the hypothesis that distorting (texturizing) images in a way that they remain indistinguishable (metameric) for humans somehow enhances their discriminability. I don't understand what's the reasoning behind this logic. Why would throwing away information and distorting an image be useful? The original image is by definition part of the equivalence class of metameric, so why should a distorted image somehow be easier to classify by a neural network than the original image? Both images are perceived in the same way by humans. It seems like a strange proposal to me, but it's possible that I'm missing something. I'd like the authors to explain their reasoning better.\n\nHaving said that, even if we put these fundamental concerns aside, there are a number of practical issues:\n\n(1) The presented images don't appear metameric to me, and the paper does not provide any psychophysical data showing that they are. For instance, for the three images labeled \"Metameric\" in Fig. 5A, I have a very hard time believing that they are metameric to human observers. Could the authors provide psychophysical evidence that this is the case?\n\n(2) I am not sure about the purpose of the \"matched-resource\" control and why SSIM is the right metric to use here. If it's about information content in the image, I think it should be evaluated in terms of mutual information between the image and the labels (which is hard); if it's about information available to human observers, then a perceptual metric is the right approach, but SSIM is a very poor one that doesn't correspond well with human perception. Therefore, I am skeptical that these baselines are really useful.\n\n(3) The claim “greater iid generalization, high spatial frequency sensitivity and robustness to occlusion emerged exclusively in our foveated texture-based models” seems to be an overstatement. There appears to be basically no difference between foveated and standard net. In all Figures (5–8) the differences are mostly within the range of 1 SD and the order of the two models changes sometimes. The only effect that might be significant is the difference in the scotoma condition for intermediate to high occlusion conditions. The only robust difference is between standard+foveation and MatchedNet+AdaGauss, where the latter seem to carry much less information (see points (1) and (2) above).\n\n\n\nMinor comments\n\n- Sec 3.4: The connection between Gaussian blur and scale invariance is not clear\n- Sec 3.4: The following statement is trivial, since the images were low-pass filtered: “We found that Foveation-Nets and Standard-Nets were more sensitive to High Pass Frequency information, while Ada-Gauss-Nets and Matched-Nets were sensitive to Low Pass Frequency stimuli”\n- Fig 8: It is unclear how it argues for shape bias.", "extra_scores": {}, "extra_text": {}}, {"review_id": "im_7f-Bas-L", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors compared Foveation-Net with three other stage-one visual systems: Standard-Net with non-foveated transform; Ada-Gauss-Net as a matched-resource control for Foveation-Net (to test the contribution of the foveation transformation comparing to other spatially-varying processing); Matched-Net as a matched-resource control for Standard-Net. Specifically, the authors did four sets of experiments on these visual systems to evaluate their generalizability, robustness, image-region bias, and spatial frequency sensitivity.\n\nOverall, the methods are solid and clearly stated. The results suggest some perceptual advantages and specific representational properties yielded by texture-based foveation transformation. \n\n##### Specific comments and questions:\n1. It is not clearly stated how to read the plot in Fig. 5B. And why using a 20-way scene categorization task to evaluate the model performance?\n2. The results in experiment 3 suggest Foveation-Net and Ada-Gauss-Net has higher foveal bias. Is this simply because the effective sampling density is higher in the image center (as suggested by Fig. 5A) for these two visual systems? Then are these two visual systems more computationally efficient in general?\n3. In section 3.4, the authors argued that the Foveation-Net may enforce a shape-bias since \"these Spatial Frequency curves show similar trend as SIN\". Are these curves refer to the ones in Fig. 8B? But by visually inspecting this figure, it seems the trend of Standard-Net is quite closed to Foveation-Net. Please let me know if I misunderstood this argument.\n4. What would be the effect on model performance if the fixation is not in the center for Foveation-Net and Ada-Gauss-Net?", "extra_scores": {}, "extra_text": {}}, {"review_id": "bvU4N0E95VZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary:\n\nIn this paper, authors study the functional advantages of a foveal transform of visual inputs. It is nicely introduced with a very comprehensive review of the literature. The method introduces a 2 two stage model of the visual system, where the first stage corresponds to the (fixed and non adaptive) foveation stage and the second stage to the higher level processing, typically associated with the categorisation operated in the ventral stream of the visual pathway. This second stage will be implemented by existing CNN architectures (AlexNet and ResNet) which are re-learned on the transformed inputs. To control for the functional consequences of the foveated processing, the first stage can also be a single isotropic blurring of the image. Both alternatives are manipulated such that their distortion (as computed with a SSIM measure) are equally balanced, leading to 1 standard mapping and three proposal retinal transformations :  Standard-NEt (unmatched) and Foveation-Net, Matched-Net and Ada-Gauss-Net (matched). Results show that for both perceptual systems which are foveated, \"Foveation-Net has the highest i.i.d generalization while Ada-Gauss- Net has the greatest o.o.d generalization\". Second result is that foveated processing allowed a better robustness to occlusions and third result is that such networks reproduce behavioural results of a Window Cue-Conflict. Last results propose to study that foveation introduces a focusing strategy, and keep high-french information on the fovea - which seem less striking results.\n\nOverall, the paper is original, technically sound and very well supported by experiments. However, there are some concerns that I highlight below.\n\n##########################################################################\nConcern:\n\nThe  paper is very dense and wants to say too much while perhaps losing on the main point: the foveal transform In particular, there is one point about foveation and another about metamerism (and crowding). While the first is studied in depth, the second is not studied fully, or at least not parametrically. This hinders the comprehension of the mechanisms that may be at play behind the functions of foveation.  Controlling for the amount of \"metameric distortion\" at different eccentricities both in models and humans would be a novel contribution to the field.\n\nMore generally, the paper seems to explore many computational alternatives but perhaps misses a key property of the visual system, that is that RFs size grow in size AND density as a function of eccentricity - it seems the metameric distortions adds a novel ingredient as it tunes the precision of textural information as a function of eccentricity. taken together, this factors could simplify the presentation of the paper by presenting different hypothesis for the evolution of RF size and density and in particular to show why a given magnification factor may be better than other. I consider the iid vs ood and occlusion results to be the most convincing, but at the end of reading the paper, I lack a comprehension of why we have such results, and what hyperparameters play the most important role.\n\nMore particularly, for the robustness to occlusions, the results are in my understanding given of the systems learned in the non-occluded case. Would you predict that results would generalise if you re-learn weights from the occluded images? The Window Cue-Conflict results are less convincing  as the effects are rather small. Perhaps using different hyper parameters (magnification ratio, metameric distortion) would help? The conclusion \"that foveation (in general) seems to induce a focusing mechanism (...) while the texture-based computation still preserves high spatial-frequency selectivity\" seem to be a mechanical consequences of the foveation transform (you focus on a point while preserving cone density in the fovea) - did I miss something?  Finally, I know the problem of page budget, but I had to go page 16 of the SM to have the definition of the dataset and get eg the value of the chance level - a synthetic overview on p.5 would be welcome.\n\nConcerning the form of the paper, clarity could be improved in some sections. Some sentences (e.g. \"same pattern of results are evident for ResNet18\" (p. 5) \"an unexpected victory for Foveation-Nets\" (p.5)  \"is still quite impressive\" (p.6), ...) are more reminiscent to stunts from the PR department than is necessary in a scientific paper. Please avoid.\nLast, missing the code \n##########################################################################\n\nminor\np.4 « guassian » _\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FbJu4UkWIiL", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I thank the authors for a thoroughly written paper studying an important question for both machine learning and neuroscience. The authors propose a biologically inspired modification to CNN architectures by introducing foveation. Several thorough experiments are performed to assess the benefit of foveation with reasonable control transformations. The proposed modifications seem novel and discuss the relevant prior work in this domain. Appreciate the detailed discussion of all implementational specifics, I'm fairly confident about the correctness of the experiments performed and results presented.\n\n# Some concerns I have about the claims:\nI agree with some of R1's comments, the claims seem to be overstated in my opinion. In Fig. 7, it looks like Foveation-net significantly outperforms other baselines only in pre-foveation scotoma transform. This is likely because compared to Standard-Net, Foveation-Net has larger unoccluded information due to the nature of the foveation transform. In the post-foveation scotoma condition wherein the amount of unoccluded area is matched, Standard-Nets outperform Foveation-Net. This is fine, but I didn't read any discussion on this observation in the submission. I see that the authors have attempted to address this question in other comments below. In summary, I doubt whether foveation truly provides more robustness to occlusions in the presented sense.\n\nFig. 9C does not seem like sufficient evidence to claim that foveation promotes shape bias over texture bias.\n\nAs mentioned earlier, find the topic studied to be a very interesting and an important one. I'm certain this work will stimulate interesting discussions and future work to better understand the functional significance of foveation. The analyses are thorough and the reported results seem to be accurate. However I doubt whether the proposed claims are justified by the analyses. The figures are great, particularly the ones that describe procedures like foveation, scotoma, etc. The ones analyzing model performance seemed a little shrunk and I had to zoom in to glean the details. I would suggest moving some/all figures that discuss the foveation/occlusion procedures (such as in Figs 7&8) to SI to give more room for the quantitative analysis figures.\n\n# Questions to authors:\n1. What are your thoughts on applying foveation-like masking throughout the network's activations and not just at the input stage?\n2. I'm very interested to know whether the proposed mechanism can buy more performance on Stylized ImageNet, and robustness to adversarial perturbations.\n3. The authors might be interested to explore how well foveated model performs on datasets like CIFAR-10C and ImageNet-C.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2wGZNn2dqzp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explores how preprocessing an image with a foveated texture-based rendering - where content in the periphery is transformed in a lossy manner but still perceptually equivalent - affects the training of a downstream neural network.  It is shown that when equated to other lossy image transforms in terms of overall rate, that the foveated system shows better robustness to occlusion, generalization, and preservation of high spatial-frequency content.\n\nThe many figures are difficult to wade through because the are so compressed and cluttered with many labels and diagrams.  I would recommend figuring out a way to prune these down so they are more readable and convey just central punchline that you are trying to show.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the use of a texture-based foveation stage in
scene categorization.  They show that the foveated system shows
presevation of high spatial-frequency information relative to other
matched transformations.

This paper engendered a lot of discussion and had a wide range of
ratings.  An extra review was requested that fell intermediate between
the high and low scores.  Generally reviewers agree that the question
is interesting but that the paper does not clearly elucidate the logic
of the paper.  That is, Reviewer 1, and another reviewer in
discussion, had issues with the logic of the paper. I also found the
motivation behind the paper, difficult to understand at first.  I had
to find the Rosenholtz paper to understand why the authors were
considering this particular representation. This should be better
explained in the paper - the main points of the paper should be
understandable by itself.


There is also concern that the claims are not validated by
the presented results.  For example, the authors claim that their
foveated network were more robust to occlusion but Reviewer 5 points
out that this is likely due to the foveation-nets having more
unoccluded information.


On the positive side, Reviewer 4 points out that the experiments are
extensive and several reviewers commented that they trust that the experiments
were done correctly.  Reviewer 2, 4 and 5 all mention that there are too many
results reported and recommend paring down to the most important
results.

In my view the paper is right at the border of acceptance.  Acceptance/rejection will
depend on capacity limits and balancing areas.  I recommend that if accepted, 
or resubmitted to another conference, that the results be pared down, and 
more space be devoted to explaining the question(s) and why they are 
interesting and relevant and why the comparison networks allow the questions 
to be answered.  

Originality - High
Quality - High 
Clarity - Could be improved
Significance - Could be better articulated and is hard to assess as is.
Pros: - interesting idea, many well done experiments
Cons: - claims not well validated, clarity could be improved to	emphasize significance
Other: paper too dense - should be pared down to improve clarity

Decision: Reject