Emergent Properties of Foveated Perceptual Systems

The goal of this work is to characterize the representational impact that foveation1 operations have for machine vision systems , inspired by the foveated human visual2 system , which has higher acuity at the center of gaze and texture-like encoding in3 the periphery . To do so , we introduce models consisting of a first-stage fixed image4 transform followed by a second-stage learnable convolutional neural network,5 and we varied the first stage component . The primary model has a foveated-6 textural input stage , which we compare to a model with foveated-blurred input7 and a model with spatially-uniform blurred input ( both matched for perceptual8 compression ) , and a final reference model with minimal input-based compression.9 We find that : 1 ) the foveated-texture model shows similar scene classification10 accuracy as the reference model despite its compressed input , with greater i.i.d.11 generalization than the other models ; 2 ) the foveated-texture model has greater12 sensitivity to high-spatial frequency information and greater robustness to occlusion,13 w.r.t the comparison models ; 3 ) both the foveated systems , show a stronger center14 image-bias relative to the spatially-uniform systems even with a weight sharing15 constraint . Critically , these results are preserved over different classical CNN16 architectures throughout their learning dynamics . Altogether , this suggests that17 foveation with peripheral texture-based computations yields an efficient , distinct,18 and robust representational format of scene information , and provides symbiotic19 computational insight into the representational consequences that texture-based20 peripheral encoding may have for processing in the human visual system , while also21 potentially inspiring the next generation of computer vision models via spatially-22 adaptive computation.23 1 Introduction24 In the human visual system , incoming light is sampled with different resolution across the retina , a25 stark contrast to machines that perceive images at uniform resolution . One account for the nature of26 this foveated ( spatially-varying ) array in humans is related purely to sensory efficiency ( biophysical27 constraints ) ( Land & Nilsson , 2012 ; Eckstein , 2011 ) , e.g. , there is only a finite amount of retinal28 ganglion cells ( RGC ) that can relay information from the retina to the Lateral Geniculate Nucleus29 ( LGN ) constrained by the thickness of the optic nerve . Thus it is “ more efficient ” to have a moveable30 high-acuity fovea , rather than a non-moveable uniform resolution retina when given a limited number31 of photoreceptors as suggested in Akbas & Eckstein ( 2017 ) . Machines , however do not have such32 wiring/resource constraints – and with their already proven success in computer vision ( LeCun et al.,33 2015 ) – this raises the question if a foveated inductive bias is necessary for vision at all.34 However , it is also possible that foveation plays a functional role at the representational level , which35 may confer perceptual advantages – as most computational approaches have mainly focused on36 saccade planning ( Geisler et al. , 2006 ; Mnih et al. , 2014 ; Elsayed et al. , 2019 ; Daucé et al. , 2020 ) .37 This idea has remained elusive in computer vision , but popular in vision science , and has been38 explored both psychophysically ( Loschky et al. , 2019 ) and computationally ( Poggio et al. , 2014 ; 39 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . Cheung et al. , 2017 ; Han et al. , 2020 ) . Other works that have suggested representational advantages of40 foveation include the work of Pramod et al . ( 2018 ) , where blurring the image in the periphery gave an41 increase in object recognition performance of computer vision systems by reducing their false positive42 rate . In Wu et al . ( 2018 ) ’ s GistNet , directly introducing a dual-stream foveal-peripheral pathway in a43 neural network boosted object detection performance via scene gist and contextual cueing . Relatedly,44 the most well known example of work that has directly shown the advantage of peripheral vision45 for scene processing in humans is Wang & Cottrell ( 2017 ) ’ s dual stream CNN that modelled the46 results of Larson & Loschky ( 2009 ) with a log-polar transform and adaptive Gaussian blurring ( RGC-47 convergence ) . Taken together , these studies present support for the idea that foveation has useful48 representational consequences for perceptual systems . Further , these computational examples have49 symbiotic implications for understanding biological vision , indicating what the functional advantages50 of foveation in humans may be , via functional advantages in machine vision systems.51 Importantly , none of these studies introduce the notion of texture representation in the periphery – a52 key property of peripheral computation as posed in Rosenholtz ( 2016 ) . What functional consequences53 does this well-known texture-based coding in the visual periphery have , if any , on the nature of54 later stage visual representation ? Here we directly examine this question . Specifically , we introduce55 perceptual systems : as two-stage models that have an image transform stage followed by a deep56 convolutional neural network . The primary model class of interest possesses a first stage image57 transform that mimics texture-based foveation via visual crowding ( Levi , 2011 ; Pelli , 2008 ; Doerig58 et al. , 2019b , a ) in the periphery as shown in Figure 1 ( Deza et al. , 2019 ) , rather than Gaussian59 blurring ( Wang & Cottrell , 2017 ; Pramod et al. , 2018 ; Malkin et al. , 2020 ) or compression ( Patney60 et al. , 2016 ; Kaplanyan et al. , 2019 ) . These rendered images capture image statistics akin to those61 preserved in human peripheral vision , and resembling texture computation at the stage of area V2 , as62 argued in Freeman & Simoncelli ( 2011 ) ; Rosenholtz ( 2016 ) ; Wallis et al . ( 2019 ) .63 Our strategy is thus to compare in terms of generalization , robustness and bias these foveation-texture64 models to three other kinds of models . The first comparison model class – foveation-blur models –65 uses the same spatially-varying foveation operations but uses blur rather than texture based input.66 The second class – uniform-blur models – uses a blur operation uniformly over the input , with the67 level of blur set to match the perceptual compression rates of the foveation-texture nets . Finally , the68 last comparison model class is the reference , which has minimal distortion , and serves as a perceptual69 upper bound from which to assess the impact of these different first-stage transforms.70 Note that our approach is different from the one taken by Wang & Cottrell ( 2017 ) , who have built71 foveated models that fit results to human behavioural data like those of Larson & Loschky ( 2009 ) .72 Rather , our goal is to explore the emergent properties in CNNs with texture-based foveation on scene73 representation compared to their controls agnostic to any behavioural data or expected outcome.74 Naturally , the results of our experimental paradigm is symbiotic as it can shed light into both75 the importance of texture-based peripheral computation in humans , and could also suggest a new76 inductive bias for advanced machine perception in scenes.77 2 Perceptual Systems78 We define perceptual systems as two-stage models with an image transform ( stage 1 , f ( ◦ ) : RD →79 RD ) , that is relayed to a deep convolutional neural network ( stage 2 , g ( ◦ ) : RD → Rd ) . Note that the80 first transform stage is a fixed operation over the input image , while the second stage has learnable81 parameters . In general , the perceptual system S ( ◦ ) , with retinal image input I : RD is defined as:82 S ( I ) = g ( f ( I ) ) ( 1 ) Such two-stage models have been growing in popularity , and the reasons these models are designed to83 not be fully end-to-end differentiable is mainly to force one type of computation into the first-stage of a84 system such that the second-stage g ( ◦ ) must figure out how to capitalize on such forced transformation85 and thus assess its f ( ◦ ) representational consequences ( See Figure 2 ) . For example , Parthasarathy & 86 Simoncelli ( 2020 ) successfully imposed V1-like computation in stage 1 to explore the learned role87 of texture representation in later stages with a self-supervised objective , and Dapello et al . ( 2020 ) 88 found that fixing V1-like computation also at stage 1 aided adversarial robustness . At a higher level,89 our objective is similar where we would like to force a texture-based peripheral coding mechanism90 ( loosely inspired by V2 ; Ziemba et al. , 2016 ) at the first stage to check if the perceptual system ( now91 foveated ) will learn to pick-up on this newly made representation through g ( ◦ ) and make ‘ good ’ use92 of it potentially shedding light on the functionality hypothesis for machines and humans.93 2.1 Stage 1 : Image Transform94 To model the computations of a texture-based foveated visual system , we employed the model95 of Deza et al . ( 2019 ) ( henceforth Foveated-Texture Transform ) . This model is inspired by the metamer96 synthesis model of Freeman & Simoncelli ( 2011 ) , where new images are rendered to have locally97 matching texture statistics ( Portilla & Simoncelli , 2000 ; Balas et al. , 2009 ) in greater size pooling98 regions of the visual periphery with structural constraints . Analogously , the Deza et al . ( 2019 ) 99 Foveation Transform uses a foveated feed-forward style transfer ( Huang & Belongie , 2017 ) network100 to latently perturb the image in the direction of its locally matched texture ( see Figure 1 ) . Altogether,101 f : RD → RD is a convolutional auto-encoder that is non-foveated when the latent space is un-102 perturbed : f0 ( I ) = D ( E ( I ) ) , but foveated ( ◦Σ ) when the latent space is perturbed via localized style103 transfer : f∗ ( I ) = D ( EΣ ( I ) ) , for a given encoder-decoder ( E , D ) pair.104 Note that with proper calibration , the resulting distorted image can be a visual metamer ( for a human ) ,105 which is a carefully perturbed image perceptually indistinguishable from its reference image ( Freeman106 & Simoncelli , 2011 ; Rosenholtz et al. , 2012 ; Feather et al. , 2019 ; Vacher et al. , 2020 ) . However,107 importantly in the present work , we exaggerated the strength of these texture-driven distortions108 ( beyond the metameric boundary ) , as our aim here is to understand the implications of this kind109 of texturized peripheral input on later stage representations ( e.g . following a similar approach as110 Dapello et al . ( 2020 ) ) . By having an extreme manipulation , we reasoned this would accentuate the111 consequences of these distortions , making them more detectable in our subsequent experiments.112 2.2 Stage 2 : Convolutional Neural Network backbone113 The transformed images ( stage 1 ) are passed into a standard convolutional neural network architecture.114 Here we tested two different base architectures : AlexNet ( Krizhevsky et al. , 2012 ) , and ResNet18 ( He115 et al. , 2016 ) . The goal of running these experiments on two different hierarchically local architectures116 is to let us examine the consequences across all image transforms ( with our main focus towards117 texture-based foveation ) that are robust to these different network architectures . Further , this CNN118 backbone ( g : RD → Rd ) should not be viewed in the traditional way of an end-to-end input/output119 system where the input is the retinal image ( I ) , and the output is a one-hot vector encoding a d-class-120 label in Rd . Rather , the CNN ( g ) acts as a loose proxy of higher stages of visual processing ( as it121 receives input from f ) , analogous to the 2-stage model of Lindsey et al . ( 2019 ) .122 2.3 Critical Manipulations : Foveated vs Non-Foveated Perceptual Systems123 Now , we can define the first two of the four perceptual systems that will perform 20-way scene124 categorization : Foveation-Texture , receives an image input , applies the foveation-texture transform125 f∗ ( ◦ ) , and relays it through the CNN g ( ◦ ) . Similarly , Reference performs a non-foveated transform126 f0 ( ◦ ) , where images are sent through the same convolutional auto-encoder D ( E ( I ) ) of f∗ ( ◦ ) , but127 with the parameter that determines the degree of texture style transfer set to 0 – producing an upper-128 bounded , compressed and non-foveated reference image – then relayed through the CNN g ( ◦ ) . Both129 of these systems are depicted in Figure 2 ( A ) . As the foveation-texture model has less information130 from the input , relative to the reference networks , we next designed two further comparison models131 which have a comparable amount of information after the input stage , but with different amounts of132 blurring in the stage 1 operations . To create matched-resources systems , our broad approach was to133 use a Rate-Distortion ( RD ) optimization procedure ( Ballé et al. , 2016 ) to match information between134 the stage 1 operations , given the SSIM ( Wang et al. , 2004 ) image quality assessment ( IQA ) metric.135 Specifically , to create matched-resource Uniform-Blur , we identified the standard deviation of the136 Gaussian blurring kernel ( the ‘ distortion ’ D ) , such that we could render a perceptually resource-137 matched Gaussian blurred image – w.r.t Reference – that matches the perceptual transmission ‘ rate ’ 138 R of Foveation-Texture via the SSIM perceptual metric ( Wang et al. , 2004 ) . This procedure yields a139 model class with uniform blur across the image , but with matched stage 1 information content as the140 Foveation-Texture . And , to create matched-resource Foveation-Blur , we carried our this same RD141 optimization pipeline per eccentricity ring ( assuming homogeneity across pooling regions at the same142 eccentricity ) , thus finding a set of blurring coefficients that vary as a function of eccentricity . This143 procedures yielded a different matched-resource model class , this time with spatially-varying blur.144 Figure 3 ( B ) summarizes our solution to this problem . Details of the RD Optimization are presented145 in Appendix A.146 Ultimately , it is important to note that the selection of the perceptual metric ( SSIM in our case ) ,147 plays a role in this optimization procedure , and sets the context in which we can call a network148 “ resource-matched ” . We selected SSIM given its monotonic relationship of distortions to human149 perceptual judgements , symmetric upper-bounded nature , sensitivity to contrast , local structure and150 spatial frequency , and popularity in the Image Quality Assessment ( IQA ) community . However151 to anticipate any possible discrepancy in the interpretability of our future results , we additionally152 computed the Mean Square Error ( MSE ) , MS-SSIM , and 11 other IQA metrics as recently explored153 in Ding et al . ( 2020 ) to compare all other image transforms to the Reference on the testing set.154 Our logic is the following : if the MSE is greater ( ↑ ) for Foveation-Texture compared to Foveation-155 Blur and Uniform-Blur , then the current distortion levels place Foveation-Texture at a resource156 ‘ disadvantage ’ relative to the other transforms , and any interesting results would not only hold but157 also be strengthened . This same logic applies to the other IQA metrics contingent on their direction158 of greater distortion . Indeed , these patterns of results were evident across IQA metrics – except those159 tolerant to texture such as DISTS ( Ding et al. , 2020 ) – as shown in Table 1 , and Appendix C.160 3 Experiments161 Altogether , the 4 previously introduced perceptual systems162 help us answer three key questions that we should have163 in mind throughout the rest of the paper : 1 ) Foveation-164 Texture vs Reference will tell us how a texture-based165 foveation mechanism will compare to its perceptual upper-166 bound – shedding light into arguments about computa-167 tional efficiency . 2 ) Foveation-Texture vs Foveation-Blur168 will tell us if any potentially interesting pattern of results169 is due to the type/stage of foveation . This will help us170 measure the contributions of the adaptive texture coding171 vs adaptive gaussian blurring ; 3 ) Foveation-Texture vs172 Uniform-Blur will tell us how do these perceptual systems173 ( one foveated , and the other one not ) behave when allo-174 cated with a fixed number of perceptual resources under175 certain assumptions – potentially shedding light on why176 biological organisms like humans have foveated texture-177 based computation in the visual field instead of uniform178 spatial processing like modern machines.179 Dataset : All previously introduced models were trained180 to perform 20-way scene categorization . Scene categories181 were selected from the Places2 dataset ( Zhou et al. , 2017 ) ,182 and were re-partitioned into a new 4500 images per cate-183 gory for training , 250 per category for validation , and 250184 per category for testing . The categories included were:185 aquarium , badlands , bedroom , bridge , campus , corridor , forest path , highway , hospital , industrial186 area , japanese garden , kitchen , mansion , mountain , ocean , office , restaurant , skyscraper , train interior,187 waterfall . Samples of these scenes coupled with their image transforms can be seen in Figure 4.188 Networks : Training : Convolutional neural networks of the stage 2 of each perceptual system were189 trained which resulted in 40 image-transform based networks per architecture ( AlexNet/ResNet18 ) :190 10 Foveation-Texture , 10 Reference , 10 Uniform-Blur , 10 Foveation-Blur ; totalling 80 trained191 networks to compute relevant error bars shown in all figures ( standard deviations , not standard errors ) 192 and to reduce effects of randomness driven by the particular network initialization . All systems were193 paired such that their stage 2 architectures g ( ◦ ) started with the same random weight initialization194 prior to training . Testing : The networks of each perceptual system were tested on the same type of195 image distribution they were trained on . Learning Dynamics : Available in Appendix H.196 3.1 Texture-based foveation provides greater i.i.d . generalization than Blur-based foveation197 How well does the foveation-texture stage classify scene images ( i.i.d . generalization ) compared to198 the other matched-resource models that use blurring and the reference ? The results can be seen in199 Figure 5 . Each bars ’ height reflects overall accuracy for each of the 10 neural network backbone200 runs ( g ( ◦ ) ) per system , with a square marker at the top indicating the i.i.d . accuracy . We found that201 Foveation-Texture had similar i.i.d . performance to the Reference – which is the the undistorted202 perceptual upper bound , and greater performance than both Uniform-Blur and Foveation-Blur . Thus203 the compression induced by foveated-texture generally maintains scene category information.204 We next performed a contrived experiment where we tested how well each perceptual system could205 classify the stage 1 outputs of the other models . For example , we showed a set of foveated blurred206 images to a network trained on foveated texture images . This experiment is in essence a test of207 out-of-distribution ( o.o.d . ) generalization . The results of these tests are also shown in Figure 5 . For208 each model , the classification accuracy for the inputs from the other stage 1 images is indicated by209 the height of the different colored diamonds , where the color corresponds to the stage 1 operation.210 This experiment yielded a rather complex set of patterns , that even differed depending on the211 architecture ( AlexNet vs ResNet18 as g ( ◦ ) ) . Generally , the Foveation-Texture model had a similar212 profile of generalization as the Reference model . However , the networks trained with different types213 of blur ( Uniform-Blur & Foveated-Blur ) in some cases showed very high o.o.d . generalization –214 though once again this is contingent on g ( ◦ ) .215 Unraveling the underlying causes to understand this last set of results sets the stage for our experiments216 in the rest of this section . So far it seems like Foveation-Texture has learned to properly capitalize the217 texture information in the periphery and still out-perform all other matched-resource systems even if218 heavily penalized under several IQA metrics ( Table 1 ) – highlighting the critical differences in texture219 vs blur for scene processing . As for the interaction of Uniform-Blur with g ( ◦ ) , is is likely that the220 residual connections are counter-productive to o.o.d . generalization ( or it has overfit ) . Interestingly,221 humans have a combination of texture and adaptive-gaussian based peripheral computation ( Ehinger222 & Rosenholtz , 2016 ) , so future work should look into the effects of continual learning , joint-training223 or a combined image transform ( Texture + Blur ) to merge gains of both i.i.d and o.o.d generalization.224 3.2 Texture-based foveated systems preserve greater high-spatial frequency sensitivity225 We next examined whether the learned feature representations of these models are more reliant on low226 or high pass spatial frequency information . To do so , we filtered the testing image set at multiple levels227 to create both high pass and low pass frequency stimuli and assessed scene-classification performance228 over these images for all models , as shown in Figure 6 . Low pass frequency stimuli were rendered by229 convolving a Gaussian filter of standard deviation σ = [ 0 , 1 , 3 , 5 , 7 , 10 , 15 , 40 ] pixels on the foveation230 transform ( f0 , f̂0 , f∗ , f̂∗ ) outputs . Similarly , the high pass stimuli was computed by subtracting the231 reference image from its low pass filtered version with σ = [ ∞ , 3 , 1.5 , 1 , 0.7 , 0.55 , 0.45 , 0.4 ] pixels232 and adding a residual . These are the same values used in the experiments of Geirhos et al . ( 2019 ) .233 We found that Foveation-Texture and Reference trained networks were more sensitive to High234 Pass Frequency information , while Foveation-Blur and Uniform-Blur were selective to Low Pass235 Frequency stimuli . Although one may naively assume that this is an expected result – as both236 Foveation-Blur and Uniform-Blur networks are exposed to a blurring procedure – it is important to237 note that : 1 ) the foveal resolution has been preserved between Foveation-Texture and Foveation-Blur238 ( See Fig . 4 ) , thus high spatial frequency sensitivity could have still predominated in Foveation-Blur239 but it did not ( though see Fig . 6 A2/B2 where these high pass Gabors are still learned , implying240 that higher layers in g ( ◦ ) overshadow their computation ) ; and 2 ) Foveation-Texture could have241 also learned to develop low spatial frequency sensitivity given the crowding/texture-like peripheral242 distortion , but this was not the case ( likely due to the weight sharing constraint embedded in the243 CNN architecture Elsayed et al. , 2020 ) . Finally , the robustness to low-pass filtering of Foveation-Blur244 suggests that foveation via adaptive gaussian blurring may implicitly contribute to scale-invariance as245 also shown in Poggio et al . ( 2014 ) ; Cheung et al . ( 2017 ) ; Han et al . ( 2020 ) .246 3.3 Texture-based foveation develops greater robustness to occlusion247 We next examined how all perceptual systems could classify scene information under conditions248 of visual field loss , either from left to right ( left2right ) , top to bottom ( top2bottom ) , center part of249 the image ( scotoma ) , or the periphery ( glaucoma ) . This manipulation lets us examine the degree250 to which learned representations relying on different parts of the image to classify scene categories.251 Critically , here we apply the occlusion after the stage 1 operation . The results are shown in Figure 7.252 Overall we found that , across all types of occlusion the Foveation-Texture modules have greater ro-253 bustness to occlusion than both the Foveation-Blur and Uniform-Blur models . Further , the Foveation-254 Texture models have nearly equivalent performance to the Reference . In contrast , both models with255 blurring , whether uniformly or in a spatially-varying way , were far worse at classifying scenes under256 conditions of visual field loss . These results highlight that the texture-based information content257 captured by the foveation-texture nets preserves scene category content in dramatically different way258 than simple lower-resolution sampling – perhaps using the texture-bias ( Geirhos et al. , 2019 ) in their259 favor ; as humans too use texture as their classification strategy for scenes ( Renninger & Malik , 2004 ) .260 In addition , the Foveation-Texture model is not overfitting . As recent work has suggested an Accuracy261 vs Robustness trade-off where networks trained to outperform under the i.i.d . generalization condition262 will do worse under other perceptual tasks – mainly adversarial ( Zhang et al. , 2019 ) – we did not263 observe such trade-off and a greater accuracy did not imply lower robustness to occlusion.264 3.4 Foveated systems learn a stronger center image bias than non-foveated systems265 It is possible that foveated systems weight visual information strongly in the foveal region than the266 peripheral region as hinted by our occlusion results ( the different rate of decay for the accuracy curves267 in the Scotoma and Glaucoma conditions ) . To resolve this question , we conducted an experiment268 where we created a windowed cue-conflict stimuli where we re-rendered our set of testing images269 with one image category in the fovea , and another one in the periphery ( all aligned with a different270 class systematically ; ex : aquarium with badlands ) . We also had an additional condition where the271 conflicting cue was now square-like and uniformly and randomly paired with a conflicting scene272 class and more finely sampled . We then systematically varied the fovea-periphery visual area ratios273 & re-examined classification accuracy for both the foveal and peripheral scenes ( Figure 8 ) .274 We found that the Foveation-Texture and Foveation-Blur transform imposed the networks g ( ◦ ) to275 learn to weigh information in the center of the image stronger than Reference & Uniform-Blur for276 scene categorization . A qualitative way of seeing this foveal-bias is by checking the foveal/peripheral277 ratio where these two accuracy lines cross . The more leftward the cross-over point ( ⊗ ) , the higher the278 foveal bias ( highlighted through the vertical bars ) . This result was unexpected as we initially predicted279 that g ( ◦ ) would weigh the peripheral information stronger as it has been implicitly regularized through280 a distortion . However this was not the case and our findings are similar to Wang & Cottrell ( 2017 ) 281 who showed this foveal bias on a foveated system with adaptive blur with a dual-stream neural282 network . Thus , these results indicate that the spatially varying computation from center to periphery283 is mainly responsible for the development of a center image bias even with a weight sharing constraint.284 Furthermore , it is possible that one of the functions of any spatially-varying coding mechanisms285 in the visual field is to enforce the perceptual system to attend on the foveal region – avoiding the286 shortcut of learning to attend the entire visual field if unnecessary ( Geirhos et al. , 2020 ) .287 4 Discussion288 The present work was designed to probe the impact of foveated texture-based input representations in289 machine vision systems . To do this we specifically compared the learned perceptual signatures in290 the second-stage of visual processing across a set of of networks trained on other image transforms.291 We found that when comparing Foveation-Texture to their matched-resource models that differed in292 computation : Foveation-Blur ( foveated w/ adaptive gaussian blur ) and Uniform-Blur ( non-foveated293 w/ uniform blur ) – that peripheral texture encoding did lead to specific representational signatures,294 particularly greater i.i.d generalization , preservation of high-spatial frequency sensitivity , and ro-295 bustness to occlusion – even as high as its perceptual upper bound ( Reference ) . We also found that296 foveation ( in general ) seems to induce a focusing mechanism , servicing the foveal/central region –297 whereas neither a perceptually upper-bounded system ( Reference ) or a non-foveated compressed298 system ( Uniform-Blur ) did not develop as strongly.299 The particular consequences of our foveation stage raises interesting future directions about what300 computational advantages could arise when trained on object categorization ( Pramod et al. , 2018 ) 301 coupled with eye-movements ( Akbas & Eckstein , 2017 ; Deza et al. , 2017 ) , as objects are typically302 centered in view and have different hierarchical/compositional priors than scenes ( Zhou et al . ( 2014 ) ; 303 Deza et al . ( 2020 ) ) in addition to different processing mechanisms ( Renninger & Malik ( 2004 ) ; 304 Ehinger & Rosenholtz ( 2016 ) ) . We are currently exploring the impact of these foveated texture-based305 representational signatures on shape vs texture bias for object recognition similar to Geirhos et al.306 ( 2019 ) and Hermann et al . ( 2020 ) , and assessing their interaction with scene representation.307 Further , a future direction is investigating the effects of texture-based foveation to adversarial308 robustness . Motivated by the recent work of Dapello et al . ( 2020 ) which has shown promise of309 adversarial robustness via enforcing stochasticity and V1-like computation by obeying the Nyquist310 sampling frequency of these filters w.r.t the image ( Serre et al. , 2007 ) in addition to a natural gamut of311 orientations and frequencies as studied in De Valois et al . ( 1982 ) , it raises the question of how much312 we can further push for robustness in hybrid perceptual systems like these , drawing on even more313 biological mechanisms . Works such as Luo et al . ( 2015 ) and recently Reddy et al . ( 2020 ) ; Kiritani & 314 Ono ( 2020 ) have already taken steps in this direction by coupling fixations with a spatially-varying315 retina . However , the representational impact of texture-based foveation on adversarial robustness,316 and its symbiotic implication for human vision still remains an open question.317 References318 Akbas , E. and Eckstein , M. P. Object detection through search with a foveated visual system . PLoS319 computational biology , 13 ( 10 ) : e1005743 , 2017.320 Balas , B. , Nakano , L. , and Rosenholtz , R. A summary-statistic representation in peripheral vision321 explains visual crowding . Journal of vision , 9 ( 12 ) :13–13 , 2009.322 Ballé , J. , Laparra , V. , and Simoncelli , E. P. End-to-end optimized image compression . arXiv preprint323 arXiv:1611.01704 , 2016.324 Cheung , B. , Weiss , E. , and Olshausen , B . Emergence of foveal image sampling from learning to325 attend in visual scenes . International Conference on Learning Representations ( ICLR ) , 2017.326 Dapello , J. , Marques , T. , Schrimpf , M. , Geiger , F. , Cox , D. D. , and DiCarlo , J. J. Simulating a327 primary visual cortex at the front of cnns improves robustness to image perturbations . BioRxiv,328 2020.329 Daucé , E. , Albiges , P. , and Perrinet , L. U . A dual foveal-peripheral visual processing model330 implements efficient saccade selection . Journal of Vision , 20 ( 8 ) :22–22 , 2020.331 De Valois , R. L. , Yund , E. W. , and Hepler , N. The orientation and direction selectivity of cells in332 macaque visual cortex . Vision research , 22 ( 5 ) :531–544 , 1982.333 Deza , A. and Eckstein , M. Can peripheral representations improve clutter metrics on complex scenes ? 334 In Advances in Neural Information Processing Systems , pp . 2847–2855 , 2016.335 Deza , A. , Peters , J. R. , Taylor , G. S. , Surana , A. , and Eckstein , M. P. Attention allocation aid for336 visual search . In Proceedings of the 2017 CHI Conference on Human Factors in Computing337 Systems , pp . 220–231 , 2017.338 Deza , A. , Jonnalagadda , A. , and Eckstein , M. P. Towards metamerism via foveated style transfer . In339 International Conference on Learning Representations , 2019 . URL https : //openreview.net/340 forum ? id=BJzbG20cFQ.341 Deza , A. , Liao , Q. , Banburski , A. , and Poggio , T. Hierarchically local tasks and deep convolutional342 networks . CBMM Memo , 2020.343 Ding , K. , Ma , K. , Wang , S. , and Simoncelli , E. Image quality assessment : Unifying structure and344 texture similarity . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2020.345 Ding , K. , Ma , K. , Wang , S. , and Simoncelli , E. P. Comparison of Image Quality Models for346 Optimization of Image Processing Systems . arXiv e-prints , art . arXiv:2005.01338 , May 2020.347 Doerig , A. , Bornet , A. , Choung , O. H. , and Herzog , M. H. Crowding reveals fundamental differences348 in local vs. global processing in humans and machines . bioRxiv , 2019a . doi : 10.1101/744268.349 URL https : //www.biorxiv.org/content/early/2019/08/23/744268.350 Doerig , A. , Bornet , A. , Rosenholtz , R. , Francis , G. , Clarke , A. M. , and Herzog , M. H. Beyond351 bouma ’ s window : How to explain global aspects of crowding ? PLoS computational biology , 15352 ( 5 ) : e1006580 , 2019b.353 Eckstein , M. P. Visual search : A retrospective . Journal of vision , 11 ( 5 ) :14–14 , 2011.354 Eckstein , M. P. , Koehler , K. , Welbourne , L. E. , and Akbas , E. Humans , but not deep neural networks,355 often miss giant targets in scenes . Current Biology , 27 ( 18 ) :2827–2832 , 2017.356 Ehinger , K. A. and Rosenholtz , R. A general account of peripheral encoding also predicts scene357 perception performance . Journal of Vision , 16 ( 2 ) :13–13 , 2016.358 Elsayed , G. , Kornblith , S. , and Le , Q. V. Saccader : Improving accuracy of hard attention models for359 vision . In Advances in Neural Information Processing Systems , pp . 700–712 , 2019.360 Elsayed , G. , Ramachandran , P. , Shlens , J. , and Kornblith , S. Revisiting spatial invariance with361 low-rank local connectivity . In International Conference on Machine Learning , pp . 2868–2879.362 PMLR , 2020.363 Feather , J. , Durango , A. , Gonzalez , R. , and McDermott , J. Metamers of neural networks reveal diver-364 gence from human perceptual systems . In Wallach , H. , Larochelle , H. , Beygelzimer , A. , d'Alché-365 Buc , F. , Fox , E. , and Garnett , R . ( eds . ) , Advances in Neural Information Processing Systems366 32 , pp . 10078–10089 . Curran Associates , Inc. , 2019 . URL http : //papers.nips.cc/paper/367 9198-metamers-of-neural-networks-reveal-divergence-from-human-perceptual-systems.368 pdf.369 Freeman , J. and Simoncelli , E. Metamers of the ventral stream . Nature neuroscience , 14 ( 9 ) :370 1195–1201 , 2011.371 Fridman , L. , Jenik , B. , Keshvari , S. , Reimer , B. , Zetzsche , C. , and Rosenholtz , R. Sideeye : A genera-372 tive neural network based simulator of human peripheral vision . arXiv preprint arXiv:1706.04568,373 2017.374 Gatys , L. A. , Ecker , A. S. , and Bethge , M. Image style transfer using convolutional neural networks.375 In Proceedings of the IEEE conference on computer vision and pattern recognition , pp . 2414–2423,376 2016.377 Geirhos , R. , Temme , C. R. , Rauber , J. , Schütt , H. H. , Bethge , M. , and Wichmann , F. A. Generalisation378 in humans and deep neural networks . In Advances in Neural Information Processing Systems , pp.379 7538–7550 , 2018.380 Geirhos , R. , Rubisch , P. , Michaelis , C. , Bethge , M. , Wichmann , F. A. , and Brendel , W. Imagenet-381 trained CNNs are biased towards texture ; increasing shape bias improves accuracy and robustness.382 In International Conference on Learning Representations , 2019 . URL https : //openreview.383 net/forum ? id=Bygh9j09KX.384 Geirhos , R. , Jacobsen , J.-H. , Michaelis , C. , Zemel , R. , Brendel , W. , Bethge , M. , and Wichmann , F. A.385 Shortcut learning in deep neural networks . arXiv preprint arXiv:2004.07780 , 2020.386 Geisler , W. S. and Perry , J. S. Real-time foveated multiresolution system for low-bandwidth video387 communication . In Human vision and electronic imaging III , volume 3299 , pp . 294–305 . Interna-388 tional Society for Optics and Photonics , 1998.389 Geisler , W. S. , Perry , J. S. , and Najemnik , J . Visual search : The role of peripheral information390 measured using gaze-contingent displays . Journal of Vision , 6 ( 9 ) :1–1 , 2006.391 Han , Y. , Roig , G. , Geiger , G. , and Poggio , T. Scale and translation-invariance for novel objects in392 human vision . Scientific Reports , 10 ( 1 ) :1–13 , 2020.393 He , K. , Zhang , X. , Ren , S. , and Sun , J . Deep residual learning for image recognition . In Proceedings394 of the IEEE conference on computer vision and pattern recognition , pp . 770–778 , 2016.395 Hermann , K. L. , Chen , T. , and Kornblith , S. The origins and prevalence of texture bias in convolutional396 neural networks . Neural Information Processing Systems , 2020.397 Huang , X. and Belongie , S. Arbitrary style transfer in real-time with adaptive instance normalization.398 In Proceedings of the IEEE International Conference on Computer Vision , pp . 1501–1510 , 2017.399 Kaplanyan , A. S. , Sochenov , A. , Leimkühler , T. , Okunev , M. , Goodall , T. , and Rufo , G. Deepfovea:400 neural reconstruction for foveated rendering and video compression using learned statistics of401 natural videos . ACM Transactions on Graphics ( TOG ) , 38 ( 6 ) :1–13 , 2019.402 Kiritani , T. and Ono , K. Recurrent attention model with log-polar mapping is robust against403 adversarial attacks . arXiv preprint arXiv:2002.05388 , 2020.404 Krizhevsky , A. , Sutskever , I. , and Hinton , G. E. Imagenet classification with deep convolutional405 neural networks . In Advances in neural information processing systems , pp . 1097–1105 , 2012.406 Land , M. F. and Nilsson , D.-E . Animal eyes . Oxford University Press , 2012.407 Laparra , V. , Ballé , J. , Berardino , A. , and Simoncelli , E. P. Perceptual image quality assessment using408 a normalized laplacian pyramid . Electronic Imaging , 2016 ( 16 ) :1–6 , 2016.409 Larson , A. M. and Loschky , L. C. The contributions of central versus peripheral vision to scene gist410 recognition . Journal of Vision , 9 ( 10 ) :6–6 , 2009.411 Larson , E. C. and Chandler , D. M. Most apparent distortion : full-reference image quality assessment412 and the role of strategy . Journal of electronic imaging , 19 ( 1 ) :011006 , 2010.413 LeCun , Y. , Bengio , Y. , and Hinton , G. Deep learning . nature , 521 ( 7553 ) :436 , 2015.414 Levi , D. M. Visual crowding . Current Biology , 21 ( 18 ) : R678–R679 , 2011.415 Lindsey , J. , Ocko , S. A. , Ganguli , S. , and Deny , S. The effects of neural resource constraints on416 early visual representations . In International Conference on Learning Representations , 2019 . URL417 https : //openreview.net/forum ? id=S1xq3oR5tQ.418 Loschky , L. C. , Szaffarczyk , S. , Beugnet , C. , Young , M. E. , and Boucart , M. The contributions of419 central and peripheral vision to scene-gist recognition with a 180 visual field . Journal of Vision , 19420 ( 5 ) :15–15 , 2019.421 Luo , Y. , Boix , X. , Roig , G. , Poggio , T. , and Zhao , Q. Foveation-based mechanisms alleviate422 adversarial examples . arXiv preprint arXiv:1511.06292 , 2015.423 Malkin , E. , Deza , A. , and tomaso a poggio . { CUDA } -optimized real-time rendering of a foveated424 visual system . In NeurIPS 2020 Workshop SVRHM , 2020 . URL https : //openreview.net/425 forum ? id=ZMsqkUadtZ7.426 Mnih , V. , Heess , N. , Graves , A. , et al . Recurrent models of visual attention . In Advances in neural427 information processing systems , pp . 2204–2212 , 2014.428 Parthasarathy , N. and Simoncelli , E. P. Self-supervised learning of a biologically-inspired visual429 texture model . arXiv preprint arXiv:2006.16976 , 2020.430 Patney , A. , Salvi , M. , Kim , J. , Kaplanyan , A. , Wyman , C. , Benty , N. , Luebke , D. , and Lefohn , A.431 Towards foveated rendering for gaze-tracked virtual reality . ACM Transactions on Graphics ( TOG ) ,432 35 ( 6 ) :179 , 2016.433 Pelli , D. G. Crowding : A cortical constraint on object recognition . Current opinion in neurobiology,434 18 ( 4 ) :445–451 , 2008.435 Poggio , T. , Mutch , J. , and Isik , L. Computational role of eccentricity dependent cortical magnification.436 arXiv preprint arXiv:1406.1770 , 2014.437 Portilla , J. and Simoncelli , E. P. A parametric texture model based on joint statistics of complex438 wavelet coefficients . International journal of computer vision , 40 ( 1 ) :49–70 , 2000.439 Pramod , R. T. , Katti , H. , and Arun , S. P. Human peripheral blur is optimal for object recognition.440 arXiv preprint arXiv:1807.08476 , 2018.441 Reddy , M. V. , Banburski , A. , Pant , N. , and Poggio , T. Biologically inspired mechanisms for442 adversarial robustness . arXiv preprint arXiv:2006.16427 , 2020.443 Renninger , L. W. and Malik , J . When is scene identification just texture recognition ? Vision research,444 44 ( 19 ) :2301–2311 , 2004.445 Rosenholtz , R. Capabilities and limitations of peripheral vision . Annual Review of Vision Science , 2:446 437–457 , 2016.447 Rosenholtz , R. , Huang , J. , Raj , A. , Balas , B. J. , and Ilie , L. A summary statistic representation in448 peripheral vision explains visual search . Journal of vision , 12 ( 4 ) :14–14 , 2012.449 Russakovsky , O. , Deng , J. , Su , H. , Krause , J. , Satheesh , S. , Ma , S. , Huang , Z. , Karpathy , A. , Khosla,450 A. , Bernstein , M. , et al . Imagenet large scale visual recognition challenge . International journal of451 computer vision , 115 ( 3 ) :211–252 , 2015.452 Serre , T. , Wolf , L. , Bileschi , S. , Riesenhuber , M. , and Poggio , T. Robust object recognition with453 cortex-like mechanisms . IEEE transactions on pattern analysis and machine intelligence , 29 ( 3 ) :454 411–426 , 2007.455 Sheikh , H. R. and Bovik , A. C. Image information and visual quality . IEEE Transactions on image456 processing , 15 ( 2 ) :430–444 , 2006.457 Shumikhin , M. M. A. Quantitative measures of crowding susceptibility in peripheral vision for large458 datasets . PhD thesis , Massachusetts Institute of Technology , 2020.459 Vacher , J. , Davila , A. , Kohn , A. , and Coen-Cagli , R. Texture interpolation for probing visual460 perception . Advances in Neural Information Processing Systems , 33 , 2020.461 Wallis , T. S. , Funke , C. M. , Ecker , A. S. , Gatys , L. A. , Wichmann , F. A. , and Bethge , M. Image462 content is more important than bouma ’ s law for scene metamers . eLife , 8 : e42512 , 2019.463 Wallis , T. S. A. , Funke , C. M. , Ecker , A. S. , Gatys , L. A. , Wichmann , F. A. , and Bethge , M. A464 parametric texture model based on deep convolutional features closely matches texture appearance465 for humans . Journal of Vision , 17 ( 12 ) , Oct 2017. doi : 10.1167/17.12.5 . URL http : //doi.org/466 10.1167/17.12.5.467 Wang , P. and Cottrell , G. W. Central and peripheral vision for scene recognition : A neurocomputa-468 tional modeling exploration . Journal of vision , 17 ( 4 ) :9–9 , 2017.469 Wang , Z. and Simoncelli , E. P. Translation insensitive image similarity in complex wavelet domain.470 In Proceedings. ( ICASSP ’ 05 ) . IEEE International Conference on Acoustics , Speech , and Signal471 Processing , 2005. , volume 2 , pp . ii–573 . IEEE , 2005.472 Wang , Z. , Simoncelli , E. P. , and Bovik , A. C. Multiscale structural similarity for image quality473 assessment . In The Thrity-Seventh Asilomar Conference on Signals , Systems & Computers , 2003,474 volume 2 , pp . 1398–1402 . Ieee , 2003.475 Wang , Z. , Bovik , A. C. , Sheikh , H. R. , and Simoncelli , E. P. Image quality assessment : from error476 visibility to structural similarity . IEEE transactions on image processing , 13 ( 4 ) :600–612 , 2004.477 Wu , K. , Wu , E. , and Kreiman , G. Learning scene gist with convolutional neural networks to improve478 object recognition . In 2018 52nd Annual Conference on Information Sciences and Systems ( CISS ) ,479 pp . 1–6 . IEEE , 2018.480 Xue , W. , Zhang , L. , Mou , X. , and Bovik , A. C. Gradient magnitude similarity deviation : A highly481 efficient perceptual image quality index . IEEE Transactions on Image Processing , 23 ( 2 ) :684–695,482 2013.483 Zhang , H. , Yu , Y. , Jiao , J. , Xing , E. , El Ghaoui , L. , and Jordan , M. Theoretically principled trade-484 off between robustness and accuracy . In International Conference on Machine Learning , pp.485 7472–7482 . PMLR , 2019.486 Zhang , L. , Zhang , L. , Mou , X. , and Zhang , D. Fsim : A feature similarity index for image quality487 assessment . IEEE transactions on Image Processing , 20 ( 8 ) :2378–2386 , 2011.488 Zhang , L. , Shen , Y. , and Li , H. Vsi : A visual saliency-induced index for perceptual image quality489 assessment . IEEE Transactions on Image processing , 23 ( 10 ) :4270–4281 , 2014.490 Zhang , R. , Isola , P. , Efros , A . A. , Shechtman , E. , and Wang , O . The unreasonable effectiveness of491 deep features as a perceptual metric . In Proceedings of the IEEE conference on computer vision492 and pattern recognition , pp . 586–595 , 2018.493 Zhou , B. , Khosla , A. , Lapedriza , A. , Oliva , A. , and Torralba , A . Object detectors emerge in deep494 scene cnns , 2014.495 Zhou , B. , Lapedriza , A. , Khosla , A. , Oliva , A. , and Torralba , A . Places : A 10 million image database496 for scene recognition . IEEE transactions on pattern analysis and machine intelligence , 40 ( 6 ) :497 1452–1464 , 2017.498 Ziemba , C. M. , Freeman , J. , Movshon , J . A. , and Simoncelli , E. P. Selectivity and tolerance for visual499 texture in macaque v2 . Proceedings of the National Academy of Sciences , 113 ( 22 ) : E3140–E3149,500 2016.501 Checklist502 1 . For all authors ... 503 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s504 contributions and scope ? [ Yes ] We have focused our experiments on implementing505 a two-stage model that has a texture-based foveation transform and compared it to a506 reference model ( a perceptual upper bound ) , and two matched resource systems : one507 foveated with blur and another one uniformly blurred.508 ( b ) Did you describe the limitations of your work ? [ Yes ] At the end of each Experiments509 Sub-Section we provide a mini-discussion of our work and how it fits or does not fit the510 literature . Mainly we provide limitations in the Discussion at the end ( See Section 4 ) 511 ( c ) Did you discuss any potential negative societal impacts of your work ? [ No ] To our512 knowledge , there are none.513 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to514 them ? [ Yes ] 515 2 . If you are including theoretical results ... 516 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] We include517 only one supplementary theoretical result and proof in the AppendixB518 ( b ) Did you include complete proofs of all theoretical results ? [ Yes ] See above.519 3 . If you ran experiments ... 520 ( a ) Did you include the code , data , and instructions needed to reproduce the main exper-521 imental results ( either in the supplemental material or as a URL ) ? [ Yes ] See Supple-522 mentary Material ( that provides access to a URL ) 523 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they were524 chosen ) ? [ Yes ] These are reported brielfy in Section 3 , and in more detail through-out525 the Appendix.526 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-527 ments multiple times ) ? [ Yes ] All experiments were ran with paired initial noise seeds528 to control for matched initial conditions derived from SGD ( though the order in which529 the networks were exposed to images was different ) . All errorbars report 1 standard530 deviation , and these can be seen throughout Sections 3.2,3.3,3.4531 ( d ) Did you include the total amount of compute and the type of resources used ( e.g.,532 type of GPUs , internal cluster , or cloud provider ) ? [ Yes ] These are specified in the533 Appendix.534 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 535 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] We use a re-partition536 of the Places2 Dataset which is cited.537 ( b ) Did you mention the license of the assets ? [ No ] Given that to our knowledge the538 Places2 dataset is widely known and free to use.539 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ No ] 540 As everything in the Supplementary Material/URL has been created/derived by us.541 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re542 using/curating ? [ N/A ] We did not run any experiments with humans.543 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable544 information or offensive content ? [ N/A ] We did not run any experiments with humans,545 and the scene classes we used were all publicly known and non-offensive places : e.g.546 ocean.547 5 . If you used crowdsourcing or conducted research with human subjects ... 548 ( a ) Did you include the full text of instructions given to participants and screenshots , if549 applicable ? [ N/A ] No human subjects were used.550 ( b ) Did you describe any potential participant risks , with links to Institutional Review551 Board ( IRB ) approvals , if applicable ? [ N/A ] No human subjects were used.552 ( c ) Did you include the estimated hourly wage paid to participants and the total amount553 spent on participant compensation ? [ N/A ] No human subjects were used.554

Summary: I thank the authors for a thoroughly written paper studying an important question for both machine learning and neuroscience. The authors propose a biologically inspired modification to CNN architectures by introducing foveation. Several thorough experiments are performed to assess the benefit of foveation with reasonable control transformations. The proposed modifications seem novel and discuss the relevant prior work in this domain. Appreciate the detailed discussion of all implementational specifics, I'm fairly confident about the correctness of the experiments performed and results presented.