Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors address hypotheses about the nature of peripheral biological vision, specifically the Foveation-Texture theory stating that peripheral representations in primate vision are not simply blurred (subsampled) but rather extract more informative texture statistics. The empirical study comparing the proposed model vs several baseline model corroborates their proposal.

main_review: This work addresses an important question: Is foveated vision an artifact of constrained resources or does it endow primate vision with an advantage in terms of learning more efficient representations?
Their empirical studies corroborate the notion that texture-foveation mechanism offers greater generalization, preserves more high spatial frequency information in periphery and develops greater robustness to occlusion than the blur-based model of peripheral representations.
The paper is well written and the claims are convincingly supported by empirical tests. 
However, the line of argumentation seems a bit confounded: while the authors state in the introduction that they focus on representational consequences of foveation, they are less interested in general such consequences of any foveation model as compared to non-foveated alternatives, but rather differences between specifically texture-foveation model vs other foveation models. It would be interesting to also address a model where the texture representation is used without the foveation model for testing these factors independently.
limitations_and_societal_impact: The authors of the study seems to have made a peculiar choice that might be undermining their conclusions: the first (texture processing) system that imitates foveation (and texture-level processing) actually acts as an encoder-decoder that generates processed images for the second stage, implemented as a standard CNN. However, a  more natural for both ML and for modeling a biological vision system would be to actually use the learnt latent codes of the autoencoder as the input to the CNN. So it would be interested to see how (if at all) the reaults would be affected by such processing pipeline.

Impact: better understanding of human vision with implications for medicine and novel inductive biases for computer vision models.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper aims to explore the effects on representation that emerge when a CNN is equipped with a foveated visual system. The authors build a model whereby images are artificially rendered into a form that captures foveation which are then fed into standard CNN architectures. Two different types of foveation are considered: one where the periphery is blurred (greater blurring as a function of eccentricity), and another where texture statistics are preserved over local regions which increase in size as a function of eccentricity.  These are compared against each other and two baselines:

- Reference: images are transformed by pushing them through a convolutional AE
- Foveation-texture: images are transformed by pushing them through a convolutional AE with a foveated texture style transfer
- Uniform-blur: each image is blurred uniformly, with the amount of blurring set such that the SSIM between the blurred image the its corresponding reference was the same as the SSIM between the corresponding foveation-texture image and reference.
- Foveation-blur: as above, but with blur additionally increasing with eccentricity

Clearly the reference images contain more information than the foveation-texture images; the SSIM matching on the is done to resource match the blurred images for comparison to foveation-texture, and the reference serves as an upper bound. Experiments are performed on a scene classification task. 
main_review: Originality
-----------

The paper is original in the sense that it's trying to give a new understanding of the potential benefits of a foveated visual system. I believe it is adequately cited and is complementary in its aims to prior work.

Quality
-------

Clearly a lot of work has gone into this paper (just from the the shear number of experiments performed for example), however I have a number of questions/issues which I'll describe below. It is of course possible that I've misinterpreted some parts of the paper, and if so, I look forward to a discussion with the authors during that phase of the review process.

__The model:__

Two-stage models like the one proposed (and e.g. Lindsay (2019) as mentioned, but also recent work looking at pre-filtering the network input with Gabor filters or similar) have become popular ways of exploring how different inductive biases affect representations in artificial visual systems. To the best of my knowledge however, all the prior work has involved using an input that is matched with the inductive biases built into the network - that is the spatial resolution of the input is constant and thus matched to the convolutions that follow. In this work, the foveated networks effectively vary spatial resolution across spatial position, so it is not in anyway clear that applying fixed convolutions (with respect to the space between taps) across this input in anyway makes sense. Have the authors considered this? Why wasn't an alternative (perhaps multiple streams of fixed spatial resolution in the 1st stage? or maybe not using shared weights? or...).

With respect to the resource matching, it isn't clear to me why matching using a perceptual metric is the correct approach. Surely the aim should be to match the amount of information that is transmitted to the second stage? in which case why wouldn't you e.g. match based on entropy?


__The results:__

The authors have chosen to do their experiments with a variant of the Places2 dataset for a scene categorization task, but there isn't really any discussion of why this was chosen in the paper, or of any of the potential pitfalls of using this dataset in the evaluation that is performed. By its very nature a scene categorization task requires information from across the entire image, at least in expectation over the entire dataset; for example, many (the majority?) of the images have very human biases which affect their overall composition - e.g.:

- photos with a person in the foreground, often near the center of the image, where the presence of a person is unrelated to the scene (e.g. ocean)
- images where if you relied just on the center you would be unlikely to predict the 'correct' scene (the badlands image in the paper is a good representative example)

These factors need to be considered when analysing results of experiments, because they will inherently cause a bias. In this case before doing any experiments, one might reasonably expect that if the periphery of the image is corrupted in some way performance would be lower. Further, because early work on scene classification (e.g. using things like bags of visual words with quantised dense SIFT descriptors, which were basically capturing local texture) actually worked rather well on these tasks, one would inherently expect that texture is going to be an important factor. With this in mind, I question if many of the results in the paper actually tell you more about the nature of the data rather than about foveation?

I've written comments/questions on each experimental section below:

_"Texture-based foveation provides greater i.i.d generalisation than Blur-based foveation"_

- The above claim could possibly be inferred from the results, but it could just be that you've shown that the task you've chosen is more easily solved with high-frequency/texture information as I suggested previously? Do the differences between uniform and foviated -blur just indicate that for this task there is _relatively more_ information to solve the task in the periphery (which shouldn't really be surprising given the nature of the dataset)?
- The claim that foveation-texture has "similar i.i.d performance" to reference seems like something of an overstatement; it's closer than blur, but still seems worse (particularly for the resnet) unless I'm missing something?
- Following on, isn't it obvious that texture is likely to be more informative given what is already known about this type of dataset?
- The reference images applied to foveated-texture showing good generalisation seems intuitive, however the uniform blur differences across networks are odd. When I looked at this I the question that arose is "why is alexnet so good (particularly w.r.t reference - you'd expect foviated-blur to be better)?"; the authors clearly thought that problem was instead with ResNet (which I'd argue has more intuitive performance with respect to the ordering of each input type) - I wonder if the authors could elaborate on this?

_"Texture-based foveated systems preserve greater high-spatial frequency sensitivity"_ 

- I don't understand the argument in note 1: yes, the foveal bit is the same, but the texture model clearly has more high frequencies, which for this task are more likely to be important. Point 2 again just suggests that higher frequencies are more important for this task.

_"Texture-based foveation develops greater robustness to occlusion"_ 

- So the takeaway is again that the solution to the task is dominated by higher frequency information, and clearly there is relevant high-frequency information in the periphery that helps significantly? Again, given the data is this really surprising?

_"Foveated systems learn a stronger center image bias than non-foveated systems"_

- I'm completely failing to understand what is surprising about this (or why the authors hypothesised it would be different); surely any learning system given a mixture of true and slightly corrupted data is going to have a preference for weighting the uncorrupted data higher?


Clarity
-------

From a technical perspective I believe enough information is given to be able to reproduce the experiments if one were minded. I did however struggle with understanding the motivation of the work (largely with respect to the model choices as outlined above). I also felt that the introduction was not sufficiently self-contained - for example the reader needs to be fully aware of the contents Rosenholtz (2016) to really understand some of the specific motivation for the _foveation-texture_ model (I had to read it; perhaps it is "well known" in the core vision sciences community, but this is a paper submitted to NeurIPS where the audience is much broader (particularly with respect to people in the computer vision community, as well as those more generally interested in understanding how different biases affect representation). I would suggest that this bit of the paper needs rewriting to make it much more self contained.


Significance
------------

Commenting on the significance is tough: having read the paper, do I know things that I didn't before? yes. Are they useful? that's completely unclear to me. As I've outlined above it's not clear to me that the model itself makes sense, or that the results actually tell us anything other than that the data used to train/evaluate the models had an inherent 'human' bias which can be exploited by foveation.
limitations_and_societal_impact: Limitations are not adequately addressed as I've described above. It's probably fair to say that there are no major potential negative societal impacts, although I'd note that the energy cost in training all of these models must have been quite large (in terms of CO2 equivalent).
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this manuscript the authors test how DNNs trained on classifying scene categories react to texture based foveation, which they compare to undisturbed images, a foveated blur and a uniform blur matched in their SSIM to the original image. In overall accuracy the texture model performs close to the performance of the undistorted images and clearly better than the other two distortions. Going beyond accuracy the authors then test the effects of the input distortions on processing filtered images, occlusion, and cue conflicts between center and periphery.

main_review: Overall, I am torn on this one. On the one hand, the authors did a lot of work and I don’t have many technical issues with their work. On the other hand, I do not see at all what the relevance of the performed experiments is or what could be learned from them. The texture based foveation the authors apply appears quite different from the way we think the human visual system implements this, which limits any interpretation towards human comparisons and the observed effects of the foveation seem rather trivial. This leaves me with the question: yes, but why?

To elaborate on the human comparison problem: This kind of peripheral texturization of images is a method for testing human peripheral perception, which is thought not to differentiate these texturized images from each other due to computing summary statistics at midlevel processing steps. Using such distorted images as input to a non-foveated processing system really does not capture the human foveated processing well in my opinion. Also this implementation provides none of the advantages of foveated processing like reduced computational power requirements for the periphery.

And on the observations not being particularly interesting:

Figure 6: Networks trained with blurred inputs react more to lower frequencies. Although, the authors dismiss this as a naive idea (in 3.2), I still find this unsurprising. Even if some higher frequency information in available for the foveated blur, it certainly sees much more low frequency content.

Figure 7: In principle, better performance for occlusions is an independent observation, but after observing a generally better performance of the Foveation-texture model this mostly looks like this advantage is simply preserved across occlusions.

Figure 8: Models trained with more information in the center weigh central information slightly stronger. That is definitely expected and if anything the effect is surprisingly small due to the non-foveated processing in the models.

limitations_and_societal_impact: The authors do not comment on their limitations and societal impact. Besides the sampling biases of the used databases I don’t think there are important societal impacts associated with the content of this paper though.
review_title: null

Meta-review: This paper received one positive review and two negative reviews (including one borderline reject). The positive reviewer did not engage in follow-up discussion which is based only on the two negative reviewers. There seems to have been some difficulty in communicating between the reviewers and the authors. As they stated in their final comments, I think the authors are right "perhaps there is a confusion on the goal (rather than the method) of our paper?". I agree that the reviewers understand what the methods is. However, it is pretty clear that they do not see the benefit of doing so either from an ML or neuroscience perspective. Even the positive reviewer calls the validity of the approach into question. Unfortunately it seems that no amount of revision will address this issue. For all these reasons, the AC recommends this paper being rejected.
Decision: Reject