Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: I thank the authors for a thoroughly written paper studying an important question for both machine learning and neuroscience. The authors propose a biologically inspired modification to CNN architectures by introducing foveation. Several thorough experiments are performed to assess the benefit of foveation with reasonable control transformations. The proposed modifications seem novel and discuss the relevant prior work in this domain. Appreciate the detailed discussion of all implementational specifics, I'm fairly confident about the correctness of the experiments performed and results presented.

# Some concerns I have about the claims:
I agree with some of R1's comments, the claims seem to be overstated in my opinion. In Fig. 7, it looks like Foveation-net significantly outperforms other baselines only in pre-foveation scotoma transform. This is likely because compared to Standard-Net, Foveation-Net has larger unoccluded information due to the nature of the foveation transform. In the post-foveation scotoma condition wherein the amount of unoccluded area is matched, Standard-Nets outperform Foveation-Net. This is fine, but I didn't read any discussion on this observation in the submission. I see that the authors have attempted to address this question in other comments below. In summary, I doubt whether foveation truly provides more robustness to occlusions in the presented sense.

Fig. 9C does not seem like sufficient evidence to claim that foveation promotes shape bias over texture bias.

As mentioned earlier, find the topic studied to be a very interesting and an important one. I'm certain this work will stimulate interesting discussions and future work to better understand the functional significance of foveation. The analyses are thorough and the reported results seem to be accurate. However I doubt whether the proposed claims are justified by the analyses. The figures are great, particularly the ones that describe procedures like foveation, scotoma, etc. The ones analyzing model performance seemed a little shrunk and I had to zoom in to glean the details. I would suggest moving some/all figures that discuss the foveation/occlusion procedures (such as in Figs 7&8) to SI to give more room for the quantitative analysis figures.

# Questions to authors:
1. What are your thoughts on applying foveation-like masking throughout the network's activations and not just at the input stage?
2. I'm very interested to know whether the proposed mechanism can buy more performance on Stylized ImageNet, and robustness to adversarial perturbations.
3. The authors might be interested to explore how well foveated model performs on datasets like CIFAR-10C and ImageNet-C.
review_title: Interesting study to assess the functional benefit of foveated perception

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################
Summary:

In this paper, authors study the functional advantages of a foveal transform of visual inputs. It is nicely introduced with a very comprehensive review of the literature. The method introduces a 2 two stage model of the visual system, where the first stage corresponds to the (fixed and non adaptive) foveation stage and the second stage to the higher level processing, typically associated with the categorisation operated in the ventral stream of the visual pathway. This second stage will be implemented by existing CNN architectures (AlexNet and ResNet) which are re-learned on the transformed inputs. To control for the functional consequences of the foveated processing, the first stage can also be a single isotropic blurring of the image. Both alternatives are manipulated such that their distortion (as computed with a SSIM measure) are equally balanced, leading to 1 standard mapping and three proposal retinal transformations :  Standard-NEt (unmatched) and Foveation-Net, Matched-Net and Ada-Gauss-Net (matched). Results show that for both perceptual systems which are foveated, "Foveation-Net has the highest i.i.d generalization while Ada-Gauss- Net has the greatest o.o.d generalization". Second result is that foveated processing allowed a better robustness to occlusions and third result is that such networks reproduce behavioural results of a Window Cue-Conflict. Last results propose to study that foveation introduces a focusing strategy, and keep high-french information on the fovea - which seem less striking results.

Overall, the paper is original, technically sound and very well supported by experiments. However, there are some concerns that I highlight below.

##########################################################################
Concern:

The  paper is very dense and wants to say too much while perhaps losing on the main point: the foveal transform In particular, there is one point about foveation and another about metamerism (and crowding). While the first is studied in depth, the second is not studied fully, or at least not parametrically. This hinders the comprehension of the mechanisms that may be at play behind the functions of foveation.  Controlling for the amount of "metameric distortion" at different eccentricities both in models and humans would be a novel contribution to the field.

More generally, the paper seems to explore many computational alternatives but perhaps misses a key property of the visual system, that is that RFs size grow in size AND density as a function of eccentricity - it seems the metameric distortions adds a novel ingredient as it tunes the precision of textural information as a function of eccentricity. taken together, this factors could simplify the presentation of the paper by presenting different hypothesis for the evolution of RF size and density and in particular to show why a given magnification factor may be better than other. I consider the iid vs ood and occlusion results to be the most convincing, but at the end of reading the paper, I lack a comprehension of why we have such results, and what hyperparameters play the most important role.

More particularly, for the robustness to occlusions, the results are in my understanding given of the systems learned in the non-occluded case. Would you predict that results would generalise if you re-learn weights from the occluded images? The Window Cue-Conflict results are less convincing  as the effects are rather small. Perhaps using different hyper parameters (magnification ratio, metameric distortion) would help? The conclusion "that foveation (in general) seems to induce a focusing mechanism (...) while the texture-based computation still preserves high spatial-frequency selectivity" seem to be a mechanical consequences of the foveation transform (you focus on a point while preserving cone density in the fovea) - did I miss something?  Finally, I know the problem of page budget, but I had to go page 16 of the SM to have the definition of the dataset and get eg the value of the chance level - a synthetic overview on p.5 would be welcome.

Concerning the form of the paper, clarity could be improved in some sections. Some sentences (e.g. "same pattern of results are evident for ResNet18" (p. 5) "an unexpected victory for Foveation-Nets" (p.5)  "is still quite impressive" (p.6), ...) are more reminiscent to stunts from the PR department than is necessary in a scientific paper. Please avoid.
Last, missing the code 
##########################################################################

minor
p.4 « guassian » _

review_title: clearly defined model; thorough experimental analysis; clarity could be improved

final_score: rating: 7: Good paper, accept
review_content: The paper explores how preprocessing an image with a foveated texture-based rendering - where content in the periphery is transformed in a lossy manner but still perceptually equivalent - affects the training of a downstream neural network.  It is shown that when equated to other lossy image transforms in terms of overall rate, that the foveated system shows better robustness to occlusion, generalization, and preservation of high spatial-frequency content.

The many figures are difficult to wade through because the are so compressed and cluttered with many labels and diagrams.  I would recommend figuring out a way to prune these down so they are more readable and convey just central punchline that you are trying to show.

review_title: A nice demonstration of the advantages of foveated texture-based image preprocessing

final_score: rating: 3: Clear rejection
review_content: # Update after discussion
The discussion has reinforced my concerns. I cannot fully follow the logic of the paper and am not convinced the controls are useful. Therefore  I stand by my original assessment.


# Original Review

The paper proposes to use texturization of the periphery in images as a proprocessing step for scene classification and claims "greater iid generalization, high spatial frequency sensitivity and robustness to occlusion" for such models. The model is essentially a concatenation of the synthesis procedure for metameric images by Deza and colleagues (2019) followed by AlexNet as a classification head.


Strengths:
+ Tries to address an interesting question: why does human vision treat the periphery mainly as texture?
+ In-depth investigation of the model and three controls


Weaknesses:
- Unclear if the premise of the paper is well-conceived
- No evidence provided that "foveated" images are actually metameric
- "Matched-resource" controls are somewhat questionable
- Definition of "foveation" seems strange


Details on major issues (all of which would need to be addressed in a convincing manner for my score to change):

My main issue with the paper is that I don't understand its logic. The results by Rosenholtz, Simoncelli, Wallis and others show that some information about the image is discarded in human peripheral vision and, to some extent, peripheral vision "cares" only about the texture statistics of an image rather than the detailed composition. The hypothesis is that this is the same effect that has been described as crowding.

Now, the paper seems to put forward the hypothesis that distorting (texturizing) images in a way that they remain indistinguishable (metameric) for humans somehow enhances their discriminability. I don't understand what's the reasoning behind this logic. Why would throwing away information and distorting an image be useful? The original image is by definition part of the equivalence class of metameric, so why should a distorted image somehow be easier to classify by a neural network than the original image? Both images are perceived in the same way by humans. It seems like a strange proposal to me, but it's possible that I'm missing something. I'd like the authors to explain their reasoning better.

Having said that, even if we put these fundamental concerns aside, there are a number of practical issues:

(1) The presented images don't appear metameric to me, and the paper does not provide any psychophysical data showing that they are. For instance, for the three images labeled "Metameric" in Fig. 5A, I have a very hard time believing that they are metameric to human observers. Could the authors provide psychophysical evidence that this is the case?

(2) I am not sure about the purpose of the "matched-resource" control and why SSIM is the right metric to use here. If it's about information content in the image, I think it should be evaluated in terms of mutual information between the image and the labels (which is hard); if it's about information available to human observers, then a perceptual metric is the right approach, but SSIM is a very poor one that doesn't correspond well with human perception. Therefore, I am skeptical that these baselines are really useful.

(3) The claim “greater iid generalization, high spatial frequency sensitivity and robustness to occlusion emerged exclusively in our foveated texture-based models” seems to be an overstatement. There appears to be basically no difference between foveated and standard net. In all Figures (5–8) the differences are mostly within the range of 1 SD and the order of the two models changes sometimes. The only effect that might be significant is the difference in the scotoma condition for intermediate to high occlusion conditions. The only robust difference is between standard+foveation and MatchedNet+AdaGauss, where the latter seem to carry much less information (see points (1) and (2) above).



Minor comments

- Sec 3.4: The connection between Gaussian blur and scale invariance is not clear
- Sec 3.4: The following statement is trivial, since the images were low-pass filtered: “We found that Foveation-Nets and Standard-Nets were more sensitive to High Pass Frequency information, while Ada-Gauss-Nets and Matched-Nets were sensitive to Low Pass Frequency stimuli”
- Fig 8: It is unclear how it argues for shape bias.
review_title: Not sure if the premise of the paper is well-conceived

final_score: rating: 7: Good paper, accept
review_content: The authors compared Foveation-Net with three other stage-one visual systems: Standard-Net with non-foveated transform; Ada-Gauss-Net as a matched-resource control for Foveation-Net (to test the contribution of the foveation transformation comparing to other spatially-varying processing); Matched-Net as a matched-resource control for Standard-Net. Specifically, the authors did four sets of experiments on these visual systems to evaluate their generalizability, robustness, image-region bias, and spatial frequency sensitivity.

Overall, the methods are solid and clearly stated. The results suggest some perceptual advantages and specific representational properties yielded by texture-based foveation transformation. 

##### Specific comments and questions:
1. It is not clearly stated how to read the plot in Fig. 5B. And why using a 20-way scene categorization task to evaluate the model performance?
2. The results in experiment 3 suggest Foveation-Net and Ada-Gauss-Net has higher foveal bias. Is this simply because the effective sampling density is higher in the image center (as suggested by Fig. 5A) for these two visual systems? Then are these two visual systems more computationally efficient in general?
3. In section 3.4, the authors argued that the Foveation-Net may enforce a shape-bias since "these Spatial Frequency curves show similar trend as SIN". Are these curves refer to the ones in Fig. 8B? But by visually inspecting this figure, it seems the trend of Standard-Net is quite closed to Foveation-Net. Please let me know if I misunderstood this argument.
4. What would be the effect on model performance if the fixation is not in the center for Foveation-Net and Ada-Gauss-Net?
review_title: The paper provides a comprehensive study to evaluate how a texture-based foveated inductive bias influences visual perception from several perspectives.

Meta-review: This paper explores the use of a texture-based foveation stage in
scene categorization.  They show that the foveated system shows
presevation of high spatial-frequency information relative to other
matched transformations.

This paper engendered a lot of discussion and had a wide range of
ratings.  An extra review was requested that fell intermediate between
the high and low scores.  Generally reviewers agree that the question
is interesting but that the paper does not clearly elucidate the logic
of the paper.  That is, Reviewer 1, and another reviewer in
discussion, had issues with the logic of the paper. I also found the
motivation behind the paper, difficult to understand at first.  I had
to find the Rosenholtz paper to understand why the authors were
considering this particular representation. This should be better
explained in the paper - the main points of the paper should be
understandable by itself.


There is also concern that the claims are not validated by
the presented results.  For example, the authors claim that their
foveated network were more robust to occlusion but Reviewer 5 points
out that this is likely due to the foveation-nets having more
unoccluded information.


On the positive side, Reviewer 4 points out that the experiments are
extensive and several reviewers commented that they trust that the experiments
were done correctly.  Reviewer 2, 4 and 5 all mention that there are too many
results reported and recommend paring down to the most important
results.

In my view the paper is right at the border of acceptance.  Acceptance/rejection will
depend on capacity limits and balancing areas.  I recommend that if accepted, 
or resubmitted to another conference, that the results be pared down, and 
more space be devoted to explaining the question(s) and why they are 
interesting and relevant and why the comparison networks allow the questions 
to be answered.  

Originality - High
Quality - High 
Clarity - Could be improved
Significance - Could be better articulated and is hard to assess as is.
Pros: - interesting idea, many well done experiments
Cons: - claims not well validated, clarity could be improved to	emphasize significance
Other: paper too dense - should be pared down to improve clarity

Decision: Reject