Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a transformer architecture which incorporates human-like peripheral vision. They modify the multi-head attention with peripheral attention, which consists of content and position attention. Content attention is the same as scaled dot product attention. Position attention is learned using a neural network which takes a fixed relative position embedding as input and is independent of the input image. The position-based attention learned by the proposed approach seems to divide the image into distinct regions similar to peripheral vision. Finally, they present quantitative and qualitative results to validate the effectiveness of their approach. 
strengths_and_weaknesses: Strengths: 
* The paper aims to design neural networks which see in a similar way to humans. The direction and the motivations are novel.
* Extensive qualitative results help in understanding the proposed approach.

Weakness:
* The motivation behind using neural networks for learning position-based attention is not explained clearly.
* It is possibly difficult to disentengle the performance gained by the particular change (like the peripheral encodings in thie case). There is huge variation withing different implementations of ViT itself. For instance, the implementation of ViT in [A], significantly outperforms the original model. ViT implementation in [A] without autoencoder fine tuning achieves 82% top1 on imageNet, bringing over 5% improvement on the original model (76.5%). It is hence natural to question approaches which bring minor gains (less than a percent). The change can clearly be achived by simpler engineering corrections.  
* A more solid approach would be to use the optimally trained ViT and then base the experiments on it. 

[A] He et al. Masked Autoencoders Are Scalable Vision Learners. CVPR 2022
questions: * What is the formulation of neighbour function, and how does it impact the learned position-based attention?

* There should be a baseline which uses fixed position embedding varying by layer to highlight the benefits of learning position embedding using NN. Did the authors try such a baseline? Would appreciate a discussion on that in the author's response.

* Can you take the ViT implementation of [A] and apply the peripheral positional encoding on it?  
limitations: Yes, the limitations are addressed adequately.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper takes inspiration from the foveated nature of human vision, in which visual scenes are processed with decreasing resolution and increased information compression as eccentricity to the center of the fovea increases.
It notes that recent vision transformers (ViTs), which use multi-headed self-attention, require a lot of data to learn useful patterns of self-attention at different layers of the feature hierarchy.
While various methods have been proposed to address this, often relying in one way or another on re-asserting locality into the processing e.g. explicitly via convolution or by introducing pyramidal structure or more structured attention, the paper here proposes adding a position-based attention mechanism into a standard data-efficient image transformer (DeIT).
In the MPA, learned position-based attention maps are combined with the (original) learned content-based attention maps to produce a  "Multi-head Peripheral Attention" (MPA) map.
Experiments on ImageNet-1k show that their Peripheral ViT (PerViT) model which uses MPA uses this additional structure in the attention to perform better at the classification task than without, to a level on par with recent pyramidal architectures, across three different approximate model sizes.
Further experiments look at the qualitative nature of the learned peripheral maps, the trade off of influence between the position-based and content-based maps, the locality of the attention at different layers, the benefit of careful initialization, and an ablation of the various model additions. 
strengths_and_weaknesses: [Paper Strengths]
* the proposed multi-head peripheral attention approach seem to be new, technically sound, and interesting 
* the experiments are well-motivated, with carefully chosen baselines and good performance
* code is available with promise of reproducible experiments (although I did not try)

[Paper Weaknesses]
Overall I think the paper presents an interesting approach backed up by some good empirical results, but there are a few reasons why I remain borderline:
* I think that the paper somewhat oversimplifies peripheral vision and oversells its involvement in the design of the model. There is only one reference to peripheral vision in the main text, and it is a bit reductive -- I encourage the authors to explore more recent literature on peripheral vision, e.g. https://visxvision.files.wordpress.com/2017/08/ruth_peripheral.pdf in order to better relate their work to peripheral vision literature. I think it is fair to be inspired by the notion of peripheral vision, but using definitive terms like "peripheral inductive bias" (L56) may be misleading. Claims like L47 "According to recent study on inner workings of vision transformers [11, 15, 36, 42, 51], their behavior is in fact closely related to how the peripheral vision functions" are not supported by the cited works (which make no mention of peripheral vision), and are arguably more speculation than scientific "fact". I believe the paper will be better served by toning down this aspect; it is ok to be inspired loosely by a notion of how peripheral vision might work, and this model might indeed model peripheral vision to a better extent than previous works, but as far as I can tell, we do not know and cannot be certain.
* Secondly, although the experiments are reasonably extensive, a few experiments are missing which I think are important for a reader to understand the value MPA. (i) One of the main motivations to impose structure into the attention (I think) is to improve training sample efficiency. Have any experiments been conducted to explore this (e.g. performance vs training set proportion on ImageNet)? (ii) My understanding is that of the three components (phi_p, C-Stem, CPE), phi_p is the main novelty of this work, while C-Stem and CPE have been adopted from prior work. I would like to see the ablation on the -S and -M models to understand the impact with/without only phi_p in those cases. (iii) Experiments on other datasets (e.g. ones which DeIT supports already like iNaturalist) would help to further validate the usefulness of the proposed method.
* Although I found the paper to have good general coverage of recent work in this area, I found the related work section to be somewhat brief, and would appreciate a more explicit explanation of why the proposed work differs from previous efforts to improve the efficacy of the self-attention mechanism for vision transformers. 

EDIT: post-rebuttal, upgraded Rating from 5 to 7 (accept).

[Minor Points / typos]
* L9 "large-scale ImageNet dataset" -- "large-scale" here may mislead readers to think of the full ImageNet dataset; in reality the 1K dataset is used in the paper
* some references are not ordered numerically e.g. L30, 53, 82
* L56 - we proposes 
* Fig 3 - bases on -> is based on the 
* Sec 3 - please define new variables when they are introduced (e.g. D_h), for better clarity. 
* Eq 3 has an unwanted comma
* L119-21 - break up the sentence for better clarity
* L123 - Eucliden
* L134 non-linearlity
* L142 the they
* please check grammar in 4.1
* L252 - double negative should be single negative?
* Please fix table ordering in 4.2 so it follows text
questions: In addition to the points mentioned above,
1. I think Figure 1 could be explained more clearly to inform a new reader - could you explain how does this learned attention map relate to peripheral vision? 
2. How was the D_r chosen - and what is the impact of varying it?
3. L142 what is K?
4. L179 as far as I'm aware, "cylindrical" is not a standard term. What is meant by the term here? Do you mean "columnar", in the sense as described by e.g. https://arxiv.org/pdf/2102.12122.pdf ? Columnar is strictly better, because cylindrical implies a circle in some plane?
5. I'm confused why w_r is appearing in Eq 9, when it doesn't appear to be part of the peripheral projection? (based on my reading of L122-L132). Is this just describing all of the parameter settings for all experiments?
6. What does classification of visual regions add in Fig 5? It seems somewhat arbitrary, since "angle" doesn't have an obvious notion in an image taken from an unknown camera. In Supp Sec B, there is some explanation of visual fields broken down by angle, but is it fair to assume that images correspond to a particular FOV? Peripheral vision only makes sense in the context of a physical eyeball which has a focal point; how does that relate to an image from an unknown camera? An alternative might be to make a scatter plot of average attention distances vs layers, in a similar vein to Fig 3 of [36]? 
7. What, if any, is the distinction between the proposed Peripheral Positional Encoding (PPE) (Table 2) and the used Convolutional Positional Encoding (CPE) (Table 1)? 
8. Table 5 of the DeIT paper https://arxiv.org/pdf/2012.12877.pdf shows that using the distillation method during training yields stronger performance (e.g. DeIT-B gives 83.4 Top-1, and higher when trained for longer). Although this is not necessarily a fair comparison, I think it could be mentioned at least. The abstract and contributions claim unconditionally (L11, L69) that the PerViT yields "state of the art performance in image classification task" but I think this is not technically true and ought to be revised?
9. parts of S3.1 could be rewritten to improve readability, in particular the explanation of Peripheral Projections from L141. "By referring neighboring relative distances around the keys" does not make sense to me - could it be rewritten? 
10. In Eq 8 - the PP function does not appear to be explicitly defined until the supplementary. Perhaps some of this section could be represented graphically, similar to Fig 1 in [50], to improve clarity?
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed a novel visual transformer model that incorporates the inductive bias of peripheral vision into the relative positional encoding. Specifically, the authors define a multi-head peripheral attention (MPA) module, where content-based and position-based attention scores are first calculated independently and then combined through an element-wise product. The design of the position-based attention allows pairs of locations ($q$,$k$) with the same Euclidean distance to have similar attention scores. The peripheral projection introduces the diversity, while the peripheral initialization ensembles the increasing spatial receptive fields. The results show better performance on ImageNet classification task than models with similar capacity. The ablation studies suggest the major contribution of the performance improvement comes from the position-based peripheral attention in the MPA.
strengths_and_weaknesses: Strength:
- This paper proposed a concise idea to incorporate peripheral vision into the transformer-based computer vision models. The definition of relative positional encoding is straightforward to apply and understand and is well-aligned with the distance-based sampling in the human visual system.
- The performance evaluation and ablation studies convincingly suggest that peripheral attention is the critical module that contributes to the improvement, and it has a reasonable and interpretable behavior (e.g., the position bias is mainly seen in early layers; larger models have farther peripheral information).

Weakness:
- The evaluation of ImageNet classification may not be the best task to show the benefit of adding a peripheral mechanism to the computer vision model. It would be great if the authors could demonstrate if this mechanism significantly increases the efficiency and robustness of the model.
- The presentation of the methods and results can be improved for clarity.
questions: Major:
- The "peripheral projections" part in Section 3.1 seems interesting and important, but it's hard to fully understand its intuition and computation from the description. Does this peripheral projection allow the attention to be calculated not from the key location itself, but from a small surrounding area (local context) around the key location? Does $PP$ in equation 8 stand for the computation of $\Phi_p$ in equation 7? It will be helpful if the authors may rephrase this part in a clearer and more intuitive way, or maybe use a visual illustration if it's helpful. I want to understand the "peripheral projection" better since it has the most contribution according to the results in Table 4.
- How is the $\mathcal{N}(\cdot)$ function defined in equation 7? Is it fixed for different layers/heads?
- Why there is a vertical discontinuity in the attention patterns after peripheral projections (Fig. 2, middle row)?
- From Table 4, it seems that adding ML (multi-layer design) to Euc doesn't help but adding $\mathcal{N}$ (peripheral projection) to Euc+ML improves the performance significantly. Is the multi-layer design an essential component or do we only need the diversity from $\mathcal{N}$?


Minor:
- The tables are not referenced in the order as they appear
- Some discussion of how peripheral vision affects model behavior in a human-like way may be inspiring when sketching future directions.
- It may be worth including a few early works that design biologically-inspired peripheral vision models with CNNs by polar transformation or using non-uniform sampling to generate foveated images, for example:
    1. Wang, P., & Cottrell, G. W. (2017). Central and peripheral vision for scene recognition: A neurocomputational modeling exploration. Journal of vision, 17(4), 9-9.
    2. Reddy, M. V., Banburski, A., Pant, N., & Poggio, T. (2020). Biologically inspired mechanisms for adversarial robustness. arXiv preprint arXiv:2006
    3. Jonnalagadda, A., Wang, W., & Eckstein, M. P. (2021). Foveater: Foveated transformer for image classification. arXiv preprint arXiv:2105.14173.


limitations: No negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors introduce a novel transformer biologically inspired transformer architecture. They introduce a change on the attention mechanism which enable models to split the visual field into different peripheral regions. Authors compare their proposal to the baseline on classification, showing improvement.

strengths_and_weaknesses: **Strengths**:
- S1. The paper proposes an interesting and useful addition to standard transformer attention. State-of-the-art methods typically lack of the ability to partition the visual field in peripheral regions, which can help focusing on the most important content for the model and make training more efficient.

- S2. The paper is well written and motivated. I think authors described well the goal of the paper and the model, as well as illustrated well the idea with plenty of useful figures.

- S3.  The ablation study in the paper is very complete, going over all posible design decisions that shaped the final architecture and model. I specially like the results in Table 1, where the reader can easily understand the importance of the different elements. 

** Weaknesses**: 
In my opinion the main weakness of the paper is the evaluation. I think authors fail to show that their proposed method is better than other proposals in the transformer area. My main concerns are:

- W1. Authors only evaluate on one ImageNet, and do not provide additional results on other datasets. In my opinion for the paper to be convincing about the usefulness of the methods, author should provide results on other datasets as well.

- W2. Focal-S performs better than the proposed model with similar number of flops and model size. As the ImageNet evaluation is the only data point we have to evaluate the model performance, it's hard to convince the reader with this evidence that the model is pushing the state-of-the-art. 

- W3. Given the main innovation of the model, I believe that an object centric dataset such as ImageNet is probably not the best dataset for evaluation. I think the proposed model is good at focusing to the different objects in the image in an efficient manner, while in ImageNet typically a single object is present.

- W4. Following up with W3, I think this model could actually shine more on tasks such as segmentation, where fine detail of the input image is important to produce a quality output.

- W5. Have the authors considered evaluating the segmentation masks generated by the attention? (Figure 6)



------------------------------
* After rebuttal*: The author's response addressed most of my concerns and I updated my review score recommending acceptance.
questions: My concerns are listed in the weaknesses section. My main questions are:

- Have the authors considered other tasks/datasets for evaluation? In my opinion the evaluation is very limited.

- How would the authors argue that the paper is still contributing to pushing the SOTA if Focal-S performs better than the proposed model.

- Have the authors evaluated the quality of the segmentation in segmentation data?
limitations: Yes, I think the discussion of limitations is correct.
review_title: null

Meta-review: metareview: The paper proposes a transformer architecture that models human-like peripheral vision. Experiment results show it achieves good performance. All the reviewers consider the paper above the bar. They like the novelty and the strong empirical performance. The AC finds no reason to object.
confidence: Certain
Decision: Accept