Peripheral Vision Transformer
Authors: Juhong Min
Yucheng Zhao
Chong Luo
Minsu Cho
Venue: NeurIPS 2022 Accept (2022)
Keywords: Vision transformers
Peripheral vision
Image recognition
Image classification
Inductive bias

Abstract: Human vision possesses a special type of visual processing systems called peripheral vision. Partitioning the entire visual field into multiple contour regions based on the distance to the center of our gaze, the peripheral vision provides us the ability to perceive various visual features at different regions. In this work, we take a biologically inspired approach and explore to model peripheral vision in deep neural networks for visual recognition. We propose to incorporate peripheral position encoding to the multi-head self-attention layers to let the network learn to partition the visual field into diverse peripheral regions given training data. We evaluate the proposed network, dubbed PerViT, on ImageNet-1K and systematically investigate the inner workings of the model for machine perception, showing that the network learns to perceive visual data similarly to the way that human vision does. The performance improvements in image classification over the baselines across different model sizes demonstrate the efficacy of the proposed method.

Peripheral Vision Transformer
Juhong Min
Yucheng Zhao 2.3
Chong Luo²
Minsu Cho
PPhang University of Science and Technology (POSTECH)
Microsoff Research Asia (MSRA)
University of Science and Technology of China (USTC)
Abstract
Human vision possesses special type of visual processing systems called periph- eral vision. Partitioning the entire visual field into multiple contour regions based the distance the center of our gaze, the peripheral vision provides the ability to perceive various visual features at different regions. In this work, we take biologically inspired approach and explore model peripheral vision in deep neural networks for visual recognition. We propose to incorporate peripheral position encoding the multi-head self- aff-aattenion layers let the network learn to partition the visual field into diverse peripheral regions given training data. We evaluate the proposed network, dubbed PerViT, ImageNet- K and systematically investigate the inner workings of the model for machine perception, showing that the network learns to perceive visual data similarly the way that human vision does. The performance improvements in image classification over the baselines across different model sizes demonstrate the efficacy of the proposed method.
Introduction
For the last ten years, convolution has been dominant fea- ture transformation in neural networks for visual recognition due to its superiority in modelling spatial configurations of images [21, 32, 34]. Despite the efficacy in learning visual patterns, the local and stationary nature of convolutional ker- nels limited the maximum extent of representation ability in flexible processing, e.g., dynamic transformations with global receptive fields. Originally devised for natural lan- guage processing (NLP), self-attention [63] shed a light this direction; equipped with adaptive input processing and the ability capture long-range interactions, it has emerged as an alternative feature transform for computer vision, being widely adopted core building block [18]. The stand-alone self-attention models, e.g., ViT [18], however, demand signif- icantly more training data [57] for competitive performance with its convolutional counterparts [6, 25, 27, 72] since they miss certain desirable property which convolution possesses, e.g., locality. These inherent pros and cons of convolution and self-attention encourage recent researches toward combi- nations of both so to enjoy the best of the both worlds but which one suits the best for effective visual processing is yet controversial in literature [8, 9, 10, 12, 35, 37, 38, 40, 41, 49, 50, 59, 60, 62, 67, 69, 71, 73, 77].
Central
Para contral
Mid peripheral
peripheral
Figure 1: This work explores blend- ing human peripheral vision (top) with attention based neural networks (bottom) for visual recognition.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Unlike the dominant visual feature transformations in machine vision, human vision possesses a special type of visual processing systems called peripheral vision [36]; it partitions the entire visual field into multiple contour regions based on the distances to the center of a gaze where each region identifies different visual aspects, As seen in Fig. 1, we have high-resolution processing near the center of our gaze, i.e., central and para-central regions, to identify highly-detailed visual elements such as geometric shapes, and low-level details. For the regions more distant from the gaze, i.e., mid and far peripheral regions, the resolution decreases to recognize abstract visual features such as motion, and high-level contexts. This systematic strategy enables us to effectively perceive important details within small fraction (1%) of the visual field while minimizing unnecessary processing of background clutter in the rest (99%), thus facilitating efficient visual processing for human brain.
According to recent study on inner workings of vision transformers [12, 18, 49, 58, 711, their behaviors are related to the aforementioned visual processing strategies in the following respect; the attention maps of early layers are learned to locally capture fine-grained geometric details at central regions while those of later layers perform global attentions to identify coarse-grained semantics and contexts from the whole visual field, covering peripheral regions. This finding reveals that imitating biological designs may potentially help in modelling an effective machine vision, and also support recent approaches towards hybrid [5, 10, 12, 24, 39] of convolution and self-attention beyond stand-alon aanod visual processing to take advantages of the two different perception strategies: fine-grained tie-grained/local and coarse- coarse-grainee grained/global, similarly to the human visual processing as in Fig. 1.
In this work, we take a biologically inspired approach and propose to inject the peripheral inductive biases' to deep neural networks for image recognition. We propose to incorporate peripheral attention mechanism to the multi-head self-attention [63] to let the network learn to partition the visual field into diverse peripheral regions given training data where each region captures different visual features. We experimentally show that the proposed network models effective visual periphery for reliable visual recognition. Our main contributions can be summarized as follows:
This work explores to narrow the gap between human and machine vision by injecting peripheral inductive biases to self-attention layers, and presents new form of feature transformation named Multi-heat Peripheral Attention (MPA).
Based on the MPA, we introduce Peripheral Vision Transformer (PerViT), and systemati- cally study the inner workings of ViT by qualitatively and quantitatively analyzing its learned attentions, which reveal that the network learns to perceive visual elements similarly to the way that human vision does without any special supervisions.
The performance improvements in image classification over columnar Transformer baselines, egg., DeiT, across different model sizes demonstrate the efficacy of the proposed approach.
2 Related Work
Feature transformations in computer vision. With notable success in NLP, Transformers [14, 63] introduced paradigm shift in computer vision [3, 4, 7, 18, 30, 50, 55, 56, 58, 59, 62]. Despite their generalization capability, pure Vision Transformers [18] require extensive amount of training data to capture spatial layout of images due to lack of certain desirable property, e,g., locality. This encouraged many recent ViT work to incorporate local inductive biases via distillation of convo- lutional biases [58], local self-attention [35, 40, 73], hybridization [10, 12, 37], and augmenting convolutions [8, 60, 67, 69], all of which convey a unified message: "Despite high generalizibility of aelf-attentions, a sufficient amount of convolutional processing must be incorporated to capture the spatial configurations of images for reliable visual processing."
Position encoding for Transformer. Witnessing the efficacy of position encoding in capturing input structures in NLP [11, 28, 54], recent vision models [18, 50, 70] have begun employing position encodings for images to model spatial structures of images. In particular, relative position encoding (RPE) plays vital role for the purpose: The work of Cordonnier et al. [9] proves that self-attention has close relationship with convolution when equipped with certain form of RPE. Wu et al. [70] explore the cxisting RPE methods used in vision transformers [1 I, 28, 54, 65] and draws a conclusion that RPEs impose convolutional processing on vision transformers. Dai et al. [10] observe that
We refer peripheral inductive bias as the prioritization of our hypothesis which any attention based neural networks use to mimic human peripheral vision by modelling shauee attentions as illustrated in Fig. I.

---

depthwise convolution and self-attention can naturally be unified via RPEs. While offering promising directions, the previous RPE work, however, limited the sense that the focus of RPE utilization is restricted to only local attention, e-g., convolution. This work exploits RPEs to devise an original visual feature transformation which naturally generalizes convolution and self-attention layers, thus enjoying the benefits of both via imitation of human visual processing system: peripheral vision.
Peripheral vision for machine perception. Along with central vision, peripheral vision plays vital role in a wide range of visual recognition tasks [33]. The fundamental mechanisms peripheral visual processing, however, have not been fully disclosed in human vision literature [52] which stimulated many researchers to reveal its inner workings and deep implications [2, 15, 16, 43, 53, 68]. The work of Rosenholtz [52] discusses pervasive myths and current findings about peripheral vision, suggesting that peripheral vision is more crucial for human perception than previously thought to perform diverse important tasks. Inspired by its importance, number of pioneering work [16, 20, 22, 23, 44, 66] investigate the linkage between peripheral vision and machine vision, e.g., CNNs, while some [31, 64] devise biologically-insiige
models for the creation of stronger machine vision. Continuing previous study, this paper explores to blend human peripheral vision with attention- based neural networks, e.g., vision transformer [18, 58], and introduces new network called Peripheral Vision Transformer.
3 Our Approach
In this section, we introduce the Peripheral Vision Transformer (PerViT) which learns to model peripheral vision for effective image recognition. We first revisit the mathematical formulation of self-attention layer and then describe how we improve it with peripheral inductive biases.
Background: Multi-Head Self-Attention. The multi-head self-attention (MHSA) [63] with N
heads performs an attention- based feature transformation by aggregating N,, attentttnnion outputs:
MHSA(X) = concat Seeff-Attentoon (b) (X) W out + bouts
kE(Ns)
where X E & a is set of input tokens and W E IR NA and b E Pave are the transformation parameters. The N2, outputs of self-attention are designed extract diverse set of features from the input representation. Formally, the self-attention at head h is defined as
Self Leff-Attention" (1) X Normalize (T(h) (X)]
where Normalize(: denotes a row-wise normalization and (.))(.) HWW is a function that provides spatial attentions based on content information to aggregate the values ((h) XW (h),
val
T (h) XX] = exp(1 XW (f) qry
(XW(()) ( exp(rQ (()))
R Pon Dr. for queries, keys, and values respectively where 7 is softmax temperature and exp(-) applies an element-wise exponential to the input matrix.
using linear projections of w-hh)
(-(h)
(((h)
key
val
3.1 Multi-head Peripheral Attention
Based on the formulation of MHSA in Eq. 1, we define Multi-head Peripheral Attention (MPA) as
MPA(X) == concat [Periperrll-Attention (h) (X, R.) W Won bouts kE[M,]
(4)
where R E #WWHWWD, is the relative position encoding with D, channel dimension. The
self-attention in MHSA is now replaced with Peripheral-Attentionn-)) consisting of content- and
position-basee based attention functions (b(h)) (X). b (h) (R) I. HWW which is formulated as follows:
Peripheral- Atternall-ttetiionn (b) XX R) = Normalize
(N) (h)
(X)
4 (R) v(h)
(5)
where 3 is Hadamard product which mixes the given pair of attentions provide mixed attention d (h) (R) e HWW For the content-based attention &.., we use expo-
& (h) R) 4!
nentiated (scaled) dot-product between queries and keys as in Eq. 3: (h) (²x((x) (X) = (
For the cosition-basee attention Tp- we design a neural network that aims to imitate human visual processing system, e.g., peripheral vision, which we discuss next.
3

---

Modelling peripheral vision: a Roadmap. Human visual field can be grouped into several regions based on the Euclidean distances from the center of gaze, each forming ring-shaped region as seen in Fig. 1, where each region captures different visual aspects: the closer to the gaze, the more complex features we process, and further from the gaze, the simpler visual features we perceive. In the context of 2-dimensional attention map +, (+).: E TR we refer the query position q E RR as the center of gaze, í.e., the position where feature of our interest lies at for the transformation. We refer the local regions around the query q central/para--entral regions and the rest mid/far peripheral regions.
Perhaps the simplest approach to divide the visual field into multiple subregions is to perform a single linear projection on the Euclidean distances, i..,, 4 (R) = a [RW where W!!) E R.D. and of-] is non- non-linearity, similarly to the previous work of Wu er al. 170β.. For straightforward imitation of peripheral vision, we use Euclidean distance for relative position input R and weigh the distances in D, different ways for the network to learn the mapping in multiple scales: Rq = ce [uc, k E IP, where [w))re[p.) is a set of learnable parameters shared across layers and heads, and qq = ||q kl|2 is the Euclidean distance between query and key positions q. k E 122. In our experiments, we choose sigmoid function for a to provide normalized (positive) weights to the content -baed attention L. A main drawback of this single-layer formulation is that @p is only able provide Gaussian-like attention map as seen in top-left in Fig. 2, thus being unable to represent peripheral regions in diverse shapes. For the encoding function to represent various (torus- shaped) peripheral regions, the distances must be processed by an MLP:
p
(b) (R) a Linear (ReLU(Linear RR; W W p2 = ReLU pl)
w (h) p2
(6)
where Wp pl Dr and w R. are the linear projection parameters and ReL.U gives non-linearity to the function. The first projection Wp1 is shared across the heads in order exchange
p2
information so each function is able to provide attention that are effective or complementary to other heads' attention. Note that given identical relative distances between fixed query point q E R²2 and key points k; kj E i.e. Rg R.,.,, Eq. 6 provides the same attention scores: (pp(R), = Lp(R)q.k, as seen in top-right of Fig. 2. This property, however, is not always desired in practical scenarios because the rotational symmetric property hardly holds for most real- rorldworld objects. To break the symmetric property in Eq. 6 while preserving peripheral design to sufficient extent, we introduce peripheral projections in which the transformation parameters are given small spatial resolutions, e.g., K K window, such that Wp E x Das and (( p2 E RR x D so that they provide similar but different attention scores, (R))). * (R)(R) given Rq.k, = Ra.ky by aggregating neighboring relative distances around the key location k as follows:
(R)) =o
Σ nE)(k)
ReLU
Σ (E
Rq.m. Wpl a- k,y
((())
n-
[]] is set of K2 neighbors around input position k. We set K= 3 for all layers and heads as K> 3 hardly brought improve- ments. Note that each linear projection in Eq. 7 is equivalent to 4-dimensional convolution [51]. tak- ing 4-dimensional input R E R MW x#W XI, to process in convolutional manner using 4-dimensional kernels in size of K x K projection considers a small subset of 4D local neighbors that pivots the query position q. similarly to the center-pivot 4D convolution [45, 46]. After each peripheral projection, we add an instance normalization layer [61] for stable optimization:
where (Kk
k
k K k+ +
1, i.e. Wpl E 1 D, Precisely, the peripheral
R' - ReLU (IN W
(h) $ ((0) (IN(PP(R'; w((h) )i T :
(8)
where 7pt. E TR. Dric and 7p2
(h) ((h)) E IR are weights/biases of the instance norms and (P(() denotes
the peripheral projection: PP(R, W) W)q.k= = EneN(k)) Rq... W.-k::: The middle row of Fig. 2
depicts learned attentions of d with peripheral projections, which provides peripheral attention maps in greater diversity compared to single- and multi- counterparts W.thout.V.
'Given
of-] == exp(-), Peripheral-Attentine (h) = Normalize(exp(Q" (h) K( (())T @ exp(RW(()) y(*) =
softmax( Q (h) ( + RW pp) V(h), which is equivalent to the bias mode RPF presented in [70].
We omit the bias terms in the linear layers for brevity.

---

Singe: over protection
Mutti-byer projection
Poripheral
Imgg embedding
Normate
Natrkmmtt
Peripheral projections
B.B. Sgmmol
Fes
FFN
Peripharal projection
Staled
Peripherol initiolization
x/k
Q
Kits
Convolational parch emoeddlng
Peripheral projection
Linor
Lnoor
Inpus
image
Figure 2: Top: representation ability of p un- der varying # layers. Middle: peripheral projec- tions. Bottom: peripheral initialization.
Figure 3: Overall architecture of PerViT which based on DeiT [12] architecture.
Peripheral initialization. Recent study [49, 58] observe that early layers of trained vision transformer learn to attend locally whereas late layers perform global attentions. To facilitate training of our network, we inject this property in the beginning of the training stage by initializing parameters of Lp for the purpose, making attention scores near the queries larger than the distant ones in the early layers while uniformly distributing them in the late layers seen in bottom row of Fig. 2. We refer this method as peripheral initialization for its resemblance to the functioning of peripheral vision [36] which also operates either locally globally perceive different visual features [76]. To formally put, given two arbitrarily chosen distances R Réne Rate
q.k. q.k, E R which satisfy
q.k: < Raee q,k, we want 4.11 layers, and $ (1.b) (R)q,k (((hh) (R) as l Nr, i.e., global attention in late layers, where Ni
(1)h) (R) >>
1. (R)q.k, as 1 + 1. i.e., local attention in early (1,))
is the total number of MPA layers. We first initialize the parameters of $ (1,4) and (wr) (wrrr(D.) to particular values. Specifically, for all layers / E [Ni] and heads h E [NVr].
w, = -c, E [Dd
w(() = K2, W p2
(I..). = C2J
7pl = 1 8 =
(9)
where CI, C2 E RR are positive reals, and JN.M E NXM refers to all-one matrix in size x M. The above initialization provides local attention after the second peripheral projection, i.e., PP(R:; ((h)),
p2 )q.k.: > PP(R'; with)
p2 )q.ks.: given Reye RReue q.k, Next, based on our findings that
biases g((.k) and the weights 7p2
(1.A) in the second instance norm control the sizes and strengths of local attention respectively, we simulate peripheral initialization by setting their initial values as Op2
g(lA) 8; and (1,h) 2 where respective (si hej) and {ut HE(M) are sets of initial values for
attention sizes and strengths. We set their values collected from uniform intervals: st [-5.0, 4.0] and ii E [3.0, 0.01] where 81-1 Si and ut-1 > to give stronger local attentions to shallow layers compared to deep ones as seen bottom row of Fig. We C1, C2 = 0.02 in our experiments. We refer the readers to the supplementary for the complete derivation of the peripheral initialization.
3.2 Peripheral Vision Transformer
Based on the proposed peripheral projections and initialization, we develop image classification models, dubbed Peripheral Vision Transformer, which is illustrated in Fig. 3. We follow similar architecture to DeiT [58] with convolutional patch embedding stem; as the original patchify stem [18] exhibits substandard optimizability due its coarse-grained early visual processing [71], many recent ViT models adopt multi-resolution pyramidal designs [40, 67, 69, 73] to mitigate the issue. While the pyramidal models have shown their efficacy in learning reliable image embeddings, the original single-ress single-resolution columnar design for PerViT because features in multiple resolution make our study less interpretable, which further requires additional techniques for combining our
stick with
We the superscript indicate both layer and head indices for the ease of demonstration.
5

---

Tiry
Small
Meedium
Figure Learned attentions bp of PerViT-T.
Figure 5: Peripheral region classification.
Position attention: 4p
Image
Mixed attention: d; ②
Laver
Figure 6: Visualization of learned attentions Dp and 4, for I E {3, 4, 8). Best viewed in electronics.
peripheral attention 2p with the existing cost-effective self-attentions mechanisms such as factorized [40], and cross- window [17]. To carry out fine tne-grained early processing while keeping single-resolution features across the layers, we adopt convolutional patch embedding layer [71] with multi-stage layouts for channel dimensions. The convolutional embedding layer consists four x 3 and one 1 x convolutions where the 3 x convolutions are followed by batch norm [29] and ReLU [47]. For additional details, refer to the supplementary materials.
attention [73], shifted-windt
Overall pipeline. Given an image, the convolutional patch embedding provides token embeddings ( E I //W x Diab, Similarly to [18, 58], the embeddings are fed to series of N; blocks each of which consists of an MPA layer and feed-forward network with residual pathways:
MPA(LN(CPE(X (")) + (
( FFN(LLN(X + -X))
(10)
where LN layer normalization [1], and FFN an MLP consisting of linear transformations with GELU activation [26]. Following the work of [35, 73), we adopt convolutional position encoding (CPE), i.e., 3 depth-wise convolution, before first layer norm for its efficacy with negligible computational cost. The output (Na) is global-average pooled form an image embedding.
Experiments
In this section, we first investigate the inner workings of PerViT trained on ImageNet- IK classification dataset examine how it benefits from the proposed peripheral projections and initialization, and then compare the method with previous state of the arts under comparable settings.
Experimental setup. Our experiments focus on image classification on ImageNet- 1K [13]. Following training recipes of DeiT [58], we train our model on ImageNet- IK from scratch with batch size of 1024, learning rate 0.001 using AdamW [42] optimizer, cosine learning rate decay scheduler. and the same data augmentations [14] for 300 epochs, including warm-up epochs. We evaluate our model with three different sizes, e.g., Tiny (T), Small (S), and Medium (M). We use stochastic depths of 0.0, 1. and 0.2 for T. S. and M respectively. We refer to the supplementary for additional details.

---

4.1 The inner workings of VirViT
Learning peripheral vision. We begin by investigating how PerViT models peripheral vision by qualitatively analyzing its learned attention of Pp. Figure 4 depicts the learned attention map of 4p.. E RBW for all layers and heads where the query position is given at the center, i.e. q= [7.77] We observe that the attentions are learned be diverse shapes of peripheral regions. Interestingly, without any special supervisions, the four attended regions (Na, 4) in most layers learned to complement each other to cover the entire visual field, capturing different visual aspects at each region (head), similarly to human peripheral vision illustrated in Fig. 1. For example, first two heads in Layer attend the central regions while the others cover the rest peripheral regions. The second and third heads in Layer 8 cover top and bottom hemicircles respectively, forming circular-shane shaped semi-globa receptive field. Moreover, a large number of early attentions is in form of central/para- central regions while those of late layers are learned to cover mid to far peripheral regions. To quantitatively inspect how VerViT models the peripheral visual system, we classify every feature transformation layer in the network into one of the four visual regions IP E {c, p, m, f] where respective elements refer to central, para-central, mid, and far peripheral regions. PerViT-AAtentinn head h at layer is classified as peripheral region p if the average of its attention scores which fall in visual region is the largest among the others:
of
PeripheralRegion(/. == arg max
Σ
(l. q.k" 1 [1q - kl12 E Ip]
(11)
[P|2
PEP
(q.k)EPxP
where P is a set of spatial positions (IP) = HW) and Ip is distance range (real-valued interval) of peripheral region p². The pie charts of Fig. 5 describe the proportions of peripheral regions for Tiny, Small, and Medium models where the bar graphs show them in layer-wise manner'. Similarly to the visualized attention maps in Fig. 4, the early layers attends central/para-central regions whereas deeper ones focus on outer region. We observe that, as the model size grows, the number of mid/far peripheral attention increases whereas that of central/para-central attention stays similar, suggesting that the models no longer require local attentions once sufficient amount of processing is done in the central region because, we hypothesize. identifying geometric patterns, e.g., corners and edges, is relatively simpler process than understanding high-level semanties.
Inspecting the impact of attentions (static vs. dynamic). To study how position- based attentions @ contribute to the mixed attentions 4. T @ Pp. we collect sample images and visualize their attention maps of Layers 3, 4 and 8 in Fig. 6. The mixed attentions Φ, at Layer 4 are formed dynamically (Tc) within antically-ffrmed region (0,) while the attentions $ at Layer 8 weakly exploit position information (Tp)) to form dynamic attentions (..). The results reveal that o plays two different roles; it imposes semi-dynamic attention if the attended region is focused in a small area whereas it serves as position bias injection when the attended region is relatively broad. In the supplementary.
constructively prove that an MPA layer in extreme case of semi-dynamic attention/position bias injection is in turn convolation/nulti-een the both transformations. To quantitatively examine the contributions of 4. and Φ to the mixed attention 4. we define a measure of 'impact' by taking inverse of difference between two attentions:
self-attention, naturally generalizing
(T.A))
(1.A)) (((.A)) [|_]
(12)
where || is Frobenius norm. The higher the measure (1(,.)) the larger the impact of position-bant based attention (.(.0)) Being averaged over all test samples, (!(,)) is similarly defined. As seen in Fig. 7, we observe a clear tendency that the impact of position-basee based attention is significantly higher in early processing, transforming features semi-dynamically, while the later layers require less position information, regarding T as a minor position bias. This tendency becomes more visible with larger models as seen in right of Fig 7: Small and Medium models exploit dynamic transformations much
$ 'The columnar design of PeTViT provides identical spatial resolution for every intermediate feature map in the network: H, W = 14, thus facilitating the ease of qualitative/quaniattive analyses of the learned attentions. "We use I. [0, 191 I, [1.19, 3.37), Im [3.37, 5.83). and I, [5.83, 7.9). We refer the readers to the supplementary materials for the justification on the these interval choices.
We classify the 3 x 3 depth-wise convolution in CPE and the two linear projections in FFN as central regions as their receptive fields approximately fall in the interval I. [0, 1.19).
7

---

Contert 122
Postion ((AA))
Content Q22A))
Postiion U0A
Impsact mokic with (7.5MM)
SYOW Vi0 (Tmm))
Figure 7: The measure of impact (X-axis: layer index, y-axis: the impact metric V.). Each bar graph shows the measure of a single head (4 heads at each layer), and the solid lines represent the trendlines which follow the average values of layers. (left: results of PerViT-T. right: results Tf S, and M.)
Conment
Position Ao
Mised (
Content RV
Postiion A1
Mixed Moadingg (g
Non Non-locaity wi (tmn)
docatty an CES Mi
Tay
Medur
Figure 8: The measure of nonlocality (x-axis: layer index, y-axis: the nonlocality metric P ).
more compared to Tiny model, especially in the later layers. Moreover, we note that the impact measures of four heads (bar graphs) within each layer are unevenly distributed, showing high variance. which imply that the network evenly utilizes both position and content information simultancously within each MPA layer as seen Layer in Fig. 6, performing both static/local and dynamic/global transformation in single-shot. These results reveal that feature transformations for effective visual recognition should not be restricted to be in either position-only [25, 41] or content-only [18, 58] design but they should be in the form of a hybrid [10, 12].
Inspecting the locality (local vs. global). We further investigate the inner workings of PeT by quantifying how locally 4, attends. Following the work of [12], we define the measure of nonlocality for (1.h) by summing all pair-wise query-key distances weighted by their attention scores:
Σ
$ (1,)) pq.k q.k.
(13)
PP
(q.k)EPxP
The metrics P. and Ω2 are similarly defined, being averaged over all test samples. As seen in Fig. 8, we observe a similar trend of locality between $ and 4., which reveals the position information play more dominant role over the content information in forming spatial attentions (#a) for feature transformation. Interestingly, we also observe that content- and position-basee attentions behave conversely; & attends globally in early layers, i.e., large scores are distributed over the whole spatial region, while being relatively local in deeper layers. We hypothesize that the proposed Φ in early layers is trained to effectively suppress unnecessary scores of t. at distant positions, thus exploiting only a few relevant ones within its local region of interest. Meanwhile, Pp at later layers gives & higher freedom in forming the spatial attention $ as described in the plots of Fig. 7, which allows the attention scores of &e to be clustered in semantically meaningful parts, e.g., eyes of the animals as seen in attentions of the first head at Layer 8 (Fig. 6), which makes O. relatively local.
4.2 Quantitative evaluation on ImageNet-11K
Ablation study on main components. In Tab. 1, we analyze the impact of each component in ViT, which is denoted as (a), where C-stem refers to convolutional patch embedding stem". We observe that the proposed attention + brings consistent gains to models (b. f, i) with relative improvements of 1. 1.4m4.2%p for (a, d, e, h) respectively. Among three main components (Tp, C-stem, CPE), hp has the most significant impact on PerViT (a), losing 1.5%p Top-1 1 accuracy without Th i.e., model (b).
'ee increase feature dimensions of the models (c) (without l.) and (f. h, i) (without C-stem) accordingly to make FLOPs comparable to the others (a-i) ensure the accuracy drops not simply due to lower FLOPs.
8

---

Table 1: Study the effect of each component in Vir
Table 2: Comparisons between dif- ferent relative position encodings with DeiT-Tiny [58] as a baseline.
Reference
P
C-stem
CPE
Top-1
Top-S
FLOPs (G)
(a)
78.8
94.3
1.6
(b)
77.3
FLOPS
(c)
76.8
93.5
1.6
Method
Top-1
(G)
(d)
77.8
94.0
1.5
/
/
78.1
94.0
DeiT-TT
72.2
1.3
(f)
X
/
X
J
76,3
93.2
1.5
+ CPVT [8]
73.4
2.1
(g)
76.7
+iRE [70]
73.7
(b)
/
K
76.5
93.4
1.5
PPE (ours)
74.4
())
72.3
Table 4: Top-1 accuracy comparisons with DeïT-S [58]
Table 3: Ablation PerViT-T/S/M: the effects of pp. C-Stem, and CPE.
under different subsampling ratios: 1100%, 50%, 25%);
Suhsampling ratio
Top-1
Top-S
C-Stem & CPE
Ref.
4p
T
S
M
DeiT.S
PerViT-S
DeiT-S
PerViT-S
(a)
/
V
78.8
82.1
82.9
100%
79.9
82.1
95.0
95.8
77.3
81.1
81.9
50%
74.6
77.4
91.8
93.1
72.2
79.8
81.8
25%
61.8
67,5
82.6
86.9
Surprisingly, PerViT without content-based attention &c, model (c), achieves decent performance, almost equalling to the accuracy of PerViT without position-based attention ppp model (b) (-0.5%p) The results verify that the proposed peripheral attention, which achieves comparable level of efficacy to the content-base based attention, learns to generate reliable spatial attentions for visual recognition. We also implement the proposed position-based attention Pp on DeiT [58] baseline and compare the results with recent state-of-the-an RPE methods. As seen in Tab. 2, the large improvements over the previous RPE methods [8, 70] further verify the efficacy of the proposed peripheral position encoding (PPE). To confirm that the impact of Φ is consistent with large models, conduct similar ablations using PerViT-S/M in Tab. 3; without dp. the accuracy consistently drops for all the three models. Comparing (b) with (c), we observe that C-Stem and CPE are less effective for large models, bringing 1.3%p and 0. 1% gains for Small and Medium respectively whereas they improve the Tiny model by 5.1 1..p. In contrast, the impact of d is consistent across different model sizes, bringing 1 1%p gains for all the three models. The better efficacy of p for larger models, we hypothesize, is due to its flexibility in modeling local/global spatial attentions while C-Stem/CPE are designed only to be local.
Sample-efficiency of ViT To investigate the training sample efficiency of our model, we train PerViT-S with ImageNet subsampled by fractions of 50% and 25% and evaluate on full-sized test set of ImageNet- 1 K. Table compare our results with DeiT [58]; our model consistently surpasses the baseline for all subsampled datasets, showing its robustness under limited training data.
Ablation study on 2p- The top section Tab. reports results of VeT-iT-T with different parameter initialization methods for D where peripheral denotes the proposed peripheral initialization, conv refers to convolutional initialization such that 8ı = -5.0 and Ut
3.0 for all [Nt], and rand refers to random initialization for all parameters in 4p: Wr. (!) (l.h) (1) Ypi B()) 7/2 and g(lA). The results show the efficacy of our peripheral initialization which is also supported by the results in Fig. 4 and 8: 1 provides early local and late global attentions, suggesting that peripheral initialization effectively reduces burden in learning such form of attentions. The bottom section of Tab. 5 studies network designs for b where A represents the proposed peripheral projection, i.e., projecting input distance representation by referring neighbors N, ML refers to multi-layer design of p. and Fuc & Lm indicate the type of embedding R: Euc is relative Fuclidean distances (Rq concat,e[D.]] [uc, R.)) where Lrn relative distances between learnable vectors (R E R HW HW HW*D., P.). Without A and ML, we observe consistent accuracy drops for Euc and Lmn by 0.8%p and 0.2%p respectively. A sole multi-layer projection hardly improves accuracy but the model performs the best when A is jointly used, meaning that both need to complement each other to provide diverse attention shapes as Fig 4. Furthermore, Fue models consistently surpasses Lm models, implying the Euclidean distance is more straightforward encoding type than learnable vectors in capturing spatial configurations images.
(L.A)
For each subsamples, we increase the number of epochs present models with fixed number of images.

---

Table Ablation study on dif- ferent initialization methods (top section) and network de- signs (bottom section) for the position-hand -aased attention p
Table 6: Model performance on ImageNet-IK [13].
Model
Size (M)
FLOPs (G)
Top-1 (%)
PVT-T [67]
1.9
75.1
CoaT-Lite-T [73]
5.7
1.6
77.5
Pyramidal
Swin-T
4.5
81.3
Vision
CoaT-LLiteSS [73]
20
4.0
81.9
Init. method
Top-1
Top-5
Transformers
Focal-T [74]
4.9
82.2
0p
(multi-resolution)
Swin-S [40]
8.7
83.0
Peripheral
CoaT-Lite-M
9.8
83.6
(ours)
78.8
94.3
Focal-8 [74]
51
9.1
83.5
Rand
78.5
93.6
DeiT:T [58]
5.7
1.3
72.2
XCTT-T12/16 [19]
1.2
77.1
Network design for Up
PerVIT-T (ours)
7.6
1.6
78.8
Top-1
Top-5
Columnar Vision
DeiT-S [58]
4.6
79.8
Euc + - ML
T2I-ViT [75]
22
6.1
81.7
(ours)
78.8
94.3
Transformers
XCTT-S12/16 [19]
4,8
82.0
For + MI.
77.9
94.0
(single-resolution)
PerVIT-S (ours)
4,4
82.1
Bac
78.0
DeiT-B
18
81.8
Lm +.V ML
77.8
94.0
T2T ViT-ViT, [75]
64
15
82.6
ML.
77.5
93.8
XCIT-S24/16 [19]
9.1
82.6
Lm
77.6
93.8
PerVIT-M (ours)
9.0
82.9
Comparison with state of the arts. Table 6 summarizes the results of our method and recent state of the arts. For fair comparison, the baselines used in our comparison are trained using 224 224 imput resolution without distillations, and are grouped into either pyramidal or columnar ViT based on the network designs, Le., multi- or single-resolution feature processing. where the results are partitioned according to model sizes within each group. As shown in the bottom section of Tab 6, the proposed method achieves consistent improvements over the recent columnar VIT methods [12, 19, 58, 75] while showing competitive results to the pyramidal counterparts.
5 Scope and Limitations
Despite the interpretability and effectiveness of ViT it still leaves much room for improvements. First, Per Atttenttttntion (Eq. 5) is based on the original self-attention formulation [63], thus directly inheriting its limitations [18, 58], e-g., quadratic complexity w.r.t. input resolution. The computational efficiency could be further improved by approximating low-rank matrices as in [7. 37, 73]. Second, given the ability to process high-resolution input with feasible complexity, the efficacy of PerViT could be improved by adopting multi-resolution pyramidal design following recent trend of ViT designs [17, 35, 37, 40, 67, 69, 73, 74]. Third, the focus of this paper is model development & exploration for image classification task but we believe the proposed idea is broadly generalizable to other vision applications such as object detection and segmentation. We leave this to future work.
Conclusion
This paper explores blending human peripheral vision with machine vision for effective visual recognition, and introduces Peripheral Vision Transformer which learns to provide diverse position- based attentions to model peripheral vision using peripheral projections and initialization. systematically investigated the inner workings of the proposed network and observed that the network enjoys the benefits of both convolution and self-attention by learning to decide level of the locality and dynamicity for the feature transformations, by the network itself given training data. The consistent improvements over the baseline models on ImageNet-I IK classification across different model sizes and in-depth ablation study confirm the efficacy of the proposed approach.
have
7 Acknowledgments and Disclosure of Funding
This work was supported by the ITTP grants (No.2021-0-01696 High- Pith-Potential Individuals Global Training Program (40%). (0.2002--0-02290 Visual Intelligence for Space-Timi Understanding and Generation based on Multi-layered Visual Common Sense (50%), No.2019-0-01999 Al Graduate School Program POSTECH (10%) funded by Ministry Science and ICT, Korea. This work was done while Juhong Min was working an intern at Microsoft Research Asia.
10

---

References
[1] J. Ba, J. Kiros, and G. Hinton. Layer normalization. arXiv preprint arxiv:6007.064500 07 2016. 6
[2] B. Balas, L. Nakano, and R. Rosenholtz. A summary-statistic representation in peripheral vision explains
visual crowding. Journal of vision, 2009. 3
[3] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In Proc. European Conference Computer Vision (ECCV), 2020. 2
[4] M. Caron, H. Touvron, L. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proc. IEEE International Conference on Computer Vision (ICCV), 2021. 2
[5] C.-F. R. Chen, R. Panda, and Q. Fan. RegionViT: Regional-to-Loone Attention for Vision Transformers. In International Conference on Learning Representations (ICLR), 2021. 2
[6] F. Chollet. Xception: Deep learning with depthwise separable convolutions. arXiv preprint
arXiv:1610.0233: 2016.
[7] K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Game, T. Sarlos, P. Hawkins, J. Davis, A. Mo- hiuddin, L. Kaiser, D. Belanger, L. Colwell, and A. Weller. Rethinking attention with performers. In International Conference on Learning Representations (ICLR), 2021. 2, 10
[8] X. Chu, 7. Tian, B. Zhang, X. Wang, X. Wei, H. Xia, and C. Shen. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2 10882, 2021. 2,
[9] J.-B. Cordonnier, A. Loukas, and M. Jaggi. On the relationship between self-attention and convolutional layers. In International Conference on Learning Representations (ICLR), 2020. 1,2
[10] Z. Dai, H. Liu, Q. V. Le, and M. Tan. Coatnet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 1, 8
[11] 7. Dai, Z. Yang. Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proc. Association for Computational Linguistics (ACL.), 2019. 2
[12] S. d' Ascoli, Touvron, M. Leavitt, Morcos, G. Biroli, and Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In Proc. International Conference on Machine Learning (ICML), 2021. 1, 2, 5, 10
[13] J. Deng, W. Dong, R. Socher, L.-J. Li, Li, and L. Fei-Fei. Imagenet: A large-scale hicrarchical image database. In Proc. JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009. 6, 10
[14] Devlin, M.-W. Chang, K. for language understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (MACC), 2019. 2,
Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers
[15] A. Deza and M. P. Eckstein. Can peripheral representations improve clutter metrics on complex scenes?
arXiv preprint arXiv: 16XXv:1608,0400 04042, 2016. 3
[16] A. Deza and T. Konkle. Emergent propertics of foveated perceptual systems. arXiv:2006.000 07991, 2021.
arXiv preprint
[17] X. Dong, J. Bao, D. Chen, W. Zhang, Yu, I.. Yuan, D. Chen, and B. Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. arXiv preprint arXiv:2 000522 2021. 6, 10
[18] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenbomn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. 1,
3, 5, 6, 10, 15
[19] A. El-Nouby, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin, I. Laptev, N. Neverova, G. Synnaeve, J. Verbeek, al. Xeit: Cross-covariance image transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 10, 15
[20] L. Fridman, B. Jenik, S. Keshvari. B. Reimer, C. Zetzsche, and R. Rosenholtz. Sideeye: A generative neural network based simulator of human peripheral vision. arXiv preprint urXiv:1706.04568 2017. 3
[21] K. Fukushima and S. Miyake. Neocognitron: A self organniggni neural network model for a mechanism of visual pattern recognition. Competition and cooperation in neural nets, 1982.
11

---

[22] S. Gould. J. Arfvidsson, A. Kaehler. B. Sapp. M. Messner, G. Bradski, P. Baumstarck, S. Chung, and A. Y. Ng. Peripheral-fover vision for real-time object recognition and tracking in video. In Proc. International Joint Conferences on Artificial Intelligence Organization (ICCAI), 2007. 3
[23] A. Harrington and A. Deza. Finding biological plausibility for adversarially robust features via metameric tasks, International Conference on Learning Representations (ICLR), 2022.
[24] H. He, J. Liu, Z. Pan, J. Cai, J. Zhang, D. Tao, and B. Zhuang. Pruning self-attentions into convolutional layers single path. arXiv preprint arXiv:2111.1 1802, 2022. 2
[25] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 1,8
[26] D. Hendryeks and K. Gimpel. Gaussian error linear units (gelus), arXiv preprint arXiv:606.08415, 2020.
[27] Hu, L. Shen, and G. Sun. Squeeze-and-exciine networks. Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[28] Z. Huang. D. Liang, P. Xu, and B. Xiang. Improve transformer models with better relative position embeddings. In Annual Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. 2
[29] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 2015, 6
[30] A. Jaegle, F. Gimeno, A. Brock, A. Zisserman, O. Vinyals, and J. Carreira. Perceiver: General perception
with iterative attention. In Proc. International Conference on Machine Learning (ICML), 2021. 2
[31] A. Jonnalagadda, W. Y. Wang, B. S. Manjunath, and M. P. Eckstein. Foveater: Foveated transformer for image classification. arXiv preprint 105 2022. 3
[32] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2012.
[33] A. M. Larson and L. C. Loschky. The contributions of central versus peripheral vision to scene gist recognition. Journal vision, 2009. 3
[34] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998. 1
[35] Lee, J. Kim, J. Willette, and S. J. Hwang. Mpvit: Multi-path vision transformer for dense prediction. In Proc. IEFF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1, 10
[36] J. Lettvin. On seeing sidelong, The Sciences, 16, 07 1976. 2,5
[37] K. Li. Y. Wang. J. Zhang, P. Gao, G. Song. Y. Liu, H. Li, and Y. Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. arXiv preprint arX 046766 2022. 1, 10
[38] H. Liu, X. Jiang, X. Li, Z. Bao, D. Jiang, and B. Ren. Nommer: Nominate synergistic context in vision transformer for visual recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[39] J. Liu, H. Li, G. Song, X. Huang, and Y. Liu. Uninet: Unified architecture search with convolution, transformer, and mlp. arXiv preprint 11X:v22110.04035 2021. 2
[40] 7. Liu, Y. Lin, Y. Cao, H. Hu, Wei, 7. Zhang, S. Lin, and Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proc. IEFE International Conference on Computer Vision (ICCV), 2021. 1, 2, 5, 6, 10, 15
[41] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xic. A convnet for the 2020s. In Proc. IEEE Conference on Computer Vision and Patiern Recognition (CVPR), 2022. 1,8
[42] Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
[43] C. L. Lou, D. Migotina, J. P. Rodrigues, J. Semedo, F. Wan, P. U. Mak, P. L. Mak, M. L Vai, Melicio, J. G. Percira, and A. Rosa. Object recognition test in peripheral vision: A study on the influence of object color. pattern and shape. In Brain Informatics, 2012.
12

---

[44] H. Lukanov, P. König, and G. Pipa. Biologically inspired deep learning model for efficient foveal- peripeeelip vision. Frontiers in Computational Neuroxcience, 15, 2021. 3
[45] J. Min and M. Cho. Convolutional hough matching networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2940-2950, June 2021. 4
[46] J. Min, D. Kang, and M. Cho. Hypercorrelation squeeze for few-shot segmentation. In Proc. IEEE International Conference on Computer Vision (ICCV), 2021. 4
[47] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proc. International Conference on Machine Learning (ICML), 2010. 6
[48] A. Paszke, S. Gross, F. Massa, A. Lerer. J. Bradbury. G. Chanan. Killeen, Z. Lin, N. Gimelshein. L. Antiga, A. Desmaison, A. Kopf. E. Yang. Z. Vetito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurlPS), 2019. 15
[49] M. Raghu, T. Unterthiner, S. Korbllith, C. Zhang, and A. Dosovitskiy. Do vision transformers see like convolutional neural networks? In Advances in Neural Information Processing Systems (NeurIPS), 2021.
1, 2,
[50] P. Ramachandran, N. Parmar, A. Vaswani, I. Bello, A. Levskaya, and J. Shlens. Stand-alone self-attention in vision models. In Advances in Neural Information Processing Systems (NeurlPS), 2019. 1,2
[51] Rocco, M. Cimpoi, R. Arandjclović, A. Torii, T. Pajdla, and J. Sivie, Neighbourhood consensus networks. In Advances in Neural Information Processing Systems (NeurIPS), 2018. 4
[52] R. Rosenholtz. Capabilities and limitations of peripheral vision. Annual review of vision science, 2016.
[53] R. Rosenholtz.
Demystifying visual awareness: Peripheral encoding plus limited decision complexity resolve the paradox of rich visual experience and curious perceptual failures. Attention, Perception, & Psychophysics, 2020.
[54] P. Shaw, Uszkoreit, and A. Vaswani. Self-attention with relative position representations. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). 2018. 2
[55] A. Srinivas, T.-Y. Lin, N. Parmar, J. Shlens, P. Abbeel, and A. Vaswani. Bottleneek transformers for visual recognition. In Pruc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[56] R. Strudel, R. Garcia, I. Laptev, and C. Schmid. Segmenter: Transformer for semantic segmentation. In Proc. IEEE International Conference on Computer Vision (ICCV), 2021. 2
[57] C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proc. IEEE International Conference on Computer Vision (ICCV), 2017.
[58] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolies, and H. Jegou, Training data-efficient image transformers & distillation through attention. In Proc. International Conference on Machine Learning (ICML), 2021. 5, 6, 8, 9, 10, 15
[59] H. Touvron, M. Cord, A. El-Nouby. P. Bojanowski, A. Joulin, G. Synnaeve. J. Verbeek, and H. Jegou. Augmenting convolutional networks with attention-based aggregation. arXiv preprint arXiv:2112.13692. 2021.
[60] 7. Tu, H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y. Li. ViT: Multi-Axis Vision Transformer. In Proc. European Conference Computer Vision (ECCV), 2022. 1,2
[61] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arxiv:1607.08022 2017. 4
[62] A. Vaswani, P. Ramachandran, A. Srinivas, N. Parmar, B. Hechtmann and J. Shlens. Scaling local self- attention for parameter efficient visual backbones. In Proc. IEEE Conference on Compuder Vision and Pattern Recognition (CVPR), 2021. 1,2
[63] A. Vaswani, N. Shazcer, N. Parmar, J. Uszkoreit, L., Jones, A. N. Gomez, L.. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 1,2, 3, 10
[64] M. R. Vuyyuru, A. Banburski, N. Pant, and T. Poggio. Biologically inspired mechanisms for adversarial robustness. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 3
13

---

[65] H. Wang. Y. Zhu, B. Green. Н. Adam, A. Yuille, and L.-C. Chen. Axial-deeplab: Stand-alone axial- attention for panoptic segmentation. In Proc. European Conference Computer Vision (ECCV), 2020.
[66] P. Wang and G. W. Cottrell. Central and peripheral vision for scene recognition: A neurocoopuutational modeling exploration. arXiv preprint arXiv:105.008166 2017.
[67] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang. T. Lu, P. Luo, and L. Shao. Pyramid vision transformer: A versatile backbone dense prediction without convolutions. Proc. IEEE International Conference on Computer Vision (ICCV), 2021. 1, 2, 5, 10, 15
[68] Wijntes and R. Rosenholtz. Context mitigates crowding: Peripheral object recognition in real-world images. Cognition, 2018. 3
[69] Wu, Xiao, Codella, M. Liu, Dai, L. Yuan, and L. Zhang. Cvt: Introducing convolutions to vision transformers. In Proc. IEEE International Conference on Computer Vision (ICCV), 2021. 1, 2, 5, 10
[70] Wu, Peng, Chen, Fu, and H. Chao. Rethinking and improving relative position encoding for vision transformer. In Proc. IEEE International Conference on Computer Vision (ICCV), 2021. 2, 4,
[71] T. Xiao, P. Dollar, M. Singh, Mintun. Darrell, and R. Girshick. Early convolutions help transformers see better. In Advances in Neural Information Processing Systems (NeurlPS), 2021. 1, 2, 5.
[72] S. Xie, R. Girshick, Dollár, Tu, and K. He. Aggregated residual transformations for deep neural networks. arXiv preprint arXiv: 1.004331,
[73] Xu, Y. Xu, T. Chang, and Z. Co-scale conv-attentional image transformers. Proc. IEEE International Conference on Computer Vision (ICCV), 2021. 1, 5, 6, 10, 15
[74] Yang, C. Li, Zhang, X. Dai, B. Xiao, L.. Yuan, Gao. Focal self-attention for local-global interactions in vision transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 10, 15
[75] L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, Z.-H. Jiang, F. E. Tay, J. Feng, and S. Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proc. IEEE International Conference on Computer Vision (ICCV), 2021. 10
[76] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In Proc. European Conference on Computer Vision (ECCV), 2014.
[77] Y. Zhao, G. Wang, C. Tang, C. Luo, W. Zeng, and 7.-J. Zha. A battle of network structures: An empirical study of can, transformer, and mlp. arXiv preprint 1:XXi:2108.13002 2021. 1
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We describe potential negative societal impacts in the supplementary.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We describe the full of assumptions theoretical results in the supplementary.
(b) Did you include complete proofs of all theoretical results? [Yes] We include complete proof's of our theoretical results in the supplementary.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material the code and instructions for reproduction in the supplementary. The training and validation data, e.g., ImageNet (ILSVRC2012), is available online.
URL)? [Yes] We include
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] We specify all the training details in the supplementary.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No] Training large-scale image classification models is very expensive
only performed one run. This is common practice in the field in cited references [18, 19, 40, 58, 67, 73, 74].
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] We include the amount and type of resources used for training in the supplementary.
4. If you are using existing assets (e.g., code, data, models) or curating/releusing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] As our baseline code, we use DeiT [58] which is implemented with PyTorch [48] framework, all of which are open-sourced.
cite all the papers that are relevant to the code, models, and datasets.
(b) Did you mention the license of the assets? [Yes] Our code refers to the licenses of the assets it relies on.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We include trained models in the supplementary for reproduction of our main results.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No] We are using only publicly available, benchmark datasets. (e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] We using only publicly available, benchmark datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] We did not croudsource any information for this research.
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] We did not croudsource any information for this research.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] We did not croudsource any information for this research.
15


Reviews:
[{"review_id": "lRp_nQ9Ggtc", "reviewer": "Reviewer_NKKB", "summary": "The paper proposes a transformer architecture which incorporates human-like peripheral vision. They modify the multi-head attention with peripheral attention, which consists of content and position attention. Content attention is the same as scaled dot product attention. Position attention is learned using a neural network which takes a fixed relative position embedding as input and is independent of the input image. The position-based attention learned by the proposed approach seems to divide the image into distinct regions similar to peripheral vision. Finally, they present quantitative and qualitative results to validate the effectiveness of their approach. ", "questions": "* What is the formulation of neighbour function, and how does it impact the learned position-based attention?\n\n* There should be a baseline which uses fixed position embedding varying by layer to highlight the benefits of learning position embedding using NN. Did the authors try such a baseline? Would appreciate a discussion on that in the author's response.\n\n* Can you take the ViT implementation of [A] and apply the peripheral positional encoding on it?  ", "limitations": "Yes, the limitations are addressed adequately.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n* The paper aims to design neural networks which see in a similar way to humans. The direction and the motivations are novel.\n* Extensive qualitative results help in understanding the proposed approach.\n\nWeakness:\n* The motivation behind using neural networks for learning position-based attention is not explained clearly.\n* It is possibly difficult to disentengle the performance gained by the particular change (like the peripheral encodings in thie case). There is huge variation withing different implementations of ViT itself. For instance, the implementation of ViT in [A], significantly outperforms the original model. ViT implementation in [A] without autoencoder fine tuning achieves 82% top1 on imageNet, bringing over 5% improvement on the original model (76.5%). It is hence natural to question approaches which bring minor gains (less than a percent). The change can clearly be achived by simpler engineering corrections.  \n* A more solid approach would be to use the optimally trained ViT and then base the experiments on it. \n\n[A] He et al. Masked Autoencoders Are Scalable Vision Learners. CVPR 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GEALQhg2Ay", "reviewer": "Reviewer_XfL6", "summary": "This paper proposed a novel visual transformer model that incorporates the inductive bias of peripheral vision into the relative positional encoding. Specifically, the authors define a multi-head peripheral attention (MPA) module, where content-based and position-based attention scores are first calculated independently and then combined through an element-wise product. The design of the position-based attention allows pairs of locations ($q$,$k$) with the same Euclidean distance to have similar attention scores. The peripheral projection introduces the diversity, while the peripheral initialization ensembles the increasing spatial receptive fields. The results show better performance on ImageNet classification task than models with similar capacity. The ablation studies suggest the major contribution of the performance improvement comes from the position-based peripheral attention in the MPA.", "questions": "Major:\n- The \"peripheral projections\" part in Section 3.1 seems interesting and important, but it's hard to fully understand its intuition and computation from the description. Does this peripheral projection allow the attention to be calculated not from the key location itself, but from a small surrounding area (local context) around the key location? Does $PP$ in equation 8 stand for the computation of $\\Phi_p$ in equation 7? It will be helpful if the authors may rephrase this part in a clearer and more intuitive way, or maybe use a visual illustration if it's helpful. I want to understand the \"peripheral projection\" better since it has the most contribution according to the results in Table 4.\n- How is the $\\mathcal{N}(\\cdot)$ function defined in equation 7? Is it fixed for different layers/heads?\n- Why there is a vertical discontinuity in the attention patterns after peripheral projections (Fig. 2, middle row)?\n- From Table 4, it seems that adding ML (multi-layer design) to Euc doesn't help but adding $\\mathcal{N}$ (peripheral projection) to Euc+ML improves the performance significantly. Is the multi-layer design an essential component or do we only need the diversity from $\\mathcal{N}$?\n\n\nMinor:\n- The tables are not referenced in the order as they appear\n- Some discussion of how peripheral vision affects model behavior in a human-like way may be inspiring when sketching future directions.\n- It may be worth including a few early works that design biologically-inspired peripheral vision models with CNNs by polar transformation or using non-uniform sampling to generate foveated images, for example:\n    1. Wang, P., & Cottrell, G. W. (2017). Central and peripheral vision for scene recognition: A neurocomputational modeling exploration. Journal of vision, 17(4), 9-9.\n    2. Reddy, M. V., Banburski, A., Pant, N., & Poggio, T. (2020). Biologically inspired mechanisms for adversarial robustness. arXiv preprint arXiv:2006\n    3. Jonnalagadda, A., Wang, W., & Eckstein, M. P. (2021). Foveater: Foveated transformer for image classification. arXiv preprint arXiv:2105.14173.\n\n", "limitations": "No negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- This paper proposed a concise idea to incorporate peripheral vision into the transformer-based computer vision models. The definition of relative positional encoding is straightforward to apply and understand and is well-aligned with the distance-based sampling in the human visual system.\n- The performance evaluation and ablation studies convincingly suggest that peripheral attention is the critical module that contributes to the improvement, and it has a reasonable and interpretable behavior (e.g., the position bias is mainly seen in early layers; larger models have farther peripheral information).\n\nWeakness:\n- The evaluation of ImageNet classification may not be the best task to show the benefit of adding a peripheral mechanism to the computer vision model. It would be great if the authors could demonstrate if this mechanism significantly increases the efficiency and robustness of the model.\n- The presentation of the methods and results can be improved for clarity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B13llK6vyqX", "reviewer": "Reviewer_5MyL", "summary": "In this paper, the authors introduce a novel transformer biologically inspired transformer architecture. They introduce a change on the attention mechanism which enable models to split the visual field into different peripheral regions. Authors compare their proposal to the baseline on classification, showing improvement.\n", "questions": "My concerns are listed in the weaknesses section. My main questions are:\n\n- Have the authors considered other tasks/datasets for evaluation? In my opinion the evaluation is very limited.\n\n- How would the authors argue that the paper is still contributing to pushing the SOTA if Focal-S performs better than the proposed model.\n\n- Have the authors evaluated the quality of the segmentation in segmentation data?", "limitations": "Yes, I think the discussion of limitations is correct. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- S1. The paper proposes an interesting and useful addition to standard transformer attention. State-of-the-art methods typically lack of the ability to partition the visual field in peripheral regions, which can help focusing on the most important content for the model and make training more efficient.\n\n- S2. The paper is well written and motivated. I think authors described well the goal of the paper and the model, as well as illustrated well the idea with plenty of useful figures.\n\n- S3.  The ablation study in the paper is very complete, going over all posible design decisions that shaped the final architecture and model. I specially like the results in Table 1, where the reader can easily understand the importance of the different elements. \n\n** Weaknesses**: \nIn my opinion the main weakness of the paper is the evaluation. I think authors fail to show that their proposed method is better than other proposals in the transformer area. My main concerns are:\n\n- W1. Authors only evaluate on one ImageNet, and do not provide additional results on other datasets. In my opinion for the paper to be convincing about the usefulness of the methods, author should provide results on other datasets as well.\n\n- W2. Focal-S performs better than the proposed model with similar number of flops and model size. As the ImageNet evaluation is the only data point we have to evaluate the model performance, it's hard to convince the reader with this evidence that the model is pushing the state-of-the-art. \n\n- W3. Given the main innovation of the model, I believe that an object centric dataset such as ImageNet is probably not the best dataset for evaluation. I think the proposed model is good at focusing to the different objects in the image in an efficient manner, while in ImageNet typically a single object is present.\n\n- W4. Following up with W3, I think this model could actually shine more on tasks such as segmentation, where fine detail of the input image is important to produce a quality output.\n\n- W5. Have the authors considered evaluating the segmentation masks generated by the attention? (Figure 6)\n\n\n\n------------------------------\n* After rebuttal*: The author's response addressed most of my concerns and I updated my review score recommending acceptance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5dV3nUbi2-G", "reviewer": "Reviewer_mNPh", "summary": "This paper takes inspiration from the foveated nature of human vision, in which visual scenes are processed with decreasing resolution and increased information compression as eccentricity to the center of the fovea increases.\nIt notes that recent vision transformers (ViTs), which use multi-headed self-attention, require a lot of data to learn useful patterns of self-attention at different layers of the feature hierarchy.\nWhile various methods have been proposed to address this, often relying in one way or another on re-asserting locality into the processing e.g. explicitly via convolution or by introducing pyramidal structure or more structured attention, the paper here proposes adding a position-based attention mechanism into a standard data-efficient image transformer (DeIT).\nIn the MPA, learned position-based attention maps are combined with the (original) learned content-based attention maps to produce a  \"Multi-head Peripheral Attention\" (MPA) map.\nExperiments on ImageNet-1k show that their Peripheral ViT (PerViT) model which uses MPA uses this additional structure in the attention to perform better at the classification task than without, to a level on par with recent pyramidal architectures, across three different approximate model sizes.\nFurther experiments look at the qualitative nature of the learned peripheral maps, the trade off of influence between the position-based and content-based maps, the locality of the attention at different layers, the benefit of careful initialization, and an ablation of the various model additions. ", "questions": "In addition to the points mentioned above,\n1. I think Figure 1 could be explained more clearly to inform a new reader - could you explain how does this learned attention map relate to peripheral vision? \n2. How was the D_r chosen - and what is the impact of varying it?\n3. L142 what is K?\n4. L179 as far as I'm aware, \"cylindrical\" is not a standard term. What is meant by the term here? Do you mean \"columnar\", in the sense as described by e.g. https://arxiv.org/pdf/2102.12122.pdf ? Columnar is strictly better, because cylindrical implies a circle in some plane?\n5. I'm confused why w_r is appearing in Eq 9, when it doesn't appear to be part of the peripheral projection? (based on my reading of L122-L132). Is this just describing all of the parameter settings for all experiments?\n6. What does classification of visual regions add in Fig 5? It seems somewhat arbitrary, since \"angle\" doesn't have an obvious notion in an image taken from an unknown camera. In Supp Sec B, there is some explanation of visual fields broken down by angle, but is it fair to assume that images correspond to a particular FOV? Peripheral vision only makes sense in the context of a physical eyeball which has a focal point; how does that relate to an image from an unknown camera? An alternative might be to make a scatter plot of average attention distances vs layers, in a similar vein to Fig 3 of [36]? \n7. What, if any, is the distinction between the proposed Peripheral Positional Encoding (PPE) (Table 2) and the used Convolutional Positional Encoding (CPE) (Table 1)? \n8. Table 5 of the DeIT paper https://arxiv.org/pdf/2012.12877.pdf shows that using the distillation method during training yields stronger performance (e.g. DeIT-B gives 83.4 Top-1, and higher when trained for longer). Although this is not necessarily a fair comparison, I think it could be mentioned at least. The abstract and contributions claim unconditionally (L11, L69) that the PerViT yields \"state of the art performance in image classification task\" but I think this is not technically true and ought to be revised?\n9. parts of S3.1 could be rewritten to improve readability, in particular the explanation of Peripheral Projections from L141. \"By referring neighboring relative distances around the keys\" does not make sense to me - could it be rewritten? \n10. In Eq 8 - the PP function does not appear to be explicitly defined until the supplementary. Perhaps some of this section could be represented graphically, similar to Fig 1 in [50], to improve clarity?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Paper Strengths]\n* the proposed multi-head peripheral attention approach seem to be new, technically sound, and interesting \n* the experiments are well-motivated, with carefully chosen baselines and good performance\n* code is available with promise of reproducible experiments (although I did not try)\n\n[Paper Weaknesses]\nOverall I think the paper presents an interesting approach backed up by some good empirical results, but there are a few reasons why I remain borderline:\n* I think that the paper somewhat oversimplifies peripheral vision and oversells its involvement in the design of the model. There is only one reference to peripheral vision in the main text, and it is a bit reductive -- I encourage the authors to explore more recent literature on peripheral vision, e.g. https://visxvision.files.wordpress.com/2017/08/ruth_peripheral.pdf in order to better relate their work to peripheral vision literature. I think it is fair to be inspired by the notion of peripheral vision, but using definitive terms like \"peripheral inductive bias\" (L56) may be misleading. Claims like L47 \"According to recent study on inner workings of vision transformers [11, 15, 36, 42, 51], their behavior is in fact closely related to how the peripheral vision functions\" are not supported by the cited works (which make no mention of peripheral vision), and are arguably more speculation than scientific \"fact\". I believe the paper will be better served by toning down this aspect; it is ok to be inspired loosely by a notion of how peripheral vision might work, and this model might indeed model peripheral vision to a better extent than previous works, but as far as I can tell, we do not know and cannot be certain.\n* Secondly, although the experiments are reasonably extensive, a few experiments are missing which I think are important for a reader to understand the value MPA. (i) One of the main motivations to impose structure into the attention (I think) is to improve training sample efficiency. Have any experiments been conducted to explore this (e.g. performance vs training set proportion on ImageNet)? (ii) My understanding is that of the three components (phi_p, C-Stem, CPE), phi_p is the main novelty of this work, while C-Stem and CPE have been adopted from prior work. I would like to see the ablation on the -S and -M models to understand the impact with/without only phi_p in those cases. (iii) Experiments on other datasets (e.g. ones which DeIT supports already like iNaturalist) would help to further validate the usefulness of the proposed method.\n* Although I found the paper to have good general coverage of recent work in this area, I found the related work section to be somewhat brief, and would appreciate a more explicit explanation of why the proposed work differs from previous efforts to improve the efficacy of the self-attention mechanism for vision transformers. \n\nEDIT: post-rebuttal, upgraded Rating from 5 to 7 (accept).\n\n[Minor Points / typos]\n* L9 \"large-scale ImageNet dataset\" -- \"large-scale\" here may mislead readers to think of the full ImageNet dataset; in reality the 1K dataset is used in the paper\n* some references are not ordered numerically e.g. L30, 53, 82\n* L56 - we proposes \n* Fig 3 - bases on -> is based on the \n* Sec 3 - please define new variables when they are introduced (e.g. D_h), for better clarity. \n* Eq 3 has an unwanted comma\n* L119-21 - break up the sentence for better clarity\n* L123 - Eucliden\n* L134 non-linearlity\n* L142 the they\n* please check grammar in 4.1\n* L252 - double negative should be single negative?\n* Please fix table ordering in 4.2 so it follows text", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept