LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate
Authors: Anthony Fuller
Daniel Kyrollos
Yousef Yassin
James R Green
Venue: NeurIPS 2024 poster (2024)
Keywords: vision transformers
position encoding
computer vision
Primary area: machine_vision
TL;DR: We introduce a position encoding method for plain ViTs, called LookHere, that improves performance at the training resolution and beyond.

Abstract: High-resolution images offer more information about scenes that can improve model accuracy. However, the dominant model architecture in computer vision, the vision transformer (ViT), cannot effectively leverage larger images without finetuning — ViTs poorly extrapolate to more patches at test time, although transformers offer sequence length flexibility. We attribute this shortcoming to the current patch position encoding methods, which create a distribution shift when extrapolating.

We propose a drop-in replacement for the position encoding of plain ViTs that restricts attention heads to fixed fields of view, pointed in different directions, using 2D attention masks. Our novel method, called LookHere, provides translation-equivariance, ensures attention head diversity, and limits the distribution shift that attention heads face when extrapolating. We demonstrate that LookHere improves performance on classification (avg. 1.6%), against adversarial attack (avg. 5.4%), and decreases calibration error (avg. 1.5%) — on ImageNet without extrapolation. With extrapolation, LookHere outperforms the current SoTA position encoding method, 2D-RoPE, by 21.7% on ImageNet when trained at $224^2$ px and tested at $1024^2$ px. Additionally, we release a high-resolution test set to improve the evaluation of high-resolution image classifiers, called ImageNet-HR.

LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate
Anthony Fuller, Daniel Kyrollos, Yousef Yassin, James Green
Department of Systems and Computer Engineering Carleton University
Ottawa, Ontario, Canada "anthony. wtw.oomyy..lleeeereeettto...ooo
Abstract
High-resolution images offer more information about scenes that can improve model accuracy. However, the dominant model architecture in computer vision, the vision transformer (ViT), cannot effectively leverage larger images without finetun- ing ViTs poorly extrapolate to more patches at test time, although transformers offer sequence length flexibility. We attribute this shortcoming to the current patch position encoding methods, which create distribution shift when extrapolating.
We propose drop-in replacement for the position encoding of plain ViTs that restricts attention heads to fixed fields view, pointed different directions, using 2D attention masks. Our novel method, called LookHere, provides translation- equivariance, ensures attention head diversity, and limits the distribution shift that attention heads face when extrapolating. We demonstrate that Look improves performance on classification (avg.1 1.6%), against adversarial attack (avg.1 1 5.4%), and decreases calibration error (avg.) 1.5%) - on ImageNet without extrapolation. With extrapolation, LookHere outperforms the current SoTA position encoding method, 2D-RoPE, by 21.7% on ImageNet when trained at 224 px and tested at 1024 px. Additionally, we release high-resolution test set to improve the evaluation of high-resolution image classifiers, called ImageNet-HR.
1
Introduction
There is decades-long trend in computer vision towards higher-resolution imagery, which contains more detailed scene information. Increasing resolution is reliable way to improve model accuracy
15 16] 7 &8 I9 20), but this comes at cost; training models for hundreds of epochs on large-scale datasets is expensive, especially at high-resolutions. There are two ways to reduce this cost and still see accuracy benefits from high-resolutions: high-resolution finetuning, which pretrains models at lower resolution, like 224 px, then finetunes them at higher resolution, like 3842 px; and @ extrapolating, which deploys models at a higher resolution, without further training. Of these two options, should aim for models that can effectively extrapolate, as it presents zero-cost solution that does not require finetuning at every target resolution. Finetuning costs aside, improvements to extrapolation should benefit high-resolution finetuning since models that better at extrapolating can adapt to higher resolutions more easily. Although extrapolation is a significant and exciting challenge, ofte-of-the-are (SoTA) model architectures extrapolate poorly.
11 [4
Vision transformers (ViTs 9)) offer SoTA performance on many computer vision tasks. ViTs are simple; they split images into non-overlapping patches, linearly project pixels to form patch embeddings, and process these "tokens" with a stack of architecturally identical transformer layers maintaining constant feature map size throughout. This non- oon-hicaarc design enables learning
patch representations, which are useful for dense prediction tasks [2] 22 23 and are fundamental
AF, DGK, and YY made significant technical contributions. AF and DGK initiated the project. AF and JRG led project. Code data are available at: /tttpp////ghatttcco////ee///////
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ImageNet- Val 1
ImageNet-v2 2
ImageNet- A 3
20
Top-1 Aecuracy [Se]
80
76
15
70
10
85
60
224
320
384
448
512
768
1024
224
320
364
1024
224
320
381
4ls
s12
768
1024
ImageNet- ReaL
ImageNet- -R (ours)
ImageNet-RR R 5
90
Top-1 Accuracy [%
85
30
25
80
20
16
75
75
10
224
320
384
438
512
768
1024
768
221
320
381
48
5/2
768
1024
Test Resolution
Test Resolution
Test Resolution
LH-180 (ours)
LH-90 (ours)
LH-45 (ours)
Factorized
2D-RoPE
2D-sincos
ID-lcam
RPE-leam
10
Fourier
2D-ALiBi
PP
Figure I: VIT-B/16 models trained for 150 epochs on ImageNet at 224 px and tested up 1024 px.
Model architectures are consistent between runs other than position encoding methods. We perform an 8-run hyperparameter sweep, per method, to ensure fair comparisons. Our three LookHere variants improve extrapolation ability, with more narrow fields of view performing best at 10243.
for vision-language models 24 25 26
The design enables efficient processing of only a subset
of patches, known as token dropping [27 28). Lastly, it enables model scaling by increasing the
embedding size and the layer count [29 30
Image-size extrapolation with ViTs can be achieved in three ways: increasing the patch size, which packs more pixels into each patch embedding: @ increasing the "patchification" stride, which skips-over pixels; and @ increasing the number of patches. Of these three options, we should aim for models that can effectively ingest more patches called "sequence length extrapolation" the natural language processing (NLP) community [1 as greater number of patches presents models with more (uncompressed) information that we hope to leverage into higher accuracy. Furthermore, methods that improve sequence length extrapolation, like our proposed method, can be fused with methods that adjust patch sizes, like FicxiViT (32). We strongly believe that patch position encoding a primary cause of the poor sequence length extrapolation ability of ViTs like it NLP, where
significant advancements have been made by improving position encoding 31 33 34 351
Adding learnable fixed sinusoidal position embeddings to patch embeddings before the first layer is the most common way ViTs encode positions. Recently, the rotary position embeddings (RoPE (36)) were extended to ViTs, as 2D-RoPE [7). showing exciting results. RoPE is different approach to position encoding that injects positional information in each self-attention layer by rotating queries and keys with fixed sinusoidal embeddings. But for these methods to ingest more patches at test time, they must either introduce new position embeddings or modify existing embeddings both options create significant distribution shift. Motivated by these observations and more, we make the following contributions:
used in SoTA language models [37 38
LookHere- We introduce novel position encoding method for plain ViTs that restricts attention heads to fixed fields of view (FOV) and points them in different directions via 2D masks. This design O attention head diversity, @ improved interpretability, and ? limits the distribution shift that attention heads face when extrapolating.
provides: cranslatio-equiivriann
Controlled Experiments - We perform an apples-to-apples comparison between seven position encoding methods for plain VITs alongside our three LookHere variants. We demonstrate that LookHere: @ improves classification, segmentation, adversarial robustness, and model calibration

---

when tested at the training resolution: ① significantly improves performance when tested beyond the training resolution; and @ increases its performance advantage after high-resolution finetuning.
@ Extrapolation Insights - We show that extrapolation: @ henefits images with small objects the most, they occupy more patches at test time: O produces class-level and dataset-level effects; and creates distribution shifts that can be visualized via attention maps.
ImageNet-HHR - We introduce the first natively high-resolution ImageNet test set (10242 px) aimed to benchmark classifiers images that were not upsampled to achieve the target image size.
2 Background and Related Work
A ViT splits an image into a grid of non-overlapping patches, flattens the grid into a sequence, and flattens the patches into vectors; i.e., RYxXxC - R,,,,N,, P8xC + R (N, C), where Y is the image-height,
is the image-width, C' is the number of channels, N,, is the grid-height N₂ is the grid-width, P is the patch height and width. A linear layer maps each vector of pixels to a patch embedding; i.e., RRPa
ERputhh E R.D, where D is the embedding dimension also known as the transformer width. We define and (z) as the sequence position and the 2D position of the 7tH patch, respectively, where N is the total number of patches, equal to Ny NN., i E f1,2,.... N), by E (1, Ny and & E (1, No). Finally, sequence length extrapolation occurs when Neest > Ntrain.
A patch embedding represents the content of patch, and contains no information representing its original location within the image. Thus, we must encode patch positions to enable spatial reasoning: otherwise, a VIT will operate on bag of patches.
We define a "plain VIT as attention-only and non- non-hierarchical. Our primary goal is to improve the extrapolation ability i.e., generalize to more patches at test time - of plain ViTs. Our work is motivationally aligned with FiexiViT [32 and ViT [6. improving the flexibility of plain ViTs. Next, we briefly describe seven position encoding methods and refer the reader to the cited studies for further details; we include them all our controlled experiments, Another method, iRPE (39, is also compatible with plain ViTs. However, we exclude it because it is more than twice as slow as other methods; nonetheless, we benchmark iRPE with our best training recipe Appendix AA.2.
Input Embeddings. This group leverages learned or fixed position embeddings, F.p00 E TRP, that added to patch embeddings the transformer input; i.e., 2; Eaatch Ep where is the input to the first transformer layer. Position embeddings represent the absolute positions of patches in an image.
ID position embeddings 9 (1D-learn for short) map to learnable embeddings. 2D sinusoidal embeddings g (2D-sincos for short) individually map iy and i, to fixed ID-sinusoidal embeddings
IR 2) ). then concatenate them along the embedding dimension. Factorized position embeddings (6) (Factorized for short) individually map ły and ig learnable embeddings (FE F.y RD),, then add them. Learnable Fourier features [11] (Fourier for short) map (iy, iz) to Fourier features 40 41, then to embeddings with a multi-layer perceptron (MLP).
(E", E
Attention Biases. This group leverages learned or fixed operations that encode positions by modifying the pairwise interactions between patches in self-attention without adding position embeddings to patch embeddings. Recall that self-attention first applies three separate linear transformations to project internal patch representations and splits the resultant vectors into H smaller vectors of length patch
DHi i.c., ND + xx x denote a specific head by h. Next, attention scores (A E NxNNN are calculated by measuring the similarity between all pairs of queries (gh: E IR P")) and keys (kij E (RR"") )) separately, for each head; i.c., Obij 9hi Kag/ v DH. where and are query and key sequence positions, and define (iy, iz) and (Jy. j=) as their 2D positions. Attention scores (Aniz) represent the amount of information moving from patch position j to whereas values (Unj E RD) represent the content of the moving information.
creating queries, keys, and values for each attention head. We
Learnable relative position encoding I00 (RPE-learn for short) biases attention scores by mapping
all possible relative positions between queries and keys to learnable embeddings (Bi E RR ) i.e., biases are function of in jy. in ja. and h. O A 2D extension of Attention with Linear Biases
3

---

(ALiBi BT) 2D-ALiBi [12 penalizes attention scores as function of the Euclidean distance between (3y, in) and (1y, 3z). and head-specific scalar, called slope. Slopes bias attention heads at different rates. @ A 2D extension of rotary position embeddings (RoPE [6]), 2D-RoPE ( rotates queries and keys as a function of their positions. Each query is rotated by the sinusoidal embedding
of by for half its dimensions and the sinusoidal embedding of iz for the other half of its dimensions; likewise, keys are rotated as a function of jy and Ju-
Non-plain ViTs. Many hybrid or hierarchical architectures have been invented that often encode positions differently 42
51 52 53 Although these architectures may be favored in some cireumstances, the plain ViT is the most common single architecture due to its simplicity, flexibility, and scalability. We benchmark many non-plain ViTs and large SoTA ViTs on extrapolation in Appendix [9.2.
48 49
50
VIT Extrapolation. Some ViTs have been tested higher resolutions than they were trained 54
4 12 55). VaV [6] benchmarked input embedding methods on extrapolation, none see the
gains at higher resolutions that we observe.
LookHere
Design Motivation. We introduce 2D attention masks that assign each attention head direction and a FOV, preventing attention outside the head's s FOV. Within a head's s FOV, attention scores are penalized based on relative patch distances. Three ideas motivate this design. 1 Attention head diversity: heads often learn redundant algorithms that can be pruned with little accuracy penalty [56] 57 [58]. Head
redundancy has also been observed in NLP [59 60 61], where diversity-encouraginn loss functions
have been leveraged to improve generalization 662 63 64 65 From a mechanistic point of view,
we can think of attention heads as an ensemble of sub-networks that "operate completely in parallel, and each add their output back into the residual stream," " [66] and the residual stream is mapped to logits. Diversity has long been a desirable property of ensembles [67 668. and constraining attention heads to focus in different directions ensures it. @ Attention head consistency: heads often learn interpretable spatial algorithms, like "attend to the area above the query," which reliably retrieves information from the internal representations above the query; however, believe these types of spatial algorithms might fail when new or modified position embeddings are introduced to encode new patch positions during extrapolation misleading the model about the information above the query, for example. We believe hard-coding both directions and distances (via attention masks and biases) will reduce the need for models learn their own spatial algorithms. @ Translation-equivarian long been a desirable property of vision models, contributing to the success of convolutional networks 69 70 71. ViTs are critiqued for weak inductive biases, leading poor sample efficiency when trained from scratch 772, /7 74). We believe that LookHere's 's stronger inductive biases, achieved via directional masking and distance penalties, can improve ViT sample efficiency.
has
Design Specifics. Let H be the number of heads. L be the number of layers, and N be the number of patches (plus one for the CLS token). We denote the LookHere matrices by Anx (XX))xXX+)X(N++ We encode positions by subtracting the LookHere matrix for a layer 1, A/ix from the learned attention matrix, AiRN = QKT / /VDH. before the softmax that normalizes the attention matrix prior to multiplying it by values (75 i.e., A' = softmax (AL - Abx)). We do not add position embeddings patch embeddings.
Let and j be query and key sequence positions, respectively, with 2Dcoorriinates (ly. iz) and (u).a)) Crucially, j is visible to i if lies within i's FOV. This attention masking technique is inspired by the 1D causal masks used in autoregressive transformer decoders used in NI.P (75). When j is visible, we bias the attention score based the Fuclidean distance between and j to encode the relative distance between patches. We scale distances via a slope function m : Nr, x NH + R, m(1, h) (I) sn (h) 8y that strengthens or weakens the distance penalty as function of the head (Sh : NH + IR) and layer (st N2 + IR), scaled by a global slope 8, E IR. Finally, the CLS token is visible to all positions.
LookHere h) [m(l h). Distance(i..) if j is visible i otherwise
(1)
100
Distance V(ly (ig - j1y) + (ix ja)
(2)

---

ALRN QK
AFIX LookHere 2
VTx
X
655 (M0) (3.0) a d R2 (4.00
023 (1 dy 3; 13,00 11 (9/9)
negative
positive
19.0)
00.00
(0. 0)
(0, 1)
(0.
26.1
(1.)
x
query
0..00
(1, 0,
(1
16.0
01.13
19.20
01.80
28.1
(5.3)
U)
(2.
(2. 2)
33.4
(2.9)
Figure 2: LookHere masks and biases (center) the learned attention matrix (left, where colors are random). Masked cells are black, encoding directions (- with a 90 FOV); biased cells are shaded bluish-greed encoding relative patch distances. (Right) An example of the FOV of the center query patch. The final attention matrix is computed as A' softmax
- Afix), at each layer l.
For example, F displays attention matrices of head that "looks right" with 90 FOV. We create three LookHere variants, the first two have FOVs of 180 and 909 (LH-180 and LH-90). We direct attention heads eight different ways, selecting the four cardinal directions (1, <- -) and the four intercardinal directions (/, / K). ViT-B models have twelve attention heads; we leave the last four attention heads undirected to allow them unrestricted attention over the full image. We create a final variant that cuts the first four LH-90 masks in two, creating eight 45 views that cover the full image without overlapping (LH-45). Visualizations of the bias matrices in Appendix A.3
Design Ablations. We offer four takeaways through extensive ablations (Appendix A.6 LookHere is robust to the choice of slope function. We set our default to linearly decrease from 1.5 to 0.5 with increasing depth (inspired by depth-wise attention distance findings (76)). This helped in preliminary experiments, but the benefits disappear in our ablations. We arbitrarily set default 8y, to 2 8 32 128 for the four undirected heads, but distance penalties on undirected heads can be removed entirely. We set 8g 1; LookHere is also robust to the choice of the global slope. We believe precisely tuning slopes is unnecessary because models can learn to scale attention logit magnitudes. @ Increasing penalties with the square square root of the distance harms extrapolation. Removing all distance penalties harms extrapolation. O Our main contribution, 2D directional masks, are crucial retain performance, but method is robust to many directional configurations.
Compute.
AFxx is precomputed and fixed, subtracting it dementtwwise from the learned attention matrices AIRN only costs H (N + 1) (N 1) floating point operations (FLOPs) per layer. For a ViT-B/16 model, these subtractions account for 0.016% of the total FLOPs. LookHere reduces FLOPs by not adding position embeddings to patch embeddings, but this amount is also negligible. Additionally, LookHere matrices offer structured sparsity (up to 7/8 for 45 FOV) that can speedup attention although exciting, this speedup requires custom kernels that we leave for future work.
Experiments
Deep neural networks including ViTs - can be sensitive to seemingly minor hyperparameter changes when trained from scratch. Dosovitskiy et al. finetuned the original ViT higher resolution, reaching 77.9% top-1 accuracy on ImageNet (we refer to ILSVRC2012 or ImageNet- as ImageNet). Steiner al. [77] searched 28 hyperparameter configurations, achieving best and average runs of 80.0% and 76.9%, respectively (average calculation omits runs without data augmentation, as they were poor). Touvron
78 ablated repeat augmentation [79), dropping accuracy by 4.8%. Touvron et al. [17] replaced cross-entrope loss with binary cross- cross-entropy loss, raising accuracy by 1.3%. Importantly, these are ViT-B/16 models trained from scratch for 300 epochs on ImageNet. Informed by these observations and more, we design controlled experiment: We search hyperparameter configurations for each position encoding method using single codebase; this offers
comparison between our three LookHere variants and seven baselines.
5

---

4.1 Setup
Our 80 training runs result from the following Cartesian product:
Position encoding: 1D-lcam, 2D-sincos,
Augmentations: RandAugment(2. 15)
3 3-Augment
Factorized, Fourier, RPE-leam,, 2D-ALiBBi,
Learning rate: 1.5 10- 3.0 10
2D-RoPE, LH-180, LH-90, LH-45
Weight decay: 0.02, 0.05
For each configuration, as a validation set called "minival", following 77 81 (see appendix[A.44]]eee other hyperparameters). We train all models from seratch for 150 epochs on 224² px images. Our results are competitive and sometimes surpass VITs trained for much longer, which validates our setup. The best models (according minival accuracy), among our 8-run hyperparameter sweep per method, are always trained using 3-Augment [17]. a 3.0 10 learning rate, and 0.05 weight decay.
train VTT-B/16 on 99% of the ImageNet training set, holding the last 1%
Test sets. We test all 80 models six ImageNet sets. This includes the original "validation" set used as a test set (Val for short []. @ the reassessed labels of the original validation set (Real. for
short [4) the independently collected and in-distribution test (v2 for short (2), the natural
adversarial test set (-A for short (3). @ the ImageNet rendition test set (-R for short (5]), and O the high-resolution test set that we introduce (-HR for short).
toucan
shovel
balloon
Figure 3: Images three classes from ImageNet-HR. (Bottom left is Anthony's niece Addison.) ImageNet-HRR, Since there are no natively high-resolution ImageNet test sets, there are two options to test the extrapolation ability of models trained on ImageNet: upsample existing test sets to higher resolutions, and @ collect high-resolution test set ourselves. However, upsampling low- resolution images introduces another distribution shift (i.e., interpolated pixels) that we may not want to test. Thus, we collect a high-resolution test set to remove this confounding variable from our analysis. We manually collect images for each ImageNet class, resulting and manually crop them to 1024 This is smaller than other test sets (v2 is 30k images, -A is 7.5k images). However, we invest considerable resources to ensure its quality with two priorities: annotation accuracy and image diversity. See Appendix A. for details. ImageNet-HR can be accessed: https: htttss//huuginggaaee. wtt.://.e..c.aaaatttttteoaccssee attplsss///oofuuuo www.rr/llteemmmmeeeetttttt
total images,
Adversarial Attacks, We perform Fast Gradient Sign Method (FGSM 82) adversarial attacks with 11.. 3
two strengths (255 255 on all models using Val images.
Calibration Estimates. We calculate the Expected Calibration Error (ECE models using Val images.
83 with 15 bins of all
Higher- Resolutin Finetuning. With the best model per method, we continue training on ImageNet for 5 epochs at 3842 px. We test at 3842 px without extrapolating.

---

Segmentation. With the best model per method, we finetune following the Segmenter protocol with a linear decoder 84 Additionally, we probe the patches by only training a linear layer to produce a low-resolution logit map which is upsampled to obtain a full resolution segmentation map. following We run these experiments on ADE20K [6 at 5122 px and Cityscapes [87 at 7682 px.
85
Patch Logit-lens. Inspired by interpretability research [88], we evaluate the quality of the learned patch representations for models leveraging LookHere compared with other methods. Following prior work [8 90]. we project frozen patch representations onto the learned class embedding space using the MLP classifier head that was learned for the CLS token. We leverage the ImageNet-S dataset [9]], which contains partial segmentation maps for 12k images from Val, covering 919 ImageNet classes.
Extrapolating. With the best model per method, test on images larger than 224 px, increasing the number of patches and we test on images smaller than 224 px, decreasing the number of patches; for both experiments, no further training is performed the models are tested on their resolution generalization ability. For 1D-learn and 2D>-sincos, we bilinearly interpolate the position embeddings used during training. For Factorized, we linearly interpolate the position embeddings for each axis. Fourier does not require adjustment since fractional positions along each axis are used as input. For RPE-learn, we interpolate the learned relative biases using the official BEiT implementation [T). 2D-ALiBi does not require adjustment either. However, we tune a parameter on minival that scales the distance penalty at each test resolution. For 2D-RoPE, we tune its base frequency on minival this is a SoTA method to extrapolate RoPE used in NLP [33]. Lastly for LookHere, we tune the global slope on minival. The benefits of tuning slopes are minimal, see Appendix A.4.4
4.2 Results and Analysis
Table 1: Top-1 acc. (%) for ViT-B models trained on ImageNet for 150 epochs; trained and tested at 224 We report the best and average results across our 8-run hyper-parameter sweep.
Val M
ReaL [
v2 (
-A 3
-R [5]
-HR (ours)
Method
Best
Avg.
Best
Avg.
Best
Avg.
Best
Avg.
Best
Avg.
Best
Avg.
1D-learn
79.45
77.35
84.97
82.87
68.49
65.17
10.97
7.58
29.64
25.73
88.28
85.22
2D-sincos
79.05
77.44
84.62
82.96
67.86
65.31
10.45
7.76
29.11
26.07
87.58
85.36
Factorized
79.86
77.29
85,30
82.99
69.11
65.34
11.00
7.16
29.99
26.18
87.86
85.37
Fourier
79.69
77.37
85.13
82.89
68.30
65.33
11.36
7.79
29.73
24.62
88.14
85.39
RPE-learn
79.86
77.26
85.46
82,88
68.57
65.19
9.85
7.18
29.10
24.62
88.22
85.17
2D-ALiBi
79.54
77.29
85.15
82.92
68.47
65.15
10.45
7.27
28.26
24.41
87.70
85.13
2D-RoPE
80.38
78.37
85.64
83.78
69.34
66.56
13.03
8.84
32.45
28.55
88.78
86.35
LH-180
81.31
80.01
86.53
85.30
70.70
68.52
13.53
10.45
32.10
28.94
89.86
87.80
LH-90
81.02
79.89
86.44
85,28
70.28
68.54
13.15
10.80
31.77
29.47
89.90
87.86
LH-45
81.06
79.74
86.23
85.07
69.65
68.18
13.41
10.21
32.12
29.51
89.46
87.43
LookHere improves ViT sample efficiency (Table Our three variants outperform the best baseline. 2D-RoPE, under almost all test conditions (the single exception being the best 2D-RoPE model on -R). LookHere further improves gains when considering averaged results i.c.. when accuracy values are averaged over 8 hyperaaraeeter configurations (please see the Appendix A.5 for individual results). For instance, LH-180 outperforms 2D-RoPE by 0.93% 1.36% on on Val v2 on our best runs and by 1.6 1.64% / 1.96% on Val / v2 on our averaged runs indicating that LookHere decreases hyperparameter sensitivity. Surprisingly, LH-1880 averages 80.01% on Val, which matches the best run trained for twice as long by Steiner et al. 77
Head Diversity by Layer
LookHere improves ViT adversarial robustness and model calibration (Tables 2[: both have been linked to ensem-
0.7
ble diversity 99 93 94 which we offer as a potential
0.6
IS Divergenes
explanation. This is an interesting finding because ad- versarial robustness and calibration can be at odds with accuracy 195. (96. We show that LookHere learns more di- verse attention heads by measuring the generalized Jensen-
0.5
0.3
Shannon divergence [97] between heads (Figure 4. In the Appendix A.8] we measure more properties of models leveraging different position encoding methods. LookHere significantly outperforms other methods on segmentation linear probing, demonstrating its ability to learn spatially-
0.2
Layer
Figure 4: LookHere learns more diverse attention heads and prevents attention
7
collapse. Legend follows Figures

---

Table 2: Fast Gradient Sign Method at- tack 82 (% top-5 acc. on Val), best and average runs.
Table 3: Expected
Table 4: Semantic Segmentation Calibration Error % (% mloU), linear probing (LP)
83 (L) on Val, best
and finetuning (FT).
1/255
3/255
and average
ADE20k
Cityscapes
Method
Best
Avg.
Best
Avg.
Best
Avg.
LP
FT
LP
FT
1D-learn
58.87
54.36
44.23
41.37
10.13
12.21
29.5
38.05
47.1
72.93
2D-sincos
60.38
55.16
45.37
41.61
10.14
11.85
29.2
38.39
45.3
72.91
Factorized
60.86
56.19
46.34
42.32
10.01
11.37
29.4
37.95
45,9
72.51
Fourier
59.91
54.74
44.99
41.90
9.65
12.13
29.8
38.26
46.2
73.60
RPE-leamn
59.81
53.36
45.04
40.19
8.66
11.42
26.4
37.25
42.9
73.87
2D-ALiii
58.07
53.68
43.32
40.30
9.26
11.24
26.2
37.56
48.4
73.92
2D-RoPE
60.59
57.16
47.11
43.77
9.60
11.48
29.9
39.74
47.0
75.53
LH-180
65.06
62.59
51.81
49.06
8.28
9.76
32.4
40.29
55.0
75.05
LH-90
63.89
61.88
50.87
48.07
8.68
9.91
32.6
40.60
55.3
74.90
LH-45
64.71
61.71
50.21
47.86
8.87
9.99
32.7
40.07
55.5
74.42
aware patch representations. LookHere also performs well with segmentation finetuning, achieving comparable performance 2D-RoPE (Table 4p.
Table 5: Top-1 acc. (%) for models trained at 224 px, finetuned and tested at 384 px.
High-resolution finetuning increases the perfor- mance advantage of all three LookHere variants over 2D-RoPE (Table 5). This aligns with our intuition that improving extrapolation methods can improve high-resolution finetuning. Lower initial finetuning loss has been linked to bet- ter retaining the general representations learned during pretraining (98). and better extrapolat- ing models have lower initial loss at higher- resolution, by definition.
Method
Val
Real.
-v2
-A
-R
-HR
ID-learn
81.46
86.46
70.69
18.80
29.80
89.82
2D-sincos
81.33
86.50
70.53
17.73
29.26
89.62
Factorized
81.50
86.62 70.95
18.05
29.98
89.50
Fourier
81.71
86.73 71.01
19.73
29.68
89.90
RPE-learn
82.01
87.17 71.66
18.13
29.53
90.20
2D-ALiBi
81.41
86.73 70.50
18.01
28.60
89.46
2D-RoPE
82.31
87.21 71.82
21.68
33.38
89.92
Using a "logit lens" [8] approach, we project patch representations onto the class embedding space [89). We observe that LookHere encodes semantic information in its patches faithful to the original patch location; these patch-level predictions act as segmentation map that can be generated without additional training. The officer in Figure is not one-off example; using ImageNet-S (91), we see that LookHere outperforms 2D-RoPE by at least 22% mloU using this natch-projection method (Figure 5) Our best explanation is that, by restricting attention, LookHere
LH-180
83.28
88.05 73.12
22.85
32.95
91.38
LH-90
83.08
87.99
72.99
23.51
32.63
91.24
LH-45
83.10
87.83 72.43
22.39
33.10
90.92
1D-learn (4.1%) 2D-sincos (2.1%) Factorized (8. 7%)
Fourier (3.3%)
RPE-learn (6.1%)
2D-ALiBi (6.5%) 2D-RoPE (8.7%) LH-180 (31.2%) LH-90 (31.1%) LH-45 (30.7%)
Figure 5: apply frozen MLP classifying heads (learned on the CLS token) on frozen patch representations. We visualize ImageNet class predictions: assault rifle (red), bulletproof vest (green), crash helmet (blue), and holster white In parentheses, we show mloU results @ 224px) on ImageNet-S 191], where we apply this technique to segment images without training.

---

ImageNet- Val
ImageNet-v2
ImageNct-A A
Top-1 Accuracy (%6
55
50
90
45
40
50
35
30
40
25
35
Ju
25
224
112 128 iás
192 208 224
ImageNet- ReaL
ImageNet-HH HR
ImageNet-R
Top-1 Accuracy 8/
85
30
25
65
70
80
20
55
15
50
45
50
10
40
35
30
25
96
112
144
176
192
224
160 192 Test Resolution
112 128
64
96 112
128
144
176 192 208
Test Resolution
Test Resolution
LII-180 (ours)
LH (ours)
LH1-45 (ours)
Factorized
6
2D- DD-RoPE
Figure 6: ViT-B/16 models trained for 150 epochs on ImageNet at 224 px and tested down to 64 px. Model architectures are consistent between other than position encoding methods.
prevents the attention collapse at deeper layers observed in Figure(4jthat divorces patch representations from their original patch locations; this collapse has been observed in other ViTs [99] [00]. We also expect that preventing attention collapse will benefit vision-language models, where frozen patch representations are used as "image tokens" that should represent their original patch locations Appendix A.7
24 25 26).
More examples and detailed analysis
LookHere significantly improves extrapolation ability (Figure T.. Our smallest FOV variant (LH-45) sees improving relative performance resolution increases. LH-45 outperforms 2D-ALiBi, which is equivalent to LookHere without our 2D directional masks, by 9.5% on Val at 1024 px. These two results demonstrate the extrapolation benefits of restricting attention fixed FOVs. LH-45 gains 1.3% on Val when extrapolating from 224 to 384² px; this is the largest gain we find in the literature, including our extensive benchmarking of SoTA models in Appendix A.2 LookHere also outperforms other methods when tested on smaller images, but the advantage narrows (Figure
Interestingly, smaller objects benefit most from extrapolation (Figure 7). which are distributed over more patches at test time. We believe this effect also explains the 6 - 8% that LookHere models gain when extrapolating on ImageNet-A from 224² to 4482 px; by inspection. ImageNet-A seems to have small objects, and other work found zooming-in on center-cropped ImageNet-A images improves
Ace. 224 + 384
Acc. 224 + 5122
Acc. 224
7682
Aec. 224
1024
Top-1 Accuracy [56]
10
-10
20
25
330
-20
131
4
1
34
1
3dd
4
14
124
gid
4
Object Size (Quintile)
L1I. 180 (ours)
LII 90 (ours)
451.45 (ours)
Factorized
2D RoPE
RPE
2D
Figure 7: The effect of object size on accuracy gains or losses due to extrapolation. Object size is measured using annotations from Kaggle's ImageNet Object Localization Challenge [01].

---

performance [022]. Finally, all LookHere variants outperform other methods on ImageNet- HR, indicating better handling of interpolated pixels generated when upsampling lower-resolution imagery is not the reason why LookHere extrapolates better.
Reducing the distribution shift faced by attention heads during extrapolation is our best explana-
224
5122
768
1024
tion for LookHere's ' large relative improvement. Figure 8 shows attention maps that are "unflat- KPE-leur tened" to visualize the image regions to which heads attend, averaged over the same 5k im- ages. We show one head per model that exhibits similar behavior at a 224² resolution. Models AD-ALIB
leveraging RPE-learn and 2D-ALiBi learn vari- ants of an algorithm that retrieve information from above the query; however, both models re- trieve information elsewhere in the image when extrapolating. LookHere hard-codes this type of algorithm, which it continues to execute when extrapolating. In Appendix A.8 we find more ex- amples of interesting attention head behaviour.
LIE-00
Figure 8: Attention maps of three attention heads across four resolutions, where the query is in the center. We use the colormap:
Extrapolation affects different datasets differ- ently; it also affects different classes differently. For example, when extrapolating, all models underpredict certain classes (bakery, church, and tights) and overpredict other classes (mobile home, threshing machine, and sports car). This investigation is inspired by the class-level effects of data augmentation [103] In Appendix A.9 we find more class-level effects extrapolation.
Closing
Limitations. The primary limitation of LookHere is it requires hand-designed directional masks and distance penalties. However, our extensive ablations demonstrate that LookHere is robust to the choice of directional masks and distance penalties. The primary limitation of our experiments is we do not scale ViTs to giant sizes. Instead, we select the most common size, the ViT-B/16, and focus computational resources on controlled experiment that extensively and fairly tunes the appropriate baselines for plain ViTs; this allows us to make confident conclusions based on our thorough experiments.
Conclusion. LookHere position encoding significantly improves the ability of plain ViTs to make inferences when provided greater number of patches than seen during training. We thoroughly demonstrate that LookHere outperforms other methods with and without extrapolation on standard image benchmarks and our high-resolution ImageNet test set called ImageNet-HiP We provide new insights into ViT extrapolation by showing object-size, class-level, and dataset-level effects. We believe LookHere will help the vision community transform higher-resolution into higher accuracy.
Future Work. We are excited to realize computational gains that LookHere makes available via sparse attention kernels, well as bring LookHere to video and 3D point-cloud applications.
Acknowledgments. Anthony thanks NSERC's Postgraduate Scholarships Doctoral program for funding his PhD.
10

---

References
[1] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 2015.
[2] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet Classifiers Generalize to ImageNet? In International Conference on Machine Learning (ICML), 2019.
[3] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural Adversarial Examples. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[4] Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with ImageNet? arXiv preprini arXiv:2006.07/55 2020.
[5] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The Many Faces of Robustness: A Critical Analysis of Out-of-Distribttion Generalization. In International Conference on Computer Vision (ICCV), 2021.
[6] Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim Alabdulmohsin, Avital Oliver, Piotr Padlewski, Alexey Gritsenko, Mario Lučić, and Neil Houlsby. Patch n' Pack: NaViT, Vision Transformer for any Aspect Ratio and Resolution. In Neural Information Processing Systems (NeurIPS), 2023.
[7] Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, and Jérôme Revaud. CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow, In International Conference on Computer Vision (ICCV), 2023.
[8] Zelun Wang and Jyh-Charn Liu. Translating math formula images to LaTeX sequences using deep neural networks with sequence-level training. International Journal on Document Analysis and Recognition, 2021.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale. In International Conference on Learinng Representations (ICLR), 2021.
[10] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT: BERT Pre-Training of Image Transformers. In International Conference on Learning Representations (ICLR), 2022.
[11] Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding. Neural Information Processing Systems (NeurlPS), 2021.
[12] Anthony Fuller, Koreen Millard, and James R Green. CROMA: Remote Sensing Representa- tions with Contrastive Radar-Optical Masked Autoencoders. In Neural Information Processing Systems (NeurIPS), 2023.
[13] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. In Neural Information Processing Systems (NeurIPS), 2019.
[14] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. arXiv preprint arXiv:1811.06965 2019.
[15] Mingxing Tan and Quoe Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In International Conference on Machine Learning (ICML), 2019.
[16] Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, and Herve Jegou. Three things everyone should know about Vision Transformers. arXiv preprint arXiv:2203.09795, 2022.
11

---

[17] Hugo Touvron, Matthieu Cord, and Hervé Jégou. DeiT III: Revenge of the VIT. In European Conference on Computer Vision (ECCV), 2022.
[18] Brandon McKinzie. Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, et al. MMI: Methods, Analysis & Insights from Multimodal LLM Pre-training. arXiv preprint arXiv:2403.09611 2024.
[19] Zhang Li, Biao Yang. Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[20] Penghao Wu and Saining Xie. V*: Guided Visual Search as Core Mechanism in Multimodal LLMs. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[21]
Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, et al. Vig Semantic Segmentation with Plain Vision Transformers. In Neural Information Processing Systems (NeurIPS), 2022.
[22] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, et al. Simple peen-Vocabulary Object Detection. In European Conference on Computer Vision (ECCV), 2022.
[23] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring Plain Vision Transformer Backbones for Object Detection. In European Conference on Computer Vision (ECCV), 2022.
[24] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. In Neural Information Processing Systems (NeurIPS), 2023.
[25] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li. Pascale N. Fung, and Steven Hoi. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. In Neural Information Processing Systems (NeurIPS), 2023.
[26] Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, et al. Language Is Not All You Need: Aligning Perception with Language Models. In Neural Information Processing Systems (NeurIPS), 2023.
[27] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. ATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text. In Neural Information Processing Systems (NeurIPS), 2021.
[28] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[29] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling Vision Trans- formers. In Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[30] Ibrahim M. Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting VIT in Shape: Scaling Laws for Compute-Optimal Model Design. In Neural Information Processing Systems (NeurIPS), 2023.
[31] Ofir Press, Noah Smith, and Mike Lewis. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. In International Conference on Learning Representations (ICLR), 2022.
[32] Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiao- hua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. FlexiViT: One Model for All Patch Sizes. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[33] Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling Laws of RoPE- based Extrapolation. In International Conference on Learning Representations (ICLR), 2024.
[34] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The Impact of Positional Encoding on Length Generalization in Transformers. In Neural Information Processing Systems (NeurIPS). 2023.
[35] Dawei Zhu, Nan Yang. Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. PoSE: Efficient Context Window Extension of LLLMs via Positional Skip-wise Training. In International Conference on Learning Representations (ICLR), 2024.
12

---

[36]
Jianlin Su, Murtadha Ahmed, Yu Lu. Shengfeng Pan, Wen Bo,, . and Yunfeng Liu. RoFormer: Enhanced transformer with Rotary Position Embedding. Neurocomputiigg 2024.
[37] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, et al. ALaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302. 2tt.222322.9999 13911, 2023.
[38] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, et al. Mixtral of Experts. arXiv preprint arXiv:2401.04088 2024.
[39] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and Improving Relative Position Encoding for Vision Transformer. International Conference on Computer Vision (ICCV), 2021.
[40] Ali Rahimi and Benjamin Recht. Random Features for Large-Scale Kernel Machines. In Neural Information Processing Systems (NeurIPS), 2007.
[41] Ali Rahimi and Benjamin Recht. Weighted Sums of Random Kitchen Sinks: Replacing minimization with randomization in learning. In Neural Information Processing Systems (NeurPPS), 2008.
[42] Juhong Min, Yucheng Zhao, Chong Luo, and Minsu Cho. Peripheral Vision Transformer. In Neural Information Processing Systems (NeurIPS), 2022.
[43] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional Positional Encodings for Vision Transformers. In International Conference on Learning Representaatinss (ICLR), 2023.
[44] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Bain- ing Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In International Conference on Computer Vision (ICCV), 2021.
[45] Qihang Fan, Huaibo Huang, Xiaoqiang Zhou, and Ran He. Lightweight Vision Transformer with Bidirectional Interaction. In Neural Information Processing Systems (NeurIPS), 2023.
[46] Yulong Shi, Mingwei Sun, Yongshuai Wang, Rui Wang, Hui Sun, and Zengqiang Chen. FVIT:
A Focal Vision Transformer with Gabor Filter. arXiv preprint arXiv: 13303, 2024.
[47] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-Scale Conv-Attentional Image Transformers. In International Conference Computer Vision (ICCV), 2021.
[48] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal Self-attention for Local-Global Interactions in Vision Transformers. In Neural Information Processing Systems (NeurIPS), 2021.
[49] Minghao Chen, Kan Wu, Bolin Ni, Houwen Peng, Bei Liu, Jianlong Fu, Hongyang Chao, and Haibin Ling. Searching the search space of vision transformer. In Neural Information Processing Systems (NeurIPS), 2021.
[50] Stéphane d' Ascoli, Hugo Touvron, Matthew L. Leavitt, Ari S. Morcos, Giulio Biroli, and Levent Sagun. ViT: Improving Vision Transformers with Soft Convolutional Inductive Biases. In International Conference on Machine Learning (ICML), 2021.
[51] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang. Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the Design of Spatial Attention in Vision Transformers. In Neural Information Processing Systems (NeurIPS), 2021.
[52] Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. Metaformer Baselines for Vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[53] Alaacldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. XCIT: Cros-Coovriaance Image Transformers. In Neural Information Processing Systems (NeurIPS), 2021.
[54] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin Transformer V2: Scaling Up Capacity and Resolution. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
13

---

[55] Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary Position Embedding for Vision Transformer. arXiv preprint arXiv:2403. 2024.
[56] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing Spar- sity in Vision Transformers: An End-to-End Exploration. In Neural Information Processing Systems (NeurIPS), 2021.
[57] Lu Yu and Wei Xiang. X-Pruner: eXplainable Pruning for Vision Transformers, In Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
[58] Fang Yu, Kun Huang, Meng Wang, Yuan Cheng, Wei Chu, and Li Cui. Width & Depth Pruning for Vision Transformers. In AAAI Conference on Artificial Intelligence, 2022.
[59] Paul Michel, Omer Levy, and Graham Neubig. Are Sixteen Heads Really Better than One? In Neural Information Processing Systems (NeurIPS), 2019.
[60] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi- head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Associa- tion Computational Linguistics (ACL), 2019.
[61] Maximiliana Behnke and Kenneth Heafield. Losing Heads in the Lottery: Pruning Transformer. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2020.
[62] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In International Confer- ence on Learning Representations (ICLR), 2017.
[63] Jian Li, Zhaopeng Tu, Baosong Yang. Michael R. Lyu, and Tong Zhang. Multi-Head Attention with Disagreement Regularization. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2018.
[64] Jian Li, Xing Wang, Zhaopeng Tu, and Michael R. Lyu. On the diversity of multi-head attention. Neurocomputing, 2021.
[65] Yao Huang, Xiaojun Chang, and Alexander Hauptmann. Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2019.
[66] Nelson Elhage. Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dode Dodds, Danny Hernandez. Andy Jones, Jackson Kernion. Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A Mathematical Framework for Transformer Circuit. In Transformer Circuits Thread, 2022.
[67] Prem Melville and Raymond Mooney. Diverse ensembles for active learning. In International Conference on Machine Learning (ICML), 2004.
[68] Gavin Brown, Jeremy Wyatt, Rachel Harris, and Xin Yao. Diversity creation methods: a survey and categorisation. Information Fusion, 2005.
[69] Kunihiko Fukushima and Sei Miyake. Neocognitron: A new algorithm for pattern recognition
tolerant of deformations and shifts in position. Pattern recognition, 1982.
[70] Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten Digit Recognition with a Back Propagation Network. Neural Information Processing Systems (NeurIPS), 1989.
[71] Jeffrey Wood and John Shawe- Taylor. Representation theory and invariant neural networks. Discrete applied mathematics, 1996.
[72] Zhiying Lu. Hongtao Xie, Chuanbin Liu, and Yongdong Zhang. Bridging the gap between vision transformers and convolutional neural networks on small datasets. In Neural Information Processing Systems (NeurIPS), 2022.
[73] Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. In International Conference on Learning Representations (ICIR), 2022.
[74] Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. Neural Information Processing Systems (NeurIPS), 1.21.
14

---

[75] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Neural Information Processing Systems (NeurIPS), 2017.
[76] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovit- skiy. Do Vision Transformers See Like Convolutional Neural Networks? In Neural Information Processing Systems (NeurIPS), 2021.
[77] Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers. Transactions on Machine Learning Research, 2022.
[78] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training eita-effiicent image transformers & distillation through attention. In International Conference on Machine Learning (ICML), 2021.
[79] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: better training with larger batches. In Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
[80] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Conference on Computer Vision and Pattern Recognition (CVPR) Workshop, 2020.
[81] Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain ViT baselines for ImageNet-Ik. arXiv preprint arXiv:2205.01 08500, 2022.
[82] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations (ICLR),
2015,
[83] Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining Well Calibrated Probabilities Using Bayesian Binning. In AAAI Conference on Artificial Intelligence, 2015.
[84] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for Semantic Segmentation. In International Conference on Computer Vision (ICCV), 2021.
[85] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint 2rX1v:2304.07193 2023.
[86] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 2019.
[87] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[88] Nostalgebraist. Interpreting gpt: The logit lens. https://www wttp://wwww.ssswo..ocoooo/oo/// http://pwwaaaooooutnnttetreeeeee
[89] Martina G. Vilas, Timothy Schaumlöffel, and Gemma Roig. Analyzing Vision Transformers for Image Classification in Class Embedding Space. In Neural Information Processing Systems (NeurIPS), 2023.
[90] Sonia Joseph. Vit prisma: A mechanistic interpretability library for vision transformers. /ttps://github. com/ www.aaaoseeevvittrrriiiiie 2023.
[91]
Shanghua Gao, Zhong- Yu Li, Ming-Hsuan Yang, Ming-Ming Cheng, Junwei Han, and Philip Torr. Large-scale Unsupervised Semantic Segmentation. JEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMII) 2022.
[92] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Neural Information Processing Systems (NeurIPS), 2017.
[93] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift, In Neural Information Processing Systems (NeurIPS), 2019.
15

---

[94] Tianyu Pang. Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning (ICML), 2019.
[95] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On Calibration of Modern Neural Networks. In International Conference on Machine Learning (ICML), 2017.
[96] Dimitris Tsipras, Shibami Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations (ICLR), 2019.
[97] Jianhua Lin. Divergence measures based on the Shannon entropy. IEEE Transactions on Information theory, 1991.
[98] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution In International Conference on Learning Representations (ICLR), 2022.
[99] Namuk Park. Wonjae Kim, Byeongho Heo, Tackyung Kim, and Sangdoo Yun. What Do Self-Supervised Vision Transformers Learn? In International Conference on Learning Representations (ICLR), 2023.
[100] Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind. Stabilizing transformer training by preventing attention entropy collapse. In International Conference Machine Learning (ICML), 2023.
[101] Wendy Kan Addison Howard, Eunbyung Park. Imagenet object localization challenge. https 2018.
/kaggle..oom com/
[102] Mohammad Reza Taesiri, Giang Nguyen, Sarra Habchi, Cor-Paul Bezemer, and Anh Nguyen. Imagenet-hard: The hardest images remaining from a study of the power of zoom and spatial biases in image classification. In Neural Information Processing Systems (NeurIPS), 2023.
[103] Polina Kirichenko, Mark Ibrahim, Randall Balestriero. Diane Bouchacourt, Shanmukha Ra- makrishna Vedantam, Hamed Firooz, and Andrew G Wilson. Understanding the detrimental class-level effects of data augmentation. In Neural Information Processing Systems (NeurIPS), 2023.
[104] Ross Wightman. Pytorch image models.
https://ggituub.com////ggg/
sntorch-image-modee 2019.
[105] Alexandra Sasha Luccioni and David Rolnick. Bugs in the data: How ImageNet misrepresents biodiversity. In AAAI Conference on Artificial Intelligence, 2023.
[106] Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BEiT v2: Masked image modeling with vector quatti-- visual tokenizers. arXiv preprint arXiv:2208.066 06366, 2022.
[107] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang. Tiejun Huang, Xinlong Wang, and Yue Cao. EVA: Exploring the limits of masked visual representation learning at scale. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[108] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin. Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning (ICML), 2021.
[109] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICIR)), 2019.
[110] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations (ICLR), 2018.
[111] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In International Conference on Computer Vision (ICCV), 2019.
16

---

A Appendix supplemental material
A.1 ImageNet-HR
We invest considerable resources to ensure ImageNet-HR's quality with two priorities. Annotation accuracy we only include images for which we are confident of their la- bel; we achieve this by: @ rounds of quality control consisting of manually re- viewing all cases where models disagreed with our annotations, using SoTA model hww.larrr_ppacchhe.11 that disagrees more often (tiny_vit_ 5m_ 224
fat___@mm_f__ii22___ wim.mimmmmmmm... ft_ ft_in22k_inik from timm (104) and a weaker model
from timm [04
con-
sulting someone with wildlife expertise to limit the annotation errors made by other test sets [00S. using multiple labels where necessary, for example, combining the "sunglass" and "sunglasses" classes, and labeling "tusker" as also an "Asian elephant," if the image of the tusked animal is an Asian elephant. @ Image diversity - when collecting images, we try to maximize the diversity of images belonging to class. Models achieve high accuracy on ImageNet-HR, likely due to less label ambiguity than other ImageNet test sets. Finally, we manually crop all images to 1024 px, resulting in the first natively high-resolution ImageNet test set.
We collect the vast majority of images from flickr and Unsplash] Unsplash images "are made be used freely" for commercial and non-commercial uses. flickr images were selected from the "All creative commons" license option. However, for some classes, we could not find enough open-access high-resolution images like "oil filter" or "hand or block plane," so we used Google search to find more. We estimate that around of 5k images were collected on flickr or Unsplash. Nine images were taken by author his family, with consent of everyone involved.
17

---

A.2
Extrapolation Results
Table 6: Extrapolation results for ViT-B models trained on ImageNet for 150 epochs; trained at 224 and tested various resolutions.
Val
ReaL
v2
-A
3
-R
[S]
-HR (ours)
Method
Res
top-1
top-5
top-1
top-5
top-1
top-5
top-1
top-5
top-1
top-5
top-1
top-5
ID-learn
320
79.89
94.59
85.17
96.26
68.78
87.88
12.91
34.36
26.73
40.04
88.76
96.94
2D-sincos
320
79.48
94.50
84.76
96.34
68.23
87.69
12.52
33.59
26.30
39.84
87.94
96.76
Factorized
320
79.73
94.70
85.08
96.37
68.71
87.83
11.44
32.01
25.89
39.33
87.78
97.08
Fourier
320
79.80
94.58
85.17
96.45
68.43
87.95
12.72
34.16
26.33
39.64
88.40
97.08
RPE-leam
320
79.94
94.73
85.50
96.63
68.56
87.83
11.17
30.51
24.21
36.71
88.12
97.26
2D-ALiBi
320"
80.44
95.10
85.66
96.66
69.25
88.65
13.33
35.00
26.54
40.47
88.62
97.36
2D-RoPE
320
81.40
95.36
86.43
96.77
70.38
88.94
16.08
39.52
29.76
44.21
89.38
97.38
LH-180
320
82.65
95.88
87.63
97.28
72.03
89.78
18.04
40.28
30.26
43.01
90.60
97.98
LH-90
320²
82.22
95.66
87.44
97.13
72.15
89.62
17.92
40.28
30.33
43.34
90.36
97.94
LH-45
320
82.45
95.84
87.45
97.17
71.99
89.92
17.99
40.96
30.53
43.31
90.30
97.66
1D-learn
3842
79.02
94.14
84.42
95.99
67.47
87.13
11.97
32.23
23.66
36.93
87.36
96.74
2D-sincos
384
78.56
94.06
83.95
96.02
66.96
87.16
11.87
31.31
23.53
36.63
87.02
96.38
Factorized
3842
78.56
94.06
84.01
95.94
66.62
87.21
10.41
29.72
23.03
35.69
86.90
97.00
Fourier
3842
78.85
94.07
84.30
96.07
67.43
87.33
12.63
32.59
23.31
35.95
87.48
96.84
RPE-learn
3842
77.96
93.64
83.72
66.54
86.51
9.51
26.85
19.41
30.38
86.38
96.44
2D-ALiBi
38/12
80.38
94.93
85.49
96.51
69.21
88.34
14.99
37.88
24.45
38.02
88.30
97.24
2D-RoPE
384
81.16
95.27
86.20
96.71
70.27
88.91
17.60
41.16
27.48
41.05
88.70
97.44
LH-180
384
82.38
95.79
87.35
97.25
72.15
89.67
19.09
42.32
27.58
39.97
89.94
97.84
LH-90
3842
82.08
95.70
87.26
97.18
71.50
89.74
19.93
42.68
27.99
40.58
90.10
97.66
LH-45
38-12
82.38
95.85
87.32
97.15
71.98
90.00
19.73
43.29
28.38
40.95
90.16
97.76
1D-leam
448a
77.52
93.45
83.06
95.47
65.77
86.10
10.44
28.79
20.93
33.86
86.04
96.36
2D-sincos
448
77.12
93.54
82.80
95.52
65.11
85.62
10.61
29.32
20.84
33.66
85.78
96.24
Factorized
418
76.98
93.33
82.63
95.32
65.06
85.70
9.67
26.63
20.21
32.50
86.00
96.46
Fourier
4482
77.47
93.46
83.05
95.51
65.57
86.05
11.37
29.59
20.57
32.76
86.18
96.46
RPE-learn
4482
75.40
92.41
81.45
94.87
63.40
84.29
8.11
23.79
15.31
25.31
84.18
95.44
2D-LLiBI
4482
79.63
94.61
84.74
96.32
68.66
87.86
15.13
37.79
22.25
35.13
87.32
96.96
2D-RoPE
448"
80.47
94.92
85.67
96.47
68.99
88.27
17.84
41.92
24.72
37.36
88.02
97.20
LH-180
448a
81.86
95.56
87.05
97.16
71.05
89.62
19.84
43.35
24.90
36.83
88.80
97.74
LH-90
448
81.91
95.54
87.00
97.08
71.39
89.49
20.57
43.39
25.71
37.70
89,20
97.52
LH-45
4482
82.19
95.67
87.02
97.08
71.93
89.58
20.77
43.76
25.97
38.35
89.62
97.66
1D-leam
5122
75.89
92.54
81.41
94.79
63.31
84.37
8.95
25.65
18.65
31.03
84.44
96.02
2D-sincos
512
75.43
92.57
81.26
94.87
62.74
84.31
25.27
18.41
30.80
84.18
95.66
Factorized
5122
74.97
92.22
80.65
94.47
62.29
83.56
8.05
22.81
17.78
29.80
84.10 95.76
Fourier
5122
75.68
92.56
81.33
94.78
63.14
83.90
9.96
26.20
18.11
30.06
84.76
95.94
RPE-leam
5122
72.59
90.74
78.81
93.52
59.35
81.83
6.45
21.00
12.17
21.45
81.64
94.36
2D- ALiBi
5122
78.86
94.18
84.02
95.96
67.28
87.05
14.23
35.83
20.23
32.23
86.42
96.56
2D-RoPE
512"
79.23
94.39
84.61
96.15
67.44
87.22
16.05
38.53
21.09
33.74
86.90
96,90
LH-180
5122
81.11
95.26
86.46
96.87
69.96
88.97
19.49
41.81
22.64
33.98
88.08
97.48
LH-90
5122
81.19
95.16
86.38
96.84
70.35
88.92
20.05
42.09
23.44
35.18
88.62
97.16
LH-45
512
81.62
95.44
86.57
96.91
71.09
88.80
20.55
43.21
23.87
35.76
89.04
97.40
1D-learn
7682
65.95
87.11
71.75
90.33
51.11
75.27
3.79
13.16
12.13
22.47
75.76
92.64
2D-sincos
768
65.48
86.90
71.19
90.19
50.64
75.36
3.84
12.77
11.45
21.22
75.46
92.06
Factorized
7682
63.71
85.36
69.15
88.81
48.58
73.17
3.04
10.95
10.70
20.32
74.64
92.24
Fourier
768
65.97
86.92
71.56
90.01
51.25
75.24
4.05
13.45
11.53
21.38
76.04
92.54
RPE-leam
7682
57.16
79.87
63.00
83.99
41.56
65.96
2.55
8.91
4.83
9.98
66.68
86.32
2D-ALiBi
768
72.97
90.64
78.13
93.26
59.19
81.53
8.48
24.00
12.83
22.54
79.96
93.76
2D-RoPE
7682
71.28
89.93
77.03
92.54
56.70
79.93
7.53
22.20
12.00
21.23
79.14
93.68
LH-180
7682
76.59
92.92
82.17
95.09
63.88
84.63
12.52
29.41
15.56
25.88
83.56
95.68
LH-90
768
77.12
93.38
82.68
95.46
64.49
85.23
12.89
30.44
17.52
28.68
84.90
96.08
LH-45
768
78.13
93.76
83.67
95.68
66.51
86.17
14.21
31.96
18.14
28.64
85,30
96.10
ID-leam
1024
55.67
80.00
61.00
83.85
40.97
65.86
1.95
7.77
8.46
17.28
65.14
86.02
2D-sincos
10242
53.71
78.36
58.91
82.32
39.57
64.82
1.48
6.04
7.08
14.93
64.62
86.16
Factorized
10242
50.46
75.08
55.22
79.17
37.41
62.22
1.33
5.43
7.00
14.51
63.86
85.08
Fourier
1024"
54.58
78.17
59.80
82.14
39.22
64.61
1.87
7.01
7.65
15.49
64.66
86.46
RPa-learn
1024
36.80
60.77
41.10
65.25
24.37
46.37
0.99
3.88
1.85
4.81
48.40
72.34
2D-ALiBi
1024
63.62
84.36
68.80
87.80
49.88
73.57
4.51
14.48
7.78
15.24
71.56
88.74
2D-RoPE
1024
51.41
75.05
56.71
79.27
37.18
60.56
2.12
8.04
4.00
9.06
60.30
82.64
LH-180
1024
69.24
88.81
75.22
91.84
55.84
78.89
6.63
18.92
11.23
20.18
76.14
92.44
LH-90
10242
71.58
90.19
77.48
92.99
58.30
80.83
8.13
21.29
12.61
22.50
79.68
93.34
LH-45
10242
73.15
91.09
79.01
93.63
60.26
82.38
8.35
22.15
14.11
23.78
80.48
93.78
18

---

A.2.1 Other Models
Table 7: Top-1 acc, (%) Val for models outside our controlled experiment, using the timm
library [104].
Name
224 320² 384²
448²
512²
768
1024
Saatt2__arge_patcchie wttppaaaah___2...._p__
wtt.iiikkkk__22222_12 inikk
87.97 87.73 80.76 60.80 40.48 10.63 5.19
85.28 85.61 84.93 84.08 83.08 77.90 70.35
caformer_b36 11166ssallin2221
1n_inkk53 52
87.24 87.29 86.34 84.78 82.96 73.55 62.60
convformer_b3e 10t0__aalnnnii33
84.59 85.06 83.98 82.34 79.48 58.24 37.54
convformer_ett stt.oormer_a...aallll.m..
82.89 83.34 81.72 78.86 73.36 39.49 17.57
eva wva_giant.patchh4__22 ft _
88.75 88.86 88.50 87.83 87.22 83.37 78.31
iRPE (our implementation 39
80.53 81,59 81,47 80.77 79.94 72.86 60.30
svit_basse_attche__etnnooorr
104.ms_ine 2o44mm__nn2k..m in1k
84.40 84,80 84.31 83.77 82.90 78.56 70.58
sxin_se_baas_2224.mm.iinn
83.86 82.61 81.34 80.39 79.30 73.54 63.78
sein_taay._aatche_ssnddoor/
80.85 80.92 79.96 79.09 78.33 72.24
wvins_poopr.bbsse.nnn1151
82.54 83.20 82.27 81.06 79 72.22 61.24
wrins__poptt.mmall..nnkk155
80.94 81.67 80.92 79.76 78.46 70.56 58.74
witne__vt_.arrge.nn11511]
83.38 83,44 82.64 82.03 80.99 76.49 69.01
85.79 85.84 85.03 84.24 83.25 76.35 66.60
vit
81.26 82.33 81.88 80.89 79.66 72.66 63.25
vit att_aarg__pacchl....i..22..e.eer
T08
87.93 88.08 87.57 87.02 86,18 81.51 75.51
81.55 82.67 82.23 81.43 80.08 73.24 64.69
rit_emaall_x26_s$222222
81.38 82.89 82.46 81.75 80.32 72.30 61.38
http:/addw........22.....o.oooe. xcit_small_12.ples
84.86 85.36 84.97 84,41 83.75 79.58 72.94
81.68 82.65 82.10
81.51
80.43 74.48 65.46
Table 8: Top-1 acc. (%) on -HR (ours) for models outside our controlled experiment, using the timm
library [104].
Name
224 320 384 4482
5122
7682 1024
beitv__large_pateeeeee _244__22222222 inik_ft_innn/ in22k, in1k 106
95.16 95.24 90.36 73.72 52.60 14.32 7.40
caformer_bbe htt.mmrrbbb.eesl_nnnffff...
93.06 93.08 92.84 92.10 90.88 85.68 79.30
94.40 94.56 94.02 93.00 91.34 82.16 70.94
92.44 92.26 90.94 88.94 85,40 70.04 59.80
convformer__1...sll_iiil..
90.98 90.84 88.40 83.30 77.24 53.18 41.02
swa_giant._attch4..22
95.96 86 95.70 95.58 95.36 92.94 89.24
iRPE (our implementatiion
89.10 89.76 76 89.56 88.8 88.12 82.40 72.60
svin_basse_pacchh__iindoo7
htt_mm_in2______._mm_m 44]
91.82 92.20 91.18 90.24 90.06 85.22 79.26
91.78 90.22 88.52 86.82 85,72 78.00 68.46
skin_tbay..attchh..iindoo...
89.06 80 87.56 86.76 85.46 77.58 68.54
wvins.popprt_basee.nn1...11
90.54 90.92 90.30 89.24 87.60 79.50 69.74
twins
wiine_poppr_cmall.11n1555
89.62 89 89 40 88, 87, 14 85.62 77 67.30
Wwins_sst_larrge.in11[]]]
91.46 91.24 90.34 89.36 87.82 82,42 76.22
vit
in1k [08
93.22 93.38 93.24 92.68 92.02 18.14 81.92
vit.base.psttchl__pppp.....
waap_25..bbbinnkk.... [00
89.36 90.84 90.14 89.22 88.22 83.36 76.54
vtt.aarg..paccia....ppp......ppp
vit
in
94.62 94.78 94.68 94.62 94. 91.10 86.64
104
89.42 90.56 90.38 89.74 89.26 83.90 77.48
rit_small__r2___3__224
89.76 90.52 90.30 89.56 88.80 82.06 72.12
ccit_meddimm_24_882222
itt.:__2222..___iiiiii_iii inik
92.54 92.94 92.34 91.94 91.20 87.70 81.80
xcit_small_12_p16_222 fb_.in2
89.54 90.14 89.88 89.02 88.22 82.82 74.86
19

---

A.3 LookHere Bias Matrices
Figure 9: LH-180 bias matrices for query patch (11,8), grid size of 1xx14.
20

---

Figure 10: LH-90 bias matrices for query patch (11,8), grid size of 14x14.
Figure 11: LH-45 bias matrices for query patch (11,8), grid size of 14x14.
21

---

A.4
Experimental Details
A.4.1
Training ViTs
Recipe. Our training recipe that consistent across configurations:
AdamW [009
using the default PyTorch implementation that does not fully decouple learning rate and weight decay
Binary cross-entropy loss - summing along the class dimension, averaging along the batch dimension
Linear warm-up for 10% of steps and cool-down using cosine decay schedule learning rate
Batch size 2048
Mixup [10] a 0.8, cutmix ||о
a
CLS token with an MLP classifying head - final linear layer weights are initialized to and biases to - 6.9 (so all class probabilities start at Toon
layer drop rate of 0.1 and MLP dropout of 0
Train for 150 epochs the first 99% of ImageNet-Ik using Huggingface's datasets library, i.e., laaa_laddaaaaaa datasedatsettttc ("imagenet-1k", split="train[:9e"
Choose checkpoint according to best minival top-1 accuracy (run after each epoch), where minival is the last 1% of the ImageNet-Ik training set, i.e., load_datased caaaassetiimmageeeeeee ("imagenet-1k" split="train [99%:]
A.4.2 Compute
Training takes around 3 days RTX 4090 GPU. Thus, all 80 training runs take around 240 GPU-days. We spend another 54 GPU-days on 18 ablations. Ablations and our iRPE run always use our best training recipe, which is 3-Augment (17] data augmentation, 3 10 learning rate, and 0.05 weight decay. iRPE (39] takes around days on an RTX 4090 GPU, even with the official custom CUDA kernel. As result, exclude from our apples-to-apples comparisons.
A.4.3 High-resolution finetuning
Following DEIT III's finetuning recipe [17], we increase the drop rate to 0.2 and the weight decay to 0.1, and fix the learning rate to 10-5 with 512 batch size.
A.4.4 Extrapolation Tuning
For 2D-ALiBi, 2D-RoPE, and LookHere models, we tune a single parameter at the target resolution on minival (Table 9). LookHere models benefit less from tuning than 2D-ALiBi and 2D-RoPE models. For example at 512" resolution, the difference top-1 accuracy on minival when using the tuned parameter versus the default value is 2. 2.1% for 2D-ALiBi, 1.3% for 2D-RoPE, and 0.15% for LH-45. Thus, LookHere does require tuning its global slope value effectively extrapolate.
Table 9: Tuned Parameter Values
Name
Tuning Parameter
224²
320
3842
4482
512
7682
1024
2D-ALiBi
8g
1.0
1.4
1.4
1.4
1.5
1.6
2D-RoPE
base frequency
100
160
190
250
700
1250
1250
LookHere
8g
1.0
1.00
0.95
0.95
0.95
0.75
0.6
A.4.5 Segmentation
For both linear probing and full finetuning we use a linear decoder. The linear decoder consists of linear layer applied to the frozen patch representations which then upsampled the original image size. Similar to (85) we add BatchNorm layer before the linear layer.
22

---

For full finetuning, we followed the Segmenter training recipe 84 exactly. For ADE20k, the base learning rate 10-3 3 for 160k iterations with a batch size of 8, at 5122 px. For Cityscapes, the base learning rate is 10-2 for 80k iterations with a batch size of 8, at 384 px. We train with SGD.
For linear probing, we freeze the backbone and pre-compute the patch representa- tions. We use the AdamW optimizer [109] and sweep the following learning rates: {0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.3, 0.5}. For both ADE20k and Cityscapes we set the batch size to 16 and train the linear decoder for 50 epochs.
23

---

A.5 Full Experimental Results
Table 10: First half of our hyper-parameter sweep. ViT-B models trained on ImageNet for 150 epochs; trained and tested at 224 RA is for RandAugment and 3A for 3-Augment.
WD
LR Data
Val
[]]
ReaL
v2 2
-A
-HR (ours)
Method
10
10
3
Aug
top-1
top-5
top-1 top-5 top-1 top-5
top-1 top-5
top-1
top-5
top-1
top-5
1D-leam
2
3.0 3A
78.51
93.42 83,91 95.45 66.86 86.11
9.64
27.71 28.21
41.85
86.54 95.88
2D-sincos
2
3.0 3A
77.96
93.29 83.63 95.34 66.35 85.84
9.55
27.57
28.29
42.01
86.24 96.16
Factorized
2
3.0 3A
78.42
93.47 84.09 95.64 66.89 85.84
8.36
25.65
28.79
42.68
86.70 96.48
Fourier
2
3.0 3A
78.78
93.37 84.21
95.50 67.32 85.89
9.56
27.08
28.90
42.29
87.24 96.28
RPE-learn
2
3.0 3A
78.92
93.76 84.46 95.74 67.75 86.28
10.00
28.01
28.70
42.45
87.06 96.24
2D-ALiBí
2
3A
78.47
93.68 84.19 95.80 66.66 86.18
9.01 26.44
26.51
39.74
86.46 96.50
2D-RoPE
2
3.0 3A
79.36
93.96 84.70 95.96 67.98 86.93
10.37
28.36
30.58
44.46
87.42 96.64
LH-180
2
3.0
3A
80.76
94.78 86.23 96.56 69.43 88.02
11.47
29.87
31.09
44.20
88.86 97.08
LH-90
2
3.0
3A
80.75 94.71 86.17
96.45 69.85 87.97
12.27
30.24
31.19
44.20
88.90 97.06
LH-45
2
3.0
3A
80.49 94.55 86.06
96.42 69.27 87.55
11.44
30.56
31.70
45.38
88.90
97.02
ID-learn
5
3.0 3A
79.45
94.30 84.97
96.10
68.49 87.59
10.97
30.59
29.64
43.48
88.28 96.76
2D-sincos
5
3A
79.05
94.25 84.62
96.14 67.86 87.01
10.45
29.41
29.11
43.24
87.58 96.48
Factorized
5
3.0 3A
79.86
94.73 85,30
96.41 69.11 87.87
11.00
31.32
29.99
44.22
87.86 97.02
Fourier
5
3.0 3A
79.69
94.41 85.13
96.36 68.30 87.66
11.36
30.93
29.73
43.90
88.14 96.96
RPE-Icamn
5
3.0 3A
79.86
94.64
85.46
96.64
68.57 87.72
9.85
29.27 29.10
43.28
88.22 97.32
2D-ALiBi
3.0 3A
79.54
94.57
85.15
96.38
68.47 87.58
10.45 29.33 28.26
41.91
87.70 96.74
2D-RoPE
5
3A
80.38
94.86
85.64
96.49
69.34
87.89
13.03 33.95
32.45
46.96
88.78 96.92
LHI-18
5
3.0 3A
81.31
95.11 86.53
96.71
70.70 88.38
13.53
32.72
32.10
45.07
89.86 97.54
LH-90
5
3.0 3A
81.02
94.92
86.44
96.68
70.28
88.34
13.15
32.89
31.77
44.74
89.90 97.20
LH-45
5
3.0 3A
81.06
94.87
86.23
96.46
69.65
88.60
13.41
32.96
32.12
45.25
89.46
97.06
1D-leam
2
3.0 RA
76.51 92.08
81.84
94.32
63.89
83.41
6.12
19.93
23.56
36.15
84.26 94.96
2D-sincos
2
3.0 RA
76.38
92.22
81.77 94.53 63.87 84.05
6.57
20.23
23.62
36.95
84.40 95.28
Factorized
2
76.45
92.18 82.16 94.53 64.31 84.10
6.57
20.97
24.30
37.35
84.34 94.90
Fourier
2
3.0 RA
76.59
92.08 82.07
94.49 64.51 84.13
7.28
21.72
24.20
37.39
83.76 94.68
RPE-learn
2
3.0 RA
76.37
92.28 81.90
94.54 63.99 83.41
6.12
18.76
23.05
36.01
83.58 94.96
2D-ALiBi
2
3.0 RA
76.08
92.16 81.52
94.45 63.67 83.22
5.61
19.08 22.17
34.74
83.20 94.78
2D-RoPE
77.31
93.10 82.84
95.22 65.06 84.75
6.07
20.63 27.05
41.09
85.24 95.76
LH-180
2
3.0 RA
80.02
94.07 85.15 95.79 68.32 86.73
9.21
25.53 27.69
40.24
87.18 96.70
LH-90
2
3.0 RA
79.36
93.83 84.67
95.64 67.64
86.43
10.00
24.99 27.86
41.01
87.20 96.16
LH-45
2
3.0 RA
79.77
93.99
84.93
95.68
68.30
86.41
9.36
25.91
28.35
41.63
86.40
96.40
1D-learn
5
3.0 RA
78.06
93.38 83.35
95.44
65.33
85.67
8.07
25.07
25.98
39.42
84.94 95.64
2D-sincos
5
3.0 RA
77.95
93.27 83.26 95.37 65.51 85.64
7.57
25.37 26.11
39.63
85.16 95.88
Factorized
5
3.0
78.55
93.96 84.00 95.91 66.88 86.19
8.05
24.05
27.08
40.76
86.36 96.28
Fourier
5
3.0 RA
78.16
93.47 83.43 95.41 66.28 85.90
8.28
25.16 26.25
39.94
85.98 95.98
RPE-Iearn
5
3.0 RA
78.15
93.57 83.50 95.51 66.50 85,91
7.56
23.64
25.10
38.51
85.62 95.64
2D-ALiBi
5
3.0 RA
77.00
92.89 82.49 94.88 64.75 85.05
6.88
21.55
23.65
36.51
84.22 95.46
2D-RoPF
5
3.0 RA
79.29
94.05 84.46 95.85 67.73 86.67
10.77
29.89
28.99
43.18
86.50 96.30
LH-180
5
3.0
80.20
94.32 85.22 96.02 68.27 86.70
10.60
27.40
27.80
40.05
87.74 96.60
LH-90
5
3.0 RA
80.35
94.33 85.47 95.96 68.98 87.29
10.67
27.89
28.54
41.64
87.76 96.56
LH-45
5
3.0 RA
80.13
94.31 85.10
95.99 68.
86.65
10.89
27.57
28.35
41.35
87.08
96.20
24

---

Table 11: Second half of our Syyper-paraeete sweep. VYT-B models trained on ImageNet for 150 epochs; trained and tested at 224". RA is for RandAugment and 3A for 3-Augment.
LR Data
1
ReaL
v2 2
-A
3
-HR (ours)
Method
10
2
10
Aug
top-1
top-5
top-1
top-5
top-1
top-5
top-1
top-5
top-1
top-5
top-1
top-5
1D-leam
2
3A
77.31
92.41
82,95 94.79 64.85 84.22
6.95
22.00
26.14
38.81
85.68 95.84
2D-sincos
2
1.5 3A
77.50
92.64 83.14 94.93 65.57 84.70
7.23
22.91
26.66
39.92
85.78 95.60
Factorized
2
1.5
3A
76.88
92.43
82.82 94.88
64.96 84,43
5.99
19.96
26.47
39.91
85.36 95.38
Fourier
2
1.5
3A
77.15
92.31
82.82
94.63
64.92 84.24
7.09
22.67
26.28
39.49
85.38 95.48
RPE-Icarn
2
1.5 3A
77.07
92.61
82.97
95.00 65.13 84.52
6.52
21.40 24.75
37.91
85.16 95.62
2D-ALiBi
2
3A
77.72
93.05
83.40
95.38
66.23 85.56
7.59
22.76
25.78
38.89
85.88
96.14
2D-RoPE
2
1.5 3A
78.14
93.19
83.74
95.40
66.67 85.57
8.20
25.76
28.78
42.64
86.26 96.14
LH-180
2
1.5
3A
80.14
94.19 85.51
96.12
68.87 87.25
11.03
27.84
29.73
42.81
88.14 96.82
LH-90
2
1.5
3A
79.88
94.18
85.51
96.12
69.34 87.07
10.83
28.32
30.88
44.23
88.32 96.92
LH-45
2
3A
79.57
94.06
85.22
96.01
68.40
87.02
9.43
27.60
30.69
44.85
87.86
96.88
1D-lcam
5
1.5 3A
77.87
93.31
83.56
95.47
66.56 85.69
8.64
25.03
27.16
40.67
86.40 95.86
2D-sincos
5
1.5 3A
78.48
93.50
83,99
95.65 66.65 86.19
8.85
25.75
27.72
41,90
86.88 96.46
Factorized
5
1.5 3A
77.34
92.98
83.12
95.32 65.62 85.33
7.24
23.51
26.86
40.51
86.42 96.02
Fourier
5
1.5 3A
77.89
93.18
83.59
95.30 66.02 84.99
8.49
25.13
26.44
39.74
86.04 96.04
RPE-learn
5
1.5 3A
77.71
93.31
83.50
95.44 66.12 85.43
7.91
24.67 25.23
38.30
86.60 96.02
2D-ALiBi
5
3A
78.56
93.77
84.32
95.88 66.34 86.57
8.60
27.16 26.97
40.76
86.62 96.60
2D-RoPE
5
1.5 3A
78.74
93.79
84.45
95.86 67.12 86.61
9.69
27.61 29.82
43.58
87.62 96.60
LH-180
5
1.5
3A
80.53
94.65
85.82
96.47 69.38 87.70
11.63
29.75 30.07
43.02
88.66 97.06
LH-90
5
1.5 3A
80.34
94.68
85.81
96.48 68.92 87.89
11.55
30.31
30.85
44.73
88.56 97.00
LH-45
5
1.5 3A
80.32 94.60
85.59
96.34
68.78
87.33
10.71
29.65
31.25
45.14
88.82 97.06
ID-learn
2
1.5 RA
75.02
90.83 80.68
93.26
62.38 81.45
4.55
16.04
21.92
33.64
82.50 94.44
2D-sincos
2
75.72
91.12 81.32 93.59 62.60 82.06
5.61
18.31
22.82
35.10
82.82 94.00
Factorized
2
1.5 RA
74.47
90.62
80.40 93.21 61.73 81.29
4.63
15.31
22.10
33,95
82.16 93.40
Fourier
2
1.5 RA
74.95
90.58 80.59 93.19 61.66 81.23
4.97
17.11
22.08
34.01
82.84 94.14
RPE-Icam
2
1,5 RA
74.65
90.50
80.28
93.11 61.42 81.04
4.44
15.64
20.22
31.96
82.14 94.12
2D-ALiBi
2
1.5
74.95
90.75
80.69
93.40
61.92 81.07
5.01
16.52
20.45
32.03
83.18 93.82
2D-RoPE
2
76.59
91.56
81.98
93.90 63.96 83.06
6.13
19.84
24.69
37.57
83.94 95.04
LH-180
2
1.5 RA
78.42
93.04
83.68
95.03 66.46 85.12
8.20
22.65
26.07
38.42
86.18 95.58
LH-90
2
1.5 RA
78.62
93.21
84.09
95.16 66.64 85.15
8.77
24.07
27.22
39.94
86.50 95.82
LH-45
2
1.5 RA
78.17
93.12
83.61
95.22
66.54
85.01
7.49
23.24
26.56
39.61
85.98 95.78
1D-leam
5
1.5 RA
76.07
91.79
81.66
94.11
63.03
83.12
5.73
19.24
23.20
35.88
83.12 94.62
2D-sineos
5
1.5 RA
76.50
92.22 81.96 94.49
64.10 83.72
6.21
19.16
24.23
37.13
84.02 94.68
Factorized
5
76.38
92.25 82.05 94.69
63.25 83.35
5.41
17.81
23.88
36.33
83.72 95.16
Fourier
5
1.5 RA
75.78
91.66 81.31 94.04
63.62 83.10
5.29
18.76
22.95
35.26
83.70 94.80
RPE-learn
5
1.5 RA
75.33
91.29 80.98
93.77 62.04 82.19
5.07
18.47
20.80
32.74
83.00 94.32
2D-ALiBi
5
1.5 RA
76.02
91.94 81.61
94.18 63.16 82.93
5.01
17.97
21.53
33.85
83.80 94.78
2D-RoPE
1.5
77.13
92.53 82.47 94.80 64.63 84.68
6.47
20.95
26.08
39.35
85.00 95.14
LH-180
5
1.5 RA
78.68
93.68 84.26 95.69 66.76 85.67
7.93
23.17 26.98
40.13
85.74 96.12
LH-90
5
1.5 RA
78.77
93.63
84.09
95.37
66.69 85.83
9.15
25.33
27.45
40.08
85,70 95.98
LH-45
5
1.5 RA
78.39
93.45
83.85
95.37
66.30 85.61
8.97
24.84
27.11
40.44
84.90 95.66
25

---

A.6 Ablations
We train 18 models to ablate the LookHere design. Each run uses our best 150 epoch training recipe. We test models without extrapolation at 224 px (Table 12 and with extrapolation at 1024 px (Table 13.. Before running extrapolation tests, we tune the global slope of each model at 1024 px to fairly compare with our three default variants. To fit in the tables, we use short forms explained here: "undir-> 90" means replacing the four undirected heads with four 90º FOV heads, "undir-> no dist" means removing the distance penalties on the four undirected heads, "invert" means inverting the layer-wise slope pattern such that sj linearly increases from 0.5 to 1.5 with depth, "mask:oo + 0" means replacing oo with 0 in equation T and "dist->no dist" means removing the distance penalties on all heads.
Table 12: LookHere design ablations without extrapolation. ViT-B models trained on ImageNet for 150 epochs; trained and tested at 224
Val M
ReaL
3
-R [
-HR (ours)
Variant
Change
top-1 top-5
top-1
top-5
top-1
top-5
top-1
top-5
top-1
top-5
top-1
top-5
LH-45
undir 90"
80.53 94.92 86.27 96.81
69,42
88.29
10.33
29.60
32.49
46.78
89.42
97.38
LHH445
undir- 180
80.72 94.90 86.19 96.74
69.66
88.51
10.81
29.59
32.14
46.13
89.44
97.26
LH-45
undir -no dist
81.14 95.08 86.44 96.74
70.53 88.48
14.17
34.07
32.61
46.14
89.84 97.54
LH-90
undir 90"
81.00 94.98 86.59 96.78
70.15 88.46
10.84
29.40
32.44
46.46
89.10 97.62
LH-90
undir 180"
80.94 95.06 86.54 96.78 69.99
88.55
12.29
30.83
31.73
45.64
89,46 97.06
LH-90
undir dist
81.01 95.13 86.38 96.79
70.37
88.68
12.41
32.39
32.27
46.51
89.34 97.52
LH-180
undir- 90"
80.82 95.02 86.56 96.
69.50
88.57
11.85
29.99
31.85
45.77
88.98 97.18
LH-180
undir 180
80.88 95.11 86.56 96.87 70.36
88.33
11.96
30.55
31.63
45.52
89.28 97.32
LH-180
undir dist
81.39 95.11 86.78 96.77 70.66
88.43
12.49
32.00
31.79
44.93
89.84 97.50
LH-90
8y + 0.125
81.20
95.03 86.48 96.69
70.14
88.33
13.63
33.27
32.14
45.44
89.22 97.08
LH-90
8y
0.25
81.08 94.92 86.28 96.53
70.02
88.05
12.64 31.43
31.46
44.73
88.88 97.06
LH-90
89 :::1- - 0.5
81.09 94.97 86.47 96.58 70.18
88.40
13.04 33.00
32.02
45.67
89.56 97.30
LH-90
8g 114
80.91 95.10 86.58 96.92 70.16
88.69
11.40 30.31
32.13
46.33
89.40 97.46
LH-90
8t : invert
81.37 95.02 86.43 96.72
70.30
88.32
13.87 33.88
32.69
46.55
89.72 97,44
LH-90
dist + dist?
80.98 95.17 86.50 96.88
70.34
88.50
11.45 30.44
32.13
46.43
89.66 97.48
LH-90
dist + dist
80.86 94.89 86.17 96.55
69.15 88.15 12.33
31.75
31.65
45.23
88.56 97.32
LH-90
mask: da
79.68 94.47 85.11 96.47
68.58
87.82
11.21 30.33 29.69 43.59
87.94 97.02
LH-90
dist dist
80.19
94.77 85.49
96.48
69.22
88.27
11.52
30.92
31.76
46.33
88.52 97.10
LH-90
$t : learn
81.35
95.06 86.55 96.64 70.40 88.55 13.08
33.03
31.93 45.88
89.56 97.20
Table 13: LookHere design ablations with extrapolation. ViT-B models trained on ImageNet for 150 epochs; trained at 224 and tested at 1024
Val 1
Real.
v2
-R 5
-HR (ours)
Variant
Change
top-1
top-5
top-1 top-5
top-1
top-5
top-1
top-5
top-1
top-5
top-1 top-5
LH-45
undir + 90"
71.94
91.01 78.26 93.58 59.23 81.53
5.89
18.47
16.02
27.72
81.40 95.26
LH-45
undir 180
69.97
89.42 76.05 92.44 56.19
79.03
5.33
16.71
13.70
24.44
78.72
94.12
LH-45
undir dist
69.72
89.35 75.69 92.33 55.97 78.76
4.77
15.76
12.92
23.42
79.14
93.46
LH-90
undir-+ 90"
69.39
88.99 75.67 92.00 55.82
78.68
5.00
16.40
11.93
21.96
78.10 93.46
LH-90
undir 180
68.80 88.62 74.95 91.72 53.89 77.43
4.16
14.49
12.72
22.84
76.60 92.48
LH-90
undir-too dist
69.24
89.40 75.29 92.46 56.03 79.38
4.84
15.65
12.67
23.08
78.24 94.08
LH-180
undir 90"
64.44
85.72 70.73 89.36 49.80 73.70
3.40
12.16
8.98
17.32
73.60 90.14
LH-180
undir 180
54.13
77.21 59.97 81.66 39.38 63.14
1.61
5.51
4.23
8.80
66.44 84.56
LH-180
undir -oo dist
66.35
86.89 72.67 90.27 51.97 75.62
4.92
14.69
9.33
17.43
74.30 91.20
LH-90
Sy 1-11-00 0.125
67.36
87.84 73.31 91.06 53.06 76.37
3.15
10.69
12.15
22.20
76.58 91.90
LH-90
Sy 1::1- +11+0000 0.25
70.46
89.68 76.43 92.52 57.11 80.17
5,43
16.19
12.45
22.25
79.32 93.14
LH-90
8y 1: 0.5
72.53
90.72 78.40 93.30 59.34 81.56
7.16
20.80
13.88
24.05
79.82 93.36
LH-90
11:1-8
55.16
78.88 61.14 83.31 41.30 66.02
2.85
9.59
5.19
10.95
68.02 86.92
LH-90
81 : invert
70.03
89.28 76.09
92.30 55.69 79.06
6.21
19.29
11.85
21.36
76.80 92.26
LH-90
dist + dist?
65.08
87.03 71.16
90.46 51.05 75.52
3.28
12.35
10.96
20.95
74.80 92.04
LH-90
dist + Vdist
66.83 87.59 72.53
90.70 52.88 76.41
3.91
12.57
10.93
19.83 76.80 92.60
LH-90
mask: 80 + 0
40.52 66.37 45.24
70.96 28.48 52.11
1.23
5.84
2.58
6.58 50.18 75.08
LH-90
dist dist
44.42
69.57
49.20
74.05
31.31
55.09
1.03
5.11
6.02
13.10
53.92 78.48
LH-90
81 : fex-m
66.13
86.72
71.96
89.82
52.54
75.71
4.44
13.80
10.59
19.27
75.82 91.66
26

---

A.7
Logit Lens
container shipking penguin
siamang
shovel
diaper
Crock Pot mountain bike
airship
Blenheim spaniel
segienttttion Image
180
UH-1
UH-90
UH-45
ZD-ALBBI
2D-RoPE
Ractorized
2D 30-sincos
ID-learn
Fourier
APE--ee leam
Figure 12: More examples from ImageNet-S and each model's logit lens predictions.
27

---

CLS Token
Patch Tokens
RFE-ccam
1.0
Ul-180
0.9
UH-45
2D-RoPE
0.8
Class idontiiblity
Factorized
2D-sincos
1D-leam
0.7
0.5
0.6
Layer
Layer
We plot the average class identifiability 89 across the model layers on 1000 images from Val for the class and patch tokens. This is a measure of how recoverable the correct class is from the class projection the token. The score ranges from to with denoting that the correct class has the highest logits and O the lowest.
Figure 13:
28

---

bass derkhasity
LH-180
layar
cass Rencrabiiny
LH-90
leyer
cos dentrabiio
L11-45
luyar
cass denemesiity
2D-ALBBI
leyar
clsso ventraallits
2D-RRPPE
leyer
cless demumestiy
Factorized
cass censitiblity
2D-sincos
leyer
ciess tcemiaaiiiyy
1D learn
layer
cles tderuneenityy
fourier
leyer
das delthasilty
RFE-leam
10
leyer
cles dentnebliig my
vit tiny pstch16 224.augreg in21k ft in1k
dles iderumentty
patch16 augregg in1k
coss Merthabity
vit base patch16 224 augreg In1k
leyer
Figure 14: Leveraging the semantic segmentation labels from the ImageNet-S, we compared the
identifiability rate of class patches (blue) vs non-class tokens [89 across the model layers on 1000
images from Val. LookHere can discriminate between class and non-class patches. Other positional encodings cannot unless they are trained for much longer.

---

A.8 Head Diversity, Attention Distance, Patch Similarity and Head Visualizations
In our paper, we show that LookHere prevents attention collapse measured by JSD (Figure Here, we measure attention diversity using L1 and L2 distance. We also measure attention distances and patch-wise representational similarity both at 224 px (Figure [5) and at all resolutions tested
(Figure 16 & 17
Head Diversity by Depth (12, 224)
Head Diversity by Depth (Metric: 11, Res: 224)
0.00040
0.009
0.00035
0.008
RPE-learn
0.00030
1D-learn
LH-45
0.00025
LH-90
Diversity
Diversity
0.006
LH-180
0.00020
20-ALBB
0.005
2D-sincos
0.00015
0.004
Fourier
2D-RoPE
0.00010
0.003
Factorized
0.00005
0.002
0.00000
2
Depth
Depth
Mean Attention Distance by Depth (Res: 224)
Patch Similarity by Depth (Res: 224)
1.0
120
Mean Attention Distance (px)
0.3
RPE-leam
100
1D-leam
Cosine Similarity
LH-45
0.5
LH-90
80
LH-180
2D-ALIBI
0.4
2D-sincus
60
Fourier
2D-RoPE
0.2
Factorized
40
2
4
6
8
10
12
2
4
6
8
10
12
Depth
Depth
Figure 15: Measurements of head diversity, attention distance, and patch similarity by layer across position encoding methods.
30

---

224
320 px
384 px
448
512"
768
1024
1eur ontrorton decopee leyes uose in
Meno enverts cevere Dowth prade 20 shvie
Mww www.rtiinns Dorrgr by Dopph Mode (urrite))
0.45
3.00
340
Hear Altertion by Depth (Mddll
Hean stance
Ststanee Depth srdell
3.114
0.40
3.40
2410
2
0.15
6.40
0100
Jenz Diuecty by Depeh (odel (ertsted)
1MAC Divertyy Deprt (MOSAl Featerr
Head Diwedby Degth Mocsl spcismi
0.43
Hevid Diuee Ty Deper
Worrdhy Doathh (Mtce) kalt))
Eiberrtbtyyyy Deathh Moode 1900
0.11
6.3
Figure 16: measure the attention distance per depth, for each model and resolution, by taking the sum of patch distances weighted by and the head diversity the generalized JSD attention matrix for
attention averaged heads [(row; col: each head, averaged over rows [[4, 2) - (7, 2)]. We report the avernge over 500 randomly selected images from minival.
31

---

224
320
384²
448
512
768
1024
Head Docnity Depth MMocee L9-331
Hoad Oiunddy Depor (Peate (8A5)
Peac Sindarrtyy Desth ghoee: 10.0em))
7
Serilanty Dopth Mezdd
Simdunts Teclerentt
WNithSemitaty Death dreech
0.13
8
200
6:5
639
Marth Apt.oom
20L8
cm
Semderty (Podd J63221
dinte Desth Meddd (X-00)
Pathh Simel Depin Podec L-AK))
0.0
6.0
100
10
033
Figure 17: The patch similarity, for each model and resolution, is measured the average of pairwise cosine-similaritios between patch representutions each layer ((7, - (10, 3)]: we report the average over the same 500 minival images.
32

---

224
320
3842
4482
5122
7682
1024
Figure 18: 1D-learn attention maps of ten attention heads across seven resolutions, where the query the center. We use the colormap:
Averaged over 5k images.
33

---

224
320
384
4482
5122
768
1024
Figure 19: 2D-sincos attention maps of ten attention heads across seven resolutions, where the query Averaged over 5k images.
is in the center. We use the colormap:
34

---

224
320
384
448
512
768
1024
Figure 20: Factorized attention maps of ten attention heads across seven resolutions, where the query AAerageed over 5k images.
is in the center. We use the colormap:
35

---

224
320
384
448
512
768
1024
Figure 21: Fourier attention maps of ten attention heads across seven resolutions, where the query is Averaged over 5k images.
in the center. We use the colormap:
36

---

224
320
384
448
5122
768
1024
Figure 22: RPE-learn attention maps of ten attention heads across seven resolutions, where the query Averaged over 5k images.
is in the center. We use the colormap:
37

---

224
320
384
4482
5122
768
1024
Figure 23: 2D-ALiBi attention maps of ten attention heads across seven resolutions, where the query Averaged over 5k images.
is in the center. We use the colormap:
38

---

224
320
384
4482
5122
768
1024
Figure 24: 2D-RoPE attention maps of ten attention heads across seven resolutions, where the query Averaged over 5k images.
is in the center. We use the colormap:
39

---

224
320
384
4482
5122
768
1024
Figure 25: LH-180 attention maps of ten attention heads across seven resolutions, where the query Averaged over 5k images.
is in the center. We use the colormap:
40

---

224
320
384
4482
512
768
1024
Figure 26: LH-90 attention maps of ten attention heads across seven resolutions, where the query is Averaged over 5k images.
in the center. We use the colormap:
41

---

224
320
384
4482
5122
768
1024
Figure 27: LH-45 attention maps of ten attention heads across seven resolutions, where the query is Averaged over 5k images.
in the center. We use the colormap:
42

---

A.9 Accuracy Gaps and Class-level Effects
LH-180 (ours)
LH-90 (ours)
LH-45 (ours)
Factorized
2D- RoPE
Disinoos
ID-leam
RPE-leam [0
Fourier
AD Bi
P
Deotctio Motel
restie (Prode spmmm)
Aeruncy olon (Medel Facheroed)
22412004
220-000
2200113
150
130
02
Aeruney yapt pernectittio (Nocel 3D Seattss
Apcunery gapt per reetl Biosee (811
Accurery ga04 per reetl (Mryded 114 60)
224
224-706
224
250
130
43
Accurayy DAOS per resoadice Medde (H-45)
230
724
302
Figure calculate difference in accuracy ImageNNt examples extrapolating from 224 px models trained 224
43

---

LH-180 (ours)
LH-90 (ours)
LH-45 (ours)
Factorized
DD-RoPE
DDsincoss
D-lccam
RPE-Icam 10
Fourier
2D-ALIBI
1
class Gardien
Church
Nodic:
Poot 100n 1860
Prcti000 1840
Proquancy clana
Praquenay Millary
Proquatrcy
12
2
8
222
7/5
2003
000
Requeenc ftr cass hobebock Computer
Frequency for closs Radio
Frequeery Sor rlsee Stup Fivel
222
400
720
2003
500
class
class
3.00
10.0
08
3.00
680
7911 180n 180
Trequency
Tequarry Chas
Trequerry
306
133
Teqquntry
1000
340
Eabol.zion 194)
300
100
400
100
Figure 29: Wo colculate the prediction froquency for each class for the first 5k ImageNot [1 and plot those frequencics for the classos with the largest decrease (11, 1) + (4, 2)1. increase [14, 3) - (7, 2]]. and spread (77, 3) + (10, 3) 1024" px.

---

(ours)
(ours)
LHH-45 (ours)
Factorized
2D sincos
1D-leamm
RPE learn 10
Fourier
2D ALiBi
Prequency for clis WWiieee
Prequency for dess Cansermm
hrqqurcy for class Allensinccter
250
7
1.3
420
400
Frecuency Squarnel
Trequency Opener
Trequsticy
300
8
100
Puaoksoo
Frequetcy caes graw
Flequency
Frecuency Squirel
100
200
300
Frequency Pirsshed
Rreeueny
Whistte
1200
12:0
120
10
Phe 2am
250
200
1000
44
130
100
3003
Figure 30: A continuation of increase, and spreaul plots.
45

---

LH-180 (ours)
LH-90 (ours)
LH-45 (ours)
Factorized 6
DD-RoPE
DDssnnos
DD-ccam
RPE-Ieam
10
Fourier
1
2D-LLBB
1
ctre Eastemn Bomnorthrrk fottiennke
Carden
Jenurnyy Cryfiih
0.-
07-
02
0.3
012
82
300
1203
210
60
120
13:0
400
1953
fr Glant Schrmuzer
diess
recuney forhop dennaddnn
0.-
0.-
02
82
120
700
1251
30
00
100
1253
cbes Spider
Accurey for coes Eng fodhound
0.7
3
0.6%
0.4
3000
Accurscy for class Asspult Rifle
Accuracy for class Casserte Mayer
Autrage 2008 poy for 200 incressing dasses
2.00
2.50
0.75
1300
0.00
2.2
2.00
0.5
02
630
250
800
150
100
3003
255
300
800
800
100
3003
1.0
0.4
64
30
700
2003
300
300
500
02
30
800
300
000
07-
0.1
02
02
700
2003
200
300
640
300
302
830
700
020
46
Figure 31: We culculate the prediction accuracy for each class among the first 5k ImageNet 1 examples, and plat these accuricies for classes with the top five classes with the largest decrease [(1, 1) - (2, 3)), increase [(3, 1) + (4. 3)], and spread [(6. 1) - (6. 3)] at 1024 px. along with the averago across live.

---

Durepeon Canden speder Son Spate (PeiS: L0VE
Whi Spueer fam plocet TDANON
Phooriann segescee and Recteee Dinnnorr (PaR (D-bww
Redicca Coodtone
Sangrey and odde Bay (loktt ID-oar!
cn Gurdin 5p and barn Spidrr 2eott: 3D-kmm
Pervocuo and Thisstnng Pedeen Ofrect IDkmm
180
35
30
500
90
X0
200
756
900
Schecnor and lovi WWedk beart
Soup Bost and Consen (oodel Lo eami
Stiggay Blecticc Bay Ploce: 20alnoal
show
02-
76
50
180
25
34
300
304
20
30
4.6
77
300
20
180
Los
0.-
30.0
1540
Thrnhhng Hichin Meeet (Dancas)
Souz Envi and Deneemie (Wadk DDoiooo
Elstin iPadd Taderoodl
Averice f.8liig and Biled frng Phote: N00
1abee Cla den Spader and Hore Spater (Madet Hactoited)
Surppain Speder and Stan Spader (Padtt 2011
300
Figure 32: We lind and plot cross-plots of class pairs that confuse models during extrapulation, indicated by a trunsfer in prediction probabiities. For the subset ImageNet examples, within the 5k, with true class (blue) we select pairs (X, Y) where P(X (224²) P(X]10243) + P(Y|10244) P(Y|2243) is maximized.
47

---

Sorder Ban Spode Nedoreed
Thinkh Meciow Slede:
Lauguer Carton Spicer bern Spider Founer
1WC0 53-
605
208
810
800
JIn-0 Foods
Thesthng Hadiin Mocel (arrer)
Secal Consenere (Model Fautted
Trredling Mock (MSSA
tooking
68
Chrosres Sodiing (wiwtedd Hodil eant
12naro and Ptwttedddd (oodet (7S--0a7)
Pace Cal and Sparts Carl (Rddt I'S 1am
Schooner and Mecel Seart
Stherry Dectro Plde: ALL0
Ourspson Carden Spider and Spkerr Madel 204.81
dingee
Rrisee gili 1ol0
605
7M
510
180
400
550
305
20
30
4.5
200
305
130
400
910
Figure Cross-plots continued.
48

---

Grand Up gtt how Ofiee:: 204US!
Nirbu Ponig 2044000
Schezna Yeel (Mbett 20-4800
605
180
305
816
1000
1006
655
Echocner and lovl (adde 20
25agriy and Elettri Say (Roddel L11-8001
Esoppes Spicer Spider
i9: Spiter and Sern Sp de IWadet (H-123)
Lesperd and jquar 2Medet LI5
Clith and Noneltry (Mool: 1-01
Grord Plar and Liritt U11300
ibovetter and Trrshhing Modias Model Un0
Schooner and Mede: J1-30C
para
1000
6/5
76
50
180
300
25
305
250
30
756
200
30
180
13th
400
180
2.5
Figure Cross-poots continued.
49

---

Purpppoon do qade www.eeiiie ove Splleriitee 101.40)
4nd www.ereaeeeee and forr Spiier (1A4))
unoviit Fuote NELU Readle geoce 11
Ereene giate saler
1000
Ober and Dierskat (Pastt JURE
1e.vetor and Thms Patline Dicee: (h-00)
137eaeet and (Macet En-
220
200
180
306
0.6
800
300
140
01000
Standerd Faddi and nature Paadic Pleett US-85
Skmiore Tank (Pdd: J6451
Dar (edd:: J143)
03
Figure 35: Cross-plots continued.
50

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction are supported by thorough experiments in our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in the Closing of our paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
51

---

Justification:
We include no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our method can be implemented following the details provided in our paper. Our method is quite simple and can easily be integrated into existing ViT codebases.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
52

---

Answer: [Yes]
Justification: We provide the code of our PyTorch implementation with a link, which also contains all images from our introduced dataset, ImageNet-Hie
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssincolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In our main paper. we outline hyperparameters that differ between runs. In our Appendix, we outline hyperparameters that are consistent between runs. We use commonly used benchmarks that are listed in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: However, we report all experimental results in the Appendix. These can be used to calculate statistical significance required.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
53

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In our "Training ViTs" Appendix section we provide details on the required GPU-hours,
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics /tttp:://neur/i http://p.........tchhhcccccceeee
Answer: [Yes]
Justification: We read the Code of Ethics. In our Appendix, discuss consent and copyright with respect to the dataset we introduce.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
54

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Our dataset consists of 5k images, we manually downloaded them from the internet and reviewed them several times. We do not believe any images pose safety risk. Our models do not pose any more of safety risk than other ViT-B models.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credit creators of datasets and models. We share the websites used to collect images and their licenses.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
55

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We share training details in the Appendix and provide code via zip file. We discuss dataset license in the Appendix and provide all images via URL.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not perform research with human subjects nor crowdsource.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our research does not require IRB approvals.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
56

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
57


Reviews:
[{"review_id": "MrbpgF3pIr", "reviewer": "Reviewer_U7cz", "summary": "Vision transformer is known for its constrained scalability across various image resolutions and this work is designed to address the generalization ability of ViT at high-resolution images. Specifically, the authors propose LookHere, a drop-in replacement for the positional encoding of standard ViTs, which restricts attention heads to fixed fields of view oriented in different directions using 2D attention masks. It offers translation-equivariance, guarantees attention head diversity, and minimizes the distribution shift encountered by attention heads during extrapolation. Extensive experiments validate the effectiveness of the proposed method.", "questions": "Apart from the questions in weakness, the reviewer has one additional question:\n\nIs there a particular reason that the authors did not compare LookHere with NaViT, which can also be tested at higher resolutions?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Evaluation is extensive. The authors provide validations across different datasets, and tasks, and offer detailed analysis to demonstrate the effectiveness of the proposed approach.\n\n2. The results seem promising. LookHere exhibits the best performance compared to other baseline methods.\n\n3. The presentation is clear and the paper is easy to follow.", "weaknesses": "1. The generalization ability of LookHere to smaller-scale images remains unverified. Although the authors particularly focus on testing at high-resolution images, it would be better if LookHere could be generalized to smaller images as well.\n\n2. The ablation study on the design choices seems missing. Although the authors provide abundant experiments and analysis, the ablation study on the specific designs is missing and their efficacy is unverified.\n\n3. The size of the proposed ImageNet-HR is too small (5 images per class). The limited number of images in each class may not be representative enough and could have a strong bias implicitly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review. In particular, their recognition of our extensive experiments, LookHere’s performance, and our submission’s presentation, as well as suggestions to improve the paper. The review lists three concerns, which we address in this rebuttal.\n\n$\\textbf{Q1)}$ Does LookHere generalize to smaller images?\n\n$\\textbf{A1)}$ Our main goal is to improve ViT performance on larger images, which is an exciting challenge with great potential impact, given the long trend of applying computer vision methods to higher-resolution imagery that contain more detailed scene information. Although deploying models on smaller images will hurt performance, some users will benefit from reduced computational costs. We thus thank the reviewer for raising this point and perform new evaluations to address it. We test all 10 position encoding methods on images ranging from 64x64 px to 208x208 px and all sizes in between (at 16 px increments). LookHere outperforms all baselines at every size, averaged across all 6 datasets. In particular, LookHere-45 outperforms 2D-RoPE by an average of 4.1% (top-1 accuracy) at 64x64 px. We display these 6 plots in Figure 1 of our rebuttal PDF, and we will add this Figure to the 10th page of our main paper, should our submission be accepted.\n\n$\\textbf{Q2)}$ Is there a missing ablation section?\n\n$\\textbf{A2)}$ We believe the reviewer missed our extensive ablations in the appendix. On page 25, we report the performance of 18 ablation runs. Specifically, we report performance on all 6 benchmarks at 224x224 px and 1024x1024 px. In our main paper, we only had the space to summarize the results of these experiments; please see lines 171 - 180. We will move some of these ablation results to our main paper if accepted. Our ablations demonstrate that our novel 2D direction masks and distance penalties are crucial to extrapolate effectively — and that LookHere is robust to both 2D direction mask types and slopes. We believe these ablations are a strength of our submission. Additionally, during this rebuttal period, we ran a 19th ablation (experimenting with learnable slopes), which reviewer cHUb enquired about.\n\n$\\textbf{Q3)}$ Is ImageNet-HR too small?\n\n$\\textbf{A3)}$ Our introduced ImageNet-HR dataset consists of 5k total images, with 5 images per ImageNet class; this is smaller than most ImageNet benchmarks. However, we do not believe it is too small. \n\nFirst, although ImageNet-Val is substantially larger, we note that ImageNet-A and ImageNet-v2 comprise 7.5k and 10k images, respectively. Both are widely used (1K+ citations each). Second, we focused on quality over quantity, agreeing with other experts in the field; for instance, Lucas Beyer publicly encouraged the creation of “small, very high-quality test data [R1].” Although ImageNet-Val comprises 50k images, it contains significant annotation error and label ambiguity [4, 105]. For instance, one study estimates that 44% of the “mistakes” made by a ViT are actually correct predictions on mislabeled samples [R2]. Labeling quality has not been thoroughly studied for v2; however, we have observed annotation error and ambiguity via visual inspection. For ImageNet-HR, we manually selected and cropped high-quality and diverse images from Unsplash and flickr. Several rounds of quality control further reduced annotation error and ambiguity. Finally, to demonstrate that performance on ImageNet-HR is a meaningful measure of a model’s capability, we measure the correlation between ImageNet-HR / v2 and ImageNet-ReaL (a re-annotated version of ImageNet-Val). Across 240 paired evaluations, ImageNet-HR top-1 accuracy is highly correlated with ImageNet-ReaL top-1 accuracy ($R^2$=0.997), even more so than between ImageNet-v2 and ImageNet-ReaL ($R^2$=0.983). Please see Figure 3 of our rebuttal PDF.\n\nWe believe we’ve addressed all concerns raised by the reviewer. The reviewer also asked why we do not compare LookHere to NaViT, which we answer below.\n\nNaViT is primarily a recipe to efficiently train ViTs on images with variable aspect ratios by processing non-square grids, i.e., variable grid-height and grid-width. It achieves this by “packing” multiple images into a sequence, preventing attention between images, and then creating a batch of these sequences to process. We believe this recipe, along with other innovative ViT training recipes, may be complementary to LookHere. Furthermore, NaViT does not see improved performance when testing on longer sequences than it saw during training (see Figure 10 b in [6]) — whereas LookHere does see improved performance when extrapolating. NaViT also introduces additive factorized position encoding — which is one of the seven baselines in our submission. We thank the reviewer for this question, and we will add some of this text to our main paper, should it be accepted, since we feel it may be valuable to readers.\n\nBased on the reviewer’s question, we examined whether LookHere can also generalize to non-square images like NaViT. During this rebuttal period, we tested the 10 position encoding methods on ImageNet-Val by resizing the largest dimension to 384 and maintaining the native aspect ratio for each image. LookHere generalizes the most effectively to non-square images (see Table 2 in our rebuttal PDF).\n\n[R1] Twitter: https://tinyurl.com/4azematv\n\n[R2] “When does dough become a bagel? Analyzing the remaining mistakes on ImageNet”, Vasudevan et al., NeurIPS 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2OZ7JcD3lU", "reviewer": "Reviewer_cHUb", "summary": "The authors explore an alternative to existing positional encoding methods for vision transformers. Within attention operations, 2D attention masks (subtracted from attention matrix i.e. key-query inner product, prior to softmax) limits feature mixing to fixed fields of view pointed in different directions. ViTs resulting from proposed method exhibit improved performance across diverse tasks, especially when extrapolating to input image dimensions larger than the train data. The authors also contribute a new high-resolution ImageNet dataset variant for evaluation.", "questions": "1. **Learning Parameters**: Have you tried learning the slope values? And maybe even the entire attention masks? These could be learned using some SSL setup like MAE.", "limitations": "Discussed.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well written with clear motivation, thorough discussion of prior works, and detailed explanation of methodology. \n2. The proposed method is simple and elegant, providing better OOD performance to ViTs. This can be highly useful to the community.   \n3. Rigorous experimentation including hyper-parameter tuning to all prior works on a common training setup. \n4. Extensive evaluations and ablations to establish usefulness of proposed method and to highlight its various interesting aspects.\n5. Useful high-res ImageNet dataset that is already released anonymously (in appendix)", "weaknesses": "1. **Translation Equivariance:** Authors claim this to be a result of proposed method. However, this is not proved theoretically or demonstrated through experiments. Please explain this in method section, compare to CNNs, and provide experiments to quantify this is possible. In case I missed something related from appendix, please highlight it better in main paper. \n\n2. **[Minor] Adversarial Attacks:** Only FGSM - maybe try newer methods like AutoAttack?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review. In particular, their recognition of our writing, proposed method, rigorous experimentation, extensive ablations, and ImageNet-HR, as well as suggestions to improve the paper. The review lists two concerns, which we address in this rebuttal.\n\n$\\textbf{Q1)}$ Is LookHere translation-equivariant?\n\n$\\textbf{A1)}$ LookHere masks and biases are translation-equivariant w.r.t. patches. However, the model’s representations are not translation-equivariant due to patchification and edges. In our view, the motivation for translation-equivariant position encoding follows from the motivation for translation-equivariant convolutions — i.e., we want the feature map of an object or a region to be insensitive to its location within the image. This reasoning motivated CNN-ViT hybrids that only encode positions via convolutions [43], also achieving translation-equivariance.\n\nTo show that LookHere is translation-equivariant, we may consider two arbitrary patches at P1(x1, y1) and P2(x2, y2); then LookHere’s contribution to the attention matrix from query P1 to key P2 is A. Now impose a patch-level translation T such that the two patches are at P1’(x1 + Tx, y1 + Ty) and P2’(x2 + Tx, y2 + Ty); then LookHere’s contribution to the attention matrix from P1’ to P2’ is B. Where A=B because LookHere’s two components depend only on the relative positions of P1' and P2', which are maintained under translation — i.e., masks are a function of the direction between query-key pairs, and biases are a function of the Euclidean distance between query-key pairs. A formal proof is straightforward from here, which we are happy to provide should the reviewer wish to see it. We thank the reviewer for this question and will elaborate on LookHere’s translation-equivariance in our main paper if it is accepted. We feel this may be valuable to readers.\n\n$\\textbf{Q2)}$ Why not try newer adversarial attack methods?\n\n$\\textbf{A2)}$ We chose FGSM because it is the most well-known and is extremely fast — allowing us to attack all 80 ViTs on 50k Val images, using multiple strengths in a reasonable amount of time. During this rebuttal period, we first considered AutoAttack’s official implementation but found that their fastest attack takes ~2 hours (on an RTX 4090) per model and strength. Alternatively, we test all 10 position encoding methods against PGD attacks during this rebuttal period. Averaged across 4 attack strengths, LookHere-180 / 90 / 45 outperforms 2D-RoPE by 3.5% / 2.8% / 4.3%, and 1D embeddings by 6.8% / 6.2% / 7.6%. Please see Table 1 in our rebuttal PDF for full results. We thank the reviewer for this question and agree that more sophisticated attacks will strengthen our claim of LookHere’s adversarial robustness — space-permitting, we will add these new results to the appendix or our main paper.\n\nThe reviewer also asked about learning, rather than hard-coding, components of LookHere. In preliminary experiments, we found that learnable slopes did not improve performance at 224x224 px. During this rebuttal period, we re-ran this experiment and confirmed that learnable slopes perform roughly on-par with fixed slopes at 224x224 px and hurt performance when extrapolating to 1024x1024 px (see Table 3 in our rebuttal PDF). We thank the reviewer for this question and will add it to our ablations section.\n\nWe did not try learning masks with LookHere, mainly because other learnable relative position encoding methods do not extrapolate well. For instance, RPE-learn could theoretically learn LookHere’s masks by biasing attention scores with large negative numbers whose entries become zero after the softmax. In preliminary experiments, we also tried another learnable RPE method that did not perform well; this method calculates the relative positions between query-key pairs and processes them with an MLP, which outputs attention biases (inspired by position encoding used in hierarchical ViTs [54]). However, we are excited for the community to build on LookHere and help us develop better position encoding methods — and we welcome learnable solutions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C8aD9y3K2m", "reviewer": "Reviewer_Fbdg", "summary": "This paper introduces Lookhere, a novel mask-based positional encoding designed to address the performance degradation of ViT when resolution changes. Lookhere restricts the receptive field of each head in the attention mechanism and enhances the diversity of information perceived by each head. Extensive experiments have demonstrated the effectiveness of Lookhere. Compared to other positional encodings, Lookhere has significantly better resolution scalability.", "questions": "Please refer to the weakness", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The author's writing and illustrations are very clear, making it easy to understand and follow the method.\n2. The authors conducted numerous experiments, thoroughly validating the effectiveness of Lookhere.\n3. The issue explored by the authors, namely the resolution generalization of vision models, is a very important but rarely investigated area. The authors' work could potentially guide future research in this direction.", "weaknesses": "In summary, I believe this is a very solid work with no obvious issues. I have only one personal concern. From Figure 6, it can be observed that the essence of the proposed positional encoding seems to be limiting the receptive field of each token, so that when the resolution changes, the number of tokens each token attends to does not change too drastically. However, this approach seems to reduce the model's overall receptive field. As a result, the fine-tuning results at a resolution of 384 do not show a particularly significant advantage compared to other positional encodings. I suspect that as the fine-tuning time increases, this advantage may disappear or even be worse than RoPE. Could the authors fine-tune for a longer period (for example, 30 epochs) to verify that this approach does not negatively impact the model's accuracy?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review. In particular, their recognition of our writing and illustrations, extensive experiments, and the importance of resolution generalization in ViTs, as well as suggestions to improve the paper. The review lists one concern, which we address in this rebuttal.\n\n$\\textbf{Q1)}$ Will the 2D masks that limit each token’s receptive field harm performance if finetuned for longer, relative to 2D-RoPE?\n\n$\\textbf{A1)}$ First, a clarification: each of the eight directed heads are restricted to attend in a given direction, but the concatenation of all heads, including the four unrestricted heads, results in each token attending to the entire image. Second, the primary design motivation and performance advantage is LookHere’s extrapolation ability. We view LookHere’s other features, like performance at the training resolution and adversarial robustness, as an outcome of our 2D masks acting like a regularizer by encouraging attention diversity; in fact, we notice that LookHere consistently achieves higher training loss and lower validation loss than 2D-RoPE (which is the desired outcome of a regularizer).\n\nRegarding finetuning performance at 384, we believe LookHere’s ~1% performance advantage on ImageNet is significant (many papers have been published at NeurIPS showing a 1% improvement on ImageNet). That being said, this advantage at 384 might be due to LookHere’s inductive biases that could potentially harm performance when training for longer — as the reviewer points out. As requested by the reviewer, we finetune LookHere-90 and 2D-RoPE for 30 epochs on ImageNet at 384x384 px. The gap narrows significantly; LookHere-90 achieves 83.18% / 88.31% and 2D-RoPE achieves 83.18% / 88.00% on ImageNet-Val / ReaL. Interestingly, LookHere-90 achieves comparable results — 83.08% / 87.99% (Table 5 in our main submission) — after 5 epochs of finetuning, underscoring LookHere’s sample efficiency. Since finetuning on ImageNet for 30 epochs at 384x384 is expensive — roughly 65 A100 hours or 90% the cost of training for 150 epochs at 224x224 — we firmly believe the ability to quickly finetune models at higher-resolutions is significantly valuable to users.\n\nHowever, the reviewer raises an excellent question: Do LookHere’s inductive biases achieved via attention masks and distance penalties, that improve ViT sample efficiency and enable extrapolation, eventually limit its performance? To answer this question we train LookHere-90 and 2D-RoPE models from scratch for 600 epochs using function matching [R1] with a highly accurate teacher. LookHere-90 achieves 84.94% / 89.39% and 2D-RoPE achieves 85.06% / 89.39% on ImageNet-Val / ReaL. These results are around the empirical upper bound of ViT-B/16 models trained on ImageNet-1k at 224x224 px. Thus, we demonstrate that LookHere’s inductive biases do not limit its performance. We also find that this LookHere model outperforms this 2D-RoPE model by 15% when tested on ImageNet-Val at 1024x1024 px — maintaining a vast extrapolation advantage.\n\n[R1] “Knowledge distillation: A good teacher is patient and consistent”, Beyer et al., CVPR 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o3MX36Cfou", "reviewer": "Reviewer_irRJ", "summary": "This paper proproses LookHere, a novel positional encoding method for Vision Transformers for dealing with high-resolution images. Specifically, LookHere explicitly constratin attention heads to attend certain directions via 2D masks. With comprehensive experiments. LookHere demonstrates strong performance across different benchmarks. It also achieves better performance than existing positional encoding methods on high-res images.", "questions": "See the weakness.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The idea is very clear. High-res images introduce new challenges on positoinal encoding due to the mismatch between the training and inference time. By introducing 2D attention masks, we can assign different heads a direction, such that they obtain explicit bias and will know how to attend on high-res images. \n2. The experiemnts are very comprehensive and strong, which well demonstrate the effectiveness of the proposed method. \n3. This paper is well-written and in a very good shape.", "weaknesses": "1. Actually the authors have clearly dicussed the weakness of their method, e.g. hand-designed masks, single scale of ViT. Beside that, have the authors considered a discussion with ConViT [50]? ConViT introduces convolutional bias to attention head, which also restricts the head to only attend to certain areas.\n2. (Minor) It would be more promising to see if this technique can be applied into multi-modal LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review. In particular, their recognition of our clear ideas, comprehensive experiments, and writing, as well as suggestions to improve the paper. The review asks two questions, which we address in this rebuttal.\n\n$\\textbf{Q1)}$ Have you considered a discussion with ConViT [50]?\n\n$\\textbf{A1)}$ We agree there are some conceptual similarities between LookHere and ConViT — e.g., local and directional inductive biases. In ConViT, they arise from carefully initializing relative position encoding such that attention behaves like convolutions early in training. ConViT can learn to overcome the initially restricted views via learnable gating. In our estimation, ConViT is most similar to RPE-learn, with a clever and effective initialization scheme. We thank the reviewer for raising this point, and we will add a short discussion of ConViT to our final paper, should it be accepted.\n\n$\\textbf{Q2)}$ Can you apply this technique to multi-modal LLMs?\n\n$\\textbf{A2)}$ We do not have the resources to train multi-modal LLMs, for instance vision-language models (VLMs) with LookHere position encoding — at least during this rebuttal period. Given our extensive experiments, we fully expect that LookHere can significantly improve the extrapolation ability of VLMs. We also believe LookHere may improve VLMs in other ways. For example, compositional reasoning is a significant limitation in VLMs [R1, R2, R3]; e.g., they may find it challenging to differentiate between “a bed to the left of a dog” and “a dog to the left of a bed.” LookHere explicitly encourages ViTs to learn direction-aware representations since each head attends to a specific direction. We appreciate the reviewer’s suggestion and are excited to bring explicit directional inductive biases to VLMs in future work!\n\n[R1] “SugarCrepe: A benchmark for faithful vision-language compositionality evaluation”, Hsieh et al, NeurIPS 2023\n\n[R2] “Image Captioners Are Scalable Vision Learners Too”, Tschannen et al., NeurIPS 2023\n\n[R3] “CREPE: Can Vision-Language Foundation Models Reason Compositionally?”, Ma et al, CVPR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their thoughtful comments and are pleased with the positive reception of our submission. In particular, we appreciate the recognition of our writing, presentation, and extensive experiments from all reviewers. Additionally, we appreciate the suggestions to improve our paper. Should our submission be accepted, we believe we will have space for these additions on a 10th page. There were no shared concerns among reviewers, so we formulate our rebuttal as replies to specific reviewers.

During this rebuttal period, we ran additional experiments that reviewers enquired about. These results can be found in a 1-page PDF attached to this comment.

Meta-review: In the discussion period, the authors addressed the concerns of the reviewers and the reviewers are happy to accept this paper. The AC also enjoyed reading this paper and the proposed idea of directionally limiting the fields of view to better process high-resolution images.
Decision: Accept (poster)