Locality-Attending Vision Transformer
Authors: Sina Hajimiri
Farzad Beizaee
Fereshteh Shakeri
Christian Desrosiers
Ismail Ben Ayed
Jose Dolz
Venue: ICLR 2026 Poster (2026)
Keywords: Vision Transformer
Semantic Segmentation
Attention Mechanism
Global Average Pooling
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A recipe for training vision transformers to learn representations better suited for segmentation

Abstract: Vision transformers have demonstrated remarkable success in classification by leveraging global self-attention to capture long-range dependencies. However, this same mechanism can obscure fine-grained spatial details crucial for tasks such as segmentation. In this work, we seek to enhance segmentation performance of vision transformers after standard image-level classification training. More specifically, we present a simple yet effective add-on that improves performance on segmentation tasks while retaining vision transformers' image-level recognition capabilities. In our approach, we modulate the self-attention with a learnable Gaussian kernel that biases the attention toward neighboring patches. We further refine the patch representations to learn better embeddings at patch positions. These modifications encourage tokens to focus on local surroundings and ensure meaningful representations at spatial positions, while still preserving the model's ability to incorporate global information. Experiments demonstrate the effectiveness of our modifications, evidenced by substantial segmentation gains on three benchmarks (e.g., over 6% and 4% on ADE20K for ViT Tiny and Base), without changing the training regime or sacrificing classification performance. The code is available at https://github.com/sinahmr/LocAtViT/.

Published as a conference paper at ICLR 2026
LOCALITY-ATENDING VISION TRANSFORMER
Sina Hajimiri S Farzad Beizaee, Fereshteh Shakeri, Christian Desrosiers, Ismail Ben Ayed, Jose Dolz
ÉTS Montreal, LIVIA, ILLS 8 seyed-mosee mohammadsina ha jimirii. smtl.mm................mm. net
ABSTRACT
Vision transformers have demonstrated remarkable success in classification by leveraging global self-attention capture long-range dependencies. However, this same mechanism can obscure fine- nee-grained spatial details crucial for tasks such as segmentation. In this work, seek to enhance segmentation performance of vi- sion transformers after standard image-level classification training. More specif- ically, we present simple yet effective add-on that improves performance on segmentation tasks while retaining vision transformers' image-level recognition capabilities. In our approach, we modulate the self-attention with a learnable Gaussian kernel that biases the attention toward neighboring patches. We fur- ther refine the patch representations to learn better embeddings at patch positions. These modifications encourage tokens to focus local surroundings and ensure meaningful representations at spatial positions, while still preserving the model's ability to incorporate global information. Experiments demonstrate the effective- ness of our modifications, evidenced by substantial segmentation gains on three benchmarks (e.g., over 6% and 4% on ADE20K for ViT Tiny and Base), without changing the training regime or sacrificing classification performance. The code is available wttttps:: /tt/t/itttttttt thub. com s http://iaaaamm//tt/iiii.
INTRODUCTION
Vision transformers (ViT, Dosovitskiy et al. 2021) have emerged as powerful visual backbones by modeling images as sequences of patch tokens, processed with self-attention. Unlike convolutional neural networks (CNN, [eCun (CNn,LeCun et al. 2015), which aggregate local information a restricted recep- tive field, ViTs can capture long-range dependencies at any layer. This global attention mechanism has proven highly effective for image classification, enabling ViT models to surpass CNN perfor- mance when sufficient data is available (Touvron et al. 202 A key factor behind this success is the ability to integrate global context that leads to more uniform and holistic representations across layers, which enhances the recognition of high-level image semantics (Raghu et al., 2021).
The same global focus that makes ViTs excel in classification, however, poses challenges for dense prediction tasks such as semantic segmentation. These tasks require precise localization and fine- grained spatial detail, properties that convolutional inductive biases naturally encourage but vanilla ViTs lack (Hassani al. 2023). As a result, the design of spatial attention and feature hierarchy has been found critical for adapting transformers to dense tasks (Wang et al.) 2021 Liu et al.) 2021). Still, tension remains between capturing global context and preserving local detail. Global atten- tion can dilute local cues, whereas purely local schemes may miss long-range dependencies needed for holistic understanding. Besides, the classification objective used by models often neglects the necessities of dense prediction, motivating need for a "'egmentation-in-mine ically, we show in Appendix [F that, in VIT trained for classification, patch tokens progressively lose distinct local structure and become increasingly aligned with the [CLS] token.
pretraining. Empir-
More recently, foundation models trained large-scale (Radford et al. 2021 Oquab et al. 2024), which learn versatile visual representations, have seen broad adoption in breadth of visual tasks. Despite the availability of more intricate designs, these models still mostly adopt vanilla ViT due to its simplicity and ease of integration. This widespread reliance underscores the practical value of enhancing VIT's capabilities rather than pursuing more complex new designs. A prominent exam- ple is CLIP (Radford et al. 2021), which couples ViT-based image encoder with text encoder

---

Published as a conference paper at ICLR 2026
VIT
LoCAtVIT
VIT
LOCAtVIT
(CLS) token
Attention Intensiity
VIT
LocAtVIT
VIT
LocAtviT
Figure Qualitative evaluation on the attention maps. The final attention maps (before the classification head) of ViT and LocAtViT for the [CLS] token and three patches are illustrated for an image with label school bus.
to align representations, enabling zero-shot classification and open-vocabulary recognition. Such representations can be repurposed for dense prediction, for instance, by comparing local features to text prompts, but this adaptation is non-trivial. Furthermore, recent studies try to harness CLIP's
knowledge for segmentation without any task-specific training (Zhou et al. 2022] Wang al. 2024 Hajimiri et al. 2025). However, as CLIP and similar models are not trained for quality local repre- sentations, their features often lack the spatial granularity needed for precise dense prediction.
Contributions. In this paper, we propose a modular Locality-Attending (LocAt) add-on, which incorporates two ideas: (i) We modulate the attention logits with a learnable Gaussian kernel centered on each token's lo- cation, ensuring that patches closer to the token receive higher attention. This acts an explicit inductive bias en- couraging each token to attend to its local neighborhood while still allowing global interactions. We denote the re- sulting self-attention module as the Gaussian- Auggmettt- (GAug) attention (Section 4.1p. (ii) We enhance patch representations for segmentation by introducing minor changes prior to the classification head, preserving the meaningfulness of spatial tokens that are most important for dense prediction. We term this procedure Patch Rep- resentation Refinement (PRR) that addresses the gradient flow issue ViT's segmentation, which is overlooked in the literature (see Section 4.2p. LocAt refers to the combination of GAug and PRR, and Figure 2 demon- strates that it improves different baselines, yielding sig- nificant segmentation performance gains (arrows pointing
10
RegViT
Avg. Segmentation mlou
ROPEVIT
38
Jumbo
36
34
Baseline
w/ LocAt
Classification Top-1 Acc.
Figure 2: LocAt considerably en- hances different baselines in segmen- tation, while preserving or even improv- ing classification.
upward), while preserving improving classification accuracy (no arrow pointing to the left). The proposed add-on also enhances the quality of attention maps, as illustrated in Figure [ LocAt is lightweight and objective-agnosicc add-on, also compatible with self-supervised pretraining. Im- portantly, the minimal architectural changes required to integrate LocAt make readily applicable to any ViT with marginal changes, facilitating usage foundation models. Our perspective is that ViT pretraining should be designed with downstream dense prediction in mind, while remaining faithful to the vanilla ViT architecture and training regime.
2 RELATED WORK
Hierarchical VIT backbones for dense prediction. While the original ViT targets image classi- fication and produces low-resolution features with weak locality priors (Dosovitskiy et al.) (02T), dense prediction has motivated backbones that retain recover spatial detail across stages. Some
2

---

Published as a conference paper at ICLR 2026
works use pyramid and loken-merging designs to introduce multi-scale features and lightweight de- coders for segmentation (Wang et al., 2021 Xie et al. 202T). while others build parallel branches for local and global processing (Chu et al. 2021). These works show that topology substantially helps dense tasks. However, they typically require non-trivial architectural changes (new stages or merging blocks) and may rely local window attention that limits full-image interaction.
Convolution-based hybrids. Another line injects convolutional priors either inside attention or in the beed-fowward network to encourage local bias while keeping global modeling. Works use con- volutional projections (Wu et al., (021a), add gated positional self-attention to softly bias toward convolutional behavior (d' Ascoli et al. 202TD, couple local convolutional features with global rep- resentations (Peng et al. 2021), or add convolutions in the feed-forward network (Li et al.) 2021) ) These hybrid models add extra modules that require tuning, and they can reduce plug-and-play compatibility with off-the-shelf ViTs, as they often introduce branches replace core components. Besides, convolution offers a spatially-shared kernel which is independent of patch information.
Locality mechanisms inside attention. Orthogonal to backbone design, many papers modify the attention pattern itself to introduce locality. Many of the works use fixed or structured windows (Liu et al. 2022 Dong al. 2022, Yang et al.] 2021). Other ideas include utilizing sliding or dilated
neighborhoods to expand receptive fields efficiently (Hassani et al. 2023 Hassani & Shi] 2022),
sampling content-relevant keys (Xia et al., 2023), selecting regions using dynamic sparse rout-
ing (Zhu et al.) 2023), or using explicit global-local mixers to balance context with locality (Ding al. [2022] Tu et al. 2022] Chen al. 2022, Hatamizadeh al., [023). Most of these approaches
restrict or mask interactions (using windows or patterns) or add mixing subsystems that complicate
design, impeding their widespread adoption.
Positional encodings that strengthen locality. Beyond absolute embeddings, relative positional encoding (RPE), and rotary positional encodings (RoPE) improve spatial awareness in ViTs (Shaw ett al. 2018; Liu et al.) 2021: Wu et al. 2021b (Su et al. 2224:; Heo et al. 2024). These approaches are orthogonal to attention locality, and we briefly mentioned them to emphasize that they encode locality as well. Our work complements rather than replaces them,
show in the experiments.
Improving token representation. Recent work on register tokens augments ViTs with dedicated auxiliary tokens that absorb non-informative computation and yield smoother feature maps helpful for dense prediction (Darcet et al., 2024). Unlike this approach, we do not require auxiliary tokens, and we also address the issue of gradient flow to spatial patch outputs, overlooked in the prior work. Some works introduce class-attention layers that specialize the last blocks to refining only the class token, while keeping patch tokens fixed in those layers, leading to suboptimal dense prediction per- formance (Touvron et al., 2021b). Finally, pooling heads such as global average pooling (GAP) and multihead attention pooling Zhai et al., 2022) aim to produce a stronger pooled representation for classification by aggregating patch tokens, while our work is explicitly designed for segmentation- in-mind training with an emphasis on improving the spatial token representations themselves rather than only the pooled vector.
Foundation models for dense prediction. Large pre-trained foundation models, such as CLIP (Radford et al., (202I), demonstrate impressive zero-shot generalization on image-level recog- nition by leveraging ViT backbones. The preference for the standard ViT backbone can be attributed its strong global attention, predictable scaling behavior with data and model size, and uniform architecture that avoids the need for complex stage-wise tuning as the model grows (Zhai et al., 2022 Alabdulmohsin et al., 2023). However, despite excelling on image-level benchmarks, such models remain less effective for dense prediction because their representations are predominantly global and aask-agi (Shao et al., 2024). As a result, additional adaptation or decoding layers usually required to repurpose them for segmentation or detection (Li et al.) 2202 Xu et al.) 2023] Luo et al. 2023). While these adaptations yield improvements, they do not fully address the core issue: foundation-model ViTs trained with classification objectives- tend emphasize global semantics over local detail (Liang et al. 2023).
A ViT backbone that natively preserves both local detail and global context could enable foundation models to excel at dense prediction without extra adaptation layers or specialized fine-tuning. In this work, we take step in that direction by refining the ViT backbone itself. Our approach aims to
3

---

Published as a conference paper at ICLR 2026
potentially bridge the gap between the powerful image-level understanding and the requirements of pixel-level prediction tasks.
PRELIMINARIES
Each ViT layer 1 takes a sequence of tokens x(1-1) [[++Aa)) input, containing a [CLS] token and hwo spatial patch tokens. Each token is C-dimensional vector, and h and w denote the number of patches in each column and row. x0) is the partitioned and flattened input after adding the positional embeddings. At each layer I, the following operations are applied, where LN, attn, and MLP denote layer normalization, self-attention, and feed-forward network, respectively:
(x-11) + +atnn (LN(x(-1)).
x' + MLP (LN())).
Each self-attention module (attn) consists of two sets of weight matrices: W9, W W RCdd to compute d-dimensional query, key, and value matrices (i.e., q, k, R R()-ww)xd) based the input, and W E RdxC for the final projection. After obtaining k, and v, we calculate:
Z softmax (k SVV/VV
Matrix Z (L+hhw))d is then transformed by W form the output of the layer. The attention logits of patch p are represented by the nd row of qk /Vd. Note that for simplicity, present the formulation of single-head self-attention.
4 METHOD
We now present LocAtViT, which enhances VIT with two modular components, GAug attention (Section 4.D and PRR (Section 4.2), and is trained with the same classification objective VIT.
4.1
ATTENTION
We introduce explicit locality into vision transformers by adding a patch-specific Gaussian kernel to the attention logits for all spatial tokens. We first present the modified self-attention, then describe how the kernel is computed, and finally define the resulting attention addition.
Modified self-attention. At each self-attention layer, we add supplement matrix S to the atten- tion logits, encouraging each patch to attend more strongly to its local neighborhood. With this addition, the self-attention formulation of Eq. 3 modified as follows, which is also depicted in Figure 3a
gk T + + Vd
= softmax
(4)
We construct S so that a patch attends more to its immediate surroundings, with the added bias decaying smoothly with distance from p. Since our locality prior is defined on the spatial grid, does not apply to the [CLS] token (which has spatial coordinates). Concretely, we first compute all the following quantities for the hw spatial tokens, and then embed them into a (1+hw) x x01 (1+hww matrix by pero-padding the row and column corresponding [CLS].
natural choice for such a distance- based locality prior is an unnormalized Gaussian centered at p (more information is available in Appendix E A Gaussian kemel provides smooth, monotone decay of influence with distance, controlled by a variance parameter o² (in the isotropic case). This gives interpretable handle the effective receptive field: small yields sharp. highly local focus, whereas large approaches a nearly uniform weighting over patches.
We parameterize the variance of the Gaussian kernel for each patch by 2D vector, stored in the th of Σ h which controls the attention span along both axes. Because patches may
4

---

Published as a conference paper at ICLR 2026
require different receptive fields, predict these variances from the spatial query matrix, i.e,, 4sp E R hgmaxd the sub-matrix of q obtained by removing the [CLS] row. Using learnable weight matrix W E Eddx (with f a scaled sigmoid ensuring positive, bounded values), we compute:
Σ f(sp W
(5)
Gaussian kernel.
For patch grid of size h x w, denote the of coordinate vectors
P P= li (1..........,), (E(1........))..(.2...........
(6)
an low x matrix. The hw hww pairwise squared difference is computed as:
(P P.mm 2 for (1,2),
D ptm
where p and denote indices of the source and target patches, and indexes the coordinate dimen- sions. Given Σ, we compute the Gaussian kernel over spatial tokens, G E B bbw
as:
2
Gp = exp -
Σ
D
ptm
Σ
pm
which determines the addition attention logits from patch p to t.
Supplement matrix. From Eq. {8 each entry of G lies in [0, I]. therefore, directly adding G to the attention logits would create a scale mismatch. To mitigate this, we use learnable weight matrix W E Rdxx that predicts per-query scaling from the spatial query matrix. Entries in scale rows of the Gaussian kernel (softplus ensures positive coefficients), which yield S after zero aaro-nadding the [CLS] row and column. Concretely:
softplu((4bp W E R
OT
S diag(a) G
E (((th)))(((+h))
(10)
Intuitively, a as a per-query, row-wise balancing factor between the original attention logits and the Gaussian locality prior. For tokens where the network predicts small values of ax the contribution of negligible and the behavior approaches standard global self-attention (weak locality), whereas larger values of a yield stronger local bias. This makes our approach a soft, data- dependeetndnnt locality mechanism rather than hard constraint. We empirically analyze the effect of this scaling. well as parameter-free alternatives, in Appendix D.3 and D.4
We refer to our modified self-attention as Gaussian-Augmennt (GAug) attention. Figure trates the generation process of the supplement matrix.
3b illus-
4.2 PATCH REPRESENTATION REFINEMENT
Problem statement. In a classification task using ViT, only the [CLS] token's 's output of the model used for computing the loss. While effective for classification, this approach has fundamental lim- itations for dense prediction from a gradient flow perspective. More concretely, the patch positions' outputs receive direct supervision, i.e., are at those positions. However, these output representations are crucial for further dense prediction. This is problematic because the fine-grain spatial information carried by individual patch tokens is not effectively learned at the final layer.
important to the model what VIT's final outputs
Some subsequent methods, such as Swin (Liu et al. 2021). remove the [CLS] token and use global average pooling (GAP) before the classification head. However, this forces an undesirable behavior from a dense prediction standpoint, i.e., a uniform gradient flow across all positions. For example, in image of bird with other objects in the background, GAP compels the model to match all patch representations -inittatiiiininnnnnnnnnn background regions-wiin with the classifier's 's prototype of bird. The uniform gradient flow means that all patch tokens receive equal importance, regardless of their relevance, potentially leading to representations particularly suboptimal for tasks like segmentation. Moreover, GAP has been shown to reduce localization in higher layers (Raghu al. 2021).
5

---

Published as a conference paper at ICLR 2026
Gaussian Augmant
Gaussian
Augment
Wo
divide sumjaxue-1
suftmax
W
attn(x)
(a) Modified self-attention
(b) Supplement matrix S.
Figure 3: Illustration of the Gaussian-Aummenter attention for grid. (a) The Gaussian addition is obtained based on the query and is added to the attention logits. The pdh row in the attention logits matrix presents the attention of patch p to patch tokens. The reshaped matrix illustrates that with GAug. both local and global attentions matrix encourages attending to the locality and is computed using the pairwise squared difference tensor D from Eq. 7 For simplicity, the [CLS] ] token is not shown in this visualization, and Gaussian variances and scaling coefficients are set to a constant value for all patches.
integrated. (b) The supplement
Proposed solution. To encourage meaningful patch representations the final layer's output, ( we apply parameter-free attention before the classification head. We reshape x into H heads, x + [x]=1 where X; (1(++u))xd and compute:
X4 x
x = softmax
(11)
then reshape back to x E ((++hw)XC This can be viewed parameter-free multi-head self- attention. This operation, which introduces no new parameters, aggregates information from all patch positions in non non- con-uniform manner, thereby preserving their unique contributions and ensuring diverse gradient flow across patch locations. The resulting representation at the [CLS] token, x5. then passed to the classification head. We refer to this strategy as Patch Representation Refinement (PRR), which can be seen as an alternative to GAP, suitable for segmentation-in inniiinn----iinddd pretraining.
Our components share the common objective of making ViT's representations more suitable for dense prediction, and they act at different stages. GAug operates inside the backbone, modifying self-attention bias information exchange toward local neighborhoods so that patch tokens can bet- ter encode fine spatial details. PRR. in contrast, acts right before the classification head and changes how tokens are aggregated explicitly route supervision and gradients to patch outputs. prac- tice, each module can be attached independently to VIT backbone (see ablations in Section 5.4). However, they are coupled through the gradient path: with standard [CLS] classification, if PRR not present, last block's GAug has little effect because its parameters receive gradient from the loss. PRR routes gradients to those GAug parameters so they can be effectively learned.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets. For the main experiments, where assess both classification and segmentation per- formance, we first train models on ImageNet-1K (Deng et al. 2009 Russakovsky et al., 2015). which contains 1.28M training images from 1,000 classes. Then, further utilize these mod- els for training on segmentation datasets: ADE20K (ADE, Zhou et al.] 2019). PASCAL Con- text (P-Context, Mottaghi al. 2014), and COCO Stuff (C-Stuff, Caesar et al. 2018 Lin et al.

---

Published as a conference paper at ICLR 2026
2014). which contain 150, 59, and 171 semantic categories, respectively. ADE20K and COCO Stuff images are resized to 512 x 512 and PASCAL Context images to 480 x 480. Furthermore, we also assess classification performance smaller scale datasets: CIFAR-100 (Krizhevsky & Hinton, 2009) and mini-ImageNet (Vinyals et al., (016), a subset of ImageNet- K, consisting of 100 classes with 500 training and 100 validation examples each. In all classification experiments, images resized to 224 x 224.
Implementation details. Our method is implemented using the PyTorch Image Models (timm) [Wightman] 2019) library. We train models on ImageNet-1K for 300 epochs, with ini- tial learning rate (LR) 0.001 and 20 epochs of linear warm-up. CIFAR-100 and mini-emageNet models are trained for 600 epochs, with LR 0.0005 and 120 epochs of linear warm-up. Global batch size set to 1024, and we use AdamW (Kingma & Ba 2015 Loshchilov & Hutter 2019) opti-
mizer with a weight decay of 0.05. Similar to Ding et al. (2022), simple triangular learning rate scheduler (Smith & Topin 2018) applied, and the stochastic depth drop rates (Huang et al. 2016) for the Tiny, Small, and Base backbones are set to 0.1, 0.2, and 0.4, respectively. We follow Liu et al. (202T) for data augmentation and use RandAugment (Cubuk et al. 2020), Mixup (Zhang et al. 2018 Cutmix (Yun et al., 2019), and random erasing (Zhong et al. (020). The sigmoid function in Eq. 5 is scaled to have a maximum of max(h, w), and shifted to satisfy f(0) =
For semantic segmentation, utilize the MMSegmentation toolbox (OpenMMLabb 2020) and em- ploy a simple 1-layer MLP on top of the frozen classification-traine models. This configuration en- sures that segmentation performance mainly reflects the discriminative power of the classification- trained backbones in dense prediction. This setup aligns with our goal of isolating and assessing patch representation quality under low-tuning regime. Training on segmentation datasets is per- formed over 20K iterations with a batch size of 32. When processing images at resolutions different from the pretraining resolution, scale GAug's variance proportionally.
5.2
MAIN RESULTS
Segmentation performance. The LocAt add-on can be applied to several ViT-based models, and Table I evaluates its effect, in terms of classification performance on ImageNet-1K, as well as seg- mentation performance on three benchmarks, when applied to five models: ViT (Dosovitskiy et al.) 2021 Swin Transformer (Liu et al. 2021), ViTs with registers (denoted as ViT, we use 4 reg- isters. Darcet et al.) 2024). Rotary Position Embedding for ViTs (denoted as RoPEVIT, Heo et al.) 2024 and Jumbo (Fuller et al. 2026). Comparing each baseline with its enhanced counterpart (gray row below), indicates that LocAt's addition is useful in improving the segmentation performance of all. For instance, LocAtViT Tiny achieves substantial improvement of +6.17% 7,, +4.86% % and +5.86% over ViT on ADE20K, PASCAL Context, and COCO Stuff, respectively. Importantly, Locehhnnadeed models' superior segmentation performance is achieved without compromising classification performance; in fact, they deliver comparable or even improved accuracy across dif- ferent models (e.g., LocAtViT outperforms VIT by +1.55% % in the Tiny backbone).
LocAt improves baselines that are architecturally close to ViT significantly, e.g., RoPEVIT, and interestingly, it brings improvements over Swin as well. We believe this is not trivial as the add-on was designed for ViT's architecture, in which there exists a [CLS] token and the attention width is not limited, while in Swin the windowed attention mechanism severely affects the extent to which LocAt can play role. Furthermore, our add-on incurs negligible increase in computational cost in terms of number of FLOPs over the corresponding counterparts (measured at 224 224 using Sovrasov] 2018 Additional experiments are presented in Appendix
Classification performance. In addition to the ImageNet- 1K classification results in Ta- ble [] Table [ investigates LocAt's classification effectiveness on small-scale datasets: mini- ImageNet (Vinyals et al., 2016) and CIFAR-100 (Krizhevsky & Hinton| 2009). Although designed to enhance segmentation, these results demonstrate LocAt's classification effectiveness even when trained on small-scale datasets. LocAt improves ViT's performance by 3-6% on mini-ImageNet and 4-7% on CIFAR- -000 while only introducing 2,340 new parameters (0.003% increase for Base). We do not report segmentation results for models trained on these datasets since due to their scale and number of classes, representations not expected generalize well to segmentation benchmarks.
7

---

Published as a conference paper at ICLR 2026
Table Segmentation performance of models and their counterparts with our LocAt extension (in gray), along with their classification performance ImageNet-1K, which the models are initially trained on. Results demonstrate that (i) LocAt substantially boosts segmentation performance (our primury focus), while preserving even improving the classification performance, and (ii) this ef- fect holds for a variety methods, for different backbone sizes. Furthermore, ((i) the segmentation gains appear not only in weaker baselines, but also in strong, high-perforning models, where clas- sification improvements are harder to achieve.
Segmentation mloU (%)
Top-1 (%) ImageNet
#iParams (M)
FLOPs
Method
ADE
P-Context
C-StufT
(G)
VIT
17.30
33.71
20.29
72.39
6
1.26
LocAt
23.47
38.57+4.86
26.1545.86
73.94
6
1.27
Swin
25.58
36.78
28.34
81.18
28
4.50
LocAi
26.52.0.94
37.65,0.87
29.09.a.75
81.43
28
4.51
Tiny
RegViT
15.98
33.45
19.58
72.90
6
1.29
LocAt
24.39+8.41
39.90+6.45
27.3847.80
74.08
6
1.30
RoPEVIT
19.17
38.16
22.75
73.60
6
1.26
LocAt
24.48.5.31
40.7942.6
27.9845.23
74.34+0.74
6
1.27
Jumbo
20.33
36.36
22.13
78.71
17
1.40
LocAt
21.62,1.29
37.22,0.86
23.87+1.74
78.78
17
1.42
ViT
28.40
43.10
30.43
80.99
86
17.58
LocAt
32.64.4.24
45.35
33.6243.19
82.31
86
17.64
Swin
31.90
40.11
33.60
83.41
88
15.46
LocAt
32.89.0.99
41.44,..33
34.20,0.60
83.43+0002
88
15.47
Base
Rig
27.93
41.81
28.99
80.71
86
17.95
LocAt
32.71
46. 14,433
34.12,5.13
82. 1
86
18.02
RoPEVIT
31.38
48.83
34.35
82.
86
17.58
LocAt
34.94.3.56
49. 49.244..41
36.37+2.02
82.54
86
17.64
Jumbo
32.20
47.31
34.65
84.42
260
19.74
LocAt
35.69.3.49
49.20,1.89
35.84
84.43+...1 +0,01
260
19.81
Table 2: Classification top-1 accuracy of ViT and LocAtViT' for different backbone sizes mini-ImageNet and CIFAR-100, showcasing LocAt's effectiveness small-scale datasets.
Table 3: Self-supervised performance of Lo- cAtViT used in DINO, demonstrating LocAt's effectiveness in the self-supervised regime.
Experiment
VIT-S/16
LocAtViT-S/16
mini- ImageNet
CIFAR-100
Linear classification
65.52
67.65.213
Size
VIT
LocAtViT
VIT
LocAtVIT
10-NN
61.69
63.96.227
Tiny
74.94
78.47-333
73.84
80.43
Nearest
20-NN
61.53
63.74-221
Small
78.98
84.30.ax2
76.33
11413
neighbor
100-NN
59.30
61.19.189
Base
79.91
84.86.495
76.90
82.20.4.0
200-NN
57.90
59.78
Foundation models. In the previous sections, described our interest in improving ViT's seg- mentation capabilities without changing their training scheme. Our experiments support that our minor modifications lead to better dense prediction performance, while performing on par or supe- rior to the vanilla models in classification. One reason for our interest in the mentioned problem is that ViTs have been widely used across computer vision foundation models and are the go-to choice for many of the recent methods (Radford et al. 2021 Kirillov et al. 2023 Caron et al.) 2021 Oquab) al. a 2024). One of the popular models that yields versatile image representations and transfers well to different computer vision tasks is DINO (Caron et al. 2021), which is trained in self-supervised manner and can serve as a general-purpose purpose backbone.
We train DINO VIT-S/16 and DINO LocA(ViT-S/16 on ImageNet-IK for 50 epochs, and evaluate on two tasks used by Caron et al. (202T): learning a linear classifier on top of the frozen back- bone and gearest-neighhon classifictiion (k-NN) on the features. We train the linear classifier for 50 epochs. Table demoonstatts that replacing ViT with LocAtViT in DINO improves its performance on both linear and k-NN classification. We report the k-NN performance k E {10, 20, 100, 200)
8

---

Published as a conference paper at ICLR 2026
Table Hummingbird dense nearest-neighhor retrieval (mloU %) of models and their counter- parts with our LocAt extension for different backbone sizes on PASCAL VOC and ADE20K.
Tiny
Base
PASCAL
ADE20K
PASCAL
ADE20K
Method
Vanilla
+ LocAt
Vanilla
+ LocAt
Vanilla
LocAt
Vanilla
LocAt
VIT
39.2
50.3.11.1
12.0
15.2,3.2
55.8
58.7.29
19.5
21.5.20
Swin
45.2
45.3.0.1
16.1
16,3,0.2
57.6
62.8,5.2
23.3
24.6.1.3
Reg VIT
39.4
52.3.129
12.5
15.9,34
55.5
60.3
19.4
22.8
RoPEVIT
50.7
54.7,4.0
16.0
17.5,15
61.0
61.4,0.4
22.4
23.7+1.3
Jumbo 40.0
45.5.5.5
13.3
14.5.1.2
58.5
63.8,5.1
21.6
23.7+2.
as advised by Caron et al. (2021). These findings reveal our objective-agnostic modifications' ef-
fectiveness in the self-supervised regime and the potential of our method on backbones that learn general-purpose representations. While interesting, further investigation on larger foundation mod- els is beyond our computational reach and lies outside the scope of this work.
Humminghird evaluation. To further assess whether LocAt improves quality of image features, we evaluate our models using Hummingbird (Balažević et al. 2023), a protocol proposed for evalu- ating in-context scene understanding in purely mrozen-feature regime. We the implementation by Pariza et al. (2024) and follow its dense nearest- neiihtt (NN) retricval setup. Given a sup- port set of images with semantic segmentation labels, each query image is segmented by retrieving the nearest visual tokens from the support set the embedding space, without any fine-tuning or decoder training. This protocol therefore measures the intrinsic spatial and contextual quality of representations produced by the backbone, which aligns well with our motivation. Table 4 shows that LocAt consistently improves NN retrieval performance relative the corresponding vanilla backbones on PASCAL VOC (Everingham et al. (2010) and ADE20K (Zhou et al.) (2019) across ar- chitectures, suggesting that LocAt enhances spatial representations, even without any task task- specifcc fine-tuning or decoder.
5.3 QUALITATIVE ANALYSIS
An interesting implication of our proposed modifications is the refinement of ViT's patch outputs, which makes it more suitable for use cases on dense prediction tasks. Figure I offers a visual comparison of attention maps from vanilla VIT and our LocAtViT, both trained for classification, for an image labeled as school bus. From the [CLS] token's attention, we observe that VIT's focus broadly dispersed, whereas LocAtViT shows more concentrated and coherent activation on key features of the bus. Furthermore, we present the attention maps of three patch tokens to other patches. For instance, a patch on the bus side attends to nearly the entire bus in LocAtViT, whereas ViT's map is harder interpret. A patch covering the child's face generates meaningful attention in both models, but VIT seems to highlight unrelated regions more. Interestingly, for a patch near the top-right corner, LocAtViT not only focuses some patches, but also extends attention to the sky and road, all corresponding to the image background. Despite being trained solely for classification, LocAtViT exhibits improved ability detect some scene structures, suggesting that our proposed local interactions can enrich the model's contextual understanding without sacrificing global attention. Further qualitative examples are presented in Appendix C
5.4 ABLATION STUDY
In this section, we provide an ablation study on the architectural choices we made. We also provide an ablation study on the self-attention module's design in the Appendix D
Effect of GAug and PRR. Part of Table ablates GAug and PRR defined in Sections 4.1 4.2 Results demonstrate that both GAug and PRR indeed enhance the performance of the model in both classification and segmentation, and their combination pushes the performance even further.

---

Published as a conference paper at ICLR 2026
Table Ablation study on model's architecture. We report segmentation performance (mIoU %) over three benchmarks and classification accuracy (top-1 %) on ImageNet-IKK PE and GAP stand for positional embeddings and global average pooling.
Tiny
Base
Method
ADE
P-Context
C-Stuff
ImageNet
ADE
P-Context
C-Stuff
ImageNet
VIT
17.30
33.71
20.29
72.39
28.40
43.10
30.43
80.99
VIT GAug
18.98
34.97
21.51
73,16
30.26
44.36
32.21
82,00
VIT PRR
21.60
37.93
25.85
73.71
29.89
44.03
32.16
82.19
LocAtVIT
23.47
38.57
26.15
73,94
32.64
45.35
33.62
82.31
ViT PE LocAtViT PE
15.13
31.94
19.35
69.36
24.59
40.18
28.79
79.39
22.69
38.15
26.05
73.10
29.73
44.69
32.17
82.
VIT
17.30
33.71
20.29
72.39
28,40
43.10
30.43
80,99
VIT GAP
19.65
34,94
22.86
72.50
27.99
41.97
29.88
81.84
VIT PRR
21.60
37.93
25.85
73.71
29.89
44.03
32.16
82.19
Effect of positional embeddings. Part of Table evaluates the impact of the default absolute positional embeddings (PE) on our proposed LocAt add-on. For both backbone sizes, LocAtViT without PE not only outperforms ViT without PE. but also surpasses VIT with PE. This indicates that LocAr captures the spatial information embedded into PE and more, with much fewer learnable parameters. worth noting that our approach is did not intend to propose new PE method. Therefore, these results are included just to demonstrate empirically that LocAt indeed captures the spatial information that the default PE captures, which is the mechanism for capturing locality in vanilla ViT. We have shown in Table [ that LocAt is applicable alongside other, newer positional encoding approaches, such RoPE, as well.
alternative positional encoding and
Comparison between PRR and GAP. As discussed in Section 4.2 PRR addresses patches' gra- dient flow issues while overcoming GAP's limitations in segmentation. Part 2 of 5J how vanilla VIT performs when equipped with PRR versus GAP. PRR shows superior segmenta- tion performance and interestingly. it improves classification accuracy more than GAP. Moreover, although GAP helps VIT in classification, hurts the segmentation performance in the Base back- bone, which is in line with the discussions in Section 4.2 about GAP's problems in segmentation.
CONCLUSION
Summary. We present the Locality-Artending Vision Transformer, modular add-on that enhances vision transformers for dense prediction while preserving image-level capabilities and integrating seamlessly into existing ViTs. The approach introduces spective: GAug softly biases attention toward local regions to capture fine-grained spatial details, and PRR ensures meaningful gradient flow to patch tokens, strengthening representations for dense prediction. Experiments across multiple ViT baselines show that LocAt delivers consistent seg- mentation performance gains without compromising classification accuracy. Rather than replacing strong existing architectures, we offer simple, largely orthogonal upgrade for classification- trained ViTs, particularly relevant given their widespread use in foundation models. We hope these minimal changes will be adopted in future ViT-based models.
cegmentation-in-nt pretraining per-
Limitations. We evaluated our method on multiple classification and segmentation benchmarks, but all of them only contain natural images. Extending the evaluation other domains (e.g., medical imaging or remote sensing) remains future work. Additionally, while we validated LocAt within small foundation model, evaluating large foundation models (e.g.. CLIP-scale) was beyond our computational budget.
ACKNOWLEDGMENTS
This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Fonds de recherche du Québec Nature technologies (FRQNT) under grant no. 369892. We also thank Calcul Québec and Compute Canada for providing the computing resources used in this work.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Ibrahim M. Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting ViT in shape: Scaling laws for compute-optimal model design. In Advances Neural Information Processing Systems, 2023.
Ivana Balaževic, Daniel Steiner, Nikhil Parthasarathy, Relja Arandjelovié, and Olivier Hénaff. Towards in-context scene understanding. In Advances in Neural Information Processing Systems,
2023.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO-Stuff: Thing and stuff classes in context. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1209-1218, 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In /EEE/CVF Inter- national Conference on Computer Vision, pp. 9650-9660, 2021.
Chun-Fu Richard Chen, Rameswar Panda, and Quanfu Fan. RegionViT: Regional-to-local attention for vision transformers. In International Conference Learning Representations, 2022.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In Ad- vances in Neural Information Processing Systems, 2021.
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang. and Chunhua Shen. Conditional positional encodings for vision transformers. In International Conference on Learning Representations, 2023.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. RandAugment: Practical automated data augmentation with a reduced search space. In IEFF/CVF Conference on Computer Vixion and Pattern Recognition Workshops, pp. 702-703, 2020.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In International Conference Learning Representations, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hier- archical image database. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. DaViT: Dual attention vision transformers. In European Conference on Computer Vision, pp. 74-92, 2022.
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. CSWin transformer: A general vision transformer backbone with cross-shaped windows. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Stéphane d' Ascoli, Hugo Touvron, Matthew L. Leavitt, Ari S. Morcos, Giulio Biroli, and Levent Sagun. Von: Improving vision transformers with soft convolutional inductive biases. In International Conference Learning Representations, 2021.
Mark Everingham, Lue Van Gool, Christopher K. Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2):303-333, 2010.
Anthony Fuller, Yousef Yassin, Daniel G. Kyrollos, Evan Shelhamer, and James R. Green. Thicker and quicker: The jumbo token for fast plain vision transformers. International Conference on Learning Representations, 2026.
11

---

Published as a conference paper at ICLR 2026
Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. Pay attention to your neighbours: Training-free open- vocabulary semantic segmentation. In IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5061-5071, 2025.
Ali Hassani and Humphrey Shi. Dilated neighborhood attention transformer, 2022. URL https:
//axxiv.org/abs/22202 15001
Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention trans- former. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6185-6194, 2023.
Ali Hatamizadeh, Hongxu Yin, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Global context vision transformers. In International Conference on Machine Learning, volume 202, pp. 12633- 12646, 2023.
Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, 2024.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision, pp. 646-661, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International
Conference on Learning Representations, 2015.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yes Lo, Piotr Dollár, and Ross Girshick. Segment anything. In JEEE/CVF International Conference on Computer Vision pp. 4015-4026, 2023.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report, 2009.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436-444 2015.
Boyi Li, Kilian Q. Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations, 2022.
Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Lue Van Gool. ViTa Bringing locality to vision transformers, 2021. URL https://a/xx orgg abs 2 2004. 05707
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Mareulescu. Open-vocabulary semantic segmentation with mask-adapted CLIP. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7061-7070, 2023.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, pp. 740-755, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Brandon Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE/CVF International Conference on Computer Vision. pp. 10012-10022, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer-
ence on Learning Representations, 2019,
Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. SegCLIP: Patch ag- gregation with learnalle centers for open-vocabulary semantic segmentation. In International Conference on Machine Learning, volume 202, pp. 23033-23044, 2023.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmen- tation in the wild. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 891-898, 2014.
12

---

Published as a conference paper at ICLR 2026
OpenMMLab. MMSegmentation: OpenMMLab semantic segmentation toolbox and benchmark. 2020.
https://githui. wwallb..co/coo
mm- aappen-mwlaa/mmmeeeettaaaa.a
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Russell Howes, Yao Huang. Hu Xu, Vasu Sharma, Shang-Wert Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nico- las Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
Valentinos Pariza, Mohammadreza Salehi, and Yuki Asano. Hummingbird evaluation for vision encoders.
2024.
Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. Conformer: Local features coupling global representations for visual recognition. In IEEE/CVF International Conference on Computer Vision, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, volume 139, pp. 8748-8763, 2021.
Maithra Raghu, Thomas Unterthiner, Simon Korbblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In Advances in Neural Information Processing Systems, volume 34, pp. 12116-12128, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115 (3):211-252, 2015.
Tong Shao, Zhuotao Tian, Hang Zhao, and Jingyong Su. Explore the potential of CLIP for training- free open vocabulary semantic segmentation. European Conference on Computer Vision, 2024.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position represen- tations. In Conference of the North American Chapter of the Association for Computational Linguistics, 2018.
Leslic N. Smith and Nicholay Topin. Super-convergence: Very fast training of residual networks
using large learning rates, 2018.
Vladislav Sovrasov. ptllops: A FLOPS counting tool for neural networks in PyTorch framework.
https://github com/ http:///ww..o..o//////////ooooa. wryouunr...tttoc................ 2018.
Jianlin Su, Murtadha Ahmed, Yu Pan, Bo Han, and Yunfeng Liu. RoFormer: Enhanced transformer
with rotary position embedding, Neurocomputing, 568: 127063, 2024.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, volume 139, pp. 10347-10357, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In IEEE/CVF International Conference on Computer Vision, 2021b.
Hugo Touvron, Matthieu Cord, and Herve Jegou. DeiT III: Revenge of the VIT. In European Conference on Computer Vision, 2022.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Max VIT: Multi-axis vision transformer. European Conference on Computer Vision, pp. 459-479, 2022.
13

---

Published as a conference paper at ICLR 2026
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Advances in Neural Information Processing Systems, vol- ume 29, pp. 3630-3638, 2016.
Feng Wang, Jieru Mei, and Alan Yuille. SCLIP: Rethinking self-attention for dense vision-language inference. In European Conference on Computer Vision, 2024.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. In IEEE/CVF International Conference Computer Vision, pp. 568-578, 2021.
Ross Wightman. PyTorch image models.
http:://ggLchub.com/rrwro/
Setoorch-image-ne moage-models. 2019.
Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. CvT: Introducing convolutions to vision transformers. In /EEE/CV International Conference on Com- puter Vision, 2021a.
Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improv- ing relative position encoding for vision transformer. IEEE/CVF International Conference on Computer Vision, 2021b.
Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. DAT++: Spatially dynamic vi- sion transformer with deformable attention, 2023. Utttpss::://axx
01430
Enze Xie, Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Seg- Former: Simple and efficient design for semantic segmentation with transformers. In Advances in Neural Information Processing Systems, volume 34, pp. 12077-12090, 2021.
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. SAN: Side adapter network for open- vocabulayy semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. In Advances in Neural Information Processing Systems, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features. In IEEE/CVF International Conference Computer Vision, pp. 6023-6032, 2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez- Paz. mixup: Beyond empiri- risk minimization. In International Conference on Learning Representations, 2018.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yang. Random erasing data augmen- tation. In AAAI Conference on Artificial Intelligence, pp. 13001-13008, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Tor- ralba. Semantic understanding of scenes through the ADE20K dataset. International Journal of Computer Vision, 127:302-321, 2019.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from CLIP. In European Conference on Computer Vision, pp. 696-712, 2022.
Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, and Rynson W. H. Lau. BiFormer: Vision transformer with bi-level routing attention. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10323-10333, 2023.
14

---

Published as a conference paper at ICLR 2026
LOCALITY-ITENDDNE VISION TRANSFORMER APPENDIX
A TECHNICAL DETAILS
A. CODE AND COMPUTE RESOURCES
We used the implementation provided by Wightman (2019) for ViT (Dosovitskiy et 2021 Swin Transformer (Liu et al. 2021), Reg (Darcet et al., ), and RoPEVIT (Heo et al. 2024). and we used the official repository of Jumbo (Fuller et al. 2026). Results for all of these meth- ods are reproduced. Jumbo is new work and its public repository was incomplete at the time of writing this paper. hence we used the available code and implemented some of the components based on the paper. Moreover, the training regime used by Fullerr et al. (2026) is more complex than ours (training for more epochs and at different resolutions with separate teachers). We trained Jumbo with scheme consistent with other models, except for leveraging distillation similar to Fuller et al. (2026), and we did not change Jumbo's teachers mid-training. Tiny Jumbo mod- els utilize wastt--bas-ppttche 6-224 deit3-large-patchhe 222- fb- 1o-in22k-fftinnnt as teachers (Touvron et al. [202]). Our experiments mostly conducted using NVIDIA RTX A6000 48GB, V100 32GB, and A100 40GB GPUs.
and Base Jumbo models use
A.2 LLM USAGE
We used LLMs to generate code for plotting figures, tables, and other LaTeX or code-related tasks. We also used LLMs improve the writing, polish, shorten the paragraphs, while double-checking the output.
B LOCATVIT COMPARISON WITH RELATED WORK
In Table T we included five baseline methods and implemented LocAt for each.
LocAtViT to multiple related works from Section 2 CvT-21 (Wu et al. 2021a). Conformer (Peng) ct al. 2021), ConViT (d' Ascosc et al. (021). Twins (Chu et al. 2023 2021). DaViT (Ding et al., 2022 and GCVIT (Hatamizadeh al. 2003). We utilized the timm library well publicly available code and checkpoints [Wightman] 2019), and evaluated the models on our segmentation pipeline, using the same segmentation protocol as described in Section Although LocAtViT does not achieve the best classification performance, LocAt helps ViT outperform methods like Twins across all three segmentation benchmarks.
Table 6: Segmentation and classification performance of Base backbones from prior work and the proposed LocAtViT.
Segmentation mloU (%) P-Context
Top-1 (%) ImageNet
Method
ADE
C-Stuff
CvT-21
21.40
40.91
29.29
82.50
Conformer
22.11
40.03
26.37
83.83
CIn
23.08
44.82
25.20
82.30
Twins
30.47
44.55
32.27
82.71
DaVIT
30.68
44.87
32.38
84.64
GCVIT
30.91
44.71
32.77
84.47
LocAtViT
32.64
45.35
33.62
82.31
15

---

Published as a conference paper at ICLR 2026
C ADDITIONAL QUALITATIVE EXPERIMENTS
Figure provides three additional images from the mini-ImageNet dataset, alongside the attention
maps of the [CLS] token and several patches for VIT and LocALViT.
VIT
LoCALVIT
VIT
LocAtVIT
(CLS)
Attention Intensity
LocAtVIT
VIT
LocALViT
VIT
LocAtViT
VIT
ArAIIT
[CLS]] token
LocALVIT
LocAtVIT
Vil
LocAtVil
VIT
LocAtVIT
[CLS] token
LocAtVIT
VIT
LocAtVIT
Figure 4: Qualitative evaluation on the attention maps. The final attention maps (before the clas- sification head) of ViT and LocAtViT for the [CLS] token and three different patches are illustrated for three different images from mini-ImageNet with labels: orange, Komondor, and corn.
D ABLATION STUDY ON SELF-ATTENTION
In this section, we perform ablations on the design choices inside the GAug self-attention module.
D. GAUSSIAN BASED ON INPUT
In the original VIT, query vector intuitively determines the information patch should be looking for. Since the Gaussian variance controls how far a patch attends to its surroundings, we compute Σ based the spatial query matrix in Eq. 5 Table compares this approach to computing Σ based
16

---

Published as a conference paper at ICLR 2026
Table Ablations on GAug attention components. A#Params shows the difference in the number of the parameters of each model compared to LocAtViT (first row). Experiments are conducted on mini-ImageNet, and the classification accuracy (top-1 %) is reported.
Tiny
Base
P#rarmss
LocAtViT (Section
78.47
84.86
Gaussian from x
79.10
85.18
+18,504, +-29,868
Isotropic Gaussian
78.71
84.66
-780
Fixed = 1
75.20
82.81
-2,340
76.41
82.65
-2,340
width = 10
75.53
82.42
-2,340
No scaling Auto a
76.26
83.07
-780
78.48
84.54
-780
on x, the self-attention input. While the latter improves performance, it significantly increases the number parameters.
D.2 VARIANCE MATRIX
To comply with general setting, assigned separate variances for each image axis. An alternative to use a single variance per patch, forming an isotropic Gaussian kernel. This simplifies Eq. (8) to:
Gpt = cxp (- Em=1 Dptm
(12)
20
The result of this modification is referred to Isotropic Gaussian in Table 7 This table also com- pares this approach with another experiment where the Gaussian kernel width fixed to different constant values, instead of being patch-specific and query-based. These results indicate that an isotropic Gaussian kernel performs comparably, but a fixed kernel width substantially diminishes performance, demonstrating the importance of our dynamic input-dependent kernel width.
D.3 No SUPPLEMENT MATRIX SCALING
In Section we introduced learnable scaling vector a to match the scale of the supplement matrix to that the attention logits. To isolate its effect, Table reports a variant (No a)) which the supplement matrix in Eq. (10) is not scaled, i.e., set 1. This no-scaling configuration corresponds to a harder use of the locality term and consistently reduces accuracy, confirming that unscaled addition of G is suboptimal and that the learnable scaling important for balancing global attention with the Gaussian prior.
D.4 AUTOMATIC SCALING OF THE SUPPLEMENT MATRIX
motivated the need for scaling the supplement matrix before adding to the attention logits in Section 4.1 We now propose a parameter-free free, input-dependent scheme, Auto a, that automati- cally matches the scale to that of the original attention logits. Concretely, define the row-wise f2-norm vectors:
r [|
12.
q [1
(13)
u
[
2
kk
(14)
Then the standard attention logits satisfy:
qk'
ru
cos (q. k)
(15)
17

---

Published as a conference paper at ICLR 2026
where denotes the Hadamard product, and cos(q, k) (l has entries cos ky).
Hence, if we set:
a
ru
E (R0tht))((tho))
(16)
vd
then the modified logits in Eq.
44) can be rewritten as:
qk
S (cos(q. k) G),
(17)
Vd
where both terms inside the parentheses are bounded (in [-1, 1] and [o, 1]. respectively). ensuring that S scales comparably the original logits.
However, using a c would independently scale each entry of G, destroying the Gaussian kernel structure (each row of G is kernel centered patch). To preserve each kernel's shape, average a across columns:
hw
a =
aij
ā = [0, a........... ahw! E 1/
(18)
hw 1=1
and then form:
S diag(a) G.
(19)
similar to Eq. 10) This row-wise scaling applies single factor to each Gaussian kernel, preserving its shape while matching its magnitude to the attention logits. Unlike the main text, G, a, and a assume entries corresponding to [CLS] in this section, which should be manually zero since [CLS] does not correspond to a spatial location.
Auto performs close to learnable in the original LocAtViT, with slightly fewer parameters. We nevertheless keep the learnable a our main model for simplicity formulation and to give the network maximal flexibility in attenuating or amplifying locality where beneficial.
E ABLATION STUDY ON ALTERNATIVE DISTANCE-BASHD KERNELS
In the main text, we model locality with Gaussian kernel added to the attention logits (Section 4.1 The choice of a Gaussian is motivated by the desire for smooth, distance-based attenuation function with a scale parameter that controls the effective receptive field, and that can be predicted from each query token. Nevertheless, other monotone distance-based kernels are also reasonable, and compare against two other kernels in what follows.
Let Tyt II, - Polla denote the Euclidean distance between patches p and in the spatial grid. We construct two alternative kemells by predicting scale parameters y and À from the queries:
Lyt exp (-7p Tpt),
(20)
denoting the Laplace kernel, and the inverse-distance kernel:
1
1p
(21)
1 1 Tps/Ap"
In both cases, the resulting kernel matrix replaces G in Eq. (T). and the rest of the GAug formula- tion (including the scaling with a is kept unchanged.
Table compares performance of different choices of the kernel. All three variants improve over the baseline ViT, confirming that introducing a smooth distance- prior is beneficial. Among them, the Gaussian kernel delivers the strongest segmentation gains on all three benchmarks, while remaining competitive in ImageNet-IK accuracy compared to the Laplace and inverse-distance kernels.
F LOCAL FEATURE ANALYSIS ACROSS LAYERS
In the main text, argue that the global attention mechanism of vanilla ViT tends obscure fine- grained local information that is important for dense prediction. Here, we provide quantitative analysis of how local patch features evolve across layers in standard VIT and in our LocAtViT. We focus on Base models VIT and LocAtViT trained on ImageNet- IK and evaluate features on the ImageNet- 1K validation set.
18

---

Published as a conference paper at ICLR 2026
Table 8: Effect of different distance-based kernels. Segmentation performance (mloU %) over three benchmarks and classification accuracy (top-1 %) on ImageNet- IK are reported.
Tiny
Base
Kernel
ADE
P-Context
C-Stuff
ImageNet
ADE
P-Context
C-Stuff
ImageNet
No (ViT) 17.30
33.71
20.29
72.39
28.40
43.10
30.43
80.99
Gaussian
23.47
38.57
26.15
73.94
32.64
45.35
33.62
82.31
Inv-dist
22.18
38.16
25.25
74.00
28.42
43.48
30.82
81.94
Laplace
21.67
37.80
25.56
74.01
29.74
44.10
31.95
82.24
VIT
LoCAtVIT
VIT
LOCAtVIT
0.30
Cosine similarity
Cosine similarity
0.65
0.20
0.80
0.55
0.10
0.50
0.45
0.00
Layer
Layer
Average cosine similarity 8 spatial neighbors.
(b) Average cosine similarity the [CLS] token.
Figure 5: Degradation of local features in vanilla ViT. Features in VIT collapse to the global
information in the last layers while LocAtViT, patch features encode local information.
Locality score. For each layer and each spatial patch token, we compute locality score defined as the cosine similarity between that patch and its 8 immediate neighbors in the surrounding 3 x 3 window. We then average this score over all spatial locations and all validation images. Intuitively, higher locality desirable as long as representations do not collapse globally. Figure 5a reports this locality score per layer. After the third layer, LocAtViT consistently achieves higher locality score than vanilla ViT, indicating that its patch features remain more coherent with their spatial neighbors as depth increases.
indicates that nearby patches share more similar representations, which is
Patch- [CLS] similarity. High neighbor similarity alone does guarantee that meaningful local structure is preserved: if all patch tokens collapse to the same global representation, their mutual similarity (including to neighbors) will also be high. To distinguish this degenerate case from gen- uine locality, we additionally measure, for each layer I, the cosine similarity between every patch token and the [CLS] token, again averaged over all patches and validation images. Figure 5b shows that in vanilla ViT this patch- [CLS] similarity steadily increases with depth and peaks in the final layers, revealing progressive pull of patch features toward shared global representation dominated by the [CLS] token. In contrast, LocAtViT maintains substantially lower patch- [CI.S] similarity across layers, while still achieving a higher locality score.
Discussion. Taken together, these two measurements show that, in vanilla ViT, patch tokens grad- ually lose distinct local information and become dominated by global [CLS]-like content as depth grows. LocAtViT, on the other hand, preserves strong locality in patch features without collapsing them onto the [ [CLS] token. This behavior aligns with design goal: enhance the preservation of local structure while retaining the benefits global attention, thereby producing representations that are better suited for dense prediction.
G STABILITY OF LEARNED STANDARD DEVIATIONS
The per- parchatch Gaussian variances are predicted from the queries through a bounded nonlinearity in Eq. (5) ensuring numerical stability; however, in principle these values could collapse to the
19

---

Published as a conference paper at ICLR 2026
lower upper end of the admissible range. Figure 6 analyzes the mean and percentile ranges of the learned standard deviations across layers for a LocAtViT Base model trained on ImageNet- IK. We find that the predicted variances remain well inside the allowed interval and do not cluster the bounds. These observations indicate that GAug learns meaningful locality scales rather than de- generately switching the Gaussian bias "off" (very small variance) "fully on" (maximal variance) everywhere.
10-90%
Median
12
30-70%
Mean
Standard deviation
10
0
Layer
Figure 6: Layer-wise statistics of the learned Gaussian standard deviation in LocAtViT. For each layer, we summarize the distribution of Iearned standard deviation values using percentile ribbons (10-90% and 30-70%) and overlay the median (solid) and mean (dashed).
H LIMITATIONS OF THE GAUSSIAN BIAS
Our design goal for the Gaussian augmentation is to gently bias attention toward local structure, rather than to strictly enforce locality. Empirically, across the backbones and tasks reported in the main text, we observe performance gains when adding GAug and PRR. However, the magnitude of the gains depends on the underlying attention topology. The largest improvements appear on back- bones with unrestricted patch-patch attention (e.g., ViT, RegViT, RoPEVIT, and Jumbo), whereas the gains windowed-attention backbone such as Swin are noticeably smaller. This suggests that GAug is most effective when attention globally connected and locality not already hard-coded by the architecture.
To further probe this limitation, we also applied our approach to GCVIT (Hatamizadeh et al. 2023 stronger windowed-atteniee model with attention confined to small grids. In this setting did not observe improvements in the performance. We attribute this negative result to the fact that when attention is restricted to narrow windows, the additional Gaussian bias has little fully reshape the locality pattern. In contrast, even for powerful unrestricted-atteniion models such Jumbo, there remains enough flexibility for GAug and PRR provide noticeable benefits.
to meaning-
20


Reviews:
[{"review_id": "lQJjpZpegL", "reviewer": "Reviewer_bk4S", "summary": "This paper proposes the Locality-Attending Vision Transformer, a new attention mechanism aimed at improving the local inductive bias of Vision Transformers (ViTs) without sacrificing global contextual modeling. The key design introduces a modular Locality-Attending (LocAt) add-on, which modulates the attention logits with a learnable Gaussian kernel centered on each query token's location. This work also enhances patch representations for segmentation by introducing minor changes prior to the classification head, preserving the meaningfulness of spatial tokens.", "questions": "My concerns are mentioned above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Conceptually simple but effective modification. In specific, this paper computes a Gaussian kernel based on the query token and incorporates local spatial information into the attention logits, thereby balancing both global and local attention, which is beneficial for segmentation tasks. In addition, to mitigate the limitations of the [CLS] token, this work applies a parameter-free \"self-attention\"-style refinement to the output of the final layer. Though the solution is simple and straightforward, it is effective.\n\n+ The related work provides a detailed discussion of methods that optimize attention locality.\n\n+ The proposed method yields noticeable improvements across different models and tasks, and the gains observed in the ablation studies are also significant.", "weaknesses": "- The paper lacks visualization of the method's effects. For example, case studies showing attention heatmaps after applying Gaussian-Augmented attention and Patch Representation Refinement.\n\n- It is recommended to highlight the motivation for using the Gaussian kernel before the Method section. For example, by listing a table that qualitatively compares convolution-based hybrids, locality mechanisms inside attention, positional encodings, etc., and explicitly points out the advantages of using Gaussian kernel in this work.\n\n- Not all tasks necessarily require explicit local information. Directly adding S to the attention logits is a rather \"hard\" approach. Would it be possible to introduce a scaling factor $\\alpha$ to balance the logits and S, for example, using $\\alpha \\cdot S$?\n\n- This paper uses a Gaussian term to enhance the locality. As shown in Figure 3, the resulting S matrix appears similar to the pattern of sliding window. The authors are encouraged to clarify how their approach, i.e., Gaussian-Augmented attention differs from sliding window and to explain the performance gain it achieves.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n6K8SVrkKl", "reviewer": "Reviewer_oh5T", "summary": "LocAtViT proposes a simple yet effective way to enhance Vision Transformers with spatial locality by introducing GAug and PRR modules. The method achieves notable segmentation improvements while maintaining classification performance and requires only minimal architectural changes. Its strength lies in practicality and plug-and-play applicability across ViT variants. However, the work lacks a clear problem analysis, a theoretical motivation for the Gaussian kernel, and clarification of the interaction between GAug and PRR, making the approach appear more empirical than principled.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. LocAtViT introduces locality awareness through the GAug and PRR modules with minimal architectural modification.\n\n2. The proposed modules yield substantial improvements on segmentation benchmarks while preserving or even slightly improving ImageNet classification accuracy.\n\n3. The work highlights a valuable perspective that ViT pretraining can be enhanced for dense prediction by refining patch-level representations.", "weaknesses": "1. Although the author pointed out that the global attention mechanism of ViT is not conducive to capturing local details, there is a lack of analysis on the degradation of local features in the baseline ViT. \n\n2. The paper directly proposed Gaussian-Augmented attention, but did not explain why a Gaussian kernel was chosen (instead of other forms of local attenuation functions) and its correspondence with human vision or signal attenuation models. The lack of theoretical or empirical support makes this design more like a heuristic attempt. \n\n3. The relationship between GAug and the PRR module was not clarified whether they are coupled or independent. Although LocAtViT is defined as a combination of the two, the paper did not discuss whether there is a complementary or redundant relationship between the two.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bZKaaMsNMQ", "reviewer": "Reviewer_iq82", "summary": "The paper proposes LocAtViT, a ViT backbone augmentation aimed at making features more locality-aware (useful for dense prediction) while preserving image-level classification strength. Two components are introduced: (i) a Gaussian-biased self-attention that softly favors nearby patches; and (ii) a patch-aware classifier refinement that aggregates patch tokens (rather than relying solely on a [CLS] token or uniform pooling). Experiments (classification on ImageNet-1k; frozen-backbone segmentation on ADE20K, PASCAL-Context, COCO-Stuff; plus a DINO self-supervised setup) show consistent segmentation gains with parity or small gains in classification. \n\nRelative to prior locality work—e.g., ConViT (soft convolutional inductive bias via GPSA) and LocalViT (injecting locality in the FFN)—the paper’s novelty is to impose a learned Gaussian locality bias directly on attention and to pair it with a patch-involving aggregation for classification, both kept lightweight. There is also prior art that explicitly studies Gaussian attention bias in ViTs; LocAtViT’s contribution is its particular way of learning and deploying such a bias for general pretraining and dense transfer.\n\nSummary of the review: The paper proposes a simple, low-cost locality bias for ViTs that improves spatial sensitivity while preserving global context, showing clear gains in frozen-backbone segmentation and self-supervised settings. Its design is elegant, lightweight, and broadly applicable, offering practical value for making ViTs more “segmentation-ready.” However, the evaluation scope is narrow—missing full fine-tuning, detection, and ablations against class-attention or pooling baselines—and more transparency on Gaussian stability would strengthen the case. Overall, this is a solid, well-motivated contribution with practical merit but limited empirical breadth, justifying a rating of 6 for clarity, utility, and moderate originality.", "questions": "1) Aggregation baselines: How does your classifier-side refinement compare to CaiT’s class-attention and token pooling / token labeling? Can you add a small table isolating this component? \n\n2) Generalization under full fine-tuning. Do the segmentation gains hold under end-to-end fine-tuning? If yes, please report one such setting (e.g, ADE20K with a standard decoder).\n\n3) Detection transfer: Have you tried COCO object detection (e.g., with a simple Detr head) to check whether locality-biased features help localization beyond conventional semantic tasks?\n\n4) Scaling + overhead: What is the measured training wall-clock and memory overhead for 224x224 → 512x512 inputs?\n\n5) In-Context Visual Understanding: Does the model gain any benefits on the recent proposed Hummingbird evaluation [2] ( implemented openly by [3])?\n\n6) Failure modes: Did you observe maybe whether there are any cases where the Gaussian bias hurt? \n\n[1] Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., & Jégou, H. (2021). Going deeper with Image Transformers. arXiv [Cs.CV]. http://arxiv.org/abs/2103.17239\n[2] Balažević, I., Steiner, D., Parthasarathy, N., Arandjelović, R., & Hénaff, O. J. (2023). Towards In-context Scene Understanding. arXiv [Cs.CV]. http://arxiv.org/abs/2306.01667 \n[3] https://github.com/vpariza/open-hummingbird-eval", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Simple, plug-in design with low overhead: Aligns with trends showing locality helps ViTs; uses a soft bias so global context remains available. \n2) Clear target and protocol: Frozen-backbone segmentation fairly isolates representational gains; positive results in self-supervised DINO suggest generality beyond supervised pretraining. \n3) Potential impact: Cost seems negligible and code is clean, thus many ViT backbones could adopt the tweak during pretraining to become more “segmentation-ready”.", "weaknesses": "1) Full fine-tuning and detection: Frozen-backbone segmentation is informative but not standard practice; include end-to-end segmentation fine-tuning and at least one object detection benchmark (e.g., COCO with a simple detector) to test if gains persist under typical training.\n\n2) Ablation transparency: Report stability and learned variance scales of the Gaussian (do they collapse or saturate?), and whether the [CLS] treatment or bias masking affects results.\n\n3) Aggregation vs. class-attention / pooling variants. The patch-aware classifier refinement should be positioned and compared to CaiT [1] (class-attention layers) and established token-pooling/labeling approaches; at least show the difference and the relation, or even at least small-scale ablations would help isolate benefits over these alternatives. \n\n[1] Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., & Jégou, H. (2021). Going deeper with Image Transformers. arXiv [Cs.CV]. http://arxiv.org/abs/2103.17239", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)