LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision
Authors: Anthony Fuller
Yousef Yassin
Junfeng Wen
Tarek Ibrahim
Daniel Kyrollos
James R Green
Evan Shelhamer
Venue: NeurIPS 2025 poster ()
Keywords: deep learning
computer vision
adaptive computation
Primary area: deep_learning
TL;DR: We introduce a selector-extractor framework that extracts high-res features without ever seeing full high-res images to save compute.

Abstract: Vision transformers are ever larger, more accurate, and more expensive to compute.
At high resolution, the expense is even more extreme as the number of tokens grows quadratically in the image size. 
We turn to adaptive computation to cope with this cost by learning to predict where to compute.
Our LookWhere method divides the computation between a low-resolution selector and a high-resolution extractor without ever processing the full high-resolution input.
We jointly pretrain the selector and extractor without task supervision by distillation from a self-supervised teacher, in effect learning where and what to compute at the same time.
Unlike prior token reduction methods, which pay to save by pruning already-computed tokens, and prior token selection methods, which 
require complex and expensive per-task optimization, LookWhere economically and accurately selects and extracts transferrable representations of images.
We show that LookWhere excels at sparse recognition on high-resolution inputs (Traffic Signs), maintaining accuracy while reducing FLOPs by 17x and time by 4x, and standard recognition tasks that are global (ImageNet classification) and local (ADE20K segmentation), improving accuracy while reducing time by 1.36x.

Look Where? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision
Anthony Fuller
Yousef Yassin
Junfeng Wen
Daniel G. Kyrollos' Tarek Ibrahim²
James R. Green
* Evan Shelhamer!
Carleton University!
University British Columbia?
Vector Institute
Co-first author
Co-advising author
Abstract
Vision transformers ever larger, accurate, and more expensive to compute. The expense is even more extreme at high resolution as the number of tokens grows quadratically with the image size. We turn to adaptive computation cope with this cost by learning to predict where to compute. Our BookWhert method divides the computation between low-resolution selector and high-resolution extractor without ever processing the full high-resolution input. We jointly pretrain the selector and extractor without task supervision by distillation from self- supervised teacher, in effect, learning where and what to compute simultancously. Unlike prior token reduction methods, which pay to save by pruning already- computed tokens, and prior token selection methods, which require complex and expensive per-task optimization, LookWhere economically and accurately selects and extracts transferrable representations of images. We show that LookWhere excels at sparse recognition on high-resolution inputs (Traffic Signs), maintaining accuracy while reducing FL.OPs by up to 34x and time by 6x. It also excels at standard recognition tasks that are global (ImageNet classification) or local (ADE20K segmentation), improving accuracy while reducing time by 1.36x. See petps://githhe womm//nttofulleee/ookeeeeeeee for the code and weights.
Introduction: A Look through Self-Supervised Eyes
Self-supervised computer vision models can identify informative inputs, as shown in Fig. 1 by what DINOv2 [1 attends and where, and without any knowledge of the task to be done. In short, these methods solve their own learning problems: given one view of visual data, predict another (by reconstructive 2 3 4 6 contrastive learning (1 g 9. When learned scale, they attend to the visually interesting by identifying what is informative for their predictions; importantly, this input may be sparser than the full input, especially at high resolution, offering an opportunity for efficiency,
In this work, we reduce the computation for recognition by predicting this visual interest without fully processing the input. We rely on attention and representation in the eyes of the self-supervised model to learn both where to look, with an efficient selector of locations, and what to see, with expressive extractor of representations. To do so, we propose LookWhere self-supervised adaptive computation that factorizes where and what, for learning and inference, with paired selector-extractor models. Our selector and extractor together represent high-res input given low-res version and @ sparse set of high-res patches selected from the high-res version. Our novel eextctor-e architecture and what-where distillation training effectively approximate deep self-supervised representations with efficient adaptive predictions.
framework for
Look Where applies imagery of any size, but can help most at high resolution. High resolution (with pixel dimensions of >000 on a side) is now common in photography [10], remote sensing
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

high-res imput
teacher attn
selector map
high-res input
teacher attn
selector map
Figure 1: Attention as Supervision. supervieee models (like DINOv2 ( interesting without tasks or labels. We use last step of their deep computation, the final attention map, to start ours; supervising an efficient selector that predicts where to process for adaptive computation.
learn what is visually
[1. autonomous driving 12 13 medical imaging 14 15
and more. These high-resolution inputs but high-res computation is costly.
offer detail and context to improve recognition [16 17
1
This cost has led to active research on adaptive computation for efficiency. These works, and ours, are motivated by the intense computation of vision transformers/ViTs ( 21], and the opportunity in their structure: by first converting an image into tokens, then only applying pair-wise attention and token-witg wise FFN operations, the amount of computation scales directly with the number of tokens (quadratically for pair-wise and linearly for token- Each token eliminated is efficiency gained.
24 25 26 27]. and token selection 28 29 3D BT in its efficiency and simplicity. Unlike token reduction methods, never processes all input tokens, which saves computation (especially on high-res inputs). Unlike prior token selection methods, is efficient and straightforward pretrain and finetune. Unlike both, jointly pretrains its selector-extractor models for transfer across diverse visual recognition tasks.
LookWhere improves on existing methods for token reduction [2]
23
We show that Where delivers state-of-the-ar (SoTA) accuracy/efficency trade-off's in compar- isons and controlled experiments across tasks and resolutions. We measure its efficiency during testing and training, and find it achieves these results with the least per-task training computation. We first demonstrate Look Where in standard benchmarks (ImageNet [32), ADE20K (33)) to prove its effectiveness. Then, we demonstrate LookWhert for high-resolution data on an established special-purpose benchmark (Traffic Signs 34) to emphasize when adaptive computation is needed.
2
Method: Selector-Extractor Computation and What-Whers Distillation
LookWhere accelerates inference by approximating full, deep representations with adaptive com- putation of predictions learned from distillation. The selector predicts where to look, the extractor predicts what to see, and each learns by distillation from the attention and tokens of a self-supervieed teacher (Fig. 2 We call this novel joint learning scheme what-where distillation.
The selector and extractor work in tandem to efficiently extract task sesk-specific representations, given minimal finetuning, based on their self-supervised pretraining of what and where to compute. We first explain inference with the selector and extractor models and their computation (Sec. [.1]. Next, we explain finetuning for task (Sec. 2.2 Last, we explain pretraining (Sec. 2.3). which is needed just once for a visual domain to enable fast task-wise finetuning and deployment.
We review ViT computation and notation as a prerequisite for our adaptive computation of ViTs. This brief summary clarifies the computations that Lhere approximates and accelerates.
Patchification. To process an image dimensions R x It with C' channels, ViTs split the image into an N x N grid of patches R RxC - N Px where P is the resolution of patch. This grid is then flattened into a sequence of N2 patches, each of which is linearly projected to a D-dimensional embedding called a token R. XNxXxxXxC + N The patch tokens Ipat E 12 N3 xD are now
ready for attention and feedforward network (FFN) computation.
Computation. ViTs process the tokens like generic transformer: through a stack architecturally- identical layers composed of pair-wise self-attention and token-wise FFN operations. The computa- tional cost scales with tokens: self-attention scales quadratically and the FFN scales linearly.
Attention. Self-attention applies three learned linear projections to each token in the sequence, producing query, key, and value vectors. The dot product similarity between queries, and keys the "attention" for the pair. The output for each query is the weighted sum of the values given by the

---

input high-res image
sparse high-res image
downscaled
selector map
top
Selector
mapp
attention map
480
global tokens
Cela
pat
self-suerviied
Teacher
Extractor
distillation
Figure 2:
Look Where is trained by distillation from self-supervised teacher to learn where to compute, from the attention map, and what to compute, from the class and patch token representations. Only the teacher sees the high-res input to make its attention and tokens for the losses. The selector predicts where to look from the low-res input. The extractor predicts what see (e.g. a boy in a hat and jersey, coffee cup. .... from the top & high-res patches (and low-res tokens) from the selector.
normalized attention for all pairs. (This is normally repeated across H distinct "heads".) Tokens with lower attention add less to these sums, and raise the potential for efficiency by not computing them.
Global Tokens. Special tokens summarize global information complement the local information in the patch tokens. ViTs augment the patch sequence pat with a learnable "class" token Tela E R", and this technique can extend to multiple "register" tokens Zreg E RxD for G tokens [35].
The full token sequence is (Iels; pat)) E RR(++++N")xD, and its length is dominated by the 22 patch tokens. This is the opportunity for adaptive computation: fewer tokens, less computation.
2.1 Selector-Extractor Inference with Split Resolution Computation
LookWhere factorizes inference into a low-resolution selector for where to compute and a high- resolution extractor for what compute. The selector operates on low-resolution edition of the input to predict the locations of informative low-resolution summaries of the input. Only the top h most informative patches are selected from the high-resolution input. The extractor processes only these selected patches and re-uses the provided low-resolution summaries to predict a complete representation the high-resolution input without ever processing all its pixels.
a selector map-and and also generates
Look admits a variety of architectures for the selector and extractor. Both must be differentiable for end-to-end learning. The selector should be fast in order to efficiently choose and provide the input for the extractor. The extractor should be able harness spatial sparsity accelerate its computation of the selected input. We satisfy these considerations by choosing ViTs for both.
The selector's input and depth reduced for efficient computation. We resize the input smaller Ruw. and truncate the model to its first Law layers; both modifications compound for higher efficiency. The selector produces latents 2low E IR [1++++N2 ) )x where the N2 patch tokens are mapped by FFN Nighh patch importance scores R
+ Nugh Naig yield the selector map Ånigh-
For effective extractor computation, we take high-resolution and low-resolution inputs and maintain higher capacity. The top & locations in Ánigh are selected as patches in the high-resolution input. The extractor tokenizes only the selected patches into pat RD and re-uses the selector's low-resolution global tokens zol computation). Given these high-resolution and low-resolution tokens, the extractor computes its patch token and global token representations following standard ViT inference. We choose the ViT-B architecture for sufficient capacity represent what is in the input. The extractor outputs latents Anigh TR. (1+++))x For full representation of the high-resolution input, the sparse set of patch tokens are then spatially interpolated to the high-resolution grid Snigh E ( XDD)
<fow and .eeg (to replace the extractor's 's own Eats and Zreg without more
Task Task-specific modeling finally outputs predictions ý from Engh depending the details of the task.

---

2.2 Finetuning Look Where for a Task by Supervised Learning
To apply Lhee to a given task, we (1) add a task-specific predictor to the extractor and (2) finetune the extractor-predd stractor-predictor by supervised learning. For simplicity, linear model on the extractor representation suffices to map from its global class token or local patch tokens to the ouiput space. Finetuning Look Where is highly efficient: we only update the predictor-extracto without updating the selector or needing the teacher. Having already pretrained our the self-supervised teacher, we can finetune only what they compute (the representation) for the task while sharing where they compute (the adaptive computation) across tasks. The selector further accelerates finetuning: its adaptive computation continues to reduce high-resolution processing at every step. LookWhere is more efficient during inference and during finetuning.
extractxrr to approximate
2.3 Pretraining by What- Where Distillation of Self-Supervision
Distillation trains a student model to predict a reference output from teacher model [36]. LhokWhere trains its selector and extractor as pair of students to approximate a deep self-supervised model as the teacher. We distill sele-super model for accuracy, efficiency, and transferability. Self-supervised
visual representations are now effective for many tasks 5 7 37 38 39 40). Furthermore, their attention maps can identify informative patches in images (per our Fig. I and prior analyses (()).. While their computation is expensive, our learned approximation need not be due to adaptive computation and the architectural decoupling of the selector-extractor enabled by distillation.
We propose simultaneous distillation of where and what to compute. We distill where from the teacher attention and what from the teacher representation. The teacher fully processes the high- resolution input to provide the attention and representation for learning. By distilling the teacher's 's attention, the selector learns to predict the location of deep, salient information given only shallow, low-resolution input. The extractor learns to predict the full representation of the high-resolution input, given only sparse high-resolution tokens and the selector's low-resolution global tokens, by distilling the teacher representation. We pretrain the selector and extractor jointly and in parallel from partial inputs that are more efficient to compute. Note that the selector and extractor learn to predict the representation of an image they never fully see, which is itself self-supervised task: the masked reconstruction of the representation from only partial inputs and computations. These partial inputs and computations make pretraining more efficient because only the teacher representation is extracted at high resolution.
Teacher. We choose DINOv2 ① as the teacher for its VIT architecture and internet-scale pretraining on diverse images without annotations. We use the variant with G=4 register tokens to reduce attention artifacts (55, as we use its attention to train the selector. The teacher processes inputs at resolution Rnigh =518 with patch size P=14 for grid of Nnigh =37 patches. (This is unaltered from DINOv2.) To distill its attention, we extract the unnormalized attention among its patch tokens at the last layer and then average over queries and heads. This approximates where patch tokens contributed to the deepest teacher representation. To distill its representation, we extract the class and patch tokens at the last layer: 2high E R (1+
We never update the teacher: its parameters are fixed.
Losses and Updates. We train LookWhere jointly with three losses for what to compute, by distillation of the class token and patch tokens, and where to compute, by distillation of attention.
Class Token Distillation: We distill the teacher's class token via mean-squared error (MSE): (f zcls This trains the selector and extractor to learn global representations of hiihh "high):
Late = MSE(MSEEE
the high-res input, given the low-res input and selected high-res patches.
Patch Token Distillation: We distill the teacher's 's patch representations via the mean-squared
eтror (MSE): Lpat = MSE( nigh+ nigh) This trains the selector and extractor to learn local representations of the high-res input, given the low-res input and selected high-res patches.
Attention Distillation: We distill the teacher's 's attention map via the Kullback-Leibler (KL) diver-
gence: Lmap = KL(Anigh
Anig.). This trains the selector to predict where the teacher computes.
Our pretraining loss is their sum L = Aeta Cos + Apat + Cmap- We set Aata: Apot = 1, and Amap 0.1. All losses are optimized end-to-end by the extractor and selector. We update the selector and extractor jointly and in parallel without custom batching, balancing, or tuning.
Initialization, We set the selector and extractor parameters to those of the teacher: DINOv2.

---

high-res imput
selector map
sparse patches
high-res input
selector map
sparse patches
Figure 3:
Looking at Adaptive Computation. We visualize the selector's prediction of where to compute and the extractor's sparse input for what to see. Left: Spatially-sparse traffic sign recognition. Right: Fine-grained bird recognition. The same selector generalizes across these images and tasks.
Table I: ImageNet Classification. LookWhers is more accurate and significantly faster than SoTA adaptive com- putation methods for ViT's. Inference memory is equal for all models. Each cell reports the results for ViT-S/ViT-B: not reported and L = layer.
Table 2: ADE20K Segmentation. We compare three levels of adaptive compu- tation (r, k): LookWhers is more accu- rate & efficient each level.
mloU FLOPs
Speed
Method
Top-1 Acc.
FLOPs
Speed
Avg Tok per T.
% f
G
K im/s
Method
%1
G
K im/s f
DINOv2 11
46.8
46.7
0.9
DINOv2
81.9/84.2
6.2/23.6
6.8/2.2
256
DTEM 25
(r=0.5)
38.9
19.0
0.7
PiToMe
(r:0.925)
79.8/
3.1/
6.6/
128
Look Where (k=512)
40.6
14.6
2.0
DTEM 25
(r=16)
79.4/80.7
2.4/9.2
7.0/3.8
94
DTEM
25
(v-0.44
42.6
22.3
ATC
24
(r=0.9)
79.9/82.0
4.0/15.3
0.4/0.4
169
Look LookWhere
(A-7GG)
43.3
23.1
1.4
LTRP
1
(koppt0.70)
182.8
/18.3
/2.3
147
DTEM 25
(+-0.3)
44.3
25.8
0.5
LookWhere (A-128)
80.3/83.0
3.8/14.8
9.5/3.2
128
Look Whoreeeee (t-1024)
44.6
32.8
1.0
3
Experiments: Accuracy & Efficiency
We evaluate standard visual recognition benchmarks in Sec. 3.1 to confirm that LookWhere is generally accurate and efficient, and on high-resolution benchmarks in Sec. 3.2 to measure adaptive computation when efficiency is key. The standard benchmarks, ImageNet- classification [32) and ADE20K semantic segmentation [33, are ubiquitous in deep learning for vision. The high-resolution benchmarks, Swedish Traffic Signs [34]. Caltech-UCSS UCSD Birds 41. and Billiard Balls (29 evaluate spatially-sparse classification tasks as established by existing work on adaptive computation. We then analyze our choice of oef-suuervviiio and ablate our selector, extractor, and distillation in Sec. 3.4
Fig. 3 shows our selector's 's choice of high-resolution patches (see Appendix A.5 for more examples). Recall that only update extractor for each task, and rely the selector to generalize when predicting where to compute.
Comparisons and Baselines. We choose state-of-the-are comparisons encompassing both token reduction (PiToMe (23], ATC [24), and DTEM (25) and token selection (DPS (29), and IPS (00).
We review their sophisticated adaptive computation methods in Sec. and Appendix A.
Measuring Computation. We measure throughput speed, FLOPs, and peak memory usage for a complete accounting of computational efficiency. We profile computation with the official torch. forch.profffl iler tool, the reference implementations of all methods, and an RTX 4090 GPU.
3.1
Comparison Experiments on Standard Recognition Benchmarks
We apply our standard finetuning of LookWhere (Sec. 2.2 to update the extractor for image classification ImageNet-IK IK [4) and semantic segmentation ADE20K (33). We include DINOY2 as a non-adaptive baseline and as the teacher for Look Where during pretraining.
ImageNet Classification. This standard benchmark of accuracy and efficiency is also used for
adaptive computation with ViT' (Tab. [].. We follow prior work [43 25 30 epochs at 224 px resolution. LookWhert achieves the best accuracy and speed for adaptive computation at ViT-S and VIT-B scales with comparable memory and FLOPss For top speed with the smaller VIT-S model, it delivers 1.36x faster computation than the second fastest method (DTEM).
23 44 and finetune for
5

---

Traffic Sign Recognition
Bird Classification
Billiard Ball Recognition
100
IPS
LW (ours)
100
DINOv2
100
DINOV2
L.W
Top-1 Acc. (%)
90
DINOy2
80
IPS
DTEM
(ours)
LW
80
IPS
O
(ours)
DPS
DPS O
O DPS
80
60
60
70
ATC
40
40
O PiToMe
PeToMe
ATC
ATC
PrToMe
60
20
20
0 0.2 0.4 0.6 0.8
1.0
1.5 2 2.5 Throughput (K images/s)
0.5
0 0.2 0.4 0.6 0.8 1.0
Throughput (K images/s)
Throughput (K images/s)
Figure 4: Nigh-Resolution Recognition Traffic Signs, Birds (CUB), and Billiards. We plot the accuracy and throughput of Look Where, its teacher (DINOv2), and SoTA token reduction and selection methods. Where reaches the Pareto frontier for sparse object recognition (left, right) and for fine- ane-grrined classification (center). To more fully measure computation, points are colored by memory usage (darker is higher/less efficient). See Appendices A.2 and A.3 for more details.
ADE20K Segmentation. This standard benchmark of semantic segmentation / pixel classification (Tab. 2 requires more spatial precision than image classification. Most adaptive computation methods do not tackle this more challenging task, but DTEM 25 does, so we compare in their setting. We vary the amount of computation measure the accuracylefficiency trade-off. LookWhers has better accuracy at 22x higher speed compared to DTEM.
For both tasks, adaptive computation improves efficiency, but with trade-off. Without adaptive computation, DINOv2 is the most accurate but less efficient. With adaptive computation, Look Where achieves the best accuracy and highest efficiency.
3.2 High-Resolution Benchmarks with Spatial Sparsity: Where Efficiency is Key
We evaluate the accuracy, efficiency, and generalization of Look Where on high-resolution inputs. Specifically we choose datasets with spatial sparsity or fine- mreeggrined detail, and finetune while limiting the ratio of high-resolution tokens computed. For these experiments we set the number of patches k for LookWher to process only 10% of the high-resolution input. This sparsity at high-resolution tests our selector's ability to prioritize and improve efficiency by adaptive computation.
In these experiments, control for factors like backbone architecture (DINOv2), training time (30 epochs of finetuning), and input size (square crops following DINOv2 pretraining). We standardize in this way for clarity of comparison; please see the end of this section for more details.
Seatially-Sparee Recognition of Traffic Signs. The Traffic Signs dataset [4 has large images of signs and class labels. The variable and often small size of the signs results in spatial sparsity. We follow the established adaptive computation benchmark 29 30 45) of recognizing speed limit signs as 50, 70, or 80 km/h or sign. We set the input size to be square: 994x994 px.
Look strikes the best accuracy- speed balance across all methods (Fig. left). Look is competitive with the state-of of-the-art: it rivals IPS in accuracy while reducing inference time by 6x and FLOPs by 34x, although IPS reaches 1.1% higher accuracy with more computation. LookWhere exceeds its teacher DINOv2 in efficiency and accuracy, even though the teacher processes the full input. Note that longer training and additional tuning can achieve higher accuracy for all methods (including Look Where, DPS, and IPS) but departs from this controlled setting for fair comparisons.
Fine-Grained Recognition Birds. We evaluate how well LookWhere represents visual detail by experimenting with fine-grained recognition on the Caltech-UCSI Birds (CUB-200-201 1) dataset 4T of 200 bird species in 11,788 images. This common benchmark has been adopted for adaptive computation 29 46). We set the input size to square 518x518 px following the default for DINOv2.
LookWhere reaches the top accuracy, least FLOPs, and fastest speed among adaptive computation methods (Fig. 4 center). Relative to its DINOv2 teacher, which processes all patches, Look Where has nearly equal accuracy To of the patches for more efficiency in speed, memory, and FLOPs.
Sparse Recognition and Inter-patch Reasoning with Billiards. The Billiard Balls dataset 29] is synthetic benchmark designed to test spatial reasoning. Each image contains 4-8 numbered balls, and

---

CUB Training Cost
Traffic Training Cost
Training Cost per Image on Traffic
95
Method
FLOPs
Memory Speed
G 4
GB J im/s f
Top-1 Acc. (%)
DINOV2 1
2698
21.3
6.8
PiToMe 23
1299
8.56
17.0
85
ATC 24
1599
10.7
1.2
80
DTEM 25
OOM
00M
00M
80
DPS 29
142
1.8
259.0
X 10
x10
IPS
50
1912
1.8
79.3
75
75
LookWhere
149
1.8
222.0
100
0
20
40
Cumulative FLOPs (G)
Cumulative FLOPs (G)
Legend: * LookWhere (ours)
DPS
IPS
DTEM
DINOx2
Figure 5: Finetuning Computation. We plot (left) the accuracy and total FLOPs used during training for adaptive computation methods on high-res data, and report (right) the cost per image. LookWhere rivals or bests the accuracy of SoTA token reduction and selection methods at fraction of the cost.
the task is identify the larger number between the left-most and right-most balls. We the input size to be square: 1008 x 1008 px.
LookWhere delivers the best eecuracy-specd trade-off: it nearly maintains its DINOv2 teacher's accuracy with significantly fewer FLOPs, while also rivaling the speed of DPS (Fig. right). DPS achieves higher throughput by incurring 28% drop in accuracy.
Experiment Setup for Controlled Comparisons. We finetune all models for 30 epochs with the AdamW optimizer 477. All methods use the DINOv2 (ViT-Base) backbone, which is the same as the teacher for Look Where. We tune all methods equally. For each method, we sweep learning rates (2e-5, 5, 5, 1e-4). selecting the best result based on final test accuracy. See Appendix A..2 for hyperparameter settings (e.g., token reduction/selection amounts) and exhaustive results.
3.3 Pretraining
We pretrain on ImageNet-1K 422 We train ViT-Base for 200 epochs and ViT-Small for 400 epochs. We augment the data with standard RandAugment [48] transformations. We evaluate the teacher on every input for its attention and representation. We set the selector's 's input size R15x=154 and depth =3 for efficiency; these perform well, and Where works with different choices. At each step, we sample & E [16, 128] (out of N2 high 113369 tokens per image) for transferrable adaptive computation at high resolution. We optimize by AdamW [47] with the same learning rate for the selector and extractor. Please see Appendix A.2 for full details.
Our whete distillation is generally robust to pretraining settings. works well across learning rates, loss weights, selector sizes, and the choice of low-res tokens shared by the selector & extractor.
3.4 Analysis and Ablations
Accounting for Computation. Efficiency depends on input size hardware acceleration. For time, LookWhere is faster at low-res (e.g., 224² px) and higher-res (e.g., >512 px). For FLOPs, LookWhere is often equal or better, but at low-res its selector relatively costly, and the total can exceed the FLOPs of methods without such predictor. For memory, LookWhere is equal or better at low-res and high-res, and can reduce training memory by >5x. While these metrics usefully summarize computation, close examination, the type and hardware compatibility of operations impact efficiency. Look Where relies only on the accelerated GPU computation of standard ViT layers. However, PiToMe, DTEM, and ATC all rely clustering algorithms, which are FLOP efficient yet slower on current hardware. As a result, at high-res LookWhere is more tine/FLLPPmeemon efficient than these SoTA methods with 10x the speed and >5x reduction in FLOPs and memory (Fig. 5).
Adversarial Robustness. To check the sparse processing due to the selector has an effect on robustness, evaluate Look Where and and its teacher against the standard attacks in AutoAttack
7

---

More specifically we evaluate LookWhere and DINOV2 models finetuned on ImageNet- K against attacks on 10K images from ImageNet- Val. Look achieves robust accuracies of 0.75% and 16.6%, while DINOv2 achieves 0. 0.11% and 7. 77% using Toc--boudded attacks with epsilons 253 and 0.5 respectively, Note that we do not claim practical adversarial robustness, and that these are weak attacks relative to the standards for adversarial attack research. No adversarial training was done and these are simply nominal models trained by regular finetuning. Nonetheless, LookWhere is significantly more robust to these attacks at small epsilon than DINOv2, which shows an effect of our architecture and sparse computation. We include this ex exploratory experiment to encourage further work at the intersection of adaptive computation with adversarial robustness, in case robustness gains could compound.
49
I
3.4.1 Where to Look for Supervision of Where to Look?
Our selector drives adaptive computation by predicting where to process high-res patches. How the selector learns is fundamental to our method, so we investigate the choice of supervision for it.
Attention Maps as Oracle Selectors. ViT's compute thousands of 2D attention maps over patch tokens. We test 35 salient choices of teacher attention maps and their aggregations as substitutes for the selector. Specifically, we pretrain LookWhere ViT-S extractors by selecting high-res patches using teacher attention maps instead of selector predictions. These experiments train on ImageNet for 100 epochs. To check generality across tasks, evaluate via &NN for classification over class token representations on ImageNet- [50] (a high-res ImageNet- IK test set) and kNN for segmentation over patch token representations ADE20K.
Results across Layers and Queries. Fig. shows how accuracy depends on which layers and query tokens to choose for the attention maps. For layers, accuracy generally improves with depth. For query tokens, classification is insensitive 1%), but segmentation is more sensitive (>13). We choose the teacher's s last layer attention over patch tokens: it balances the accuracy of both tasks.
Classification
Layer Aggregation
Segmentation
Layer Aggregation
lirst half
last. half
last third
last only
all layers
lirst half
last
last third
last only
Query Aggregution
Query Aggregation
ball
layers
cls
69.3
72.1
72.4
73.4
cls
64.3
63.8
63.4
64.5
reg
69.1
70.2
71.3
73.4
60.6
reg
64.5
64.9
65.0
64.1
65.2
pat
67.0
71.9
72.0
73.1
71.6
put
65.5
67.4
67.3
66.2
67.5
cls+rog
69.3
70.4
72.7
73.7
71.2
cls+reg
64.5
64.7
64.8
63.2
65.2
clsepat
69.6
72.6
73.0
73.7
cls+pat
65.4
65.7
64.9
63.7
tegtpal
68.9
71.3
72.1
74.3
70.9
reg+pat
65.1
66.00
66.1
[6.1
clstregtpat
69.2
72.2
73.0
73.4
71.6
elstregtpat
64.9
65.8
65.2
63.8
65.8
Figure 6: Teacher Attention for Selection. We evaluate teacher attention maps as selector maps to choose distillation targets. For classification (&NN on ImageNet-HR). the deepest attention maps and class queries perform best. For segmentation (kNN on ADE20K), averaging over all layers and patch queries perform best. We choose the last layer's patch token attention maps to balance the tasks.
3.4.2 Ablating Selector- Extractrr Architecture and What- Where Distillation
Design Choices. We ablate four design choices of our architecture and pretraining (Fig. 7. We experiment with 5182 resolution, VIT-S architecture, and k=72 (out of 1, 369 high-res patches). (1) We vary the depth and input size of the selector (a). Larger inputs tend to help more than deeper architecture, so we choose 1542 px resolution a 3 layers. (2) We experiment with training the selector map end-to-end from the extractor's 's distillation losses alongside the attention distillation loss. We try REINFORCE (52) (inspired by PatchDrop (28), and Gumbel Top-k [5T] for this purpose and observe no improvement from either (b). (3) We confirm the effectiveness of passing global tokens from the selector to the extractor for efficiency (c). (4) We measure the trade-off in class token vs. patch token loss weights across classification and segmentation (see Appendix A..4p.
Random Selection Ablation. We ablate our selector by selecting high-res patches at random. On Traffic Signs, accuracy drops by 25% which is only marginally better than choosing the majority class. The drop on CUB is 11%, and on ImageNet, is 2.1% (Fig. (()). The smaller drop on ImageNet could be due to its object-eentric images making where less important.
8

---

(a) Deeper selectors & larger inputs are more ac- curate but slower.
(b) Training the selector map teacher attention alone is best.
(d) Random selection hurts. Varying k during finetuning enables flexible inference at different k.
case
cls. seg.
none (stop grad)
63.0
60.2
pretrained selector, fixed 80
depth res.
cls. seg.
Gumbel Top-K [51
61.4
58.9
ImageNet (top-1 acc.)
252 66.2 60.5
REINFORCE S2
61.8
59.6
3
252
69.7 64.7
(c) Giving the extractor global
random selector, fixed
154
68.8 62.7
tokens from the selector helps.
6
6
154
982
63.0 60.2
69.7 63.0
cls.
seg.
75
12
98²
69.6 61.0
60.7
56.9
pretrained selector, varied &
12
12a
17.3 54.6
cls-only
62.2
58.7
reg-only
62.0
59.8
50
100 150
clstreg
63.0
60.2
# high-res patches (k)
Figure 7: Ablating Architecture and Training "cls." ANN image/class token classification on ImageNet-HR (top-1 acc.) and "seg." KNN local/patch token classification ADE20K (top-1 acc.). Default settings are shared across ablations; final setting is used on downstream experiments. Subfigure (d) finetunes ImageNet and evaluates ImageNet- Val for comparability with Tab. T
high-res input
DINOv2 I
Franca 53
MAE 2
Figure 8: Alternative Teachers. We visualize attention maps from different teachers for where to look. DINOv2 and Franca both identify the visually interesting and serve as effective teachers for LookWhere. By contrast, MAE's attention is occasionally unfocused, making it weaker teacher.
Varying for Flexibility. We randomly vary l by sampling from [1,256] during finetuning on ImageNet. This enables choosing & at inference for faster/more accurate computation (Fig. d)).
Pretraining Using Other Teachers. Thus far our teacher has been DINOv2. However, attending to the visually interesting is not specific to DINOv2: as Figure & demonstrates, other models may also guide where look (see Appendix A.6 for more examples). Here, we experiment with Franca (53) teacher and pretrain selector-extractor using our what-where distillation. Specifically, the selector's target Franca's 's last layer CLS-token attention (chosen because the Franca paper shows it is meaningful). The extractor's target is Franca's 's last layer representation. We evaluate Where-Francal extracted features with &NN on ImageNet-HR: / 71.1% vs. LookWheree werrWeeeeeD-OO21 73.1% 80.3% / 81.8% for & high-res patches 16 / /2 / 128 (both sets of results are at the ViT-Base scale). LooeWhere-aranca thus performs well enough as feature extractor but worse than Look W Where-DINOW the Traffic Signs, Birds, and Billiard Balls datasets and report results in Appendix A.2 In summary, Look LookWhere Where- Franca outperforms prior work but not NookWhereDINOO. To make Look LookWherd fully
achieves 57.0% 68.9 68.9%
We also finetune LookWheree Where-Franca on

---

up-to-date, we train a Look LookWhere results using Franca or DINOv3 teachers can surely be achieved by searching for better selector targets, e.g. our DINOv2 search (Fig. 6), which we leave for future work.
[54] to achieve 43.8% / 73.4% 77.6% accuracy. Better
4 Related Work
Token reduction drops or merges already-computed tokens layer-by-layer to save further computation. Many of these methods cleverly combine similar tokens by soft bipartite matching algorithms 22 |6 26 55 [2],, while others incorporate dynamic filtering [56 57 or differentiable pruning [44]. Nonetheless, such gradual reduction requires processing all tokens at the first layer, nearly all at the second, and on. This is already too much in our case: any processing of all tokens is too expensive at high resolution. By contrast, our selector processes a low-resolution input, and then our extractor receives small set of high-resolution patches. LookWhere never processes the full-resolution input to attain efficiency unobtainable by reduction.
Token selection chooses the inputs for computation, and so can save more, but must learn how to select and do so efficiently. Prior works scale to megapixel images but require complex and expensive per-task optimization [28 29 30) IPS 30) iterates over sets of patches to identify the most salient, which reduces memory but still takes time. DPS [2] and PatchDrop [28] train predictors that process low-resolution versions of the input to score high-resolution patches by discrete optimization. While efficient for inference, their optimization requires sampling, careful tuning of gradient approximation 12 58 and multi-stage training (PatchDrop). LookWhere also processes low and high resolutions for efficient inference, but with simple and efficient training during both pretraining (by distillation) and finetuming (by extractor-only updates).
Self-supervision for Adaptive Computation. Most existing token reduction and selection methods rely on task-specific optimization that does not harness transfer across tasks. LTRP 31 instead pretrains general selector by self-supervision, like LookWhere but lacks a general extractor and scores patches differently. Specifically, LTRP relies on multiple forward passes over perturbed inputs to measure the patch-wise sensitivity of a self-supervised model (MAE (2]) to supervise its selector. This results in expensive pretraining (> 10x ours) without an extractor to transfer. By contrast, LookWhere directly distills the teacher's attention and representation, which only needs one forward pass to compute. Furthermore, Look Where simultancously pretrains the selector and extractor for efficient re-use and transfer across tasks.
Distillation. Many works train the attention maps of students to mimic the attention maps of teachers 59 CD 61 62
and last layer teacher attention maps in particular 63.. However, our what-where distillation trains our student, the selector, to predict single map that summarizes multiple teacher attention maps. This strategy allows our selector to predict a map at higher resolution than its input- e.g., receiving Noww Noow patches and predicting an Nnigh X Nhigh map. Furthermore, existing methods do not distill attention for adaptive computation: we are the first to turn self-supervised attention maps into the supervision of how to prioritize tokens for adaptive computation.
Conclusion
We introduce our selector-extractor architecture and what-where distillation for accurate, efficient, and general adaptive computation on diverse visual recognition tasks. LookWherd is simple but effective: selector predicts 2D map (where to look) for extractor that predicts full-res image representations (what to see). This simplicity enables efficiency on current hardware: Look Where applies standard transformer operations used just right. Together, our selector & extractor represent images they never fully see for efficient finetuning and deployment, especially at high resolutions.
Limitations and Future Work. There is more to learn for selection and there are more dimensions for adaptive computation. LookWhere does not finetune the selector. Although this achieves SoTA results, by pretraining task-general selector, finetuning task-specific selector could do better. Our selector-extractor is only spatial, but could be temporal (Look When) for video or spectral (LookWhich) for remote sensing bands.
10

---

Acknowledgements
AF is primarily supported by an NSERC PGS-D scholarship. ES is supported by Canada CIFAR AT Chair. JW acknowledges the support of NSERC, RGP1N-2024-05337. YY is primarily supported by an Ontario Graduate Scholarship (OGS) and a Vector Scholarship in AL. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. We thank the Google TPU Research Cloud (TRC) for providing the TPUs on which we conducted all pretraining experiments. We thank Eric Tzeng for their helpful review and feedback on the experiments and analysis.
References
[1] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[2] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[3] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. arXiv preprinl 100.::2006.08254.. 2021.
[4] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2022.
[5] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint- embedding predictive architecture. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619-15629, 2023.
[6] Yibing Wei, Abhinav Gupta, and Pedro Morgado. Towards latent masked image modeling for self-supervised visual representation learning. In European Conference Computer Vision, pages 1-17. Springer, 2024.
[7] Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9733 2020.
[8] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong comi-supervisee learners. Advances in neural information processing systems, 13:22243-22255, 2020.
[9] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 15750-15758, 2021.
[10] Qianqian Shen, Yunhan Zhao, Nahyun Kwon, Jeceun Kim, Yanan Li, and Shu Kong. A high- resolution dataset for instance detection with nie object capture. Advances in Neural Information Processing Systems, 36:42064 42076, 2023.
[11] Ritwik Gupta, Shufan Li, Tyler Zhu, Jitendra Malik, Trevor Darrell, and Karttikeya Mangalam. xt: Nested tokenization for larger context in large images. In Forty-firs International Conference on Machine Learning, 2024.
[12] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset. In
CVPR Workshop on the Fulure of Datasets Vision, volume 2, page 1, 2015.
[13] Hongbo Zhao, Lue Fan, Yuntao Chen, Haochen Wang, Xiaojuan Jin, Yixin Zhang, Gaofeng Meng, ZHAO-XIANG ZHANG, et al. Opensatmap: fine- fine-grained high-resolutie satellite dataset for scale map construction. Advances in Neural Information Processing Systems, 37:59216-59235. 2024.
[14) Wisdom Tkezogwo, Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mo- hammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt- Im: One million
11

---

image-text pairs for histopathology. Advances in neural information processing systems, 38837995-38017, 2023.
[15] Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, et al. A whole-slide foundation model for digital pathology from real-world data. Nature, 630(8015):181-1558 2024.
[16] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng. Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence,
43(10))3349-3333 336433444 2020.
[17] Chen Jin, Ryutaro Tanno, Thomy Mertzanidou, Eleftheria Panagiotaki, and Daniel C. Alexander. Learning to downsample for segmentation of ultra-high resolution images. In International Conference on Learning Representations, 2022.
[18] Lianghui Zhu, Bencheng Liao, Qian Zhang. Xinlong Wang. Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In International Conference on Machine Learning, pages 62429 62442. PMLR, 2024.
[19] Penghao Wu and Saining Xie. V?: Guided visual search as a core mechanism in multimodal Ilms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084-13094, 2024.
[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[21] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. International Conference on Machine Learning, pages 7480-7512. PMLR, 2023.
[22] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint (tXxx:221100946111 2022.
[23] Chau Tran, Duy MH Nguyen, Manh-Duy Nguyen, TrungTin Nguyen, Ngan Le, Pengtao Xie, Daniel Sonntag, James Y Zou, Binh Nguyen, and Mathias Niepert. Accelerating transformers with spectrum -rreeerrnnn token merging. Advances in Neural Information Processing Systems, 37:30772-308 10, 2024.
[24] Joakim Bruslund Haurum, Sergio Escalera, Graham W Taylor, and Thomas B Moeslund. Agglomerative token clustering. In European Conference on Computer Vision, pages 200-218. Springer, 2024.
[25] Dong Hoon Lee and Seunghoon Hong. Learning to merge tokens via decoupled embedding for efficient vision transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[26] Minchul Kim, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. Token fusion: Bridging the gap between token pruning and token merging. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1383-1392, 2024.
[27]
Joakim Bruslund Haurum, Sergio Escalera, Graham W. Taylor, and Thomas B. Moeslund. Which tokens to use? investigating token reduction in vision transformers, 2023.
[28] Burak Uzkent and Stefano Ermon. Learning when and where zoom with deep reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12345-12354, 2020.
[29] Jean-Baptiste Cordonnier, Aravindh Mahendran, Alexey Dosovitskiy, Dirk Weissenborn, Jakob Uszkoreit, and Thomas Unterthiner. Differentiable patch selection for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2351-2366 2021.
[30] Benjamin Bergner, Christoph Lippert, and Aravindh Mahendran. Iterative patch selection for high-resolution image recognition. In The Eleventh International Conference on Learning Representations, 2023.
12

---

[31] Yang Luo, Zhineng Chen, Peng Zhou, Zuxuan Wu, Xieping Gao, and Yu-Gan Gang Jiang. Learning to rank patches for unbiased image redundancy reduction. In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition, pages 22831-228880 2024.
[32] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai I.i, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[33] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. International Journal of Computer Vision 127:302-32 321, 2019.
Journaa
[34] Fredrik Larsson and Michael Felsberg. Using fourier descriptors and spatial models for traffic sign recognition. In Scandinavian conference on image analysis, pages 238-249. Springer, 2011.
[35] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024.
[36] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 503.02531, 2015.
[37] Anthony Fuller, Koreen Millard, and James Green. Croma:
Remote sensing representations with contrastive radar-optical masked autoencoders, Advances in Neural Information Processing Systems, 36:5506-55333 5538,, 2023.
[38] Gabriel Tseng, Anthony Fuller, Marlena Reil, Henry Herzog, Patrick Beukema, Favyen Bastani, James R. Green, Evan Shelhamer, Hannah Kerner, and David Rolnick. Galileo: Learning global and local features in pretrained remote sensing models, 2025.
[39] Aiham Taleb, Winfried Loetzsch, Noel Danz, Julius Severin, Thomas Gaertner, Benjamin Bergner, and Christoph Lippert. 3d self supervved methods for medical imaging. Advances in neural information processing systems, 33:18158 18172, 2020.
[40] Xiwen Liang, Yangxin Wu, Jianhua Han, Hang Xu, Chunjing Xu, and Xiaodan Liang. Effective adaptation multi-task co-training for unified autonomous driving. Advances in Neural Information Processing Systems, 35: 19645-9-99658, 2022.
[41] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongic. Caltech-uesd birds-200-200 Technical Report CNS-TR-201 1-00000 California Institute of Technology, 2011.
[42] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115:211-2533 2015.
[43] Lingehen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser- Nam Lim. Adavit: Adaptive vision transformers for efficient image recogmition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12309-12318, 2022.
[44] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamievit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:1 13937-13949, 2021.
[45] Angelos Katharopoulos and François Flcuret. Processing megapixel images with deep attention- sampling models. In International Conference Machine Learning, pages 3282-3291. PMLR.
2019.
[46] Ze Yang, Tiange Luo, Dong Wang, Zhiqiang Hu, Jun Gao, and Liwei Wang. Learning to navigate for fine-grained classification. In Proceedings of the European conference on computer vision (ECCV), pages 420-435, 2018.
[47] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representationss 2019.
[48] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with a reduced search space. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18613-18624. Curran Associates, Inc., 2020.
13

---

[49] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter attacks. In ICML, 2020.
[50] Anthony Fuller, Daniel Kyrollos, Yousef Yassin, and James Green. Lookhere: Vision trans- formers with directed attention generalize and extrapolate. Advances in Neural Information Processing Systems, 37:19683-19739, 2024.
[51] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax.
arXiv preprint 1r11v:1611.01144, 2016.
[52] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8:229-256, 1992.
[53] Shashanka Venkataramanan, Valentinos Pariza, Mohammadreza Salehi, Lukas Knobel, Spyros Gidaris, Elias Ramzi, Andrei Bursuc, and Yuki M Asano. Franca: Nested matryoshka clustering for scalable visual representation learning. arXiv preprint arXiv:2507. 14137, 2025.
[54] Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. Dinov3. arXiv preprint arXiv:2508. 10104, 2025.
[55] Sanghyeok Lee, Joonmyung Choi, and Hyunwoo J Kim. Multi-criteria token fusion with one-step-ahead attention for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15741-15750, 2024.
[56] Benjamin Bergner, Christoph Lippert, and Aravindh Mahendran. Token cropr: Faster vits for quite few tasks. arXiv preprini (ttiv:22222000660 2024.
[57] Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11809-10818, 2022.
[58] Quentin Berthet, Mathieu Blondel, Olivier Teboul, Marco Cuturi, Jean- Phillippp Vert, and Francis Bach. Learning with differentiable pertubed optimizers. Advances in neural information processing systems, 33:9508-9519, 2020.
[59] Yang Zhou, Xu Gao, Zichong Chen, and Hui Huang, Attention distillation: A unified approach to visual characteristics transfer. arXiv preprint 2tti::25022202335. 2025.
[60] Kai Wang, Fei Yang, and Joost van de Weijer. Attention distillation: self-supervised vision transformer students need more guidance. arXiv preprint arXiv:22 0.00944, 2022.
[61] Alex Li, Yuandong Tian, Beidi Chen, Deepak Pathak, and Xinlei Chen. On the surprising effectiveness of attention transfer for vision transformers. Advances in Neural Information Processing Systems, 37:1 13913-113999, 2024.
[62] Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Do- ermann, and Arun Innanje. Ensemble attention distillation for privacy- swweacypreservvng federated learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 15076-15086, 2021.
[63] Wenhui Wang, Furu Wei, Li Dong, Hangho Bao, Nan Yang, and Ming Zhou. Minilm: Deep self self-attention distillation for task aano-agn compression of pre-trained transformers. Advances in neural information processing systems, 33:5776-5788 2020.
[64] Sang Michael Xie and Stefano Ermon. Reparameterizable subset sampling via continuous relaxations. arXiv preprint 1901: 2019.
[65] Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In European conference on computer vision, pages 516-533. Springer, 2022.
[66] Gui-Song Xia, Jingwen Hu, Fan Hu, Baoguang Shi, Xiang Bai, Yanfei Zhong, Liangpei Zhang, and Xiaoqiang Lu. Aid: A benchmark data set for performance evaluation of acrial scene classification. IEEE Transactions on Geoscience and Remote Sensing, 55(7):3965-3981 2017.
[67] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks weakly- supervised classification and localization of common thorax diseases. Proceedings of the IEEE conference computer vision and pattern recognition, pages 2097-2106, 2017.
14

---

[68] Aakash Kaku, Sahana Upadhya, and Narges Razavian. Intermediate layers matter in momentum contrastive self supervised learning. Advances Neural Information Processing Systems, $4:24063-240744 2021.
[69] Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng, and Pranav Rajpurkar. Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature biomedical engineering, 6(12):1399-1406 2022.
[70] Mingxing Tan and Quoe Le. Efficientnetv2: Smaller models and faster training. In International conference machine learning, pages 10096-10106. PMLR, 2021.
15

---

A Appendix
A.1
Adaptive Token Method Descriptions
A.1.1
Token Reduction
2 builds on Bipartite Soft Matching (BSM), which partitions tokens into two sets and merges pairs across both according to key-based similarity. Instead considering all tokens for merging, PiToMe proposes an energy score- defined aggregate similarity of a token with all others distinguish redundant (high-energy) from informative (low-energy) tokens. Only the top 2k high-energy tokens are considered for merging via BSM, preserving the most informative tokens.
Protect Informative Tokens before Merging (PiToMe)
Agglomerative Token Clustering (ATC) [24] replaces BSM with bottom-up, hierarchical clustering approach for token merging. Each token starts as its own cluster, and clusters iteratively merged based on a similarity score defined between clusters. At each step, the two most similar (i.c., redundant) clusters are merged, continuing until target number of tokens remains.
Decoupled Token Embedding for Merging (DTEM) 25 introduces a trainable module that decouples token merging from representation learning. It projects intermediate token embeddings to extract perging-speciffe features for estimating pairwise token similarities. Unlike traditional hard assignment grouping schemes, DTEM uses differentiable top-k operator [64] to softly group tokens, followed by BSM for merging. Notably, the soft merging retains all tokens during training, with discretization applied only during inference for actual token reduction.
A.1.2 Patch Selection
formulates selection as Reinforcement Learning problem train a patch selection policy. Given a down-sampled input, the policy returns Berouuli distributions over high-resolution patches from which samples are drawn. The policy is initially trained using the REINFORCE [52] algorithm with a fixed pretrained classifier, receiving rewards that encourage both classification accuracy and sparse sampling. A joint finetuning stage then updates the policy and classifier together for improved performance.
PatchDrop 28
Differentiable Patch Selection (DPS)
casts patch selection ranking problem, where scoring network assigns relevance scores to high-resolution patches based on a low-resolution input. differentiable top-k module, based the perturbed maximum method (58) selects the top scoring patches for downstream processing by feature and patch-aggregation networks. The entire pipeline is trained end-to-end under supervision.
29
Iterative Patch Selection (IPS) (30 performs ranking-based patch selection directly on the high-res input through sequential glimpses. A scoring module maintains a buffer of the top k most salient patches, which is updated auto-regressively by evaluating patches at time in no-gradient mode. The selected patches then re-embedded with gradients enabled and passed to cross-attention transformer-base classifier, enabling end-to-end training. This same cross-attention module is shared with the scoring network guide saliency prediction.
Learning Rank Patches (LTRP) 31 trains a patch ranking model via sulf-superviioon Using pretrained masked autoencoder, the method randomly masks patches and estimates each visible patch's importance by measuring the change image reconstruction when the patch removed- yielding pseudo-relevance score. A ranking model is then trained to predict the resulting ranks such that, following pretraining, they may used to retain the top most salient patches for downstream tasks.
A.2 Experimental Details
A.2.1 Pretraining Hyperparameters
We pretrain our ViT-S for 400 epochs and ViT-B for 200 epochs on ImageNet- [32 using the AdamW optimizer 47 We use a batch size of 1024, image size of 518x518 px, weight decay of 0.05, learning rate warmup of 10%, peak learning rate of 2e-4 with cosine decay to 15-05, and RandAugment 48 data augmentation.
16

---

A.2.2 ImageNet Hyperparameters
We finetune for 30 epochs on ImageNet- I 32 using the AdamW optimizer [47. We use a batch size of 128, image size of x24 224 px, weight decay of 0.02, learning rate warmup of 10%, peak learning rate of 2e-5 with cosine decay to le- and 3-Augment [5] data augmentation.
A.2.3 ADE20K Hyperparameters
We finetune for 160K steps on ADE20K (3 using the Adam optimizer [47]. We use batch size of 16, image size of 518x518 px. weight decay of 0.02, learning rate warmup of 10%. peak learning rate of 2e-5 with cosine decay to 1e-5, and data augmentation: random crop scale [0.7, 1.0), horizontal flip, random choice (gray scale, solarize, gaussian blur), J, and color jitter (0.3). We compute all patch-token representations by interpolating sparse representations with the same interpolation method used during pretraining. We use simple linear head that maps patch-token representations to pixel-wise class predictions.
A.2.4 Traffic Signs Recognition
Traffic Signs Setup. We use the annotated subset of the Swedish Traffic Signs dataset (34), containing 747 training and 684 test images. All images are resized and cropped to square dimensions of 994x994 px for compatibility with DINOv2 ViT backbones pretrained on square images. IPS and DPS use 980x980 px due to compatibility with their ViT patch extraction methods. We finetune all models for 30 epochs using the AdamW optimizer [77]. We sweep learning rates over (2e-5, 5e5,, Se Se-5, 5, 1e-4) with a batch size of 16, following 30 and report the best final test accuracy achieved by each method. We apply a weight decay of 0.02, a 10% learning rate warmup, with cosine decay to 10-05, and data augmentation including random cropping (scale range [0.8, 1.0) and standard RandAugment transformations
[48]. DTEM and PiToMe merge tokens after every layer, ATC merges after layers 4, 7, and 10 (following (56)).
Tabular Traffic Signs Results. Table reports full experimental results complement Figure 4
Table 3: Traffic Signs results. LookWhere remains competitive with SoTA token selection methods, being more efficient than IPS and rivaling DPS for speed, memory, and FLOPs. LookWhere-R uses random masking instead of the selector, but still receives the selector's low-res global tokens for context. DTEM runs out of memory (00M).
Method
Top-1 Acc.
Memory (GB) +
FLOPs (G) 4
Speed (im/s) t
% f
Test
Train
Test
Train
Test
Train
DINOv2
94.0
3.1
21.3
900
2698
24.1
6,8
PiToMe 23
(r=0.9)
66.5
3.1
8.6
440
1300
51.4
17.0
ATC 24 (--0.7)
69.3
2.7
10.7
537
1599
1.3
1.2
DTEM 251
00M
0OM
OOM
0OM
0OM
00M
00M
DPS (--0.1)
29
83.6
0.8
1.8
49
142
942.4
259.0
IPS (r=0.1)
30
96.3
0.8
1.8
1819
1912
94.1
79.3
Look Where-R (k-504)
70.6
0.8
1.8
53
149
609.8
222.0
LookWhers (k-504)
94.6/90.2
0.8
1.8
53
149
609.8
222.0
Look (f=1008)
95.2/91.4
0.9
2.5
110
320
288.1
99.1
Where accuracies are reported for DINOv2/Franca teachers. Compute is the same.
A.2.5 Fine-grained Bird Classification
Fine-grained Bird Classification Setup. We finetune for 30 epochs on the Caltech-UCSD Birds 1) dataset [1 of 200 bird species in 11,788 images. We sweep learning rates over (2c-5, 5, 5c 8c-5, 5, 1c-4) using a batch size of 64. Images are resized to 518x518 px by default (588 x 588 px for IPS and DPS for 98x98 pixel patches fed to the ViT feature extractor). We use a weight decay of 0.02, a 10% learning rate warmup, and cosine decay to a minimum learning rate of le-05. Data augmentation includes random cropping (scale range [0.8, 1.0) and standard Randenggent transformations [88.. DTEM and PiToMe merge tokens after every layer, ATC merges after layers 4, 7, and 10 (following (56).
17

---

Tabular Bird Results. Table reports full experimental results to complement Figure
Table 4: Bird results. Wher achieves the Pareto frontier with the fastest speed, and lowest memory and FLOPs among SoTA token reduction and selection methods while retaining competitive accuracy. LookWheree uses random masking instead of the selector, but still receives the selector's s low-res global tokens for context.
Method
Top-1 Acc.
Memory (GB) +
FLOPs (G) 4
Speed (im/s)
% f
Eval
Train
Eval
Train
Eval
Train
DINOv2
90.5
0.9
3.1
152
455
209.4 .
67.4
PiToMe
(v=0.9)
34.3
0.9
1.9
81
239
391.1
128.4
ATC
24
(r=0.7)
34.4
0.9
2.1
95
285
21.2
18.2
DTEM 25
90.7
1.0
5.5
84
464
261.8
52.3
DPS (r=0.1)
20
85.4
19
57
2479.0
697.7
DPS (r-0.2)
88.3
0.7
1.8
33
98
1442.2
432.1
IPS (==0.1)
80
89.0
0.8
1.8
653
690
261.8
213.8
IPS (==0.2)
90.2
0.8
1.8
639
704
241.8
179.4
Look BookeRe (I=136)
78.2
0.8
16
2322.4
865.6
Lhore (A-136)
89.0/86.6
16
2322.4
865.6
LookWhere (X-273)f
90.1/88..1 1
0.8
1.7
29
79
1333.6
479.0
LookWhere (K-G84)1
90.4/88.2
0.8
1.9
71
206
515.8
173.6
LookWhere accuracies are reported for DINOv2/Franca teachers. Compute is the same.
Pretrained with wider sampled patch range (256-512) to better support larger K at finetuning.
A.2.6 Billiard Ball Inter-patch Reasoning
Billiard Ball Setup. We finetune for 30 epochs the Billiard Ball dataset (29 of 8k training images, and 10k for test. Each image contains 4-8 balls, numbered through 9, with the task of identifying the larger number among the left-most and right-most balls. We resize images to 1008 1008 px for compatibility with DINO's patch size; DPS and IPS use 1072 x 1072 for compatibility with their ViT patch extraction methods. We train most methods with batch size of 64 following [29], and 32 for IPS / DPS due to memory constraints. The remaining setup follows identically from Traffic Signs (see A.2.4.
Tabular Billiard Ball Results. Table 5 reports full experimental results to complement Figure
Table 5: Billiard Ball results. Look attains SoTA accuracy on the Billiard Balls task, matching DINO's performance while receiving 1 of the patches significantly improve efficiency; achieving the least memory and FLOP consumption and speeds comparable to DPS.
10
uses random
masking instead of the selector, but still receives the selector's low-res global tokens for context.
Method
Top-1 1 Acc.
Memory (GB) |
FLOPs (G) |
Speed (im/s) 1
% t
Test
Train
Test
Train
Test
Train
DINOv2
97.6
2.9
OOM
939
OOM
22.9
6.4
PiToMe
(--07))
23.3
3.2
8.9
458
1353
49.1
15.8
ATC
24
23.6
3.2
11.2
559
1667
1.2
1.0
DTEM 25
0OM
OOM
0OM
OOM
00M
00M
00M
DPS (r=0.1)
29
69.1
0.8
1.8
59
170
780.0
215.5
IPS (rw0.1)
90.4
1.8
1810
1921
77.9
65.2
Look LookWhere-A (k-518)
27.8
0.8
1.8
55
154
583.1
209.6
Look Where (A=518)
97.3/97.1
0.8
1.8
55
154
583.1
209.6
Look Where (k=1037)
97.5/97.0
0.9
2.5
114
331
285.4
98.5
Look (A=25022)
97.4/97.0
1.4
7.0
350
1041
77.0
25.8
Loook Where accuracies are reported DINOv2/Franca teachers. Compute same.
18

---

A.3 Additional Finetuning Experiments
We further assess the generalization of LookWhere's selector by evaluating on more diverse, medical and remote sensing datasets.
A.3.1 Remote Sensing Classification
AID Sctup. We finetune for 30 epochs on the Aerial Image Dataset (AID) [66] of 10k images for aerial scene classification across 30 classes. We use a 50/50 train/test split, batch size of 64, and resize images from 600 px 602 x 602 px match DINOv2's patch size. The remaining setup follows identically from Traffic Signs (see A.2.4). Results are reported in Table 6
Table 6: AID results. Look Where rivals DINOv2's performance on remote sensing classification using a fraction of the patches. Accuracy improves with increased computation.
Methad
Top-1 Accuracy (%) t
DINOv2 !
98.7
Look Where (k-185)
97.4
WhokWhere (k=370)
98.0
LookWhere (845/25)
98.3
A.3.2 Medical Chest X-ray Classification
NIH Chest X-ray Setup. We the NIH Chest X-ray [67 dataset of 12k medical images (90k train / 22k test) for multi-label clinical diagnosis across 14 labels. Images are resized from 1024x 1024 px x022 X122x100 1022 px and finetuning for 10 epochs using a batch size of 64. The remaining setup follows identically from Traffic Signs (see A.2.4). Following prior work [68] 69,, we report AUC-ROC scores T
Table 7: NIH Chest X-ray results. On multi-label medical classification, Lhore remains competitive with DINOv2 despite processing only fraction of the patches. Performance scales with the allocated computation.
Method
AUC-ROC ?
DINOv2 !I
82.2
LookWhers (I=533)
78.7
LookWhere (k=1066)
79.6
LookWhere (K-2666)
81.1
A.4 Detailed Ablation Setup and Results
We ablate several design choices, including: (1) teacher attention targets for selector maps, (2) distillation loss weights, (3) selector map training schemes, (4) low-resolution conditioning tokens for the extractor, and (5) the input size and depth of the selector. Unless stated otherwise, all ablations are conducted over 400 epochs of pretraining on ImageNet-IK 1K [32] at resolution of 518x518 pixels using the ViT-S architecture. During pretraining, we sample k [16, 128] uniformly at random, and report results at test time for fixed values k E {16, 72, 128} for general classification
and segmentation downstream tasks. Unless stated otherwise, ablations use our default settings, which are highlighted in blue.
To ablate LookWhere efficiently,
simple kNN-based evaluation. For classification, process all ImageNet-HR (50) images with Lher and store the resulting class-token representa- tions sels We then compute top-1 accuracy using leave-one-out kNN approach; specifically, for "high"
each ImageNet-HRR image, retrieve the 3 nearest neighbors, excluding the query itself and assign the most frequent class among them as the prediction. For segmentation, we process all ADE20K spot 33 images with LookWhert and store 10 patch-token representations "high per sample. We then compute top-1 accuracy using the validation set as queries and the training set as keys. We fetch 20 patch tokens for each query and assign the most common class the predicted label.
19

---

A.4.1 Teacher Attention Targets for Selection
To distill the teacher's attention, we extract the unnormalized attention among its patch tokens at select layers and query tokens and then average over layers, queries and heads (Fig. (9. This approximates where the selected patches contributed to the teacher representation. In this section, we explore different combinations of layers and query tokens used for attention aggregation.
last
only
last
third
last half
all
first
half
Tels Treg
Epat
Figure 9: Attention Aggregation. We aggregate attention by averaging unnormalized attention scores originating from the cls, reg, and patch queries across different layers (all, first 3, last last 2, and last only). Score maps from each query type weighted equally when several are used (i.e., we average the maps among tokens each type first, before averaging across types).
Classification
Layer Aggregation
Segmentation
Layer Aggregation
first half
last half
last only
all layers
first half
last half
last third
last only
all
Query Aggregation
third
Query Aggregation
layers
cls
48.5
52.9
54.8
57.6
52.1
cls
52.3
51.8
51.6
51.8
52.1
47.8
50.1
56.5
49.2
reg
53.4
53.2
52.7
53.7
pat
46.9
49.8
50.3
54.1
48.8
pat
53.6
54.2
53.6
53.3
54.1
cls+reg
48.5
49.0
53.8
57.2
50.9
cls+reg
53.4
52.2
52.5
51.7
53.0
cls+pat
48.9
52.2
37.7
50.8
cls+pat
52.5
52.7
52.6
52.3
52.7
pat+pat
47.5
49.4
51.0
56.3
48.6
reg+pat
53.6
53.1
53.6
53.2 33.2
53.9
cls+reg+ppa
47.9
51.5
52.8
57.3
51.3
cls+reg+pat
53.8
53.1
52.
52.0
53.4
Figure 10: Teacher Attention for Selection (k=16). We evaluate teacher attention maps selector maps to choose distillation targets when testing using k= selected patches.
20

---

Classification
Layer Aggregation
Segmentation
Layer Aggregation
first half
last half
last
all
first half
last half
last
last
all
Query Aggregation
third
only
layers
Query Aggregation
third
only
layers
cls
65.0
68.6
69.2
70.3
67.8
cls
61.4
60.8
60.4
59.4
60.8
64.1
66.2
70.2
65.5
rag
61.5
61.6
pat
62.4
69.0
68.9
70.4
67.5
pat
62.1
64.2
63.7
62.7
64.2
clstreg
63.4
66.4
68.8
70.4
67.1
cls+rreg
61.2
61.0
61.0
60.1
62.2
cls+pat
65.0
68.9
70.9
68.3
cls+pat
62.0
62.0
61.0
60.9
62.0
ret+pat 64.1
67.5
68.4
70.6
66.6
reg+pat
61.7
62.4
62.7
61.1
62.6
cls+reg+pat
64.9
68.0
69.2
70.6
67.6
clstreg+pat
61.5
62.2
61.4
60.5
62.4
Figure 11: Teacher Attention for Selection (k=72). We evaluate teacher attention maps as selector maps to choose distillation targets when testing using k= selected patches.
Classification
Layer Aggregation
Segmentation
Layer Aggregation
first half
last half
last third
last only
all layers
first half
last half
last third
last
Query Aggregation
Query Aggregation
only
layers
cls
69.3
72.1
72.4
73.4
71.9
cls
64.3
63.8
63.4
63.4
64.5
reg
69.1
70.2
71.3
73.4
69.6
reg
64.5
64.9
65.0
64.1
65.2
pat
67.0
71.9
73.1
71.6
pat
65.5
67.4
67.3
66.2
67.5
cls+reg
69.3
70.4
72.7
73.7
71.2
cls+treg
64.5
64.7
64.88
63.2
65.2
cls+pat
69.6
72.6
73.0
73.7
72.0
cls+pat
65.4
65.7
64.9
63.7
65.7
rat 68.9
71.3
72.1
74.3
70.9
ret+pat
65.1
66.0
66.
64.6
66.1
cls+reg+pat
69.2
72.2
73.0
73.4
71.6
cls+reg+pat
64.9
65.8
65.2
63.8
65.8
Figure 12: Teacher Attention for Selection (k=128). We evaluate teacher attention maps as selector maps to choose distillation targets when testing using k= 128 selected patches.
21

---

A.4.2 Classification Accuracy vs. Distillation Losses
We ablate distillation weights and feature-interpolation parameters with 100 epochs of pretraining. During pretraining (and when using ADE20K downstream), interpolate the sparse/visible high-res patch-token representations to compute full 2D grid. We compute the predicted dense patch-token representations the weighted sum of the N nearest neighbors 2D space, among the sparse/visible patch tokens. The weights in the weighted sum are the Euclidean distances raised to the pow*h power, then normalized s.t. all weights sum 1. We experiment with N {5, 163 and pow {1, 2). and summarize our results in Tables through
Table 8: Distillation Losses (pow=1, neighbors). We evaluate different combinations of distillation loss weights for pretraining the selector-extractor (Nets, Apor) and selector map (Amap). We consider spatial patch interpolation using Euclidean distance, with selected neighbours. cls. (by k)
Acts
Apat
Amap
Emap
seg. (by k)
16
128
72
17.7
61.7
66.4
51.6
59.6
62.6
0.1
KL
46.8
61.5
66.7
51.0
59.5
62.3
0.1
1
KL
48.1
62.5
67.4
48.2
57.3
60.8
0.1
0.1
KL
48.9
63.0
67.4
47.8
56.7
60.7
0.1
KI.
44.9
60.3
66.1
51.6
59.8
63.4
KI.
45.0
60.2
65.3
51.3
59.7
62.9
0.1
0.1
1
KI.
46.1
62.2
66.9
51.0
59.5
63.1
0.1
0.1
0.1
KL
48.7
62.0
66.7
50.8
59.8
62.5
MSE
50.2
61.8
66.2
51.2
58.5
62.0
Table 9: Distillation Losses (pow=1, neighbors). We evaluate different combinations of distillation loss weights for pretraining the selector-extractor (Noles )pat) and selector map (Ampp) ) We consider spatial patch interpolation using Euclidean distance, with selected neighbours. (by k)
Acta
Apat
Amap
Lmap
cls. (by k)
16
72
128
16
72
128
1
KL
48.9
62.2
66.8
49.4
58,1
61.5
1
0.1
KL
48,1
61.9
66.9
48,9
57.4
61.0
1
0.1
1
KI.
47.4
62.4
67.0
47.4
56.7
60.2
1
0.1
0.1
KI.
48.3
62.2
66.2
47.5
55.9
60.2
0.1
1
KI.
44.6
60.4
65.5
49.8
58.8
62.0
0.1
1
0.1
KL
45.8
60.0
66.0
49.2
58.5
61.6
0.1
0.1
1
KL
46.5
62.6
67.0
49.0
58.2
62.1
0.1
0.1
0.1
KL
47.8
62.5
67.1
49.6
58.4
61.6
Table 10: Distillation Losses (pow=2, neighbors). We evaluate different combinations of distillation loss weights for pretraining the selector-extracte (Aets, Apat) and selector map (Amap). We consider spatial patch interpolation using squared Euclidean distance, with selected neighbours. seg. (by k)
Adta
Aput
Amap
Lmap
cls. (by k)
72
128
16
72
128
KI.
46.7
61.9
67.2
50.7
59.0
62.8
1
0.1
KI.
47.0
61.5
65.8
50.5
59.6
62.5
1
0.1
1
KI.
49.3
62.6
67.0
48.3
56.7
60.6
0.1
0.1
KL.
48.0
62.6
67.2
48.3
56.7
60.6
0.1
1
KL
44.8
60.6
66.0
48.0
48.0
0.1
1
0.1
KL
43.9
60.3
65.1
51.3
59.8
63.2
0.1
0.1
1
KL
45.7
62.3
67.0
50.6
59.0
62.5
0.1
0.1
0.1
KL
47.6
62.3
66.3
50.7
59.1
62.6
22

---

Table Distillation Losses (pow=2, 16 neighbors). We evaluate different combinations of distil- lation loss weights for pretraining the selector-extrace (Aets Apet) and selector map (Amap). We consider spatial patch interpolation using squared Euclidean distance, with 16 selected neighbours.
Adts Apat
Amap
Lmap
cls. (by k)
seg. (by k)
16
72
128
16
72
128
KL
48.3
62.1
66.9
50.2
58.2
61.7
0.1
KL
47.6
61.5
66.4
49.8
57.5
61.6
0.1
1
KL
48.7
62.6
67.5
47.7
56.7
60.1
0.1
0.1
KL
49.1
62.6
66.9
17.6
6.1
59.8
KL
41.1
61.0
65.3
59.5
62.6
0.1
0.1
KL
44.2
60.8
65.7
50.4
58.8
61.7
0.1
0.1
1
KL
45.9
62.3
67.2
49.8
58.4
62.2
0.1
0.1
0.1
KL
47.8
62.5
67.1
50.3
58.7
62.2
A.4.3 Selector Map Training
In addition distilling the teacher's attention map, explore differentiable learning of the selector map to better leverage the extractor's 's signal for optimal patch selection. To enable differentiable selection, experiment with (1) Gumbel Top-K K [51) and (2) REINFORCE [52].
both approaches, selector map treated as logits defining a softmax probability distribution over patches; we sample the map using the Gumbel Top-K trick. We make sampling differentiable through the straight-throug! estimator (i.e., differentiable Gumbel Top-K) and sweep over learning rates and sampling temperatures. With REINFORCE, we model pretraining problem: low-resolution images define the state, actions are binary masks selecting k patches (as in PatchDrop (28). and the reward is derived from the loss. Specifically, we compute an advantage as the difference in distillation loss between a greedy top-k policy and the sampled action. We backpropogate through the extractor for both samples. We explore various learning rates, weightings for distillation and policy gradient losses, and the use of an entropy bonus to promote exploration.
one-step (contextual) bandit
We observe that training fails to converge without map distillation loss (which effectively acts analogous to a KI. regularizing term with respect to the teacher's "policy"). Table reports the best results across all configuration consistently showing that teacher distillation alone performs best. Gumbel Top-K pretrains for 400 epochs, while REINFORCE pretrains for 300 epochs (to roughly balance compute across all methods since REINFORCE backpropogates through both the policy sample and greedy baseline).
Table 12: Selector Map Training Schemes. We consider different training schemes for the selector map in addition to distillation. Stop grad refers to no additional scheme, while Gumbel Top-K and REINFORCE leverage sampling to estimate a gradient from the extractor's signal. We find that distillation, alone, is optimal.
cls. (by k)
seg. (by k)
16
72
128
16
72
128
none (stop grad)
50.9
63.0
66.5
51.1
60.2
62.9
Gumbel Top-K [1
44.5
61.4
66.2
49.1
58.9
62.5
REINFORCE 52
47.9
61.8
66.0
50.7
59.6
62.8
A.4.4 Selector Layer Initialization
explored several selector initialization schemes as alternatives to truncating the teacher's first Ltow layers. The results, summarized in Table show that Look is generally robust to initialization, especially at larger values. At smaller values, we find performance improves slightly when initializing with deeper layers.
23

---

Table 13: Selector Initialization. We experiment with different layer initialization schemes for Look Where's s selector. We find that performance is relatively unchanged for larger K, while smaller benefits slightly from initializing with deeper layers. cls. (by k)
case
seg. (by k)
16
72
128
256
16
72
128
256
0-1-2
38.52
55.90
62.76
68.68
48.10
57.5
60.33
63.81
3-4-5
39.88
56.68
62.20
68.36
47.78
57
60.45
64.31
6-7-8
39.90
56.66
62.42
68.44
48.06
57.77
60.24
64.24
9-10-11
40.54
57.32
63.02
68.34
48.49
57.68
60.61
64.32
A.4.5
Alternative Selector Architectures
LookWher generalizes to architectures beyond ViTs, although the extractor and selector require different considerations. Here, we demonstrate this flexibility by using a CNN for the selector architecture.
The extractor relies on partitioning of the input, typically into tokens, that enables: selection and processing of sparse inputs, and ii) distillation using partial inputs to learn semantic representations. Input patchification satisfies these needs. making the extractor compatible with various downstream architectures. For instance, methods like DPS and IPS also use patch-based inputs with CNN components.
The selector has a simpler role: approximating the teacher's s saliency map. We hypothesize that any modern image processing architecture can perform this task effectively. To validate this, pretrained two models for 100 epochs: one with our default DINOv2-initializeed ViT selector and another with CNN selector initialized from EfficientNetV2 770. As shown in Table [4 the kNN performance is comparable. After finetuning on ImageNet, the EfficientNet-basse selector achieved top-1 accuracy of 82.3%. Overall, these results confirm that Looke generalizes well beyond ViT-only designs.
Table 14: Alternative CNN-based Selectors. We experiment with using CNN-based Selectors, initialized with EfficientNet22, for LookWhere. Results are comparable using a DINOv2-based selector.
case
cls. (by
seg. (by k)
16
72
128
16
72
128
DINOv2 1
66.2
74.9
78.4
52.3
61.8
64.6
EfficientNet22
71.8
81.2
82.5
49.3
58.8
62.3
A.4.6 Extractor Conditioning on Low-Resolution Global Tokens
To provide the extractor with additional global image context, experiment with initializing its class token Iela and/or register tokens Treg by the selector's 's final representations for each: z and "low respectively. Table 15 summarizes results; find that adding global tokens generally helps, although performance interestingly degrades for classification when selecting k=128 patches specifically.
Table 15: Extractor Conditioning, We experiment with feeding the extractor different combinations of global low-resolution tokens resulting from the selector's processing. We find that giving both the cls token and all register generally helps performance.
case
cls. (by k)
seg. (by k))
16
72
128
72
128
none
28.2
60.7
68.4
39.7
56.9
cls-only
48.8
62.2
67.2
49.8
58.7
61.8
reg-only
49.4
62.0
67.3
51.0
59.8
62.4
cls+reg
50.9
63.0
66.5
51.1
60.2
62.9
24

---

A.4.7 Selector Input Size and Depth
We experiment with varying the selector's size by trading off network depth against input resolution while keeping computational cost relatively low. Our results are summarized in Table [6
Table 16: Selector Sizes. Both shallow networks with large inputs and deeper networks with smaller inputs result in suboptimal performance. The best results are achieved by balancing depth and input resolution.
depth res.
(by k))
seg. (by k)
16
72
128
16
72
128
252
43.7
66.2 70.8 49.0
60.5 64.4
3
252
65.8
72.1 73.7
56.1
64.7
67.2
154
58.7 68.8 71.2 53.9
62.7 65.1
6
1542
65.0
69.7 72.1 54.4
63.0 65.4
6
98*
50.9
63.0 66.5
51.1 60.2 62.9
12
984
65.8
69.6
70.9
53.0
61.0 63.3
12
422
25.5
47.3
57.1
43.8
54.6
58.1
A.5 Selector Maps
Our selector generalizes to diverse downstream images and tasks. particular, we visualize general examples for the recognition of birds (Fig. 13). traffic signs (Fig. 14). and billiard balls (Fig. 15).
25

---

A.5.1
Bird Recognition (CUB)
high-res ipput
selector map
sparse patches
high-res input
selector map
sparse patches
Figure 13: Adaptive Computation for Recognizing Birds. We visualize the selector's prediction of where compute and the extractor's 's sparse input for what to see. We do not finetune the selector; each pair shows different generalization scenarios, specifically for bird recognition on CUB 41
26

---

A.5.2
Traffic Signs
high-res input
selector map
sparse patches
high-res input
selector map
sparse patches
Figure 14: Adaptive Computation for Traffic Sign Recognition. We visualize the selector's prediction of where to compute and the extractor's s sparse input for what to see. We do not finetune the selector; each pair shows different generalization scenarios, specifically for recognizing traffic signs on the Swedish Traffic Signs dataset (34
27

---

A.5.3
Billiard Balls
high-res imput
selector map
sparse patches
high-res input
selector map
sparse patches.
0
Figure 15: Adaptive Computation for Billiard Ball Recognition. We visualize the selector's prediction of where to compute and the extractor's s sparse input for what to see. We do not finetune the selector; each pair shows different generalization scenarios, specifically for recognizing and reasoning across several billiard ball configurations 29
28

---

A.6 Comparing Selector Maps from Different Teachers
LookWhere generalizes other teachers beyond DINOv2. In particular, we visualize general examples for the recognition of birds (Fig. T6), traffic signs (Fig. [7p, and billiard balls (Fig. [8] for three different teachers: DINOv2 M, Franca [53], and MAE [2] (we upsample MAE's maps to
match the resolution of DINOv2/Franca for visual consistency).
29

---

A.6.1
Bird Recognition (CUB)
high-res input
DINOv2 1
Franca 533
MAE 2
Figure 16: Alternative Teachers for Recognizing Birds. We visualize attention maps from different
teachers for where to look on Bird Recognition tasks.
30

---

A.6.2
Traffic Signs
high-res input
DINOv2 1
Franca 533
MAE 2
Figure 17: Alternative Teachers for Traffic Signs. We visualize attention maps from different teachers for where to look on Traffic Signs Recognition.
31

---

A.6.3
Billiard Balls
high-res input
DINOv2 1
Franca 533
MAE 2
Figure 18: Alternative Teachers for Billiard Ball Recognition. We visualize attention maps from different teachers for where look for recognizing and reasoning across several billiard ball configurations [29].
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, contributions in the abstract and introduction accurately reflect the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We briefly discuss limitations the very end of our paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
33

---

Justification:
We do no not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The most important experimental details are shared in the paper, but not all. In supplement, will include all information needed to reproduce results.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
34

---

Answer: [Yes]
Justification:
We provide code in PyTorch and/or JAX.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
www.SSubbisiooooiiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The most important experimental details are shared in the paper, but not all. In our supplement, will include all information needed reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not have the resources to run experiments multiple times and include eтror bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
35

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We will include these details the supplement.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:/ppp....aeo..oooo/o/oo//aaa
Answer: [Yes]
Justification:
We have read the Code of Ethics and feel our research conforms to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We will discuss both potential positive societal impacts and negative societal impacts in the supplement.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
36

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not believe our paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite the original creators of assets to the best of our knowledge.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
37

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
38

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
We only use LLMs for tikz help.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
39


Reviews:
[{"review_id": "mhoRtViwF0", "reviewer": "Reviewer_J83A", "summary": "The authors propose LookWhere, a method for efficient processing of high-resolution images with vision transformers. LookWhere consists of two models, the selector and the extractor. The selector processes the input image in low resolution to decide which image patches (tokens) should be processed in higher resolution. Then, the extractor receives as input only these patches from the high resolution image, and outputs the final representation that can be used for downstream tasks. The main source of efficiency is the described selective processing, since only a small part of the input image is processed in high resolution.\n\nThe selector and extractor models are trained together end-to-end, by distilling an existing self-supervised teacher, i.e., DINOv2. The authors use 3 simple loss terms, (1) MSE between the class token of the extractor and that of the teacher, (2) MSE between the patch tokens of the extractor and those of the teacher, (3) KL divergence between the output of the selector (it is an attention map over the patch sequence of the full resolution image), and a combination of attention maps from the teacher.\n\nThe authors test their model on image classification and semantic segmentation, by finetuning the extractor on the respective datasets along with a task-specific head; the selector is expected to generalize after distillation, and is not finetuned for different tasks. Baselines consist of other adaptive computation methods, and DINOv2 for reference, since it is the teacher model. The authors first test the accuracy and efficiency of their method on ImageNet-1K and ADE20K, which are not of particularly high resolution, but are well established benchmarks. Then they test on Traffic Signs dataset with images of resolution $994 \\times 994$ px, where the regions of interest (signs) are relatively small. The last dataset is CUB Birds, which again is not of particularly high resolution, but requires fine-grained classification, so it can test a different capability of the model. Overall, LookWhere compares favorably to the adaptive baselines, while DINOv2 usually has the best performance, but is also considerably slower. Finally, the authors conduct a number of ablations to justify their design choices.", "questions": "My main question is why the authors didn’t explore the compute-performance tradeoff that LookWhere could provide? Especially since DINOv2 achieved better performance in all but the Traffic Signs dataset, indicating that there is value in processing more image patches.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- Originality:\n1. The idea of selective processing by decoupling “where” and “what”, along with using a low resolution image for guidance, are not novel.\n2. However, the proposed method has simple and straightforward losses, which I assume is good for training stability, and takes advantage of modern powerful models through distillation.\n- Quality:\n1. I can see both strengths and weaknesses related to the experiments. On the strengths side, I think the authors use appropriate baselines, and they test on both classification and segmentation, which is not a common task for adaptive models. In addition, as I described in the previous Section, different experiments test for different capabilities, for example ImageNet-1K contributes as a popular benchmark, while Traffic Signs can test the attention policy of the selector.\n2. Another strength of the paper is the ability of the selector to generalize across datasets without finetuning.\n3. On the weaknesses side, I found surprising that the authors did not provide compute-performance trade-off curves by varying $k$, which determines the number of high resolution patches processed by the extractor. In general, I think it is important for an adaptive model to be able to trade off compute for performance and vice versa, so it can adjust to scarcity of resources or increased performance demands. The authors vary $k$ only in the segmentation experiments in Table 2, and in a brief ablation experiment in Fig. 7 (d), however, I think it would be appropriate to test different values of $k$ in all experiments. For example, in the fine-grained classification experiments in Fig. 4, we see that LookWhere strikes a good balance between throughput and accuracy, but it's neither the fastest nor the most accurate, so, I think it would strengthen the results if there was a throughput-accuracy trade-off curve to show that LookWhere can beat all baselines for comparable throughput (increase $k$ to beat more accurate baselines and decrease $k$ to beat faster baselines at similar throughput). I see that in Tables 3 and 4 in the Appendix, the authors provide LookWhere results for different $k$, but LookWhere never reaches the accuracy of the top performing baselines, while it remains considerably faster, so, I think higher values of $k$ could be used; for example, in Table 4 in the Appendix, LookWhere has about 1% lower accuracy compared to DTEM, but its eval speed is about $\\times 5$.\n5. I think the authors could have used more diverse high-resolution datasets, e.g., satellite or medical images.\n6. I think the parameters of all models, including the baselines, should be provided in at least one of the Tables, e.g., Table 1, for easy comparison.\n- Clarity:\n1. The paper is very well written, and easy to follow. The authors provide a lot of implementation details, and a lot of ablation studies, which can be very useful for people who are interested to build on this work in the future.\n2. I only found Fig. 5 a bit confusing, what do the authors mean by cumulative FLOPs?\n3. Minor typos, e.g., ln 263: “resolution a 3 layers” should be “and”.\n- Significance:\n1. Overall, it is a very well written paper, with carefully designed experiments, however, given that the originality of the ideas is limited, I think the results should have been stronger.", "was_revised": true, "final_justification": "The authors provided additional experiments during the rebuttal period, showing the efficacy of LookWhere in a wider range of datasets. i.e., AID, NIH Chest X-ray dataset, and Billiard Ball Dataset. In addition, they trained LookWhere with a wider range of $k$ values, addressing my main concern regarding the ability of LookWhere to trade off compute for performance and vice versa. LookWhere was able to perform comparably to its Teacher in a number of datasets for less compute, demonstrating its ability to utilize different $k$ when appropriately trained, and its potential to further improve in light of a stronger Teacher model. For these reasons, I increased my score from $3$ to $5$.", "rebuttal": "# Response to Reviewer J83A\n\nWe thank the reviewer for their thoughtful feedback, particularly their recognition of our extensive ablations, carefully chosen baselines, and clear presentation. We reply to clarify the originality of our work and provide additional ablations as requested, focusing on LookWhere’s compute-performance tradeoff and its generalization to more diverse, high-resolution datasets.\n\n\\\n**Originality of LookWhere:** \\\nWhile we agree that decoupling “where” and “what” in selective processing is not novel, we believe LookWhere introduces a key innovation through its scalable and straightforward joint pretraining of the selector and extractor via distillation; simultaneously learning where to attend from attention and what to extract from representation, unlike all prior work. As the reviewer notes, it is precisely this distillation that enables LookWhere’s simple yet effective losses, promoting stable and efficient training while leveraging the capabilities of powerful modern models.\n\nIn the related work section, we explain LookWhere’s relationship to prior token selection methods such as IPS, DPS, and PatchDrop, which also adopt a decoupled approach. However, we emphasize that these methods rely on complex, task-specific, or multi-stage training pipelines. In contrast, LookWhere achieves superior efficiency and performance through a single-pass distillation process that pretrains both the selector and extractor jointly. By leveraging self-supervised representations, we find the selector generalizes across tasks, requiring only the extractor to be finetuned. We appreciate the reviewer’s feedback and will further underline our novelty by this dual distillation in the revised related work section.\n\n\\\n**Generalization to More Domains:** \\\nWe thank the reviewer for suggesting an evaluation on more diverse datasets and fully agree that doing so strengthens our generalization claims. During the rebuttal period, we extended our evaluation to three additional datasets. Following our established methodology, we finetuned both LookWhere and DINOv2 on these new downstream tasks. The results continue to demonstrate LookWhere’s strong performance, as summarized below:\n\n- **Aerial Image Dataset (AID) [R1]** of 10K images for aerial scene classification across 30 classes. Images were resized from 600px to 602px to match DINOv2’s patch size. We used a 50/50 train/test split and finetuned for 30 epochs. We measure top-1 accuracies of 97.4%, 98.0%, and 98.3% using K values of 184 (10%), 369 (20%), and 924 (50%), respectively, which continues to be competitive with DINOv2’s 98.7% when processing _all_ patches.\n\n- **NIH Chest X-ray Dataset [R2]** of 112K medical images (90K train / 22K test) for multi-label clinical diagnosis across 14 labels. Images were resized from 1024px to 1022px and finetuned for 10 epochs. Following prior work [R3, R4], we measure AUC-ROC scores of 78.7%, 79.6%, and 81.1% using K values of 532 (10%), 1065 (20%), and 2664 (50%), respectively. DINOv2 achieves a comparable AUC-ROC of 82.2% while processing _all_ patches.\n\n- **Billiard Ball Dataset [R5]** of 18K images (8K train / 10K test) used to evaluate interpatch reasoning. Each image contains 4-8 balls numbered 1-9; the task is to identify the larger number between the left-most and right-most balls. Images were resized from 1000px to 1008px and finetuned for 30 epochs. We measure top-1 accuracies of 97.3%, 97.5%, and 97.4% using K values of 518 (10%), 1036 (20%), and 2592 (50%), respectively. LookWhere achieves nearly the same accuracy as DINOv2 (97.6%) while only processing _a fifth_ of the patches.\n\n\\\n**LookWhere’s Compute Performance Tradeoff:** \\\nWe agree that evaluating LookWhere’s ability to trade off compute for performance is important. As the reviewer notes, we already explored this in two settings: segmentation (Table 2) and ImageNet classification (Figure 7d). During this rebuttal, we vary K more extensively on AID, NIH Chest X-ray, Billiard Balls, and CUB (see below) and will plot accuracy-throughput tradeoff curves, similar to Figure 7d, for these experiments in our paper, should it be accepted. \n\nWe initially observed no performance gains using our original LookWhere model during finetuning on CUB across a wider K range, beyond the ratio of 0.2 already included in our paper. This lack of improvement is likely due to the model’s pretraining, which sampled 16 - 128 tokens (out of 1369). To explore this further, we pretrained another model during the rebuttal period using a wider range of token samples (256 - 512) to better support larger K during finetuning.\n\nWe then finetuned this model on CUB using our standard approach and measured top-1 accuracies of 88.7%, 90.1% and 90.4% using K values of 136 (10%), 273 (20%) and 684 (50%), respectively. This updated model matches DINOv2’s performance of 90.4% using only half the patches, outperforming all baselines except DTEM, and coming within 0.6% of DTEM’s accuracy while reducing FLOPs by 4.42x. These results further support LookWhere’s ability to reduce computation cost with minimal performance loss. We also note that LookWhere’s performance is closely tied to that of the teacher model, and could improve further with more powerful teachers.\n\n\\\n**Parameters of all Models:** \\\nWe agree that including parameter counts for all models, including baselines, facilitates fair comparison. \nDINOv2, DTEM, PiToME, and ATC are all plain ViT-B architectures with 86 million parameters. In addition to a plain ViT-B, LTRP adds a 22-million-parameter ViT-Small ranking model using 108 million parameters in total. In addition to a plain ViT-B, LookWhere adds 3/12 of a ViT-B along with an FFN to realize its selector, using 109.8 million parameters in total. However, we measured total inference memory usage across all models to be approximately equal.\n\n\\\n**Cumulative FLOPs:** \\\nIn Figure 5, “cumulative FLOPs” refers to the total FLOPs consumed during training up to a given point. For example, if a forward and backward pass over a batch requires X FLOPs and there are Y batches per epoch, the cumulative FLOPs after the first epoch would be Y * X, after the second epoch 2 * Y * X, and so on. We will provide further clarification in Figure 5’s caption.\n\n\\\n**Minor typos:** \\\nThank you for pointing this out. We will correct the typo in the final version.\n\n[R1] G.-S. Xia et al., “AID: A benchmark data set for performance evaluation of aerial scene classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 7, pp. 3965–3981, 2017.\n\n[R2] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2097–2106.\n\n[R3] A. Kaku, S. Upadhya, and N. Razavian, “Intermediate layers matter in momentum contrastive self supervised learning,” Advances in Neural Information Processing Systems, vol. 34, pp. 24063–24074, 2021.\n\n[R4] E. Tiu, E. Talius, P. Patel, C. P. Langlotz, A. Y. Ng, and P. Rajpurkar, “Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning,” Nature biomedical engineering, vol. 6, no. 12, pp. 1399–1406, 2022.\n\n[R5] J.-B. Cordonnier, A. Mahendran, A. Dosovitskiy, D. Weissenborn, J. Uszkoreit, and T. Unterthiner, “Differentiable patch selection for image recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2351–2360."}, {"review_id": "rvoJSVOa9U", "reviewer": "Reviewer_HzUw", "summary": "This paper proposed a self-supervised learning based distillation framework comprised of a selector for processing low-resolution images, a FFN to ‘upscale’ attention map, an extractor for tokenization and a task-specific predictor for downstream fine-tuning. And then authors conducted experiments on ImageNet-1K, Caltech-UCSD Birds and Traffic Signs dataset for image classification and ADE20K for semantic segmentation to evaluate proposed method’s effectiveness and efficiency.", "questions": "1. In this paper, authors initialized the selector and extractor with the **first** $L_{low}$ layer from the teacher model DINOv2 and provided an ablation of layer depth in `Fig.7.a`. What about the layer location’s effect on distillation(e.g., 3 layers of 1-3, 4-6, 7-9, or 9-12)? There are some previous research revealing that bottom layer gains more semantic information than top layers, which may bring more gains for distillation.\n2. Could this proposed distillation method also be applied on other architectures beyond ViT? After the process of selector, top-k tokens may loss its 2D structure, which may bring a limitation of the extractor. While the architecture of selector that process full low-resolution images has more choices beyond ViT. Could author make a try of more architectures?", "limitations": "See `Weakness` and `Question` part.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Well-motivated: The authors comprehensively introduced the motivation of each section or design.\n2. Clear writing: This paper offers a clear and logical structure that is easy to follow.\n\nWeaknesses:\n1. About `Fig.2`’s demonstration: According to the `Sec.2.1`, this figure may loss some key details about FFN for high-resolution selector map, which also makes the $L_{map}$ a little confusing.\n2. According to the ablation in `Fig.7.c`, the register token brings large performance gain on both cls and seg task. Would the effectiveness of distillation still exist when other models (e.g. contrastive learning based CLIP or reconstruction based MAE) without register token are used as teacher model?\n3. This paper provided many metrics(e.g. FLOPS, memory, IPS) for efficiency comparison. Could author additionally provide the whole ViT-S or -B based model parameters information (including FFN for high-resolution selector map) for a more comprehensive with compared methods in `Tab.1-2`, `Fig.4-5`.\n\nI'd be more than willing to revise my score based on the author's further responses.", "was_revised": true, "final_justification": "Authors provided detailed responses and additional experiment results on alternative selector initialization schemes, which have addressed most of my previous concerns.", "rebuttal": "# Response to Reviewer HzUw\n\nWe thank the reviewer for their thoughtful feedback, including their recognition of our clear writing and well-motivated approach. We reply to address concerns regarding effectiveness without registers, in addition to new experimental results investigating layer initialization and LookWhere’s generalization to other architectures.\n\n\\\n**Does the effectiveness of distillation exist when the teacher lacks registers?:** \\\nWe thank the reviewer for this thoughtful question and agree that registers are indeed beneficial for distillation. However, based on experiments conducted during the rebuttal period, we confirm that registers are not essential for LookWhere’s effectiveness. Registers offer two key benefits: (i) they pass global context from the low-resolution image via the selector, and (ii) they typically produce cleaner attention maps with fewer artifacts, which helps the selector learn better maps. However, these benefits are not unique to registers. Global context can also be delivered through mechanisms such as a CLS token or pooling patches, and teachers without registers can still produce semantic attention maps.\n\nTo verify LookWhere’s effectiveness in the absence of registers, we conducted new experiments during the rebuttal period using Franca [R1] – a new visual foundation model – with a ViT-B backbone as the teacher, keeping all other hyperparameters fixed. We note that Franca was released after our submission, and demonstrates that LookWhere can be easily applied to and benefit from future models. Franca uses a CLS token for global context and does not employ registers, yet we observed that distillation remains effective. Following Section 3.1, our LookWhere-Franca model with k = 128 (9.3% of tokens) achieves 71.2% accuracy on ImageNet-HR and 61.4% on ADE20K. Furthermore, our finetuned LookWhere-Franca model achieves 81.0% on ImageNet which compares favorably to DINOv2 (Table 1). We also finetuned this LookWhere-Franca model on downstream tasks, obtaining the following results:\n- **Traffic:** 90.2%, 91.4% and 93.4% using K values of 518 (10%), 1036 (20%), and 2592 (50%).\n- **Birds:** 86.6%, 88.1% and 88.2% using K values of 136 (10%), 273 (20%) and 684 (50%).\n- **Billiards:** 97.1%, 97.0%, and 97.0% using K values of 518 (10%), 1036 (20%), and 2592 (50%) (see \"Generalization to More Domains\" in our response to reviewer  J83A) \n\nThese results compare favorably to those obtained with DINOv2, and we will reference them alongside our other finetuning results in section 3.2 while also including them near Tables 3 and 4. We reiterate our thanks to the reviewer for this insight that broadens the generality of LookWhere; our experiments confirm its effectiveness, regardless of whether the teacher has registers.\n\n\\\n**Selector and Extractor Layer Initialization:** \\\nWe want to clarify that only the selector is initialized with the first $L_{\\text{low}}$ layers from the teacher; the extractor retains all layers. While a similar strategy could be applied to the extractor, our primary goal is to reduce the selector’s depth for improved efficiency. We retain the extractor’s depth to maintain performance while ensuring efficiency by processing fewer tokens.\n\nFollowing the reviewer’s suggestion, we explored alternative selector initialization schemes during the rebuttal period. Specifically, we trained four ViT-S models for 100 epochs each using a learning rate of 0.0002, and initializing the selector with different sets of teacher layers: {[0, 1, 2], [3, 4, 5], [6, 7, 8], [9, 10, 11]}. We summarize these results in Tables T1 and T2 below and generally find LookWhere to be robust to layer initialization – especially with larger K values. At smaller values of K, LookWhere benefits slightly from initializing with deeper layers, aligning with the reviewer’s insight. We thank the reviewer for this suggestion and will include these results alongside our other ablations in Figure 7.\n\n**Table T1. Layer Initialization ImageNet kNN Accuracy (%) for Varying Number of Patches (K)**\n\n| Layer Initialization | K=16   | 72   | 128  | 256  |\n|----------------------|------|------|------|------|\n| 0-1-2                | 38.52| 55.90| 62.76| 68.68|\n| 3-4-5                | 39.88| 56.68| 62.20| 68.36|\n| 6-7-8                | 39.90| 56.66| 62.42| 68.44|\n| 9-10-11              | 40.54| 57.32| 63.02| 68.34|\n\n**Table T2. Layer Initialization ADE kNN Accuracy (%) for Varying Number of Patches (K)**\n\n| Layer Initialization |  K=16   | 72   | 128  | 256  |\n|----------------------|------|------|------|------|\n| 0-1-2                | 48.10| 57.88| 60.33| 63.81|\n| 3-4-5                | 47.78| 57.58| 60.45| 64.31|\n| 6-7-8                | 48.06| 57.77| 60.24| 64.24|\n| 9-10-11              | 48.49| 57.68| 60.61| 64.32|\n\n\\\n**Could the proposed distillation method be applied beyond ViTs?:**\n\nYes, LookWhere can generalize beyond ViTs, though different considerations are needed for the extractor and selector architectures.\n\nThe extractor relies on a partitioning of the input, typically into tokens, that enables: i) selection and processing of sparse inputs, and ii) distillation using partial inputs to learn semantic representations. These two requirements are satisfied by patchification of the high-resolution input, regardless of how the selected patches are later processed. For example, both DPS and IPS use CNN-based selectors and explore different extraction methods, such as pooling or shallow transformers. While alternative extractor architectures are viable, we focus on ViTs due to their effectiveness at seamlessly processing sparse inputs.\n\nRegarding 2D structure, we appreciate the reviewer’s concern that selecting the top-k tokens may disrupt spatial coherence. However, we emphasize that the selector is explicitly trained to select the most informative tokens, meaning any dropped tokens are less critical to the image representation. During pretraining, LookWhere’s patch and class-token distillation losses help ensure that full representation of the high-resolution input can be approximated using only sparse high-resolution tokens which is aided by 2D positional embeddings, also helping ensure structure is maintained.\n\nThe selector plays a simpler role of approximating the teacher’s saliency map, which we realize through attention. Hence, any architecture capable of image processing is suitable here. To validate this, we conducted additional experiments during the rebuttal period, training two ViT-B models for 100 epochs each: one using a DINOv2-initialized selector and the other using a CNN initialized EfficientNetV2 and summarize kNN results on ImageNet-HR classification and ADE20K segmentation below (akin to Tables 1 and 2 from our paper) using K values of 16 (1.2%), 72 (5.3%), and 128 (9.3%):\n- **LookWhere w/ EfficientNetV2 Selector**\n    - **ImageNet-HR:** 71.84%, 81.18% and 82.48%\n    - **ADE20K:** 49.34%, 58.80% and 62.25%\n- **LookWhere w/ DINOv2 Selector**\n    - **ImageNet-HR:** 66.16%, 74.92% and 78.42%\n    - **ADE20K:** 49.34%, 58.80% and 62.25%\n\nWe also finetuned LookWhere with an EfficientNetV2 Selector for 10 epochs on ImageNet and obtained top-1 accuracy of 82.3%. Overall, we confirm EfficientNet comparable performance to DINOv2 as a selector across all these experiments and validate the ability of LookWhere to generalize beyond ViTs. We will add a new subsection referencing these results, and discussing alternative architectures, to our experiments should our paper be accepted.\n\n\\\n**Figure 2’s demonstration:** \\\nWe appreciate the reviewer’s concern regarding the clarity of our central figure. We intentionally abstract away the internals of both the selector (including the FFN) and the extractor to simplify the depiction of LookWhere’s computational flow. We aim to avoid tying the diagram to specific implementation details. While our paper focuses on a ViT-based instantiation of LookWhere, the framework is not limited to ViTs. For example, as demonstrated above, the selector can also be implemented using a CNN. We will revise our figure, clarifying that the output of the selector is a _high-resolution_ selector map to ease interpretation of the losses.\n\n\\\n**ViT-S or -B based model parameters information:** \\\nWe agree that including parameter counts for all models, including baselines, facilitates fair comparison. \nDINOv2, DTEM, PiToME, and ATC are all plain ViT-B architectures with 86 million parameters. In addition to a plain ViT-B, LTRP adds a 22-million-parameter ViT-Small ranking model using 108 million parameters in total. In addition to a plain ViT-B, LookWhere adds 3/12 of a ViT-B along with an FFN to realize its selector, using 109.8 million parameters. However, we measured total inference memory usage across all models to be approximately equal.\n\n[R1] S. Venkataramanan et al., “Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning,” arXiv preprint arXiv:2507.14137, 2025."}, {"review_id": "BhgjO02RYP", "reviewer": "Reviewer_HJW7", "summary": "The paper proposes LookWhere, a self-supervised method for vision transformers aiming to reduce computational cost. The method splits inference into two components: a low-resolution selector determining \"where to look\" by predicting regions of visual interest, and a high-resolution extractor processing only the selected regions to determine \"what to see.\" The selector and extractor pair is jointly pretrained via distillation from a self-supervised teacher model (DINOv2), enabling efficient representation learning. Experiments demonstrate LookWhere's efficacy in reducing computational overhead while maintaining or improving performance on different benchmarks.", "questions": "1. How dependent is LookWhere's performance on the choice of the self-supervised teacher? Please discuss the robustness of LookWhere when trained using alternative teachers.\n2. What prevents fine-tuning the selector for task-specific scenarios?\n3. Could the selective mechanism make LookWhere more sensitive or robust to adversarial examples?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper uses an interesting split-resolution selector-extractor strategy, which effectively utilizes adaptive computation.\n2. The paper presents an interesting joint distillation framework (what-where) from a self-supervised teacher that enhances the computational efficiency.\n3. Strong experimental results showing efficiency gains.\n\nWeaknesses:\n\n1. Relying on a single self-supervised teacher (DINOv2) for distillation might limit the generalizability.\n2. The exploration of selector fine-tuning for task-specific optimization is limited.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed all my concerns and questions. After reading the rebuttal and other reviewers' comments, I decide to keep my score and recommend accept.", "rebuttal": "# Response to Reviewer HJW7\n\nWe thank the reviewer for their thoughtful, constructive and encouraging feedback. In particular, we appreciate their recognition of our strong experimental results and are grateful for their suggestions for improving our paper, which we address in this rebuttal.\n\n\\\n**Robustness of LookWhere Using Alternative Teachers:** \\\nLookWhere relies on DINOv2 – and any other teacher model – in two key ways: to distill where to look via its saliency maps, and what to see via its representations. While many modern models provide high-quality patch representations suitable for training the extractor, our choice of DINOv2 is primarily motivated by its highly semantic and clean attention maps, which allow the selector to identify visually salient regions without supervision from labels or tasks.\n\nImportantly, any teacher model with semantic attention maps could be used effectively within our framework. To verify LookWhere’s effectiveness using other teachers, we conducted new experiments during the rebuttal period using Franca [R1] – a new visual foundation model – with a ViT-B backbone as the teacher, keeping all other hyperparameters fixed. We note that Franca was released after our submission, and demonstrates that LookWhere can be easily applied to and benefit from future models. Franca uses a CLS token for global context and does not employ registers, yet we observed that distillation remains effective. Following Section 3.1, our LookWhere-Franca model with k = 128 (9.3% of tokens) achieves 71.2% accuracy on ImageNet-HR and 61.4% on ADE20K. Furthermore, our finetuned LookWhere-Franca model achieves 81.0% on ImageNet which compares favorably to DINOv2 (Table 1). We also finetuned this LookWhere-Franca model on downstream tasks, obtaining the following results:\n- **Traffic:** 90.2%, 91.4% and 93.4% using K values of 518 (10%), 1036 (20%), and 2592 (50%).\n- **Birds:** 86.6%, 88.1% and 88.2% using K values of 136 (10%), 273 (20%) and 684 (50%).\n- **Billiards:** 97.1%, 97.0%, and 97.0% using K values of 518 (10%), 1036 (20%), and 2592 (50%) (see “Generalization to More Domains” in our response to reviewer J83A) \n\nThese results compare favorably to those obtained with DINOv2, and we will reference them alongside our other finetuning results in section 3.2 while also including them near Tables 3 and 4. We reiterate our thanks to the reviewer for this insight that broadens the generality of LookWhere; our experiments confirm its effectiveness, regardless of the teacher used.\n\n\\\n**What prevents finetuning the selector for task-specific scenarios?:** \\\nWhile nothing fundamentally prevents finetuning the selector for task-specific scenarios, at least in theory, its _discrete_ patch selection makes standard gradient descent inapplicable without modifying the (supervised) training scheme or overall model architecture. One possible direction is adapting a differentiable top-k selection method, such as DPS, or using a reinforcement learning approach as in PatchDrop, to enable selector finetuning after pretraining.\n\nOur results suggest the pretrained selector already generalizes well across tasks – evidenced by the selector maps for Traffic Sign Recognition and CUB (Figures 12 and 13), and strong performance on the newly added Billiard Ball task (see our response to reviewer J83A). In such cases, especially with a suitable teacher, we expect limited gains from selector finetuning. Nevertheless, we agree with the reviewer and acknowledge that finetuning the selector could still improve performance and enhance LookWhere’s flexibility, as well as other adaptive selection schemes such as LTRP, which is why we explicitly mention this limitation in our paper. Given our positive results regarding selector generalization, we hope our findings encourage future research into selector specialization through finetuning.\n\n\\\n**Could the selective mechanism make LookWhere more sensitive or robust to adversarial examples?:** \\\nWe thank the reviewer for this insightful question and investigated it further during the rebuttal period by evaluating our LookWhere model under a PGD attack. We find that LookWhere significantly increases robustness to adversarial examples compared to DINOv2, often achieving an additional 20+% accuracy. We summarize these results below and will include them in a new subsection focused on robustness (given an additional room from a 10th page).\n\n**PGD attacks with epsilon=1/255**\n- **LookWhere-Base**\n    - **3 steps:** 32.1% acc. on ImageNet-Val\n    - **5 steps:** 27.5% acc. on ImageNet-Val\n    - **10 steps:** 24.8% acc. on ImageNet-Val\n- **DINOv2-Base**\n    - **3 steps:** 8.9% acc. on ImageNet-Val\n    - **5 steps:** 6.5% acc. on ImageNet-Val\n    - **10 steps:** 5.3% acc. on ImageNet-Val\n\n[R1] S. Venkataramanan et al., “Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning,” arXiv preprint arXiv:2507.14137, 2025."}, {"review_id": "Zsge5cEAtN", "reviewer": "Reviewer_7H5B", "summary": "This paper proposes \"LookWhere,\" an adaptive computation method to address the prohibitive computational cost of Vision Transformers, especially on high-resolution images. The method divides computation into a low-cost 'selector' that predicts 'where to look' from a low-resolution input, and an 'extractor' that processes 'what to see' using only the selected high-resolution patches. For training, it employs a novel 'what-where distillation' technique that simultaneously distills the attention (where) and representations (what) from a self-supervised model, enabling task-agnostic generalization and efficient fine-tuning.\nExperiments demonstrate that the method achieves a superior accuracy-efficiency trade-off compared to existing methods across diverse recognition benchmarks. Notably, on the high-resolution traffic sign recognition task, it shows remarkable results, reducing FLOPs by up to 34x and inference time by 6x while maintaining accuracy.", "questions": "- How much information is lost in the global tokens extracted from a low-resolution input compared to their original high-resolution counterparts? Can it be expected that this loss is small enough to avoid significant performance degradation on tasks that rely heavily on global object information?\n- Is the choice of DINOv2 as the teacher model critical? To what extent does the proposed distillation framework depend on the specific properties of DINOv2?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe idea of separating the process into an inexpensive selector that decides \"where to look\" and an expensive extractor that processes \"what to see\" is intuitive and compelling. By not needing to process the entire input image at high resolution from the start, it can achieve fundamentally higher computational efficiency than existing token pruning methods.\n\n**Major Weaknesses**\n\nThe core of the proposed method lies in reusing global tokens (CLS/register) extracted from a computationally cheap, low-resolution input. While Figure 7(c) demonstrates the effectiveness of this approach, a crucial analysis is absent: a direct comparison with the original tokens from a high-resolution input to quantify how much global contextual information is retained in this low-resolution version. \nTo convincingly justify the validity of the efficiency gains, it is essential to quantitatively verify that this cost reduction does not incur a significant loss of important contextual information.\n\n**Minor Weaknesses**\n\n- The paper selects DINOv2 as the teacher model. A discussion on why DINOv2 is particularly suitable for this task, and the potential implications of using other self-supervised models, would strengthen the paper's claims of generality. Without this context, it is difficult for the reader to gauge to what extent the framework's effectiveness is tied to the specific properties of DINOv2, as opposed to being a more universally applicable distillation strategy.\n\n- The number of selected patches, k, is a critical hyperparameter that determines the trade-off between accuracy and efficiency. While the paper evaluates several fixed values for k, it does not deeply discuss a mechanism for dynamically determining k based on the task or input image. Although a training method that allows for flexible selection of k at inference is mentioned, the criterion for this selection is left to the user.", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 7H5B\n\nWe thank the reviewer for their detailed and thoughtful comments, and appreciate their recognition of LookWhere’s intuitive design and its ability to achieve strong performance while maintaining computational efficiency. The reviewer raises three key concerns, which we address below.\n\n\\\n**How much information is lost in the global tokens extracted from a low-resolution input compared to their high-resolution counterparts?:** \\\nThis is a valid concern that we are happy to address. First, we clarify that LookWhere not only reuses global tokens from the low-resolution input but also relies heavily on its selector to determine where to look – i.e., to identify the most relevant high-resolution patches. We believe this selective mechanism is also central to LookWhere’s effectiveness.\n\nAs emphasized in the introduction, many vision tasks involve spatial redundancy, where only small portions of the image are essential for accurate prediction. LookWhere exploits this sparsity by selecting only those high-resolution patches that receive high attention from the teacher. Low-attention patches contribute less to the global context and can be safely discarded with minimal impact on task performance.\n\nWe completely agree with the reviewer that a direct comparison with the original tokens from a high-resolution input is crucial to demonstrate the effectiveness of our approach. However, we believe this is already addressed through our experiments, which all include DINOv2 as a baseline. DINOv2 receives the full high-resolution input, and since LookWhere distills a DINOv2 teacher, the performance of the latter serves as a natural upper bound for LookWhere. As shown in Tables 1 and 2, LookWhere achieves comparable performance to DINOv2 on ImageNet classification and ADE20k segmentation using only a fraction of the patches, supporting that LookWhere’s carefully designed cost reduction does not incur a significant loss of meaningful context. Given this helpful feedback we will revise our explanation of the teacher baselines to clarify that they offer the full high-resolution comparison without sparse processing.\n\nFurthermore, in some downstream tasks, LookWhere even outperforms DINOv2. For example, in Traffic Sign Recognition (Table 3), it achieves higher accuracy with 20% of the patches, and, in the Billiard Ball task (see our response to reviewer J83A), it nearly matches DINOv2’s performance with only 20% of the patches. We attribute this strong performance to the selector’s ability to focus precisely on task-relevant regions.\n\nLookWhere also maintains competitive performance across other downstream tasks – including fine-grained bird classification (Table 4), aerial scene classification, and clinical diagnosis (see new experiments in our response to reviewer J83A). These results further support that our selection strategy retains the _relevant_ global context, even without full high-resolution input.\n\n\\\n**To what extent does the proposed distillation framework depend on the specific properties of DINOv2?:** \\\nLookWhere relies on DINOv2 – and any other teacher model – in two key ways: to distill where to look via its saliency maps, and what to see via its representations. While many modern models provide high-quality patch representations suitable for training the extractor, our choice of DINOv2 is primarily motivated by its highly semantic and clean attention maps, which allow the selector to identify visually salient regions without supervision from labels or tasks.\n\nImportantly, any teacher model with semantic attention maps could be used effectively within our framework. To test this generality, we conducted new experiments during the rebuttal period using Franca [R1] – a new visual foundation model – with a ViT-B backbone as the teacher, keeping all other hyperparameters fixed. We note that Franca was released after our submission, and demonstrates that LookWhere can be easily applied to and benefit from future models. Franca uses a CLS token for global context and does not employ registers, yet we observed that distillation remains effective. Following Section 3.1, our LookWhere-Franca model with k = 128 (9.3% of tokens) achieves 71.2% accuracy on ImageNet-HR and 61.4% on ADE20K. Furthermore, our finetuned LookWhere-Franca model achieves 81.0% on ImageNet which compares favorably to DINOv2 (Table 1). We also finetuned this LookWhere-Franca model on downstream tasks, obtaining the following results:\n- **Traffic:** 90.2%, 91.4% and 93.4% using K values of 518 (10%), 1036 (20%), and 2592 (50%).\n- **Birds:** 86.6%, 88.1% and 88.2% using K values of 136 (10%), 273 (20%) and 684 (50%).\n- **Billiards:** 97.1%, 97.0%, and 97.0% using K values of 518 (10%), 1036 (20%), and 2592 (50%) (see “Generalization to More Domains” in our response to reviewer J83A) \n\nThese results compare favorably to those obtained with DINOv2, and we will reference them alongside our other finetuning results in section 3.2 while also including them near Tables 3 and 4. We agree with the reviewer that discussing the specific advantages of DINOv2, and the general properties favorable within a teacher would strengthen the paper’s generality, and we will incorporate this discussion into Section 2.3 on Pretraining.\n\n\\\n**A mechanism for dynamically determining K:** \\\nFirst, we note that – consistent with prior work on token selection using fixed K values – we observe that most instances within a given task can be handled effectively with a similar number of patches. That said, we agree with the reviewer that dynamically selecting K is a promising direction for improving both performance and efficiency, potentially enabling richer adaptive models. Several strategies could support dynamic K selection, such as thresholding the selector map scores or conditioning K on the input image and a desired accuracy or efficiency level.\n\nWe explored one approach during the rebuttal period by selecting high-res patches based on logit thresholds rather than top-K selection (our default in the paper). For both methods, we swept values and measured accuracies on ImageNet-1K. We found that both approaches performed similarly with respect to accuracy and speed. For example, using a logit threshold of 0.03 resulted in 79.48% accuracy and 78.35 tokens on average; whereas setting K=80 tokens resulted in 79.72% accuracy. Given the broader relevance of this idea – extending beyond LookWhere to adaptive token selection more generally – we leave a deeper investigation to future work.\n\n[R1] S. Venkataramanan et al., “Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning,” arXiv preprint arXiv:2507.14137, 2025."}]

Author rebuttal:

Meta-review: This paper introduces LookWhere, a method for building inference-efficient vision transformer. It consists of a low-resolution selector (to identify informative regions) and a high-resolution extractor (to process the patches). The approach is trained by distilling self-supervised models and achieves strong accuracy-efficiency trade-offs across several benchmarks. The reviewers generally praised the paper in their initial reviews, and the authors successfully answered all of the reviewers concerns, strengthening the claims. The additional evidence (more datasets and more teacher models) made reviewers improve their scores. Overall this is a good paper and I recommend acceptance as a poster.
Decision: Accept (poster)