Title: LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision
Venue: NeurIPS 2025
Accepted: yes
Keywords: deep learning

computer vision

adaptive computation
Abstract: Vision transformers are ever larger, more accurate, and more expensive to compute.
At high resolution, the expense is even more extreme as the number of tokens grows quadratically in the image size. 
We turn to adaptive computation to cope with this cost by learning to predict where to compute.
Our LookWhere method divides the computation between a low-resolution selector and a high-resolution extractor without ever processing the full high-resolution input.
We jointly pretrain the selector and extractor without task supervision by distillation from a self-supervised teacher, in effect learning where and what to compute at the same time.
Unlike prior token reduction methods, which pay to save by pruning already-computed tokens, and prior token selection methods, which 
require complex and expensive per-task optimization, LookWhere economically and accurately selects and extracts transferrable representations of images.
We show that LookWhere excels at sparse recognition on high-resolution inputs (Traffic Signs), maintaining accuracy while reducing FLOPs by 17x and time by 4x, and standard recognition tasks that are global (ImageNet classification) and local (ADE20K segmentation), improving accuracy while reducing time by 1.36x.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes \"LookWhere,\" an adaptive computation method to address the prohibitive computational cost of Vision Transformers, especially on high-resolution images. The method divides computation into a low-cost 'selector' that predicts 'where to look' from a low-resolution input, and an 'extractor' that processes 'what to see' using only the selected high-resolution patches. For training, it employs a novel 'what-where distillation' technique that simultaneously distills the attention (where) and representations (what) from a self-supervised model, enabling task-agnostic generalization and efficient fine-tuning.\nExperiments demonstrate that the method achieves a superior accuracy-efficiency trade-off compared to existing methods across diverse recognition benchmarks. Notably, on the high-resolution traffic sign recognition task, it shows remarkable results, reducing FLOPs by up to 34x and inference time by 6x while maintaining accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe idea of separating the process into an inexpensive selector that decides \"where to look\" and an expensive extractor that processes \"what to see\" is intuitive and compelling. By not needing to process the entire input image at high resolution from the start, it can achieve fundamentally higher computational efficiency than existing token pruning methods.\n\n**Major Weaknesses**\n\nThe core of the proposed method lies in reusing global tokens (CLS/register) extracted from a computationally cheap, low-resolution input. While Figure 7(c) demonstrates the effectiveness of this approach, a crucial analysis is absent: a direct comparison with the original tokens from a high-resolution input to quantify how much global contextual information is retained in this low-resolution version. \nTo convincingly justify the validity of the efficiency gains, it is essential to quantitatively verify that this cost reduction does not incur a significant loss of important contextual information.\n\n**Minor Weaknesses**\n\n- The paper selects DINOv2 as the teacher model. A discussion on why DINOv2 is particularly suitable for this task, and the potential implications of using other self-supervised models, would strengthen the paper's claims of generality. Without this context, it is difficult for the reader to gauge to what extent the framework's effectiveness is tied to the specific properties of DINOv2, as opposed to being a more universally applicable distillation strategy.\n\n- The number of selected patches, k, is a critical hyperparameter that determines the trade-off between accuracy and efficiency. While the paper evaluates several fixed values for k, it does not deeply discuss a mechanism for dynamically determining k based on the task or input image. Although a training method that allows for flexible selection of k at inference is mentioned, the criterion for this selection is left to the user.\n\nQUESTIONS:\n- How much information is lost in the global tokens extracted from a low-resolution input compared to their original high-resolution counterparts? Can it be expected that this loss is small enough to avoid significant performance degradation on tasks that rely heavily on global object information?\n- Is the choice of DINOv2 as the teacher model critical? To what extent does the proposed distillation framework depend on the specific properties of DINOv2?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

comments: SUMMARY:
The paper proposes LookWhere, a self-supervised method for vision transformers aiming to reduce computational cost. The method splits inference into two components: a low-resolution selector determining "where to look" by predicting regions of visual interest, and a high-resolution extractor processing only the selected regions to determine "what to see." The selector and extractor pair is jointly pretrained via distillation from a self-supervised teacher model (DINOv2), enabling efficient representation learning. Experiments demonstrate LookWhere's efficacy in reducing computational overhead while maintaining or improving performance on different benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The paper uses an interesting split-resolution selector-extractor strategy, which effectively utilizes adaptive computation.
2. The paper presents an interesting joint distillation framework (what-where) from a self-supervised teacher that enhances the computational efficiency.
3. Strong experimental results showing efficiency gains.

Weaknesses:

1. Relying on a single self-supervised teacher (DINOv2) for distillation might limit the generalizability.
2. The exploration of selector fine-tuning for task-specific optimization is limited.

QUESTIONS:
1. How dependent is LookWhere's performance on the choice of the self-supervised teacher? Please discuss the robustness of LookWhere when trained using alternative teachers.
2. What prevents fine-tuning the selector for task-specific scenarios?
3. Could the selective mechanism make LookWhere more sensitive or robust to adversarial examples?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a self-supervised learning based distillation framework comprised of a selector for processing low-resolution images, a FFN to ‘upscale’ attention map, an extractor for tokenization and a task-specific predictor for downstream fine-tuning. And then authors conducted experiments on ImageNet-1K, Caltech-UCSD Birds and Traffic Signs dataset for image classification and ADE20K for semantic segmentation to evaluate proposed method’s effectiveness and efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Well-motivated: The authors comprehensively introduced the motivation of each section or design.\n2. Clear writing: This paper offers a clear and logical structure that is easy to follow.\n\nWeaknesses:\n1. About `Fig.2`’s demonstration: According to the `Sec.2.1`, this figure may loss some key details about FFN for high-resolution selector map, which also makes the $L_{map}$ a little confusing.\n2. According to the ablation in `Fig.7.c`, the register token brings large performance gain on both cls and seg task. Would the effectiveness of distillation still exist when other models (e.g. contrastive learning based CLIP or reconstruction based MAE) without register token are used as teacher model?\n3. This paper provided many metrics(e.g. FLOPS, memory, IPS) for efficiency comparison. Could author additionally provide the whole ViT-S or -B based model parameters information (including FFN for high-resolution selector map) for a more comprehensive with compared methods in `Tab.1-2`, `Fig.4-5`.\n\nI'd be more than willing to revise my score based on the author's further responses.\n\nQUESTIONS:\n1. In this paper, authors initialized the selector and extractor with the **first** $L_{low}$ layer from the teacher model DINOv2 and provided an ablation of layer depth in `Fig.7.a`. What about the layer location’s effect on distillation(e.g., 3 layers of 1-3, 4-6, 7-9, or 9-12)? There are some previous research revealing that bottom layer gains more semantic information than top layers, which may bring more gains for distillation.\n2. Could this proposed distillation method also be applied on other architectures beyond ViT? After the process of selector, top-k tokens may loss its 2D structure, which may bring a limitation of the extractor. While the architecture of selector that process full low-resolution images has more choices beyond ViT. Could author make a try of more architectures?\n\nLIMITATIONS:\nSee `Weakness` and `Question` part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose LookWhere, a method for efficient processing of high-resolution images with vision transformers. LookWhere consists of two models, the selector and the extractor. The selector processes the input image in low resolution to decide which image patches (tokens) should be processed in higher resolution. Then, the extractor receives as input only these patches from the high resolution image, and outputs the final representation that can be used for downstream tasks. The main source of efficiency is the described selective processing, since only a small part of the input image is processed in high resolution.\n\nThe selector and extractor models are trained together end-to-end, by distilling an existing self-supervised teacher, i.e., DINOv2. The authors use 3 simple loss terms, (1) MSE between the class token of the extractor and that of the teacher, (2) MSE between the patch tokens of the extractor and those of the teacher, (3) KL divergence between the output of the selector (it is an attention map over the patch sequence of the full resolution image), and a combination of attention maps from the teacher.\n\nThe authors test their model on image classification and semantic segmentation, by finetuning the extractor on the respective datasets along with a task-specific head; the selector is expected to generalize after distillation, and is not finetuned for different tasks. Baselines consist of other adaptive computation methods, and DINOv2 for reference, since it is the teacher model. The authors first test the accuracy and efficiency of their method on ImageNet-1K and ADE20K, which are not of particularly high resolution, but are well established benchmarks. Then they test on Traffic Signs dataset with images of resolution $994 \\times 994$ px, where the regions of interest (signs) are relatively small. The last dataset is CUB Birds, which again is not of particularly high resolution, but requires fine-grained classification, so it can test a different capability of the model. Overall, LookWhere compares favorably to the adaptive baselines, while DINOv2 usually has the best performance, but is also considerably slower. Finally, the authors conduct a number of ablations to justify their design choices.\n\nSTRENGTHS AND WEAKNESSES:\n- Originality:\n1. The idea of selective processing by decoupling “where” and “what”, along with using a low resolution image for guidance, are not novel.\n2. However, the proposed method has simple and straightforward losses, which I assume is good for training stability, and takes advantage of modern powerful models through distillation.\n- Quality:\n1. I can see both strengths and weaknesses related to the experiments. On the strengths side, I think the authors use appropriate baselines, and they test on both classification and segmentation, which is not a common task for adaptive models. In addition, as I described in the previous Section, different experiments test for different capabilities, for example ImageNet-1K contributes as a popular benchmark, while Traffic Signs can test the attention policy of the selector.\n2. Another strength of the paper is the ability of the selector to generalize across datasets without finetuning.\n3. On the weaknesses side, I found surprising that the authors did not provide compute-performance trade-off curves by varying $k$, which determines the number of high resolution patches processed by the extractor. In general, I think it is important for an adaptive model to be able to trade off compute for performance and vice versa, so it can adjust to scarcity of resources or increased performance demands. The authors vary $k$ only in the segmentation experiments in Table 2, and in a brief ablation experiment in Fig. 7 (d), however, I think it would be appropriate to test different values of $k$ in all experiments. For example, in the fine-grained classification experiments in Fig. 4, we see that LookWhere strikes a good balance between throughput and accuracy, but it's neither the fastest nor the most accurate, so, I think it would strengthen the results if there was a throughput-accuracy trade-off curve to show that LookWhere can beat all baselines for comparable throughput (increase $k$ to beat more accurate baselines and decrease $k$ to beat faster baselines at similar throughput). I see that in Tables 3 and 4 in the Appendix, the authors provide LookWhere results for different $k$, but LookWhere never reaches the accuracy of the top performing baselines, while it remains considerably faster, so, I think higher values of $k$ could be used; for example, in Table 4 in the Appendix, LookWhere has about 1% lower accuracy compared to DTEM, but its eval speed is about $\\times 5$.\n5. I think the authors could have used more diverse high-resolution datasets, e.g., satellite or medical images.\n6. I think the parameters of all models, including the baselines, should be provided in at least one of the Tables, e.g., Table 1, for easy comparison.\n- Clarity:\n1. The paper is very well written, and easy to follow. The authors provide a lot of implementation details, and a lot of ablation studies, which can be very useful for people who are interested to build on this work in the future.\n2. I only found Fig. 5 a bit confusing, what do the authors mean by cumulative FLOPs?\n3. Minor typos, e.g., ln 263: “resolution a 3 layers” should be “and”.\n- Significance:\n1. Overall, it is a very well written paper, with carefully designed experiments, however, given that the originality of the ideas is limited, I think the results should have been stronger.\n\nQUESTIONS:\nMy main question is why the authors didn’t explore the compute-performance tradeoff that LookWhere could provide? Especially since DINOv2 achieved better performance in all but the Traffic Signs dataset, indicating that there is value in processing more image patches.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}