From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection
Authors: Lincan Cai
Jingxuan Kang
Shuang Li
Wenxuan Ma
Binhui Xie
Zhida Qin
Jian Liang
Venue: ICML 2025 poster (2025)
Keywords: Vision-language models
Attention-Based
Feature selection
Primary area: deep_learning->other_representation_learning
TL;DR: We propose a training-free method that enhances vision-language model performance by using attention maps for fine-grained raw space selection and holistic feature selection.

Abstract: Pretrained vision-language models (VLMs), e.g., CLIP, demonstrate impressive zero-shot capabilities on downstream tasks. Prior research highlights the crucial role of visual augmentation techniques, like random cropping, in alignment with fine-grained class descriptions generated by large language models (LLMs), significantly enhancing zero-shot performance by incorporating multi-view information. However, the inherent randomness of these augmentations can inevitably introduce background artifacts and cause models to overly focus on local details, compromising global semantic understanding. To address these issues, we propose an **A**ttention-**B**ased **S**election (**ABS**) method from local details to global context, which applies attention-guided cropping in both raw images and feature space, supplement global semantic information through strategic feature selection. Additionally, we introduce a soft matching technique to effectively filter LLM descriptions for better alignment. **ABS** achieves state-of-the-art performance on out-of-distribution generalization and zero-shot classification tasks. Notably, **ABS** is training-free and even rivals few-shot and test-time adaptation methods.

From Local Details to Global Context:
Advancing Vision-Language Models with Attention-Based Selection
Lincan Cai
Jingxuan Kang" 2
Shuang Li Wenxuan Ma²
Binhui Xie
Zhida Qin
Jian Liang*
Abstract
T= photo Samoyed"
Pretrained lasion-language models (VLMs), e.g.. CLIP, demonstrate impressive zero-shot capabil- ities on downstream tasks. Prior research high- lights the crucial role of visual augmentation tech- niques, like random cropping, in alignment with fine- eeeigea class descriptions generated by large language models (LLMs), significantly enhanc- ing zero-shot performance by incorporating multi- view information. However, the inherent random- ness of these augmentations can inevitably in- troduce background artifacts and cause models to overly focus on local details, compromising global semantic understanding. To address these issues, we propose an Attention-Bal Based Selection (ABS) method from local details global context, which applies attention-guidel cropping in both raw images and feature space, supplement global semantic information through strategic feature se- lection. Additionally, we introduce a soft match- ing technique to effectively filter LLM descrip- tions for better alignment. ABS achieves state-of- the-art performance on out-of-distribution gener- alization and zero-shot classification tasks. No- tably, ABS is training-free and even rivals few- shot and test-time adaptation methods. Our code is available at https//github.oom/BBTtAAAAA
Resolution: 30 x 30
Resolution: 90 x 90
sim ()) 28.6
sim .T)
sim T 63.1
sim( 7) = 31.9
sim
67.0
Figure 1: Random cropping for visual augmentation may capture background objects unrelated to the category (red box), which have lower similarity the text compared semantically meaningful objects (green box). Addition- ally, the randomness in crop size may result in background objects having a higher resolution than the main objects, leading to misjudgments when attempting to filter back- grounds based on image similarity.
(Zhou et al., 2022b;a; Khattak et al., 2023) introduce learn- able text or image prompts while keeping VLM's pre-trained backbone fixed. Similarly, test-time adaptation (TTA) ap- proaches (Shu et al., 2022; Feng et al., 2023; Karmanov et al., 2024) also achieve impressive results by finetuning VLMs online using test data. Although these methods pre- forgetting of pre-trained knowledge in VLMMs by freez- ing the backbone and only finetuning learnable prompts adapters. overfitting or decline in generalization of VLMs still inevitably occurs (Ma et al., 2023; Zhu et al., 2023).
To maximize the generalization potential of VLM pretrain- ing, recent studies show that manually designed prompts can significantly improve VLM performance on downstream tasks (Zhou al., 2022b). However, the need for domain expertise and substantial time investment makes such ap- proaches impractical for real-world applications. To address this issue, Pratt et al. (2023) uses category information as a prompt to guide large language models (LLMs) in generat- ing fine-grained descriptions, effectively enriching the text prompt with detailed nuances. Furthermore, Li al. (2024) find that text deseriptions are often more precise for local image details. To improve alignment, they apply a random cropping operation to enhance image diversity, ensuring better matching between the image and text modalities.
1. Introduction
Vision-language models (VLMs) (Radford al., 2021; Alayrac al., 2022; Jia et al., 2021; Xue al., 2021) garner significant attention for their remarkable ability to perform zero-shot generalization across various downstream tasks. To better adapt VL.Ms, several prompt-tuning methods
'Equal contribution BBeijing Institute of Technology Uniiversity of Illinois Urbana-Champaign 'Beihang University 4 "Kuaishou Technology. Correspondence to: Shuang Li <shuan- Hlai@buaa.edu.cn..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Despite the benefits of the random cropping, this technique can inadvertently crop out background objects, leading to

---

From Local Details to Global Context: Advancing Vision-Language Models with Attention-Bass Selection
misjudgments by the model. Although Li et al. (2024) as- signs weights cropped images based on their similarity to the original image, we find that the effectivvnesss of this similarity calculation is influenced by the crop size. As illustrated in Fig. 1, the green box provides a more semanti- cally meaningful image while the red box cropped image is semantically meaningless. However, the red box, with its higher resolution, shows greater similarity to the original im- age. This suggests that using image similarity to assess the importance of cropped image is not universally applicable. Therefore, before applying cropping, it is essential to focus on the primary objects within the image to avoid cropping backgrounds, which can mislead the model's judgment.
The bear's eyes (v:)
The bear's (12)
Text Encoder
The monkey's eyes (3/)
The monkey's ears (v4)
Ti
T2
Ts
T4
crop
Image Encoder
1
28.8
X
Grop
Image
Encoder
1/
28.7
25.8 218
20.9
Figure 2: Similarity between the cropped image obtained in the image raw space and the cropped feature obtained at the feature map with the text descriptions. Although both crops can focus on the "eyes" as a local feature through cropping, the crop from the feature space retains the semantic infor- mation "bear", while the crop from the raw space misleads the model to identify it as "monkey".
To tackle this, we propose an Attention- Based Selection (ABS) method. The attention map of DINO (Caron et al., 2021) effectively highlights key objects within the image, making it ideal for guiding cropping. By leveraging this, we can target regions with higher attention values for crop- ping, ensuring that the focus remains on the main objects in the image. However, cropping at the image raw space alone can help the model focus on local object features but may result in the loss of global semantic information. As shown in Fig. 2, cropped images focusing on the bear's eyes may still be misclassified as monkey due to the loss of global context, despite attending to local features. This situation highlights the limitation of image-level cropping in preserving the semantic integrity of the object's category.
model retains semantic understanding while focusing on local features. (fii) We propose soft matching approach, enabling targeted matching of text descriptions differ- ent patches. ABS achieves state-of-the-art performance in zero-shot classification and out- distriiiiittnon datasets, even outperforming methods that require finetuning.
2. Related Work
2.1. Vision-Language Models
To supplement the global semantic information of cropped images, we introduce feature selection that performing attention-guided cropping in feature space. By using the original images as input, we crop on the feature map before the model's 's final layer, extracting the crop features corre- sponding to the image-level crops. Since these features are derived from the original images, the model can retain global category information when extracting features. As shown in Fig. 2. the feature cropped from the feature map preserves the global semantic information of the bear, which would otherwise be lost in purely image-level crop. By using these cropped features, we can enrich the global information of the corresponding cropped images, ultimately achieving better alignment between the image and the text descriptions, thereby enhancing VLM performance on downstream tasks. Additionally, we propose soft matching method, which allows us to filter out text descriptions with low relevance to each crop, enabling more targeted matching.
In recent years, Vision-Language Models (VLMs) make remarkable advancements in the domain of computer vi- sion (Radford et al., 2021; Jia et al., 2021; Xue et al., 2021; Liu et al., 2023). These models acquire rich multimodal representations through joint pretraining on language and visual data, thereby outperforming traditional models (Doso- vitskiy et al., 2020; He et al., 2016) that rely exclusively on image supervision. For instance, CLIP Radford al. (2021) and ALIGN Jia et al. (2021) are trained on a dataset consisting of large number of pairs of images and text. Further research, including BLIP (Li et al., 2022; 2023) and LLaVA (Liu et al., 2023), leverages CLIP and frozen LLMs as backbones, driving advancements in the VLM field.
Despite the impressive zero-shot capabilities and transfer- ability demonstrated by VLMs, these models often overlook task-specific nuances, which can lead to suboptimal perfor- mance on downstream tasks (Zhou et al., 2022b). Conse- quently, effectively harnessing their representation capabil- ities presents a significant challenge. This study seeks to address this limitation by proposing a novel Attention- Selection mechanism from local details to global context.
In a nutshell, our contributions are summarized as follows: (i) We propose an Attention- Based Selection (A (ABS)(e to guide the cropping process, focusing on the main objects in the image and minimizing the risk of cropping background ob- jects. (ii) We introduce feature selection that cropping at the feature map of the original image to supplement the cropped images with global information, ensuring that the
2.2. Adapt VLMs To Downstream Tasks
The performance of pretrained Dision-Language Models (VLMSs) in downstream tasks is significantly influenced by
2

---

From Local Details to Global Context: Advancing Vision-Language Models with Attention- Selection
the design of text prompts (Radford et al., 2021). Prompts can either be hand-crafted for specific tasks (Zhang al., 2021; Gao et al., 2024) or learned automatically during the fine-tuning process (Zhou et al., 2022b:a). For better per- formance, the former approach often necessitates distinct prompt designs tailored to varying tasks and datasets. In contrast, the latter approach involves optimizing a continu- ous set of prompt vectors within the language branch of the model, enhancing alignment with the specifie task. How- ever, these techniques typically require few-shot data from the downstream task to effectively train the prompts. This process can be time-consuming and costly.
focus on localized features of individual objects fail to address the subsequent loss of global context. In contrast, our approach leverages DINO's s attention map, known for highlighting key objects, not only highlights local object characteristics but also preserves crucial semantic informa- tion, offering a more comprehensive solution, which can fully match fine-grained text descriptions, thereby enhanc- ing CLIP's zero-shot performance.
3. Method
3.1. Preliminary
Menon & Vondrick (2022) as well as Pratt et al. (2023) illustrate the effectiveness of enhancing textual representa- tions. This enhancement is achieved by integrating insights from large language models (LLMs) (Brown et al., 2020), which enable the automatic generation of descriptions tai- lored to specific classes. WCA (Li et al., 2024) suggests that local visual areas, obtained through random cropping, can be cross-aligned with more detailed descriptions by constructing a similarity matrix using a pre-trained Visual Language Model (VLM). However, the randomness inherent in these augmentations can introduce background artifacts and cause the model to overemphasize local information, potentially compromising its global semantic understanding. In contrast, our study employs DINO's (Caron et al., 2021) attention maps to guide data augmentation in both raw space and feature maps, effectively enhancing the model's ability to capture and integrate global semantic information.
Problem setting, An image classification task involves an image space X' and a label space Y' where y is a set of classname corresponding to each image, such as y = {cat, dog. car). The goal of zero-shot classi- fication task is to adapt pretrained VLMs to a downstream classification task without additional training. In pre- trained VLM, we demote ʃ as the image encoder and g as the text encoder, which transforms input images I and labels into a shared feature space of dimension d. Next, we will introduce the zero-shot classification capabilities of the CLIP model and discuss recent methods for generating visual and text prompts.
Zero-shot classification of CLIP. CLIP (Radford et al., 2021) is a VLM pretrained on 400 million image-text pairs using contrastive learning. It performs zero-shot classifi- cation by computing the cosine similarity between a given image and a set of labels. The scoring function is:
2.3. Attention Guided Study
sim(r, (r, 3) = cos(f(z),
(I)
Local features of an image often contain finer details. Region-CLII (Zhong et al., 2022) aims to focus the model on these local features through data augmentation augmentation. RedCir-
where x is the given image, y is one of the candidate labels, and cos represents cosine similarity. A higher score indi- cates closer semantic alignment The predicted label y is the one with the highest score for x among all y E Y. The orig- inal CLIP constructs input text using hand-crafted prompts like Phe = "a photo of a {y}." Enhanced performance was achieved by manually designing 80 diverse prompts.
cle (Shtedritski et al., 2023) also shows that encircling object with a red circle can effectively direct model's 's atten- tion to that specific area. As we all know, attention, as a key component of transformer models, weights the relationships between image patches, identifying main objects. Thus, attention maps are effective tools for guiding focus on local features. FALIP (Zhuang al., 2025) incorporates foveal attention within the image, allowing the model to transition more smoothly between the focal area and the background. In addition, ACEN (Chen et al.. 2022a) generates attention maps to crop and randomly erasing regions to force the model to focus on key areas, ProxyCLIP (Lan et al., 2024) combines features from VFMs with CLIP through Proxy At- tention, which enhances the prominence of primary objects, and zero-seg (Rewatboworwwong et al., 2023) proposes to balance global and local contexts within CLIP's attention layers by analyzing attention values to estimate region-wise saliency. However, these methods either fail to focus the model on local object features and lack the capability to
Zero-shot classification using visual and text prompts. Recent work (Pratt et al., 2023) utilizes category information as prompts to guide LLMs in generating detailed descrip- tions. For a classname y E V, the descriptions generated are " = (LLM where M is the number of descrip- tions. Building on this approach, (Li et al., 2024) propose a visual prompting method using random cropping aimed at generating fine- fine-grained image regions that align better with Pim- The score function for (Li et al., 2024) is defined as:
N
M
simwea (x, =
m w.u.sim(a.. v!",,
(2)
3

---

From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection
space Selection
Feature Selection
(1) pet Samoyed typiealy has thisk covers body.
DINO
Guide
Samoyed sylcaaly while Bully dog with blue
NsensonnKsg
Patch El Prtsedbin
Soft Matching
(1. 1)
Festure Selection
LayestNonn
MSA
LayerNorm
Layer
Encoder
Blocks
"Samoyed"
Raw space Selection
Figure 3: Framework overview. Raw space selection: We use DINO's attention map to guide image cropping, avoiding the inclusion of background objects. Feature selection: The original image is used as input and performs cropping on the feature map corresponding to the fine-gr ane-grained selection before the final layer, to preserve global semantic information. Soft matching: We calculate a weight matrix to filter out irrelevant text descriptions for each crop, enabling better alignment.
where wy and i/ are weights for filtering irrelevant im- ages and descriptions, and N indicate the number of crops. These approaches enhance image-text alignment, improving CLIP's s zero-shot performance.
Based on the computed probability distribution, we sample the top-k patches N times. The sampling process can be represented by PPa..Po......PP Sampling (Pop-- Prob(p;). NN), where Sampling() refers to performing N samples according to the probability distribu- tion of the top-k patches.
3.2. Attention-Based Raw Space Selection
Random cropping images can yield more refined re- gions, but due to its randomness, the position and size of the cropped areas are uncertain, potentially cropping out background objects unrelated to the category. Although (Li et al., 2024) attempts to mitigate this with w;, our analysis in Fig. shows that w; is also influenced by crop size, making difficult to effectively filter out background objects.
For each sampled patch, assuming its center position is Qi we randomly select crop size centered at e and perform the cropping operation to obtain the final fine-grained selection:
p(z) {α. 6(2, c, s)(i ,,...,,,, .N.
(6)
where & is the cropping operation and crop size s (rand(a, 9,,, rand H)). Finally, we can obtain local features from raw space selection Fr,(x) = f(x)) By leveraging Dino's attention map, we ensure that the center of each crop is focused on the main object within the image, while the crop size is random. This approach guarantees both fine-grained character and diversity of the cropped images. preventing backgrounds from being cropped.
Therefore, we propose an attention- based method to guide image cropping named Attention-Based Selection (ABS), aiming to avoid cropping background objects. Since DINO's (Caron et al., 2021) attention map is widely recognized for effectively capturing the main objects an image, we use the attention map from the last transformer layer of DINO model for image x, denoted as A E PxPh, where P and h represents the number of patches and attention heads respectively. We average attention maps from all heads:
3.3. Attention-Based Feature Selection
By guiding image cropping with the attention map, we enable the model to focus more on the object's local fea- tures, resulting in better alignment with the fine-grained text description. However, shown in Fig. 2, we find that while the crop helps the model focus local features, it may also lead to the loss of global semantic information. Therefore, we propose an attention- feature selection method, which supplements the lost global information by performing crop operation on the original image's s features, corresponding the raw space selection.
h
A
(3)
1=1
where A, denotes the attention map of the i-th attention head. We then sort values in Á and select the top-k patches:
Piet (4) where pj represents the value corresponding to the i-th patch in the A. For the selected top-k patches, we apply the soft- max to obtain the probability of each patch being selected:
Piop-k (Pii P where Ps, > Pie
We take the original image x as input and extract features Fmnd = ft-1 (x) just before the final transformer layer, where is the number of transformer layers. Since the original image serves as the initial input, the extracted fea- tures at this stage retain global semantic information. We then perform the same cropping operation on the feature
Prob(p:) = exp(p:)
i {1, 21,2,...,AA .... (5)
Σ/= exp(p;)

---

From Local Details to Global Context: Advancing Vision-Language Models with Attention- Based Selection
map as the fine-grainet attention-based selection, cropping out the corresponding N feature maps:
Algorithm 1 Attention- Selection
Require: input image x E Rlxwx3, DINO sampled
Ptea ( {āo mimm Ce, s)li = 1....,..J N}.
patches P {p: 1
Crop size bounds a, B E (0,1)
mid_fea (z layer = 1)
These cropped feature maps are resized back to the original size using bicubic interpolation and then reintroduced into the model to obtain the final features of feature selection F/-- Through attention- based raw space and feature selection, for each image, we obtain N features from raw space selection and corresponding N features from feature selection, which preserve global information alongside local focus.
2: for each patch p P do
3: Sample crop size: Csize uU(a, 9)
4: # Raw Space Selection:
5: Terop = ((x. p.center, Csize)
6:
Saw = CLIP(
7:
raw raw_cropp www.crop..oo wwtpss.ppendd((cm frww
# Feature Space Selection:
femp o (0mmd__faa p.center, Csize)
3.4. Soft Matching
10: fresize = Interpolate( (foup)
11: fim CLIP final (freiiee))))
At this point, we obtain the final feature F which contains
12: fea.crops.append (fica)
the fine-grained features Frs and holistic features Fjs cor- responding to an image. But intuitively, not all descriptions are suitable for each cropped image. For example, if the cropped image of dog's eye but the text description refers to its ears tail, the image and description do not match. Forcing such mismatch into the final score could interfere with the model's output. To address this, we propose soft matching approach. First, we compute the similarity sim' between each crop and all descriptions across categories:
13: end for
14: com_fea = Concat(raw.crops @ fea_crops)
ImageNerV2 (Recht et al., 2019) presents distribution shift that simulates real-world scenarios, while ImageNet- Sketch (Wang et al., 2019) consists of black-and-whic sketches that challenge models to recognize objects based on outlines rather than photographic details. ImageNet- A (Hendrycks et al., 2021b) includes naturally occurring images that serve adversarial examples, testing the robust- ness of classification models against atypical inputs. Lastly, ImageNet-R (Hendrycks et al., 2021a) features a diverse set of images that vary in style. blurriness, geographic location, and camera operation, aiming to evaluate the adaptability of models to different visual conditions. For the zero-shot classification benchmark, we adhere to the methodology outlined in (Menon & Vondrick, 2022). This benchmark encompasses several datasets, including ImageNet (Deng et al., 2009), comprehensive object recognition dataset; CUB (Welinder et al.), which focuses on fine-grained bird classification; Oxford Pets (Parkhi et al., 2012), an animal classification dataset; DTD (Cimpoi et al., 2014), a texture recognition dataset; Food101 (Bossard et al., 2014), which contains diverse range of food images; and Place365 (Zhou et al., 2017), designed for scene classification tasks
sim = cos (os ((f))) for j = I,..., M; c == I,..., K
(8)
where K is the number of categories. The description weight vector w is obtained via softmax:
w = So Softmaz(sim)).
The final score function is:
2N M
simats (z.y) v)
Σ w cos( Fi, g(y,"))
(10)
The algorithm of ABS is illustrated in Alg. 1. For additional details of our method, please refer to the Appendix A.1.
4. Experiment
In this section, we first introduce datasets and basclines that relevant to our work, and our implementation details. Then, we validate the effectiveness of ABS on two benchmark with three different backbones, comprising a total of 10 datasets. Finally, through a series of analytical experiments including component ablation, parameter sensitivity and visualization and so on, we showcase the superiority of each module within ABS when compared to alternative approaches.
Baselines. In the context of the zero-shot classification and out-of-distribution (OOD) generalization benchmark, we evaluate our method using three different backbones against several baseline approaches. The baselines are as follows: (i) CLIP (Radford et al., 2021): Utilizes photo of class as the text prompt for classification; (ii) CLIP- E (Radford et al., 2021): An enhanced variant of CLIP that employs an ensemble of hand-crafted prompts; (iii) CLIP-D (Menon & Vondrick, 2022): Leverages LLMs for generating descriptive text associated with the classes; (iv) CuPL. (Pratt et al., 2023): Improves upon CLIP-D by
Datasets. In alignment with recent studies (Li et al., 2024), we conduct evaluations across two established bench- marks: (1) out-of-distribution generalization and (2) zero- shot classification. For the -iitt-i-ddttiittionn generaliza- tion, we evaluate our methods on the variants of ImageNet.
5

---

From Local Details to Global Context: Advancing Vision-Language Models with Attention- Selection
Table 1: The Top-1 accuracy (%) of the outoo--distrbuttoon generalization benchmark using three different CLIP backbones (ViT-B/32, B/16, and L/14), the bold values highlight the highest accuracy in the table. a represents the standard deviation and A indicates the improvement of our method over the tor-performing baseline, which is underlined.
ImageNet
ImageNet-V2
ImageNet-R
ImageNet-S
ImageNet-A
Average
Method
B/16
L/14
B/32
1/14
B/32
B3/6
1/14
IV/2
B/16
L/14
B/32
B/16
1/14
B/32
B/16
L/14
CLIP
62.05
66. 74
73.48
54,79
60.83
67.65
66.24
73.98
85.40
40.78
10
57.81
29.56
47.75
68.83
50.68
59.08
70.68
CLIP-E
63:37
68.37
75.52
55.97
61,90
69.8 89.85
69.33
77.68
87.82
42.29
48.25
59.60
31.61
49.93
70.77
52.51
61.23
72.71
CLIP-D
63.01
68.04
75.03
56,25
61.32
68,
66.29
74.69
86.08
40.83
46.91
57.99
30.57
48.79
69.16
51.39
59.95
71.43
Waflle
63.30
68.12
75.31
55,72
61.71
69.35
67.34
75.85
86.96
41.48
48.30
58.76
31.25
50.31
70,28
51.81
60.86
72.13
CuPI.
64.37
69.61
76.62
57.09
63.27
70.72
68.36
77.06
87.69
42.46
49.00
59.00
31,15
50.69
71.85
52.69
61.93
73.18
WCA
71.08
77.32
59.81
64.71
71.46
69.47
78.06
43.86
50.18
59.77
25.58
56.13
25.63
55.11
64.03
74.48
ABS
67.74
71.92
77.62
61,41
66.19
72.07
72.16
79.57
88.77
44.37
50.54
60.16
41.79
61.80
77,80
57.49
66.00
75.28
0.03
0.03
0.02
0.04
0.09
0.05
0.06
0.03
0.03
0.01
0.02
0.05
0.10
0.05
0.03
+0.90
+0.84
+0.30
+1.60
+1.48
+0.61
+2.69
+0.55
+0.51
+0.36
+6.21
+5.67
+2.17
+2.38
+1.97
Table 2: The Top-1 accuracy (%) of the zero-shot classification benchmark using three different CLIP backbones (VIT-B/32, B/16 and L/14). The bold values highlight the highest accuracy in the table and underlining indicates the second-best results.
ImageNet
CUB
Oxford Pets
DTD
Food101
P6ace365
Method
B/32
B/16
L/14
B/32
B/16
L/14
B/32
B/16
L/14
B/32
B/16
L/14
B/32
B/16
L/14
B/32
B/16
L/14
CLIP
62.05
66.74
73.48
51.21
56.01
62. 12
85.04
88. 14
93.24
42.93
42.98
52.61
82,60
88.40
92.55
38.51
39:27
39.63
CLIP-E
63.37
68.37
75.52
52.74
62.53
87.38
93.62
43.83
45.27
55.43
83.93
88.83
93.07
39.28
40.30
40.55
CLIP-D
63.01
68.04
75.03
52.69
57,08
63.26
84.46
87.52
93.30
44.20
46.17
55.05
84.12
88.85
93.03
39.9
40.34
40.55
Waffie
63.30
68.12
75.31
52.04
56.89
62.27
85, $5.50
86,51
91.55
42.90 12.98
44.68
54.31
83,98
89.06
93.33
39:47
40.76
40.89
CaPL.
64.37
69.61
76.62
49.76
62.15
87.03
91.14
94.33
50.53
60.59
84.20
88,98
93.37
39.08
39.83
40.77
WCA
66.84
71.08
77.32
56.91
59.78
65.24
89.89
92.23
94.66
49.
52.79
61.78
86.40
90.01
93.96
40.66
41.43
42.23
ABS
67.74
71,92
77.62
57.42
61.01
67.06
90.39
92.64
94,88
51,65
54,26
61.80
85.66
89.69
93.05
41.22
41.88
42.27
generating higher-quality descriptions; (vi) Waffle (Roth et al., 2023): Substitutes LLM- -LM-generated descriptions with randomly generated character and word descriptions; (v) WCA (Li et al., 2024): Implements random cropping and visual-text cross-alignment to enhance classification perfor- mance. In addition, we conduct comparative analysis of our method against several fine-tuning approaches within the context of OOD generalization benchmarks. Specifi- cally, we evaluate our method alongside CoOp (Zhou et al., 2022b), CoCoOp (Zhou et al., 2022a), UPT (Zang et al., 2022), ProGrad (Zhu et al., 2023), KgCoOp (Yao et al., 2023), TPT (Shu et al., 2022), DiffTPT (Feng al., 2023), TDA (Karmanov et al., 2024) and GDA (Wang et al., 2024).
CLIP backbones: ViT-B/16, ViT-B/32, and ViT-L/14. As shown in Table 1, ABS achieved statc-of-the-ar results across all datasets and backbones. On individual datasets, we improved top-performing baselines by up to 6.21%, and on average, we achieved 2.38% improvement, demonstrating the effectiveness of our method.
Zero-shot classification. In the zero-shot classification experiment, we used three different CLIP backbones: ViT- B/16, ViT-B/32, and ViT-L/14, and compared ABS with six zero-shot baselines. As shown in Table 2, ABS achieved the best results on five out of six datasets, demonstrating its superiority on fine-grained category datasets.
Implementation details. Our experiments are conducted using the CLIP model with various backbones, including ViT-B/32, VIT-B/16, and ViT-L/14. All experiments are performed on an NVIDIA 4090 GPU. Our method incorpo- rates four key parameters: the crop lower and upper bound (a,, (8), the top importance of the patch (K), and the number of crops (N). In our study, we maintain consistent param- eters across all architectures and datasets. Specifically, we set a= = 0.5, 3 = 0.9, K = 20, N = 60, and A = 50.
Comparing with finetuning methods. The results in Ta- ble compare ABS with a series of fine-tuning approaches on the Out-of-Distrbution generalization benchmark, using the ViT-B/16 backbone. As shown in Table 3, ABS achieved the best results on all datasets except ImageNet, where it slightly lagged behind UPT. Notably, ABS is a training-fres zero-shot approach, yet it outperforms fine-tuning methods like few-shot learning and test-time adaptation, highlighting the effectiveness and superiority of our approach.
4.1. Overall Results
4.2. Analytic Experiments
fut-of-distribution generalization. In the out-of- distribution generalization benchmark, we compare our method with six zero-shot baselines using three different
A Ablation study. Our component ablation study is pre- sented in Table 4, where we use ViT-B/16 as the back- bone and perform experiments on three datasets: Ima-
6

---

From Local Details to Global Context: Advancing Vision-Language Models with Attention- Barioo Selection
Table 3: The Top-1 accuracy (%) of the out-of-distribution generalization benchmark using ViT-B/16 as the CLIP backbone compared with some finetuning methods, such as fewshot and TTA methods. "Tuned" means the model is finetuned on ImageNet and tested on target datasets.
Source
Target
Method
Tuned?
ImageNet
ImageNet-V2
ImageNet-R
ImageNet-S
ImageNet-A
Average
CoOp (Zhou al., 2022b)
71.51
64,20
75.21
47.99
49.71
61.72
CoCoOp (Zhou et al., 2022a)
71.02
64.07
76. 18
48.75
50.63
62.13
UPT (Zang et al., 2022)
72.63
64.35
76.24
48.66
50.66
62.51
ProGrad (Zhu et al., 2023)
72.24
64.73
74.58
47.99
49.39
61.79
KgCoOp (Yao et al., 2023)
71.20
64.10
76.70
48.97
50.69
62.33
TPT (Shu al., 2022)
69.70
64.30
73.90
46,40
53.67
61.59
DinTPT (Feng al., 2023)
70.30
65.10
75.00
46.80
55.68
62.58
TDA (Karmanov al., 2024)
69.51
64.67
80.24
50.54
60.11
65.01
CuPL (Pratt et al., 2023)
69.61
63.27
77.10
48.80
50.77
61.91
GDA (Wang et al., 2024)
72.23
65.04
76.97
48.96
50.51
60.37
WCA (Li al., 2024)
71.08
64.71
78.06
50.18
56.13
64.03
ABS
71.92
66.19
79.57
50.54
61,80
66.00
geNet(Deng et al., 2009), DTD(Cimpoi et al., 2014), and ImageNet- V2(Rehht et al., 2019), reporting top-1 accuracy. We take CuPL(Pratt et al., 2023) as the baseline and pro- gressively add components of our method for the ablation study. Specifically, Fss refers to the use of attention- raw space selection, Fis refers to the use of attention-based feature selection, and Soft-M refers to the application of soft matching for alignment. From Table 4, we observe that both Frs and Fis individually improve the results by approxi- mately 0.3% compared to the CuPL. The small improvement observed when using the raw space and feature selection individually because: Using either selection alone may lead to excessive focus on either local or global information. When both selection methods are employed simultaneously, they complement each other, leading to improvement of approximately 1.3%. In addition, the use of cropping will focus on specific local regions. While this enables better alignment with I.LM descriptions that match the currently focused regions, it weakens the alignment for those unre- lated these regions. Consequently, without soft matching to filter irreeevat descriptions, many unrelated descriptions would adversely affect the current crop's alignment. When combined with soft matching, the results improve by about 1.5%. In the final row, ABS, which integrates both local and global information along with soft matching, achieves a 2.98% improvement, highlighting the importance of the complementary nature of local details and global context and the significance of filtering irrelevant descriptions.
Table 4: Ablation Study on three datasets: ImageNet, DTD, and Imagenet- V2 using ViT-B/16 as the backbone. The bold values highlight the highest accuracy in the table. The first row represents the (Pratt al., 2023) which only uses LLM descriptions for alignment, and the last row represents our method. A indicates the average improvement of these three datasets compared to the (Pratt et al., 2023).
Component
Datasets
4 (Avg.)
Fs.
Ps. Soft-M ImageNet
DTD
Imagenet-V V2
69.61
50.53
63.27
69.34
$1.52
63.56
+0.33
69.34
51.81
63.10
+0,28
70.03
52,89
63.68
+1.06
v
70.32
$2.66
64.34
+1.30
70.98
$2.72
65.51
+1.93
70.31
53.88
63.92
+1.56
71.92
54.26
66.19
+2.98
raw space, augmentations included mask, highlight, redcir- cle (Shtedritski al., 2023), and crop. Here, the mask sets non-selected area pixels to zero, and the highlight increases the brightness of the selected area. In the feature map, aug- mentations included mask, highlight (fea.), highlight (attn.), and crop. Highlight (fea) and highlight (attn) refer to high- lighting in the feature map and attention map respectively. As shown in Table 5, crop outperform other augmentation methods, whereas mask performs poorly in both raw and feature space. We believe this is because CLIP's pretraining augmentation primarily uses crops, making it less adaptable to other augmentation methods. We aim to explore more VLMs and related visual augmentations in the future.
Different visual augmentation ways. Both prior work (Li et al., 2024) (Jia et al., 2022) and our experiments demon- strate the importance of visual augmentation in enhancing the model's zero-shot generalization capability. Therefore, we conduct experiments using various visual augmenta- tion methods in both raw space and feature space. In the
Parameter sensitivity. In this subsection, we analyze the sensitivity of three hyperparameters on the ImageNet dataset

---

From Local Details to Global Context: Advancing Vision-Language Models with Attention-Bass Selection
Table 5: Ablation study on different visual augmentation ways in raw space and feature space. highlight (fea.) and highlight (attn.) represent the highlight the feature map
B/16
B:2
CuPL
CLIPD
Aozumey
68
and attention map respectively.
raw space ImageNet
feature space
ImageNet
0.1
0.3
0.5
0.7
10
30
60
30
60
(5). Crop ratlo a
(0). Number of crops
(c). Top-K
mask
53.65
mask
62.72
highlight
66.44
highlight (fea.)
65.03
Figure 4: The sensitivity of three hyperparameter: crop ratio a, number of crops N and value of Top-k on on ImageNet dataset using different CLIP backbones, ViT-B/16 and ViT-
redcircle
62.63
highlight (attn.)
67.37
crop 70.98
crop
70.31
B/32, comparing with two baselines CLIP-D and CuPL.
using two different CLIP backbones, ViT-B/16 and ViT- B/32. The first hyperparameter the crop ratio. We fix and test the sensitivity of a, ranging from (0.1,0.7, As shown in Fig. 4(a), the model's performance not sensitive to the crop ratio, it generally forms horizontal line. We observe that, unlike WCA(Li et al., 2024), where perfor- mance increases with crop ratio, the model performs better with a smaller crop ratio in some cases. For example, the result with a crop ratio of 0.5 outperforms the result with 0.7. We attribute this to the inclusion of Fis, which supple- ments global semantic information to the cropped image. As result, when the crop ratio is small, the cropped im- age focuses more on local features without losing semantic context, leading to better performance.
crop around the top-k values of the attention map, ensuring the focus remains on the main objects in the image. The cropped images in Fig. 5 contain features from different regions of the main objects while avoiding background ob- jects that are completely unrelated to the image category. Moreover, we find that DINO's s attention map is superior to CLIP's. While both DINO and CLIP focus on the main objects in an image, DINO's focus more distributed, avoid- ing the extreme values seen in CLIP. This results in cropped images with greater diversity, capturing more aspects of objects. Additional crop visualization experiments can be found in the appendix.
Effect of attention map guiding. In this paper, we experi- mentally show that guiding cropping with DINO's attention map improves the quality of cropped images and enhances model performance. In Table 6, we investigate the impact of different attention maps by comparing random cropping with three other attention map-guided cropping methods. We use CLIP V as the backbone and experiment with random crop, CLIP attention map, DINO-S/16, and DINO-B/16 guidance across three datasets. As shown in Table 6, random crop yields the worst results due to its inherent randomness, which often crops background ob- jects, misleading the model's judgment. Among the three attention map-guided methods, DINO-B/16 performs the best. This is because, based on visualization analysis in Fig. 5, DINO's 's attention map outperforms CLIP's. There- fore, we can conclude that using stronger attention maps better guides cropping, ultimately improving model's zero- shot performance.
The second parameter is N, which is the number of crops. As shown in Fig. 4(b), our results remain stable across all values of N, even when N is as small as 10. This demon- strates the effectiveness of our attention-based selection method, which ensures that even with small number of crops, the performance is not compromised by randomly cropping background objects. Notably, when N is small, the efficiency of our method improves, meaning that we can reduce the inference time without sacrificing accuracy.
The third parameter is top-k, we select the top-k patches based the attention values from DINO's attention map, using their attention values as probabilities for sampling. The sampled patches are then used as centers for cropping. As shown in Fig. 4(c), our results are minimally affected by the choice of k, with the optimal performance achieved at = 20. This is because we sample based on the proba- bilities of the patches, so even with a larger k. the smaller attention values lead lower sampling probabilities, which minimizes the impact the results. This demonstrates the robustness of our approach.
Table 6: Comparison of different attention map guiding methods, including random cropping, attention map of CLIP-B/16, DINO-S/1,6, and DINO-B/16.
Visualization. Due to the randomness of random crop- ping. it is unavoidable to crop background objects, which can mislead the model's classification. Therefore, we use DINO's (Caron et al., 2021) attention map to guide the im- age cropping. As shown in Fig. 5, DINO's attention map effectively locates key objects the image. We center the
ImageNet
CUB
DTD
ImageNet-a
Average
Random Crop
69.60
53.58
52.40
54.31
57.47
CLIP-B/16
70.37
59.82
54.05
60.11
60.84
DINO-S/16
71.77
60.92
54.12
59.45
61.66
DINO-B/16
71.92
61.01
54.26
61.80
62.25
8

---

From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection
Table 7: Ablation study on using feature space selection in different transformer layers.
Layer id
2
3
4
5
6
7
8
10
11
acc. (%)
66.14
66.42
66.26
66.41
66.82
66.99
67.10
67.23
67.4
67.54
67.74
Attention
Crop
Attention
Crop
Origin
DINO
Origin
DINO
CLIP
CLIP
Figure 5: The visualization of the DINO and CLIP attention map and the cropped images guided by the attention maps.
Integrating with other VLMs. To further validate the effectiveness and transferability of method, we con- ducted additional experiments on ImageNet using multi- ple VL.M backbones, including ALIGN (Jia et al., 2021), AItLIP (Chen et al., 2022b), and GiT (Xu et al., 2022). While these models share general similarities with CLIP (Radford et al.. 2021), they exhibit distinct architec- tural designs pretraining configurations. As shown in Table 8 (where "CLIP" denotes directly using the single- image and "CLIP prompt" features obtained from differ- ent VLMs for classification), the consistent performance gains across all benchmarks demonstrate the superiority and robustness our approach. Additional results of more advanced VI.P model like BL.IP-2 (Li et al., 2023) can be found in Appendix A.3.
raw space selection form more comprehensive features, thereby enhancing model performance.
5. Conclusion
In this paper, we propose Attention-Bass Selection (ABS) from local details global context, which leverages DINO's attention map to guide cropping in both raw space and inter- mediate feature maps. This approach yields crops that focus local object characteristics as well those containing global semantic information. Additionally, we filter text deseriptions for each crop using soft matching to achieve better feature matching. Our method achieves state-of-the art results across two benchmarks including ten datasets using different backbones.
Limitation and future work. During the course of this research, we identify some limitations in the current version and directions for future improvement: Firstly, our method improves model performance by leveraging stronger atten- tion maps. However, the attention map of a single model can be limited. We look forward exploring segmentation models, such SAM, to further guide the cropping process. Secondly, have found that crop-based augmentation is currently the most effective for CLIP. We aim to explore more diverse augmentation techniques to complement each other and provide richer features for CLIP. Thirdly, our method is currently limited to image-text modalities, and we plan explore its applicability across additional modali- ties and different tasks.
Table 8: Comparison with different methods across various VLMs on ImageNet.
VLM
CLIP
CLIP-E
CLIP-D
Waffle
CuPL
WCA
ABS
ALIGN
65.24
65.79
65.08
65.22
66.24
66.77
67.85
AKCLIP
73.79
74.86
74.48
74.29
75.74
76.20
76.85
GTT
42.72
40.10
45.27
46.96
Feature selection in different layers. Table 7 presents the results of performing feature space selection at different layers of the transformer in CLIP. It can be observed that the model's accuracy generally increases with deeper lay- ers. We attribute this to the model's improved extraction of global semantic features at deeper layers. Cropping features at shallow layers yields similar to not using Fis, indicating that the model has not captured category information, mak- ing it almost identical to cropping in the raw space. This suggests that the resulting feature I's lacks global contex- tual information, as shallow-layer cropping tends to over- emphasize local patterns. In contrast, cropping at deeper layers captures global representations, complementing the
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

---

From Local Details to Global Context: Advancing Vision-Language Models with Attention-Bass Selection
Acknowledgements
Feng, C.-M., Yu, K., Liu, Y., Khan, S., and Zuo, W. Diverse data augmentation with diffusions for effective test-time prompt tuning. In ICCV, pp. 2704-2714, 2023.
This paper was supported by the National Natural Science Foundation of China (No. 62376026), Beijing Nova Pro- gram (No. 20230484296) and Beijing Nova Programme Interdisciplinary Cooperation Project.
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., and Qiao, Y. Clip-adapter: Better vision- -sionnlanguage models with feature adapters. IJC,, 132(2):581-595 2024.
References
Alayrac, J., Donahue, J., Luc, P., Miech, A., Barr, I., Has- son, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M.. Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J. L., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. Flamingo: visual language model for few-shot learning. In NeurIPS, pp. 23716-23736, 2022.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F,, Dorundo, E.,, Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis out-oo-distributton generalization. In ICCV, pp. 8340- 8349, 2021a.
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. CVPR, pp. 15262-15271, 2021b.
Bossard, L.., Guillaumin, M., and Van Gool, L. Food-101 mining discriminative components with random forests. In ECCV, 2014.
Jia, C., Yang, Y., Xia, Y., Chen, Y.TT., Parekh, Z., Pham, H., Le, Q., Sung, Y.HH., Li, Z., and Duerig, T. Scaling up visual and vision- lision-language representation learning with noisy text supervision. In ICML, pp. 4904-4916. PMLR, 2021.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., et al. Language models are few-shot learners. NeurlPS, 33:1877-1901, 2020.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In ECCV, pp. 709-727. Springer, 2022.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In ICCV, pp. 9650- 9660, 2021.
Karmanow, A., Guan, D., Lu, S., El Saddik, A., and Xing, E. Efficient test-time adaptation of vision-language models. In CVPR, pp. 14162-14171, 2024.
Chen, J., Li, H., Liang J., Su, X., Zhai, Z., and Chai, X. Attention-based cropping and erasing learning with coarse-to- refinement for fine-grained visual classifi- cation. Neurocomputing, 501:359-369, 2022a.
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. Maple: Multi-modal prompt learning. In CVPR, pp. 19113-19122, 2023.
Chen, Z., Liu, G., Zhang, B.- Ye, F., Yang, Q., and Wu, L. Altelip: Altering the language encoder in clip for extended language capabilities. arXiv preprint arXiv:2211.06679 2022b.
Lan, M., Chen, C.,, Ke, Y., Wang, X., Feng, L., and Zhang, W. Proxyclip: Proxy attention improves clip for open- vocabulary segmentation. In ECCV, pp. 70-88. Springer, 2024.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In CVPR, pp. 3606-3613, 2014.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, pp. 12888- 12900. PMLR, 2022.
Deng, J., Dong. W., Socher, R., Li, L.-J.. Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
Li, J., Li, D., Savarese, S., and Hoi, S. C. H. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pp. 19730-19742, 2023.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn,
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In
Li, J., Li, H., Erfani, S., Feng. L., Bailey, J., and Liu, F. Visual-text cross alignment: Refining the similarity score in vision-language models. CoRR, abx/2406.0299 2024.
ICLR, 2020.
10

---

From Local Details to Global Context: Advancing Vision-Language Models with Attention- Selection
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tuning. In NeurlPS, 2023.
X., De Mello, S., Liu, S., Byeon, W., Breuel, T., Kautz, J., and Wang, X. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, pp. 18134-18144, 2022.
Ma, C., Liu, Y., Deng, J., Xie, I.,, Dong, W., and Xu, C. Understanding and mitigating overfitting in prompt tun- ing for vision-language models. IEEE Transactions on Circuits and Systems for Video Technology, 33(9):4616- 4629, 2023.
Xue, H., Huang, Y., Liu, B., Peng, H., Fu, J., Li, H., and Luo, J. Probing inter-modality: Visual parsing with self- attention for vision-and-languaal pre-training. NeurIPS, 34:4514 41251444528, 2021.
Menon, S. and Vondrick, C. Visual classification via descrip- tion from large language models. CoRR, abs/2210.07/883, 2022.
Yao, H., Zhang, R., and Xu, C. Visual-language prompt tuning with knowledge-guided context optimization. In CVPR, pp. 6757-6767, 2023.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In CVPR, pp. 3498-3505, IEEE, 2012.
Zang, Y., Li, W., Zhou, K., Huang, C., and Loy, C. C. Unified vision and language prompt learning. CoRR, abs/2210007225. 2022.
Pratt, S., Covert, I., Liu, R., and Farhadi, A. What does platypus look like? generating customized prompts for zero-shot image classification. In ICCV, pp. 15691- 15701, 2023.
Zhang, R., Fang, R., Zhang, W., Gao, Li, K., Dai, J.. Qiao, Y., and Li, H. Tip-adapter: Training-fres clip-adapter for better vision-language modeling. CoRR, abx/2111.03330, 2021.
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763, 2021.
Zhong, Y., Yang. J., Zhang, P., Li, C., Codella, N.. Li, L. H., Zhou, I.,, Dai, X., Yuan, I.., Li, Y., et al. Regionclip: Region-based language-image pretraining. In CVPR, pp. 16793-16803, 2022.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? In ICML, pp. 5389-5400. PMLR, 2019.
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Tor- ralba, A. Places: A 10 million image database for scene recognition. TPAMI, (006))
2017.
Rewatbowornwong, P., Chatthee, N., Chuangsuwanich, E., and Suwajanakor,, S. Zero-guidance segmentation using segment labels. In ICCV, pp. 1162- 172, 2023.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Conditional prompt learning for vision-language models. In CVPR, pp. 168 16825,, 2022a.
Roth, K., Kim, M., Koepke, A., Vinyals, O., Schmid, C., and Akata, Z. Waffling around for performance: Visual classification with random words and broad concepts. In ICCV. pp. 15746-15757, 2023.
Zhou, K., Yang. J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. IJV, 130(9):2337- 2348, 2022b.
Zhu, B., Niu, Y.. Han, Y., Wu, Y., and Zhang, H. Prompt- aligned gradient for prompt tuning. In ICCV, pp. 15659 15669, 2023.
Shtedritski, A., Rupprecht, C., and Vedaldi, A. What does clip know about a red circle? visual prompt engineering for vlms. In ICCV, pp. 11987-11997 2023.
Zhuang. J., Hu, J., Mu, L., Hu, R., Liang, X., Ye, J., and Hu, H. Falip: Visual prompt foveal attention boosts clip zero-shot performance. In ECCV, pp. 236-253. Springer, 2025.
Shu, M., Nie, W., Huang, D.-A., Yu, Z., Goldstein, T., Anandkumar, A., and Xiao, C. Test-time prompt tuning for zero-shot generalization in vision-language models. NeurlPS, 35:14274-14289 2022.
Wang, H., Ge, S., Lipton, 7., and Xing, F. Learning ro- bust global representations by penalizing local predictive power. NeurIPS, 32, 2019.
Wang, Z., Liang, J.. Sheng, L., He, R., Wang, Z., and Tan, T. A hard-to-beat baseline for training- free clip-based adaptation. CoRR, abs/:2402.04087, 2024.
Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., and Perona, P. Caltech-ucsd birds 200.

---

From Local Details to Global Context: Advancing Vision-Language Models with Attention-Bass Selection
A. Appendix
A.I. Algorithm of ABS
The algorithm of ABS is illustrated in Alg. 1. Firstly, we select N patches of the raw space image from DINO's attention map using top-k sampling. Next, we determine the crop size by randomly sampling between a and 3 based on the positional centers of the selected N patches. Using these centers and the derived crop sizes, we perform cropping operations in both the space (for images) and the feature space (for features), thereby obtaining crops that incorporate both local and global characteristics.
The specific operations and dimensional transformations in the feature space are as follows: feature selection is performed before the forward the final transformer layer. Given input features with dimensions [bs, 197, 768), we first separate the [CLS] token ([bs, 1, 768J) from the remaining tokens ([bs, 196, 768]). The remaining tokens are reshaped into 2D size [bs, 14, 14, 768]. Based on DINO's attention map. then select N crops from this feature map. Each crops is interpolated to the original feature map size, and concatenated with the [CLS] token, reconstructing the feature into [bs, N, 197, 768]. This modified feature is fed into the final transformer layer. During this layer's forward, the [CLS] token interacts with the crops to capture diverse local features enriched with global semantic information.
A.2. Visualization
As shown in Fig. 7, select images from various datasets for visualization, including DINO's attention and the cropped images obtained from raw space selection guided by the attention map. It is evident that with effective guidance from the attention map, our cropped areas focus on the main objects in the images and capture different features of the objects.
A.3. Additional Experiments
Time cost of raw space selection and feature selection. As shown in Table 9, we calculate the time consumption for the "crop+preprocess" and "Encoding" stages. "crop+preprocess" includes the time for raw space selection, while "Encoding" covers feature selection. Although ABS takes more time than CLIP, the performance improvement is significant.
Table 9: The "Crop+Preprocess" and "Encoding" time cost of CLIP and ABS for different numbers of crops in seconds.
N
Process Step
CLIP
10
20
30
40
50
Crop+Preprocess
0.0004
0.0011
0.0028
0.0036
0.0045
0.0060
Encoding
0.0004
0.0094
0.0168
0.0244
0.0322
0.0340
Total
0.0008
0.0105
0.0196
0.0280
0.0367
0.0400
Accuracy
66.74
71.72
71.92
71.98
71.97
71.93
More advanced VLP model. We employ the Biin2ForlmageTexxRetieee (Li al., 2023) model architecture and compare ABS with other baseline methods (where "CLIP" denotes directly using the single-imap and "CLIP prompt" features obtained from BLIP-2 for classification). As shown in the Table 10, ABS outperforms all other approaches across two different datasets, demonstrating its effectiveness and adaptability.
Table 10: Comparison with other methods using BLIP-2 as backbone on DTD and ImageNet-A datasets.
Base model: BLIP-2
CL.IP
CLIP-E
CLIP-D
Waftle
CuPL.
WCA
ABS
DTD
42.55
46.54
51.70
45.64
50.74
54.47 72.79
56.12
ImageNet-A
59.96
58.08
62.99
61.43
63.51
74.39
A.4. Failure Case Discussion
The relatively modest performance of our method on the Food101 dataset can be attributed to the following factors: The Food101 differs from conventional multi-object datasets, it contains inherently multi-label images but provides only single-
12

---

From Local Details to Global Context: Advancing Vision-Language Models with Attention-Bant Selection
label annotations. For instance, image labeled "french fries" may actually contain multiple objects (e.g., fries, steak, and salad), where non-target objects could occupy larger visual proportion than the labeled subject (as shown in Fig. 6). Because all these objects fall within the predefined label set, the single-label assignment introduces ambiguity. These inherent properties could cause our method to identify unlabeled object categories within the images.
Label: cheesecake
Label: pork_chop Predict: baby_ Lal__back__aaa ribs
Label: sandwshhhiich Predict: french_fries
Label: hamburger Predict: french_fries
Predict: Chocolate cake
Label: hamburger Predict: onion_ringe
Label: filet_migg jet_minnonn Predict: french fries
Label: cheesecake Predict: cup_ cakes
Label: cheese Lopeseeplatte__et Predict: bread_ _acad__dding
Label: lobster bisque Predict: oysters
Label: pancakes
Label: french fries
Label: filet_ sett_mmigon
Predict: hetwberry_shortckke
Predict: Fish and chips
Predict: Lobster bisque
Figure 6: Examples of Food101 dataset.
13

---

From Local Details to Global Context: Advancing Vision-Language Models with Attention-Bass Selection
Origin
Attention
Crop
Origin
Attention
Crop
Oxford Pet
CUB200
Food101
DTD
Origin
Attention
Crop
Origin
Attention
Crop
ImageNet
ImageNet-R
ImageNet-S
Place365
Figure The visualization of the DINO attention map and the cropped images guided by attention maps.
14


Reviews:
[{"review_id": "YUWUn9s11m", "reviewer": "Reviewer_C2Mc", "summary": "This paper introduces ABS, a training-free Attention-Based Selection method that uses Vision-Language Pretraining (VLP) model’s attention maps (e.g., DINO and CLIP) to guide cropping in both raw image and feature space, effectively integrating local details with global semantic context via soft matching to achieve SoTA performance in some zero-shot classification tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The concerns in Claims And Evidence.\n\n**A1:** Thanks.\n1. The table below compares applying soft matching alone vs. combined two selections to the baseline. Combined with the ablations in our paper, it shows that individual components yield improvement when used independently, but integration works best. It is noteworthy that the combined two selections outperforms using soft matching alone, proving these key modules significantly enhance overall performance.\n2. The small improvement observed when using the raw space and feature selection individually because: Using either selection alone may lead to excessive focus on either local or global information. Moreover, employing cropping will focus on specific local regions. While this enables better alignment with LLM descriptions that match the currently focused regions, it weakens the alignment for those unrelated to these regions. Consequently, without soft matching to filter irrelevant descriptions, many unrelated descriptions would adversely affect the current crop's alignment. This phenomenon is also observed in the ablations in WCA, demonstrating that filtering or weighted integration of crops is crucial.\n3. We replace the components in WCA with ours in the table below. The results prove that our components deliver performance gains compared to the original WCA, proving superiority of our design.\n||ImageNet|DTD|ImageNetv2|$\\Delta$Avg\n|-|-|-|-|-\n|Baseline|69.61|50.53|63.27|\n|Baseline w/ soft m.|70.03|52.89|63.68|+1.06\n|Baseline w/ two sele.|70.32|52.66|64.34|+1.30\n|Ours|**71.92**|**54.26**|**66.19**|+2.98\n|WCA|71.08|52.79|64.71|\n|WCA w/ soft m.|71.37|53.60|65.67|+0.69\n|WCA w/ two sele.|71.42|53.76|65.83|+0.81\n\n**Q2:** More advanced VLP model.\n\n**A2:** Thanks, as your suggestion, we employ the Blip-2 and compare with other baselines. The table below shows that ABS outperform all others across two datasets, demonstrating our effectiveness and adaptability. More advanced VLMs (e.g. ALIGN, AltCLIP and GroupViT) please refer to **our answer A3 to Reviewer dz4u.**\n\n||Waffle|CuPL|WCA|ABS\n|-|-|-|-|-\n|DTD|45.64|50.74|54.47|**56.12**\n|ImageNet-A|61.43|63.51|72.79|**74.39**\n\n**Q3:** The comment of Broader Scientific Literature.\n\n**A3:** Thanks. Please refer to **our answer A1 to Reviewer bJ6A.**\n\n**Q4:** About attention‐based cropping methods.\n\n**A4:** Thanks. [1] generates attention maps to crop and randomly erasing regions to force the model to focus on key areas. Although their method focus local regions, it fails to address the subsequent loss of global context. In contrast, our approach compensates by using feature selection. We experiment with [1] on our task (as shown in the table below), ABS demonstrate superior performance. This advantage stems from our global semantic compensation for cropped regions.\n||ImageNet\n|-|-\n|ACEN [1]|61.62\n|Ours|**71.92**\n\nAlthough [2] and [3] are both attention-related works, [2] uses geographical information, making it only applicable to specific scenarios. [3] requires training for its modules, which contradicts zero-shot tasks. Moreover, neither method considers supplementing global information. In the revision, we will discuss and compare with these methods.\n\n**Q5:** About technical contribution.\n\n**A5:** Thanks. Please refer to **A1 and our answer A1 to Reviewer bJ6A.**\n\n**Q6:** Concern about Food101.\n\n**A6:** Thanks. ABS demonstrates performance improvements across multi-object or complex backgrounds datasets including Place365, only with the exception of Food101. \n\nThe Food101 differs from conventional multi-object datasets, it contains inherently multi-label images but provides only single-label annotations. For instance, an image labeled as \"french fries\" may actually contain multiple objects (e.g., fries, steak, and salad), where non-target objects could occupy a larger visual proportion than the labeled subject (We show several examples in our anonymous link: https://anonymous.4open.science/r/Submission_4487-1B78). Because all these objects fall within the predefined label set, the single-label assignment introduces ambiguity. These inherent properties could cause our method to identify unlabeled object categories within the images.\n\nWCA’s randomness may inadvertently benefit from this properties. However, our experiments confirm substantial improvements in most of classification tasks including multi-object or complex backgrounds datasets.\n\n**Q7:** About prompt and pseudocode.\n\n**A7:** Thanks.\n1. For the generation of LLM descriptions, we simply follow CuPL, utilizing their publicly released JSON files. The specific prompts can be found in the CuPL.\n2. For details and pseudocode of the alignment process, please refer to **our answer A2 to Reviewer dz4u and A4 to Reviewer iUpr.**\n\nWe will supplement these in future versions of our paper.\n\n**Q8:** Weakness 4&5.\n\n**A8:** Thanks. We will revise our paper thoroughly.\n\n**Q9:** Other comments and questions. \n\n**A9:**  Thanks. Please refer to the answer above.", "claims_and_evidence": "The primary claims may not be supported by convincing evidence. Specifically, the effectiveness of the attention‐based raw space and feature selection components is not fully supported by the ablation study provided. In Table 4, these components contribute only a marginal average improvement (around +0.3%). It makes me concerns about whether these key modules significantly impact overall performance.", "methods_and_evaluation_criteria": "The paper leverages standard zero-shot visual classification and domain generalization benchmarks (i.e., various ImageNet variants, CUB, Oxford Pets, DTD, Food101, and Place365) which effectively assess both global and fine-grained performance.", "theoretical_claims": "The paper offers only limited theoretical claims without detailed proofs, relying mostly on intuitive arguments.", "experimental_designs_or_analyses": "The experimental design is generally sound with thorough ablation studies. However, given that the quality of the attention map significantly impacts performance, evaluating the method with a more advanced VLP model (e.g., BLIP-2) would better validate its robustness.", "relation_to_broader_scientific_literature": "The paper builds on WCA (ICML, 2024), by incrementally enhancing its pipeline. Whereas WCA uses random cropping and LLM-generated descriptions, this paper integrates attention-based selection to more accurately focus on semantically important regions.", "essential_references_not_discussed": "The paper overlooks several key studies on attention‐based cropping methods [1,2,3] that are essential for highlighting its contributions.\n\n[1] Chen J, Li H, Liang J, et al. Attention-based cropping and erasing learning with coarse-to-fine refinement for fine-grained visual classification[J]. Neurocomputing, 2022, 501: 359-369.\n\n[2] Wang Y, Zhang Z, Feng L, et al. A new attention-based CNN approach for crop mapping using time series Sentinel-2 images[J]. Computers and electronics in agriculture, 2021, 184: 106090.\n\n[3] Wang W, Shen J. Deep cropping via attention box prediction and aesthetics assessment[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2186-2194.", "other_strengths_and_weaknesses": "Strengths \n\n1.\tThe experiment of this paper is extensive.\n\n2.\tIt is an interesting idea of attention-based feature selection to enhance the model's performance.\n\nWeaknesses\n\n1.\tLimited technical contribution. The proposed ABS closely resembles WCA (ICML, 2024) with only incremental changes, and the Soft Matching module appears to drive most of the performance gains (Please refer to “Relation To Broader Scientific Literature”).\n\n2.\tConcerns about the performance on images with multiple objects (e.g., Food101) or complex backgrounds (e.g., Place365). For instance, the Food101 dataset shows lower performance compared to WCA and even negative gains on Place365.\n\n3.\tConcerns about the reproducibility. the paper insufficient details on the LLM's prompt generation, alignment process between raw and feature spaces. It is a recommendation for including a pseudo-code.\n\n4.\tFormatting issues in some formulas (e.g., Formula 8) that need optimization.\n\n5.\tOverall language and clarity could be improved for better readability.", "other_comments_or_suggestions": "My main concern remains the incremental contribution, particularly the limited gains from the attention-based selection compared to the soft matching component. Therefore, I give a \"Weak Reject\" recommendation and may improve my rating if the rebuttal satisfies.\n\nUpdate:\n\nI appreciate the authors' efforts in their rebuttal. It effectively addresses most of my concerns. I believe the proposed method is simple but efficient, and as a result, I decide to raise my rating to ``weak accept'' recommendation", "questions_for_authors": "Why does ABS perform worse on Food101? Is it related to the complexity of food images, the presence of multiple objects, or another factor?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "l2TT4lsJlA", "reviewer": "Reviewer_bJ6A", "summary": "The paper proposes an Attention-Based Selection (ABS) method to improve zero-shot classification and out-of-distribution generalization capabilities of vision-language models (VLMs). ABS leverages DINO’s attention maps to guide the cropping of images, thus preventing random crops from focusing on irrelevant background areas. The method also introduces feature-level cropping to supplement global semantic context. Finally, a soft matching mechanism filters large language model (LLM)-generated text descriptions to improve visual-textual alignment. The authors report state-of-the-art results on multiple benchmarks, demonstrating the effectiveness of ABS.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** About contribution.\n\n**A1:** Thanks. Our core contribution resides not in proposing incremental adjustments to established frameworks, but rather in advancing systematic methodologies that demystify stochastic factors while enhancing holistic semantic comprehension for this research domain.\n\nFirstly, the adoption of attention-based mechanisms to mitigate cropping randomness plays a pivotal role in zero-shot classification task, enabling consistent and robust performance improvements. \n\nMoreover, our key insight lies not merely in employing attention-based selection to focus on critical regions, but also in being the first (to the best of our knowledge) to perform **feature cropping** that complement raw-space crops by integrating global semantic information. This dual mechanism proves crucial for aligning with LLM descriptions, while also establishing a novel paradigm for future research in multimodal alignment.\n\nAdditionally, our approach demonstrates high flexibility, enabling seamless integration with various advanced VLMs (please refer to **our answer A2 to Reviewer C2Mc and A3 to Reviewer dz4u**) to enhance their performance through plug-and-play adaptation.\n\n**Q2:** Table 6 is not properly cited. \n\n**A2:** Thanks for pointing it out. We will fix it in the future version of our paper.\n\n**Q3:** Maintain “cos” consistency.\n\n**A3:** Thanks for pointing it out. We will fix it in the future version of our paper.\n\n**Q4:** Capitalization consistency.\n\n**A4:** Thanks for pointing it out. We will fix it in the future version of our paper.\n\n**Q5:** About hyperparameters set.\n\n**A5:** Thanks. Our method involves three critical hyperparameters: crop_ratio, num_crops, and top-k, which we consistently set to 0.5, 60, and 20 respectively across all datasets. This unified configuration serves dual purposes: First, it demonstrates our approach's capability to achieve consistent performance improvements without dataset-specific tuning, and second, it facilitates convenient application in industrial deployments. As evidenced by the hyperparameter sensitivity analysis presented in the table below or in the Figure5 in our paper, our method exhibits remarkable robustness to parameter variations. In fact, alternative configurations (e.g. crop ratio=0.6, num_crops=30 and top-k=30) might potentially yield superior results to those reported in our paper. Nevertheless, we maintain fixed parameters throughout all experiments to ensure fair comparative evaluation and rigorous empirical validation of the proposed methodology.\n\n| crop_ratio | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 |\n| --- | --- | --- | --- | --- | --- | --- |\n| ABS | 71.75 | 71.87 | 71.92 (reported in paper) | **71.99** | 71.88 | 71.84 |\n| **num_crops** | **10** | **20** | **30** | **40** | **50** | **60** |\n| ABS | 71.72 | 71.88 | **71.98** | 71.97 | 71.93 | 71.92 (reported in paper) |\n| **top-k** | **10** | **20** | **30** | **40** | **50** | **60** |\n| ABS | 71.83 | 71.92 (reported in paper) | **71.98** | 71.97 | 71.69 | 71.56 |\n\n**Q6:** About soft matching.\n\n**A6:** Thanks. The soft matching mechanism is proposed to addresses the inherent mismatch issue in CLIP's original matching strategy, specifically the \"description-crop misalignment\" phenomenon where textual descriptions partially deviate from cropped image regions. This problem arises because our selection process emphasizes local object features - compared to using full images, these focused crops may induce partial mismatches with LLM descriptions during semantic alignment. The soft matching resolves this through adaptive weighting: it suppresses irrelevant LLM descriptions while amplifying semantically aligned ones. As demonstrated in the table below, the ablation studies confirm that removing soft matching causes performance degradation, verifying its critical role in mitigating mismatch effects. Notably, applying soft matching to WCA further improves accuracy, empirically proving its general effectiveness in filtering noisy descriptions.\n\n|  | ImageNet | DTD | ImageNetV2 |\n| --- | --- | --- | --- |\n| Ours w/o soft matching  | 71.22 | 53.36 | 65.23 |\n| Ours | **71.92** | **54.26** | **66.19** |\n| WCA | 71.08 | 52.79 | 64.71 |\n| WCA w/ soft matching | **71.37** | **53.60** | **65.67** |", "claims_and_evidence": "The claims made by the authors are supported by the experimental results, and the motivation behind the proposed approach is sound.", "methods_and_evaluation_criteria": "The proposed method, including the use of DINO's attention maps for guiding cropping, feature-level cropping, and soft matching of textual descriptions, is conceptually clear, well-motivated. The evaluation on standard benchmarks aligned with current practices in the vision-language modeling literature.", "theoretical_claims": "The paper does not explicitly present theoretical proofs or formal theoretical claims.", "experimental_designs_or_analyses": "The comparisons across multiple widely recognized datasets and three different CLIP backbones (ViT-B/32, ViT-B/16, ViT-L/14) clearly support the viewpoint.", "relation_to_broader_scientific_literature": "The paper is effectively situated within the broader literature of vision-language modeling, specifically highlighting recent advances such as WCA, CuPL, and methods employing attention-based augmentation.", "essential_references_not_discussed": "The paper provides a comprehensive summary of recent relevant literature.", "other_strengths_and_weaknesses": "**Strengths**\n1. The paper is clearly written and provides strong motivation for its proposed method.\n\n2. Extensive experiments validate the effectiveness of the proposed method, and fair comparisons were conducted.\n\n3. The visualizations clearly reflect the authors' intentions.\n\n**Weaknesses**\n1. Although the method is effective, its contribution to the field is relatively minor, merely proposing a small modification to an existing problem.\n\n2.  Table 6 (Lines 425-539) is not properly cited; instead, numerical results are presented directly in the text.", "other_comments_or_suggestions": "1. In Equation (1) and (10), it is recommended to use an upright font for \"cos\" to maintain consistency throughout the paper.\n\n2. There are inconsistencies in the capitalization of certain headings.", "questions_for_authors": "1. How were the hyperparameters set, and why were they unified across all datasets?\n\n2. How sensitive is the method to the precision of soft matching? Would the performance drop significantly if the matching precision is imperfect or noisy (e.g., descriptions slightly mismatching the cropped areas)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9OL6BrcG57", "reviewer": "Reviewer_dz4u", "summary": "Recent studies have explored the use of multiple image crops obtained through random cropping, utilizing text descriptions generated by LLMs to assess the similarity between image and text embeddings for zero-shot classification tasks. This paper builds on that concept by addressing the noise caused by random cropping operations. It does this by selectively sampling crops and feature subsets that focus on the salient regions identified by the attention map from DINO. Experimental evaluations across several benchmarks show performance improvement.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Claims And Evidence.\n\n**A1:** Thanks. The purpose of employing cropping is to enable the model to focus on local features of objects, thereby achieving better alignment with certain LLM descriptions. However, random crop exhibits two inherent limitations: When using smaller crop sizes, it introduces randomness and compromises global semantic coherence; while with larger crop sizes, although uncertainty diminishes (as you mentioned), this comes at the cost of sacrificing the ability to concentrate on local features, instead providing more global information similar to using full images.\n\nOur method addresses this by enabling the model to eliminate randomness while preserving global semantics when focusing on local and pivotal features. Through ablations in the table below, we observe that our approach does not exhibit monotonically increasing accuracy with larger crop sizes. Conversely, it achieves superior results with smaller crop sizes, whereas WCA demonstrates an incremental pattern with increasing crop sizes. This contrast underscores the significance of our method's dual capability in eliminating randomness and maintaining global semantic integrity during local feature focusing, ultimately enhancing the model's optimal performance.\n\n|crop_ratio|0.3-0.9|0.4-0.9|0.5-0.9|0.6-0.9|0.7-0.9|0.8-0.9\n|-|-|-|-|-|-|-\n|WCA|70.80|70.89|70.95|70.99|71.03|**71.03**\n|ABS|71.75|71.87|71.92|**71.99**|71.88|71.84\n\n**Q2:** Clarification of feature crop operation.\n\n**A2:** Thanks. In our method, feature selection is performed before the forward of the final transformer layer. Given input features with dimensions [bs, 197, 768], we first separate the [CLS] token ([bs, 1, 768]) from the remaining tokens ([bs, 196, 768]). The remaining tokens are reshaped into 2D size [bs, 14, 14, 768]. Based on DINO’s attention map, we then select N crops from this feature map. Each crops is interpolated to the original feature map size, and concatenated with the [CLS] token, reconstructing the feature into [bs, N, 197, 768]. This modified feature is fed into the final transformer layer. During this layer's forward, the [CLS] token interacts with the crops to capture diverse local features enriched with global semantic information. We will supplement this detail in subsequent versions of our paper.\n\n**Q3:** Integrating with other VLMs.\n\n**A3:** Thanks. As shown in the table below, we apply our method to a broader range of VLMs (e.g. ALIGN, AltCLIP and GroupViT) in ImageNet and achieved superior performance compared to other approaches.\n\n|VLM|Waffle|CuPL|WCA|ABS\n|-|-|-|-|-\n|ALIGN|65.22|66.24|66.77|**67.85**\n|AltCLIP|74.29|75.74|76.20|**76.85**\n|GroupViT|42.42|44.53|45.27|**46.96**\n\n**Q4:** The impact of different components of our method.\n\n**A4:** Thanks. Please refer to **our answer A1 to Reviewer C2Mc.**\n\n**Q5:** Contribution of the work.\n\n**A5:** Thanks. Please refer to **our answer A1 to Reviewer bJ6A.**\n\n**Q6:** About other attention-based strategy methods.\n\n**A6:** Thanks. [2] employs heatmaps to localize target regions. However, it relies on dynamically updating bounding boxes during training, which is incompatible with our zero-shot scenario. Furthermore, while [2] can successfully locate target regions, it fails to address the preservation of global information after cropping, which may lead to inter-class confusion in the model.\n\n[3] combines features from VFMs with CLIP through Proxy Attention, which enhances the prominence of primary objects. [4] proposes to balance global and local contexts within CLIP's attention layers by analyzing attention values to estimate region-wise saliency. However, these methods fail to focus the model on local object features and lack the capability to focus on localized features of individual objects. In contrast, our approach not only highlights local object characteristics but also preserves crucial semantic information, offering a more comprehensive solution.\n\nWe implement the core modules of [3] and [4] within our framework and conduct experiment on the ImageNet dataset. As shown in the table below, our method demonstrates superior performance, validating the advantage of our approach in simultaneously focusing on local features while preserving global semantic information. A comprehensive comparison with methods [2]-[4] will be presented in the subsequent version of our paper.\n\n||ImageNet\n|-|-\n|Proxyclip [3]|70.36\n|zero-seg [4]|68.37\n|Ours|**71.92**\n\n**Q7:** The clarity regarding this paper's position.\n\n**A7:** Thanks. The key insight of ABS compared to previous attention-based methods lies in two aspects: not only do we leverage attention-guided cropping in the raw space to focus on local regions while eliminating randomness, but more importantly (and to the best of our knowledge) we are the first to implement attention-guided feature space selection to complement global semantics. This dual-space cropping enables the crops to better align with LLM descriptions.", "claims_and_evidence": "The claim regarding uncertainty from random cropping in Figure 1 is not sufficiently valid. The only sample sizes provided are 30×30 and 90×90, which fall outside the interval (0.4, 0.9) used in this paper. At higher resolutions, the uncertainty introduced by the cropping operation may significantly decrease. Therefore, a more comprehensive empirical evaluation is necessary to clarify this claim.", "methods_and_evaluation_criteria": "- The operation described in lines 208-211, which involves \"reintroducing the interpolated cropped feature maps into the model,\" raises some concerns. Do the authors concatenate the original [CLS] token with the interpolated feature map as the input for the final attention residual block? If so, how can the [CLS] token effectively capture the local semantics associated with the cropped region?  A more detailed clarification or analysis would be appreciated.", "theoretical_claims": "There is no theoretical analysis provided.", "experimental_designs_or_analyses": "- The experiment results only demonstrate compatibility with CLIP.  Since the feature selection strategy ( $F_{fs}$ ) may not be model-agnostic, I'm curious whether the proposed pipeline can integrate with other VLMs. A related study, specifically WCA, has shown results using ALIGN, AltCLIP, and GroupViT (please refer to Table 10 of [1]). Conducting a similar experiment could provide valuable insights.\n\n- The results shown in Table 4 of the ablation study indicate that the factors ( $F_{rs}$ ) and ($ F_{fs} $) contribute to performance degradation on some benchmarks. It appears that the main improvement in performance is primarily attributable to the soft-matching strategy, while the impacts of ($ F_{rs} $) and ($ F_{fs} $) remain unclear.\n\n[1] Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models. ICML 2024.", "relation_to_broader_scientific_literature": "- The proposed framework is mainly based on WCA [1], with its own contributions of handling the cropping uncertainty. \n\n- Utilizing network feedback mechanisms such as attention to guide the cropping process [2] and zero-shot inference [3,4] is a well-explored technique. The attention-based strategy presented in this paper resembles those earlier studies.\n\n[1] Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models. ICML 2024.\n\n[2] Crafting better contrastive views for siamese representation learning. CVPR 2022.\n\n[3] Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. ECCV 2024.\n\n[4] Zero-guidance segmentation using zero segment labels. CVPR 2023.", "essential_references_not_discussed": "- The clarity regarding this paper's position in recent attention-guided studies is still ambiguous (Sec.2.3).", "other_strengths_and_weaknesses": "### Pros: \n\n- The paper is well organized.\n\n- The motivation is clear, and the method is simple and intuitive.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OzmbroOLGY", "reviewer": "Reviewer_iUpr", "summary": "The paper introduces **Attention-Based Selection (ABS)**, a method to enhance vision-language models (VLMs) like CLIP by addressing limitations of random cropping, which often introduces background noise and compromises global semantic understanding. ABS leverages **DINO's attention maps** to guide cropping in **Raw Image Space** and **Feature Space**.  Additionally, a **soft matching** technique filters irrelevant text descriptions for each crop, improving alignment between visual and textual modalities.  \n\nThe main results indicate that ABS achieves **state-of-the-art performance** on 10 datasets across two benchmarks (zero-shot classification and out-of-distribution generalization), outperforming methods like WCA and CuPL.  Besides, it matches or surpasses fine-tuning approaches (e.g., CoOp, TPT) without requiring additional training data in the out-of-distribution generalization benchmark. Ablation studies also validate the necessity of each component, with ABS improving accuracy by up to **2.98%** over baselines.  \n\nThe main findings are that introducing DINO feature maps to select key regions for cropping is effective, and cropping both the original images and the intermediate feature maps is crucial.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Experimental details in Table 3.\n\n**A1:** Thanks. The results of baselines in Table 3 follow WCA protocol: Tuning-based methods are 16-shot source-trained and target-evaluated for OOD generalization. Notably, our method requires no fine-tuning, operating in a zero-shot manner on both source and target datasets and even surpasses fine-tuned methods. \n\nRegarding your concern, we include in the table below the results of Coop fine-tuned directly on target data. Since our method functions as a plug-and-play module without requiring fine-tuning, integrating it with Coop further enhances performance.\n\n||ImageNet-S|ImageNetv2\n|-|-|-\n|Coop ft. on tar.|49.79|70.59\n|Coop ft. on tar. + ABS|**51.47**|**72.63**\n\n**Q2:** About negative impact of DINO.\n\n**A2:** Thanks.\n1. DINO establishes a new contrastive learning framework that learns superior visual representations. Its exceptional attention maps, which have gained prominence for reliably pinpointing primary objects across diverse datasets, provide the foundation for our feature selection mechanism. And also a lot of works [2][3] using DINO for assistance. This reliability in object localization motivates our strategic adoption of DINO's attention maps to guide our selection process.\n2. We observe that DINO's attention maps consistently localize semantically relevant object regions in multiple datasets. While occasional inaccuracies may occur, our adaptive crop_ratio enhances diversity in the cropped regions to mitigate such cases. Furthermore, our soft matching strategy effectively downweights irrelevant image crops, thereby reducing potential negative impacts from DINO's misalignments.\n3. We observe that different attention heads in DINO may focus on distinct regions. In our paper, we simply average all attention heads. To further mitigate potential negative effects from DINO, we use attention maps from heads with higher variance, ensuring the selection maintain strong focus on salient objects. The following table shows the results demonstrate nearly identical performance to our original method. This suggests that ABS already achieves low error rates in attention map selection, making further error reduction less noticeable. \n\nRegarding more explicit filtering of negative attention maps, we will conduct further research.\n\n||ImageNet|ImageNet-A\n|-|-|-\n|Ours w. diverse head|71.96|61.86\n|Ours w. mean head|71.92|61.80\n\nref:\n\n[2] Zero-guidance segmentation using zero segment labels. CVPR 2023.\n\n[3] Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. ECCV 2024.\n\n**Q3:** About inference time overhead.\n\n**A3:** Thanks. When processing and encoding images, we do need to augment a single image into 2N crops. However, our hyperparameter sensitivity experiments reveal that the results are not sensitive to the num_crop. Therefore, we can reduce the value of N to lower time costs, while achieving average improvements of approximately **2%** on Zero-shot classification and **5%** on OOD datasets compared to CuPL, both of which represent significant gains. Additionally, we can follow the approach of WCA by pre-storing image features prior to inference. This ensures that the final similarity computation stage incurs comparable time costs to CuPL, while delivering substantially enhanced performance.\n\n||encoding time|zero-shot avg. acc.|OOD avg. acc.\n|-|-|-|-\n|CuPL|0.0008|39.83|61.93\n|ABS (N=10)|0.0105|41.65|65.59\n\n**Q4:** About dimension in Equation 7.\n\n**A4:** Thanks. The specific process of interpolation please refer to **our answer A2 to Reviewer dz4u** and the pseudo-code is as follows:\n```\n# s_p: sampled patches from DINO attention\nraw_crops = []\nfea_crops = []\nmid_fea = CLIP(x, layer=l-1)\nfor p in s_p:\n    c_size = random(α, β)\n    c_img = crop_img(x, p.center, c_size)\n    raw_fea = CLIP(c_img)\n    raw_crops.append(raw_fea)\n\n    c_fea = crop_fea(mid_fea, p.center, c_size)\n    r_fea = interpolate(c_fea, original_size)\n    f_fea = CLIP.final_layer(r_fea)\n    fea_crops.append(f_fea)\ncom_fea = concatenate(raw_crops + fea_crops)\n```\n\n**Q5:** About negative impact of DINO.\n\n**A5:** Thanks. Please refer to **A2**.\n\n**Q6:** Comparison with other weighted methods.\n\n**A6:** Thanks. FILIP [1] proposes a token-level matching approach for VLM pretraining. However, since its pretrained model is not publicly available, we cannot directly apply its token-level matching to VLMs that were not pretrained with this method. As an alternative, we experiment with the weight-based matching approach proposed in WCA and common entropy weighted methods and compared them with our soft matching. The results demonstrate the superior performance of our soft matching, indicating its enhanced capability to select more semantically aligned image-text pairs for effective matching.\n\n||ImageNet|DTD|ImageNetV2\n|-|-|-|-\n|Ours w/ WCA weighted|71.22|53.36|65.23\n|Ours w/ entropy weighted|70.76|53.12|64.94\n|Ours|**71.92**|**54.26**|**66.19**", "claims_and_evidence": "Yes,  the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No, because this paper doesn't has proofs for theoretical claims.", "experimental_designs_or_analyses": "Yes, I have reviewed all the experiments mentioned in the Section 4. I have identified the following issues:\n1. The experimental details in Table 3 are not described clearly enough. For example, it is not clear how many shots are specifically used, or whether the whole data from ImageNet are used for fine-tuning. Besides, fine-tuning the model with the data from ImageNet itself may actually reduce the generalization ability of the model on other distributions of ImageNet, because the fine-tuning process may causes the model to overfit the distribution of the original ImageNet. Therefore, the baselines that this method needs to compare with should be models fine-tuned using the few-shot datasets from target datasets like ImageNet-V2 and ImageNet-R.", "relation_to_broader_scientific_literature": "The method proposed in this paper can help VLMs improve their performance in tasks such as image recognition.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strengths:**\n1. The method proposed in this paper is simple, effective, and intuitive.\n2. The paper is clearly written and quite readable.\n\n**Weaknesses:**\n\nI have two main concerns regarding the ABS method proposed in this paper:\n1. The ABS method strongly depends on the quality and accuracy of the Attention Map generated by DINO. Therefore, when the attention map of DINO is of poor quality (for example, it focuses on the background or misidentifies objects), this method may have a negative impact on the object classification accuracy of the model.\n2. There is a significant increase in inference time overhead. For single - image classification, this method changes the number of images for feature extraction from 1 to 2N, thus significantly reducing the corresponding inference speed. This situation is also shown in Table 8. Although Table 8 indicates that the performance is greatly improved compared to the original CLIP, I think ABS should not only compare it with the original CLIP but also with methods like CuPL. Since these methods also only need to infer one image, the performance improvement compared to them may not be significant enough to ignore the growing reasoning overhead.", "other_comments_or_suggestions": "NA.", "questions_for_authors": "1. What is the dimension of $F_{mid}$ in Equation 7? The specific process of Feature Map interpolation could be described in more detail.\n2. How can we maximize the avoidance of the negative impact of inaccurate attention maps of DINO on the method?\n3. Soft Matching essentially weights the image-text similarities of different images and texts. The goal of this weighting is to increase the relative ratio of the image-text similarities between similar and dissimilar image-text pairs. Have the authors tried other weighting methods, such as the approach similar to cross-modal late interaction in FILIP [1]? It would be better if there were some exploratory experiments in this regard. \n\n[1] Yao, Lewei, et al. \"FILIP: Fine-grained Interactive Language-Image Pre-Training.\" International Conference on Learning Representations.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes Attention-Based Selection (ABS), a training-free approach for enhancing zero-shot classification by leveraging attention maps from models like DINO to guide cropping in both raw image and feature space. Combined with a soft matching mechanism to filter LLM-generated descriptions, ABS aims to mitigate the randomness of standard cropping and better integrate local object details with global semantic context. The authors demonstrate state-of-the-art results on several zero-shot and out-of-distribution benchmarks across various VLM backbones. While initial concerns were raised about the incremental nature of the contribution and the effectiveness of individual components, the authors' rebuttal provided convincing additional evidence and clarifications, leading one reviewer to raise their score.

**Strengths:**

* **Novelty & Effectiveness:** The core idea of using attention for dual-space (raw image + feature) cropping combined with soft matching is intuitive and demonstrates strong empirical performance, achieving SOTA results on multiple benchmarks and improving over strong baselines like WCA and CuPL (All Reviewers).
* **Broad Applicability & Robustness:** The method shows effectiveness across various VLM backbones (CLIP, Blip-2, ALIGN, AltCLIP, GroupViT) and exhibits robustness to hyperparameter choices (bJ6A, dz4u, C2Mc).
* **Experimental Rigor:** The paper is supported by extensive experiments, thorough ablation studies validating component contributions, and clear visualizations (C2Mc, bJ6A, iUpr).
* **Clarity & Motivation:** The paper is generally well-written with clear motivation for the proposed approach (bJ6A, dz4u, iUpr).

**Concerns:**

* **Contribution Scope:** Initial concerns about the contribution being incremental over WCA (C2Mc, bJ6A). Authors argued for the novelty of the systematic methodology and dual-space cropping; C2Mc accepted this post-rebuttal, bJ6A acknowledged.
* **Inference Overhead:** The method introduces significant inference time cost due to multiple crops, although authors argue performance gains justify it and cost can be mitigated (iUpr; acknowledged rebuttal).
* **Reliance on Attention Quality:** Performance potentially depends on the quality of the DINO attention maps (iUpr; authors discussed mitigations, reviewer acknowledged).
* **Dataset-Specific Performance:** Initial weaker performance on Food101 was noted (C2Mc; authors provided plausible explanation related to dataset annotation ambiguity, C2Mc accepted).
* **Minor Issues:** Formatting, citation, and clarity issues were noted (All Reviewers; authors promised fixes, reviewers acknowledged).
Decision: Accept (poster)