Enhancing Pre-trained ViTs for Downstream Task Adaptation: A Locality-Aware Prompt Learning Method

Authors: shaokun  wang
yifan  yu
yihong  gong
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 10:12:31

Abstract: Vision Transformers (ViTs) excel in extracting global information from image patches. However, their inherent limitation lies in effectively extracting information within local regions, hindering their applicability and performance. Particularly, fully supervised pre-trained ViTs, such as Vanilla ViT and CLIP, face the challenge of locality vanishing when adapting to downstream tasks. To address this, we introduce a novel LOcality-aware pRompt lEarning (LORE) method, aiming to improve the adaptation of pre-trained ViTs to downstream tasks. LORE integrates a data-driven Black Box module (i.e.,a pre-trained ViT encoder) with a knowledge-driven White Box module. The White Box module is a locality-aware prompt learning mechanism to compensate for ViTs’ deficiency in incorporating local information. More specifically, it begins with the design of a Locality Interaction Network (LIN), which treats an image as a neighbor graph and employs graph convolution operations to enhance local relationships among image patches. Subsequently, a Knowledge-Locality Attention (KLA) mechanism is proposed to capture critical local regions from images, learning Knowledge-Locality (K-L) prototypes utilizing relevant semantic knowledge. Afterwards, K-L prototypes guide the training of a Prompt Generator (PG) to generate locality-aware prompts for images. The locality-aware prompts, aggregating crucial local information, serve as additional input for our Black Box module. Combining pre-trained ViTs with
our locality-aware prompt learning mechanism, our Black-White Box model enables the capture of both global and local information, facilitating effective downstream task adaptation. Experimental evaluations across four downstream tasks demonstrate the effectiveness and superiority of our LORE.

Main paper: 
 introduction: 
 In recent years, Vision Transformers [7, 39] (ViTs) have achieved significant progress and become the mainstream of computer vision. It demonstrates strong performance on various vision tasks, including image classification [14] , image retrieval [43] , instance segmentation [6] , etc. Despite its promising progress, researchers have recently identified a fundamental limitation of the ViT models, i.e., though excelling at extracting global information of the image patches, ViTs are inferior in extracting information within local regions [21, 35] . This deficiency in incorporating local information can be attributed to the inherent design of the architecture, which prioritizes holistic patterns and structures while neglecting the intricate details prevalent in foreground local regions. Specifically, for a fully-supervised pre-trained ViT 1 model such as Vanilla ViT [38] and CLIP [34] , it suffers from a locality vanishing problem when adapting to downstream tasks. As shown in Fig. 1 (a) , when employing a pre-trained ViT (e.g., CLIP) to extract features for several downstream task images, the majority of high-attention-weight tokens 1) are sparsely distributed in the background rather than in the foreground object region, and 2) exhibit similar and lowinformation semantics. This reveals that when pre-trained ViTs encounter unknown downstream task images, they primarily pay attention to global information while neglecting local information in crucial regions. Consequently, the applicability and performance of the pre-trained ViTs may be compromised when adapting a pre-trained model to downstream tasks. There are methods [8, 27, 44, 48] proposed to improve the ViTs by extracting local information of the foreground objects. However, they are train-from-scratch designed, thus requiring substantial computational resources and time for deployment in downstream tasks. To overcome this deficiency, an effective strategy involves adopting Parameter-Efficient-Tuning (PET) techniques to adapt pre-trained ViTs to downstream tasks. The PET methods, such as visual prompt learning (VPL) [14, 16, 52, 53] and adapter tuning [10, 49] , train only a small number of additional parameters for downstream tasks, while keeping all pre-trained ViT parameters frozen. Efficient as they are, these methods are data-driven and regard the pre-trained model as a Black Box model [40] . The inherent opacity and agnosticism of these methods present a significant challenge in comprehending the internal workings and interpreting the additional parameters of PET.Different from the data-driven Black Box model mentioned above, White Box models [11, 17] are knowledge-driven, characterized by explicit rules and logic, facilitating interpretability of their internal workings and logical rules, while performing inferior in complex scenarios. Recent advances in a Black-White Box model theory [28, 37] suggest that combining Black Box and White Box models to construct a unified Black-White Box model provides a promising approach to enhance both the performance and interpretability of Black Box models.Motivated by the Black-White Box model theory, we introduce a novel LOcality-aware pRompt lEarning method (LORE) aimed at enhancing the adaptation of pre-trained ViTs to downstream tasks. Our LORE consists of i) a data-driven Black Box module: i.e.,a pre-trained ViT encoder, and ii) a knowledge-driven White Box To demonstrate the effectiveness and superiority of our proposed LORE method, we conduct comprehensive experiments on a total number of 16 benchmark datasets on 4 different downstream tasks, including image classification, image retrieval, point correspondence, and video object segmentation. On the task of image classification, the LORE steadily and significantly outperforms the existing state-of-the-art (SOTA) methods on 12 benchmarks. Besides, experiments on the other three tasks demonstrate the generality of our LORE. Ablation studies further demonstrate the effectiveness of the proposed components. In summary, the main contributions include:• We propose a novel LOcality-aware pRompt lEarning method (LORE) consisting of a data-driven Black Box module and a knowledge-driven White Box module for downstream task adaptation. Motivated by the success of prompt learning for pre-trained language models in the NLP field [18, 24, 51] , investigating prompt learning for pre-trained vision models has emerged as a prominent research area. Visual prompt learning (VPL) aims to fine-tune only a small number of task-specific parameters while freezing the entire pre-trained model [14, 23] . In comparison to alternative fine-tuning strategies (e.g.,Full Fine-tuning, Adapter Tuning [10, 36, 49] , and Prefix Tuning [20] ), VPL achieves remarkable performance and substantially reduces per-task storage requirements. Today, VPL is mainly used to adapt vision-only models [14, 42, 50] and Vision-Language Models (VLMs) [2, 15, 16, 29, 45, [52] [53] [54] to downstream tasks. Specifically, VPT [14] is the first to adopt prompt learning for pre-trained vision models. It investigates the applicability and viability of VPL and opens up an innovative avenue for downstream task adaptation. CoOp [53] proves that learnable prompts (i.e.,continuous prompts) perform better than hand-crafted prompts (i.e.,discrete prompts, like "a photo of an apple. ") in terms of performance and robustness of downstream task adaptation. Maple [16] proposes multi-modal prompts to improve alignment between vision and language representations in VLMs. ProGrad [54] presents a prompt-aligned gradient method for downstream task adaptation. These methods design learnable prompts instead of hand-craft prompts and show the superiority of learnable prompts. However, it is difficult to interpret what the learnable prompts mean and how they help pre-trained models adapt to downstream tasks. In this paper, we attempt to provide a more interpretable perspective on VPL. Our locality-aware prompt learning mechanism is designed to generate locality-aware prompts, aggregating crucial local information and thus addressing the deficiency in pre-trained ViTs' capacity to incorporate local information for downstream tasks. 
vits with locality mechanism: 
 ViTs rely on self-attention mechanisms to extract global information among image patches [7, 21, 35, 39] . However, the lack of a locality mechanism (e.g.,the convolutions in CNNs) makes it difficult to capture critical local regions in foreground objects, especially in downstream tasks. The aforementioned issue restricts pre-trained ViTs' ability to adapt to downstream tasks [25] . To improve the capacity to extract local information, recent studies [9, 13, 26, 27, 31, 48] have concentrated on tokenization techniques and self-attention mechanisms. Slide-Transformer [31] proposes the slide attention for local relationship modeling. T2T-ViT [48] proposes a progressive tokenization strategy that can better encode the critical local structure for image patches. Swin Transformer [27] uses a shifted windowing scheme to provide better cross-window connections within local windows. Additionally, there has been a trend of designing hybrid architectures [8, 22, 44, 46, 47] of convolutional layers and self-attention layers in a way that local mechanisms are introduced to ViTs. For instance, ConViT [8] and CvT [44] bring locality to ViTs by adding convolutions within the transformer blocks. However, these methods are train-from-scratch designed.In light of this limitation, our LORE is intended to implement a locality-aware prompt learning mechanism for pre-trained ViTs. 
methodology: 
 As shown in Fig. We formulate our Black Box module as follows. The 𝐹 is a pretrained ViT encoder with 𝐿 transformer layers. Given an input of image X, the image is reshaped to 𝑀 flattened 2D patches. These patches are then projected into image patch tokensE 0 ∈ R 𝑀 ×𝑑 𝑒 ,where 𝑀 is the token length and 𝑑 𝑒 is the dimension of each patch token. Furthermore, patch tokens E 0 and a CLS token C 0 are fed into the 𝐹 . Formally, for the 𝑖-th transformer block 𝐹 𝑖 :EQUATIONwhere 𝑖 = 1, 2, ..., 𝐿. Notably, all parameters of the Black Box module are frozen in our method. Taking the E 0 as input, we construct a directed neighbor graph G = (V, E) for each image, where V is the node set and E is the edge set. More specifically, 𝑀 patch tokens of an image E 0 = [e 1 , e 2 , ..., e 𝑀 ] are defined as the node set V. Each node indicates an image patch token e 𝑖 ∈ R 1×𝑑 𝑒 . For a node e 𝑖 , we establish a directed edge originating from node e 𝑗 to e 𝑖 when e 𝑗 is identified as one of the TopK nearest neighbors of e 𝑖 , denoted as e 𝑗 ∈ N (e 𝑖 ). Obviously, this graph G represents local relationships between image patch tokens, which can be used as a prior to help characterize local regions. Afterwards, we use the max-relative graph convolution Θ [12, 19] to enhance the locality interaction between an image patch token e 𝑖 and its neighbors N (e 𝑖 ). In this way, the E 0 can be updated to E ′ ∈ R 𝑀 ×𝑑 𝑒 :EQUATIONEQUATIONwhere 𝑚𝑎𝑥 (•) is a max-pooling feature aggregator to pool the difference of features between e 𝑖 and its neighbors. Eq. ( 3) further details the operation of Θ from the perspective of an image patch token e 𝑖 . And then, we design a residual connect module ℎ(•, •) to alleviate over-smoothing problem [5] of the max-relative graph convolution. Finally, the locality-enhanced tokens Ê ∈ R 𝑀 ×𝑑 𝑒 of an image is defined as: where 𝜎 is the relu non-linearity, W 1 ∈ R 𝑑 𝑒 ×𝑑 𝑒 and W 2 ∈ R 𝑑 𝑒 ×𝑑 𝑒 are fully-connected layers. Ê enhances the local relationships between image patches in comparison to E 0 .EQUATIONAttention. Ê includes the entire regions of an image. There are still a few local regions in Ê that are unimportant in terms of representing the image's salient characteristics. To this end, we introduce the KLA, guided by relevant semantic knowledge, which helps in capturing important local regions of Ê while discarding unimportant ones.We formulate the problem of capturing important local regions as a semantic knowledge-locality matching between the semantic knowledge embedding q and the locality-enhanced tokens Ê. More specifically, we encode auxiliary prompts into semantic embeddings q utilizing a semantic knowledge encoder denoted as 𝑆, represented as q = 𝑆 (Prompt). These auxiliary prompts contain relevant semantic knowledge corresponding to the foreground objects in images, which are handcrafted instructions like "This is a photo of a [CLASS]. ". In KLA, we use q as the input of the Q (query) and Ê as the input of the K (key) and V (value). Then we apply linear transformations to generate Q, K, and V, respectively:EQUATIONwhereW q ∈ R 𝑑 𝑞 ×𝑑 𝑒 , W k ∈ R 𝑑 𝑒 ×𝑑 𝑒 , W v ∈ R 𝑑 𝑒 ×𝑑 𝑒 , Q ∈ R 1×𝑑 𝑒 , K ∈ R 𝑀 ×𝑑 𝑒 ,and V ∈ R 𝑀 ×𝑑 𝑒 . Furthermore, we define an 𝐴𝑇𝑇 𝑁 (•) function to match and aggregate the crucial local regions into A:EQUATIONwhere √ 𝑑 𝑒 is a scaling factor. The 𝜑 𝑁 (•) denotes a row-wise top-N filter, which sets the top-N values unchanged and the rest to 0, aiming to ensure that 𝐴𝑇𝑇 𝑁 (•) focuses on the most relevant locality-enhanced tokens. For simplicity, the concept of multiple heads here is omitted. In Eq.( 6), matrix product of QK ⊤ ∈ R 1×𝑀 represents the similarity between the semantic embedding q and all locality-enhanced tokens Ê of an image. Finally, a multi-layer perceptron (𝑀𝐿𝑃) and a residual connection are adopted to enhance the representation ability of A. The Knowledge-Locality (K-L) prototype Â is formulated as:EQUATIONwhere 𝐵𝑁 indicates the batch normalization operation. This K-L prototype Â contains critical local information about an image.Taking the K-L prototype Â as input, a straightforward method for generating locality-aware prompt U is to directly feed Â into the PG. However, this solution necessitates the model to compute the corresponding K-L prototype for each image during inference, increasing computation complexity and requiring auxiliary prompts for the prototype generation, which is impractical for real-world applications. To overcome this limitation, we develop a K-L prototype-guided constraint L 𝑘𝑝 . It facilitates PG training with the use of the K-L prototypes Â, wherein K-L prototypes are not necessary for the inference phase. We firstly formulate the architecture of the PG and then describe the details of L 𝑘𝑝 .PG has two lightweight bottleneck architectures. These bottleneck architectures reduce the number of parameters while ensuring effectiveness. Using the first bottleneck architecture Ψ 1 parameterized by 𝜃 1 , we map the Ê to a latent space. We define a latent embedding Y ∈ R 1×𝑑 𝑒 as:EQUATIONwhere 𝑎𝑣𝑝 (•) is the average pooling operation and Ψ 1 consists of two 1 × 1 convolutions. Initially, it reduces the feature dimension to 𝑟 , followed by an expansion of the dimension back to 𝑑 𝑒 . 𝑟 ≪ 𝑑 𝑒 . Afterwards, the second bottleneck architecture Ψ 2 parameterized by 𝜃 2 is used to generate the locality-aware prompt U:EQUATIONΨ 2 comprises two 1×1 convolution operations that initially decrease the feature dimension to 𝑟 and then expand the dimension back to 𝑑 𝑒 . The normalization operations are omitted here. K-L prototype-guided constraint L 𝑘𝑝 optimizes the training objective of the PG. More specifically, we first introduce the positive and negative K-L prototypes. For a downstream task image X with a true label 𝑡, we can obtain a predicted label 𝑝 using our Black Box module. 𝑝 = 𝑡 denotes a correct prediction, whereas 𝑝 ≠ 𝑡 denotes a wrong prediction. When 𝑝 ≠ 𝑡, we define the positive / negative prompt like "This is a photo of a [CLASS 𝑡 / CLASS 𝑝].". For instance, for a "dog" image of the downstream task, the B-B module incorrectly predicts it as a "cat". We verbalize the wrong prediction as a negative prompt, such as "This is a photo of a cat", which demonstrates the B-B's wrong understanding of this image. We hope to correct the wrong understanding during PG's training phase. To this end, we compute the corresponding positive/negative K-L prototype Â𝑡 / Â𝑝 using Eq. ( 5) and (6) . Â𝑝 is the negative K-L prototype that contains misleading local information about the image. And then, we define L 𝑘𝑝 to guide the latent embedding Y that are similar to Â𝑡 but not similar to Â𝑝 in a joint latent space:EQUATIONwhereI(•) is the indicator function. If 𝑡 ≠ 𝑝 is true, I(𝑡 ≠ 𝑝) is 1. Otherwise, I(𝑡 ≠ 𝑝) is 0. 𝑐𝑜𝑠 (•, •)is the cosine similarity.During the training phase, the overall training loss of our model is formulated as:EQUATIONwhere L 𝑡𝑎𝑠𝑘 is the downstream task loss. L 𝑘𝑝 is used to optimize the training of PG, and we describe the details of L 𝑘𝑝 in Eq. (10) . 𝜆 indicates the hyper-parameter to balance the contributions of the two losses. The LIN, PG, and KLA are trainable.During the inference phase, we first compute the locality-enhanced tokens Ê for an image by using Eq. ( 2), (3), and (4). And then, Eq. ( 8) and ( 9) are used to generate the locality-aware prompt U. Finally, U is used as additional input tokens containing the critical local information, fed into our pre-trained ViT along with image patch tokens. The 𝑖-th transformer block 𝐹 𝑖 in our model is formulated as:EQUATIONwhere 𝑖 = 1, 2, ..., 𝐿. The outputs of 𝐹 𝐿 are utilized to get downstream task results. 
experiments: 
 In the following part of this section, we first provided the experimental setup, then presented the evaluation results and visualization results, and finally showed ablation studies. For the point correspondences task, we reported results on SPair-71k dataset. For the video object segmentation task, we conducted experiments on DAVIS 2017 dataset. Implementation Details. We implemented our LORE2 in Py-Torch with two NVIDIA RTX 3090 GPUs. We adopted the visual encoder of pre-trained CLIP ViT-B/16 [34] , pre-trained CLIP ViT-B/32 [34] , and Vanilla ViT-B/16 pre-trained on ImageNet-21k [38] as our Black Box modules, respectively. We adopted the text encoder of pre-trained CLIP as our semantic knowledge encoder. Our models are trained using the SGD optimizer with a mini-batch size of 32/64. In the few-shot classification task and the first stage of the easy-to-hard classification task, we set the initial learning rate to 0.003 and decreased it to 0.0001 by the cosine annealing rule. In the second stage of the easy-to-hard classification task, we set the initial learning rate to 0.0005 and decrease it to 0.0001 by the cosine annealing rule. We trained our method for 50 epochs. We followed the same data augmentation strategies in our comparison methods. The hyper-parameter 𝜆 is set to 0.1. The length of locality-aware prompts is set to 10. The dimension 𝑟 in Ψ 1 and Ψ 2 is set to 32.4.2.1 Classification. We compared our model with SOTA prompt learning methods ( CoOp [53] , Co-CoOp [52] , Maple [16] , and Pro-Grad [54] ), representative adapter-based method ( Clip-Adapter [10] ), and Zero-shot CLIP [34] . For a fair comparison, we adopted the same pre-trained CLIP as the base model of our comparison methods, and we used the dataset split strategy as the same as our method to reproduce these methods. In this task, LORE's task head is trainable.Few-shot classification. Fine-tuning models with few-shot training samples that are randomly selected, followed by performance evaluation on the total test set. Notably, we used the same training samples in our comparison methods.We compared our LORE with all these methods on 12 datasets with 16-shot and 8-shot settings. As shown in Table 1 , our LORE outperforms the SOTA methods with the 16-shot setting: (1) it has achieved the highest accuracies on all datasets. ( 2) it has achieved the highest average accuracy. LORE surpasses the best method Clip-Adapter by 7.09% in average accuracy. Besides, our LORE also outperforms these methods with the 8-shot setting: (1) it has Easy-to-Hard classification. To test whether our White Box module can improve pre-trained ViTs when encountered with hard samples, we devise a two-stage classification task comprising an easy curriculum stage and a hard curriculum stage. More specifically, we assess sample difficulty based on their distribution in the feature space of our Black Box module. For a sample X belonging to class 𝑗, the difficulty function D is definable using the distance between its feature (i.e.,CLS token) C 𝐿 and its class centroid o 𝑗 :D = 𝑑𝑖𝑠 (C 𝐿 , o 𝑗 ), where 𝑑𝑖𝑠 (•, •) is the cosine distance. A lower difficulty score indicates that X is closer to its class centroid o 𝑗 and is thus considered an easy sample, suitable for initial adaptation to downstream tasks. Conversely, a higher difficulty score suggests that X is farther from its class centroid o 𝑗 and is more likely to be misclassified, making it a suitable candidate for further adaptation of the model to downstream tasks. In the easy curriculum stage, we select the 𝑁 easiest samples to train the LORE-e. Subsequently, in the hard curriculum stage, we select the 𝑁 hardest samples based on the LORE-e. These selected samples are then utilized to further train the LORE-h. Importantly, both LORE-e and LORE-h share an identical framework. It is worth noting that the training samples of each class are given 16/stage and 8/stage, respectively. We did not test the 16/stage setting on Flowers102, StanfordCars, DTD, and UCF101. Because the number of training samples in the 4 datasets is not insufficient to select 16 qualified hard samples. According to Table 3 , our method has shown obvious advantages over the SOTA methods: (1) our LORE-h has obtained the highest accuracies on all datasets and the highest average accuracies for the 16/stage and 8/stage settings; (2) our LORE-e has achieved the highest average accuracies for the 16/stage and 8/stage settings; (3) our LORE-e has achieved the highest accuracies or compatible one on all datasets for the two settings; (4) our LORE-h outperforms our LORE-e by 1.66% and 2.04% for both settings, respectively, which is the most significant improvement compared to SOTA methods. In conclusion, our LORE-h outperforms our LORE-e, demonstrating the superiority of our locality-aware prompt learning mechanism in improving the performance of our Black Box module when encountered with difficult samples.To further verify the effectiveness of our Black-White Box model in improving pre-trained ViTs, we conduct experiments on other downstream tasks. Our White Box module, pre-trained on the classification task, Table 3 : Comparison with the SOTA methods on 12 datasets under the easy-to-hard classification task, where the Black Box module is CLIP ViT-B/16. 'xxx-e' and 'xxx-h' represent being trained only in the easy curriculum stage and being trained in the easy curriculum stage followed by the hard curriculum stage, respectively. The highest accuracy in the easy/hard curriculum is indicated with an underline / in bold. Absolute improvements from the easy to hard curriculum are indicated in parentheses. Image Retrieval. We used the evaluation protocols of [3] to compare the performance of off-the-shelf features in our Black-White Box model and the Black Box model. Concretely, the pretrained feature (i.e.,CLS token) is frozen and used directly for image retrieval using the K-NN strategy without any fine-tuning. We reported the Mean Average Precision (mAP) for the Medium (M) and Hard (H) split on ROxford5k and RParis6k datasets [33] . As depicted in Table 4 , our Black-White Box models have clear advantages over Black Box models (i.e.,Vanilla ViT-B/16 and CLIP ViT-B/16).Point Correspondences. In the point correspondences task, given a source image with annotated keypoints, the objective is to predict the locations of corresponding keypoints in a target image. We followed the evaluation protocols outlined in [1, 41] . More specifically, image patch tokens are mapped to the size of the original image by bi-linear interpolation. We compute the key point features in the source image and match the mutual nearest neighbors in the target image. In Table 4 , we reported the Percentage of Correct Keypoint (PCK) on SPair-71k [30] dataset. Our Black-White Box models outperform their Black Box counterparts.Video Object Segmentation. We conducted experiments on DAVIS 2017 [32] for the task of video object segmentation. It is a semi-supervised task that aims to propagate the first frame's segmentation mask to subsequent frames. We followed the evaluation protocols outlined in [4] . Image patch tokens of video frames are extracted to segment scenes without any fine-tuning. As illustrated in Table 4 , we reported the mean region similarity and contourbased accuracy (J&F M) of our Black-White Box models and the corresponding Black Box models. We can observe that our Black-White Box models perform better in this dense recognition task.We reported visualization results to make intuitive explanations for our method. Specifically, we visualized the self-attention maps generated by our Black Box module and our Black-White Box model. These self-attention maps indicate the self-attention of the CLS token and other image tokens across the heads of the last ViT layer. Additionally, we visualized the attention map of our KLA mechanism, which illustrates the attention between the semantic knowledge embedding q and the locality-enhanced tokens Ê. As shown in Fig. 3 We designed ablation studies to further investigate the effectiveness of our method. Ablation studies are conducted on the 16-shot classification task. The Black Box modules here are Vanilla ViT-B/16 and CLIP ViT-B/16. First, we removed the White Box module and adopted visual prompt tuning [14] to fine-tune the Black Box module (i.e.,"B-B with VPT"), which is a commonly used prompt learning method. Second, we removed the K-L prototype-guided constraint L 𝑘𝑝 to obtain another baseline "LORE w/o L 𝑘𝑝 ". Notably, our KLA is also removed in "LORE w/o L 𝑘𝑝 ". As shown in Table 5 , when using the Vanilla ViT-B/16 as the Black Box module, our method achieves the best performance compared with "LORE w/o L 𝑘𝑝 " and "B-B + VPT", leading to improvements of 1.14% and 2.37%, respectively. When using the CLIP ViT-B/16 as the Black Box module, LORE outperforms the two baselines by 1.03% and 4.28%, respectively. These results demonstrate that the White Box module helps adapt pre-trained ViTs to downstream tasks. 
conclusions and future works: 
 In this work, we propose a locality-aware prompt learning method for downstream task adaptation. Specifically, we utilize the pretrained ViT encoder as our Black Box module, while designing the locality-aware prompt learning mechanism, referred to as the White Box module. In our White Box module, the LIN, KLA, and PG collaborate to generate the locality-aware prompts, which can enhance the local information incorporating capacity of the Black Box module. We showcase LORE's superiority and effectiveness on 4 downstream tasks. A promising future research direction is utilizing rules from external knowledge graphs to develop more interpretable White Box modules. 
 
 Appendix: The comprehensive statistics of the classification datasets are presented in Table 6 . For StanfordDogs, CIFAR-10, CIFAR-100, DTD, ImageNet, Resisc45, and Pattern, we followed the official dataset split strategy. For Flowers102, StanfordCars, Aircraft, EuroSAT, and UCF, we followed the split strategy used in CoOp.For image retrieval datasets, ROxford5k and RParis6k contain 4,993 and 6,322 high-resolution (1024 × 768) images, respectively, and each dataset has 70 queries from 11 landmarks.For point correspondences dataset, SPair-71k comprises 70,958 image pairs from 18 classes with diverse variations in viewpoint and scale, of which 53340 pairs serve as the training set, 5384 pairs serve as the validation set, and 12234 pairs serve as the test set.For video object segmentation dataset, DAVIS consists of 50 video sequences with 3455 densely annotated frames in pixel level. 30 videos with 2079 frames are for training, and 20 videos with 1376 frames are for validation.As shown in Table 7 , the number of trainable parameters, GFLOPs, and FPS of our LORE during the inference phase remain comparable to those of "B-B" and "B-B+VPT". Moreover, our LORE significantly improves performance on downstream tasks and offers an interpretable perspective for visual prompt learning. Therefore, we believe these trade-offs are justified. Our LORE aims to enhance the adaptation of pre-trained ViTs to downstream tasks. Therefore, the primary objective of our experimental design is to validate the effectiveness of pre-trained ViTs in adapting to downstream tasks. To ensure comprehensive evaluations, we compared our LORE with several effective and representative PET methods, including CoOp, Co-CoOp, Maple, ProGrad, Clipadapter, and VPT. Some of these methods have demonstrated outstanding performance in domain generalization and cross-dataset transfer evaluations using pre-trained CLIP models. However, it is important to note that domain generalization and cross-dataset transfer evaluations assess the CLIP-based model's generalization ability, which is beyond the scope of this study. We would like to further investigate the generalization ability problem in future work.The code is available at https://github.com/Mysteriousplayer/KGPT. 


Decision: The paper initially got four consistent positive scores. The authors have provided a rebuttal. After checking the rebuttal, three of the reviewers are satisfied with it and all gave weak accept. However, the Reviewer 7C63 reduced the score from borderline accept to borderline reject due to lack of compared the generalization ability with related works. After carefully checked the paper and the rebuttal, the AC thinks this paper mainly focuses on the adaptation to downstream tasks and thus it is fine that missing the discussion about the generalization ability. The AC thus thinks this paper can be accepted as an effective task adaptation approach but strongly encourages the authors to provide the comparisons on generalization ability and cross-dataset evaluation, which can better reflect the effectiveness or drawback of the proposed method.

Reviews: review: paper_summary: paper_summary: This article introduces a test-time augmentation method named LORE, designed to optimize Vision Transformer (ViT) models. The LORE method effectively mitigates the locality vanishing problem commonly observed in ViT models when processing downstream datasets. The study conducts experimental evaluations on multiple downstream tasks and compares the results with various state-of-the-art (SOTA) methods, thereby demonstrating the effectiveness of the LORE method.
strength_weakness: strength_weakness: [+] The LORE method uniquely employs a combination of white-box and black-box approaches to alleviate the locality vanishing problem. The design principles of integrating a data-driven black box with a knowledge-driven white box are particularly intriguing.

[+] The authors conducted extensive experiments across various downstream tasks and datasets to demonstrate the effectiveness of their method. These tasks include few-shot image classification, easy-to-hard classification, image retrieval, point correspondences, and video object segmentation. The experimental results are quite compelling.

[+] The visualization of the model's attention effectively demonstrates the impact of LORE, supporting the authors' claimed innovations.
limitations: limitations: [-] The increase in parameter count and computational overhead introduced by the new module needs to be detailed. The authors should clarify the computational cost of the LORE module and analyze its impact on the model's inference speed.

[-] Attention should be paid to writing issues. For instance, references should be numbered according to their order of appearance. The presentation of Figure 2 is somewhat convoluted and might cause misunderstandings.

[-] The explanations of the equations are somewhat obscure. The equation descriptions in section 3.2.1 should include more detailed explanations, and the readability of the equations needs to be further improved to avoid any potential misunderstandings.

I'll raise my score if the authors can address my above concerns.
paper_summary: paper_summary: The paper addresses the limitation of Vision Transformers (ViTs) in extracting local information, which hampers their performance in downstream tasks. It introduces a LOcality-aware pRompt lEarning (LORE) method that integrates a data-driven Black Box (pre-trained ViT encoder) with a knowledge-driven White Box (locality-aware prompt learning mechanism). This includes a Locality Interaction Network (LIN) and Knowledge-Locality Attention (KLA) mechanism to enhance local relationships and capture critical local regions. By combining global and local information, LORE improves ViTs' performance across various downstream tasks.
strength_weakness: strength_weakness: - Overall the paper is a solid one. The method looks integral and makes sense. 
- The visualization results demonstrate that the KLA mechanism effectively captures local information.
- The authors provide sufficient experiment and ablation studies to demonstrate the effectiveness of the proposed method.
- This paper is clear and well-written.
limitations: limitations: - Could you provide clarification on the process for generating negative prompts that demonstrate the Black Box module’s wrong understanding of images?
- Seems like the primary contribution comes from W-B. It is OK but somewhat limited in terms of novelty.
- Although K-L prototypes are not necessary for the inference phase, could you explain the extent to which the inference cost is increased by this method?
- Fig. 1 (c) might be better presented as a separate figure. Its connection to the rest of Figure 1 is not as strong as the connections among the other parts.
paper_summary: paper_summary: This paper proposes a method, LOcality-aware pRompt lEarning (LORE), aiming to address the challenge of locality vanishing when adapting pre-trained ViTs to downstream tasks. LORE integrates a data-driven Black Box module (i.e.,a pre-trained ViT encoder) with a knowledge-driven White Box module which is a locality-aware prompt learning mechanism to compensate for ViTs’deficiency in incorporating local information. The results are good and comprehensive, which validates the efficacy of the proposed method.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to follow, the idea is intuitive and clear. Visualizations properly demonstrate the concept of the proposed framework.
2. The performances improvement of the proposed method are impressed.
limitations: limitations: 1. It would be better to add domain generalization experiments to verify the robustness of the proposed method against distribution shifts, as demonstrated in ProGrad[1] and Maple[2].
2. It would be better to add cross-dataset transfer evaluation, as demonstrated in ProGrad[1] and Maple[2].
3. What impact do the hyper-parameter $\lambda$ and the dimension $r$ have on the performance of the proposed method? It would be better to provide a detailed explanation.
---
[1] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. 2023. Prompt-aligned Gradient for Prompt Tuning. In International Conference on Computer Vision.

[2] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. 2023. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19113–19122.
paper_summary: paper_summary: The manuscript introduces a method called LORE, which leverages a knowledge-driven white box module to enhance local information for Vision Transformers (ViTs). In particular, the white box module introduces Knowledge-Locality Attention (KLA) to improve the local connections between image patches, utilizes Knowledge-Locality Attention (KLA) to identify important features based on the prompt, and incorporates a Prompt Generator (PG) to create the prompt, which is guided by knowledge locality matching. Comprehensive experiments illustrate the effectiveness of the LORE.
strength_weakness: strength_weakness: - Novelty and technical correctness: The method addresses the issue of insufficient local information in ViTs by enhancing local relationships and generating prompts explicitly.
- Comprehensive evaluation: The datasets and tasks used for evaluation are appropriate, and both quantitative and qualitative results confirm the effectiveness of the proposed method.
- Clarity: The writing and logic are clear, with the authors providing explanations for the motivation behind each module.
- Practical applications: The method is valuable for the implementation of ViTs in various downstream tasks.
limitations: limitations: 1. Figure 2 is a bit confusing, It's difficult to differentiate different modules. The authors should consider adjusting the font size and color to enhance the clarity of the framework. Besides, the framework should show the training and inference process.
2. Table 5 lacks of ablation study on the Locality Interaction Network and Prompt Generator.