KNN Transformer with Pyramid Prompts for Few-Shot Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-28 12:46:16

Abstract: Few-Shot Learning (FSL) aims to recognize new classes with limited labeled data. Recent studies have attempted to address the challenge of rare samples with textual prompts to modulate visual features. However, they usually struggle to capture complex semantic relationships between textual and visual features. Moreover, vanilla self-attention is heavily affected by useless information in images, severely constraining the potential of semantic priors in FSL due to the confusion of numerous irrelevant tokens during interaction. To address these aforementioned issues, a K-NN Transformer with Pyramid Prompts (KTPP) is proposed to select discriminative information with K-NN Context Attention (KCA) and adaptively modulate visual features with Pyramid Cross-modal Prompts (PCP). First, for each token, the KCA only selects the K most relevant tokens to compute the self-attention matrix and incorporates the mean of all tokens as the context prompt to provide the global context in three cascaded stages. As a result, irrelevant tokens can be progressively suppressed. Secondly, pyramid prompts are introduced in the PCP to emphasize visual features via interactions between text-based class-aware prompts and multi-scale visual features. This allows the ViT to dynamically adjust the importance weights of visual features based on rich semantic information at different scales, making models robust to spatial variations. Finally, augmented visual features and class-aware prompts are interacted via the KCA to extract class-specific features. Consequently, our model further enhances noise-free visual representations via deep cross-modal interactions, extracting generalized visual representation in scenarios with few labeled samples. Extensive experiments on four benchmark datasets demonstrate significant gains over the state-of-the-art methods, especially for the 1-shot task with 2.28% improvement on average due to semantically enhanced visual representations.

Main paper: 
 related work 2.1 few-shot learning: 
 In general, methods for Few-Shot Learning can be divided into two types. One is optimization-based methods [16, 42, 49, 74] , aimed at quickly adapting models to new data. The other type is metricbased methods adopted in this work aims to classify by measuring distances between support and query samples within the feature space. ProtoNet [50] uses Euclidean distance. Deepemd [68] utilizes the Earth Mover's Distance. others [12, 21, 36, 64, 71] have integrated Transformer layers as classifiers. Moreover, recent works have incorporated textual modality as auxiliary information to boost generalization on novel classes. AM3 [61] merges semantic and visual prototypes through an adaptive fusion mechanism. Multiple-Semantics [66] aims to enhance prototype representation by integrating multiple semantic information TRAML [31] leverages semantic similarities between classes to improve classification accuracy by a task-relevant adaptive margin loss. SVAE [62] generates additional visual features from semantic information through a Conditional Variational Autoencoder (CVAE) [51] , resulting in more reliable prototypes. KTN [45] utilizes graph convolutional networks and knowledge graphs to explicitly leverage semantic information for learning classifiers. These methods apply semantic information at a higher level such as prototypes and classifiers, which lack foundational visual feature interactions and ignore distribution differences. In contrast, we utilize ViT as the backbone, leveraging semantic information in the extraction process to modulate visual features adaptively.To diminish the dependency on extensive training datasets and enhance the flexibility of pre-trained models for downstream tasks, several prompt-based approaches [6, 10, 23, 46, 53, 72, 73] have been proposed. CLIP [46] utilizes constructed prompts like "a photo of a class" to compute similarity scores between image embeddings and textual prompts for classification. CoOp [73] introduces learnable continuous prompt tokens for fine-tuning the pre-trained CLIP. Further building on CoOp, CoCoOp [72] employs a Meta-Net to transform each prompt token into instance-specific contexts, significantly improving model adaptability. MAPLE [23] designs corresponding prompts for each modality to guide the learning of the respective modality. VPT [22] introduces learnable parameters as visual prompts to fine-tune the CLS token of the ViT, enhancing the flexibility and adaptability of the model. Frozen [53] conducts multimodal few-shot learning by using image features paired with text as prompts. SP [6] utilizes textual labeled embedding matching images as prompts to supplement visual features by employing linear layers for direct and single-scale fusion. In this paper, we propose Pyramid Cross-modal Prompts to learn pyramid prompts, which can dynamically adjust the importance weights of visual features based on rich semantic information at different scales, making models robust to spatial variations.Vision Transformers (ViTs) have shown remarkable success in numerous computer vision tasks [2, 5, 26, 34, 37, 43, 59, 70] , benefiting2𝐶 × 𝐻 2 × 𝑊 2 4𝐶 × 𝐻 4 × 𝑊 4"A photo of {class}"CLIP Pyramid Cross-modal Prompts (PCP) from their proficiency to capture long-range dependencies among image patches. To match or surpass the performance of CNNs of similar size trained on ImageNet [11] , ViTs are usually trained on large-scale datasets. Few recent works [4, 13, 20, 58] have explored the adaptation of ViTs on small datasets. IDMM [4] pretrains ViT using instance discrimination loss, followed by fine-tuning on the target dataset. SUN [13] introduces local supervision to enhance dependency learning of tokens. FewTURE [20] also achieves excellent performance with a fully Transformer-based architecture. SP [6] utilizes semantic prompts to enhance the visual feature by crossmodal interactions based on self-attention. Despite these advances, the fully connected self-attention mechanism in these models usually introduces noisy tokens, which can degrade the quality of the representation. KVT [58] attempts to refine this mechanism, yet the performance is limited in scenarios with few labeled samples due to excessive focus on local feature context. In this paper, we propose a 𝑘-NN Context Attention mechanism that progressively filters out irrelevant information in visual modality via three cascaded stages, captures discriminative features in cross-modal interactions guided by semantic information, and enhances global context information in FSL. To the best of our knowledge, we are the first to consider the way of integrating cross-modal data and context prompts for improving 𝑘-NN attention and applying it in FSL.Classification Fusion K V K V K V Q Q Q Query Support 𝑍 t 
method: 
 We first overview the whole pipeline, and then introduce the preliminary of FSL. Next, two novel components of K-NN Context Attention (KCA) and Pyramid Cross-modal Prompts (PCP) are presented.Fig. 2 shows the framework of our proposed KTPP. To suppress irrelevant information, the KCA progressively filters out noisy tokens in three cascaded stages. To exploit rich semantic priors, class-aware prompts are extracted by inputting class names into the CLIP. To capture complex semantic relationships between different modalities, pyramid prompts are obtained to enhance support features by applying the PCP at different scales, enabling the ViT to dynamically adjust the importance weights of support features based on semantic information. Consequently, models are robust to spatial variations. To select discriminative support features, enhanced support features and class-aware prompts are interacted via the KCA, which can extract class-specific support features guided by semantic information. Finally, the ViT can generate generalized class prototypes for classification.Few-Shot Learning (FSL) focuses on generalizing the knowledge learned from training classes 𝐶 train to test classes 𝐶 test with only a few labeled samples, where the training and test categories are disjoint (𝐶 train ∩ 𝐶 test = ∅). This task is typically formalized as an N-way M-shot problem where the training involves N distinct categories and each category has M labeled images. An episodic training strategy [55] is adopted, where each episode consists of two parts: a support set 𝑆 = {(𝑥 𝑖 , 𝑦 𝑖 )} 𝑁 ×𝑀 𝑖=1 with 𝑁 × 𝑀 labeled samples, and a query set 𝑄 = {(𝑥 𝑖 , 𝑦 𝑖 )} 𝑁 ×𝐻 𝑖=1 with 𝑁 × 𝐻 test samples to evaluate the performance.In the FSL approach, prototype-based inference is commonly utilized. Specifically, N class prototypes are obtained by averaging 3 all samples of each class as follows:N-1 N ••• 1 2 3 N-1 N ••• 1 1 1 1 1 ••• N N N N N ••• ••• 1 2 3 N-1 N ••• 1 2 3 N-1 N ••• •••EQUATIONwhere 𝑥 𝑗 ∈ 𝑆 𝑖 . 𝑆 𝑖 denotes the i-th support class. |𝑆 𝑖 | represents the number of samples in each support class. 𝑓 is the feature extractor. For a query image 𝑞 𝑖 , the similarity score between 𝑞 𝑖 and all support classes is determined by calculating the distance function between 𝑞 𝑖 and N class prototypes. The probability of 𝑞 𝑖 belonging to the n-th support class is calculated as follows:EQUATION)where 𝛾 is a temperature parameter. 𝑑 () denotes the distance function. The support class corresponding to the maximum probability is regarded as the classification result.Following [55] , the vanilla self-attention formula is defined as:EQUATIONThe fully connected self-attention mechanism allocates weights to all tokens without distinguishing noise, making ViTs susceptible to noisy tokens when only a limited number of samples are available.To mitigate this issue, we propose a 𝐾-NN Context Attention (KCA) mechanism, which aims to progressively filter out irrelevant tokens in a coarse-fine manner via three cascaded stages. Specifically, the KCA computes the attention matrix 𝐴 ∈ R 𝑛×𝑛 by the dot product of all query-key pairs, like vanilla attention:EQUATION)We further construct a masked attention matrix Â ∈ R 𝑛×𝑛 , selecting the 𝑘 most relevant keys for each query, as shown in Fig. 3 . This is achieved by performing the top-𝑘 operation on each row of the attention matrix to obtain the highest 𝑘 scores and their indexes.In the Â, the selected elements remain unchanged and all others are set to negative infinity. The formulation is defined as follows:EQUATIONwhere 𝐼 ∈ R 𝑛×𝑘 represents the collection of 𝑛 × 𝑘 indexes. Therefore, the KCA naturally forms a sparse attention matrix, concentrating the focus of the ViT on class-relevant features. In the proposed framework, image patches are sequentially fed into three stages to perform the KCA, thereby progressively filtering out irrelevant tokens by selecting similar tokens and refining features in a coarse-fine manner.In vanilla self-attention, attention weights are calculated based on the relationships between the position and all other positions, resulting in 𝑂 (𝑁 2 ) computational complexity for a sequence of length 𝑁 . In contrast, the KCA considers only the 𝐾 nearest neighbors for each position, reducing computational complexity to approximately 𝑂 (𝑁 𝐾), where 𝐾 is much smaller than 𝑁 . This approximation significantly reduces computational costs and maintains ins relatively high performance.However, a potential risk is the over-focus on local features to the detriment of global context. This limits the generalization of the ViT to new classes with few samples. To address this problem, an average representation of the entire tokens is used as the context prompt to capture global information:EQUATIONwhere 𝑧 𝑐𝑝 represents context prompts. Incorporating 𝑧 𝑐𝑝 into each token can enhance global context information.The whole KCA formula can be defined as:EQUATIONwhere 𝛼 is a learnable parameter, allowing the ViT to adaptively adjust the coefficient between local and global context information.The proposed Pyramid Cross-modal Prompts (PCP) leverage semantic embeddings to enhance support representations, allowing the ViT to adaptively modulate support features based on semantic information, capturing complex semantic correlations between textual and visual modalities via deep cross-modal interactions. The key component of PCP is the cross-modal enhancement module which is illustrated in Fig. 4 . It illustrates cross-modal attention to leverage semantic information and generate semantic-based visual features. Specifically, to exploit rich and unbiased semantic information, class-aware prompts 𝑍 𝑐𝑝 𝑡 ∈ R 𝐿 𝑡 ×𝑑 𝑡 from class names are obtained by the CLIP which has an excellent capability of cross-modal feature alignments. Additionally, three support features 𝑍 𝑆𝑖 𝑣 ∈ R 𝐿 𝑣 ×𝑑 𝑣 at different scales are extracted via three stages.Given weight matrices 𝑊 𝑄 𝑡 ∈ R 𝑑 𝑡 ×𝑑 , 𝑊 𝐾 𝑣 ∈ R 𝑑 𝑣 ×𝑑 , and 𝑊 𝑉 𝑣 ∈ R 𝑑 𝑣 ×𝑑 , the query 𝑄, key 𝐾, and value 𝑉 in the cross-modal attention are computed as follows:𝑍 𝑣 𝑆𝑖 ∈ ℝ 𝐿𝑣×𝑑𝑣 𝑊 𝑄𝑡 ∈ ℝ 𝑑𝑡×𝑑 𝑍 t 𝑐𝑝 ∈ ℝ 𝐿𝑡×𝑑𝑡 𝑊 𝐾𝑣 ∈ ℝ 𝑑𝑣×𝑑 𝑊 𝑉𝑣 ∈ ℝ 𝑑𝑣×𝑑 𝑄 ∈ ℝ 𝐿𝑡×𝑑 𝐾 ∈ ℝ 𝐿𝑣×𝑑 𝑉 ∈ ℝ 𝐿𝑣×𝑑 𝑊 ∈ ℝ 𝐿𝑡×𝐿𝑣 𝑍 v→t 𝑓𝑐𝑝𝑖 ∈ ℝ 𝐿𝑡×𝑑EQUATIONNext, the attention weight matrix of textual modality for visual modality is obtained as:EQUATIONwhere 𝑊 𝑣→𝑡 is the attention weight matrix and 𝛽 is a scaling parameter. Due to discrepancies in modality distributions, the initial training phase may yield limited relevance between textual and visual features, resulting in small values within the weight matrix.To enhance the learning of model parameters, we introduce the hyperparameter 𝛽 to scale the matrix before applying softmax.After applying the identical operation to the support features of the remaining two scales, we derive the pyramid prompts 𝑍 𝑓 𝑝𝑝 𝑣→𝑡 , representing the inherent semantic details of visual features chosen by textual features across various levels. Utilizing the multi-scale operation, ViT captures diverse spatial information, thereby enhancing its capability to adapt to spatial variations.The pyramid prompts are linearly mapped to the dimension of the support feature 𝑍 𝑆3 𝑣 for fusion, defined as:EQUATIONwhere 𝑍 𝑆 𝑣 represents the support features emphasized by the pyramid prompts, 𝑁 is typically set to 3, and 𝜆 as a learnable parameter.𝑣 and class-aware prompts 𝑍 𝑐𝑝 𝑡 are interacted in the ViT via the KCA. The output 𝑦 𝑆 𝑣 is calculated as follows:EQUATIONwhere ∥ represents the concatenation operation along the spatial dimension. The KCA enables the ViT to extract discriminative support features due to the semantically guided capture of classspecific features, followed by obtaining more generalized class prototypes for classification via Eq. 1.The similarity scores between the query samples and class prototypes are computed via the cosine distance, defined as follows:EQUATIONwhere cos() denotes the cosine distance function. The classification results of the query image 𝑞 𝑖 is the support class with maximum probability.The ViT is trained by minimizing the Cross-Entropy (CE) loss function, which can be formulated as follows:EQUATIONwhere L CE represents CE loss. 𝑓 () represents the ViT backbone. 𝑞 𝑖 and 𝑦 𝑖 donate a query image and its corresponding ground-truth label. 
experiments: 
 We first present the four benchmark datasets. Next, we introduce the details of the experiments, followed by the few-shot classification performance and the model analysis.The effectiveness of the proposed method is evaluated on four fewshot datasets: miniImageNet [56] , tieredImageNet [47] , CIFAR-FS [30] , and FC100 [44] . MiniImageNet and TieredImageNet are both subsets of ImageNet [11] . Backbone. In all experimental setups, we employ Visformer-T [8] as the feature extractor. The Visformer-T model boasts a more small number of parameters, rendering our model smaller compared to Resnet-12 models [48, 50, 68] , or even several times smaller than the ViT-S/16 and WRN-28 models [20, 49, 57] . KCA Block consists of two Batchnorm layers, a two-layer MLP, and Multi-Head KCA. Furthermore, we employ ViT-B/32 CLIP as a text encoder, extracting an output dimension of 512.Training Details. Our approach follows a two-phase training procedure, similar to traditional frameworks such as Meta-baseline [7] , including pre-training and meta-tuning stages. Throughout both phases, KCA serves as the attention mechanism for the ViT. Moreover, PCP is employed in the meta-tuning phase. For input [39] with a learning rate of 5e-4. During the pre-training phase, the number of training epochs is set to 500 for the mini-ImageNet, CIFAR-FS, and FC100 datasets, and 300 epochs for the tieredImageNet dataset. A batch size of 512 is employed, and a cosine learning rate scheduler [38] is utilized for learning rate adjustment. In the meta-tuning phase, the model is trained for 100 epochs using the episodic training strategy. All experiments are conducted on an NVIDIA GeForce RTX 3090 GPU, utilizing PyTorch for code implementations. Evaluation Protocol. The proposed method is evaluated in 5-way 1-shot and 5-way 5-shot settings. Each class has 15 test instances. We randomly sampled 2,000 episodes from the test class. The average accuracy with 95% confidence interval is reported.To evaluate the effectiveness of our proposed KTPP, extensive experiments are conducted on four benchmark datasets. Tabs. 1 and 2 present the results of KTPP compared to state-of-the-art methods under 5-way 1-shot and 5-way 5-shot settings.KTPP demonstrates substantial superiority over the counterparts across all settings. Specifically, in the 1-shot settings, KTPP exhibits notable advancements, outperforming state-of-the-art results by 1.45% to 3.06% across the four benchmarks. KTPP significantly outperforms previous textual modality-based methods (AM3 [61] , KTN [45] , TRAML [31] , SVAE [62] , and SP [6] ), improving 1-shot accuracy by 4.4% and 5-shot accuracy by 3.04% on miniImageNet compared to SP. In addition, Compared to previous ViT-based methods (SUN [13] , FewTure [20] ), KTPP achieves superior results with fewer parameters, improving 1-shot accuracy by 7.81% and 5-shot accuracy by 3.27% on TieredImageNet.The effectiveness of KTPP is confirmed by the results. This is attributed to the ability of KTPP to progressively filter out classirrelevant features in a coarse-fine manner and capture discriminative in cross-modal interactions via the KCA. Moreover, pyramid prompts learned in the PCP contribute to enhancing visual features through deep cross-modal interactions, improving adaptability to spatial variations via the pyramid structure.To fairly prove the effectiveness of our method, we leverage the ViT as the baseline. KCA and PCP are introduced to investigate their impact, as shown in Table 3 . First KCA is integrated as the attention mechanism into the baseline, improving the 1-shot and 5-shot accuracy by 3.25% and 2.89% in Table 2 : Results (%) for the 5-way 1-shot and 5-way 5-shot settings on CIFAR-FS and FC100. The average accuracy with 95% confidence interval are reported. Bold font indicates the best results. Blue font indicates the sub-optimal results.Venue Backbone Params CIFAR-FS 5-way FC100 5-shot 1-shot 5-shot 1-shot 5-shot ProtoNet [50] NeurIPS 6 shows the results of different Prompt and Sparse Attention methods. To fairly compare performance, we use the same ViT architectures in the FSL settings. We simply replace PCP with VPT [22] and SP [6] in the meta-tuning phase. However, VPT lacks semantic information by directly introducing some prompt parameters. Moreover, SP struggles to capture complex semantic relationships due to direct fusion. Instead, KTPP employs PCP to learn pyramid prompts, which allow the ViT to adaptively adjust visual features based on semantic information, making it robust to spatial variations via pyramid structure. Next, we directly replace KCA with KVT [58] in the whole training process. However, KVT has a poor generalization to new classes in sparse scenarios with only few labeled samples. Unlike KVT, we utilize semantic information to further extract discriminative visual features via cross-modal interaction in KCA. Meanwhile, the mean of all tokens as context prompts to enhance the global context information of each token. As a result, Our KTPP significantly outperforms the state-of-the-art prompt and sparse attention methods. 7 shows three different classifiers and the results on miniImageNet are reported. Classification is achieved based on the distance between the support and query features in the feature space. Cosine distance to achieve nearest neighbor classification gets better results than Euclidean distance. Moreover, the Linear logistic regression classifier gets the best results due to constructing more support samples.4.4.6 Image Resolution. Table 8 shows three common image resolutions in FSL. The ViT gets similar results at different resolutions, reflecting the adaptability of the proposed method to image resolution due to the full utilization of diverse spatial information.  
: 
 [ 15, 25] . Instead, humans can recognize new objects from only a few images. To emulate human learning ability and reduce the dependency on annotated data, Few-Shot Learning (FSL) [50] is proposed to learn from a limited number of data, which has drawn considerable concern in recent years. Despite of remarkable progress [1, 7, 13, 14, 20, 50, 60, 68] , achieving human-level performance remains challenging, especially in the 1-shot scenario with only one labeled sample.Most methods [7, 13, 14, 50] utilize support features to infer class prototypes for classification. However, the limited number of labeled samples makes it challenging to learn discriminative features, leading to semantic biases in the generation of class-specific prototypes. To address this issue, an alternative is to involve integrating additional semantic information (e.g., category descriptions) with visual prototypes. Following this view, some studies [6, 31, [61] [62] [63] 73] reveal that integration of semantic priors can enhance prototype representation. Despite their success, most methods integrate semantic embeddings at a higher hierarchical level (e.g., prototypes and classifiers), lacking underlying feature interactions and ignoring distribution differences between modalities. Moreover, a recent study [6] attempted to modulate visual features by semantic embeddings of labels as prompts during the extraction process. However, it struggles to capture complex semantic relationships between textual and visual features by employing linear layers for direct and single-scale fusion.Moreover, several approaches [6, 13, 20] perform interactions within or across modalities in Vision Transformers (ViTs) via vanilla self-attention. However, for vanilla self-attention where each patch is forced to interact with all tokens for computing the attention matrix, noisy tokens are inevitably introduced into the calculation. This incorporation of noisy tokens renders image features vulnerable to spurious correlations, especially in cluttered backgrounds and occlusions. As shown in Fig. 1 , the image annotated as "dog" consists of one person, apple, wall, tree, and other class-irrelevant objects, making the ViT falsely associate these objects with the label when only few labeled samples are available. Therefore, the potential of semantic priors is severely constrained in cross-modal interactions via vanilla self-attention due to the confusion of numerous irrelevant tokens during interaction. As a result, designing an effective cross-modal fusion strategy and improving the selfattention mechanism are crucial to achieving deep cross-modal interactions and filtering out class-irrelevant visual features in FSL frameworks.To address these aforementioned issues, a 𝑘-NN Transformer with Pyramid Prompts (KTPP) is proposed to fully leverage semantic information and capture class-specific support features. Specifically, the KTPP mainly consists of two modules: 𝐾-NN Context Attention (KCA) and Pyramid Cross-modal Prompts (PCP). Firstly, the KCA selects the 𝑘 most relevant tokens for each token to compute attention weights instead of all tokens, effectively excluding irrelevant information and reducing the computational burden. As a result, irrelevant tokens can be progressively suppressed in a coarse-fine manner via three cascaded stages, as shown in Fig. 1 . To avoid the issue of paying too much attention to the local context and neglecting the discriminative information of global feature [19, 32] in the top-𝑘 operation, the mean of all tokens is regarded as context prompts which are incorporated to enhance the global context information of each token. Secondly, the pre-trained CLIP model [46] has an excellent capability of cross-modal feature alignments, extracting rich and unbiased semantic embeddings. Therefore, we exploit the class-aware prompts encoded from class names via the CLIP. the PCP enables these class-aware prompts to interact with multi-scale support features, learning pyramid prompts that capture complex semantic relationships between textual and visual features. The pyramid structure allows the ViT to dynamically focus on the support features most relevant to the semantic information, making it robust to spatial variations. Finally, the augmented support features and the class-aware prompts are then interacted via the KCA, which enables the ViT to further capture discriminative support features due to the semantically guided class-specific features. As shown in Fig. 1 , the KTPP progressively filters out noisy tokens at different stages and further enhances noise-free visual representations via deep cross-modal interactions guided by prompts.Overall, our contributions are summarized as follows:• A KTPP method is proposed to fully exploit cross-modal information and select discriminative visual features. • A 𝐾-NN Context Attention module is proposed to filter out class-irrelevant features in a coarse-fine manner, capturing class-specific features in cross-modal interactions and improving computational efficiency. • Pyramid Cross-modal Prompts are proposed to enhance visual features via deep cross-modal interactions, improving the adaptivity ability to spatial variations. • The proposed method achieves significant performance in four FSL benchmarks, particularly improving 1-shot accuracy by 1%-4% compared to the state-of-the-art approaches. 
 
 Appendix:  


Decision: The KNN Transformer idea was deemed as quite interesting and new. The results are strong.
The author rebuttal was effective, which served to convince one reviewer to improve his/her rating (and that reviewer's initial comments were quite positive too, despite the initial rating). Overall, it should be a good paper to include for the conference.

Reviews: review: paper_summary: paper_summary: For Few-Shot Learning, this paper proposes a 𝑘-NN Transformer with Pyramid Prompts (KTPP), which involves two novel modules named 𝑘-NN Context Attention (KCA) and Pyramid Cross-modal Prompts (PCP). Specifically, KCA achieves progressively noisy filtration via a coarseto-fine manner in the three cascaded stages and extracts discriminative features guided by semantic information. In PCP, pyramid prompts are generated to enhance visual features via deep crossmodal interactions between textual and multi-scale visual features. Experimental results show that the KTPP model is effective.
strength_weakness: strength_weakness: 1. Paper is easy to follow, and motivation is clear.
2. The proposed 𝑘-NN Context Attention (KCA) and Pyramid Cross-modal Prompts (PCP) are interesting, and are effective for few-shot learning according to the experimental results.
3. The experimental results are good, the proposed KTPP model achieves new state-of-the-art performance.
limitations: limitations: 1. In related work section, please add more discussion about the relevant works of CLIP for few-shot learning.
2. The Fusion module in Figure 2, fuses three features including the support features $𝑍_𝑣^𝑆$, text-based class-aware prompts $𝑍_𝑡^{𝑐𝑝}$, and pyramid prompts $𝑍_𝑡^{𝑝𝑝}$. Please provide more detail ablation study to explore the effectiveness of these three features. For example, $𝑍_𝑣^𝑆$, $𝑍_𝑣^𝑆$ + $𝑍_𝑡^{𝑐𝑝}$, $𝑍_𝑣^𝑆$ + $𝑍_𝑡^{𝑝𝑝}$.
paper_summary: paper_summary: This paper addresses the problem of few-shot learning. The authors first propose an improved attention calculation module, KCA, to enable the model to focus only on relevant tokens. Additionally, they introduce language knowledge into the model framework through the CLIP model and design corresponding knowledge fusion modules (PcP and Fusion).  On multiple benchmarks for this task, the proposed method achieves state-of-the-art performance.
strength_weakness: strength_weakness: 1.  The motivation is reasonable, and the specific methods closely align with the motivation.
2.  The writing and illustrations are relatively clear and easy to understand.
3. The proposed modules are complex and detailed, reflecting the authors' deep thinking.
limitations: limitations: 1. The KCA module, which constructs the attention map based on the top k keys, seems to have already been proposed [2]. Therefore, regarding the improvement of attention, this might not provide much additional insight. Additionally, in the ViT or transformer community, there are already many improved self-attention methods that use more sparse attention to enhance model performance or accelerate model convergence, such as Deformable Attention in Deformable DETR. It seems that the authors did not discuss or compare these methods in the paper.
2. The main experiments lack sufficient comparisons. Some recent methods that use language knowledge or the CLIP module have not been compared, such as [1].
3.  The ablation study of the PCP module and its related context is not thorough enough. In the paper, the authors conducted an ablation study on the entire PCP module, which seems to be too coarse-grained. First, the Fusion module behind stage3 and the PcP module takes 𝑍^{S3}{v}, 𝑍^{fp}{t}, 𝑍^{cp}{t} as inputs. Does it make sense to use these inputs and combine them? For example, what would happen to the performance if 𝑍^{cp}{t} is not concatenated to the tokens? Second,  PCP calculates prompts 𝑍^{ppi}{t} for each stage module. Does the prompts for each stage help the subsequent model process? I think the authors can try ablation studies on prompts 𝑍^{ppi}{t} outputted by different stages.
4. Some expressions in the paper are not very clear. For example, how is 𝑍^{cp}{t} obtained from the text encoder of CLIP? How is L{t} obtained?
5. Some typo. For example, in Eqn 6, A_{ij} should be A; in line 541, q_{i} should be a support image.

While this paper still has these limitations, I believe the authors have put in a lot of effort to complete this work. If the authors can supplement the experiments regarding the second and third point, I would consider giving it a higher score.

[1] Self-Prompt Mechanism for Few-Shot Image Recognition
[2] Mixed Autoencoder for Self-supervised Visual Representation Learning
paper_summary: paper_summary: This paper proposes a k-NN Transformer with pyramid prompts, designed to select discriminative information through k-NN contextual attention and to adaptively modulate visual features using Pyramid Cross-modal Prompts (PCP). The cross-modal prompts are generated by CLIP's text encoder. The proposed method achieves promising results on several popular datasets.
strength_weakness: strength_weakness: 1. This paper is well-written and easy to follow. 
2. The proposed KCA attention mechanism, which filters tokens by calculating similarity, is novel. Additionally, the integration of CLIP's text encoder and the design of pyramid prompts assist in few-shot learning. 
3. The experiments comprehensively demonstrate the effectiveness of the proposed method, showing significant improvements across various datasets.
limitations: limitations: 1. The proposed method of designing prompts using CLIP's text encoder is complex. CLIP's text encoder already contains strong class priors, and directly adding a linear layer for training can also achieve good results, as demonstrated by SemFew[1].
2. Using auxiliary information to aid few-shot learning is only applicable to the ViT architecture and is not compatible with ResNet.
[1] Zhang, H., Xu, J., Jiang, S., & He, Z. Simple Semantic-Aided Few-Shot Learning. CVPR2024.
paper_summary: paper_summary: The paper proposes a KNN Transformer with pyramid prompts in few shot learning. In this approach, KNN is considered in attention mechanism, and pyramid is combined into prompt learning. Extensive experiments on four benchmark datasets demonstrate the effectiveness of proposed approach.
strength_weakness: strength_weakness: 1. It is soundable for considering KNN into attention mechanism to filter out the background noise. 
2. text feature is combined with different visual feature level by attention, it is a good idea, while the fusion stage should consider much more details.
3. The experiment is fine.
limitations: limitations: 1. For example, as shown in fig. 1, the dog is the hot point in this figure. Since the KCA (combining KNN into attention) is done based on Vit, there is a question as follow. 
How to really solve the problem of noise from the background? where  backgrounds in different images also have strong attention weights?
2. There is no cross domain dataset in experiment.
3. 'Figure 4: Illustration of cross-modal enhancement module in the PCA.' PCA is PCP?