Rethinking the Effect of Uninformative Class Name in Prompt Learning

Authors: fengmao  lv
changru  nie
jianyang  zhang
guowu  yang
guosheng  lin
xiao  wu
tianrui  li
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 09:42:39

Abstract: Large pre-trained vision-language models like CLIP have shown amazing zero-shot recognition performance. To adapt pre-trained vision-language models to downstream tasks, recent studies have focused on the "learnable context + class name" paradigm, which learns continuous prompt contexts on downstream datasets. In practice, the learned prompt context tends to overfit the base categories and cannot generalize well to novel categories out of the training data. Recent works have also noticed this problem and have proposed several improvements. In this work, we draw a new insight based on empirical analysis, that is, uninformative class names lead to degraded base-to-novel generalization performance in prompt learning, which is usually overlooked by existing works. Under this motivation, we advocate to improve the base-to-novel generalization performance of prompt learning by enhancing the semantic richness of class names. We coin our approach as the Information Disengagement based Associative Prompt Learning (IDAPL) mechanism which considers the associative, meanwhile, decoupled learning of prompt context and class name embedding. IDAPL can effectively alleviate the phenomenon of learnable context overfitting to base classes, meanwhile, learning more informative semantic representation of base classes by fine-tuning the class name embedding, leading to improved performance on both base and novel classes. Experimental results on eleven widely used few-shot learning benchmarks clearly validate the effectiveness of our proposed approach. Code is available at https://github.com/tiggers23/IDAPL

Main paper: 
 introduction: 
 It is well known that the recognition performance of CLIP is usually sensitive to the prompt template. Prompt engineering based on trial and error usually takes a large amount of time in word tuning. To address this problem, existing works [8, 14, 45] assign a set of learnable vectors as the prompt template and optimizes them on downstream datasets. In practice, however, the learned prompt context usually tends to overfit the base categories and cannot generalize to novel categories well. Hence, recent studies along this line mainly focus on improving the cross-category generalizability of prompt learning by learning image conditional prompt [44] , multimodal prompt [14] and regularization stratages [15, 46] . To further improve the generalizability of prompt learning, we turn our attention to analyzing the mechanism of cross-category generalizability degradation in prompt learning. [21] with base class names of different qualities, including manually-defined code-names, original name embedding, name embedding enhanced by hierarchical label set [25] , and name embedding refined by images [27] . "Codename" indicates replacing the original base class names with code-names (e.g., "00", "01"). For fair comparisons, the testing procedure is conducted with original name embedding of novel categories.To answer the above question, we propose a hypothesis from the perspective of prompt composition, that is, one of the key reasons for the learned prompts overfitting to base categories is the lack of semantics carried in class name embedding. The focus of existing works on prompt learning mainly lies in the context part, but ignores that a prompt is composed of a prompt context and a class name (i.e., the "learnable context + class name" paradigm). In general, class names are assumed to carry the discriminative information for classification, while the prompt context should be responsible for aligning the text encoder with the corresponding visual features, acting as a universal template. In practice, however, class names are usually coarsely-defined or non-descriptive (e.g., "A340-200" representing an aircraft) [23] , which may degrade the discriminative performance of CLIP. When the semantic information provided by class names is insufficient, the learnable prompt context will try to derive discriminative information from training data, in order to remedy the uninformative drawback of class names. As a result, the learned context will be biased to base categories and cause performance degradation on novel categories out of the training dataset. This violates the original intention of context optimization, i.e., automatically learning a prompt context acting as a universal template (e.g., "a photo of a"). To verify this hypothesis, we conduct experiments by implementing context optimization with class names of different qualities. From the results shown in Figure 1 , we can see that context prompt learned with class names of richer semantic richness (subclass names or optimized class names) can be more generalizable to novel categories. When using manuallydefined code-names to replace the class names of training data, the learned context will have very poor recognition performance on novel categories. These experimental phenomena well confirm our hypothesis.Motivated by the above observations, this work advocates improving the base-to-novel generalization performance of prompt learning by enhancing the semantic richness of class names. To this end, we propose the Associative Prompt Learning (APL) mechanism to associatively optimize both the learnable context and class name embedding. Specifically, by introducing learnable classspecific residual vectors, APL sets class name embedding as trainable variables to refine them. During the learning process, the semantic richness of class name embedding can be improved by deriving class-related discriminative information from training data. With trainable class name embedding to remedy the drawback of uninformative class names, the prompt context can hence concentrate on its original role in finding a universal template. As a result, the learned prompt context can achieve improved generalizability on novel categories. In addition, APL can also use class semantic descriptions generated by GPT [26] as additional training data to refine class name embedding following the implementation of [8] .Moreover, although improving the semantic richness of class name embedding can reduce the learnable context overfitting to base classes, there is still no guarantee to prevent learnable contexts from carrying base class information, especially when both context and class name embedding are optimized by a unified crossentropy loss. To achieve this guarantee, we further propose the Universal and Discriminative Information Disengagement (UDID) mechanism to ensure the prompt context and class name embedding to concentrate on their original role in finding a universal template and carrying the discriminative information of classes, respectively. Specifically, UDID adopts an embedding discriminator to decouple the prompt context and class names. As class name embedding already carries most of the class information, decoupling the prompt context and class names can further prevent it from carrying base class information, as a result, avoiding the learned prompts overfitting to the base classes. We coin our approach as Information Disengagement based Associative Prompt Learning (IDAPL) considering both the associative and disengagement mechanism of prompt context and class name embedding. The overall architecture of IDAPL is shown in Figure 2 . IDAPL can effectively alleviate the phenomenon of learnable context overfitting to base classes, meanwhile, learning more informative semantic representation of base classes by fine-tuning the class name embedding, leading to improved performance on both base and novel classes. The IDAPL mechanism can be easily applied into existing prompt learning methods (e.g., CoOp [45] and MaPLe [14] ). Extensive experiments on diverse few-shot learning benchmarks clearly demonstrate the effectiveness of our approach. Compared with the previous works relying on regularization with human-engineered prompt temples [15, 46] , which potentially requires a lot of human costs, IDAPL achieves better performance as well. This result further validates the effectiveness of our approach. 
related works 2.1 vision-language models: 
 VLMs [13, 29, 40] mainly focus on learning aligned image-language representations. In general, vision-language models consist of a visual encoder and a language encoder and bridge the two modalities by training two encoders jointly. The recent studies on VLMs pay attentions on pre-training large-scale vision-language models via contrastive representation learning [3, 9] on web-scale datasets. For instance, CLIP [29] and ALIGN [13] respectively exploit about 400 million and 1.8 billion noisy image-text pairs curated from the Internet for pre-training. Based on the rich supervision provided by natural language, large-scale pre-trained VLMs such as CLIP and ALIGN have been recognized to learn powerful visual representations that are transferable across a wide range of downstream tasks.Inspired by the NLP community [17, 18] , the prompt learning paradigm is also applied for VMLs adaptation [19, [34] [35] [36] 45] . For example, Yu et al. [42] propose to add learnable residuals on classification weights. Zhou et al. [45] propose the CoOp approach to learn continuous prompt contexts on downstream datasets. Yin et al. [41] propose test time prompt tuning to avoid collecting labeled training data for prompt learning. In practice, the learned prompts usually tend to overfit the base categories and cannot well generalize to novel categories out of the training data. Hence, recent works mainly focus on improving the cross-category generalization performance of the prompt learning paradigm [14, 20, 39, 44, 46] . For example, Chen et al. propose to apply optimal transport to match learnable prompt with visual features. Khattak et al. [14] propose MaPLe to improve alignment between the vision and language embedding. Zhou et al. [44] propose the Conditional CoOp (CoCoOp) approach to improve the cross-category generalizability of prompt learning. Moreover, Maniparambil et al. [46] and Roy et al. [31] propose using GPT generated class description to refine the learnable prompts. Zhu et al. [46] propose the Prompt-aligned Gradient (ProGrad) mechanism to prevent prompt tuning from damaging the prior knowledge of pre-trained VLMs. Khattak et al. [15] propose a self-regularization strategy to improve the cross-category generalizability of learned prompt. However, the above two approaches rely on regularization with human-engineered prompt temples, which potentially require a lot of human costs. In addition,In general, existing works still lack in-depth analyses of how prompt learning effects the cross-category generalizability of VLMs. In practice, as we found, the uninformative class name embedding could significantly reduce the cross-category generalizability of the learned prompt context. Several recent works have also found the importance of improving the richness of class names for VLMs. Novack et al. [25] and Ge et al. [7] improve the class name embedding via hierarchical label set. Yu et al. [27] further propose the Class name Optimization (CnOp) mechanism to learn optimal class names in prompt learning by replacing the class name embedding with learnable vectors which are trained on downstream tasks. However, the above works lack further insight on the connection between uninformative class names and biased context optimization. In this work, we utilize class name optimization to improve the generalizability of prompt learning in new classes. 
methodology: 
 In this section, we present our proposed Information Disengagement based Associative Prompt Learning approach, with the motivation of improving the base-to-novel generalizability of prompt learning by enhancing the semantic richness of class names.Revisiting CLIP. CLIP [29] consists of an image encoder V and a text encoder G. After pre-trained by a contrastive loss on large-scale multimodal corpus, CLIP can be used for zero-shot classification. For each test image 𝑥, the image encoder V encodes it into a visual embedding f = V (𝑥). The classification weight vectors {w 𝑖 } 𝐾 𝑖=1 are derived by using hand-engineered text prompts (e.g., "a photo of a [classname] i ") to query the textual encoder G, where 𝐾 is the number of classes and [classname] i represents the name of class 𝑖 (e.g., "cat", "dog" and "car"). The prediction probability that image 𝑥 belongs to class 𝑖 is calculated as:EQUATIONwhere sim (•, •) denotes the cosine similarity and 𝜏 is a learned temperature parameter.Prompt Learning in CLIP. As CLIP's performance is sensitive to human-defined prompt context (e.g., "a photo of a"), the context optimization mechanism [45] is proposed to learn the optimal context by introducing a set of learnable vectors 𝑃 = {𝑣 1 , 𝑣 2 , ..., 𝑣 𝑀 } to replace the human-defined context. The prompts are hence reformulated as 𝑡 𝑖 = {𝑣 1 , 𝑣 2 , ..., 𝑣 𝑀 , 𝐶 𝑖 }, where 𝐶 𝑖 represents the word embedding of class names, and 𝑀 is the number of learnable context vectors. The prediction probability is computed as:EQUATION)The prompt context vectors {𝑣 1 , 𝑣 2 , ..., 𝑣 𝑀 } are learned by minimizing the cross-entropy loss between 𝑝 (𝑦 = 𝑖 |𝑥) and ground-truth labels.In a desired prompt 𝑡 𝑖 = {𝑣 1 , 𝑣 2 , ..., 𝑣 𝑀 , 𝐶 𝑖 }, the class name embedding should be responsible for carrying the discriminative information for classification, while the prompt context should be responsible for aligning G(𝑡 𝑖 ) with the corresponding visual features, acting as a universal template. Existing works put their main focus on optimizing prompt contexts [45] , in order to adapt CLIP to downstream datasets, with the assumption that the class name embedding can reveal the visual difference between categories. In practice, however, the class names are usually coarsely-defined or non-descriptive (e.g., "A340-200" representing an aircraft). In this case, the context vectors {𝑣 1 , 𝑣 2 , ..., 𝑣 𝑀 } will be trained to derive discriminative information from the training dataset, in order to remedy the uninformative drawback of class names. As a result, the learned prompt context will inevitably be biased towards base categories and cannot generalize well to novel categories out of the training dataset. Hence, the class name embedding should be refined, not only for improving the discriminative ability but also for improving the base-to-novel generalizability of the learned context.Figure 2 shows the overall architecture of our EQUATIONwhere each 𝑟 (𝑖,𝑗 ) is a learnable vector with the same dimension as the class name embedding 𝑐 (𝑖,𝑗 ) and 𝛼 is a hyper-parameter. By replacing the original class embedding 𝐶 𝑖 in prompt 𝑡 𝑖 with 𝐶 * 𝑖 , the refined prompt 𝑡 * 𝑖 = {𝑣 1 , 𝑣 2 , ..., 𝑣 𝑀 , 𝐶 * 𝑖 } is received. Thereby, the classification weight vector w * 𝑖 will be generated by passing the refined prompt 𝑡 * 𝑖 through the text encoder G as w * 𝑖 = G 𝑡 * 𝑖 . Similar to existing prompt learning approaches [14, 44, 45] , we optimize prompts using base class images through the standard classification loss function, cross-entropy loss, formally,EQUATIONwhere The objective of D is to learn a binary classifier to classify whether the input embedding is the general context or class name embedding. This decoupling process is performed using the binary cross-entropy loss, formally,𝔇EQUATIONwhere ℭ 𝐵 is the set of base classes. In this way, the similarity between prompt context and class names can be reduced. As class name embedding already carries most of the discriminative information, reducing the similarity can prevent it from carrying base class information. Therefore, the disengagement mechanism can further avoid the learned prompt context from overfitting the base classes.Optimization and Inference. The overall training object of our IDAPL is the combination of the above two loss functions, formally,EQUATIONwhere 𝛽 is a hyper-parameter. In the inference stage, we put the optimized prompt context {𝑣 1 , 𝑣 2 , ..., 𝑣 𝑀 } and the original novel class name embedding into the text encoder G to derive the classification weights for novel classes. Based on the above APL and UDID mechanisms, our IDAPL approach can effectively alleviate the phenomenon of learnable context overfitting to base classes, meanwhile, learning more informative semantic representation of base classes by fine-tuning the class name embedding, leading to improved performance on both base and novel classes. 
experiments 4.1 experimental setup: 
 We conduct comprehensive experiments on diverse benchmarks to validate the core idea of our work. Both base-to-novel and crossdataset generalization settings [14, 44, 45] are considered to demonstrate the effectiveness of our proposed approach in improving the cross-category generalizability of the prompt learning mechanism. Specifically, in the base-to-novel setting, each dataset is divided into subsets of base and novel categories, and only base categories provide images for training. In the cross-dataset generalization setting, we adopt ImageNet [32] for training and test the performance on other datasets with categories different from the training categories.Following existing works on prompt learning [14, 44, 45] , we conduct experiments on eleven standard image recognition benchmarks which focus on different categories, including: i) two genericobject recognition benchmarks (i.e., ImageNet [32] and Caltech101 [5] ); ii) five fine-grained recognition benchmarks (i.e., OxfordPets [28] , StanfordCars [16] , Flowers102 [24] , Food101 [1] , and variant level task of FGVCAircraft [21] ); iii) one scene benchmark (i.e., SUN397 [38] ); iv) one action recognition benchmark (i.e., UCF101 [33] ); v) one texture benchmark (i.e., DTD [4] ); vi) one satelliteimage benchmark (i.e., EuroSAT [10] ).In all the experiments, following prior works [14, 44] , we randomly select 16 samples per category for training, under the few-shot learning setting. In our experiments, we implement the IDAPL mechanism under the training paradigm of CoOp [45] and MaPLe [14] respectively. Specifically, the pre-trained ViT-B/16 CLIP model is utilized as the backbone of our model. We directly adopt the GPT visual descriptions collected by Maniparambil et al. [22] as the additional training data of base classes. The training epoch number is set to 50 and 5 for CoOp w/ IDAPL and MaPLe w/ IDAPL, respectively. The adaptation stage utilizes the same iteration number as that of the training stage. The hyper-parameter 𝛼 & 𝛽 are set to 0.1 & 5, respectively. The rest hyper-parameter settings (e.g., learning rate, batch size, context length, prompt depth, etc) follow the original implementation of CoOp [45] and MaPLe [14] . In addition, to warm up the discriminator D, we optimize it by Eq. 4 with the original general context (i.e., "a photo of a") and class name embedding before the formal training process. The hyperparameters are fixed across all datasets. The experiments are conducted on one 4090 GPU. The implementation code will be released online upon acceptance.Baselines. We compare our approach with the recent state-ofthe-art works on VLMs adaptation, including CLIP-Adapter (CLIP-A) [6] , Context Optimization (CoOp) [45] , Class name Optimization (CnOp) [27] , Conditional CoOp (CoCoOp) [44] , Prompt Learning with Optimal Transport (CoPLOT) [2] , Regularized Mask Tuning(R-AMT) [43] , Multi-modal Prompt Learning (MaPLe) [14] , CLIP-A-Self [22] , PromptSRC [15] , and ProGrad [46] . Of these, CLIP-A [6] , R-AMT [43] and CoOp [44] are pioneer works on adapting CLIP towards downstream tasks. Specifically, CLIP-A introduces lightweight additional feature adapters on either visual or language branches, R-AMT fine-tunes VLMs via masking selected parameters, while CoOp focuses on learning continuous context vectors. CnOp [27] proposes to adapt CLIP towards downstream datasets by optimizing class name embedding. CoCoOp [44] , CoPLOT [2] , MaPLe [14] , PromptSRC [15] , and ProGrad [46] draw attentions on improving the base-to-novel generalization performance of prompt learning. CLIP-A-Self [22] further proposes to enhance prompt learning by utilizing GPT descriptions. Base-to-Novel Generalization. This setting splits datasets into base and novel categories. In this setting, the model is first trained on base categories and then tested on novel categories. Table 1 displays the performance comparison of each approach. From Table 1 , we can draw the following observations. First, the performance achieved by our approach is on par with or even better than that of the compared baselines on most of the benchmarks (except FGV-CAircraft, Caltech101, and ImageNet), as well as on average. By improving the generalizability of prompt context and enriching the class name embedding together, our approach can obtain consistent performance improvement over both base and novel categories. Second, by applying the IDAPL mechanism into the training paradigm of CoOp, our approach can achieve consistent performance improvement over the vanilla CoOp approach. By applying the IDAPL mechanism into the training paradigm of MaPLe, our approach outperforms the vanilla MaPLe approach on most of the benchmarks (except Caltech101), as well as on average. In addition, our approach can clearly outperform the CLIP-A-Self baseline which also utilizes GPT descriptions to enhance prompt learning. Finally, our approach can clearly outperform the regularization strategy based baselines, PromptSRC [15] and ProGrad [46] , which potentially require a lot of human costs. This result shows the superiority of enhancing the semantic richness of class names and decoupling the embedding of prompt context and class names in improving the generalizability of prompt learning. Cross-Dataset Generalization. In this setting, the model is first trained on base categories of one source dataset and then tested on novel categories of other datasets. Table 2 displays the performance comparison of each approach. From Table 2 , we can draw similar observations as in the above setting. In general, our approach can achieve improved performance compared with baseline models and can achieve comparable (3/10) or improved (6/10) performance compared with other comparison models on target benchmarks (except EuroSAT). EuroSAT is a satellite-image benchmark, which is very different from other datasets and contains only 10 classes, making its performance unstable. Fair comparisons, we adopt the same hyperparameters for baseline models, (i.e., CoOp and MaPLe) and our implementation.Ablation Study. In order to further demonstrate the effectiveness of the proposed associative learning mechanism, we conduct ablation studies and show the corresponding results in Table 3 .Comparing CLIP with the context learning mechanism, we can see that learning prompt context can significantly improve base class accuracy, but limit the cross-category generalizability as the Sensitivity Analysis. To demonstrate the robustness of our approach, we further conduct sensitivity analysis for hyper-parameters and report the corresponding results in Figure 3 . From the upper part of Figure 3 , we can see that the performance of our approach progressively improves as the epoch number grows, and is stabilized after 50 epochs, while the performance of CoOp is sensitive to the epoch number. This observation further shows that our proposed associative learning mechanism is robust against base-tonovel performance degradation during context optimization. From the middle and bottom part of Figure 3 , we can see that our approach can tolerate wide ranges of the trade-off parameters 𝛼 & 𝛽, which are used to incorporate category residuals 𝑅 𝑖 into class name embedding and balance the loss functions, respectively. Domain Generalization. The associative prompt learning mechanism is originally proposed for improving the cross-category generalization performance of context optimization. Considering that our approach usually obtains better performance on base categories, we have concerns on whether the introduced IDAPL mechanism will cause extra bias towards the domain style of training data. To this end, we further examine the performance of our approach on the domain generalization setting and report the corresponding results in Table 4 . Specifically, the model is first trained on ImageNet and then evaluated on datasets of other domain styles (i.e., ImageNet-V2 [30] , ImageNet-S [37] , ImageNet-A [12] and ImageNet-R [11] ). From Table 4 , we can see that the introduced IDAPL mechanism does not lead to clear performance degradation on new domains. The domain generalization performance obtained by our approach can be on par with or even better than the corresponding baselines, although cross-domain generalization is not the original purpose of our approach. Prompt Complexity. As shown in Table 5 , we compare the number of trainable parameters of CoOp+IDAPL and MaPLe+IDAPL with their baselines. We can see that the IDAPL mechanism only introduces about 0.1M extra trainable parameters (∼0.1% parameters of CLIP) to their baselines, but brings about clear performance improvement on the harmonic mean metrics. Notably, since APL does not change the visual branch, its inference speed is consistent with the original baseline model. 
conclusion: 
 This work focuses on improving the generalization ability of prompt learning for vision-language models. To achieve this, we draw a novel empirical-analysis-based insight on the effect of uninformative class names causing novel class performance degradation in prompt learning. Motivated by this observation, we propose our IDAPL approach, which consists of APL and UDID mechanisms. APL alleviates the phenomenon of learnable context overfitting to base classes by enhancing the semantic richness of class names, meanwhile, semantically enriched class name embedding also brings performance improvement on base classes. In addition, as APL cannot prevent the learnable context from learning base class information, we further propose UDID to ensure the prompt context and class name embedding to concentrate on their original role in finding a universal template and carrying the discriminative information of classes, respectively, by decoupling the embedding of prompt context and class names. In this way, the learnable context is protected from biasing to base classes. We conduct extensive experiments on eleven widely used few-shot learning benchmarks. Experimental results clearly validate the effectiveness of IDAPL. 
 
 Appendix:  


Decision: This paper examines the impact of uninformative class names on the generalization performance of prompt learning. The authors find that semantically poor class names cause the prompt context to overfit to base categories, resulting in poor performance on novel categories. To address this, they propose the Information Disengagement based Associative Prompt Learning (IDAPL) method, which optimizes both prompt context and class name embeddings to enhance generalizability. IDAPL includes two main components: Associative Prompt Learning (APL), which introduces class-specific residual vectors to improve class name embeddings, and Universal and Discriminative Information Disengagement (UDID), which uses an embedding discriminator to separate the prompt context from class name embeddings. Experimental results show that IDAPL outperforms existing methods on various few-shot learning benchmarks, improving classification performance on both base and novel categories.

The proposed method is simple yet effective. The paper is well-written and easy to understand, presenting a novel idea that marks a significant advancement in the field.

More discussions with respect to recent methods is desried:
[1]PLOT: Prompt Learning with Optimal Transport for Vision-Language Models (ICLR23) 
[2]Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Models (ICCV23) 
[3]Task Residual: Tuning Vision-Language Models in One Line of Code (CVPR23)
[4] In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-language Model. CoRR abs/2403.06126 (2024)

Reviews: review: paper_summary: paper_summary: This paper introduce a new method which improves the base-to-novel generalization performance of prompt learning by enhancing the semantic richness of class names.
strength_weakness: strength_weakness: IDAPL can be combined with multiple SOTA methods (CoOP and MaPLe) without introducing much learnable params.
limitations: limitations: The improvement on performance is limited according Table1/2. Moreover, according to Table.3, CoOp+IDAPL uses gpt generated prompts. Is the comparison reported in Table.1 fair enough? Because the results of CoCoOp in this paper is same as the results reported by CoCoOp itself, which did not use gpt generated prompts.

More SOTAs ([1][2][3) should be included for comparison in Table.1 and Fig.3.

What is APL in Fig.3? I think it might be IDAPL.

[1]PLOT: Prompt Learning with Optimal Transport for Vision-Language Models (ICLR23)
[2]Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Models (ICCV23)
[3]Task Residual: Tuning Vision-Language Models in One Line of Code (CVPR23)
paper_summary: paper_summary: This paper investigates the impact of uninformative class names on the generalization performance of prompt learning. The study reveals that the lack of semantically rich class names leads to prompt context overfitting to base categories, resulting in poor performance on novel categories. To address this issue, the authors propose the Information Disengagement based Associative Prompt Learning (IDAPL) method, which enhances the generalizability of prompts by optimizing both the prompt context and class name embeddings. IDAPL consists of two main mechanisms: Associative Prompt Learning (APL) and Universal and Discriminative Information Disengagement (UDID). APL improves class name embeddings by introducing class-specific residual vectors, while UDID uses an embedding discriminator to decouple the prompt context from class name embeddings. Experimental results demonstrate that IDAPL outperforms existing methods on various few-shot learning benchmarks, improving classification performance on both base and novel categories.
strength_weakness: strength_weakness: 1. The proposed method is simple yet effective.
2. The paper is well writen and easy to understand.
3. The overall idea is novel.
limitations: limitations: 1. During inference stage, is the  class-specific category residuals used for base class? Will this cause inconsistent inference model for base and novel class? For more realistic settings such as evaluation among all classes[a], the proposed method might be invalid.
2. The details of GPT generated class semantic descriptions are not provided.



[a]. Shu Y, Guo X, Wu J, et al. Clipood: Generalizing clip to out-of-distributions[C]//International Conference on Machine Learning. PMLR, 2023: 31716-31731.
paper_summary: paper_summary: This paper proposes to enhance the semantic richness of class names to improve the base-to-novel generalization performance of prompt learning. Experimental results validate the effectiveness of our proposed approach.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to follow.
2. The analysis in Lines 102-170 is clear and insightful.
2. The proposed UDID mechanism is interesting.
limitations: limitations: 1. APL can assist categories with insufficient information such as "A340-200", but such categories only make up a small portion. For other conventional categories, it may already have sufficient information to distinguish them (comparing their category embedding similarity may indicate this), and the performance benefits mainly come from the additional information provided by APL and GPT, which may conflict with the motivation of the paper.
2. Based on the first point of view, I speculate that the main improvement comes from the additional information brought by GPT, rather than APL. I am curious about the performance of removing the APL module and only using the GPT-generated description.
3. The proposed strategy does not bring benefits to Domain Generalization, which is somewhat puzzling. I hope the author can discuss this more.
paper_summary: paper_summary: This paper introduces the Information Disengagement based Associative Prompt Learning (IDAPL) mechanism to enhance the base-to-novel generalization performance of prompt learning by enriching the semantic content of class names. Experimental results on eleven few-shot learning benchmarks demonstrate that IDAPL improves performance on both base and novel classes.
strength_weakness: strength_weakness: The motivation of this paper is very clear. This paper optimize the context prompt with class names of  semantic richness to generalize pretrained model to novel categories.

The paper is well-organized and well-written.

The experiments are thorough and validate the effectiveness of the method.
limitations: limitations: The author proposes deriving class-related discriminative information from training data by learning residuals. Could this objective also be achieved by directly learning class names? If optimization is difficult, could the corresponding learning rate be adjusted?

Ablation studies were only conducted on 11 fine-grained recognition datasets. Verification of the method’s effectiveness is also needed on ImageNet variant datasets.

In line 25, “meanwhile” is confusing.

By applying the proposed IDAPL to the training paradigm of MaPLe and CoOp, the performance of vanilla MaPLe and CoOp has improved on most benchmarks. However, similar results were not observed on Caltech101. The authors should provide some explanations. Additionally, on most datasets, the proposed method does not outperform PromptSRC in terms of base performance, which also requires an explanation.