Category-Prompt Refined Feature Learning for Long-Tailed Multi-Label Image Classification

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 04:52:53

Abstract: Real-world data consistently exhibits a long-tailed distribution, often spanning multiple categories. This complexity underscores the challenge of content comprehension, particularly in scenarios requiring long-tailed multi-label image classification (LTMLC). In such contexts, imbalanced data distribution and multi-object recognition pose significant hurdles. To address this issue, we propose a novel and effective approach for LTMLC, termed Category-Prompt Refined Feature Learning (CPRFL), utilizing semantic correlations between different categories and decoupling category-specific visual representations for each category. Specifically, CPRFL initializes category-prompts from the pretrained CLIP's embeddings and decouples category-specific visual representations through interaction with visual features, thereby facilitating the establishment of semantic correlations between the head and tail classes. To mitigate the visual-semantic domain bias, we design a progressive Dual-Path Back-Propagation mechanism to refine the prompts by progressively incorporating context-related visual information into prompts. Simultaneously, the refinement process facilitates the progressive purification of the category-specific visual representations under the guidance of the refined prompts. Furthermore, taking into account the negative-positive sample imbalance, we adopt the Asymmetric Loss as our optimization objective to suppress negative samples across all classes and potentially enhance the head-to-tail recognition performance. We validate the effectiveness of our method on two LTMLC benchmarks and extensive experiments demonstrate the superiority of our work over baselines.

Main paper: 
 related work 2.1 long-tailed visual recognition: 
 Real-world data often follows a long-tailed distribution, presenting a significant challenge for traditional models due to the imbalanced class distributions. Common methods to address this challenge include direct resampling of training samples to balance category distribution, which may result in over-fitting of the tail categories [2, 3, 18, 43] . Another strategy involves loss re-weighting based on label frequencies of training samples to rebalance the uneven positive gradients among classes [4, 11, 21, 51] . More recently, researchers have explored some techniques like transfer learning [34, 61] and self-supervised learning [22, 56] to address imbalanced class distribution. As the Vision-Language Pretrained (VLP) models like CLIP [41] exhibit strong zero-shot adaptation performance, some strategies have been proposed to adapt them to downstream long-tailed learning tasks [13, 35, 44, 48] . These VLP-based methods incorporate additional language data to generate auxiliary confidence scores and fine-tune the CLIP-based model on long-tailed data. However, the enhanced performance of tail classes in these methods is largely attributed to CLIP's image encoder, which has been pretrained on a vast dataset containing many tail samples. This reliance on the image encoder may inadvertently leverage prior exposure to a large amount of visual data. In our work, we specifically utilize only CLIP's text encoder to extract category semantics, allowing us to avoid this potential bias and establish a more robust method for addressing the long-tailed problem.For the Multi-Label image Classification (MLC) tasks, early solutions involved training separate binary classifiers for each label [49] . Consequently, recent research addresses the MLC problem by utilizing category semantics to model label semantic correlations. CNN-based methods [6, 15, 50, 60] utilize Recurrent Neural Networks (RNNs) to extract features sensitive to label dependencies, implicitly capturing these dependencies. Similarly, statistical cooccurrence graphs are constructed to leverage Graph Convolutional Network (GCNs) for representing label correlations [7, 9, 36, 54] . For example, SSGRL [7] incorporates category semantics to guide learning semantic-specific representations and explores their interactions via a graph propagation mechanism. With the rise of Transformer across various computer vision tasks, Transformerbased methods [27, 32, 38] utilize the core attention mechanisms in Transformer [33] to explore label correlations. For instance, C-Tran [27] leverages category semantics extracted from word embedding to facilitate visual-semantic interaction within the Transformer. However, these methods often have complex designs and may not generalize well to LTMLC. Recently, VLP models [41] have been adapted for downstream MLC tasks to address few-shot and zeroshot problems [1, 17, 20] . Neglecting the imbalanced distribution of samples, they either have poor generalization on the LTMLC tasks. Figure 1 : Overall framework of our CPRFL for long-tailed multi-label image classification. Overall, our approach consists of two sub-networks: Prompt Initialization (PI) network and Visual-Semantic Interaction (VSI) network. The initial prompts 𝑃 are extracted from CLIP's text embedding within the PI network, and then these prompts are employed to interact with visual features 𝐹 within the VSI network, facilitating the decoupling of category-specific visual representations 𝑃 ′ . Finally, we compute the similarities between category-specific features 𝑃 ′ and corresponding prompts 𝑃 to obtain the prediction probability for each category and utilize a progressive Dual-Path Back-Propagation mechanism to refine the prompts. To further address the negative-positive imbalance problem inherent in multiple categories, we incorporate a Re-Weighting (RW) strategy.In recent years, research on addressing class imbalance in multilabel settings has been relatively limited. Similar to re-weighting strategies, Wu et al. [52] propose a distribution-balanced (DB) loss to slow down the optimization rate of negative labels based on binary-cross-entropy loss. Based on DB loss, Lin et al. [29] aim to flexibly adjust the training probability and further reduce the probability gap between positive and negative labels. To adapt resampling strategies to multi-label settings, Chen et al. [8] introduce a group sampling strategy, while Guo et al. [16] adopt collaborative training on both uniform and re-balanced samplings to alleviate the class imbalance. Additionally, a prompt-tuning method [53] has been proposed to adapt pretrained CLIP [41] to LTMLC. However, these methods may partly compromise the performance of the head classes while improving tail classes. In this paper, we utilize pretrained CLIP's text embedding to extract category semantics and guide visual-semantic information interaction, which explicitly establishes semantic correlations between the head and tail classes and decouple category-specific visual representations,leading to synchronous improvements in head-to-tail performance in LTMLC. 
overview: 
 In this section, we introduce the proposed CPRFL approach, consisting of two sub-networks, i.e., Prompt Initialize (PI) network and Visual-Semantic Interaction (VSI) network. Firstly, we leverage the pretrained CLIP's text embedding to initialize category-prompts within the PI network, utilizing category semantics to encode semantic correlations between different categories. Subsequently, these initialized prompts interact with extracted visual features using a Transformer encoder within the VSI network. This interaction process facilitates the decoupling of category-specific visual representations, enabling the framework to discern context-related visual information associated with each category. Finally, we compute the similarity between the category-specific features and their corresponding prompts at the category level to obtain the prediction probability for each category. To mitigate the visual-semantic domain bias, we employ a progressive Dual-Path Back-Propagation mechanism guided by category-prompt learning to refine the prompts and progressively purify the category-specific visual representations over the training iterations. To further address negative-positive imbalance issue, we optimize the framework adopting a Re-Weighting strategy (i.e., an Asymmetric loss (ASL) [42] ), which helps suppress negative samples across all classes. Fig. 1 illustrates the entire pipeline of the proposed CPRFL approach. Feature Extraction. Given an input image 𝑥 from the dataset 𝐷, we first utilize a backbone network to extract local image features 𝑓 𝑥 𝑙𝑜𝑐 ∈ R ℎ×𝑤×𝑑 0 , where 𝑑 0 , ℎ, 𝑤 denote the number of channels, height and width, respectively. In this paper, we employ a convolutional network such as ResNet-101 [19] and obtain the local features by removing the last pooling layer. After that, we add a linear layer 𝜑 to project the features from dimension 𝑑 0 to 𝑑 into a visual-semantic joint space to match the dimension of categoryprompts:EQUATIONUtilizing the local features, we conduct visual-semantic information interaction between them and the initial category-prompts to discern category-specific visual information.Semantic Extraction. Formally, the pretrained CLIP [41] comprises an image encoder 𝑓 (•) and a text encoder 𝑔(•). For our purpose, we solely utilize the text encoder to extract category semantics. Specifically, we adopt a classic predefined template "a photo of a [CLASS]" as the input text for the text encoder. Then the text encoder maps the input text (class 𝑖, 𝑖 = 1, ..., 𝑐) to the text embedding W = 𝑔(𝑖) = {𝑤 1 , 𝑤 2 , ..., 𝑤 𝑐 } ∈ R 𝑐 ×𝑚 ,where 𝑐 represents the number of classes, and 𝑚 denotes the dimension length of the embedding.This extracted text embedding serves as the category semantics for initializing the category-prompts.In order to bridge the gap between the semantic and visual domains, recent work [27, 50] has attempted to project semantic word embeddings into a visual-semantic joint space using linear layers. Instead of emloying linear layers for this projection directly, we opt for nonlinear structures to handle category semantics derived from pretrained CLIP's text embedding. This approach allows us to achieve a more sophisticated projection from the semantic space to the visual-semantic joint space.Specifically, we design a Prompt Initialize (PI) network, which consists of two fully connected layers followed by a nonlinear activation function. Through the nonlinear transformation performed by the PI network, we map the pretrained CLIP's text embedding W to the initial category-prompts P = {𝑝 1 , 𝑝 2 , ..., 𝑝 𝑐 } ∈ R 𝑐 ×𝑑 . Eq. 2 illustrates the entire initialization process:EQUATIONwhere 𝑊 1 ,𝑊 2 , 𝑏 1 , 𝑏 2 denote the weight matrices and bias vectors of the two linear layers, and 𝐺𝐸𝐿𝑈 represents the nonlinear activation function. Here, 𝑊 1 ∈ R 𝑚×𝑡 ,𝑊 2 ∈ R 𝑡 ×𝑑 , 𝑡 = 𝜏 × 𝑑 and 𝜏 is the expansion coefficient controlling the dimension of hidden layers. Typically, 𝜏 is set to 0.5 in our experiments, and we will investigate its impact further in the supplementary material.The PI network plays a crucial role in extracting category semantics from the pretrained CLIP's text encoder, leveraging its powerful semantic representation capability to establish semantic correlations between different categories without relying on ground-truth labels. By initializing category-prompts with category semantics, the PI network facilitates the projection from the semantic space to the visual-semantic joint space. Additionally, the nonlinear design of our PI network enhances the visual-semantic interaction capacity of the extracted category-prompts, thereby improving the subsequent visual-semantic information interaction.With the widespread adoption of Transformer in the field of computer vision, as evidenced by recent works [10, 27, 32] showcasing the capability of typical attention mechanisms to enhance the interaction between visual-semantic cross-modal features, we are motivated to design a Visual-Semantic Interaction (VSI) network. This network incorporates a Transformer encoder, which takes as input the initial category-prompts and visual features. The Transformer encoder performs visual-semantic information interaction to discern context-related visual information specific to each category. This interaction process effectively decouples category-specific visual representations, thereby facilitating better feature classification for each category.To facilitate visual-semantic information interaction between category-prompts and visual features, we concatenate the initial category-prompts P ∈ R 𝑐 ×𝑑 with the visual features F ∈ R 𝑣×𝑑 , forming a combined set of embeddings 𝑍 = (F , P) ∈ R (𝑣+𝑐 ) ×𝑑 . These embeddings are then input to the VSI network for the visualsemantic information interaction. Within the VSI network, each embedding 𝑧 𝑖 ∈ 𝑍 undergoes calculation and updating through the multi-head self-attention mechanism inherent to the Transformer encoder. Notably, we focus solely on updating the category-prompts P, as these represent the decoupling part of category-specific visual representations. The attention weight 𝛼 𝑝 𝑖 𝑗 and the subsequent updating process are computed as follows:EQUATIONEQUATIONEQUATIONwhere 𝑊 𝑞 ,𝑊 𝑘 ,𝑊 𝑣 are the query, key, and value weight matrices, 𝑊 𝑟 ,𝑊 𝑜 are transformation matrices, and 𝑏 3 , 𝑏 4 are bias vectors.To streamline the complexity of the VSI network, we opt for one single layer of Transformer encoders without stacking. The resulting output of the VSI network and the category-specific visual features are denoted as𝑍 ′ = {𝑓 ′ 1 , 𝑓 ′ 2 , ..., 𝑓 ′ 𝑣 , 𝑝 ′ 1 , 𝑝 ′ 2 , ..., 𝑝 ′ 𝑐 } and P ′ = {𝑝 ′ 1 , 𝑝 ′ 2 , ..., 𝑝 ′ 𝑐 }, respectively. Within the self-attention mechanism described in Eq. 3, each category-prompt embedding comprehensively considers its attention towards all local visual features and other category-prompt embeddings. This comprehensive attention mechanism effectively discerns context-related visual information from the samples, leading to the decoupling of category-specific visual representations.Following the interaction between visual features and initial prompts via the VSI network, the resulting output P ′ serves as categoryspecific features for classification. In traditional Transformer-based methods [10, 27, 32] , the specific output features obtained from the Transformer are typically projected onto the label space using linear layers for final classification. Different from these methods, we employ the category-prompts P as the classifier and compute the similarity between the category-specific features and the categoryprompts to conduct classification within the feature space. The classification probability 𝑠 𝑖 for class 𝑖 can be calculated by Eq. 6:EQUATIONIn the context of the unique data characteristics intrinsic to the multi-label setting, we compute the dot-product similarity between the category-specific feature vector of each category and the corresponding prompt vector to determine the probability, which computes absolute similarity. We deviate from the traditional similarity pattern using relative measurement between category-specific feature vector of each category and all prompt vectors. The reason is to mitigate the computational redundancy incurred by calculating similarity between the feature vector of each category and unrelated category prompts, which is unnecessary. The initial prompts lack crucial visual-context information, leading to a substantial data bias between the semantic and visual domains during information interaction. This discrepancy results in imprecise initial prompts, consequently affecting the quality of category-specific visual representations. To mitigate this issue, we introduce a progressive Dual-Path Back-Propagation mechanism guided by category-prompt learning. This mechanism involves two paths of gradient optimization during model training (shown in Figure 2a ): one path through the VSI network and another path directly to the PI network. The former path also optimizes the VSI network to enhance its ability for visual-semantic information interaction. By employing a series of dual-path gradient back-propagations, the prompts are gradually refined over the training iterations, allowing for the progressive accumulation of context-related visual information. Concurrently, the refined prompts guide the generation of more accurate category-specific visual representations, leading to the progressive purification of category-specific features. We term this entire process "Prompt Refined Feature Learning", and it will be iteratively conducted until convergence, as illustrated in Figure 2b .To further address the negative-positive sample imbalance inherent in multiple categories, we integrate the Re-Weighting (RW) strategy, commonly utilized in such scenarios. Specifically, we adopt the Asymmetric Loss (ASL) [42] as our optimization objective, which is a variant of focal loss [30] with different 𝛾 values for positive and negative samples. Given an input image 𝑥 𝑖 , our model predicts its final category probabilities 𝑆 𝑖 = {𝑠 𝑖 1 , 𝑠 𝑖 2 , ..., 𝑠 𝑖 𝑐 } and its ground truth is 𝑌 𝑖 = {𝑦 𝑖 1 , 𝑦 𝑖 2 , ..., 𝑦 𝑖 𝑐 }. We train the whole framework using ASL as shown in Eq. 7:EQUATION)where 𝑐 is the number of classes. s𝑖 𝑗 is the hard threshold in ASL, denoted as s𝑖 𝑗 = 𝑚𝑎𝑥 (𝑠 𝑖 𝑗 -𝜇, 0). 𝜇 is a threshold used to filter out negative samples with low confidence. By default, we set 𝛾 + = 0 and 𝛾 -= 4 in our experiments. In our framework, ASL effectively suppresses negative samples across all classes, potentially improving head-to-tail category performance in LTMLC tasks. 
experiments 4.1 experimental setup: 
 Datasets. Following the settings outlined in [52] , we conduct experiments on two datasets for long-tailed multi-label visual recognition: VOC-LT and COCO-LT. These two datasets are artificially sampled from two well-known multi-label recognition benchmarks: Pascal VOC [14] and MS-COCO [31] , respectively.VOC-LT is created from the 2012 train-val set of Pascal VOC, following the guidelines provided in [52] . The training set comprises 1,142 images annotated with 20 class labels. The number of images per class varies from 4 to 775. To simulate a long-tailed distribution, all classes are categorized into three groups based on the number of training samples per class: head classes (more than 100 samples), medium classes (20 to 100 samples), and tail classes (less than 20 samples). After splitting, the ratio of head, medium, and tail classes Table 1 : The mAP (%) performance of the proposed CPRFL and comparison methods on two long-tailed multi-label datasets. We present the mAP results on overall, head, medium, and tail classes. "CPRFL-GloVe" denotes our CPR with GloVe word embedding, and "CPR-CLIP" denotes our CPRFL with CLIP-RN50 's text embedding. Bold indicates the best scores. Similarly, COCO-LT is derived from the MS-COCO 2017 dataset using a comparable approach. The training set of COCO-LT comprises 1,909 images annotated with 80 class labels. The number of images per class ranges from 6 to 1,128. The distribution of head, medium, and tail classes in COCO-LT is set to 22:33:25. The performance evaluation is conducted on the MS-COCO 2017 test set, which contains 5,000 images.Implementation Details. For the pretrained CLIP models, we adopt CLIP ResNet-50 or ViT-Base/16 [41] and use the corresponding CLIP Transformer as the text encoder. For network optimization, we use the Adam optimizer [24] with a weight decay of 1𝑒 -4 and the training epochs are set to 30. The batch size is 32, and the learning rates for COCO-LT, and VOC-LT are empirically initialized with 1𝑒 -5, 5𝑒 -5. We use mean average precision (mAP) as the evaluation metric to assess the performance of long-tailed multi-label visual recognition across all classes. More implementation details can be found in the supplementary material.To validate the effectiveness of our proposed method, we compare it with previous state-of-the-art methods on two long-tailed multilabel datasets. The state-of-the-art methods include popular longtailed learning methods and multi-label algorithms, i.e., Empirical Risk Minimization (ERM), a smooth version of Re-Weighting (RW) using the inverse proportion to the square root of class frequency, ML-GCN [9] , OLTR [34] , LDAM [4] , Class-Balanced (CB) Focal [11] , BBN [58] , Distribution-Balanced (DB) Focal [52] , ASL [42] . We also compare our CPRFL with previous LTMLC methods, including:• LTML [16] : collaborative training on the uniform and rebalanced samplings with learnable logit compensation.• CDRS+AFL [46] : the combination of copy-decoupling resampling strategy and adaptive weighted focal loss. • Bilateral-TPS [28] : the modification of a bilateral structure that samples both the original and proposed sampling distributions to represent tail classes. • Probability Guided (PG) Loss [29] : an improved version of DB focal that can flexibly adjust the training probability and further reduce the probability gap between positive and negative labels. • COMIC [55] : an end-to-end learning framework that corrects missing labels, adaptively modifies the attention to different samples, and balances the classifier on head-to-tail learning. • CAE-Net [5] : a class-aware embedding network that learns robust class-based representations.For the proposed CPRFL, we present the results using different text embeddings for fair comparisons, i.e., GloVe word embedding [39] ("CPRFL-Glove") and CLIP's text embedding ("CPRFL-CLIP"). Notably, we don't compare our CPRFL with previous CLIP-based methods, because the superior performances of tail classes in these methods may be due to the image encoder pretrained on large-scale datasets, which may potentially involve a considerable number of tail-category samples and thereby lead to an unfair comparison. Table 1 illustrates the mAP performance of different methods. Experimental results show that our proposed method outperforms previous methods by a large margin. Specifically, our proposed CPRFL achieves outstanding results, with total mAP scores of 86.28% and 66.69% on VOC-LT and COCO-LT, respectively. Compared to the previous SOTA method (CAE-Net), CPRFL exhibits notable improvements of approximately 4.67% and 9.05% in total mAP on VOC-LT and COCO-LT. Moreover, CPRFL excels in all three class subsets (head, medium, and tail classes), with particularly noteworthy gains observed in tail classes, achieving a remarkable 3.7% mAP increase on COCO-LT. In VOC-LT, performance enhancements are primarily observed in both head and medium classes. These results fully demonstrate the efficacy of CPRFL in achieving synchronous improvements in head-to-tail recognition performance.To thoroughly investigate the impact of each component within the proposed CPRFL framework, we conducted a series of ablation studies on the influence of classification loss, category semantics used for prompt initialization, and the components of CPRFL.Multi-Label Classification Loss. To further address the negativepositive sample imbalance problem, we adopt the Asymmetric Loss (ASL) [41] as the multi-label classification loss, which effectively suppresses negative samples across all classes. In this part, we compare several classification losses for optimizing our CPRFL approach, including Binary Cross-Entropy Loss (BCE), Multi-Label Soft Margin Loss (MSL), Class-Balanced Focal Loss (CB Focal) [11] , Distribution-Balanced Loss (DB Focal) [52] , a No-Focal version of DB Loss, and ASL [41] . Table 2 lists the comparison results on VOC-LT and COCO-LT. The results demonstrate that ASL consistently outperforms other classification losses for the LTMLC tasks. This superior performance can be attributed to ASL's ability to account for the dominance of negative-positive imbalance across all classes in LTMLC, a key aspect that the other losses do not adequately address.Category Semantics for Prompt Initialization. We further conduct a comparative analysis of CPRFL's performance with various types of category semantics for prompt initialization to explore whether the notable improvement of our approach is primarily attributed to the powerful pretrained linguistic knowledge in CLIP. Specifically, we evaluate three different word embeddings, i.e., Randn (random initialization embedding), Word2Vec [37] , GloVe [39] , and CLIP-RN50, CLIP-ViT16. The results on COCO-LT are presented in Figure 2 . From the figure, it is clear that CPRFL utilizing CLIP outperforms the Word2Vec and GloVe embeddings across all classes. This can be attributed to CLIP's multi-modal pre-training, which enhances the alignment of semantic embedding with visual concepts and strengthens the representation ability of category semantics. Additionally, our CPRFL with Randn embedding (63.9%) surpasses all previous state-of-the-art methods (57.6%) as reported in Table 1 . This suggests that even without leveraging CLIP's pretrained semantic knowledge, our CPRFL approach still outperforms other methods in the LTMLC tasks.Components Analysis of CPRFL. To evaluate the contributions of various components to our method for long-tailed multi-label classification, we conduct a series of ablation studies, with the results summarized in Table 3 . Our baseline experiment uses focal loss [30] and achieves mAP performances of 73.88% on VOC-LT and 49.46% on COCO-LT. Our CPRFL framework comprises two key components: Visual-Semantic Interaction (VSI) and Prompt Initialization (PI). VSI leverages category-semantic prompts to decouple category-specific visual representations from samples, while PI establishes semantic correlations between different categories using CLIP. Adding these two components leads to significant improvements across head, medium, and tail classes, and the overall mAP outperforms the baseline by 10.34% on VOC-LT and 14.80% on COCO-LT. We attribute these gains to the Dual-Path Back-Propagation mechanism guided by this two components, which can refine the category-prompts and enhance visual-semantic interaction, thereby progressively purifying category-specific features during iterative training. Additionally, integrating VSI, PI with the Re-Weighting (RW) strategy further improves mAP performance, particularly for the tail classes, with increases of 4.27% on VOC-LT and 4.13% on COCO-LT. To better understand how our method handles long-tailed multilabel data, we conduct qualitative experiments using ResNet-50, CLIP, and our proposed CPRFL.  
conclusion: 
 To tackle the challenges of head-to-tail imbalance and multi-object recognition in long-tailed multi-label image classification, we propose a novel and effective approach, termed Category-Prompt Refined Feature Learning (CPRFL). CPRFL capitalizes on CLIP's text encoder to extract category semantics, leveraging its robust semantic representation capability. This allows for the establishment of semantic correlations between the head and tail classes. The derived category semantics are then utilized as category-prompts, facilitating the decoupling of category-specific visual representations. Through a series of dual-path gradient back-propagations, we refine these prompts to effectively mitigate the visual-semantic domain bias. Simultaneously, the refinement process aids in purifying the category-specific visual representations under the guidance of the refined prompts. To our knowledge, this is the pioneering work to leverage category semantic correlations for mitigating head-to-tail imbalance in LTMLC, offering an innovative solution tailored to the unique characteristics of the data.  
: 
 image classification tasks [19, 26, 45, 47] . This progress greatly relies on many mainstream balanced benchmarks (e.g., CIFAR [25] , ImageNet ILSVRC [12] , MS COCO [31] ), which have two key characteristics: 1) they provide a relatively balanced and sufficient number of samples across all classes, and 2) each sample belongs to only one category. However, in real-world applications, the distribution of different categories often follows a long-tailed pattern [34, 57] , where deep networks tend to underperform on tail classes. Meanwhile, unlike the classical single-label classification, practical scenarios frequently involve images associated with multiple labels [9, 27, 32, 50] , adding complexity and challenge to the task. To address these issues, an increasing number of works focus on the problem of Long-Tailed Multi-Label image Classification (LTMLC) [8, 16, 29, 52] .As the samples of tail classes are relatively scarce, mainstream methods for solving LTMLC focus on addressing the head-to-tail imbalance by employing various strategies, such as resampling the number of samples for each category [2, 3, 43, 52] , re-weighting the loss for different categories [4, 11, 16, 51] , and decoupling the learning of representation and classification head [23, 58] . For example, the Decoupling approach [23] designs four distinct sampling strategies evaluated for representation learning of long-tailed data. Similarly, Label-Distribution-Aware Margin (LDAM) [4] enforces class-dependent margin factors for different classes based on their training label frequencies. Although these methods have made significant contributions, they have often overlooked two crucial aspects. Firstly, it is of great importance to consider the semantic correlations between the head and tail classes in long-tailed learning. Leveraging such correlations can substantially improve the performance of tail classes with the support of head classes. Secondly, real-world images often encompass a variety of objects, scenes, or attributes, adding complexity to the classification task. The aforementioned methods typically consider extracting the visual representation of images from a global perspective. However, this global visual representation contains mixed features from multiple objects, hindering effective feature classification for each category. Therefore, how to explore semantic correlations between categories and extract local category-specific features in long-tailed data distributions remains a critical area of research.Recently, Visual-Language Pretrained (VLP) models [40, 41, 59, 60] have been successfully adapted to various downstream visual tasks. For instance, CLIP [41] , pretrained on billions of samples of image-text pairs, contains abundant linguistic knowledge from the Natural Language Processing (NLP) corpora in its text encoder.The text encoder demonstrates substantial potential in encoding semantic context representations within the text modality. Hence, it is feasible to leverage CLIP's text embedding representations to encode the semantic correlations between the head and tail classes. Furthermore, in numerous studies, CLIP's text embedding has been successfully employed as semantic prompts to decouple local category-specific visual representations from global mixed features [7, 27, 32] .To tackle the challenges inherent in Long-Tailed Multi-Label Classification (LTMLC), we propose a novel and effective approach named Category-Prompt Refined Feature Learning (CPRFL). CPRFL leverages CLIP's text encoder to extract category semantics, thereby enabling the establishment of semantic correlations between the head and tail classes. This is achieved through the robust semantic representation capabilities of CLIP's text encoder. Subsequently, the extracted category semantics are utilized to initialize prompts for all categories, which interact with visual features in order to discern context-related visual information specific to each category. Such visual-semantic interaction can effectively decouple category-specific visual representations from the input samples. However, these initial prompts lack visual-context information, resulting in a significant data bias between the semantic and visual domains during information interaction. In essence, the initial prompts may not be precise, thereby compromising the quality of category-specific visual representations. To mitigate this issue, we introduce a progressive Dual-Path Back-Propagation mechanism to iteratively refine the prompts. This mechanism progressively accumulates context-related visual information into the prompts. Concurrently, the category-specific visual representations are purified under the guidance of the refined prompts, enhancing their relevance and accuracy. Finally, to further address the negativepositive imbalance problem inherent in multiple categories, we incorporate the Re-Weighting (RW) strategy, commonly utilized in such scenarios. Specifically, we employ the Asymmetric Loss (ASL) [42] as our optimization objective, which effectively suppresses negative samples across all classes and potentially improves headto-tail category performance in LTMLC tasks.The contributions of our work can be summarized as follows:• We propose a novel prompt-learning approach termed Category Prompt Refined Feature Learning (CPRFL), for Long-Tailed Multi-Label image Classification (LTMLC). CPRFL leverages CLIP's text encoder to extract category semantics, harnessing its powerful semantic representation capability. This facilitates the establishment of semantic correlations between the head and tail classes. The extracted category semantics serve as category-prompts to enable the decoupling of category-specific visual representations. To the best of our knowledge, this is the first work to utilize category semantic correlations to mitigate the head-to-tail imbalance problem in LTMLC, offering a pioneering solution tailored to the distinctive characteristics of the data. • We design a progressive Dual-Path Back-Propagation mechanism aimed at refining the category-prompts by progressively incorporating context-related visual information into prompts during visual-semantic interaction. By employing a series of dual-path gradient back-propagations, we effectively counteract the visual-semantic domain bias stemming from the initial prompts. Simultaneously, the refinement process facilitates the progressive purification of categoryspecific visual representations.• We conduct experiments on two LTMLC benchmarks, including the publicly available datasets COCO-LT and VOC-LT . Extensive experiments not only validate the effectiveness of our approach but also highlight its significant superiority over the recent state-of-the-art approaches. 
 
 Appendix:  


Decision: The paper proposes a Category Prompt Refined Feature Learning (CPRFL) method for Long-Tailed Multi-Label Image Classification (LTMLC). CLIP’s text encoder is used to establis semantic correlations between the head and tail classes, and a dual-path gradient back-propagations is used to refine the prompts.  Three reviewers give positive scores, and two eviews raise their scores after rebuttal. The AC is inclined  to accept.

Quality: Using head-to-tail correlation to mitigate the imbalance of long-tail problems is reasonable, and the proposed method is effective.  

Clarity: The paper is well written but it should  be re-organized to fill the blank of lines 149-152.

Originality: The motivation of the paper is good, and the method is simple yet effective. 

Significance： The proposed method is simple and effective, and thus can be applied to other long-tailed tasks and real-world scenarios.

Reviews: review: paper_summary: paper_summary: Authors propose a new method called Category-Prompt Refined Feature Learning (CPRFL) for long-tailed multi-label image classification. A text encoder is firstly used to extract category semantics and further establish the semantic association between the head and tail classes. Then, a progressive dual-path back-propagation mechanism is proposed to iteratively accumulates context-related visual information into the category-prompts, and purify the category-specific visual representations under the guidance of the refined prompts. Lastly, the commonly utilized Re-Weighting strategy is incorporated to address the negative-positive imbalance problem in multiple categories.
strength_weakness: strength_weakness: The novelty of this paper primarily reflect in two main aspects: (a) The category semantics are extracted to establish the semantic association between the head and tail classes, and (b) the progressive dual-path back-propagation mechanism iteratively purify the category-specific visual representations and refine the category-prompts.
Comprehensive experiments are conducted to prove the performance of the proposed method, including the quantitative comparison with other methods, the qualitative analysis of retrieval results, and the ablation study. The experimental results show that the proposed method outperforms the compared methods.
The paper is well wrote.
limitations: limitations: (1) The used encoders in the paper should be explained.
(2) Figure 1 in the paper describes the overall framework, which focuses on the long-tail problems. However, how to deal with the multi-label image classification problem is not shown clearly.
(3) In the experimental part, it is better to add some comparative experiments to verify the effectiveness of the proposed method. For example, authors can try other text encoders such as Word2Vec, GloVe, and so on, to prove the superiority of CLIP.
(4) In the conclusion part of this paper, the possible research direction in the future should be suggested.
paper_summary: paper_summary: This paper introduces the  Category-Prompt Refined Feature Learning (CPRFL) to address Long-Tailed Multi-Label image Classification (LTMLC).The authors introduced the semantic prior and conducted tests on two datasets with varying configurations. Furthermore, the authors illustrate that the proposed approach outperforms others.
strength_weakness: strength_weakness: 1. This paper is generally well-structured and articulated.
2. The authors have conducted extensive experiments across various datasets and settings, achieving strong performance.
limitations: limitations: 1. From Table 3, the effect of PI network improvement is limited, and the authors mention CoOp[1],CoCoOp[2] in the supplemental materials, a more interesting ablation study is to replace the PI with CoOp and CoCoOp while keeping other modules unchanged to show the effectiveness of PI module.

2.  The pretrained text encoder may contain implicitly image information, since CLIP aligns the text embedding space and image feature space, so whether leveraging pretrained text encoder is a too-strong prior knowledge also needs to be validated, and the authors need to ablate on the initialization parameters parameters of text encoders.

3. The author claims the enhanced tail-class performance attributes to the pretrained image encoder, how about the performance finetuning using the pretrained image encoder.   

Overall, the the innovation points are unattractive. Although relatively good performance has been achieved, there is a lack of experiments that sufficiently illustrate the effectiveness of the contributions.

[1]  Learning to prompt for vision-language models. 
[2] Conditional prompt learning for vision-language models
paper_summary: paper_summary: The paper proposes a novel method, Category-Prompt Refined Feature Learning (CPRFL), to address the challenges of Long-Tailed Multi-Label image Classification (LTMLC). The primary goals are to mitigate the imbalanced data distribution and enhance multi-object recognition. CPRFL utilizes semantic correlations between different categories and decouples category-specific visual representations for each category. This is achieved by initializing category-prompts from pretrained CLIP embeddings and refining them through a progressive Dual-Path Back-Propagation mechanism. The approach aims to suppress negative samples across all classes using Asymmetric Loss, thereby improving head-to-tail recognition performance. The effectiveness of CPRFL is validated through extensive experiments on two LTMLC benchmarks, demonstrating its superiority over existing methods.
strength_weakness: strength_weakness: * __Adequate Evaluation:__ The paper provides comprehensive experimental validation of the proposed method. CPRFL is tested on two well-known LTMLC benchmarks, COCO-LT and VOC-LT, and the results demonstrate significant improvements over state-of-the-art methods. The experiments are thorough, with performance metrics (mean Average Precision) reported for overall, head, medium, and tail classes, highlighting the method's effectiveness across different categories. Additionally, ablation studies and comparisons with various baseline methods further substantiate the robustness and superiority of CPRFL.
* __Clarity:__ The paper is generally well-written, with clear explanations of the methodology, experimental setup, and results. The use of figures and diagrams, such as the overall framework of CPRFL and the visual-semantic interaction process, aids in understanding the complex interactions and processes involved. The structured presentation of contributions and detailed description of the dual-path back-propagation mechanism enhance the readability and comprehensibility of the paper.
limitations: limitations: * __Lack of Detailed Comparison with Similar Methods:__ The paper does not thoroughly compare CPRFL with other CLIP-based methods (such as https://arxiv.org/pdf/2305.04536). While it mentions avoiding direct comparisons due to the potential pretraining bias of CLIP's image encoder, it is essential to include a detailed discussion on how CPRFL differs from or improves upon these methods. This omission leaves a gap in understanding the relative advantages of CPRFL over similar state-of-the-art techniques.
﻿* __Handling of Negative-Positive Sample Imbalance:__ Although the paper adopts Asymmetric Loss to address the negative-positive sample imbalance, there is limited discussion on why this particular loss function was chosen over others and how it compares to other imbalance-handling techniques in detail. Additionally, the paper does not explore the impact of different settings of the Asymmetric Loss parameters (γ+ and γ−) on the final performance, which could be crucial for fine-tuning in various applications.
paper_summary: paper_summary: This paper proposes a very simple method that uses the text-encoder of CLIP to extract initial category semantics, and an interaction module between categories and visual information to update the category semantics considering both object relationships and visual inputs.  
Then the input category semantics of the interaction module are used as classifiers to tag the output updated category semantics.
strength_weakness: strength_weakness: 1. A simple method to learn relationship information from CLIP to classify tail categories of LTMLC.
2. Not only the CLIP, but the GloVe embedding also works for the proposed method with only a little performance drop.
limitations: limitations: 1.  The RW is a useful trick to improve performance, which is not the concern of this paper. Despiting the RW, the results of the penultimate line in Table 3,  are inferior on tail categories compared with the results in Table 1.  
2. Though the paper only uses the text encoder of CLIP discarding the image encoder,  the text encoder has been trained along with the image encoder using text image paired data. So claims in lines 204-207 are somewhat untrustworthy.  
3. eq.(7) missing the labels Y.