LMPT: Prompt Tuning with Class-Specific Embedding Loss for Long-Tailed Multi-Label Visual Recognition (2024)

Abstract: Long-tailed multi-label visual recognition (LTML) task is a highly challenging task due to the label co-occurrence and imbalanced data distribution. In this work, we propose a unified framework for LTML, namely prompt tuning with class-specific embedding loss (LMPT), capturing the semantic feature interactions between categories by combining text and image modality data and improving the performance synchronously on both head and tail classes. Specifically, LMPT introduces the embedding loss function with class-aware soft margin and re-weighting to learn class-specific contexts with the benefit of textual descriptions (captions), which could help establish semantic relationships between classes, especially between the head and tail classes. Furthermore, taking into account the class imbalance, the distribution-balanced loss is adopted as the classification loss function to further improve the performance on the tail classes without compromising head classes. Extensive experiments are conducted on VOC-LT and COCO-LT datasets, which demonstrates that our method significantly surpasses the previous state-of-the-art methods and zero-shot CLIP in LTML. Our codes are fully public at https: //github.com/richard-peng-xia/LMPT.

heading: 1 Introduction
text: Long-tailed multi-label visual recognition (LTML) (Wu et al., 2020; Guo and Wang, 2021) is a common and practical task owing to the highly imbalanced data distribution (Zhang et al., 2021b) and diverse objects of real-world images (Wang et al., 2017; Ju et al., 2023). Compared with longtailed recognition and multi-label recognition tasks, LTML is more complex and challenging, because it requires capturing multiple categories and the label co-occurrence in individual images (Chen et al., 2019a), which needs to compensate for the negative impacts caused by the long-tailed distribution (i.e., low performance on the tail classes).
Several approaches have been proposed to address the LTML problem from different perspectives, such as re-sampling (Buda et al., 2018; Dong et al., 2017; Guo and Wang, 2021), re-weighting (Cao et al., 2019; Wu et al., 2020) and modeling more powerful structures (Chen et al., 2019a; Wang et al., 2016, 2017). Despite their great contributions, these works neglect to take into account two crucial aspects. First of all, the importance of semantic feature interaction between classes to capture label co-occurrence. However, these methods are limited to balancing the distribution of categories from the perspective of samples, without considering the feature correlation between different classes. Second, synchronous improvements in head-to-tail category performance, while some of these works improve the performance of tail classes at the expense of the head classes.
Recently, graphic models have been introduced
26
to model the semantic label correlation in a few works (Chen et al., 2019a; Wang et al., 2016), whereas these works are complex and are modeling label dependencies mainly based on the image modality without additional semantic information from other modal data. Vision-language models (VLMs) (Radford et al., 2021; Jia et al., 2021; Tian et al., 2022; Huang et al., 2022; Xia et al., 2024a) demonstrate the huge potential of text modality on semantic context feature for downstream visual tasks, especially for the prompt tuning methods (Schick and Schütze, 2021; Shin et al., 2020; Yao et al., 2021; Xia et al., 2023), which provide an efficient way to transfer pre-trained VLMs to downstream tasks by learning the task-specific prompts rather than finetuning the entire model. Nonetheless, the existing prompt tuning methods (Zhou et al., 2022b,a; Sun et al., 2022) for visual recognition simply minimize prediction errors using the classification loss (e.g., cross-entropy loss) with respect to the learnable prompts, which may lead to learning general embeddings or inaccurate classrelated embeddings. For instance, when presented with an image (Fig.1a) that contains both a head class [person] and a tail class [horse], the zero-shot method (Fig.1b) relies solely on the rich knowledge of the pre-trained VLMs to assess the similarity between the image and the word embeddings of the class names, while the existing prompt tuning method (Fig.1c) further learns more generalized prompt tokens to improve model performance. However, these methods do not consider the interclass relationships, particularly between head and tail classes, which is a critical factor for LTML. This underscores the need for approaches that incorporate such relationships to improve performance in such scenarios.
Therefore, to address these issues, we present the class-specific embedding loss for prompt tuning on long-tailed multi-label visual recognition, called LMPT. The abundance of image-caption data facilitates prompt learning that encompasses more nuanced and specific textual descriptions, as well as the semantic inter-dependencies between categories (Fig.1d) that share information, such as similar features or common descriptions. This attribute is particularly critical in the identification of both head and tail classes. More specifically, we propose the class-specific embedding loss to enhance the inclusivity of class-related embeddings within prompts. By gradually approaching
the embeddings of the corresponding caption, our proposed approach enables prompt tokens to effectively judge the association between different classes with the aid of textual modality. Aiming for class imbalance and consistency improvements between head classes and tail classes, we integrate class-aware soft margin and re-weighting into the class-specific embedding loss, which serves to assign larger margins and more weights to tail classes. Notably, for images containing both head and tail classes, our approach outperforms visual models and current prompt tuning methods. Moreover, we adopt the distribution-balanced loss (Wu et al., 2020) as the classification loss. To sum up, the main contributions of this work include:
• We propose the LMPT framework to adapt pretrained VLMs to tackle long-tailed multi-label visual recognition, where captions are easily accessible from public image-caption datasets or generated by powerful image-caption models (Wang et al., 2022).
• We present a novel class-specific embedding loss with class-aware soft margin and re-weighting to learn more fine-grained and class-related embeddings that build semantic relationships across head and tail classes with shared semantic information. Such design can benefit performance in tail classes and hard-to-recognize classes with the help of text modality.
• We verify the effectiveness of the proposed method by achieving new state-of-the-art (SOTA) results on two datasets, which outperform previous SOTA (Guo and Wang, 2021) by 9/6% and zero-shot CLIP by 6/2% on VOC-LT / COCOLT.

heading: 2 Related Work

heading: 2.1 Long-Tailed Visual Recognition
text: Real-world training data usually exhibits longtailed distribution (Zhang et al., 2021b), which presents a challenge for traditional methods due to the imbalanced class distribution. To address this problem, several approaches (Cui et al., 2022; Menon et al., 2020; Ouyang et al., 2016; Samuel and Chechik, 2021; Xia et al., 2024b) have been proposed from different aspects. One common method is to directly re-sample the training data to balance the class distribution (Drummond et al., 2003; Buda et al., 2018; Dong et al., 2017), by adjusting the sampling rate of head classes and tail
classes, yet it might lead to the overfitting of tail classes. A better solution is to design re-weighted loss functions (Khan et al., 2017; Huang et al., 2016; Cao et al., 2019) that assign more weight to tail classes or ignore negative gradients (Tan et al., 2020) for tail classes. In addition, researchers also propose to use techniques such as transfer learning (Liu et al., 2019; Zhu and Yang, 2020) and self-supervised learning (Kang et al., 2020; Zhang et al., 2021a) to alleviate the class imbalance problem. Recently, some studies (Ma et al., 2021; Tian et al., 2022) also explore the possibility of text modality by refining visual-language representations on the long-tailed recognition tasks.

heading: 2.2 Multi-Label Visual Recognition
text: For multi-label visual recognition, some early methods include treating it as multiple binary image classifications (Tsoumakas and Katakis, 2007; Zhang and Zhou, 2013) or finding k-nearest neighbors (Zhang and Zhou, 2007). To locate regions of interest, some researchers (Wang et al., 2016, 2017) proposed to introduce recurrent neural networks (e.g., RNN, LSTM) to learn a joint imagelabel embedding. In addition, Chen et al. (Chen et al., 2019a) proposed to model the label correlations by constructing a graph based on the label co-occurrence and Ye et al. (Ye et al., 2020) updated static graph to dynamic graph convolutional network (GCN) for robust representation. Wu et al. (Wu et al., 2020) proposed a distributionbalanced loss and Guo et al. (Guo and Wang, 2021) adopted collaborative training on the uniform and re-balanced samplings to alleviate the class imbalanced problem. There is also a popular trend to align between visual and textual features (Xu et al.,
2022; Liu et al., 2021; Huang et al., 2022; Ridnik et al., 2023) for multi-label recognition.

heading: 2.3 Prompt Tuning for Vision-Language Models
text: Prompt tuning (Schick and Schütze, 2021; Shin et al., 2020; Yao et al., 2021) is a parameterefficient technique used to utilize the representation ability of pre-trained vision-language models to achieve better performance instead of fine-tuning the whole model on downstream tasks. Meanwhile, large-scale vision-language models (e.g., CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021)) have demonstrated impressive power to learn visual and textual features. CoOp (Zhou et al., 2022b) learns soft prompts via minimizing the classification loss and CoCoOp (Zhou et al., 2022a) further formulates the prompts in an imageconditional way to improve its generalization to unseen classes. DualCoOp (Sun et al., 2022) firstly adapts CLIP to multi-label image recognition by learning pairs of positive and negative prompts for each class, then TaI-DPT (Guo et al., 2023) extracts both coarse-grained and fine-grained embedding by treating texts as images in prompt tuning. Different from the above work, LMPT focuses on exploring the transfer ability to address long-tailed multi-label visual recognition.

heading: 3 Methodology
text: In this section, we present our proposed prompting tuning method, i.e., LMPT, for adapting pretrained vision-language models for long-tailed multi-label visual recognition.

heading: 3.1 Preliminaries
text: Consider D as the dataset we use, N as the number of the dataset, C as the number of classes, and L as the fixed length of contexts for optimization. Then (xk, yk, tk) ∈ Dtrain, k ∈ {1, ..., N}, where xk is an input single image, yk =[ yk1 , ..., y k C ] ∈ {0, 1}C is the multi-label ground-
truth and tk = [ tk1, ..., t k L ] is the corresponding text embedding of text description (caption). But during the test phase, only (xk, yk) ∈ Dtest. Let ni = ∑N k=1 y k i denote the number of training examples that contain class i. Please note that labels for computing the class-specific embedding loss need to be processed into ỹk = [ ỹk1 , ..., ỹ k C ] =[
2 ∗ yk1 − 1, ..., 2 ∗ ykC − 1 ]
∈ {−1, 1}C , where {−1, 1} indicates negative and positive.

heading: 3.2 Approach Overview
text: In order to make effective use of the linguistic modality in the long-tailed multi-label visual recognition task, we propose a novel framework (i.e., LMPT), as depicted in Fig. 2. Text encoder from the pre-trained CLIP is used to encode the prompts and text descriptions (captions) of images. Only the parameters in the prompts are optimized, while the text encoder and image encoder are both kept frozen. We introduce two sorts of trainable prompts to obtain class embedding, which are jointly optimized by the classification loss Lcls and classspecific embedding loss Lcse. Details of the aforementioned loss functions will be introduced in the later sections.

heading: 3.3 Prompt Tuning
text: Formally, the vision-language model consists of an image encoder f(·) and a text encoder g(·). Following (Zhou et al., 2022a), a prompt is defined as:
oi|M1 = [V]1 [V]2 ... [V]m ... [V]M [CLASS] , (1)
where i ∈ {1, ..., C}, m ∈ {1, ...,M}, the [CLASS] token is replaced by the specific class name (e.g., “cat,” “dog”, “car”), each [V]m is a learnable word embedding with the same dimension as normal word embeddings in the vocabulary (i.e., 512 for CLIP), and M is a hyper-parameter specifying the number of context tokens. The prediction probability (classification output) z is then
computed as:
p(y = i | x) = exp (cos (g (oi) ,f (x)) /τ)∑C j=1 exp (cos (g (oj) ,f (x)) /τ) , (2) where τ is a temperature parameter learned by CLIP and cos(·, ·) represents cosine similarity.

heading: 3.4 Class-Specific Embedding Loss
text: We introduce the class-specific embedding (CSE) loss to optimize the trainable fine-grained instance prompts by learning from text embeddings of captions. It tries to minimize the cosine distance of matching patches and to increase the cosine distance of non-matching patches above the margin. Embedding loss is then computed as
ℓebd =
{ ∆ki , if ỹ k i = 1,
max ( 0, µ−∆ki ) , if ỹki = −1,
∆ki = 1− cos ( tki , oi|Mm ) ,
(3)
where µ is the margin factor. Intuitively the embedding loss penalizes positive (i.e., prompts of matching classes) pairs that have large distances and negative (i.e., prompts of non-matching classes) pairs that have small distance (less than µ).
LDAM (Cao et al., 2019) has inspired the development of a decision boundary that is both robust and generalizable, capable of accurately classifying features that vary within a certain range. However, when applied to long-tailed datasets characterized by a significant class imbalance, models tend to exhibit greater sensitivity to more frequent classes. As a result, the performance of these models in less frequent classes is often poor.
To address this issue, CSE loss employs the class-aware soft margin strategy to encourage the model to have the optimal trade-off between perclass margins by stimulating the minority classes to have larger margins, which can be viewed as
regularization (Wei et al., 2018). More specifically, as illustrated in Fig. 3, blue samples (head classes) are classified incorrectly, and the model update gradient is shown with pointed arrows. Green samples (medium classes) are classified correctly outside of the margin and the gradient is shown. Intuitively, the embedding loss does not give special consideration to the minority categories, but with the help of class-aware soft margin, the trade-off of µ1 (in Fig. 3) can be optimized by shifting the decision boundary to encourage the tail classes to have larger margins. So yellow samples (tail classes) are classified correctly outside of the original margin but within the enlarged margin, and the embedding loss has no gradient for these samples. Following the trade-off between the class margins, we adopt a class-aware margin for multiple classes of the form
µ̃i ∝ n−1/4i = η
n 1/4 i
. (4)
Here η is a hyper-parameter to be tuned. Therefore, when yki = −1, the loss can be computed as max { 0, µ̃i −∆ki } .
Meanwhile, our loss can be combined with a re-weighting strategy to be more efficient when it comes to long-tailed distribution data. We then define the reference weight based on the empirical class frequencies {n1, ..., nC} on the training set:
wi = (1/ni)
γ
∑C i=1 (1/ni) γ , (5)
where γ is a scale hyper-parameter to provide more flexibility. Hence, the re-weighted class-specific embedding loss is defined as:
ℓcse =
{ wi∆ k i , if ỹ k i = 1,
max { 0, wi ( µ̃i −∆ki )} , if ỹki = −1,
(6)
Lcse = ∑N
k=1 ℓcse N . (7)
The overall process of class-specific embedding loss is outlined in Algorithm 1.

heading: 3.5 Multi-Label Classification Loss
text: Our method can be easily combined with the existing multi-label classification loss functions (Ridnik et al., 2021; Lin et al., 2017; Cui et al., 2019; Wu et al., 2020), regardless of whether they are designed for long-tailed distributions or not. By
Algorithm 1: Class-Specific Embedding Loss
Input: Text embeddings of textual descriptions (captions) t, labels ỹ, prompt o
Output: Class-Specific Embedding Loss Lcse 1 for k = 1, 2, ..., N do 2 ℓcse = 0; 3 for i = 1, 2, ..., C do 4 Calculate class-aware soft margin µ̃i by Eq. 4; 5 Calculate weight wi by Eq. 5; 6 Calculate ∆ki = 1− cos ( tki , oi|Mm ) ; 7 if ỹki = 1 then 8 ℓcse = wi∆ k i ; 9 else 10 ℓcse = ReLU ( wi ( µ̃i −∆ki )) ; 11 Calculate Lcse by Eq. 7.
blending the classification loss functions with our proposed CSE loss, our method facilitates prompt learning of more refined class descriptions and semantic relationships between categories, particularly between head and tail classes.
In this study, we introduce the distributionbalanced loss (Wu et al., 2020) as the classification loss function, which can be formulated as:
r = α+ σ ( β × ( 1 ni∑C i=1 1 ni − θ )) , (8)
vi = −κ×− log ( 1 ni/N − 1 ) , (9)
ℓcls =
{ −r ( 1− qki )γ log ( qki ) , if yki = 1,
− rζ ( qki )γ log ( 1− qki ) , if yki = −1,
(10) where qki = σ ( zki − vi ) is for positive instances,
qki = σ ( ζ ( zki − vi )) is for negative ones and α, β, θ, κ, ζ are hyperparameters. Then Lcls =∑N k=1 ℓcls/N . Hence, the overall training loss can be written as: L = λLcls + (1− λLcse) , (11)
where λ ∈ [0, 1] is a hyperparameter to balance Lcls and Lcse.

heading: 4 Experiment

heading: 4.1 Benchmark Setting
text: Following (Wu et al., 2020; Guo and Wang, 2021), we conduct experiments on two datasets for longtailed multi-label visual recognition: VOC-LT and COCO-LT (Wu et al., 2020). They are artificially
sampled from two multi-label recognition benchmarks, PascalVOC (Everingham et al., 2015) and MS-COCO (Lin et al., 2014), respectively.

heading: 4.2 Experimental Settings
text: Metrics. As in (Liu et al., 2019), the classes are split into three groups by the number of their training examples: head classes each contain over 100 samples, medium classes each have between 20 and 100 samples, and tail classes with under 20 samples each. We use mean average precision (mAP) to evaluate the performance of long-tailed multilabel visual recognition for all the classes. Implementation Details. We adopt CLIP ResNet50 (He et al., 2016) or ViT-B/16 (Dosovitskiy et al., 2020) as the visual encoder and use the corresponding CLIP Transformer as the text encoder. During training, the parameters of both the two encoders are kept frozen, and only learnable prompts are optimized. SGD optimizer is adopted to learn prompt tokens, and the training epochs are set to 30. The learning rates for COCO-LT and VOC-LT are empirically initialized with 1e-4, 5e-4, and decay by the cosine annealing rule during training. For loss functions, η in Eq. 4, γ in Eq. 5 and λ in Eq. 11 are
set as 1.0, 1.0 and 0.5, respectively. Other hyperparameters in DB loss are set as the same as (Wu et al., 2020).

heading: 4.3 Long-Tailed Multi-Label Visual Recognition
text: To evaluate the effectiveness of the proposed method, firstly we compare it with previous methods of image-only models on the two long-tailed multi-label datasets. The compared methods include Empirical Risk Minimization (ERM), a smooth version of Re-Weighting (RW) using the inverse proportion to the square root of class frequency, Re-Sampling (RS) (Shen et al., 2016), Focal Loss (Lin et al., 2017), ML-GCN (Chen et al., 2019b), OLTR (Liu et al., 2019), LDAM (Cao et al., 2019), Class-Balanced (CB) Focal (Cui et al., 2019), BBN (Zhou et al., 2020), DistributionBalanced (DB) Focal (Wu et al., 2020) and LTML (Guo and Wang, 2021). The mAP performance of different methods is shown in Table 1. The prior best performance is achieved by LTML – mAP of 81.44% over all classes on VOC-LT and 56.90% over all classes on COCO-LT.
Furthermore, we compare zero-shot and prompt
learning methods based on CLIP on the two benchmarks. The mAP performance of these methods is shown in Table 1 as well. For a fair comparison, we initialize the prompt as the default hand-crafted one “a photo of a" for all the methods. The results show that when using ViT-B/16 as the backbone, even the overall mAP performance of zero-shot CLIP reaches 85.77% and 60.17%, which outperforms previous SOTA LTML by 4.33 points (85.77% vs.81.44%) and 3.27 points (60.17% vs.56.90%) on the two datasets, respectively. Therefore, it is meaningful to explore how to use prompt tuning based on CLIP effectively for better performance. From the perspective of prompt tuning methods, when using ResNet-50 as the backbone, the performance of our method on VOC-LT is more promising, which is 4.1 points, 6.81 points, 4.41 points and 1.69 points better than CoOp, CoCoOp, DualCoOp and TaI-DPT, which are popular prompt learning methods for single-label and multi-label recognition. The performance on COCO-LT is similar to that on VOC-LT, which is 4.03 points, 12.95 points, and 5.86 points better than CoOp, CoCoOp, and DualCoOp. When replacing the backbone with ViT-B/16, the overall mAP performance of our method can further boost up to 87.88% and 66.19% on VOC-LT and COCO-LT, which is the current new state-of-the-art of the two datasets.

heading: 4.4 Ablation Analysis
text: Components Analysis. To further analyze which component makes our methods performant for LTML, we conduct a set of ablation studies and report the results in Table 3. We first conduct exper-
iments with CLIP and the mAP performances are 85.77% on VOC-LT, 60.17% on COCO-LT, which surprisingly outperforms the prior SOTA LTML. It indicates that pre-trained VLMs demonstrate a robust capability for visual recognition, providing a solid foundation for our approach. However, the mAP performance of the tail classes outperforms the head classes by nearly 30 points on both VOC-LT and COCO-LT. Then CoOp is benefited from soft prompts and the mAP performance is improved to 86.02% on VOC-LT and 60.68% on COCO-LT, with 0.25% and 0.51% increments. Besides, we design the class-specific embedding loss with class-aware soft margin and re-weighting to learn more fine-grained and class-related prompts that build semantic relationships across different classes, especially for the tail classes by encouraging those classes to have larger margins and weights. The mAP performances of head, medium, and tail classes after adding the embedding loss are all significantly improved and the overall mAP surpasses CoOp by 1.26% and 4.66% on VOC-LT and COCO-LT, which demonstrates our embedding loss can help prompts learn fine-grained classes descriptions and semantic relationships across the classes. Finally, the integration of CASM and RW strategy further improves the mAP performance slightly, mainly for the tail performance by 0.65% and 1.12% on VOC-LT and COCO-LT. Multi-Label Classification Loss Functions. We compare a number of multi-label classification loss functions, including Binary Cross-Entropy Loss (BCE), Multi-Label Soft Margin Loss (MSL), Focal Loss, CB Loss, R-BCE-Focal, Asymmetric Loss (ASL) and DB Focal. As illustrated in Table 2, DB Focal loss that takes the co-occurrence of labels and the dominance of negative labels into account works significantly better than other multilabel classification loss for the LTML task. Effectiveness of Text Supervision. We further compare our method with fine-tuning CLIP’s image encoder when using ResNet-50 as the backbone to explore whether the significant effect of our approach is due to text supervision or simply because the CLIP’s image encoder is so powerful. In order to prevent interference with the trained CLIP’s image encoder during the fine-tuning phase, we only fine-tune a fully connected layer added at the end of the image encoder. The results are shown in Fig. 4. Obviously, fine-tuning the image encoder shows promising results, but still largely
underperforms LMPT, which suggests that the gradients that went through the text encoder provide more useful information.

heading: 4.5 Case Analysis
text: To better understand how our method deals with long-tailed multi-label data, we performed qualitative experiments with ResNet, CLIP, and ours on COCO-LT and VOC-LT. Fig. 5 shows several cases where the model justifies its abilities for the prediction. For example, in the third column, ResNet only recognizes [person] (belongs to head classes) and fails to classify the image to [train] (belongs to tail classes), which is a pervasive challenge encountered by image-only models. The emergence of CLIP is a great remedy for this issue, owing to its huge training data and effective text supervision. Nevertheless, simple hand-crafted templates as prompts still cannot accurately identify categories as they cannot describe the characteristics of each category. Understanding the inter-class relationships, particularly among head and tail categories, presents a formidable challenge in multi-label visual recognition, which is essential for achieving optimal performance in this domain. With the aid of our approach, utilizing prompts that learn from a large corpus of image-caption data, it has become feasible to discern the semantic relationships between categories and accurately predict the relevant categories of simple objects, even in challenging scenarios such as identifying [stop sign] from images. Therefore, our proposed method demonstrates significant advantages in effectively address-
ing the intricate relationship among multiple labels and the long-tailed problem with the aid of text supervision.

heading: 5 Conclusion
text: In this work, we propose a new view of prompt tuning for long-tailed multi-label visual recognition by learning class-specific contexts from the alignment of prompts and textual description (caption), which complements more fine-grained features and builds semantic relationships across head and tail classes. Considering the class imbalance, a novel class-specific embedding loss with the class-aware soft margin and re-weighting strategy is introduced to promote increased generalization among the tail classes. Furthermore, we integrate a distributionbalanced loss as the classification loss function in consideration of its empirical efficacy compared to alternative loss functions. Our method exhibits significant improvement over the previous state-of-theart (SOTA) and zero-shot CLIP on VOC-LT and COCO-LT. Additionally, We hope our approach will inspire future work in this field.

heading: Acknowledgement
text: We thank Monash-Airdoc Research donation Phase 2, MRFF NCRI GA89126, and MRFF AI-Eye NCRI GA89126 for their grant.

heading: Future Work
text: Additionally, We hope our approach will inspire future work in this field