Text-Region Matching for Multi-Label Image Recognition with Missing Labels

Authors: dengdi  sun
haifeng  zhao
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-15 06:20:56

Abstract: Recently, large-scale visual language pre-trained (VLP) models have demonstrated impressive performance across various downstream tasks. Motivated by these advancements, pioneering efforts have emerged in multi-label image recognition with missing labels, leveraging VLP prompt-tuning technology. However, they usually cannot match text and vision features well, due to complicated semantics gaps and missing labels in a multi-label image. To tackle this challenge, we propose Text-Region Matching for optimizing Multi-Label prompt tuning, namely TRM-ML, a novel method for enhancing meaningful cross-modal matching. Compared to existing methods, we advocate exploring the information of category-aware regions rather than the entire image or pixels, which contributes to bridging the semantic gap between textual and visual representations in a one-to-one matching manner. Concurrently, we further introduce multimodal contrastive learning to narrow the semantic gap between textual and visual modalities and establish intra-class and inter-class relationships. Additionally, to deal with missing labels, we propose a multimodal category prototype that leverages intra- and inter-category semantic relationships to estimate unknown labels, facilitating pseudo-label generation. Extensive experiments on the MS-COCO, PASCAL VOC, Visual Genome, NUS-WIDE, and CUB-200-211 benchmark datasets demonstrate that our proposed framework outperforms the current state-of-the-art methods by a significant margin. Our code is available here.

Main paper: 
 introduction: 
 Collecting high-quality and complete annotations for each image is time-consuming and labor-intensive, which hinders the application and promotion of multi-label image learning [11, 46] . An emerging research interest is relaxing the full-supervised setting to multi-label image recognition with missing labels (MLR-ML), meaning that only a subset of full labels is annotated in a multilabel image. MLR-ML has become increasingly popular owing to its reduced labeling costs, rendering it more practical for extensive use. However, this shift has also resulted in previous multi-label image recognition methods losing working (i.e., incorrect utilization of incomplete annotated data or misuse), introducing new challenge.To tackle this problem, some researchers [1, 13] have explored a few flexible solutions to study the task, such as utilizing known labels or assuming unknown labels as negative ones. In addition, Graph-based methods [6, 32] propose to model the correlation between images and labels to solve the MLR-ML problem through the semantic transfer or representation blending. Although they achieved acceptable results, their performance is significantly inferior when compared to prompt tuning based methods. For example, DualCoOp [36] pioneers the application of CoOp [55] to the MLR-ML task and achieves favorable performance, indicating that prompt tuning on the CLIP [33] is a promising avenue for advancing MLR-ML. Additionally, TaI-DPT [16] and LLM-PT [49] advocate texts as images to augment training examples, while SCPNet [12] explores structured prior knowledge to learn label relationships.Nevertheless, these prompt tuning-based methods still exhibit limitations: 1) In terms of how text and visual representations match, SCPNet focuses on text-image matching, neglecting the rich semantic information within visual data. DualCoOp utilizes text-pixels matching, which can introduce irrelevant noise due to its emphasis on pixel-level details. As shown in Figure 1(a) , there are more negatively correlated pixel-level representations than positively correlated ones for arbitrary textual representations under text-pixels matching. Regarding text-image matching in Figure 1(b ) (e.g., CoOp, SCPNet), image-level representations encompass multiple semantic objects and corresponding scenes, making it challenging for textual representations to distinguish different visual concepts. As a result, both text-image and text-pixel matching methods struggle to efficiently match the textual representation with its corresponding visual counterpart. Moreover, as described in Figure 1(c ), text-text matching (e.g., Tai-DPT, LLM-PT [49] and TaI-Adapter [56] ) is susceptible to underperforming on test images because of the heterogeneity between training and testing data.2) Current prompt tuning methods [12, 16, 36, 49, 56] struggle to effectively leverage known annotation information, leading to discarding valuable data on unknown annotations. These methods often mask out unknown labels during the calculation of binary cross-entropy loss, hindering the model's ability to learn accurate label correlations. Specifically, taking an image with categories "person", "dog", and "frisbee" as an instance, when the category "dog" is unlabeled, the model will fail to build the correlation between "dog" and any of {"person", "frisbee"} during training, which hinders recognition of "dog" in inference. Facing extremely missing labels, e.g., single positive label, the performance of existing methods will further decrease dramatically. 3) Within the joint embedding space, existing prompt tuning based methods [12, 15, 16, 36] fail to effectively align visual and text representations. This results in a significant semantic gap between modalities, hindering text-vision matching. Furthermore, these methods neglect both intra-class and inter-class semantic relationships within each modality, as well as the semantic relationships between modalities themselves. Although SCPNet [12] acknowledges the inter-class relationship within the text modality, it overlooks the intra-and inter-class relationships among visual representations and the interplay between visual and textual modalities.Based on the above investigations, as shown in Figure 1 (d), we design text-region matching for optimizing multi-label prompt tuning, which is collaborative with multimodal category prototype and multimodal contrastive learning. Firstly, we propose a Category-Aware Region Learning Module based on cross-modal attention to learn semantically relevant region-level visual representations corresponding to textual descriptions. In this way, one-to-many (text-pixels) or one-to-agnostic matching (text-image) is converted to one-to-one (text-region) matching. Secondly, visual representations are generally highly stochastic due to various contexts with external objects and object representations. Therefore, we represent each category with multiple visual prototypes to describe the object. Besides, we construct a corresponding text prototype for each category to leverage the textual knowledge embedded in CLIP effectively. Then, Multimodal Category Prototypes estimate unknown labels by leveraging intra-and inter-class semantic relationships within and across modalities. Finally, to bridge the semantic gap between vision and text and establish intra-and inter-class relationships, we propose Multimodal Contrastive Learning that maximizes intra-class similarity while minimizing inter-class similarity. This idea leverages the potential advantages of contrastive learning in learning representations [5, 42] .Our contributions to this work are summarized as follows:1) The text-region matching strategy is proposed to narrow the impact of irrelevant visual information and the semantic gap between text and vision, which is a major contribution of this work.2) Multimodal Category Prototype is designed to estimate unknown labels and generates pseudo-labels via intra-and intercategory semantic relationships.3) Multimodal Contrastive Learning is introduced to further align the models between textual and visual representations. Besides, implicit inter-class relationships can also be learned to obtain fine visual representations.4) We demonstrate through mathematical analysis how textregion matching is effective. The detailed mathematical analysis is presented in the appendix due to limited space. 
related work: 
 Multi-label Image Recognition. Multi-label image recognition (MLIR) [40, 48] is a crucial task in computer vision that involves assigning corresponding labels to various objects or semantic content contained in an image. As far as methods are concerned, they can be summarized into three categories: 1) modeling label relationship [8, 39, 43, 45] ; 2) object localization or attention mechanism [7, 50, 51] ; 3) improving loss function [34, 44] . Given the substantial cost of acquiring complete annotations, multi-label image recognition with missing labels tasks have attracted research attention. A naïve solution treats unknown labels as negative since there are more positive than negative labels per image [21, 35] . Nevertheless, these methods mistake positive labels as negative ones, resulting in undesired results. Inversely, some researchers only exploit known labels or generate pseudo-labels. For example, Durand et al. [13] propose a partial-BCE that only considers known labels and ignores unknown labels. Kim et al. [23] modify the label based on the largest loss value and use the changed label for the next training. Ben-Baruch et al. [1] design a partial asymmetric loss to select an appropriate threshold for each class to generate pseudo-labels by estimating the class distribution. Later, some researchers employ relational modeling to solve missing labels problems. SST [6] considers the structured semantic relationship, within and cross-images, to complete unknown label information. SARB [32] blends different label-level semantic representations to generate new labels and representations, then models the relationship between new representations on a co-occurrence graph.Prompt Tuning in Visual Tasks. Visual prompt tuning comes from natural language processing (NLP) [27] , a parameter-efficient learning method that is flexibly and efficiently adapted to various downstream vision tasks. CoOp [55] and its extension research [54] introduce content-optimizable prompt learning, which can perform satisfactorily with a few labeled images for each category. However, optimizing and utilizing text effectively descriptions is still a challenging problem. To alleviate this problem, ProDA [28] and PPL [25] aim to optimize the probability distribution of multiple prompts for the same visual representation using Gaussian Mixture Models. Besides, some prompt tuning-based methods are devoted to solving the multi-label recognition problem, e.g. DualCoOp [36] , TaI-DPT [16] , SCPNet [12] , HSPNet [38] , T2I-PAL [15] , LLM-PT [49] and TaI-Adapter [56] . However, unlike the impractical text and visual matching approaches of the seven methods mentioned above, we optimize matching strategies to achieve one-to-one matching between text and corresponding visual regions.Contrastive Learning. In recent years, contrastive learning [30, 41] has significantly advanced the domain of computer vision and steadily gained prominence as a mainstream technique. This methodology endeavors to learn discriminative representations by aligning similar sample pairs within the same embedding space while effectively separating dissimilar sample pairs. For example, InfoNEC [30] leverages information entropy to learn discriminative representations from unlabeled data. SimCLR [5] learns representations by maximizing the correlation between representations of augmented views of the same image. MoCo [18] uses a momentum encoder to learn representations that are invariant to transformations. Yuan et al. [52] utilize inherent data attributes within each modality and cross-modal semantic information to construct crossmodal contrastive learning. Although contrastive learning has been explored in MLIR tasks [4, 17, 29, 47] , to our knowledge, this is the first attempt to introduce multimodal contrastive learning into this task with missing labels. 
method 3.1 problem definition and model overview: 
 In the multi-label image recognition with missing labels (MLR-ML) task, an image in a dataset of 𝐶 categories can be positive, negative, or unknown, with corresponding labels of 1, -1, and 0, respectively. Similar to prior work [16] , we aim to leverage prompt tuning to learn a set of text representations that are functionally equivalent to classifiers.As illustrated in Figure 2 , to be specific, given an image 𝒙, the CLIP [33] visual encoder F (•) maps it to a visual representation 𝒇 ∈ R ℎ×𝑤×𝑑 , also called feature maps, where 𝑤, ℎ, 𝑑 are width, height and dimension respectively. Note that, visual regions 𝑹 = {𝒓 1 , 𝒓 2 , . . . , 𝒓 𝑐 } indicate salient information about the different categories in an image or its feature maps. For the 𝑐-th category, its visual region 𝒓 𝑐 = (𝒑 𝑐 , 𝒆 𝑐 ) contains the set of pixel indices 𝒑 𝑐 and their corresponding energies 𝒆 𝑐 , where each element 𝑝 = (𝑖, 𝑗) of 𝒑 𝑐 represents a spatial position on the feature map and its corresponding energy 𝑒 ∈ 𝒆 𝑐 represents the probability of that pixel belonging to the 𝑐-th category. Then, we can leverage these regions to obtain region-level representation 𝑭 = {𝒇 1 , 𝒇 2 , . . . , 𝒇 𝑐 } for each category. To achieve one-to-one matching between text and images, we define text prompts for each category: 𝒕 𝑐 = {𝜔 1 , 𝜔 2 , . . . , 𝜔 𝐿 , CLS 𝑐 }, where 𝜔 𝑖 is learnable prompt embedding and CLS 𝑐 is class token for 𝑐-th category. Then, the text encoder G(•) takes all {𝒕 1 , 𝒕 2 , . . . , 𝒕 𝑐 } as input to generate text representations 𝑮 = {𝒈 1 , 𝒈 2 , . . . , 𝒈 𝑐 }. Finally, the prediction score belonging to the 𝑐-th category is calculated as follows:EQUATIONwhere 𝜏 is a learnable parameter, Matching (•) denotes 𝜏-normalized cosine similarity.Visual region construction plays a key role in our text-region matching framework. A reliable visual region ensures that all pixels of a category are collected, resulting in accurate matching between the category-aware text representation and the region-level representation. Consequently, prompt tuning could be optimized to learn category-aware knowledge guided by accurate visual regions. The naïve way to construct a visual region 𝑹 is to use a binary mask to indicate the absence or presence of the category 𝑐 in the feature maps, i.e., 𝒓 𝑐 = {𝑟 (𝑖,𝑗 ) |𝑖 ∈ [1, ℎ], 𝑗 ∈ [1, 𝑤], 𝑟 (𝑖,𝑗 ) ∈ {0, 1}}. However, this is infeasible because we do not have access to pixellevel labels. To remedy this, we extend the naïve region from hard form to soft form, which includes not only spatial positions but also present probabilities corresponding to these positions. The 𝒓 𝑐 , which performs the easy-to-hard form, can be redefined as follows:EQUATIONwhere 𝒑 𝑐 and 𝒆 𝑐 consist of related positions and their present probability (energy), respectively. Specifically, we utilize CLIP text encoder with prompt tuning [55] or random initialization [3] to generate learnable category embeddings 𝑸 = {𝒒 1 , 𝒒 2 , . . . , 𝒒 𝑐 }. Next, we perform cross-attention using a decoder like a transformerdecoder [29] , with category embeddings 𝑸 as queries and visual representations 𝒇 as keys and values. This produces visual region 𝑹 and query-level representations𝑭 𝑞 = {𝒇 𝑞 1 , 𝒇 𝑞 2 , . . . , 𝒇 𝑞 𝑐 }.The above process is written as follows:EQUATION)where 𝑹 is directly generated by Cross_Att. However, the transformer decoder contains multiple linear layers, which could disrupt the alignment of visual and textual representations in the joint space of CLIP. To address this problem, we leverage visual regions 𝑹 as soft mask to directly guide image representations 𝒇 to filter out irrelevant visual information and obtain region-level representationsEQUATIONwhere 𝒆 𝑐 is the energies of category 𝑐 in the visual representation at spatial location.Processing high-quality visual regions plays a pivotal role in acquiring region-level representations. However, due to incomplete category annotation or missing pixel-level annotation, the visual regions learned by the transformer-decoder are often of poor quality (e.g., unable to locate objects, containing too much noise). This phenomenon could cause the two modalities between the text and visual to be unable to ensure good alignment. Thanks to CLIP's rich prior knowledge and generalization capabilities, we can transfer region-level representation 𝑭 𝑟 knowledge to query-level representation 𝑭 𝑞 to obtain higher-quality visual region 𝑹.To be specific, according to Eq.( 1), the query-level region representations 𝑭 𝑞 and region-level representations 𝑭 𝑟 are matched with the corresponding text representation 𝑮 to obtain prediction scores:EQUATIONSubsequently, we introduce knowledge distillation [20] to promote consistency between the above two types of representations, 𝑭 𝑟 and 𝑭 𝑞 . This procedure can be expressed as follows:EQUATION)where KL is Kullback-Leibler divergence [20] , 𝑝 𝑟 and 𝑝 𝑞 serve as the teacher prediction and student prediction, respectively. To illustrate pseudo-label estimation simply and clearly, we select one of the four color samples as an example.Intuitively, in a large-scale visual language pre-trained model (VLP), image-level, pixel-level, and text-level representations belonging to the same category are similar in terms of encoded features. Inspired by this, we hypothesize that region-level representations from the same category are highly similar to the corresponding visual or text prototype, and region-level representations from different images of the same category are also highly similar in the embedding space. To this end, we design a multimodal category prototype that includes visual and textual modalities.Visual Prototype. To make the visual prototype more discriminative, we derive it from multiple pixel-level representations with high-energy rather than region-level representations. Specifically, we design a visual memory bank of size 𝑐 × ℓ × 𝑑 to store the pixel-level visual representations as visual prototypes. The visual repository consists of 𝑐 queues with a queue length of ℓ. Given visual regions of an image, if the label of category 𝑐 is 1, we extract the top-k high response values on the 𝑐-th visual region as indexes and extract the representation of the corresponding position as visual prototypes to store them in the memory bank:EQUATIONwhere 𝑽 𝑐 denotes the visual prototype of the 𝑐-th category. Then, 𝑽 𝑐 is input to the 𝑐-th queue of the memory bank, using a first-infirst-out (FIFO) rule for the columns.Text Prototype. In contrast to [22] , we opt not to employ encoded representations of GPT-3 generated text descriptions as text prototypes [2] . This approach necessitates additional manual design efforts. Therefore, we adopt the text representation trained in the first stage as a text category prototype 𝑻 = {𝒕 1 , 𝒕 2 , . . . , 𝒕 𝑐 } to simplify and improve acquisition efficiency.Pseudo-Label Estimation. Given a minibatch of input images 𝑿 = {(𝒙 𝑖 , 𝒚 𝑖 )} 𝑛 𝑖=1 , we can obtain a set of region-level representations 𝑭 𝑟 = {𝒇 𝑟 𝑖,𝑗 | 𝑖 ∈ [1, 𝑛], 𝑗 ∈ [1, 𝑐]} using category-aware region learning, as described in the Sec. 3.2. The cosine similarity between the region-level representation 𝒇 𝒓 and the visual prototype 𝒗 is formulated as follows:EQUATION)As shown in Figure 3 , we calculate the average cosine similarity between all region-level representations 𝒇 𝑐 for the 𝑐-th category and the corresponding all visual prototypes 𝒗 𝑐 :EQUATIONwhere 𝑠 𝑝 𝑐 signifies the average similarity associated with positive labels, 𝑠 𝑛 𝑐 denotes the average similarity linked to negative labels, D 𝑝 𝑐 and D 𝑛 𝑐 represent the positive label set and the negative label set, respectively. To estimate the label of a missing category 𝑐 in an image 𝑚, we calculate the average similarity between the visual prototype 𝒗 𝑐 and the region representations 𝒇 𝑚,𝑐 , and then compare it to two thresholds 𝜃 𝑐 positive and 𝜃 𝑐 negative :EQUATIONwhere D 𝑢 𝑐 denotes the unknown label set. Moreover, the thresholds for different categories are dynamically and adaptively updated using an exponential moving average (EMA):EQUATIONwhere 𝜂 is the moving average parameter. On the other hand, we adopt a similar strategy to estimate unknown labels ỹ𝑛 𝑐;𝑡 based on text prototypes 𝑻 . Finally, the pseudo-labels estimated from visual prototypes and textual prototypes are combined:EQUATION)To enhance the consistency between text and visual modalities and establish intra-and inter-class relationships, we design multimodal contrastive learning to bridge the semantic gap between modalities, enhance intra-class similarity, and diminish inter-class similarity. Specifically, we use the text representation of the 𝑐-th category as the anchor, and we select the region-level representation belonging to the same category from the current batch and memory bank as the positive sample 𝑷 while collecting the region-level representations from other categories as the negative samples 𝑵 . We adopt the InfoNCE loss [30] as the contrastive loss:EQUATIONwhere 𝜏 is a temperature term that control the sharpness of the output. In addition, to strengthen semantic relationships (within and across categories) and minimize semantic gaps, we implement an additional multilayer perceptron (MLP) to enhance representation.Learning. During the training phase, we apply classification losses to both the text-query matching prediction score and textregion matching prediction score, respectively:L 𝑐𝑙𝑠 = L 𝑞 𝑐𝑙𝑠 + 𝛼 L 𝑟 𝑐𝑙𝑠 . Following prior work [36] , we utilize Asymmetric Loss (ASL) [34] as the classification loss optimization function:EQUATIONwhere 𝑝 𝑚 = max(𝑝 -𝑐, 0) represents the shifted probability for very easy negative samples. 𝛾+, 𝛾-, and 𝑐 values are set to 1, 2, and 0.05, respectively, by default. By combining classification loss with multimodal contrastive learning and knowledge distillation loss, we obtain the total loss function as follows:EQUATIONwhere 𝛽 and 𝛾 are hyperparameters.Inference. For inference, we utilize the text-region matching score as the final prediction score, obtained after eliminating the multi-modal category prototype and loss function from our proposed model. It is worth noting that through Eq.( 1), the prediction score obtained by our method is better than that obtained by pixellevel matching and image-level matching. Due to the limited space, we provide a detailed analysis of this conclusion in the Appendix. 
experiment 4.1 experimental setup: 
 Datasets & Evaluation. To evaluate the effectiveness of our proposed method under the partial labels setting on the MLR-ML task, we conduct experiments on three benchmarks, MS-COCO [26] , VOC 2007 [14] and Visual Genome [24] , following the approach of SARB [32] . As the most widely used multi-label image recognition benchmark, MS-COCO contains 80 common categories, 82,081 images for training, and 40,137 images for testing. The VOC 2007 dataset consists of 9,963 images, split into 5,011 training images and 4,952 test images. The images are labeled with 20 object classes. Visual Genome (VG) is a large-scale image dataset with over 100,000 images, but most categories have few samples. Therefore, we select the 200 most frequent categories as a subset, also called VG-200, following the settings of SARB [32] . These datasets are fully annotated, so we randomly sample labels at a ratio of 10% to 90%, including both positive and negative labels. Furthermore, we perform validation to evaluate the effectiveness of the proposed method under the single positive label setting across four datasets, which include MS-COCO, NUS-WIDE [9] , VOC 2012 [14] , and CUB-200-2011 [37] . NUS-WIDE is a substantial web-based dataset frequently employed in multi-label image learning tasks, featuring a total of 81 categories. VOC 2012 is an extension of VOC 2007 with more images. The CUB-200-2011, a widely recognized benchmark dataset in fine-grained image recognition with 200 categories and 312 attributes, is also commonly used for multi-label image recognition tasks. Following Table 1 : Compare our method with SOTA methods on the MS-COCO, VOC2007, and VG-200 benchmark datasets with partial labels. ★ indicates results we reproduced. Bolded means best performance, and all metrics are in %. All models adopt ResNet101 as the backbone. The "Vanilla" indicates weights pre-trained on ImageNet, while the "CLIP" utilizes weights derived from CLIP.Venue Backbone 10% 20% 30% 40% 50% 60% 70% 80% 90% Avg. MS-COCO Curriculum labeling [13] CVPR ' prior researchs [12, 16, 32] , we employ mean Average Precision (mAP) as the primary evaluation metric and provide an average mAP score for a comprehensive assessment. Implementation details. To facilitate a fair comparison, we employ ResNet [19] as the visual encoder, and Transformer as the text encoder, both initialized with parameters from CLIP pretraining [33] , also known as CLIP ResNet. The weights for vanilla ResNet are obtained from PyTorch [31] . The class-specific text prompts adopt the dual semantic strategy, initialized with Gaussian noise sampled from N (0, 0.02), and the length of the prompts is 16. Note that both the visual encoder and text encoder are frozen during training phase. The transformer decoder has only one attention head. We use AdamW as the optimizer and OneCycleLR as the lr_scheduler, with a maximum learning rate of 5e-4 and a batch size of 128. The input resolution of all images is 448 × 448, and the data augmentation settings are consistent with TaI-DPT [16] . We train for 40 epochs on all benchmark datasets. The default values for the hyperparameters are 𝛼 = 1, 𝛽 = 0.05, and 𝛾 = 0.01.MLR-ML on partial labels setting. To evaluate the effectiveness of our proposed method for the multi-label image recognition with partial labels task, we compare it to three types of methods: conventional (e.g., Curriculum labeling [13] , Patial-BCE [13] ), graph-based (e.g., ML-GCN [8] , SSGRL [7] , SST [6] , SARB [32] ), and prompt tuning (e.g., DualCoOp [36] , TaI-DPT [16] , SCPNet [12] , HSPNet [38] , TaI-Adapter [56] ). Although prompt tuning methods adopt CLIP ResNet101 as their visual encoder, which is better than other methods, previous work, e.g., TaI-DPT, has confirmed that most prompt tuning methods are superior to the other two types of methods. As shown in Table 1 , our proposed method outperforms existing methods on all benchmark datasets by a large margin. The proposed method demonstrates superior performance compared to HSPNet, SCPNet, and TaI-DPT on the MS-COCO dataset, surpassing them by 2.0%, 2.2%, and 1.3%, respectively. Although it outperforms TaI-DPT by 0.4% on VOC 2007, a combined approach incorporating both methods results in a notable 0.6% enhancement. Remarkably, TaI-DPT or TaI-Adapter combined with DualCoOp surpasses most methods on VOC 2007, especially in settings with a low proportion of known labels. This difference arises from the substantial disparity in both the volume and completeness of training data between text and image. Text data typically comprises a larger dataset with more comprehensive annotations compared to image data. Furthermore, when evaluated on the more demanding VG-200 dataset, the proposed method outperforms the suboptimal method by a notable margin of 3.4%. MLR-ML on single positive labels setting. To fairly evaluate the effectiveness of our proposed method for the multi-label image recognition with single positive labels task, we adopt the experimental settings of LargeLoss [23] and SPLC [53] . Table 2 clearly demonstrates that our proposed method surpasses existing methods across all four benchmark datasets. Remarkably, even when employing a frozen CLIP ResNet50 as the visual encoder, our method outperforms HSPNet and SCPNet, exhibiting substantial improvements, with 1.7% and 2.4% enhancement on the VOC 2012 dataset and impressive 3.8% and 3.2% increase on the MS-COCO dataset, particularly under the LargeLoss setup. Overall, the performance under the SPLC setup is 1.9% superior to the LargeLoss setup. However, compared to SCPNet, our method achieves a 2.0% improvement under the LargeLoss setting and a 1.8% improvement under the SPLC setup.In this section, to better understand how our method improves multi-label image recognition with missing labels, we conduct a series of experiments on the VOC 2007 and MS-COCO datasets.The effect of key components. To investigate the impact of different components on the model's performance and validate its effectiveness, we conduct an ablation study, systematically removing or modifying different components and evaluating the resulting model's performance. In this work, we employ the modified Du-alCoOp as a baseline, retaining the dual semantic prompt, text encoder, and visual encoder upon integrating our proposed component and omitting the attention module. As shown in Table 3 , our baseline method outperforms the SATO method, and integrating our proposed components with the baseline method steadily improves performance compared to the baseline alone. This demonstrates the effectiveness and significance of our proposed components in addressing missing label tasks through prompt tuning.The effect of visual regions. We further evaluate the impact of high-quality visual regions on text prompt tuning. To this end, we remove multimodal category prototypes and multimodal contrastive learning, train the model under complete annotation, and load the trained category-aware region learning module into a new model. Finally, we train the new model with partial label data. As presented in Table 4 , we report mAP for 10%-50% annotation proportions, with the average mAP shown in the last column. Leveraging high-quality visual regions, we achieve an average mAP improvement of 1.0% and 0.8% on VOC 2007 and MS-COCO datasets, respectively. The most notable enhancements are observed at 10% and 20% annotation rates, with mAP gains of 2.8% and 1.0%, respectively, for VOC 2007, and 2.3% and 0.9%, respectively, for MS-COCO. These results demonstrate the effectiveness of high-quality visual regions in optimizing text prompt tuning.Number of parameters and computational overhead. Described in Table 5 , evaluated on MS-COCO, these methods have Table 4 : Ablation study examines the influence of highquality visual regions on text prompt tuning learning in the MS-COCO and VOC 2007 datasets. The "complete" refers to the category-aware region learning module, trained using complete annotation data. All metrics are in %.Method 10% 20% 30% 40% 50% Avg. comparable total parameters and FLOPS, with little difference. TRM-ML has slightly more learnable parameters than DualCoOp, but significantly less than SCPNet. We achieve a higher mAP than both. In general, our method is superior to the other two methods.Text prototype and visual prototype valid independently. We conduct experiments on the VOC 2007. Notably, multimodal contrastive learning has been removed. In Table 6 , text prototypes and visual prototypes significantly improve model performance. In isolation, text prototypes achieve a 1.05% improvement, while visual prototypes achieve a 0.91% improvement. Combining both prototypes yields an even greater improvement of 1.52%.To demonstrate the effectiveness of category-aware region learning in constructing visual regions associated with specific categories, we present a comparison between the visual regions generated by the presented method and the class activation maps corresponding to DualCoOp, along with the corresponding prediction scores, as depicted in Figure 4 . From the figure, the four common scene images in MS-COCO are shown: outdoor, food, sports, and transportation. 1) Compared to DualCoOp, the proposed method excels in extracting visual information from category-specific regions while mitigating the impact of noise, as demonstrated by the examples of "person" in Figure 4 (a) and "motorcycle" in Figure 4 (d). 2) Our proposed method exhibits improved accuracy in perceiving certain objects, particularly smaller objects within complex scenes, e.g., "tie" in Figure 4 (a), "knife" in Figure 4 (b) and "person" in Figure 4 (c). Please observe that within the input image depicted in Figure 4 (d), both a "motorcycle" and a "person" are delineated within the red box and our method can locate them. In cases where certain categories are present in an image, our proposed method yields higher prediction scores than DualCoOp. In summary, category-aware region learning allows for a more precise emphasis on relevant visual regions, leading to enhanced prediction accuracy. 
conclusion: 
 In this work, to address the text-visual optimization challenges encountered by visual and language pre-trained models when dealing with multi-label image recognition with missing labels (MLR-ML) tasks, we introduce a novel framework called TRM-ML. To this end, we propose a category-aware region learning module that transforms the one-to-many (pixel-level) or one-to-agnostic (imagelevel) matching problem into a one-to-one (region-level) matching problem, enabling more effective multi-label image recognition. Furthermore, we combine multimodal contrastive learning with the category-aware region learning module to reduce the semantic gaps between textual and visual representations. When these three components are integrated, our method achieves new state-of-the-art performance on diverse multi-label benchmark datasets. 
 
 Appendix:  


Decision: Initially, the paper received three borderline Accept scores and one borderline Reject score. The reviewers liked the proposed approach to multi-label image recognition with missing labels by exploring category-aware regions' information to narrow the semantic gap between text and vision. However, they were concerned about the lack of multi-run experiments, the rationale of methodology designs, and the lack of necessary analysis for model complexity and hyper-parameters. These issues made it difficult to evaluate the proposed method. The rebuttal addressed the reviewers’ points by offering additional experimental results and explanations.

After rebuttal, Reviewer 7oeJ and WaPw kept the borderline Accept scores, and Reviewer WRAp changed the rating from borderline Reject to borderline Accept.  They recommended acceptance, acknowledging that the rebuttal successfully addressed their concerns.  

The AC has read the paper, reviews, and rebuttal. Reviewer gecv rated borderline Accept initially and didn't confirm with final recommendation. However, the AC believes the authors' response is reasonable, and Reviewer gecv's concerns will not overturn the overall rating. The authors are encouraged to improve the final paper version by following reviewer recommendations, in particular including a clearer ablation analysis and more in-depth discussion, as well as improving the clarity of the writing and presentation.

Reviews: review: paper_summary: paper_summary: This paper proposes a text-region matching for multi-label image recognition. The experiments prove the effectiveness of the proposed  method.
strength_weakness: strength_weakness: The paper presents a one-to-one (category-to-region) matching between textual and visual representations.

The proposed TRM-ML reaches the SOTA performance.
limitations: limitations: The labels are randomly sampled for training, according to sec4.1.The paper should include the standard deviation in the performance of TRM-ML when using different seeds.

Why the baseline of TRM-ML surpasses some SOTA methods?

The paper should include the analysis on the influence of 𝜂 (moving average parameter).
paper_summary: paper_summary: The paper introduces Text-Region Matching for Multi-Label prompt tuning (TRM-ML), aiming to enhance cross-modal matching in multi-label image recognition with missing labels. It emphasizes category-aware regions, utilizes multimodal contrastive learning, and introduces a multimodal category prototype to estimate unknown labels. Experimental results demonstrate the superiority of TRM-ML over existing methods.
strength_weakness: strength_weakness: 1）This paper is easy to understand, with a great introduction that thoroughly analyzes the shortcomings of existing work.
2）The related experiments are conducted comprehensively.
3）Proposed a relatively novel approach is proposed.
limitations: limitations: 1）The authors pointed out in the introduction that text-to-pixel level matching may introduce irrelevant noise. However, in Section 3.4, they chose to use pixel-level representations to construct visual prototypes, which seems contradictory to their previous statement. Why not use existing region representations to build visual prototypes, which could also save computational time?
2）If high-energy pixels are used for filtering to address pixel noise issues, is it worthwhile to explore whether integrating high-energy pixels directly into DualCoop could achieve similar effects without adding extra computational burden?
3）In generating region representations, the cross-attention mechanism of the transformer-decoder is employed. However, considering the actual spatial information in the images might yield better results.such as SCLIP[1]. 
[1] SCLIP:Rethinking Self-Attention for Dense Vision-Language Inference.
4）The framework diagram contains too many lines, which can lead to confusion. For example, while generating visual prototypes, the text mentions using pixel-level representations, but the diagram shows lines from Fr​ pointing to visual prototypes, which seems incorrect.
Minor Issues:
1)Table 6 should indicate that the last column represents the mean retention rate of labels varying from 10% to 50%.
2)There are minor errors, such as "SATO" on line 842 of page 8, which should be "SOTA".
paper_summary: paper_summary: This paper introduces a novel method, Text-Region Matching (TRM-ML), for optimizing multi-label prompt tuning to improve multi-label image recognition with incomplete annotations. TRM-ML narrows down the semantic gap between text and vision by exploring the information of category-aware regions rather than the entire image or pixels. It incorporates multimodal contrastive learning and a multimodal category prototype to establish intra-class and inter-class relationships and estimate unknown labels, aiding pseudo-label generation.
strength_weakness: strength_weakness: 1.The paper presents some focused and customized designs for the task of multi-label image recognition with incomplete annotations.

2.It conducts relatively comprehensive experiments, providing a robust validation of its proposed methods. The proposed methods clearly demonstrate significant improvement over existing strategies, showing the effectiveness of the proposed solutions.
limitations: limitations: 1.While the authors claim to have four main innovative points, it would be helpful for them to condense and emphasize their core contribution, particularly focusing on the methodology outlined in Figure 2. At present, the work feels like a combination of several smaller methods, making it difficult to understand the systematized value of the proposed work.

2.The study could benefit from more comprehensive experiments. The authors could analyze the performance on larger category datasets like OpenImages to extend the applicability of their method beyond the current scope. These additional investigations would increase the paper’s relevance to real-world applications and widen its usability.
paper_summary: paper_summary: This paper introduces a novel framework called TRM-ML that addresses the challenge of missing labels in multi-label image recognition tasks. By integrating a Category-Aware Region Learning Module, Multimodal Contrastive Learning, and Multimodal Category Prototypes, the framework effectively bridges the semantic gap between textual and visual representations, achieving state-of-the-art performance on several multi-label benchmark datasets. The authors demonstrate the effectiveness and superiority of their approach through a series of ablation studies and visualization analyses, particularly in dealing with incomplete annotations.
strength_weakness: strength_weakness: 1.The methodology is technically sound, with a clear explanation of how the text-region matching strategy works in conjunction with multimodal contrastive learning and the multimodal category prototype. 
2.The authors have conducted extensive experiments on multiple benchmark datasets, which demonstrates the effectiveness of their proposed framework.
3.The manuscript is well-written, and the figures and illustrations are clear and aid in understanding the proposed method.
limitations: limitations: 1.The proposed TRM-ML framework incorporates several new components, including a category-aware region learning module and a multimodal category prototype. However, the paper does not discuss the computational complexity of these additions, which could be significant, especially for large-scale datasets. An analysis of the computational overhead would be beneficial.

2.Including qualitative case studies or examples where the model succeeds or fails could provide deeper insights into the model's capabilities and limitations.

3.A discussion on the sensitivity to hyperparameter choices would be beneficial, as this can greatly affect the performance of models.