Prototypical Prompting for Text-to-image Person Re-identification

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 04:35:04

Abstract: In this paper, we study the problem of Text-to-Image Person Re-identification (TIReID), which aims to find images of the same identity described by a text sentence from a pool of candidate images. Benefiting from Vision-Language Pre-training, such as CLIP (Contrastive Language-Image Pretraining), the TIReID techniques have achieved remarkable progress recently. However, most existing methods only focus on instance-level matching and ignore identity-level matching, which involves associating multiple images and texts belonging to the same person. In this paper, we propose a novel prototypical prompting framework (Propot) designed to simultaneously model instance-level and identity-level matching for TIReID. Our Propot transforms the identity-level matching problem into a prototype learning problem, aiming to learn identity-enriched prototypes. Specifically, Propot works by ‘initialize, adapt, enrich, then aggregate’. We first use CLIP to generate high-quality initial prototypes. Then, we propose a domain-conditional prototypical prompting (DPP) module to adapt the prototypes to the TIReID task using task-related information. Further, we propose an instance-conditional prototypical prompting (IPP) module to update prototypes conditioned on intra-modal and inter-modal instances to ensure prototype diversity. Finally, we design an adaptive prototype aggregation module to aggregate these prototypes, generating final identity-enriched prototypes. With identity-enriched prototypes, we diffuse its rich identity information to instances through prototype-to-instance contrastive loss to facilitate identity-level matching. Extensive experiments conducted on three benchmarks demonstrate the superiority of Propot compared to existing TIReID methods.

Main paper: 
 related work 2.1 text-to-image person re-identification: 
 TIReID [30] has gained significant attention in recent years. Existing studies in TIReID can be broadly divided into the following categories: better model architectures and optimization losses, better alignment strategies, and richer prior information. Previous methods [3, 62, 64] have focused on designing network and optimization loss to learn globally aligned image and text features in a joint embedding space. These methods are simple and efficient, but ignore detailed information and fine-grained correspondences. To address these limitations, subsequent methods have refined matching strategies to mine fine-grained correspondence between modalities. Some methods [22, 30, 39, 47, 63, 66] have emerged to achieve fine-grained matching through interactions between local parts of images and texts. While effective, these approaches require significant computational resources. To mitigate computational overhead, other works [5, 8, 53] adopt local image parts as references to guide the generation of locally aligned text features, avoiding pairwise interactions. However, the effectiveness of these methods depends on the quality of explicitly acquired local parts. As an alternative, diverse aggregation schemes [27, 41, 44, 57] have been proposed to adaptively aggregate images and text into modality-shared local features, avoiding explicit local part acquisition.Recently, visual-language pre-training models (VLP) [40] have made significant progress. To leverage their rich multi-modal knowledge, recent advanced methods [2, 15, 20, 56] have proposed diverse strategies to tailor VLP for TIReID, resulting in notable performance enhancements. However, despite these advancements, existing methods often overlook the specific nature of TIReID as a challenge involving matching multi-view image-text pairs at the identity level, rather than merely at the instance level. Addressing this concern, LCR 2 S [55] devised a teacher-student network to reason about a comprehensive representation with multi-view information from a single image or text. This approach yielded notable performance gains, it had limitations in fully integrating multi-view information and exhibited lower training efficiency. In this study, we present an end-to-end framework to learn a prototype with comprehensive information for each identity using the training set, and transfer multi-view information to individual samples for identity-level matching. This approach enhances both efficiency and the integration of multi-view data for TIReID.Nowadays, the "pre-training and fine-tuning" paradigm stands as a foundational approach in the computer vision community, in which pre-training models that can provide rich prior knowledge for various downstream vision tasks have gained increasing attention. Previous prevailing practice is rooted in the supervised unimodal pre-training [9, 16] on ImageNet [6] . However, to improve representation capabilities and overcome annotation constraints, a new paradigm called language-supervised vision pre-training (vision-language pre-training, VLP) has emerged. Within VLP, investigating the interaction between vision and language has become a central research focus. Several works [4, 25, 26, 33, 49] have been proposed to model the interaction of vision and language based on some multi-modal reasoning tasks, such as masked language/region modeling, and image captioning. Recently, contrastive representation learning [19, 40, 59] has gained attention, which learns representations by contrasting positive pairs against negative pairs. The representative work, Contrastive Language-Image Pretraining (CLIP) [40] , has strong multi-modal semantic representation and zero-shot generalization capabilities, which is trained on 400 million image-text pairs. CLIP has shown promising adaptability for various downstream tasks like video-text retrieval [10, 34] , referring image segmentation [51] , and person re-identification [56] . Our work builds upon CLIP and utilizes its ample multi-modal knowledge to learn identity-enriched prototypes.Prompt learning [21, 43] , originating from natural language processing (NLP), is a method used to customize pre-training models for different tasks by providing instructions in the form of sentences, known as prompts. Early prompts were manually designed for specific tasks, but recent studies have introduced prompt learning, where task-specific prompts are automatically generated during fine-tuning. This approach addresses issues like instability and knowledge bias. Prompt learning has now been extended to computer vision. CoOp [65] pioneered the application of prompt learning to adapt large vision-language models in computer vision, while CoCoOp [46] built upon CoOp to introduce a conditional prompt learning framework, improving generalization. Chen et al. [23] proposed efficiently adapting the CLIP model to the video understanding task by optimizing a few continuous prompt vectors. CLIP-ReID [29] designed a two-stage framework to generate coarse descriptions of pedestrians, leveraging CLIP's capabilities for ReID. Inspired by these, in this work we propose using prompt learning to learn comprehensive prototype representations to model identity-level matching for TIReID. 
the propot framework: 
 Our Propot is a conceptually simple end-to-end trainable framework, and the overview is depicted in Figure 2 . We first extract features of images and texts through visual and textual encoders, followed by instance-level and identity-level matching.Previous studies [2, 20, 56] have underscored the effectiveness of CLIP [40] in tackling TIReID challenges. To harness the vast multimodal knowledge in CLIP, we leverage its image and text encoders to initialize Propot's image and text backbones. Concretely, for an image-text pair (𝐼,𝑇 ), we exploit CLIP pre-trained ViT model to extract visual representations for the image 𝐼 , resulting in the global visual feature 𝒗 ∈ R 𝑑 . For the text caption 𝑇 , the CLIP's textual encoder is utilized to generate the global textual feature 𝒕 ∈ R 𝑑 .Instance-level matching involves directly aligning 𝒗 with 𝒕. For identity-level matching, we transform it into an identity-enriched prototype learning problem. The aim is to learn a rich prototype containing all instance identity information for each identity in the training set. In Propot, the prototype learning method is crucial. While it is conceivable to learn a prototype for each identity from scratch, akin to previous method [29] , such an approach can encounter challenges in network convergence and may not ensure prototype quality. To address these concerns, we introduce a novel 'initialize, adapt, enrich, then aggregate' prototype learning scheme, detailed in the subsequent subsections.We start our approach by utilizing the CLIP model to extract features for each instance in the TIReID training set, leveraging its strong semantic information extraction and multi-modal alignment capabilities. We then cluster these instance features based on shared identity labels to produce initial prototypes for each modality. Specifically, given the training set {𝐼 𝑖 ,𝑇 𝑖 , 𝑌 𝑖 } 𝑁 𝑠 𝑖=1 , where 𝑌 𝑖 ∈ {𝐿 1 , 𝐿 2 , ..., 𝐿 𝑁 } represents the identity label of the image-text pair (𝐼 𝑖 ,𝑇 𝑖 ), 𝑁 𝑠 denotes the number of pairs, and 𝑁 denotes the number of identity label, we employ the pre-trained CLIP visual and textual encoders to obtain the visual and textual features {𝒗 𝑖 , 𝒕 𝑖 } 𝑁 𝑠 𝑖=1 of all image-text pairs. We then perform feature clusters on the image and text features. Taking identity 𝐿 𝑖 as an example, We generate initial prototypes 𝒑𝒕 𝑣 𝑖 and 𝒑𝒕 𝑡 𝑖 for identity 𝐿 𝑖 as follows: ... pt2 pt2 ptN ptN It includes instance-level matching and identity-enriched prototype learning. For instancelevel matching, each image and its annotated text are directly aligned through SDM loss (Baseline). For prototype learning, we first utilize pre-trained CLIP to generate initial prototypes (𝒑𝒕 𝑣 and 𝒑𝒕 𝑡 ). We then adapt the initial prototypes to TIReID through the DPP module, resulting in task-adapted prototypes (𝒑 𝑣 𝑎 and 𝒑 𝑡 𝑎 ). The IPP module updates the prototypes conditioned on a batch of intra-modal and inter-modal instances, generating intra-modal and inter-modal enriched prototypes (𝒑 𝑣 𝑒𝑛 , 𝒑 𝑡 𝑒𝑛 , 𝒑 𝑣 𝑒𝑜 and 𝒑 𝑡 𝑒𝑜 ). The multiple prototypes are aggregated using Adaptive Prototypical Aggregation (APA) to generate final prototypes (𝒑 𝑣 and 𝒑 𝑡 ). Their rich identity information is then diffused to each instance using prototype-to-instance contrastive loss (L 𝑝2𝑣 , L 𝑝2𝑡 ) to model identity-level matching. Moreover, we introduce the MLM module to enhance fine-grained matching. During testing, only visual and textual encoders are used for inference.EQUATION.. v 2 v 2 v B v B v 1 v 1 ... v 2 v B v 1 ... p 2 v p 2 v p N v p N v p 1 v p 1 v ... p 2 v p N v p 1 v Textual[Xi]1 [Xi]1[Xi]2 [Xi]2 [Xi]K [Xi]K ... pti pti pti [Xi]1[Xi]2 [Xi]K ...v 2 v 2 v B v B v 1 v 1 ... v 2 v B v 1 ... t 2 t 2 t B t B t 1 t 1 ... t 2 t B t 1 Q / K V / Self Attention Feed Forward Cross Attention 4 L L mlm L mlm v i v i t i t i Q K V L L id L id + pt1 pt1 ... ... p eo,1 v p eo,1 v p eo,1 v p eo,where 𝑁 𝑖 denotes the number of instances under the identity 𝐿 𝑖 . Therefore, for the entire training set, we can generate visual and textual initial prototype sets𝒑𝒕 𝑣 = [𝒑𝒕 𝑣 1 , 𝒑𝒕 𝑣 2 , ..., 𝒑𝒕 𝑣 𝑁 ] ∈ R 𝑁 ×𝑑 and 𝒑𝒕 𝑡 = [𝒑𝒕 𝑡 1 , 𝒑𝒕 𝑡 2 , ..., 𝒑𝒕 𝑡 𝑁 ] ∈ R 𝑁 ×𝑑 .Benefiting from CLIP's powerful capabilities, the initial prototype effectively captures some identity information to model identitylevel matching, leading to improved performance (as shown in Table 4 ), which confirms the validity of our idea. However, a notable challenge arises due to the domain gap between the pretraining data of CLIP and the TIReID dataset. Consequently, the identity information mined by CLIP's features falls short, significantly diminishing the impact of the initial prototype. To address this challenge, inspired by the Contextual Optimization (CoOp) framework [65] , which introduces learnable context vectors to adapt VLP to downstream tasks, we introduce a Domain-conditional Prototypical Prompting (DPP) module to adjust the initial prototype to the TIReID task. Specifically, we add a set of learnable contextual prompt vectors before each initial prototype. These vectors undergo training on the TIReID dataset concurrently with the network, gaining domain knowledge specific to the TIReID task. These vectors then transmit the domain knowledge to each initial prototype through the self-attention encoder (SAE), aiding in the adaptation of prototypes to the TIReID task. Formally, for each initial prototype 𝒑𝒕 𝑖 , we add a set of learnable contextual prompt vectors{[𝑿 𝑖 ] 1 , [𝑿 𝑖 ] 2 , ..., [𝑿 𝑖 ] 𝐾 } ∈ R 𝐾 ×𝑑 before it, where [𝑿 𝑖 ] is the visual contextual prompt vector 𝒄 𝑣,𝑖 for the visual prototype 𝒑𝒕 𝑣 𝑖 , and [𝑿 𝑖 ] is the textual contextual prompt vector 𝒄 𝑡,𝑖 for the textual prototype 𝒑𝒕 𝑡 𝑖 . Then, we feed{[𝑿 𝑖 ] 1 , [𝑿 𝑖 ] 2 , ..., [𝑿 𝑖 ] 𝐾 , 𝒑𝒕 𝑖 } ∈ R (𝐾+1)×𝑑 into SAE to pass the information to the initial prototype for updating it.EQUATIONwhere 𝒑 𝑎,𝑖 ∈ R 𝑑 represents the task-adaptive prototype. SAE is comprised of 𝑁 𝑎 blocks, with each block containing a multi-head self-attention layer and a feed-forward network layer. The above process allows us to generate modality-specific task-adaptive prototypes, 𝒑 𝑣 𝑎,𝑖 and 𝒑 𝑡 𝑎,𝑖 , for the input prototypes of different modalities.So far, we have generated a prototype adapted to the TIReID task. As expected, the resulting prototype brought significant performance improvements. However, as depicted in Figure 1 , instances of the same identity showcase notable diversity due to factors like view changes and camera parameters. The previously derived prototype fails to account for this diversity, resulting in the loss of some discriminative identity information. To solve this problem, inspired by Conditional Context Optimization (CoCoOp) [46] , we propose an Instance-conditional Prototypical Prompting (IPP) module. This module updates the prototype conditioned on each batch of the input instances, enabling the comprehensive mining of instancespecific information and ensuring diversity within the prototype. Specifically, for a batch of 𝐵 image-text pairs {𝐼 𝑖 ,𝑇 𝑖 } 𝐵 𝑖=1 , after feature encoding, we have 𝑽 𝐵 = [𝒗 1 , 𝒗 2 , ..., 𝒗 𝐵 ] ∈ R 𝐵×𝑑 and 𝑻 𝐵 = [𝒕 1 , 𝒕 2 , ..., 𝒕 𝐵 ] ∈ R 𝐵×𝑑 . Next, the instance features 𝑽 𝐵 /𝑻 𝐵 and the prototypes 𝒑𝒕 𝑣 /𝒑𝒕 𝑡 are processed in a cross-attention decoder (CAD). This decoder enables interaction between instance features and prototypes, extracting identity-relevant information from instances to enrich the prototypes. There are 𝑁 𝑒 blocks in CAD, each featuring a multi-head cross-attention layer and a feed-forward network layer. For the cross-attention decoder, the prototypes 𝒑𝒕 𝑣 /𝒑𝒕 𝑡 serve as query, while the instance features 𝑽 𝐵 /𝑻 𝐵 are designated as key and value, which is formulated as follows:EQUATIONwhere 𝒑 𝑣 𝑒𝑛,𝑖 ∈ R 𝑑 and 𝒑 𝑡 𝑒𝑛,𝑖 ∈ R 𝑑 represent the intra-modal instanceenriched visual and textual prototypes, respectively. This process enhances the prototypes using intra-modal instance features. Additionally, we extend this enrichment by incorporating inter-modal instance features, fostering the network to extract modality-shared identity information comprehensively and minimize the modal gap. We formulate them asEQUATIONwhere 𝒑 𝑣 𝑒𝑜,𝑖 ∈ R 𝑑 and 𝒑 𝑡 𝑒𝑜,𝑖 ∈ R 𝑑 represent the inter-modal instanceenriched visual and textual prototypes, respectively.For both images and texts, we generate three distinct modalityspecific prototypes: 𝒑 𝑚 𝑎,𝑖 ∈ R 𝑑 , 𝒑 𝑚 𝑒𝑛,𝑖 ∈ R 𝑑 , and 𝒑 𝑚 𝑒𝑜,𝑖 ∈ R 𝑑 , each containing different information for each identity 𝐿 𝑖 , where 𝑚 ∈ {𝑣, 𝑡 }.To seamlessly integrate all available information, we introduce an Adaptive Prototype Aggregation (APA) module. This module effectively aggregates diverse prototypes to form a comprehensive identity-enriched prototype. Since 𝒑𝒕 𝑚 𝑖 is directly derived by CLIP, ensuring high-quality prototypes due to its robust semantic understanding capabilities. We designate 𝒑𝒕 𝑚 𝑖 as the prototype baseline for aggregation. The aggregation weights are determined based on the correlation of other prototypes with 𝒑𝒕 𝑚 𝑖 . This approach enables the suppression of spurious identity prototype information in 𝒑 𝑚 𝑎,𝑖 , 𝒑 𝑚 𝑒𝑛,𝑖 , 𝒑 𝑚 𝑒𝑜,𝑖 , while amplifying the correct ones during aggregation. The calculation of prototype correlation, serving as the aggregation weights for the three prototypes, is as follows:𝒘 𝑚 𝑎,𝑖 = 𝒑𝒕 𝑚 𝑖 (𝒑 𝑚 𝑎,𝑖 ) 𝑇 , 𝒘 𝑚 𝑒𝑛,𝑖 = 𝒑𝒕 𝑚 𝑖 (𝒑 𝑚 𝑒𝑛,𝑖 ) 𝑇 , 𝒘 𝑚 𝑒𝑜,𝑖 = 𝒑𝒕 𝑚 𝑖 (𝒑 𝑚 𝑒𝑜,𝑖 ) 𝑇 .(5) Then, we utilize the softmax function to normalize the weights and obtain the final identity-enriched prototype asEQUATIONwhere 𝑘 ∈ {𝑎, 𝑒𝑛, 𝑒𝑜 }. Thus, for the entire training set, we can generate the final visual and textual prototype sets𝒑 𝑣 = [𝒑 𝑣 1 , 𝒑 𝑣 2 , ..., 𝒑 𝑣 𝑁 ] ∈ R 𝑁 ×𝑑 and 𝒑 𝑡 = [𝒑 𝑡 1 , 𝒑 𝑡 2 , ..., 𝒑 𝑡 𝑁 ] ∈ R 𝑁 ×𝑑. Subsequently, we propagate the rich identity information encapsulated in the prototypes to each instance through prototype-to-instance contrastive loss to model identity-level matching for TIReID.The goal of Propot is to model both instance-level and identity-level matching for TIReID. To this end, we optimize Propot through crossmodal matching loss, cross-entropy loss, prototype-to-instance contrastive loss, and mask language modeling loss.Given a batch of 𝐵 image-text pairs {𝐼 𝑖 ,𝑇 𝑖 } 𝐵 𝑖=1 , we generate the global visual and textual features as𝑽 𝐵 = [𝒗 1 , 𝒗 2 , ..., 𝒗 𝐵 ] ∈ R 𝐵×𝑑 and 𝑻 𝐵 = [𝒕 1 , 𝒕 2 , ..., 𝒕 𝐵 ] ∈ R 𝐵×𝑑 .To align each image 𝐼 𝑖 and its annotated text 𝑇 𝑖 , we utilize the similarity distribution matching (SDM) [20] as the cross-modal matching loss to model instance-level matching between them.EQUATIONEQUATIONEQUATIONwhere 𝑠𝑖𝑚(•) denotes the cosine similarity function, 𝜏 denotes the temperature factor to control the distribution peaks, and 𝜖 is a small number to avoid numerical problems. 𝑞 𝑖,𝑗 denotes the true matching probability. 𝛼 indicates the margin. L 𝑡 2𝑖 can be obtained by exchanging 𝒗 and 𝒕 in Eqs. 8 and 9. Moreover, to ensure the discriminability of features 𝒗 and 𝒕, we calculate cross-entropy loss L 𝑖𝑑 on them to classify them into corresponding identity labels. Through the prototype learning process described above, we generate two identity-rich prototypes 𝒑 𝑣 𝑖 and 𝒑 𝑡 𝑖 for each identity 𝐿 𝑖 . To effectively diffuse the rich identity information encapsulated in the prototypes to instances of the same identity, we employ a prototype-to-instance contrastive loss, denoted as L 𝑝2𝑖 . This loss operates in tandem with the cross-modal matching loss, collectively contributing to the modeling of identity-level matching.EQUATIONEQUATIONEQUATIONwhere 𝑃 (𝐿 𝑖 ) represents a set of instance indices of identity 𝐿 𝑖 , and |𝑃 (𝐿 𝑖 )| denotes the cardinality of 𝑃 (𝐿 𝑖 ). To further improve performance, we follow [20] to introduce a mask language modeling task L 𝑚𝑙𝑚 to model fine-grained matching between modalities.Propot is a single-stage and end-to-end trainable framework, and the overall objective function L for training is as follows:EQUATIONwhere 𝜆 1 and 𝜆 2 balance the contribution of different loss terms.The prototype learning process only exists during training. During inference, we only use Propot's visual and textual encoders to extract the global features of the test samples. The retrieval results are obtained by calculating the cosine similarity. 
experiments 4.1 experiment settings: 
 Datasets and Metrics: The evaluations are conducted on three datasets for TIReID. CUHK-PEDES [30] Implementation Details: For input images, we uniformly resize them to 384×128 and augment them with random horizontal flipping, random crop with padding, and random erasing. For input texts, the length of the token sequence is unified to 77 and augmented by randomly masking out some tokens with a 15% probability [7] . Propot's visual encoder is initialized with the CLIP-ViT-B/16 version of CLIP, where the dimension 𝑑 of global visual and textual features is 512. The SAE and CAD consist of 𝑁 𝑎 = 1 and 𝑁 𝑒 = 3 blocks, respectively, each with 8 heads. The length 𝐾 of the learnable prompt vector in the DPP module is set to 4. The temperature factor 𝜏 is 0.02. The loss balance factors are set to 𝜆 1 = 0.2 and 𝜆 2 = 1.0. Model training utilizes the Adam optimizer with a weight decay factor of 4e-5. Initial learning rates are 1e-5 for the visual/textual encoder and 1e-4 for other network modules. We employ a cosine learning rate decay strategy, stopping training at 60 epochs. The learning rate linearly decays by a factor of 0.1 within the first 10% of the training epochs for warmup. All experiments are implemented in PyTorch library, and models are trained with a batch size of 64 on a single RTX3090 24GB GPU.In this section, we compare our Propot with current state-of-the-art (SOTA) approaches on all three TIReID benchmarks. The methods for comparison are categorized into two sections: methods (w/o CLIP) based on single-modal pre-training models (ResNet [16] , ViT [9] , BERT [7] ) and methods (w/ CLIP) based on multi-modal pretraining CLIP [40] models. Propot falls under the second section.The performance comparison with SOTA methods on CUHK-PEDES is summarized in Table 1 . The proposed Propot framework demonstrates competitive performance at all metrics and outperforms all compared methods except [31] , achieving remarkable R@1, R@5, and R@10 accuracies of 74.89%, 89.90% and 94.17%, respectively. While our R@1 accuracy is slightly lower (-0.13%) Table 1 : Performance comparison with state-of-the-art methods on CUHK-PEDES. R@1, R@5, and R@10 are listed.Pre Ref R@1 R@5 R@10 SRCF [45] w/o CLIP ECCV ' compared to the optimal method DCEL [31] , it is essential to note that DECL introduces both mask language modeling and globallocal semantic alignment to mine fine-grained matching, resulting in higher computational cost. In contrast, Propot employs only mask language modeling for fine-grained matching. Additionally, as observed in Table 4 #7, even without a local matching module, Propot achieves a noteworthy 74.37% R@1 accuracy, surpassing most compared methods. Table 2 reports the comparative results on ICFG-PEDES. Our Propot establishes a new SOTA performance on this dataset, with R@1, R@5, and R@10 accuracy scores of 65.12%, 81.57%, and 86.97%, respectively. Notably, Propot surpasses the current SOTA solution TBPS-CLIP [2] by 1.23% in R@5 and 1.50% in R@10. The comparative analysis with SOTA methods on RSTPReid is summarized in Table 3 . Propot demonstrates commendable performance, achieving competitive results over recent SOTA methods, specifically attaining 61.83%, 83.45%, and 89.70% on R@1, R@5, and R@10. Although our method achieves slightly lower performance (-0.08%) than the optimal method TBPS-CLIP [2] , which incorporates CLIP into the TIReID task using various data augmentation and training tricks. In contrast, our approach uses only basic data augmentation without additional tricks. In summary, propot achieves superior performance on all three benchmarks. This is attributed to the fact that our prototypical prompting can simultaneously model instance-level and identity-level matching.We conduct ablation experiments for Propot on CUHK-PEDES using the default settings above. Baseline solely includes visual and textual encoders initialized by CLIP, which is trained using SDM and crossentropy loss, with the training settings aligned with those of Propot. Contributions of Proposed Components: In Table 4 , we assess the contribution of each module of Propot, including the initial prototype (IniPt) generated by CLIP, the DPP module, the IPP module, and the MLM module. Baseline achieves a notable R@1 accuracy of 72.73% due to the rich multi-modal knowledge of CLIP. Introducing our prototype learning process based on Baseline, aimed at modeling identity-level matching, yields several key observations. Firstly, using only the initial prototype generated by CLIP to supervise identity-level matching leads to distinct performance improvements (+0.35% R@1 improvement over Baseline), affirming the feasibility of our approach and the importance of identity-level matching. Secondly, incorporating the DPP module to update the initial prototype results in a 1.38% R@1 accuracy improvement over Baseline, demonstrating the effective adaptation of the initial prototype to the TIReID task. Thirdly, when the IPP module is employed to update the prototype, whether conditioned on intra-modal or inter-modal instances, substantial performance enhancements are observed (+1.1% or +0.92% R@1 improvement over Baseline). The performance is further elevated when both IPP modules are utilized concurrently, underscoring the IPP module's capacity to enrich the prototype with instance information. Fourthly, the collaborative use of DPP and IPP modules further enhances the R@1 accuracy to 74.37%, surpassing most state-of-the-art methods in Table 1 . This is attributed to our Propot's ability to model instance-level and identity-level matching simultaneously. Finally, incorporating the local matching module gets the best performance for Propot.Impact of contextual vector length 𝐾 in DPP: To facilitate the initial prototype adapt to the TIReID task, we introduce a set of learnable contextual prompt vectors for each prototype in the DPP module. The length 𝐾 of these vectors is a crucial parameter affecting the prototype's adaptation. To explore the impact of different vector lengths, we vary 𝐾 from 1 to 6 and report the results in Figure 3 (a) . The observed trend indicates that larger values of Table 3 : Performance comparison with state-of-the-art methods on RSTPReid. R@1, R@5, and R@10 are listed.Pre Ref R@1 R@5 R@10 LBUL [52] w 𝐾 result in better performance, as they provide more contextual parameters to capture TIReID task information. Notably, when 𝐾 equals 4, the performance reaches a peak of 74.89%. However, excessively large 𝐾 values may introduce redundant information, leading to overfitting and increased computational costs. Hence, we set 𝐾 to 4 to strike a balance between performance and efficiency.Influence of 𝑁 𝑎 , 𝑁 𝑒 : The parameters 𝑁 𝑎 and 𝑁 𝑒 determine the number of blocks in SAE and CAD, respectively. Their impact on performance is shown in Figure 3 (b) and (c). Regarding 𝑁 𝑎 , we observe a notable performance drop when its value exceeds 3. This suggests that too many SAE parameters might impede the learning of contextual prompt vectors. Therefore, we set 𝑁 𝑎 to 1 in our experiments, which yields the optimal result. Conversely, our model shows less sensitivity to 𝑁 𝑒 , with its curve displaying a relatively stable trend. However, excessively large 𝑁 𝑒 values would introduce unnecessary parameters, leading to increased computational costs. Hence, we set 𝑁 𝑒 to 3 to achieve superior performance while maintaining efficiency. Impact of weight 𝜆 1 in the objective function: The parameter 𝜆 1 determines the strength of identity-level matching. To assess its effect, we conduct experiments by varying 𝜆 1 from 0.01 to 10, as depicted in Figure 3 (d) . Results indicate that increasing 𝜆 1 leads to gradual performance improvement, peaking at 0.2. However, further increase in 𝜆 1 results in performance decline, collapsing when 𝜆 1 reaches 10. This pattern occurs because too small 𝜆 1 fails to effectively spread identity information to instances, making identity-level matching ineffective. Conversely, excessively large 𝜆 1 disrupts instance-level matching by over-diffusing identity information. Therefore, we set 𝜆 1 to 0.2 to balance instance-level and identity-level matching.Different Prototype Aggregation Schemes: We introduced the Adaptive Prototype Aggregation (APA) module to adaptively combine multiple prototypes generated by different modules. To demonstrate its effectiveness, we compared APA with several common aggregation schemes: (1) summing prototypes, (2) averaging prototypes, (3) using a multi-layer perceptron (MLP) to assign weights to prototypes, and (4) learning weights for prototypes simultaneously with the network. Table 5 summarizes the comparison results. From the analysis, we draw the following conclusions: Aggregation methods with learnable weights perform worse than those without. This is because adding parameters makes optimization more difficult and uncertain for the network. Simple summation or averaging of prototypes yields superior performance, highlighting the effectiveness of our approach. The strength of our aggregation method lies in its use of the CLIP-generated initial prototype as a baseline for combining other prototypes adaptively. The quality of the initial prototype is guaranteed by CLIP's strong semantic understanding capabilities.Qualitative Results: To showcase the effectiveness of Propot, we present retrieval results in Figure 4 . For each query text, Figure 4 displays the top-10 gallery images retrieved by both the baseline and Propot. Baseline solely focuses on instance-level matching, whereas Propot incorporates prototype prompting to include identity-level Balding male with gray hair, wearing glasses, dark colored shirt over a colored shirt, dark pants and dark colored shoes.A person with dark hair, wearing a blue short sleeve shirt, pink shorts and a pair of open heel shoes.The lady wears a black and white shirt black and white shorts with beige wedge she carries two shoulder bags black and brown.The woman is wearing a white cardigan over a long floaty dress and black shoes. She is carrying a black carryon bag.Top-10 Retrieval Results matching. The examples show that Propot excels where the baseline struggles, ensuring images with the same identity as the given query text rank high. This highlights the importance of identity-level matching, with Propot outperforming by modeling both instancelevel and identity-level matching simultaneously. 
conclusion: 
 In this study, to model identity-level matching for TIReID, we present Propot, a conceptually simple framework for identity-enriched prototype learning. The framework follows the 'initialize, adapt, enrich, then aggregate' pipeline. Initially, we generate robust initial prototypes using CLIP. Then, we employ the Domain-conditional Prototypical Prompting (DPP) module to prompt and adapt the initial prototypes to the TIReID task. To ensure prototype diversity, the Instance-conditional Prototypical Prompting (DPP) is devised, enriching the prototypes with both intra-modal and inter-modal instances. Finally, we use an adaptive prototype aggregation module to effectively combine multiple prototypes and diffuse their rich identity information to instances, thereby enabling identity-level matching. Through extensive experiments conducted on three popular benchmarks, we demonstrate the superiority and effectiveness of the proposed Propot framework.  
: 
 A girl in her twenties has long black hair. She is wearing an offwhite quilted jacket with white fur on its hood. She is also wearing black sweat pants and black ankle boots with heels.A young woman with straight long black hair is wearing a beige scarf around her neck. She is wearing a white overcoat and brown capri pants. She is wearing black socks and black gloves. She is holding a black purse in one hand.A woman in her thirties with straight long black hair is wearing an off-white puffer coat and beige scarf. She is also wearing brown pants and black wedge boots. She is carrying a black bag and has black gloves in her hand.A man in his late twenties with short black hair is wearing a black jacket with a yellow hood. He is wearing gray jogging pants with letterings on the side and blue sneakers.A man in his thirties with medium length straight black hair is wearing a yellow patched blue quilted puffer jacket. He is also wearing a pair of black trousers and a maroon sweater. His running shoes are black with grey patterns and soles.A middle-aged man with short black hair and a small bad patch is wearing a hooded navy-blue quilted puffer jacket. He is also wearing grey pants. He is wearing black socks and grey crocs. garnered increasing interest due to its huge practical value. In recent years, we have witnessed great progress on image-to-image person re-identification (IIReID) [24, 28, 50, 58, 61] , which has been successfully applied in various practical scenarios. However, ReID is still challenging when pedestrian images from some cameras are missing. In contrast, textual descriptions are more readily accessible and freer than images collected by specialized equipment, which can be obtained from witnesses at the scene. Thus, text-to-image person re-identification (TIReID) [30] has received a lot of research attention recently owing to it being closer to real-world scenarios.Compared to IIReID, TIReID faces a significant challenge in bridging the modal gap between images and texts while modeling their correspondence. Various methods have been devised to address this challenge by cross-modal interactive attention mechanisms [39, 63, 66] , semantically aligned local feature learning [5, 8, 41, 57] , and fine-grained auxiliary task-enhanced global feature learning [37, 38] . Nowadays, the emergence of vision-language pretraining models (VLP) has propelled the advancements in computer vision, showcasing exceptional capabilities in semantic understanding, multi-modal alignment, and generalization. It is intuitive to consider that the rich multi-modal prior information of VLP can be harnessed to enhance TIReID models in modeling inter-modal correspondence. Some follow-up CFine [56] , IRRA [20] extend VLP for TIReID, greatly promoting the progress of this task.Despite the success of VLP-based methods, the performance of TIReID still lags behind that of IIReID, especially in complex scenes like multiple cameras and drastic view changes. This is primarily due to the specificity of the TIReID task. As shown in Figure 1 (a), there are multiple images of the same identity, each annotated with specific text at the instance level. The goal of TIReID is to match all images of the same identity given a text, meaning to correctly associate each text with all images of the same identity. However, most existing methods, constrained by the alignment objective function [55] , only consider instance-level matching between each image and its annotated text, ignoring matches with other non-annotated texts of the same identity, called identitylevel matching. An intuitive idea to this problem is to directly match all images and texts under the same identity, but this can disrupt instance-level matching relationship [55] due to huge view variations, leading to performance collapse. So far, there have been limited solutions [8, 55] proposed to model identity-level matching for TIReID. SSAN [8] introduces an enhanced alignment loss to simultaneously align the image with its annotated text and another text sharing the same identity. LCR 2 S [55] designs a teacher-student network architecture to randomly fuse two images/texts under the same identity and then align them to model identity-level matching. While effective, they do not comprehensively model the matching relationship between all images and texts under the same identity. Moreover, the two-stage teacher-student framework is costly to train and lacks practicality. In this paper, we thus ask: Can an endto-end efficient TIReID model be trained to simultaneously model instance-level and identity-level matching?To address this problem, we propose a novel Prototypical prompting framework (Propot) that enables the network to model instancelevel and identity-level matching for TIReID simultaneously. As shown in Figure 1 , existing TIReID methods only model instancelevel matching between each image and its labeled text. In contrast, our Propot further introduces identity-level matching based on instance-level matching. We learn a prototype containing rich identity information for each identity and diffuse the rich information from the prototype to each instance, indirectly modeling identitylevel matching. This transforms the identity-level matching problem into an identity-enriched prototype learning problem.Specifically, our Propot follows the 'initialize, adapt, enrich, then aggregate' pipeline. Initially, we leverage CLIP's strong multi-modal alignment capability to cluster instance (image/text) features under each identity, forming the initial prototype for each identity. Since there's a gap between CLIP and TIReID data, this prototype is not fully adapted to TIReID. To address this, we propose a domain-conditional prototypical prompting (DPP) module inspired by CoOp [65] , which introduces a set of learnable prompt tokens to learn target domain knowledge and adapt the initial prototype to the TIReID task. However, instances under the same identity exhibit significant diversity due to factors like view changes and camera parameters. Ignoring this diversity can lead to a monotonous prototype, losing rich identity information. Thus, inspired by CoCoOp [46] , we propose an instance-condition prototypical prompting (IPP) module. This module generates two prototypical prompts conditioned on a batch of instances, leveraging both intra-modal and inter-modal instances to enhance diversity and bridge the modal gap. To integrate the multiple prototypes generated above, an adaptive prototype aggregation (APA) module is designed, which treats the initial CLIP-generated prototype as the baseline and adaptively ensemble these generated prototypes as the final prototype. Finally, we utilize a prototype-to-instance contrastive loss to diffuse the rich identity information from the prototype to each instance, enabling effective modeling of identitylevel matching. Our Propot is single-stage and end-to-end trainable. During inference, only the backbone of the network is used for inference, which is simple and efficient.Here are the main contributions of our paper: (1) We propose an end-to-end trainable prototypical prompting framework to model instance-level and identity-level cross-modal matching for TIReID simultaneously. (2) We transform the identity-level matching problem into an identity-enriched prototype learning problem. We use CLIP to generate initial prototypes and propose a domainconditional prototypical prompting (DPP) module and an instancecondition prototypical prompting (IPP) module to generate multiple identity-enriched prototypes. An adaptive prototype aggregation (APA) module is designed to fully integrate these prototypes. (3) Extensive experiments have been conducted to validate the effectiveness of Propot, and it achieves superior performance on the CUHK-PEDES, ICFG-PEDES, and RSTPReid benchmarks. 
 
 Appendix:  


Decision: Although all reviewers have upgraded their ratings (Weak Reject->Borderline Accept, Weak Accept->Accept, Borderline Reject->Weak Accept), the following concerns still need to be addressed.
1. The idea of using CLIP for Text-to-Image retrieval is not novel.
2. The authors' feedback cannot address the concerns raised by the reviewers. In particular, "the structure of the model is somewhat intricate".
3. This paper lacks key references, which all reviewers indicated.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel prototypical prompting framework, termed Propot, for the task of text-to-image person re-identification (TIReID). Propot addresses the identity-level matching challenge by developing identity-enriched prototypes. These prototypes are initially created using CLIP, adapted to the TIReID domain, enriched with additional instance information, and eventually aggregated into the final prototypes. The comprehensive identity information from these prototypes is then disseminated to individual instances, facilitating effective identity-level matching.
strength_weakness: strength_weakness: 1.	The identity-level matching issue in TIReID has not been adequately resolved. This paper proposes an end-to-end trainable framework that concurrently models both instance-level and identity-level matching for TIReID.
limitations: limitations: 1.	Overall, the structure of the model is somewhat intricate, and its performance does not surpass the state-of-the-art (SOTA), proving to be less than satisfactory. The comparison with SOTA methods such as [1-2] is absent.

2.	Some illustrations are unclear.

[1] 23IJCAI-Rasa: Relation and sensitivity aware representation learning for text-based person search

[2] 23MM-Towards Unified Text-based Person Retrieval: A Large-scale Multi-Attribute and Language Search Benchmark
paper_summary: paper_summary: Based on the observation that most methods only focus on instance-level matching, this paper additionally introduces the identity-level matching, which performs the alignment between multiple images and texts of the same person. Towards this end, this paper proposes an end-to-end prototypical prompting framework (Propot) to "initialize, adapt, enrich, and then aggregate" the prototypes. Extensive experiments on three benchmarks demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. **Good writing**: This paper exhibits an overall clear storyline. I can easily catch the motivation and the contribution of this work.
2. **Novelty**: This paper demonstrates a high level of novelty in adapting to the specific nature of identity-level matching in TIReID, which correspondingly  presents a chain of "initialize, adapt, enrich, then aggregate" to effectively achieve identity-instance alignment.
3. **Adequate experiments**:  This paper provides a thorough experimental evaluation of the proposed method in TIReID. The authors demonstrate the effectiveness and robustness of their approach through a comprehensive set of experiments.
limitations: limitations: 1. **Lack of critical reference**: This paper lacks the inclusion of several pivotal references[1] that are highly relevant to the motivation and the idea, which also delve into the identity-level matching. It is better to make a comprehensive comparison and deep analysis with the most relevant works.

2. **Inadequate Explanation**: 

    (1). The methodology described in Equation 1 for generating initial prototypes needs further clarification. Traditionally, prototypes are calculated as the average values of the features of individual instances. However, Equation 1 appears to accumulate the features of each instance instead. 

    (2). The dimension of $p_{a,i}$ in Equation 2 after self-attention encoder should be $(K+1) \times d$, which appears to be inconsistent with the description in Line 452.

    (3). I am having difficulty understanding the motivation behind the implementation of the cross-attention decoder, where the prototypes serve as query, while the instance features in a **batch** as key and value. It is worth considering that the instances in the batch may not encompass the entire range of identities. In cases where the prototypes lie outside the instances of the batch, how does the model handle this situation?

    (4). As the training goes on, the visual encoder and the textual encoder gradually adapt to the TIReID task. Given this adaptation process, it raises a question as to why frozen CLIP encoders are used to extract prototypes. Have the authors tried to use the on-training encoders to generate prototype features?

    Overall, the authors should provide a more detailed explanation and justification for this approach.



[1]. Bai Y, Cao M, Gao D, et al. RaSa: relation and sensitivity aware representation learning for text-based person search[C]//Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. 2023: 555-563.
paper_summary: paper_summary: This paper proposes a novel Prototypical Prompting framework (Propot) for the text-to-image person re-identification (TIReID) task, which addresses the limitations of existing methods by simultaneously modeling instance-level and identity-level matching. The key innovation lies in transforming the identity-level matching problem into an identity-enriched prototype learning problem. The proposed framework follows an 'initialize, adapt, enrich, then aggregate' pipeline. The final prototype-to-instance contrastive loss diffuses rich identity information from the prototype to each instance, enabling effective modeling of identity-level matching. Propot is designed to be single-stage and end-to-end trainable, enhancing efficiency.
strength_weakness: strength_weakness: 1、Pioneering the use of prompt learning for solving the challenge of multi-view matching in TIReID.

2、The proposed method achieves state-of-the-art performance on multiple benchmark datasets.
limitations: limitations: 1、In Table4, there is an error in the experiment serial number.

2、Q1:How are $pt_i^v$ and $pt_i^t$ initialized? Reinitialized before each epoch training or Initialized only once before model training. Q2: In the experiment Baseline+IniP, are the prototypes frozen or learnable, consistent with the DPP and IPP setup?

3、In the section "Comparisons with State-of-the-art Models", For "DECL introduces both mask language modeling and global-local semantic alignment to mine fine-grained matching, resulting in higher computational cost.".  This paper introduces an additional number of parameters to perform prototype hint learning, and there are no relevant experiments to prove that Propot has lower computational cost than DCEL. 

4、In the table of comparison results with state-of-the-art models, do the bold numbers represent the case of optimal performance? It is recommended that the authors give an explanation to improve readability. For example, the bold results in Table 3 are not the highest.

5、Many state-of-the-art methods have demonstrated mAP performance, and it is recommended that the authors add this metric to their experimental results to further validate method validity.

6、Line 666-667, DECL should be DCEL. You can correct it and cite DECL: Deep evidential learning with noisy correspondence for cross-modal retrieval.

7、The SOTA approaches are not exhaustive and the paper lacks citations. For example, Noisy-Correspondence Learning for Text-to-Image Person Re-identification