Fine-grained Semantic Alignment with Transferred Person-SAM for Text-based Person Retrieval

Authors: yihao  wang
meng  yang
rui 2024 cao
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 14:51:46

Abstract: Addressing the disparity in description granularity and information gap between images and text has long been a formidable challenge in text-based person retrieval (TBPR) tasks. Recent researchers tried to solve this problem by random local alignment. However, they failed to capture the fine-grained relationships between images and text, so the information and modality gaps remain on the table. We align image regions and text phrases at the same semantic granularity to address the semantic atomicity gap. Our idea is first to extract and then exploit the relationships between fine-grained locals. We introduce a novel Fine-grained Semantic Alignment with Transferred Person-SAM (SAP-SAM) approach. By distilling and transferring knowledge, we propose a Person-SAM model to extract fine-grained semantic concepts at the same granularity from images and texts of TBPR and its relationships. With the extracted knowledge, we optimize the fine-grained matching via Explicit Local Concept Alignment and Attentive Cross-modal Decoding to discriminate fine-grained image and text features at the same granularity level and represent the important semantic concepts from both modalities, effectively alleviating the granularity and information gaps. We evaluate our proposed approach on three popular TBPR datasets, demonstrating that SAP-SAM achieves state-of-the-art results and underscores the effectiveness of end-to-end fine-grained local alignment in TBPR tasks.

Main paper: 
 introduction: 
 The woman has shoulder length dark, straight hair and is wearing a short sleeve chambray blouse and light cropped pants. She is carrying a large, beige shoulder bag. The granularity and information gaps make it difficult for previous methods to consider the details of finegrained tasks. Our approach, SAP-SAM, first obtains the relationships of fine-grained features by Person-SAM and then exploits them from two perspectives: fine-grained matching via explicit local concept alignment and informative concept representing via attentive cross-modal decoding.Text-Based Person Retrieval (TBPR) [7, 33, 42, 49] is a finegrained cross-modal retrieval task, which aims to search for images corresponding to a given textual description in a person gallery. Compared to traditional image-based Person Re-Identification (Re-ID) [11, 19, 53, 60-62, 74, 76] , TBPR retrieves images in a crossmodal manner by only requiring text descriptions that are easier to obtain and more lenient, expanding its usage scenarios. As a fine-grained sub-task of image-text retrieval, TBPR focuses on the perception of human beings, one of the most important research objects, benefiting various significant applications of personal albums, surveillance, and public safety.To realize such a flexible and accurate image-text retrieval algorithm, it is necessary to address the intra-identity variations present in images and textual descriptions, which have been extensively studied in the vision and language communities and have achieved exciting progress. For the fine-grained cross-modal TBPR task, it is much more crucial to address the unique challenges, such as the granularity gap (e.g., as shown in the right of Figure 1 , images provide detailed descriptions at the quantitative and pixel level, while textual descriptions are more conceptualized and coarse) and the information gap (e.g., as shown in the top of Figure 1 , there are some focuses in the textural descriptions but not in the image).The granularity gap results in the fineness misalignment of semantic concepts due to the difficulty of extracting the fine-grained feature in the same granularity, preventing the effective cross-modal feature matching between text descriptions and images. Many methods struggle to extract local features (e.g., dividing images into patches straightforwardly [7, 16, 67] , using pose estimation [26] , performing an auxiliary attribute segmentation [63] and employing sentence analysis tools [43] ) or implicitly learn the relationships between cross-modal local parts after extracting their features through deep networks [14, 24, 52, 54, 71, 72] . These methods ignore the granularity gap between text partition (e.g., phrases) and image partition (e.g., image patches). Even if the text is sliced into much smaller units (e.g., words), they still have a larger granularity than an image region corresponding to a specific phrase. For example, a "shirt" is a coarser semantic concept than a specific region of "red wool shirt." The explicit image and text local features in existing methods are not semantically accurate enough, and the implicit alignment methods cannot guarantee that local features, such as image patches and text words, are of the same granularity, preventing further fine-grained alignment learning.The information gap results in an exact misalignment of semantic concepts due to the characteristic of vision and language, where the former has an overall and redundant visual representation, but the latter has focal and abstract text descriptions. Early studies [32, 33] employed contrastive learning for cross-modal matching, but the information gap is still significant due to the separate encodings of the image and text. The emergence of the vision-language pretraining model has generally alleviated the information gap between images and texts, and variants of contrastive representation learning [17, 37, 73] have been proposed for the task of TBPR. Even through contrastive learning is applied to the cross-modal data, the focal and meaningful semantic concept has not been well extracted from images. The information gap still hinders the effective crossmodal alignment between important semantic concepts of vision and language.A straightforward idea to overcome these gaps is to extract and align image and text features at the same granularity level, and generate an informative cross-modal attention for exact semantic concepts of vision and language. Following this idea, we proposed a novel Fine-grained Semantic Alignment with Transferred Person-SAM (SAP-SAM) for Text-Based Person Retrieval.With the idea of distilling knowledge from large pre-trained models (e.g., SAM [28] , BLIP [30] ) and transferring the knowledge to person retrieval, our approach first captures fine-grained semantic concept relationships in the same granularity from images and texts. To address the granularity gap, we designed Person-SAM, a person segmentation model driven by the text phrase, which is fine-tuned on a human parsing dataset with fine-grained annotations distilled from an off-the-shelf pretrained multi-modal model. Immediately after that, we slice the text in TBPR into fine-grained phrases, and then transfer the trained Person-SAM to extract the segmentation regions corresponding to these fine-grained phrases. It is worth noting that we only apply Person-SAM offline in the training phase, and the inference phase is consistent with existing methods.By exploiting these fine-grained semantic pairs in the same granularity and focusing more on important semantic concepts, we design fine-grained semantic alignment via Explicit Local Concept Alignment (ELCA) module and Attentive Cross-Modal Decoding (ACMD) method, respectively. The former pushes the model via cross-modal attention to distinguish whether there is a match between two fine-grained cross-modal concepts. The latter forces the model to predict the selected fine-grained conceptual content with a multi-modal context. We validate our approach on three mainstream benchmarks with state-of-the-art results, and these experiment results demonstrate the effectiveness of our approach. Our contribution consists of the following three points.1. Using the idea of knowledge distillation and transfer, we propose the Person-SAM model for capturing fine-grained feature relationships in TBPR, effectively alleviating the granularity gap in feature extraction.2. We design an Explicit Local Concept Alignment (ELCA) module to discriminate fine-grained similar features across modalities, effectively alleviating the granularity gap in model learning.3. We propose an Attentive Cross-Modal Decoding (ACMD) method to understand the important fine-grained context based on information from different modalities, effectively overcoming the information gap. 
related work 2.1 global alignment in tbpr: 
 Early research primarily involved extracting features from texts or images using neural networks, followed by feature aggregation and straightforward cross-modal alignment [7, 32] . Some methods [51, 52, 54] utilize refined single-modality pre-trained models, such as ViT [13] , BERT [10] , and DeiT [56] , at the text and image data, respectively, and align global representations of these models. Others [24, 36] utilize pre-trained multi-modal models, such as CLIP [46] , or pre-trained for pedestrian contrast learning, such as UniPT [50] , and then fine-tuned on TBPR tasks. However, these methods need more exact image-text feature interaction or finegrained cross-modal feature alignment, making it challenging to address the modality gap in TBPR.Previous research frequently relied on loss functions such as Triple Loss [16] , Ranking Loss [27, 41] , and Instance Loss [82] for the alignment of text-image representation. However, these methodologies have proven insufficient to address the information-gap and granularity-gap challenges inherent to TBPR. As a response, some researchers [5, 6, 20, 63] have explored innovative approaches that leverage contrastive losses [44] . Additionally, concepts like KL divergence and label smoothing have been introduced by the following studies [23, 24, 79] . While these refinements in loss functions have indeed boosted the performance, they remain predominantly focused on global alignment, which falls short of meeting the demands of fine-grained tasks such as TBPR.Researchers have used local information for fine-grained alignment. To varying degrees, we classify these methods into two categories: explicit and implicit methods. The former [7, 8, 42, 43, 80] usually possesses obvious correspondences in the data domain, i.e., it pushes an embedding of image part to be similar to an embedding of the text part, which corresponds to meaningful visual regions. Subsequent studies have attempted to utilize hyperpixels like keypoints [12, 29] and additional information like pose [26, 63] for slicing. However, these methods, which tend to be fine-grained at the image level, are not fine-grained at the text level. Meanwhile, due to the limited accuracy and insufficient samples, the image division, hyper-pixels, and pose information are practically challenging to accommodate the multi-scale information in the person concept.The implicit methods, on the contrary, utilize learnable neural networks to focus on data parts and enhance the local similarity across modalities. Some implicit methods [14, 54, 59, 72] utilize neural networks for local feature correspondence in the embedding space, while still borrowing knowledge (e.g., posture, position, or color) from the explicit data. However, lack of labeling data causes their alignment to be inefficient. Others [24, 51, 52, 64, 71 ] have adopted a completely implicit feature learning approach, i.e., completely discard this explicit information or prior knowledge. These methods lack a relationship between textual descriptions and image concepts, which causes them to be agnostic. There are also several approaches [15, 24] that borrow ideas from some generalized methods, such as mask language modeling and mask image modeling. However, it is still being determined whether the network in these methods utilizes the corresponding portion of one modality in the reconstruction process of the other modality. 
method 3.1 overview: 
 We seek a new way to resolve the granularity and information gaps mentioned above between person images and text descriptions. To address the granularity gap, we innovatively learn an enhanced fine-grained feature extraction model with distilling knowledge from SAM and transfer the model to extract fine-grained imagetext relationships for TBPR, which are then discriminated in a fine-grained manner across modalities. Moreover, we make the model focus on the informative image and text semantic concept representation to address the information gap. Following this idea, we proposed a novel model of Fine-grained Semantic Alignment with Transferred Person-SAM (SAP-SAM) for explicitly extracting, aligning and focusing on fine-grained image and text features as shown in Fig. 2 . Specifically, we first designed Person-SAM to capture the fine-grained one-to-one relationships between text phrases and image regions, which is the first step. Our Person-SAM is driven by entirely fine-grained texts, not several words or coarse semantic categories, i.e., 'white shirt with blue band trim' rather than 'shirt' or one-hot vector. Due to needing such a fine-grained dataset, we transformed the existing humanparsing dataset for training Person-SAM. In the second step, we transfer the trained Person-SAM to the TBPR dataset according to knowledge distillation and transferring while obtaining image regions corresponding to fine-grained phrases of TBPR. In the third step, we explicitly use these relationships to propose the design of Explicit Local Concept Alignment and Attentive Cross-modal Decoding for these exact local details to push the model's learning of fine-grained semantic concepts, respectively. To the best of our knowledge, we are the first to completely extract and align the fine-grained features of both text and images in TBPR.Let I and T denote the image and text modalities, respectively. Given a pair of person image and text I and T, we further let 𝑆 𝑖 and 𝐴 𝑖 denote the 𝑖-th fine-grained image region and attribute description in the same granularity.It is very challenging to extract fine-grained semantic concept relationships in TBPR. SAM shows a strong segmentation ability in universal tasks but performs poorly in fine-grained text-region semantics learning of TBPR. Based on SAM, we introduce text encoder and image region prediction modules and realize an end-to-end text-driven segmentation model to directly obtain one-to-one relationships from phrases to regions. This change allows us to extract relationships at the same semantic granularity, simultaneously addressing the granularity gap.Question: What kind of hair is she? Answer: It is ... (Right) The structure of Person-SAM. We introduce language models to SAM [28] as textual prompts, and then train on a transformed human parsing dataset. Finally, we will transfer the trained Person-SAM as a knowledge base to generate localized relationships in TBPR.For training such a fine-grained feature extraction model for TBPR, there are still numerous challenges, one of the most significant being the unavailability of a training dataset. Here, we borrowed the idea of knowledge distillation, transformed the dataset from approximate domains to a usable form, designed Person-SAM, and transferred the trained model to the TBPR training data. As shown in Table 1 , we achieved this by adding only a small number of training parameters. Finally, we use only the relationships extracted by Person-SAM in the training phase, while we do not use Person-SAM and any relationships in the validation phase.Table 1 : Trainable Parameters of our method in different phases.Trainable Parameters BiLMa [15] 220 M BLIP-2 (Section 3.2.2) 0 Person-SAM (Section 3.2.1) 5 M Alignment (Section 3.3) 195 MWe first introduce our proposed Person-SAM and how to train it., i.e., Step: 01 Person-SAM Training. An ideal situation would be if we could have a one-to-one mapping of text phrases to image regions, i.e., (𝐴 𝑖 , I) 𝜃 -→ 𝑆 𝑖 , where 𝐴 𝑖 is the text prompt, 𝑆 𝑖 is the segmented image region, 𝜃 is the mapping to be learned. For the implementation of 𝜃 , we build a segmentation model based on SAM with text input (that is, 𝐴 𝑖 ) instead of coarse class labels (e.g., using "red plaid shirts" instead of "clothing"). Person-SAM Components. We utilize the ViT model 𝜃 vit in vanilla SAM [28] to encode the image I into the image feature 𝑓 I . At the same time, we freeze all parameters of the image encoder. To capture fine-grained text features completely, we discard the dense prompt encoder in vanilla SAM and carefully design the text encoder in our Person-SAM. We keep the decoder 𝜃 dec in the vanilla SAM, trainable, and decode the segmentation mask corresponding to the text features.Text Prompt Design. Encoding text by introducing Word2Vec [40] can only work for coarse-grained words. Therefore, we introduce a language model, for example, BERT, to enable vanilla SAM to encode textual phrase prompts, that is, 𝐴 𝑖 . We freeze the parameters of the language model while keeping some learnable parameters 𝑓 𝑝 after the language model to adapt to the changing fine-grained semantic features in the text. Here, the learnable parameters 𝑓 𝑝 act as a class-agnostic prototype to push the language model to adapt to the domain-specific information in these phrases. (We describe more details in the Appendix.)Image Region Prediction. As shown in Figure 3 , we set a Multihead Cross-Attention (MCA) layer and make 𝑓 𝑝 as the query 𝑄 and text features as the values 𝐾, 𝑉 , i.e., f 𝐴 𝑖 = 𝜃 MCA 𝑓 𝑝 , 𝑓 𝐴 𝑖 , 𝑓 𝐴 𝑖 . Then the image region 𝑆 ′ 𝑖 predicted from the attribute phrase 𝐴 𝑖 through the Transformer decoder is denoted as 𝑆 ′ 𝑖 = 𝜃 dec 𝑓 I , f 𝐴 𝑖 . Similar to vanilla SAM [28] , we also train our Person-SAM by the Dice loss. However, we need a fine-grained text-region dataset that can train Person-SAM. Therefore, we considered training Person-SAM by transforming other task datasets of a similar domain.Here, we describe how to transform an existing dataset to train Person-SAM. We propose to generate a fine-grained semantic concept of the person using the available human-parsing dataset [38, 39] and multimodal generative model. However, there are two challenges related to the granularity gap in utilizing the existing dataset and the generative model for generating the fine-grained semantic concept. The first obstacle is that the provided category (denoted by 𝐶 𝑖 for the 𝑖-th image region) is coarser than the image region in the human-parsing data. For instance, the category is 'hair' regardless of the hair types of 𝑆 𝑖 , like 'blonde curly' and 'short black hair. ' The second obstacle is that the generative model cannot distinguish the granularity of image regions in the human-parsing dataset. For instance, the image region 𝑆 𝑖 is recognized as 'leg' regardless of the 'left leg' or 'right leg. ' To solve these obstacles, we propose merging the body parts on both sides (e.g., 'left leg' and 'right leg') as a single image region and using the multi-modal generative model to generate the finegrained attribute description. In addition, we purify this part of the image rather than using key points or other visual prompts for phrase generation. We asked the model about the image content based on the category 𝐶 𝑖 of the merged image region and designed diverse textural prompts (an example of 𝑃 𝑖 is: 'Question: What kind The woman has shoulder length dark, straight hair and is wearing a short sleeve chambray blouse and ...Cross-Modal Attention of hat is this? Answer: It is' , Note: We describe these details in the Appendix.),EQUATIONwhere merge is the merging operation optional for 𝑆 𝑖 and its other side 𝑆 𝑖 ′ and 𝑃 𝑖 is the predefined textural prompt with 𝐶 𝑖 for 𝑆 𝑖 , as shown in the left part of Figure 3 . 𝐴 𝑖 is the obtained fine-grained attribute description corresponding to the image region 𝑆 𝑖 . Finally, we choose BLIP-2 [30] as the generative model. Moreover, we blur the human-parsing image to mitigate the stylistic gap between it and TBPR. In this way, we transformed the humanparsing dataset into an image-text one-to-one fine-grained conceptual dataset for Person-SAM training.We then describe how to transfer Person-SAM to TBPR data, i.e., Step: 02 Person-SAM Transferring. Before extracting fine-grained local correspondences from TBPR data using the pre-trained Person-SAM, it is necessary to extract fine-grained phrase descriptions from the TBPR text. We extract text phrase segmentation from the TBPR text utilizing a simple language model with grammar rules like those of previous methods [43, 52] .For each sentence, as shown in Figure 4 , we extract the set of phrases Ā𝑖 | 𝑁 atr 𝑖=1 corresponding to the TBPR sentence, with 𝑁 atr representing the number of phrases. Then, we take these phrases as prompts and feed them into the trained Person-SAM separately, along with the person images I. Finally, we get the local part S𝑖 corresponding to each phrase Ā𝑖 , that is, S𝑖 = Person-SAM I, Ā𝑖 . The outputs of the model are paired as matched fine-grained imagetext relationships Φ = Ā𝑖 , S𝑖 | 𝑁 atr 𝑖=1 . These concept pairs and the original sample pairs will be used as training inputs. We show some Person-SAM extracted examples in Figure 6 . Let 𝑓 𝑖,𝑗 1 be the first token of the cross-modal attention result. For a semantic concept pair, (𝑖, 𝑗) ∈ POS, 𝑓 𝑖,𝑗 1 is the cross-modal interaction at the same granularity level, indicating the matching information. For the local feature with the index ī of a given modality, we randomly select another fine-grained localization j of the other modality as a negative sample and compute 𝑓 ī, j 1 , denoted by ( ī, j) ∈ NEG. Then we use a linear classifier, P, to predict whether the fused feature comes from a matched pairEQUATIONTo overcome the information gap, instead of using a random mask in cross-modal decoding, we proposed an attentive mask to ensure informative representation in the modalities of image and text. Random mask: In conventional cross-modal decoding [15, 24] , a randomized mask is utilized to predict the masked text words from the image. However, random decoding equal for all words would contain many task-irrelevant words (e.g., "is", "this", "he", "and", etc.) that contribute little to the task.Attentive mask: Given the index X Ā of the text phrases contained in each person description T, the idea is to perturb these task-relevant fine-grained parts more, i.e., the attention mask probability for local concept 𝑝 𝑎 is much larger than the random mask probability 𝑝 𝑟 for common words. Denote the attentive mask by Mask 𝑎 . By destroying and decoding these key details at X Ā more frequently, we push the model to learn those semantic details that are truly meaningful, f 𝑡 = CMA(Mask 𝑎 (𝑓 𝑡 , X Ā, 𝑝 𝑎 , 𝑝 𝑟 ), 𝑓 𝑣 , 𝑓 𝑣 ) s.t. 𝑝 𝑎 > 𝑝 𝑟 (4) where Mask 𝑎 (𝑓 𝑡 , X Ā, 𝑝 𝑎 , 𝑝 𝑟 ) is the masked text features and f 𝑡 is the fused feature using a cross-attention block based on the image features. Finally, the text decoding loss L 𝑑𝑒𝑐-𝑡 is conducted via the cross-entropy loss between the predicted words from f 𝑡 and the ground-truth T.We choose L 𝑆𝐷𝑀 proposed by [24] to optimize the global feature similarity of our model, which constrains the similarity of image and text global features to be consistent with the actual distribution through bidirectional KL divergence, i.e., L 𝑆𝐷𝑀 = KL(sim t2i ||𝑦 𝑡 2𝑖 ) + KL(sim i2t ||𝑦 𝑖2𝑡 ), where sim t2i /𝑦 𝑡 2𝑖 and sim i2t /𝑦 𝑖2𝑡 represent the text-to-image and image-totext similarity/truth distributions, respectively. We train our model via minimizing the multi-task loss L as shown in Equation 5.EQUATION 
experiments 4.1 settings: 
 We validate the performance of the proposed method using publicly available datasets that are frequently used in TBPR tasks, including CUHK-PEDES [33] , ICFG-PEDES [12] , RSPTReid [84] and ATR Dataset [38, 39] . More datasets information are shown in the Appendix.Following the established practice, we use the Rank-𝑘 metrics (𝑘 = 1, 5, 10) for evaluation. Given a caption from a query sentence, the model retrieves a corresponding person in the image gallery. If any image of the corresponding person is in the top-𝑘 retrieved images, we call it a successful search.We show the results of our method on the three popular benchmarks in Tables 2, 3 ,4, respectively. We train the baseline using Encoder and SDM loss [24] . CUHK-PEDES We first evaluate the proposed method on the most common benchmark, CUHK-PEDES. As shown in Table 2 , our method outperforms all state-of-the-art methods, achieving 75.05% Rank-1 accuracy, 89.93% Rank-5 accuracy and 93.73% Rank-10 accuracy, respectively. We have characterized how these methods use local features, where "-" means not used with "E" standing for explicit methods and "I" for implicit methods. The results show that our methods surpass previous explicit methods and also recent implicit methods, in particular, 1.67% and 1.02% Rank-1 outperform compared to IRRA [24] and BiLMa [15] , respectively. The results indicate that explicitly learning these fine-grained relationships of local features is beneficial. Compared to APTM [75] , we are still 1.48% behind on R-1, mainly because APTM built a training dataset of over 1, 500𝐾 images and used this large dataset for training, while CUHK-PEDES only has 34𝐾 image data for training (more than 40× times smaller). Nevertheless, even so, our SAP-SAM still leads on R-5 and R-10.Table 3 : Main result of our SAP-SAM on ICFG-PEDES.Ref.Type R@1 R@5 R@10 SSAN [65] arXiv21 E. 54. 23 ICFG-PEDES The performance of the ICFG-PEDES dataset is shown in Table 3 . Likewise, the proposed method outperforms all existing methods reported on ICFG-PEDES, achieves Rank-1 of 63.97%, Rank-5 of 80.84% and Rank-10 of 86.17%. We still gained some improvement in all metrics compared to previous randomized [24] or explicit [50] methods.Table 4 : Main result of our SAP-SAM on RSTPReid.Ref.Type R@1 R@5 R@10 DSSL [ RSTPReid We also validate the effectiveness of our method on a newly proposed dataset, RSTPReid. As shown in Table 4 , our method achieves Rank-1 of 62.85%, Rank-5 of 82.65% and Rank-10 of 89.85%, respectively, outperforming previous explicit and implicit methods. In particular, compared to the recently proposed BiLMA [15] , a SOTA method, ours outperforms 1.65%, 1.15% and 1.05% the Rank-1, Rank-5, and Rank-10 leads, respectively.Collectively considering the results of three datasets with different scales, sizes, and focuses, our method outperforms previous methods and achieves state-of-the-art results, once again proving our approach is practical and robust. In contrast to IRRA, we no longer randomize but introduce attentive mask based on the relationships of text phrases, and these results demonstrate the effectiveness of exploiting textual relationships. Compared to BiLMa, we introduce ELCA, and the better results illustrate the importance of For ACMD component (Section 3.3.2), we add both fine-grained text phrases ( '+L 𝑑𝑒𝑥-𝑡 ' 𝑣𝑠. 'Baseline', '+L 𝑚𝑎𝑡𝑐ℎ +L 𝑑𝑒𝑐-𝑡 ' 𝑣𝑠. '+ L 𝑚𝑎𝑡𝑐ℎ ') and fine-grained image regions ('+L 𝑑𝑒𝑥-𝑖 ' 𝑣𝑠. 'Baseline', '+L 𝑚𝑎𝑡𝑐ℎ + L 𝑑𝑒𝑐-𝑖 ' 𝑣𝑠. '+ L 𝑚𝑎𝑡𝑐ℎ ') to the decoding brought significant benefits, yet performance slightly decreased when combining these two approaches ('+L 𝑚𝑎𝑡𝑐ℎ + L 𝑑𝑒𝑐-𝑡 + L 𝑑𝑒𝑐-𝑖 ' 𝑣𝑠. '+L 𝑚𝑎𝑡𝑐ℎ + L 𝑑𝑒𝑐-𝑖 ', '+L 𝑚𝑎𝑡𝑐ℎ + L 𝑑𝑒𝑐-𝑡 + L 𝑑𝑒𝑐-𝑖 ' 𝑣𝑠. '+L 𝑚𝑎𝑡𝑐ℎ + L 𝑑𝑒𝑐-𝑡 '). We hypothesize that images contain a more homogeneous information density than text. Text, on the other hand, contains more information only for key phrases due to morphological differences in the data source itself.Person-SAM relationships. We used the relationships extracted by Person-SAM on the LGUR [51] and SSAN [12] methods to verify whether these relationships are valid for other methods. The LGUR and the SSAN methods learn fixed fine-grained conceptual divisions, whereas an improvement is obtained when using our relations. As shown in Table 6 , the results show that adding the relations extracted by Person-SAM can bring an R-1 improvement of 1.22% to LGUR and an R-1 improvement of 1.37% to SSAN, which demonstrates that our Person-SAM can effectively drive the model's alignment of fine-grained features. 7 compares our original and attentive random masks. This experiment proves that masking the more important parts works in fine-grained tasks. We also explored the effect of the mask probability 𝑝 𝑎 . The results are shown in Figure 5 . We started with the probability 𝑝 𝑎 = 0.15 and gradually increased the size of 𝑝 𝑎 by 0.05 as a step. We obtain the best score when 𝑝 𝑎 = 0.35. Furthermore, performance will drop as 𝑝 𝑎 increases. We hypothesize that this is due to a high masking probability that can make it difficult for the model to understand the meaning of these key parts of the word, thus impairing performance.We visualize some segmentation results of Person-SAM, as shown in Fig. 6 . The phrases on top of the figure drive these results. Although there are some pixel-level deviations in the Person-SAM produced by TBPR compared to those of the exact image segmentation data, these regions roughly contain the blocks where the objects are present. Person-SAM achieves good results in these examples, whether they are long text phrases (such as the leftmost example at the bottom), small examples (such as the hair, shoes, or bag in the figure), or confusing examples (such as the rightmost example at the top). Admittedly, these results still have some flaws, but they are fine for TBPR.Figure 7 illustrates an example of Top-10 retrieval results for our SAP-SAM and the baseline. As a comparison, our method retrieves more accurate results when the baseline model fails. Further, we found that our approach can capture even small fine-grained semantic concepts. Fine-grained semantic matches can be retrieved even in the wrong cases, as shown by the boxes in the figure. This phenomenon further illustrates the effectiveness of aligning at a fine-grained semantic level and proves that learning the relationship between fine-grained cross-modal semantics is crucial. A man with short black hair is wearing a grey and black jacket, a pair of black pants, black shoes and a black backpack.A man in a white shirt, a pair of black shorts and a pair of black and white shoes.  
conclusion: 
 We propose a novel Fine-grained Semantic Alignment with Transferred Person-SAM (SAP-SAM) method to address the granularity and information gaps in TBPR. First, with knowledge distillation and transferring, we successfully trained the proposed Person-SAM and exactly captured the relationships between fine-grained concepts. Then, in the fine-grained feature matching, our designed Explicit Local Concept Alignment (ELCA) effectively enhances the model's discrimination of fine-grained features, addressing the granularity gap in previous approaches. In addition, our proposed Attentive Cross-Modal Decoding (ACMD) has enhanced the model's understanding of fine-grained content, addressing the information gap in previous approaches. We have validated and achieved optimal result on three existing benchmarks, proving our method's effectiveness. The success of utilizing fine-grained crossmodal conceptual relationships also provides new ideas for future research. We validate the performance of the proposed method using publicly available datasets that are frequently used in TBPR tasks, including CUHK-PEDES [33] , ICFG-PEDES [12] , RSPTReid [84] and ATR Dataset [38, 39] .CUHK-PEDES [33] is a classical TBPR dataset with 40, 206 images and 80, 412 textual descriptions for 13, 003 identities. The pedestrian image of CUHK-PEDES comes from five existing person reidentification datasets, CUHK03 [35] , Market-1501 [81] , SSM [69] , VIPER [18] , and CUHK01 [34] . The training and testing sets comprise 11, 003/1, 000 persons with 34, 054/3, 074 images and 68, 108/6, 156 sentence descriptions, respectively.ICFG-PEDES [12] contains 54, 522 pedestrian images of 4, 102 different identities with more fine-grained text descriptions. The images in ICFG-PEDES are collected from the MSMT17 database [67] . The training set contains 34, 674 image-text pairs from 3, 102 pedestrians, while the test set contains 19, 848 image-text pairs for the remaining 1, 000 pedestrians.RSPTReid [84] is a real scenario text-based person re-identification dataset based on MSMT17 [67] . It contains 20, 505 images of 4, 101 persons from 15 cameras in total. The training set consists of 3, 701 people, 18, 505 images, and 37, 010 sentence descriptions. The test set includes 1, 000 images and 2, 000 textual descriptions of 200 pedestrians.ATR Dataset [38, 39] is a human parsing dataset with 17, 706 images and 18 semantic categories. The images in ATR come from diverse sources. Each image is assigned several semantic categories and labeled with fine pixel-level annotations. We randomly split the ATR dataset into the train, valid, and test sets with the ratio of 8 : 1 : 1. 
 
 Appendix: We loaded the pre-trained CLIP-B/16 for the text and image encoders and randomly initiated the rest of the modules. During training and testing, all images are uniformly scaled to 384 × 128, and the maximum length of the text is set to 77. We train the model using the Adam optimizer and set the learning rate of the pre-training module to 1𝑒-5 with the cosine decay strategy. For the other modules, we set the learning rate 5𝑒-5. The mask probability 𝑝 ′ is set to 0.35. The size of the image patch is 16. For the training of the segmentation model, we use SAM-Base and BERT-base-uncased for the text model. We freeze the base encoder and train only the decoder and other parameters. We set the learning rate to 1𝑒-4, and train 20 epochs. The image input size of the model is 1024 × 1024, and the maximum length of each phrase is 16.We performed a semantic merging on the ATR dataset when training Person-SAM in the main text, and here we explain in detail why we conducted this operation. First, the original ATR data consists of 18 categories (including the background). At the same time, it contains, for example, categories with positional information such as "left-shoe" and "right-shoe, " as shown in Fig. 8 . These categories usually appear in entirely different shapes due to the masking of the parts. As shown in the three left columns of Figure 8 , we demonstrate some masked semantic classes that are unfavorable for training text-driven semantic segmentation. Therefore, we need to merge these similar semantics to mitigate some of the effects due to masking. In addition to this, for example, as shown in the two right columns of Figure 8 , smaller regions, such as glasses, are challenging to achieve detailed generation of linguistic descriptions when used alone, whereas the generative model can work well when merged with the face. Finally, as shown in Table 8 , we will merge the semantic classes. Instead of merging, we generate the phrases corresponding to these regions separately for the rest of the semantic classes.In addition to this, inspired by some recent research on visual prompts [9, 57, 77] , we still tried to utilize some visual prompts to directly generate descriptions of the corresponding regions, as shown in Fig. 9 . We attempted to use visual prompts for points, regions, and boxes, respectively, and asked for the content of the corresponding regions via text. However, this result was not satisfactory, and these attempts invariably produced very noisy results while still generating many errors when confronted with those fine semantic classes. Therefore, we directly filter as much irrelevant interference as possible during the actual generation process and target the prompts for each semantic class.After merging the semantic classes and processing the image regions, we next need to design the textual prompts to make the model output fine-grained phrases as correctly as possible. Some existing research also suggests that textual prompts [3, 4, 25, 45, 47, 55] , even some punctuation marks that seem small to humans, may contain very unique semantic information to the model. Therefore, we need to design these linguistic prompts very carefully. We take a natural language perspective, keeping sentences as fluent as possible, and ask questions about the current semantic class. This operation aims to narrow down the model's choices of what to output so that the model outputs more accurate results than without containing the current semantic class word. Taking hair as an example, we show the design of some semantic classes, as shown in Table 9 . Finally, we manually evaluate the generation quality of each text prompt on several samples and select the most accurate one for the template. We perform this operation for each semantic class, and finally, we get the specific text prompt templates for all semantic classes as shown in Table 10 . Question: What kind of hair is this? Answer:The kind of hair she has is short.Question: What kind of hair is she? Answer: She has short, curly hair We also have two strategies for linear features: (3) First, the image I is masked immediately after Patch Embedding and the masked tokens are fed to the subsequent attention layer in ViT.(4) The other is to mask the image features 𝑓 𝑣 acquired by ViT. Unlike the discrete strategy, both strategies use MSE loss ℓ 𝑀𝑆𝐸 to compute the decoding loss L 𝑑𝑒𝑐-𝑖 . In addition to that, the rest of the masking and decoding strategies remain unchanged.The results of these strategies are shown in Table 11 , with strategy (3) achieving the best results. Although image cross-modal decoding alone is effective, performance impairment occurs when combining text cross-modal decoding with image cross-modal decoding, as shown in the main text. How to solve this problem is also part of our subsequent research.In the design of Person-SAM, we introduced a text encoder to accommodate text-driven prompts to generate exactly corresponding fine-grained regions. We explored a few ways to use the text features, and the segmentation result in the ATR dataset is shown in Table 12 , and we used mAP as the evaluation criterion.We then briefly describe these methods. Method 𝑓[CLS] 𝐴 𝑖 means using the [CLS] token feature of the text 𝐴 𝑖 as text prompt. Method 𝑓 𝐴 𝑖 meas using dense text features (i.e., features of all 𝐴 𝑖 tokens.[CLS] 𝐴 𝑖• 𝑓 I means using dot products between dense textual prompts with image features as prompts for the decoder. Method 𝜃 MCA 𝑓 𝑝 , 𝑓 𝐴 𝑖 , 𝑓 𝐴 𝑖 , which we used, means using Multi-head Cross-Attention(MCA) to obtain the fused textual features between some learnable tokens 𝑓 𝑝 and the dense textual features as input to the decoder's prompts.As shown in Table 12 , method 𝜃 MCA 𝑓 𝑝 , 𝑓 𝐴 𝑖 , 𝑓 𝐴 𝑖 is better than the other methods, so we use this method in the Person-SAM structure. The reason for this phenomenon is that fine-grained features require an exact representation, so methods with dense features 𝑓 𝐴 𝑖 are better than methods using 𝑓[CLS] 𝐴 𝑖 . At the same time, the learnable parameters provide domain adaptation, which helps Person-SAM to focus more easily on details related to pedestrians.In addition to the alignment strategies described in Section 3.3.1, inspired by previous work, like GLIP [31] , X-VLM [78] , we still explored some other alignment strategies. Method 'avgPool' means using average pooling to aggregate each local feature, i.e., f 𝑖 𝑣 = avgPool f 𝑖 𝑣 and f 𝑖 𝑡 = avgPool f 𝑖 𝑡 , and then employing the In-foNCE loss to constrain the cosine similarity between image and text local features. Method 'Conv1D' means using 1-𝑑 convolution network instead of average pooling in method 'avgPool' with Table 13 shows the influence of different matching approaches. The benefit of retaining features for all tokens compared to aggregated features, such as 'avgPool', suggests that the aggregated features lose some fine-grained information that is trivial in traditional tasks but critical in TBPR. Our ELCA can push the model more strongly to discriminate semantic differences between localizations by interacting with fine-grained features. This result demonstrates the specificity of the TBPR task, i.e., it is a fine-grained task, and the importance of aligning fine-grained semantic features. We still trained SAP-SAM on a larger backbone network, such as CLIP(ViT/L-14), and the results are shown in Table 14 . Our SAP-SAM achieved better results, but we did not use this result in the text for a fair comparison.We mainly propose a fine-grained local feature alignment method for images and text to improve the quality of the model's representation of cross-modal features through fine-grained feature identification and understanding, thus improving the model's performance in downstream tasks. We focus on some problems in the TBPR task from a fine-grained perspective. However, due to resource constraints, our approach still has some problems:• In Person-SAM transfer, due to the limited computational resources, the model we chose is small, which may limit some of the model's capabilities and lead to less fine-grained results obtained. • Since there are still some domain differences between the ATR and TBPR datasets in the transfer process, this problem still exists even though we have performed some data style transformations. • Since we retained all the feature blocks, the computation process took up more time during the learning process of fine-grained features. In the future, we will also investigate how to learn this relationship faster. 


Decision: All the reviewers recognize that the technical contribution and experimental results are acceptable. 
This submission is acceptable if room

Reviews: review: paper_summary: paper_summary: 1、This paper primarily addresses the issue of the granularity gap between textual descriptions and real image information in text-based person retrieval tasks by aligning text phrases and images at the same semantic granularity to bridge the semantic gap.
2、To solve this problem, the authors first propose a method called Person-SAM (SAP-SAM) to align fine-grained image and text phrases. They extract knowledge from SAM through a pre-training approach and transfer it to extract fine-grained image-text relationships using the TBPR task format dataset. During training, they optimize fine-grained matching through explicit local concept alignment and attention-based cross-modal decoding, distinguishing fine-grained image and text features at the same granularity level.
strength_weakness: strength_weakness: The paper innovatively proposes a new text-image retrieval method based on fine-grained semantic alignment. By fine-tuning the transferred knowledge from the SAM large model to adapt it to the TBPR task, the Person-SAM structure is proposed. To address two shortcomings of Person-SAM, the Explicit Local Concept Alignment (ELCA) and Attention-based Cross-Modal Decoding (ACMD) modules are introduced. The effectiveness of the experiments is validated on three benchmark datasets.
limitations: limitations: 1. The method proposed in this paper consists of a pre-training stage and a training stage. What about the training time complexity?
2. The experimental results of the RaSa method in Table 2 differ significantly from the original paper, while the reason is not explained. Also no comparison of the experimental results of the RaSa method in Tables 3 and 4. Please supplement the experiments or provide an explanation.
3. The reason for the lower comparison results between this method and the APTM method as stated is that APTM proposed a new large-scale dataset for pre-training. This argument is not fully substantiated. Can this method also be pre-trained on the large-scale dataset proposed by APTM? 
4. The impact of the Explicit Local Concept Alignment (ELCA) and Attentive Cross-Modal Decoding (ACMD) modules on the final validation results lacks ablation experiment verification.
5. The experimental data in the method diagram of the article is too extensive and can be appropriately omitted.
paper_summary: paper_summary: This paper proposes a Fine-grained Semantic Alignment model with Transferred Person-SAM (SAP-SAM) to effectively address the granularity and information gaps inherent in Text-based Person Retrieval task.
strength_weakness: strength_weakness: 1. This paper reveals the necessity of explicit alignment and fine-grained alignment, constructs fine-grained annotations for text-based person retrieval task and proposes a Person-SAM model to capture fine-grained feature relationships in TBPR.
2. This paper proposes the ELCA and ACMD modules to exploit the relationship of fine-grained semantic concepts in the same granularity.
limitations: limitations: 1. When the fine-grained text phrases are obtained, the proposed Person-SAM can be replaced by the human-parsing model proposed by VITAA[1]. If so, only the inference step is needed to construct fine-grained region-phrase pairs, and there's no need to train the model.
2. Why not do the fine-grained semantic alignment directly by pretrained Person-SAM instead of designing the ELCA module?
3. Implementation details should be included in the main text rather than in the appendix.
4. There are many logical inconsistencies and unclear causal expressions in the paper. (For instance, L156, L307, L375, L505)

[1] Zhe Wang, Zhiyuan Fang, Jun Wang, and Yezhou Yang. 2020. Vitaa: Visualtextual attributes alignment in person search by natural language. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16. Springer, 402–420.
paper_summary: paper_summary: This paper proposed a novel approach called Fine-grained Semantic Alignment with Transferred Person-SAM (SAP-SAM). SAP-SAM addressed the disparity in description granularity and information gaps between images and text in TBPR tasks by extracting and aligning fine-grained textual and visual features using a transferred Person-SAM model, and exploiting the captured relationships via Explicit Local Concept Alignment (ELCA) and Attention Cross-Modal Decoding (ACMD) modules to enhance fine-grained discrimination and cross-modal understanding. SAP-SAM was evaluated on CUHK-PEDES, ICFG-PEDES, and RSTPReid, outperforming SoTA methods. Ablation experiments demonstrate the effectiveness of the proposed components.
strength_weakness: strength_weakness: 1. By fine-tuning the person segmentation (Person-SAM) model on human parsing datasets, it learns to capture fine-grained relationships between text phrases and image regions at the same semantic granularity.
2. The ELCA module enhances the model's ability to distinguish matched fine-grained concepts across modalities, which improves fine-grained discrimination. 
3. The ACMD focuses on informative semantic concepts by using an attentive mask. This helps the model understand the important fine-grained context based on information from different modalities.
4. SAP-SAM leverages the idea of knowledge distillation and transfer to train the Person-SAM model on existing datasets. This allows the model to learn from related domains and improves its ability to capture fine-grained relationships in TBPR data.
limitations: limitations: 1. The authors acknowledge pixel-level deviations compared to the exact image segmentation data. These deviations may affect the precision of the captured fine-grained relationships.
2. The authors compare SAP-SAM with the APTM method, which uses a large-scale pre-training dataset of over 1,500K images. In contrast, the CUHK-PEDES dataset for training SAP-SAM has only 34K images. While SAP-SAM still outperforms APTM on Rank-5 and Rank-10 metrics, it falls short on Rank-1 accuracy. This suggests that large-scale pre-training could potentially further improve SAP-SAM’s performance.
3. While the authors emphasize that Person-SAM is only used offline during training, the computational overhead of this process is not explicitly discussed.
4. The interpretability of the learned relationships and SAP-SAM’s decision-making process is not extensively discussed in the paper.
paper_summary: paper_summary: The paper introduces a Fine-grained Semantic Alignment with Transferred Person-SAM (SAP-SAM) model to address the granularity gap in text-based person retrieval (TBPR). The Person-SAM model is fine-tuned on a human parsing dataset using fine-grained annotations derived from an off-the-shelf pre-trained multi-modal model. Text in TBPR is segmented into fine-grained phrases, and these segments are aligned with corresponding segmented areas in images using the trained Person-SAM model. The alignment focuses on semantic pairs of the same granularity, emphasizing important semantic concepts through Explicit Local Concept Alignment (ELCA) and Attentive Cross-Modal Decoding (ACMD). ELCA uses cross-modal attention to distinguish matches between fine-grained cross-modal concepts, while ACMD predicts fine-grained conceptual content within a multi-modal context. The approach is validated on three mainstream benchmarks, demonstrating its effectiveness with state-of-the-art results. This method significantly improves the granularity and information alignment between images and text in TBPR tasks.
strength_weakness: strength_weakness: The paper is grammatically fluent and structurally clear, and the ablation study provided is very detailed. Furthermore, unlike previous methods that employed random alignment, this paper innovatively uses pre-trained multi-modal models to add fine-grained annotations to a human parsing dataset for fine-tuning the Person-SAM model. This approach enhances the model's ability to accurately segment fine-grained textual phrases and align them with corresponding image regions, thus significantly improving the alignment capabilities between text and images.
limitations: limitations: 1. Figure 1 has a spelling error: 'Overall & Redudant' should be corrected to 'Overall & Redundant'.

2. The APTM method mentioned in line 673 of this paper also includes a cross-modal attention matching module and masked language modeling. Given the inconsistency in training data scale, a direct and fair performance comparison cannot be conducted. Therefore, could the Person-SAM model be added on top of the APTM framework to facilitate a performance comparison with APTM? Moreover, it seems that the performance is not as stated in the text: 'Compared to APTM [58], we are still 1.12% behind on R-1, … Nevertheless, even so, our SAP-SAM still leads on R-5 and R-10.' In fact, this paper reports being 1.48% lower on R-1 than APTM, and also lower on R-5 and R-10.

3. Although the Person-SAM is only applied offline during the training phase, using this model along with fine-grained semantic alignment could potentially result in higher computational costs. The supplementary materials mention that fine-grained semantics consume more computational resources; however, they do not provide specific details on the model's runtime and the computational resources required, which are critical considerations for practical applications.

4. During the inference phase, it is necessary to input text features and image features in pairs into the cross-modal attention to obtain matching scores. This seems to require sorting the features output by the encoder, followed by selecting the top-k pairs of images and texts for input. The paper does not mention the specific implementation process for this procedure.