Selection and Reconstruction of Key Locals: A Novel Specific Domain Image-Text Retrieval Method

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 05:22:37

Abstract: In recent years, Vision-Language Pre-training (VLP) models have demonstrated rich prior knowledge for multimodal alignment, prompting investigations into their application in Specific Domain Image-Text Retrieval (SDITR) such as Text-Image Person Re-identification (TIReID) and Remote Sensing Image-Text Retrieval (RSITR). Due to the unique data characteristics in specific scenarios, the primary challenge is to leverage discriminative fine-grained local information for improved mapping of images and text into a shared space. Current approaches interact with all multimodal local features for alignment, implicitly focusing on discriminative local information to distinguish data differences, which may bring noise and uncertainty. Furthermore, their VLP feature extractors like CLIP often focus on instance-level representations, potentially reducing the discriminability of fine-grained local features. To alleviate these issues, we propose an Explicit Key Local information Selection and Reconstruction Framework (EKLSR), which explicitly selects key local information to enhance feature representation. Specifically, we introduce a Key Local information Selection and Fusion (KLSF) that utilizes hidden knowledge from the VLP model to select interpretably and fuse key local information. Secondly, we employ Key Local segment Reconstruction (KLR) based on multimodal interaction to reconstruct the key local segments of images (text), significantly enriching their discriminative information and enhancing both inter-modal and intra-modal interaction alignment. To demonstrate the effectiveness of our approach, we conducted experiments on five datasets across TIReID and RSITR. Notably, our EKLSR model achieves state-of-the-art performance on two RSITR datasets.

Main paper: 
 introduction: 
 With the surge in data proliferation, the Internet and social media platforms have been overwhelmed with an abundance of multimodal content, including both images and text. There is a great demand to automatically retrieve useful information from these vast amounts of data [12, 23, 39, 46, 50, 51] . In response to this demand, significant progress has been made in Cross-modal Image-Text Retrieval (CMITR) in the past few years [12, 17, 23, 46, 52] . Recently, with the emergence of Vision-Language Pre-training (VLP) models [22, 24, 25, 29, 39, 40] , CMITR has undergone further advancements.However, it is worth noting that VLP models like CLIP exhibit less favorable performance in Specific Domain Image-Text Retrieval (SDITR), such as Text-Image Person Re-identification (TIReID) and Remote Sensing Image-Text Retrieval (RSITR), in comparison to their performance in CMITR tasks within the general domain, as depicted in Figure 1 (a). The discrepancies in these metrics suggest that specific domains possess unique data characteristics that differ from those of the general domain. Specific domain data typically exhibit high image similarity [53] , with semantic nuances often confined to key local segments, such as object regions in images or content-rich words in the text, as seen in Figure 1 (b). Even minor changes in these segments can significantly alter the entire content, highlighting the importance of these segments. Thus, SDITR necessitates that models concentrate on key local segment information [49] to enhance the representation of image-text features in a shared space and improve image-text alignment.To facilitate the utilization of key local segment information, early bottom-up attention methods [23, 46] captured discriminative local image features, boosting retrieval accuracy in general domains. However, their cross-domain applicability is limited for specific domain image-text tasks. Subsequent advancements have shifted towards leveraging the robust prior knowledge of pre-trained models for specific domains. Instead of directly extracting key local information, these methods [5, 9, 45, 48, 50] employ interactions among all local features in both images and text to implicitly guide the model's attention to key local information. While these methods enhance retrieval performance, they suffer from two main issues.Firstly, interactions among all local features bring inevitable noise and uncertainty, as they indiscriminately engage with all the local features rather than focusing on key local features. As shown in Figure 1 (c), these methods [5, 9, 45, 48, 50] can yield false correlations among extraneous local features, bringing noise and causing misaligned image-text pairs that lower retrieval accuracy [21] . Selecting and utilizing key local features from the multitude is vital for enhancing image-text feature representation. Secondly, VLP models like CLIP [39] have not been optimized for local features, resulting in a lack of discriminability for local features. Throughout the pre-training stage, CLIP predominantly employs contrastive losses that favor instance-level features [49] over local region features. . . . . . Consequently, as shown in Figure 1 (d), insufficiently optimized key local features exhibit reduced discriminative capacity and become entangled with other local features. In specific scenarios, the lack of discriminative key local features hampers the differentiation between local segments, thus diminishing the efficacy of subsequent feature interaction modules. Therefore, enhancing the discriminability of key local features is crucial.To address the aforementioned two issues, we introduce the Explicit Key Local information Selection and Reconstruction (EKLSR) framework based on CLIP. Unlike previous methods [5, 9, 45, 48, 50] that utilize all local features without distinction, our approach not only interpretably selects key local features but also bolsters the discriminability of these features through a multimodal interactionbased reconstruction task. EKLSR consists of a Key Local information Selection and Fusion (KLSF) module and Key Local segment Reconstruction (KLR) based on multimodal interaction.KLSF directly selects and utilizes key local features from the multitude to enhance image-text feature representation. It leverages hidden knowledge from the CLIP to interpretably select key local information, thereby enhancing the final image-text feature representation. Specifically, it employs VLP model hidden priors to assign an importance factor to each local feature, gauging its significance. In the key feature selection process, local features with higher importance factors are more likely to be selected. Selected key local features capture object details but lack associations between the objects. Thus, KLSF fuses key local features with instance-level global features representing object relationships, significantly enhancing the final multimodal feature representation. Additionally, KLSF operates independently on image and text branches, maintaining the dual-stream structure of the model and enabling offline computation during inference.KLR enhances the discriminability of the key local features. Inspired by works [2, 61] that enhance feature specificity through feature reconstruction tasks, KLR applies both Masked Language Modeling (MLM) and Masked Visual Modeling (MVM) tasks to the reconstruction of selected key local features. These two tasks ensure that the features retain their unique information, thereby preserving their discriminative properties. MLM and MVM are generally utilized in the pre-training stage of Visual-Language Pre-training (VLP) [6, 25, 32, 43] , we make the first attempt to demonstrate the effectiveness of MLM and MVM in downstream fine-tuning tasks. Innovatively, to ensure the key local segments are reconstructed, we reconstruct key segments selectively rather than random reconstruction of all segments. Furthermore, we predict masked segments through the integration of intra-modal and inter-modal interaction information. This not only strengthens multimodal fine-grained alignment but also aids the backbone network in extracting more discriminative features.The contributions of this paper can be summarized as follows:• We propose EKLSR to make CLIP more adaptable to finegrained downstream tasks without substantial additional supervision and inference costs.  
related work 2.1 vision-language pre-training: 
 Vision-Language Pre-training (VLP) aims to learn the semantic correspondence between the vision and text by pre-training on a large-scale dataset. Inspired by the success of Transformer-based language model BERT [8] and vision model ViT [11] , most current multimodal models [6, 18, 22, 24, 25, 29, 37, 39, 40, 44] adopt their variants to learn multimodal representations. Existing VLP models can be categorized into two types: single-stream and dualstream models. Single-stream models [18, 22, 25, 37, 44] combine extracted text and vision local embeddings as input to a Transformer structure to extract a joint representation of image-text pairs. They are more effective in capturing fine-grained relationships between images and text, leading to higher retrieval accuracy compared to dual-stream models. Furthermore, the complex interaction mechanism leads to slower inference speed, which does not meet the real-time requirements of specific domain tasks. On the other hand, dual-stream models [12, 39, 44] utilize independent encoders to map images or text into global embeddings and align them on the common space. Recent Transformer-based dual-stream models [19, 25, 39] have chosen to improve their performance by leveraging additional large-scale data. Despite their impressive performance in image-text retrieval tasks, their heavy reliance on instance-level representations limits their ability to capture local features effectively. Consequently, they face constraints when applied to Specific Domain Image-Text Retrieval (SDITR) tasks.TIReID: TIReID is a multimodal task [38, 56] , first introduced by [28] . Compared to general cross-modal retrieval tasks, TIReID demands models to pay more attention to fine-grained information and the correspondence between modalities to distinguish the differences among pedestrians. Initial global matching approaches [57, 58] aligned images and text in a joint embedding space through cross-modal matching loss functions, neglecting direct emphasis on fine-grained local information. Recently, several approaches [5, 9, 45, 48] have emerged that leverage single-modal pre-trained models (such as ResNet [16] , ViT [11] , and BERT [8] ), or VLP models like CLIP, as backbones to incorporate powerful external knowledge. Han et al. [15] first introduced a CLIP model for text-to-image person retrieval. Later, CFine [49] builds upon VLP and introduces a token selection module to directly choose informative key local embeddings. However, this selection process lacks interpretability. IRRA [21] learns relations between local visual-textual tokens and enhances global image-text matching. However, it overlooked the fact that CLIP heavily relies on instance-level representations, resulting in limited capability to represent local features. Our model introduces a novel task of reconstructing key local segment features in images (text) through multimodal interaction, which greatly enhances their discriminative information. RSITR: RSITR refers to recalling required RS images with text. Initial studies [1, 34] prioritized global image-text representations, where multimodal information was encoded and merged to formulate a shared semantic representation. Subsequent research [1, 34, 35, 54] introduced additional information to enhance modality representations. Yuan et al. [54] employed a shared modality transmission module to facilitate communication across modalities. Cheng et al. [7] devised a semantic alignment module to effectively identify latent correspondences between images and text. Yuan et al. [53] developed an asymmetric multimodal feature matching network and employed multi-scale feature information. Later, [30, 31, 55] found that the knowledge CLIP can be transferred to the remote sensing domain, but there is a lack of in-depth research on remote sensing image-text characteristics. In this study, we employed CLIP as the backbone and introduced the KLSF module to extract fused discriminative local features, mitigating the issue of high similarity of remote sensing image-text data. 
method: 
 In this section, we will introduce our proposed Explicit Key Local information Selection and Reconstruction (EKLSR) framework. An overview of EKLSR is shown in Figure 2 (a) . It consists of a dual-stream feature extraction backbone, Key Local information Selection and Fusion (KLSF), and Key Local segment Reconstruction (KLR) based on multimodal interaction. We leverage CLIP [39] as the initialization of the backbone. KLSF leverages hidden knowledge of CLIP to select interpretably and fuse key local features to form the final feature representations of the image and text. KLR masks key segments of images and text, and it predicts these segments using the contextual information from unmasked image (text) and the global information from the paired text (image). KLR does not participate in the inference process. The details of EKLSR will be discussed in the following subsections.Inspired by the success of transferring knowledge from CLIP [39] to text-image retrieval [21] , we directly initialize our EKLSR framework with the CLIP pre-trained model weight.Image Encoder. Given an input image 𝐼 ∈ 𝑅 𝐻 ×𝑊 ×𝐶 , we employ the CLIP pre-trained Vision Transformer model to obtain both global and local token embeddings of the image. Firstly, the image I is split into a sequence of 𝑁 = 𝑊 × 𝐻 /𝑃 2 fixed-size patches, where P represents the patch size. The patch sequence is then mapped to a token sequence {𝑣 𝑖 } 𝑁 𝑖=1 through a trainable linear projection. By injecting position embeddings and an additional [𝐶𝐿𝑆] token, the token sequence 𝑉 = {𝑣 𝑐𝑙𝑠 , 𝑣 1 , . . . , 𝑣 𝑁 } is inputted to the Transformer, enabling multi-head self-attention to obtain 𝑓 𝑣 𝑐𝑙𝑠 ,All sentences from the RSITMD test set the importance factor of each local feature, as follows:EQUATION)EQUATION)Where 𝑓 𝑣 𝑐𝑙𝑠 and 𝑓 𝑡 𝑐𝑙𝑠 denote instance-level features of images and text, respectively, while 𝑓 𝑣 𝑖 and 𝑓 𝑡 𝑖 represent their local features. 𝑠 𝑣 𝑖 (𝑠 𝑡 𝑖 ) stands for the importance factor of i-th local feature in images (text). We visualized the distribution of image and text importance factors on the test set of the RSTMD dataset, as shown in Fig. 3 . The visualization indicates that the segments of high importance factors correlate with the main subject regions in images and meaningful words in the text, confirming that these factors can adaptively gauge the importance of local features. Furthermore, we interpretably select key local features based on their importance factors. We employ a polynomial probability selection method that selects features based on their importance factors. Specifically, the likelihood of being selected increases proportionally with the magnitude of their importance factor. This method can select not only the majority of key local features but also a few non-key local features that are beneficial for feature representation, outperforming random and top-k selection methods. Through polynomial probability selection, key local features are selectively extracted.EQUATIONwhere 𝑆 𝑣 = {𝑠 𝑖 } 𝑁 𝑖=0 ,Features. Key local features selected capture local details but lack associations among these details. Therefore, we fuse key local tokens with instance-level global features that represent relationships among local details to foster complementarity between their respective feature information. As shown in Figure 2 (b), for images, we apply Softmax to the importance factors to serve as weight coefficients for the key local features. We multiply these coefficients by the key local features and accumulate the results to obtain the key fused representation. To prevent the loss of global contextual features, we integrate it with the key fused representation using a residual connection, resulting in the joint feature representation 𝑓 𝑣 key . The procedure is shown below:EQUATIONThe joint feature 𝑓 𝑣 key may still contain redundant feature representations or even misleading information [54] . Naturally, we perform a secondary feature transformation on the 𝑓 𝑣 key to suppress irrelevant information. We map the joint feature to a high-dimensional hidden space, randomly deactivating some neurons to filter out redundant and irrelevant information. Subsequently, we linearly map the result to the common space. These steps are formulated as follows:𝑓 𝑣 keyh = Dropout GELU Linear 𝑓 𝑣 key (6) f 𝑣 key = Linear 𝑓 𝑣 keyh (7) The global representation 𝑓 𝑣 𝑔 extracted from CLIP contains valuable hidden knowledge of the global relationship, which is crucial in the final feature representation. The fusion is performed as follows:EQUATIONwhere 𝐹 𝑣 is the final image feature. A similar process is performed for the text modality to obtain the final text feature 𝐹 𝑡 . Finally, we use contrastive loss for optimization. The loss function can be expressed as 𝐿 𝑐𝑒 = -where 𝐹 𝑣 𝑗 is the final feature representation of the j-th image in the current batch data. This is a bidirectional loss function, where the first half calculates the loss for the image-to-text retrieval task, and the second half represents the loss calculation for the text-to-image retrieval task. This loss function aims to bring paired image-text features closer together in a common space while pushing unrelated pairs further apart.CLIP has not been optimized for local feature representation, resulting in a lack of discriminability for local features. This does not facilitate the mutual complementation of local feature information.We propose a KLR task to augment the discriminability of local features. Unlike previous methods [18, 22, 25, 37, 44] that randomly reconstruct segments, KLR ensures the reconstruction of key local segments and enhance the interaction between multimodal local features. The KLR is only utilized during model training and is not employed during inference.Interaction (Text KLR). We employ an approach similar to MLM of BERT [8] to reconstruct the key segments. However, there are two points of difference as shown in the upper part of Figure 2 (c). Firstly, to ensure the masking of key local tokens, we select segments with higher importance factors in a proportion of 𝛾. Then, we apply masking to the selected segments. SpecificallyEQUATIONWhere 𝑇 = 𝑡 sos , 𝑡 1 , 𝑡 mask 2 , 𝑡 mask 3 . . . , 𝑡 𝑛 , 𝑡 eos denotes the text encoding after masking. In this case, the number of masked words is 𝛾 * 𝑛. Subsequently, 𝑇 is input to the text Transformer encoder:EQUATIONwhere𝐹 𝑡 = f 𝑡 sos , f 𝑡 1 , f tmask 2 , f tmask 3 , . . . , f 𝑡 𝑛 , f 𝑡 eos is masked text features.Secondly, to enhance the interaction between image and text modalities, we predict masked text segments with unmasked text segment features and paired image instance-level features. The prediction process can be expressed as:EQUATION)EQUATIONWhere 𝑦 𝑖,𝑗 denotes the label at the j-th mask position of the i-th text. ŷ𝑡 𝑖,𝑗 denotes the predicted result.To reconstruct the key local segment features in the image, we adopt a similar approach to Text KLR. As shown in the lower part of Figure 2 (c) . EQUATIONEQUATIONWe tokenize image segments{𝑣 𝑖 } 𝑁 𝑖=1 using the image tokenizer of BEIT [3] , obtaining labels 𝑌 𝑣 = 𝑦 𝑣 1 , 𝑦 𝑣 2 , . . . , 𝑦 𝑣 𝑁 * 𝛼 for all masked patches. Then, we predict the labels of masked image segments with the unmasked image segment features and paired text global features.ŷ𝑣EQUATION)EQUATIONFinally, the total loss of our EKLSR framework is defined as:EQUATION)where L 𝑐𝑒 denotes the image-text matching loss. L 𝑡𝑘𝑙𝑟 and L 𝑖𝑘𝑙𝑟 respectively denote the reconstruction loss of key segments for the text and image. 
experimental 4.1 datasets: 
 In this paper, we experimented with two Remote Sensing Image-Text Retrieval (RSITR) datasets and three Text-Image Person Reidentification (TIReID) datasets.We conducted experiments on two remote sensing image-text datasets, RSICD [33] , RSITMD [53] . The RSICD dataset consists of 10,921 RS images in 30 semantic categories, each of which has 5 captions. The RSITMD dataset consists of 4743 images in 32 semantic categories, with a total of 23,715 captions. 4.2.1 Evaluation Metrics. The experimental metrics are R@K and mR, where R@K (K=1, 5, and 10) is defined as the similarity ranking of matching pairs included in the top K retrieval results, and a higher value of R@K indicates better performance. mR represents the average of all R@K, which is more reasonable for evaluating the overall performance of the model.For KLSF, the probability of selecting key features in the image, 𝛼, is set to 0.3 for RSITR and 0.55 for TIReID. The probability of selecting key features in the text, 𝛽, is set to 0.3 for RSITR and 0.70 for TIReID. For KLR, the probability of masking the image, 𝜏, is set to 0.5 for RSITR and 0.2 for TIReID, and the probability of masking the text, 𝛾, is set to 0.5 for RSITR and 0.2 for TIReID. Both 𝜆 and 𝜂 are set to 0.1. We employ a cosine annealing learning rate strategy with a warm-up period of 2000 steps. The learning rate is set to 1e-05, the batch size is 128, and the total number of epochs is 10 for RSITR and 50 for TIReID. The initialization of our framework is based on CLIP-ViT-B/16. The experiments are conducted using the PyTorch on an NVIDIA RTX 3090 GPU for RSITR task and an A100 GPU for TIReID task.We conducted an extensive evaluation of the EKLSR performance across two specific domains: RSITR (RSITMD and RSICD) and TIReID (CUHK-PEDE, ICFG-PEDES, and RSTPReid). We benchmarked our EKLSR against the best comparison model and the CLIP (baseline), as depicted in Figure 4 , clearly demonstrating our method's superior performance. Detailed analyses for each dataset are presented below.In this section, we compare our approach with state-of-the-art methods on two RSITR benchmark datasets, RSICD and RSITMD, as shown in Table 1 . The "Type" column classifications "F," "R," "V," and "C" correspond to image backbones based on bottom-up attention models, ResNet, VIT, and CLIP, respectively. Text Retrieval refers to image-to-text retrieval. Image Retrieval refers to text-to-image retrieval.Performance Comparisons on RSITMD. As indicated in Table 1, EKLSR outperforms all methods across all recall rates R@K, achieving mR accuracy of 51.59%, which is 3.03% higher than the baseline and 3.45% higher than the current best method, TGKT [30] . Notably, our directly fine-tuned CLIP baseline already surpasses the advanced TGKT [30] method with an mR accuracy of 48.56%. It is evident from the "Type" column in Table 1 that the robust feature extraction backbones in RSITR are key, with VLP-based methods becoming increasingly dominant. This underscores the significance of our research in optimizing VLP models for specific domain tasks.Performance Comparisons on RSICD. As shown in Table 1 , the baseline exceeds the most recent state-of-the-art results by +2.11% in mR accuracy. Furthermore, our proposed EKLSR surpasses all methods across all R@K accuracy, significantly outperforming the latest advanced method, TGKT, by +3.73% in mR accuracy.EKLSR consistently delivers state-of-the-art performance across all metrics on two RSITR datasets. This underscores the efficacy of EKLSR in effectively leveraging VLP model knowledge for RSITR.. In this section, we evaluate the generalization capability of our proposed EKLSR model on TIReID by conducting experiments across three public TIReID benchmark datasets (CUHK-PEDES, ICFG-PEDES, and RSTPReid).Performance Comparisons on CUHK-PEDES. We evaluated our EKLSR model on the widely-used CUHK-PEDES dataset, with performance comparisons shown in Table 2 . EKLSR achieved anImage Retrieval Text Retrieval Image Retrieval Type R@1 R@5 R@10 R@1 R@5 R@10 mR R@1 R@5 R@10 R@1 R@5 R@10 mR SCAN [23] Performance Comparisons on ICFG-PEDES and RSTPReid. We compared our EKLSR against previous works on two additional benchmarks, RSTPReid and ICFG-PEDES, as illustrated in Tables 3 and Table 4 . EKLSR consistently outperforms the baselines on both datasets. Specifically, on the RSTPReeid dataset, EKLSR significantly surpasses the same CLIP-based method, CFine, achieving 55.30% (+4.95) and 73.81% (+5.60%) in Rank-1 and mR accuracy, respectively. Similarly, on the ICFG-PEDES, it significantly outperforms the baseline and achieves performance comparable to the CFine. These results demonstrate the robustness and generalizability of EKLSR.To fully demonstrate the impact of different components in EKLSR, we use the CLIP-ViT-B/16 models as the baseline and conduct experiments on the RSITMD (RSITR task) and CUHK-PEDES (TIReID task) datasets. Refer to Table 5 for the experimental results. The "ours (w/o interaction)" denotes that the global information from paired modalities is not used during masked segment reconstruction.(1) By comparing No.0 and No.1, we demonstrate the effectiveness of the KLSF module, showing an approximately 1% improvement in mR accuracy on the RSITMD and CUHK-PEDES datasets. This indicates that the KLSF module can extract key local information from the powerful multimodal knowledge representation of CLIP to enhance the final feature representation.(2) Comparing No.2 and No.1 on the RSITMD dataset shows text retrieval gains with R@1, R@5, and R@10 increasing by 0.44%, 1.77%, and 1.11%, respectively. However, image retrieval improvements are negligible across datasets. This can be attributed to Text KLR's focus on text reconstruction, only improving the discriminability of text local features and benefiting unidirectional text Methods KLSF Text KLR Image KLR R@1 R@5 R@10 R@1 R@5 R@10 mR R@1 R@5 R@10 mR 0 baseline Text Retrieval Image Retrieval Image Retrieval No. Module random topk our R@1 R@5 R@10 R@1 R@5 R@10 mR R@1 R@5 R@10 mR 0 ✓ 25. 22 provements in all R@K accuracy for image retrieval on the RSITMD dataset and in R@1 for image retrieval on the CUHK-PEDES dataset.Text KLR and Image KLR are more focused on improving the unidirectional retrieval performance, and when used together, they improve the bidirectional retrieval performance. Comparing No.2 and No.1, significant increases in all R@K accuracy were observed on both RSITMD and CUHK-PEDES datasets, with the mR for bidirectional retrieval rising by 3.03% and 0.96%, respectively. The above experiments demonstrate the effectiveness of KLR.(3) By comparing No.5 with No.4, we observe that the reconstruction with crossmodal information consistently outperforms the reconstruction with only intra-modal information, with an approximate 1% improvement in the mR on the RSITMD dataset. This indicates that crossmodal interaction benefits reconstruction tasks.The above experiments demonstrate the effectiveness of each component in our EKLSR framework.Different Key Local Selection and Mask Strategies. We investigate the selection method for key local features in the KLSF module and the masking methods in the KLR task in Table 6 . The random represents a random selection (masking) method. The topk indicates selecting (masking) only the top k local features (segments) based on their importance factor ranking. The our represents the polynomial probability selection method, the characteristic of which is that segment features with higher importance factors have a greater probability of being selected, while also allowing segment features with lower importance factors to be selected.Comparisons between No.2 and No.1 with No.0 indicate methods focusing on key local feature selection (topk and our) significantly outperform random selection on the mR metric. Similar trends are observed in comparisons between No.5 and No.4 with No.3, with topk and our methods outperforming random mask. These highlight the pivotal role of key local features in specific domain tasks. Additionally, the our method performs better than the topk method in KLSF and KLR because it may also select (mask) image background regions and text words with low-importance factors. Although these segments have low importance factor, they are still essential for the overall understanding of the image and text. 
conclusion: 
 To adapt the VLP model like CLIP to specific domain image-text retrieval tasks such as Text-Image Re-identification (TIReID) and Remote Sensing Image Text Retrieval (RSITR), we introduce the Explicit Key Local information Selection and Reconstruction (EKLSR) framework tailored for high similarity characteristics in specific domain data. Our Key Local information Selection and Fusion (KLSF) module leverages interpretable importance factors from CLIP's prior knowledge to identify key local features. These key token features, selected based on importance factors, are dynamically fused with instance-level global features to enhance feature representation in a shared space. Additionally, our approach employs Key Local segment Reconstruction based on multimodal interaction (KLR) to reconstruct key local segments of images (and text) using intra-modal contextual information and global information from matched modalities. This not only enriches the discriminative information of key local features but also intensifies the interaction between multimodal local features. Furthermore, our framework facilitates offline inference, catering to the real-time demands of specific domain applications. Ultimately, our EKLSR model attains substantial performance on two benchmark RSITR datasets and three TIReID datasets. In future work, we plan to conduct experiments on a broader range of VLP models.  
 
 Appendix:  


Decision: There is a consensus that this paper is borderline, but three out of four are in favor of acceptance.
Two of the reviewers found that the authors' rebuttal provided relevant and satisfactory responses to the reviewers' comments in their rebuttal. One didn't discuss the rebuttal
There is one reviewer that is concerned about the explanation of Important Factor that is still not clear, I suggest to focus on this for the camera ready.
I decided to accept the rebuttal even if the style has been slightly modified, as mentioned by one reviewer.

Reviews: review: paper_summary: paper_summary: The paper proposes a novel framework named Explicit Key Local information Selection and Reconstruction (EKLSR) for Specific Domain Image-Text Retrieval (SDITR), particularly focusing on Text-Image Person Re-identification (TIReID) and Remote Sensing Image-Text Retrieval (RSITR). The framework enhances the discriminability of local features through two main components: Key Local information Selection and Fusion (KLSF) and Key Local segment Reconstruction (KLR). The authors demonstrate the effectiveness of their approach with experiments conducted on five datasets, achieving state-of-the-art performance on two RSITR datasets.
strength_weakness: strength_weakness: Novel Approach: The EKLSR framework addresses specific domain challenges by explicitly selecting and reconstructing key local features, which is a well-motivated and innovative solution.
Comprehensive Evaluation: The paper provides extensive experimental results on multiple datasets, showing significant improvements over existing methods.
Technical Depth: The integration of multimodal interaction-based reconstruction tasks (MLM and MVM) in fine-tuning is a novel application that enhances the discriminability of local features.
Clarity: The paper is well-written and clearly explains the proposed method, its components, and the underlying motivation.
limitations: limitations: Complexity: The proposed framework adds significant complexity to the baseline VLP models, which may impact the scalability and practical applicability in real-time systems.
Ablation Studies: While the ablation studies are thorough, additional insights into the interaction between different components could be beneficial. For instance, more detailed analysis on the impact of the polynomial probability selection method compared to simpler methods would add value.
Generalization: The paper primarily focuses on two specific domains (TIReID and RSITR). While the results are promising, the generalizability of the approach to other specific domains or broader applications remains uncertain.
paper_summary: paper_summary: Focusing on specific-domain image-text retrieval problem, this paper investigates on two main problems: 1) The interaction among local features leads to noise and uncertainty; 2) VLP models like CLIP are not specialized for local features, which perform unsatisfactorily. Therefore, “important factor” is proposed to measure the importance of local features, and a novel framework named Explicit Key Local Information Selection and Reconstruction is developed. In details, KLSF (Key Local information Selection and Fusion) selects key local features based on their importance factor, to avoid noise and uncertainty. KLR (Key Local segment Reconstruction) is proposed with Masked Image and Language Modeling with importance-factor-based masking for better local features.
strength_weakness: strength_weakness: This paper proposes a novel concept “importance factor”, and conducts extensive experiments on 5 datasets on Text-Image Person ReID and Remote Sensing Image-Text Retrieval.
limitations: limitations: A. The interpretability of “importance factor” is relatively low, where the visualization cannot support such too strong assumption. 

According to Eq. 1 and 2, the closer to the global feature, the more important the local features are. 

However, on one hand, if the background occupies the whole image like Figure 3, it is possible that there are some background information in the global features. As shown in the top-left and bottom-left picture in Figure 3, many no-object background parts are also highlighted with high “importance factor”. 

On the other hand, if the foreground is the main part of image, like persons in Text-Image Person ReID task, it is also possible that some human parts are not comprised in global features. But no visualization of “importance factor” is provided for Text-Image Person ReID task like Figure 3. Similar uninterpretable highlighted parts can be also found in Figure 6 of supplementary materials.

Please provide more explanation about how such strong assumption is proven, as well as the visualization of “importance factor” for Text-Image Person ReID task. 

B. The performance gain by the proposed method is marginal, which is conflict with the motivation.

In Figure 4 and Table 2-4, only about 3% performance gain are achieve in main-stream Text-Image Person ReID benckmarks. Moreover, each component merely obtain 1% performance gain in ablation study in Table 5 and 6 in Text-Image Person ReID task. 

In Table 1, the proposed method brings 5% more improvements on RSITMD dataset for Remote Sensing Image-Text Retrieval, but still only 1% on RSICD dataset, compared with its baseline. 

Given such marginal performance gain, it is confusing that whether the proposed method really solves the problem investigated in Figure 1. When placing the gained performance from Figure 4 into Figure 1(a), the results are still imbalanced with general domain Flickr30K. 

Please explain such a conflict between the marginal performance gains and the motivation of balanced domain performances. 

C. Some writing problems

1) In Row 2 and 3 of Table 5, KLSF is misspelled as “KTSF”. 

2) In Eq. 12, masked features and global features are used to predict masked features. It should be non-masked features and global features for masked features.
paper_summary: paper_summary: The primary challenge of applying vision-language pre-training (VLP) models, such as CLIP, to Specific Domain Image-Text Retrieval (SDITR) lies in effectively leveraging discriminative fine-grained local information. Current approaches interact with all multimodal local features for alignment, which may introduce noise and uncertainty. To mitigate these issues, this paper proposes an Explicit Key Local information Selection and Reconstruction framework (EKLSR), which explicitly selects key local information to enhance feature representation. EKLSR introduces a Key Local information Selection and Fusion (KLSF) module and a Key Local segment Reconstruction (KLR) module based on multimodal interaction. Experiments conducted on five datasets across TIReID and RSITR validate the effectiveness of EKLSR. Notably, the EKLSR model achieves state-of-the-art performance on two RSITR datasets.
strength_weakness: strength_weakness: + This paper proposes an Explicit Key Local information Selection and Reconstruction framework (EKLSR), which identifies key local features through Key Local information Selection and Fusion (KLSF) and Key Local segment Reconstruction (KLR), enriching the discriminative information of key local features.
+ The Key Local information Selection and Fusion part is simple and effective. Unlike MLM, KLR reconstructs key local information in images and text, which is also intriguing.
+ EKLSR achieves good performance on two specific domain image-text retrieval tasks.
+ The main text and supplementary materials present comprehensive experimental results.
+ The paper is written clearly and understandably.
limitations: limitations: + In Section 4.2.2, how are the various hyperparameters ($\beta $, ...) determined?
+ The textual content and mathematical symbols in the figures must be consistent. For example, in Figure 2, is the CLS feature of the text right? What is the relationship between CLS and SOS/EOS? 
+ Are all the parts that should be bolded in the table correctly bolded? 
+ In the supplementary materials, the numbering of tables and figures should ideally follow the corresponding content in the main text.
paper_summary: paper_summary: The paper presents an Explicit Key Local information Selection and Reconstruction Framework (EKLSR) to efficiently apply Vision-Language Pre-training (VLP) models in Specific Domain Image-Text Retrieval (SDITR) pursuits. The EKLSR overcomes issues of noise and uncertainty in current methods by selectively using key local information for feature representation. Demonstrated through experiments on five datasets, the EKLSR model outperforms existing models in Text-Image Person Re-identification and Remote Sensing Image-Text Retrieval scenarios.
strength_weakness: strength_weakness: 1. The method proposed in this paper has been validated on RS and reID retrieval datasets, achieving good results. Additionally, ablation experiments have been conducted, proving that the method is effective.

2. The method is very intuitive.
limitations: limitations: 1. The inference setting is not introduced in the paper. When performing retrieval on test datasets, does the author need to pre-process the entire dataset to get all importance factors for each token? And when selecting key local features, do you pick a fixed number based on importance factors (without randomness)?

2. The authors could explain more about why local features with lower similarity to the instance-level feature are more important. And the author could provide concrete examples to better illustrate what words are important and what are not.

3. The authors could conduct experiment to compare the proposed method to the baseline, i.e. CLIP, on the general domain. The proposed method seems not only works on specific domain, so it should be beneficial to the general domain.

4. In terms of writing, in 3.2 and 3.3, the authors would better decouple the intuitive and concise introductions from equations and formulas. And an error is found in figure 3: the color palette is with wrong color.